System and method for contextual synthetic media generation

Aspects of the subject disclosure may include, for example, obtaining, at a first location, media content capturing an event occurring at a capture location. The media content is analyzed proximal to the first location to obtain an analysis result, a context of the event is determined, and a rule is applied according to the context to obtain a rule application. The media content is modified according to the rule application to obtain a modified media content adapted for distribution to a media consumer at a second location, while preventing distribution of the media content for consumption by the media consumer. Other embodiments are disclosed.

FIELD OF THE DISCLOSURE

The subject disclosure relates to a system and method for contextual synthetic media generation.

BACKGROUND

Synthetic media generally refers to the artificial production, manipulation, and modification of data and media by automated means, especially through the use of artificial intelligence algorithms. Such synthetic content is often referred to as a “deepfake”—a term derived from the combination of “deep learning” and “fake” meaning “not true.” For example, a face of one subject in an image may be swapped with a face of another, sometimes referred to as “FaceSwap.”

Deepfake adopts techniques from machine learning and artificial intelligence to manipulate and/or generate visual and/or audio content with a high capability/probability to deceive. To this end, machine learning techniques may be used to create deepfakes based on deep learning that involves training generative neural network architectures. Such neural networks may be leveraged to generate synthetic content in which a subject, e.g., a person, in an existing image or video is replaced with another's likeness. For example, face swapping technology may use auto-encoder and decoder pairs in which an autoencoder extracts features of a facial image and a decoder reconstructs the facial image.

DETAILED DESCRIPTION

The subject disclosure describes, among other things, illustrative embodiments for synthetic media generation according to contextual information. Other embodiments are described in the subject disclosure.

One or more aspects of the subject disclosure include a process that includes recording, by a processing system including a processor at a first location, original media content of a live event, analyzing, by the processing system, the original media content proximal to the first location to obtain an analysis result, and identifying, by the processing system, a context of the live event. The process further includes applying, by the processing system, a rule proximal to the first location according to the context to obtain a rule application, modifying, by the processing system, the original media content proximal to the first location according to the rule application to obtain modified media content adapted for distribution to media consumers, and preventing, by the processing system, distribution of the original media content for consumption by the media consumers.

One or more aspects of the subject disclosure include a media processing system, which includes a processing system including a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations. The operations include generating, at a source location, media content capturing an event occurring at the source location, analyzing the media content proximal to the source location to obtain an analysis result and identifying a context of the event. The operations further include applying a rule proximal to the source location according to the context to obtain a rule application, modifying the media content proximal to the source location according to the rule application to obtain modified media content adapted for distribution to media consumers, and preventing distribution of the media content to the media consumers.

One or more aspects of the subject disclosure include a non-transitory, machine-readable medium, that includes executable instructions that, when executed by a processing system including a processor, facilitate performance of operations. The operations include obtaining, at a first location, media content capturing an event occurring at a capture location, analyzing the media content proximal to the first location to obtain an analysis result, and determining a context of the event. The operations further include applying a rule according to the context to obtain a rule application, wherein the media content is modified according to the rule application to obtain a modified media content adapted for distribution to a media consumer at a second location, and preventing distribution of the media content for consumption by the media consumer.

It is appreciated that media content generation, such as user-based content upload, may accidentally capture someone and/or something who should not be seen, e.g., according to their personal preference, a user preference, and/or any laws and/or regulations, such as child privacy laws, security, e.g., corporate security, and so on. Consider crowd-based scenarios with an expectation of privacy, but camera captures crowd, e.g., at a celebrity party. Other examples include wartime reporting that inadvertently capture someone/something, e.g., disclosing personal identity that may endanger the person. On-the-spot reporting, e.g., timely news, is often accomplished by lightweight reporting teams with limited editing and/or production capacity that would make it unfeasible to have immediate and/or local access a full production facility.

Some of the example solutions to such problems disclosed herein include applications of contextual synthetic media generation, including image modification that utilizes convincing image replacement segments, e.g., “deepfakes,” implemented as close as possible to the media source, without necessarily requiring corresponding compute power at the source. To this end, the processing may be accomplished at media capture equipment, and/or as nearby as practical, e.g., providing resources that are no more than “one hop” up the network path, e.g., at a point-of-entry cell tower, or further up the network, e.g., at an edge node, but preferably before content is edited and/or published, to protect identities.

The convincing image replacement approaches, when applied to the foregoing situations, yield positive results, intended to appear convincing to a content consumer in that the results look real and are not detectable as replacements, in order to preserve rights of privacy of persons, places and/or things that may be captured in the media content. Examples of convincing image replacement segments include face swapping that may be used to make one face look like another. Alternatively or in addition, the convincing image replacement segments may include aspects of puppeteering by which a replacement face is adapted to conform to actions, e.g., head movement, expressions and/or speech, e.g., lip-syncing of the face being replaced.

Referring now toFIG.1, a block diagram is shown illustrating an example, non-limiting embodiment of a system100in accordance with various aspects described herein. For example, system100can facilitate in whole or in part obtaining, at a first location, media content that captures an occurrence of an event, analyzing the media content proximal to the first location to obtain an analysis result, determining a context of the event, applying a rule according to the context to obtain a rule application, modifying the media content according to the rule application to obtain a modified media content adapted for consumption by a media consumer at a second location, and preventing distribution of the media content for consumption by the media consumer. In particular, a communications network125is presented for providing broadband access110to a plurality of data terminals114via access terminal112, wireless access120to a plurality of mobile devices124and vehicle126via base station or access point122, voice access130to a plurality of telephony devices134, via switching device132and/or media access140to a plurality of audio/video display devices144via media terminal142. In addition, communication network125is coupled to one or more content sources175of audio, video, graphics, text and/or other media. While broadband access110, wireless access120, voice access130and media access140are shown separately, one or more of these forms of access can be combined to provide multiple access services to a single client device (e.g., mobile devices124can receive media content via media terminal142, data terminal114can be provided voice access via switching device132, and so on).

In at least some embodiments, one or more of the data terminals114and/or the access terminal112at a first location may include a module, application and/or executable instructions182a,182b, adapted for obtaining media content, and/or analyzing the media content, and/or determining a context of an event portrayed in the media content, and/or applying a rule according to the context, and/or modifying the media content according to the rule, and/or preventing distribution of the media content for consumption by a media consumer at a second location remote from the first location. Likewise, in at least some embodiments, one or more of the display devices144via media terminal142at a first location may include a module, application and/or executable instructions184a,184b, adapted for obtaining media content, and/or analyzing the media content, and/or determining a context of an event portrayed in the media content, and/or applying a rule according to the context, and/or modifying the media content according to the rule, and/or preventing distribution of the media content for consumption by a media consumer at a second location remote from the first location. Similarly, in at least some embodiments, one or more of the mobile devices124,126and/or access points122at a first location may include a module, application and/or executable instructions186a,186b, adapted for obtaining media content, and/or analyzing the media content, and/or determining a context of an event portrayed in the media content, and/or applying a rule according to the context, and/or modifying the media content according to the rule, and/or preventing distribution of the media content for consumption by a media consumer at a second location remote from the first location.

According to the illustrative example, the system100may include a media processor180in communication with one or more of the data terminals114, the access terminal112, the display devices144, the media terminal142, the mobile devices124,126and/or the access points122via the communications network125. The media processor180may be adapted to perform one or more of the processing steps disclosed herein in relation to processing of media content to determine a context of the media content, applying a rule according to the context, modifying the media content according to the application of the rule, and preventing a distribution of the unmodified media content. To this end, the media processor180may perform at least a portion of the processing steps that are distributed among another media processor, which may include one or more of the network elements150,152,154,156, and/or one or more of the data terminals114, the access terminal112, the display devices144, the media terminal142, the mobile devices124,126and/or the access points122via the communications network125.

FIG.2Ais a schematic diagram illustrating an example, non-limiting embodiment of interactions occurring within a contextual synthetic media generation system200functioning within the communication network ofFIG.1in accordance with various aspects described herein. The contextual synthetic media generation system200includes a device201, an edge analyzer203, an orchestrator204and one or more rule sets205. The example device201is in communication with the edge analyzer203, which is in further communication with the orchestrator204. The orchestrator204, in turn, is in further communication with the one or more rule sets205. In at least some embodiments, the one or more rule sets205may be stored according to a rule file or script repository. Storage of the one or more rule sets205may be accomplished locally, e.g., proximal to one or more of the device201, the edge analyzer203and/or the orchestrator204. Alternatively or in addition, at least a portion of the one or more rule sets205may be stored remotely and accessible via network communications, e.g., as an accessible rule repository or database.

The device201may be adapted to perform one or more of receiving, capturing, retrieving from storage, storing, transmitting, and/or displaying media content. The media content may include, without limitation, one or more of digital images, video, computer generated imagery (CGI), and/or audio. Alternatively or in addition, the device201may include a media recording device, such as a digital camera, a video camera, an audio recorder, a digital video recorder (DVR), and the like. Alternatively or in addition, the device201may include a media player adapted to access pre-recorded media content for presentation via a display device and/or an audio player as one or more of still images, video images or audio. In at least some embodiments, the device201may include a game console and/or a game controller as may be manipulated by the user202during gameplay. In at least some embodiments, the device201may include a smart phone, a smart television, a personal computer, a tablet device, and so on.

The edge analyzer203may be adapted to analyze one or more of the digital images, video, computer generated imagery (CGI), and/or audio received and/or otherwise accessed by the device201. The edge analyzer203may be employed wholly or at least partially proximal to the device201. In at least some embodiments, the edge analyzer203may be incorporated into a media capture device, such as a camera, a video camera, a smart phone, a tablet, and the like. For example, the edge analyzer203may include an application program or app that may be stored and/or otherwise executed locally at and/or near the device201. Alternatively or in addition, the edge analyzer203may be provided as a service, e.g., having local and/or back-end functionality that may be accessible via a network connection.

According to at least some configurations, the edge analyzer202may be implemented at, near or within a device in network communication with one or more of the device201, the orchestrator204and the one or more rule sets205. For example, the edge analyzer203may be implemented at, near or within a processing device, such as a server, as may be accessible by the device201using a wired and/or wireless local area network. Alternatively or in addition, the edge analyzer203may be implemented at least partially within a wireless access point and/or a business or residential gateway. In at least some embodiments, the edge analyzer203may be implemented at, near or within local equipment servicing a cellular communication terminal, e.g., a radio access network and/or a base transceiver station. Alternatively or in addition, at least a portion of the edge analyzer203may be located at, near or within an edge node of a service provider network. According to any of the illustrative examples, an analysis performed by the edge analyzer203of media content obtained by the device201may be performed proximal to the device201, e.g., before network transport of the media content to a destination, such as a media editing and/or production facility, a media storage facility, and/or before broadcast and/or dissemination of the media content to any consumers of the media content. It is understood that in at least some applications, it is preferable to apply any modifications and/or adaptation of media content as close as practical to the source to prevent and/or otherwise limit the possibility of unauthorized access to any unmodified media content.

In operation, the edge analyzer203may employ one or more analysis techniques to interpret at least a portion of the media content. For example, an image, such as a video frame, may be analyzed to identify one or more subjects of the media content discernable according to presentation of the media content. Subjects may include objects, e.g., persons, places, things, may be discernable within a presentation of an image according to image analysis tools, e.g., object and/or facial recognition. Alternatively or in addition, objects may be discernable according within an audio presentation, e.g., according to a sound characteristic and/or voice signature or profile.

In at least some embodiments, the analysis techniques may parse an image of the media content into segments that may be representative of one or more objects of a scene portrayed within the image. In at least some embodiments, the analysis may categorize the one or more objects, e.g., identifying them as people, faces, animals, inanimate objects, such as clothing, logos, brands, monikers. Analysis techniques may further distinguish a foreground portion of an image from a background portion of the image, e.g., associating one or more of the objects with the foreground or the background, as interpreted. In at least some embodiments in which the objects include people, the analysis techniques may further distinguish and/or otherwise estimate a person's gender, approximate age, race, ethnicity, nationality, employment, group affiliation, and so on. Still further, analysis techniques may identify a person's position, posture, and/or action, and in at least some instances interpret the person's mood and/or intent, e.g., according to body language.

Without limitation, analysis may include encoding at least a portion of the media content, e.g., at least a portion of an image, a video frame, three-dimensional (3D) images and/or video, such as those represented according to voxels, and/or audio, to obtain an encoding. The encoding may be analyzed further utilizing aspects of artificial intelligence, e.g., machine learning.

In at least some embodiments, the orchestrator204may be adapted to facilitate adaptation and/or modification of the media content according to at least one rule of the one or more rule sets205. Adaptation may include, without limitation, a substitution, e.g., replacement, of at least a portion of the media content with a substitute portion, such as replacement of an entire person, a subject's face, hair, and/or an article(s) of clothing of a subject portrayed within an image and/or a video portion of the media content. Any such substitution or replacement may be performed in a convincing manner, such that the modified media content would be indistinguishable to any subsequent consumer of the modified media content from the original media content, but for the substitution.

In at least some embodiments, the orchestrator204adopts techniques from machine learning and/or artificial intelligence to manipulate and/or generate visual and/or audio content with a high capability and/or probability to be convincing or otherwise not detectable as a replacement. For example, the orchestrator204may include one or more auto-encoder and/or decoder pairs. The autoencoder is adapted to extract features of an image, such as dormant features of facial image, whereas the decoder is adapted to reconstruct the image, e.g., the facial image.

Examples of such adaptations that include convincing image replacement segments are sometimes referred to as “deepfakes” in which artificial media is used to replace a person's face obtained from an image or a video with a face of another person. Accordingly, a convincing image replacement segments is a method that digitally swaps a face of one subject in an image with another, sometimes referred to as “Face Swap.” For example, a face of an adult female may be replaced with a distinguishable face of another adult female. The other adult female may be a recognizable figure, e.g., a person of some notoriety, such as a celebrity, a politician, or a historical figure. Alternatively or in addition, the distinguishable face may be a machine generated face, e.g., constructed according to an algorithm, adapted to provide certain realistic features that are distinguishable from the original face, such as complexion, eye color, shape, proportion, hair color, style, and the like. It is understood that in at least some instances the original and replacement faces may differ in one or more of gender, race, ethnicity, age, and the like, such that a face of a white male adult, may be replaced with the face of an adult female of color.

In some embodiments, the orchestrator204generates a replacement portion of the media content based on one or more original portions of the media content. For example, a replacement face for a face swap of one subject within a scene of an image and/or video may be generated according to one or more features of a face or faces of other individuals also portrayed within the same scene. Thus, the orchestrator204may generate the replacement face to adapt an otherwise distinguishable subject into a crowd or setting. Likewise, the orchestrator204may generate a replacement face to further distinguish an otherwise indistinguishable subject within the scene.

In at least some embodiments, the orchestrator204may obtain analysis results of the media content from the edge analyzer203. Analysis results may include a description and/or assessment of one or more portions of the media content, such as one or more portions of the image. The assessment may include an indication of a background portion of an image and/or video, e.g., an assessment of another subject or subjects, e.g., a crowed, within which a subject was captured within the image and/or video. The assessment may include a description and/or characterization of the other subject(s), e.g., providing estimates of one or more of genders, races, ethnicities, ages, and the like. Alternatively or in addition, the assessment may include encoded information as may have been obtained using AI techniques, e.g., a deep neural network, directed towards the other subject, crowd and/or background.

In addition to replacing a face of one person with another, the adaptation, e.g., face swapping, techniques disclosed herein may also be adapted to manipulate other features, such as a subject's position, posture, facial expression, and so on. It is further understood that one or more of the various adaptation techniques may be applied to audio, e.g., convincing audio replacement segments, permitting post-production changes to dialogue in a newscast, a movie and/or video without a need for reshoot.

Continuing with an illustrative example, in which the device201comprises a video recording device, such as a video camera, and the user202represents a person operating the device201. According to first arrow211, the user202may inform the device201of one or more elements of context and/or one or more specific identities of interest. In at least some embodiments, this may occur during a preliminary or pre-recording phase, e.g., according to a configuration or setup procedure. The user202may further configure the device201for video capture according to second arrow212. Continuing with the configuration procedure, the orchestrator204, in turn, may receive one or more rule sets205according to arrow213, e.g., by way of a rule set update obtained from a rule set(s) repository, such as the example preference database229.

The device201having been informed of a context and/or identity of interest may distribute an indication of the same to the orchestrator204according to arrow214. The orchestrator204may store one or more of the context and/or identity of interest, which it may provide to the edge analyzer203according to arrow217for analysis, e.g., to determine whether a subject of interest is present within a scene of the captured video content corresponds to the identity of interest.

Having completed the configuration procedure, the orchestrator204may inform the device201that it may begin recording, e.g., according to arrow218. The device201may record the video content providing at least a portion of the same to the edge analyzer203for analysis. According to the illustrative example, a subject may enter and/or otherwise be present in a scene of the video content, e.g., according to arrow215. The orchestrator204and/or the edge analyzer203, alone or in combination, may determine, e.g., from an analysis obtained via the edge analyzer203according to the information provided by the orchestrator204, whether the subject who entered the scene corresponds to the identity of interest.

Responsive to an identification of the subject of interest206within a scene of the recorded video content, one or more rules of the one or more rule sets205may be received and/or applied according to arrows224, e.g., by the orchestrator204, which, in turn, may determine at arrow216, whether the subject206is to be recorded and/or modified, e.g., according to the rule set update received according to arrow213. In this regard, the orchestrator204may communicate with the edge analyzer203, e.g., to determine from an analysis obtained by the edge analyzer203whether the subject of interest206is present within the scene and/or whether a modification regarding the subject of interest should be applied to the recorded video content before further distribution of the video for editing, publication and/or distribution to end consumers.

In at least some embodiments, the one or more rule sets205may identify the subject206and/or one or more contexts according to which the subject206may be identified. Alternatively or in addition, the one or more rule sets205may identify the subject206and/or one or more contexts according to which the subject206may be modified and/or otherwise obfuscated and/or adapted. To the extent the orchestrator204and/or the edge analyzer203determines that adaptation is required, the orchestrator determines an adaptation request, e.g., an obfuscate person request. The adaptation and/or obfuscation, e.g., according to arrow226, may be determined by the orchestrator204according to or more adaptation parameters identified according to the one or more rule sets205. In at least some embodiments, adaptation parameters may be obtained according to arrows225, from one or more preferences as may be prescribed, e.g., retrievable from a preferences file or preference database229. Alternatively or in addition, adaptation parameters may be obtained at least in part according to the one or more rule sets205, e.g., in which at least some of the adaptation parameters may depend upon a predetermined rule and/or policy alone or in combination with one or more of the contexts associated with the recorded video content.

According to arrow227, the orchestrator204may identify and/or request an adaptation, e.g., an obfuscation, of the subject of interest206identified within a scene of the recorded video content. An adapted version of the recorded video content is obtained according to arrow228, e.g., according to one or more of the orchestrator204, the edge analyzer203and/or the device201. The adapted video content may be disseminated from the device201, as if it were the originally captured video content, but for the introduced adaptation. Accordingly a recipient of the adapted video content obtained directly from the device201may not detect that the video content has been adapted, e.g., according to an obfuscation and/or face swap of the subject of interest206. Alternatively or in addition, the adapted video content may be disseminated from one or more of the edge analyzer203and/or the orchestrator204, without necessarily requiring that it be provided to the device201. However, in at least some instances in which the adapted content may be disseminated from one or more of the edge analyzer203and/or the orchestrator204, the adapted video content may also be provided to the device201, e.g., to provide feedback to the user202as to a confirmation that the adaptation has occurred and/or an assurance as to a quality of the adaptation.

It is envisioned that in at least some embodiments, the user202may adjust an adaptation responsive to an initial introduction of the adaptation. Such adjustments may be introduced in a manual and/or automated feedback manner to provide adjustments to reduce errors and/or otherwise improve an extent of an adaptation, a quality of the adaptation, and/or a feature of the adaptation.

In at least some embodiments, e.g., after recording has begun according to arrow281, the edge analyzer203may determine a scene change for analysis at arrow219. The scene change may be identified according to abrupt discernable differences between adjacent video frames or frame segments, according to metadata and/or according to input from the device201, e.g., a user stops recording, then begins again at a later time. To the extent it is determined at arrow221that a screen change may have occurred, the orchestrator204may query the edge analyzer203regarding an existing and/or otherwise know entity, e.g., any subjects of interest that may have been tracked in the preceding scene and/or to recheck the new scene according to a comprehensive list of subjects of interest. In at least some embodiments, the edge analyzer203and/or the orchestrator204may consult the one or more rule sets205according to arrow223. Likewise any subjects of interest identified within the new scene may be adapted and/or modified according to one or more rules and/or preferences.

FIG.2Bis a block diagram illustrating an example, non-limiting embodiment of a contextual synthetic media generation system230functioning within the communication network100ofFIG.1in accordance with various aspects described herein. In at least some embodiments, the contextual synthetic media generation system230may operate according to the example interactions occurring within the contextual synthetic media generation system200ofFIG.2A. The example contextual synthetic media generation system230includes a content source231, a content processor250, a content distribution network233and one or more content consumers232. The content source231may include any of the example sources provided herein, such as an example digital camera, video recorder, audio recorder, and/or DVR. The example content processor250is in communication between the content source231and an edge node234of the content distribution network233, and is adapted to process media content, e.g., video, obtained from the content source231to obtain processed media content. The processed media content is provided to the one or more content consumers232via the content distribution network233in place of the originally received or captured media content. Accordingly, the processed media content is obtained from the content processor250before being distributed to the one or more content consumers232, and in at least some embodiments, before transport of the processed media content from the edge node234to other communication nodes of the network233. In at least some embodiments, one or more of the content source231, the content processor250and/or the edge node234are adapted to prevent recording, distribution and/or presentation of the unmodified media content. It is envisioned that content consumers may include end users, e.g., receiving the processed media content according to a subscription, channel selection, order and/or request.

By way of example media content may include video content235as may be obtained from a video recorder, a streaming media source and/or a DVR, e.g., adapted to play previously recorded video data. The video content235generally includes a sequence of video frames, each video frame236including one or more objects of a scene captured by the video content235and observable within a presentation of the video content235. The scene of the example video frame236includes a first person237located in a central, foreground region of the frame236, a group of persons240located in a background portion along a right edge of the video frame236, and an object242located towards an upper left portion of the video frame236. The first person237can be further defined according to a head and/or facial portion239, a body portion238, and optionally, a badge, moniker and/or other distinguishable tag243.

In more detail, the content processor250includes an analyzer251, an orchestrator252, and a rules engine253. The analyzer is in communication with the content source231, receiving at least a portion of the video content235therefrom. The analyzer251is adapted to analyze one or more aspects of the video content235to obtain analysis results. The analysis may include one or more of segmenting the video frame236, distinguishing and/or otherwise identifying one or more objects within the video frame236, identifying one or more features of the video frame236, segments of the video frame236and/or objects within the frame. The features may include, without limitation, one or more of sizes, shapes, orientations, colors, shadows, arrangements, textures, motion or lack thereof, and so on.

In at least some embodiments, the rules engine253is in communication with one or more of the analyzer251and the orchestrator252. The rules engine may include one or more predetermined rules and/or policies adapted to control and/or otherwise manage one or more of the analysis and/or adaptation of the video content235as disclosed herein, or otherwise generally known. By way of example, rules enforced by the rules engine253may identify one or more subjects of interest, e.g., according to individual identities, ages, occupations, roles, affiliations, and so on. Alternatively or in addition, rules enforced by the rules engine253may identify one or more adaptation parameters. The adaptation parameters may be implemented by one or more of the analyzer251and the orchestrator252to adapt at least a portion of the video content235, e.g., at least a portion of one or more of the video frames236, according to enforcement of the rules.

By way of further example, the analyzer may parse a video frame236into a foreground portion, e.g., including the first person237, and a background portion, e.g., including the group of persons240. A first rule may identify that any persons of minor age should be obfuscated and/or otherwise masked. The analyzer may separately analyze the first person237and the group of persons240according to the minor age requirement. To the extent any one of minor age is identified, a second rule may identify how to obfuscate such persons, e.g., face swap if the minor person is in the foreground and/or blur if the minor person(s) are in the background portion of the video frame236.

The analyzer251, the orchestrator252or a combination of both may implement the rules to obtain adapted, or modified video content235′, including modified video frames236′, each portraying corresponding scenic elements to the unmodified frames236, but for application of any rules that may adapt one or more portions of the video frame236′. According to the illustrative example, a head or facial portion239of the first person237of the original frame236has been replaced, e.g., according to a face swap, or convincing image replacement segments, adaptation providing an adapted head or facial portion244of the first person237′ of the modified video frame236′. Other portions of the modified video frame236′, e.g., the object242′, the clothing238′ and/or any tag243′ may remain substantially unchanged from the original video frame236.

According to the illustrative embodiment, the content processor250may include a storage element254in communication with one or more of the rules engine253, the orchestrator252and/or the analyzer251. The storage element254may be adapted to store one or more data elements related to processing of the video content235, such as one or more rules, preferences, algorithms, temporary storage during generate of replacement content, and so on. Alternatively or in addition, the contextual synthetic media generation system230may include a user interface255. The user interface255is shown as part of the content processor250and is in communication with one or more of the analyzer251, the rules engine253, the storage element254and/or the orchestrator252. Alternatively or in addition, the user interface255may be provided at least partially within the content source, or as a separate device, e.g., a portal that may be accessible via the network233.

In operation, the user interface255may be adapted to accept user input related to one or more of rules, preferences, managing authorization and/or access, e.g., to the content processor250and the like. In at least some embodiments, the user interface255may accept user input related to identification of any subjects of interest. User inputs may include preferences of any adaptations, e.g., whether blurring is applied, versus filtering and/or swapping of image portions. In at least some embodiments, user input may include a measure of reality to be applied during any adaptation. The example user interface includes a reality dial257, which a user may adjust to obtain a desired level of reality with any adaptations, e.g., relatively low level of reality that may be easily identified by content consumers232, versus a relatively high level of reality that may be indiscernible by the content consumers232. It is understood that a system and/or process complexity may vary according to a selected reality level, e.g., with a greater level of processing power, storage, and/or other resources, such as AI resources being required to achieve a greater reality.

FIG.2Cdepicts an illustrative embodiment of a process270for contextual deep fake generation in accordance with various aspects described herein. According to the process270, media content is received at271. Without limitation, the media content may be received using one or more media processing devices, such as a camera, a video camera, an audio recorder, a smart phone, a tablet device, a personal computer, a smart television, a DVR, a personal digital assistant, and/or a smart speaker system, such as the Alexa® or Echo® smart speaker devices. The media content may include, without limitation, any of the various examples disclosed herein, such as digital images, video and audio. In at least some embodiments, the media content may be received from a media recording device, such as a camera, a video camera, a microphone and so on. Alternatively or in addition, the media content may be received from a playback of a media recording, e.g., from a DVR or audio recorder. Without limitation, the media content may also be received from a media broadcast, e.g., according to a live broadcast and/or a preprogrammed channel lineup, from an on-demand source, such as Netflix® or Hulu® on-demand services, from other streaming sources, such as YouTube® streaming media service.

Analyze media content according to subject(s) at272. In at least some embodiments, the media content may be analyzed to identify one or more subjects of interest as may be present with them media content. A subject of interest may include one or more of persons, places and/or things. Any subjects of interest may be determined before or after analyzing the media content at272. For example, a user may identify a subject of interest during a configuration of a media processing system adapted to perform at least a portion of the illustrative processing270. Alternatively or in addition, the subject of interest may be determined according to a profile, and/or a configuration file, and/or by one or more rules to be applied to one or more of the processing steps of the illustrative process270.

Identification of the subject of interest may apply generally understood image parsing, image interpretation, and/or image recognition algorithms. At least some of the algorithms may employ aspects of machine learning or AI. In at least some embodiments, identification of the subject of interest may apply a facial recognition algorithm, a voice recognition algorithm and/or a combination of both. It is understood that reference to a subject of interest may include specific subjects, such as a specific object or brand, e.g., a Coke® cola drink, a specific place, e.g., the U.S. Capital building, and or a specific person, e.g., President Biden. Alternatively or in addition, reference to a subject of interest may be more general. For example, a subject of interest may be a type of object, such as a soda can, a school building, a man, a child, a police officer, and so on.

According to the illustrative process270, a context may be identified at273. In at least some embodiments, contexts may refer to a subject matter of the media content, e.g., persons, places, things observable within a presentation of the media content. Alternatively or in addition, the context may relate to a manner in which the media content was captured, stored and/or otherwise accessed. For example, a context may indicate that the media content relates to a live event or a prerecorded event, or that the media content is unedited or edited, and so on. In at least some embodiments, contextual information may relate to one or more of a location or setting, an event, an individual person and/or group of persons of a subject matter of the media content and/or associated with generation, e.g., recording, of the media content.

Contextual information may be obtained automatically, e.g., as reported by equipment used to record and/or otherwise generate the media content. Example of such information may include geolocation reported by a GPS receiver of a smart phone, a temperature, barometric pressure, weather conditions, biometric data of individuals, e.g., heart rate, blood pressure, body temperature. Alternatively or in addition, contextual information may be reported by a user. The contextual information may be provided via a user interface. Alternatively or in addition, the contextual information may be provided by metadata associated with the media content. Users may include, without limitation, a user of the media processing device performing one or more steps of the illustrative process270, a user involved with a recording, editing and/or production of the media content, and/or a content provider.

At least one rule ay be applied at274. In at least some embodiments, the rule may be applied towards identification of a subject of interest. Alternatively or in addition, the rule may be applied towards adaptation of the media content. For example, the rule may relate to when the media content should be adapted and/or a manner in which the media content may be adapted. The rule may be based on at least a portion of the contextual information. For example, the rule may be configured to protect a news reporter when reporting from within a dangerous location. Accordingly, the rule may identify any news anchor of a particular news station or group of news stations, and/or a particular news correspondent or group of correspondents as subject matters of interest. The rule may identify further that any identified subjects of interest within the media content should be obfuscated, or otherwise adapted according to a convincing image replacement segment, only when present within a particular region, e.g., a war zone, or a hostile territory.

A determination is made at275, as to whether the media content should be modified. Such a determination may be based upon detection of a subject of interest within the media content. Alternatively or in addition, such a determination may be based further upon a rule that sets a condition as to whether the media content should be modified. It is understood that at least some rules may apply to a nature of media modification, not necessarily setting any conditions as to when and/or whether the modification should be applied. To the extent it is determined at275that the content should not be modified, the process270proceeds to receive media content at271. However, to the extent it is determined that the content should be modified, the media content is modified at276.

Modification may include, without limitation, occlusion and/or blurring of at least a portion of an image, e.g., blurring a face, or a brand name. Alternatively or in addition, modification may include replacement of at least a portion of an image with a replacement image portion. For example, a face of a subject of interest may be replaced with a face of another. The replacement face may be of another individual and/or computer generated, e.g., determined according to an algorithm. In at least some embodiments, the process may include the user of machine learning and/or AI as disclosed herein. Similar techniques may be applied to audio portions of media content, e.g., modifying audio of a subject of interest with audio of another individual and/or according to an algorithm and/or AI technique. Advantageously, the replacement portions of an image, video and/or audio may be adapted to mask and/or otherwise prevent disclosure of the subject of interest.

In at least some embodiments, the modification may be obtained using one or more filters or algorithms. Such filters may be applied to at least a portion of the media content to an image, video and/or audio to obtain the modified media content item. Filters may be used to change one or more of a size, a resolution, a color and/or a tone of an image, a video and/or an audio of the media content. Other filters may be adapted to change an appearance of an individual, e.g., according to gender, race, ethnicity, height, weight, affiliation, occupation, and so on. Some filters may be adapted to change an appearance of a subject of interest according to an arbitrary or fanciful trait, e.g., replacing a person with a celebrity or an animated character. The modifications may be prescribed, e.g., according to the rules, to modify the media content in a directed manner. Alternatively or in addition, at least a portion of the modifications may be applied in a random manner, e.g., changing a trait, such as a skin tone, hair color, gender according to a randomized selection.

In at least some embodiments, the modified media content may be provided at277for storage and or consumption. In some applications, it may be advantageous to prevent storage, distribution and/or presentation of the original, unmodified media content. In this regard, steps may be implemented to ensure that the media modification occurs as close as possible to a source of the media content. For example, a media recorder adapted to generate or capture media content may be further adapted to modify the media content according to at least some of the various techniques disclosed herein. Alternatively or in addition, at least a portion of the modification of the media content may be accomplished by one or more other processing devices within a relatively close proximity to the media recorder, e.g., within a wireless range of a personal area network and/or a wired and/or wireless local area network. In at least some embodiments, modification of the media content may be accomplished within an edge node of a wide area network. Preferably the edge node is proximal, in a network sense, to the media recorder, to limit exposure of any unmodified media content to a minimum area, physical range and/or network range.

FIG.2Ddepicts an illustrative embodiment of another process280for contextual deep fake generation in accordance with various aspects described herein. According to the illustrative process, media content is accessed at281, e.g., according to any of the various media access techniques disclosed herein. A context, such as any of the various examples disclosed herein is identified at282and a subject of interest is identified at273. The media content is analyzed according to the subject of interest at284, and a determination is made at285as to whether the subject of interest is present within the media content. To the extent it is determined that the subject of interest is not present, the process280proceeds to access media content at281. However, to the extent it is determined that the subject of interest is present, a rule is identified at286.

A determination is made at287as to whether an alteration of the media content is required according to the rule and, to the extent it is determined that an alteration is not required, the process280proceeds to access media content at281. However, to the extent it is determined at287that an alteration is required, a detail of the alteration is determined at288.

A further determination is made at289as to whether an alteration of the media content is available according to the detail. To the extent it is determined that an alteration according to the detail not available, an alteration is generated at290according to the detail. However, to the extent it is determined that the alteration according to the detail is available, the process accesses the alteration at291. For example, if frames of a video segment are processed individually, the same subject may appear within at least a contiguous group of frames, or frames of a common scene of the media content, such that any adaptation may be accomplished in a similar manner, without necessarily having to determine the alterations and/or generate the alterations, instead utilizing alterations that may have already been determined and/or otherwise prepared during a processing of a preceding frame or frames. In either instance, the process280continues to modify the media content according to the alteration at292, providing the modified media content for storage and or consumption at293.

Referring now toFIG.3, a block diagram300is shown illustrating an example, non-limiting embodiment of a virtualized communication network in accordance with various aspects described herein. In particular a virtualized communication network is presented that can be used to implement some or all of the subsystems and functions of system100, the subsystems and functions of systems100,200,230,300and processes270,280presented inFIGS.1,2A,2B,2C,2D and3. For example, virtualized communication network300can facilitate in whole or in part obtaining, at a first location, media content that captures an occurrence of an event, analyzing the media content proximal to the first location to obtain an analysis result, determining a context of the event, applying a rule according to the context to obtain a rule application, modifying the media content according to the rule application to obtain a modified media content adapted for consumption by a media consumer at a second location, and preventing distribution of the media content for consumption by the media consumer.

According to the illustrative example, the system300may include one or more virtual media processors380,381in communication with one or more of the broadband access110, the media access140, the wireless access120, and/or the content sources175. The virtual media processors380,381may be adapted to perform one or more of the processing steps disclosed herein in relation to processing of media content to determine a context of the media content, applying a rule according to the context, modifying the media content according to the application of the rule, and preventing a distribution of the unmodified media content. To this end, the virtual media processors380,381may perform at least a portion of the processing steps that are distributed among another media processor, which may include one or more of the VNEs330,332,334, and/or one or more of the broadband access110, the media access140, the wireless access120, and/or the content sources175, that may be adapted to include one or more media processing components382,386,384,390adapted to work independently and or in cooperation with one another and/or with one or both of the virtual media processors380,381to perform one or more of the processing steps disclosed herein in relation to processing of media content to determine a context of the media content, applying a rule according to the context, modifying the media content according to the application of the rule, and preventing a distribution of the unmodified media content.

Turning now toFIG.4, there is illustrated a block diagram of a computing environment in accordance with various aspects described herein. In order to provide additional context for various embodiments of the embodiments described herein,FIG.4and the following discussion are intended to provide a brief, general description of a suitable computing environment400in which the various embodiments of the subject disclosure can be implemented. In particular, computing environment400can be used in the implementation of network elements150,152,154,156, access terminal112, base station or access point122, switching device132, media terminal142, and/or VNEs330,332,334, etc. Each of these devices can be implemented via computer-executable instructions that can run on one or more computers, and/or in combination with other program modules and/or as a combination of hardware and software. For example, computing environment400can facilitate in whole or in part obtaining, at a first location, media content that captures an occurrence of an event, analyzing the media content proximal to the first location to obtain an analysis result, determining a context of the event, applying a rule according to the context to obtain a rule application, modifying the media content according to the rule application to obtain a modified media content adapted for consumption by a media consumer at a second location, and preventing distribution of the media content for consumption by the media consumer.

Turning now toFIG.5, an embodiment500of a mobile network platform510is shown that is an example of network elements150,152,154,156, and/or VNEs330,332,334, etc. For example, platform510can facilitate in whole or in part obtaining, at a first location, media content that captures an occurrence of an event, analyzing the media content proximal to the first location to obtain an analysis result, determining a context of the event, applying a rule according to the context to obtain a rule application, modifying the media content according to the rule application to obtain a modified media content adapted for consumption by a media consumer at a second location, and preventing distribution of the media content for consumption by the media consumer. In one or more embodiments, the mobile network platform510can generate and receive signals transmitted and received by base stations or access points such as base station or access point122. Generally, mobile network platform510can comprise components, e.g., nodes, gateways, interfaces, servers, or disparate platforms, which facilitate both packet-switched (PS) (e.g., internet protocol (IP), frame relay, asynchronous transfer mode (ATM)) and circuit-switched (CS) traffic (e.g., voice and data), as well as control generation for networked wireless telecommunication. As a non-limiting example, mobile network platform510can be included in telecommunications carrier networks, and can be considered carrier-side components as discussed elsewhere herein. Mobile network platform510comprises CS gateway node(s)512which can interface CS traffic received from legacy networks like telephony network(s)540(e.g., public switched telephone network (PSTN), or public land mobile network (PLMN)) or a signaling system #7 (SS7) network560. CS gateway node(s)512can authorize and authenticate traffic (e.g., voice) arising from such networks. Additionally, CS gateway node(s)512can access mobility, or roaming, data generated through SS7 network560; for instance, mobility data stored in a visited location register (VLR), which can reside in memory530. Moreover, CS gateway node(s)512interfaces CS-based traffic and signaling and PS gateway node(s)518. As an example, in a 3GPP UMTS network, CS gateway node(s)512can be realized at least in part in gateway GPRS support node(s) (GGSN). It should be appreciated that functionality and specific operation of CS gateway node(s)512, PS gateway node(s)518, and serving node(s)516, is provided and dictated by radio technology(ies) utilized by mobile network platform510for telecommunication over a radio access network520with other devices, such as a radiotelephone575.

In at least some embodiments, the mobile device576may include a module, application and/or executable instructions590, adapted for obtaining media content, and/or analyzing the media content, and/or determining a context of an event portrayed in the media content, and/or applying a rule according to the context, and/or modifying the media content according to the rule, and/or preventing distribution of the media content for consumption by a media consumer at a second location remote from the first location. Alternatively or in addition, the mobile network platform510may include a module, application and/or executable instructions592, adapted for obtaining media content, and/or analyzing the media content, and/or determining a context of an event portrayed in the media content, and/or applying a rule according to the context, and/or modifying the media content according to the rule, and/or preventing distribution of the media content for consumption by a media consumer at a second location remote from the first location. It is envisioned that the example modules590,592may work independent and/or cooperatively, e.g., distributing processing according to one or more of processing capacity, storage capacity, power consumption, battery power reserve, network conditions, and the like.

Turning now toFIG.6, an illustrative embodiment of a communication device600is shown. The communication device600can serve as an illustrative embodiment of devices such as data terminals114, mobile devices124, vehicle126, display devices144or other client devices for communication via either communications network125. For example, computing device600can facilitate in whole or in part obtaining, at a first location, media content that captures an occurrence of an event, analyzing the media content proximal to the first location to obtain an analysis result, determining a context of the event, applying a rule according to the context to obtain a rule application, modifying the media content according to the rule application to obtain a modified media content adapted for consumption by a media consumer at a second location, and preventing distribution of the media content for consumption by the media consumer.

The media analysis and adaptation techniques disclosed herein may be applied at production time, e.g., for marketing/ad material (fast turnaround for stuntman or marketing video). For example, a generic advertising and/or marketing scene may be recorded, but regionalize and/or modified for particular intended purposes. Scenes may use body double for actual shots and replace individuals with other identities as desired. Consider a recorded video scene of kayakers, in which images of persons captured in the originally recorded scene may be replaced with different personas, possibly even younger versions of those captured in the recording. Other adaptations may include gender and/or ethnicity, while preserving features such as body positions, actions and/or general appearance. In at least some embodiments, a predetermined list of entities to be adapted, e.g., a “block list” and/or an “opt-in” list, as the case may be.

Identities may be contextually discovered, e.g., by geolocation, by assignment and/or role. For example a media adaptation system may infer a dangerous and/or sensitive place, e.g., wartime, military facility, government installation, corporate environment, laboratory, elementary school. Having determined the media was obtained from a sensitive location, other rules may be applied to determine what to adapt, modify or block and/or what may be permitted, in which instance the adaptation, modification and/or blocking would be applied to those aspects of the media content not permitted. A scenario may include one/more as persons who should not be blocked, e.g., according to the rules, in which instance the adaptation system would automatically block anyone else who may have been captured within the media content. Others may be selectively blocked, e.g., based on gender, ethnicity, age, uniform, etc. In at least some embodiments, analysis of captured media content may be adapted to detect a token, e.g., a news moniker as may be worn by a reporter, or attached to a microphone held by a reporter. Such monikers, when detected may as in indication of who/what should or should not be adapted, modified and/or otherwise blocked.

It is envisioned that the media adaptation techniques disclosed herein may be used to preserve privacy and/or to offer protection to persons, places and/or entities who may be captured in media content. Alternatively or in addition, the techniques may be applied in a context of moderation, e.g., to detect violence, or scary content that could be blocked according to the rules, perhaps applied according to a graded application, e.g., according to an age of an intended media consumer, with more being blocked according to younger age groups. Still other applications include brand safety, in which an advertiser and/or sports team may not want their spokesperson seen drinking alcohol. Accordingly, the adaptations may be applied to anonymize such individuals according to detected activities, e.g., according to predetermined rules.

In at least some applications, the adaptation techniques may be applied according to a timing rule that may depend upon a time of day, a date, a duration, an age and so on. By way of example, an identity captured within a media content item may be associated with a time-to-live, such as an expiring identity, in which instance, the identity may be adapted and/or not adapted according to the timing rule. Alternatively or in addition, the timing rule may apply to the adaptation, e.g., replacing a face of a president recorded in a previous administration with a face of a president of the current administration. Similar to time sensitive rules, other rules may relate to events. For example in a sporting game, an identity of a real athlete portrayed in a game may be adapted based on a trade, a retirement, and so on, in order to maintain a freshness of the game relevance. Consider a baseball scenario in which a team, e.g., the Yankees recorded generated in one era, but certain individuals may be replaced with caricatures, and/or more current and/or relevant players of a later/earlier era—e.g., substitute Babe Ruth or Lou Gehrig for Aaron Judd. Still other timing and/or event-based rules may be used in relation to advertising in general or to movies, e.g., replacing brands and/or an actor(s) to keep movies/tv series relevant/fresh.

In at least some embodiments, the techniques disclosed herein may be applied to media content that includes, without limitation, at least portions of one or more of an image, a video frame, 3D images and/or 3D video, including media content represented by voxels, and/or audio. Images and/or video content may include extended range content, such as those used in panoramic applications and/or, so-called, 360-video applications. The media content subject to and/or produced by the various techniques disclosed herein may be obtained from and/or otherwise adapted for use in coordination with one or more of virtual reality (VR) applications, extended reality (XR) applications, augmented reality (AR) applications, diminished reality (DR) applications, gaming applications, cinematic presentations, media recording applications, e.g., DVR, broadcast applications, video-on-demand applications, streaming applications, live broadcast and/or live streaming applications, and so on.

In at least some embodiments, media capture and/or media processing devices may include wearable device VR/AR that may be applied in a collaborative environment, e.g., a meeting, in which an individual may be wearing an AR headset, but does not want to appear as such to other participants/audience members. Some social media platforms use an Avatar Codec, in which an inward looking camera observes a user's mouth, eyes etc., to extract local features from VR headset. It is envisioned that similar techniques may be utilized according to the techniques disclosed herein, e.g., using an Avatar Codec, to encode faces adjacent to the recording camera, but not do the full processing, defer that to edge or elsewhere that has more computer power.

In at least some embodiments, application of a convincing image replacement segment may apply preferences and/or rules, e.g., to have an individual blend in with a crowd. The analyzer may review a crowd to determine one or more generalized characteristics, e.g., gender, race, ethnicity, age, affiliations, then define changes to be applied to one or more subjects of interest based on same.

In at least some embodiments, a media adaptation system according to the techniques disclosed herein may be configured to permit modifications (a “dial”), by a participant, e.g., one being recorded, an observer, a third party, some combination thereof. Similarly, a person and/or AI may be applied in the loop, to provide and/or otherwise assist in any of the processing steps disclosed herein in relation to the disclosed adaptation techniques.

In at least some embodiments, a user purchase of a non-fungible token (NFT) may be associated with a view of a video and/or news item in which a purchaser purchases the original and/or an original modification, such that access is constrained and/or changed according to one NFT. For example, a rough cut of a video may be preserved as an NFT.

In at least some embodiments, the analysis may identify an individual and provide a query to the individual, e.g., via SMS, and/or social media, indicating that they've been tagged in a particular recording and requesting whether they prefer to remain in the recording or to be replaced, e.g., “faked.” The adaption rules may be adapted to initiate the query and to modify and/or otherwise configure any subsequent adaptation responsive to a response to the inquiry. A default rule may be established to provide that in the absence of a reply, the individual will be faked.

In at least some embodiments, media adaptation or convincing replacement of media segments, encompasses audio as well. For example, the adaptation rules may be configured to apply a voice changing to a voice of a recognized image and/or recognized voice, and/or metadata. The voice adaptation may apply a pitch shift to anonymize the individual. Alternatively or in addition, the adaptation may be configured to simulate a voice based on an image of a person, e.g., a synthesized voice and/or audio characteristic of a representative mature, white female may replace the actual voice of a mature white female to anonymize the voice alone or in addition to an anonymizing of the person or face.

In at least some embodiments, the techniques disclosed herein may be applied to adapt and/or otherwise replace a background scene of an image, a background of a video scene and/or background audio, e.g., crowd noise, environment noise, a soundtrack, and the like.

In one or more embodiments, any and all modifications may be performed according to applicable laws, rules, regulations, permissions, digital rights management (DRM) policies, authorizations, and so forth.

U.S. Pat. No. 10,810,324, entitled “Methods, Systems and Algorithms for Providing Anonymization”, and directed to anonymization that may be provided such as to implement, for example, privacy as a service, is incorporated herein in its entirety.