Patent Description:
Advertisement insertion has been in use for some time. That is, targeted advertising may be inserted between two-dimensional (2D) scenes in audio/video (a/v) content, such as a movie or television show. Another form of advertising delivery encompasses product placement in production, such as deliberate use of a particular brand of cereal in a breakfast 2D scene in a movie. A process called rotoscoping allows identification of the outline of objects in a scene to allow post-production insertion of content into a completed video scene.

Published US patent application <CIT> concerns the dynamic augmentation of images and videos. User input and image processing is used to both manually and automatically identify spots in images and videos where content can be convincingly inserted to form native advertisements. Ad servers can identify targeted ads for a particular viewer and automatically supply those ads for insertion in the identified spots when the viewer views the content, thus providing a low-impact targeted advertising experience.

The invention is a system, medium and method as defined in the appended claims.

The figures depict a preferred embodiment for purposes of illustration only. One skilled in the art may readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

Disclosed are embodiments for insertion of three-dimensional (3D) media elements within two-dimensional video. To prepare for insertion of a 3D media element, the disclosed embodiments generate a 3D representation of a scene within a 2D video stream. Once a 3D representation of the scene is available, empty, or void space within the scene are identified. The void spaces are then prioritized for possible insertion of a 3D media element. The prioritization of the void spaces may consider several factors, including factors that relate to both technical and business aspects. For example, void spaces having a relatively flat horizontal bottom may be prioritized over those other spaces that do not. The flat horizontal bottom, such as a void space above a table top, provides, in some embodiments, a platform upon which to insert a realistically appearing object. Other factors include the size of the void space, lighting within the scene, and a length of time the void space appears in the scene without being blocked and/or transgressed. A trackability of the void space may also be determined. Once the identified void spaces are prioritized, one or more of the highest priority spaces may be selected for insertion of a 3D media element. After a 3D media element has been inserted, the 3D representation of the scene is saved to a non-transient computer readable storage medium (e.g. hard disk), transmitted over a network, or displayed via a 2D output device.

The disclosed embodiments use of a 3D representation of 2D video provides advances when compared to systems that operate within an exclusively 2D environment. For example, by generating a 3D representation of a scene and inserting a 3D object, the 3D representation may be used when multiple perspectives of the scene are displayed. For example, many movies, TV shows, and other content shoot a scene via multiple camera, which captures a scene from multiple perspectives. Solutions that operate in exclusively 2D environments are not able to insert media elements so as to maintain realism when switching between multiple perspectives of a scene. While a 2D based insertion environment may provide for small changes in scene perspectives, for example, a <NUM> degree change in perspective, the disclosed embodiments facilitate multiple views of scene from virtually any angle offset. For example, a scene that shows a dialogue between two actors sitting at a table may be shot using two cameras, with each of the two cameras focused toward a respective one of the two actors. An inserted item sitting on the table between the two actors may be accurately represented from both camera perspectives, via use of the disclosed embodiments, which can display a 2D video image from each perspective using a common representation of the scene and a common representation of an inserted 3D media element. In contrast, an insertion environment relying exclusively on 2D video processing requires duplicative efforts to render the item from the multiple perspectives. This results in a more costly and less realistic solution when compared to the disclosed embodiments.

Camera tracking information may be utilized to add a 3D media element to a scene. In some embodiments, the tracking information is used to correctly align a 3D media element with geometry of a set represented by the scene as viewed from the camera. In some embodiments, camera tracking is done using knowledge of the geometry of the set or some component in it, for example, a table of a known size. In some cases, construction plans for the set are available and can be used to facilitate 3D media element placement.

In certain genres, media content is captured using fixed camera positions typically using surveillance style cameras which are mounted on the set and can rotate and zoom. These cameras are positioned around the set offering views from different angles.

In a manner similar to systems called volumetric video or spatial tracking of systems such as NCam (http://www. com/), output of those cameras can be used to construct an accurate CG model of the set. This method is different in that it requires no particular set up on the set, the cameras used are those used to shoot the program, and it can be done at any point after production is complete.

Such a system and method benefits both original and aftermarket production companies by allowing a "refresh" of original content to include or replace media elements for localization, target audience, or topical relevance. A by-product of the technology is a marketplace for the identification of available spaces in a production, the time on screen and prominence of the available space, as well as the drawing power of the actors/director. The marketplace can be used to match available spaces and to potential items that can be placed in order to determine a value for the space in view of the product. The system allows a "market of one" so that each viewing of a media object can be customized, either at the content delivery source or at the content viewing point (TV, smartphone, tablet, etc.).

Integrating a Computer Generated (CG) element (whether an inert or animated object) into a scene requires particular processing to avoid a result that does not look natural in the context of the original AV content. It may be desirable to enable parties outside of an AV production process to add elements to the AV content after it has been produced. The creation of a 3D media element includes generation of mesh or wire frame information, which is a computer representation of the structure of the element. In some embodiments, the mesh defines a number of polygons connected at the vertices. The mesh is overlaid with a texture, effectively a computer-generated skin that covers the mesh. The texture defines characteristics of the object the element represents. As one example, if the object were an orange, the texture would be shades of orange, have the familiar look of the irregular rind and the way an orange looks at the point where the stem was attached.

As discussed in more detail below, after a void space is identified in a video, a value of the space is determined. The 'value' of the space may be based on a type of object to be displayed in the space. For example, in the product placement arena, if a space suitable for the placement of a bottle is on the table at a family breakfast, the space has a first value with respect to a first product placement (e.g. orange juice) and a second value with respect to a second product placement (e.g. vodka).

A machine learning algorithm determines the level of match by taking attributes of the object, both of its physical properties and requirements such as proximity to an actor and assessing the match with attributes of available spaces. Acceptance or rejection of the matches by the operator drives the algorithm to refine its results in future analysis. Some attributes of the space can be measured. Examples may include a size of the void space, a distance from a camera capturing the scene, a distance from the center of the scene's frame, a duration that the space is in focus, proximity of the void space to each actor in the scene, including whether the void space is handled by an actor, lighting in the scene, a duration within the scene in which the space appears, objective attributes of the content itself, identities of actors that are in the scene, a cast of the content (e.g. movie, episode, or series), a physical location of the scene, a date and/or time of day (e.g. setting) of the scene. Additional attributes include, in some embodiments, a genre or sub-genre of the content (e.g. move, TV show), a 'mood' of the scene, what is the scene about, an intended audience of the scene, a summary of a story told by the content, a widr context of the scene, such as, for example, a relative position of the scene within the content or story.

The mesh and the texture are the first step. The second step is to render the object so that it blends/integrates into the scene. This requires that the element be lit to match the scene and that the relationship of the element to its surroundings, for example the reflection in a shiny surface, are correctly reproduced.

A 3D media element to be inserted into 2D video content includes a mesh definition and a texture definition, which describe properties of the 3D media element that do not necessarily vary across a variety of scenes in which that 3D media element is inserted. However, some attributes of the 3D media element may vary based on one or more properties or attributes of a scene or particular location within a scene in which the 3D media element is inserted. For example, to integrate a 3D element into a scene, a lighting characteristic of the scene is used to determine particular shading of the 3D media element.

Specific modifications to a 3D element to integrate the 3D element into a scene are stored in a template data structure in some embodiments. Access to the template data structure is necessary in some embodiments when including the 3D element in a scene. However, rights holders of the content seek to control access to the template data and also track usage of the 3D media element resulting from application of the template.

In some of the disclosed embodiments, a blockchain ledger is used to fulfill this function. Carried by the blockchain are both the contract (a smart contract) and the template data structure referenced above (e.g. also described below as template <NUM>). In some embodiments, the contract and data template are a payload of the blockchain. In some other embodiments, the blockchain stores a reference (e.g. URL) to the contract and/or data template.

In one embodiment of the blockchain, the ledger may contain a formula, and/or the template for rendering the 3D media element so as to match the scene. In other words, the data necessary to go from the general 3D media element (e.g. mesh and texture) to the specific (rendered video) is accessible via the ledger. In some embodiments, a separate blockchain entry may be used for each scene, meaning the sections of a scene from cut to cut. When a creator of the 3D media element model, the artist, has completed a rendering of the element the result is entered into the ledger. An entry in the ledger may then contain reference to the model, the template and the final rendered video. In some embodiments, the preferred implementation is a permissioned or private blockchain allowing the rights holder for the AV content to maintain control either directly or through a service.

A refinement may be the expression of that space in terms of regular shapes (cubes, spheres, etc.) and their suitability for placement, for example, whether the object provides a base with a horizontal surface or whether the entire volume is hanging in space.

Another refinement may be the use of parameters that limit the volume that must be analyzed. For example, the maximum depth that is of interest may be <NUM> meters and the analysis can exclude the region of the scene that is further than <NUM> meters from the camera.

A further analysis pass may determine whether the identified volumes of space are available across more than one shot. Video content is made up of a sequence of shots. Each shot is part of a scene. Each scene takes place on a set. A set is used for one or more scenes and appears in one or more episodes. A comprehensive analysis determines whether the identified space exists in more than one scene. However, such an analysis may work in concert with another form of analysis. A semantic analysis may determine how the volume may be used in the context of continuity and story. Continuity is the process in video production of ensuring visual consistency across multiple shots. For example, if there is a kettle on a table in a scene it needs to be in the same place in every shot in the scene. The story, in this context, is how the use of the volume fits in with the scene's setting and action. For example, using the volume to place a cereal box on a table fits in with the story of a scene if it is breakfast but does not fit if it is dinner time.

The semantic analysis may be assisted by image recognition of what is in the scene, speech recognition of the dialog, and the script which has dialog and staging instructions. Closed captions and audio descriptions for the visually impaired may also be used.

Other attributes may be assigned to each volume of space to express 'usefulness' of the volume. This may not be a single value, it may be a set of values that describe the commercial value of the volume of space for the placement of a CG object and the difficulty level for each process used to create the CG element that will be placed in the volume of space. For example, the commercial value may be greater for a volume close to the camera and less if the volume is out of focus. The difficulty level for rotoscoping would reflect the difficulty of rotoscoping the scene. Rotoscoping is the process of tracing the outline of an object in the scene. Rotoscoping hair.

<FIG> is an overview diagram showing a 2D scene. The scene shows two actors 102a-b. Actor 102b is holding a cereal box <NUM>. The scene <NUM> also shows a countertop <NUM> and a window <NUM>. The scene <NUM> provides an opportunity for product placement of the cereal box <NUM>. Traditional 2D methods of insertion would facilitate such an opportunity. Such an opportunity is typically facilitated via human artists that manually insert the product placement into the scene <NUM>. Because of the relatively high overhead associated with product placement, such a traditional placement is generally global and perpetual. The placement is sold once and remains indefinitely. A brand manager for the cereal also does not receive any visibility into who watched tor interacted with the scene.

<FIG> shows a second overview diagram demonstrating product placement opportunities enabled by the disclosed embodiments. <FIG> shows a scene <NUM> that could be processed by one or more of the disclosed embodiments. <FIG> shows that the cereal box <NUM> may be replaced with various other versions of the cereal box 106a-d, depending on, for example, a locality where a video including the scene is presented. The ability to perform dynamic placement via contextual insertion based on viewer, region, time of day, and viewing distance, among other parameters, creates new revenue opportunities for funding content distribution.

<FIG> is a data flow diagram illustrating software components and data flows that are implemented in one or more of the disclosed embodiments. The system <NUM> discussed below with respect to <FIG> provides for dynamic 3D content insertion into 2D video.

<FIG> shows three software components, a marketplace <NUM>, player <NUM>, and content processing engine <NUM>. <FIG> also illustrates three data stores, an insertion specifications data store 308a, void spaces data store 308b, and a content data store 308c. The content processing engine <NUM> reads content data from the content data store 308c. The content data store stores data defining multimedia content. For example, the content data store stores data defining one or more of movies, TV shows, or other multimedia content. The content processing engine306 processes this content to identify void spaces included in the content that may be candidates for insertion of an additional 3D media element. As discussed further below, the content processing engine <NUM> is configured to generate, from 2D content included in the content data store 308c, 3D media elements representing objects included in a scene represented by the 2D content. For example, if the 2D content includes a scene showing a table and chair, the content processing engine generates 3D media elements for each of the table and the chair. The content processing engine <NUM> then identifies one or more void spaces within the scene. Information defining these void spaces is then written by the content processing engine <NUM> to the void spaces data store 308b. Once the void spaces are identified, they are accessed by the content marketplace application <NUM>. The content marketplace application <NUM> provides an opportunity for content presentation opportunities within content (identified via the content data store 308c). Content can be inserted within the void spaces identified by the content processing engine. As content presentation opportunities are purchased via the marketplace application <NUM>, information defining these purchased opportunities is stored in the insertion specifications data store 308a.

The content player application <NUM> then reads the content from the content data store 308c and insertion specifications (e.g. <NUM> discussed below) from the insertion specifications data store 308a to dynamically insert media elements into the content from the content data store 308c. Different versions of content may be generated for delivery to different regions (state, country, region) or for a particular set of users within a region (for example, English dialog is replaced with Spanish dialog for a Spanish language station broadcasting in Los Angeles).

The content with the inserted media elements is then provided to content displays 310a-c. In some embodiments, content displays 310a-c are computing devices that include a display device, such as any of a mobile phone (e.g. smart phone), smart TV, laptop, or desktop computer. In some embodiments, the content player <NUM> executes on the display device itself. In other embodiments, the content player <NUM> executes on back-end server type hardware, or is implemented via a cloud-based topology.

In the disclosed embodiments, audio video content (AV) is stored on a back-end or cloud-based server system. The server may implement, for example, a Content Distribution Network (CDN) of an Over the Top (OTT) streaming service. In some aspects, the content distribution server is a playout center of a broadcast service such as a cable company or an Over the Air (OTA) station.

<FIG> shows example data structures implemented by one or more of the disclosed embodiments. While the data structures of <FIG> are discussed below as relational database tables, one of skill would understand that some of the disclosed embodiments utilize alternative data structure organizations without departing from the scope of the data discussed below. For example, various embodiments implement unstructured data stores and/or traditional in-memory structures such as linked lists, arrays, heaps, or other data organizational structures.

<FIG> shows a scene table <NUM>, media element table <NUM>, void space table <NUM>, content properties table <NUM>, content data table <NUM>, insertable element table <NUM>, and template table <NUM>. The scene table <NUM> includes a scene identifier field <NUM>, height field <NUM>, width field <NUM>, depth field <NUM>, content identifier field <NUM>, elapsed time field <NUM>, date/time field <NUM>, location field <NUM>, and actors field <NUM>. The scene identifier field <NUM> uniquely identifies a scene. The height field <NUM>, width field <NUM>, and depth field <NUM> define a height, width, and depth of the scene respectively. The content id field <NUM> identifies content that the scene (identified via <NUM>) is included in (e.g. content data table <NUM> discussed below). The elapsed time field <NUM> defines a length, in time units, of the scene. The date/time field <NUM> identifies a date/time of when the scene occurs. In some embodiments, the date/time reflects an actual real-world time. In some other embodiments, the date/time reflects a simulated time, which indicates a setting for the scene. The location field <NUM> defines a location of the scene, such as a geographical location (e.g. GPS coordinates). The actors field <NUM> defines identifies of one or more actors present in the scene.

The media elements table <NUM> includes a media element identifier field <NUM>, volume definition field <NUM>, position definition field <NUM>, and a scene identifier. The media element identifier field <NUM> uniquely identifies a single media element. A media element is a distinct three-dimensional object that appears in a scene. The volume definition field <NUM> defines a volume occupied by the element. The position definition field <NUM> identifies a position of the media element within the scene. The position may be provided with respect to a particular corner of a total 3D space of the scene. The scene identifier field <NUM> identifies a scene in which the media element appears. The scene identifier field <NUM> is cross referenceable with the scene identifier field <NUM> and/or scene identifier(s) <NUM>, discussed below.

The void space table <NUM> includes a void space identifier <NUM>, a scene identifier field <NUM>, a volume definition field <NUM>, position definition field <NUM>, distance from center field <NUM>, duration field <NUM>, and a priority field <NUM>. The void space identifier field <NUM> uniquely identifies a single void space. The scene identifier field <NUM> identifies a scene in which the void space is present (cross referenceable with the scene id fields <NUM>, <NUM>, and/or <NUM>, discussed below). The volume definition field <NUM> defines a volume inhabited or occupied by the void space (identified via <NUM>). In some embodiments, the volume definition field <NUM> defines a plurality of polygons that define the volume. The position definition field <NUM> defines a location of the void space within the scene. In some embodiments, the position is specified relative to a reference corner of a total 3D volume of the scene. The distance from center field <NUM> defines a distance of the void space (e.g. a centroid of the void space) from a center of the scene (identified via field <NUM>). The distance from center information stored in field <NUM> is derived, in some embodiments, from the position definition information <NUM>. The duration field <NUM> stores a duration, in time units (e.g. milliseconds or seconds), the void space is present in the scene. The priority field <NUM> defines a priority of the void space for insertion of a three-dimension media element, discussed further below. Note that the void spaces data store 308b discussed above with respect to <FIG> includes the void spaces table <NUM> in at least some embodiments.

The content attributes table <NUM> includes a content identifier field <NUM>, a cast field <NUM>, and an attributes field <NUM>. The content identifier field <NUM> uniquely identifies content, such as a particular movie, tv show, video, or other content. The cast field <NUM> identifies a cast of the content (e.g. movie cast, tv show cast, etc). The attributes field <NUM> identifies one or more attributes of the content, such as date of production, title, producer, director, run time, rating (e.g. PG-<NUM>, etc) or other attribute.

The content data table <NUM> includes a content identifier <NUM>, and a plurality of content data pairs, each pair including 2D image data <NUM><NUM>. n and scene identifier <NUM><NUM>. The content identifier <NUM> uniquely identifies content, and is cross referenceable with the content identifier field <NUM>. The 2D image data represents a 2D scene present in the content. The scene identifier <NUM> identifies a scene in which the 2D image data scene is represented. Thus, a plurality of 2D image data field <NUM> may share a common scene identifier value in their respective scene identifier fields <NUM>. Note that the content data store 308c discussed above with respect to <FIG> includes the content table <NUM> in at least some embodiments. Additionally, one or more of the scene table <NUM>, content attributes table <NUM> are included in the content table 408c in at least some embodiments.

The insertable element table <NUM> includes an insertable element identifier <NUM>, mesh data field <NUM>, texture data field <NUM>, and a template identifier field <NUM>. The insertable element identifier field <NUM> uniquely identifies an insertable 3D media element. The mesh data field <NUM> stores data defining a 3D mesh. The 3D mesh may define a plurality of polygons that are joined at their vertexes to create a 3D space. The texture data <NUM> defines a texture of the insertable element. The template identifier field <NUM> defines a template to use when inserting the insertable 3D element into a scene.

The template table <NUM> includes a template identifier field <NUM>, scene identifier field <NUM>, and a formula field <NUM>. The template identifier field <NUM> uniquely identifies a template. The scene identifier field <NUM> identifies a scene for which the template is used to render a 3D insertable object into the scene. The formula field <NUM> defines a one or more transformation operations performed on the 3D insertable element before the element is inserted into a scene.

The insert specifications table <NUM> includes a void space identifier <NUM>, region identifier field <NUM>, date/time field <NUM>, a view properties field <NUM>, and an insertable element identifier <NUM>. The void space identifier field <NUM> identifies a particular void space within content of the content table <NUM>. The void space identifier field <NUM> can be cross referenced with void space identifier field <NUM>. The region identifier field <NUM> identifies a region where the insertion (identified by the particular row of the insert specifications table <NUM>) is to occur. The region identifier field <NUM> identifies a particular geographical region, such as a country, state, or other geographic boundary within which the insertion is to be performed. The region identified is applicable to where the content is ultimately displayed to a viewer and not necessarily where the insertion is physically performed. The date/time field <NUM> identifies one or more of the date range and/or time range when the insertion is to be performed. As with the region, the date/time field <NUM> applies to where content is displayed to a viewer and not necessarily where the insertion is physically performed. The view properties field <NUM> identifies one or more other properties of content display that are used to identify content display subject to the insertion into the void space (identified via <NUM>). The insertable element id field <NUM> identifies a 3D media element to be inserted into the void space. The insertable element id field <NUM> is cross referenced with the insertable element id field <NUM>, discussed above. In some embodiments, the insertion specification table <NUM> is included in the insertion specifications data store 308a, discussed above with respect to <FIG>.

<FIG> show a flowchart of a process for inserting a 3D media element into a 3D representation of a 2D image. In some aspects, one or more of the functions discussed below with respect to <FIG> is performed by hardware processing circuitry. For example, in some embodiments, one or more hardware memories store instructions that when executed configure the hardware processing circuitry to perform one or more of the functions discussed below with respect to <FIG>.

In operation <NUM>, a 2D image is received. The image represents a two-dimensional scene. In some embodiments, the 2D image is part of a video. Thus, in these embodiments, a plurality of 2D images may be received with each of the 2D images representing a portion of the video.

In operation <NUM>, a 3D representation of media elements present in the scene is generated. Thus, for example, while image 102a of <FIG> shows a two-dimensional image, operation <NUM> generates, in this example, 3D representations of each of the two actors 102a and 102b, and the cereal box <NUM>. Additional 3D representations of the counter <NUM>, chair <NUM>, and window <NUM> are also generated. In some embodiments, the 3D representations of each media element are defined via a mesh. In some embodiments, the mesh is comprised of a plurality of polygons connected at the vertices to form a 3D representation of a media element. Each 3D representation also includes position information defining a position of the 3D element within the scene. The position information, along with the mesh definition provides for a determination of space occupied by the 3D media element within the scene. Collectively, the 3D representation(s) of media element in the scene form a 3D representation of the scene itself.

In operation <NUM>, a total three-dimensional volume of the scene is determined. The total 3D volume of the scene may be determined by multiplying a width by a height of the scene, and further multiplying by a distance of an object furthest from a perspective or camera capturing the scene. The distance furthest from the perspective or camera may be identified by one the 3D representations of the media elements discussed above with respect to operation <NUM>. For example, operation <NUM> may search the 3D media elements of operation <NUM> to identify one of the media elements occupying a volume furthest from a perspective or camera capturing the scene.

In operation <NUM>, a 3D volume representation of the scene is generated. The 3D volume representation is generated by aggregating the 3D representations of the media elements within the scene that are generated in operation <NUM>. Thus, continuing with the example discussed above with respect to operation <NUM>, operation <NUM> aggregates volume occupied by the two actors 102a and 102b, cereal box <NUM>, counter <NUM>, chair <NUM>, a window <NUM> (among other items in the scene <NUM> not identified here).

In operation <NUM>, the 3D volume representation generated in operation <NUM> is subtracted from the total 3D volume of the 3D representation.

In operation <NUM>, a 3D void space is defined based on the difference resulting from the subtraction of operation <NUM>.

Continuing to <FIG>, operation <NUM> segments the 3D void space defined by operation <NUM> into a plurality of 3D segments. In some aspects, the 3D segments are of equivalent size. In some embodiments, some of the 3D segments are generated to conform to one or more of the 3D media element representations generated in operation <NUM>.

In operation <NUM>, each of the 3D segments is scored based on one or more of tracking characteristics of the 3D segment, lighting of the 3D segment, duration of time the segment is in focus or otherwise included in the 2D content, a size of the 3D segment, and a distance of the 3D segment from a center of a scene including the 3D segment. Tracking characteristics of the scene may be derived from planer camera tracking characteristics, for example, those obtained from Visual Effects Software (VFX). In some aspects, additional factors are considered when determining a score of a respective segment. For example, other actors considered include, in various embodiments, one or more of a distance from a viewing perspective of the scene or a distance from a camera shooting the scene, a proximity of the 3D segment to one or more actors in the scene. In some embodiments, whether the 3D segment is handled by an actor is considered.

Additional factors considered include identities of actors included in the scene (e.g. field <NUM>), the cast included in the content (e.g. field <NUM>), a location of the scene (e.g. field <NUM>), a year and time of day of a scene including the 3D segment (e.g. field <NUM>). In some embodiments, operation <NUM> considers one or more of a genre and/o sub-genre of the content, a mood of a scene including the 3D element, or a theme of the content. Information about a viewing user may also be used to select content for insertion. For example, each of the content displays 302a-c discussed above with respect to <FIG> may be transmitted to devices associated with individual user accounts. Each of the user accounts define one or more attributes of the user, such as the user's age, gender, interests, past purchase history, and other characteristics. One or more of these factors are considered, in at least some embodiments, when scoring each of the 3D segments.

In operation <NUM>, a segment is selected based on the respective score of each segment. For example, in some aspects, a segment having a highest numerical score is selected. In some embodiments, the segment is selected by a machine learning model. For example, in some embodiments, a model is trained via a data set that identifies 3D segments in a scene. The identified 3D segments are annotated with score or priority information. In some aspects, the annotations are generated via human based supervised learning techniques. From the training data, the model learns which 3D segments are most valuable for insertion of additional 3D media elements. In some embodiments, the model is provided with one or more of a 3D segment's attributes as described above (size, duration, actors in the scene, distance from a center of the scene, etc). The model is provided with a plurality of segments, for example, multiple segments occurring in a particular content scene. The model then ranks the segments based on the learned data from the training. The selection of operation <NUM> is then configured, in these embodiments, to select a highest ranked segment of the scene for insertion of an additional 3D media element.

In operation <NUM>, a 3D media element is inserted into the selected 3D segment.

In operation <NUM>, an output signal is generated based on the 3D representation of the scene including the inserted 3D media element. In some embodiments, operation <NUM> includes displaying a 2D version of the 3D representation on an electronic display. In some embodiments, the output signal writes an output file storing a 2D version of the scene. In some embodiments, operation <NUM> includes projecting the 3D media elements included in the 3D representation (including the inserted 3D media element) of the scene onto a 2D image.

In some embodiments, the output signal represents a 'flat' video (conventional video that is a sequence of individual images displayed on a screen or some other mechanism for making the content viewable). In some embodiments, the inserted 3D element is included in a first video that is overlaid on another (the primary) video. In these embodiments, the primary video includes an indication of transparency of various objects included in a scene such that the inserted 3D media element is correctly displayed with other objects included in the scene.

<FIG> illustrates a block diagram of an example machine <NUM> upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. The machine <NUM> may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a smart phone, a web appliance, a network router, switch or bridge, a server computer, a database, conference room equipment, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. In various embodiments, machine <NUM> may perform one or more of the processes described above with respect to <FIG> above.

Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or mechanisms (all referred to hereinafter as "modules").

The term "machine readable medium" may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine <NUM> and that cause the machine <NUM> to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine readable medium examples may include solid-state memories, and optical and magnetic media. Specific examples of machine readable media may include: nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; Random Access Memory (RAM); Solid State Drives (SSD); and CD-ROM and DVD-ROM disks. In some examples, machine readable media may include non-transitory machine readable media. In some examples, machine readable media may include machine readable media that is not a transitory propagating signal.

The instructions <NUM> may further be transmitted or received over a communications network <NUM> using a transmission medium via the network interface device <NUM>. The machine <NUM> may communicate with one or more other machines utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) <NUM> family of standards known as Wi-Fi®, IEEE <NUM> family of standards known as WiMax®), IEEE <NUM>. <NUM> family of standards, a Long Term Evolution (LTE) family of standards, a Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device <NUM> may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network <NUM>. In an example, the network interface device <NUM> may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. In some examples, the network interface device <NUM> may wirelessly communicate using Multiple User MIMO techniques.

In an example, the whole or part of one or more computer systems (e.g., a standalone, client, or server computer system) or one or more hardware processors may be configured by firmware or software (e.g., instructions, an application portion, or an application) as a module that operates to perform specified operations. In an example, the software may reside on a machine-readable medium.

Claim 1:
An apparatus for modifying media content, the apparatus comprising:
means for obtaining one or more two-dimensional, 2D, images representing a 2D scene from within the media content;
means for generating, based on the one or more of the 2D images, a three-dimensional, 3D, representation of media elements present in the 2D scene, two of the three dimensions identifying a location within a respective image of the media element and a third dimension identifying a distance of the media element from a captured perspective of the scene;
means for creating a 3D volume representation of the scene by aggregating the 3D representations;
means for creating a map of 3D void space in the 2D scene by subtracting the 3D volume representation from a total 3D volume of the 2D scene;
means for segmenting the 3D void space into a plurality of 3D void segments;
means for scoring each of the plurality of 3D void segments based on one or more of tracking characteristics of the 3D void segment, lighting characteristics of the respective 3D void segment, a duration of time the respective 3D void segment is in focus or otherwise included in the 2D scene, a size of the respective 3D void segment, or a distance of the respective 3D void segment from a center of the 2D scene;
means for selecting one of the 3D void segments based on the respective score of each 3D void segment;
means for inserting a 3D media element into the one selected 3D void segment; and
means for modifying the one or more 2D images to include the inserted 3D media element within the selected 3D void segment.