Real-time objects insertion into content based on frame identifiers

Disclosed herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for inserting objects in real-time into content based on frame identifiers. An example embodiment operates by identifying a first object in a frame of a content and generating an identifier for the first object in the frame. The identifier includes an array of numbers representing context and visual features of the first object in the frame. The example embodiments further operates by identifying a second object based on the array of numbers representing the first object and inserting the second object into the frame of the content.

BACKGROUND

Field

This disclosure is generally directed to real-time insertion of objects into content during playback of the content, and more particularly to inserting personalized and localized objects in real-time into one or more frames of content during playback of the content based on identifiers of the one or more frames.

Background

A typical media system operates to receive and play out (e.g., output for playout) an analog or digital media stream representing media content such as video and/or audio content and to output the media content and/or forward the stream for presentation of the content on a user interface such as a display screen and/or an audio speaker. In general, the multimedia content can be delivered from a content source device operated by a content provider to millions of viewers. Examples of the media systems include televisions, computer monitors, projection systems, loudspeakers, headphones, set top boxes (e.g. cable or satellite TV receivers), digital video recorders, radios, personal computers, gaming consoles, streaming media players, and the like.

By way of example, the media system could be a television (TV), which could receive a media stream (e.g., an over-the-air broadcast stream, a media stream output from a set top box, a media stream played from a digital storage medium, an Internet stream, or another such stream) and could present the media content of that stream to a viewer. TV can offer viewers access to content via subscription to cable or satellite services or through over-the-air broadcasts. As another example, the media system could be a set top box or digital video or audio recorder, which could receive a broadcast stream from a multi-channel video program distributor and/or from data storage and could output the media content of that stream via a high-definition multimedia interface (HDMI) cable or other interface to a television and/or audio/video receiver for playout. Numerous other examples are possible as well.

SUMMARY

Provided herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for real-time insertion of objects into content based on frame identifiers. In some embodiments, an object identification system can identify an object in one or more frames of the content. The object identification system can generate an identifier for the identified object. In some embodiments, the identifier can include an array of numbers representing context and visual features of the identified object in the one or more frames. In some embodiments, the content may include streaming content. In some embodiments, the insertion may occur prior to content being streamed to a media device.

In some embodiments, the object identification system can identify a target object similar to the identified object based on the array of numbers. In some embodiments, the object identification system can identify a personalized and localized target object based on user preference and/or user location. In some embodiments, the object identification system can generate an additional array of numbers for the target object. The object identification system can identify the target object similar to the identified object by comparing the arrays of numbers for the identified object and the target object. In some embodiments, the comparison process can include determining a distance between the arrays of numbers for the identified object and the target object. In some embodiments, when the content is paused on the frame having the identified object, a content modification system can display the identified similar target object on the frame. For example, the identified similar target object may correspond to an item that is similar to the object identified within the frame and the displayed target object may be associated with an option to allow a user to purchase the target object.

In some embodiments, the object identification system can identify a target object based on the array of numbers and one or more content modification rules. In some embodiments, the object identification system can identify an appropriate surface on the identified object in the one or more frames. In some embodiments, an appropriate surface may be determined based on certain criteria such as whether there is a match between the surface of the identified object and the properties of the target object. For example, an appropriate surface for a logo target object may be a rectangular flat surface. In some embodiments, an appropriate surface may be a homogeneous surface. In some embodiments, a content modification system can insert the identified target object into the frame such that it is displayed on the homogeneous surface during playback of the content. In some embodiments, the content modification system can perform additional operations on the inserted target object to fit the target object on the homogeneous surfaces, such as resizing and rotation.

An example embodiment of a system can include a storage module and at least one processor each coupled to the storage module and configured to perform various operations to inserting objects in real-time into content based on frame identifiers. In an example, the at least one processor can be configured to identify an object in a frame of a content and generate an identifier for the identified object in the frame. In some embodiments, the object may be identified in one or more frames of the content. The identifier can include an array of numbers representing context and visual features of the identified object in the frame. Afterwards, the at least one processor can be configured to identify a target object based on the array of numbers representing the identified object. In addition, the at least one processor can be further configured to insert the second object into the frame of the content.

DETAILED DESCRIPTION

With the technology advances for multimedia and communication, many types of media content are readily available for streaming and/or display. For example, media content can be delivered via various communication technologies so that the media content can be easily accessed, watched, or listened to anywhere and anytime by viewers. Compared to the early days when media content may be limited to printed publications or delivered by radio, current media content can be available in various forms such as television shows, videos, movies, advertisements, audio files, text, etc., and any combination thereof. In general, media content may be referred to as content, which may include one or more content items, where one content item can include a plurality of scenes and each scene can include a sequence of frames.

Typically, media content includes advertisements inserted between certain frames or in one or more frames as a banner that is separate from the content in the frames. There are issues with these approaches. Advertisements between certain frames may interrupt the flow of the media content. Viewers may have to watch the advertisements and wait for the subsequent frames of the content. Advertisements as banners in one or more frames may obstruct critical visual aspects of the one or more frames, which may lower viewers' experience of the media content. How to effectively insert advertisements in media content without lowering viewers' experience can be of value to the viewers as well as the content creators. Additionally, typical advertisements in media content have time constraint or frame space constraint and may not take much of the time and frame space of the media content.

Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for real-time insertion of personalized and localized objects into one or more frames of content. In some embodiments, the content can be processed to identify a first object in one or more frames of the content and generate an identifier for the one or more frames. In some embodiments, the processing of content may occur at a backend device prior to streaming the processed content to a media device. The identifier can include an array of numbers representing the context and visual features of the first object in the one or more frames. A second object can be identified based on the array of numbers representing the first object. In some embodiments, the second object can be an image, a logo, a string of text, a banner, a combination thereof, or other advertisement objects. A homogenous surface of the first object can be identified in the one or more frames. The homogenous surface can be marked as a frame object marker for object insertion. The frame object marker can be used to track object impressions during playback of the content. The second object can be inserted in real-time into the homogenous surface of the one or more frames. For example, in some embodiments, the second object can be an outfit, a furniture, a toy, an electronic device, or other advertisement merchandises. For example, in some embodiments, during a pause of the content on the one or more frames, the second object can be identified in real-time based on the identifier of the first object in the one or more frames and can be displayed on the one or more frames for viewers to purchase the advertisement merchandise that is associated with the second object. In some embodiments, the second object can be identified based on a preference of the viewer and/or a location of the viewer. The preference and location of the viewer can be obtained from a user account used for playback of the content. Hence, the inserted/displayed second object on the one or more frames of the content may not interrupt the flow of the content and can improve viewer's experience. In some embodiments, selection of the second object may be based on one or more parameters such as the characteristics of the homogenous surface, the characteristics of the first object, or characteristics of the content within the one or more frames. Additionally, additional advertisement objects can be inserted into more frames of the content without decreasing viewer's experience, thus increasing the time and space of advertisements in the content.

Various embodiments of this disclosure may be implemented using and/or may be part of a multimedia environment102shown inFIG.1. It is noted, however, that multimedia environment102is provided solely for illustrative purposes, and is not limiting. Embodiments of this disclosure may be implemented using and/or may be part of environments different from and/or in addition to the multimedia environment102, as will be appreciated by persons skilled in the relevant art(s) based on the teachings contained herein. An example of the multimedia environment102shall now be described.

Multimedia Environment

FIG.1illustrates a block diagram of a multimedia environment102including a content modification system to insert in real-time to a frame of a content an object identified based on an identifier of the frame, according to some embodiments. Multimedia environment102illustrates an example environment, architecture, ecosystem, etc., in which various embodiments of this disclosure may be implemented. However, multimedia environment102is provided solely for illustrative purposes, and is not limiting. Embodiments of this disclosure may be implemented and/or used in environments different from and/or in addition to multimedia environment102ofFIG.1, as will be appreciated by persons skilled in the relevant art(s) based on the teachings contained herein.

In a non-limiting example, multimedia environment102may be directed to streaming media. However, this disclosure is applicable to any type of media (instead of or in addition to streaming media), as well as any mechanism, means, protocol, method and/or process for distributing media.

The multimedia environment102may include one or more media systems104. A media system104could represent a family room, a kitchen, a backyard, a home theater, a school classroom, a library, a car, a boat, a bus, a plane, a movie theater, a stadium, an auditorium, a park, a bar, a restaurant, or any other location or space where it is desired to receive and play streaming content. User(s)132may operate with the media system104to select and consume content, such as content122. Viewers and users (and similar parties and entities) are used interchangeably in the current description.

Each media system104may include one or more media device(s)106each coupled to one or more display device(s)108. It is noted that terms such as “coupled,” “connected to,” “attached,” “linked,” “combined” and similar terms may refer to physical, electrical, magnetic, logical, etc., connections, unless otherwise specified herein.

Media device106may be a streaming media device, a streaming set-top box (STB), cable and satellite STB, a DVD or BLU-RAY device, an audio/video playback device, a cable box, and/or a digital video recording device, to name just a few examples. Display device108may be a monitor, a television (TV), a computer, a computer monitor, a smart phone, a tablet, a wearable (such as a watch or glasses), an appliance, an internet of things (IoT) device, and/or a projector, to name just a few examples. In some embodiments, media device106can be a part of, integrated with, operatively coupled to, and/or connected to its respective display device108.

Each media device106may be configured to communicate with network118via a communication device114. The communication device114may include, for example, a cable modem or satellite TV transceiver. The media device106may communicate with the communication device114over a link116, wherein the link116may include wireless (such as WiFi) and/or wired connections.

Media system104may include a remote control110. The remote control110can be any component, part, apparatus and/or method for controlling the media device106and/or display device108, such as a remote control, a tablet, laptop computer, smartphone, wearable, on-screen controls, integrated control buttons, audio controls, or any combination thereof, to name just a few examples. In an embodiment, the remote control110wirelessly communicates with the media device106and/or display device108using cellular, Bluetooth, infrared, etc., or any combination thereof. The remote control110may include a microphone112, which is further described below.

The multimedia environment102may include a plurality of content server(s)120(also called content providers, channels, or sources). Although only one content server120is shown inFIG.1, in practice the multimedia environment102may include any number of content server(s)120. Each content server120may be configured to communicate with network118. Content server120, media device106, display device108, may be collectively referred to as a media system, which may be an extension of media system104. In some embodiments, a media system may include system server126as well.

Each content server120may store content122and metadata124. Content122may include any combination of music, videos, movies, TV programs, multimedia, images, still pictures, text, graphics, gaming applications, advertisements, programming content, public service content, government content, local community content, software, and/or any other content or data objects in electronic form. Content122may be the source displayed on display device108.

In some embodiments, metadata124comprises data about content122. For example, metadata124may include associated or ancillary information indicating or related to identifiers for each frame of content122, writer, director, producer, composer, artist, actor, summary, chapters, production, history, year, trailers, alternate versions, related content, applications, and/or any other information pertaining or relating to the content122. Metadata124may also or alternatively include links to any such information pertaining or relating to the content122. Metadata124may also or alternatively include one or more indexes of content122, such as but not limited to a trick mode index. In some embodiments, content122can include a plurality of content items, each content item can include a plurality of scenes, and each scene can include a plurality of frames having metadata about the corresponding frame (seeFIG.3).

The multimedia environment102may include one or more system server(s)126. The system server(s)126may operate to support the media device(s)106from the cloud. It is noted that the structural and functional aspects of the system server(s)126may wholly or partially exist in the same or different ones of the system server(s)126. System server(s)126and content server120together may be referred to as a media server system. An overall media system may include a media server system and media system104. In some embodiments, a media system may refer to the overall media system including the media server system and media system104.

The media device(s)106may exist in thousands or millions of media systems104. Accordingly, the media device(s)106may lend themselves to crowdsourcing embodiments and, thus, the system server(s)126may include one or more crowdsource server(s)128.

For example, using information received from the media device(s)106in the thousands and millions of media systems104, the crowdsource server(s)128may identify similarities and overlaps between closed captioning requests issued by different user(s)132watching a particular movie. Based on such information, the crowdsource server(s)128may determine that turning closed captioning on may enhance users' viewing experience at particular portions of the movie (for example, when the soundtrack of the movie is difficult to hear), and turning closed captioning off may enhance users' viewing experience at other portions of the movie (for example, when displaying closed captioning obstructs critical visual aspects of the movie). Accordingly, the crowdsource server(s)128may operate to cause closed captioning to be automatically turned on and/or off during future streaming of the movie. In some embodiments, crowdsource server(s)128can be located at content server120. In some embodiments, some part of content server120functions can be implemented by system server126as well.

The system server(s)126may also include an audio command processing module130. As noted above, the remote control110may include a microphone112. The microphone112may receive audio data from user(s)132(as well as other sources, such as the display device108). In some embodiments, the media device106may be audio responsive, and the audio data may represent verbal commands from the user(s)132to control the media device106as well as other components in the media system104, such as the display device108.

In some embodiments, the audio data received by the microphone112in the remote control110is transferred to the media device106, which is then forwarded to the audio command processing module130in the system server(s)126. The audio command processing module130may operate to process and analyze the received audio data to recognize the user(s)132's verbal command. The audio command processing module130may then forward the verbal command back to the media device106for processing.

In some embodiments, the audio data may be alternatively or additionally processed and analyzed by an audio command processing module216in the media device106(seeFIG.2). The media device106and the system server(s)126may then cooperate to pick one of the verbal commands to process (either the verbal command recognized by the audio command processing module130in the system server(s)126, or the verbal command recognized by the audio command processing module216in the media device106).

FIG.2illustrates a block diagram of an example media device106, according to some embodiments. Media device106may include a streaming module202, processing module204, storage/buffers208, and user interface module206. As described above, user interface module206may include audio command processing module216and one or more sensing module(s)218. In some embodiments, sensing module(s)218can include cameras, microphones, infra-red sensors, and touch sensors, to name just some examples. Sensing module(s)218can capture image and audio signals within a vicinity of sensing module(s)218.

The media device106may also include one or more audio decoders212and one or more video decoders214. Each audio decoder212may be configured to decode audio of one or more audio formats, such as but not limited to AAC, HE-AAC, AC3 (Dolby Digital), EAC3 (Dolby Digital Plus), WMA, WAV, PCM, MP3, OGG GSM, FLAC, AU, AIFF, and/or VOX, to name just some examples.

Now referring to bothFIGS.1and2, in some embodiments, the user(s)132may interact with the media device(s)106via, for example, the remote control110. For example, the user132may use the remote control110to interact with the user interface module206of the media device106to select content, such as a movie, TV show, music, book, application, game, etc. The streaming module202of the media device106may request the selected content from the content server(s)120over the network118. The content server(s)120may transmit the requested content to the streaming module202. The media device106may transmit the received content to the display device108for playback to the user(s)132.

In streaming embodiments, the streaming module202may transmit the content to the display device108in real time or near real time as it receives such content from the content server(s)120. In non-streaming embodiments, the media device106may store the content received from content server(s)120in storage/buffers208for later playback on display device108.

FIG.3illustrates a block diagram of content server(s)120storing content with frame identifiers, according to some embodiments. As shown inFIG.3, content server(s)120can include a plurality of content items, such as content122-1and content122-2. Content122-2can have a similar structure as content122-1. ThoughFIG.3illustrates two content items in content server(s)120, content server(s)120can include more than two content items having a similar structure as content122-1. The discussion of elements of content122-1applies to content122-2, unless mentioned otherwise. And like reference numerals generally indicate identical, functionally similar, and/or structurally similar elements.

In some embodiments, as shown inFIG.3, content122-1can include content metadata124-1. Similarly, content122-2can include content metadata124-2. In some embodiments, content122-1can include a plurality of scenes, such as scene322-1and scene322-2. Content122-2can have a similar structure as content122-1. In some embodiments, each scene can include a plurality of frames. As an example, scene322-1can include frame332-1and frame332-2. Scene322-2can have a similar structure as scene322-1. Scene322-1can include scene metadata324-1. Frame332-1can further include frame metadata334-1and frame identifier336-1. In some embodiments, frame identifier336-1can be included in frame metadata334-1. Similarly, frame332-2can further include frame metadata334-2and frame identifier336-2.

In some embodiments, frame metadata334-1and334-2, and content metadata124-1and124-2may include associated or ancillary information similar to content metadata124as described above. In some embodiments, the associated and ancillary information can be generated by the content creators or by content server(s)120. In some embodiments, frame metadata334-1and334-2may include color contrast, brightness, histogram of color spectrum, a number of objects, a trajectory of objects contained in frames332-1and332-2, or a frame feature contained in frames332-1and332-2. Scene metadata324-1related to scene322-1may include information about objects, people, places, actions, genre, time-frame, caption data text of scene322-1, and audio information related to scene322-1. The time-frame information can include an approximate time of a scene occurring from one frame to another frame. In some embodiments, the time-frame information can help identify an object in a particular scene. In some embodiments, Content metadata124-1and124-2related to respective content122-1and122-2can include information about respective genre of the content, keywords, a description, and reviews of content122-1and122-2.

Frame identifiers336-1and336-2can include context and visual features of objects in frames332-1and332-2, such as object locations in frames332-1and332-2. In some embodiments, each of frame identifiers336-1and336-2can be a matrix having an array of numbers to represent the context and visual features of objects in frames332-1and332-2. For example, as shown inFIG.7, images702(1)-(N) can be represented by identifiers706(1)-(N). In some embodiments, the numbers in the arrays of identifiers706(1)-(N) can indicate a number of objects, a trajectory of objects, a category of objects, color, size, location, and background of objects, and other semantic and visual features of the objects in corresponding images702(1)-(N).

In some embodiments, frames332-1and332-2can be pre-processed and frame identifiers336-1and336-2can be generated to represent context and visual features of objects in frames332-1and332-2, for example, as shown inFIG.7. In some embodiments, frame identifiers336-1and336-2can be provided by the content provider and can be associated with respective frames332-1and332-2.

FIG.4illustrates storage/buffers208that stores information relating to user accounts and content modification rules, according to some embodiments. As shown inFIG.4, storage/buffers208can include one or more content modification rules438and one or more user accounts, such as user account432. User account432may be shared and accessible among multiple users, such as one or more members of a household. User account432can further include user preference434and view history436. User preference434may be related to all the members of the household. In some embodiments, user preference434can include respective user preferences for each member of the household associated with user account432. User preference434can be related to and store information about user settings of media systems104and media content by user(s)132accessed through user account432. For example, user preference434may include user preferred image properties and sound effects, user's favorite genres, and other user requirements and settings. Additionally, user preference434can include one or more images and/or one or more audio recordings of user(s)132for user identification.

View history436can include information related to media content viewed by user(s)132, for example, the name of a show user(s)132viewed, a time user(s)132viewed the show, a frame user stopped viewing, and other related information. In some embodiments, view history436can include a portion of content122being viewed by user(s)132.

Content modification rules438can include rules to modify content122to be played on display device(s)108. In some embodiments, content modification rules438can include rules for inserting an object into one or more frames of content122. In some embodiments, the inserted object can be an image, a logo, a string of text, a banner, a combination thereof, or other advertisement objects. In some embodiments, content modification rules438can include a list of appropriate object categories for insertion into identified objects in the one or more frames. For examples, content modification rules438may specify furniture categories for the frames including identified tables. In some embodiments, content modification rules438can include locating, sizing, rotating, and other operations of the inserted object to fit the inserted object onto the identified object in the frames.

Content Modification Based on Identified Objects

FIG.5illustrates a block diagram of a system500for inserting personalized and localized objects in real-time into a content based on identifiers for one or more frames of the content, according to some embodiments. As shown inFIG.5, system500can include user account432, content modification rules438, content storage542, object storage544, content modification system546, and processed content550with object marker552. Content modification system546can further include object identification system548, image encoder554, and quality control system556.

In some embodiments, content storage542can store one or more media content for processing. Examples of media content can include any combination of music, videos, movies, TV programs, multimedia, images, still pictures, text, graphics, gaming applications, advertisements, programming content, public service content, government content, local community content, software, and/or any other content or data objects in electronic form. In some embodiments, content storage542can be located on content server(s)120as shown inFIG.1and/or storage/buffers208as shown inFIG.2.

In some embodiments, object storage544can include target objects to be inserted into media content. Object storage544can store any combination of images, still pictures, text, graphics, logos, banners, outfits, furniture, toys, electronic devices, and/or any other advertisement objects in electronic form. In some embodiments, object storage544can be located on content server(s)120as shown inFIG.1and/or storage/buffers208as shown inFIG.2.

In some embodiments, content modification system546can be located on content server(s)120as shown inFIG.1and/or storage/buffers208as shown inFIG.2. In some embodiments, content modification system546can process media content in content storage542to identify objects in the frames of the media content with object identification system548. In some embodiments, object identification system548can identify the objects in an image using computer vision image segmentation algorithms. For example, as shown inFIG.6, original image600A can include a mirror602A, a lamp604A, a pillow606A, a large sofa608A, a table610A, a small sofa612A, and other objects in a living room setting. Object identification system548can recognize the living room setting and the objects based on the context and visual features of original image600A. Object identification system548can further generate segmented image600B and can identify mirror602B, lamp604B, pillow606B, large sofa608B, table610B, small sofa612B, and other objects in segmented image600B. In some embodiments, object identification system548can generate segmented image600B using a machine learning model trained to recognize objects in images, such as original image600A. In some embodiments, the machine learning model can recognize the objects based on any text description in original image600A. In some embodiments, the machine learning model can recognize the objects based on any metadata associated with original image600A. In some embodiments, the machine learning model can be trained with modified images. For example, an object in an image can be removed from the image, variations of the object can be filled back into the image. The machine learning model can be trained to recognize the variations and similarities of the objects.

In some embodiments, content modification system546can generate an identifier for an image with image encoder554to represent context and visual features of the objects in the image. For example, as shown inFIG.7, image encoder554can generate identifiers706(1)-(N) for images702(1)-(N). Identifiers706(1)-(N) can also be referred to as I1to IN. In some embodiments, as shown inFIG.7, identifiers706(1)-(N) can have arrays of numbers representing context and visual features of the objects in respective images702(1)-(N). For example, the number matrix in identifier706(1) can represent a dog in image702(1), a location of the dog, a color of the dog, and other features of the dog in image702(1). In some embodiments, the array of numbers in identifiers706(1)-(N) can have multiple dimensions, for example, 512 dimensions. Each dimension can represent one or more features of the object in corresponding images702(1)-(N). In some embodiments, a combination of multiple dimensions can represent one or more features of the object in corresponding images702(1)-(N). In some embodiments, a value of the number in each dimension can be a binary value of 0 or 1 and can be a decimal value ranging from about −1 to about 1. The value differences of one number across the arrays of numbers in identifiers706(1)-(N) can represent variations of the one or more features of the object across images702(1)-(N). In some embodiments, an array of numbers can be referred to as “an embedding” for a corresponding image. In some embodiments, identifies706(1)-(N) can include a matrix of numbers of multiple dimensions, In some embodiments, each of identifiers706(1)-(N) can be treated as a fingerprint or a signature of corresponding images702(1)-(N). In some embodiments, the machine learning model can recognize the objects in images702(1)-(N) and generate identifiers706(1)-(N) for corresponding images702(1)-(N). In some embodiments, identifiers706(1)-(N) can be embedded in images702(1)-(N) as metadata. In some embodiments, identifiers706(1)-(N) can be a part of metadata of corresponding images702(1)-(N).

In some embodiments, content modification system546can generate an identifier for a target object with image encoder554. For example, as shown inFIG.8, image encoder554can recognize target objects802(1),802(2),802(3), and802(4), and generate identifiers806(1),806(2),806(3), and806(4) having arrays or numbers to represent corresponding target objects. Similarly, in some embodiments, identifiers806(1)-(4) can be embedded in target objects802(1)-(4) as fingerprints for corresponding target objects. In some embodiments, identifiers806(1)-(4) can be a part of metadata of corresponding target objects802(1)-(4).

In some embodiments, content modification system546can generate identifiers for one or more frames of a media content with image encoder554. The identifiers can represent the context and visual features of identified objects in the frames. For example, as shown inFIG.9, content modification system546can identify objects in one or more frames of media content900and generate identifiers906having arrays of numbers for the one or more frames. In some embodiments, though identifiers906can have similar array or numbers for each frame of media content900as shown inFIG.9, identifiers906for each frame of media content900can have different array of numbers. In some embodiments, media content900can be preprocessed by content modification system546and identifiers906can be embedded in the frames of processed media content908. In some embodiments, identifiers906for the frames can be stored in a file910(e.g., an XML file) associated with processed media content908. In some embodiments, file910can be included in metadata of the frames of media content900. In some embodiments, a new standard can be set up for processed media content908with embedded identifiers906or associated file910including identifiers906. File910may contain information identifying identified objects within frames within a scene. In some embodiments, the information may be organized based on different scenes (i.e., sequence of frames), based on object type or identifier, or some other parameter for organizing the object information. File910may then be associated with each media content; for example, there may be a file for a particular movie or episode of a TV show such that the file may be utilized to provide object information whenever the media content is requested for streaming. File910may therefore be utilized to avoid having to process the requested media content at run-time.

In some embodiments, content modification system546can identify target objects similar to the identified objects in an image with object identification system548. For example, as shown inFIG.10, object identification system548can segment original images1000(1) and1000(2) and recognize the objects in original images1000(1) and1000(2), such as identified objects1002(1)-(4). Image encoder554can generate identifiers for original images1000(1) and1000(2) including identified objects1002(1)-(4). In some embodiments, Image encoder554can further generate identifiers for identified objects1002(1)-(4). Based on the identifiers of identified objects1002(1)-(4), object identification system548can identify target objects1012A,1012B,1012C similar to identified object1002(1), target object1014similar to identified object1002(3), target object1016similar to identified object1002(2), and target object1018similar to identified object1002(4). In some embodiments, object identification system548can retrieve original images1000(1) and1000(2) from content storage542. In some embodiments, object identification system548can identify target objects1012A-C,1014,1016, and1018in object storage544.

In some embodiments, object identification system548can identify similar objects using a machine learning model. In some embodiments, the machine learning model can compare the identifiers of the identified objects and the target objects to find target objects in object storage544similar to the identified objects. For example, object identification system548can compare the identifiers (e.g., arrays of numbers) of identified object1002(1) with target object identifiers in object storage544and identify similar target objects1012A,1012B, and1012C. In some embodiments, the comparison process can include computing a maximum deviation between the arrays of numbers for the identified objects and the target object, and determine if the maximum deviation is within a predefined tolerance. In some embodiment, if the arrays of numbers are binary values, the comparison process can include computing a Boolean determination or a Hamming distance (as a count of mismatches between respective bit locations in the arrays of numbers). In some embodiment, if the arrays of numbers are decimal values as shown in identifiers906, the comparison process can include determining a distance, such as a Euclidean distance, between the arrays of numbers.

In some embodiments, object identification system548can iterate the comparison process to find appropriate target objects similar to the identified objects. For example, as shown inFIG.10, after first iteration, target objects1012A-C can be identified for identified object1002(1) and target object1018can be identified for identified object1002(4). After additional iterations, object identification system548may rank target objects1012A-C according to their similarities to identified object1002(1). Target object1012A may be identified as the most similar target object for identified object1002(1). Additionally, after more iterations, object identification system548may determine that though target object1018and identified object1002(4) are in a same category of stuffed toys, they actually have different animal figures and thus may not be similar objects.

In some embodiments, content modification system546can insert the identified similar target objects into the frame having identified objects. For example, as shown inFIG.11, during playback of a media content on display device108, the media content may be paused on frame1100by a viewer. In some embodiments, object identification system548can segment and identify object1102(e.g., an outfit) on frame1100. Object identification system548can further identify target objects1112A-C (e.g., similar outfits) similar to identified object1102. Target objects1112A-C can be inserted to frame1100and can be displayed on display device108to the viewer. As target objects1112A-C are be presented on display device108during a pause of the media content, target objects1112A-C may not interrupt the flow of the media content and the viewer's experience can be improved. Additionally, as shown inFIG.11, price tags1116A-C and checkout links1118A-C can be displayed together with corresponding target objects1112A-C, which can provide convenience for the viewer to buy target objects1112A-C and promote the sales of target objects1112A-C. After play button1122is clicked, playback of the media content can continue on display device108and target objects1112A-C may not be displayed during the playback.

In some embodiments, object identification system548can identify similar target objects personalized and localized for the viewer. For example, as shown inFIG.5, content modification system546may retrieve user preference434and view history436from user account432. Object identification system548can identify personalized similar target objects in object storage544based on the identified objects in the original content, user preference434, and view history436. In some embodiments, content modification system546can determine a location of the viewer from user account432and use the viewer's location to identify similar target objects in the local area.

In some embodiments, object identification system548can identify similar target objects using audio information of the media content. For example, object identification system548can obtain the audio information for multiple frames prior to paused frame1100. In some embodiments, object identification system548can capture the audio information with sensing modules(s)218shown inFIG.2. In some embodiments, object identification system548can process the audio information and identify similar target objects in object storage544based on the audio information. For example, characters in the media content may talk about a stuffed toy monkey prior to the pause on frame1100, content modification system546can identify target object1018in object storage544and display target object1018on paused frame1100.

In some embodiments, content modification system546can identify an appropriate surface on identified objects in one or more frames and insert identified target objects onto the appropriate surface of the identified objects. In some embodiments, an appropriate surface may be determined based on visual characteristics of the target object (e.g., round, rectangular, flat, curved, dimensions) and the visual characteristics of the appropriate surface. In some embodiments, content modification system546may identify a surface in the one or more frames based on matching the visual characteristics of the surface and the target object. In some embodiments, the appropriate surface can be marked as a frame object marker in the one or more frames for object insertion. In some embodiments, the frame object marker can be in the form of XML, JSON, or other suitable data formats and can be included in metadata of the one or more frames, for example, as shown inFIG.9.

In some embodiments, the appropriate surface may be identified as a homogeneous surface. In some embodiments, a homogeneous surface can have uniform properties (e.g., color, contrast, texture, dimensions) across the surface. For example, as shown inFIG.12, during playback of a media content on display device108, object identification system548can segment frame1200and identify object1202(e.g., a mailbox) in frame1200. Object identification system548can further identify homogeneous surface1204on identified object1202in frame1200. Content modification system546can insert appropriate target objects on homogeneous surface1204. In some embodiments, homogeneous surface1204can have any shape, such as a square, a rectangle, a circle, an oval, and a triangle. In some embodiments, homogeneous surface1204may not have text or images.

In some embodiments, object identification system548can identify appropriate target objects in object storage544based on content modification rules438and/or user account432, as shown inFIG.5. For example, object identification system548may identify target objects having an appropriate dimension for homogeneous surface1204(as noted above), and/or an appropriate category for identified object1202. For example, identified objects and target objects may be organized into categories (e.g., apparel, electronic devices, furniture, toys) and a target object may be identified based on matching the categories of the identified object1202and the target object. Content modification system546can insert the identified appropriate target object on homogeneous surface1204. In some embodiments, content modification system546can perform additional manipulations on the inserted target objects, for example, rotation, resizing, and so on.

In some embodiments, the target objects can be displayed on homogeneous surface1204in multiple frames during playback of the media content without interrupting the flow of the media content. For example, as shown inFIG.13, media content1300can include a plurality of scenes, such as scene1360. Scene1360can include a plurality of frames1302A(1)-(4). In some embodiments, content modification system546can identify homogeneous surface1204on identified object1202in frames1302A(1)-(4). Content modification system546can further identify appropriate target objects to insert on homogeneous surface1204, such as ad object1306shown inFIG.13. After the insertion, modified frames1302B(1)-(4) can include ad object1306marked on homogeneous surface1204. During playback of modified media content1308, ad object1306can be displayed in modified frames1302B(1)-(4) without interrupting the flow of modified media content1308. Accordingly, viewer experience can be improved and advertisement time and space for target objects can be increased. In some embodiments, the location of homogeneous surface1204may change slightly from frame1302A(1) to frame1302A(4), content modification system546can adjust inserted ad object1306based on different locations of homogeneous surface1204.

In some embodiments, quality control system556can review the processed content, remove artifacts in processed content, and improve the image quality of inserted target objects. For example, as shown inFIG.13, scene1360may include a number of frames that include a series of motions. During these motions, homogeneous surface1204may change locations and/or vary in size from frames1302A(1) to frames1320A(4). Accordingly, in some embodiments, ad object1306may need to be enlarged or shrunk to fit within homogeneous surface1204as it changes locations and/or varies in size in frames1302A(1)-(4). The resizing of ad object1306may introduce artifacts in frames1302A(1)-(4). Quality control system556can review the processed content and reduce artifacts introduced after the content modification. In some embodiments, objects inserted to frames of high motion video content may introduce additional artifacts. Quality control system556can remove the inserted target objects from frames of high motion video content.

FIG.14is a flowchart illustrating a method1400for inserting an object to a frame of a content during playback of the content, according to some embodiments. Method1400can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG.14, as will be understood by a person of ordinary skill in the art. Method1400shall be described with reference toFIGS.5-9,12, and13. However, method1400is not limited to that example embodiment.

Referring toFIG.14, in step1402, a first object is identified in a frame of a content. For example, as shown inFIG.6, mirror602A, lamp604A, pillow606A, large sofa608A, table610A, small sofa612A, and other objects in original image600A can be segmented and identified by object identification system548. The identified mirror602B, lamp604B, pillow606B, large sofa608B, table610B, small sofa612B, and other objects can be demonstrated in segmented image600B. In another example shown inFIG.10, object identification system548can identify objects1002(1)-(4). Additionally, object identification system548can identify object1102(e.g., an outfit) in frame1100inFIG.11and object1202(e.g., a mailbox) in frame1200inFIG.12. In some embodiments, object identification system548can use a machine learning model to segment objects in an image and to recognize the objects in the image. In some embodiments, the machine learning model can be trained with semantic and context information of the objects in the image. In some embodiments, the machine learning model can be trained with the images modified by replacing the objects with different variants.

In step1404, an identifier for the first object can be generated. The identifier can include an array of numbers representing context and visual features of the first object in the frame, such as the object location of the first object in the frame. For example, as shown inFIG.7, identifiers706(1)-(N) can be generated for images702(1)-(N). Each of identifiers706(1)-(N) can include an array of numbers representing context and visual features, such as the object location, of the object in images702(1)-(N). In another example shown inFIG.9, identifiers906can be generated for each frame of media content900. Identifiers906can include arrays of numbers representing context and visual features of the objects in the frames of media content900. In some embodiments, as shown inFIG.5, image encoder554can generate identifiers706(1)-(N) for images702(1)-(N) and identifiers906for the frames of media content900. In some embodiments, the array of numbers can be generated by a machine learning model to identify objects and compare objects.

In step1406, a homogeneous surface of the first object can be identified. For example, as shown inFIG.12, homogeneous surface12can be identified by object identification system548. In some embodiments, homogeneous surface1204can have any shape, such as a square, a rectangle, a circle, an oval, and a triangle. In some embodiments, homogeneous surface1204may not have any text or images.

In step1408, a second object can be identified based on one or more content modification rules and the array of numbers representing the first object. For example, as shown inFIG.13, ad object1306can be identified based on the array of numbers representing identified object1202(e.g., a mailbox). The array of numbers can represent context and visual features of identified object1202in frames1302A(1)-(4), such as homogeneous surface1204on a mailbox object1202. In some embodiments, object identification system548can identify ad object1306based on the context and visual features of identified object1202. Additionally, object identification system548may identify ad object1306based on one or more content modification rules438. Content modification rules438may include a list of appropriate object categories to insert onto the homogeneous surfaces of identified objects. In some embodiments, the characteristics of the homogeneous surface1204may be considered when identifying and selecting a target object to be inserted into the one or more frames.

In step1410, the second object is inserted on the homogeneous surface of the first object in the frame of the content. For example, as shown inFIG.13, ad object1306can be inserted on homogeneous surface1204in frames1302B(1)-(4). In some embodiments, content modification system546can perform further operations on ad object1306to fit in homogeneous surface1204, such as resizing and rotation.

FIG.15is a flowchart illustrating a method1500for displaying a target object based on an identified object in a frame of a content during a pause of the content on the frame, according to some embodiments. Method1500can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG.15, as will be understood by a person of ordinary skill in the art. Method1500shall be described with reference toFIGS.5-11. However, method1500is not limited to that example embodiment.

Referring toFIG.15, in step1502, a first object is identified in a frame of a content. In some embodiments, step1502can be similar to step1402. For example, as shown inFIG.6, mirror602A, lamp604A, pillow606A, large sofa608A, table610A, small sofa612A, and other objects in original image600A can be segmented and identified by object identification system548. The identified mirror602B, lamp604B, pillow606B, large sofa608B, table610B, small sofa612B, and other objects can be demonstrated in segmented image600B. In another example shown inFIG.10, object identification system548can identify objects1002(1)-(4) in original images1000(1) and1000(2). In some embodiments, object identification system548can use a machine learning model to segment objects in an image and to recognize the objects in the image. In some embodiments, the machine learning model can be trained with semantic and context information of the objects in the image. In some embodiments, the machine learning model can be trained with the image modified by replacing the objects with different variants.

In step1504, an identifier for the first object can be generated. The identifier can include an array of numbers representing context and visual features of the first object in the frame. In some embodiments, step1504can be similar to step1404. For example, as shown inFIG.7, identifiers706(1)-(N) can be generated for images702(1)-(N). Each of identifiers706(1)-(N) can include an array of numbers representing context and visual features of the object in images702(1)-(N). In another example shown inFIG.9, identifiers906can be generated for each frame of media content900. Identifiers906can include arrays of numbers representing context and visual features of the objects in the frames of media content900. In some embodiments, as shown inFIG.5, image encoder554can generate identifiers706(1)-(N) for images702(1)-(N) and identifiers906for the frames of media content900. In some embodiments, the array of numbers can be generated by a machine learning model to identify objects and compare objects.

In step1506, a second object is identified based on the array of numbers representing the first object. For example, as shown inFIG.10, based on the arrays of numbers for identified objects1002(1)-(4), object identification system548can identify target objects1012A,1012B,1012C similar to identified object1002(1), target object1014similar to identified object1002(3), target object1016similar to identified object1002(2), and target object1018similar to identified object1002(4). In some embodiments, similarities between target objects and identified objects may be based on one or more of the categories of the target and identified objects. In another example shown inFIG.11, based on the arrays of numbers for identified object1102(e.g., an outfit) on frame1100, object identification system548can identify target objects1112A-C (e.g., similar outfits) similar to identified object1102. In some embodiments, object identification system548can identify similar objects using a machine learning model. In some embodiments, the machine learning model can compare the identifiers of the identified objects and the target objects to find target objects in object storage544similar to the identified objects. In some embodiments, the comparison process can include determining a distance between the arrays of numbers for the identified objects and the target objects. In some embodiments, the target object may be further identified based on the characteristics of the surface (e.g., dimensions, shape) in the frame onto which the target object will be displayed.

In step1508, the second object is selected and displayed in the frame during a pause of the content on the frame. For example, as shown inFIG.11, when the media content is paused on frame1100, the identified similar target objects1112A-C (e.g., similar outfits) can be displayed on display device108. As target objects1112A-C are be presented on display device108during a pause of the media content, target objects1112A-C may not interrupt the flow of the media content and the viewer's experience can be improved. Additionally, as shown inFIG.11, price tags1116A-C and checkout links1118A-C can be displayed together with corresponding target objects1112A-C, which can provide convenience for the viewer to buy target objects1112A-C and promote the sales of target objects1112A-C. After play button1122is clicked, playback of the media content can continue on display device108and target objects1112A-C may not be displayed during the playback.

Example Computer System

Various embodiments may be implemented, for example, using one or more well-known computer systems, such as computer system1600shown inFIG.16. For example, the media device106may be implemented using combinations or sub-combinations of computer system1600. Also or alternatively, one or more computer systems1600may be used, for example, to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof.

Computer system1600may include one or more processors (also called central processing units, or CPUs), such as a processor1604. Processor1604may be connected to a communication infrastructure or bus1606.

Computer system1600may also include user input/output device(s)1603, such as monitors, keyboards, pointing devices, etc., which may communicate with communication infrastructure1606through user input/output interface(s)1602.

Computer system1600may also include a main or primary memory1608, such as random access memory (RAM). Main memory1608may include one or more levels of cache. Main memory1608may have stored therein control logic (i.e., computer software) and/or data.

Computer system1600may also include one or more secondary storage devices or memory1610. Secondary memory1610may include, for example, a hard disk drive1612and/or a removable storage device or drive1614. Removable storage drive1614may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and/or any other storage device/drive.

Removable storage drive1614may interact with a removable storage unit1618. Removable storage unit1618may include a computer usable or readable storage device having stored thereon computer software (control logic) and/or data. Removable storage unit1618may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/any other computer data storage device. Removable storage drive1614may read from and/or write to removable storage unit1618.

Secondary memory1610may include other means, devices, components, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system1600. Such means, devices, components, instrumentalities or other approaches may include, for example, a removable storage unit1622and an interface1620. Examples of the removable storage unit1622and the interface1620may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB or other port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

Computer system1600may further include a communication or network interface1624. Communication interface1624may enable computer system1600to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference number1628). For example, communication interface1624may allow computer system1600to communicate with external or remote devices1628over communications path1626, which may be wired and/or wireless (or a combination thereof), and which may include any combination of LANs, WANs, the Internet, etc. Control logic and/or data may be transmitted to and from computer system1600via communication path1626.

In some embodiments, a tangible, non-transitory apparatus or article of manufacture comprising a tangible, non-transitory computer useable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system1600, main memory1608, secondary memory1610, and removable storage units1618and1622, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system1600or processor(s)1604), may cause such data processing devices to operate as described herein.

CONCLUSION