HDMI customized ad insertion

Disclosed herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for ad insertion by a display device coupled to a media device via a high-definition media interface (HDMI) connection, where the media device provides media content and/or a control signal. When the media device pauses the media content, the display device can determine that a pause event has occurred and insert an ad shown on the display device. Further, some embodiments include determining the context and/or content of the media content that is paused, and determining an ad that is customized to the determined context and/or content to be displayed on the display device. In some embodiments, the display device can determine additional information from the control signal that may also be used to determine the ad to be displayed on the display device.

BACKGROUND

Field

This disclosure is generally directed to inserting ads for display and more particularly to inserting customized ads for display based on paused media content.

Background

Content, such as a movie or TV show, is typically displayed on a television or other display screen for watching by users. Accordingly, a user's experience of the content is typically confined to the TV and to speakers connected to the TV. A user can control the consumption of media services by selecting media content for streaming, playing, pausing, and unpausing streaming media content.

SUMMARY

Provided herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for high-definition multimedia interface (HDMI) customized ad insertion. An example embodiment operates by detecting a pause event in one or more frames (e.g., audio or video frames) received via a HDMI connection, and during the pause event, recognizing content within the one or more frames. Some embodiments include receiving a control signal via the HDMI connection and determining information about the source device, and/or information about the application running on the source device. Some embodiments include determining an ad based on i) the content within the one or more frames, and/or ii) information about the source device and/or information about the application running on the source device. The ad can be displayed on the display device.

Some embodiments include a display device detecting a pause event in one or more frames received via an HDMI connection, and during the pause event, recognizing content within the one or more frames. Some embodiments include determining an ad based on the content within the one or more frames, and presenting the ad on the display device. Detecting the pause event can include: i) receiving a remote control pass through pause signal; ii) detecting a silent audio signal via the HDMI connection, and determining that a video frame of the one or more frames has not changed; and/or iii) detecting a pause icon from the one or more frames using computer vision (CV) technology, and detecting no change from a first video frame to a second video frame, where the first video frame and the second video frame are of the one or more frames.

In some embodiments, recognizing the content includes analyzing a first video frame or a first audio frame of the one or more frames using automatic content recognition (ACR) technology, and determining a fingerprint, a watermark, or a cue tone corresponding to the first video frame or the first audio frame, where the fingerprint, watermark, and/or cue tone is used to identify information about content corresponding to the first video frame or the first audio frame. In some embodiments, recognizing the content includes analyzing a first video frame of the one or more frames using CV technology, and determining metadata corresponding to the first video frame, where the metadata is used to identify one or more objects corresponding to the first video frame.

Some embodiments include receiving a control signal via the HDMI connection and determining service product description (SPD) from the control signal, where the ad is determined based on the SPD. Some embodiments further include detecting an auto low latency mode (ALLM) from the control signal, where the ad is based on the ALLM. Some embodiments further include detecting a variable refresh rate (VRR) from the control signal, where the ad is based on the VRR.

In some embodiments, determining the ad includes transmitting the VRR to a network, and receiving the ad corresponding to the VRR from the network. In some examples, determining the ad includes transmitting a fingerprint, watermark, cue tone, or metadata corresponding to the content to a network, and receiving a corresponding ad from the network.

Some embodiments include transmitting an SPD or an ALLM corresponding to a control signal received via the HDMI connection, to a network, and receiving an ad corresponding to the SPD or the ALLM from the network. Some embodiments include presenting the ad on a graphics plane of the display device, blending the graphics plane with a video plane comprising a first video frame of the one or more frames, and presenting the blended planes on the display device.

Further embodiments, features, and advantages of the present disclosure, as well as the structure and operation of the various embodiments of the present disclosure, are described in detail below with reference to the accompanying drawings.

DETAILED DESCRIPTION

Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for ad insertion by a display device coupled via a high-definition media interface (HDMI) connection, to a media device where the media device provides media content. When the media device pauses the media content, the display device can determine that a pause event has occurred and insert an ad shown on the display device. In other words, the detection of the pause event can trigger the insertion of a relevant ad. After the pause event is detected, some embodiments include determining the context and/or content of the media content that is paused, and selecting an ad that is customized to the determined context and/or content to be displayed on the display device. In some embodiments, the display device may be coupled via a media interface connection other than HDMI to a media device.

Various embodiments of this disclosure may be implemented using and/or may be part of a multimedia environment102shown inFIG.1. It is noted, however, that multimedia environment102is provided solely for illustrative purposes, and is not limiting. Embodiments of this disclosure may be implemented using and/or may be part of environments different from and/or in addition to the multimedia environment102, as will be appreciated by persons skilled in the relevant art(s) based on the teachings contained herein. An example of the multimedia environment102shall now be described.

Multimedia Environment

FIG.1illustrates a block diagram of a multimedia environment102, according to some embodiments. In a non-limiting example, multimedia environment102may be directed to streaming media. However, this disclosure is applicable to any type of media (instead of or in addition to streaming media), as well as any mechanism, means, protocol, method and/or process for distributing media.

The multimedia environment102may include one or more media systems104. A media system104could represent a family room, a kitchen, a backyard, a home theater, a school classroom, a library, a car, a boat, a bus, a plane, a movie theater, a stadium, an auditorium, a park, a bar, a restaurant, or any other location or space where it is desired to receive and play streaming content. User(s)132may operate with the media system104to select and consume content.

Each media system104may include one or more media devices106each coupled to one or more display devices108. It is noted that terms such as “coupled,” “connected to,” “attached,” “linked,” “combined” and similar terms may refer to physical, electrical, magnetic, logical, etc., connections, unless otherwise specified herein.

Media device106may be a streaming media device, DVD or BLU-RAY device, audio/video playback device, cable box, and/or digital video recording device, to name just a few examples. Display device108may be a monitor, television (TV), computer, smart phone, tablet, wearable (such as a watch or glasses), appliance, internet of things (IoT) device, and/or projector, to name just a few examples. In some embodiments, media device106can be a part of, integrated with, operatively coupled to, and/or connected to its respective display device108.

Each media device106may be configured to communicate with network118via a communication device114. The communication device114may include, for example, a cable modem or satellite TV transceiver. The media device106may communicate with the communication device114over a link116, wherein the link116may include wireless (such as WiFi) and/or wired connections.

Media system104may include a remote control110. The remote control110can be any component, part, apparatus and/or method for controlling the media device106and/or display device108, such as a remote control, a tablet, laptop computer, smartphone, wearable, on-screen controls, integrated control buttons, audio controls, or any combination thereof, to name just a few examples. In an embodiment, the remote control110wirelessly communicates with the media device106and/or display device108using cellular, Bluetooth, infrared, etc., or any combination thereof. The remote control110may include a microphone112, which is further described below.

The multimedia environment102may include a plurality of content servers120(also called content providers, channels or sources). Although only one content server120is shown inFIG.1, in practice the multimedia environment102may include any number of content servers120. Each content server120may be configured to communicate with network118.

In some embodiments, metadata124comprises data about content122. For example, metadata124may include associated or ancillary information indicating or related to writer, director, producer, composer, artist, actor, summary, chapters, production, history, year, trailers, alternate versions, related content, applications, and/or any other information pertaining or relating to the content122. Metadata124may also or alternatively include links to any such information pertaining or relating to the content122. Metadata124may also or alternatively include one or more indexes of content122, such as but not limited to a trick mode index.

The multimedia environment102may include one or more system servers126. The system servers126may operate to support the media devices106from the cloud. It is noted that the structural and functional aspects of the system servers126may wholly or partially exist in the same or different ones of the system servers126.

The system servers126may also include an audio command processing module130.

As noted above, the remote control110may include a microphone112. The microphone112may receive audio data from users132(as well as other sources, such as the display device108). In some embodiments, the media device106may be audio responsive, and the audio data may represent verbal commands from the user132to control the media device106as well as other components in the media system104, such as the display device108.

In some embodiments, the audio data received by the microphone112in the remote control110is transferred to the media device106, which is then forwarded to the audio command processing module130in the system servers126. The audio command processing module130may operate to process and analyze the received audio data to recognize the user132's verbal command. The audio command processing module130may then forward the verbal command back to the media device106for processing.

In some embodiments, the audio data may be alternatively or additionally processed and analyzed by an audio command processing module216in the media device106(seeFIG.2). The media device106and the system servers126may then cooperate to pick one of the verbal commands to process (either the verbal command recognized by the audio command processing module130in the system servers126, or the verbal command recognized by the audio command processing module216in the media device106).

FIG.2illustrates a block diagram of an example media device106, according to some embodiments. Media device106may include a streaming module202, processing module204, storage/buffers208, and user interface module206. As described above, the user interface module206may include the audio command processing module216.

The media device106may also include one or more audio decoders212and one or more video decoders214.

Each audio decoder212may be configured to decode audio of one or more audio formats, such as but not limited to AAC, HE-AAC, AC3 (Dolby Digital), EAC3 (Dolby Digital Plus), WMA, WAV, PCM, MP3, OGG GSM, FLAC, AU, AIFF, and/or VOX, to name just some examples.

Now referring to bothFIGS.1and2, in some embodiments, the user132may interact with the media device106via, for example, the remote control110. For example, the user132may use the remote control110to interact with the user interface module206of the media device106to select content, such as a movie, TV show, music, book, application, game, etc. The streaming module202of the media device106may request the selected content from the content server(s)120over the network118. The content server(s)120may transmit the requested content to the streaming module202. The media device106may transmit the received content to the display device108for playback to the user132.

In streaming embodiments, the streaming module202may transmit the content to the display device108in real time or near real time as it receives such content from the content server(s)120. In non-streaming embodiments, the media device106may store the content received from content server(s)120in storage/buffers208for later playback on display device108.

HDMI Customized Ad Insertion

Referring toFIG.1, media device106may be a streaming media device (e.g., a set top box or a gaming device) coupled to display device108via an HDMI connection107(e.g., an HDMI cable). For example, user132may select HDMI as a source input for media streaming on display device108. User132may use remote control110to communicate with media device106via radio interface111. In previous solutions, when user132paused media content provided by media device106(e.g., a source device) the corresponding video frames may be paused (e.g., halted). In some examples media device106may transmit a pause icon in a signal and the pause icon may be presented on display device108(e.g., a smart TV). In previous solutions, display device108did not detect a pause event. Consequently, display device108would not insert an ad during the pause event, or analyze the content and/or context of the paused media content to determine a customized ad, and display the customized ad on a graphics plane of display device108.

FIG.6illustrates a block diagram of media device600, according to some embodiments. For illustration purposes and not a limitation, media device600may be described with reference to elements from other figures in the disclosure. For example, HDMI output680may support HDMI connection107ofFIG.1, where HDMI connection107is coupled to display device108ofFIG.1. User interface module620and audio command processing module625can correspond to user interface module206and audio command processing module216ofFIG.2. Processor630may correspond to processing module204ofFIG.2. Buffers/storage640, audio decoder660, and video decoder650may correspond to storage/buffers208, audio decoder(s)212, and video decoder(s)214ofFIG.2.

Media device600includes applications610that can include streaming applications612, broadcast/tuner614, USB inputs616, and gaming application618. In some embodiments, media device600can be a gaming device. Applications610can be a source of information. Processor630can extract audio and video data from corresponding applications and place the extracted audio and video data into buffers/storage640. After audio and video decoding, post processing module670can perform scaling of images, color improvements, and lip synching functions, for example, before outputting video frames via HDMI output680to a display device (e.g., display device108ofFIG.1, display device108ofFIG.3, display device700ofFIG.7A, display device705ofFIG.7B, and/or display device900ofFIG.9). In some embodiments, media device600transmits a service product description (SPD), auto low latency mode (ALLM), or a variable refresh rate (VRR) to display device700ofFIG.7A, display device705ofFIG.7B, or display device900ofFIG.9.

FIG.8illustrates a block diagram of media device800, according to some embodiments. For illustration purposes and not a limitation, media device800may be described with reference to elements from other figures in the disclosure. For example, HDMI output880, processor830, buffers/storage840, audio decoder860, and video decoder850media device800may correspond to HDMI output680, processor630, buffers/storage640, audio decoder660, and video decoder650of media device600ofFIG.6.

Media device800includes applications810that may be a data source including but not limited to input application816, network application814, and media application817. Input application816can include but is not limited to input applications corresponding to a joystick, a microphone, and/or a remote control device. Network application814can include but is not limited to Wi-Fi™, Bluetooth™, Zigbee™, and/or a tuner. Media application817can include but is not limited to a transport stream, a movie, a game, and/or music.

Processor830can extract audio and video data from corresponding applications and place the extracted audio and video data into buffers/storage840. After audio and video decoding, output video frames are transmitted via HDMI output880to a display device (e.g., display device108ofFIG.1, display device108ofFIG.3, display device700ofFIG.7A, display device705ofFIG.7B, and/or display device900ofFIG.9). In some embodiments, media device800transmits an SPD, an ALLM, or a VRR to display device700, display device705ofFIG.7B, or display device900ofFIG.9.

FIG.3illustrates a block diagram of display device108, according to some embodiments. For illustration purposes and not a limitation, display device108may be described with reference to elements from other figures in the disclosure. For example, HDMI module310may support HDMI connection107ofFIG.1, where HDMI connection107is coupled to media device106ofFIG.1orFIG.2. Display device108can include ACR module320, CV module325, ad insertion module330, processing module340, video scaling and frame rate conversion (FRC) module350, graphics module360, display module370, and memory380. Bus390can coupled the elements of display device108.

ACR module320can analyze a video frame and generate a unique fingerprint and/or unique watermark with information about the content of the video frame. For example, the unique fingerprint and/or watermark may include multiple patches of content within the video frame. The unique fingerprint and/or watermark may be compared against a known reference fingerprint or reference watermark to identify the video content based on the match. The match may identify for example, a movie title, actor, rating, where the movie can be found for viewing, production house, etc. In some examples, the unique fingerprint and/or watermark can correspond to content particular to the video frame itself. The fingerprints and/or watermark generated from different consecutive video frames can be compared. If two or more generated fingerprints and/or watermarks are the same, that can indicate no change in content (e.g., the video or movie may be paused.)

In some embodiments, ACR module320can analyze an audio frame and generate a unique fingerprint and/or unique watermark with information about the content of the audio frame. For example, the unique fingerprint and/or watermark may include multiple patches of content within the audio frame. The unique fingerprint and/or watermark may be compared against a known reference fingerprint or reference watermark to identify the audio content based on the match. The match may identify for example, a movie title, actor, rating, where the movie can be found for viewing, production house, etc. In some examples, the unique fingerprint and/or watermark can correspond to content particular to the audio frame itself. The fingerprints and/or watermark generated from different consecutive audio frames can be compared. If two or more generated fingerprints and/or watermarks are the same, that can indicate no change in content (e.g., the audio or movie may be paused.)

In some embodiments ACR module320can analyze a video frame and/or an audio frame and generate a unique cue tone (e.g., a cue tone compliant with American National Standards Institute (ANSI)/Society of Cable and Telecommunications Engineers (SCTE) standards defined in ANSI/SCTE 35 2013) with information about the content of the video frame and/or audio frame.

CV module325can analyze a video frame and generate metadata. The metadata can be used to recognize objects within the video frame, or one or more video frames. The metadata can for example, recognize images and/or objects within the video frame. Examples of the metadata can include for example, items depicted in the video frame such as a bicycle, car, a beer bottle, a mountain, and/or scenery.

Display device108can detect a pause event by using: i) a remote control pass through function; ii) a silent audio signal; and/or iii) pause icon recognition.

In some embodiments, display device108may use a remote control pass through function to detect a pause event. For example, media device106may be configured to support Consumer Electronics Control (CEC) (e.g., remote control pass through) functions including but not limited to: power on, power off, pause key, play key, and so on. For example, user132using remote control110can transmit a pause key via radio interface111to the media device which transmits the pause key signal to display device108. In this example, display device108can readily detect a pause event based on the pause key signal being received from the media device across HDMI connection107. The detection of the pause event may be performed in hardware by processing module340, by processing module340executing software stored in memory380, or a combination thereof. Processing module340may include one or more processors, for example.

In some embodiments, display device108can detect a pause event based on a silent audio signal received from media device106as well as determining that one or more video frames received from media device106is not changing. The unchanging video frames in conjunction with the silent audio signal can be used to determine that a pause event is detected. A silent audio signal can be an audio signal corresponding to one or more video or audio frames received from media device106that may not be within a human audible frequency range. In other words, the silent audio signal volume is so low that user132may not hear the silent audio signal, but the silent audio signal is present. In addition to detecting a silent audio signal, display device108can determine that one or more video frames received are not changing by utilizing periodic sampling. In some examples, ACR module320and/or CV module325can be used to determine that contents are not changing between consecutive video frames.

In some embodiments, a fingerprint and/or watermark can be generated by ACR module320for each frame received and the generated fingerprints or watermarks can be compared. If the generated fingerprints and/or watermarks are different, then the one or more video frames are changing and a pause event is unlikely, even if a silent audio signal is detected. If however, the fingerprints and/or watermarks are the same (e.g., fingerprints and/or watermarks from 3-4 consecutive frames are unchanged) and a silent audio signal is detected, then display device108determines that a pause event is detected. In other words, user132may have paused streaming data at media device106, perhaps by using remote control110. The detection of the silent audio signal and/or the comparison of the fingerprints and/or watermarks from different video frames can be performed in hardware by processing module340, by processing module340executing software stored in memory380, or a combination thereof.

In some embodiments, metadata can be generated by CV module325for each frame received and the generated metadata can be compared. If the generated metadata are different, then the one or more video frames are changing and a pause event is unlikely, even if a silent audio signal is detected. If however, the metadata for each frame are the same (e.g., metadata from 3-4 consecutive frames are unchanged) and a silent audio signal is detected, then display device108determines that a pause event is detected. In other words, user132may have paused streaming data at media device106, perhaps by using remote control110. The detection of the silent audio signal and/or the comparison of the metadata from different video frames can be performed in hardware by processing module340, by processing module340executing software stored in memory380, or a combination thereof.

In some embodiments, display device108can detect a pause event based on a recognition of a pause icon. For example, CV module325can analyze media content (e.g., one or more video frames) received via HDMI connection107from media device106to generate metadata that identifies objects in a video frame (e.g., a type of car, an actor, a pause icon, a penguin, or a bee on a flower.) When a pause icon is detected in one or more consecutive video frames received via HDMI module310, then a pause event is detected. The detection of the pause icon in one or more consecutive video frames can be performed in hardware by processing module340, by processing module340executing software stored in memory380, or a combination thereof.

After display device108determines that a pause event is detected (e.g., based on a remote control pass through function, a silent audio detected with unchanging video frames, or a pause icon detected with no motion in video frames), display device108can perform content recognition on the video frame and/or audio frame of the pause event (e.g., a scene of a movie that is paused). In some embodiments, ACR module320may generate a fingerprint and/or watermark for the video frame and/or the audio frame. In some embodiments, CV module325may generate metadata that corresponds to objects identified or recognized within the video frame.

Display device108can use the fingerprint, watermark, and/or the metadata to determine one or more appropriate ads (e.g., advertisements), add the ad(s) to a graphics plane, and present the ad(s) on display device108for a portion of or a duration of the pause event. For example, after the pause event detection (e.g., ad insertion module330may receive a notice of the pause event detection from processing module340) the corresponding fingerprint and/or watermark from ACR module320, and/or the corresponding metadata from CV module325. Ad insertion module330can use the corresponding fingerprint, watermark, and/or metadata to select one or more relevant ads to be presented on display device108. The fingerprint and/or watermark may correspond to a certain movie title, famous actor, and movie genre. The metadata may correspond to a champagne bottle and mountain scenery. The relevant ads may include the famous actor, a type of champagne, vacation opportunities that include the mountain scenery recognized. The one or more relevant ads may be static (e.g., a still image) or dynamic (e.g., a gif, a short video, an animation, an interactive screen). In some embodiments, input from user132regarding the interactive screen (e.g., a survey) can be collected and transmitted to crowdsource server128. In some embodiments in response to input corresponding to the one or more relevant ads, additional information may be transmitted to a mobile device corresponding to a customer account associated with display device108(e.g., email, text).

Ad insertion module330may transmit the one or more relevant ads to graphics module360to be added to a graphics plane (not shown) of display device108. The one or more video frames that are received may be received and processed by video scaling and FRC350. The one or more video frames can be added to a video plane of display device108(e.g., a last video frame used in the detection of the pause event can be added to the video plane). In some embodiments, the video plane and the graphics plane can be blended (e.g., merged) together and the data of the blended planes can be received by display module370for presentation on display device108. In some embodiments the graphics plane may be displayed in front of the video plane (e.g., overlaid to be viewed on top of the video plane). In some embodiments the images and/or ads of the graphics plane can be transparent. In some embodiments a selection function from remote control110can be passed through media device106to display device108to make a selection from the graphics plane. The functions above may be performed by processing module340, executing software of the various modules that may be stored in memory380, or a combination of hardware and software.

FIG.7Aillustrates a block diagram of display device700, according to some embodiments. For example, display device700may utilize control signal information received via an HDMI connection in addition to video frames, to determine appropriate and relevant ads to present on display device700. In some embodiments the relevant ads are presented on display device700during a pause event.

For illustration purposes and not a limitation, display device700may be described with reference to elements from other figures in the disclosure. For example, HDMI input710may support HDMI connection107ofFIG.1, where HDMI connection107is coupled to media device106ofFIG.1. HDMI input710, ACR730, computer vision740, ad insertion750, graphics module770, memory780, and video scaling and FRC782, and display panel790can correspond respectively to HDMI module310, ACR module320, CV module325, ad insertion module330, graphics module360, memory380, video scaling and FRC350, and display module370ofFIG.3.

In some embodiments, display device700may be coupled via HDMI connection to a media device (e.g., media device600ofFIG.6or media device800ofFIG.8). In some embodiments, media device600can be a gaming device. Display device700can receive video frames as well as one or more control signals from media device600via HDMI input710. The control signals can include a product information signal, a latency mode signal, or a refresh rate signal from media device600. Product information function722can be a function of processor720that determines SPD about the source, such as media device600, from the product information signal. Examples of SPD include the type of gaming device platform, and/or the model of the gaming device platform. As an example, the product information signal can include an SPD InfoFrame that communicates the name and product type of a source device such as media device600(e.g., a video game console) to a sink such as display device700(e.g., a television (TV)). When a video game console is coupled to a TV, the video game console may transmit the following SPD via the product information signal to display device700. The SPD can include the following:

Vendor Name=Company A

Product Description=Video game console model1

Using the above SPD information, display device700can display an advertisement of a newer model of media device600, in this example, a newer model than video game console model1. Display device700can also display an advertisement of a competitor product.

Latency mode function724can be a function of processor720that determines an ALLM from a latency mode signal received from media device600. The ALLM can indicate whether or not gaming application618running (e.g., executing) on media device600is a low latency application. Accordingly, display device700utilizes resources to enable corresponding low latency performance supporting gaming application618. For example, the ALLM enables media device600(e.g., a source like a video game console) to automatically enable or disable the low latency performance of display device700(e.g., a sink like a TV) without requiring a user to navigate to menus of display device700(e.g., the sink's menus) to set the optimal latency corresponding to the content of media device600(e.g., gaming application618such as the video game console gaming content). Media device600may transmit the latency mode signal via an HDMI Forum Vendor Specific Info Frame (HF-VSIF). The 5th byte in an HF-VSIF may include an ALLM_Mode field which can be either 0 or 1. If ALLM_Mode=1, media device600(e.g., a source like the video game console) requests that display device700(e.g., a sink like a TV) switches to game mode. If ALLM_Mode=0, media device600requests that display device700turns off game mode. Accordingly, latency mode function724can determine (a value of) the ALLM that can be used as an indication that the user has started playing a game and hence relevant advertisements related to the game can be displayed to the user via display device700.

Refresh rate function726can be a function of processor720that determines a VRR for advanced gaming applications from the refresh rate signal. In some embodiments, the VRR can be a dynamic display refresh rate that can continuously and seamlessly vary on the fly (e.g., change during streaming). The VRR can be transmitted by media device600using a Video Timing Extended Metadata Packet (EMP) in the refresh rate signal. Using the VRR, display device700can advertise sources and/or applications that offer similarly advanced types of VRR and/or more advanced types of VRR (e.g., VRRs supported by Free Sync offered by AMD graphics cards or GSync offered by NVIDIA graphics cards). In some embodiments, display device700can utilize the control signal information (e.g., SPD from product information function722, ALLM from latency mode function724, and/or VRR from refresh rate function726) to determine relevant ads to be presented on display panel790.

In some embodiments, display device700may detect a pause event by using: i) a remote control pass through function; ii) a silent audio signal; and/or iii) pause icon recognition. For example, display device700can readily detect a pause event based on a pause key signal (e.g. a remote control pass through function) being received from the media device across an HDMI connection (e.g., HDMI connection107) via HDMI input710. The detection of a pause event may be performed in hardware by processor720, by processor720executing software stored in memory780, or a combination thereof. Processor720may include one or more processors, for example. In some embodiments, processor720may execute software stored in a different memory (not shown).

In some embodiments, display device700can detect a pause event based on a silent audio signal received from the media device (e.g., media device106, media device600, or media device800) as well as determine that one or more video frames received from the media device is not changing. For example, a fingerprint and/or watermark can be generated by ACR730for each frame received and the generated fingerprints and/or watermarks can be compared. In some embodiments, CV740can generate metadata for each video frame received and generated metadata from consecutive video frames can be compared. When the fingerprints, watermarks, and/or metadata from consecutive video frames are unchanged in conjunction with a silent audio signal, then a pause event is detected. The detection of a pause event based on the silent audio signal and the unchanging video frames may be performed in hardware by processor720, by processor720executing software stored in memory780, or a combination thereof. Processor720may include one or more processors, for example. In some embodiments, processor720may execute software stored in a different memory (not shown).

In some embodiments, display device108can detect a pause event based on pause icon recognition. For example, CV740can analyze media content (e.g., one or more video frames) received via HDMI input710from the media device to generate metadata that identifies objects (e.g., a pause icon) in a video frame. When metadata from one or more consecutive video frames received include a pause icon, and a determination is made that the video frames are not changing, a processor can determine that a pause event has been detected. The detection of a pause icon and the unchanging video frames may be performed in hardware by processor720, by processor720executing software stored in memory780, or a combination thereof. Processor720may include one or more processors, for example. In some embodiments, processor720may execute software stored in a different memory (not shown).

After display device700determines that a pause event is detected (e.g., based on a remote control pass through function, silent audio detected with unchanging video frames, and/or a pause icon recognition with no motion of the video frames), display device700can perform content recognition on the video frame of the pause event (e.g., a scene of a movie that is paused) to determine relevant ads to be presented on display panel790for a period of or duration of the pause event. For example, ACR730may generate a fingerprint and/or watermark for the video frame that enables determination of the video (e.g., the movie) including the video frame, and data associated with the video (e.g., actors, movie title, genre). In some embodiments, computer vision740may generate metadata that corresponds to objects (e.g., champagne bottle, mountain scenery) identified or recognized within the video frame. The fingerprints, watermarks, and/or metadata generated can be used to determine relevant ad to be displayed on display device700for a portion of or the duration of the pause event.

In some embodiments, after display device700determines that a pause event is detected (e.g., based on a remote control pass through function, silent audio detected with unchanging video frames, and/or a pause icon recognition with no motion of the video frames), display device700can perform content recognition on an audio frame of the pause event (e.g., a scene of a movie that is paused) to determine relevant ads to be presented on display panel790for a period of or duration of the pause event. For example, ACR730may generate a fingerprint and/or watermark for the audio frame that enables determination of the content (e.g., the movie) including the audio frame, and data associated with the audio frame (e.g., actors, movie title, genre). The fingerprints, watermarks, and/or metadata (from computer vision740) generated can be used to determine relevant ad to be displayed on display device700for a portion of or the duration of the pause event.

In some embodiments, after display device700determines that a pause event is detected (e.g., based on a remote control pass through function, silent audio detected with unchanging video frames, and/or a pause icon and no motion of the video frames), display device700can utilize the information obtained from control signals (e.g., SPD, ALLM, and/or VRR) to determine one or more appropriate ads to be presented on display panel790during a portion or duration of the pause event. In some embodiments, display device700utilizes the content recognition data of one or more video frames (e.g., fingerprints and/or watermarks from ACR730and/or metadata from computer vision740) in conjunction with SPD, ALLM, and/or VRR to determine one or more appropriate ads to be presented on display panel790during a portion or duration of the pause event.

For example, after a pause event is detected (e.g., ad insertion750may receive a notice of the pause event detection from processor720), the corresponding fingerprint and/or watermark from ACR730, the corresponding metadata from CV740, SPD from product information function722, ALLM from latency mode function724, and/or VRR from refresh rate function726can be utilized by ad insertion750to select one or more relevant ads to be presented on display device700. For example, the fingerprint and/or watermark may correspond to a certain gaming application618. The metadata may correspond to the genre of gaming application618(e.g., magical creatures, sports arena, vehicles). The SPD may indicate the particular gaming system or gaming console and the model, and the ALLM may indicate low latency is needed along with a certain VRR for advanced gaming. The SPD can include a unique ID that corresponds to a particular hardware device and/or model of the hardware device. In some embodiments, ad insertion750determines one or more relevant ads. A relevant ad may include a new model gaming system, a competitor gaming system, and/or one or more gaming applications that utilize a corresponding ALLM and/or corresponding VRR within a similar genre. Many variations are possible. In some embodiments, ad insertion750transmits the corresponding fingerprint and/or watermark from ACR730, the corresponding metadata from CV740, SPD from product information function722, ALLM from latency mode function724, and/or VRR from refresh rate function726to network760(e.g., a cloud network) for the determination of the relevant ads and receive the relevant ads from network760.

The one or more relevant ads may be static (e.g., a still image), dynamic (e.g., a gif, a short video, an animation, an interactive screen), or a link for purchasing one or more items in the relevant ad. In some embodiments, input from user132regarding the interactive screen (e.g., a survey) can be collected and transmitted to crowdsource server128. In some embodiments in response to input corresponding to the one or more relevant ads, additional information may be transmitted to a mobile device corresponding to a customer account associated with display device700(e.g., email, text). In some embodiments the corresponding fingerprint and/or watermark from ACR730, the corresponding metadata from CV740, SPD from product information function722, ALLM from latency mode function724, VRR from refresh rate function726, and/or the one or more relevant ads may be stored in a historical profile corresponding to user132and/or media device600. Ad insertion750and/or network760may utilize the historical profile in conjunction with current data (e.g., fingerprint and/or watermark from ACR730, metadata from CV740, SPD from product information function722, ALLM from latency mode function724, and/or VRR from refresh rate function726, to determine current relevant ads.

Ad insertion750may transmit the one or more relevant ads to graphics module770to be added to a graphics plane775of display device700. The one or more video frames that are received may be stored in memory780, and processed by video scaling and FRC782. The one or more processed video frames can be added to video plane784of display device700(e.g., a last video frame used in the detection of the pause event can be added to video plane784). In some embodiments, video plane784and graphics plane775can be blended (e.g., merged) together and the data of the blended planes can be received by display panel790for presentation on display device700. In some embodiments graphics plane775may be displayed in front of video plane784(e.g., overlaid to be viewed on top of video plane784). In some embodiments the images and/or ads of graphics plane775can be transparent. In some embodiments a selection function from remote control110can be passed through media device600to display device700to make a selection from the graphics plane. The functions above may be performed by processor720, executing software of the various modules that may be stored in memory780, or a combination of hardware and software.

FIG.7Billustrates a block diagram of display device705, according to some embodiments. For illustration purposes and not a limitation, elements of display device705may be described with reference to elements from other figures in the disclosure. For example, functions of display device705may be similar to that of display device700ofFIG.7A, but network765includes the following functions: product information function722, latency mode function724, refresh rate function726, ACR730and/or computer vision740, determines relevant ads to be presented on display panel790, and transmits the relevant ads to ad insertion750.

FIG.9illustrates a block diagram of display device900, according to some embodiments. For illustration purposes and not a limitation, elements of display device900may be described with reference to elements from other figures in the disclosure. For example, functions of display device900may be similar to that of display device700ofFIG.7A. For example, HDMI input910, product information function922, latency mode function924, refresh rate function926, ACR930, computer vision940, processor920, ad insertion950, graphics module970, graphics plane975, video plane984, memory980, and video scaling and FRC982, and display panel990can correspond respectively to HDMI input710, product information function722, latency mode function724, refresh rate function726, ACR730, computer vision740, processor720, ad insertion750, graphics module770, graphics plane775, video plane784, memory780, and video scaling and FRC782, and display panel790ofFIG.7A.

In some embodiments, display device900can execute one or more functions product information function922, latency mode function924, refresh rate function926, ACR930and/or computer vision940via corresponding processor(s) (not shown) that are different from processor920. In some embodiments one or more of the functions product information function922, latency mode function924, refresh rate function926, ACR930and/or computer vision940may be performed by network960, processor920, and/or a different processor (not shown).

In some embodiments, HDMI input910receives video frames and control signals from media device600and/or media device800. In some embodiments, network960provides the video frames to HDMI input910.

FIG.4illustrates method400for a display device, according to some embodiments. For illustration purposes and not a limitation, method400may be described with reference to elements from other figures in the disclosure. For example, method400may be performed by a display device (e.g., display device108ofFIG.1andFIG.3, display device700ofFIG.7A, display device705ofFIG.7B, and/or display device900ofFIG.9).

At405, the display device can determine whether CEC (e.g., remote control pass through) functions are enabled. When CEC functions are enabled (e.g., set), method400proceeds to410. Otherwise, method400proceeds to435.

At410, display device108determines whether a pause key signal has been received. For example, user132may have selected a pause key on remote control110. Since CEC functions are enabled, the pause key signal passes through a media device (e.g., media device106ofFIG.1orFIG.2, media device600ofFIG.6, and/or media device800ofFIG.8) to a display device (e.g., display device108ofFIG.1andFIG.3, display device700ofFIG.7A, display device705ofFIG.7B, or display device900ofFIG.9) via an HDMI connection (e.g., HDMI connection107ofFIG.1) or HDMI input710, or HDMI input910. When a pause key signal has been received, the display device determines that a pause event has been detected, and method400proceeds to415and418. Otherwise, method400returns to405.

At415, when a pause event has been detected, the display device performs content recognition on the one or more video frames received from the media device106(e.g., the one or more video frames from which the pause event is determined). In some embodiments, when a pause event has been detected, the display device performs content recognition on the one or more audio frames received from the media device106(e.g., the one or more audio frames from which the pause event is determined). The content recognition may be performed by an ACR function (e.g., ACR Module320, ACR730, network760, ACR930, or Network960) that determines corresponding fingerprints and/or watermarks, or by a CV function (e.g., CV module325, computer vision740, network760, computer vision940, and/or network960) that determines corresponding metadata.

At418, when a pause event has been detected, the display device performs source device recognition and/or application recognition based on received control signals including but not limited to a product information signal, a latency mode signal, or a refresh rate signal. For example, a product information function enables the display device to determine the SPD that can include the type of gaming device or gaming platform (e.g., a video game console) and the model number. The latency mode function enables the display device to determine the ALLM, whether an application in use works best with low latency (e.g., gaming application618, input application816, or media application817). The refresh rate function enables the display device to determine the VRR and that the application in use may be an advanced (e.g., sophisticated) gaming application.

At420, the display device determines an appropriate ad corresponding to the determined fingerprints, watermarks, metadata, and/or information from the control signals (e.g., SPD, ALLM, VRR). For example, an ad insertion function (e.g., ad insertion module330, ad insertion750, or ad insertion950) can determine the appropriate ad corresponding to the fingerprints, watermarks, metadata, and/or information from the control signals (e.g., SPD, ALLM, VRR). In some embodiments, the ad insertion function communicates with a network (e.g., network118via communication device114, network760, or network960) to exchange the fingerprints, watermarks, metadata, and/or information from the control signals (e.g., SPD, ALLM, VRR) and/or receive one or more relevant ads to be displayed on the display device (e.g., display device108, display device700, display device705, or display device900) during the pause event. The one or more relevant ads may include a static or dynamic image.

In some embodiments, the display device transmits the video frames and/or control signals received via an HDMI connection to a network (e.g., network118, network760, and/or network960). In some embodiments the network performs one or more of the ACR functions, CV functions, and control signal analysis to determine the one or more fingerprints, watermarks, metadata, and/or information from the control signals (e.g., SPD, ALLM, VRR). Further, the network can determine an appropriate ad corresponding to the fingerprints, watermarks, metadata, and/or information from the control signals (e.g., SPD, ALLM, VRR). The network can transmit the appropriate ad to the ad insertion function (e.g., ad insertion module330, ad insertion750, or ad insertion950).

In some embodiments the network performs one or more of the ACR functions, CV functions, and control signal analysis to determine the one or more fingerprints, watermarks, metadata, and/or information from the control signals (e.g., SPD, ALLM, VRR). A processor (e.g., processor720, processor920, or other processor (not shown) of the display device performs the remaining functions that the network does not perform, and transmits the output to the network (e.g., directly or via the ad insertion function). The network utilizes the one or more fingerprints, watermarks, metadata, and/or information obtained from the control signals (e.g., SPD, ALLM, and/or VRR) to determine the one or more appropriate ads, and transmits the one or more appropriate ads to the ad insertion function.

In some embodiments, the processor (e.g., processor720, processor920, or other processor (not shown) of the display device performs the remaining functions that the network does not perform, and transmits the output to the ad insertion function. The ad insertion function also receives the one or more fingerprints, watermarks, metadata, and/or information obtained from the control signals (e.g., SPD, ALLM, and/or VRR) determined by the network. The ad insertion function uses the received data from the processor and the network to determine one or more appropriate ads.

The ad insertion function communicates the one or more relevant ads to be displayed on the display device (e.g., display device108, display device700, display device705, or display device900) during the pause event. The one or more relevant ads may include a static or dynamic image.

At425, the display device can add one or more of the one or more appropriate ads to a graphics plane of the display device. For example, the ad insertion function may transmit the one or more relevant ads to a graphics module (e.g., graphics module360, graphics module770, graphics module970) to be added to the graphics plane (e.g., graphics plane775, graphics plane975).

At430, the display device presents the one or more appropriate ads during the pause event. For example, the graphics plane and the video plane may blended (e.g., aligned) and a display module (e.g., display module370, display panel790, display panel990) may present the one or more appropriate ads on a screen of the display device. In some embodiments, the display device determines that the pause event ends, and the display device stops presenting the one or more appropriate ads on the screen. The determination that the pause event ends can include but is not limited to: receiving a play key input (e.g., when CEC is set) from remote control110; detecting that silent audio is not received or that video frames are changing; and/or determining that a pause icon is no longer detected or motion is detected in the video frames (e.g., via a CV function.)

Returning to435, when CEC functions are not set, the display device can determine whether a silent audio signal is detected. When a silent audio signal is detected, method400proceeds to440. Otherwise, method400proceeds to455.

At440, when silent audio has been detected, display device108determines whether one or more video frames is unchanging (e.g., stopped). For example, processing module340can perform periodic sampling on a received video frame. In some examples, the determination of the change may be by performing content recognition functions periodically. When the video frame is determined to be unchanging, method400proceeds to415. When the video frames are changing, method400returns to405to detect a pause event.

At455, the display device determines whether a pause icon is detected among the one or more video frames received. For example, the CV function can analyze the content of one or more video frames to recognize a pause icon. When a pause icon is detected, method400proceeds to460. Otherwise, method400returns to405.

At460, display device108determines whether there is a change in the one or more audio and/or video frames received. For example, the CV function can determine whether the content and/or context of one or more of the video frames received changes. In some embodiments, an ACR function determines based on video and/or audio frames whether the content of the video and/or audio frames received have changed. Display device can use the corresponding fingerprints, watermarks, and/or metadata to determine whether content has changed. In some examples, the determination of the change may be performed by content recognition functions periodically. When no change in the audio and/or video frames is detected method400proceeds to415and/or418. Otherwise, method400returns to405.

Example Computer System

Various embodiments may be implemented, for example, using one or more well-known computer systems, such as computer system500shown inFIG.5. For example, media device106, media device600, media device800, display device108, display device700, display device705, or display device900, and/or method400may be implemented using combinations or sub-combinations of computer system500. Also or alternatively, one or more computer systems500may be used, for example, to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof.

Computer system500may include one or more processors (also called central processing units, or CPUs), such as a processor504. Processor504may be connected to a communication infrastructure506(that can be a bus).

Computer system500may also include user input/output device(s)503, such as monitors, keyboards, pointing devices, etc., which may communicate with communication infrastructure506through user input/output interface(s)502.

Computer system500may also include a main or primary memory508, such as random access memory (RAM). Main memory508may include one or more levels of cache. Main memory508may have stored therein control logic (i.e., computer software) and/or data.

Computer system500may also include one or more secondary storage devices or memory510. Secondary memory510may include, for example, a hard disk drive512and/or a removable storage device or drive514. Removable storage drive514may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and/or any other storage device/drive.

Removable storage drive514may interact with a removable storage unit518. Removable storage unit518may include a computer usable or readable storage device having stored thereon computer software (control logic) and/or data. Removable storage unit518may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/any other computer data storage device. Removable storage drive514may read from and/or write to removable storage unit518.

Computer system500may further include a communication or network interface524. Communication interface524may enable computer system500to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference number528). For example, communication interface524may allow computer system500to communicate with external or remote devices528over communications path526, which may be wired and/or wireless (or a combination thereof), and which may include any combination of LANs, WANs, the Internet, etc. Control logic and/or data may be transmitted to and from computer system500via communication path526.

In some embodiments, a tangible, non-transitory apparatus or article of manufacture comprising a tangible, non-transitory computer useable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system500, main memory508, secondary memory510, and removable storage units518and522, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system500or processor(s)504), may cause such data processing devices to operate as described herein.

CONCLUSION