Abstract:
Aspects relate to transmission of metadata from a source to a sink device, and optionally through one or more intermediaries. A source device encodes metadata into what would have been a blanking area of a field to be transmitted, according to a current video format. The source device encodes a timing for an active video data signal that is modified from a timing that would be used only for transmission of video data at a current resolution. A separate indicator from the source, or a negotiation between source and sink allows the sink to determine what part of the data indicated as being active video data is metadata, and to use that metadata for controlling aspects of the video display, and to use other parts of the received video data as video data for display. A sink can signal supported capabilities to a source.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application claims priority from U.S. Provisional App. No. 62/075,554, filed on Nov. 5, 2015, which is incorporated by reference in its entirety for all purposes. 
    
    
     BACKGROUND 
     Field 
     The following relates, in one aspect, to systems and methods for transferring metadata from one device to another over standard AudioVideo (AV) interfaces. 
     Related Art 
     Transmission of AV data from a source device to a sink device is typically implemented according to a pre-defined standard. Providing a standard for communication of such data allows source and sink devices to be built to the standard by different entities and have an expectation that those devices will be able to be used together. 
     For example, High Definition Multimedia Interface (HDMI) provides a standard that is widely adopted and used in devices for AV data communication. HDMI implements EIA/CEA standard video formats and waveforms. Digital video signals can be broken into three main parts: the vertical blanking region, the horizontal blanking region, and the active picture region. The vertical blanking region contains the vertical sync pulse that signifies a new field or frame (field and frame are used interchangeably herein, unless explicitly indicated otherwise). There are multiple lines of video data in each field. 
     For example, a Full High-Definition (HD) 1080p/60 Hz video stream has a vertical blanking interval every 16.67 ms (60 Hz) to signify a new frame and a horizontal blanking interval every 14.8 μs to signify a new horizontal line. Within the 14.8 μs horizontal line duration there are 2,200 pixels, which include the horizontal blanking period and the active picture. This requires a pixel clock of 148.5 MHz (1/14.8 μs horizontal line duration*2,200 pixels) to generate the video frame composed of the three regions. 
     The active picture region is the location of the pixels that a user would see on the display. The pixel color value is stored in three channels that are typically RGB- or YUV (YPbPr)-encoded. The color levels can be encoded in a defined number of bits per channel, such as from between 6 to 16 bits per channel, meaning that each pixel has a 24- to 48-bit color value. A digital monitor communicates Enhanced Display Identification Data (EDID) to a source, in order to describe what video formats and timings that monitor supports. 
     Information about a current video format is communicated from a source to a sink. In HDMI, a video format is indicated as a standard VIC code in an InfoFrame. An InfoFrame is a packet of data that is provided in a vertical blanking interval (IE: in a period where the video signaling indicates that there is no active video data). Resolution also can be communicated via other mechanisms, such as a DCDB record (as in SuperMHL). 
     For example,  FIG. 1  shows a timing diagram that illustrates how audio and video data are transmitted and timed relative to each other for a frame in HDMI.  FIG. 1  depicts an active video area  110 , an area containing audio sample packets  115 , and unused blanking areas  120   a  and  120   b .  FIG. 1  also depicts that part of the blanking area, e.g., blanking area  120   a  can be filled with infoframe and data island packets. HDMI interleaves video, audio and auxiliary data using three different packet types, called the Video Data Period, the Data Island Period and the Control Period. During the Video Data Period, the pixels of an active video line are transmitted. Data Island packets can be transmitted in a pre-defined location  122  within a blanking interval used for Infoframe and data island packet transmission. 
     SUMMARY 
     In one aspect, implementations of the disclosure provide a robust, end-to-end mechanism for signaling metadata from one device to another. Metadata is data about the main payload or other information that is used in formatting, processing, or interacting with the main payload. For example, metadata may include sampling formats. 
     In one implementation, the active video area is extended into the regions that are not indicated as being active video data, such as the blanking area. In HDMI or MHL, a video data preamble is used to distinguish video data from other data (a portion of a frame transmission period during which data is encoded as video is called a “video period”). A video data preamble contains character sequence(s) that are not permitted in a video data period. This disclosure encodes non-video data into the video data period, but at a physical layer level, this non-video data is encoded and transmitted indistinguishably from video data. 
     In one implementation, character sequences indicative of a start of a video period may be provided earlier and/or provided later than what would be required purely based on the timing required for the video itself. These extra lines appear to HDMI/MHL receivers and transmitters as video data, and thus there is no need to have HW changes in receivers (sinks), transmitters (sources), including at bridge and switch nodes. 
     A difference between the standard (expected/determined according to the video mode) and the modified timing can be signaled using an existing source to sink communication method. In such an approach, to determine the timing of actual video data, a sink has to know VIC code and the information on where and how many metadata lines are added 
     In an aspect, hardware forward error correction (FEC) can be added for metadata only. In such implementation, a CPU can perform the FEC processing in a software-implemented stack. FEC could be provided for all Active Lines, and then FEC would likely be implemented with hardware acceleration. Other approaches to signaling a change to an active video period can be provided for different audio/video data transmission systems and approaches. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  depicts a prior art approach of encoding metadata into non-video areas (data islands) of a frame transmission; 
         FIG. 2  depicts a prior art approach to an HDMI sink that receives video and metadata transmitted according to  FIG. 1  and separately sends the metadata from data islands to a CPU for processing while video data travels over a different path to a video processor; 
         FIG. 3  depicts that in implementations of the disclosure, video and metadata can be received and transmitted to a video processor over the same bus; 
         FIG. 4  depicts an example implementation of extending an active video area in order to encode metadata as video data in the extended active video area(s); 
         FIG. 5  depicts an example implementation of a source that encodes metadata according to the disclosure; 
         FIG. 6  depicts an example implementation of a sink that extracts metadata according to the disclosure; 
         FIG. 7  depicts an example implementation of a sink that supports High Dynamic Range (HDR) processing with an HDR decoder in a system on chip (SOC) that receives HDR metadata over an SPI interface from a port processor that separates metadata within an active video area from actual video data; 
         FIG. 8  depicts an example implementation of a sink that supports HDR processing with an HDR decoder in a port processor, where the HDR decoder uses metadata from an active video data area; and 
         FIG. 9  depicts an example implementation of a sink that supports High Dynamic Range (HDR) processing with an HDR decoder in a system on chip (SOC) that receives video and metadata from a framebuffer, such that video and metadata transmitted as video data can be handled together. 
     
    
    
     DETAILED DESCRIPTION 
     As shown in  FIG. 1 , HDMI and MHL provide a capability to transmit Infoframes within a blanking area  120   a ,  120   b . InfoFrame have a relatively small size limit of 27 (31) bytes. This leads to situations where metadata may need to be sent as fragments in multiple infoframes, and then the fragments need to be reassembled. Fragmentation means that less information can be sent, and also that multiple fields may be needed to signal all of the metadata required to indicate a change in some property of the video data. Fragmentation also leads to complications in a sink node, which must extract metadata from multiple infoframes and reassemble the metadata. Infoframes also have a fixed size, which means that sources must pad a data length, in situations which metadata to be communicated is less than the size of the infoframe. 
     With reference to  FIG. 2 , in modern devices, InfoFrames also are de-facto asynchronous, in that there is no standard transport mechanism for sending Data Islands over a parallel video bus (i.e., on the same channel as video and audio data) and instead video and InfoFrames (e.g., with metadata) follow different paths within a processing system.  FIG. 2  depicts that HDMI encoded video and metadata arrives at a HDMI sink or repeater  150 . 
     HDMI sink or repeater  150  contain a receiver  152  that is capable of operating the physical layer of HDMI, and extracting the video and metadata. Extracted video  153  is transferred over a parallel bus  155  to a video processor (if a sink) and to an HDMI transmitter (if a repeater)  154 . Metadata  160  is received by a CPU  156  and then transferred to the video processor/HDMI transmitter  154 . As shown, transfer of metadata can involve transfer through CPUs and interfaces, such as Inter Integrated Circuit Bus (I2C) make it difficult to pass metadata synchronously with video. In some cases, one or more repeaters may exist between a source and a sink. Each repeater between the source and the sink can increase the problem of de-synchronization. In  FIG. 2 , the metadata  158  is extracted from data islands that are located in pre-defined areas of the video blanking area as shown with respect to  FIG. 1 . 
       FIG. 3  depicts a first example of how an HDMI sink or repeater  175  would handle metadata according to the disclosure. As in  FIG. 2 , HDMI formatted video and metadata  151  arrives at an HDMI sink or repeater  175 . However, HDMI receiver  152  in  FIG. 3  does not separately extract metadata from data islands, as in  FIG. 2 . Rather, HDMI receiver  152  simply receives signals that are encoded as video and places the data contained in those signals on the parallel bus  155 . In  FIG. 3  however, the video signals also encode metadata that would have been encoded in data islands according to prior approaches to metadata transfer. Video processor or HDMI transmitter  154  can receive the video and metadata (signaled as video)  182  from bus and either process (if a sink) or transmit (if a repeater). Thus,  FIG. 3  presents an improvement over approaches according to  FIG. 2 , in that metadata does not need to be separated from video and transmitted via a separate channel, and then returned to video processor/HDMI transmitter. Rather, video data has the same path as metadata within a device and the metadata information can be transferred over video bus  155  as if it is actual video. 
     One advantage of implementations according to  FIG. 3  is that the involvement of CPU  156  in metadata transfer may be eliminated, which may allow for a lower power and/or cheaper CPU to be used in HDMI sink or repeater  175 , for example. Approaches according to  FIG. 3  can lead to further advantages. One advantage is that significantly more metadata may be encoded with one field or frame than what can be encoded in an InfoFrame or a data island packet. Also, synchronization between video and metadata is naturally maintained throughout the entire video path, because metadata is treated as video data. The approach is compatible with current HDMI and MHL hardware architecture, although some hardware may require a software or firmware update as described below. 
       FIG. 4  depicts an example of how video fields can be encoded in order to make use implementations according to  FIG. 3 .  FIG. 4  depicts that rather than including metadata in data islands, metadata is encoded into an extended portion  400   a ,  400   b  of the active video area (two extensions are depicted here, but implementations may include only  400   a  or  400   b ). In other words, one or more blanking intervals are shortened, and the active video area is increased. Metadata is encoded as though it represents video data for display. The active video area can be extended in HDMI by manipulating a timing of when character sequences indicative of the Video Data Period are provided. 
     Within a receiver, a Data Enable (DE) signal can be generated in response to detection of the character sequences indicative of the Video Data Period. A level of the DE signal can indicate whether data being transmitted is or is not video data. Thus, in one approach, the Video Data Period begins earlier and/or continues longer (and consequently, a DE signal would be brought high earlier and/or maintained high later while preserving other video timing) in order to extend what HDMI receiver  152  would identify as video data. This approach to metadata transmission does not need to take the place of data islands. Metadata could be transferred in data islands also, or data islands could be used for encoding or transmitting other data. 
     Particulars of how metadata can be encoded into the extended active video area can vary among implementations, although a standardized approach would allow increased inter-operability. In one approach, metadata can be encoded directly into the extended (also can be called “overscan” area in that the extended “video” area is not intended for actual display) or into Least Significant Bits, or at fixed locations, such as particular lines, or fixed positions within lines in the extended area. An implementation can use a similar structure to data islands. A generic packet structure can be defined to carry various metadata payloads. To be clear, the approach to encoding metadata into the extended active video area in an implementation decision, in that transmitters and receivers will treat the physical transmission and reception of such metadata simply as video data. 
     In one implementation, one HDMI/MHL field in a sequence will be encoded with metadata for a subsequent-in-display-order (e.g., a next) field. Extracted metadata is kept in a buffer until a VSYNC pulse. On the pulse, the stored metadata is applied to configure processing for the next field or frame to be displayed (e.g., to determine resolution, color encoding, and so on). 
     Some existing repeaters may need a SW or firmware update to allow the capabilities propagate from sink to source and be ready to tolerate the additional “video” lines. 
     In one approach, a standard Video Identification Code (VIC), provided in an AVI InfoFrame (for HDMI) or in a Device Connection Database (DCDB) for MHL indicates video resolution (i.e., the active video area), which indicates the number of video data lines being transmitted for each field. A Vendor Specific InfoFrame (VSIF) (for HDMI) or a DCDB for MHL is used to indicate modified video timing due to encoding of metadata as active video data. 
     A sink can indicate capability to support metadata encoding according to the disclosure to a source by transmitting a capability indication through a control channel. For example, using a display data channel via SCDC for HDMI or control bus via DCDB for MHL. In one approach, the sink&#39;s support for receiving metadata as active video and also support for each separate capability, such as Dynamic HDR, is also separately indicated. 
     Examples of an encoding format for metadata is a structure that allows embedding different sub-structures that communicate different kinds of metadata. Examples of metadata that can be provided from a source include metadata about High Dynamic Resolution (HDR), Display Stream Compression (DSC) PPS. HDMI includes a capability indication SCDC (for HDMI) and DCDB (for MHL) need to have a flag indicating sink ability to receive the metadata in video lines. Each sub-structure, within the general structure, can have a separate capability indication flag, for example, a sub-structure for Dynamic HDR can have a separate flag to indicate support for that capability. 
     An example data line structure can support that all pixels in a line are used for metadata encoding. So, a number of bits of metadata that can be stored per pixel and per line would vary based on resolution and color depth. In each video field, information in each next metadata line can be a continuation of the information of the previous metadata line. In one approach, metadata encoding is not continued across boundaries of fields, but in another implementation, metadata can be continued from one field to another. 
     Forward Error Correction (FEC) can be used to provide error detection and/or error correction capability for the metadata. FEC could be implemented in software on a CPU for only the metadata. If FEC were used for the entire active video area (including metadata), then hardware support would be desirable. 
     The following figures present various examples of how embodiments according to the disclosure can be implemented.  FIG. 5  depicts an example HDMI source System on Chip (SOC)  202 . A source of MPEG content  204  (MPEG content source  204 ) outputs an MPEG stream  206  to a one frame delay  208 , while metadata from the MPEG content is loaded by a metadata encoder  205  into an extended frame buffer  216 . MPEG stream  210  is output from one frame delay  208 , and provided to MPEG decoder  212 . Uncompressed video  214  is then loaded into extended frame buffer  216  in appropriate locations, and data from the frame buffer is transferred, such as over a parallel system bus) to an HDMI transmitter  220 , which outputs an HDMI signal, including a DE signal with timing modified to reflect the extension of the active video area in which metadata is encoded. While this example is of a one-frame delay of video to metadata (i.e., metadata leads video data by one frame), other implementations can be provided. For example, a zero frame delay can be implemented, such that metadata transmitted can be applied directly to video data in that same frame. A multiple frame delay (n frames) can be implemented. An amount of delay can be specified by the metadata as a field. A sequence or selection of frames to which particular metadata is to be applied can be specified; different metadata for different frames in a sequence can be specified. 
       FIG. 6  depicts an HDMI receiver  232  receiving an HDMI signal  230  and providing active video data (which includes encoded metadata in an extended active video region) over a bus  234  to an HDMI sink SOC  225 . Sink  225  includes metadata extraction circuitry  236 , which extracts metadata  244  from extended active regions and causes metadata  244  to be stored in a metadata buffer  242 , and also transmits video  240  to a video processor  250 . Metadata  244  is outputted from metadata buffer  242  in response to a VSYNC  246  received through HDMI receiver  232 . Metadata  244  is parsed through a metadata processing circuitry to extract relevant features (such as dynamic HDR data) and then provides that parsed metadata (extracted features) for configuring a video processor  250  to process a subsequent field of video data that will be received by HDMI receiver  232 . An LCD panel  255  receives output video data from video processor  250 . 
       FIGS. 7-9  depict example implementations of sinks that can receive and use metadata according to the disclosure. In particular, these figures depict different locations at which metadata extraction and application of metadata to video data can be performed.  FIG. 7  depicts an HDMI or MHL signal  306  being received at a port processor  305 . Port processor  305  separates metadata from extended active video regions over a fast System Peripheral Interface (SPI) bus to a System on Chip ( 316 ) that includes an HDR decoder  318  that consumes the metadata for its configuration. Using SPI is an example. Video data  314  extracted by port processor  305  is transmitted by an embedded Transition Minimized Differential Signal (eTMDS) channel  312  to SOC  316 , which stores video data  314  in frame buffer  322 . Video  325  is output from frame buffer  322  to HDR decoder  318  for processing and then output to LCD panel  320 . Transfer of video data  314  to and from frame buffer  322  can be accomplished by a variety of approaches, including Direct Memory Access (DMA) circuitry that can be located in SOC  316 . 
       FIG. 8  depicts an alternate approach in which like circuitry or features are given like numbers.  FIG. 8  depicts that HDR decoder  318  can be located in port processor  305 . In such case, metadata extracted by port processor can be provided to HDR decoder  318  within port processor, and does not need to transit eTMDS  312 . HDR decoder  318  applies the metadata for video processing and outputs decoded and processed video data  360  over eTMDS  312  to frame buffer  322 , which then supplies decoded video  362  for display on LCD panel  320 . Decoded video data  360  and  362  are given different numbers to indicate that certain video data may be retrieved (e.g. for one frame in a sequence) while other video data is being stored for a subsequent frame). In the example of  FIG. 8 , separate transfer of metadata is not required to SoC  316 . 
       FIG. 9  presents a third example implementation where like circuitry is given like numbers.  FIG. 7  depicted that port processor  305  extracted metadata from the extended active video region and transmitted metadata separately to SOC  316 .  FIG. 8  depicted that HDR decoder  318  could be located in port processor  305 .  FIG. 9  presents an example where port processor  305  performs receiver functions, but transmits all data identified as active video data (both actual video data and metadata encoded in an extended active video region) over eTMDS  312  to SOC  316 , for storage in frame buffer  322  and subsequent retrieval. HDR decoder  318  then extracts and processes that metadata from the extended active video region. In the example where metadata encoded in one field is applied in to a subsequent field, HDR decoder  318  buffers the metadata and then effects changes indicated by the metadata in order to decode the subsequent field.