Patent Description:
To service compressed moving images by way of broadcasting, networks, or the like, there is an upper limit on replayable frame frequency depending on decoding capability of a receiver. Therefore, service providers need to limit their services to low-frame frequency services or provide concurrently high-frame frequency services and low-frame frequency services, with consideration given to replaying capabilities of the prevailing receivers.

To correspond to high-frame frequency services, the receivers become higher in cost, which is a disincentive to popularization. When inexpensive receivers dedicated to low-frame frequency services are initially in widespread use and service providers start high-frame frequency services in the future, customers cannot receive the high-frame frequency services without new receivers, which is a disincentive to proliferation of the new services.

For example, there is proposed time-direction scalability by subjecting image data of pictures constituting moving image data to hierarchical encoding by high efficiency video coding (HEVC) (refer to Non-patent Document <NUM>). At the reception side, the levels of the pictures can be identified based on temporal ID (temporal_id) information inserted in the header of a network abstraction layer (NAL) unit, which allows selective decoding up to the level corresponding to decoding capability.

Document <CIT> discloses a video playback apparatus capable of adaptively setting a speed of slow playback in accordance with a frame rate of video content that is being played back.

An object of the subject technique is to allow favorable decoding at the reception side.

A concept of the subject technique lies in a transmission device including:.

According to the subject technique, the image encoding unit encodes the image data of the pictures constituting the moving image data to generate the video stream (encoded stream). In this case, the image data of the pictures constituting the moving image data is classified into a plurality of levels and encoded to generate the video stream having the image data of the pictures at the respective levels. The hierarchical composition is equalized between the low-level side and the high-level side. Corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded.

The transmission unit transmits the container in the predetermined format including the foregoing video stream. For example, the image encoding unit may generate a single video stream having the encoded image data of the pictures at the respective levels or divide the plurality of levels into two sets of the upper-level side and the lower-level side and generate two video streams having the encoded image data of the pictures in the respective level sets.

According to the subject technique, the hierarchical composition is equalized between the low-level side and the high-level side, and corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded. This allows the reception side to decode the encoded image data of the pictures on the low-level side and the high-level side with a smaller buffer size and a reduced decoding delay.

In the subject technique, for example, a hierarchical information insertion unit that inserts hierarchical information into a layer of the container may further be included. In this case, for example, the hierarchical information may have information on level specified values for the respective levels. In addition, in this case, for example, the hierarchical information insertion unit may insert the hierarchical information into the layer of the container at positions in synchronization with the encoded image data of the pictures in the video stream.

For example, the hierarchical information insertion unit may insert the hierarchical information into an extension field of a PES packet. In this case, the hierarchical information insertion unit may insert the hierarchical information into the extension field of the PES packet at least for each coded video sequence. In addition, in this case, for example, an information insertion unit that inserts information for describing explicitly whether the hierarchical information is inserted into the extension field of the PES packet may further be included under a program map table.

In addition, for example, the hierarchical information insertion unit may insert the hierarchical information under a program map table. In addition, for example, the hierarchical information insertion unit may insert the hierarchical information under an event information table.

The hierarchy information is inserted in the layer of the container, and the reception side can refer to the hierarchy information to retrieve selectively from the video stream the encoded image data of the pictures up to the level commensurate with the capability of the decoder in an easy manner.

In addition, another concept of the subject technique lies in a reception device including a reception unit that receives a container in a predetermined format that contains a video stream having encoded image data of pictures obtained by classifying image data of the pictures constituting moving image data into a plurality of levels and encoding the same, wherein.

According to the subject technique, the reception unit receives the container in the predetermined format. The container contains the video stream having image data of the pictures at the respective levels obtained by classifying the image data of the pictures constituting the moving image data into a plurality of levels and encoding the same. In this case, in the process of encoding, the hierarchical composition is equalized between the low-level side and the high-level side, and corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded.

The processing unit processes the received container. For example, the processing unit may be configured to retrieve selectively the encoded image data of the pictures at a predetermined level and lower ones from the video stream and decode the same based on the hierarchy information, thereby obtaining the image data of the pictures at the predetermined level and lower ones.

As described above, according to the subject technique, in the video stream contained in the received container, the hierarchical composition is equalized between the low-level side and the high-level side, and corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded. This makes it possible to decode the encoded image data of the pictures on the low-level side and the high-level side with a smaller buffer size and a reduced decoding delay.

According to the subject technique, hierarchical information may be inserted into a layer of the container, and
the processing unit may retrieve selectively from the video stream the encoded image data of the pictures at a predetermined level and lower ones and decode the same, based on the hierarchical information, to obtain the image data of the pictures at the predetermined level and lower ones. In this case, it is easy to retrieve selectively from the video stream the encoded image data of the pictures at the level commensurate with the capability of the decoder in an easy manner.

According to the subject technique, the reception side can perform favorable decoding. The advantages of the technique are not limited to the ones described here but may be any of advantages described in the subject disclosure.

An embodiment for carrying out the invention (hereinafter, referred to as "embodiment") will be described below. The descriptions will be given in the following order:.

<FIG> illustrates a configuration example of a transmission/reception system <NUM> as an embodiment. The transmission/reception system <NUM> has a transmission device <NUM> and a reception device <NUM>.

The transmission device <NUM> transmits a transport stream TS as a container carried on broadcast waves or in packets over a network. The transport stream TS contains a video stream in which image data of pictures constituting moving image data is classified into a plurality of levels and encoded data of the image data of the pictures at the respective levels is included. In this case, the transport stream TS contains a single video stream having the encoded image data of the pictures at the respective levels or two video streams in which the plurality of levels is divided into two sets of high-level side and low-level side and the encoded image data of the pictures at the respective level sets is included.

For example, the referenced pictures are encoded according to H. <NUM>/AVC or H. <NUM>/HEVC such that they belong to their levels and/or lower ones. In this case, the hierarchical composition is equalized between the low-level side and the high-level side, and corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded. Performing such encoding allows the reception side to decode the encoded image data of the pictures on the low-level side and the high-level side with a smaller buffer size and a reduced decoding delay.

Level identification information is added to the encoded image data of the pictures at the respective levels to identify the levels to which the pictures belong. In the embodiment, the level identification information ("nuh_temporal_id_plus1" indicative of temporal_id) is added to the headers of the NAL units (nal_unit) of the pictures. Adding the level identification information allows the reception side to retrieve selectively the encoded image data at the predetermined level and lower ones for decode processing.

Hierarchical information including information on level specified values at the respective levels and others is inserted into the layer of the container. The reception side can refer to the hierarchical information to retrieve selectively from the video stream the encoded image data of the pictures at the level commensurate with the capability of the decoder in an easy manner. For example, the hierarchical information is inserted under a program map table (PMT) or under an event information table (EIT).

In addition, the hierarchical information is inserted into PES extension fields of the headers of PES packets at positions in synchronization with the encoded image data of the pictures in the video stream, for example. This allows the reception side to, even with changes in the hierarchical composition, retrieve selectively from the video stream the encoded image data of the pictures at the level commensurate with the capability of the decoder.

When the hierarchical information is inserted into the extension field of the PES packet as described above, identification information indicating that the hierarchical information is inserted into the extension field of the PES packet is inserted under the program map table. In this case, the reception side can identify a situation that the hierarchical information is inserted into the extension field of the PES packet based on the identification information.

The reception device <NUM> receives the transport stream TS sent from the transmission device <NUM> on broadcast waves or in packets over a network. The reception device <NUM> processes the transport stream TS. In this case, the reception device <NUM> retrieves selectively from the video stream the encoded image data of the pictures at a predetermined level and lower ones commensurate with the capability of the decoder and decodes the same based on the hierarchy information contained in the layer of the container, thereby obtaining the image data of the pictures at the predetermined level and lower ones.

<FIG> illustrates a configuration example of the transmission device <NUM>. The transmission device <NUM> has a central processing unit (CPU) <NUM>, an encoder <NUM>, a compressed data buffer (cpb: coded picture buffer) <NUM>, a multiplexer <NUM>, and a transmission unit <NUM>. The CPU <NUM> is a control unit that controls the operations of the components of the transmission device <NUM>.

The encoder <NUM> inputs uncompressed moving image data VD to perform hierarchical encoding. The encoder <NUM> classifies the image data of pictures constituting the moving image data VD into a plurality of levels. Then, the encoder <NUM> encodes the classified image data of the pictures at the respective levels to generate a video stream having the encoded image data of the pictures at the respective levels.

The encoder <NUM> performs encoding such as H. <NUM>/AVC or H. <NUM>/HEVC. At that time, the encoder <NUM> performs encoding such that the referenced pictures belong to their levels and/or lower ones. The encoder <NUM> also divides the plurality of levels into low-level side and high-level side, and equalizes the hierarchical composition between the low-level side and the high-level side, and combines corresponding pictures on the low-level side and the high-level side into one set and encodes the same sequentially.

<FIG> illustrates an example of hierarchical encoding performed by the encoder <NUM>. In this example, the image data of the pictures is classified into six levels of <NUM> to <NUM>, and is subjected to encoding.

The vertical axis indicates the levels. The values <NUM> to <NUM> are set as the temporal_id (level identification information) added to the headers of the NAL units (nal_unit) constituting the encoded image data of the pictures at the levels <NUM> to <NUM>. Meanwhile, the horizontal axis indicates the picture order of composition (POC), the display time is earlier with increasing proximity to the left side and is later with increasing proximity to the right side.

<FIG> illustrates a structural example (Syntax) of the NAL unit header, and <FIG> illustrates the contents (Semantics) of major parameters in the structural example. The <NUM>-bit field "Forbidden_zero_bit" is essentially <NUM>. The <NUM>-bit field "Nal_unit_type" indicates the NAL unit type. The <NUM>-bit field "Nuh_layer_id" is <NUM> as a precondition. The <NUM>-bit field "Nuh_temporal_id_plus1" indicates temporal_id and takes the value increased by one (<NUM> to <NUM>).

Returning to <FIG>, rectangular frames indicate pictures, and numbers in the rectangular frames indicate the order of coded pictures, that is, the encoding order (the decoding order at the reception side). For example, eight pictures of "<NUM>" to "<NUM>" constitute a sub group of pictures, and the picture "<NUM>" becomes the first picture in the sub group of pictures. Several sub groups of pictures are collected into a group of pictures (GOP).

In this example, three levels of <NUM> to <NUM> are on the low-level side and three levels of <NUM> to <NUM> are on the high-level side. As illustrated in the drawing, the hierarchical composition is equalized between the low-level side and the high-level side, and corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded. For example, the picture "<NUM>" on the low-level side and the picture "<NUM>" on the high-level side are first combined into one set and subjected to encoding, and then the picture "<NUM>" on the low-level side and the picture "<NUM>" on the high-level side are combined into one set and subjected to encoding. The pictures at the following levels are encoded in the same manner. In this case, the low levels are limited to levels lower than a specific level. Accordingly, to decode the pictures at the low levels, only the pictures at the limited low levels can be decoded and displayed in a stable manner. This matter also applies even when the pictures are not divided into the low levels and the high levels.

Returning to <FIG>, solid-line and broken-line arrows indicate the reference relationships between the pictures in encoding. For example, the picture "<NUM>" is an intra picture (I picture) that needs no reference to other pictures, and the picture "<NUM>" is a P picture that is encoded with reference to the "<NUM>" picture. The picture "<NUM>" is a B picture that is encoded with reference to the "<NUM>" picture and a picture in the previous sub group of pictures (not illustrated). The picture "<NUM>" is a B picture that is encoded with reference to the "<NUM>" and "<NUM>" pictures. Similarly, the other pictures are encoded with reference to pictures close to them in the picture order of composition. The code "D" indicates how much each picture is distant from the referenced picture in the picture order of composition. Without the indication of "D," D = <NUM>.

<FIG> illustrates another example of hierarchical encoding performed by the encoder <NUM>. Although no detailed description will be provided, in the example of <FIG>, the picture order of composition on the high-level side is one picture behind the picture order of composition on the low-level side, whereas, in the example of <FIG>, the picture order of composition on the high-level side is one picture ahead of the picture order of composition on the low-level side. In this case, the hierarchical composition is equalized between the low-level side and the high-level side, and corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded.

The encoder <NUM> generates a video stream having the encoded image data of the pictures at the respective levels. For example, the encoder <NUM> generates a single video stream having the encoded image data of the pictures at the respective levels or generates two video streams having the encoded image data of the pictures on the upper-order level side and the lower-order level side.

<FIG> illustrates a configuration example of encoded image data of the pictures. The encoded image data of the first picture of the GOP is composed of NAL units of AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS. Meanwhile, the pictures other than the first picture of the GOP are composed of NAL units of AUD, PPS, PSEI, SLICE, SSEI, and EOS. The unit VPS can be transmitted together with the unit SPS once per sequence (GOP), and the unit PPS can be transmitted for each picture. The unit EOS may not exist.

The bit stream level specified value "general_level_idc" is inserted into the sequence parameter set (SPS). In addition, when the pictures belonging to the levels indicated by "temporal_id" are bound into sub layers (sub_layer) and "Sublayer_level_presented_flag" is set to "<NUM>," the bit rate level specified value for each sub layer "sublayer_level_idc" can also be inserted into the SPS. This matter is applied to not only the SPS but also the VPS.

For example, the example of hierarchical encoding illustrated in <FIG> will be discussed. The value of "general_level_idc" inserted into the SPS is a level value including all the pictures at the levels <NUM> to <NUM>. For example, when the frame rate is 120P, the value is "Level <NUM>. " The value of "sublayer_level_idc[<NUM>] " inserted into the SPS becomes a level value including only the pictures at the levels <NUM> to <NUM>. For example, when the frame rate is 60P, the value is "Level <NUM>.

Returning to <FIG>, the compressed data buffer (cpb) <NUM> accumulates temporarily the video stream containing the encoded image data of the pictures at the respective levels generated by the encoder <NUM>. The multiplexer <NUM> reads the video stream accumulated in the compressed data buffer <NUM>, turns the same into a PES packet, and further turns the same into a transport packet to multiplex the same, thereby obtaining a transport stream TS as a multiplexed stream. The transport stream TS contains one or more video streams as described above.

The multiplexer <NUM> inserts the hierarchical information into the layer of the container. The transmission unit <NUM> transmits the transport stream TS obtained by the multiplexer <NUM> on broadcast waves or in packets over a network to the reception device <NUM>.

The insertion of the hierarchical information by the multiplexer <NUM> will be further explained. The multiplexer <NUM> inserts the hierarchical information to the layer of the container by any of the following methods (A), (B), and (C), for example:.

The transport stream TS contains a PMT as program specific information (PSI). The PMT has a video elementary loop (video ES1 loop) with information related to each video stream. In the video elementary loop, information such as stream type and packet identifier (PID) is arranged and descriptors describing information related to each video stream are also arranged in correspondence with the video stream.

The multiplexer <NUM> inserts a layer decoding descriptor (Layer_decoding_descriptor) newly defined as one of the descriptors. <FIG> illustrate a structural example (Syntax) of the layer decoding descriptor. <FIG> illustrates the contents (Semantics) of major information in the structural example.

The <NUM>-bit field "Layer_decoding_descriptor_tag" indicates descriptor type, and in this example, layer decoding descriptor. The <NUM>-bit field "Layer_decoding_descriptor_length" indicates the length (size) of the descriptor as the number of bytes of the subsequent "layer_information().

<FIG> illustrates a structural example (Syntax) of "layer_information(). " The <NUM>-bit field "layer-minimum LMI" indicates the level (layer) indicated by the minimum value of "temporal_id. " The <NUM>-bit field "layer_maximum LMX" indicates the level (layer) indicated by the maximum value of "temporal_id. " In this example, the number of layers to which "temporal_id" is assigned is (LMX - LMI + <NUM>). The <NUM>-bit field "layer_level_idc[i]" indicates "level_idc" as level specified value of the bit rate at each level.

The transport stream TS also contains EIT as SI (serviced information) for management of each event. The multiplexer <NUM> arranges the layer decoding descriptor described above (see <FIG>) under the EIT. In this case, the multiplexer <NUM> also arranges a conventionally known component descriptor under the EIT to make a link with the PES stream.

PES extension field can be provided in the header of the PES packet. The multiplexer <NUM> inserts PES extension field data having hierarchical information into the extension field. In this manner, the multiplexer <NUM> provides the PES extension field in the header of the PES packet to insert the PES extension field data having hierarchical information at least for each coded video sequence (CVS), for example. <FIG> illustrates a structural example (Syntax) of the PES extension field data "pes_extension_field_data().

The "PES_extension field length" is given outside the syntax structure. The <NUM>-bit field "start_sync_byte" indicates the code value representing the start of the extension field. The <NUM>-bit field "extension_field_type" indicates the type of the extension field, which means the supply of hierarchical information in this example. The "layer_information()" has fields "layer_minimum LMI," "layer_minimum LMX, " and "layer_level_idc[i]," as described above (see <FIG>).

In this case, the multiplexer <NUM> arranges a PES extension descriptor (PES_extension_descriptor) as one of the descriptors in the video elementary loop to describe explicitly that the hierarchical information is inserted into the PES extension field.

<FIG> illustrates a structural example (Syntax) of the PES extension descriptor (PES_extension_descriptor). <FIG> illustrates the contents (Semantics) of major information in the structural example. The <NUM>-bit field "PES_extention_descriptor_tag" indicates the type of the descriptor, which means the PES extension descriptor in this example.

The <NUM>-bit field "PES_extention_descriptor_length" indicates the length (size) of the descriptor as the number of subsequent bytes. The <NUM>-bit field "PES_extension_existed" indicates whether the PES extension field of the applicable PES stream is encoded. The value "<NUM>" indicates that the PES extension field is encoded, and the value "<NUM>" indicates that the PES extension field is not encoded.

<FIG> illustrates a configuration example of the transport stream TS in the case where single-stream distribution is performed and the hierarchical information is inserted under the program map table (PMT) (the foregoing case (A)). In this configuration example, there is a PES packet "video PES1" of a video stream having image data encoded by HEVC of pictures at a plurality of levels, for example.

The encoded image data of the pictures have NAL units such as VPS, SPS, PPS, SLICE, and SEI. As described above, the level identification information ("nuh_temporal_id_plus1" indicative of temporal_id) for the picture is arranged in the headers of the NAL units. The level specified value of the bit stream "general_level_idc" is inserted into the SPS. In addition, the pictures belonging to the levels indicated by "temporal_id" are bound into sub layers (sub_layer) and "Sublayer_level_presented_flag" is set to "<NUM>," whereby the bit rate level specified value for each sub layer "sublayer_level_idc" is inserted into the SPS.

The transport stream TS also contains the program map table (PMT) as program specific information (PSI). The PSI is information describing to which program each elementary stream contained in the transport stream belongs.

The PMT has a program loop describing information related to the entire program. The PMT also has an elementary loop with information related to each elementary stream. In this configuration example, there exists a video elementary loop (video ES loop).

In the video elementary loop, information such as stream type and packet identifier (PID) is arranged in correspondence with the video stream (video PES1), and descriptors describing information related to the video stream are also arranged. As one of the descriptors, the layer decoding descriptor (Layer_decoding_descriptor) described above is inserted.

For example, in the examples of hierarchical encoding illustrated in <FIG> and <FIG>, the contents described by the descriptor are as follows: "layer_minimum LMI" = <NUM>, "layer_maximum LMX" = <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, and "layer_level_idc[<NUM>]" = Level <NUM>.

<FIG> illustrates a configuration example of the transport stream TS in the case where two-stream distribution is performed and the hierarchical information is insertedunde r the program map table (PMT) (the foregoing case (A)). In this configuration example, a plurality of levels is divided into two sets of low-level side and high-level side, and there exist PES packets "video PES1" and "video PES2" of the video streams having image data encoded by HEVC of the pictures of the two sets, for example.

The encoded image data of the pictures on the low-level side have NAL units such as VPS, SPS, PPS, SLICE, and SEI. The hierarchical identification information ("nuh_temporal_id_plus1" indicative of temporal_id) of the picture is arranged in the header of the NAL units. The level specified value of the bit stream "general_level_idc" is inserted into the SPS. In addition, the pictures belonging to the levels indicated by "temporal_id" are bound into sub layers (sub_layer) and "sublayer_level_presented_flag" is set to "<NUM>," whereby the bit rate level specified value for each sub layer "sublayer_level_idc" is inserted into the SPS.

Meanwhile, the encoded image data of the pictures on the high-level side have NAL units such as PPS and SLICE. The hierarchical identification information ("nuh_temporal_id_plus1" indicative of temporal_id) of the picture is arranged in the headers of the NAL units.

The PMT has a program loop describing information related to the entire program. The PMT also has an elementary loop with information related to each elementary stream. In this configuration example, there exist two video elementary loops (video ES1 loop and video ES2 loop).

In the video elementary loop, information such as stream type and packet identifier (PID) is arranged in correspondence with the video streams (video PES1 and video PES2), and descriptors describing information related to the video streams are also arranged. As one of the descriptors, the layer decoding descriptor (Layer_decoding_descriptor) described above is inserted.

For example, in the examples of hierarchical encoding illustrated in <FIG> and <FIG>, the contents described by the descriptors corresponding to the PES packets "video PES1" and "video PES2" are as follows: the descriptor corresponding to the PES packet "video PES1" describes "layer_minimum LMI" = <NUM>, "layer_maximum LMX" = <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, and "layer_level_idc[<NUM>]" = Level <NUM>; and the descriptor corresponding to the PES packet "videoPES1"describes "layer_minimumLMI" =<NUM>, "layer_maximum LMX" = <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, and "layer_level_idc[<NUM>]" = Level <NUM>.

<FIG> illustrates a configuration example of the transport stream TS in the case where single-stream distribution is performed and the hierarchical information is inserted under the event information table (EIT) (the foregoing case (B)). In this configuration example, as in the configuration example of <FIG>, there exists the PES packet "video PES1" of the video stream having the image data encoded by HEVC of the pictures at a plurality of levels, for example.

The transport stream TS contains the program map table (PMT) as program specific information (PSI). The PSI is information describing to which program each elementary stream contained in the transport stream belongs.

The PMT has a program loop describing information related to the entire program. The PMT also has elementary loops with information related to each elementary stream. The PMT has a program loop describing information related to the entire program. The PMT also has an elementary loop with information related to each elementary stream. In this configuration example, there exists a video elementary loop (video ES loop). In the video elementary loop, information such as stream type and packet identifier (PID) is arranged in correspondence with the video stream (video PES1), and descriptors describing information related to the video stream are also arranged.

The transport stream TS also contains EIT as SI (serviced information) for management of each event. The layer decoding descriptor (Layer_decoding_descriptor) described above is arranged under the EIT. Although not explained in detail, the contents described by the descriptor are the same as those in the configuration example of <FIG>. A conventionally known component descriptor is arranged under the EIT to make a link with the PES packet "video PES1.

<FIG> illustrates a configuration example of the transport stream TS in the case where two-stream distribution is performed and the hierarchical information is inserted under the event information table (EIT) (the foregoing case (B)). In this configuration example, a plurality of levels is divided into two sets of low-level side and high-level side, and there exist PES packets "video PES1" and "video PES2" of the video streams having image data encoded by HEVC of the pictures of the two sets, for example, as in the configuration example of <FIG>.

The PMT has a program loop describing information related to the entire program. The PMT also has an elementary loop with information related to each elementary stream. In this configuration example, there exist two video elementary loops (video ES1 loop and video ES2 loop). In the video elementary loop, information such as stream type and packet identifier (PID) is arranged in correspondence with the video streams (video PES1 and video PES2), and descriptors describing information related to the video streams are also arranged.

The transport stream TS also contains EIT as serviced information (SI) for management of each event. The layer decoding descriptors (Layer_decoding_descriptor) corresponding to the PES packets "video PES1" and "video PES2" are arranged under the EIT. Although not explained in detail, the contents described by the descriptors are the same as those in the configuration example of <FIG>. A conventionally known component descriptor is arranged under the EIT to make links with the PES packets "video PES1" and "video PES2.

<FIG> illustrates a configuration example of the transport stream TS in the case where single-stream distribution is performed and the hierarchical information is inserted into the extension field of the header of the PES packet (the foregoing case (C)). In this configuration example, there is a PES packet "video PES1" of a video stream having image data encoded by HEVC of pictures at a plurality of levels, for example, as in the configuration example of <FIG>.

A PES extension field is provided in the header of the PES packet, and PES extension field data "pes_extension_field_data()" having "layer_information()" is inserted into the PES extension field. Although not described in detail, the contents described in "layer_information()" are the same as those described by the layer decoding descriptor in the configuration example of <FIG>.

In the video elementary loop, information such as stream type and packet identifier (PID) is arranged in correspondence with the video streams (video PES1 and video PES2), and descriptors describing information related to the video streams are also arranged. As one of the descriptors, a PES extention descriptor (PES_extention_descriptor) is inserted. The PES extention descriptor is a descriptor to describe explicitly that the hierarchical information is inserted into the PES extension field.

<FIG> illustrates a configuration example of the transport stream TS in the case where two-stream distribution is performed and the hierarchical information is inserted into the extension field of the header of the PES packet (the foregoing case (C)). In this configuration example, a plurality of levels is divided into two sets of low-level side and high-level side, and there exist PES packets "video PES1" and "video PES2" of the video streams having image data encoded by HEVC of the pictures of the two sets, for example, as in the configuration example of <FIG>.

A PES extension field is provided in the header of the PES packet "video PES1", and PES extension field data "pes_extension_field_data()" having "layer_information()" is inserted into the PES extension field. Although not described in detail, the contents described in "layer_information()" are the same as those described by the layer decoding descriptor corresponding to the PES packet "video PES1" in the configuration example of <FIG>.

A PES extension field is provided in the header of the PES packet "video PES2", and PES extension field data "pes_extension_field_data()" having "layer_information()" is inserted into the PES extension field. Although not described in detail, the contents described in "layer_information()" are the same as those described by the layer decoding descriptor corresponding to the PES packet "video PES2" in the configuration example of <FIG>.

The transport stream TS also contains the PMT (program map table) as PSI (program specific information). The PSI is information describing to which program each elementary stream contained in the transport stream belongs.

Operation of the transmission device <NUM> illustrated inFig. <NUM> will be briefly described. Uncompressed moving image data VD is input into the encoder <NUM>. The encoder <NUM> subjects the moving image data VD to hierarchical encoding. Specifically, the encoder <NUM> classifies image data of pictures constituting the moving image data VD into a plurality of levels and encodes the same, thereby generating a video stream having encoded image data of the pictures at the respective levels.

In this case, the referenced pictures are encoded such that they belong to their levels and/or lower ones. In this case, a plurality of levels is divided into two of low-level side and high-level side, and the hierarchical composition is equalized between the low-level side and the high-level side, and corresponding pictures on the low-level side and the high-level side are combined into one set and are sequentially encoded. Also in this case, a single video stream having the encoded image data of the pictures at the respective levels is generated, or two video streams having the encoded image data of the pictures on the upper-order level side and the lower-order level side are generated.

The video stream generated by the encoder <NUM> and containing the encoded data of pictures at the respective levels is supplied to the compressed data buffer (cpb) <NUM> and is temporarily accumulated there. The multiplexer <NUM> reads the video stream from the compressed data buffer <NUM>, turns the same into PES packet, further turns the same into transport packet for multiplexing, thereby obtaining the transport stream TS as a multiplexed stream. The transport stream TS contains one or more video streams as described above.

When the multiplexer <NUM> generates the transport stream TS, the hierarchical information is inserted in the layer of the container under the program map table (PMT), under the event information table (EIT), or in the extension field of the header of the PES packet. The transport stream TS generated by the multiplexer <NUM> is sent to the transmission unit <NUM>. The transmission unit <NUM> transmits the transport stream TS on broadcast waves or in packets over a network to the reception device <NUM>.

<FIG> illustrates a configuration example of the reception device <NUM>. The reception device <NUM> has a central processing unit (CPU) <NUM>, a reception unit <NUM>, a demultiplexer <NUM>, and a compressed data buffer (cpb: coded picture buffer) <NUM>. The reception device <NUM> also has a decoder <NUM>, a decompressed data buffer (dpb: decoded picture buffer) <NUM>, a post-processing unit <NUM>, and a display unit <NUM>. The CPU <NUM> constitutes a control unit that controls operations of the components of the reception device <NUM>.

The reception unit <NUM> receives the transport stream TS on broadcast waves or in packets over a network transmitted from the transmission device <NUM>. The demultiplexer <NUM> retrieves selectively from the transport stream TS the encoded image data of the pictures at the level commensurate with the capability of the decoder <NUM>, and sends the same to the compressed data buffer (cpb: coded picture buffer) <NUM>. In this case, the demultiplexer <NUM> refers to the value of "nuh_temporal_id_plus1" indicative of "temporal_id" arranged in the headers of the NAL units (nal_unit) of the pictures.

In this case, the demultiplexer <NUM> extracts the hierarchical information inserted in the layer of the container, recognizes "layer_level_idc" at the respective levels from the hierarchical information, and detects up to which level decoding is enabled according to the capability of the decoder <NUM>. For example, in the examples of hierarchical encoding of <FIG> and <FIG>, it is assumed that "layer_level_idc[<NUM>]" = Level5, "layer_level_idc[<NUM>]"=Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, and "layer_level_idc[<NUM>]" = Level <NUM>. In this case, when the decoder <NUM> has a capability of 60P, that is, "Level <NUM>," the demultiplexer <NUM> detects that decoding is enabled up to the level <NUM>. In addition, in this case, when the decoder <NUM> has a capability of 120P, that is, "Level <NUM>," the demultiplexer <NUM> detects that decoding is enabled up to the level <NUM>.

The compressed data buffer (cpb) <NUM> accumulates temporarily the encoded image data of pictures at the respective levels sent from the demultiplexer <NUM>. The decoder <NUM> reads and decodes the encoded image data of the pictures accumulated in the compressed data buffer <NUM> at decode timings given by decoding time stamps (DTS) of the pictures, and sends the same to the decompressed data buffer (dpb) <NUM>.

The decompressed data buffer (dpb) <NUM> accumulates temporarily the image data of the pictures decoded by the decoder <NUM>. The post-processing unit <NUM> matches the frame rate for the image data of the pictures read sequentially at display timings given by presentation time stamps (PTS) from the decompressed data buffer (dpb) <NUM> with the display capability.

For example, when the frame rate of image data of the pictures after decoding is <NUM> fps and the display capability is <NUM> fps, the post-processing unit <NUM> performs interpolation in the image data of the pictures after decoding such that the time-direction resolution becomes doubled, and sends the same as image data of <NUM> fps to the display unit <NUM>.

The display unit <NUM> is composed of a liquid crystal display (LCD), an organic electro-luminescence (EL) panel, or the like, for example. The display unit <NUM> may be an external device connected to the reception device <NUM>.

Operations of the reception device <NUM> illustrated in <FIG> will be described briefly. The reception unit <NUM> receives the transport stream TS on broadcast waves or in packets over a network from the transmission device <NUM>. The transport stream TS is sent to the demultiplexer <NUM>. The demultiplexer <NUM> retrieves selectively from the transport stream TS the encoded image data of pictures at the level commensurate with the capability of the decoder <NUM> based on the hierarchical information inserted in the layer of the container, and sends the same to the compressed data buffer (cpb) <NUM> for temporary accumulation.

The decoder <NUM> retrieves the encoded image data of pictures at the respective levels accumulated in the compressed data buffer <NUM>. The decoder <NUM> then decodes the retrieved encoded image data of the pictures at the respective decode timings for the pictures, sends the same to the decompressed data buffer (dpb) <NUM> for temporary accumulation.

Then, the image data of the pictures read sequentially at the display timings from the decompressed data buffer (dpb) <NUM> is sent to the post-processing unit <NUM>. The post-processing unit <NUM> subjects the image data of the pictures to interpolation or sub sampling to match the frame rate with the display capability. The image data of the pictures processed by the post-processing unit <NUM> is supplied to the display unit <NUM> for display of moving images.

As described above, in the transmission/reception system <NUM> illustrated in <FIG>, the transmission device <NUM> equalizes the hierarchical composition between the low-level side and the high-level side, and combines corresponding pictures on the low-level side and the high-level side into one set and encodes the same sequentially. Accordingly, the reception device <NUM> can decode the encoded image data of the pictures on the low-level side and the high-level side at one collective timing, thereby reducing the buffer size and decreasing decode delay.

In addition, in the transmission/reception system <NUM> illustrated in <FIG>, the transmission device <NUM> inserts the hierarchical information into the layer of the container to generate a transport stream containing a video stream having the image data of the encoded pictures at the respective levels. Accordingly, the reception device <NUM> can refer to the hierarchical information to retrieve selectively from the video stream the encoded image data of the pictures up to the level commensurate with the capability of the decoder in an easy manner, for example.

In the transmission/reception system <NUM> illustrated in <FIG>, the transmission device <NUM> inserts the hierarchical information into the PES extension field of the header of the PES packet in the position synchronized with the encoded image data of the pictures of the video stream at least for each coded video sequence (CVS). This allows the reception side to, even with changes in the hierarchical composition, retrieve selectively from the video stream the encoded image data of the pictures up to the level commensurate with the capability of the decoder.

<FIG> illustrates an example of correspondence between signaling of "level_idc" and hierarchical information of encoded image data with changes in hierarchical composition. In this example, the hierarchical composition changes from a first 50P CVS system in which encoding is performed at three levels of <NUM> to <NUM> to a second 50P CVS system in which encoding is performed at four levels of <NUM> to <NUM>, and further changes to a 100P CVS system in which encoding is performed at six levels of <NUM> to <NUM>. In the illustrated example, the hierarchical information is inserted under the PMT. However, the foregoing matter also applies to the case where the hierarchical information is inserted under the EIT or into the PES extension field as described above.

In the period of the first 50P CVS system, data is distributed in a single video stream. The value of "general_level_idc" inserted into the SPS of the encoded image data is set to "Level <NUM>" as a level value containing all the pictures at the levels of <NUM> to <NUM>. The value of "sublayer_level_idc[<NUM>]" as the level specified value of bit rate at the level of <NUM> is set to "Level <NUM>. " In this case, the hierarchical information is described as "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, and "layer_level_idc[<NUM>]" = Level <NUM>.

In the period of the second 50P CVS system, data is distributed in a single video stream. The value of "general_level_idc" inserted into the SPS of the encoded image data is set to "Level <NUM>" as a level value containing all the pictures at the levels of <NUM> to <NUM>. The value of "sublayer_level_idc[<NUM>]" as the level specified value of bit rate at the level of <NUM> is set to "Level <NUM>. " In this case, the hierarchical information is described as "layer_level_idc [<NUM>]" = Level <NUM>, "layer_level_idc [<NUM>] " = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>", and "layer_level_idc[<NUM>]" = Level <NUM>.

In the period of the 100P CVS system, data is distributed in two video streams. The value of "general_level_idc" inserted into the SPS of the encoded image data is set to "Level <NUM>" as a level value containing all the pictures at the levels of <NUM> to <NUM>. The value of "sublayer_level_idc[<NUM>]" as the level specified value of bit rate at the level of <NUM> is set to "Level <NUM>. " In this case, the hierarchical information is described as "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" =Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, "layer_level_idc[<NUM>]" = Level <NUM>, and "layer_level_idc[<NUM>]" = Level <NUM>.

While the hierarchical composition changes as illustrated in the drawing, if the decoder <NUM> of the reception device <NUM> corresponds to 50P, for example, the demultiplexer <NUM> retrieves the levels of <NUM> to <NUM> in the period of the first 50P CVS system, retrieves the levels of <NUM> to <NUM> in the second 50P CVS system, and retrieves the levels of <NUM> to <NUM> in the period of the 100P CVS system, based on the hierarchical information, and sends the same to the compressed data buffer <NUM>. The decoder <NUM> decodes the encoded image data of the pictures at their respective decode timings to obtain 50P image data.

In the foregoing embodiment, the transmission/reception system <NUM> is composed of the transmission device <NUM> and the reception device <NUM>. However, the configuration of the transmission/reception system to which the subject technique is applicable is not limited to this. For example, part of the reception device <NUM> may be formed as a set-top box and a monitor connected via a digital interface such as a high-definition multimedia interface (HDMI). The "HDMI" is a registered trademark.

In the foregoing embodiment, the container is a transport stream (MPEG-<NUM> TS). However, the subject technique is also applicable to other systems in which data is distributed to reception terminals via a network such as the Internet. In the Internet delivery, data is frequently distributed by a container in MP4 or other formats. That is, the container may be a transport stream (MPEG-<NUM> TS) employed under digital broadcasting standards, or any other container in various formats such as MP4 used in the Internet delivery.

Claim 1:
A reception device comprising
a reception unit (<NUM>) configured to receive a container containing a first stream having encoded image data of pictures on a low-level side corresponding to the one or more lowest levels of a plurality of levels of a hierarchical level structure, a second stream having encoded image data of pictures on a high-level side corresponding to the one or more highest levels of the plurality of levels of the hierarchical level structure, and a program map table, PMT,
wherein said first and second stream are generated by subjecting image data of the picture constituting moving image data to hierarchical encoding,
wherein the PMT includes a first descriptor describing a level specified value of the first stream in correspondence with the first stream and a second descriptor describing a level specified value of a bit stream formed from a combination of the first stream and the second stream, the first descriptor corresponding to the first stream and the second descriptor corresponding to the second stream;
wherein the encoded image data has a network abstraction layer, NAL, unit structure, and the level specified value of the first stream is inserted together with the level specified value of the bit stream formed from a combination of the first stream and the second stream into an NAL unit of SPS of the first stream; and
wherein the image data of the picture are hierarchical encoded such that the picture order of composition of the encoded image data of pictures on the high-level side is either one picture behind or one picture ahead of the picture order of composition of the encoded image data of pictures on the low-level side;
a processing unit (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) configured to:
extract the first descriptor and the second descriptor from the container,
detect up to which level decoding is enabled according to a decoding capability,
decode the first stream or both the first stream and the second stream contained in the received container according to the decoding capability to obtain the image data of the pictures constituting moving image data,
perform a frame rate interpolation or a frame rate sub-sampling to the decoded image data of the pictures based on a display capability of a display unit for displaying the moving image data; and
supply the interpolated or sub-sampled image data to the display unit.