Patent Description:
When providing compressed moving images over broadcasting or network services and the like, the upper limit of the frame frequency that may be played back is restricted by the performance of the receiver. Consequently, the service side is required to take the playback performance of prevalent receivers into account, and restrict the service to low frame frequency only, or simultaneously provide multiple high-grade and low-grade services.

Adding support for high frame frequency services increases the cost of the receiver, and becomes a barrier to adoption. If only low-cost receivers dedicated to low frame frequency services are widespread, and in the future the service side starts a high frame frequency service, the new service is completely unwatchable without a new receiver, which becomes a barrier to adoption of the service.

Moving image compression schemes such as H. <NUM>/AVC (Advanced Video Coding) (see Non-Patent Literature <NUM>) are generally made up of the following three types of pictures.

Utilizing this property, frame-decimated playback is possible to some extent, such as by playing only I pictures and P pictures, for example. However, with this method, finely decimated playback is difficult, and usage as a practical service is challenging.

Document<NPL>, as well as document <NPL>), disclose signalling of scalability features in SVC.

Document <NPL>), discloses background art on HEVC high-level syntax and reference picture management.

Document<NPL>), section "<NUM> Carriage of SVC over MPEG-<NUM> Systems", discloses that "an operation point can be signaled in the PMT (Program Map Table) or PSM (Program Stream Table). Multiple operation points can be signaled in the PMT or PSM", which corresponds to signalling of the highest/lowest frame rate, as well as number of layers, in the PMT.

Non-Patent Literature <NUM>: ITU-T H. <NUM> (<NUM>/<NUM>), "Advanced video coding for generic audiovisual services.

An objective of the present technology is to achieve with ease a high frame frequency service.

According to an aspect of the present technology, there is provided a receiving device as defined in claim <NUM> and a receiving method as defined in claim <NUM>.

According to the present technology, it is possible to easily achieve a high frame frequency service.

Hereinafter, embodiments for carrying out the invention (hereinafter designated the exemplary embodiments) will be described. Hereinafter, the description will proceed in the following order.

<FIG> illustrates an exemplary configuration of a television (TV) transmitting/receiving system <NUM> as an exemplary embodiment. The TV transmitting/receiving system <NUM> includes a TV transmitter <NUM> and a TV receiver <NUM>.

The TV transmitter <NUM> transmits a transport stream TS that acts as a container on a carrier wave. In the transport stream TS, the image data of each picture constituting moving image data is classified into multiple layers, and the transport stream TS includes a single video stream holding the coded data of the image data in each layers. In this case, coding such as H. <NUM>/AVC is performed, for example, so that a referenced picture belongs to a layer of referencing image data and/or a lower layer than the layer of the referencing image data.

In this case, the image data of each picture constituting the moving image data is classified into multiple layers so that, except for the lowest layer, the pictures belonging to each layer are equal in number to the pictures belonging to all lower layers, and in addition, are positioned in the temporal centers between the pictures belonging to all lower layers. With such a classification, the frame frequency doubles every time the layer is raised by one, and thus on the receiving side, it becomes possible to easily recognize the frame frequency in each layer with only the frame frequency information of the pictures in the lowest layer.

For each picture, layer identification information for identifying the containing layer is added to the coded image data of each layer. In this exemplary embodiment, layer identification information (temporal_id) is placed in the header part of the NAL unit (nal_unit) of each picture. As a result of layer identification information being added in this way, on the receiving side, it is possible to conduct good selective retrieval of coded image data in a prescribed layer and lower layers.

Frame frequency information of the pictures in the lowest layer and layer number information indicating the number of the multiple layers is inserted into the transport stream TS. This information is inserted into the transport layer or the video layer. For example, this information is inserted into statements under a video elementary loop under a program map table (PMT). As another example, this information is inserted as an SEI message in the "SEIs" part of an access unit. As a result of frame frequency information and layer number information being inserted in this way, on the receiving side, it becomes possible to acquire this information easily.

The TV receiver <NUM> receives the above transport stream TS sent from the TV transmitter <NUM> on a carrier wave. The TV receiver <NUM> selectively retrieves and decodes the coded image data of a prescribed layer and lower layers from the video stream included in the transport stream TS, acquires the image data of each picture, and conducts image playback. In this case, the speed of image playback according to the decoded image data of each picture is adjusted to match the frame frequency of the pictures in the prescribed layer.

As discussed earlier, frame frequency information of the pictures in the lowest layer and layer number information indicating the number of the multiple layers is inserted into the transport stream TS. At the TV receiver <NUM>, the decoding layer is controlled on the basis of this information and the decoding performance of the TV receiver <NUM> itself, and in addition, the image playback speed is controlled.

<FIG> illustrates an exemplary configuration of the TV transmitter <NUM>. The TV transmitter <NUM> includes an original moving image data supply section <NUM>, a decoding device <NUM>, a hierarchical classification section <NUM>, an image coding section <NUM>, an audio coding section <NUM>, a multiplexing section <NUM>, an additional information producing section <NUM>, and a modulation/transmitting antenna section <NUM>.

The original moving image data supply section <NUM> retrieves original moving image data (image data, audio data) stored in an appropriate professional compression format on a device such as a hard disk drive (HDD), and supplies the retrieved original moving image data to the decoding device <NUM>. The decoding device <NUM> decodes the original moving image data, and outputs uncompressed image data and uncompressed audio data.

The hierarchical classification section <NUM> classifies the image data of each picture constituting the uncompressed image data into multiple layers. For example, as illustrated in the drawing, image data is classified into the three layers of a first layer, a second layer, and a third layer. Herein, the hierarchical classification section <NUM> conducts classification so that, except for the lowest layer, the pictures belonging to each layer are equal in number to the pictures belonging to all lower layers, and in addition, are positioned in the temporal center between the pictures belonging to all lower layers.

The image coding section <NUM> encodes the classified image data of each layer, and generates a video stream (video elementary stream) holding the coded image data of each layer. Herein, the image coding section <NUM> conducts coding such as H. <NUM>/AVC, for example, so that a referenced picture belongs to a layer of referencing image data and/or a lower layer than the layer of the referencing image data.

<FIG> illustrates an example of hierarchical classification and image coding. This example is an example of classifying the image data of each picture into three layers from a first layer to a third layer. In this example, I pictures (intra pictures) and P pictures (predictive pictures) are made to belong to the first layer. An I picture does not reference another picture, while a P picture only references an I picture or a P picture. For this reason, the first layer is decodable with just first layer pictures.

In addition, B pictures (bi-directional predictive pictures) are placed in the temporal center positions between the respective pictures in the first layer, and are made to belong to the second layer. The B pictures in the second layer are encoded so as to reference only pictures belonging to a combined layer of the second layer and/or the first layer.

In this example, B pictures in the second layer are made to reference only I pictures and P pictures in the first layer. For this reason, the second layer is decodable with just the first/second combined layer. Also, compared to the case of decoding the first layer only, the frame frequency is doubled when decoding the first/second combined layer.

In addition, B pictures are placed in the temporal center positions between the respective pictures in the first/second combined layer, and are made to belong to the third layer. The B pictures in the third layer are made to reference only pictures belonging to the third layer and/or the first/second combined layer. For this reason, the third layer is decodable with just the first to third combined layer. Also, compared to the case of decoding the first/second combined layer only, the frame frequency is doubled when decoding the first to third combined layer.

In <FIG>, the dashed lines indicate picture reference relationships. A P picture in the first layer references only the immediately previous I picture or P picture. A B picture in the second layer references only the immediately previous or immediately following I picture or P picture in the first layer. A B picture in the third layer references only the immediately previous or immediately following I picture, P picture, or B picture in the first/second combined layer.

For each picture, the image coding section <NUM> adds layer identification information for identifying the layer containing the picture to the coded image data of each layer. In other words, the image coding section <NUM> places layer identification information (temporal_id) in the header part of the NAL unit (nal_unit) of each picture.

<FIG> illustrates the placement position of the layer identification information (temporal_id). Namely, the layer identification information (temporal_id) is placed in the NAL unit header SVC extension (Header svc extension), for example. Additionally, as illustrated in <FIG>, "temporal _id=<NUM>" is assigned to pictures belonging to the first layer, "temporal_id=<NUM>" is assigned to pictures belonging to the second layer, and "temporal_id=<NUM>" is assigned to pictures belonging to the third layer.

In the example of <FIG>, when the frame frequency of the first layer only is <NUM> fps, the frame frequency of the first/second combined layer is <NUM> fps, and the frame frequency of the first to third combined layer is <NUM> fps. Also, although not illustrated in the drawing, it is possible to similarly construct a fourth layer and fifth layer.

Returning to <FIG>, the audio coding section <NUM> performs coding such as MPEG-<NUM> Audio or AAC on the uncompressed audio data, and generates an audio stream (audio elementary stream). The multiplexing section <NUM> multiplexes the elementary streams output from the video encoder <NUM> and the audio encoder <NUM>. The multiplexing section <NUM> then outputs a transport stream TS as transport data.

The additional information producing section <NUM> produces, and sends to the multiplexing section <NUM>, frame frequency information of the pictures in the lowest layer and layer number information indicating the number of the multiple layers. The multiplexing section <NUM> inserts this information into the transport layer. For example, in the descriptor loop under "ES_info_length" of a program map table (PMT), the multiplexing section <NUM> places a newly defined FPS descriptor (fps_descriptor) stating the frame frequency information and the layer number information, as illustrated in <FIG>. This descriptor loop is the place that states the property information of each elementary stream (elementary_stream). The FPS descriptor is treated as one descriptor included among the above.

<FIG> illustrates example syntax of the FPS descriptor. The <NUM>-bit field "descriptor_tag" indicates the class of the descriptor, and herein indicates that the descriptor is the FPS descriptor. For example, the currently unused "0xf0" is assigned. The <NUM>-bit field "descriptor_length" indicates the immediately following byte length, and herein is "0x02".

The <NUM>-bit field "base" expresses the frame frequency information of pictures in the lowest layer, or in other words the frame frequency information of the first layer. For example, in the case of <NUM> fps as in the example illustrated in <FIG>, the value is "0x1e" indicating <NUM>. The <NUM>-bit field "max" expresses layer number information indicating the number of the multiple layers. For example, in the case of layers up to the third layer as in the example illustrated in <FIG>, the value is "0x03" indicating <NUM>.

In this way, by adding the FPS descriptor on the transmitting side (coding side), frame-decimated playback becomes easy on the receiving side (decoding side). In other words, it is known from the stated content of the FPS descriptor that the frame frequency is <NUM> fps with the first layer only, <NUM> fps with the first/second combined layer, and <NUM> fps with the first to third combined layer. For example, if the decoding performance on the receiving side goes up to a maximum of <NUM> fps, from this information it is known that up to the first/second combined layer is decodable. Additionally, it is known that it is sufficient to decode the pictures with "temporal_id=<NUM>" and "temporal_id=<NUM>". Also, it is known that it is sufficient to play back decoded pictures at <NUM> fps.

Note that inserting the frame frequency information and the layer number information in the video layer, such as, for example, an SEI message in the "SEIs" part of an access unit, is also conceivable. In this case, the additional information producing section <NUM> transmits this information to the image coding section <NUM>, as indicated by the dashed line. As illustrated in <FIG>, the image coding section <NUM> inserts FPS info (fps_info) including the "base" and "max" information as an "fps_info SEI message" in the "SEIs" part of the access unit.

In the case of using an SEI message in this way, the multiplexing section <NUM> inserts identification information identifying the existence of that SEI message in the transport layer. For example, in the descriptor loop under "ES_info_length" of the program map table (PMT), the multiplexing section <NUM> places a newly defined FPS exist descriptor (fps_exit_descriptor), as illustrated in <FIG>.

The <NUM>-bit field "descriptor_tag" indicates the class of the descriptor, and herein indicates that the descriptor is the FPS exist descriptor. For example, the currently unused "0xf2" is assigned. The <NUM>-bit field "descriptor_length" indicates the immediately following byte length, and herein is "0x01". The <NUM>-bit field "fps_exit" indicates the existence of an SEI message with inserted FPS info (fps_info). For example, "fps_exit=<NUM>" indicates that the SEI message does not exist, whereas "fps_exit=<NUM>" indicates that the SEI message exists.

In this way, by adding the FPS exist descriptor on the transmitting side (coding side), the receiving side (decoding side) knows of the existence of an SEI message with inserted FPS info (fps_info) that includes the frame frequency information and the layer number information. If the FPS exist descriptor indicates the existence of an SEI message, the receiving side (decoding side) extracts fps_info, and is able to know, from the values of "base" and "max" inside, which pictures have the "temporal_id" that the receiving side (decoding side) itself should decode. On the basis thereof, the receiving side (decoding side) decodes pictures with the desired "temporal_id".

Returning to <FIG>, the modulation/transmitting antenna section <NUM> modulates the transport stream TS according to a modulation scheme suited to broadcasting, such as QPSK/OFDM. The modulation/transmitting antenna section <NUM> then transmits an RF modulated signal from a transmitting antenna.

Operations of the TV transmitter <NUM> illustrated in <FIG> will be described. Original moving image data (image data, audio data) stored in an appropriate professional compression format is supplied from the original moving image data supply section <NUM> to the decoding device <NUM>. In the decoding device <NUM>, the original moving image data is decoded, and uncompressed image data and uncompressed audio data are obtained.

The uncompressed image data obtained by the decoding device <NUM> is supplied to the hierarchical classification section <NUM>. In the hierarchical classification section <NUM>, the image data of each picture constituting the uncompressed image data is classified into multiple layers. In this case, pictures are classified so that, except for the lowest layer, the pictures belonging to each layer are equal in number to the pictures belonging to all lower layers, and in addition, are positioned in the temporal center between the pictures belonging to all lower layers (see <FIG>).

The image data of each layer hierarchically classified in this way is supplied to the image coding section <NUM>. In the image coding section <NUM>, the classified image data of each layer is decoded, and a video stream (video elementary stream) holding the coded image data of each layer is generated. In this case, coding such as H. <NUM>/AVC is conducted, so that a referenced picture belongs to a layer of referencing image data and/or a lower layer than the layer of the referencing image data.

In this case, in the image coding section <NUM>, for each picture, layer identification information for identifying the layer containing the picture is added to the coded image data of each layer. In other words, in the image coding section <NUM>, layer identification information (temporal_id) is placed in the header part of the NAL unit (nal _unit) of each picture (see <FIG>).

In addition, the uncompressed audio data obtained by the decoding device <NUM> is supplied to the audio coding section <NUM>. In the audio coding section <NUM>, coding such as MPEG-<NUM> Audio or AAC is performed on the uncompressed audio data, and an audio stream (audio elementary stream) is generated.

The video stream generated by the image coding section <NUM> and the audio stream generated by the audio coding section <NUM> are supplied to the multiplexing section <NUM>. In the multiplexing section <NUM>, the elementary streams are multiplexed, and a transport stream TS is obtained as transport data. In the multiplexing section <NUM>, frame frequency information of the pictures in the lowest layer and layer number information indicating the number of the multiple layers is produced, and added to the transport layer (container layer). For example, in the multiplexing section <NUM>, the FPS descriptor (fps_descriptor) stating the frame frequency information and the layer number information is placed in the descriptor loop under "ES_info_length" of the program map table (PMT) (see <FIG> and <FIG>).

Note that the frame frequency information and the layer number information may also be inserted in the video layer, such as, for example, an SEI message in the "SEIs" part of the access unit. In this case, FPS info (fps_info) including the information is inserted as an "fps_info SEI message" in the "SEIs" part of the access unit (see <FIG>). Subsequently, in this case, identification information identifying the existence of the SEI message is inserted into the transport layer (container layer). For example, in the multiplexing section <NUM>, the FPS exist descriptor (fps_exit_descriptor) is placed in the descriptor loop under "ES_info_length" of the program map table (PMT) (see <FIG>).

The transport stream TS generated by the multiplexing section <NUM> is sent to the modulation/transmitting antenna section <NUM>. In the modulation/transmitting antenna section <NUM>, the transport stream TS is modulated according to a modulation scheme suited to broadcasting, such as QPSK/OFDM, and an RF modulated signal is generated. Subsequently, in the modulation/transmitting antenna section <NUM>, the RF modulated signal is transmitted from a transmitting antenna.

<FIG> illustrates an exemplary configuration of the TV receiver <NUM>. The TV receiver <NUM> includes a receiving antenna/demodulation section <NUM>, a demultiplexing section <NUM>, a control section <NUM>, an image decoding section <NUM>, a playback speed adjustment section <NUM>, an image display section <NUM>, an audio decoding section <NUM>, and an audio output section <NUM>.

The receiving antenna/demodulation section <NUM> demodulates an RF modulated signal received with a receiving antenna, and acquires a transport stream TS. The demultiplexing section <NUM> respectively extracts the video stream and the audio stream from the transport stream TS. In the video stream, the image data of each picture constituting moving image data is classified into multiple layers, in which the image data is coded so that a referenced picture belongs to a layer of referencing image data and/or a lower layer than the layer of the referencing image data.

In addition, the demultiplexing section <NUM> extracts, and transmits to the control section <NUM>, various information inserted into the transport layer (container layer) of the transport stream TS. At this point, the FPS descriptor (fps_descriptor) placed in the descriptor loop under "ES_info_length" of the program map table (PMT) is also extracted. In the FPS descriptor, frame frequency information of the pictures in the lowest layer and layer number information indicating the number of the multiple layers is stated.

Alternatively, if the frame frequency information and the layer number information is inserted into the video layer, such as an SEI message in the "SEIs" part of the access unit, for example, the FPS exist descriptor (fps_exit_descriptor) placed in the descriptor loop under "ES_info_length" of the program map table (PMT) may be extracted.

The image decoding section <NUM> selectively retrieves and decodes the coded image data in a prescribed layer and lower layers from the video stream demultiplexed by the demultiplexing section <NUM>, and obtains the image data of each picture. At this point, the image decoding section <NUM> retrieves and decodes the coded image data of pictures in a desired layer on the basis of layer identification information (temporal _id) placed in the header part of the NAL unit of each picture. The playback speed adjustment section <NUM> adjusts the speed of image playback according to the decoded image data of each picture, so as to match the frame frequency of the pictures in the prescribed layer. In other words, the playback speed adjustment section <NUM> successively outputs the decoded image data of each picture to match the frame frequency (frame rate) of pictures in the prescribed layer.

The control section <NUM> controls the operation of each part of the TV receiver <NUM>. The control section <NUM> controls the decoding layer by transmitting, to the image decoding section <NUM>, decoding layer information specifying the prescribed layer and lower layers to be decoded. In addition, the control section <NUM> controls the image playback speed by transmitting, to the playback speed adjustment section <NUM>, playback speed information corresponding to the frame frequency of the pictures in the prescribed layer, such as a synchronization signal, for example.

The control section <NUM> controls the decoding layer in the image decoding section <NUM> and the image playback speed in the playback speed adjustment section <NUM> on the basis of the frame frequency information, the layer number information, and the decoding performance of the TV receiver <NUM> itself. For example, consider the case of the FPS descriptor (fps_descriptor) having stated content as illustrated in <FIG>.

In this case, the control section <NUM> knows that the frame frequency is <NUM> fps with the first layer only, <NUM> fps with the first/second combined layer, and <NUM> fps with the first to third combined layer. Additionally, if decoding capability of the TV receiver <NUM> itself goes up to a maximum of <NUM> fps, from this information the control section <NUM> knows that up to the first/second combined layer is decodable. Additionally, the control section <NUM> knows that it is sufficient to decode the pictures with "temporal_id=<NUM>" and "temporal_id=<NUM>". Also, the control section <NUM> knows that it is sufficient to play back decoded pictures at <NUM> fps.

The image display section <NUM> is made up of a display such as a liquid crystal display (LCD). The image display section <NUM> displays images according to the image data of each picture output from the playback speed adjustment section <NUM>. The audio decoding section <NUM> performs decoding on the audio stream demultiplexed by the demultiplexing section <NUM>, and obtains audio data corresponding to the image data obtained by the image decoding section <NUM>. The audio output section <NUM> is made up of components such as an amp and speakers. The audio output section <NUM> outputs audio according to the audio data output from the audio decoding section <NUM>.

Operations of the TV receiver <NUM> illustrated in <FIG> will be described. In the receiving antenna/demodulation section <NUM>, an RF modulated signal received with a receiving antenna is demodulated, and a transport stream TS is acquired. This transport stream TS is supplied to the demultiplexing section <NUM>. In the demultiplexing section <NUM>, the video stream and the audio stream are respectively extracted from the transport stream TS. Herein, in the video stream, the image data of each picture constituting moving image data is classified into multiple layers, in which the image data is coded so that a referenced picture belongs to a layer of referencing image data and/or a lower layer than the layer of the referencing image data.

In addition, in the demultiplexing section <NUM>, various information inserted into the transport layer (container layer) of the transport stream TS is extracted and transmitted to the control section <NUM>. At this point, the FPS descriptor (fps_descriptor) placed in the descriptor loop under "ES_info_length" of the program map table (PMT) is also extracted. In the FPS descriptor, frame frequency information of the pictures in the lowest layer and layer number information indicating the number of the multiple layers is stated.

In the control section <NUM>, it is determined up to which layer is decodable, on the basis of the frame frequency information, layer number information, and decoding performance of the TV receiver <NUM> itself. In addition, by this control section <NUM>, the decoding layer in the image decoding section <NUM> is controlled, and the image playback speed in the playback speed adjustment section <NUM> is controlled.

The video stream demultiplexed by the demultiplexing section <NUM> is supplied to the image decoding section <NUM>. In the image decoding section <NUM>, under control by the control section <NUM>, the coded image data in a prescribed layer and lower layers is selectively retrieved and decoded from the video stream, and the image data of each picture is successively obtained. The image data of each picture decoded in this way is supplied to the playback speed adjustment section <NUM>.

In the playback speed adjustment section <NUM>, under control by the control section <NUM>, the speed of image playback according to the image data of each picture is adjusted so as to match the frame frequency of the pictures in the prescribed layer. In other words, from the playback speed adjustment section <NUM>, the image data of each picture is successively output to match the frame frequency (frame rate) of pictures in the prescribed layer. The image data is supplied to the image display section <NUM>, and images according to the image data of each picture in the prescribed layer and lower layers are displayed.

Also, the audio stream demultiplexed by the demultiplexing section <NUM> is supplied to the audio decoding section <NUM>. In the audio decoding section <NUM>, decoding is performed on the audio stream, and audio data corresponding to the image data obtained by the image decoding section <NUM> is obtained. The audio data is supplied to the audio output section <NUM>, and audio corresponding to the displayed images is output.

The flowchart in <FIG> illustrates an example of a transmitting processing sequence in the TV transmitter <NUM> illustrated in <FIG>, in the case in which the FPS descriptor (fps_descriptor) is placed under the PMT. Note that in the TV transmitter <NUM> illustrated in <FIG>, in the image coding section <NUM>, a single video stream holding the coded image data of pictures in respective layers is generated, as discussed earlier.

First, in step ST1, the TV transmitter <NUM> starts the transmitting process. Subsequently, in step ST2, the TV transmitter <NUM> decodes original moving image data, and generates uncompressed image data and audio data.

Next, in step ST3, the TV transmitter <NUM> classifies the image data of each picture into multiple layers. In this case, the pictures (frames) are divided into two, and every other one is put into the third layer. Additionally, the other pictures (frames) are divided into two again, and every other one is put into the second layer, while the remaining are put into the first layer.

Next, in step ST4, the TV transmitter <NUM> encodes the image data of each hierarchically classified picture. In this case, the first layer is encoded. In this case, references are made possible only within the first layer. Also, the second layer is encoded. In this case, references are made possible within the first layer and the second layer. Also, the third layer is encoded. In this case, references are made possible within the first layer to the third layer. At this point, the TV transmitter <NUM> places layer identification information (temporal _id) in the header part of the NAL unit (nal_unit) of each picture.

Next, in step ST5, the TV transmitter <NUM> encodes the audio data. Subsequently, in step ST6, the TV transmitter <NUM> generates the FPS descriptor (fps_descriptor) and the PMT containing the FPS descriptor.

Next, in step ST7, the TV transmitter <NUM> multiplexes the coded image data, audio data, and PMT into a transport stream TS. Subsequently, in step ST8, the TV transmitter <NUM> modulates and transmits the transport stream TS. After that, in step ST9, the TV transmitter <NUM> ends the process.

The flowchart in <FIG> illustrates an example of a receiving processing sequence in the TV receiver <NUM> illustrated in <FIG>, in the case in which the FPS descriptor (fps_descriptor) is placed in the descriptor loop under "ES_info_length" of the PMT. This receiving processing sequence corresponds to the transmitting processing sequence illustrated by the flowchart in <FIG> discussed above.

First, in step ST11, the TV receiver <NUM> starts the receiving process. Subsequently, in step ST12, the TV receiver <NUM> receives and demodulates the RF modulated signal (broadcast signal), and obtains the transport stream TS.

Next, in step ST13, the TV receiver <NUM> extracts image data, audio data, and the PMT from the transport stream TS. Subsequently, in step S14, the TV receiver <NUM> extracts the FPS descriptor (fps_descriptor) from the PMT, compares the FPS descriptor to the decoding performance of the TV receiver <NUM> itself, and decides the layer to decode.

Next, in step ST15, the TV receiver <NUM> decodes the image data of pictures in the layer decided in step ST14. Subsequently, playback is conducted at a suitable playback speed from the content of the FPS descriptor (fps_descriptor). Additionally, in step ST16, the TV receiver <NUM> decodes and plays back audio data. After that, in step ST17, the TV receiver <NUM> ends the process.

The flowchart in <FIG> illustrates an example of a transmitting processing sequence in the TV transmitter <NUM> illustrated in <FIG>, in the case of adding an FPS info (fps_info) SEI message. Note that in the TV transmitter <NUM> illustrated in <FIG>, in the image coding section <NUM>, a single video stream holding the coded image data in respective layers is generated, as discussed earlier.

First, in step ST21, the TV transmitter <NUM> starts the transmitting process. Subsequently, in step ST22, the TV transmitter <NUM> decodes original moving image data, and generates uncompressed image data and audio data.

Next, in step ST23, the TV transmitter <NUM> classifies the image data of each picture into multiple layers. In this case, the pictures (frames) are divided into two, and every other one is put into the third layer. Additionally, the other pictures (frames) are divided into two again, and every other one is put into the second layer, while the remaining are put into the first layer.

Next, in step ST24, the TV transmitter <NUM> encodes the image data of each hierarchically classified picture. In this case, the first layer is encoded. In this case, references are possible only within the first layer. Also, the second layer is encoded. In this case, references are possible within the first layer and the second layer. Also, the third layer is encoded. In this case, references are possible within the first layer to the third layer. At this point, the TV transmitter <NUM> places layer identification information (temporal_id) in the header part of the NAL unit (nal_unit) of each picture. In addition, the TV transmitter <NUM> adds an FPS info (fps_info) SEI message.

Next, in step ST25, the TV transmitter <NUM> encodes the audio data. Subsequently, in step ST26, the TV transmitter <NUM> generates the FPS exist descriptor (fps_exist_descriptor) and the PMT containing the FPS exist descriptor.

Next, in step ST27, the TV transmitter <NUM> multiplexes the coded image data, audio data, and PMT into a transport stream TS. Subsequently, in step ST28, the TV transmitter <NUM> modulates and transmits the transport stream TS. After that, in step ST29, the TV transmitter <NUM> ends the process.

The flowchart in <FIG> illustrates an example of a receiving processing sequence in the TV receiver <NUM> illustrated in <FIG>, in the case in which an FPS info (fps_info) SEI message is added. This receiving processing sequence corresponds to the transmitting processing sequence illustrated by the flowchart in <FIG> discussed above.

First, in step ST31, the TV receiver <NUM> starts the receiving process. Subsequently, in step ST32, the TV receiver <NUM> receives and demodulates the RF modulated signal (broadcast signal), and obtains the transport stream TS.

Next, in step ST33, the TV receiver <NUM> extracts image data, audio data, and the PMT from the transport stream TS. In step S34, the TV receiver <NUM> extracts the FPS exist descriptor (fps_exit_descriptor) from the PMT, and looks at "fps_exit". Then, in step ST35, the TV receiver <NUM> judges whether or not "fps_exit=<NUM>".

When "fps_exit=<NUM>", in step ST36, the TV receiver <NUM> extracts the FPS info (fps_info) added as an SEI message, compares the FPS info to the decoding performance of the TV receiver <NUM> itself, and decides the layer to decode. In step ST37, the TV receiver <NUM> decodes the image data of pictures in the layer decided in step ST36. Subsequently, playback is conducted at a suitable playback speed from the content of the FPS info (fps_info). Additionally, in step ST38, the TV receiver <NUM> decodes and plays back audio data. After that, in step ST39, the TV receiver <NUM> ends the process.

Also, when "fps_exit=<NUM>" in step ST35, in step ST40, the TV receiver <NUM> decodes and plays back the image data normally. Additionally, in step ST38, the TV receiver <NUM> decodes and plays back audio data. After that, in step ST39, the TV receiver <NUM> ends the process.

As described above, in the TV transmitting/receiving system <NUM> illustrated in <FIG>, the image data of each picture constituting moving image data is classified into multiple layers, and a video stream holding the coded image data of each layer is transmitted. For this reason, on the transmitting side, by simply transmitting one program or one file, a service supporting various frame frequencies may be provided, and a reduction in operating costs becomes possible.

Meanwhile, on the receiving side, the coded image data in a prescribed layer and lower layers may be selectively retrieved and decoded, enabling playback at a frame frequency suited to the playback performance of the receiving side itself, thereby effectively promoting the adoption of receivers. Herein, image data is coded so that a referenced picture belongs to a layer of referencing image data and/or a lower layer than the layer of the referencing image data, and at a receiver, the playback performance of the receiving side itself may be used effectively without needing to decode layers higher than the prescribed layer.

Also, in the TV transmitting/receiving system <NUM> illustrated in <FIG>, the image coding section <NUM> generates a single video stream holding the encoded image data of each layer, and for each picture, adds layer identification information (temporal_id) for identifying the layer containing the picture to the coded image data of each layer. For this reason, on the receiving side, it is possible to conduct good selective retrieval of coded image data in a prescribed layer and lower layers, on the basis of the layer identification information.

Also, in the TV transmitting/receiving system <NUM> illustrated in <FIG>, the hierarchical classification section <NUM> classifies the image data of each picture constituting the moving image data into multiple layers so that, except for the lowest layer, the pictures belonging to each layer are equal in number to the pictures belonging to all lower layers, and in addition, are positioned in the temporal center between the pictures belonging to all lower layers. For this reason, the frame frequency doubles every time the layer is raised by one, and thus on the receiving side, it becomes possible to easily recognize the frame frequency in each layer with only the frame frequency information of the pictures in the lowest layer.

Also, in the TV transmitting/receiving system <NUM> illustrated in <FIG>, frame frequency information of the pictures in the lowest layer and layer number information indicating the number of the multiple layers is inserted into the container layer (transport layer) or the video layer. For this reason, on the receiving side, it becomes possible to easily acquire the frame frequency information of the pictures in the lowest layer and the layer number information indicating the number of the multiple layers.

Note that the foregoing embodiment illustrates an example in which, in the image coding section <NUM>, a single video stream holding the coded image data of each layer is generated, or in other words, an example of the same PID. However, in the image coding section <NUM>, it is also conceivable for multiple video streams holding the image data of each of multiple layers to be generated.

In this case, as illustrated in <FIG>, a different PID is assigned to each layer. Respectively different PIDs are assigned when multiplexing the NAL units of each layer separated by the hierarchical layering of the video layer into transport stream packets. In comparison to the case of putting all layers into the same PID as in the embodiment discussed above, differences such as the following exist.

In the case of different PIDs, a structure descriptor (structure_descriptor) is placed in the descriptor loop under "program_info_length" of the PMT, for example. <FIG> illustrates example syntax of the structure descriptor. The <NUM>-bit field "descriptor_tag" indicates the class of the descriptor, and herein indicates that the descriptor is the structure descriptor. For example, the currently unused "0xf1" is assigned. The <NUM>-bit field "descriptor_length" indicates the immediately following byte length.

Inside the for loop, the PIDs assigned to each layer (layer_PID) are all stated. The statement order is sequential from the first layer, for example. On the decoding side, the TS packets of which PIDs should be acquired is known from the value of "base" and the listed PIDs.

In addition, it is also conceivable to use the FPS info (fps_info) SEI message illustrated in <FIG> with different PIDs. In this case, the structure descriptor (structure_descriptor) illustrated in <FIG> is placed in the descriptor loop under "program_info_length". On the receiving side (decoding side), TS packets of the PID of the first layer stated at the beginning of the for loop of the structure descriptor are acquired, and the SEI message inside, that is, the FPS info (fps_info), is extracted. The layer to be decoded is judged from the value of "base", the PIDs of the TS packets to be acquired are detected from the "layer_PID" of the structure descriptor, and the desired TS packets are acquired and decoded.

The flowchart in <FIG> illustrates an example of a transmitting processing sequence for the case of being configured so that the TV transmitter <NUM> codes the image data of each layer in different PIDs, and the FPS descriptor (structure_descriptor) is placed under the PMT.

First, in step ST51, the TV transmitter <NUM> starts the transmitting process. Subsequently, in step ST52, the TV transmitter <NUM> decodes original moving image data, and generates uncompressed image data and audio data.

Next, in step ST53, the TV transmitter <NUM> classifies the image data of each picture into multiple layers. In this case, the pictures (frames) are divided into two, and every other one is put into the third layer. Additionally, the other pictures (frames) are divided into two again, and every other one is put into the second layer, while the remaining are put into the first layer.

Next, in step ST54, the TV transmitter <NUM> encodes the image data of each hierarchically classified picture. The first layer is encoded. In this case, references are possible only within the first layer. Also, the second layer is encoded. In this case, references are possible within the first layer and the second layer. Also, the third layer is encoded. In this case, references are possible within the first layer to the third layer.

Next, in step ST55, the TV transmitter <NUM> encodes the audio data. Subsequently, in step ST56, the TV transmitter <NUM> generates the structure descriptor (structure_descriptor) and the PMT containing the FPS exist descriptor.

Next, in step ST57, the TV transmitter <NUM> multiplexes the coded image data, audio data, and PMT into a transport stream TS. Subsequently, the TV transmitter <NUM> multiplexes the image data with different PIDs for each layer. Subsequently, in step ST58, the TV transmitter <NUM> modulates and transmits the transport stream TS. After that, in step ST59, the TV transmitter <NUM> ends the process.

The flowchart in <FIG> illustrates an example of a receiving processing sequence in the TV receiver <NUM> illustrated in <FIG>, in the case in which the image data of each layer is encoded with different PIDs, and the structure descriptor (structure_descriptor) is placed under the PMT. This receiving processing sequence corresponds to the transmitting processing sequence illustrated by the flowchart in <FIG> discussed above.

First, in step ST61, the TV receiver <NUM> starts the receiving process. Subsequently, in step ST62, the TV receiver <NUM> receives and demodulates the RF modulated signal (broadcast signal), and obtains the transport stream TS.

Next, in step ST63, the TV receiver <NUM> extracts image data, audio data, and the PMT from the transport stream TS. Subsequently, in step S64, the TV receiver <NUM> extracts the structure descriptor (structure_descriptor) from the PMT, compares the structure descriptor to the decoding performance of the TV receiver <NUM> itself, and decides the layer to decode.

Next, in step ST65, the TV receiver <NUM> decodes, from the TS packets of each PID, the image data of pictures in the layer decided in step ST64. Subsequently, playback is conducted at a suitable playback speed from the content of the structure descriptor (structure_descriptor). Additionally, in step ST66, the TV receiver <NUM> decodes and plays back audio data. After that, in step ST67, the TV receiver <NUM> ends the process.

The flowchart in <FIG> illustrates an example of a transmitting processing sequence for the case in which the TV transmitter <NUM> codes the image data of each layer with different PIDs, and adds an FPS info (fps_info) SEI message.

First, in step ST71, the TV transmitter <NUM> starts the transmitting process. Subsequently, in step ST72, the TV transmitter <NUM> decodes original moving image data, and generates uncompressed image data and audio data.

Next, in step ST73, the TV transmitter <NUM> classifies the image data of each picture into multiple layers. In this case, the pictures (frames) are divided into two, and every other one is put into the third layer. Additionally, the other pictures (frames) are divided into two again, and every other one is put into the second layer, while the remaining are put into the first layer.

Next, in step ST74, the TV transmitter <NUM> encodes the image data of each hierarchically classified picture. The first layer is encoded. In this case, references are possible only within the first layer. Also, the second layer is encoded. In this case, references are possible within the first layer and the second layer. Also, the third layer is encoded. In this case, references are possible within the first layer to the third layer. At this point, the TV transmitter <NUM> adds an FPS info (fps_info) SEI message.

Next, in step ST75, the TV transmitter <NUM> encodes the audio data. Subsequently, in step ST76, the TV transmitter <NUM> generates the structure descriptor (structure_descriptor) and the PMT containing the FPS exist descriptor.

Next, in step ST77, the TV transmitter <NUM> multiplexes the coded image data, audio data, and PMT into a transport stream TS. Subsequently, the TV transmitter <NUM> multiplexes the image data with different PIDs for each layer. Subsequently, in step ST78, the TV transmitter <NUM> modulates and transmits the transport stream TS. After that, in step ST79, the TV transmitter <NUM> ends the process.

The flowchart in <FIG> illustrates an example of a receiving processing sequence in the TV receiver <NUM> illustrated in <FIG>, in the case in which the image data of each layer is coded with different PIDs, and an FPS info (fps_info) SEI message is added. This receiving processing sequence corresponds to the transmitting processing sequence illustrated by the flowchart in <FIG> discussed above.

First, in step ST81, the TV receiver <NUM> starts the receiving process. Subsequently, in step ST82, the TV receiver <NUM> receives and demodulates the RF modulated signal (broadcast signal), and obtains the transport stream TS.

Next, in step ST83, the TV receiver <NUM> extracts image data, audio data, and the PMT from the transport stream TS. In step S84, the TV receiver <NUM> extracts the structure descriptor (structure_descriptor) from the PMT. Then, in step ST85, the TV receiver <NUM> judges whether or not the structure descriptor exists.

When the structure descriptor exists, in step ST86, the TV receiver <NUM> extracts the FPS info (fps_info) added as an SEI message, compares the FPS info to the decoding performance of the TV receiver <NUM> itself, and decides the layer to decode. In step ST77, the TV receiver <NUM> decodes, from the TS packets of each PID, the image data of pictures in the layer decided in step ST76. Subsequently, playback is conducted at a suitable playback speed from the content of the FPS info (fps_info). Additionally, in step ST88, the TV receiver <NUM> decodes and plays back audio data. After that, in step ST89, the TV receiver <NUM> ends the process.

Also, when the structure descriptor does not exist in step ST85, in step ST90, the TV receiver <NUM> decodes and plays back the image data normally. Additionally, in step ST88, the TV receiver <NUM> decodes and plays back audio data. After that, in step ST89, the TV receiver <NUM> ends the process.

<FIG> illustrates a comparison of additional information for the above four methods of (a) syntax statements with the same PID (PES) and in the PMT, (b) syntax statements with the same PID (PES) and in the SEI, (c) syntax statements in different PIDs (PES) and in the PMT, and (d) syntax statements in different PIDs (PES) and in the SEI.

Also, the foregoing embodiments illustrates an example of classifying the image data of each picture constituting the moving image data into multiple layers so that, except for the lowest layer, the pictures belonging to each layer are equal in number to the pictures belonging to all lower layers, and in addition, are positioned in the temporal center between the pictures belonging to all lower layers. However, the classification method is not limited to such an example. For example, classification methods like the following are also possible.

<FIG> illustrates another example of hierarchical classification and image coding. This example is an example of classifying the image data of each picture into the two layers of a first layer and a second layer. In this example, I pictures and P pictures are made to belong to the first layer. An I picture does not reference another picture, while a P picture only references an I picture or a P picture. For this reason, the first layer is decodable with just first layer pictures.

In addition, two B pictures are placed at equal intervals temporally between each picture in the first layer, and are made to belong to the second layer. The B pictures in the second layer are encoded so as to only reference pictures belonging to the second layer and/or the first layer. For this reason, the second layer is decodable with just the first/second combined layer. Also, compared to the case of decoding the first layer only, the frame frequency is tripled when decoding the first/second combined layer. Consequently, as illustrated in the drawing, when the frame frequency of the first layer only is <NUM> fps, the frame frequency of the first/second combined layer is <NUM> fps.

Likewise in this example, for each picture, layer identification information for identifying the layer containing the picture is added to the coded image data of each layer. In other words, layer identification information (temporal_id) is placed in the header part of the NAL unit (nal _unit) of each picture. In this example, "temporal_id=<NUM>" is assigned to pictures belonging to the first layer, and "temporal_id=<NUM>" is assigned to pictures belonging to the second layer.

<FIG> illustrates example syntax of the FPS descriptor (fps_descriptor) in the case in which hierarchical classification and image coding as illustrated in <FIG> is conducted. The <NUM>-bit field "descriptor_tag" indicates the class of the descriptor, and herein indicates that the descriptor is the FPS descriptor. For example, the currently unused "0xf0" is allocated. The <NUM>-bit field "descriptor_length" indicates the immediately following byte length.

The <NUM>-bit field "base" expresses the frame frequency information of pictures in the lowest layer, or in other words the frame frequency information of the first layer. In this example, the value is "0x28" indicating <NUM>. The <NUM>-bit field "max" expresses layer number information indicating the number of the multiple layers. In this example, the value is "0x02" indicating <NUM>. Also, inside the for loop, the multiples of the frame frequency in the combined layer up to each layer in the second layer and subsequent layers versus the frame frequency of the first layer are all stated. In this example, the value is "0x03" for the second layer, which states that the multiple is 3x.

<FIG> also illustrates another example of hierarchical classification and image coding. This example is an example of classifying the image data of each picture into the two layers of a first layer and a second layer. In this example, I pictures and P pictures are made to belong to the first layer. An I picture does not reference another picture, while a P picture only references an I picture or a P picture. For this reason, the first layer is decodable with just first layer pictures.

In addition, four B pictures are placed at equal intervals temporally between each picture in the first layer, and are made to belong to the second layer. The B pictures in the second layer are encoded so as to only reference pictures belonging to the second layer and/or the first layer. For this reason, the second layer is decodable with just the first/second combined layer. Also, compared to the case of decoding the first layer only, the frame frequency is five times when decoding the first/second combined layer. Consequently, as illustrated in the drawing, when the frame frequency of the first layer only is <NUM> fps, the frame frequency of the first/second combined layer is <NUM> fps.

Likewise in this example, for each picture, layer identification information for identifying the layer containing the picture is added to the coded image data of each layer. In other words, layer identification information (temporal_id) is placed in the header part of the NAL unit (nal _unit) of each picture. In this example, "temporal _id=<NUM>" is assigned to pictures belonging to the first layer, and "temporal_id=<NUM>" is assigned to pictures belonging to the second layer.

The <NUM>-bit field "base" expresses the frame frequency information of pictures in the lowest layer, or in other words the frame frequency information of the first layer. In this example, the value is "0x18" indicating <NUM>. The <NUM>-bit field "max" expresses layer number information indicating the number of the multiple layers. In this example, the value is "0x02" indicating <NUM>. Also, inside the for loop, the multiples of the frame frequency in the combined layer up to each layer in the second layer and subsequent layers versus the frame frequency of the first layer are all stated. In this example, the value is "0x05" for the second layer, which states that the multiple is 5x.

<FIG> also illustrates another example of hierarchical classification and image coding. This example is an example of classifying the image data of each picture into the four layers from the first layer to the fourth layer. In this example, I pictures and P pictures are made to belong to the first layer. An I picture does not reference another picture, while a P picture only references an I picture or a P picture. For this reason, the first layer is decodable with just first layer pictures.

In addition, B pictures (bi-directional predictive pictures) are placed in the temporal center positions between the respective pictures in the first layer, and are made to belong to the second layer. The B pictures in the second layer are encoded so as to reference only pictures belonging to a combined layer of the second layer and/or the first layer. For this reason, the second layer is decodable with just the first/second combined layer. Also, compared to the case of decoding the first layer only, the frame frequency is doubled when decoding the first/second combined layer. Consequently, as illustrated in the drawing, when the frame frequency of the first layer only is <NUM> fps, the frame frequency of the first/second combined layer is <NUM> fps.

In addition, four B pictures are placed at equal intervals temporally between each picture in the first layer, and are made to belong to the third layer. The B pictures in the third layer are encoded so as to only reference pictures belonging to the third layer and/or the second layer or below. For this reason, the third layer is decodable with from the first to third combined layers only. Also, compared to the case of decoding the first layer only, the frame frequency is five times when decoding from the first to third combined layers. Also, compared to the first and second combined layers, the frame frequency is <NUM> times. Consequently, as illustrated in the drawing, when the frame frequency of the first layer only is <NUM> fps, the frame frequency of the first to third combined layers is <NUM> fps.

In addition, B pictures (bi-directional predictive pictures) are placed in the temporal center positions between the respective pictures in the first layer and the third layer, and are made to belong to the fourth layer. However, a part of the pictures are missing, because they are the same as the pictures in the second layer. The B pictures in the fourth layer are encoded so as to only reference pictures belonging to the fourth layer and/or the third layer or below. For this reason, the fourth layer is decodable with the first to fourth combined layer only. Also, compared to the case of decoding the first layer only, the frame frequency is ten times when decoding from the first to fourth combined layers. Consequently, as illustrated in the drawing, when the frame frequency of the first layer only is <NUM> fps, the frame frequency of the first to second combined layers is <NUM> fps.

Likewise in this example, for each picture, layer identification information for identifying the layer containing the picture is added to the coded image data of each layer. In other words, layer identification information (temporal _id) is placed in the header part of the NAL unit (nal _unit) of each picture. In this example, "temporal _id=<NUM>" is assigned to pictures belonging to the first layer, "temporal_id=<NUM>" is assigned to pictures belonging to the second layer, "temporal_id=<NUM>" is assigned to pictures belonging to the third layer, and "temporal_id=<NUM>" is assigned to pictures belonging to the fourth layer.

The <NUM>-bit field "base" expresses the frame frequency information of pictures in the lowest layer, or in other words the frame frequency information of the first layer. In this example, the value is "0x0C" indicating <NUM>. The <NUM>-bit field "max" expresses layer number information indicating the number of the multiple layers. In this example, the value is "0x04" indicating <NUM>. Also, inside the for loop, the multiples of the frame frequency in the combined layer up to each layer in the second layer and subsequent layers versus the frame frequency of the first layer are all stated. In this example, the value is "0x03" for the second layer, which states that the multiple is 2x. In addition, the value is "0x05" for the third layer, which states that the multiple is 5x. Further, the value is "0x0a" for the fourth layer, which states that the multiple is 10x.

Also, although the foregoing embodiments illustrate a TV transmitting/receiving system <NUM> made up of the TV transmitter <NUM> and the TV receiver <NUM>, the configuration of a TV transmitting/receiving system to which the present technology is applicable is not limited thereto. For example, part of the TV receiver <NUM> may also be a configuration of a set-top box and a monitor or the like connected by a digital interface such as High-Definition Multimedia Interface (HDMI), for example.

Also, the foregoing embodiments illustrate an example in which the container is a transport stream (MPEG-<NUM> TS). However, the present technology is similarly applicable to systems configured for delivery to a receiving terminal using a network such as the Internet. With Internet delivery, content is often delivered in a container for MP4 or some other format. In other words, for the container, containers of various formats, such as the transport stream (MPEG-<NUM> TS) adopted in digital broadcasting standards, or MP4 being used for Internet delivery.

Claim 1:
A receiving device (<NUM>) comprising:
a receiving section configured to receive a container in a prescribed format, the container including a video stream of image data, the image data being classified into a plurality of layers, each layer having image data of pictures constituting moving image data;
an image decoding section (<NUM>) configured to selectively retrieve and decode coded image data of a selected layer and one or more layers lower than the selected layer from the video stream included in the received container, to obtain decoded moving image data of the selected layer;
a playback speed adjustment section (<NUM>) configured to adjust a speed of image playback according to the decoded image data to match a frame frequency of pictures in the selected layer;
a control section (<NUM>) configured to control the playback speed adjustment section and to control the image decoding section, and
a demultiplexing section (<NUM>) configured to classify the image data of each picture constituting moving image data into a plurality of layers so that, except for a lowest layer, pictures belonging to each layer are equal in number to pictures belonging to all lower layers, and in addition, are positioned in a temporal center between the pictures belonging to all lower layers, so that the frame frequency doubles every time the layer is raised by one;
wherein information included in the container includes frequency information of pictures in a lowest layer and layer number information indicating the number of the plurality of layers; and
wherein the information included in the container includes layer identification information for identifying each of the plurality of layers, and the image decoding section is configured to selectively retrieve and decode coded image data in the selected layer and lower layers from the video stream on the basis of the layer identification information;
characterized in that
the control section (<NUM>) is configured to determine the highest decodable layer on the basis of the frequency information of the pictures in the lowest layer, the number of the plurality of layers in the container, and decoding performance of the receiving device.