IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD

The present technique relates to an image processing apparatus and an image processing method capable of suppressing deterioration in accuracy of a predicted image and reducing the number of storable reference images. A motion prediction/compensation unit generates a predicted image of an encoding target image by using a reference image. A frame memory is, for example, a decoded picture buffer (DPB) and preferentially stores the reference image of which display order is close to that of the encoding target image. The present technique may be applied to, for example, an encoding device using a high efficiency video coding (HEVC) scheme.

TECHNICAL FIELD

The present technique relates to an image processing apparatus and an image processing method and, more particularly, to an image processing apparatus and an image processing method capable of suppressing deterioration in accuracy of a predicted image and reducing the number of storable reference images.

BACKGROUND ART

In recent years, apparatuses according to moving picture experts group phase (MPEG) schemes or the like in which image information is treated as digital data and, at this time, for the purpose of high-efficiency information transmission and accumulation, compression is performed through orthogonal transform such as discrete cosine transform and motion compensation by using redundancy unique to the image information have been widely used for information distribution of broadcasting stations or the like and for information reception at ordinary homes.

Particularly, the MPEG-2 (ISO/IEC 13818-2) scheme is defined as a general-purpose image encoding scheme, and as a standard covering both of interlaced scanning images and sequential scanning images and covering standard resolution images and high-accuracy images, the MPEG-2 scheme is widely used for a wide range of applications of professional uses and consumer uses. By using the MPEG-2 scheme, for example, a bit rate of 4 to 8 Mbps is allocated to an interlaced scanning image having a standard resolution of 720×480 pixels, and a bit rate of 18 to 22 Mbps is allocated to an interlaced scanning image having a high resolution of 1920×1088 pixels, so that a high compression rate and a good image quality may be implemented.

The MPEG-2 is mainly applied to high image quality encoding which is suitable for broadcasting, but it does not correspond to an encoding scheme having a bit rate lower than that of the MPEG-1, that is, an encoding scheme having a higher compression rate. With the spread of mobile phones, needs for the encoding scheme are expected to be increased, and accordingly, the MPEG-4 encoding scheme is standardized.

With respect to the image encoding scheme of the MPEG-4, the ISO/IEC 14496-2 standard was approved as an international standard in December, 1998.

In addition, in recent years, for the purpose of image encoding for TV conference, standardization called H.26L (ITU-T Q6/16 VCEG) has been promoted. It is known that, in comparison with the encoding schemes such as the MPEG-2 or the MPEG-4 in the related art, in the H.26L, although a large calculation amount is needed for encoding and decoding, a higher encoding efficiency is implemented.

In addition, at present, as a part of activities of the MPEG-4, standardization which is based on the H.26L and incorporates functions which are not supported in the H.26L to implement a higher encoding efficiency is performed as Joint Model of Enhanced-Compression Video Coding. The standard was approved as an international standard on the basis named H.264 and MPEG-4 Part 10 (AVC (Advanced Video Coding)) in March, 2003.

In addition, as extension thereof, standardization of Fidelity Range Extension (FRExt) including RGB, encoding tools necessary for business such as 4:2:2 or 4:4:4, 8×8 DCT defined by the MPEG-2, and quantization matrix was completed in February, 2005. Accordingly, the AVC became an encoding scheme capable of representing film noise included in a movie with a good quality. Therefore, the AVC has been used for a wide range of applications such as Blu-Ray (registered trade mark) disc.

However, recently, needs for high compression rate encoding, for example, a need to compress images of about 4000×2000 pixels which is four times of a high-vision image or a need to distribute a high-vision image in a limited-transmission-rate environment such as the Internet have been further increased. Therefore, in the video coding expert group (VCEG) under the ITU-T, improvement of an encoding efficiency continues to be studied.

Moreover, at present, for the purpose of further improvement of the encoding efficiency in comparison with the H.264/AVC, standardization of an encoding scheme called high efficiency video coding (HEVC) has been promoted by the joint collaboration team-video coding (JCTVC) as a joint standardization body of the ITU-T and the ISO/IEC. With respect to the HEVC standard, Committee Draft as a first draft specification was issued in February, 2012 (for example, refer to Non-Patent Document 1).

CITATION LIST

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

In the HEVC scheme, there is a desire to further reduce the number of reference images storable in a decoded picture buffer (DPB).

The present technique is to suppress a deterioration in accuracy of a predicted image and to reduce the number of storable reference images.

Solutions to Problems

An image processing apparatus according to an aspect of the present technique includes: a predicted image generation unit which generates a predicted image of an image by using a reference image; and a storage unit which preferentially stores the reference image of which display order is close to that of the image.

An image processing method according to an aspect of the present technique corresponds to an image processing apparatus according to an aspect of the present technique.

In an aspect of the present technique, a predicted image of an image is generated by using a reference image, and the reference image of which display order is close to that of the image is preferentially stored.

Effects of the Invention

According to the present technique, it is possible to suppress a deterioration in accuracy of a predicted image, and it is possible to reduce the number of storable reference images.

MODE FOR CARRYING OUT THE INVENTION

Embodiment

Configuration Example of Embodiment of Encoding Device

FIG. 1is a block diagram illustrating a configuration example of an embodiment of an encoding device employing the present technique.

The encoding device11illustrated inFIG. 1is configured to include an A/D converter31, a screen rearrangement buffer32, an arithmetic unit33, an orthogonal transform unit34, a quantization unit35, a lossless encoding unit36, an accumulation buffer37, an inverse quantization unit38, an inverse orthogonal transform unit39, an addition unit40, a deblocking filter41, an adaptive offset filter42, an adaptive loop filter43, a frame memory44, a switch45, an intra prediction unit46, a motion prediction/compensation unit47, a predicted image selection unit48, and a rate control unit49.

More specifically, the A/D converter31of the encoding device11A/D-converts frame-unit images input as input signals and outputs the A/D-converted images to the screen rearrangement buffer32to store the A/D-converted images. The screen rearrangement buffer32rearranges the frame-unit images, which are stored in the display order, in the order for encoding according to a GOP structure and outputs the rearranged images to the arithmetic unit33, the intra prediction unit46, and the motion prediction/compensation unit47.

The arithmetic unit33performs encoding by calculating a difference between a predicted image supplied from the predicted image selection unit48and an encoding target image output from the screen rearrangement buffer32. More specifically, the arithmetic unit33performs encoding by subtracting the predicted image supplied from the predicted image selection unit48from the encoding target image output from the screen rearrangement buffer32. The arithmetic unit33outputs the image obtained as a result thereof as residual information to the orthogonal transform unit34. In addition, in the case where the predicted image is not supplied from the predicted image selection unit48, the arithmetic unit33outputs the image read out from the screen rearrangement buffer32without change as the residual information to the orthogonal transform unit34.

The orthogonal transform unit34performs orthogonal transform on the residual information from the arithmetic unit33to generate an orthogonal transform coefficient. The orthogonal transform unit34supplies the generated orthogonal transform coefficient to the quantization unit35.

The quantization unit35performs quantization on the orthogonal transform coefficient supplied from the orthogonal transform unit34by using quantization parameters supplied from the rate control unit49. The quantization unit35inputs the coefficient obtained as a result thereof to the lossless encoding unit36.

The lossless encoding unit36acquires information (hereinafter, referred to as intra prediction mode information) representing an optimal intra prediction mode from the intra prediction unit46. In addition, the lossless encoding unit36acquires information (hereinafter, referred to as inter prediction mode information) representing an optimal inter prediction mode, motion vectors, and the like from the motion prediction/compensation unit47. In addition, the lossless encoding unit36acquires the quantization parameters from the rate control unit49.

In addition, the lossless encoding unit36acquires a storage flag, an index or an offset, and type information as offset filter information from the adaptive offset filter42and acquires a filter coefficient from the adaptive loop filter43.

The lossless encoding unit36performs lossless encoding such as variable length encoding (for example, context-adaptive variable length coding (CAVLC) or the like) and arithmetic encoding (for example, context-adaptive binary arithmetic coding (CABAC) or the like) on the quantized coefficient supplied from the quantization unit35.

In addition, the lossless encoding unit36performs lossless encoding on intra prediction mode information or inter prediction mode information, motion vectors, information for identifying a reference image or the like, quantization parameters, offset filter information, and a filter coefficient as encoding information on the encoding. The lossless encoding unit36supplies the lossless-encoded encoding information and the lossless-encoded coefficient as the encoding data to the accumulation buffer37to accumulate the encoding data. In addition, the lossless-encoded encoding information may be considered to be header information (slice header) of the lossless-encoded coefficient.

The accumulation buffer37temporarily stores the encoding data supplied from the lossless encoding unit36. In addition, the accumulation buffer37outputs the stored encoding data.

In addition, the quantized coefficient output from the quantization unit35is also input to the inverse quantization unit38. The inverse quantization unit38performs inverse quantization on the coefficient quantized by the quantization unit35by using the quantization parameters supplied from the rate control unit49and supplies the orthogonal transform coefficient obtained as a result thereof to the inverse orthogonal transform unit39.

The inverse orthogonal transform unit39performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit38. The inverse orthogonal transform unit39supplies residual information obtained as a result of the inverse orthogonal transform to the addition unit40.

The addition unit40obtains a locally decoded image by adding the residual information supplied from the inverse orthogonal transform unit39and the predicted image supplied from the predicted image selection unit48. In addition, in the case where the predicted image is not supplied from the predicted image selection unit48, the addition unit40defines the residual information supplied from the inverse orthogonal transform unit39as the locally decoded image. The addition unit40supplies the locally decoded image to the deblocking filter41and supplies the locally decoded image to the frame memory44to accumulate the locally decoded image.

The deblocking filter41performs an adaptive deblocking filtering process for removing block distortion on the locally decoded image supplied from the addition unit40and supplies the image obtained as a result thereof to the adaptive offset filter42.

The adaptive offset filter42performs an adaptive offset filter (SAO: Sample Adaptive Offset) process for mainly removing ringing on the image after the adaptive deblocking filtering process of the deblocking filter41.

More specifically, the adaptive offset filter42determines a type of the adaptive offset filtering processes for each largest coding unit (LCU) that is a maximum unit of encoding and obtains an offset used for the adaptive offset filtering process. The adaptive offset filter42performs the determined type of the adaptive offset filtering process on the image after the adaptive deblocking filtering process by using the obtained offset. Next, the adaptive offset filter42supplies the image after the adaptive offset filtering process to the adaptive loop filter43.

In addition, the adaptive offset filter42is configured to include a buffer which stores the offset. The adaptive offset filter42determines whether or not the offset used for the adaptive deblocking filtering process in each LCU is stored in the buffer in advance.

In the case where it is determined that the offset used for the adaptive deblocking filtering process is stored in the buffer in advance, the adaptive offset filter42sets a storage flag indicating that the offset is stored in the buffer to a value (herein, 1) representing that the offset is stored in the buffer.

Next, the adaptive offset filter42supplies the storage flag that is set to 1, an index indicating a storage position of the offset in the buffer, and the type information representing the type of the performed adaptive offset filtering process to the lossless encoding unit36in each LCU.

On the other hand, in the case where the offsets used for the adaptive deblocking filtering process are not yet stored in the buffer, the adaptive offset filter42stores the offsets in the buffer in order. In addition, the adaptive offset filter42sets the storage flag to a value (herein, 0) representing that the offset is not stored in the buffer. Next, the adaptive offset filter42supplies the storage flag that is set to 0, the offset, and the type information to the lossless encoding unit36in each LCU.

The adaptive loop filter43performs, for example, an adaptive loop filtering (ALF) process on the image after the adaptive offset filtering process supplied from the adaptive offset filter42in each LCU. As the adaptive loop filtering process, for example, a process using two-dimensional Wiener filter is used. In addition, filters other than the Wiener filter may be used.

More specifically, the adaptive loop filter43calculates a filter coefficient used for the adaptive loop filtering process in each LCU so that the residual between the original image that is the image output from the screen rearrangement buffer32and the image after the adaptive loop filtering process is minimized. Next, the adaptive loop filter43performs the adaptive loop filtering process on the image after the adaptive offset filtering process by using the calculated filter coefficient in each LCU.

The adaptive loop filter43supplies the image after the adaptive loop filtering process to the frame memory44. In addition, the adaptive loop filter43supplies the filter coefficient to the lossless encoding unit36.

In addition, herein, the adaptive loop filtering process is performed in each LCU. However, the processing unit of the adaptive loop filtering process is not limited to the LCU. If the processing unit of the adaptive offset filter42and the processing unit of the adaptive loop filter43are in accordance with each other, these processes may be efficiently performed.

The frame memory44is a DPB and accumulates the image supplied from the adaptive loop filter43or the image supplied from the addition unit40as the decoded image. The decoded image accumulated in the frame memory44is output as a reference image to the intra prediction unit46or the motion prediction/compensation unit47through the switch45.

The intra prediction unit46performs intra prediction processes in all the candidate intra prediction modes by using the reference image read out from the frame memory44through the switch45to generate a predicted image as an encoding target image.

In addition, the intra prediction unit46calculates cost function values (described later in detail) for all the candidate intra prediction modes based on the image read out from the screen rearrangement buffer32and the predicted image generated as a result of the intra prediction process. Next, the intra prediction unit46determines the intra prediction mode where the cost function value is minimized as an optimal intra prediction mode.

The intra prediction unit46supplies the predicted image generated in the optimal intra prediction mode and the corresponding cost function value to the predicted image selection unit48. In the case where the selection of the predicted image generated in the optimal intra prediction mode is notified from the predicted image selection unit48, the intra prediction unit46supplies the intra prediction mode information to the lossless encoding unit36.

In addition, the cost function value is also referred to as a rate distortion (RD) cost and is calculated based on any of methods of a high complexity mode or a low complexity mode which is defined by a joint model (JM) that is reference software in the H.264/AVC scheme.

More specifically, in the case where the high complexity mode is employed as the method of calculating the cost function value, the processes including the decoding are performed on all the candidate prediction modes, and the cost function value expressed by the following Formula (1) is calculated for each prediction mode.

D denotes a difference (distortion) between the original image and the decoded image; R denotes an occurrence bit rate including the coefficient of the orthogonal transform; and λ denotes a Lagrange multiplier given as a function of a quantization parameter QP.

On the other hand, in the case where the low complexity mode is employed as the method of calculating the cost function value, the predicted images are generated and the bit rate of the encoding information is calculated for all the candidate prediction modes, and the cost function expressed by the following Formula (2) is calculated for each prediction mode.

D denotes a difference (distortion) between the original image and the predicted image; Header_Bit denotes the bit rate of the encoding information; and QPtoQuant denotes a function given as a function of the quantization parameter QP.

In the low complexity mode, only the predicted image may be generated with respect to all the prediction modes, and since there is no need to generate the decoded image, a calculation amount may be small.

The motion prediction/compensation unit47performs a motion prediction/compensation process in all the candidate inter prediction modes. More specifically, the motion prediction/compensation unit47detects motion vectors in all the candidate inter prediction modes based on the image supplied from the screen rearrangement buffer32and the reference image read out from the frame memory44through the switch45. Next, the motion prediction/compensation unit47functions as a predicted image generation unit to apply a compensation process to the reference image based on the motion vector to generate the predicted image of the encoding target image.

At this time, the motion prediction/compensation unit47calculates the cost function values for all the candidate inter prediction modes based on the image supplied from the screen rearrangement buffer32and the predicted image and determines the inter prediction mode where the cost function value is minimized as an optimal inter prediction mode. Next, the motion prediction/compensation unit47supplies the cost function value in the optimal inter prediction mode and the corresponding predicted image to the predicted image selection unit48. In addition, in the case where the selection of the predicted image generated in the optimal inter prediction mode is notified from the predicted image selection unit48, the motion prediction/compensation unit47outputs inter prediction mode information, the corresponding motion vectors, information for identifying the reference image, and the like to the lossless encoding unit36.

The predicted image selection unit48determines the mode where the corresponding cost function value is small among the optimal intra prediction mode and the optimal inter prediction mode as an optimal prediction mode based on the cost function values supplied from the intra prediction unit46and the motion prediction/compensation unit47. Next, the predicted image selection unit48supplies the predicted image in the optimal prediction mode to the arithmetic unit33and the addition unit40. In addition, the predicted image selection unit48notifies the selection of the predicted image in the optimal prediction mode to the intra prediction unit46or the motion prediction/compensation unit47.

The rate control unit49determines the quantization parameter used for the quantization unit35based on the encoding data accumulated in the accumulation buffer37so that overflow or underflow does not occur. The rate control unit49supplies the determined quantization parameter to the quantization unit35, the lossless encoding unit36, and the inverse quantization unit38.

(First Example of Reference Image Stored in Frame Memory)

FIG. 2is a diagram illustrating the reference image stored in the frame memory44in the case where the number of reference images storable in the frame memory44is 6.

As illustrated inFIG. 2, in the case where the number of reference images storable in the frame memory44is 6, the decoded image of one encoding target image and the decoded images of 5 or less encoding-completed images are stored in the frame memory44. Namely, the frame memory44is configured to include a temporary storage area which temporarily stores the decoded image of one encoding target image and a long-term storage area which stores the decoded images of 5 or less encoding-completed images.

In addition, inFIG. 2, I indicates an I picture, and B indicates a B picture. In addition, the numbers following I or B indicate the display orders of the corresponding pictures. InFIG. 2, in the uppermost row, pictures are arranged and written in the encoding order (decoding order). In the second row from the top, the display orders (picture order counts (POCs)) of the pictures in the uppermost row are written. In the third row from the top, pictures displayed at the time of decoding the pictures in the uppermost row are written.

In addition, in the fourth to eighth rows from the top, pictures stored in the long-term storage area of the frame memory44at the time of encoding the pictures in the uppermost row are written. In the ninth row from the top, the display orders of the pictures used as the reference images in the L0 prediction at the time of encoding the pictures in the uppermost row are written. In the tenth row from the top, the display orders of the pictures used as the reference images in the L1 prediction at the time of encoding the pictures in the uppermost row are written. These are the same inFIGS. 3 to 5described later.

As illustrated inFIG. 2, the frame memory44stores the pictures of which displaying is not yet completed at the time of decoding the encoding target picture in the long-term storage area. On the other hand, the frame memory44does not store the pictures of which displaying is completed at the time of decoding the encoding target picture and which are not used as the reference images in the long-term storage area.

In addition, the frame memory44preferentially stores the pictures of which quantization parameter is small rather than the picture of which display order is close to the display order of the encoding target picture in the long-term storage area. For example, at the time of encoding the B picture (B5 picture) of which display order is 5, the frame memory44preferentially stores the I picture (I0 picture) of which display order is 0 and of which quantization parameter is small rather than the B picture (B2 picture) of which display order is 2 and is close to the display order of the B5 picture in the long-term storage area.

(Second Example of Reference Image Stored in Frame Memory)

FIG. 3is a diagram illustrating a first example of the reference image stored in the frame memory44in the case where the number of reference images storable in the frame memory44is 5.

As illustrated inFIG. 3, in the case where the number of reference images storable in the frame memory44is 5, the decoded image of one encoding target image and the decoded images of 4 or less encoding-completed images are stored in the frame memory44. Namely, the frame memory44is configured to include a temporary storage area which temporarily stores the decoded image of one encoding target image and a long-term storage area which stores the decoded images of 4 or less encoding-completed images.

As illustrated inFIG. 3, the frame memory44stores the pictures of which displaying is not yet completed at the time of decoding the encoding target picture in the long-term storage area. On the other hand, the frame memory44does not store the pictures of which displaying is completed at the time of decoding the encoding target picture and which are not used as the reference images in the long-term storage area.

In addition, the frame memory44preferentially stores the pictures of which display orders are close to the display order of the encoding target picture in the long-term storage area. For example, at the time of encoding the B picture (B6 picture) of which display order is 6, the frame memory44preferentially stores the B2 picture of which display order is 2 and is close to the display order of the B6 picture rather than the I0 picture of which display order is 0 in the long-term storage area.

(Third Example of Reference Image Stored in Frame Memory)

FIG. 4is a diagram illustrating a second example of the reference image stored in the frame memory44in the case where the number of reference images storable in the frame memory44is 5.

As described above, in the case where the number of reference images storable in the frame memory44is 5, the frame memory44is configured to include a temporary storage area which temporarily stores the decoded image of one encoding target image and a long-term storage area which stores the decoded images of 4 or less encoding-completed images.

As illustrated inFIG. 4, the frame memory44stores the pictures of which displaying is not yet completed at the time of decoding the encoding target picture in the long-term storage area. On the other hand, the frame memory44does not store the pictures of which displaying is completed at the time of decoding the encoding target picture and which are not used as the reference images in the long-term storage area.

In addition, the frame memory44partially preferentially stores the pictures of display order is close to the display order of the encoding target picture in the long-term storage area. For example, at the time of encoding the B6 picture of which display order is 6, the frame memory44preferentially stores the I0 picture of which display order is 0 and of which quantization parameter is small rather than the B2 picture of which display order is 2 and is close to the display order of the B6 picture in the long-term storage area.

On the other hand, at the time of encoding the B picture (B7 picture) of which display order is 7, the frame memory44preferentially stores the B picture (B4 picture) of which display order is 4 and is close to the display order of the B7 picture rather than the I0 picture of which display order is 0 in the long-term storage area.

(Fourth Example of Reference Image Stored in Frame Memory)

FIG. 5is a diagram illustrating a third example of the reference image stored in the frame memory44in the case where the number of reference images storable in the frame memory44is 5.

As described above, in the case where the number of reference images storable in the frame memory44is 5, the frame memory44is configured to include a temporary storage area which temporarily stores the decoded image of one encoding target image and a long-term storage area which stores the decoded images of 4 or less encoding-completed images.

As illustrated inFIG. 5, the frame memory44stores the pictures of which displaying is not yet completed at the time of decoding the encoding target picture in the long-term storage area. On the other hand, the frame memory44does not store the pictures of which displaying is completed at the time of decoding the encoding target picture and which are not used as the reference images in the long-term storage area.

In addition, the frame memory44preferentially stores the pictures of which quantization parameter is small rather than the picture of which display order is close to the display order of the encoding target picture in the long-term storage area. For example, at the time of encoding the B6 picture of which display order is 6, the frame memory44preferentially stores the I0 picture of which display order is 0 and of which quantization parameter is small rather than the B2 picture of which display order is 2 and is close to the display order of the B6 picture in the long-term storage area.

In addition, the number of reference images storable in the frame memory44is defined according to the size of the encoding target image, that is, the level of profile, or the like. For example, in the case where the encoding target image is large, the number of reference images storable in the frame memory44is set to 5; and in the case where the encoding target image is small, the number of reference images storable in the frame memory44is set to 6.

In addition, in the case where the number of reference images storable in the frame memory44is 5, the frame memory44may stores the reference images according to any of the methods illustrated inFIGS. 3 to 5. In addition, the methods illustrated inFIGS. 3 to 5may be switched according to the type of the encoding target image or the like. In this case, for example, in the case where the encoding target image is a moving image, the method illustrated inFIG. 3is used; and in the case where the encoding target image is a still image, the method illustrated inFIG. 5is used.

(Description of Processes of Encoding Device)

FIG. 6is a flowchart illustrating details of an encoding process of the encoding device11illustrated inFIG. 3.

In step S31illustrated inFIG. 6, the A/D converter31of the encoding device11A/D-converts frame-unit images input as input signals and outputs the A/D-converted images to the screen rearrangement buffer32to store the A/D-converted images.

In step S32, the screen rearrangement buffer32rearranges the frame-unit images, which are stored in the display order, in the order for encoding according to a GOP structure. The screen rearrangement buffer32supplies the frame-unit images after the rearrangement to the arithmetic unit33, the intra prediction unit46, and the motion prediction/compensation unit47.

In step S33, the intra prediction unit46performs intra prediction processes in all the candidate intra prediction modes. In addition, the intra prediction unit46calculates cost function values for all the candidate intra prediction modes based on the image read out from the screen rearrangement buffer32and the predicted image generated as a result of the intra prediction process. Next, the intra prediction unit46determines the intra prediction mode where the cost function value is minimized as an optimal intra prediction mode. The intra prediction unit46supplies the predicted image generated in the optimal intra prediction mode and the corresponding cost function value to the predicted image selection unit48.

In addition, the motion prediction/compensation unit47performs motion prediction/compensation processes in all the candidate inter prediction modes. In addition, the motion prediction/compensation unit47calculates cost function values for all the candidate inter prediction modes based on the image supplied from the screen rearrangement buffer32and the predicted image and determines the inter prediction mode where the cost function value is minimized as an optimal inter prediction mode. Next, the motion prediction/compensation unit47supplies the cost function value in the optimal inter prediction mode and the corresponding predicted image to the predicted image selection unit48.

In step S34, the predicted image selection unit48determines the mode where the cost function value is minimized among the optimal intra prediction mode and the optimal inter prediction mode as an optimal prediction mode based on the cost function values supplied from the intra prediction unit46and the motion prediction/compensation unit47through the process of step S33. Next, the predicted image selection unit48supplies the predicted image in the optimal prediction mode to the arithmetic unit33and the addition unit40.

In step S35, the predicted image selection unit48determines whether or not the optimal prediction mode is an optimal inter prediction mode. In the case where it is determined in step S35that the optimal prediction mode is an optimal inter prediction mode, the predicted image selection unit48notifies the selection of the predicted image generated in the optimal inter prediction mode to the motion prediction/compensation unit47.

Next, in step S36, the motion prediction/compensation unit47supplies inter prediction mode information, the corresponding motion vectors, information for identifying the reference image, and the like to the lossless encoding unit36, and the process proceeds to step S38.

On the other hand, in the case where it is determined in step S35that the optimal prediction mode is not an optimal inter prediction mode, that is, in the case where the optimal prediction mode is an optimal intra prediction mode, the predicted image selection unit48notifies the selection of the predicted image generated in the optimal intra prediction mode to the intra prediction unit46. Next, in step S37, the intra prediction unit46supplies the intra prediction mode information to the lossless encoding unit36, and the process proceeds to step S38.

In step S38, the arithmetic unit33performs encoding by subtracting the predicted image supplied from the predicted image selection unit48from the image supplied from the screen rearrangement buffer32. The arithmetic unit33outputs the image obtained as a result thereof as residual information to the orthogonal transform unit34.

In step S39, the orthogonal transform unit34performs orthogonal transform on the residual information from the arithmetic unit33and supplies the orthogonal transform coefficient obtained as a result thereof to the quantization unit35.

In step S40, the quantization unit35performs quantization on the coefficient supplied from the orthogonal transform unit34by using the quantization parameters supplied from the rate control unit49. The quantized coefficient is input to the lossless encoding unit36and the inverse quantization unit38.

In step S41illustrated inFIG. 7, the inverse quantization unit38performs inverse quantization on the quantized coefficient supplied from the quantization unit35by using the quantization parameters supplied from the rate control unit49and supplies the orthogonal transform coefficient obtained as a result thereof to the inverse orthogonal transform unit39.

In step S42, the inverse orthogonal transform unit39performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit38and supplies the residual information obtained as a result thereof to the addition unit40.

In step S43, the addition unit40obtains a locally decoded image by adding the residual information supplied from the inverse orthogonal transform unit39and the predicted image supplied from the predicted image selection unit48. The addition unit40supplies the obtained image to the deblocking filter41and supplies the obtained image to the frame memory44.

In step S44, the deblocking filter41performs a deblocking filtering process on the locally decoded image supplied from the addition unit40. The deblocking filter41supplies the image obtained as a result thereof to the adaptive offset filter42.

In step S45, the adaptive offset filter42performs an adaptive offset filtering process on the image supplied from the deblocking filter41in each LCU. The adaptive offset filter42supplies the image obtained as a result thereof to the adaptive loop filter43. In addition, the adaptive offset filter42supplies a storage flag, an index or an offset, and type information as offset filter information to the lossless encoding unit36in each LCU.

In step S46, the adaptive loop filter43performs an adaptive loop filtering process on the image supplied from the adaptive offset filter42in each LCU. The adaptive loop filter43supplies the image obtained as a result thereof to the frame memory44. In addition, the adaptive loop filter43supplies a filter coefficient used for the adaptive loop filtering process to the lossless encoding unit36.

In step S47, as described inFIGS. 2 to 5, the frame memory44accumulates the image supplied from the adaptive loop filter43or the image supplied from the addition unit40. The image accumulated in the frame memory44is output as a reference image to the intra prediction unit46or the motion prediction/compensation unit47through the switch45.

In step S48, the lossless encoding unit36performs lossless encoding on intra prediction mode information or inter prediction mode information, motion vectors, information for identifying the reference image or the like, quantization parameters from the rate control unit49, offset filter information, and a filter coefficient as encoding information.

In step S49, the lossless encoding unit36performs lossless encoding on the quantized coefficient supplied from the quantization unit35. Next, the lossless encoding unit36generates encoding data from the lossless-encoded encoding information and the lossless-encoded coefficient which are lossless-encoded in step S48.

In step S50, the accumulation buffer37temporarily accumulates the encoding data supplied from the lossless encoding unit36.

In step S51, the rate control unit49determinates the quantization parameter used for the quantization unit35based on the encoding data accumulated in the accumulation buffer37so that overflow or underflow does not occur. The rate control unit49supplies the determined quantization parameter to the quantization unit35, the lossless encoding unit36, and the inverse quantization unit38.

In step S52, the accumulation buffer37outputs the stored encoding data.

In addition, in the encoding process illustrated inFIGS. 6 and 7, for simplification of the description, the intra prediction process and the motion prediction/compensation process are always performed. However, in actual cases, only one thereof may be performed according to the type of picture or the like.

As described above, the frame memory44of the encoding device11performs storing in the manner as described inFIGS. 3 to 5, so that it is possible to reduce the number of storable reference images down to 5. In addition, as described inFIGS. 3 and 4, the frame memory44preferentially stores the decoded image of which display order is close to the display order of the encoding target image as the reference image, so that in the case where the encoding target image is a moving image or the like, it is possible to suppress a deterioration in accuracy of the predicted image.

In addition, as described inFIGS. 4 and 5, the frame memory44preferentially stores the decoded image of which quantization parameter is small as the reference image, so that in the case where the encoding target image is a still image or the like, it is possible to suppress a deterioration in accuracy of the predicted image.

(Configuration Example of Embodiment of Decoding Device)

FIG. 8is a block diagram illustrating a configuration example of an embodiment of a decoding device employing the present technique which decodes an encoding stream transmitted from the encoding device11illustrated inFIG. 3.

The decoding device113illustrated inFIG. 8is configured to include an accumulation buffer131, a lossless decoding unit132, an inverse quantization unit133, an inverse orthogonal transform unit134, an addition unit135, a deblocking filter136, an adaptive offset filter137, an adaptive loop filter138, a screen rearrangement buffer139, a D/A converter140, a frame memory141, a switch142, an intra prediction unit143, a motion compensation unit144, and a switch145.

The accumulation buffer131of the decoding device113receives encoding data transmitted from the encoding device11illustrated inFIG. 3and accumulates the encoding data. The accumulation buffer131supplies the accumulated encoding data to the lossless decoding unit132.

The lossless decoding unit132performs lossless decoding such as variable length decoding or arithmetic decoding on the encoding data from the accumulation buffer131to obtain a quantized coefficient and encoding information. The lossless decoding unit132supplies the quantized coefficient to the inverse quantization unit133. In addition, the lossless decoding unit132supplies intra prediction mode information and the like as the encoding information to the intra prediction unit143and supplies motion vectors, inter prediction mode information, information for identifying a reference image, and the like to the motion compensation unit144.

In addition, the lossless decoding unit132supplies the intra prediction mode information or the inter prediction mode information as the encoding information to the switch145. The lossless decoding unit132supplies offset filter information as the encoding information to the adaptive offset filter137and supplies a filter coefficient to the adaptive loop filter138.

The inverse quantization unit133, the inverse orthogonal transform unit134, the addition unit135, the deblocking filter136, the adaptive offset filter137, the adaptive loop filter138, the frame memory141, the switch142, the intra prediction unit143, and the motion compensation unit144performs the same processes of those of the inverse quantization unit38, the inverse orthogonal transform unit39, the addition unit40, the deblocking filter41, the adaptive offset filter42, the adaptive loop filter43, the frame memory44, the switch45, the intra prediction unit46, and the motion prediction/compensation unit47illustrated inFIG. 4, so that the image is decoded.

More specifically, the inverse quantization unit133performs inverse quantization on the quantized coefficient from the lossless decoding unit132and supplies an orthogonal transform coefficient obtained as a result thereof to the inverse orthogonal transform unit134.

The inverse orthogonal transform unit134performs inverse orthogonal transform on the orthogonal transform coefficient from the inverse quantization unit133. The inverse orthogonal transform unit134supplies residual information obtained as a result of the inverse orthogonal transform to the addition unit135.

The addition unit135performs decoding by adding the residual information as a decoding target image supplied from the inverse orthogonal transform unit134and the predicted image supplied from the switch145. The addition unit135supplies the image obtained as a result of the decoding to the deblocking filter136and supplies the image to the frame memory141. In addition, in the case where the predicted image is not supplied from the switch145, the addition unit135supplies the image that is the residual information supplied from the inverse orthogonal transform unit134as the image obtained as a result of the decoding to the deblocking filter136and supplies the image to the frame memory141.

The deblocking filter136performs an adaptive deblocking filtering process on the image supplied from the addition unit135and supplies the image obtained as a result thereof to the adaptive offset filter137.

The adaptive offset filter137is configured to include a buffer which stores the offsets supplied from the lossless decoding unit132in order. In addition, the adaptive offset filter137performs an adaptive offset filtering process on the image after the adaptive deblocking filtering process of the deblocking filter136based on the offset filter information supplied from the lossless decoding unit132in each LCU.

More specifically, in the case where the storage flag included in the offset filter information is 0, the adaptive offset filter137performs an adaptive offset filtering process corresponding to the type indicated by the type information on the image after the deblocking filtering process in each LCU by using the offset included in the offset filter information.

On the other hand, in the case where the storage flag included in the offset filter information is 1, the adaptive offset filter137reads out the offset stored in the position indicated by the index included in the offset filter information with respect to the image after the deblocking filtering process in each LCU. Next, the adaptive offset filter137performs the adaptive offset filtering process corresponding to the type indicated by the type information by using the read-out offset. The adaptive offset filter137supplies the image after of the adaptive offset filtering process to the adaptive loop filter138.

The adaptive loop filter138performs the adaptive loop filtering process in each LCU on the image supplied from the adaptive offset filter137by using the filter coefficient supplied from the lossless decoding unit132. The adaptive loop filter138supplies the image obtained as a result thereof to the frame memory141and the screen rearrangement buffer139.

The screen rearrangement buffer139stores the images supplied from the adaptive loop filter138in units of a frame. The screen rearrangement buffer139rearranges the frame-unit images, which are stored in the order for encoding, in the original display order and supplies the rearranged images to the D/A converter140.

The D/A converter140D/A-converts the frame-unit image supplied from the screen rearrangement buffer139and outputs the D/A-converted images as output signals.

Similarly to the frame memory44, the frame memory141is a DPB and accumulates the image supplied from the adaptive loop filter138or the image supplied from the addition unit135as the decoded image. More specifically, the information designating the decoded image stored in the frame memory44illustrated inFIG. 1, the information designating the methods illustrated inFIGS. 2 to 5, or the like is transmitted from the encoding device11. Similarly to the frame memory44, the frame memory141controls the storing of the decoded image based on the information transmitted from the encoding device11. The image accumulated in the frame memory141is read out as the reference image and is supplied to the motion compensation unit144or the intra prediction unit143through the switch142.

The intra prediction unit143performs the intra prediction process in the intra prediction mode indicated by the intra prediction mode information supplied from the lossless decoding unit132by using the reference image read out from the frame memory141through the switch142. The intra prediction unit143supplies the predicted image of the decoding target image generated as a result thereof to the switch145.

The motion compensation unit144reads out the reference image from the frame memory141through the switch142based on the information for identifying the reference image supplied from the lossless decoding unit132. The motion compensation unit144functions as a predicted image generation unit to perform a motion compensation process in the optimal inter prediction mode indicated by the inter prediction mode information by using the motion vector and the reference image. The motion compensation unit144supplies the predicted image of the decoding target image generated as a result thereof to the switch145.

In the case where the intra prediction mode information is supplied from the lossless decoding unit132, the switch145supplies the predicted image supplied from the intra prediction unit143to the addition unit135. On the other hand, in the case where the inter prediction mode information is supplied from the lossless decoding unit132, the switch145supplies the predicted image supplied from the motion compensation unit144to the addition unit135.

(Description of Processes of Decoding Device)

FIG. 9is a flowchart illustrating details of a decoding process of the decoding device113illustrated inFIG. 8.

In step S131illustrated inFIG. 9, the accumulation buffer131of the decoding device113receives the frame-unit encoding data transmitted from the encoding device11and accumulates the frame-unit encoding data. The accumulation buffer131supplies the accumulated encoding data to the lossless decoding unit132.

In step S132, the lossless decoding unit132performs lossless decoding on the encoding data from the accumulation buffer131to obtain the quantized coefficient and the encoding information. The lossless decoding unit132supplies the quantized coefficient to the inverse quantization unit133. In addition, the lossless decoding unit132supplies the intra prediction mode information or the like as the encoding information to the intra prediction unit143and supplies the motion vectors, the inter prediction mode information, the information for identifying the reference image, and the like to the motion compensation unit144.

In addition, the lossless decoding unit132supplies the intra prediction mode information or the inter prediction mode information as the encoding information to the switch145. The lossless decoding unit132supplies the offset filter information as the encoding information to the adaptive offset filter137and supplies the filter coefficient to the adaptive loop filter138.

In step S133, the inverse quantization unit133performs inverse quantization on the quantized coefficient from the lossless decoding unit132and supplies the orthogonal transform coefficient obtained as a result thereof to the inverse orthogonal transform unit134.

In step S134, the motion compensation unit144determines whether or not the inter prediction mode information is supplied from the lossless decoding unit132. In the case where it is determined in step S134that the inter prediction mode information is supplied, the process proceeds to step S135.

In step S135, the motion compensation unit144reads out the reference image based on the information for identifying the reference image supplied from the lossless decoding unit132and performs a motion compensation process in the optimal inter prediction mode indicated by the inter prediction mode information by using the motion vector and the reference image. The motion compensation unit144supplies the predicted image generated as a result thereof to the addition unit135through the switch145, and the process proceeds to step S137.

On the other hand, in the case where it is determined in step S134that the inter prediction mode information is not supplied, that is, in the case where the intra prediction mode information is supplied to the intra prediction unit143, the process proceeds to step S136.

In step S136, the intra prediction unit143performs an intra prediction process in the intra prediction mode indicated by the intra prediction mode information by using the reference image read out from the frame memory141through the switch142. The intra prediction unit143supplies the predicted image generated as a result of the intra prediction process to the addition unit135through the switch145, and the process proceeds to step S137.

In step S137, the inverse orthogonal transform unit134performs inverse orthogonal transform on the orthogonal transform coefficient from the inverse quantization unit133and supplies the residual information obtained as a result thereof to the addition unit135.

In step S138, the addition unit135adds the residual information supplied from the inverse orthogonal transform unit134and the predicted image supplied from the switch145. The addition unit135supplies the image obtained as a result thereof to the deblocking filter136and supplies the image to the frame memory141.

In step S139, the deblocking filter136performs a deblocking filtering process on the image supplied from the addition unit135to remove block distortion. The deblocking filter136supplies the image obtained as a result thereof to the adaptive offset filter137.

In step S140, the adaptive offset filter137performs the adaptive offset filtering process in each LCU on the image after the deblocking filtering process of the deblocking filter136based on the offset filter information supplied from the lossless decoding unit132. The adaptive offset filter137supplies the image after the adaptive offset filtering process to the adaptive loop filter138.

In step S141, the adaptive loop filter138performs an adaptive loop filtering process on the image supplied from the adaptive offset filter137in each LCU by using the filter coefficient supplied from the lossless decoding unit132. The adaptive loop filter138supplies the image obtained as a result thereof to the frame memory141and the screen rearrangement buffer139.

In step S142, the frame memory141accumulates the image supplied from the addition unit135or the image supplied from the adaptive loop filter138by the method illustrated inFIGS. 2 to 5similarly to the frame memory44illustrated inFIG. 1. The image accumulated in the frame memory141is supplied as the reference image to the motion compensation unit144or the intra prediction unit143through the switch142.

In step S143, the screen rearrangement buffer139stores the images supplied from the adaptive loop filter138in units of a frame, rearranges the frame-unit images, which are stored in the order for encoding, in the original display order, and supplies the rearranged images to the D/A converter140.

In step S144, the D/A converter140D/A-converts the frame-unit images supplied from the screen rearrangement buffer139and outputs the D/A-converted images as output signals, and after that, the process is ended.

As described above, similarly to the frame memory44, the frame memory141of the decoding device113stores the decoded image by the method illustrated inFIGS. 3 to 5, so that it is possible to reduce the number of storable reference images down to 5. In addition, the frame memory141preferentially stores the decoded image of which display order is close to the display order of the encoding target image as the reference image by the method illustrated inFIGS. 3 and 4, so that in the case where the encoding target image is a moving image or the like, it is possible to suppress a deterioration in accuracy of the predicted image.

In addition, the frame memory141preferentially stores the decoded image of which quantization parameter is small as the reference image by the method illustrated inFIGS. 4 and 5, so that in the case where encoding target image is a still image or the like, it is possible to suppress a deterioration in accuracy of the predicted image.

(Application to Multiple Viewpoint Image Encoding/Multiple Viewpoint Image Decoding)

A series of the processes described above may be applied to multiple viewpoint image encoding/multiple viewpoint image decoding.FIG. 10illustrates an example of a multiple viewpoint image encoding scheme.

As illustrated inFIG. 10, the multiple viewpoint image includes images of multiple viewpoints, and the image of a predetermined viewpoint among the multiple viewpoints is designated as an image of a base view. The image of each viewpoint other than the image of the base view is treated as an image of a non-base view.

In case of performing the multiple viewpoint image encoding illustrated inFIG. 10, the image of each view is encoded/decoded. However, the methods of the above-described embodiments may be applied to the encoding/decoding of each view. By performing the above-described processes, it is possible to suppress a deterioration in accuracy of the predicted image, and it is possible to reduce the number of storable reference images.

In addition, a difference between the quantization parameters may be taken in each view (the same view).

In case of performing the multiple viewpoint image encoding, a difference between the quantization parameters may be taken in each view (different view).

In this case, a combination of the above-described (1) to (4) may be used. For example, in the non-base view, a method (a combination of 3-1 and 2-3) of taking a difference in quantization parameter at a slice level between the base view and the non-base view and a method (a combination of 3-2 and 2-1) of taking a difference in quantization parameter at an LCU level between the base view and the non-base view are considered. In this manner, by repetitively applying the difference, even in the case where the multiple viewpoint encoding is performed, it is possible to improve an encoding efficiency.

Similarly to the above-described methods, a flag identifying whether or not a dQP of which value is not 0 exits may be set with respect to each dQP described above.

(Configuration Example of Multiple Viewpoint Image Encoding Device)

FIG. 11is a diagram illustrating a multiple viewpoint image encoding device which performs the above-described multiple viewpoint image encoding. As illustrated inFIG. 11, the multiple viewpoint image encoding device600is configured to include an encoding unit601, an encoding unit602, and a multiplexer603.

The encoding unit601encodes a base view image to generate a base view image encoding stream. The encoding unit602encodes a non-base view image to generate s non-base view image encoding stream. The multiplexer603multiplexes the base view image encoding stream generated in the encoding unit601and the non-base view image encoding stream generated in the encoding unit602to generate a multiple viewpoint image encoding stream.

The encoding device11may be applied to the encoding unit601and the encoding unit602of the multiple viewpoint image encoding device600. In this case, the multiple viewpoint image encoding device600sets a difference value between the quantization parameter set by the encoding unit601and the quantization parameter set by the encoding unit602and transmits the difference value.

(Configuration Example of Multiple Viewpoint Image Decoding Device)

FIG. 12is a diagram illustrating a multiple viewpoint image decoding device which performs the above-described multiple viewpoint image decoding. As illustrated inFIG. 12, the multiple viewpoint image decoding device610is configured to include a demultiplexer611, a decoding unit612, and a decoding unit613.

The demultiplexer611demultiplexes a multiple viewpoint image encoding stream where a base view image encoding stream and a non-base view image encoding stream are multiplexed to extract the base view image encoding stream and the non-base view image encoding stream. The decoding unit612decodes a base view image encoding stream extracted by the demultiplexer611to obtain a base view image. The decoding unit613decodes a non-base view image encoding stream extracted by the demultiplexer611to obtain a non-base view image.

The decoding device113may be applied to the decoding unit612and the decoding unit613of the multiple viewpoint image decoding device610. In this case, the multiple viewpoint image decoding device610sets a quantization parameter from a difference value between the quantization parameter set by the encoding unit601and the quantization parameter set by the encoding unit602and performs inverse quantization.

(Application to Hierarchical Image Encoding/Hierarchical Image Decoding)

A series of the processes described above may be applied to hierarchical image encoding/hierarchical image decoding.FIG. 13illustrates an example of a multiple viewpoint image encoding scheme.

As illustrated inFIG. 13, the hierarchical image includes images of multiple layers so as to have a scalability function for a predetermined parameter, and the image of a predetermined layer among the multiple layers is designated as an image of a base layer. The image of each layer other than the image of the base layer is treated as an image of a non-base layer.

In case of performing the hierarchical image encoding illustrated inFIG. 13, a difference of the quantization parameters may be taken in each layer (the same layer).

In case of performing the hierarchical encoding, a difference between the quantization parameters may be taken in each layer (different layer).

In this case, a combination of the above-described (1) to (4) may be used. For example, in the non-base layer, a method (a combination of 3-1 and 2-3) of taking a difference in quantization parameter at a slice level between the base layer and the non-base layer and a method (a combination of 3-2 and 2-1) of taking a difference in quantization parameter at an LCU level between the base layer and the non-base layer are considered. In this manner, by repetitively applying the difference, even in the case where the hierarchical encoding is performed, it is possible to improve an encoding efficiency.

Similarly to the above-described methods, a flag identifying whether or not a dQP of which value is not 0 exits may be set with respect to each dQP described above.

In the hierarchical image encoding/hierarchical image decoding (scalable encoding/scalable decoding), a parameter having a scalability function is arbitrary. For example, a spatial resolution illustrated inFIG. 14may be defined as the parameter (spatial scalability). In case of the spatial scalability, the resolution of the image is different among the layers. Namely, in this case, as illustrated inFIG. 14, each picture is hierarchized into two layers of a base layer having a resolution lower than that of an original image and an enhancement layer having an original spatial resolution by combining with the base layer. The number of layers is exemplary, and each picture may be hierarchized into an arbitrary number of layers.

In addition, besides, as the parameters providing the scalability, for example, a temporal resolution illustrated inFIG. 15may be applied (temporal scalability). In case of the temporal scalability, the frame rate is different among the layers. Namely, in this case, as illustrated inFIG. 15, each picture is hierarchized into two layers of a base layer having a resolution lower than that of an original moving image and an enhancement layer having an original frame rate by combining with the base layer. The number of layers is exemplary, and each picture may be hierarchized into an arbitrary number of layers.

In addition, as the parameters providing the scalability, for example, a signal-to-noise ratio (SNR) may be applied (SNR scalability). In case of the SNR scalability, the SN ratio is different among the layers. Namely, in this case, as illustrated inFIG. 16, each picture is hierarchized into two layers of a base layer having an SNR lower than that of an original image and an enhancement layer having an original SNR by combining with the base layer. The number of layers is exemplary, and each picture may be hierarchized into an arbitrary number of layers.

Besides the above-described examples, other parameters providing the scalability may be used. For example, as the parameters providing the scalability, a bit depth may also be used (bit-depth scalability). In case of the bit-depth scalability, the bit depth is different among the layers. In this case, for example, the base layer is configured with an 8-bit image, and by adding the enhancement layer to the base layer, a 10-bit image may be obtained.

In addition, as the parameters providing the scalability, a chroma format may also be used (chroma scalability). In case of the chroma scalability, the chroma format is different among the layers. In this case, for example, the base layer is configured with a 4:2:0-format component image, and by adding the enhancement layer to the base layer, a 4:2:2-format component image may be obtained.

(Configuration Example of Hierarchical Image Encoding Device)

FIG. 17is a diagram illustrating a hierarchical image encoding device which performs the above-described hierarchical image encoding. As illustrated inFIG. 17, the hierarchical image encoding device620is configured to include an encoding unit621, an encoding unit622, and a multiplexer623.

The encoding unit621encodes a base layer image to generate a base layer image encoding stream. The encoding unit622encodes a non-base layer image to generate a non-base layer image encoding stream. The multiplexer623multiplexes the base layer image encoding stream generated in the encoding unit621and the non-base layer image encoding stream generated in the encoding unit622to generate a hierarchical image encoding stream.

The encoding device11may be applied to the encoding unit621and the encoding unit622of the hierarchical image encoding device620. In this case, the hierarchical image encoding device620sets a difference value between the quantization parameter set by the encoding unit621and the quantization parameter set by the encoding unit622and transmits the difference value.

(Configuration Example of Hierarchical Image Decoding Device)

FIG. 18is a diagram illustrating a hierarchical image decoding device which performs the above-described hierarchical image decoding. As illustrated inFIG. 18, the hierarchical image decoding device630is configured to include a demultiplexer631, a decoding unit632, and a decoding unit633.

The demultiplexer631demultiplexes a hierarchical image encoding stream where a base layer image encoding stream and a non-base layer image encoding stream are multiplexed to extract the base layer image encoding stream and the non-base layer image encoding stream. The decoding unit632decodes the base layer image encoding stream extracted by the demultiplexer631to obtain a base layer image. The decoding unit633decodes the non-base layer image encoding stream extracted by the demultiplexer631to obtain a non-base layer image.

The decoding device113may be applied to the decoding unit632and the decoding unit633of the hierarchical image decoding device630. In this case, hierarchical image decoding device630sets a quantization parameter from a difference value between the quantization parameter set by the encoding unit621and the quantization parameter set by the encoding unit622and performs inverse quantization.

(Description of Computer Employing the Present Technique)

A series of the above-described processes may be executed by hardware or by software. In the case where a series of the processes is executed by software, a program constituting the software is installed in a computer. Herein, the computer includes a computer which is assembled in dedicated hardware, a general-purpose personal computer where various programs are installed to execute various functions, and the like.

FIG. 19is a block diagram illustrating a configuration example of hardware of the computer which executes a series of the above-described processes by a program.

In the computer, a central processing unit (CPU)801, a read only memory (ROM)802, and a random access memory (RAM)803are connected to each other via a bus804.

In addition, an input/output interface805is connected to the bus804. An input unit806, an output unit807, a storage unit808, a communication unit809, and a drive810are connected to the input/output interface805.

The input unit806is configured with a keyboard, a mouse, a microphone, or the like. The output unit807is configured with a display, a speaker, or the like. The storage unit808is configured with a hard disk, a non-volatile memory, or the like. The communication unit809is configured with a network interface or the like. The drive810drives a removable medium811such as a magnetic disk, an optical disk, an optical magnetic disk, or a semiconductor memory.

In the computer having the above-described configuration, for example, the CPU801loads a program stored in the storage unit808on the RAM803through the input/output interface805and the bus804and executes the program, so that a series of the above-described processes are performed.

The program executed by the computer (CPU801) may be provided in a manner that the program is recorded in the removable medium811, for example, a package medium, or the like. In addition, the program may be provided through a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the removable medium811is mounted on the drive810, so that the program may be installed in the storage unit808through the input/output interface805. In addition, the program may be received by the communication unit809through the wired or wireless transmission medium to be installed in the storage unit808. Otherwise, the program may be installed in the ROM802or the storage unit808in advance.

In addition, the program executed by the computer may be a program which performs processes in time sequence according to a procedure described in the specification, or the program may be a program which performs processes in parallel or at a necessary timing such as a time when a call is made.

(Configuration Example of Television Apparatus)

FIG. 20illustrates an example of a schematic configuration of a television apparatus employing the present technique. The television apparatus900is configured to include an antenna901, a tuner902, a demultiplexer903, a decoder904, a video signal processing unit905, a display unit906, an audio signal processing unit907, a speaker908, and an external interface unit909. In addition, the television apparatus900is configured to further include a controller910, a user interface unit911, and the like.

The tuner902selects a desired channel from broadcast wave signals received through the antenna901to perform demodulation and outputs an obtained encoded bit stream to the demultiplexer903.

The demultiplexer903extracts video or audio packets of a program which is to be viewed from the encoded bit stream and outputs data of the extracted packets to the decoder904. In addition, the demultiplexer903supplies the packets of the data such as an electronic program guide (EPG) to the controller910. In addition, in case of performing scrambling, descrambling is performed by the demultiplexer or the like.

The decoder904performs a packet decoding process and outputs video data generated by the decoding process to the video signal processing unit905and outputs audio data to the audio signal processing unit907.

The video signal processing unit905performs noise removing, video processing according to user setting, or the like on the video data. The video signal processing unit905generates video data of the program displayed on the display unit906, image data by a process based on an application supplied through the network, or the like. In addition, the video signal processing unit905generates video data for displaying a menu screen of item selection and the like and overlaps the video data of the program with the video data for displaying the menu screen. The video signal processing unit905generates a drive signal based on the video data generated above to drive the display unit906.

The display unit906drives a display device (for example, a liquid crystal display device or the like) based on the drive signal from the video signal processing unit905to display the video and the like of the program.

The audio signal processing unit907applies a predetermined process such as noise removing on the audio data, performs a D/A conversion process or an amplification process on the audio data after the process, and supplies the audio data to the speaker908, so that the audio outputting is performed.

The external interface unit909is an interface for connecting to an external device or a network and performs data transmission/reception of the video data, the audio data, and the like.

The user interface unit911is connected to the controller910. The user interface unit911is configured with a manipulation switch, a remote control signal reception unit, and the like and supplies a manipulation signal according to user manipulation to the controller910.

The controller910is configured by using a central processing unit (CPU), a memory, and the like. The memory stores programs executed by the CPU, various data necessary for the CPU to perform processes, EPG data, data acquired through the network, or the like. The program stored in the memory is read out and executed by the CPU at a predetermined timing such as a startup time of the television apparatus900. The CPU executes the program to control each component so that the television apparatus900is operated according to user manipulation.

In addition, in the television apparatus900, a bus912is installed so as to connect the controller910to the tuner902, the demultiplexer903, the video signal processing unit905, the audio signal processing unit907, the external interface unit909, and the like.

In the television apparatus having the above-described configuration, the functions of the image processing apparatus (image processing method) according to the present application are installed in the decoder904. Therefore, it is possible to suppress a deterioration in accuracy of the predicted image, and it is possible to reduce the number of storable reference images.

(Configuration Example of Mobile Phone)

FIG. 21illustrates an example of a schematic configuration of a mobile phone employing the present technique. The mobile phone920is configured to include a communication unit922, an audio codec923, a camera unit926, an image processing unit927, a multiplexing/separating unit928, a recording/reproducing unit929, a display unit930, and a controller931. These components are connected to each other via a bus933.

In addition, an antenna921is connected to the communication unit922, and a speaker924and a microphone925are connected to the audio codec923. In addition, a manipulation unit932is connected to the controller931.

The mobile phone920performs various operations such as transmission/reception of audio signals, transmission/reception of electronic mails or image data, image capturing, or data recording in various modes such as an audio communication mode or a data communication mode.

In the audio communication mode, the audio signal generated by the microphone925is converted into audio data or data-compressed by the audio codec923and is supplied to the communication unit922. The communication unit922performs a modulation process, a frequency conversion process, and the like on the audio data to generate a transmission signal. In addition, the communication unit922supplies the transmission signal to the antenna921to transmit the transmission signal to a base station (not shown). In addition, the communication unit922performs amplification, a frequency conversion process, a demodulation process, and the like on a reception signal received through the antenna921and supplies the obtained audio data to the audio codec923. The audio codec923performs data decompression of the audio data or conversion of the audio data to an analog audio signal and outputs the obtained signal to the speaker924.

In addition, in the data communication mode, in case of performing mail transmission, the controller931receives character data input by manipulation of the manipulation unit932and displays the input characters on the display unit930. In addition, the controller931generates mail data based on user instruction or the like in the manipulation unit932and supplies the mail data to the communication unit922. The communication unit922performs a modulation process, a frequency conversion process, and the like on the mail data and transmits the obtained transmission signal from the antenna921. In addition, the communication unit922performs amplification, a frequency conversion process, a demodulation process, and the like on a reception signal received through the antenna921to restore the mail data. The mail data are supplied to the display unit930, so that displaying of a content of the mail is performed.

In addition, the mobile phone920may also store the received mail data in a storage medium by using the recording/reproducing unit929. The storage medium is an arbitrary rewritable storage medium. For example, the storage medium is a semiconductor memory such as a RAM or a built-in flash memory, a removable medium such as a hard disk, a magnetic disk, an optical magnetic disk, an optical disk, a USB memory, or a memory card, or the like.

In the case where the image data are transmitted in the data communication mode, the image data generated by the camera unit926are supplied to the image processing unit927. The image processing unit927performs an encoding process on the image data to generate encoding data.

The multiplexing/separating unit928multiplexes the encoding data generated by the image processing unit927and the audio data supplied from the audio codec923in a predetermined scheme and supplies the multiplexed data to the communication unit922. The communication unit922performs a modulation process, a frequency conversion process, and the like on the multiplexed data and transmits the obtained transmission signal from the antenna921. In addition, the communication unit922performs amplification, a frequency conversion process, a demodulation process, and the like on a reception signal received through the antenna921to restore the multiplexed data. The multiplexed data are supplied to the multiplexing/separating unit928. The multiplexing/separating unit928performs separation on the multiplexed data, supplies the encoding data to the image processing unit927, and supplies the audio data to the audio codec923. The image processing unit927performs a decoding process on the encoding data to generate the image data. The image data are supplied to the display unit930, so that displaying of the received image is performed. The audio codec923converts the audio data into an analog audio signal and supplies the analog audio signal to the speaker924to output the received audio.

In the mobile phone apparatus having the above-described configuration, the functions of the image processing apparatus (image processing method) according to the present application are installed in the image processing unit927. Therefore, it is possible to suppress a deterioration in accuracy of the predicted image, and it is possible to reduce the number of storable reference images.

FIG. 22illustrates an example of a schematic configuration of a recording/reproducing apparatus employing the present technique. The recording/reproducing apparatus940, for example, records audio data and video data of a received broadcast program in a recording medium and provides the recorded data to a user at a timing according to user instruction. In addition, the recording/reproducing apparatus940, for example, may acquire audio data or video data from another apparatus and record the data in the recording medium. In addition, the recording/reproducing apparatus940decodes the audio data or the video data recorded in the recording medium and outputs the decoded data, so that image displaying on a monitor device or the like or audio outputting may be performed.

The recording/reproducing apparatus940is configured to include a tuner941, an external interface unit942, an encoder943, a hard disk drive (HDD) unit944, a disk driver945, a selector946, a decoder947, an on-screen display (OSD) unit948, a controller949, and a user interface unit950.

The tuner941selects a desired channel from broadcast signals received through an antenna (not shown). The tuner941outputs an encoded bit stream obtained by demodulating a reception signal of the desired channel to the selector946.

The external interface unit942is configured with at least one of an IEEE1394 interface, a network interface unit, a USB interface, a flash memory interface, and the like. The external interface unit942is an interface for connecting to an external device, a network, a memory card, or the like and performs data reception of to-be-recorded video data, audio data, and the like.

The encoder943performs encoding on the video data or the audio data in a predetermined scheme when the video data or the audio data supplied from the external interface unit942are not encoded and outputs an encoded bit stream to the selector946.

The HDD unit944records contents data of video, audio, and the like, various programs, other data, and the like in a built-in hard disk and reads out the data and the like from the hard disk at a reproducing time or the like.

The disk driver945performs signal recording and signal reproducing on a mounted optical disk. The optical disk is, for example, a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+R, DVD+RW, or the like), a Blu-ray (registered trade mark) disk, or the like.

At a recording time of video or audio, the selector946selects an encoded bit stream from any of the tuner941and the encoder943and supplies the encoded bit stream to any of the HDD unit944and the disk driver945. In addition, at a reproducing time of video or audio, the selector946supplies the encoded bit stream output from the HDD unit944or the disk driver945to the decoder947.

The decoder947performs a decoding process on the encoded bit stream. The decoder947supplies the video data generated by performing the decoding process to the OSD unit948. In addition, the decoder947outputs the audio data generated by performing the decoding process.

The OSD unit948generates video data for displaying a menu screen of item selection and the like and overlaps the video data output from the decoder947with the video data for displaying the menu screen to output the video data.

The user interface unit950is connected to the controller949. The user interface unit950is configured with a manipulation switch, a remote control signal reception unit, and the like and supplies a manipulation signal according to user manipulation to the controller949.

The controller949is configured by using a CPU, a memory, and the like. The memory stores programs executed by the CPU or various data necessary for the CPU to perform processes. The program stored in the memory is read out and executed by the CPU at a predetermined timing such as a startup time of the recording/reproducing apparatus940. The CPU executes the program to control each component so that the recording/reproducing apparatus940is operated according to user manipulation.

In the recording/reproducing apparatus having the above-described configuration, the functions of the image processing apparatus (image processing method) according to the present application are installed in the decoder947. Therefore, it is possible to suppress a deterioration in accuracy of the predicted image, and it is possible to reduce the number of storable reference images.

(Configuration Example of Imaging Apparatus)

FIG. 23illustrates an example of a schematic configuration of an imaging apparatus employing the present technique. The imaging apparatus960captures an image of an object to display the image of the object on a display unit or to record the image of the object as image data in a recording medium.

The imaging apparatus960is configured to include an optical block961, an imaging unit962, a camera signal processing unit963, an image data processing unit964, a display unit965, an external interface unit966, a memory unit967, a media drive968, an OSD unit969, and a controller970. In addition, a user interface unit971is connected to the controller970. In addition, the image data processing unit964, the external interface unit966, the memory unit967, the media drive968, the OSD unit969, the controller970, and the like are connected to each other via a bus972.

The optical block961is configured by using a focus lens, a stop, and the like. The optical block961focuses an optical image of the object on an image plane of the imaging unit962. The imaging unit962is configured by using a CCD or CMOS image sensor and generates an electric signal according to the optical image through photoelectric conversion to supply the electric signal to the camera signal processing unit963.

The camera signal processing unit963performs various camera signal processes such as knee correction, gamma correction, and color correction on the electric signal supplied from the imaging unit962. The camera signal processing unit963supplies the image data after the camera signal processing to the image data processing unit964.

The image data processing unit964performs an encoding process of the image data supplied from the camera signal processing unit963. The image data processing unit964supplies the encoding data generated by performing the encoding process to the external interface unit966or the media drive968. In addition, the image data processing unit964performs a decoding process on the encoding data supplied from the external interface unit966or the media drive968. The image data processing unit964supplies the image data generated by performing the decoding process to the display unit965. In addition, the image data processing unit964supplies the image data supplied from the camera signal processing unit963to the display unit965or overlaps the image data with the data for display acquired from the OSD unit969and supplies the overlapped data to the display unit965.

The OSD unit969generates the data for displaying such as menu screens and ions which are configured with symbols, characters, or figures and outputs the data for display to the image data processing unit964.

The external interface unit966is configured with, for example, USB input/output ports and the like, and in case of printing images, the external interface unit966is connected to a printer. In addition, if necessary, a drive is connected to the external interface unit966, and a removable medium such as a magnetic disk or an optical disk is appropriately mounted. A computer program read out from the removable medium is installed if necessary. In addition, the external interface unit966is configured to include a network interface which is connected to a predetermined network such as a LAN or the Internet. For example, the controller970may read out the encoding data from the media drive968according to instruction from the user interface unit971and supply the encoding data from the external interface unit966to other devices connected via the network. In addition, the controller970may acquire the encoding data or the image data supplied from other devices via the network through the external interface unit966and supply the encoding data or the image data to the image data processing unit964.

As a recording medium driven by the media drive968, for example, an arbitrary readable/writable removable medium such as a magnetic disk, an optical magnetic disk, an optical disk, or a semiconductor memory is used. In addition, the recording medium is arbitrary in terms of the type as the removable medium. The recording medium may be a tape device, a disk, or a memory card. In addition, the recording medium may be a non-contact integrated circuit (IC) card, or the like.

In addition, the media drive968and the recording medium are integrated, so that the recording medium may be configured with a non-portable storage medium such as a built-in hard disk drive or a solid state drive (SSD).

The controller970is configured by using a CPU. The memory unit967stores programs executed by the controller970, various data necessary for the controller970to perform processes, or the like. The program stored in the memory unit967is read out and executed by the controller970at a predetermined timing such as a startup time of the imaging apparatus960. The controller970executes the program to control each component so that the imaging apparatus960is operated according to user manipulation.

In the imaging apparatus having the above-described configuration, the functions of the image processing apparatus (image processing method) according to the present application are installed in the image data processing unit964. Therefore, it is possible to suppress a deterioration in accuracy of the predicted image, and it is possible to reduce the number of storable reference images.

<Example of Application of Scalable Encoding>

Next, a specific use example of scalable encoding data which are scalable-encoded (hierarchical-encoded) will be described. For example, like an example illustrated inFIG. 24, scalable encoding is used for selecting data which are to be transmitted.

In a data transmission system1000illustrated inFIG. 24, a distribution server1002reads out scalable encoding data stored in a scalable encoding data storage unit1001and distributes the scalable encoding data to terminal devices such as a personal computer1004, an AV device1005, a tablet device1006, and a mobile phone1007via a network1003.

At this time, the distribution server1002selects and transmits encoding data having an appropriate quality according to a capability of the terminal device, a communication environment, or the like. Although the distribution server1002unnecessarily transmits data having high quality, an image having a high image quality may not be obtained in the terminal device, and it may be a cause of occurrence of delay or overflow. In addition, a communication band may be unnecessarily occupied, or the load of the terminal device may be unnecessarily increased. On the contrary, although the distribution server1002unnecessarily transmits data having low quality, an image having a sufficient image quality may not be obtained in the terminal device. Therefore, the distribution server1002appropriately reads out the scalable encoding data stored in the scalable encoding data storage unit1001as the encoding data having a quality appropriate to a capability of the terminal device, communication environment, and the like and transmits the encoding data.

For example, the scalable encoding data storage unit1001stores scalable encoding data (BL+EL)1011which are scalable-encoded. The scalable encoding data (BL+EL)1011are encoding data including both of the base layer and the enhancement layer and data from which both of the image of the base layer and the image of the enhancement layer are obtained by decoding.

The distribution server1002selects an appropriate layer according to the capability of the terminal device which transmits the data, communication environment, and the like and reads out data of the layer. For example, with respect to the personal computer1004or the tablet device1006having a high processing capability, the distribution server1002reads out the scalable encoding data (BL+EL)1011having a high quality from the scalable encoding data storage unit1001and transmits the scalable encoding data (BL+EL)1011without change. On the contrary, for example, with respect to the AV device1005or the mobile phone1007having a low processing capability, the distribution server1002extracts the data of the base layer from the scalable encoding data (BL+EL)1011and transmits the data as scalable encoding data (BL)1012having a quality which is lower than that of the scalable encoding data (BL+EL)1011although the scalable encoding data (BL)1012are data of the same content as that of the scalable encoding data (BL+EL)1011.

If the scalable encoding data is used in this manner, it is possible to easily adjust the data amount, so that it is possible to suppress occurrence of delay or overflow or to suppress an unnecessary increase in load of the terminal device or the communication medium. In addition, in the scalable encoding data (BL+EL)1011, since redundancy between the layers is decreased, it is possible to reduce the data amount in comparison with the case where the encoding data of each layer are treated as individual data. Therefore, it is possible to efficiently use the storage area of the scalable encoding data storage unit1001.

In addition, similarly to the personal computer1004to the mobile phone1007, since various devices may be applied to the terminal device, hardware performance of the terminal device is different among the devices. In addition, since various applications may be executed by the terminal device, software capability is also different. In addition, since any communication network including a wired communication network, a wireless communication network, or both thereof such as the Internet or a local area network (LAN) may be applied to the network1003which is a communication medium, the data transmission capability is different. Furthermore, the data transmission capability may be changed according to other communications or the like.

Therefore, before starting the data transmission, the distribution server1002may perform communication with the terminal device which is a destination of the data transmission to obtain information on the capabilities of the terminal device such as hardware performance of the terminal device or performance of applications (software) executed by the terminal device and information on the communication environment such as an available bandwidth of the network1003. Next, the distribution server1002may select an appropriate layer based on the information obtained above.

In addition, layer extraction may be performed in the terminal device. For example, the personal computer1004may decode the transmitted scalable encoding data (BL+EL)1011to display the image of the base layer or to display the image of the enhancement layer. In addition, for example, the personal computer1004may extract the scalable encoding data (BL)1012of the base layer from the transmitted scalable encoding data (BL+EL)1011to store the scalable encoding data (BL)1012, to transmit the scalable encoding data (BL)1012to anther device, or to decode the scalable encoding data (BL)1012to display the image of the base layer.

Of course, the number of scalable encoding data storage units1001, the number of distribution servers1002, the number of networks1003, and the number of terminal devices are arbitrary. In addition, heretofore, although the example where the distribution server1002transmits data to the terminal device is described, the use example is not limited thereto. The data transmission system1000may be applied to an arbitrary system which selects an appropriate layer according to the capabilities of the terminal device, the communication environment, and the like and performs transmission when the data transmission system1000transmits the scalable-encoded encoding data to the terminal device.

In addition, for example, like an example illustrated inFIG. 25, the scalable encoding is used for transmission via a plurality of communication media.

In a data transmission system1100illustrated inFIG. 25, a broadcasting station1101transmits scalable encoding data (BL)1121of a base layer through terrestrial broadcast1111. In addition, the broadcasting station1101transmits a scalable encoding data (EL)1122of an enhancement layer via an arbitrary network1112configured with a wired communication network, a wireless communication network, or both thereof (for example, transmits packetized data).

The terminal device1102has a reception function of the terrestrial broadcast1111broadcast by the broadcasting station1101to receive the scalable encoding data (BL)1121of the base layer transmitted through the terrestrial broadcast1111. In addition, the terminal device1102further has a communication function of implementing communication via the network1112to receive the scalable encoding data (EL)1122of the enhancement layer transmitted via the network1112.

The terminal device1102obtains the image of base layer by decoding the scalable encoding data (BL)1121of the base layer acquired through the terrestrial broadcast1111, stores the data, or transmits the data to other devices, for example, according to user instruction or the like.

In addition, the terminal device1102obtains the scalable encoding data (BL+EL) by combining the scalable encoding data (BL)1121of the base layer acquired through the terrestrial broadcast1111and the scalable encoding data (EL)1122of the enhancement layer acquired through the network1112, obtains the image of the enhancement layer by decoding the data, stores the data, or transmits the data to other devices, for example, according to user instruction or the like.

In this manner, the scalable encoding data may be transmitted, for example, through different communication medium for each layer. Therefore, it is possible to share the load, so that it is possible to suppress occurrence of delay or overflow.

In addition, the communication medium used for transmission may be selected for each layer according to the situation. For example, the scalable encoding data (BL)1121of the base layer which has a relatively large data amount may be transmitted through the communication medium having a wide bandwidth, and the scalable encoding data (EL)1122of the enhancement layer which has a relatively small data amount may be transmitted through the communication medium having a narrow bandwidth. In addition, for example, the communication medium through which the scalable encoding data (EL)1122of the enhancement layer are to be transmitted may be switched between the network1112and the terrestrial broadcast1111according to the available bandwidth of the network1112. Of course, the same is applied to data of an arbitrary layer.

By controlling in this manner, it is possible to further suppress increase in load of the data transmission.

Of course, the number of layers is arbitrary, and the number of communication media used for transmission is also arbitrary. In addition, the number of terminal devices1102which are destinations of data distribution is also arbitrary. In addition, heretofore, the example of broadcasting from the broadcasting station1101is described. However, the use example is not limited thereto. The data transmission system1100may be applied to an arbitrary system which separates the scalable-encoded encoding data into multiple data in units of a layer and transmits the data through multiple communication lines.

In addition, for example, like an example illustrated inFIG. 26, the scalable encoding is used for storing the encoding data.

In an imaging system1200illustrated inFIG. 26, an imaging apparatus1201performs scalable encoding on image data obtained by capturing an image of an object1211and supplies the data as scalable encoding data (BL+EL)1221to a scalable encoding data storage device1202.

The scalable encoding data storage device1202stores the scalable encoding data (BL+EL)1221supplied from the imaging apparatus1201with a quality according to a situation. For example, in case of a normal period, the scalable encoding data storage device1202extracts data of a base layer from the scalable encoding data (BL+EL)1221and stores the data as scalable encoding data (BL)1222of the base layer having a small data amount with a low quality. On the contrary, for example, in case of an attention period, the scalable encoding data storage device1202stores the scalable encoding data (BL+EL)1221having a large data amount with a high quality as it is.

By doing in this manner, the scalable encoding data storage device1202may store the image with a high image quality only if necessary. Therefore, it is possible to suppress an increase in data amount while suppressing a decrease in value of the image due to a deterioration in image quality, and it is possible to improve a utilization efficiency of a storage area.

For example, the imaging apparatus1201is a surveillance camera. In the case where a surveillance target (for example, an intruder) does not appear on the captured image (in case of a normal period), since the possibility that the content of the captured image is not important is high, decreasing of the data amount is given priority, and the image data (scalable encoding data) are stored with a low quality. On the contrary, in the case where the surveillance target appears as the object1211on the captured image (in case of an attention period), since the possibility that the content of the captured image is important is high, the image quality is given priority, and the image data (scalable encoding data) are stored with a high quality.

In addition, the determination as to whether the situation is in a normal period or an attention period may be performed, for example, by the scalable encoding data storage device1202analyzing the image. In addition, the determination may be performed by the imaging apparatus1201, and a result of the determination may be transmitted to the scalable encoding data storage device1202.

In addition, the criterion of the determination as to whether the situation is in a normal period or an attention period is arbitrary, and the content of the image defined as the criterion of the determination is arbitrary. Of course, other conditions other than the content of the image may be defined as the criterion of the determination. For example, the normal and attention periods may be switched according to the magnitude, waveform, or the like of the recorded audio; the normal and attention periods may be switched every predetermined time; or the normal and attention periods may be switched according to external instruction such as user instruction.

In addition, heretofore, although the example where the two states of the normal period and the attention period are switched is described, the number of states is arbitrary. Three or more states of, for example, a normal period, a weak attention period, an attention period, a strong attention period, and the like may be switched. However, the upper limit of the number of switching states depends on the number of layers of the scalable encoding data.

In addition, the imaging apparatus1201may determine the number of layers in the scalable encoding according to the state. For example, in case of the normal period, the imaging apparatus1201may generate the scalable encoding data (BL)1222of the base layer having a small data amount with a low quality and supply the data to the scalable encoding data storage device1202. In addition, for example, in case of the attention period, the imaging apparatus1201may generate the scalable encoding data (BL+EL)1221of the base layer having a large data amount with a high quality and supply the data to the scalable encoding data storage device1202.

Heretofore, although the example of the surveillance camera is described, the applications of the imaging system1200are arbitrary and are not limited to the surveillance camera.

In addition, the LCU denotes a coding unit (CU) having the largest size, and a coding tree unit (CTU) is a unit including a coding tree block (CTB) of the LCU and a parameter at the time of the process in a LCU base (level). In addition, a CU constituting the CTU is a unit including a coding block (CB) and a parameter at the time of the process in a CU base (level).

The present invention may be applied to apparatuses used for transmitting/receiving image information (bit stream) compressed by orthogonal transform such as discrete cosine transform and motion compensation through a network medium such as satellite broadcasting, cable TV, the Internet, or mobile phones or apparatuses used for performing processes on a storage medium such as an optical disk, a magnetic disk, or a flash memory like MPEG, H.26x, or the like.

In addition, the encoding scheme according to the present invention may be an encoding scheme other than the HEVC scheme.

In addition, embodiments of the present technique are not limited to the above-described embodiments, but various modifications are available within the scope without departing from the spirit of the present technique.

In addition, the present technique may also have the configuration described hereinafter.

(1) An image processing apparatus including:

a predicted image generation unit which generates a predicted image of an image by using a reference image; and

a storage unit which preferentially stores the reference image of which display order is close to that of the image.

(2) The image processing apparatus according to (1) above, wherein in the case where the image is a moving image, the storage unit preferentially stores the reference image of which display order is close to the image, and in the case where the image is a still image, the storage unit preferentially stores the reference image of which quantization parameter is small.

(3) The image processing apparatus according to (2) above, wherein in the case where the image is a still image, the storage unit preferentially stores an I picture as the reference image.

(4) The image processing apparatus according to any of (1) to (3) above, wherein the number of reference images storable in the storage unit is determined based on a size of the image.

(5) An image processing method using an image processing apparatus, the method including:

a predicted image generating step of generating a predicted image of an image by using a reference image; and

a storing step of preferentially storing the reference image of which display order is close to that of the image.

REFERENCE SIGNS LIST