Source: http://patents.com/us-9609326.html
Timestamp: 2018-01-17 01:38:57
Document Index: 732189623

Matched Legal Cases: ['Application No. 2011', 'art 4', 'art 2', 'art 1', 'art 4', 'art 6', 'art 9', 'art 1', 'art 1', 'art 4', 'art 6', 'art 4', 'art 5', 'art 2', 'art 6', 'art 7', 'art 6', 'art 1', 'art 7', 'art 53', 'art 53', 'art 56', 'art 4', 'art 53', 'art 51', 'art 51', 'art 53', 'art 2', 'art 1', 'art 1', 'art 2', 'art 1', 'art 1', 'art 2', 'art 4', 'art 2', 'art 5', 'art 4', 'art 1', 'art 4', 'art 4', 'art 6', 'art 9', 'art 13', 'art 4', 'art 5', 'art 1', 'art 4', 'art 5', 'art 6', 'art 4', 'art 5', 'art 2', 'art 7', 'art 6', 'art 7', 'art 1', 'art 8', 'art 13', 'art 7', 'art 8', 'art 1', 'art 9', 'art 8', 'art 9', 'art 4', 'art 5', 'art 9', 'art 11', 'art 4', 'art 6', 'art 9', 'art 13', 'art 4', 'art 4', 'art 13', 'art 13', 'art 51', 'art. 51', 'art 1', 'art 51', 'art 51', 'art 51', 'art 51', 'art 51', 'art 53', 'art 51', 'art 54', 'art 53', 'art 4', 'art 53', 'art 4', 'art 4', 'art 4', 'art 4', 'art 53', 'art 4', 'art 54', 'art 56', 'art 58', 'art 58', 'art 56', 'art 4', 'art 53', 'arts 4', 'art 4', 'art 4', 'art 4', 'art 4', 'art 4', 'art 13', 'art 4', 'art 51', 'art 53', 'art 51', 'art 53', 'art 53']

US Patent # 9,609,326. Image encoding device, image decoding device, image encoding method, and image decoding method for generating a prediction image - Patents.com
United States Patent 9,609,326
Minezawa , et al. March 28, 2017
When carrying out an average prediction, the intra predictor carries out a filtering process on target pixels of the intra prediction located at an upper end and a left end of the block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel. The intra predictor sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel. As a result, prediction errors locally occurring can be reduced, and the image quality can be improved.
Minezawa; Akira (Tokyo, JP), Sugimoto; Kazuo (Tokyo, JP), Sekiguchi; Shunichi (Tokyo, JP)
Family ID: 1000002488404
14/977,170
US 20160156930 A1 Jun 2, 2016
13979357 9299133
PCT/JP2012/000061 Jan 6, 2012
Jan 12, 2011 [JP] 2011-004038
Current CPC Class: H04N 19/117 (20141101); G06T 5/20 (20130101); G06T 9/004 (20130101); H04N 19/11 (20141101); H04N 19/176 (20141101); H04N 19/182 (20141101); H04N 19/44 (20141101); H04N 19/593 (20141101); H04N 19/61 (20141101); H04N 19/80 (20141101); H04N 19/82 (20141101)
Current International Class: G06T 9/00 (20060101); H04N 19/80 (20140101); H04N 19/82 (20140101); H04N 19/593 (20140101); H04N 19/44 (20140101); H04N 19/176 (20140101); H04N 19/61 (20140101); H04N 19/117 (20140101); H04N 19/182 (20140101); H04N 19/11 (20140101); G06T 5/20 (20060101)
9299133 March 2016 Minezawa
2007/0053443 March 2007 Song
2007/0206872 September 2007 Song
2008/0232705 September 2008 Sohn et al.
2009/0110069 April 2009 Jung et al.
2011/0255600 October 2011 Lin et al.
2012/0014445 January 2012 Ma et al.
2012/0044993 February 2012 Sato
2013/0034158 February 2013 Kirchhoffer et al.
2013/0272405 October 2013 Jeon et al.
2262267 Dec 2010 EP
9-187008 Jul 1997 JP
10322695 Dec 1998 JP
10-2010-003438 Mar 2010 KP
WO 2009/120040 Oct 2009 WO
WO 2010/002214 Jan 2010 WO
WO 2010/143427 Dec 2010 WO
WO 2011/132400 Oct 2011 WO
WO 2012/008125 Jan 2012 WO
WO 2012/042720 Apr 2012 WO
"Advanced video coding for generic audiovisual services", MPEG-4 AVC(ISO/IEC 14496-10) / ITU-T H.264, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services--Coding of moving video, Mar. 2010, pp. 1-676. cited by applicant .
"Simplified Intra Smoothing" by Yunfei Zheng et al., Joint Collaborative Team on Video Coding (JCT-VC) and ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, 3rd Meeting: Guangzhou, CN, Oct. 2010, JCTV-C234.sub.--rl, pp. 1-6. cited by applicant .
Davies,"BBC's Response to the Call for Proposals on Video Compression Technology", Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JCTVC-A125, 1st Meeting: Dresden, DE, Apr. 2010, pp. 1-11. cited by applicant .
Ichigaya et al.,"Description of video coding technology proposal by NHK and Mitsubishi", Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JCTVC-A122, 1st Meeting: Dresden, DE, Apr. 2010, pp. 1-5, 15-16. cited by applicant .
McCann et al.,"Samsung's Response to the Call for Proposals on Video Compression Technology", Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JCTVC-A124, 1st Meeting: Dresden, DE, Apr. 2010, pp. 1-11. cited by applicant .
Demin Wang et al. "Improvement of H.26L Intra Block Prediction" --ITU -- Telecommunications Standardization Sector--Study Group 16 Question 6--Video Coding Experts Group (VCEG) Document: VCEG-L09 cited by applicant .
Detlev Marpe et al. "H.264/MPEG4-AVC Fidelity Range Extensions:--Tools, Profiles, Performance, and Application Areas" --2005 IEEE. cited by applicant .
Kazuo Sugimoto et al. "LUT-based adaptive filtering on intra prediction samples" - Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 Document: JCTVC-D109. cited by applicant .
Peng Zjang et al. "Multiple Modes Intra-Prediction in Ingra-Coding" --2004 IEEE International Conference on Multimedia and Expo. cited by applicant.
This application is a Divisional of application Ser. No. 13/979,357, filed on Jul. 11, 2013, now U.S. Pat. No. 9,299,133, which was filed as PCT International Application No. PCT/JP2012/000061 on Jan. 6, 2012, which claims the benefit under 35 U.S.C. .sctn.119(a) to Patent Application No. 2011-004038, filed in Japan on Jan. 12, 2011, all of which are hereby expressly incorporated by reference into the present application.
1. An image encoding device comprising: an intra predictor for, when a coding mode corresponding to one of coding blocks into which an inputted image is divided is an intra coding mode, carrying out an intra-frame prediction process on each block which is a unit for prediction of the coding block to generate a prediction image; and an encoder for entropy-encoding coding mode information and an intra prediction parameter indicating an average prediction, wherein when the intra predictor carries out the average prediction, the intra predictor carries out a filtering process on target pixels of intra prediction located at an upper end and a left end of the block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the intra predictor sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
2. An image decoding device comprising: a decoder for entropy-decoding coding mode information and an intra prediction parameter; and an intra predictor for, when the coding mode information associated with a coding block is an intra coding mode, carrying out an intra-frame prediction process on each block which is a unit for prediction of the coding block to generate a prediction image, wherein when the intra prediction parameter indicates an average prediction, the intra predictor carries out a filtering process on target pixels of intra prediction located at an upper end and a left end of the block based on an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the intra predictor sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
3. An image encoding method comprising: carrying out an intra-frame prediction process on each block which is a unit for prediction of a coding block to generate a prediction image, when a coding mode corresponding to the coding block into which an inputted image is divided is an intra coding mode; and entropy-encoding coding mode information and an intra prediction parameter indicating an average prediction, wherein when the average prediction is carried out, a filtering process is carried out on target pixels of intra prediction located at an upper end and a left end of the block which is a unit for prediction of the coding block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the filtering process sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
4. An image decoding method comprising: entropy-decoding coding mode information and an intra prediction parameter; and carrying out an intra-frame prediction process on each block which is a unit for prediction of a coding block to generate a prediction image, when the coding mode information associated with the coding block is an intra coding mode, wherein when the intra prediction parameter indicates an average prediction, a filtering process is carried out on target pixels of the-intra prediction located at an upper end and a left end of the block which is a unit for prediction of the coding block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the filtering process sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
For example, in accordance with an international standard video encoding method, such as MPEG (Moving Picture Experts Group) or "ITU-T H.26x", an inputted video frame is divided into rectangular blocks (encoding target blocks), a prediction process using an already-encoded image signal is carried out on each encoding target block to generate a prediction image, and orthogonal transformation and a quantization process is carried out on a prediction error signal which is the difference between the encoding target block and the prediction image in units of a block, so that information compression is carried out on the inputted video frame.
For example, in the case of MPEG-4 AVC/H.264 (ISO/IEC 14496-10|ITU-T H.264) which is an international standard method, an intra prediction process from already-encoded adjacent pixels or a motion-compensated prediction process between adjacent frames is carried out (for example, refer to nonpatent reference 1). In the case of MPEG-4 AVC/H.264, one prediction mode can be selected from a plurality of prediction modes for each block in an intra prediction mode of luminance. FIG. 10 is an explanatory drawing showing intra prediction modes in the case of a 4.times.4 pixel block size for luminance. In FIG. 10, each white circle shows a pixel in a coding block, and each black circle shows a pixel that is used for prediction, and that exists in an already-encoded adjacent block.
In this case, the block size for luminance to which an intra prediction is applied can be selected from 4.times.4 pixels, 8.times.8 pixels, and 16.times.16 pixels. In the case of 8.times.8 pixels, nine intra prediction modes are defined, like in the case of 4.times.4 pixels. In contrast with this, in the case of 16.times.16 pixels, four intra prediction modes which are called plane predictions are defined in addition to intra prediction modes associated with an average prediction, a vertical prediction, and a horizontal prediction. Each intra prediction associated with a plane prediction is a mode in which pixels created by carrying out an interpolation in a diagonal direction on the adjacent pixels in the upper block and the adjacent pixels in the left block are provided as predicted values.
In an intra prediction mode in which a directional prediction is carried out, because predicted values are generated along a direction predetermined by the mode, e.g., a direction of 45 degrees, the prediction efficiency increases and the code amount can be reduced when the direction of a boundary (edge) of an object in a block matches the direction shown by the prediction mode. However, a slight displacement may occur between the direction of an edge and the direction shown by the prediction mode, and, even if the direction of an edge in the encoding target block does not match the direction shown by the prediction mode, a large prediction error may occur locally for the simple reason that the edge is slightly distorted (swung, bent, or the like). As a result, the prediction efficiency may drop extremely. In order to prevent such a reduction in the prediction efficiency, when performing an 8.times.8-pixel directional prediction, a prediction process is carried out to generate a smoothed prediction image by using already-encoded adjacent pixels on which a smoothing process has been carried out, thereby reducing any slight displacement in the prediction direction and prediction errors which occur when a slight distortion occurs in an edge.
Because the conventional image encoding device is constructed as above, the generation of a smoothed prediction image can reduce prediction errors occurring even if a slight displacement occurs in the prediction direction or a slight distortion occurs in an edge. However, according to the technique disclosed in nonpatent reference 1, no smoothing process is carried out on blocks other than 8.times.8-pixel blocks, and only one possible smoothing process is carried out on even 8.times.8-pixel blocks. A problem is that also in a block having a size other than 8.times.8 pixels, a large prediction error actually occurs locally due to a slight mismatch in an edge even when the prediction image has a pattern similar to that of the image to be encoded, and therefore a large reduction occurs in the prediction efficiency. Another problem is that when a quantization parameter which is used when quantizing a prediction error signal, the position of each pixel in a block, the prediction mode, or the like differs between blocks having the same size, a process suitable for reducing local prediction errors differs between the blocks, but only one possible smoothing process is prepared, and therefore prediction errors cannot be sufficiently reduced. A further problem is that when carrying out an average prediction, a prediction signal for a pixel located at a boundary of a block easily becomes discontinuous with those for adjacent encoded pixels because the average of pixels adjacent to the block is defined as each of all the predicted values in the block, while because the image signal generally has a high spatial correlation, a prediction error easily occurs at the block boundary due to the above-mentioned discontinuity.
In accordance with an aspect of present invention, there is provided an image encoding device comprising: an intra predictor for, when a coding mode corresponding to one of coding blocks into which an inputted image is divided is an intra coding mode, carrying out an intra-frame prediction process on each block which is a unit for prediction of the coding block to generate a prediction image; and an encoder for encoding coding mode information and an intra prediction parameter indicating an average prediction, wherein when the intra predictor carries out the average prediction, the intra predictor carries out a filtering process on target pixels of the intra prediction located at an upper end and a left end of the block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the intra predictor sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
In accordance with an aspect of present invention, there is provided an image decoding device comprising: a decoder for decoding coding mode information and an intra prediction parameter; and an intra predictor for, when the coding mode information associated with a coding block is an intra coding mode, carrying out an intra-frame prediction process on each block which is a unit for prediction of the coding block to generate a prediction image, wherein when the intra prediction parameter indicates an average prediction, the intra predictor carries out a filtering process on target pixels of the intra prediction located at an upper end and a left end of the block based on an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the intra predictor sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
In accordance with yet another aspect of present invention, there is provided an image encoding method comprising: carrying out an intra-frame prediction process on each block which is a unit for prediction of a coding block to generate a prediction image, when a coding mode corresponding to the coding block into which an inputted image is divided is an intra coding mode; and encoding coding mode information and an intra prediction parameter indicating an average prediction, wherein when the average prediction is carried out, a filtering process is carried out on target pixels of the intra prediction located at an upper end and a left end of the block which is a unit for prediction of the coding block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the filtering process sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
In accordance with another aspect of present invention, there is provided an image decoding method comprising: decoding coding mode information and an intra prediction parameter; and carrying out an intra-frame prediction process on each block which is a unit for prediction of a coding block to generate a prediction image, when the coding mode information associated with the coding block is an intra coding mode, wherein when the intra prediction parameter indicates an average prediction, a filtering process is carried out on target pixels of the intra prediction located at an upper end and a left end of the block which is a unit for prediction of the coding block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the filtering process sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
In accordance with another aspect of present invention, there is provided a non-transitory computer readable medium comprising coded data for each of coding blocks, the coded data including: coding mode information causing a decoder to determine a type of coding mode, and causing the decoder to carry out an intra-frame prediction process on each block which is a unit for prediction of a coding block to generate a prediction image, when the coding mode information associated with the coding block indicates an intra coding mode; and an intra prediction parameter causing the decoder to determine a type of intra prediction, wherein when the intra prediction parameter indicates an average prediction, a filtering process is carried out on target pixels of the intra prediction located at an upper end and a left end of the block which is a unit for prediction of the coding block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the filtering process sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the filtering process sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block.
Because the image encoding device in accordance with the present invention is constructed in such a way that when carrying out an average prediction, the intra predictor carries out a filtering process on target pixels of the intra prediction located at an upper end and a left end of the block, the filtering process using an intermediate prediction value, which is an average value of adjacent pixels of the block, and at least one adjacent pixel of the target pixel, wherein the intra predictor sets a filter coefficient to 1/2, associated with the intermediate prediction value for a target pixel at an upper left corner of the block, and sets a filter coefficient to 1/4, associated with an adjacent pixel adjacent to an upper side or a left side of the target pixel, wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the upper end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the upper side of the target pixel at the upper end of the block, and wherein the intra predictor sets a filter coefficient to 3/4, associated with the intermediate prediction value for a target pixel at the left end of the block other than the target pixel at the upper left corner, and sets a filter coefficient to 1/4, associated with the adjacent pixel adjacent to the left side of the target pixel at the left end of the block, there is provided an advantage of being able to reduce prediction errors occurring locally, thereby being able to improve the image quality.
FIG. 6(A) is an explanatory drawing showing a distribution of partitions into which a block to encoded is divided, and FIG. 6(B) is an explanatory drawing showing a state in which a coding mode m(B.sup.n) is assigned to each of the partitions after a hierarchical layer division is performed by using a quadtree graph;
FIG. 7 is an explanatory drawing showing an example of intra prediction parameters (intra prediction modes) which can be selected for each partition P.sub.i.sup.n in a coding block B.sup.n;
FIG. 8 is an explanatory drawing showing an example of pixels which are used when generating a predicted value of each pixel in a partition P.sub.i.sup.n in the case of I.sub.i.sup.n=m.sub.i.sup.n=4;
FIG. 10 is an explanatory drawing showing intra prediction modes described in nonpatent reference 1 in the case of a 4.times.4 pixel block size for luminance;
The intra prediction part 4 carries out a process of, when receiving the coding block, which is generated through the division by the block dividing part 2, from the selection switch 3, carrying out an intra prediction process on the coding block to generate a prediction image for each partition by using an already-encoded image signal in the frame on the basis of the intra prediction parameter outputted thereto from the encoding controlling part 1. After generating the above-mentioned prediction image, the intra prediction part 4 selects a filter from one or more filters which are prepared in advance according to the states of the various parameters which must be known when the moving image decoding device generates the same prediction image as the above-mentioned prediction image, carries out a filtering process on the above-mentioned prediction image by using the filter, and outputs the prediction image on which the intra prediction part has carried out the filtering process to a subtracting part 6 and an adding part 9. Concretely, the intra prediction part uniquely determines a filter according to the state of at least one of the following four parameters which are provided as the above-mentioned various parameters: Parameter (1) The block size of the above-mentioned prediction image Parameter (2) The quantization parameter determined by the encoding controlling part 1 Parameter (3) The distance between the already-encoded image signal in the frame which is used when generating the prediction image and a target pixel to be filtered Parameter (4) The intra prediction parameter determined by the encoding controlling part 1 An intra prediction unit is comprised of the selection switch 3 and the intra prediction part 4.
The subtracting part 6 carries out a process of subtracting the prediction image generated by the intra prediction part 4 or the motion-compensated prediction part 5 from the coding block, which is generated through the division by the block dividing part 2, to generate a difference image (=the coding block--the prediction image). The subtracting part 6 constructs a difference image generating unit. The transformation/quantization part 7 carries out a process of performing a transformation process (e.g., a DCT (discrete cosine transform) or an orthogonal transformation process, such as a KL transform, in which bases are designed for a specific learning sequence in advance) on the difference signal generated by the subtracting part 6 in units of a block having a transform block size included in the prediction error encoding parameters outputted thereto from the encoding controlling part 1, and also quantizing the transform coefficients of the difference image by using a quantization parameter included in the prediction error encoding parameters to output the transform coefficients quantized thereby as compressed data of the difference image. The transformation/quantization part 7 constructs an image compression unit.
The intra prediction part 53 carries out a process of performing an intra-frame prediction process on the coding block to generate a prediction image for each partition by using an already-decoded image signal in the frame on the basis of the intra prediction parameter outputted thereto from the selection switch 52. After generating the above-mentioned prediction image, the intra prediction part 53 selects a filter from one or more filters which are prepared in advance according to the states of the various parameters which are known when generating the above-mentioned prediction image, carries out a filtering process on the above-mentioned prediction image by using the filter, and outputs the prediction image on which the intra prediction part has carried out the filtering process to an adding part 56. Concretely, the intra prediction part uniquely determines a filter according to the state of at least one of the following four parameters which are provided as the above-mentioned various parameters. The intra prediction part predetermines one or more parameters to be used which are the same as the previously-mentioned one or more parameters which are used by the moving image encoding device. More specifically, the parameters which the moving image encoding device uses and those which the moving image decoding device uses are made to be the same as each other in such a way that when the intra prediction part 4 carries out the filtering process by using the parameters (1) and (4) in the moving image encoding device, the intra prediction part 53 similarly carries out the filtering by using the parameters (1) and (4) in the moving image decoding device, for example. Parameter (1) The block size of the above-mentioned prediction image Parameter (2) The quantization parameter variable-length-decoded by the variable length decoding part 51 Parameter (3) The distance between the already-decoded image signal in the frame which is used when generating the prediction image and a target pixel to be filtered Parameter (4) The intra prediction parameter variable-length-decoded by the variable length decoding part 51 An intra prediction unit is comprised of the selection switch 52 and the intra prediction part 53.
When receiving the video signal showing the inputted image, the block dividing part 2 divides the inputted image shown by the video signal into coding blocks each having the maximum size determined by the encoding controlling part 1, and also divides each of the coding blocks into blocks hierarchically until the number of hierarchical layers reaches the upper limit on the number of hierarchical layers which is determined by the encoding controlling part 1. FIG. 5 is an explanatory drawing showing a state in which each coding block having the maximum size is hierarchically divided into a plurality of coding blocks. In the example of FIG. 5, each coding block having the maximum size is a coding block B.sup.0 in the 0th hierarchical layer, and its luminance component has a size of (L.sup.0, M.sup.0) . Further, in the example of FIG. 5, by carrying out the hierarchical division with this coding block B.sup.0 having the maximum size being set as a starting point until the depth of the hierarchy reaches a predetermined depth which is set separately according to a quadtree structure, coding blocks B.sup.n can be acquired.
At the depth of n, each coding block B.sup.n is an image region having a size of (L.sup.n, M.sup.n). Although L.sup.n can be the same as or differ from M.sup.n, the case of L.sup.n=M.sup.n is shown in the example of FIG. 5. Hereafter, the size of each coding block B.sup.n is defined as the size of (L.sup.n, M.sup.n) in the luminance component of the coding block B.sup.n.
Because the block dividing part 2 carries out a quadtree division, (L.sup.n+1, M.sup.n+1)=(L.sup.n/2, M.sup.n/2) is always established. In the case of a color video image signal (4:4:4 format) in which all the color components have the same sample number, such as an RGB signal, all the color components have a size of (L.sup.n, M.sup.n), while in the case of handling a 4:2:0 format, a corresponding color difference component has a coding block size of (L.sup.n/2, M.sup.n/2). Hereafter, a coding mode selectable for each coding block B.sup.n in the nth hierarchical layer is expressed as m(B.sup.n).
Each coding block B.sup.n is further divided into one or more prediction units (partitions) by the block dividing part, as shown in FIG. 5. Hereafter, each partition belonging to each coding block B.sup.n is expressed as P.sub.i.sup.n (i shows a partition number in the nth hierarchical layer). How the division of each coding block B.sup.n into partitions P.sub.i.sup.n belonging to the coding block B.sup.n is carried out is included as information in the coding mode m(B.sup.n). While the prediction process is carried out on each of all the partitions P.sub.i.sup.n according to the coding mode m(B.sup.n), an individual prediction parameter can be selected for each partition P.sub.i.sup.n.
The encoding controlling part 1 produces such a block division state as shown in, for example, FIG. 6 for a coding block having the maximum size, and then determines coding blocks B.sup.n. Hatched portions shown in FIG. 6(a) show a distribution of partitions into which the coding block having the maximum size is divided, and FIG. 6(b) shows a situation in which coding modes m(B.sup.n) are respectively assigned to the partitions generated through the hierarchical layer division by using a quadtree graph. Each node enclosed by a square symbol shown in FIG. 6(b) is the one (coding block B.sup.n) to which a coding mode m(B.sup.n) is assigned.
When the encoding controlling part 1 selects an optimal coding mode m(B.sup.n) for each partition P.sub.i.sup.n of each coding block B.sup.n, and the coding mode m(B.sup.n) is an intra coding mode (step ST3), the selection switch 3 outputs the partition P.sub.i.sup.n of the coding block B.sup.n, which is generated through the division by the block dividing part 2, to the intra prediction part 4. In contrast, when the coding mode m(B.sup.n) is an inter coding mode (step ST3), the selection switch outputs the partition P.sub.i.sup.n of the coding block B.sup.n, which is generated through the division by the block dividing part 2, to the motion-compensated prediction part 5.
When receiving the partition P.sub.i.sup.n of the coding block B.sup.n from the selection switch 3, the intra prediction part 4 carries out an intra prediction process on the partition P.sub.i.sup.n of the coding block B.sup.n to generate an intra prediction image P.sub.ihu n by using an already-encoded image signal in the frame on the basis of the intra prediction parameter outputted thereto from the encoding controlling part 1 (step ST4). After generating the above-mentioned intra prediction image P.sub.i.sup.n, the intra prediction part 4 selects a filter from the one or more filters which are prepared in advance according to the states of the various parameters which must be known when the moving image decoding device generates the same prediction image as the above-mentioned intra prediction image P.sub.i.sup.n, and carries out a filtering process on the intra prediction image P.sub.i.sup.n by using the filter. After carrying out the filtering process on the intra prediction image P.sub.i.sup.n, the intra prediction part 4 outputs the intra prediction image P.sub.i.sup.n on which the intra prediction part has carried out the filtering process to the subtracting part 6 and the adding part 9. In order to enable the moving image decoding device shown in FIG. 2 to also be able to generate the same intra prediction image P.sub.i.sup.n, the intra prediction part outputs the intra prediction parameters to the variable length encoding part 13. The outline of the process carried out by the intra prediction part 4 is as mentioned above, and the details of this process will be mentioned below.
When receiving the partition P.sub.i.sup.n of the coding block B.sup.n from the selection switch 3, the motion-compensated prediction part 5 carries out a motion-compensated prediction process on the partition P.sub.i.sup.n of the coding block B.sup.n to generate an inter prediction image P.sub.i.sup.n by using one or more frames of reference images stored in the motion-compensated prediction frame memory 12 on the basis of the inter prediction parameters outputted thereto from the encoding controlling part 1 (step ST5). Because a technology of carrying out a motion-compensated prediction process to generate a prediction image is known, the detailed explanation of this technology will be omitted hereafter.
After the intra prediction part 4 or the motion-compensated prediction part 5 generates the prediction image (an intra prediction image P.sub.i.sup.n or an inter prediction image P.sub.i.sup.n), the subtracting part 6 subtracts the prediction image (the intra prediction image P.sub.i.sup.n or the inter prediction image P.sub.i.sup.n) generated by the intra prediction part 4 or the motion-compensated prediction part 5 from the partition P.sub.i.sup.n of the coding block B.sup.n, which is generated through the division by the block dividing part 2, to generate a difference image, and outputs a prediction error signal e.sub.i.sup.n showing the difference image to the transformation/quantization part 7 (step ST6).
When receiving the prediction error signal e.sub.i.sup.n showing the difference image from the subtracting part 6, the transformation/quantization part 7 carries out a transformation process (e.g., a DCT (discrete cosine transform) or an orthogonal transformation process, such as a KL transform, in which bases are designed for a specific learning sequence in advance) on the difference image in units of a block having the transform block size included in the prediction error encoding parameters outputted thereto from the encoding controlling part 1, and quantizes the transform coefficients of the difference image by using the quantization parameter included in the prediction error encoding parameters and outputs the transform coefficients quantized thereby to the inverse quantization/inverse transformation part 8 and the variable length encoding part 13 as compressed data of the difference image (step ST7).
When receiving the compressed data of the difference image from the transformation/quantization part 7, the inverse quantization/inverse transformation part 8 inverse-quantizes the compressed data of the difference image by using the quantization parameter included in the prediction error encoding parameters outputted thereto from the encoding controlling part 1, performs an inverse transformation process (e.g., an inverse DCT (inverse discrete cosine transform) or an inverse transformation process such as an inverse KL transform) on the compressed data inverse-quantized thereby in units of a block having the transform block size included in the prediction error encoding parameters, and outputs the compressed data on which the inverse quantization/inverse transformation part has carried out the inverse transformation process to the adding part 9 as a local decoded prediction error signal e.sub.i.sup.n hat ("^" attached to an alphabetical letter is expressed by hat for reasons of the restrictions on electronic applications) (step ST8).
When receiving the local decoded prediction error signal e.sub.i.sup.n hat from the inverse quantization/inverse transformation part 8, the adding part 9 adds the local decoded prediction error signal e.sub.i.sup.n hat and the prediction signal showing the prediction image (the intra prediction image P.sub.i.sup.n or the inter prediction image P.sub.i.sup.n) generated by the intra prediction part 4 or the motion-compensated prediction part 5 to generate a local decoded image which is a local decoded partition image P.sub.i.sup.n hat or a local decoded coding block image which is a group of local decoded partition images (step ST9). After generating the local decoded image, the adding part 9 stores a local decoded image signal showing the local decoded image in the memory 10 for intra prediction and also outputs the local decoded image signal to the loop filter part 11.
The moving image encoding device repeatedly carries out the processes of steps ST3 to ST9 until the moving image encoding device completes the processing on all the coding blocks B.sup.n into which the inputted image is divided hierarchically, and, when completing the processing on all the coding blocks B.sup.n, shifts to a process of step ST12 (steps ST10 and ST11).
Next, the process carried out by the intra prediction unit 4 will be explained in detail. FIG. 7 is an explanatory drawing showing an example of the intra prediction parameters (intra prediction modes) which can be selected for each partition P.sub.i.sup.n in the coding block B.sup.n. In the example shown in FIG. 7, intra prediction modes and prediction direction vectors represented by each of the intra prediction modes are shown, and it is designed that a relative angle between prediction direction vectors becomes small with increase in the number of selectable intra prediction modes.
The intra prediction part 4 carries out an intra prediction process on the partition P.sub.i.sup.n on the basis of the intra prediction parameter for the partition P.sub.i.sup.n and a selection parameter for a filter which the intra prediction part uses for the generation of an intra prediction image P.sub.i.sup.n. Hereafter, an intra process of generating an intra prediction signal of the luminance signal on the basis of the intra prediction parameter (intra prediction mode) for the luminance signal of the partition P.sub.i.sup.n will be explained.
Hereafter, the partition P.sub.i.sup.n is assumed to have a size of l.sub.i.sup.n.times.m.sub.i.sup.n pixels. FIG 8 is an explanatory drawing showing an example of pixels which are used when generating a predicted value of each pixel in the partition P.sub.i.sup.n in the case of l.sub.i.sup.n=m.sub.i.sup.n=4. Although the (2.times.l.sub.i.sup.n+1) pixels in the already-encoded upper partition which is adjacent to the partition P.sub.i.sup.n and the (2.times.m.sub.i.sup.n) pixels in the already-encoded left partition which is adjacent to the partition P.sub.i.sup.n are defined as the pixels used for prediction in the example of FIG. 8, a larger or smaller number of pixels than the pixels shown in FIG. 8 can be used for prediction. Further, although one row or column of pixels adjacent to the partition are used for prediction in the example shown in FIG. 8, two or more rows or columns of pixels adjacent to the partition can be alternatively used for prediction.
When the index value indicating the intra prediction mode for the partition P.sub.i.sup.n is 2 (average prediction), the intra prediction part generates an intermediate prediction image by using the average of the adjacent pixels in the upper partition and the adjacent pixels in the left partition as each of the predicted values of all the pixels in the partition P.sub.i.sup.n. When the index value indicating the intra prediction mode is other than 2 (average prediction), the intra prediction part generates the predicted value of each pixel in the partition P.sub.i.sup.n on the basis of a prediction direction vector v.sub.p=(dx, dy) shown by the index value. In this case, the relative coordinate of the pixel (the pixel at the upper left corner of the partition is defined as the point of origin) for which the predicted value is to be generated (target pixel for prediction) in the partition P.sub.i.sup.n is expressed as (x, y). Each reference pixel which is used for prediction is located at a point of intersection of A shown below and an adjacent pixel.
The intra prediction part then carries out a filtering process, which will be mentioned below, on the intermediate prediction image which consists of the predicted values in the partition P.sub.i.sup.n generated according to the above-mentioned procedure to acquire a final intra prediction image P.sub.i.sup.n, and outputs the intra prediction image P.sub.i.sup.n to the subtracting part 6 and the adding part 9. The intra prediction part also outputs the intra prediction parameter used for the generation of the intra prediction image P.sub.i.sup.n to the variable length encoding part 13 in order to multiplex them into a bitstream. Hereafter, the filtering process will be explained concretely.
The intra prediction part selects a filter to be used from one or more filters which are prepared in advance by using a method which will be mentioned below, and carries out a filtering process on each pixel of the intermediate prediction image according to the following equation (1). s(p.sub.0)=a.sub.0s(p.sub.0)+a.sub.1s(p.sub.1)+ . . . +a.sub.N-1s(p.sub.N-1)+a.sub.N (1)
In the equation (1), an (n=0, 1, . . . , N) is filter coefficients which consist of coefficients (a.sub.0, a.sub.1, . . . , a.sub.N-1) associated with the reference pixels, and an offset coefficient a.sub.N. p.sub.n (n=0, 1, . . . , N-1) shows the reference pixels of the filter including the target pixel p.sub.0 to be filtered. N is an arbitrary number of reference pixels. s(p.sub.n) shows the luminance value of each reference pixel, and s hat (p.sub.0) shows the luminance value of the target pixel p.sub.0 to be filtered on which the filtering process has been carried out. The filter coefficients can be formed so as not to include the offset coefficient a.sub.N. Further, the luminance value of each pixel of the intermediate prediction image can be defined as the luminance value s(p.sub.n) of each reference pixel located in the partition P.sub.i.sup.n. As an alternative, the filtered luminance value can be defined as the luminance value s(p.sub.n) only at the position of each pixel on which the filtering process has been carried out. An encoded luminance value (luminance value to be decoded) is set as the luminance value s(p.sub.n) of each reference pixel located outside the partition when the pixel is in an already-encoded region, while a signal value to be used in place of the luminance value s(p.sub.n) is selected from the luminance value s(p.sub.n) of each reference pixel located in the partition P, which is defined in the above-mentioned way, and the encoded luminance value in the already-encoded area according to a predetermined procedure (for example, the signal value of a pixel at the nearest position is selected from among those of pixels which are candidates) when the pixel is in a yet-to-be-encoded region. FIG. 9 is an explanatory drawing showing an example of the arrangement of the reference pixels in the case of N=5.
When carrying out the above-mentioned filtering process, a nonlinear edge or the like occurs in the inputted image more easily and hence a displacement from the prediction direction of the intermediate prediction image occurs more easily with increase in the size (l.sub.i.sup.n.times.m.sub.i.sup.n) of the partition P.sub.i.sup.n. Therefore, it is preferable to smooth the intermediate prediction image. In addition, the larger quantized value a prediction error has, the larger quantization distortion occurs in the decoded image and hence the lower degree of prediction accuracy the intermediate prediction image generated from already-encoded pixels which are adjacent to the partition P.sub.i.sup.n has. Therefore, it is preferable to prepare a smoothed prediction image which roughly expresses the partition P.sub.i.sup.n. Further, even a pixel in the same partition has a displacement, such as an edge, occurring between the intermediate prediction image and the inputted image more easily with distance from the already-encoded pixels adjacent to the partition P.sub.i.sup.n which are used for the generation of the intermediate prediction image. Therefore, it is preferable to smooth the prediction image to suppress the rapid increase in the prediction error which is caused when a displacement occurs.
Further, the intra prediction at the time of generating the intermediate prediction image is configured in such a way as to use either of the two following different methods: an average prediction method of making all the predicted values in a prediction block be equal to one another, and a prediction method using the prediction direction vector v.sub.p. In addition, also in the case of the prediction using the prediction direction vector v.sub.p, a pixel not located at an integer pixel position is generated through interpolation on both a pixel for which the value of a reference pixel at an integer pixel position is set as its predicted value just as it is, and at least two reference pixels, the location in the prediction block of a pixel having the value of the generated pixel as its predicted value differs according to the direction of the prediction direction vector v.sub.p. Therefore, because the prediction image has a property different according to the intra prediction mode, and the optimal filtering process also changes according to the intra prediction mode, it is preferable to also change the intensity of the filter, the number of reference pixels to be referred to by the filter, the arrangement of the reference pixels, etc. according to the index value showing the intra prediction mode.
Therefore, the filter selecting process is configured in such a way as to select a filter in consideration of the four following parameters (1) to (4). (1). The size of the partition P.sub.i.sup.n(l.sub.i.sup.n.times.m.sub.i.sup.n) (2) The quantization parameter included in the prediction error encoding parameters (3) The distance between the group of already-encoded pixels ("pixels which are used for prediction" shown in FIG. 8) which are used at the time of generating the intermediate prediction image, and the target pixel to be filtered (4) The index value indicating the intra prediction mode at the time of generating the intermediate prediction image.
More specifically, the filter selecting process is configured in such a way as to use a filter having a higher degree of smoothing intensity or a filter having a larger number of reference pixels with increase in the size (l.sub.i.sup.n.times.m.sub.i.sup.n) of the partition P.sub.i.sup.n, with increase in the quantized value determined by the quantization parameter, and with distance between the target pixel to be filtered and the group of already-encoded pixels which are located on the left side and on the upper side of the partition P.sub.i.sup.n. An example of the distance between the target pixel to be filtered and the group of already-encoded pixels which are located on the left side and on the upper side of the partition P.sub.i.sup.n is listed in FIG. 11. Further, the filter selecting process is configured in such a way as to also change the intensity of the filter, the number of reference pixels to be referred to by the filter, the arrangement of the reference pixels, etc. according to the index value showing the intra prediction mode. More specifically, an adaptive selection of a filter according to the above-mentioned parameters is implemented by bringing an appropriate filter selected from among the group of filters which are prepared in advance into correspondence with each of combinations of the above-mentioned parameters. Further, for example, when combining the parameters (3) and (4), the definition of the "distance between the target pixel to be filtered and the group of already-encoded pixels" of the parameter (3) can be changed adaptively according to the "intra prediction mode" of the parameter (4). More specifically, the definition of the distance between the target pixel to be filtered and the group of already-encoded pixels is not limited to the one fixed as shown in FIG. 11, and can be a distance depending upon the prediction direction, such as the distance from a "reference pixel" shown in FIG. 8. By doing in this way, the intra prediction part can implement an adaptive filtering process which also takes into consideration a relationship between the plurality of parameters such as the parameters (3) and (4). Further, a combination for not carrying out any filtering process can be prepared as one of combinations of these parameters while being brought into correspondence with "no filtering process." In addition, as a definition of the intensity of the filter, the weakest filter can be defined as "no filtering process." Further, because the four parameters (1) to (4) are known in the moving image decoding device, no additional information to be encoded required to carry out the above-mentioned filtering process is generated. As previously explained, by preparing a necessary number of filters in advance and adaptively selecting one of them, the intra prediction part switches among the filters. As an alternative, by defining a function of the above-mentioned filter selection parameters as each filter in such a way that a filter is computed according to the values of the above-mentioned filter selection parameters, the intra prediction part can implement switching among the filters.
a.sub.0=3/4, a.sub.1=1/8, a.sub.2=1/8
a.sub.0=1/2, a.sub.1=1/4, a.sub.2=1/4
a.sub.0=1/4, a.sub.1=3/8, a.sub.2=3/8
a.sub.0=1/4, a.sub.1= 3/16, a.sub.2= 3/16, a.sub.3= 3/16, a.sub.4= 3/16
In this case, it is assumed that the filtering process is based on the equation (1) from which the offset coefficient a.sub.N is eliminated (a.sub.N=0), three types of filters are used, and each of these filters has such an arrangement of reference pixels to be referred to thereby as shown in FIG. 12.
FIG. 13 is an explanatory drawing showing an example of a table showing filters which are used in each intra prediction mode for each size of the partition P.sub.i.sup.n. In this example, it is assumed that the partition P.sub.i.sup.n has one of possible sizes of 4.times.4 pixels, 8.times.8 pixels, 16.times.16 pixels, 32.times.32 pixels, and 64.times.64 pixels, and there is a correspondence, as shown in FIG. 7, between index values each showing an intra prediction mode and intra prediction directions. Further, the filter index of 0 shows that no filtering process is carried out. In general, because there are tendencies as will be shown below when using a directional prediction or an average prediction, by bringing which filter is to be used into correspondence with each combination of the parameters (1) and (4) in the table in consideration of the characteristics of the image in intra predictions, as shown in the table shown in FIG. 13, the intra prediction part can implement the selection of an appropriate filter by referring to the table.
Because an image signal generally has high spatial continuity, it is preferable to carry out a smoothing process on pixels located in the vicinity of the block boundaries on the left and upper sides of the partition P.sub.i.sup.n, thereby improving the continuity, when using an average prediction which impairs the continuity between the partition P.sub.i.sup.n and already-encoded pixels adjacent to the partition P.sub.i.sup.n.
In general, when the partition size becomes too large, a spatial change of the signal value in the partition becomes diversified, so that the use of a directional prediction or an average prediction results in a very rough prediction, and hence a region where it is difficult to carry out a high-accurate prediction increases. Because no improvement in the prediction efficiency can be expected by simply carrying out a smoothing process to make the image become blurred in such a region, it is preferable not to carry out any filtering process in the case of such a large partition size because it is not necessary to increase the computational complexity unnecessarily (for example, in the table shown in FIG. 13, there is a setting not to carry out any filtering process in the case of a partition size of 32.times.32 pixels or more).
In addition, in a case in which the luminance value of the intermediate prediction image is used as the luminance value of each reference pixel when each reference pixel at the time that a filtering process is carried out is a pixel in the partition P.sub.i.sup.n, there is a case in which the filtering process can be simplified. For example, when the intra prediction mode is an average prediction, the filtering process on the partition P.sub.i.sup.n can be simplified to the following filtering process for each region shown in FIG. 14. Region A (pixel at the upper left corner of the partition P.sub.i.sup.n)
a.sub.0=3/4, a.sub.1=1/8, a.sub.2=1/8 (the number of reference pixels N=3)
a.sub.0=1/2, a.sub.1=1/4, a.sub.2=1/4 (the number of reference pixels N=3)
a.sub.0=1/4, a.sub.1=3/8, a.sub.2=3/8 (the number of reference pixels N=3)
a.sub.0=5/8, a.sub.1= 3/16, a.sub.2= 3/16 ((the number of reference pixels N=3)
Region B (pixels at the upper end of the partition P.sub.i.sup.n other than the region A)
a.sub.0=7/8, a.sub.2=1/8 (the number of reference pixels N=2)
a.sub.0=3/4, a.sub.2=1/4 (the number of reference pixels N=2)
a.sub.0=5/8, a.sub.2=3/8 (the number of reference pixels N=2)
a.sub.0= 13/16, a.sub.2= 3/16 (the number of reference pixels N=2)
Region C (pixels at the left end of the partition other than the region A)
a.sub.0=7/8, a.sub.1=1/8 (the number of reference pixels N=2)
a.sub.0=3/4, a.sub.1=1/4 (the number of reference pixels N=2)
a.sub.0=5/8, a.sub.1=3/8 (the number of reference pixels N=2)
a.sub.0= 13/16, a.sub.1= 3/16 (the number of reference pixels N=2)
Region D (pixels in the partition other than the regions A, B, and C)
Although the table shown in FIG. 13 is used in the above-mentioned example, another table can be alternatively used. For example, when greater importance is placed on a reduction in the computational complexity caused by the filtering process than on the degree of improvement in the encoding performance, a table shown in FIG. 19 can be used instead of the table shown in FIG. 13. Because the intra prediction unit carries out the filtering process only on the average prediction of the partition P.sub.i.sup.n whose size is 4.times.4 pixels, 8.times.8 pixels, or 16.times.16 pixels in the case of using this table, the number of prediction modes in each of which the filtering process is carried out is less than that in the case of using the table shown in FIG. 13, and therefore the increase in the computational complexity caused by the filtering process can be reduced. At this time, by using a simplification of the filtering process in the case in which the above-mentioned intra prediction mode is an average prediction, the filtering process can be implemented with very low computational complexity. In addition, when importance is placed on the ease of implementation, the intra prediction unit can carry out the filtering process only on the average prediction, like in the case of carrying out the above-mentioned filtering process, and can use the same filter (e.g., the filter of filter index of 2) at all times without not having to change the filter to be used according to the size of the partition P.sub.i.sup.n. In that case, while the degree of improvement in the encoding performance using the filter is reduced by a degree corresponding to the elimination of the process according to the size of the partition P.sub.i.sup.n, the circuit scale of the intra prediction unit installed in the device (the number of lines in the code in the case of implementing the intra prediction unit via software) can be reduced. This filtering process is simplified to a filter which takes into consideration only the parameter (4) among the four parameters (1) to (4).
The filtering process does not have to be implemented in a form in which a filter having a corresponding filter index is selected through reference to the table, and can be alternatively implemented in a form in which the filter is installed directly in the intra prediction part. For example, the filtering process is implemented in a form in which a filtering process to be carried out for each of the possible sizes of the partition P.sub.1.sup.n is incorporated directly into the intra prediction part, or a filtering process to be carried out for each pixel position in each of the possible sizes of the partition P.sub.i.sup.n is incorporated directly into the intra prediction part. As long as the prediction image which is acquired as the result of carrying out the filtering process without referring to the table in this way is equivalent to that acquired as the result of carrying out the filtering process by referring to the table, the form of the implementation is not an issue.
Even in a case in which the intra prediction part 4 is constructed in such a way as to set already-encoded pixels adjacent to the partition P.sub.i.sup.n on which the intra prediction part has carried out the smoothing process as the reference pixels at the time of generating an intermediate prediction image of the partition P.sub.i.sup.n, like in a case in which a smoothing process is carried out on a reference image at the time of an intra prediction on an 8.times.8-pixel block in MPEG-4 AVC/H.264 explained previously, the intra prediction part 4 can carry out the filtering process on an intermediate prediction image similar to that shown in the above-mentioned example. On the other hand, because there is an overlap between the effect of the smoothing process on the reference pixels at the time of generating an intermediate prediction image and that of the filtering process on the intermediate prediction image, there is a case in which even if both the processes are used simultaneously, only a very small performance improvement is produced as compared with a case in which one of the processes is carried out. Therefore, in a case in which importance is placed on reduction in the computational complexity, the intra prediction part can be constructed in such a way as not to carry out the filtering process on the intermediate prediction image of the partition P.sub.i.sup.n for which the intra prediction part has carried out the smoothing process on the reference pixels at the time of generating the intermediate prediction image. For example, there can be a case in which when carrying out the filtering process on the intermediate prediction image, the intra prediction part carries out the filtering process only on an average prediction, as shown in the table of FIG. 19, while when carrying out the smoothing process on the reference pixels at the time of generating the intermediate prediction image, the intra prediction part carries out the smoothing process by referring to the table, as shown in FIG. 20, showing that only specific directional predictions are subjected to the smoothing process. In FIG. 20, `1` shows that the smoothing process is carried out and `0` shows that the smoothing process is not carried out.
The intra prediction part outputs the intra prediction parameter used for the generation of the intra prediction image Pi to the variable length encoding part 13 in order to multiplex them into a bitstream. The intra prediction part also carries out an intra prediction process based on the intra prediction parameter (intra prediction mode) on each of the color difference signals of the partition according to the same procedure as that according to which the intra prediction part carries out the intra prediction process on the luminance signal, and outputs the intra prediction parameters used for the generation of the intra prediction image to the variable length encoding part 13. The intra prediction part can be constructed in such a way as to carry out the above-explained filtering process for the intra prediction of each of the color difference signals in the same way that the intra prediction part does for the luminance signal, or not to carry out the above-explained filtering process for the intra prediction of each of the color difference signals.
Next, the processing carried out by the moving image decoding device shown in FIG. 2 will be explained. When receiving the bitstream outputted thereto from the image encoding device of FIG. 1, the variable length decoding part 51 carries out a variable length decoding process on the bitstream to decode information having a frame size in units of a sequence which consists of one or more frames of pictures or in units of a picture (step ST21 of FIG. 4). The variable length decoding part. 51 determines a maximum size of each of coding blocks which is a unit to be processed at a time when an intra prediction process (intra-frame prediction process) or a motion-compensated prediction process (inter-frame prediction process) is carried out according to the same procedure as that which the encoding controlling part 1 shown in FIG. 1 uses, and also determines an upper limit on the number of hierarchical layers in a hierarchy in which each of the coding blocks having the maximum size is hierarchically divided into blocks (step ST22). For example, when the maximum size of each of coding blocks is determined according to the resolution of the inputted image in the image encoding device, the variable length decoding part determines the maximum size of each of the coding blocks on the basis of the frame size information which the variable length decoding part has decoded previously. When information showing both the maximum size of each of the coding blocks and the upper limit on the number of hierarchical layers is multiplexed into the bitstream, the variable length decoding part refers to the information which is acquired by decoding the bitstream.
Because the information showing the state of the division of each of the coding blocks B.sup.0 having the maximum size is included in the coding mode m(B.sup.0) of the coding block B.sup.0 having the maximum size which is multiplexed into the bitstream, the variable length decoding part 51 specifies each of the coding blocks B.sup.n into which the image is divided hierarchically by decoding the bitstream to acquire the coding mode m(B.sup.0) of the coding block B.sup.0 having the maximum size which is multiplexed into the bitstream (step ST23). After specifying each of the coding blocks the variable length decoding part 51 decodes the bitstream to acquire the coding mode m(B.sup.n) of the coding block B.sup.n to specify each partition P.sub.i.sup.n belonging to the coding block B.sup.n on the basis of the information about the partition P.sub.i.sup.n belonging to the coding mode m(B.sup.n). After specifying each partition P.sub.i.sup.n belonging to the coding block B.sup.n, the variable length decoding part 51 decodes the encoded data to acquire the compressed data, the coding mode, the prediction error encoding parameters, and the intra prediction parameter/inter prediction parameter for each partition P.sub.i.sup.n (step ST24).
More specifically, when the coding mode m(B.sup.n) assigned to the coding block B.sup.n is an intra coding mode, the variable length decoding part decodes the encoded data to acquire the intra prediction parameter for each partition P.sub.i.sup.n belonging to the coding block. In contrast, when the coding mode m(B.sup.n) assigned to the coding block B.sup.n is an inter coding mode, the variable length decoding part decodes the encoded data to acquire the inter prediction parameters for each partition P.sub.i.sup.n belonging to the coding block. The variable length decoding part further divides each partition which is a prediction unit into one or more partitions which is a transformation process unit on the basis of the transform block size information included in the prediction error encoding parameters, and decodes the encoded data of each of the one or more partitions which is a transformation process unit to acquire the compressed data (transform coefficients on which transformation and quantization are carried out) of the partition.
When the coding mode m(B.sup.n) of the partition P.sub.i.sup.n belonging to the coding block B.sup.n, which is specified by the variable length decoding part 51, is an intra coding mode (step ST25), the selection switch 52 outputs the intra prediction parameters outputted thereto from the variable length decoding part 51 to the intra prediction part 53. In contrast, when the coding mode m(B.sup.n) of the partition P.sub.i.sup.n is an inter coding mode (step ST25), the selection switch outputs the inter prediction parameters outputted thereto from the variable length decoding part 51 to the motion-compensated prediction part 54.
When receiving the intra prediction parameter from the selection switch 52, the intra prediction part 53 carries out an intra-frame prediction process on the partition P.sub.i.sup.n of the coding block B.sup.n to generate an intra prediction image P.sub.i.sup.n by using an already-decoded image signal in the frame on the basis of the intra prediction parameter (step ST26), like the intra prediction part 4 shown in FIG. 1. After generating the above-mentioned intra prediction image P.sub.i.sup.n, the intra prediction part 53 selects a filter from one or more filters, which are prepared in advance, according to the states of the various parameters which are known at the time of generating the above-mentioned intra prediction image P.sub.i.sup.n by using the same method as that which the intra prediction part 4 shown in FIG. 1 uses, and carries out a filtering process on the intra prediction image P.sub.i.sup.n by using the filter and sets the intra prediction image P.sub.i.sup.n on which the intra prediction part has carried out the filtering process as a final intra prediction image. More specifically, the intra prediction part selects a filter by using the same parameters as those which the intra prediction part 4 uses for the filter selection and by using the same method as the filter selection method which the intra prediction part 4 uses, and carries out the filtering process on the intra prediction image. For example, in a case in which the intra prediction part 4 brings the case of not carrying out the filtering process into correspondence with the filter index of 0, and further brings four filters which are prepared in advance into correspondence with filter indexes of 1 to 4 respectively, and carries out the filtering process by referring to the table shown in FIG. 13, the intra prediction part 53 is constructed in such a way as to also define the same filters and filter indexes as those for use in the intra prediction part 4, and carry out a filter selection according to the size of the partition P.sub.i.sup.n and the index showing an intra prediction mode which is an intra prediction parameter by referring to the table shown in FIG. 13 and carry out the filtering process.
When receiving the inter prediction parameters from the selection switch 52, the motion-compensated prediction part 54 carries out an motion-compensated prediction process on the partition P.sub.i.sup.n of the coding block B.sup.n to generate an inter prediction image P.sub.i.sup.n by using one or more frames of reference images stored in the motion-compensated prediction frame memory 59 on the basis of the inter prediction parameters (step ST27).
The moving image decoding device repeatedly carries out the processes of steps ST23 to ST29 until the moving image decoding device completes the processing on all the coding blocks B.sup.n into which the image is divided hierarchically (step ST30). When receiving the decoded image signal from the adding part 56, the loop filter part 58 compensates for an encoding distortion included in the decoded image signal, and stores the decoded image shown by the decoded image signal on which the loop filter part performs the encoding distortion compensation in the motion-compensated prediction frame memory 59 as a reference image (step ST31). The loop filter part 58 can carry out the filtering process for each coding block having the maximum size of the local decoded image signal outputted thereto from the adding part 56 or each coding block. As an alternative, after the local decoded image signal corresponding to all the macroblocks of one screen is outputted, the loop filter part can carry out the filtering process on all the macroblocks of the one screen at a time.
Further, because the intra prediction part 4 in accordance with this Embodiment 1 is constructed in such a way as to select a filter in consideration of at least one of the following parameters: (1) the size of the partition P.sub.i.sup.n(l.sub.i.sup.n.times.m.sub.i.sup.n); (2) the quantization parameter included in the prediction error encoding parameters; (3) the distance between the group of already-encoded pixels which are used at the time of generating the intermediate prediction image, and the target pixel to be filtered; and (4) the index value indicating the intra prediction mode at the time of generating the intermediate prediction image, there is provided an advantage of preventing a local prediction error from occurring when, for example, an edge of the image to be encoded becomes distorted slightly in a nonlinear shape or a slight displacement occurs in the angle of an edge in the image to be encoded when carrying out a directional prediction, and preventing a prediction error from occurring at a boundary between blocks due to a loss of the continuity with the signal of an already-encoded pixel adjacent to the partition when carrying out an average prediction, thereby being able to improve the prediction efficiency.
Further, because the intra prediction part 53 in accordance with this Embodiment 1 is constructed in such a way as to select a filter in consideration of at least one of the following parameters: (1) the size of the partition P.sub.i.sup.n (l.sub.i.sup.n.times.m.sub.i.sup.n); (2) the quantization parameter included in the prediction error encoding parameters; (3) the distance between the group of already-encoded pixels which are used at the time of generating the intermediate prediction image, and the target pixel to be filtered; and (4) the index value indicating the intra prediction mode at the time of generating the intermediate prediction image, there are provided an advantage of preventing a local prediction error from occurring when, for example, an edge of the image to be encoded becomes distorted slightly in a nonlinear shape or a slight displacement occurs in the angle of an edge in the image to be encoded when carrying out a directional prediction, and preventing a prediction error from occurring at a boundary between blocks due to a loss of the continuity with the signal of an already-encoded pixel adjacent to the partition when carrying out an average prediction, and another advantage of making it possible for the moving image decoding device to also generate the same intra prediction image as that generated by the moving image encoding device.
Each of the intra prediction parts 4 and 53 in accordance with above-mentioned Embodiment 1 is constructed in such a way as to select a filter from one or more filters which are prepared in advance according to the states of various parameters associated with the encoding of a target block to be filtered. While each of the intra prediction parts can select an appropriate filter from the one or more selection candidates in consideration of the four parameters (1) to (4), each of the intra prediction parts cannot carry out "optimal filtering" when an optimal filter other than the one or more selection candidates exists. This Embodiment 2 is characterized in that while a moving image encoding device designs an optimal filter on a per picture basis and carries out a filtering process, and also encodes the filter coefficients of the filter, and so on, a moving image decoding device decodes the filter coefficients and so on, and carries out a filtering process by using the filter.
An intra prediction part 4 of the moving image encoding device carries out an intra-frame prediction process on each partition P.sub.i.sup.n of each coding block B.sup.n to generate an intra prediction image P.sub.i.sup.n, like that according to above-mentioned Embodiment 1. The intra prediction part 4 also selects a filter from one or more filters which are prepared in advance according to the states of various parameters associated with the encoding of a target block to be filtered by using the same method as that the intra prediction part according to above-mentioned Embodiment 1 uses, and carries out a filtering process on the intra prediction image P.sub.i.sup.n by using this filter. After determining intra prediction parameters for each of all coding blocks B.sup.n in the picture, for each area in which an identical filter is used within the picture (each area having the same filter index), the intra prediction part 4 designs a Wiener filter which minimizes the sum of squared errors between the inputted image in the area and the intra prediction image (mean squared error in the target area).
The filter coefficients w of the Wiener filter can be determined from an autocorrelation matrix R.sub.s's' of an intermediate prediction image signal s', and a cross correlation matrix R.sub.ss' of the inputted image signal s and the intermediate prediction image signal s' according to the following equation (4). The size of the matrices R.sub.s's' and R.sub.ss' corresponds to the number of filter taps determined. w=R.sub.s's'.sup.-1R.sub.ss' (4)
After designing the Wiener filter, the intra prediction part 4 expresses the sum of squared errors in the target area for filter design in the case of carrying out a filtering process using the Wiener filter as D1, the code amount at the time of encoding information (e.g., filter coefficients) associated with the Wiener filter as R1, and the sum of squared errors in the target area for filter design in the case of carrying out a, filtering process using a filter which is selected by using the same method as that shown in above-mentioned Embodiment 1 as D2, and then checks to see whether or not the following equation (5) is established. D1+.lamda.R1<D2 (5) Where .lamda. is a constant.
When carrying out a filtering process by using the Wiener filter, the intra prediction part 4 requires filter update information showing the filter coefficients of the Wiener filter and indexes each indicating a corresponding filter which is replaced by the Wiener filter. More specifically, when the number of filters selectable in the filtering process using filter selection parameters is expressed as L, and indexes ranging from zero to L-1 are assigned to the filters, respectively, when the designed Wiener filter is used for each index, a value of "1" needs to be encoded for the index as the filter update information, whereas when a prepared filter is used for each index, a value of "0" needs to be encoded for the index as the filter update information. A variable length encoding part 13 variable-length-encodes the filter update information outputted thereto from the intra prediction part 4, and multiplexes encoded data of the filter update information into a bitstream.
A variable length decoding part 51 of a moving image decoding device variable-length-decodes the encoded data multiplexed into the bitstream to acquire the filter update information. An intra prediction part 53 carries out an intra-frame prediction process on each partition P.sub.i.sup.n of each coding block B.sup.n to generate a intra prediction image P.sub.i.sup.n according to above-mentioned Embodiment 1. When receiving the filter update information from the variable length decoding part 51, the intra prediction part 53 refers to the filter update information to check to see whether or not there is an update to the filter indicated by the corresponding index.
When determining from the result of the check that the filter for a certain area is replaced by a Wiener filter, the intra prediction part 53 reads the filter coefficients of the Wiener filter which are included in the filter update information to specify the Wiener filter, and carries out a filtering process on the intra prediction image P.sub.i.sup.n by using the Wiener filter. In contrast, for an area in which no filter is replaced by a Wiener filter, the intra prediction part selects a filter by using the same method as that which the intra prediction part according to above-mentioned Embodiment 1 uses, and carries out a filtering process on the intra prediction image P.sub.1 by using the filter.
Previous Patent US 9,609,325 | Next Patent US 9,609,327