Source: https://patents.justia.com/patent/10158871
Timestamp: 2019-08-26 05:27:51
Document Index: 8414463

Matched Legal Cases: ['§ 119', 'Application No. 03021858', 'Application No. 2003', 'Application No. 10', 'art 09', 'Application No. 10180810']

US Patent for Video encoding method, video decoding method, video encoding apparatus, video decoding apparatus, video encoding program, and video decoding program Patent (Patent # 10,158,871 issued December 18, 2018) - Justia Patents Search
Justia Patents PluralUS Patent for Video encoding method, video decoding method, video encoding apparatus, video decoding apparatus, video encoding program, and video decoding program Patent (Patent # 10,158,871)
Nov 17, 2017 - NTT DOCOMO, INC.
A video processing system provided with video encoding apparatus 1 and video decoding apparatus 2. The encoding apparatus 1 outputs a maximum delay time that is incurred by backward prediction, in addition to encoded data D1 resulting from encoding of video data D0. The decoding apparatus 2 effects input of the maximum delay time that is incurred by backward prediction, in addition to encoded data D1 from the encoding apparatus 1. Then the decoding apparatus 2 decodes the encoded data D1 with reference to the input maximum delay time to generate motion video data D2.
This application is a continuation of application Ser. No. 12/537,600 filed Aug. 7, 2009, which is a continuation of application Ser. No. 10/671,527 filed Sep. 29, 2003, which claims priority under 35 U.S.C. § 119 to Japanese Patent Application Nos. JP2002-291610 filed Oct. 3, 2002 and JP2003-190634 filed on Jul. 2, 2003. The entire contents of all of these applications are incorporated herein by reference.
Video signal encoding techniques are used for transmission and storage-regeneration of video signals. The well-known techniques include, for example, the international standard video coding methods such as ITU-T Recommendation H.263 (hereinafter referred to as H.263), ISO/IEC International Standard 14496-2 (MPEG-4 Visual, hereinafter referred to as MPEG-4), and so on. Another known newer encoding method is a video coding method scheduled fear joint international standardization by ITU-T and ISO/IEC; ITU-T Recommendation H.264 and ISO/IEC International Standard 14496-10 (Joint Final Committee Draft of Joint Video Specification, hereinafter referred to as H.26L).
FIGS. 7A and 7B are diagrams showing (A) decoding and (B) output of the frames in the case of the bidirectional prediction shown in FIG. 6. For example, in the decoding of MPEG-4, where the current frame A1 is decoded by bidirectional interframe prediction, frame A0 being one temporally previous frame and frame A2 being one temporally subsequent frame with respect to the current frame A1 are first decoded as frames decoded by intraframe prediction without use of interframe prediction or as frames decoded by forward interframe prediction, prior to decoding of the current frame A1, and they are retained as reference frames. Thereafter, the current frame A1 is decoded by bidirectional prediction using these two frames AD, A2 thus retained (FIG. 7A).
Some of the recent video coding methods permit the foregoing interframe prediction to be carried out using multiple reference frames, instead of one reference frame in the forward direction and one reference frame in the backward direction, so as to enable prediction from a frame with a smaller change from the current frame, as shown in FIG. 8. FIG. 8 shows an example using two temporally previous frames B0, B1 and two temporally subsequent frames B3, B4 with respect to the current frame B2, as reference frames the current frame B2.
FIGS. 9A and 9B are diagrams showing (A) decoding and (B) output of the frames in the case of the bidirectional prediction shown in FIG. 8. For example, in the decoding of H.26L, a plurality of reference frames can be retained within a range up to a predetermined upper bound of the number of reference frames and, on the occasion of carrying out interframe prediction, an optimal reference frame is arbitrarily designated out of them. In this case, where the current frame B2 is decoded as a bidirectionally predicted frame, the reference frames are first decoded prior to the decoding of the current frame B2; the reference frames include a plurality of temporally previous frames (e.g., two frames B0, B1) and a plurality of temporally subsequent frames (e.g., two frames B3, B4) with respect to the current frame B2, which are decoded and retained as reference frames. The current frame B2 can be predicted from a frame arbitrarily designated as the one used for prediction out of those frames B0, B1, B3, and B4 (FIG. 9A).
Application of the maximum delay time can be a method of applying at to entire encoded data or a method of applying it to each frame. Another application method is a method of applying the maximum delay time to each of the frames subsequent to the announcement of the information of the maximum delay time, i.e., to the frame for which the maximum delay time is indicated and to each of the frames temporally subsequent to that frame. The output, input, application, etc. of the maximum delay time in these methods will be specifically detailed later.
In the case where the maximum number of reference frames used for backward prediction is 2 and where the frame rate is 15 frames/second, as shown in FIG. 2, the time interval of one frame is 1/15 second. In this case, therefore, the maximum delay time is 2×( 1/15)= 2/15 second.
FIG. 3 is a block diagram showing an example of the configuration of the video encoding apparatus used in the present embodiment. The video encoding apparatus 1 shown in FIG. 3 is comprised of an encoder 10 for encoding a frame (image) by the predetermined method, a controller (CPU) 15 for controlling operations of respective parts in the encoding apparatus 1, a frame memory 11 disposed between input terminal 1a and encoder 10, and a multiplexer 12 disposed between output terminal 1b and encoder 10. The controller 15 has a maximum delay time calculator 16 for calculating the maximum delay time, as a function thereof. The encoder 10 is provided with an output buffer 13.
In the video encoding in the present encoding apparatus 1, conditions for encoding of video are entered through input terminal 1c. In this entry of the conditions, the encoding conditions are generally selected or entered through an input device such as a keyboard. In the present embodiment, specifically, the encoding conditions entered include the size of a frame as a target for encoding, the frame rate, and the bit rate and, in addition thereto, the encoding conditions also include a predictive reference structure of the video (whether backward prediction is applied), the number of frames temporarily stored and used as reference frames (corresponding to the capacity of output buffer 13), and the number of reference frames used in backward prediction. These conditions may be set so as to vary with time. The encoding conditions entered through the input terminal 1c are stored into the controller 15.
With a start of the encoding operation, the controller 15 sends the encoding conditions to the encoder 10, where the encoding condition are set. On the other hand, a frame as an encoded object is entered through the input terminal 1a and is fed through the frame memory 11 to the encoder 10 to be encoded therein. The input frame is temporarily saved in the frame memory 11, because the order of frames is changed for execution of backward prediction. For example, in the example shown in FIG. 2, frame F2 is entered through the input terminal 1a before frames F3, F4, but it is encoded after the frames F3, F4; therefore, the frame F2 is temporarily saved in the frame memory 11.
The encoder 10 encodes the frame on the basis of the algorithm of H.26L. Then the encoded data is fed to the multiplexer 12 to be multiplexed with other related information and then the multiplexed data is outputted through the output terminal 1b. The frame used for the prediction is reproduced in the encoder 10 and is stored as a reference frame for encoding of the next frame into the buffer 13.
In the present embodiment, the maximum delay time calculator 16 of the controller 15 calculates the maximum delay time dpb_output_delay, based on the number of reference frames and the frame rate entered through the input terminal is and used for the backward prediction. Then the multiplexer 12 adds the maximum delay time to encoded image data. In addition, an identifier (N) indicating a display turn for identification of each frame is also added together to the encoded data of each frame.
The parameter dpb_output_delay is defined as a syntax for carrying the maximum delay time. It is assumed here that the parameter dpb_output_delay uses the same time unit used in the other syntaxes indicating the time in H.26L and that it indicates the maximum delay time in the time unit of 90 kHz, A numeral indicated in the time unit is encoded and transmitted by a 32-bit unsigned fixed-length code. For example, where the maximum delay time is 2/15 second as described above, dpb_output_delay is ( 2/15)×90000=12000.
FIG. 4 is a block diagram showing an example of the configuration of the video decoding apparatus used in the present embodiment. The video decoding apparatus 2 shown in FIG. 4 is comprised of a decoder 20 for decoding encoded data to generate a regenerated image, a controller (CPU) 25 for controlling operations of respective parts in the decoding apparatus 2, an input buffer 21 disposed between input terminal 2a and decoder 20, and an output buffer 22 disposed between output terminal 2b and decoder 20. The controller 25 has an image output time calculator 26 for calculating an output time for display of a frame, as a function thereof.
In the video decoding in the present decoding apparatus 2, data as a decoded object is entered through the input terminal 2a. This data is multiplexed data of the encoded data of each frame encoded by the encoding apparatus 1 shown in FIG. 3, the maximum delay time dpb_output_delay, and the identifier (N) indicating the display turn of each frame.
To(n)=dpb_output_delay+N×frame interval
According to the output times To(n) obtained in this way by the controller 25, the frames in the output buffer 22 are outputted at constant intervals to the output terminal 2h, as indicated by frames F0, F1, F2, and F3 shown in FIG. 5B. Although not illustrated, the output terminal 2b is connected to a display device such as a monitor.
In the present embodiment, the time Tr(n) for encoding of each frame is also encoded together. Taking FIG. 2 as an example, D= 2/15 second, and Tin(n)=0, 1/15, 2/15, 3/15, or 4/15 (n=0, 1, 2, 3, or 4). Because of change in the order of encoding, Tr(n) becomes as follows: Tr(n)=0, 1/15, 4/15, 2/15, or 3/15 (n=1, 2, 3, or 4). Here dpb_output_delay(n) of each frame is obtained as follows.
n = 0 , dpb_output ⁢ _delay ⁢ ( 0 ) = 0 + 2 / 15 - 0 = 2 / 15 n = 1 , dpb_output ⁢ _delay ⁢ ( 1 ) = 1 / 15 + 2 / 15 - 1 / 15 = 2 / 15 n = 2 , dpb_output ⁢ _delay ⁢ ( 2 ) = 2 / 15 + 2 / 15 - 4 / 15 = 0 n = 3 , dpb_output ⁢ _delay ⁢ ( 3 ) = 3 / 15 + 2 / 15 - 2 / 15 = 3 / 15 n = 4 , dpb_output ⁢ _delay ⁢ ( 4 ) = 4 / 15 + 2 / 15 - 3 / 15 = 3 / 15
In the present embodiment the syntax for transmitting the maximum delay time was the syntax by fixed-length codes added into the Picture Parameter Set, and it is a matter of course that the codes and syntax for transmitting it, the time unit for expressing the maximum delay time are not limited to these, of course. The fixed-length codes can be replaced by variable-length codes, and the maximum delay time can be announced in any of various syntaxes capable of announcing the information to be applied to each frame.
1. A method of decoding a sequence of encoded pictures, the method comprising computer implemented steps performed by a processor of a computer system to implement the following steps:
receiving the encoded pictures, a decoding time of each of the encoded pictures, and delay time syntaxes associated respectively with the encoded pictures, wherein the delay time syntaxes represent at least a maximum delay time that is incurred by backward prediction uniquely defined for the individual encoded pictures;
decoding the received encoded pictures into decoded pictures, based on the received decoding time;
holding at least some of the decoded pictures in a decoded picture buffer before outputting the at least some of the decoded pictures from the decoded picture buffer at respective output times, wherein the decoded picture buffer holds at least one decoded picture used for reference to decode at least one of the encoded pictures; and
calculating the respective output times of the decoded picture, based on the respective decoding time and the respective maximum delay times.
2. A decoding apparatus for decoding a sequence of encoded pictures, comprising:
a receiver configured to receive the encoded pictures, a decoding time of each of the encoded pictures and delay time syntaxes associated respectively with the encoded pictures, wherein the delay time syntaxes represent at least a maximum delay time that is incurred by backward prediction uniquely defined for the individual encoded pictures;
a decoder configured to decode the received encoded pictures into decoded pictures, based on the received decoding time;
a decoded picture buffer configured to be able to hold the decoded pictures; and
an output control configured to hold at least some of the decoded pictures in the decoded picture buffer before outputting the at least some of the decoded pictures from the decoded picture buffer at respective output times, wherein the decoded picture buffer holds at least one decoded picture used for reference to decode at least one of the encoded pictures,
wherein the output control is further configured to calculate the respective output times of the at least some decoded pictures, based on the respective decoding times and the respective maximum delay times.
5828787 October 27, 1998 Uehara
5952943 September 14, 1999 Walsh et al.
6188700 February 13, 2001 Kato et al.
6205177 March 20, 2001 Girod
6393153 May 21, 2002 Ozaki
6904089 June 7, 2005 Sueyoshi et al.
6934335 August 23, 2005 Liu et al.
7596178 September 29, 2009 Adachi et al.
9826228 November 21, 2017 Adachi
2 268 036 December 2010 EP
08-205146 August 1996 JP
10-248066 September 1998 JP
2002-218461 August 2002 JP
10-0269548 October 2000 KR
WO 96/33573 October 1996 WO
Wiegand, Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification, Mar. 2003, JVT & ISO/IEC 14496-10 AVC, JVT-G050, pp. 204-238.
European Search Report from European Application No. 03021858.0, dated Mar. 31, 2005, 3 pages.
“Hypothetical Reference Decoder”, ITU-T Rec. H.264 ISO/IEC 14496-10 AVC, Jul. 22-26, 2002, 27 pages.
Office Action from Japanese Application No. 2003-190634, dated Sep. 29, 2003, 9 pages (with English translation).
Office Action from Korean Application No. 10-2003-0069014, dated Sep. 28, 2005, 2 pages.
Office Action from U.S. Appl. No. 10/671,527, dated Oct. 14, 2008, 8 pages.
Ono, Fumitaka et al., “Basic Technologies on International Image Coding Standards”, Corona Publishing Co., Ltd., Mar. 20, 1998, pp. 236-267 (with English translation).
Wiegand, T., N4920 Text of Final Committee Draft of Joint Video Specification (ITU-T Rec. H.264 ISO/IEC 14496-10AVC)—Annex C, “Hypothetical Reference Decoder”, ISO IEC JTC1 SC29 WG11, Coding of Moving Pictures and Audio, XP-002321520, Jul. 2002, pp. 157-161.
Wiegand, T., “Draft ITU-T Recommendation and Final Draft International Standard Joint Video Specification (ITU-T Rec. H.264 ISO/IEC 14496-10 AVC”, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6) 7th Meeting: Pattaya, Thailand, Mar. 7-14, 2003, pp. 204-238.
Knoll A et al., “Videokonferenz Und Bi Ldfernsprechen”, Fernmelde-Ingenieur, Bad Winsheim, DE, vol. 52, No. Part 09/10, Sep. 1, 1998 (Sep. 1, 1998), pp. 1-54.
Tian et al., “DPB Management”, 1,25. JVT Meeting; 53. MPEG Meeting; Sep. 12, 2002-Dec. 13, 2002, Awaji, JP; (Joint Video Team of ISO/IEC JTCI/SC29/WGI1 and ITU-T SG.16) No. JVT-F047-L, Dec. 13, 2002.
European Search Report issue in a counterpart EPO application dated Apr. 11, 2012.
Office Action in corresponding European Application No. 10180810.3, dated Feb. 3, 2015, 4 pages.
Patent number: 10158871
Patent Publication Number: 20180077422
Inventors: Satoru Adachi (Yokohama), Choong Seng Boon (Yokohama), Sadaatsu Kato (Yokosuka), Minoru Etoh (Yokohama), Thiow Keng Tan (Jalan Sindor)
Primary Examiner: Shawn S An
Application Number: 15/816,119
International Classification: H04N 7/12 (20060101); H04N 19/44 (20140101); H04N 19/50 (20140101); H04N 19/503 (20140101); H04N 19/46 (20140101);