Source: http://www.google.com/patents/US6078616?dq=KOI-18
Timestamp: 2017-11-19 04:05:41
Document Index: 64191238

Matched Legal Cases: ['art 1', 'art 2', 'art 1', 'art 2', 'application No. 08', 'application No. 08', 'application No. 08']

Patent US6078616 - Methods and apparatus for error concealment utilizing temporal domain motion ... - Google Patents
A method and apparatus for concealing errors during decoding of a video bit stream utilize estimates, if possible, motion vectors in the temporal domain. If estimation in the temporal domain is not possible, motion vectors are estimated in the spatial domain. A macroblock is then estimated based upon...http://www.google.com/patents/US6078616?utm_source=gb-gplus-sharePatent US6078616 - Methods and apparatus for error concealment utilizing temporal domain motion vector estimation
Publication number US6078616 A
Application number US 08/816,867
Also published as CN1256048A, DE19882177T0, DE19882177T1, US6175597, US6285715, US6449311, WO1998041028A1
Publication number 08816867, 816867, US 6078616 A, US 6078616A, US-A-6078616, US6078616 A, US6078616A
Inventors Taner Ozcelik, Gong-san Yu, Shirish C. Gadre
Patent Citations (24), Non-Patent Citations (37), Referenced by (43), Classifications (17), Legal Events (7)
Methods and apparatus for error concealment utilizing temporal domain motion vector estimation
US 6078616 A
A method and apparatus for concealing errors during decoding of a video bit stream utilize estimates, if possible, motion vectors in the temporal domain. If estimation in the temporal domain is not possible, motion vectors are estimated in the spatial domain. A macroblock is then estimated based upon the estimated motion vector. If estimation in the spatial domain is not possible, macroblock estimation is made without the use of an estimated motion vector.
1. A method of error concealment in decoding a coded video bit stream, comprising the steps of:
estimating a motion vector in a temporal domain, the step of estimating the motion vector in the temporal domain including the steps of:
detecting a first decoded motion vector for a forward reference frame at a macroblock positioned by a vector;
detecting a second decoded motion vector for the difference between the forward reference frame at the macroblock positioned by the vector and the detected decoded motion vector for the forward reference frame at the macroblock positioned by the vector; and
determining in a temporal domain an estimated motion vector for a current frame at the macroblock positioned by the vector based upon the detected second decoded motion vector;
if the step of estimating the motion vector in the temporal domain is unsuccessful, estimating the motion vector in a spatial domain;
updating a motion vector for use in estimating the macroblock based on the estimated motion vector determined in either the temporal or the spatial domain; and
estimating the macroblock based on the estimated motion vector.
2. The method of claim 1 wherein the estimated motion vector for the current frame determined in the temporal domain equals the detected second decoded motion vector.
5. The method of claim 1 wherein the step of estimating the motion vector in the spatial domain includes the steps of:
detecting a third decoded motion vector for the current frame at a macroblock positioned immediately above the current macroblock; and
determining in a spatial domain the estimated motion vector for the current frame at the macroblock positioned by the vector based upon the detected third decoded motion vector.
6. The method of claim 5 wherein the macroblock positioned immediately above the current macroblock is defined by a vector having a row index of 1 and a column index of 0.
7. An apparatus for error concealment in decoding a coded video bit stream, comprising:
means for estimating a motion vector in a temporal domain, the means for estimating the motion vector in the temporal domain including:
means for detecting a first decoded motion vector for a forward reference frame at a macroblock positioned by a vector;
means for detecting a second decoded motion vector for the difference between the forward reference frame at the macroblock positioned by the vector and the detected decoded motion vector for the forward reference frame at the macroblock positioned by the vector; and
means for determining in a temporal domain an estimated motion vector for a current frame at the macroblock positioned by the vector based upon the detected second decoded motion vector;
means for estimating a motion vector in a spatial domain, wherein when the means for estimating the motion vector in the temporal domain is unsuccessful, the means for estimating the motion vector in the spatial domain determines the estimated motion vector;
means for updating a motion vector for use in estimating the macroblock based on the estimated motion vector determined in either the temporal or the spatial domain; and
means for estimating the macroblock based on the estimated motion vector.
8. The apparatus of claim 7 wherein the estimated motion vector for the current frame in the temporal domain equals the detected second decoded motion vector.
11. The apparatus of claim 7 wherein said means for estimating said motion vector in said spatial domain includes:
means for detecting a third decoded motion vector for the current frame at a macroblock positioned immediately above the current macroblock; and
means determining in a spatial domain the estimated motion vector for the current frame at the macroblock positioned by the vector based upon the detected third decoded motion vector.
12. The apparatus of claim 11 wherein the macroblock positioned immediately above the current macroblock is defined by a vector having a row index of 1 and a column index of 0.
13. An apparatus for error concealment in decoding a coded video bit stream, comprising:
a first motion vector estimating unit for estimating a motion vector in a temporal domain, the first motion vector estimating unit including:
a first detection unit for detecting a first decoded motion vector for a forward reference frame at a macroblock positioned by a vector;
a second detection unit for detecting a second decoded motion vector for the difference between the forward reference frame at the macroblock positioned by the vector and the detected decoded motion vector for the forward reference frame at the macroblock positioned by the vector; and
a first determining unit for determining in a temporal domain an estimated motion vector for a current frame at the macroblock positioned by the vector based upon the detected second decoded motion vector;
a second motion vector estimating unit for estimating a motion vector in a spatial domain, wherein when the second motion vector estimating unit is configured to estimate the motion vector in the spatial domain when the first motion vector estimating unit is unsuccessful in estimating the motion vector in a temporal domain;
an updating unit for updating a motion vector for use in estimating the macroblock based on the estimated motion vector determined in either the temporal or the spatial domain; and
an estimating unit for estimating the macroblock based on the estimated motion vector.
14. The apparatus of claim 13 wherein the estimated motion vector for the current frame in the temporal domain equals the detected second decoded motion vector.
16. The apparatus of claim 15 wherein said estimating unit includes:
a third detection unit for detecting a third decoded motion vector for the current frame at a macroblock positioned immediately above the current macroblock; and
a second determining unit for determining in a spatial domain the estimated motion vector for the current frame at the macroblock positioned by the vector based upon the detected third decoded motion vector.
17. The apparatus of claim 16 wherein the macroblock positioned immediately above the current macroblock is defined by a vector having a row index of 1 and a column index of 0.
19. The apparatus of claim 18 wherein said estimating unit includes:
20. The apparatus of claim 19 wherein the macroblock positioned immediately above the current macroblock is defined by a vector having a row index of 1 and a column index of 0.
21. The apparatus of claim 13 wherein said estimating unit includes:
22. The apparatus of claim 21 wherein the macroblock positioned immediately above the current macroblock is defined by a vector having a row index of 1 and a column index of 0.
In further detail, FIG. 1 illustrates a simplified block diagram of an MPEG2 encoder 100. A video stream consisting of macroblock information and motion compensation information is provided to both a discrete cosine transformer 102 and a motion vector generator 104. Each 8×8 block (of pixels or error terms) is processed by the discrete cosine transformer 102 to generate an 8×8 block of horizontal and vertical frequency coefficients. The quantizer 106 quantizes the 8×8 block of frequency-domain error coefficients, thereby limiting the number of allowed values.
It is an object of the invention to provide a method and apparatus for concealing errors during decoding of compressed video signals.
FIG. 1 is a simplified block diagram of a MPEG 2 video encoder.
Referring now to FIG. 4, a block diagram of an MPEG 2 decoder 400 is shown. Decoder 400 utilizes two internal busses, a GBUS 402 and an RBUS 404. In the preferred embodiment of the invention, GBUS 402 is a 64 bit bus which is utilized for data transfer between DRAM 406 and specific blocks of decoder 400 which are described below. In the preferred embodiment of the invention, DRAM 406 is a static dynamic random access memory, although other types of memories may be utilized. RBUS 404 is an 8 bit but used primarily for control of specific blocks through reduced instruction set computing ("RISC") CPU 408. RISC CPU 408, which is coupled to both GBUS 402 and RBUS 404, operates to control the functionality of specific blocks, as more particularly described below, as well as performing a portion of video bitstream decoding.
Letter box unit 418 is essentially a vertical decimation filter with downloadable coefficients. Letter box unit 418 operates to decimate an active area of a frame which has a ratio of 4:3. Therefore, for PAL sequences, letter box unit 418 converts a 720×576 frame to a 720×432 frame. For NTSC sequences, letter box unit 418 converts a 720×480 frame to a 720×360 frame. However, in both cases, the active picture area is centered with respect to a display area.
Returning to FIG. 7, at step 704, it is determined whether motion vector estimation in the temporal domain was successful. If so, the algorithm proceeds to step 706, where based upon the estimated motion vector, the motion vector to be used for estimating the subject macroblock is updated. If the motion vector estimation in the temporal domain was not successful, the algorithm proceeds to step 708, where motion vector estimation is performed in the spatial domain. The algorithm for such estimation is shown in FIG. 9. The algorithm starts at step 900 of FIG. 9 and proceeds to step 902, where it is determined whether a decoded motion vector for the macroblock located immediately above the estimating macroblock is available. Such a motion vector is designated by MV(k,p-(1,0)). If not, a failure is indicated at step 904. If a decoded motion vector for the macroblock located immediately above the estimating macroblock is available, then, at step 906, the motion vector for the current frame, the k-th frame at the macroblock positioned by the vector p, ˜MV(k,p), is estimated to be equal to the decoded motion vector, the macroblock located immediately above the estimating macroblock, MV(k,p-(1,0)), where (1,0) is a vector indicating a row index as 1 and a column index as 0. The algorithm then proceeds to step 908 which indicates a successive motion vector estimation in the spatial domain.
As shown in FIG. 10, at step 1000, macroblock estimation with the estimated motion vector is started. At step 1002, the estimated macroblock for the current frame, the k-th frame, at the macroblock positioned by the estimated motion vector p, ˜MB(k,p), is estimated to be equal to the decoded macroblock of the difference of (1) the forward reference frame at the macroblock positioned by the vector p; and (2) the estimated motion vector for the current frame, the k-th frame, at the macroblock positioned by the vector p. This decoded macroblock is designated as MB(k-m,p-˜MV(k,p)), where m is the frame index difference between the current frame and a forward reference frame. Upon completion of estimation of the current macroblock at step 712, the algorithm is completed at step 714.
US5552831 * Jun 6, 1995 Sep 3, 1996 Matsushita Electric Industrial Co., Ltd. Digital video signal decoding apparatus and presumed motion vector calculating method
US5561532 * Mar 30, 1994 Oct 1, 1996 Canon Kabushiki Kaisha Image reproducing apparatus
US5604822 * Nov 12, 1993 Feb 18, 1997 Martin Marietta Corporation Methods and apparatus for centroid based object segmentation in object recognition-type image processing system
EP0727910A2 * Feb 5, 1996 Aug 21, 1996 THOMSON multimedia Temporal-spatial error concealment apparatus and method for video signal processors
1 * Applied Optics, vol. 21, No. 15, pp. 2758 2769, Aug. 1, 1982, J.R. Fienup, Phase Retrieval Algorithms: a Comparison.
2 Applied Optics, vol. 21, No. 15, pp. 2758-2769, Aug. 1, 1982, J.R. Fienup, "Phase Retrieval Algorithms: a Comparison."
3 B. Benson, "Television Engineering Handbook," McGraw Hill, New York, Publication Date Dec. 1985, pp. 2.7 & 2.8.
4 * B. Benson, Television Engineering Handbook, McGraw Hill, New York, Publication Date Dec. 1985, pp. 2.7 & 2.8.
5 H. Sun et al., "Adaptive Error Concealment Algorithm for MPEG Compressed Video," Visual Communications and Image Processing 1992, Boston, SPIE vol. 1818, Nov. 18, 1992, pp. 814-824.
6 * H. Sun et al., Adaptive Error Concealment Algorithm for MPEG Compressed Video, Visual Communications and Image Processing 1992, Boston, SPIE vol. 1818, Nov. 18, 1992, pp. 814 824.
7 IEEE Nov. 1994, P. Ferreira et al., "Errorless Restoration Algorithms for Band-Limited Images," pp. 157-161.
8 * IEEE Nov. 1994, P. Ferreira et al., Errorless Restoration Algorithms for Band Limited Images, pp. 157 161.
9 * IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. ASSP 29, No. 4, pp. 830 845, Aug. 1981, A. Jain, Extrapolation Algorithms for Discrete Signals With Application in Spectral Estimation.
10 IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. ASSP-29, No. 4, pp. 830-845, Aug. 1981, A. Jain, "Extrapolation Algorithms for Discrete Signals With Application in Spectral Estimation."
11 * IEEE Transactions on Circuits and Systems, vol. CAS 22, No. 9, pp. 735 742, Sep. 1975, A. Papoulis, A New Algorithm in Spectral Analysis and Band Limited Extrapolation. .
12 IEEE Transactions on Circuits and Systems, vol. CAS-22, No. 9, pp. 735-742, Sep. 1975, A. Papoulis, "A New Algorithm in Spectral Analysis and Band-Limited Extrapolation.".
13 * IEEE Transactions on Medical Imaging, vol. MI 1, No. 2, pp. 81 94, Oct. 1982, D.C. Youla et al., Image Restoration by the Method of Convex Projections: Part 1 Theory.
14 * IEEE Transactions on Medical Imaging, vol. MI 1, No. 2, pp. 95 101. Oct. 1982, M.I. Sezan et al., Image Restoration by the Method of Convex Projections: Part 2 Applications and Numerical Results.
15 IEEE Transactions on Medical Imaging, vol. MI-1, No. 2, pp. 81-94, Oct. 1982, D.C. Youla et al., Image Restoration by the Method of Convex Projections: Part 1--Theory.
16 IEEE Transactions on Medical Imaging, vol. MI-1, No. 2, pp. 95-101. Oct. 1982, M.I. Sezan et al., "Image Restoration by the Method of Convex Projections: Part 2--Applications and Numerical Results."
17 * IEEE Transactions on Signal Processing, vol. 39, No. 10, pp. 2275 2285, Oct. 1991, M. Sezan et al., Prototype Image Constraints for Set Theoretic Image Restoration.
18 IEEE Transactions on Signal Processing, vol. 39, No. 10, pp. 2275-2285, Oct. 1991, M. Sezan et al., "Prototype Image Constraints for Set-Theoretic Image Restoration."
19 J. Beyerer, "Suppression of Stochastically Placed, Straight Toolmarks to Enhance Objects and Defects," Techniques Messen TM 1982-1988, vol. 59, No. 10, Oct. 1992, pp. 389-397.
20 * J. Beyerer, Suppression of Stochastically Placed, Straight Toolmarks to Enhance Objects and Defects, Techniques Messen TM 1982 1988, vol. 59, No. 10, Oct. 1992, pp. 389 397.
21 K. Popat et al., "Novel Cluster-Based Probablity Model for Texture Synthesis, Classification, and Compression", Massachusettes Institute of Technology, Media Laboratory, SPIE vol. 2094, pp. 756-768, Nov. 1993.
22 * K. Popat et al., Novel Cluster Based Probablity Model for Texture Synthesis, Classification, and Compression , Massachusettes Institute of Technology, Media Laboratory, SPIE vol. 2094, pp. 756 768, Nov. 1993.
23 * Optica Acta, 1974, vol. No. 9, 709 720, R.W. Gerchberg, Super Resolution Through Error Energy Reduction. (Received Sep. 13, 1973, Revision Received Jan. 15, 1974).
24 Optica Acta, 1974, vol. No. 9, 709-720, R.W. Gerchberg, "Super-Resolution Through Error Energy Reduction." (Received Sep. 13, 1973, Revision Received Jan. 15, 1974).
25 * Optical Society of America, vol. 6, No. 6, pp. 844 851, Jun. 1989, H. Peng et al., Signal Recovery With Similarity Constraints,
26 Optical Society of America, vol. 6, No. 6, pp. 844-851, Jun. 1989, H. Peng et al., "Signal Recovery With Similarity Constraints,"
27 * Optical Society of America, vol. 9, No. 7, pp. 1041 1051, Jul. 1992, J. Pang et al., Signal Recovery From Subband Energy Constraints.
28 Optical Society of America, vol. 9, No. 7, pp. 1041-1051, Jul. 1992, J. Pang et al., "Signal Recovery From Subband Energy Constraints."
29 * Proceedings of the IEEE, vol. 69, No. 4, pp. 432 450, Apr. 1981, R. Schafer et al., Constrained Iterative Restoration Algorithms.
30 Proceedings of the IEEE, vol. 69, No. 4, pp. 432-450, Apr. 1981, R. Schafer et al., "Constrained Iterative Restoration Algorithms."
31 S. H. Lee Et Al., "Transmission Error Detection, Resynchronization, and Error Concealment for MPEG Video Decoder," Processing of the SPIE, vol. 2094, Nov. 8, 1993, pp. 195-204.
32 * S. H. Lee Et Al., Transmission Error Detection, Resynchronization, and Error Concealment for MPEG Video Decoder, Processing of the SPIE, vol. 2094, Nov. 8, 1993, pp. 195 204.
33 S. Inoue, "An Object Extraction Method for Image Synthesis," Proceedings of SPIE, The International Society of Optical Engineering, Conference Date Nov. 1991, vol. J74-D-11, No. 10, pp. 1411-1418.
34 * S. Inoue, An Object Extraction Method for Image Synthesis, Proceedings of SPIE, The International Society of Optical Engineering, Conference Date Nov. 1991, vol. J74 D 11, No. 10, pp. 1411 1418.
35 * U.S. application No. 08/561,377, Mitsunaga et al., filed Nov. 21, 1995.
36 * U.S. application No. 08/733,953, Hirani et al., filed Oct. 21, 1996.
37 * U.S. application No. 08/899,349, Hirani et al., filed Jul. 23, 1997.
US6285712 * Dec 28, 1998 Sep 4, 2001 Sony Corporation Image processing apparatus, image processing method, and providing medium therefor
US6697061 * Jan 21, 1999 Feb 24, 2004 Hewlett-Packard Development Company, L.P. Image compression featuring selective re-use of prior compression data
US6965647 * Nov 28, 2000 Nov 15, 2005 Sony Corporation Robust time domain block decoding
US7051262 Jan 9, 2002 May 23, 2006 Huawei Technologies Co., Ltd. Method for processing error code of compressed image in transmission
US7221710 Apr 2, 2004 May 22, 2007 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US7613351 * Oct 13, 2004 Nov 3, 2009 Broadcom Corporation Video decoder with deblocker within decoding loop
US7742529 * Nov 27, 2000 Jun 22, 2010 British Telecommunications Plc Video coding and decoding
US7773677 Jan 5, 2005 Aug 10, 2010 Lg Electronics Inc. Image block error concealing apparatus and method using weight filtering in mobile communication system
US8428141 Mar 6, 2007 Apr 23, 2013 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8532193 May 7, 2013 Sep 10, 2013 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8565316 Feb 5, 2013 Oct 22, 2013 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8571109 Feb 5, 2013 Oct 29, 2013 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8571110 Feb 5, 2013 Oct 29, 2013 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8625675 Feb 5, 2013 Jan 7, 2014 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8731063 Sep 4, 2013 May 20, 2014 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8743966 Sep 6, 2013 Jun 3, 2014 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8902990 Sep 4, 2013 Dec 2, 2014 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US8953691 Sep 6, 2013 Feb 10, 2015 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US20040013200 * May 12, 2003 Jan 22, 2004 Samsung Electronics Co., Ltd. Of Republic Of Korea Advanced method of coding and decoding motion vector and apparatus therefor
US20040059983 * Jan 9, 2002 Mar 25, 2004 Jing Wang Method for processing error code of compressed image in transmission
US20040153937 * Nov 7, 2003 Aug 5, 2004 Lg Electronics Inc. Video error compensating method and apparatus therefor
US20050190843 * Jan 5, 2005 Sep 1, 2005 Lg Electronics Inc. Image block error concealing apparatus and method in mobile communication system
US20050259879 * Oct 13, 2004 Nov 24, 2005 Timothy Hellman Video decoder with deblocker within decoding loop
US20060012719 * Jul 14, 2004 Jan 19, 2006 Nokia Corporation System and method for motion prediction in scalable video coding
US20070147513 * Mar 6, 2007 Jun 28, 2007 Lg Electronics Inc. Block error compensating apparatus of image frame and method thereof
US20080260046 * Sep 15, 2006 Oct 23, 2008 Koninklijke Philips Electronics, N.V. Coding/Decoding Method and Apparatus For Improviding Video Error Concealment
US20080285651 * May 17, 2007 Nov 20, 2008 The Hong Kong University Of Science And Technology Spatio-temporal boundary matching algorithm for temporal error concealment
US20140126637 * Jan 18, 2013 May 8, 2014 Nvidia Corporation System and method for decoding a video
U.S. Classification 375/240.27, 375/E07.105, 348/E05.077, 375/E07.281, 375/E07.256, 375/240.16, 375/E07.119
International Classification H04N19/895, H04N5/21
Cooperative Classification H04N19/51, H04N19/895, H04N19/56, H04N5/21
European Classification H04N7/26M2, H04N7/36C, H04N7/26M4I, H04N7/68
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:OZCELIK, TANER;YU, GONG-SAN;GADRE, SHIRISH C.;REEL/FRAME:008771/0231