Source: https://patents.google.com/patent/US9635308B2/en
Timestamp: 2019-04-20 21:46:42
Document Index: 252535790

Matched Legal Cases: ['Application No. 201080024719', 'Application No. 201080024719', 'Application No. 10', 'Application No. 09', 'Application No. 10', 'Application No. 13']

US9635308B2 - Preprocessing of interlaced video with overlapped 3D transforms - Google Patents
Preprocessing of interlaced video with overlapped 3D transforms Download PDF
US9635308B2
US9635308B2 US12/791,941 US79194110A US9635308B2 US 9635308 B2 US9635308 B2 US 9635308B2 US 79194110 A US79194110 A US 79194110A US 9635308 B2 US9635308 B2 US 9635308B2
US12/791,941
US20110298984A1 (en
2010-06-02 Application filed by Cisco Technology Inc filed Critical Cisco Technology Inc
2010-06-02 Priority to US12/791,941 priority Critical patent/US9635308B2/en
2010-06-02 Assigned to CISCO TECHNOLOGY, INC. reassignment CISCO TECHNOLOGY, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SCHOENBLUM, JOEL W.
2011-12-08 Publication of US20110298984A1 publication Critical patent/US20110298984A1/en
2017-04-25 Publication of US9635308B2 publication Critical patent/US9635308B2/en
In one method embodiment, partitioning a block matched reference frame into plural n×m non-overlapping pixel superblocks, where n and m are non-negative integer numbers; designating each of the n×m pixel superblocks as field or frame; and field processing by overlapped block processing logic two n×n blocks of an n×m overlapped superblock if one of first plural n×m superblocks intersected by the overlapped superblock has a field designation, otherwise frame processing, by the overlapped block processing logic, the two n×n blocks of the overlapped superblock.
Having described an example environment in which the VDN system 200 may be employed, attention is directed to FIGS. 2A-2C, which comprise schematic diagrams that conceptually illustrate data flows and/or processing implemented by various example embodiments of VDN systems and methods. Progressing from FIG. 2A to FIG. 2B and then to FIG. 2C represents a reduction in processing complexity, and hence like-processing throughout these three figures are denoted with the same numerical reference and an alphabetical or alphanumeric suffix (e.g., a, b, and c, or a-l, etc.) that may change from each figure for a given component or diagram depending on whether there is a change or reduction in complexity to the component or represented system 200. Further, each “F” (e.g., F0, F1, etc.) shown above component 220 a in FIG. 2A (and likewise shown and described in association with other figures) is used to denote frames that have not yet been matched to the reference frame (F4), and each “M” (e.g., M0, M1, etc.) is used to denote frames that have been matched (e.g., to the reference frame). Note that use of the term “component” with respect to FIGS. 2A-2C does not imply that processing is limited to a single electronic component, or that each “component” illustrated in these figures are necessarily separate entities. Instead, the term “component” in these figures graphically illustrates a given process implemented in the VDN system embodiments, and is used instead of “block,” for instance, to avoid confusion in the term, block, when used to describe an image or pixel block.
BREF ⁢ ⁢ 2 ⁢ ( i , j ) = BF_LREF ⁢ ⁢ 2 ⁢ ( 4 ⁢ ⁢ i : 4 ⁢ ⁢ i + 3 , 4 ⁢ ⁢ j : 4 ⁢ ⁢ j + 3 ) , ⁢ where ⁢ ⁢ i = 0 , 1 , … ⁢ ⁢ N hor - 1 4 ⁢ ⁢ and ⁢ ⁢ j = 0 , 1 , … ⁢ ⁢ N ver - 1 4 . Eq . ⁢ ( 4 )
S ⁢ ⁢ A ⁢ ⁢ D ⁢ ⁢ 4 ⁢ ⁢ x ⁢ ⁢ 4 ⁢ ( y , x ) = ∑ u = 0 3 ⁢ ∑ v = 0 3 ⁢  BF_LF ⁢ ⁢ 2 ⁢ ( 4 ⁢ ⁢ i + y + u , 4 ⁢ ⁢ j + x + v ) - BREF ⁢ ⁢ 2 ⁢ ( i + u , j + v ) 
(Eq. (6)), where −24≦x≦24, −12≦y≦12. The values of y and x which minimize the SAD 4×4 function in Eq. (6) define the best matching block in BF_LF2. The offset in pixels from the current BREF2 block in the vertical (y) and horizontal (x) directions defines a motion vector to the best matching block. If a motion vector is denoted by my, then mv.y denotes the motion vector vertical direction, and mv.x denotes the horizontal direction. Note that throughout the present disclosure, reference is made to SAD and SAD computations for distance or difference measures.
The DECBM module 514 omits from a search the zero vector (mv.x=0, mv.y=0) as a candidate motion vector output and any motion vectors within one pixel of the zero vector since, in one embodiment, the zero vector is always input to the next stage processing in addition to the candidate motion vectors from the DECBM module 514. Therefore, if N_BEST_DECBM_MATCHES=3, then the three (3) motion vectors consist of non-zero motion vectors, since any zero motion vectors are omitted from the search. Any zero-motion vector is included as one of the added motion vectors of the output of the DECBM module 514. In some embodiments, zero vectors are not input to the next stage and/or are not omitted from the search. The full pixel BM module 516 receives the set of candidate motion vectors, MV_BEST, and performs a limited, full pixel block matching using the filtered, undecimated frames (BF_LE, BF_LREF). In other words, the full pixel BM module 516 takes as input the motion vectors obtained in the DECBM module-implemented process, in addition to the zero motion vector and the motion vector from a neighboring block as explained above, and chooses a single refined motion vector from the candidate set. In some embodiments, a neighboring motion vector is not included as a candidate.
mvfull(k)·x=2×mv_best(k)·x 0≦k<N_BEST_DECBM_MATCHES,
mvfull(k)·y=2×mv_best(k)·y 0≦k<N_BEST_DECBM_MATCHES (Eq. (8))
S ⁢ ⁢ A ⁢ ⁢ D ⁢ ⁢ 8 ⁢ ⁢ x ⁢ ⁢ 8 ⁢ ( i , j , mvfull ⁡ ( k ) · y + m , mvfull ⁡ ( k ) · x + n ) = ∑ u = 0 7 ⁢ ∑ v = 0 7 ⁢  BF_LF ⁢ ( 8 ⁢ ⁢ i + mvfull ⁡ ( k ) · y + n + u , 8 ⁢ ⁢ j + mvfull ⁡ ( k ) · x + m + v ) - BREF ⁡ ( i + u , j + v )  Eq . ⁢ ( 9 )
[ kf ] = Min k ⁢ { λ * MVDIST ⁡ ( k ) + DISTBIAS ⁢ ⁢ ( k ) + SAD ⁢ ⁢ 8 ⁢ ⁢ x ⁢ ⁢ 8 ⁢ ( i , j , mvrfull ⁡ ( k ) · y , mvrfull ⁡ ( k ) · x ) } Eq . ⁢ ( 10 )
mvf(i,j)·x=mvrfull(kf)·x
S ⁢ ⁢ A ⁢ ⁢ D ⁢ ⁢ 8 ⁢ ⁢ x ⁢ ⁢ 8 ⁢ ( i , j , mvf ⁡ ( i , j ) · y + m , mvf ⁡ ( i , j ) · x + n ) = ∑ u = 0 7 ⁢ ∑ v = 0 7 ⁢  LF ⁡ ( 8 ⁢ ⁢ i + mvf ⁡ ( i , j ) · y + n + u , 8 ⁢ ⁢ j + mvf ⁡ ( i , j ) · x + m + v ) - REF ⁡ ( i + u , j + v )  Eq . ⁢ ( 13 )
mvr ⁡ ( i , j ) · x = mvf ⁡ ( i , j ) · x + mref ⁢ ⁢ mvr ⁡ ( i , j ) · x = mvf ⁡ ( i , j ) · x + mref ⁢ ⁢ mvr ⁡ ( i , j ) · y = mvf ⁡ ( i , j ) · x + nref ⁢ ⁢ mvr ⁡ ( i , j ) · y = mvf ⁡ ( i , j ) · x + nref ⁢ ⁢ where ⁢ ⁢ i = 0 , 1 , … ⁢ ⁢ N hor - 1 8 , and ⁢ ⁢ j = 0 , 1 , … ⁢ ⁢ N ver - 1 8 . Eq . ⁢ ( 14 )
Let I(x,y) denote the luminance intensity for a pixel at position x, y for all non-overlapping superblocks at positions x=0, 16, 32, 48, . . . and y=0, 8, 16, 24, 32 . . . and compute Aframe Afield as follows:
A frame = ∑ x = 0 13 ⁢ ∑ y = 0 7 ⁢  I ⁡ ( x , y ) - I ⁡ ( x + 1 , y )  ( a ) A field = ∑ x = 0 13 ⁢ ∑ y = 0 7 ⁢  I ⁡ ( x , y ) - I ⁡ ( x + 2 , y )  ( b )
Explaining the above in further detail, it is noted that one complication in the transition from field-frame processing to overlapped block processing is that the field-frame map corresponds to field-frame indications for non-overlapping 8×16 superblocks, and as explained further below, overlapped block processing contains many more blocks due to the overlapping. As shown in FIG. 7, an overlapped block 702 (e.g., a superblock) intersects four (4) non-overlapping superblocks in the field-frame map 700. One approach implemented by the field-frame logic 380 of FIG. 3 comprises the following procedure: for each overlapped block 702, if the overlapped block position in the frame intersects any field blocks 704 (e.g., field-designated superblocks, where the field designation is represented by “F1” in FIG. 7) in the field-frame 700, the overlapped block 702 (e.g., the sub-blocks of the overlapped block or superblock 702) is marked for field processing. If the overlapped block position overlaps only frame blocks 706 (e.g., frame-designated superblocks, where the frame designation is represented by “FR” in FIG. 7) in the field-frame map 700, then the block 702 (e.g., the sub-blocks of the overlapped block or superblock 702) is marked for frame processing. For progressive video input, the field-frame map 700 is set to all ones (1s) indicating frame processing only.
denotes element by element multiplication.
One example of scaling factors that may be used is given below:
SubbandSAD ⁢ ⁢ ( k ) = ∑ z = 0 9 ⁢  bs ⁡ ( z , k ) - bs ⁡ ( z , 1 )  , Eq . ⁢ ( 20 )
SubbandCountFWD ⁢ ⁢ 4 = ∑ k = 1 3 ⁢ SetToOneOrZero ( SubbandSAD ⁢ ⁢ ( k ) < Tsubbandsad ⁢ ) Eq . ⁢ ( 21 ⁢ a ) SubbandCountBAK ⁢ ⁢ 4 = ∑ k = 0 1 ⁢ SetToOneOrZero ( SubbandSAD ⁢ ⁢ ( k ) < Tsubbandsad ⁢ ) Eq . ⁢ ( 21 ⁢ b ) SubbandCount ⁢ ⁢ 8 = ∑ k = 0 3 ⁢ SetToOneOrZero ( SubbandSAD ⁢ ⁢ ( k ) < Tsubbandsad ⁢ ) Eq . ⁢ ( 21 ⁢ c )
bst ⁡ ( z , t ) = { 0 if ⁢ ⁢  bs ⁡ ( z , 1 )  < T_ ⁢ 2 ⁢ D ⁡ ( j ) ⁢ ⁢ and S_ ⁢ 2 ⁢ D ⁡ ( j , 0 ) ≤ z ≤ S_ ⁢ 2 ⁢ D ⁡ ( j , 1 ) bs ⁡ ( z , 1 ) otherwise Eq . ⁢ ( 22 )
If TemporalMode==FWD4 threshold [L20, H02, H11] Haar Subbands;
providing a field-frame map by partitioning a block matched frame into plural non-overlapping superblocks, wherein the block matched frame is determined by frame matching, and wherein the frame matching comprises:
collapsing a first number of earliest matched frames of the matched frames into a first summations frame;
collapsing a second number of matched frames comprising one or more matched frames from a second subsequent matched frame after the first number of earliest matched frames to a next to last matched frame into a second summations frame; and
providing only the first summations frame, the second summations frame, a matched frame immediately subsequent after the number of earliest matched frames, and a last matched frame to a 2D+1 accumulation buffer;
designating each of the plural non-overlapping superblocks of the field-frame map as either field or frame, wherein designating each of the plural non-overlapping superblocks comprises:
determining a frame intensity for each of the plural non-overlapping superblocks by summing plural luminance intensity values for pixels at first positions on each of the plural non-overlapping superblocks,
determining a field intensity for each of the plural non-overlapping superblocks by summing luminance intensity for pixels at second positions on each of the plural non-overlapping superblocks, the second positions being successive to respective first positions;
determining a difference in the determined field intensity and frame intensity,
designating a non-overlapping superblock as a field block when the determined difference is more than a user defined operational parameter, and
designating a non-overlapping superblock as a frame block when the determined difference is less than the user defined operational parameter;
comparing an overlapped superblock to first plural non-overlapping superblocks of the field-frame map;
field processing by overlapped block processing logic two overlapped blocks of the overlapped superblock when one of the plural non-overlapping superblocks intersected by the overlapped superblock has a field designation; and
frame processing, by the overlapped block processing logic, the two overlapped blocks of the overlapped superblock when one of the plural non-overlapping superblocks intersected by the overlapped superblock has a frame designation.
2. The method of claim 1, wherein the plural non-overlapping superblocks comprises pixel block, and wherein the pixel blocks comprise luma pixel blocks.
3. The method of claim 1, wherein the field processing comprises simultaneously processing the two overlapped blocks configured as two vertically interleaved blocks of the overlapped superblock for luma.
4. The method of claim 1, wherein the frame processing comprises processing the two overlapped blocks configured as two vertically adjacent top and bottom frame blocks of the overlapped superblock for luma.
5. The method of claim 1, wherein the plural non-overlapping superblocks of the field-frame map are each overlapped at least in part by the overlapped superblock.
6. The method of claim 1, wherein comparing comprises receiving the field-frame map and a coordinate position of the overlapped superblock.
7. The method of claim 1, further comprising chroma overlap block processing, wherein the chroma overlap block processing is based on two n×n fields spanning 2×n pixels vertically, where n is a non-negative integer number.
8. The method of claim 1, wherein designating a non-overlapping superblock as the frame block comprises designating each of the plural non-overlapping superblocks as frames for a progressive video input.
9. The method of claim 1, wherein intermediate denoised pixel blocks output from the block processing logic are signaled to be accumulated according to the field or frame designations.
10. A non-transitory computer-readable medium, comprising:
block matching logic configured to provide a block matched frame, wherein the block matched frame is determined by frame matching, and wherein the frame matching comprises:
collapsing a second number of matched frames comprising the matched frames from a second subsequent matched frame after the first number of earliest matched frames to a next to last matched frame into a second summations frame; and
field-frame mapping logic configured to provide a field-frame map by partitioning the frame into plural non-overlapping superblocks, the field-frame mapping logic further configured to designate each of the plural non-overlapping superblocks as either field or frame, wherein the field-frame mapping logic configured to designate comprises the field-frame mapping logic further configured to:
determine a frame intensity for each of the non-overlapping superblocks by summing plural luminance intensity values for pixels at first positions on each of the non-overlapping superblocks,
determine a field intensity for each of the non-overlapping superblocks by summing luminance intensity for pixels at second positions on each of the non-overlapping superblocks, the second positions being successive to respective first positions;
determine a difference in the determined field intensity and frame intensity,
designate a non-overlapping superblock as a field block when the determined difference is more than a user defined operational parameter, and
designate a non-overlapping superblock as a frame block when the determined difference is less than the user defined operational parameter;
field-frame logic configured to compare an overlapped superblock to first plural non-overlapping superblocks of the field-frame map; and
overlapped block processing logic configured to:
field process two blocks of the overlapped superblock when one of the plural non-overlapping superblocks intersected by the overlapped superblock has a field designation; and
frame process the two blocks of the overlapped superblock when one of the plural non-overlapping superblocks intersected by the overlapped superblock has a frame designation.
11. The non-transitory computer-readable medium of claim 10, wherein the overlapped block processing logic is configured to field process by simultaneously processing two vertically interleaved blocks of the overlapped superblock for luma.
12. The non-transitory computer-readable medium of claim 10, wherein the overlapped block processing logic is configured to frame process by processing two vertically adjacent top and bottom frame blocks of the overlapped superblock for luma.
13. The non-transitory computer-readable medium of claim 10, wherein the field-frame logic is configured to receive the field-frame map and a coordinate position of the overlapped superblock.
14. The non-transitory computer-readable medium of claim 10, wherein the plural non-overlapping superblocks of the field-frame map are each overlapped at least in part by the overlapped superblock.
15. The non-transitory computer-readable medium of claim 10, further comprising chroma overlap block processing logic, wherein the chroma overlap block processing logic is configured to implement chroma overlap block processing based on two n×n fields spanning 2×n pixels vertically, where n is a nonnegative integer number.
partitioning a block matched frame into plural n×m non-overlapping pixel superblocks, where n and m are non-negative integer numbers, wherein the block matched frame is determined by frame matching, and wherein the frame matching comprises:
providing only the first summations frame, the second summations frame, a matched frame immediately subsequent after the first number of earliest matched frames, and a last matched frame to a 2D+1 accumulation buffer;
designating each of the plural n×m non-overlapping pixel blocks as field or frame, wherein designating each of the plural n×m non-overlapping superblocks comprises:
determining a frame intensity for each of the plural n×m non-overlapping superblocks by summing plural luminance intensity values for pixels at first positions on each of the plural n×m non-overlapping superblocks,
determining a field intensity for each of the plural n×m non-overlapping superblocks by summing luminance intensity for pixels at second positions on each of the plural n×m non-overlapping superblocks, the second positions being successive to respective first positions;
designating a non-overlapping superblock as a frame block when the determined difference is less than the user defined operational parameter; and
field processing, by overlapped block processing logic, two n×n blocks of an n×m overlapped superblock when one of first plural n×m non-overlapping superblocks intersected by the overlapped block has a field designation; and
frame processing, by the overlapped block processing logic, the two n×n blocks of the overlapped superblock when one of the first plural n×m non-overlapping superblocks intersected by the overlapped block has a field designation.
17. The method of claim 16, wherein the field processing comprises simultaneously processing the two overlapped blocks configured as two vertically interleaved blocks of the overlapped superblock for luma.
18. The method of claim 16, wherein the frame processing comprises processing the two overlapped blocks configured as two vertically adjacent top and bottom frame blocks of the overlapped superblock for luma.
US12/791,941 2010-06-02 2010-06-02 Preprocessing of interlaced video with overlapped 3D transforms Active 2032-07-31 US9635308B2 (en)
US12/791,941 US9635308B2 (en) 2010-06-02 2010-06-02 Preprocessing of interlaced video with overlapped 3D transforms
US20110298984A1 US20110298984A1 (en) 2011-12-08
US9635308B2 true US9635308B2 (en) 2017-04-25
ID=45064206
US12/791,941 Active 2032-07-31 US9635308B2 (en) 2010-06-02 2010-06-02 Preprocessing of interlaced video with overlapped 3D transforms
US (1) US9635308B2 (en)
2010-06-02 US US12/791,941 patent/US9635308B2/en active Active
US20160119508A1 (en) 2009-06-05 2016-04-28 Cisco Technology, Inc. Processing Prior Temporally-Matched Frames in 3D-Based Video Denoising
ALEKSANDRA PIZURICA, VLADIMIR ZLOKOLICA AND WILFRIED PHILIPS: "Noise Reduction in Video Sequences Using Wavelet-Domainand Temporal Filtering", PROCEEDINGS OF SPIE, SPIE, US, vol. 5266, 27 February 2004 (2004-02-27), US, pages 48 - 59, XP002596108, ISSN: 0277-786x
Chinese First Office Action dated Jan. 30, 2014 cited in Application No. 201080024719.9, 16 pgs.
Chinese Second Office Action dated Sep. 22, 2014 cited in Application No. 201080024719.9, 18 pgs.
European Office Action dated Feb. 6, 2015 cited in Application No. 10 730 897.5, 5 pgs.
European Office Action dated Jun. 30, 2016 cited in Application 09 770 924.01, 4 pgs.
European Office Action dated Jun. 30, 2016 cited in Application 13 189 789.4, 6 pgs.
European Office Action mailed Feb. 19, 2014 in European Application No. 09 770 924.01, 6 pgs.
European Summons to Attend Oral Proceedings dated Dec. 1, 2016 cited in Application No. 10 730 897.5, 8 pgs.
Extended European Search Report dated Feb. 27, 2014 cited in Application No. 13 189 789.4, 12 pgs.
KOSTADIN DABOV, ALESSANDRO FOI, AND KAREN EGIAZARIAN: "VIDEO DENOISING BY SPARSE 3D TRANSFORM-DOMAINCOLLABORATIVE FILTERING", PROCEEDINGS / EUSIPCO 2007, 15TH EUROPEAN SIGNAL PROCESSING CONFERENCE : SEPTEMBER 3 - 7, 2007, POZNAÑ, POLAND, POZNAN, POLAND, 3 September 2007 (2007-09-03) - 7 September 2007 (2007-09-07), Poznan, Poland, pages 1 - 5, XP002596106
MAGAREY J, KOKARAM A, KINGSBURY N: "Robust Motion Estimation Using Chrominance Information in Colour Image Sequences", PROCEEDINGS OF ICIAP 97. 9TH INTERNATIONAL CONFERENCE ON IMAGE ANALYSIS AND PROCESSING - 17-19 SEPT. 1997 - FLORENCE, ITALY, SPRINGER-VERLAG, BERLIN, GERMANY, vol. 1310, 17 September 1997 (1997-09-17) - 19 September 1997 (1997-09-19), Berlin, Germany, pages 486 - 493, XP002596107, ISBN: 354-0-635-07648-6
Notice of Allowance and Issue Fee mailed Mar. 10, 2014 in U.S. Appl. No. 13/646,795, 8 pgs.
Notice of Allowance mailed Jan. 15, 2016 in U.S. Appl. No. 13/926,703, 12 pgs.
Rusanovskyy et al. ("Video denoising algorithm in sliding 3D DCT domain", in Advanced Concepts for Intelligent Vision Systems, ACIVS, (Antwerp, Belgium), Sep. 2005). *
U.S. Appl. No. 12/146,369, filed Jun. 25, 2008, entitled "Combined Deblocking and Denoising Filter", Inventor: Joe Schoenblum.
U.S. Appl. No. 14/154,502, filed Jan. 14, 2014, entitled "Summating Temporally-Matched Frames in 3D-Based Video Denoising", Inventors: Joel W. Schoenblum et al.
U.S. Appl. No. 14/992,234, filed Jan. 11, 2016, entitled "Processing Prior Temporally-Matched Frames in 3D-Based Video Denoising", Inventor: Joel Schoenblum.
U.S. Final Office Action dated Sep. 9, 2016 cited in U.S. Appl. No. 12/791,987, 40 pgs.
U.S. Office Action mailed Oct. 24, 2013 cited in U.S. Appl. No. 13/646,795, 5 pgs.
US20110298984A1 (en) 2011-12-08
EP2271116B1 (en) 2013-09-11 Image decoding method and image decoder
US8508597B2 (en) 2013-08-13 Method and system for video quality assessment
EP1980109B1 (en) 2018-07-11 Methods and apparatus for edge-based spatio-temporal filtering
EP1788815A2 (en) 2007-05-23 Rate control for video transcoding with resolution conversion
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SCHOENBLUM, JOEL W.;REEL/FRAME:024470/0675