Source: http://www.google.co.uk/patents/US9628674
Timestamp: 2018-01-21 07:03:06
Document Index: 203556899

Matched Legal Cases: ['art(0', 'Application No. 201080024719', 'Application No. 201080024719', 'Application No. 10730897', 'Application No. 10', 'Application No. 09', 'Application No. 10', 'Application No. 13']

Patent US9628674 - Staggered motion compensation for preprocessing video with overlapped 3D ... - Google Patents
In one method embodiment, receiving at a frame matching module a first frame comprising first plural blocks and plural frames that each comprise a plurality of blocks to be matched to the first plural blocks of the first frame, the first plural blocks and the plurality of blocks comprising luma blocks;...http://www.google.co.uk/patents/US9628674?utm_source=gb-gplus-sharePatent US9628674 - Staggered motion compensation for preprocessing video with overlapped 3D transforms
Publication number US9628674 B2
Application number US 12/791,987
Also published as US20110298986
Publication number 12791987, 791987, US 9628674 B2, US 9628674B2, US-B2-9628674, US9628674 B2, US9628674B2
Inventors Joel W. Schoenblum
Patent Citations (86), Non-Patent Citations (77), Classifications (7), Legal Events (1)
Staggered motion compensation for preprocessing video with overlapped 3D transforms
US 9628674 B2
In one method embodiment, receiving at a frame matching module a first frame comprising first plural blocks and plural frames that each comprise a plurality of blocks to be matched to the first plural blocks of the first frame, the first plural blocks and the plurality of blocks comprising luma blocks; for each of the frame pair matchings, selecting one border configuration among a plurality of border configurations, the border configuration selected for the each of the frame pair matchings unique; appending a border of pixels to the frames of the each of the frame pair matchings based on the selected border configuration; and block matching the first plural blocks with the plurality of blocks.
receiving at a frame matching module a first frame comprising first plural blocks and plural frames that each comprise a plurality of blocks to be matched to the first plural blocks of the first frame, the first plural blocks and the plurality of blocks comprising luma blocks and chroma blocks;
for each of the frame pair matchings, selecting one border configuration among a plurality of border configurations, the border configuration selected for the each of the frame pair matchings unique, the plurality of border configurations received from a staggered motion compensated (SMC) pre-processing sub-system in a border configuration signal defining quantities of border pixels for the luma blocks, wherein selecting the one border configuration comprises selecting a varying border configuration of pixels around the first frame for each of the frame pair matchings, wherein selecting the varying border configuration comprises selecting a varying coordinate positions for seams among the first plural blocks, wherein selecting the varying coordinate positions for the seams comprises selecting the varying coordinate positions to evenly spread out the seams, wherein selecting the varying border configurations further comprises:
selecting a different border configuration between different pairing of a reference frame and another frame, wherein selecting the different pairing comprises:
selecting a first border configuration for a first pairing of the reference frame and the first frame, and
selecting a second border configuration different from the first border configuration for a second pairing of the reference frame and a second frame, and
selecting a different border configuration per fields of the reference frame,
wherein the SMC pre-processing sub-system positions the seams for the luma blocks in different places each time a frame match is performed, and wherein the quantities of the border configuration for the chroma blocks are determined as downsampled by two of the quantities for the luma blocks;
appending a border of pixels to the frames of the each of the frame pair matchings based on the selected border configuration; and
block matching the first plural blocks with the plurality of blocks.
2. The method of claim 1, further comprising removing the appended border of pixels at a time corresponding to completion of the block matching.
3. The method of claim 1, wherein the plurality of border configurations differ from one another in number of pixels for at least one of the borders.
4. The method of claim 1, further comprising providing block matched frames to a 3D video denoising module, wherein the effect of the varying coordinate positions for the seams among the first plural blocks results an amelioration of seam artifacts in corresponding observed video.
5. The method of claim 1, wherein the plurality of border configurations each comprises an even quantity of total border pixels.
6. The method of claim 1, further comprising receiving chroma versions of the first and the plural frames associated with the each of the frame pair matchings and appending to the chroma versions of the frames a downsampled version of the unique configuration of border pixels according to the selected border configuration for the each of the frame pair matchings.
7. The method of claim 1, wherein for interlaced video, each field of the first frame comprises a different border configuration.
8. The method of claim 1, wherein the appended border of pixels comprise black pixels.
a frame matching module comprising border add logic, the border add logic configured to:
receive a first frame comprising first plural blocks and plural frames that each comprise a plurality of blocks to be matched to the first plural blocks of the first frame, the first plural blocks and the plurality of blocks comprising luma blocks and chroma blocks;
for each of the frame pair matchings, select one border configuration among a plurality of border configurations, the border configuration selected for the each of the frame pair matchings unique, the plurality of border configurations received in a border configuration signal received from a staggered motion compensated (SMC) pre-processing sub-system defining quantities of border pixels for the luma blocks, wherein a varying border configuration of pixels is selected around the first frame for each of the frame pair matchings, wherein a varying coordinate positions is selected for seams among the first plural blocks, wherein the varying coordinate positions are selected to evenly spread out the seams, wherein the border add logic being configured to select one border configuration further comprises the border add logic module being configured to:
select a different border configuration between different pairing of a reference frame and another frame, wherein the border add logic being configured to select the different border configuration further comprises the border add logic module being configured to:
select a first border configuration for a first pairing of the reference frame and the first frame, and
select a second border configuration different from the first border configuration for a second pairing of the reference frame and a second frame, and
select a different border configuration per fields of the reference frame, wherein the SMC pre-processing sub-system positions the seams for the luma blocks in different places each time a frame match is performed, and wherein the quantities of the border configuration for the chroma blocks are determined as downsampled by two of the quantities for the luma blocks;
append a border of pixels to the frames of the each of the frame pair matchings based on the selected border configuration; and
motion estimation (ME) and motion compensation (MC) logic of the frame matching module, the ME and MC logic configured to: block match the first plural blocks with the plurality of blocks.
10. The system of claim 9, further comprising border extract logic configured to remove the appended border of pixels at a time corresponding to completion of the block matching.
11. The system of claim 9, wherein the plurality of border configurations differ from one another in number of pixels for at least one of the borders.
12. The system of claim 9, further comprising a 3D video denoising module configured to receive the block matched frames post-pixel border removal, the 3D video denoising module configured to provide for output the denoised first frame with the varying coordinate positions for the seams among the first plural blocks resulting in an amelioration of seam artifacts in corresponding observed video.
13. The system of claim 9, wherein the plurality of border configurations each comprises an even quantity of total border pixels.
14. The system of claim 9, further comprising second border add logic corresponding to chroma processing, the second border add logic configured to receive chroma versions of the first and the plural frames associated with the each of the frame pair matchings and append to the chroma versions of the frames a downsampled version of the unique configuration of border of pixels according to the selected border configuration for the each of the frame pair matchings.
15. The system of claim 9, wherein for interlaced video, each field of the first frame comprises a different border configuration.
16. The system of claim 9, wherein the appended border of pixels comprise black pixels.
17. The method of claim 1, wherein selecting the border configuration comprises:
selecting the border configuration for the luma block; and
applying the border configuration selected for the luma block to the chroma block.
18. The system of claim 9, wherein a tally of two border configurations per frame for each field are received for each frame-pair matching.
Filtering of noise in video sequences is often performed to obtain as close to a noise-free signal as possible. Spatial filtering requires only the current frame (e.g., picture) to be filtered and not surrounding frames in time. Spatial filters, when implemented without temporal filtering, may suffer from blurring of edges and detail. For this reason and the fact that video tends to be more redundant in time than space, temporal filtering is often employed for greater filtering capability with less visual blurring. Since video contains both static scenes and objects moving with time, temporal filters for video include motion compensation from frame to frame for each part of the moving objects to prevent trailing artifacts of the filtering.
FIG. 1 is a block diagram that illustrates an example environment in which certain embodiments of preprocessing sub-systems and methods of video denoising (VDN) systems and methods can be implemented.
FIG. 3 is a block diagram that illustrates one example VDN system embodiment comprising frame alignment and overlapped block processing modules and respectively associated preprocessing sub-systems.
FIG. 4 is a block diagram that illustrates one example embodiment of a frame alignment module.
FIG. 5 is a block diagram that illustrates one example embodiment of a frame matching module, comprising example preprocessing sub-systems, of a frame alignment module.
FIG. 6A is a block diagram that illustrates example luma border pixel configurations applied by an embodiment of a staggered motion compensation sub-system residing in an example frame matching module.
FIG. 6B is a block diagram that illustrates an example constellation of a border configuration resulting from staggered motion compensation processing.
FIG. 7 is a block diagram that illustrates an overlapped superblock applied to an example field-frame map as implemented by a field-frame processing sub-system.
FIGS. 8A-8D are block diagrams that illustrate a modified one-dimensional (1D) transform used in an overlapped block processing module, the 1D transform illustrated with progressively reduced complexity.
FIG. 9 is a block diagram that conceptually illustrates the use of temporal modes in an overlapped block processing module.
FIG. 10 is a block diagram that illustrates an example mechanism for thresholding.
FIG. 11 is a block diagram that illustrates an example scene change detection mask utilized for detecting scene changes between matched frame pairs.
FIG. 12A is a flow diagram that illustrates an example method embodiment implemented by an embodiment of a field-frame processing sub-system in conjunction with an overlapped block processing module.
FIG. 12B is a flow diagram that illustrates another example method embodiment implemented by an embodiment of a field-frame processing sub-system in conjunction with an overlapped block processing module.
FIG. 13 is a flow diagram that illustrates an example method embodiment implemented by an embodiment of a scene change detection sub-system.
FIG. 14 is a flow diagram that illustrates an example method embodiment implemented by an embodiment of a staggered motion compensation sub-system.
Disclosed herein are various example embodiments of preprocessing sub-systems and methods of a video denoising (VDN) system. One embodiment of a preprocessing sub-system, referred to herein as a field-frame processing (FFP) sub-system, comprises logic that simultaneously processes two interleaved n×n (e.g., 8×8) pixel sub-blocks of an n×m (e.g., 8×16) pixel superblock, where n and m are non-negative integer numbers. For a frame, a superblock is referred to herein as two vertically adjacent blocks (top/bottom), and for a field, a superblock is referred to herein as two interleaved blocks having a vertical span of two adjacent blocks. For purposes of brevity and for illustration only, each example luma superblock is described as having an 8×16 size, with the understanding that superblocks of other sizes (e.g., 16×32, etc.) as well as different-sized sub-blocks (i.e., the two blocks that make up a superblock) of a superblock are contemplated to be within the scope of the disclosure. The simultaneous, coupled processing for interlaced video as interleaved fields (e.g., single channel processing) as disclosed herein is in contrast to decoupled processing as two separate fields (e.g., splitting the two fields of frame pictures into separate channels), hence resulting in some implementations in reduced complexity. In one embodiment, the FFP sub-system provides (e.g., once for a reference frame) a field-frame map by logically partitioning a block matched frame into plural non-overlapping superblocks, the block matched frame destined to be denoised via an overlapped block process. For each superblock in the field-frame map, the FFP sub-system provides a respective frame or field designation. Based on the field-frame map and the coordinate position of an overlapped block (e.g., overlapped superblock) that is subject to comparison with the superblocks of the field-frame map, the FFP sub-system further determines whether the sub-blocks of the overlapped block at the particular coordinate position are to each undergo overlapped block processing (e.g., video denoising) as a frame or field.
Another embodiment of a preprocessing sub-system, referred to herein as a scene change detection (SCD) sub-system, comprises logic that provides a mechanism to detect scene changes and that further cooperates with mode decision logic that handles the detected scene change in one or more temporal depth modes. In one embodiment, the SCD sub-system is implemented during a block matching stage involving iterative respective sets of noise-filtered blocks from two full-size (e.g., not decimated) frames. As the iteration of block matching between the sets progresses, a decision is made, block-by-block, as to the closeness of the match until it is determined that a scene change detection has been triggered. In one embodiment, a bit mask is used to track the location of a detected scene change among a window of frames to facilitate implementation of a suitable temporal mode.
Yet another embodiment of a preprocessing sub-system, referred to herein as a staggered motion compensation (SMC) sub-system, comprises logic that applies a varying border configuration of pixels around a reference frame, the variation occurring between different pairings of the reference frame with another frame (and further difference in border configurations per fields of the same reference frame) in an intended block matching operation. In other words, each time a frame is matched to the reference frame, the same border configuration is used between the reference frame and the other frame, yet a different border is used for each pairing of the reference frame with another frame (and then removed at the completion of frame matching). The SMC sub-system effectively positions the seams of the n×m blocks for luma in different places each time a frame match is performed, eliminating or mitigating from the denoised frames blocking or other artifacts resulting from aggregated seams.
As indicated above, the various preprocessing sub-system embodiments are described in the context of a VDN system. In one embodiment, the VDN system comprises a frame alignment module and an overlapped block processing module, the overlapped block processing module configured to denoise video in a three-dimensional (3D) transform domain using motion compensated overlapped 3D transforms. In particular, certain embodiments of VDN systems motion compensate a set of frames surrounding a current frame, and denoise the frame using 3D spatio-temporal transforms with thresholding of the 2D and/or 3D transform coefficients. One or more VDN system embodiments provide several advantages or distinctive features over brute force methods of conventional systems, including significantly reduced computational complexity that enable implementation in real-time silicon (e.g., applicable to real-time applications, such as pre-processing of frames for real-time broadcasting of encoded pictures of a video stream), such as non-programmable or programmable hardware including field programmable gate arrays (FPGAs), and/or other such computing devices. Several additional distinctive features and/or advantages of VDN system embodiments, explained further below, include the decoupling of block matching and inverse block matching from an overlapped block processing loop, reduction of accumulation buffers from 3D to 2D+n (where n is an integer number of accumulated frames less than the number of frames in a 3D buffer), and the “collapsing” of frames (e.g., taking advantage of the fact that neighboring frames have been previously frame matched to reduce the amount of frames entering the overlapped block processing loop while obtaining the benefit of the information from the full scope of frames from which the reduction occurred for purposes of denoising). Such features and/or advantages enable substantially reduced complexity block matching. Further distinctive features include, among others, a customized-temporal transform and temporal depth mode selection, also explained further below.
These advantages and/or features, among others, are described hereinafter in the context of an example subscriber television network environment, with the understanding that other video environments may also benefit from certain embodiments of the preprocessing sub-systems and VDN systems and methods and hence are contemplated to be within the scope of the disclosure. It should be understood by one having ordinary skill in the art that, though specifics for one or more embodiments are disclosed herein, such specifics as described are not necessarily part of every embodiment.
FIG. 1 is a block diagram of an example environment, a subscriber television network 100, in which certain embodiments of preprocessing sub-systems of a VDN system may be implemented. The subscriber television network 100 may include a plurality of individual networks, such as a wireless network and/or a wired network, including wide-area networks (WANs), local area networks (LANs), among others. The subscriber television network 100 includes a headend 110 that receives (and/or generates) video content, audio content, and/or other content (e.g., data) sourced at least in part from one or more service providers, processes and/or stores the content, and delivers the content over a communication medium 116 to one or more client devices 118 through 120. The headend 110 comprises an encoder 114 having video compression functionality, and a pre-processor or VDN system 200 configured to receive a raw video sequence (e.g., uncompressed video frames or pictures), at least a portion of which (or the entirety) is corrupted by noise. Such noise may be introduced via camera sensors, from previously encoded frames (e.g., artifacts introduced by a prior encoding process from which the raw video was borne, among other sources). The VDN system 200 is configured to denoise each picture or frame of the video sequence and provide the denoised pictures or frames to the encoder 114, enabling, among other benefits, the encoder to encode fewer bits than if noisy frames were inputted to the encoder. In some embodiments, at least a portion of the raw video sequence may bypass the VDN system 200 and be fed directly into the encoder 114. The VDN system 200 further comprises a preprocessing sub-system 300, the pre-processing sub-system including, among other components or sub-systems described below, one or a combination of a field-frame processing (FFP) sub-system, scene change detection (SCD) sub-system, and/or staged motion compensation (SMC) sub-system as explained further below.
The VDN system 200 (and associated sub-systems, such as the preprocessing sub-system 300) may be implemented in hardware, software, firmware, or a combination thereof (collectively or individually also referred to herein as logic). To the extent certain embodiments of the VDN system 200 or a portion thereof are implemented in software or firmware, executable instructions or code for performing one or more tasks of the VDN system 200 are stored in memory or any other suitable computer readable medium and executed by a suitable instruction execution system. In the context of this document, a computer readable medium is an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in connection with a computer related system or method.
Overall, VDN system embodiment, denoted 200 a in FIG. 2A, can be subdivided into frame matching 210 a, overlap block processing 250 a, and post processing 270 a. In frame matching 210 a, entire frames are matched at one time (e.g., a single interval of time or single processing stage), and hence do not need to be matched during implementation of overlapped block processing 250 a. In other words, block matching in the frame matching process 210 a is decoupled (e.g., blocks are matched without block overlapping in the frame matching process) from overlapped block processing 250 a, and hence entire frames are matched and completed for a given video sequence before overlapped block processing 250 a is commenced for the given video sequence. By decoupling frame matching 210 a from overlapped block processing 250 a, block matching is reduced by a factor of sixty-four (64) when overlapped block processing 250 a has a step size of s=1 pixel in both the vertical and horizontal direction (e.g., when compared to integrating overlapped block processing 250 a with the frame matching 210 a). If the step size is s=2, then block matching is reduced by a factor of sixteen (16). One having ordinary skill in the art should understand that various step sizes are contemplated to be within the scope of the disclosure, the selection of which is based on factors such as available computational resources and video processing performance (e.g., based on evaluation of PSNR, etc.).
Shown in component 220 a are eight (8) inputted contiguous frames, F0(t) through F7(t) (denoted above each symbolic frame as F0, F1, etc.). The eight (8) contiguous frames correspond to a received raw video sequence of plural frames. In other words, the eight (8) contiguous frames correspond to a temporal sequence of frames. For instance, the frames of the raw video sequence are arranged in presentation output order (which may be different than the transmission order of the compressed versions of these frames at the output of the headend 110). In some embodiments, different arrangements of frames and/or different applications are contemplated to be within the scope of the disclosure. Note that quantities fewer or greater than eight frames may be used in some embodiments at the inception of processing. Frame matching is symbolized in FIGS. 2A-2C by the arrow head lines, such as represented in component 220 a (e.g., from F0 to F4, etc.).
As illustrated by the arrow head lines, frames F0(t) through F3(t) are matched to F4(t), meaning that blocks (e.g., blocks of pixels or image blocks, such as 8×8, 8×4, etc.) have been selected from those frames which most closely match blocks in F4(t) through a motion estimation/motion compensation process as explained below. The result of frame matching is a set of frames M0(t) through M7(t), where M4(t)=F4(t), as shown in component 230 a. Frames M0(t) through M7(t) are all estimates of F4(t), with M4(t)=F4(t) being a perfect match. M4(t) and F4(t) are used interchangeably herein.
Overlapped block processing 250 a is symbolically represented with components 252 (also referred to herein as a group of matched noisy blocks or similar), 254 (also referred to herein as 3D denoising or similar), 256 (also referred to as a group or set of denoised blocks or similar), and 260 a (also referred to herein as pixel accumulation buffer(s)). Overlapped block processing 250 a moves to a pixel location i,j in each of the matched frames (e.g., the same, co-located, or common pixel location), and for each loop, takes in an 8×8 noisy block b(i,j,t) (e.g., 240 a) from each matched frame M0 through M7, with the top left corner at pixel position i,j, so that b(i,j,t)=Mt(i:i+7, j:j+7). Note that i,j are vertical and horizontal indices which vary over the entire frame, indicating the position in the overlapped processing loop. For instance, for a step size s=1, i,j takes on every pixel position in the frame (excluding boundaries of 7 pixels). For a step size s=2 i,j takes on every other pixel. Further note that 8×8 is used as an example block size, with the understanding that other block sizes may be used in some embodiments of overlapped block processing 250 a. The group of eight (8) noisy 8×8 blocks (252) is also denoted as b(i,j, 0:7). Note that the eight (8) noisy blocks b(i,j, 0:7) (252) are all taken from the same pixel position i,j in the matched frames, since frame alignment (as part of frame matching processing 210 a) is accomplished previously. Frame matching 210 a is decoupled from overlapped block processing 250 a.
For each loop, there are eight (8) denoised blocks (256), but in some embodiments, not all of the blocks of bd(i,j, 0:7) are accumulated to the pixel accumulation buffers 260 a, as symbolically represented by the frames and blocks residing therein in phantom (dashed lines). Rather, the accumulation buffers 260 a comprise what is also referred to herein as 2D+c accumulation buffers 260 a, where c represents an integer value corresponding to the number of buffers for corresponding frames of denoised blocks in addition to the buffer for A4. A 2D accumulation buffer corresponds to only A4 (the reference frame) being accumulated using bd(i,j,4) (e.g., denoised blocks bd(i,j,4) corresponding to frame A4 are accumulated). In this example, another buffer corresponding to c=1 is shown as being accumulated, where the c=1 buffer corresponds to denoised blocks bd(i,j,7) corresponding to frame AM7.
It follows that for an eight (8) frame window, a 2D+7 accumulation buffer equals a 3D accumulation buffer. Further, it is noted that using a 2D+1 accumulation buffer is analogous in the time dimension to using a step size s=4 in the spatial dimension (i.e. the accumulation is decimated in time). Accordingly, c can be varied (e.g., from 0 to a defined integer value) based on the desired visual performance and/or available computational resources. However, in some embodiments, a 3D accumulation buffer comprising denoised pixels from plural overlapped blocks is accumulated for all frames.
Ultimately, after the respective blocks for each accumulated frame have been accumulated from plural iterations of the overlapped block processing 250 a, the denoised and processed frame FD4 is output to the encoder or other processing devices in some embodiments. As explained further below, a time shift is imposed in the sequence of frames corresponding to frame matching 210 a whereby one frame (e.g., F0) is removed and an additional frame (not shown) is added for frame matching 210 a and subsequent denoising according to a second or subsequent temporal frame sequence or temporal sequence (the first temporal sequence associated with the first eight (8) frames (F0-F7) discussed in this example). Accordingly, after one iteration of frame processing (e.g., frame matching 210 a plus repeated iterations or loops of overlapped block processing 250 a), as illustrated in the example of FIG. 2A, FD4(t) is output as a denoised version of F4(t). As indicated above, all of the frames F0(t) through F7(t) shift one frame (also referred to herein as time-shifted) so that F0(t+1)=F1(t), F1(t+1)=F2(t), etc., and a new frame F7(t+1) (not shown) enters the “window” (component 220 a) of frame matching 210 a. Note that in some embodiments, greater numbers of shifts can be implemented to arrive at the next temporal sequence. Further, F0(t) is no longer needed at t+1 so one frame leaves the window (e.g., the quantity of frames outlined in component 220 a). For the 8-frame case, as one non-limiting example, there is a startup delay of eight (8) frames, and since three (3) future frames are needed to denoise F4(t) and F5 through F7, there is a general delay of three (3) frames.
FSUM ( t - 4 ) = ∑ j = 4 7 Mj ( t - 4 ) FSUM ( t - 4 ) = ∑ j = 4 7 Mj ( t - 4 ) . Eq . ( 1 )
Having described conceptually example processing of certain embodiments of VDN systems 200, attention is now directed to FIG. 3, which comprises a block diagram of VDN system embodiment 200 c-1. It should be understood by one having ordinary skill in the art in the context of the present disclosure that the various components shown in FIG. 3 may comprise, in one embodiment, software modules comprising code executed by a processor(s) (e.g., central processing unit or CPU, digital signal processor or DSP, etc.) of an FPGA or other device. In some embodiments, one or more of the components may be configured in hardware, or in some embodiments, the VDN system embodiment 200 may be configured as a mix of hardware and software components. It is noted that the architecture and functionality described hereinafter is based on VDN system embodiment 200 c described in association with FIG. 2C, with the understanding that similar types of architectures and components for VDN system embodiments 200 a and 200 b can be derived by one having ordinary skill in the art based on the teachings of the present disclosure without undue experimentation. VDN system embodiment 200 c-1 comprises a frame alignment module 310 that further comprises a staggered motion compensation (SMC) sub-system 300A and a scene change detection (SCD) sub-system 300B. The VDN system embodiment 200 c-1 further comprises a field-frame processing (FFP) sub-system 300C, the FFP sub-system comprising a field-frame mapping logic 380 and a field-frame logic 390, each of which are further described below. Note that in some embodiments, functionality for the field-frame mapping logic 380 and field-frame logic 390 may be combined, or in some embodiments, further distributed among a greater number of modules and/or combined with functionality of other modules. The VDN system embodiment 200 c-1 further comprises an overlapped block processing module 350 (that in one embodiment comprises, among other logic, the field-frame logic 390 of the field-frame processing sub-system 300C that cooperates with the field-frame mapping logic 380), an accumulation buffer 360, and a normalization module 370 (post-accumulation processing). It is noted that frame processing 250 in FIGS. 2A-2C correspond to the processing implemented by the frame alignment module 310, and overlap block processing 250 corresponds to the processing implemented by the overlapped block processing module 350. In addition, the 2D accumulation buffer 360 and the normalization module 370 implement processing corresponding to the components 260 and 270, respectively, in FIG. 2C. Note that in some embodiments, functionality may be combined into a single component or distributed among more or different modules.
As shown in FIG. 3, the frame alignment module 310 receives plural video frames F4(t), F6(t), and F7(t), where F4(t) is the earliest frame in time, and t is the time index which increments with each frame. The frame FSUM0123(t)=FSUM4567(t−4) represents the first four (4) frames F0(t) through F3(t) (F4(t−4) through F7(t−4)) which have been matched to frame F0(t) (F4(t−4)) previously at time t=t−4. The frame alignment module 310 produces the following frames for processing by the overlapped block processing module 350, the details of which are described below: M4(t), MSUM67(t), M5(t), and MSUM0123(t).
Before proceeding with the description of the overlapped block processing module 350, an example embodiment of the frame alignment module 310 is explained below and illustrated in FIG. 4. One example embodiment of a frame alignment module 310 a, shown in FIG. 4, receives frames F4(t), F6(t), F7(t), and FSUM0123(t). It is noted that M5(t) is the same as M76(t−2), which is the same as M54(t). F7(t) is frame-matched to F6(t) at frame match module 402, producing M76(t). After a delay (404) of two (2) frames, M76(t) becomes M54(t), which is the same as M5(t). Note that blocks labeled “delay” and shown in phantom (dotted lines) are intended to represent delays imposed by a given operation, such as access to memory. F6(t) is summed with M76(t) at summer 406, resulting in FSUM67(t). FSUM67(t) is frame-matched to F4(t) at frame match module 408, producing MSUM67(t). MSUM67(t) is multiplied by two (2) and added to F4(t) and M5(t) at summer 410, producing FSUM4567(t). FSUM4567(t) can be viewed as frames F5(t) through F7(t) all matched to F4(t) and summed together along with F4(t). FSUM4567(t) is delayed (412) by four (4) frames producing FSUM0123(t) (i.e., FSUM4567(t−4)=FSUM0123(t)). FSUM0123(t) is frame-matched to F4(t) at frame match module 414 producing MSUM0123(t). Accordingly, the output of the frame alignment module 310 a comprises the following frames: MSUM0123(t), M4(t), M5(t), and MSUM67(t).
One having ordinary skill in the art should understand in the context of the present disclosure that equivalent frame processing to that illustrated in FIG. 4 may be realized by the imposition of different delays in the process, and hence use of different time sequences of frames in a given temporal sequence as the input. For instance, an extra frame delay corresponding to the derivation of frames F6(t) and F7(t) from an FSUM87(t) may be inserted (e.g., immediately after summer 406), resulting in frames FSUM67(t), enabling all frame-matching operations to work out of memory.
LREF, LMAT,
CREF LF, CF CMAT Description
F4(t) FSUM67(t) MSUM67(t) Frame Match FSUM67(t) to
Normalize FSUM67(t) with
divide by 2 prior to Frame
F4(t) from both F6(t) and
F4(t) FSUM0123(t) MSUM0123(t) Frame Match FSUM0123(t) to
Normalize FSUM(t − 4) with
divide by 4 prior to Frame
MSUM0123(t) is an estimate
of F4(t) from F0(t), F1(t),
F2(t), F3(t) (or equivalently,
F4(t − 4), F5(t − 4), F6(t − 4),
F7(t − 4))
BF_X ( i , j ) = ∑ m = - 1 1 ∑ n = - 1 1 x ( m , n ) G ( i - m , j - n ) , Eq . ( 2 )
G ( i , j ) = 1 16 ( 1 2 1 2 4 2 1 2 1 ) Eq . ( 3 )
BREF 2 ( i , j ) = BF_LREF 2 ( 4 i : 4 i + 3 , 4 j : 4 j + 3 ) , where i = 0 , 1 , … N hor - 1 4 and j = 0 , 1 , … N ver - 1 4 . Eq . ( 4 )
S A D 4 x 4 ( y , x ) = ∑ u = 0 3 ∑ v = 0 3  BF_LF 2 ( 4 i + y + u , 4 j + x + v ) - BREF 2 ( i + u , j + v ) 
(Eq. (6)), where −24≦x≦24, −12≦y≦12. The values of y and x which minimize the SAD 4×4 function in Eq. (6) define the best matching block in BF_LF2. The offset in pixels from the current BREF2 block in the vertical (y) and horizontal (x) directions defines a motion vector to the best matching block. If a motion vector is denoted by mv, then mv.y denotes the motion vector vertical direction, and mv.x denotes the horizontal direction. Note that throughout the present disclosure, reference is made to SAD and SAD computations for distance or difference measures.
It should be understood by one having ordinary skill in the art that other such difference measures, such as sum-squared error (SSE), among others well-known to those having ordinary skill in the art can be used in some embodiments, and hence the example embodiments described herein and/or otherwise contemplated to be within the scope of the disclosure are not limited to SAD-based difference measures.
The DECBM module 514 omits from a search the zero vector (mv.x=0, mv.y=0) as a candidate motion vector output and any motion vectors within one pixel of the zero vector since, in one embodiment, the zero vector is always input to the next stage processing in addition to the candidate motion vectors from the DECBM module 514. Therefore, if N_BEST_DECBM_MATCHES=3, then the three (3) motion vectors consist of non-zero motion vectors, since any zero motion vectors are omitted from the search. Any zero-motion vector is included as one of the added motion vectors of the output of the DECBM module 514. In some embodiments, zero vectors are not input to the next stage and/or are not omitted from the search. The full pixel BM module 516 receives the set of candidate motion vectors, MV_BEST, and performs a limited, full pixel block matching using the filtered, undecimated frames (BF_LF, BF_LREF). In other words, the full pixel BM module 516 takes as input the motion vectors obtained in the DECBM module-implemented process, in addition to the zero motion vector and the motion vector from a neighboring block as explained above, and chooses a single refined motion vector from the candidate set. In some embodiments, a neighboring motion vector is not included as a candidate.
BREF ( i , j ) = BF_LREF ( 8 i : 8 i + 7 , 8 j : 8 j + 7 ) , where i = 0 , 1 , … N hor - 1 8 , and j = 0 , 1 , … N ver - 1 8 . Eq . ( 7 )
mvfull(k).x=2×mv_best(k).x 0≦k≦N_BEST_DECBM_MATCHES,
mvfull(k).y=2×mv_best(k).y 0≦k≦N_BEST_DECBM_MATCHES (Eq. (8))
S A D 8 x 8 ( i , j , mvfull ( k ) · y + m , mvfull ( k ) · x + n ) = ∑ u = 0 7 ∑ v = 0 7  BF_LF ( 8 i + mvfull ( k ) · y + n + u , 8 j + mvfull ( k ) · x + m + v ) - BREF ( i + u , j + v )  Eq . ( 9 )
[ kf ] = Min k { λ * MVDIST ( k ) + DISTBIAS ( k ) + S A D 8 x 8 ( i , j , mvrfull ( k ) · y , mvrfull ( k ) · x ) } Eq . ( 10 )
λ=operational parameter, e.g. λ=4
In other words, the larger the motion vector, the lower the SAD value to justify the larger motion vector as a winning candidate. For instance, winners comprising only marginally lower SAD values likely results in random motion vectors. The 12, 20, 40 values described above forces an increased justification (lower SAD values) for increasingly larger motion vectors. In some embodiments, other values and/or other relative differences between these values may be used (e.g., 1, 2, 3 or 0, 10, 20, etc.). Therefore, the final motion vector result from the full pixel block mode operation of full pixel BM module 516 is given by:
REF ( i , j ) = LREF ( 8 i : 8 i + 7 , 8 j : 8 j + 7 ) , where i = 0 , 1 , … N hor - 1 8 , and j = 0 , 1 , … N ver - 1 8 . Eq . ( 12 )
S A D 8 x 8 ( i , j , mvf ( i , j ) · y + m , mvf ( i , j ) · x + n ) = ∑ u = 0 7 ∑ v = 0 7  LF ( 8 i + mvf ( i , j ) · y + u + n , 8 j + mvf ( i , j ) · x + v + m ) - REF ( i + u , j + v )  Eq . ( 13 )
mvr ( i , j ) · x = mvf ( i , j ) · x + mref mvr ( i , j ) · x = mvf ( i , j ) · x + mref mvr ( i , j ) · y = mvf ( i , j ) · x + nref mvr ( i , j ) · y = mvf ( i , j ) · x + nref where i = 0 , 1 , … N hor - 1 8 , and j = 0 , 1 , … N ver - 1 8 . Eq . ( 14 )
Motion compensation is a well-known method in video processing to produce an estimate of one frame from another (e.g., to “match” one frame to another). MVRL and MVRC are input to motion compensation (MC) processes performed at luma MC module 522 and chroma MC module 524, respectively, which import the blocks indicated by the MVRL and MVRC motion vectors. With reference to the luma MC module 522, after the block matching has been accomplished as described hereinabove, the LF frame is frame-matched to LREF by copying the blocks in LF indicated by the motion vectors MVRL. For each block, if the block has been flagged as a BAD_MC_BLOCK by the full pixel BM module 516, instead of copying a block from the LF frame, the reference block in LREF is copied instead. For chroma operations, the same process is carried out in chroma MC module 508 on 4×4 blocks using MVRCb for Cb and MVRCr for Cr, and hence discussion of the same is omitted for brevity. Note that 8×8 and 4×4 were described above for the various block sizes, yet one having ordinary skill in the art should understand that in some embodiments, other block sizes than those specified above may be used.
Having described the overall operation of an embodiment of the frame match module 402, attention is now directed to the SMC sub-system 300A. In the block matching process, the 8×8 blocks (4×4 for chroma) are in the same pixel positions in the matched frames M0123, M5, and M67. Accordingly, the “seams” of the 8×8 blocks of the matched frames are in the same pixel positions. In circumstances where there exists a high level of filtering, some of these seams may become visible in the denoised frames. Further, every pixel is matched with a group of pixels in its designated block, so there is a bias for that pixel to be grouped with certain pixels. Even without the seams becoming visible, an undesirable blocking effect may become visible.
Certain embodiments of an SMC sub-system 300A implement a mechanism referred to herein as staggered motion compensation (SMC), which enables 8×8 blocks to be positioned in different positions in the M0123, M5 and M67 frames, and further, in the fields of these frames. As shown in example frame match module 402, SMC sub-system 300A comprises border add logic 526 and 528 at the input of the luma ME 502 and chroma ME 504, respectively, and border extract logic 530 and 532 at the output of the luma MC 506 and chroma MC 508, respectively. In general, the border add logic 526 receives the luma LF and LREF and outputs the corresponding frames to be matched with the added pixel borders to the binomial filter 510 and the luma refinement BM 518. The border add logic 528 receives the chroma CF and CREF and outputs the corresponding frames with the added pixels to the chroma refinement BM 520 and the chroma MC 524. The border extract logic 530 residing in one embodiment in the luma MC 506 receives the output from the luma MC 522 and strips away the added pixel border and outputs LMAT (without the border). Similarly, the border extract logic 532 of the chroma MC 508 receives the output from the chroma MC 524, strips away the added pixel border, and outputs CMAT (without the border). Each of the above-described logic (e.g., 526, 528, 530, and 532) of the SMC sub-system 300A receives border configuration (BC) signals from a processor (not shown) in or associated with the VDN system.
Explaining further, to accomplish SMC in one embodiment, a border is placed by the border add logic 526 and 528 around the frames to be matched (LF and LREF, CF and CREF in FIG. 5). The total amount of left border pixels plus the total amount of right border equals, in this example, eight (8) for luma and four (4) for chroma for any of the matching frames (and same for top/bottom border). For example, a border of 2-left, 6-right, 4-top, 4-bottom is one example luma border configuration signaled (e.g., by the BC or border_configuration) to the border add logic 526 of the SMC sub-system 300A. Note that the BC is downsampled by two (2) in the border add logic 528 so that the chroma effectively has the same BC as the luma. The border itself in one embodiment is simply black pixels. One having ordinary skill in the art should understand, in the context of the present disclosure, that other quantities of pixels may be used for the border.
For interlaced video, each field is block matched separately. Therefore, a tally of two (2) different border configurations per frame is contemplated in some embodiments, one for each field. For example, since M0123, M5 and M67 are the result of three (3) frame-matching operations, there may exist a total of six (6) border configurations (e.g., signalled to the SMC logic), putting the seams in six (6) different locations. Some example border configurations for luma are given in example Table 600 shown in FIG. 6A, which shows example pixel border configurations for the top, bottom, left, and right of each field frame to be matched. It should be appreciated, in the context of the present disclosure, that for these or other quantities of frame matching operations, a different quantity of border pixel configurations may be applied.
An example constellation 650 of the six (6) border configurations is shown in FIG. 6B (denoted symbolically by “X”). As shown, the border configurations of Table 600 evenly spread out the seams of the block matching process. It is further noted that the border configurations of Table 600 are all even amounts, since in at least one embodiment, the chroma block matching uses exactly the same border downsampled by two (2) to properly re-use motion vectors. Therefore, for chroma, the border configurations are the same as Table 600 divided by two (2). As an example, M5 for chroma on Field 1 (F1) uses [2,2,2,2]. Also note that the border size for chroma is four (4) pixels total for left-right, and four (4) pixels total for top/bottom. For progressive video, there is only one (1) large field, so only the F1 border configuration is used.
After block matching, the border is stripped away (e.g., removed) by the border extract logic 530 and 532 and has no effect on the rest of the overlapped block processing (e.g., is transparent to the overlapped block processing module 350).
With reference to FIG. 3, having described an example embodiment of the various modules or logic that comprise the frame alignment module 310 for the VDN system embodiment 200 c−1, attention is directed to the overlapped block processing module 350. In general, after frame matching, the overlapped block processing module 350 denoises the overlapped 3D blocks and accumulates the results, as explained in association with FIGS. 2A-2C. In one embodiment, looping occurs by stepping j by a step size s in pixels (e.g. s=2). The overlapped block processing moves horizontally until j==NHor−1, after which j is set to 0 and i is incremented by s. Note that in some embodiments, seven (7) pixels are added around the border of the frame to enable the border pixels to include all blocks. For simplicity, these pixels may be a DC value equal to the touching border pixel.
Before proceeding further on the discussion of the overlapped block processing loop, some VDN system embodiments include preprocessing sub-systems 300 that utilize single channel field processing as indicated above. Accordingly, in one embodiment, it is noted that an output of the frame alignment module 310 is provided to the field-frame processing (FFP) sub-system 300C. In one embodiment, for field-frame processing, the field-frame mapping logic 380 partitions (e.g., logically partitions) a matched frame into plural superblocks, each superblock comprising in one embodiment 8×16 non-overlapping pixel blocks (e.g., 8-pixel horizontal, 16-pixel vertical). An 8×16 non-overlapping pixel block comprises two (2) sub-blocks (e.g., each 8×8, collectively spanning sixteen (16) pixels vertically for luma). For chroma, both dimensions are halved, though in some embodiments, other integer decimations are contemplated for chroma processing. The field-frame mapping logic 380 of the FFP sub-system 300C provides (e.g., computes) a field-frame map (e.g., once) for the entire F4 frame with a binary indication 1=field, 0=frame for each of the non-overlapping 8×16 superblocks from the luma as explained below. The chroma processing uses the luma field-frame map, though in some embodiments, a separate field-frame map for chroma may be provided.
Let I(x,y) denote the luminance intensity for a pixel at position x, y for all non-overlapping superblocks at positions x=0, 16, 32, 48, . . . and y=0, 8, 16, 24, 32 . . . and compute Aframe, Afield as follows:
A frame = ∑ x = 0 13 ∑ y = 0 7  I ( x , y ) - I ( x + 1 , y )  ( a ) A field = ∑ x = 0 13 ∑ y = 0 7  I ( x , y ) - I ( x + 2 , y )  ( b )
Further, consider, Adiff given by:
A diff =A frame −A field (c)
If Adiff>TADIFF, then the 8×16 luma superblock is marked as a field block, otherwise the 8×16 luma superblock is marked as a frame block, where TADIFF is an operational parameter that may be set once at startup (e.g. TADIFF=50), or in some embodiments set more often and/or at different stages. One having ordinary skill in the art should understand, in the context of the present disclosure, that though equations (a) through (c) provide one method for determining a field or frame, other mechanisms known to those having ordinary skill in the art may be employed and hence are contemplated to be within the scope of the disclosure.
The field-frame map is used in the overlapped block processing (e.g., AVC-DCT, Haar, thresholding, Inverse, etc.) as explained in further detail below. In general, for each overlapped block, a field-frame indication is derived from the field-frame map. If the indication is field, then the two vertically interleaved 8×8 field blocks of the superblock are simultaneously processed by the overlapped block processing for luma. In other words, there is a direct coupling of the processing of fields, as opposed to splitting and processing as separate fields. If the indication is frame, then two 8×8 frame blocks (e.g., top/bottom) are processed for luma. As indicated above, for one embodiment, chroma processing uses the field-frame map generated for the F4 frame from luma and performs analogous processing on, in one embodiment, 4×4 fields for frame blocks. Note that at a time corresponding to completion of (e.g., after) the overlapped block processing, the denoised F4 blocks are accumulated in the appropriate position in the accumulation buffer (e.g., either as two vertically interleaved sub-blocks of the superblock or top/bottom adjacent sub-blocks of the superblock).
Explaining the above in further detail, it is noted that one complication in the transition from field-frame processing to overlapped block processing is that the field-frame map corresponds to field-frame indications for non-overlapping 8×16 superblocks, and as explained further below, overlapped block processing contains many more blocks due to the overlapping. As shown in FIG. 7, an overlapped block 702 (e.g., a superblock) intersects four (4) non-overlapping superblocks in the field-frame map 700. One approach implemented by the field-frame logic 380 of FIG. 3 comprises the following procedure: for each overlapped block 702, if the overlapped block position in the frame intersects any field blocks 704 (e.g., field-designated superblocks, where the field designation is represented by “FI” in FIG. 7) in the field-frame 700, the overlapped block 702 (e.g., the sub-blocks of the overlapped block or superblock 702) is marked for field processing. If the overlapped block position overlaps only frame blocks 706 (e.g., frame-designated superblocks, where the frame designation is represented by “FR” in FIG. 7) in the field-frame map 700, then the block 702 (e.g., the sub-blocks of the overlapped block or superblock 702) is marked for frame processing. For progressive video input, the field-frame map 700 is set to all ones (1s) indicating frame processing only.
Further, note that for a step size vertically (same as horizontally) of two (2) in the overlapped block processing, there are 4×4 or sixteen (16) overlapped blocks accumulated for all blocks except the borders. On the bottom border, both the bottom row of 8×8 blocks and the row immediately above have no vertical step, so these blocks (other than left/right borders) have four (4) blocks accumulated instead of sixteen (16). This situation arises because the overlapped processing works on 8×16 blocks, and it is not possible to shift vertically once the processing has reached the bottom sixteen (16) pixels of the frame.
Block matching (motion estimation and compensation) described hereinbefore is unaffected by field-frame processing, since all block matching proceeds on two independent fields (i.e., there is no field-frame in block matching). For progressive inputs, block matching proceeds using one large field per frame instead of two (2) fields (i.e., there is no field processing).
Referring again to FIG. 3, and assuming hereinafter frame processing (since if field processing is assumed, the only significant effect is that there should be a communication to the 2D buffer accumulation as to the presence of interleaved field blocks or top/bottom frame blocks to enable a determination of where to accumulate the blocks), the overlapped block processing module 350 receives the matched frames M4(t), M5(t), MSUM67(t), and MSUM0123(t), and extracts co-located blocks of 8×8 pixels from these four (4) frames. Denote the four (4) blocks extracted at a particular i, j pixel position in the four (4) frames as b(i, j, t) with t=0 . . . 3, then after reordering of the blocks (as explained below in the context of the 1D transform illustrated in FIGS. 8A-8D), the following terminology is described:
b(i, j, 0) is an 8×8 block from MSUM0123(t) at pixel position i, j
b(i, j, 1) is an 8×8 block from M4(t) at pixel position i, j
b(i, j, 2) is an 8×8 block from M5(t) at pixel position i, j
b(i, j, 3) is an 8×8 block from MSUM67(t) at pixel position i, j
Starting with i=0 and j=0, the upper left corner of the frames, a 2D transform module 304 (also referred to herein as transform logic) extracts the four (4) blocks b(0, 0, 0:3) and performs a 2D transform. That is, the 2D transform is taken on each of the four (4) temporal blocks b(i, j, t) with 0≦t≦3, where i, j is the pixel position of the top left corner of the 8×8 blocks in the overlapped block processing. In some embodiments, a 2D-DCT, DWT, among other well-known transforms may be used as the spatial transform. In an embodiment described below, the 2D transform is based on an integer DCT defined in the Advanced Video Coding (AVC) standard (e.g., an 8×8 AVC-DCT), which has the following form:
H ( X ) = DCT ( X ) = C · X · C T where , C = [ 8 8 8 8 8 8 8 8 12 10 6 3 - 3 - 6 - 10 - 12 8 4 - 4 - 8 - 8 - 4 4 8 10 - 3 - 12 - 6 6 12 3 - 10 8 - 8 - 8 8 8 - 8 - 8 8 6 - 12 3 10 - 10 - 3 12 - 6 4 - 8 8 - 4 - 4 8 - 8 4 3 - 6 10 - 12 12 - 10 6 - 3 ] · 1 / 8 Eq . ( 15 )
X=(C T ·H(X)·C)
S i,j, Eq. (16)
where Cs=C
Si,j and the symbol
denotes element by element multiplication. One example of scaling factors that may be used is given below:
S i , j = 1 ∑ i , j = 0 7 C ( i , j ) 2 = 0.0020 0.0017 0.0031 0.0017 0.0020 0.0017 0.0031 0.0017 Eq . ( 18 )
After the 2D transform, the overlapped block processing module 350 changes the spatial dimension from 2D to 1D by zig-zag scanning the output of each 2D transform from low frequency to highest frequency, so the 2D transformed block bs(i, j, f) becomes instead bs(zz_index, f), where 0≦zz_index≦63, 0≦f≦3. The mapping of (i, j) to zz_index is given by the zig_zag_scan vector below, identical to the scan used in MPEG-2 video encoding. In some embodiments, the 2D dimension may be retained for further processing. If the first row of the 2D matrix is given by elements 0 through 7, the second row by 8 through 16, then zig_zag_scan[0:63] specifies a 2D to 1D mapping as follows:
At a time corresponding to computation of the 2D transform (e.g., subsequent to the computation), the temporal mode (TemporalMode) is selected by temporal mode module 302 (also referred to herein as temporal mode logic). When utilizing a Haar 1-D transform, the TemporalMode defines whether 2D or 3D thresholding is enabled, which Haar subbands are thresholded for 3D thresholding, and which spatial subbands are thresholded for 2D thresholding. The temporal mode may either be SPATIAL_ONLY, FWD4, BAK4, or MODE8, as further described hereinbelow. The temporal mode is signaled to the 2D threshold module 306 and/or the 3D threshold module 310 (herein also collectively or individually referred to as threshold logic or thresholding logic). If the TemporalMode==SPATIAL_ONLY, then the 2D transformed block bs(zz_index, 1) is thresholded yielding bst(zz_index, 1). If the temporal mode is not SPATIAL_ONLY, then bst(zz_index, t) is set to bs(zz_index, f).
Following spatial thresholding by the 2D threshold module 306, the bst(zz_index, t) blocks are 1D transformed at the 1D transform module 308 (also referred to herein as transform logic), yielding bhaar(zz_index, f). The 1D transform module 308 takes in samples from the 2D transformed 8×8 blocks that have been remapped by zig-zag scanning the 2D blocks to 1D, so that bs(zz_index, f) represents a sample at 0≦zz_index≦63 and 0≦f≦3 so that the complete set of samples is bs(0:63, 0:3). Whereas the 2D transform module 304 operates on spatial blocks of pixels from the matched frames, the 1D transform module 308 operates on the temporal samples across the 2D transformed frame blocks at a given spatial index 0≦zz_index≦63. Therefore, there are sixty-four (64) 1D transforms for each set of four (4) 8×8 blocks.
As indicated above, the 1D Transform used for the VDN system embodiment 200 c−1 is a modified three-level, 1D Haar transform, though not limited to a Haar-based transform or three levels. That is, in some embodiments, other 1D transforms using other levels, wavelet-based or otherwise, may be used, including DCT, WHT, DWT, etc., with one of the goals comprising configuring the samples into filterable frequency bands. Before proceeding with processing of the overlapped block processing module 350, and in particular, 1D transformation, attention is re-directed to FIGS. 8A-8D, which illustrates various steps in the modification of a 1D Haar transform in the context of the reduction in frame matching described in association with FIGS. 2A-2C. It should be understood that each filter in the evolution of the 1D Haar shown in respective FIGS. 8A-8D may be a stand-alone filter that can be used in some VDN system embodiments. A modified Haar wavelet transform is implemented by the 1D transform module 308 (and the inverse in inverse 1D transform module 312) for the temporal dimension that enables frame collapsing in the manner described above for the different embodiments. In general, a Haar wavelet transform in one dimension transforms a 2-element vector according to the following equation:
( y ( 1 ) y ( 2 ) ) = T · ( x ( 1 ) x ( 2 ) ) , where T = 1 2 ( 1 1 1 - 1 ) . Eq . ( 19 )
A signal flow diagram 800A for the Haar transform, including the forward and inverse transforms, is shown in FIG. 8A. The signal flow diagrams 800A, 800B, 800C, or 800D in FIGS. 8A-8D are illustrative of example processing (from top-down) of 2D transform samples that may be implemented collectively by the 1D transform module 308 and the inverse 1D transform module 312. The signal flow diagram 800A is divided into a forward transform 802 and an inverse transform 804. The normalizing term 1/√{square root over (2)} may be removed if the transform is resealed on inverse (e.g., as shown in FIGS. 8A-8D by factors of four (4) and two (2) with ×2 and ×4, respectively, in the inverse transform section 804). In addition, while FIG. 8A shows the inverse Haar transform 804 following directly after the forward transform 802, it should be appreciated in the context of the present disclosure that in view of the denoising methods described herein, thresholding operations (not shown) may intervene in some embodiments between the forward transform 802 and the inverse transform 804.
In a dyadic wavelet decomposition, a set of samples are “run” through the transformation (e.g. as given by Eq. (19)), and the result is subsampled by a factor of two (2), known in wavelet theory as being “critically sampled.” Using eight (8) samples (e.g., 2D transformed, co-located samples) as an example in FIG. 8A, the samples b0 through b7 are transformed in a first stage 806 by Eq. (19) pair-wise on [b0 b1], [b2 b3], . . . [b6 . . . b7], producing the four (4), low frequency, sum-subband samples [L00, L01, L02, L03] and four (4), high frequency, difference-subband samples [H00, H01, H02, H03]. Since half the subbands are low-frequency, and half are high-frequency, the result is what is referred to as a dyadic decomposition. An eight (8) sample full decomposition continues by taking the four (4), low-frequency subband samples [L00, L01, L02, L03] and running them through a second stage 808 of the Eq. (19) transformation with critical sampling, producing two (2) lower-frequency subbands [L10, L11] and two (2) higher frequency subbands [H10, H11]. A third stage 610 on the two (2) lowest frequency samples completes the full dyadic decomposition, and produces [L20, H20].
The 1D Haar transform in FIG. 8A enables a forward transformation 802 and inverse transformation 804 when all samples (e.g., bd(i, j, 0:7)) are retained on output. This is the case for a 3D accumulation buffer. However, as discussed hereinabove, the output of the 2D+1 accumulation buffer (see FIGS. 2A-2B) requires only bd(i, j, 4) and bd(i, j, 7). Therefore, simplification of the flow diagram of 800A, where only the bd(i, j, 4) and bd(i, j, 7) are retained, results in the flow diagram denoted as 800B and illustrated in FIG. 8B.
In FIG. 8B, since bd(i, j, 4) and bd(i, j, 7) are the desired outcome, the entire left-hand side of the transformation process requires only the summation of the first four (4) samples b(i, j, 0:3). As described below, this summation may occur outside the 1D transform (e.g., as part of the frame matching process). On the right side, there has been a re-ordering of the samples when compared to the flow diagram of FIG. 8A (i.e., [b4 b5 b6 b7] to [b4 b7 b6 b5]). In addition, only the sum of b(i, j, 5) and b(i, j, 6) is required (not a subtraction). Accordingly, by using this simplified transform in flow diagram 800B, with sample reordering, and frame-matching the sum of the first four (4) frames to the reference frame F4(t), the first four (4) frames may be collapsed into a single frame. For frames 5 and 6, an assumption is made that both frames have been frame matched to the reference producing M5(t) and M6(t), and hence those frames may be summed prior to (e.g., outside) the 1D transform.
When the summing happens outside the 1D transform, the resulting 1D transform is further simplified as shown in the flow diagram 800C of FIG. 8C, where b0+b1+b2+b3 represents the blocks from the frame-matched sum of frames F0(t) through F3(t), that is MSUM0123(t), prior to the 1D transform, and b5+b6 represents the sum of blocks b(i, j, 5) and b(i, j, 6) from frame-matched and summed frames M5(t) and M6(t), that is MSUM56(t), the summation implemented prior to the 1D transform. Note that for 2D+1 accumulation embodiments corresponding to Haar modifications corresponding to FIGS. 8B and 8C, there is a re-ordering of samples such that b7 is swapped in and b5 and b6 are moved over.
Referring to FIG. 8D, shown is a flow diagram 800D that is further simplified based on a 2D accumulation buffer, as shown in FIG. 2C. That is, only one input sample, bd4, has been retained on output to the 2D accumulation buffer. Note that in contrast to the 2D+1 accumulation buffer embodiments where sample re-ordering is implemented as explained above, the Haar modifications corresponding to FIG. 8D involve no sample re-ordering.
Continuing now with the description pertaining to 1D transformation in the overlapped block processing module 350, and with reference to FIG. 3, at each zz_index, the 1D transform 308 processes the four (4) samples bs(zz_index, 0:3) to produce bhaar(0:63, 0:3), which includes Haar subbands [L20, H20, H02, H11]. The first index is from the stages 0, 1, 2, so L20 and H20 are from the third stage 810 (FIG. 8D), H02 is from the first stage 806 (FIG. 8D), and H11 is from the second stage (808). These Haar subbands are as illustrated in FIG. 8D and are further interpreted with respect to the matched frames as follows:
H02: For 2D+1 Accumulation Buffer (FIG. 8C), difference between matched blocks b4 and b7 from frames M4(t), and M7(t), respectively. For 2D Accumulation Buffer (FIG. 8D), difference between matched blocks b4 and b5 from frames M4(t), and M5(t), respectively.
H11: For 2D+1 Accumulation Buffer (FIG. 8C), difference between sum (b4+b7) matched blocks from M4(t) and M7(t) respectively, and 2×MSUM56(t). For 2D Accumulation Buffer (FIG. 8D), difference between sum (b4+b5) matched blocks from M4(t) and M5(t) respectively, and 2×MSUM56(t).
With reference to FIG. 9, shown is a schematic diagram 900 that conceptually illustrates the various temporal mode selections that determine whether 3D or 2D thresholding is enabled, and whether Haar or spatial subbands are thresholded for 3D and 2D, respectively. As shown, the selections include MODE8, BAK4, FWD4, and SPATIAL, explained further below. The temporal mode selections make it possible for the VDN systems 200 to adapt to video scenes that are not temporally correlated, such as scene changes, or other discontinuities (e.g., a viewer blinks his or her eye or turns around during a scene that results in the perception of a discontinuity, or discontinuities associated with a pan shot, etc.) by enabling a determination of which frames of a given temporal sequence (e.g., F0(t)-F7(t)) can be removed from further transform and/or threshold processing. The TemporalMode selections are as follows:
(d) MODE8: when the TemporalMode is set to MODE8, all samples are used. The TemporalMode is computed for every overlapped set of blocks (e.g., its value is computed at every overlapped block position). Therefore, implicitly, TemporalMode is a function of i, j, so TemporalMode(i, j) denotes the value of TemporalMode for the i, j-th pixel position in the overlapped block processing. The shorthand, “TemporalMode” is used throughout herein with the understanding that TemporalMode comprises a value that is computed at every overlapped block position of a given frame to ensure, among other reasons, proper block matching is achieved. In effect, the selected temporal mode defines the processing (e.g., thresholding) of a different number of subbands (e.g., L20, H20, etc.).
Having described the various temporal modes implemented in the VDN systems 200, attention is now directed to a determination of which temporal mode to implement. To determine TemporalMode, the SubbandSAD is computed (e.g., by the 2D transform module 304 and communicated to the temporal mode module 302) between the blocks bs(0:63, k) and bs(0:63, 1) for 0≦k≦3 (zero for k=1), which establishes the closeness of the match of co-located blocks of the inputted samples (from the matched frames), using the low-frequency structure of the blocks where the signal can mostly be expected to exceed noise. Explaining further, the determination of closeness of a given match may be obscured or skewed when the comparison involves noisy blocks. By rejecting noise, the level of fidelity of the comparison may be improved. In one embodiment, the VDN system 200 c-1 effectively performs a power compaction (e.g., a forward transform, such as via DCT) of the blocks at issue, whereby most of the energy of a natural video scene are power compacted into a few, more significant coefficients (whereas noise is generally uniformly distributed in a scene). Then, a SAD is performed in the DCT domain between the significant few coefficients of the blocks under comparison (e.g., in a subband of the DCT, based on a predefined threshold subband SAD value, not of the entire 8×8 block), resulting in removal of a significant portion of the noise from the computation and hence providing a more accurate determination of matching.
SubbandSAD ( k ) = ∑ z = 0 9  bs ( z , k ) - bs ( z , 1 )  , Eq . ( 20 )
SubbandCountFWD 4 = ∑ k = 1 3 SetToOneOrZero ( SubbandSAD ( k ) < Tsubbandsad ) Eq . ( 21 a ) SubbandCountBAK 4 = ∑ k = 0 1 SetToOneOrZero ( SubbandSAD ( k ) < Tsubbandsad ) Eq . ( 21 b ) SubbandCount 8 = ∑ k = 0 3 SetToOneOrZero ( SubbandSAD ( k ) < Tsubbandsad ) Eq . ( 21 c )
where the function SetToOneOrZero(x) equals 1 when its argument evaluates to TRUE, and zero otherwise, and Tsubbandsad is a parameter. In effect, Eqns. 21a-21c are computed to determine how many of the blocks are under the subband SAD threshold, and hence determine the temporal mode to be implemented. For instance, referring to Eq. 21c, for MODE8, the DCT of b0+b1+b2+b3 should be close enough in the lower frequencies to the DCT of b4 (and likewise, the DCT of b5 and b6+7 should be close enough in the lower frequencies to b4). Note that k=0 to k=3 since there is b0+b1+b2+b3, b4 (though k=1 is meaningless since the subband SAD of b4 with itself is zero), b5, and b6+7.
Reference is made to FIG. 10, which is a schematic diagram 1000 that illustrates one example embodiment of time-space frequency partitioning by thresholding vector, T_2D, and spatial index matrix, S_2D, each of which can be defined as follows:
S_2D(0:3,2): 4×2 element matrix of spatial indices
Together, T_2D and S_2D define the parameters used for thresholding the 2D blocks bs(0:63, 1). For only the 8×8 2D transformed block bs(0:63, 1), the thresholded block bst(0:63, 1) may be derived according to Eq. (22) below:
bst ( z , t ) = { 0 if  bs ( z , 1 )  < T_ 2 D ( j ) and S_ 2 D ( j , 0 ) ≤ z ≤ S_ 2 D ( j , 1 ) bs ( z , 1 ) otherwise for j = 0 … 3. Eq . ( 22 )
In Eq. (22), T_2D(j) defines the threshold used for block 1, bs(0:63, 1) from M4(t), over a span of spatial indices S_2D(j, 0) to S_2D(j, 1) for j=0.3. Equivalently stated, elements of bs(S_2D(j, 0): S_2D(j, 1), 1) are thresholded by comparing the values of those elements to the threshold T_2D(j), and the values in bs(S_2D(j, 0): S_2D(j, 1), 1) are set to zero when their absolute values are less than T_2D(j). Note that none of the matched frames MSUM0123(t), MSUM67(t) or M5(t) undergo 2D thresholding, only blocks from M4(t), bs(0:63, 1).
The spatial index matrix S_2D together with T_2D define a subset of coefficients in the zig-zag scanned spatial frequencies as illustrated in FIG. 10. The 2D space has been reduced to 1D by zig zag scanning
Thresholding (at 3D threshold module 310) of the 3D transformed blocks bhaar(0:63, 0:3) output from 1D transform module 308 is performed when TemporalMode is set to either FWD4, BAK4 or MODE8. Otherwise, when TemporalMode is set to SPATIAL, the output of 3D thresholding module 310 (e.g., bhaart(0:63, 0:3)) is set to the input of the 3D thresholding module 310 (e.g., input equals bhaar(0:63, 0:3)) without modification. The 3D thresholding module 310 uses threshold vectors T_3D(j) and spatial index matrix S_3D(j, 0:1) defined hereinabove in 2D thresholding 306, except using eight (8) thresholds so 0≦j<8.
An additional threshold vector TSUB(j, 0:1) is needed for 3D thresholding 310, which defines the range of temporal subbands for each j. For example, TSUB(0,0)=0 with TSUB(0, 1)=1 along with T_3D(0)=100 and S_3D(0, 0)=0 and S_3D(0, 1)=32 indicates that for j=0, 3D thresholding with the threshold of 100 is used across Haar subbands L20 and H20 and spatial frequencies 0 to 32.
Thresholding 310 of the 3D blocks is followed identically to the 2D thresholding 306 case following Eq. (22), but substituting T_3D and S_3D for T_2D and S_2D. For 3D thresholding 310, unlike 2D, all four (4) blocks bhaar(0:63, 0:3) are thresholded.
If TemporalMode==MODE8 threshold [L20, H20, H02, H11] Haar Subbands;
If TemporalMode==FWD4 threshold [H20, H02, H11] Haar Subbands;
If TemporalMode==BAK4 threshold [L20, H20] Haar Subbands.
The temporal mode determinations described above are further facilitated by another preprocessing sub-system embodiment, referred to herein as the SCD sub-system 300B shown in FIGS. 3 and 5. In one embodiment, logic of the SCD sub-system 300 resides in the full pixel BM 516 as shown (FIG. 5), though one having ordinary skill in the art should appreciate that other locales are contemplated for the SCD sub-system functionality in some embodiments. Generally, scene change detection and handling are utilized for proper temporal mode decisions. During a scene change, for example, even if 99% of the temporal mode decisions are fairly optimal, there exists a chance of a few bad decisions bleeding into the denoised frame and becoming visible. In one embodiment, scene changes are detected for the F7 frame (see, e.g., FIGS. 2C and 4) during FULLBM (or an F8 and delayed one frame). In general, all modes are reverted to backward and spatial-only (no forward) during a scene change, and forward and spatial-only (no backward)—four (4) frames later—as the old scene shifts into F0 to F3. When the scene change is completely flushed (F0 on previous frame), then mode decision processing reverts to normal operation in the manner described above.
More specifically, scene change detection occurs during full pixel block matching (e.g., corresponding to full pixel BM 516), defined above in association with FIG. 5, for luma only (chroma reuses the luma scene change detect, although in some embodiments, not limited to exclusively re-using luma). For each of the 8×8 luma blocks in full pixel block matching, if the final SAD in Eq. (14) is above a defined threshold, TSAD_SCENE_DETECT_THRESH, a counter is incremented for the frame. When the counter exceeds another threshold, T_NUM_SAD_ABOVE_SCENE_DETECT_THRESH, a scene change detection is triggered. The T_NUM_SAD_ABOVE_SCENE_DETECT_THRESH parameter is set as some percentage of the total 8×8 blocks in the frame (e.g., 40%), though in some embodiments, other criteria may be used for the setting configuration. Although the discussion above is described in the context of incrementing a counter and triggering of scene change detection responsive to “exceeding” certain thresholds, one having ordinary skill in the art should understand that other event triggering criteria (e.g., greater than or equal to, etc.) may be employed in some embodiments.
In one embodiment, scene change detection is triggered only during the F7→F6 block matching, since this is the most distant frame later in time from F4. Also, since block matching proceeds for both fields independently in interlaced video, a scene change detection is triggered whenever either of the fields detects a scene change. FIG. 11 shows a diagram 1100 of an index or table of matched frames 1102 (M0, M1, . . . M7) and a corresponding scene change detection mask (SCDM) 1104 maintained by logic of the SCD sub-system 300B. When a scene change is detected, a bit is set in the SCDM 1104. In this example embodiment, the SCDM 1104 comprises a mask which indicates the scene change detection history for the current time and previous seven (7) frame times. For each frame processing interval, this mask 1104 is shifted to the left one bit and a new bit is set on the right, denoting the scene change status (scene change detected) for the current M7. If matching F8→F7 (and delaying, instead of F7→F6), then everything is delayed one frame with equivalent effect. It should be understood by one having ordinary skill in the art in the context of the present disclosure that the use of eight (8) frames is only one example embodiment used for illustrative purposes, and other quantities of frames may be used, enabling larger or smaller masks than the mask 1104 shown in FIG. 11.
From the description above, it should be appreciated that the SCD sub-system 300B implements a scene change detection process that yields the SCDM 1104, the SCDM 1104 indicating the scene change history for frames M0 through M7. The SCDM 1104 is communicated to the temporal mode module 302. With a bit of the SCDM 1104 set for a given window of frames, the temporal mode decision described above in association with FIG. 9 is altered as given in the example pseudocode below:
Mode = GetTemporalMode
if Scene Change Mask Forward Bits Set
if Mode == BI8
Mode = BAK4
else if Mode == FWD4
Mode = SPATIAL
if Any Scene Change Mask Backward Bits Set
Mode = FWD8
else if Mode == BAK4
Note that chroma processing uses the luma SCDM 1104 (e.g., the mask 1104 is set once in the luma pixel BM process). Addressing the cooperation between the temporal mode determinations described above in association with FIG. 9 and the SCD sub-system 300B, one method embodiment comprises first determining the temporal mode in the manner as described in association with FIG. 9. Then, if the SCDM 1104 has any forward bits set, in accordance with the pseudocode hereinabove, the mode is reverted to backward only or spatial only modes. If the SCDM 1104 has any backward bits set, then the mode is reverted to forward only or spatial mode. Note that both forward and backward bits may be set for very fast scene changes (e.g., less than eight (8) frames), during a continual cross-fade, etc. If no bits are set in the SCDM 1104, then the temporal mode as initially determined is left unaltered.
Having described example embodiments of thresholding in VDN systems, attention is again directed to FIG. 3 and inverse transform and output processing. Specifically, inverse transforms include the inverse 1D transform 312 (e.g., Haar) followed by the inverse 2D transform (e.g., AVC integer DCT), both described hereinabove. It should be understood by one having ordinary skill in the art that other types of transforms may be used for both dimensions, or a mix of different types of transforms different than the Haar/AVC integer combination described herein in some embodiments. For a 1D Haar transform, the inverse transform proceeds as shown in FIG. 8D and explained above.
FD 4 ( i , j ) = A ( i , j ) W ( i , j ) Eq . ( 23 )
Having described various preprocessing sub-systems of the example VDN systems 200 disclosed herein, it should be appreciated that one method embodiment 1200A, implemented by the field-frame processing sub-system 300C in conjunction with an overlapped block processing module 350 and shown in FIG. 12A, comprises providing a field-frame map by partitioning a block matched frame into plural non-overlapping superblocks (1202); designating each of the plural superblocks of the field-frame map as either field or frame (1204); comparing an overlapped superblock to first plural superblocks of the field-frame map (1206); and field processing by overlapped block processing logic two overlapped blocks of the overlapped superblock if one of the first plural blocks intersected by the overlapped superblock has a field designation, otherwise frame processing, by the overlapped block processing logic, the two overlapped blocks of the overlapped superblock (1208).
Another method embodiment, 1200B, implemented by the field-frame processing sub-system 300C in conjunction with an overlapped block processing module 350 and shown in FIG. 12B, comprises partitioning a block matched reference frame into plural n×m non-overlapping pixel superblocks, where n and m are non-negative integer numbers (1210); designating each of the n×m pixel superblocks as field or frame (1212); and field processing by overlapped block processing logic two n×n blocks of an n×m overlapped superblock if one of first plural n×m blocks intersected by the overlapped superblock has a field designation, otherwise frame processing, by the overlapped block processing logic, the two n×n blocks of the overlapped superblock (1214).
Another method embodiment 1300, shown in FIG. 13, implemented by the SCD sub-system 300B, comprises receiving noise-filtered plural blocks of a first frame and noise-filtered plural blocks of a second frame (1302); for each of the plural blocks to be matched, determining whether an indication of closeness in match between the each of the plural blocks exceeds a first threshold (1304); incrementing a counter value each time the first threshold is exceeded for closeness of the block matching of a particular block (1306); determining whether the counter value exceeds a second threshold, the exceeding of the second threshold indicating that a defined quantity of blocks has exceeded the first threshold (1308); and responsive to determining that the counter value exceeds the second threshold, triggering a scene change detection (1310).
Another method embodiment 1400, shown in FIG. 14, implemented by the SMC sub-system 300A, comprises receiving at a frame matching module a first frame comprising first plural blocks and plural frames that each comprise a plurality of blocks to be matched to the first plural blocks of the first frame, the first plural blocks and the plurality of blocks comprising luma blocks (1402); for each of the frame pair matchings, selecting one border configuration among a plurality of border configurations, the border configuration selected for the each of the frame pair matchings unique (1404); appending a border of pixels to the frames of the each of the frame pair matchings based on the selected border configuration (1406); and block matching the first plural blocks with the plurality of blocks (1408).
Any process descriptions or blocks in flow charts or flow diagrams should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the present disclosure in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art. In some embodiments, steps of a process identified in FIGS. 12A-14 using separate boxes can be combined. Further, the various steps in the flow diagrams illustrated in conjunction with the present disclosure are not limited to the architectures described above in association with the description for the flow diagram (as implemented in or by a particular module or logic) nor are the steps limited to the example embodiments described in the specification and associated with the figures of the present disclosure. In some embodiments, one or more steps may be added to one or more of the methods described in FIGS. 12A-14, either in the beginning, end, and/or as intervening steps.
It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations, merely set forth for a clear understanding of the principles of the VDN systems and methods and their associated sub-systems. Many variations and modifications may be made to the above-described embodiment(s) without departing substantially from the spirit and principles of the disclosure. Although all such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims, the following claims are not necessarily limited to the particular embodiments set out in the description.
US4442454 15 Nov 1982 10 Apr 1984 Eastman Kodak Company Image processing method using a block overlap transformation procedure
US5699336 18 Jan 1996 16 Dec 1997 Sony Corporation Reproducing apparatus having buffer memory and capable of rapidly restarting reproduction and method of controlling the apparatus
US5764805 23 Oct 1996 9 Jun 1998 David Sarnoff Research Center, Inc. Low bit rate video encoder using overlapping block motion compensation and zerotree wavelet coding
US6091460 10 Oct 1997 18 Jul 2000 Mitsubishi Denki Kabushiki Kaisha Video signal encoding method and system
US6442203 * 5 Nov 1999 27 Aug 2002 Demografx System and method for motion compensation and frame rate conversion
US6735342 29 Mar 2001 11 May 2004 Koninklijke Philips Electronics N.V. Video encoding method using a wavelet transform
US6754234 21 May 1999 22 Jun 2004 Ati International Srl Method and apparatus for asynchronous frame synchronization
US6801573 21 Dec 2000 5 Oct 2004 The Ohio State University Method for dynamic 3D wavelet transform for video compression
US7206459 31 Jul 2001 17 Apr 2007 Ricoh Co., Ltd. Enhancement of compressed images
US7554611 19 Apr 2005 30 Jun 2009 Samsung Electronics Co., Ltd. Method and apparatus of bidirectional temporal noise reduction
US7916951 23 Feb 2009 29 Mar 2011 Landwehr Val R System and method for detecting and classifying objects in images, such as insects and other arthropods
US8130828 7 Apr 2006 6 Mar 2012 Microsoft Corporation Adjusting quantization to preserve non-zero AC coefficients
US8275047 * 19 Sep 2002 25 Sep 2012 Xilinx, Inc. Method and device for block-based conditional motion compensation
US8285068 25 Jun 2008 9 Oct 2012 Cisco Technology, Inc. Combined deblocking and denoising filter
US8358380 5 Jun 2009 22 Jan 2013 Cisco Technology, Inc. Efficient spatial and temporal transform-based video preprocessing
US8472725 2 Jun 2010 25 Jun 2013 Cisco Technology, Inc. Scene change detection and handling for preprocessing video with overlapped 3D transforms
US8520731 5 Jun 2009 27 Aug 2013 Cisco Technology, Inc. Motion estimation for noisy frames based on block matching of filtered blocks
US8571117 5 Jun 2009 29 Oct 2013 Cisco Technology, Inc. Out of loop frame matching in 3D-based video denoising
US8615044 5 Jun 2009 24 Dec 2013 Cisco Technology, Inc. Adaptive thresholding of 3D transform coefficients for video denoising
US8619881 5 Jun 2009 31 Dec 2013 Cisco Technology, Inc. Estimation of temporal depth of 3D overlapped transforms in video denoising
US8638395 5 Jun 2009 28 Jan 2014 Cisco Technology, Inc. Consolidating prior temporally-matched frames in 3D-based video denoising
US8781244 8 Oct 2012 15 Jul 2014 Cisco Technology, Inc. Combined deblocking and denoising filter
US9237259 14 Jan 2014 12 Jan 2016 Cisco Technology, Inc. Summating temporally-matched frames in 3D-based video denoising
US9342204 25 Jun 2013 17 May 2016 Cisco Technology, Inc. Scene change detection and handling for preprocessing video with overlapped 3D transforms
US20020196857 26 Mar 2002 26 Dec 2002 Fujitsu Limited Video decoding device and method, and program product therefor
US20030081685 31 Oct 2001 1 May 2003 Montgomery Dennis L. Method and apparatus for determining patterns within adjacent blocks of data
US20030086623 31 Jul 2001 8 May 2003 Kathrin Berkner Enhancement of compressed images
US20030128761 9 Dec 2002 10 Jul 2003 Minhua Zhou Image and video coding
US20050078872 10 Oct 2003 14 Apr 2005 Ramin Samadani Reducing artifacts in compressed images
US20050094893 29 Oct 2003 5 May 2005 Ramin Samadani Transformations for denoising images
US20050100235 7 Nov 2003 12 May 2005 Hao-Song Kong System and method for classifying and filtering pixels
US20050100236 14 Oct 2004 12 May 2005 Hao-Song Kong System and method for classifying pixels in images
US20050100237 14 Oct 2004 12 May 2005 Hao-Song Kong System and method for filtering artifacts in images
US20050100241 27 Apr 2004 12 May 2005 Hao-Song Kong System and method for reducing ringing artifacts in images
US20050262276 * 11 May 2005 24 Nov 2005 Ittiam Systamc (P) Ltd. Design method for implementing high memory algorithm on low internal memory processor using a direct memory access (DMA) engine
US20060002472 30 Jun 2004 5 Jan 2006 Mehta Kalpesh D Various methods and apparatuses for motion estimation
US20060039624 20 Aug 2004 23 Feb 2006 Hao-Song Kong System and method for fuzzy filtering images
US20060056724 29 Jul 2005 16 Mar 2006 Le Dinh Chon T Apparatus and method for adaptive 3D noise reduction
US20060133682 17 Dec 2004 22 Jun 2006 Microsoft Corporation Reversible overlap operator for efficient lossless data compression
US20060198439 1 Mar 2005 7 Sep 2006 Qin-Fan Zhu Method and system for mode decision in a video encoder
US20060204046 * 6 Mar 2006 14 Sep 2006 Yu Xia Method and apparatus for motion estimation in video signal decoding
US20060245497 14 Apr 2006 2 Nov 2006 Tourapis Alexis M Device and method for fast block-matching motion estimation in video encoders
US20070041448 17 Aug 2005 22 Feb 2007 Miller Casey L Artifact and noise reduction in MPEG video
US20070071095 23 Sep 2005 29 Mar 2007 Kyoungwon Lim De-blocking and de-ringing systems and methods
US20070092007 24 Oct 2005 26 Apr 2007 Mediatek Inc. Methods and systems for video data processing employing frame/field region predictions in motion estimation
US20070140352 19 Dec 2005 21 Jun 2007 Vasudev Bhaskaran Temporal and spatial analysis of a video macroblock
US20070237238 7 Apr 2006 11 Oct 2007 Microsoft Corporation Making interlace frame level coding mode decisions
US20080055477 30 Aug 2007 6 Mar 2008 Dongsheng Wu Method and System for Motion Compensated Noise Reduction
US20080123740 21 Sep 2004 29 May 2008 Ye Jong C Video De-Noising Algorithm Using Inband Motion-Compensated Temporal Filtering
US20080151101 * 8 Nov 2006 26 Jun 2008 Qualcomm Incorporated Preprocessor method and apparatus
US20080211965 2 Apr 2008 4 Sep 2008 Realnetworks, Inc. Automated inverse telecine conversion
US20080253457 9 Apr 2008 16 Oct 2008 Moore Darnell J Method and system for rate distortion optimization
US20080260033 17 Apr 2007 23 Oct 2008 Horizon Semiconductors Ltd. Hybrid hierarchical motion estimation for video streams
US20080285650 14 May 2007 20 Nov 2008 Samsung Electronics Co., Ltd. System and method for phase adaptive occlusion detection based on motion vector field in digital video
US20090002553 27 Oct 2006 1 Jan 2009 Sony United Kingdom Limited Video Processing
US20090016442 10 Jan 2006 15 Jan 2009 Vvond, Inc. Deblocking digital images
US20090067504 5 Sep 2008 12 Mar 2009 Alexander Zheludkov Real-time video coding/decoding
US20090097566 12 Oct 2007 16 Apr 2009 Yu-Wen Huang Macroblock pair coding for systems that support progressive and interlaced data
US20090167952 18 Nov 2008 2 Jul 2009 Kabushiki Kaisha Toshiba Noise reduction device and noise reduction method
US20090238535 21 Mar 2008 24 Sep 2009 Sony Corporation Merging video with time-decimated high-resolution imagery to form high-resolution video frames
US20090327386 25 Jun 2008 31 Dec 2009 Joel Warren Schoenblum Combined deblocking and denoising filter
US20100020880 25 Jun 2009 28 Jan 2010 Mathstar, Inc. Field programmable object array having image processing circuitry
US20100027897 24 Jul 2009 4 Feb 2010 Joel Sole Method and apparatus for detecting image blocking artifacts
US20100033633 21 Dec 2007 11 Feb 2010 Gokce Dane Detecting block artifacts in coded images and video
US20100091862 28 Dec 2008 15 Apr 2010 Sy-Yen Kuo High-Performance Block-Matching VLSI Architecture With Low Memory Bandwidth For Power-Efficient Multimedia Devices
US20100309377 5 Jun 2009 9 Dec 2010 Schoenblum Joel W Consolidating prior temporally-matched frames in 3d-based video denoising
US20100309379 5 Jun 2009 9 Dec 2010 Schoenblum Joel W Efficient spatial and temporal transform-based video preprocessing
US20100309979 5 Jun 2009 9 Dec 2010 Schoenblum Joel W Motion estimation for noisy frames based on block matching of filtered blocks
US20100309989 5 Jun 2009 9 Dec 2010 Schoenblum Joel W Out of loop frame matching in 3d-based video denoising
US20100309990 5 Jun 2009 9 Dec 2010 Schoenblum Joel W Estimation of temporal depth of 3d overlapped transforms in video denoising
US20100309991 5 Jun 2009 9 Dec 2010 Schoenblum Joel W Adaptive thresholding of 3d transform coefficients for video denoising
US20100316129 16 Aug 2010 16 Dec 2010 Vixs Systems, Inc. Scaled motion search section with downscaling filter and method for use therewith
US20110298984 2 Jun 2010 8 Dec 2011 Cisco Technology, Inc. Preprocessing of interlaced video with overlapped 3d transforms
US20110299781 2 Jun 2010 8 Dec 2011 Cisco Technology, Inc. Scene change detection and handling for preprocessing video with overlapped 3d transforms
US20120230423 10 Mar 2011 13 Sep 2012 Esenlik Semih Line memory reduction for video coding and decoding
US20130028525 8 Oct 2012 31 Jan 2013 Cisco Technology, Inc. Combined Deblocking and Denoising Filter
US20130290849 25 Jun 2013 31 Oct 2013 Cisco Technology, Inc. Scene Change Detection and Handling for Preprocessing Video with Overlapped 3D Transforms
US20140125869 14 Jan 2014 8 May 2014 Cisco Technology, Inc. Summating Temporally-Matched Frames in 3D-Based Video Denoising
US20160119508 11 Jan 2016 28 Apr 2016 Cisco Technology, Inc. Processing Prior Temporally-Matched Frames in 3D-Based Video Denoising
CN1154044A 5 Sep 1996 9 Jul 1997 松下电器产业株式会社 Ringing detector and filter
EP0762738A2 2 Sep 1996 12 Mar 1997 Matsushita Electric Industrial Co., Ltd. Ringing detector and filter
WO1991008547A1 28 Nov 1990 13 Jun 1991 Eastman Kodak Company Histogram/variance mechanism for detecting presence of an edge within block of image data
1 A. Nosratinia, "Denoising JPEG images by re-application of JPEG", in 1998 IEEE Second Workshop on Multimedia Signal Processing, pp. 611-615, New York: Institute of Electrical and Electronics Engineers, Dec. 1998.
2 ALEKSANDRA PIZURICA, VLADIMIR ZLOKOLICA AND WILFRIED PHILIPS: "Noise Reduction in Video Sequences Using Wavelet-Domainand Temporal Filtering", PROCEEDINGS OF SPIE, SPIE, US, vol. 5266, 27 February 2004 (2004-02-27), US, pages 48 - 59, XP002596108, ISSN: 0277-786x
3 C. Wu et al., "Adaptive postprocessors with DCT-based block classifications", in IEEE Transactions on Circuits and Systems for Video Technology, pp. 365-375, New York: Institute of Electrical and Electronics Engineers, vol. 13, No. 5, May 2003.
4 Chinese First Office Action dated Aug. 1, 2012 cited in Appl. No. 200980124315.4, 14 pages.
5 Chinese First Office Action dated Jan. 30, 2014 cited in Application No. 201080024719.9, 16 pgs.
6 Chinese Second Office Action dated Sep. 22, 2014 cited in Application No. 201080024719.9, 18 pgs.
7 * Dabov et al., "Video Denoising by Sparse 3D Transform-Domaincollaborative Filtering," Processing Conference, Sep. 7, 2007, pp. 1-5, XP002596106.
8 * Dabov, Foi, and Egiazarian, "Video Denoising by Sparse 3D Transform-Domain Collaborative Filtering," Sep. 3-7, 2007.
9 Dmytro Rusanovskyy et al., "Moving-window varying size 3D transform-based video denoising", VPQM'06, Scottdale, USA 2006, pp. 1-4.
10 Dmytro Rusanovskyy et al., "Video denoising algorithm in sliding 3D DCT domain", ACIVS 2005, Antwerp, Belgium, 8 pages.
11 European Communication dated Mar. 14, 2013 cited in Application No. 10730897.5, 4 pages.
12 European Office Action dated Feb. 6, 2015 cited in Application No. 10 730 897.5, 5 pgs.
13 European Office Action dated Jun. 30, 2016 cited in Application 09 770 924.01, 4 pgs.
14 European Office Action dated Jun. 30, 2016 cited in Application 13 189 789.4, 6 pgs.
15 European Office Action dated Nov. 15, 2011 cited in Appl. No. 09 770 924.01, 4 pgs.
16 European Office Action mailed Feb. 19, 2014 in European Application No. 09 770 924.01, 6 pgs.
17 European Summons to Attend Oral Proceedings dated Dec. 1, 2016 cited in Application No. 10 730 897.5, 8 pgs.
18 Extended European Search Report dated Feb. 27, 2014 cited in Application No. 13 189 789.4, 12 pgs.
19 G. Triantafyllidis et al., "Blocking artifact detection and reduction in compressed data", in IEEE Transactions on Circuits and Systems for Video Technology, pp. 877-890, New York: Institute of Electricl and Electronics Engineers, vol. 12, No. 10, Oct. 2002.
20 Huang et al., "Predictive subband image coding with wavelet transform", Signal Processing: Image Communication, vol. 13, No. 3, Sep. 1998, pp. 171-181.
21 International Search Report and Written Opinion dated Aug. 24, 2010 cited in Application No. PCT/US2010/0037189.
22 International Search Report dated Dec. 4, 2009 cited in Application No. PCT/US2009/048422.
23 J. Canny, "A computational approach to edge detection", in IEEE Trans Pattern Analysis and Machine Intelligence, pp. 679-698, Washington D.C.: IEEE Computer Society, vol. 8, No. 6, Nov. 1986.
24 Kostadin Dabov et al., "Image denoising by sparse 3D transform-domain collaborative filtering", Aug. 2007, vol. 16, No. 8, pp. 1-16.
25 * KOSTADIN DABOV, ALESSANDRO FOI, AND KAREN EGIAZARIAN: "VIDEO DENOISING BY SPARSE 3D TRANSFORM-DOMAINCOLLABORATIVE FILTERING", PROCEEDINGS / EUSIPCO 2007, 15TH EUROPEAN SIGNAL PROCESSING CONFERENCE : SEPTEMBER 3 - 7, 2007, POZNAÑ, POLAND, POZNAN, POLAND, 3 September 2007 (2007-09-03) - 7 September 2007 (2007-09-07), Poznan, Poland, pages 1 - 5, XP002596106
26 Kostadin Dabove et al., "Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space", Jul. 2007, pp. I-313 to I-316.
27 Kostadin Dabove et al., "Video denoising by sparse 3D transform-domain collaborative filtering", Sep. 2007, European Signal Processing Conference, Poznan, Poland, 5 pages.
28 Magarey et al., "Robust motion estimation using chrominance information in colour image sequences," In: "Lecture Notes in Computer Science," Dec. 31, 1997, Springer, vol. 1310/97, pp. 486-493, XP002596107.
29 MAGAREY J, KOKARAM A, KINGSBURY N: "Robust Motion Estimation Using Chrominance Information in Colour Image Sequences", PROCEEDINGS OF ICIAP 97. 9TH INTERNATIONAL CONFERENCE ON IMAGE ANALYSIS AND PROCESSING - 17-19 SEPT. 1997 - FLORENCE, ITALY, SPRINGER-VERLAG, BERLIN, GERMANY, vol. 1310, 17 September 1997 (1997-09-17) - 19 September 1997 (1997-09-19), Berlin, Germany, pages 486 - 493, XP002596107, ISBN: 354-0-635-07648-6
30 Marpe et al., "H.264/MPEG4-AVC Fidelity Range Extensions: Tools, Profiles, Performance, and Application Areas," IEEE Conference on Image Processing, Nov. 14, 2005, 4 pages.
31 Notice of Allowance and Issue Fee mailed Mar. 10, 2014 in U.S. Appl. No. 13/646,795, 8 pgs.
32 Notice of Allowance mailed Jan. 15, 2016 in U.S. Appl. No. 13/926,703, 12 pgs.
33 Notice of Allowance mailed Sep. 2, 2015 in U.S. Appl. No. 12/791,941, 12 pgs.
34 P. Dragotti et al., "Discrete directional wavelet bases for image compression", Visual Communications and Image Processing, Jul. 2003, Proceedings of SPIE, vol. 5150, pp. 1287-1295.
35 Pier Luigi Dragotti et al., "Discrete directional wavelet bases and frames for image compression and denoising," Jul. 8, 2003, 9 pgs.
36 Pizurica et al., "Noise reduction in video sequences using wavelet-domain and temporal filtering," Proceedings of SPIE, vol. 5266 Feb. 27, 2004, pp. 1-13, XP002596108.
37 R. Samadani et al., "Deringing and deblocking DCT compression artifacts with efficient shifted transforms", in 2004 International Conference on Image Processing, pp. 1799-1802, Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, vol. 3, No. Oct. 2004.
38 Steve Gordon et al., "Simplified Use of 8×8 Transforms-Updated Proposal & Results", Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, Mar. 2004, Munich, Germany, pp. 1-17.
39 T. Kasezawa, "Blocking artifacts reduction using discrete cosine transform", in IEEE Transactions on Consumer Electronics, pp. 48-55, New York: Institute of Electrical and Electronics Engineers, vol. 43, No. 1, Feb. 1997.
40 U.S. Appl. No. 12/146,369, filed Jun. 25, 2008, entitled "Combined Deblocking and Denoising Filter", Inventor: Joe Schoenblum.
41 U.S. Appl. No. 12/479,018, filed Jun. 5, 2009, entitled "Out of Loop Frame Matching in 3D-Based Video Denoising", Inventor: Joe Schoenblum.
42 U.S. Appl. No. 12/479,065, filed Jun. 5, 2009, entitled "Consolidating Prior Temporally-Matched Frames in 3D-Based Video Denoising", Inventor: Joe Schoenblum.
43 U.S. Appl. No. 12/479,104, filed Jun. 5, 2009, entitled "Efficient Spatial and Temporal Transform-Based Video Preprocessing", Inventor: Joe Schoenblum.
44 U.S. Appl. No. 12/479,147, filed Jun. 5, 2009, entitled "Estimation of Temporal Depth of 3D Overlapped Transforms in Video Denoising", Inventor: Joe Schoenblum.
45 U.S. Appl. No. 12/479,194, filed Jun. 5, 2009, entitled "Adaptive Thresholding of 3D Transform Coefficients for Video Denoising", Inventor: Joe Schoenblum.
46 U.S. Appl. No. 12/479,244, filed Jun. 5, 2009, entitled "Motion Estimation for Noisy Frames Based on Block Matching of Filtered Blocks", Inventor: Joe Schoenblum.
47 U.S. Appl. No. 12/791,941, filed Jun. 2, 2010, entitled "Preprocessing of Interlaced Video With Overlapped 3D Transforms", Inventor: Joe Schoenblum.
48 U.S. Appl. No. 12/791,970, filed Jun. 2, 2010, entitled "Scene Change Detection and Handling for Preprocessing Video With Overlapped 3D Transforms", Inventor: Joe Schoenblum.
49 U.S. Appl. No. 14/154,502, filed Jan. 14, 2014, entitled "Summating Temporally-Matched Frames in 3D-Based Video Denoising", Inventors: Joel W. Schoenblum et al.
50 U.S. Appl. No. 14/992,234, filed Jan. 11, 2016, entitled "Processing Prior Temporally-Matched Frames in 3D-Based Video Denoising", Inventor: Joel Schoenblum.
51 U.S. Final Office Action mailed Aug. 10, 2012 in U.S. Appl. No. 12/479,194, 17 pages.
52 U.S. Final Office Action mailed Aug. 7, 2012 in U.S. Appl. No. 12/479,244, 20 pages.
53 U.S. Final Office Action mailed Dec. 2, 2013 in U.S. Appl. No. 12/791,941, 16 pages.
54 U.S. Final Office Action mailed Jul. 3, 2012 in U.S. Appl. No. 12/479,018, 19 pages.
55 U.S. Final Office Action mailed May 20, 2013 in U.S. Appl. No. 12/479,147, 14 pages.
56 U.S. Final Office Action mailed Nov. 29, 2012 in U.S. Appl. No. 12/479,147, 14 pages.
57 U.S. Non-Final Office Action mailed Aug. 6, 2012 in U.S. Appl. No. 12/479,065, 18 pages.
58 U.S. Non-Final Office Action mailed Feb. 12, 2013 in U.S. Appl. No. 12/479,194, 15 pages.
59 U.S. Non-Final Office Action mailed Jan. 3, 2013 in U.S. Appl. No. 12/479,244, 18 pages.
60 U.S. Non-Final Office Action mailed Jan. 31, 2013 in U.S. Appl. No. 12/479,018, 14 pages.
61 U.S. Non-Final Office Action mailed Jul. 17, 2012 in U.S. Appl. No. 12/479,147, 13 pages.
62 U.S. Non-Final Office Action mailed Jun. 4, 2012 in U.S. Appl. No. 12/479,104, 29 pages.
63 U.S. Non-Final Office Action mailed Mar. 4, 2013 in U.S. Appl. No. 12/479,065, 11 pages.
64 U.S. Non-Final Office Action mailed May 21, 2013 in U.S. Appl. No. 12/791,941, 23 pages.
65 U.S. Non-Final Office Action mailed Nov. 23, 2011 in U.S. Appl. No. 12/479,244.
66 U.S. Non-Final Office Action mailed Oct. 11, 2011 in U.S. Appl. No. 12/479,018.
67 U.S. Non-Final Office Action mailed Oct. 12, 2011 in U.S. Appl. No. 12/479,147.
68 U.S. Non-Final Office Action mailed Oct. 20, 2011 in U.S. Appl. No. 12/479,194.
69 U.S. Office Action dated Jul. 12, 2016 cited in U.S. Appl. No. 12/791,941, 32 pgs.
70 U.S. Office Action dated Nov. 8, 2012 in U.S. Appl. No. 12/791,970, 12 pgs.
71 U.S. Office Action mailed Apr. 20, 2015 in U.S. Appl. No. 14/154,502, 9 pgs.
72 U.S. Office Action mailed Aug. 22, 2014 in U.S. Appl. No. 14/154,502, 19 pgs.
73 U.S. Office Action mailed Aug. 6, 2015 in U.S. Appl. No. 12/791,941, 25 pgs.
74 U.S. Office Action mailed Jan. 15, 2015 in U.S. Appl. No. 12/791,941, 20 pgs.
75 U.S. Office Action mailed Oct. 24, 2013 in U.S. Appl. No. 13/646,795, 5 pages.
76 U.S. Office Action mailed Sep. 24, 2015 in U.S. Appl. No. 123/926,703, 28 pgs.
77 Y. Nie et al., "Fast adaptive fuzzy post-filtering for coding artifacts removal in interlaced video", in IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. ii/993-ii/996, New York: Institute of Electrical and Electronics Engineers, vol. 2, Mar. 2005.
International Classification H04N11/04, H04N5/14, H04N7/12, H04N11/02, H04N5/213
Cooperative Classification H04N5/145, H04N5/213
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SCHOENBLUM, JOEL W.;REEL/FRAME:024470/0536