Motion estimation aided noise reduction

A method and apparatus for performing motion estimation aided noise reduction encoding and decoding are provided. Motion estimation aided noise reduction encoding can include identifying a motion vector for encoding a current block in a video frame based on a reference block in a reference frame, identifying a noise reduction block for denoising the current block, aligning the noise reduction block with the current block, denoising the current block, identifying a motion vector for encoding the denoised block, generating a residual block for encoding the denoised block, and encoding the denoised block. Denoising the current block can include identifying a filter coefficient for a pixel in the current block based on a corresponding pixel in the noise reduction block, producing a denoised pixel based on the coefficient and the corresponding pixel, and determining whether to use the denoised pixel or the current pixel for encoding the block.

TECHNICAL FIELD

This application relates to video encoding and decoding.

BACKGROUND

Digital video can be used, for example, for remote business meetings via video conferencing, high definition video entertainment, video advertisements, or sharing of user-generated videos. Accordingly, it would be advantageous to provide high resolution video transmitted over communications channels having limited bandwidth.

SUMMARY

A method and apparatus for performing motion estimation aided noise reduction encoding and decoding are provided. Motion estimation aided noise reduction encoding can include identifying a motion vector for encoding a current block in a video frame based on a reference block in a reference frame, identifying a noise reduction block for denoising the current block, aligning the noise reduction block with the current block, denoising the current block, identifying a motion vector for encoding the denoised block, generating a residual block for encoding the denoised block, and encoding the denoised block. Denoising the current block can include identifying a filter coefficient for a pixel in the current block based on a corresponding pixel in the noise reduction block, producing a denoised pixel based on the coefficient and the corresponding pixel, and determining whether to use the denoised pixel or the current pixel for encoding the block.

DETAILED DESCRIPTION

Digital video is used for various purposes including, for example, remote business meetings via video conferencing, high definition video entertainment, video advertisements, and sharing of user-generated videos. Digital video streams can include formats such as VP8, promulgated by Google, Inc. of Mountain View, Calif., and H.264, a standard promulgated by ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG), including present and future versions thereof.

Video encoding and decoding can include performing motion estimation to generate a motion vector for encoding a block in a current frame based on a corresponding block in a reference frame of a video signal. The video signal can include noise, such as random variations of pixel values. For example, a pixel, such as a background pixel, may include noise, such as a random change in color or brightness when compared to a corresponding pixel in a previous frame. Noise reduction can be performed at the encoder to reduce the noise in the video signal. Motion estimation aided noise reduction can include using the motion vector identified during motion estimation in conjunction with a noise reduction block to efficiently reduce noise in the signal while minimizing blurring and/or artifacting, for example, by replacing a pixel value with an average pixel value. For example, the average pixel value can be generated using a motion compensated noise reduction pixel from the noise reduction block and the pixel value.

A noise reduction block can include pixels used for noise reduction of a block in another frame, such as previous frame in the video sequence. For example, a noise reduction block can be used for denoising a first block in a first frame in the video signal. The noise reduction frame can include information, such as denoised pixel values, from denoising the first block, and can be used for denoising a second block in a second frame in the video signal.

FIG. 1is a schematic of a video encoding and decoding system. An exemplary transmitting station12can be, for example, a computing device having an internal configuration of hardware including a processor such as a central processing unit (CPU)14and a memory16. The CPU14can be a controller for controlling the operations of the transmitting station12. The CPU14is connected to the memory16by, for example, a memory bus. The memory16can be random access memory (RAM) or any other suitable memory device. The memory16can store data and program instructions which are used by the CPU14. Other suitable implementations of the transmitting station12are possible. As used herein, the term “computing device” includes a server, a hand-held device, a laptop computer, a desktop computer, a special purpose computer, a general purpose computer, or any device, or combination of devices, capable of processing information, programmed to perform the methods, or any portion thereof, disclosed herein.

A network28can connect the transmitting station12and a receiving station30for encoding and decoding of the video stream. Specifically, the video stream can be encoded in the transmitting station12and the encoded video stream can be decoded in the receiving station30. The network28can, for example, be the Internet. The network28can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), a mobile phone network, or any other means of transferring the video stream from the transmitting station12.

The receiving station30, in one example, can be a computing device having an internal configuration of hardware including a processor such as a central processing unit (CPU)32and a memory34. The CPU32is a controller for controlling the operations of the receiving station30. The CPU32can be connected to the memory34by, for example, a memory bus. The memory34can be RAM or any other suitable memory device. The memory34stores data and program instructions which are used by the CPU32. Other suitable implementations of the receiving station30are possible.

A display36configured to display a video stream can be connected to the receiving station30. The display36can be implemented in various ways, including by a liquid crystal display (LCD) or a cathode-ray tube (CRT). The display36can be configured to display a video stream decoded at the receiving station30.

Other implementations of the encoder and decoder system10are possible. For example, one implementation can omit the network28and/or the display36. In another implementation, a video stream can be encoded and then stored for transmission at a later time by the receiving station30or any other device having memory. In an implementation, the receiving station30receives (e.g., via network28, a computer bus, and/or some communication pathway) the encoded video stream and stores the video stream for later decoding. In another implementation, additional components can be added to the encoder and decoder system10. For example, a display or a video camera can be attached to the transmitting station12to capture the video stream to be encoded.

FIG. 2is a diagram of a typical video stream50to be encoded and decoded. The video stream50includes a video sequence52. At the next level, the video sequence52includes a number of adjacent frames54. While three frames are depicted in adjacent frames54, the video sequence52can include any number of adjacent frames. The adjacent frames54can then be further subdivided into a single frame56. At the next level, the single frame56can be divided into a series of blocks58. Although not shown inFIG. 2, a block58can include pixels. For example, a block can include a 16×16 group of pixels, an 8×8 group of pixels, an 8×16 group of pixels, or any other group of pixels. Unless otherwise indicated herein, the term ‘block’ can include a macroblock, a segment, a slice, or any other portion of a frame. A frame, a block, a pixel, or a combination thereof can include display information, such as luminance information, chrominance information, or any other information that can be used to store, modify, communicate, or display the video stream or a portion thereof.

FIG. 3is a block diagram of an encoder70in accordance with one embodiment. The encoder70can be implemented, as described above, in the transmitting station12such as by providing a computer software program stored in memory16, for example. The computer software program can include machine instructions that, when executed by CPU14, cause transmitting station12to encode video data in the manner described inFIG. 3. Encoder70can also be implemented as specialized hardware included, for example, in transmitting station12. Encoder70can also be implemented as specialized hardware included, for example, in transmitting station12. The encoder70encodes an input video stream50. The encoder70has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or a compressed bitstream88: an intra/inter prediction stage72, a transform stage74, a quantization stage76, and an entropy encoding stage78. The encoder70can also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of further blocks. The encoder70has the following stages to perform the various functions in the reconstruction path: a dequantization stage80, an inverse transform stage82, a reconstruction stage84, and a loop filtering stage86. Other structural variations of the encoder70can be used to encode the video stream50.

When the video stream50is presented for encoding, each frame56within the video stream50is processed in units of blocks. At the intra/inter prediction stage72, each block can be encoded using either intra-frame prediction, which may be within a single frame, or inter-frame prediction, which may be from frame to frame. In either case, a prediction block can be formed. In the case of intra-prediction, a prediction block can be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block can be formed from samples in one or more previously constructed reference frames.

Next, still referring toFIG. 3, the prediction block can be subtracted from the current block at the intra/inter prediction stage72to produce a residual block. The transform stage74transforms the residual block into transform coefficients in, for example, the frequency domain. Examples of block-based transforms include the Karhunen-Loève Transform (KLT), the Discrete Cosine Transform (DCT), and the Singular Value Decomposition Transform (SVD). In one example, the DCT transforms the block into the frequency domain. In the case of DCT, the transform coefficient values are based on spatial frequency, with the lowest frequency (i.e. DC) coefficient at the top-left of the matrix and the highest frequency coefficient at the bottom-right of the matrix.

The quantization stage76converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients or quantization levels. The quantized transform coefficients are then entropy encoded by the entropy encoding stage78. Entropy encoding can include using a probability distribution metric. The entropy-encoded coefficients, together with the information used to decode the block, which may include the type of prediction used, motion vectors, and quantizer value, are then output to the compressed bitstream88. The compressed bitstream88can be formatted using various techniques, such as run-length encoding (RLE) and zero-run coding.

The reconstruction path inFIG. 3(shown by the dotted connection lines) can be used to help provide that both the encoder70and a decoder100(described below) with the same reference frames to decode the compressed bitstream88. The reconstruction path performs functions that are similar to functions that take place during the decoding process that are discussed in more detail below, including dequantizing the quantized transform coefficients at the dequantization stage80and inverse transforming the dequantized transform coefficients at the inverse transform stage82to produce a derivative residual block. At the reconstruction stage84, the prediction block that was predicted at the intra/inter prediction stage72can be added to the derivative residual block to create a reconstructed block. The loop filtering stage86can be applied to the reconstructed block to reduce distortion such as blocking artifacts.

Other variations of the encoder70can be used to encode the compressed bitstream88. For example, a non-transform based encoder70can quantize the residual block directly without the transform stage74. In another embodiment, an encoder70can have the quantization stage76and the dequantization stage80combined into a single stage.

FIG. 4is a block diagram of a decoder100in accordance with another embodiment. The decoder100can be implemented in a device, such as the receiving station30described above, for example, by providing a computer software program stored in memory34. The computer software program can include machine instructions that, when executed by CPU32, cause receiving station30to decode video data in the manner described inFIG. 4. Decoder100can also be implemented as specialized hardware included, for example, in transmitting station12or receiving station30.

The decoder100, similar to the reconstruction path of the encoder70discussed above, includes in one example the following stages to perform various functions to produce an output video stream116from the compressed bitstream88: an entropy decoding stage102, a dequantization stage104, an inverse transform stage106, an intra/inter prediction stage108, a reconstruction stage110, a loop filtering stage112and a deblocking filtering stage114. Other structural variations of the decoder100can be used to decode the compressed bitstream88.

When the compressed bitstream88is presented for decoding, the data elements within the compressed bitstream88can be decoded by the entropy decoding stage102(using, for example, Context Adaptive Binary Arithmetic Decoding) to produce a set of quantized transform coefficients. The dequantization stage104dequantizes the quantized transform coefficients, and the inverse transform stage106inverse transforms the dequantized transform coefficients to produce a derivative residual block that can be identical to that created by the inverse transformation stage84in the encoder70. Using header information decoded from the compressed bitstream88, the decoder100can use the intra/inter prediction stage108to create the same prediction block as was created in the encoder70. At the reconstruction stage110, the prediction block can be added to the derivative residual block to create a reconstructed block. The loop filtering stage112can be applied to the reconstructed block to reduce blocking artifacts. The deblocking filtering stage114can be applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream116.

Other variations of the decoder100can be used to decode the compressed bitstream88. For example, the decoder100can produce the output video stream116without the deblocking filtering stage114.

FIG. 5is a block diagram of motion estimation aided noise reduction encoding for a series of frames in accordance with one embodiment of this disclosure. Implementations of motion estimation aided noise reduction encoding can include using a first noise reduction frame500, a first unencoded frame502, or both to generate a first denoised frame504. In an implementation, generating the first denoised frame504can include including a pixel from the first noise reduction frame500, a pixel from the first unencoded frame502, a pixel based on a pixel from the first noise reduction frame500and a pixel from the first unencoded frame502, or a combination thereof, in the first denoised frame504. The first unencoded frame502can be a frame from a video sequence, such as the video sequence52shown inFIG. 2, for example.

Motion estimation aided noise reduction encoding can include using the first denoised frame504to generate a first encoded frame506. Motion estimation aided noise reduction encoding can include using the first noise reduction frame500, the first denoised frame504, or both to generate a second noise reduction frame510. Generating the second noise reduction frame510can include including a pixel from the first noise reduction frame500, a pixel from the first denoised frame504, a pixel based on a pixel from the first noise reduction frame500and a pixel from the first denoised frame504, or a combination thereof, in the second noise reduction frame510.

The second noise reduction frame510, a second unencoded frame512, or both can be used to generate a second denoised frame514. The second unencoded frame512can be a second frame in the video sequence. The second denoised frame514can be used to generate a second encoded frame516. The second noise reduction frame510, the second denoised frame514, or both can be used to generate a third noise reduction frame520. The third noise reduction frame520, a third unencoded frame522, or both can be used to generate a third denoised frame524. The third unencoded frame514can be a third frame in the video sequence. The third denoised frame524can be used to generate a third encoded frame526. The first encoded frame506, the second encoded frame516, the third encoded frame526, or a combination thereof can be used to generate an encoded video stream, such as the compressed bitstream88shown inFIGS. 3 and 5.

FIG. 6is a diagram of motion estimation aided noise reduction encoding in accordance with one embodiment of this disclosure. Implementations of motion estimation aided noise reduction encoding can include identifying a frame at600, identifying a current block in the frame at610, identifying a motion vector for encoding the current block at620, denoising the block at630, identifying a motion vector for encoding the denoised block at640, generating a residual block at650, generating an encoded block at660, determining whether to encode another block at670, determining whether to encode another frame at680, or a combination thereof. In an implementation, a device, such as the transmitting station12shown inFIG. 1, can perform motion estimation aided noise reduction encoding. For example, motion estimation aided noise reduction encoding, or any portion thereof, can be implemented in an encoder, such as the encoder70shown inFIG. 3.

As an example, a frame, such as frame56shown inFIG. 2, can be identified for encoding at600. For example, the frame can include an 8×8 matrix of blocks as shown inFIG. 2. Identifying the frame can include identifying a current frame, a reference frame, a noise reduction frame, or a combination thereof.

A block (current block) in the current frame can be identified at610. For example, the current block can be identified based on Cartesian coordinates. The current block can include pixels. For example, the current block can include a 16×16 matrix of pixels.

A motion vector (MV) for encoding the current block can be identified at620. For example, the motion vector can be identified based on the current block and a reference block from a reference frame using a method of motion estimation, such as a motion search. Identifying a motion vector can include generating a prediction mode for encoding the current block, generating a sum of squared errors (SSE) for the current block, or both. Identifying the motion vector can include identifying a zero magnitude motion vector (MV0). The zero magnitude motion vector MV0can indicate a block in the reference frame that is collocated with the current block in the current frame.

The current block can be denoised at630. Denoising the current block can include identifying a noise reduction block at632and generating a denoised block at634. Identifying the noise reduction block at632can include using a noise reduction block from a noise reduction frame (e.g.,500), determining whether a difference between the current block and the reference block is noise, aligning the noise reduction block with the current block, or a combination thereof.

Determining whether a difference between the current block and the reference block is noise can include determining whether the magnitude of the motion vector is less than a threshold (T1), which can indicate noise. On a condition that the magnitude of the motion vector is greater than the threshold T1, the noise reduction block can be aligned with the current block. The motion vector can, for example, include Cartesian coordinates, such as an X coordinate, a Y coordinate, or both, and the magnitude of the motion vector can be a function of the coordinates, such as a square root of the sum of the X coordinate squared and the Y coordinate squared.

The motion vector (MV) identified at620can be a non-zero motion vector for the current block that is identified based on noise, such as random variations in pixel values in the current frame, the reference frame, or both. For example, the motion search may determine that a location of the reference block in the reference frame that best matches the current block is slightly different than the location of the current block in the current frame based on noise.

Determining whether a difference between the current block and the reference block is noise can include determining whether the difference between a sum of squared errors (SSE) of the motion vector and an SSE of a zero magnitude motion vector is less than a threshold (T2). The SSEs can be determined based on the sum of squared differences between the pixels of the reference block and the current block.

For example, the current block may include an M×N matrix of pixels, such as a 16×16 matrix of pixels, and a pixel in the current block can be expressed as B(M,N). The reference block indicated by the motion vector may include an M×N matrix of pixels, and a pixel in the reference block can be expressed as RB(M,N). The block in the reference frame indicated by the zero magnitude motion vector MV0may include an M×N matrix of pixels, and a pixel in the block in the reference frame indicated by the zero magnitude motion vector MV0can be expressed as RB0(M,N). The motion vector can indicate a row offset MVxand a column offset MVy. In an implementation, the SSE associated with the motion vector SSErmay be expressed as:
SSEr=Σx=1;y=1M;NB(x,y)−RB(x,y).  [Equation 1]

The SSE associated with the zero magnitude motion vector SSE0may be expressed as:
SSE0=Σx=1;y=1M;NB(x,y)−RB0(x,y).  [Equation 2]

On a condition that the magnitude of the motion vector identified at620is less than T1and the difference between SSErand SSE0is less than T2, the motion vector may be a small motion vector that indicates noise. On a condition that the magnitude of the motion vector is less than T1, or the difference between SSErand SSE0is greater than T2, the noise reduction block may be aligned with the current block.

At634, a denoised block can be generated using the current block and the noise reduction block. For example, a denoised block can be generated as shown inFIG. 7. As shown inFIG. 6, a motion vector for encoding the denoised block can be identified at640. In an implementation, identifying the motion vector for encoding the denoised block can include using the motion vector identified for encoding the current block at620as the motion vector for encoding the denoised block.

In an implementation, identifying the motion vector for encoding the denoised block can include determining whether to use a zero magnitude motion vector to a reference frame for encoding the denoised block. For example, the motion vector identified for encoding the current block may indicate an intra prediction mode and a zero magnitude motion vector to a reference frame can be used for encoding the denoised block. In an implementation, a zero magnitude motion vector to the reference frame may produce a smaller residual than the motion vector identified at620, and the zero magnitude motion vector can be used for encoding the denoised block. Alternatively, or in addition, identifying the motion vector for encoding the denoised block at640can include motion estimation, which can include generating a motion vector for encoding the denoised block based on the denoised block and the reference frame.

A residual block can be generated based on the denoised block at650, and an encoded block can be generated using the residual block at660. Whether to encode another block can be determined at670. For example, unencoded blocks in the current frame can be encoded. Whether to encode another frame can be determined at680. For example, the noise reduction block can be used to encode another frame, such as a future frame, in the video sequence.

Although not shown inFIG. 6, an encoded video stream, such as the compressed bit stream88shown inFIG. 3, can be generated using the encoded block(s) generated at660. The encoded video stream can be transmitted, stored, further processed, or a combination thereof. For example, the encoded video stream can be stored in a memory, such as the memory16and/or34shown inFIG. 1. The encoded video stream can also be transmitted to a decoder, such as the decoder100shown inFIG. 4.

Other implementations of the diagram of motion estimation aided noise reduction encoding as shown inFIG. 6are available. In implementations, additional elements of motion estimation aided noise reduction encoding can be added, certain elements can be combined, and/or certain elements can be removed. For example, in an implementation, motion estimation aided noise reduction encoding can include an additional element involving determining whether to denoise a block, and if the determination is not to denoise, the element at630of denoising a block can be skipped and/or omitted for one or more blocks and/or frames. Alternatively, or in addition, motion estimation aided noise reduction encoding can include controlling an amount of noise reduction, which can be based on an estimate of the amount of noise, such as an estimate of noise variance or a probability density function.

FIG. 7is a diagram of generating a denoised block (e.g., at634) in accordance with one embodiment of this disclosure. Implementations of, generating a denoised block can include identifying pixels at700, identifying a coefficient at710, applying a weight (e.g., weighting the coefficient) at720, producing a denoised pixel at730, evaluating the denoised pixel at740, processing the pixels at750, determining whether to produce another denoised pixel at760, or a combination thereof.

More specifically, as an example, pixels for generating the denoised block can be identified at700. Identifying the pixels can include identifying a current pixel (Pk) in a current block (e.g., as the block identified at610) and/or identifying a noise reduction pixel (NRPk) in a noise reduction block (e.g., the noise reduction block identified at632). Identifying the noise reduction pixel can be based on the current pixel. For example, the location of the current pixel in the current block may correspond with the location of the noise reduction pixel in the noise reduction block.

A coefficient (alpha′) can be identified at710. The coefficient can be identified based on the current pixel and the noise reduction pixel. For example, the coefficient can be identified based on a function of the magnitude of the difference between the current pixel and the noise reduction pixel. In one implementation, the coefficient may be calculated as follows:
alpha′=1/(1+(|Pk−NRPk|)/8).  [Equation 3]

The coefficient can be weighted at720. The coefficient can be weighted based on the motion vector. For example, the coefficient can be weighted based on a function of the coefficient and a magnitude of the motion vector (e.g., the motion vector identified at620). In one implementation, the magnitude of the motion vector and a threshold can be used to indicate whether the current block includes noise. If noise is indicated in the current block, the coefficient can be increased. If noise is not indicated in the current block, the coefficient can be decreased. For example, in one implementation, the weighting can be expressed as:
alpha′=(|MV|2<2*T1)→alpha′+alpha′/(3+|MV|2/10);
alpha′=(|MV|2≧8*T1)→0;
alpha′=[0,1].  [Equation 4]

A denoised pixel Pk′ can be produced at730. The denoised pixel can be produced based on a function of the current pixel Pk, the coefficient alpha′, the noise reduction pixel NRPk, or a combination thereof. In one implementation, a denoised pixel may be calculated (produced) as follows:
Pk′=alpha′*NRPk+(1−alpha′)*Pk.  [Equation 5]

The denoised pixel Pk′ can be evaluated at740. Evaluating the denoised pixel Pk′ can include determining whether to include the denoised pixel in the denoised block being generated, determining whether to include the denoised pixel in the noise reduction block being used for generating the denoised block, determining whether to include the current pixel in the denoised block being generated, determining whether to include the current pixel in the noise reduction block being used for generating the denoised block, or a combination thereof. In at least one implementation, the current pixels and/or denoised pixels can be included in a noise reduction block used for generating a next denoised block instead of or in addition to the noise reduction block being used for generating the denoised block.

The evaluation can be based on the denoised pixel and the current pixel. For example, the difference between the current pixel and the denoised pixel can be less than a threshold (T3) and can indicate a small change. In one example, if a small change is indicated, the denoised pixel Pk′ can be included in the denoised block being generated and the noise reduction block being used for generating the denoised block. In another example, if a small change is not indicated, the current pixel Pkcan be included in the denoised block being generated and the noise reduction block being used for generating the denoised block. Alternatively or additionally, if a small change is not indicated, the current pixel Pkcan be included in the denoised block being generated and the denoised pixel Pk′ can be included in the noise reduction block being used for generating the denoised block. In an implementation, the noise reduction block used for generating the denoised block can include pixels from the current block, denoised pixels, or both, and can be used for denoising a block in a future frame.

Evaluating the denoised pixel Pk′ can also include evaluating the current block. Evaluating the current block can include determining whether the sum of squared errors (SSE) of the motion vector identified at620SSEris less than a threshold (T4), which can indicate that the current block includes a shift in mean. For example, the SSErcan be less than or equal to the threshold T4and the denoised pixel Pk′ can be included in the denoised block being generated and the noise reduction block being used for generating the denoised block. In another example, the SSErcan be greater than the threshold T4and the current pixel Pkcan be included in the denoised block being generated and the noise reduction block being used for generating the denoised block.

Although described separately, evaluating the difference between the current pixel and the denoised pixel, and evaluating the SSErcan be performed in combination. For example, the difference between the current pixel and the denoised pixel can be less than a threshold (T3) and the SSErcan be less than the threshold T4, and the denoised pixel Pk′ can be included in the denoised block being generated and the noise reduction block being used for generating the denoised block.

The pixels, such as the current pixel, the denoised pixel, the noise reduction pixel, or a combination thereof, can be processed at750. Processing the pixels can include including the denoised pixel in the denoised block, including the current pixel in the denoised block, including the denoised pixel in the noise reduction block, including the current pixel in the noise reduction block, or a combination thereof. The denoised pixel, the current pixel, or a combination thereof can be included in the denoised block, the noise reduction block, or a combination thereof based on the evaluation above.

For example, in an implementation, the denoised pixel produced at730can be included in the denoised block being generated at634and the noise reduction block being used to generate the denoised block being generated at634on a condition that the evaluation indicates a small difference between the current pixel and the denoised pixel, which can be expressed as (Pk−Pk′)2<T2, and a small SSEr, which can be expressed as SSEr<T3. Including the denoised pixel in the noise reduction block can include replacing the noise reduction pixel with the denoised pixel. In an implementation, the difference between the current pixel and the denoised pixel may be greater than a threshold or the SSErmay be greater than a threshold, and the current pixel can be included in the noise reduction block, the denoised block, or both. In another implementation, the difference between the current pixel and the denoised pixel may be greater than a threshold or the SSErmay be greater than a threshold, and the denoised pixel can be included in the noise reduction block.

Although described separately for clarity, and while the terms are not used interchangeably, the current block and the denoised block can be the same block. For example, including a denoised pixel in the denoised block can include replacing the current pixel with the denoised pixel in the current block. The noise reduction block can be included in a noise reduction frame (e.g.,510) and can be used for encoding another frame, such as a future frame, in the video sequence.

Other implementations of the diagram of generating a denoised block as shown inFIG. 7are available. In implementations, additional elements of generating a denoised block can be added, certain elements can be combined, and/or certain elements can be removed. For example, in an implementation, generating a denoised block at730can include a combination of evaluating the pixels as shown at740and processing the pixels as shown at750.

AlthoughFIG. 7describes an example of denoising using one noise reduction block for simplicity, implementations of this disclosure can include using any number of noise reduction blocks. For example, motion estimation aided noise reduction can include using multiple blocks from N previous frames, which may be denoised frames, as noise reduction blocks.FIG. 8is a diagram of generating a denoised block using multiple noise reduction blocks in accordance with one embodiment of this disclosure. Motion estimation aided noise reduction can include identifying noise reduction blocks at800, identifying taps at810, applying weights (e.g., weighting the taps) at820, producing a denoised pixel at830, evaluating the denoised pixel at840, processing the pixels at850, determining whether to produce another denoised pixel at860, or a combination thereof.

More specifically, as an example, noise reduction blocks can be identified at800. Each noise reduction block (NRBi,j) can be associated with a block (j) of a frame (i). For example, a first noise reduction block NRB1,jcan be associated with a block from a first frame, which may be a previous frame in the video sequence, and a second noise reduction block NRB2,jcan be associated with a block from a second frame, which may be a previous frame in the video sequence. The location of the first noise reduction block in the first frame can correspond with the location of the second noise reduction block in the second frame. The noise reduction blocks can be denoised blocks. The noise reduction blocks can be unencoded blocks. Identifying the noise reduction block can include aligning each of the noise reduction blocks with the current block. Aligning the noise reduction blocks can include finding the noise reduced block which, according to the motion vectors, match the current block. In an implementation, aligning can include adjusting positions of the noise reduction blocks within an noise reduction frame based on the motion vectors. In an implementation, a noise reduction pixel (NRPi,j,k) can be a pixel (k) in a noise reduction block (j) in a frame (i).

Taps can be identified at810. A tap (TAPi,j,k) can be associated with a pixel (k) of a block (j) of a frame (i) and can indicate a pixel (k) value weighting metric. For example, a tap TAP0,j,kcan be associated with the current block of a current frame and can indicate a metric for weighting the current pixel Pk. A tap TAP1,j,kcan be associated with the first noise reduction block NRB1,1and can indicate a metric for weighting a noise reduction pixel in the first noise reduction block NRB1. A tap TAP2,j,kcan be associated with the second noise reduction block NRB2and can indicate a metric for weighting a noise reduction pixel in the second noise reduction block.

The taps can be weighted at820. The taps can indicate equal pixel value weights. For example, the magnitude of the motion vector can be zero and the weighting metrics can be equal. In an implementation, the taps can be weighted based on proximity, such as in cases where the magnitude of the motion vector is greater than zero. For example, a first tap can be associated with a first noise reduction block. The first noise reduction block can be based on a first previous frame, which can precede the current frame in the video sequence. A second tap can be associated with a second noise reduction block. The second noise reduction block can be based on a second previous frame, which can precede the first previous frame in the video sequence. The magnitude of the motion vector can be non-zero and the first tap can weighted more heavily than the second tap.

A denoised pixel Pk′ can be produced at830. Producing the denoised pixel can include identifying a noise reduction pixel NRPi,j,kfrom each noise reduction block NRBi,jbased on the location of the current pixel Pkin the current block of the current frame. Producing a denoised pixel Pk′ can include determining a weighted pixel value NRPi,j,k′ for each noise reduction pixel NRPi,j,kbased on an associated tap, determining a weighted pixel value for the current pixel, and determining a sum of the weighted pixel values. For example, a first tap TAP1,1,kcan be associated with the first noise reduction block NRB1,1and can indicate a metric of ¼ for weighting a noise reduction pixel NRP1,1,1in the first noise reduction block. For example, producing a denoised pixel using N taps may be expressed as:
Pk′=TAP0,1,k*Pk+Σj=1N(TAPi,j,k*NRPi,j,k).  [Equation 6]

Producing the denoised pixel can include determining the variance among the weighted pixel values. For example, the variance can be the difference between the sum of the weighted pixel values and the mean of the weighted pixel values. Determining the variance may be expressed as:
Var(k)=(Σj=1N(NRPi,j,k2))−(1/NΣj=1NNRPi,j,k)2.  [Equation 7]

The denoised pixel can be evaluated at840. The evaluation can be based on the denoised pixel and the current pixel. For example, the difference between the current pixel and the denoised pixel can be less than a threshold and can indicate a small change. The evaluation can include determining whether the variance is less than a threshold. For example, the variance can be less than a threshold and can indicate a small variance.

The pixels can be processed at850based on the indication of small change and/or the indication of small variance. Processing the pixels can include including the denoised pixel in the denoised block, including the current pixel in the denoised block, or a combination thereof. For example, the evaluation can indicate a small change and a small variance, and the denoised pixel can be included in the denoised block. Processing the pixels can include including the denoised pixel in a noise reduction block, including the current pixel in a noise reduction block, or a combination thereof. For example, the denoising can include using four noise reducing blocks, (e.g., identified at800), and the denoised pixel can be included in the first noise reducing block.

Other implementations of the diagram of generating a denoised block using multiple noise reduction blocks as shown inFIG. 8are available. In implementations, additional elements of generating a denoised block using multiple noise reduction blocks can be added, certain elements can be combined, and/or certain elements can be removed. For example, in an implementation, generating a denoised block using multiple noise reduction blocks can include combining in one element the evaluation of the pixels (as shown at840) and the processing of the pixels (as shown at850).

Motion estimation aided noise reduction encoding, or any portion thereof, can be implemented in a device, such as the transmitting station12shown inFIG. 1. For example, an encoder, such as the encoder70shown inFIG. 3, can implement motion estimation aided noise reduction encoding, or any portion thereof, using instruction stored on a tangible, non-transitory, computer readable media, such as memory34shown inFIG. 1.

Further, for simplicity of explanation, although the figures and descriptions herein may include sequences or series of steps or stages, elements of the methods disclosed herein can occur in various orders and/or concurrently. Additionally, elements of the methods disclosed herein may occur with other elements not explicitly presented and described herein. Furthermore, not all elements of the methods described herein may be required to implement a method in accordance with the disclosed subject matter.

The embodiments of encoding and decoding described above illustrate some exemplary encoding and decoding techniques. However, it is to be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or any other processing or change of data.

The embodiments of the transmitting station12and/or the receiving station30(and the algorithms, methods, instructions, etc. stored thereon and/or executed thereby) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of the transmitting station12and the receiving station30do not necessarily have to be implemented in the same manner.

Further, in one embodiment, for example, the transmitting station12or the receiving station30can be implemented using a general purpose computer or general purpose/processor with a computer program that, when executed, carries out any of the respective methods, algorithms and/or instructions described herein. In addition or alternatively, for example, a special purpose computer/processor can be utilized which can contain specialized hardware for carrying out any of the methods, algorithms, or instructions described herein.

The transmitting station12and receiving station30can, for example, be implemented on computers in a real-time video system. Alternatively, the transmitting station12can be implemented on a server and the receiving station30can be implemented on a device separate from the server, such as a hand-held communications device. In this instance, the transmitting station12can encode content using an encoder70into an encoded video signal and transmit the encoded video signal to the communications device. In turn, the communications device can then decode the encoded video signal using a decoder100. Alternatively, the communications device can decode content stored locally on the communications device, for example, content that was not transmitted by the transmitting station12. Other suitable transmitting station12and receiving station30implementation schemes are available. For example, the receiving station30can be a generally stationary personal computer rather than a portable communications device and/or a device including an encoder70may also include a decoder100.