Patent Description:
One technique for compression uses a reference frame to generate a prediction block corresponding to a current block to be encoded. Differences between the prediction block and the current block can be encoded, instead of the values of the current block themselves, to reduce the amount of data encoded.

<CIT> discloses implementations of coding video data with an alternate reference frame generated using a temporal filter. The alternate reference frame is generated by determining a first weighting factor, for each corresponding block of a respective frame of a filter set, that represents a temporal correlation of the block with the corresponding block, determining a second weighting factor, for each pixel for each corresponding block of the respective frame of the filter set, that represents a temporal correlation of the pixel to a spatially-correspondent pixel in the block, determining a filter weight for each pixel in the block and for each spatially-correspondent pixel is each corresponding block based on the first weighting factor and the second weighting factor, and generating a weighted average pixel value for each pixel position in the block to form a block of the alternate reference frame based on the filter weights.

A first aspect of this disclosure is a method for generating and using an alternate reference frame (ARF). The method includes selecting an anchor frame and video frames from a source input video stream, where the anchor frame includes an anchor block, and the anchor block includes anchor pixels, and identifying, for the anchor block of the anchor frame, respective reference blocks in the video frames. The method further includes, for each pixel of a plurality of the anchor pixels, determining, for the anchor pixel and using an anchor patch, respective distances between the anchor pixel and respective co-located reference pixels of the respective reference blocks, where the anchor patch includes a set of pixels in the anchor frame in a neighbourhood of the anchor pixel, and a respective distance, of the respective distances, between the anchor pixel and a respective co-located reference pixel in the respective reference block is determined using a distance between the anchor patch and a co-located patch around the respective co-located reference pixel in the respective video frame, wherein the distance between the anchor patch and the co-located patch is determined using distances between each of the anchor patch pixels and a corresponding reference pixel of the co-located patch, determining, using the respective distances, respective weights, and determining, using the respective weights, the anchor pixel and the respective co-located reference pixels of the respective reference blocks, an ARF pixel within the ARF frame that is co-located with the anchor pixel. The method also includes encoding, in a compressed bitstream, the ARF.

A second aspect is an apparatus for using an alternate reference frame (ARF). The apparatus includes a memory and a processor. The processor is configured to execute instructions stored in the memory to carry out a method according to the first aspect of the disclosure.

A third aspect is a method for decoding using an alternate reference frame (ARF). The method includes decoding, from a compressed bitstream, the ARF, and decoding, using the ARF, frames from the compressed bitstream. The ARF was generated and encoded in the compressed bitstream by an encoder according to the first aspect of the disclosure.

A fourth aspect is an apparatus for decoding using an alternate reference frame (ARF). The apparatus includes a memory and a processor. The processor is configured to execute instructions stored in the memory to carry out a method according to the third aspect.

In each of these aspects, other ARF pixels may be similarly determined using additional anchor blocks of the anchor frame, each having anchor pixels.

These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying figures.

The description herein makes reference to the accompanying drawings described below wherein like reference numerals refer to like parts throughout the several views unless otherwise noted.

A video stream can be compressed by a variety of techniques to reduce bandwidth required transmit or store the video stream. A video stream can be encoded into a bitstream, which involves compression, and is then transmitted to a decoder that can decode or decompress the video stream to prepare it for viewing or further processing. Compression of the video stream often exploits spatial and/or temporal correlation of video signals through spatial and/or motion compensated prediction. Motion compensated prediction, for example, uses one or more motion vectors to generate a block (also called a prediction block) that resembles a current block to be encoded using previously encoded and decoded pixels. By encoding the motion vector(s), and the difference between the two blocks, a decoder receiving the encoded signal can re-create the current block. Motion compensated prediction may also be referred to as inter prediction.

Each motion vector used to generate a prediction block in the inter-prediction process refers to one or more frames (also referred to as reference frames) other than a current frame. Reference frames can be located before or after the current frame in the sequence of the video stream and may be frames that are reconstructed before being used as reference frames. In some cases, there may be three or more reference frames used to encode or decode blocks of the current frame of the video sequence. One may be a frame that is referred to as a golden frame. Another may be a most recently encoded or decoded frame. Another may be an Alternative Reference Frame (also referred to in this disclosure as an ARF).

An alternative reference frame is a reference frame usable for backwards prediction. While some ARFs are displayable by a decoder, implementations according to this disclosure relate ARFs that may not be displayed by a decoder because they do not directly correspond to a frame in the source video stream.

One or more forward and/or backward reference frames can be used to encode or decode a block. The efficacy of a reference frame when used to encode or decode a block within a current frame can be measured based on a resulting signal-to-noise ratio or other measures of rate-distortion.

As mentioned above, encoding video frames can occur, for example, using so-called "alternate reference frames" (ARFs) that may not be temporally neighboring to the frames coded immediately before or after them.

An ARF, according to implementations of this disclosure, can be a synthesized frame that does not occur in the input video stream and can be used for prediction. This disclosure related to synthesized alternate reference frames. A synthesized ARF is simply referred to as ARF in this disclosure.

An ARF is a frame of image data that is encoded into the bitstream and serves to improve the encoding (and the decoding) of other transmitted frames. An ARF can be used to provide a temporal filtered reference frame that can be used to filter out acquisition noise within one or more source frames.

Unlike a conventional reference frame, an ARF is not shown to the user after decoding. The ARF may not have the same dimensions as the video stream's raw image frames or the frames displayed to the user. Instead, the ARF serves as a predictor, giving frames a better predictive choice than actual past or future frames might offer.

Creating the best possible ARF is typically a task that is left to the encoder, which then encodes the ARF in a compressed bitstream for use by a decoder. This provides a benefit in that the decoder need not re-perform the computations used to create the ARF. Computationally-expensive processes can be used by the encoder to derive the ARF, thus permitting faster, lighter and more efficient decoding.

The alternate reference frame (ARF) can be rendered (i.e., synthesized, created, etc.) by applying temporal filtering to several original frames. The several original frames can be consecutive frames. Thus, common information of the several original frames can be captured in the ARF, which is encoded, in addition to the regular frames (i.e., the frames of the source video stream), in the compressed bitstream. The reconstructed ARF can serve as one motion-compensated reference frame, alongside other regular reconstructed frames. Some video codecs (such as VP8, VP9, and AV1) employ an ARF to achieve significant compression performance gains.

The temporal filtering can capture the common information across the consecutive frames. The efficacy of the temporal filtering technique used can substantially impact the overall compression performance. Reference frames that are used for generating an ARF (such as described with respect to <FIG>) are to be differentiated from reference frames that are used for inter-prediction. That is, for example, such reference frames may not be added to a reference frame buffer and/or used for encoding and decoding other video frames into a compressed bitstream. That is, these reference frames may not necessarily be used for determining motion vectors and/or residuals that are encoded in the compressed bitstream.

In an approach of creating an ARF, temporal filtering can be used to evaluate the similarity between two blocks, an anchor block and a reference block of a reference frame, aligned in a motion trajectory, to determine a weight coefficient to be used for the pixels of the reference block. The weight is uniformly applied to all the pixels in the block to create a temporally filtered block. Such an approach largely ignores the variation in statistics across the processing block unit. For example, in a <NUM>×<NUM> processing block unit, there are <NUM> pixels. In this approach, the same weight is applied to each of the <NUM> pixels. As such, this approach ignores (e.g., does not make use of, is not sensitive to, etc.) the local information in the neighborhoods of each of the <NUM> pixels.

Implementations according to this disclosure use adaptive temporal filtering. Adaptive temporal filtering aligns the blocks (e.g., <NUM>×<NUM>-pixel blocks) in a motion trajectory, as further described with respect to <FIG>. For example, given an anchor block, adaptive temporal filtering finds motion-aligned reference blocks in reference frames. However, instead of, as described with respect to the approach of applying a weight coefficient uniformly to each of the pixels of the block, each pixel is processed individually taking into consideration the local variations in the neighborhood of the pixel.

In an example, a pixel patch (e.g., a pixel patch of size <NUM>×<NUM>, or simply, a <NUM>×<NUM>-pixel patch) surrounding an anchor pixel can be used to determine a weight that is used for a reference pixel (i.e., a pixel in a reference frame) when determining the value of an ARF pixel in the ARF that is co-located with the anchor pixel. A distance (between the pixel patch in the anchor frame and a corresponding (e.g., based on motion search) patch in the reference frame can then be used to determine the weight for a pixel of the reference frame. The weight of the target pixel is then used in the temporal filtering, as further described below. The distance can be in L2 norm (i.e., the mean squared error). The distance can be evaluated to decide the weight coefficient value per pixel.

As such, an encoder according to implementations of this disclosure can identity temporal consistencies at the pixel level (i.e., on a per-pixel basis), thereby only filtering those pixels that belong to the same motion trajectory as a target pixel and leaving out those that are not from the same motion trajectory (e.g., pixels that may be parts of objects other than the object of the target pixel). The temporal filtering described herein to generate an alternate reference frame can better preserve common information in the frames that are used to generate the ARF, thereby reducing any unique noise that may be embedded in a source frame that is used as an anchor frame. The noise can be acquisition noise related to a video or image capture device.

In some implementations, and as further described below, the collocated luminance and chrominance components can be jointly considered to better classify whether the aligned pixels belong to the same motion trajectory.

Implementations according to this disclosure can improve temporal filtering accuracy, which can result in substantial compression performance gain.

References to "pixel value" can be understood to mean, as the context makes clear, the pixel value of at least one of the color components of the pixel. For example, in a case where the RGB color system is used, then "pixel value" can mean, the red value of the pixel, the green value of the pixel, the blue value of the pixel, or a combination thereof. In the case where the YUV color system is used, then a pixel value can mean, the luminance (i.e., Y) value of the pixel, one of the color-difference chrominance components (i.e., U and/or V), or a combination thereof. As such, "pixel value" can be one or more color component values.

Further details of adaptive temporal filtering for alternate reference frame rendering are described herein with initial reference to a system in which the teachings herein can be implemented.

<FIG> is a schematic of a video encoding and decoding system <NUM>. A transmitting station <NUM> can be, for example, a computer having an internal configuration of hardware such as that described in <FIG>. However, other suitable implementations of the transmitting station <NUM> are possible. For example, the processing of the transmitting station <NUM> can be distributed among multiple devices.

A network <NUM> can connect the transmitting station <NUM> and a receiving station <NUM> for encoding and decoding of the video stream. Specifically, the video stream can be encoded in the transmitting station <NUM> and the encoded video stream can be decoded in the receiving station <NUM>. The network <NUM> can be, for example, the Internet. The network <NUM> can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), cellular telephone network or any other means of transferring the video stream from the transmitting station <NUM> to, in this example, the receiving station <NUM>.

The receiving station <NUM>, in one example, can be a computer having an internal configuration of hardware such as that described in <FIG>. However, other suitable implementations of the receiving station <NUM> are possible. For example, the processing of the receiving station <NUM> can be distributed among multiple devices.

Other implementations of the video encoding and decoding system <NUM> are possible. For example, an implementation can omit the network <NUM>. In another implementation, a video stream can be encoded and then stored for transmission at a later time to the receiving station <NUM> or any other device having a non-transitory storage medium or memory. In one implementation, the receiving station <NUM> receives (e.g., via the network <NUM>, a computer bus, and/or some communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, a real-time transport protocol (RTP) is used for transmission of the encoded video over the network <NUM>. In another implementation, a transport protocol other than RTP may be used, e.g., a Hypertext Transfer Protocol (HTTP) based video streaming protocol.

When used in a video conferencing system, for example, the transmitting station <NUM> and/or the receiving station <NUM> may include the ability to both encode and decode a video stream as described below. For example, the receiving station <NUM> could be a video conference participant who receives an encoded video bitstream from a video conference server (e.g., the transmitting station <NUM>) to decode and view and further encodes and transmits its own video bitstream to the video conference server for decoding and viewing by other participants.

<FIG> is a block diagram of an example of a computing device <NUM> that can implement a transmitting station or a receiving station. For example, the computing device <NUM> can implement one or both of the transmitting station <NUM> and the receiving station <NUM> of <FIG>. The computing device <NUM> can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like.

A CPU <NUM> in the computing device <NUM> can be a central processing unit. Alternatively, the CPU <NUM> can be any other type of device, or multiple devices, capable of manipulating or processing information now-existing or hereafter developed. Although the disclosed implementations can be practiced with one processor as shown, e.g., the CPU <NUM>, advantages in speed and efficiency can be achieved using more than one processor.

A memory <NUM> in computing device <NUM> can be a read only memory (ROM) device, a random-access memory (RAM) device, other type of memory, or a combination thereof. Any other suitable type of storage device or non-transitory storage medium can be used as the memory <NUM>. The memory <NUM> can include code and data <NUM> that is accessed by the CPU <NUM> using a bus <NUM>. The memory <NUM> can further include an operating system <NUM> and application programs <NUM>, the application programs <NUM> including at least one program that permits the CPU <NUM> to perform the methods described here. For example, the application programs <NUM> can include applications <NUM> through N, which further include a video coding application that performs the methods described here. Computing device <NUM> can also include a secondary storage <NUM>, which can, for example, be a memory card used with a mobile computing device. Because the video communication sessions may contain a significant amount of information, they can be stored in whole or in part in the secondary storage <NUM> and loaded into the memory <NUM> as needed for processing.

The computing device <NUM> can also include one or more output devices, such as a display <NUM>. The display <NUM> may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display <NUM> can be coupled to the CPU <NUM> via the bus <NUM>. Other output devices that permit a user to program or otherwise use the computing device <NUM> can be provided in addition to or as an alternative to the display <NUM>. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display or light emitting diode (LED) display, such as an organic LED (OLED) display.

The computing device <NUM> can also include or be in communication with an image-sensing device <NUM>, for example a camera, or any other image-sensing device <NUM> now existing or hereafter developed that can sense an image such as the image of a user operating the computing device <NUM>. The image-sensing device <NUM> can be positioned such that it is directed toward the user operating the computing device <NUM>. In an example, the position and optical axis of the image-sensing device <NUM> can be configured such that the field of vision includes an area that is directly adjacent to the display <NUM> and from which the display <NUM> is visible.

The computing device <NUM> can also include or be in communication with a sound-sensing device <NUM>, for example a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near the computing device <NUM>. The sound-sensing device <NUM> can be positioned such that it is directed toward the user operating the computing device <NUM> and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device <NUM>.

Although <FIG> depicts the CPU <NUM> and the memory <NUM> of the computing device <NUM> as being integrated into one unit, other configurations can be utilized. The operations of the CPU <NUM> can be distributed across multiple machines (wherein individual machines can have one or more of processors) that can be coupled directly or across a local area or other network. The memory <NUM> can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of the computing device <NUM>. Although depicted here as one bus, the bus <NUM> of the computing device <NUM> can be composed of multiple buses. Further, the secondary storage <NUM> can be directly coupled to the other components of the computing device <NUM> or can be accessed via a network and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards. The computing device <NUM> can thus be implemented in a wide variety of configurations.

<FIG> is a diagram of an example of a video stream <NUM> to be encoded and subsequently decoded. The video stream <NUM> includes a video sequence <NUM>. At the next level, the video sequence <NUM> includes a number of adjacent frames <NUM>. While three frames are depicted as the adjacent frames <NUM>, the video sequence <NUM> can include any number of adjacent frames <NUM>. The adjacent frames <NUM> can then be further subdivided into individual frames, e.g., a frame <NUM>. At the next level, the frame <NUM> can be divided into a series of planes or segments <NUM>. The segments <NUM> can be subsets of frames that permit parallel processing, for example. The segments <NUM> can also be subsets of frames that can separate the video data into separate colors. For example, a frame <NUM> of color video data can include a luminance plane and two chrominance planes. The segments <NUM> may be sampled at different resolutions.

Whether or not the frame <NUM> is divided into segments <NUM>, the frame <NUM> may be further subdivided into blocks <NUM>, which can contain data corresponding to, for example, 16x16 pixels in the frame <NUM>. The blocks <NUM> can also be arranged to include data from one or more segments <NUM> of pixel data. The blocks <NUM> can also be of any other suitable size such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.

<FIG> is a block diagram of an encoder <NUM> according to implementations of this disclosure. The encoder <NUM> can be implemented, as described above, in the transmitting station <NUM> such as by providing a computer software program stored in memory, for example, the memory <NUM>. The computer software program can include machine instructions that, when executed by a processor such as the CPU <NUM>, cause the transmitting station <NUM> to encode video data in the manner described in <FIG>. The encoder <NUM> can also be implemented as specialized hardware included in, for example, the transmitting station <NUM>. In one particularly desirable implementation, the encoder <NUM> is a hardware encoder.

The encoder <NUM> has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or compressed bitstream <NUM> using the video stream <NUM> as input: an intra/inter prediction stage <NUM>, a transform stage <NUM>, a quantization stage <NUM>, and an entropy encoding stage <NUM>. The encoder <NUM> may also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of future blocks. In <FIG>, the encoder <NUM> has the following stages to perform the various functions in the reconstruction path: a dequantization stage <NUM>, an inverse transform stage <NUM>, a reconstruction stage <NUM>, and a loop filtering stage <NUM>. Other structural variations of the encoder <NUM> can be used to encode the video stream <NUM>.

When the video stream <NUM> is presented for encoding, respective frames <NUM>, such as the frame <NUM>, can be processed in units of blocks. At the intra/inter prediction stage <NUM>, respective blocks can be encoded using intra-frame prediction (also called intra-prediction) or inter-frame prediction (also called inter-prediction). In any case, a prediction block can be formed. In the case of intra-prediction, a prediction block may be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block may be formed from samples in one or more previously constructed reference frames. The designation of reference frames for groups of blocks is discussed in further detail below.

Next, still referring to <FIG>, the prediction block can be subtracted from the current block at the intralinter prediction stage <NUM> to produce a residual block (also called a residual). The transform stage <NUM> transforms the residual into transform coefficients in, for example, the frequency domain using block-based transforms. The quantization stage <NUM> converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients, using a quantizer value or a quantization level. For example, the transform coefficients may be divided by the quantizer value and truncated. The quantized transform coefficients are then entropy encoded by the entropy encoding stage <NUM>. The entropy-encoded coefficients, together with other information used to decode the block, which may include for example the type of prediction used, transform type, motion vectors and quantizer value, are then output to the compressed bitstream <NUM>. The compressed bitstream <NUM> can be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream <NUM> can also be referred to as an encoded video stream or encoded video bitstream, and the terms will be used interchangeably herein.

The reconstruction path in <FIG> (shown by the dotted connection lines) can be used to ensure that the encoder <NUM> and a decoder <NUM> (described below) use the same reference frames to decode the compressed bitstream <NUM>. The reconstruction path performs functions that are similar to functions that take place during the decoding process that are discussed in more detail below, including dequantizing the quantized transform coefficients at the dequantization stage <NUM> and inverse transforming the dequantized transform coefficients at the inverse transform stage <NUM> to produce a derivative residual block (also called a derivative residual). At the reconstruction stage <NUM>, the prediction block that was predicted at the intralinter prediction stage <NUM> can be added to the derivative residual to create a reconstructed block. The loop filtering stage <NUM> can be applied to the reconstructed block to reduce distortion such as blocking artifacts.

Other variations of the encoder <NUM> can be used to encode the compressed bitstream <NUM>. For example, a non-transform based encoder can quantize the residual signal directly without the transform stage <NUM> for certain blocks or frames. In another implementation, an encoder can have the quantization stage <NUM> and the dequantization stage <NUM> combined in a common stage.

<FIG> is a block diagram of a decoder <NUM> according to implementations of this disclosure. The decoder <NUM> can be implemented in the receiving station <NUM>, for example, by providing a computer software program stored in the memory <NUM>. The computer software program can include machine instructions that, when executed by a processor such as the CPU <NUM>, cause the receiving station <NUM> to decode video data in the manner described in <FIG>. The decoder <NUM> can also be implemented in hardware included in, for example, the transmitting station <NUM> or the receiving station <NUM>.

The decoder <NUM>, similar to the reconstruction path of the encoder <NUM> discussed above, includes in one example the following stages to perform various functions to produce an output video stream <NUM> from the compressed bitstream <NUM>: an entropy decoding stage <NUM>, a dequantization stage <NUM>, an inverse transform stage <NUM>, an intra/inter prediction stage <NUM>, a reconstruction stage <NUM>, a loop filtering stage <NUM> and a post filtering stage <NUM>. Other structural variations of the decoder <NUM> can be used to decode the compressed bitstream <NUM>.

When the compressed bitstream <NUM> is presented for decoding, the data elements within the compressed bitstream <NUM> can be decoded by the entropy decoding stage <NUM> to produce a set of quantized transform coefficients. The dequantization stage <NUM> dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and the inverse transform stage <NUM> inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the inverse transform stage <NUM> in the encoder <NUM>. Using header information decoded from the compressed bitstream <NUM>, the decoder <NUM> can use the intralinter prediction stage <NUM> to create the same prediction block as was created in the encoder <NUM>, e.g., at the intralinter prediction stage <NUM>. At the reconstruction stage <NUM>, the prediction block can be added to the derivative residual to create a reconstructed block. The loop filtering stage <NUM> can be applied to the reconstructed block to reduce blocking artifacts.

Other filtering can be applied to the reconstructed block. In this example, the post filtering stage <NUM> may be a deblocking filter that is applied to the reconstructed block to reduce blocking distortion. The result is output as the output video stream <NUM>. The output video stream <NUM> can also be referred to as a decoded video stream, and the terms will be used interchangeably herein. Other variations of the decoder <NUM> can be used to decode the compressed bitstream <NUM>. For example, the decoder <NUM> can produce the output video stream <NUM> without the post filtering stage <NUM>.

As is known in the art, a reference frame buffer may store reference frames (such as an ARF) used to encode or decode blocks of frames of a video sequence. For example, reference frames may be identified as a last frame LAST_FRAME, a golden frame GOLDEN_FRAME, or an alternative reference frame ALTREF_FRAME. The reference buffer can include additional reference frames. In an example, up to eight reference frames can be stored in the reference frame buffer and used in inter prediction. A frame header of a reference frame may include a virtual index to a location within the reference frame buffer at which the reference frame is stored. A reference frame mapping can map the virtual index of a reference frame to a physical index of memory at which the reference frame is stored. Where two reference frames are the same frame, those reference frames will have the same physical index even if they have different virtual indexes. The number and type of reference frames stored within a reference frame buffer may differ.

The reference frames stored in a reference frame buffer can be used to identify motion vectors for predicting blocks of frames to be encoded or decoded. Different reference frames may be used depending on the type of prediction used to predict a current block of a current frame. For example, in bi-prediction, blocks of the current frame can be forward predicted using either frame stored as the LAST_FRAME or the GOLDEN_FRAME, and backward predicted using a frame stored as the ALTREF_FRAME. More reference frames can also be available.

<FIG> is a diagram of a group of pictures (GOP) <NUM> in a display order of the video sequence according to implementations of this disclosure. The GOP <NUM> can also be referred to as a group of frames. The GOP <NUM> includes a consecutive group of frames of a video stream. In this example, the GOP <NUM> includes eight frames, namely the frames <NUM>-<NUM>. However, a GOP can have more or fewer pictures (i.e., video frames). The number of frames forming each group of pictures can vary according to the video spatial and/or temporal characteristics and other encoder configurations, such as the key frame interval selected for random access or error resilience, for example.

The GOP <NUM> is also shown to include an ARF <NUM>. The ARF <NUM> is shown as shaded as it is not a frame of the video stream. The ARF <NUM> can be thought of as logically being at a location in the video sequence following the frame <NUM>. The ARF <NUM> is a derived (i.e., constructed, synthesized, etc.) frame according to implementations of this disclosure.

In an example, the frame <NUM>, which is the first frame of the GOP <NUM>, can be referred to as a key frame. No block within the frame <NUM> is inter predicted. The predicted blocks within the frame <NUM> may only be predicted using intra prediction. In another example, the frame <NUM> can be referred to as an overlay frame, which is an inter-predicted frame that can be a reconstructed frame of a previous group of frames. In an inter-predicted frame, at least some of the predicted blocks can be predicted using inter prediction.

The coding order for a GOP can differ from the display order. This allows a frame located after a current frame in the video sequence to be used as a reference frame for encoding the current frame. A decoder, such as the decoder <NUM>, can share a common group coding structure with an encoder, such as the encoder <NUM>. A group coding structure assigns different roles that respective frames within the group may play in the reference buff (e.g., a last frame, an alternative reference frame, etc.) and defines or indicates the coding order for the frames within a group.

<FIG> is a diagram of an example of a coding order <NUM> for the group of frames of <FIG>. Because the encoding and decoding order is the same, the order shown in <FIG> is generally referred to herein as a coding order. The key or overlay frame <NUM> can be designated as the golden frame (e.g., GOLDEN_FRAME) in a reference frame buffer. The ARF <NUM> can be encoded next and is designated as an alternative reference frame (e.g., ALTREF_FRAME) in the reference frame buffer. In this coding order, the ARF <NUM> is coded out of the display order after the frame <NUM> so as to provide a backward reference frame for each of the remaining frames <NUM>-<NUM>. Blocks of the ARF <NUM> can be inter-predicted (for example, the frame <NUM> can serve as an available reference frame for at least some blocks of the ARF <NUM>), intra-predicted, or a combination thereof. The ARF <NUM> can be encoded using, as reference frames, the golden frame and reference frames reconstructed from frames of prior GOPs.

<FIG> is only one example of a coding order for a group of frames. The important aspect of the coding order <NUM> is that the ARF <NUM>, which is not a frame that is part of the video stream, is encoded (e.g., is a second frame to be encoded) in a compressed bitstream that is received by a decoder, and that the ARF is encoded before most of the frames of the GOP <NUM>.

While the GOP <NUM> is described as including eight frames and that only one ARF is created, implementations according to this disclosure are not so limited. For example, more than one ARF can be generated. As such, given a group of pictures that includes N frames, an encoder can encode N+M frames for the GOP, where M is the number of alternate reference frames. A decoder can decode the N+M frames and display only the N frames of the GOP. The M alternate frames, along with other reference frames, can be used in decoding at least some of the N frames.

By way of background information in order to better understand the invention, <FIG> is an example <NUM> of a technique of generating an alternate reference frame. The example <NUM> illustrates temporal filtering through five consecutive source frames to generate an ARF <NUM>. The five consecutive source frames include an anchor frame <NUM>, two predecessor frames <NUM>-<NUM>, and two successor frames <NUM>-<NUM>. The anchor frame <NUM> is used for generating the ARF <NUM>. The anchor frame <NUM> can be divided into blocks. Each of the blocks can be of size M×N. In an example, the block can be of size <NUM>×<NUM>; however, other sizes are possible. One such block is an anchor block <NUM>.

In an example, the anchor frame <NUM> can be the frame <NUM> of the GOP <NUM> of <FIG>. That is, the anchor frame <NUM> can be the last frame of a group of pictures. As such, the frames <NUM> and <NUM> correspond, respectively, to the frames <NUM> and <NUM> of <FIG>; and the ARF <NUM> can correspond to the ARF <NUM> of <FIG>. Similarly, the frames <NUM> and <NUM> correspond, respectively, to a first frame and a second frame of the GOP (not shown) that follows the GOP <NUM>.

In the example <NUM>, a block corresponding to (i.e., co-located with) each M×N block (e.g., each <NUM>×<NUM> block) of the anchor block is generated as described below. For example, an ARF block <NUM> of the ARF <NUM> corresponds to the anchor block <NUM>. As such, each M×N block in the anchor frame <NUM> can be processed (i.e., by an encoder, such as the encoder <NUM> of <FIG>) as an operating unit. For each of the frames to be used for generating the ARF, a motion search can be performed (e.g., by an intra/inter prediction stage, such as the intra/inter prediction stage <NUM> of <FIG>) to find, for the anchor block, respective reference blocks in each of the frames. As five frames are used in the example <NUM>, five reference blocks are found; namely, reference blocks <NUM>, <NUM>, <NUM>, and <NUM> in the frames <NUM>, <NUM>, <NUM>, and <NUM>, respectively.

In this example <NUM>, the distance between an anchor block (e.g., the anchor block <NUM>) and a reference block (e.g., each of the blocks <NUM>-<NUM>) in a frame can be used to determine a weight for that frame. The distance can be in L2 norm (e.g., mean squared error). The distance can be indicative of the level of distortion between the anchor block and the reference block.

Without loss of generality, let B denote an anchor block in the anchor frame and let R denote a reference block in a frame. As such, B designates the anchor block <NUM>; and R(n-<NUM>), R(n-<NUM>), R(n+<NUM>), and R(n+<NUM>) designate, respectively, the reference blocks <NUM>, <NUM>, <NUM>, and <NUM>.

To calculate the distance, a reference block (e.g., R(n-<NUM>)) is subtracted, pixel-wise, from the anchor block (i.e., B) and the sum of squares of the differences are summed. As such, the distance, D(B, R(k)), between the block B and a block R(k), where k corresponds to each of the reference blocks <NUM>-<NUM>, can be calculated using formula (<NUM>): <MAT>.

In an example, if the distance is greater than a threshold, the block can be ignored. For example, a weight of zero can be assigned to the block if the sum is greater than the threshold. As such, the weight can be determined using a clamping function as shown in formula (<NUM>). In formula (<NUM>), the threshold used is <NUM>; however, other threshold values can be used. In another example, a threshold of <NUM> can be used.

In the formula (<NUM>), c(k) is the weight to be used for the pixels of the reference block in the frame k, where k corresponds to the reference blocks R(n-<NUM>), R(n-<NUM>), R(n+<NUM>), and R(n+<NUM>). The clamping function clamp() of formula (<NUM>) takes <NUM> arguments: the first argument corresponds to a lower limit; the second argument corresponds to an upper limit; and the third argument is the value to be clamped to a value that is between the lower limit and the upper limit, inclusive. As such, if (B, R(k)) is less than <NUM>, then <NUM> is assigned to c(k); if (B, R(k)) is greater than <NUM>, then <NUM> is assigned to c(k). The formula (<NUM>) illustrates that a higher distance D(B, R(k)) results in a lower weight c(k).

The values of the ARF block <NUM> can be calculated using formula (<NUM>): <MAT>.

As mentioned above, the "<NUM>" of formula (<NUM>) corresponds to the "<NUM>" of the clamping function of formula (<NUM>). Formula (<NUM>) illustrates that temporal filtering, to generate an alternate reference block (e.g., the ARF block <NUM>) of the alternate reference frame (i.e., the ARF <NUM>), can be accomplished by summing the weighted reference blocks and the anchor block (i.e., as shown in the numerator of the formula (<NUM>)) and normalizing the result (i.e., as shown in the denominator of the formula (<NUM>)). In formula (<NUM>), the weights c(k) are scalar values; B and R(k) are two-dimensional matrices of pixel values.

In an alternative technique, the filter coefficients can be computed on a pixel-by-basis by, for example, comparing corresponding pixels in two motion-aligned blocks, and using the distance between the two pixels to form the filter weight. This alternative technique may better capture the statistical difference within the pixel block because it can identify pixel level misalignment due to non-translational motion activities. However, the technique can easily be trapped by acquisition noise (or film grains) that randomly perturbs pixel values even if the pixels belong to the same motion trajectory. Implementations according to this disclosure solve this problem by introducing a patch-based distance measurement to form an adaptive temporal filter kernel.

<FIG> is a flowchart diagram of a process <NUM> for using an alternate reference frame (ARF) according to implementations of this disclosure. The process <NUM> can be implemented, for example, as a software program that can be executed by computing devices such as transmitting station <NUM> or receiving station <NUM>. The software program can include machine-readable instructions that can be stored in a memory such as the memory <NUM> or the secondary storage <NUM>, and that, when executed by a processor, such as CPU <NUM>, can cause the computing device to perform the process <NUM>. The process <NUM> can be implemented in whole or in part in the intralinter prediction stage <NUM> of the encoder <NUM>. The process <NUM> can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, can be used.

Using an anchor frame and other video frames, the process <NUM> generates an alternate reference frame (ARF), which can be used by the encoder (and a decoder) when performing inter prediction. The anchor frame and the video frames can be frames of a source input video stream, such as the video stream <NUM> of <FIG>.

As compared to the distance measurement described above, where the same weight is applied to each pixel of the anchor block, the process <NUM> uses the localized information about (e.g., around, in the neighborhood of, etc.) a pixel to determine the weights to be used in the temporal filtering. As such, different weights, which are based on the respective local neighborhoods, are used.

The process <NUM> is explained with reference to <FIG> is an example <NUM> of generating an alternate reference frame according to implementations of this disclosure. In the example <NUM>, an ARF <NUM> is generated, such as by the process <NUM>, using an anchor frame <NUM> and other video frames.

As is known in the art, a pixel can have an associated color space. For example, in a YCrCb or YUV color space, Y is a luminance component, and Cr or U and Cb or V are color difference components. As such, a pixel can include information representing an image captured in the frame, such as luminance information and color information. A pixel can also include location information. As such, a block (e.g., the anchor block <NUM>, the reference block <NUM>, and the reference block <NUM>) can include a luminance block (not shown) and two chrominance blocks (not shown), such as a U or Cb chrominance block, and a V or Cr chrominance block. Various sampling formats have been defined, including <NUM>:<NUM>:<NUM>, <NUM>:<NUM>:<NUM>, and <NUM>:<NUM>:<NUM>.

"Pixel" as used herein, and unless otherwise the context indicates, can refer to the value of a color component of a pixel at a location. For example, reference to "the anchor pixel <NUM>" can mean (e.g., indicate, refer to, etc.) the pixel that is at Cartesian coordinates (<NUM>, <NUM>) of the anchor block <NUM>, the luminance value at that location, the U chrominance value at that location, the V chrominance value at that location, or a combination thereof.

At <NUM> of <FIG>, the process <NUM> selects an anchor frame and video frames. As used in this disclosure, "select" means to create, form, produce, identify, construct, determine, specify, generate, or other select in any manner whatsoever.

In the example <NUM> of <FIG>, two video frames are shown; namely, a frame <NUM> and a frame <NUM>. The example <NUM> shows the anchor frame <NUM> and the frames <NUM> and <NUM> as being consecutive frames in the input video stream: the frame <NUM>, the anchor frame <NUM>, and the frame <NUM> are, respectively, frame numbers n-<NUM>, n, and n+<NUM> in the input video stream. However, that need not be the case.

In an example, and as described with respect to <FIG>, the anchor frame <NUM> can be the last frame of a group of pictures (GOP) and some of the video frames (i.e., predecessor frames in display order) used to generate the ARF <NUM> can be frames from the same GOP as the anchor frame <NUM> and some others of the video frames (i.e., successor frames in display order) can be frames from a subsequent GOP. As such, the video frames can include first video frames and second video frames, the first video frames and the anchor frame can be frames of a first group of pictures, and the second video frames can be frames of a second group of pictures that is different from the first group of pictures.

In an example, the anchor frame can be a frame that is between the first and the last frame in a GOP. As such, some of the predecessor frames can be frames in the same GOP, some of the predecessor frames can be frames in a preceding GOP, some of the successor frames can be frames of the same GOP as the anchor frame, some of the successor frames can be frames of the succeeding GOP as the anchor frame, or a combination thereof.

As described with respect to the example <NUM> of <FIG>, four other frames can be used. In other examples, more than four frames can be used. The number of video frames can be even (i.e., a multiple of <NUM> that is greater than <NUM>) and the anchor frame can be centered between the video frames.

As mentioned with respect to <FIG>, the anchor frame <NUM> can be divided into blocks. Blocks of the anchor frame <NUM> are referred to herein as anchor blocks to differentiate them from blocks of the other video frames. As such, the anchor frame <NUM> can be partitioned into anchor blocks of size M×N pixels. In an example, each anchor block can be of size <NUM>×<NUM> pixels; that is, M=N=<NUM>. However, other sizes are possible. Assuming a raster scan order of the anchor blocks, anchor blocks at the right and/or bottom boundaries of the anchor frame may be smaller than M×N, depending on the size of the anchor frame. An anchor block includes anchor pixels. For example, an anchor block of size M×N (e.g., <NUM>×<NUM>) includes M*N (e.g., <NUM>) pixels. An anchor block <NUM> of <FIG> is an example of an anchor block. The anchor block <NUM> is shown as being of size <NUM>×<NUM> pixels (i.e., an <NUM>×<NUM> block); however, the size of the anchor block can be different. The anchor block <NUM> includes <NUM>*<NUM>=<NUM> anchor pixels, which include an anchor pixel <NUM>.

At <NUM> of <FIG>, the process <NUM> identifies, for the anchor block of the anchor frame, respective reference blocks in the video frames. The respective reference blocks can be identified in any number of ways.

For example, a prediction unit, such as a unit of the intra/inter prediction stage <NUM> of <FIG> can conduct motion search in each of the video frames to identify a closest matching block within respective search windows in each of the video frames. <FIG> illustrates the respective reference blocks. A reference block <NUM> of the frame <NUM> can be the reference block that is identified using motion search, as indicated by a motion vector <NUM>. A reference block <NUM> of the frame <NUM> can be the reference block that is identified using motion search, as indicated by a motion vector <NUM>. A reference block can be at an integer pixel location or at a sub-pixel location.

In another example, the respective reference frames can be identified as being the co-located blocks in each of the reference frames. For example, if the top-left pixel of the anchor block <NUM> is at Cartesian location (x, y) (e.g., (<NUM>, <NUM>)) of the anchor frame, then the reference blocks <NUM> and <NUM> can be the <NUM>×<NUM> blocks whose top-left pixels are at locations (x, y) (e.g., (<NUM>, <NUM>)) of the frames <NUM> and <NUM>, respectively.

At <NUM> of <FIG>, the process <NUM> determines, for the anchor pixel and using an anchor patch, respective distances between the anchor pixel and respective co-located reference pixels of the respective reference blocks.

An anchor patch includes a set of pixels about the anchor pixel and/or in the neighborhood of the anchor pixel. In an example, the anchor patch can be a <NUM>×<NUM> window that is centered at/by the anchor pixel, as illustrated by an anchor patch <NUM> of <FIG>. The anchor patch <NUM> includes the anchor pixel <NUM> and the eight surrounding pixels (i.e., the shaded pixels). The anchor patch can include more or fewer pixels. The anchor patch can have a square, rectangular, or any other shape.

In an example, pixels of an anchor patch that are not part of the anchor block can be excluded from the anchor patch. For example, with respect to anchor pixels <NUM> and <NUM>, the corresponding anchor patches include only those shaded pixels that are part of the anchor block <NUM>. In another example, the anchor patch can include pixels that are outside of the anchor block but that are within the anchor frame.

A respective distance between the anchor pixel and a respective co-located reference pixel is determined using the anchor patch pixels and co-located reference pixels. In the reference block <NUM>, the co-located reference pixels of the anchor pixels of the anchor patch in the reference blocks <NUM> and <NUM> are indicated by the bounding boxes <NUM> and <NUM>, respectively.

The respective distance can be calculated using formula (<NUM>). Let b (x, y) denotes a pixel in the anchor block (denoted B) at position (x, y). Similarly, let r(x, y) denote a pixel in a reference block R at the same position. With respect to <FIG>, the block B can be the anchor block <NUM> and the reference block R can be the reference block <NUM> or the reference block <NUM>. The distance measurement between the two pixels, D(b(x, y), r(x, y)), is given by formula (<NUM>): <MAT>.

As such, formula (<NUM>) can be used to determine a first distance between the anchor pixel <NUM> (i.e., b(x, y)) and a co-located reference pixel <NUM> (e.g., r(x, y)) in the reference block <NUM>, and a second distance between the anchor pixel <NUM> (i.e., b(x, y)) and a co-located reference pixel <NUM> (e.g., r(x, y)) in the reference block <NUM>. The formula (<NUM>) is used to calculate a distance between two patches: an anchor patch of an anchor frame and a co-located anchor patch in the reference frame.

At <NUM> of <FIG>, the process <NUM> determines (e.g., identifies, calculates, etc.), using the respective distances, respective weights. The distances (e.g., the first distance and the second distance) can be clamped using formula (<NUM>) to determine the respective weight for a pixel of a reference frame: <MAT>.

In formula (<NUM>), "<NUM>" corresponds to the number of terms in formula (<NUM>); cR(x, y) is the weight to be assigned to the pixel at location (x, y) of the reference block R (i.e., the reference frame what contains the pixel r(x, y)).

At <NUM> of <FIG>, the process <NUM> determines, using the respective weights, an ARF pixel that is co-located with the anchor pixel. That is, the weights can be used to calculate (e.g., determine, etc.) pixel values of the ARF. As such, the value of a pixel (e.g., an ARF pixel <NUM>) at location (x, y) (i.e., co-located with the anchor pixel <NUM>) of the ARF (i.e., the ARF <NUM>) can be calculated using formula (<NUM>): <MAT>.

In formula (<NUM>), ARF(x, y) denotes the value of the pixel of the alternate reference frame at location (x, y); P denotes the number of reference frames (e.g., <NUM> frames in example <NUM>; namely, the frames <NUM> and <NUM>); and k is a variable that denotes a specific reference frame. The weight ck(x, y) denotes the weight to be applied to (e.g., used with, multiplied by, etc.) the pixel rk(x, y) of the reference block k. The weight ck(x, y) is as described with respect to formulae (<NUM>) and (<NUM>). Further, b(x, y) is the anchor pixel, and the "<NUM>" corresponds to the maximum clamping value. As such, the highest weight value can be assigned to the anchor pixel.

The formulae (<NUM>)-(<NUM>), can be used to determine (e.g., calculate, etc.) respective values of each of the pixels of the ARF <NUM> that are co-located with the anchor pixels of the anchor block <NUM>. Similarly, the formulae (<NUM>)-(<NUM>) can be used to determine values of ARF pixels corresponding to other anchor blocks of the anchor frame1002.

At <NUM> of <FIG>, the process <NUM> encodes the ARF in a compressed bitstream. The compressed bitstream can be the compressed bitstream of <FIG>. Encoding the ARF can be as described with respect to encoding any other frame of the video stream such as described with respect to <FIG>.

As mentioned above, the ARF is a reference frame and, as such, can be stored in a reference frame buffer and can be used for encoding other video frames. A decoder, such as the decoder <NUM> of <FIG>, can receive the ARF in the compressed bitstream, decode the ARF, and use it in decoding other frames that are encoded in the compressed bitstream.

The formulae (<NUM>)-(<NUM>) can be used to separately calculate, for a pixel of the ARF, values for each of the color components (e.g., Red, Green, and Blue, in the case that an RGB color system is used. For example, formulae (<NUM>)-(<NUM>) can be used to separately calculate, for a pixel of the ARF a luminance value, a chrominance U value, and a chrominance V value. For example, when calculating the luminance Y value, then b(x, y), r(x, y), and ARF(x, y) each corresponds to the luminance value of the respective pixel. Similarly, when calculating a chrominance value (e.g., U or V value), then b(x, y), r(x, y), and ARF(x, y) each corresponds to the chrominance value. That is, the technique described above uniformly applies to both luminance and chrominance component planes. Each color plane operates independently of the other color planes. That is, for example, the respective distances can be distances in the luminance color plane and the ARF pixel can be a luminance pixel. For example, the respective distances can be distances in a chrominance color plane and the ARF pixel can be in the same chrominance color plane. More generally, the respective distances can be distances in a particular color plane and the ARF pixel can be a pixel value in the particular color plane.

The patch-based distance measurement described above, and which determines the temporal filtering weight, optimizes the trade-off between flexibility for pixel-level temporal consistency detection and stability over the acquisition noise imposed on the pixel values. The patch-based distance (or, equivalently, weight) determination in adaptive temporal filtering for alternate reference frame described herein can improve the compression performance by <NUM>-<NUM>%.

In some examples, the color component values are not independently determined. It is observed that the luminance and the collocated chrominance pixel, together, form a colored pixel representation in the frame. As such, the chrominance and the co-located chrominance values of at a pixel location likely belong to the same motion object. Thus, in some examples, the distortion metrics (i.e., the distances) for the luminance and the chrominance components can be modified as described below.

For a luminance pixel value b(x, y) and a luminance reference pixel r(x, y), the patch-based distance measurement of formula (<NUM>) can be modified to further include at least one of the chrominance components from the U or V planes that are collocated with the luminance component. Formula (<NUM>) shows a modified distance measurement, Dm(x, y), that includes both U and V components: <MAT>.

In formula (<NUM>), D(b(x, y), r(x, y)) is as described with respect to formula (<NUM>) for calculating a distance based on the luminance values; bu(x, y), ru(x, y), bv(x, y), and rv(x, y) correspond, respectively, to the chrominance U value of the anchor pixel, the chrominance U value of the co-located reference pixel of a reference frame, the chrominance V value of the anchor pixel, and the chrominance V value of the co-located reference pixel of a reference frame; and the value of the denominator (i.e., <NUM>) corresponds to the number of terms in the numerator.

For a chrominance pixel value b(x, y) (either the U chrominance component or the V chrominance component) and a chrominance reference pixel r(x, y), the patch-based distance measurement of formula (<NUM>) can be modified to further include the luminance component from the Y plane that is collocated with the chrominance component. As such, in a case of determining a chrominance value of the ARF pixel, a Y luminance value that is collocated with a chrominance component of the anchor pixel can be included in the determining the respective distance. Formula (<NUM>) shows a modified distance measurement, Dm(x, y), that includes the luminance component: <MAT>.

In formula (<NUM>), D(b(x, y), r(x, y)) is as described with respect to formula (<NUM>) for calculating a distance based on the chrominance values; bl(x, y) and rl(x, y) correspond, respectively, to the luminance Y value of the anchor pixel and the luminance Y value of the co-located reference pixel of a reference frame; and the denominator, DENOMINATOR, is equal to the number of terms in the numerator. The number of terms in the numerator, in turn, depends on the sampling format used. As mentioned above, two of the formats used are the <NUM>:<NUM>:<NUM> (also referred to as YUV444) and <NUM>:<NUM>:<NUM> (also referred to as YUV420) formats.

The YUV444 format has the same number of luminance and chrominance pixels. As such, DENOMINATOR in formula (<NUM>) is equal to <NUM> (i.e., <NUM> chrominance pixels in the same plane plus <NUM> luminance pixel in the luminance plane).

In the YUV420 format, every <NUM>×<NUM> luminance pixels correspond to <NUM> U chrominance plane pixel and <NUM> V chrominance plane pixel. That is, in the YUV420 format, <NUM> chrominance pixel has <NUM> collocated luminance pixels. As such, in formula (<NUM>), a luminance value actually includes four luminance values. As such, in the YUV420 format, the luminance pixel term in formula (<NUM>) (i.e., (bl(x, y) - rl(x, y))<NUM>) contains four pixel differences (i.e., one squared difference for each of the luminance values). As such, the normalization term (i.e., DENOMINATOR in formula (<NUM>)) is equal to <NUM>.

The cross-plane referencing, described with respect to formulae (<NUM>)-(<NUM>), has been found to result in PSNR_U and PSNR_V metric improvements of by <NUM>-<NUM>% in the coding of the chrominance components, and a <NUM>% coding gains of the luminance component.

For simplicity of explanation, the process <NUM> is depicted and described as a series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and/or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a method in accordance with the disclosed subject matter.

The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it is to be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or any other processing or change of data.

The word "example" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" is intended to present concepts in a concrete fashion. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise, or clear from context, "X includes A or B" is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied under any of the foregoing instances. In addition, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from context to be directed to a singular form. Moreover, use of the term "an implementation" or "one implementation" throughout is not intended to mean the same embodiment or implementation unless described as such.

Implementations of the transmitting station <NUM> and/or the receiving station <NUM> (and the algorithms, methods, instructions, etc., stored thereon and/or executed thereby, including by the encoder <NUM> and the decoder <NUM>) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors or any other suitable circuit. In the claims, the term "processor" should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms "signal" and "data" are used interchangeably. Further, portions of the transmitting station <NUM> and the receiving station <NUM> do not necessarily have to be implemented in the same manner.

Further, in one aspect, for example, the transmitting station <NUM> or the receiving station <NUM> can be implemented using a general-purpose computer or general-purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms and/or instructions described herein. In addition, or alternatively, for example, a special purpose computer/processor can be utilized that contains other hardware for carrying out any of the methods, algorithms, or instructions described herein.

The transmitting station <NUM> and the receiving station <NUM> can, for example, be implemented on computers in a video conferencing system. Alternatively, the transmitting station <NUM> can be implemented on a server and the receiving station <NUM> can be implemented on a device separate from the server, such as a hand-held communications device. In this instance, the transmitting station <NUM> can encode content using an encoder <NUM> into an encoded video signal and transmit the encoded video signal to the communications device. In turn, the communications device can then decode the encoded video signal using a decoder <NUM>. Alternatively, the communications device can decode content stored locally on the communications device, for example, content that was not transmitted by the transmitting station <NUM>. Other suitable transmitting and receiving implementation schemes are available. For example, the receiving station <NUM> can be a generally stationary personal computer rather than a portable communications device and/or a device including an encoder <NUM> may also include a decoder <NUM>.

Further, all or a portion of implementations of the present disclosure can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example, tangibly contain, store, communicate, or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or a semiconductor device. Other suitable mediums are also available.

Claim 1:
A method for generating and using an alternate reference frame, ARF, comprising:
selecting an anchor frame (<NUM>) and video frames (<NUM>, <NUM>) from a source input video stream, wherein
the anchor frame comprises an anchor block (<NUM>), and
the anchor block comprises anchor pixels;
identifying, for the anchor block of the anchor frame, respective reference blocks (<NUM>, <NUM>) in the video frames;
for each anchor pixel of a plurality of the anchor pixels of the anchor block (<NUM>):
determining, for the anchor pixel and using an anchor patch, respective distances between the anchor pixel (<NUM>) and respective co-located reference pixels (<NUM>, <NUM>) of the respective reference blocks (<NUM>, <NUM>), wherein:
the anchor patch (<NUM>) comprises a set of pixels in the anchor frame in a neighbourhood of the anchor pixel (<NUM>), and
a respective distance, of the respective distances, between the anchor pixel (<NUM>) and a respective co-located reference pixel (<NUM>) in the respective reference block is determined using a distance between the anchor patch (<NUM>) and a co-located patch (<NUM>) around the respective co-located reference pixel (<NUM>) in the respective video frame, wherein the distance between the anchor patch (<NUM>) and the co-located patch (<NUM>) is determined using distances between each of the anchor patch pixels and a corresponding reference pixel of the co-located patch;
determining, using the respective distances, respective weights; and
determining, using the respective weights, the anchor pixel and the respective co-located reference pixels of the respective reference blocks, an ARF pixel (<NUM>) within the ARF that is co-located with the anchor pixel (<NUM>); and
encoding, in a compressed bitstream, the ARF.