Patent Description:
The first version of the HEVC standard was finalized in Oct. <NUM> and offers approximately <NUM>% bit-rate saving or equivalent perceptual quality compared to the prior generation video coding standard H. <NUM>/MPEG AVC. Although the HEVC standard provides significant coding improvements over its predecessor, there is evidence that superior coding efficiency can be achieved with additional coding tools over HEVC. Based on that, both VCEG and MPEG started the exploration work of new coding technologies for future video coding standardization. A Joint Video Exploration Team (JVET) was formed in Oct. <NUM> by ITU-T VECG and ISO/IEC MPEG to begin significant study of advanced technologies that could enable substantial enhancement of coding efficiency. Reference software called joint exploration model (JEM) was maintained by the JVET by integrating several additional coding tools on top of the HEVC test model (HM).

<NUM>, a joint call for proposals (CfP) on video compression with capability beyond HEVC was issued by ITU-T and ISO/IEC. <NUM>, <NUM> CfP responses were received and evaluated at the 10th JVET meeting, with demonstrating compression efficiency gain over the HEVC around <NUM>%. Based on such evaluation results, the JVET launched a new project to develop the new generation video coding standard called Versatile Video Coding (WC). In the same month, a reference software codebase, called WC test model (VTM), was established for demonstrating a reference implementation of the WC standard. For the initial VTM-<NUM>, most of the coding modules, including intra prediction, inter prediction, transform/inverse transform and quantization/de-quantization, and in-loop filters follow the existing HEVC design, except that a multi-type tree-based block partitioning structure is used in the VTM. Meanwhile, to facilitate the assessment of new coding tools, another reference software base called benchmark set (BMS) was also generated. In the BMS codebase, a list of coding tools inherited from the JEM, which provides higher coding efficiency and moderate implementation complexity, are included on top of the VTM and used as the benchmark when evaluating similar coding technologies during the WC standardization process. JEM coding tools integrated in BMS-<NUM> include <NUM> angular intra prediction directions, modified coefficient coding, advanced multiple transform (AMT) +<NUM>×<NUM> non-separable secondary transform (NSST), affine motion model, generalized adaptive loop filter (GALF), advanced temporal motion vector prediction (ATMVP), adaptive motion vector precision, decoder-side motion vector refinement (DMVR) and LM chroma mode.

<CIT> discloses methods, apparatus, and computer readable media configured to receive compressed video data, wherein the compressed video data is related to a set of frames. A decoder-side predictor refinement technique is used to calculate a new motion vector for a current frame from the set of frames, wherein the new motion vector estimates motion for the current frame based on one or more reference frames. An existing motion vector associated with a different frame from a motion vector buffer is retrieved. The new motion vector is calculated based on the existing motion vector using a decoder-side motion vector prediction technique, such that the existing motion vector is in the motion vector buffer after calculating the new motion vector.

<NPL>, describes the coding features that are under coordinated test model study by JVET as potential enhanced video coding technology.

<CIT> discloses techniques for sub-prediction unit (PU) based motion prediction for video coding in HEVC and 3D-HEVC. In one example, the techniques include an advanced temporal motion vector prediction (TMVP) mode to predict sub-PUs of a PU in single layer coding for which motion vector refinement may be allowed. The advanced TMVP mode includes determining motion vectors for the PU in at least two stages to derive motion information for the PU that includes different motion vectors and reference indices for each of the sub-PUs of the PU. In another example, the techniques include storing separate motion information derived for each sub-PU of a current PU predicted using a sub-PU backward view synthesis prediction (BVSP) mode even after motion compensation is performed. The additional motion information stored for the current PU may be used to predict subsequent PUs for which the current PU is a neighboring block.

<NPL> discloses an approach to use hardware accelerator of entropy coding to precalculate boundary strength of deblocking filter.

Some embodiments include methods that are used in video encoding and decoding (collectively "coding"). In some described examples a block-based video coding method includes: at a first block, refining a first non-refined motion vector and a second non-refined motion vector to generate a first refined motion vector and a second refined motion vector; using one or both of the first non-refined motion vector and the second non-refined motion vector, predicting motion information of a second block, the second block being a spatial neighbor of the first block; and predicting the first block with bi-prediction using the first refined motion vector and the second refined motion vector.

In an example of a video coding method, a first non-refined motion vector and a second non-refined motion vector associated with a first block are identified. Motion information of a second block neighboring the first block is predicted using one or both of the first non-refined motion vector and the second non-refined motion vector. The first non-refined motion vector and the second non-refined motion vector are refined, e.g. using decoder-side motion vector refinement (DMVR). The refined motion vectors are used to generate a first refined motion vector and a second refined motion vector, which may be used for bi-prediction of the first block. The use of the non-refined motion vector(s) to predict motion information of the second block may be performed using one or more techniques such as spatial advanced motion vector prediction (AMVP), temporal motion vector prediction (TMVP), advanced temporal motion vector prediction (TMVP), and using the non-refined motion vector(s) as spatial merge candidates. In the case of patial prediction, the second block may be spatial neighbor of the first block; in the case of temporal prediction, the second block may be a collocated block in a subsequently-coded picture. In some embodiments, deblocking filter strength for the first block is determined based at least in part on the first non-refined motion vector and the second non-refined motion vector.

In another example of a video coding method, a first non-refined motion vector and a second non-refined motion vector associated with a first block are identified. The first non-refined motion vector and the second non-refined motion vector are refined to generate a first refined motion vector and a second refined motion vector, e.g. using DMVR. Motion information of a second block is predicted using either spatial motion prediction or temporal motion prediction, wherein (i) if spatial motion prediction is used, one or both of the first non-refined motion vector and the second non-refined motion vector are used to predict the motion information, and (ii) if temporal motion prediction is used, one or both of the first refined motion vector and the second refined motion vector are used to predict the motion information.

In another example of a video coding method, at least one predictor is selected for predicting motion information of a current block. The selection is made from among a set of available predictors, where the available predictors include (i) at least one non-refined motion vector from a spatially neighboring block of the current block and (ii) at least one refined motion vector from a collocated block of the current block.

In another example of a video coding method, at least two non-overlapping regions in a slice are determined. A first non-refined motion vector and a second non-refined motion vector associated with a first block in the first region are identified. The first non-refined motion vector and the second non-refined motion vector are refined to generate a first refined motion vector and a second refined motion vector. In response to a determination that motion information of a second block neighboring the first block is predicted using motion information of the first block, the motion information of the second block is predicted (i) using one or both of the first non-refined motion vector and the second non-refined motion vector if the first block is not on the bottom edge or the right edge of the first region and (ii) using one or both of the first refined motion vector and the second refined motion vector if the first block is on a bottom edge or a right edge of the first region.

In another example of a video coding method, at least two non-overlapping regions in a slice are determined. A first non-refined motion vector and a second non-refined motion vector associated with a first block in the first region are identified. The first non-refined motion vector and the second non-refined motion vector are refined to generate a first refined motion vector and a second refined motion vector. In response to a determination that motion information of a second block neighboring the first block is predicted using motion information of the first block, the motion information of the second block is predicted (i) using one or both of the first non-refined motion vector and the second non-refined motion vector if the second block is in the first region and (ii) using one or both of the first refined motion vector and the second refined motion vector if the second block is not in the first region.

In another example of a video coding method, at least two non-overlapping regions in a slice are determined. A first non-refined motion vector and a second non-refined motion vector associated with a first block in the first region are identified. The first non-refined motion vector and the second non-refined motion vector are refined to generate a first refined motion vector and a second refined motion vector. Motion information of a second block is predicted using either spatial motion prediction or temporal motion prediction, wherein (i) if the first block is not on the bottom edge or the right edge of the first region, and if spatial motion prediction is used, one or both of the first non-refined motion vector and the second non-refined motion vector are used to predict the motion information, and (ii) if the first block is on the bottom edge or the right edge of the first region, or if temporal motion prediction is used, one or both of the first refined motion vector and the second refined motion vector are used to predict the motion information.

In another example of a video coding method, at least two non-overlapping regions are defined in a slice. A set of available predictors is determined for prediction of motion information of a current block in a first region, wherein the set of available predictors is constrained not to include motion information of any block in a second region different from the first region.

In one example, a first non-refined motion vector and a second non-refined motion vector are determining for a current block. A first prediction I(<NUM>) is generated using the first non-refined motion vector and a second prediction I(<NUM>) is generated using the second non-refined motion vector. An optical flow model is used to determine a motion refinement ( <MAT>) for the current block. The first non-refined motion vector and a second non-refined motion vector are refined using the motion refinement to generate a first refined motion vector and a second refined motion vector. The current block is predicted with bi-prediction using the first refined motion vector and the second refined motion vector.

In another example of a video coding method, a first non-refined motion vector and a second non-refined motion vector are determining for a current block. A first prediction I(<NUM>) is generated using the first non-refined motion vector and a second prediction I(<NUM>) is generated using the second non-refined motion vector. A motion refinement ( <MAT>) is determined for the current block, where <MAT> where θ is a set of coordinates of all samples within the current block, and where <MAT>.

The first non-refined motion vector and second non-refined motion vector are refined using the motion refinement to generate a first refined motion vector and a second refined motion vector. The current block is predicted with bi-prediction using the first refined motion vector and the second refined motion vector.

In another example of a video coding method, a first motion vector and a second motion vector are determined for a current block. The first motion vector and the second motion vector are refined by iteratively performing steps including the following:.

Further embodiments include encoder and decoder (collectively "codec") apparatuses configured to perform the methods described herein. Such systems may include a processor and a non-transitory computer storage medium storing instructions that are operative, when executed on the processor, to perform the methods described herein. Additional examples include non-transitory computer-readable media storing a video encoded using the methods described herein.

In view of <FIG>, and the corresponding description, one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown).

Like HEVC, VVC is built upon the block-based hybrid video coding framework. <FIG> is a functional block diagram of an example of a block-based hybrid video encoding system. The input video signal <NUM> is processed block by block. The blocks may be referred to as coding units (CUs). In VTM-<NUM>, a CU can be up to 128x128 pixels. However, as compared to HEVC, which partitions blocks only based on quad-trees, in VTM-<NUM>, a coding tree unit (CTU) may be split into CUs to adapt to varying local characteristics based on quad/binary/ternary-tree. Additionally, the concept of multiple partition unit type in HEVC may be removed, such that the separation of CU, prediction unit (PU) and transform unit (TU) is not used in the WC; instead, each CU may be used as the basic unit for both prediction and transform without further partitions. In the multi-type tree structure, a CTU is firstly partitioned by a quad-tree structure. Then, each quad-tree leaf node can be further partitioned by a binary and ternary tree structure. As shown in <FIG>, there may be five splitting types: quaternary partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.

In <FIG>, spatial prediction (<NUM>) and/or temporal prediction (<NUM>) may be performed. Spatial prediction (or "intra prediction") uses pixels from the samples of already coded neighboring blocks (which are called reference samples) in the same video picture/slice to predict the current video block. Spatial prediction reduces spatial redundancy inherent in the video signal. Temporal prediction (also referred to as "inter prediction" or "motion compensated prediction") uses reconstructed pixels from the already-coded video pictures to predict the current video block. Temporal prediction reduces temporal redundancy inherent in the video signal. Temporal prediction signal for a given CU is usually signaled by one or more motion vectors (MVs) which indicate the amount and the direction of motion between the current CU and its temporal reference. Also, if multiple reference pictures are supported, a reference picture index is additionally sent, which is used to identify from which reference picture in the reference picture store (<NUM>) the temporal prediction signal comes.

After spatial and/or temporal prediction, the mode decision block (<NUM>) in the encoder chooses the best prediction mode, for example based on the rate-distortion optimization method. The prediction block is then subtracted from the current video block (<NUM>), and the prediction residual is de-correlated using transform (<NUM>) and quantized (<NUM>). The quantized residual coefficients are inverse quantized (<NUM>) and inverse transformed (<NUM>) to form the reconstructed residual, which is then added back to the prediction block (<NUM>) to form the reconstructed signal of the CU. Further in-loop filtering, such as deblocking filter, may be applied (<NUM>) on the reconstructed CU before it is put in the reference picture store (<NUM>) and used to code future video blocks. To form the output video bit-stream <NUM>, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit (<NUM>) to be further compressed and packed to form the bit-stream.

<FIG> is a functional block diagram of a block-based video decoder. The video bit-stream <NUM> is unpacked and entropy decoded at entropy decoding unit <NUM>. The coding mode and prediction information are sent to either the spatial prediction unit <NUM> (if intra coded) or the temporal prediction unit <NUM> (if inter coded) to form the prediction block. The residual transform coefficients are sent to inverse quantization unit <NUM> and inverse transform unit <NUM> to reconstruct the residual block. The prediction block and the residual block are then added together at <NUM>. The reconstructed block may further go through in-loop filtering before it is stored in reference picture store <NUM>. The reconstructed video in reference picture store is then sent out to drive a display device, as well as used to predict future video blocks.

As mentioned earlier, BMS-<NUM> adheres to the same encoding/decoding workflow of the VTM-<NUM> as shown in <FIG> and <FIG>. However, several coding modules, especially the ones associated with temporal prediction, are further extended and enhanced. In the following, some inter tools that are included in BMS-<NUM> or the previous JEM are briefly described.

Like HEVC, to reduce the overhead of signaling motion information, both VTM and BMS include two modes to code the motion information of each CU, namely merge mode and non-merge mode. In merge mode, the motion information of the current CU is directly derived from spatial and temporal neighboring blocks, and a competition-based scheme is applied to select the best neighboring block out of all the available candidates; correspondingly, only the index of the best candidate is sent for reconstructing the motion information of the CU at the decoder. If an inter-coded PU is coded in non-merge mode, the MV will be differentially coded using a MV predictor derived from an advanced motion vector prediction (AMVP) technique. Like the merge mode, AMVP derives the MV predictor from spatial and temporal neighboring candidates. Then, the difference between the MV predictor and the actual MV, and the index of the predictor are transmitted to the decoder.

<FIG> shows an example for spatial MV prediction. In the current picture to be coded (CurrPic), the square CurrCU is the current CU, which has the best matching block CurrRefCU in the reference picture (CurrRefPic). CurrCU's MV, i.e., MV2, is to be predicted. The current CU's spatial neighborhood could be the upper, left, upper-left, bottom-left, upper-right neighboring CU of the current CU. In <FIG>, the neighboring CU is shown as the upper neighbor, NeighbCU. NeighbCU's reference picture (NeighbRefPic) and MV (MV1) are both known, because NeighbCU has been coded before CurrCU.

<FIG> shows an example for temporal MV prediction (TMVP). There are four pictures (ColRefPic, CurrRefPic, ColPic, CurrPic) shown in <FIG>. In the current picture to be coded (CurrPic), the square CurrCU is the current CU, which has the best matching block (CurrRefCU) in the reference picture (CurrRefPic). CurrCU's MV, i.e., MV2, is to be predicted. The current CU's temporal neighborhood is specified as the collocated CU (CoICU) in the neighboring picture (CoIPic). CoICU's reference picture (CoIRefPic) and MV (MV1) are both known, because ColPic has been coded before CurrPic.

For spatial and temporal motion vector prediction, given limited time and space, the MVs between different blocks are treated as translational with uniform velocity. In the examples of <FIG> and <FIG>, the temporal distance between the CurrPic and CurrRefPic is TB, and the temporal distance between CurrPic and NeighbRefPic in <FIG>, or between ColPic and ColRefPic in <FIG>, is TD. The scaled MV predictor may be calculated as <MAT>.

In VTM-<NUM>, each merge block has at most one set of motion parameters (one motion vector and one reference picture index) for each prediction direction L0 and L1. In contrast, an additional merge candidate based on advanced temporal motion vector prediction (ATMVP) is included in BMS-<NUM> to enable the derivation of motion information at the sub-block level. Using such a mode, the temporal motion vector prediction is improved by allowing a CU to derive multiple MVs for the sub-blocks in the CU. In general, the ATMVP derives the motion information of the current CU in two steps, as shown in <FIG>. The first step is to identify the corresponding block of the current block (which is referred to as the collocated block) in a temporal reference picture. The selected temporal reference picture is called the collocated picture. The second step is to split the current block into sub-blocks and derive the motion information of each sub-block from the corresponding small block in the collocated picture.

In the first step, the collocated block and the collocated picture are identified by the motion information of the spatial neighboring blocks of the current block. In the current design, the first available candidate in the merge candidate list is considered. <FIG> illustrates this process. Specifically, in the example of <FIG>, block A is identified as the first available merge candidate of the current block based on the scanning order of the merge candidate list. Then, the corresponding motion vector of block A (MVA) as well as its reference index are used to identify the collocated picture and the collocated block. The location of the collocated block in the collocated picture is determined by adding the motion vector of block A (MVA) to the coordinate of the current block.

In the second step, for each sub-block in the current block, the motion information of its corresponding small block (as indicated by the small arrows in <FIG>) in the collocated block is used to derive the motion information of the sub-block. Specifically, after the motion information of each small block in the collocated block is identified, it is converted to the motion vector and reference index of the corresponding sub-block in the current block in the same way as the TMVP.

For the merge mode in VTM, when the selected merge candidate is bi-predicted, the prediction signal of the current CU is formed by averaging the two prediction blocks using the two MVs associated with the reference lists L0 and L1 of the candidate. However, the motion information of the merge candidate (which is derived from either spatial or temporal neighbors of the current CU) may not be accurate enough to represent the true motion of the current CU and therefore may compromise the inter prediction efficiency. To further improve the coding performance of merge mode, a decoder-side motion vector refinement (DMVR) method is applied in BMS-<NUM> to refine the MVs of the merge mode. Specifically, when the selected merge candidate is bi-predicted, a bi-prediction template is firstly generated as the average of two prediction signals based on the MVs from the reference list L0 and L1, respectively. Then, block-matching based motion refinement is performed locally around the initial MVs using the bi-prediction template as the target, as explained below.

<FIG> illustrates a motion refinement process that is applied in DMVR. In general, DMVR refines the MVs of a merge candidate by the following two steps. As shown in <FIG>, in the first step, the bi-prediction template is generated by averaging the two prediction blocks using the initial MVs in L0 and L1 (i.e., MV<NUM> and MV<NUM>) of the merge candidate. Then, for each reference list (i.e., L0 or L1), a block-matching based motion search is performed in the local region around the initial MVs. For each MV, i.e., MV<NUM> or MV<NUM> of the corresponding reference list around the initial MV in that list, the cost values (e.g., sum of absolute difference (SAD)) between the bi-prediction template and the corresponding prediction blocks using that motion vector are measured. For each of two prediction directions, the MV that minimizes the template cost in that prediction direction is considered as the final MV in the reference list of the merge candidate. In the current BMS-<NUM>, for each prediction direction, eight neighboring MVs surrounding the initial MV (with one integer sample offset) are considered during the motion refinement process. At the end, the two refined MVs (MV<NUM>' and MV<NUM>' as shown in <FIG>) are used to generate the final bi-prediction signal of the current CU. Additionally, in conventional DMVR, to further improve the coding efficiency, the refined MVs of a DMVR block are used to predict the motion information of its spatial and temporal neighboring blocks (e.g., based on spatial AMVP, spatial merge candidates, TMVP and ATMVP) and to calculate the boundary strength value of the deblocking filter that is applied to the current CU. <FIG> is a flow chart of an example of a DMVR process, where "spatial AMVP" and "spatial merge candidates" refer to the spatial MV prediction processes for the spatial neighboring CUs that are in the current picture and are coded after the current CU per the coding order of CUs; "TMVP" and "ATMVP" refer to the temporal MV prediction processes for the future CUs in the following pictures (the pictures that are coded after the current picture based on the picture coding order); and "deblocking" refers to the deblocking filtering processes of both the current block and its spatial neighboring blocks.

In the method illustrated in <FIG>, a bi-prediction template is generated at <NUM>. At <NUM>, motion refinement is performed for the L0 motion vector, and at <NUM>, motion refinement is performed for the L1 motion vector. At <NUM>, the final bi-prediction is generated using the refined L0 and L1 motion vectors. In the method of <FIG>, the refined motion vectors are used to predict the motion of subsequently-coded blocks. For example, the refined motion vectors are used for spatial AMVP (<NUM>), TMVP (<NUM>), and ATMVP (<NUM>). The refined motion vectors are also used as spatial merge candidates (<NUM>) and to calculate the boundary strength value of the deblocking filter that is applied to the current CU (<NUM>).

Bi-prediction in VTM/BMS-<NUM> is a combination of two temporal prediction blocks obtained from the reference pictures that are already reconstructed using averaging. However, due to the limitation of the block-based motion compensation, there may be remaining small motion that can be observed between the two prediction blocks, thus reducing the efficiency of motion-compensated prediction. To address this issue, bi-directional optical flow (BIO) was used in JEM to compensate such motion for every sample inside a block. Specifically, BIO is a sample-wise motion refinement that is performed on top of the block-based motion-compensated predictions when bi-prediction is used. The derivation of the refined motion vector for each sample in one block is based on the classical optical flow model. Let I(k)(x, y) be the sample value at the coordinate (x, y) of the prediction block derived from the reference picture list k (k = <NUM>, <NUM>), and ∂I(k)(x, y)/ ∂x and ∂I(k)(x, y)/ ∂y are the horizontal and vertical gradients of the sample. The modified bi-prediction signal by BIO is obtained as: <MAT> where τ<NUM> and τ<NUM> are the temporal distances of the reference pictures Ref0 and Ref1 associated with I(<NUM>) and I(<NUM>) to the current picture. Further, the motion refinement (vx, vy ) at the sample location (x, y) is calculated by minimizing the difference Δ between the values of the samples after motion refinement compensation, as shown as <MAT>.

Additionally, to provide for the regularity of the derived motion refinement, it is assumed that the motion refinement is consistent within a local surrounding area centered at (x, y); therefore, the values of (vx, vy ) are derived by minimizing an optical flow error metric Δ inside the 5x5 window Ω around the current sample at (x, y) as <MAT>.

It should be mentioned that, different from DMVR, the motion refinement (vx, vy) derived by BIO is only applied to enhance the bi-prediction signal but not to modify the motion information of the current CU. In other words, the MVs that are used to predict the MVs of the spatial and temporal neighboring blocks and to decide the deblocking boundary strength of the current CU are still the original MVs (i.e., the MVs that are used to generate the block-based motion compensation signals I(<NUM>)(x, y) and I(<NUM>)(x, y)) of the CU before BIO is applied).

Like HEVC and its predecessors, VTM-<NUM> employs motion compensated prediction (MCP) to efficiently reduce the temporal redundancy between pictures, thus achieving high inter coding efficiency. Because the MVs that are used to generate the predication signal of one CU are either signaled in the bitstream or inherited from its spatial/temporal neighbors, there is no dependency between the MCPs of spatial neighboring CUs. As a result, the MCP processes of all the inter blocks in the same picture/slice are independent from each other. Thus, for VTM-<NUM> and for HEVC, the decoding processes of multiple inter blocks can be done in parallel, e.g., they can be assigned to different threads to exploit the parallelism.

As described above, the DMVR tool is applied in BMS-<NUM>. To avoid introducing extra signaling overhead, the motion refinements are derived using the two prediction signals associated with the original L0 and L1 MVs of a CU. Thus, when the motion information of a CU is predicted from one of its spatial neighbors (e.g., by AMVP and merge mode) that is coded by the DMVR, its decoding process waits until the MVs of the neighboring block are fully reconstructed by the DMVR. This could significantly complicate the pipeline design, especially at the decoder side, therefore leading to significant complexity increase for the hardware implementation.

To illustrate the coding latency caused by DMVR, <FIG> and <FIG> show examples to compare the decoding process of VTM-<NUM> and BMS-<NUM>. To facilitate the explanation, a case is described in which there are four CUs of equal block-size and all four CUs are coded by the DMVR, each being decoded by a separate decoding thread; the decoding complexity of each individual decoding module (e.g., the MCP, the DMVR, the dequantization and the inverse transform) is assumed to be the same for four CUs. As shown in <FIG>, because the four CUs can be decoded in parallel, the total decoding time of VTM-<NUM> is equal to the decoding time of one CU, i.e., TMCP + Tde-quant + Tinv-trans. Due to the dependency introduced by the DMVR, for the decoding process of BMS-<NUM> (as shown in <FIG>), the decoding of each individual coding block cannot be invoked until the DMVR of its spatial neighboring blocks are fully finished. Thus, the total decoding time of the four CUs for BMS-<NUM> is equal to Ttotal = <NUM> * (TMCP + TDMVR) + Tde-quant + Tinv-trans. As can be seen, the usage of the prediction samples to refine the motion information by the DMVR introduces dependency among neighboring inter blocks, therefore significantly increasing the latency for both encoding and decoding processes.

Methods are proposed in the present disclosure to remove or reduce the encoding/decoding latency of the DMVR while preserving its main coding performance. Specifically, various embodiments of the disclosure include one or more of the following aspects.

Unlike in the current DMVR method in BMS-<NUM> where the refined DMVR motion of one block are always used to predict the motion of its spatial/temporal neighboring blocks and derive the deblocking filter strength, it is proposed in some embodiments to completely or partially use the non-refined MVs of a DMVR block (the MVs that are used to generate the original bi-prediction signal) for the MV prediction and deblocking processes. Given that the original MVs can be obtained directly from parsing and motion vector reconstruction (motion vector predictor plus parsed motion vector difference) without the DMVR, there is no dependency between neighboring blocks, and the decoding processes of multiple inter CUs can be done in parallel.

Since the non-refined MVs may be less accurate than the refined MVs, this may result in some coding performance degradation. To reduce such loss, it is proposed in some embodiments to divide a picture/slice into multiple regions. Moreover, additional constraints are proposed in some embodiments such that the decoding of multiple CUs inside the same region or multiple CUs from different regions can be performed independently.

In some embodiments, motion derivation methods based on optical flow are proposed to replace the block-matching based motion search for calculating the motion refinements of each DMVR CU. Compared to the block-matching based method which performs motion search in a small local window, some embodiments directly calculate motion refinements based on the spatial and temporal sample derivatives. This may not only reduce the computational complexity but may also increase the motion refinement precision, because the value of the derived refined motion is not limited to the search window.

As pointed out above, using the refined MVs of one DMVR block as the MV predictors of its neighboring blocks is unfriendly to parallel encoding/decoding for real codec design, because the encoding/decoding of the neighboring blocks is not performed until the refined MVs of the current block is fully reconstructed though DMVR. Based on such analysis, methods are proposed in this section to remove the coding latency caused by DMVR. In some embodiments, the core design of DMVR (e.g., block-matching based motion refinement) remains the same as the existing design. However, the MVs of DMVR blocks that are used to perform MV predictions (e.g., AMVP, merge, TMVP and ATMVP) and deblocking are modified such that the dependency between neighboring blocks caused by the DMVR may be removed.

In some embodiments, instead of using the refined motion, it is proposed to always perform MV predictions and deblocking using the non-refined motion of DMVR blocks. <FIG> illustrates a modified DMVR process after such a method is applied. As shown in <FIG>, instead of using the refined MVs, the non-refined MVs (the original MVs before the DMVR) are used to derive the MV predictors and determine the boundary strength of the deblocking filter. The refined MVs are only used to generate the final bi-prediction signal of the block. Because the dependency between the refined MVs of the current block and the decoding of its neighboring blocks does not exist, such embodiments may be used to remove the encoding/decoding latency of the DMVR.

In the example of <FIG>, the non-refined MVs of a DMVR block are used to derive the temporal motion predictors for the collocated blocks in future pictures through TMVP and ATMVP and to calculate the boundary strength for the deblocking filter between the current block and its spatial neighbors. Because the non-refined MVs may be less accurate than the refined MVs, this could lead to some coding performance losses. On the other hand, temporal motion prediction (TMVP and ATMVP) predicts the MVs in the current picture using the MVs of previously decoded pictures (specifically the collocated picture). Therefore, before performing the temporal motion prediction for the current picture, the refined MVs of the DMVR CUs in the collocated picture are already reconstructed. A similar situation is also applicable to the deblocking filter process: because the deblocking filter is applied to reconstructed samples, it can only be invoked after the samples of the current block are fully reconstructed though MC (including the DMVR), dequantization and inverse transform. Therefore, before the deblocking is applied to a DMVR block, the refined MVs are already available.

In the method illustrated in <FIG>, non-refined motion vectors for a first block are identified at <NUM>. The non-refined motion vectors may have been signaled for the first block using any of a variety of available MV signaling techniques. The non-refined motion vectors are used at <NUM> to generate a bi-prediction template. At <NUM>, motion refinement is performed for the L0 motion vector, and at <NUM>, motion refinement is performed for the L1 motion vector. At <NUM>, the final bi-prediction of the first block is generated using the refined L0 and L1 motion vectors. In the method of <FIG>, the non-refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a second block). For example, the non-refined motion vectors are used for spatial AMVP (<NUM>), TMVP (<NUM>), and ATMVP (<NUM>). The non-refined motion vectors are also used as spatial merge candidates (<NUM>) and to calculate the boundary strength value of the deblocking filter (<NUM>).

In another example to address these issues and to achieve a better coding performance, it is proposed to use different MVs of a DMVR block (the non-refined MVs and the refined MVs) for the spatial motion prediction, the temporal motion prediction and the deblocking filter. Specifically, in this example the non-refined MVs are only used to derive the MV predictors for the spatial motion prediction (e.g., the spatial AMVP and spatial merge candidates), while the refined MVs are used to not only derive the final prediction of the block but also generate the MV predictors for temporal motion prediction (TMVP and ATMVP) and calculate the boundary strength parameter of the deblocking filter. <FIG> illustrates a DMVR process according to this second example.

In the method illustrated in <FIG>, non-refined motion vectors for a first block are identified at <NUM>. The non-refined motion vectors may have been signaled for the first block using any of a variety of available MV signaling techniques. The non-refined motion vectors are used at <NUM> to generate a bi-prediction template. At <NUM>, motion refinement is performed for the L0 motion vector, and at <NUM>, motion refinement is performed for the L1 motion vector. At <NUM>, the final bi-prediction of the first block is generated using the refined L0 and L1 motion vectors. In the method of <FIG>, the non-refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a second block) within the same picture as the first block. For example, the non-refined motion vectors are used for spatial AMVP (<NUM>) and as spatial merge candidates (<NUM>). The refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a third block) in other pictures, for example using TMVP (<NUM>) or ATMVP (<NUM>). The refined motion vectors are also used to calculate the boundary strength value of the deblocking filter (<NUM>).

In the example of <FIG>, different MVs of a DMVR block are used for the spatial motion prediction and the deblocking filter. On the other hand, unlike the MVs used for temporal motion prediction (which are stored in external memory), the MVs that are used for spatial motion prediction and deblocking are often stored using on-chip memories for practical codec design to increase the data accessing speed. Therefore, some implementations of the method of <FIG> call for two different on-chip memories to store both the non-refined and refined MVs for each DMVR block. This could double the line-buffer size that is used to cache the MVs, which may be undesirable for hardware implementations. To keep the total on-chip memory size of the MV storage the same as in VTM-<NUM>, it is proposed in a further embodiment to use the non-refined MVs of DMVR blocks for the deblocking process. <FIG> illustrates an example of a DMVR process according to this embodiment. Specifically, like the method in <FIG>, the refined DMVR MVs are also used to generate the temporal motion predictors through TMVP and ATMVP in addition to the generation of the final bi-prediction signal. However, in the embodiment of <FIG>, the non-refined MVs are used to not only derive the spatial motion predictors (spatial AMVP and spatial merge) but also to determine the boundary strength for the deblocking filter of the current block.

In the method illustrated in <FIG>, non-refined motion vectors for a first block are identified at <NUM>. The non-refined motion vectors may have been signaled for the first block using any of a variety of available MV signaling techniques. The non-refined motion vectors are used at <NUM> to generate a bi-prediction template. At <NUM>, motion refinement is performed for the L0 motion vector, and at <NUM>, motion refinement is performed for the L1 motion vector. At <NUM>, the final bi-prediction of the first block is generated using the refined L0 and L1 motion vectors. In the method of <FIG>, the non-refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a second block) within the same picture as the first block. For example, the non-refined motion vectors are used for spatial AMVP (<NUM>) and as spatial merge candidates (<NUM>). The non-refined motion vectors are also used to calculate the boundary strength value of the deblocking filter (<NUM>). The refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a third block) in other pictures, for example using TMVP (<NUM>) or ATMVP (<NUM>).

The embodiments of <FIG> may reduce or remove the encoding/decoding latency caused by the DMVR, given that the dependency of the decoding of one block on the reconstruction of the refined MVs of its spatial neighboring DMVR blocks is not present in those embodiments. Based on the same example in <FIG>, <FIG> illustrates an example of a parallel decoding process when one of the methods of <FIG> is applied. As shown in <FIG>, because the decoding of multiple DMVR blocks can be performed in parallel, there is no decoding latency between neighboring blocks. Correspondingly, the total decoding time may be equal to the decoding of one block, which may be represented as TMCP + TDMVR +Tde-quant + Tinv-trans.

As pointed out above, one cause of encoding/decoding latency for DMVR is the dependency between the reconstruction of the refined MVs of a DMVR block and the decoding of its neighboring blocks, which is incurred by spatial motion prediction (e.g., spatial AMVP and spatial merge mode). Although methods such as those of <FIG> can remove or reduce the coding latency of the DMVR, this reduced latency may comes at the expense of degraded coding efficiency due to the less accurate non-refined MVs being used for the spatial motion prediction. On the other hand, as shown in <FIG>, the worst-case encoding/decoding latency introduced by the DMVR is directly related to the maximum number of consecutive blocks that are coded by the DMVR mode. To address these issues, in some embodiments, region-based methods are used to reduce the encoding/decoding latency while reducing the coding losses caused by using non-refined MVs for spatial motion prediction.

Specifically, in some embodiments, a picture is divided into a plurality of non-overlapped segments, and the non-refined MVs of each DMVR block in a segment are used as the predictors to predict the MVs of its neighboring blocks in the same segment. But, when a DMVR block is located on the right or bottom boundary of a segment, its non-refined MVs will not be used; instead, the refined MVs of the block are used as the predictor to predict the MVs of the blocks from the neighboring segment for better efficiency of spatial motion prediction.

<FIG> illustrates an example of a DMVR process according to one embodiment, and <FIG> illustrates an example in which the blank blocks represent the DMVR blocks that use the non-refined MVs for spatial motion prediction, spatial merge, and deblocking, and the patterned blocks represent the DMVR blocks that use the refined MVs for spatial motion prediction, spatial merge, and deblocking. In the example of <FIG>, the encoding/decoding of different inter blocks inside the same segment can be performed independently from each other, while the decoding of the blocks from different segments are still dependent. For example, because the blocks on the left boundaries of segment #<NUM> may use the refined MVs of the neighboring DMVR blocks in segment #<NUM> as spatial MV predictors, their decoding processes cannot be started until the DMVR of those neighboring blocks in segment #<NUM> are fully done. Additionally, as shown in <FIG>, similar to the method in <FIG>, the same MVs of one DMVR block are used for the spatial motion predictions and the deblocking filter to avoid increasing the on-chip memory for storing MVs. In another embodiment, it is proposed to always use refined MVs for the deblocking process.

In the method illustrated in <FIG>, non-refined motion vectors for a first block are identified at <NUM>. The non-refined motion vectors may have been signaled for the first block using any of a variety of available MV signaling techniques. The non-refined motion vectors are used at <NUM> to generate a bi-prediction template. At <NUM>, motion refinement is performed for the L0 motion vector, and at <NUM>, motion refinement is performed for the L1 motion vector. At <NUM>, the final bi-prediction of the first block is generated using the refined L0 and L1 motion vectors.

At <NUM>, a determination is made of whether the first block is located on a right or bottom segment boundary. If the first block is not located on a right or bottom segment boundary, then the non-refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a second block) within the same picture as the first block. For example, the non-refined motion vectors are used for spatial AMVP (<NUM>) and as spatial merge candidates (<NUM>). The non-refined motion vectors are also used to calculate the boundary strength value of the deblocking filter (<NUM>). On the other hand, if the first block is located on a right or bottom segment boundary, then the refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a second block) within the same picture as the first block (e.g. with AMVP <NUM> and spatial merge candidates <NUM>), and the refined motion vectors are also used to calculate the boundary strength value of the deblocking filter (<NUM>). Regardless of the outcome of the determination at <NUM>, the refined motion vectors are used to predict the motion of subsequently-coded blocks (e.g. a third block) in other pictures, for example using TMVP (<NUM>) or ATMVP (<NUM>).

In the embodiment of <FIG>, the refined MVs are only enabled for spatial motion prediction of the blocks lying on the left/top boundaries of the segments inside one picture. However, depending on the segment size, the overall percentage of the blocks where the refined MVs can be applied for spatial motion prediction may be relatively small. The result may still be a non-negligible performance drop for the spatial motion prediction. To further improve the performance, it is proposed in some embodiments to allow the refined MVs of the DMVR blocks inside one segment to predict the MVs of the neighboring blocks inside the same segment. As a result, however, the decoding of multiple blocks inside one segment cannot be done in parallel. To improve the encoding/decoding parallelism, in this method, it is also proposed to prohibit the current block from using the MVs (either non-refined MV or refined MV) of a neighboring block that is from another segment as the predictors for the spatial motion predictions (e.g., spatial AMVP and spatial merge). Specifically, by such method, if a neighboring block is from a different segment to the current block, it will be treated as unavailable for spatial motion vector prediction.

One such embodiment is illustrated in <FIG>. In <FIG>, the blank blocks represent the CUs that are allowed to use the neighboring MVs (the neighboring MV could be refined MVs if the neighboring block is one DMVR block or non-refined MV if otherwise) for spatial motion predictions; the patterned blocks represent the CUs that are prevented from using the MVs of its neighboring blocks from a different segment for spatial motion prediction An embodiment according to <FIG> allows parallelized decoding of inter blocks across segments, but not within one segment.

In general, DMVR is only enabled for the bi-directional predicted CDs which have both the forward and backward prediction signals. Specifically, DMVR calls for use of two reference pictures: one with smaller picture order count (POC) and the other with larger POC than the POC of the current picture. In contrast, low-delay (LD) pictures are predicted from reference pictures that both precede the current picture in display order, with the POCs of all the reference pictures in L0 and L1 being smaller than the POC of the current picture. Therefore, the DMVR cannot be applied to LD pictures, and the coding latency caused by the DMVR does not exist in LD pictures. Based on such analysis, in some embodiments, when the DMVR is applied, it is proposed to only apply the above DMVR parallelism constraint (disabling the spatial motion predictions across segment boundaries) for non-LD pictures. For LD pictures, the constraint is not applied, and it is still permitted to predict the MVs of a current block based on the MVs of its spatial neighbors from another segment. In a further embodiment, the encoder/decoder determines whether the constraint is applied or not based on examining the POC of all the reference pictures in L0 and L1 without additional signaling. In another embodiment, it is proposed to add a picture/slice-level flag to indicate whether or not the DMVR parallelism constraint is applied to the current picture/slice.

In some embodiments, the number of segments and the position of each segment inside a picture/slice are selected by the encoder and signaled to the decoder. The signaling may be performed analogously to other parallelism tools in the HEVC and JEM (e.g., slices, tiles and wave-front parallel processing (WPP)). Various selections can lead to different trade-off between coding performance and encoding/decoding parallelism. In one embodiment, it is proposed to set the size of each segment equal to that of one CTU. In terms of signaling, syntax elements may be added at the sequence and/or picture level. For example, the number of CTUs in each segment may be signaled in the Sequence Parameter Set (SPS) and/or the Picture Parameter Set (PPS), or may be signaled in the slice header. Other variations of syntax elements may be used, for example, the number of CTU rows may be used, or the number of segment in each picture/slice may be used, among other alternatives.

Additional embodiments described herein operate to replace the block-matching motion search for calculating the DMVR motion refinement. Compared to a block-matching based method which performs motion search in a small local window, example embodiments directly calculate a motion refinement based on the spatial and temporal sample derivatives. Such embodiments reduce the computational complexity and may increase the motion refinement precision because the value of the derived refined motion is not limited to the search window.

As discussed above, BIO was used in the JEM to provide sample-wise motion refinement on top of the block-based motion compensated prediction when a block is bi-predicted. Based on the current design, BIO only enhances the motion compensated prediction samples as the outcome of the refinement without updating the MVs that are stored in the MV buffers and used for the spatial and temporal motion prediction and deblocking filter. This means that, as opposed to the current DMVR, BIO does not introduce any encoding/decoding latency between neighboring blocks. However, in the current BIO design, the motion refinement is derived on a small unit (e.g., 4x4). This may incur non-negligible computational complexity, especially at the decoder-side. This is undesirable for hardware codec implementations. Therefore, to address latency of the DMVR while maintaining acceptable coding complexity, it is proposed in some embodiments to use block-based BIO to calculate the local motion refinement for the video blocks that are coded by DMVR. Specifically, in a proposed embodiment, the BIO core design (e.g., the calculation of the gradients and the refined motion vectors) is kept the same as in the existing design to calculate the motion refinement. However, to reduce the complexity, the amount of motion refinement is derived based on CU-level, with a single value being aggregated for all the samples inside a CU and used to calculate a single motion refinement; and all samples inside the current CU will share the same motion refinement. Based on the same notations used above with respect to BIO, an example of a proposed block-level BIO motion refinement is derived as <MAT> where θ is set of the coordinates of the samples within the current CU, and where Δ(x, y) is an optical flow error metric as set forth in Equation <NUM>, above.

As indicated above, a motivation of BIO is to improve the precision of prediction samples based on the local gradient information at each sample location inside the current block. For large video blocks that contain many samples, it is possible that the local gradient at different sample locations may show quite varying characteristics. In such case, the above block-based BIO derivation may not provide a reliable motion refinement for the current block, therefore leading to coding performance loss. Based on such consideration, in some embodiments, it is proposed to only enable the CU-based BIO motion derivation for the DMVR blocks when its block size is small (e.g., no larger than one given threshold). Otherwise, the CU-based BIO motion derivation is disabled; instead, the existing block-matching based motion refinement (with the proposed DMVR latency removal/reduction methods described above being applied in some embodiments) will be used to derive the local motion refinement for the current block.

As noted above, BIO estimates the local motion refinement based on the assumption that the derived L0 and L1 motion refinements at each sample position are symmetric about the current picture, i.e., <MAT>, where <MAT> and <MAT> are the horizontal and vertical motion refinement associated with the prediction list L0 and L1. However, such assumption may be not true for the blocks that are coded by the DMVR. For example, in the existing DMVR (as shown in <FIG>), two separate block-matching based motion searches are performed for L0 and L1 such that the MVs that minimize the template costs of the L0 and L1 prediction signals may be different. Due to such symmetric motion constraint, the motion refinements derived by the BIO may not always be sufficiently accurate to enhance the prediction quality (sometimes may even degrade the prediction quality) for the DMVR.

In some embodiments, an improved motion derivation method is used to calculate the motion refinement for the DMVR. The classical optical flow model that states that the brightness of a picture keeps constant with the change of time, expressed as follows, <MAT> where x and y represent spatial coordinate and t represent time. The right-hand side of the equation can be expanded by Talyor's series about (x, y, t). After that, the optical flow equation becomes, to first order, <MAT>.

Using a camera's capturing time as the basic unit of time (e.g. setting dt = <NUM>), Eq. (<NUM>) can be discretized by changing the optical flow function from continuous domain to discrete domain. Let I(x, y) be the sample value captured from camera, then Eq. (<NUM>) becomes <MAT>.

In various embodiments, one or more error metrics may be defined based on an extent to which the expression on the left in Eq. (<NUM>) is not equal to zero. Motion refinements may be employed to substantially minimize the error metric.

In some embodiments, it is proposed to use the discretized optical flow model to estimate the local motion refinements in L0 and L1. Specifically, a bi-prediction template is generated by averaging the two prediction blocks using the initial L0 and L1 MVs of the merge candidate. However, instead of performing block-matching motion search in a local region, the optical flow model in Eq. (<NUM>) is used in some proposed embodiments to directly derive the refined MVs for each reference list L0/L1, as depicted as <MAT> where P<NUM> and P<NUM> are the prediction signals that are generated using the original MVs for the reference list L0 and L1, respectively, Ptmp is the bi-prediction template signal; <MAT> and <MAT> are the horizontal/vertical gradients of prediction signals P<NUM> and P<NUM>, which can be calculated based on different gradient filters, e.g., the Sobel filter or the 2D separable gradient filters used by BIO, as described in <NPL>. Equation (<NUM>) represents a set of equations: one equation for each sample in the prediction signal P<NUM> or P<NUM> for which one individual <MAT> and Ptmp - Pk can be calculated. With two unknown parameters Δxk and Δyk, the overdetermined problem can be solved by minimizing the sum of squared errors of Equation <NUM> as <MAT> where <MAT> is temporal difference between the L0/L1 prediction signal and the bi-prediction template signal and θ is the set of coordinates within the coding block. By solving the linear least mean squared error (LLMSE) problem in Equation (<NUM>), we can obtain the analytical expression of (Δx, Δy)*k as <MAT> <MAT> k = <NUM>,<NUM>.

Based on Equation (<NUM>), in some embodiments, to improve the precision of the derived MVs, such a method may select the motion refinements (i.e., (Δx, Δy)*k) in a recursive manner. Such embodiments may operate by generating the initial bi-prediction template signal using the original L0 and L1 MVs of the current block and calculating the corresponding delta motion (Δx, Δy)*k based on Eq. (<NUM>); the refined MVs are then used as the motion to generate the new L0 and L1 prediction samples as well as the bi-prediction template samples, which are then used to update the values of the local refinement (Δx, Δy)*k. This process may be repeated until the MVs are not updated or the maximum number of iterations is reached. One example of such a process is summarized by the following procedures, as illustrated in <FIG>.

At <NUM>, a counter l is initialized to l = <NUM>. At <NUM>, the initial L0 and L1 prediction signals <MAT> and <MAT> and the initial bi-prediction template signal <MAT> are generated using the original MVs <MAT> and <MAT> of the block. The local L0 and L1 motion refinements <MAT> and <MAT> based on Equation (<NUM>) at <NUM> and <NUM>, and the MVs of the block are updated as <MAT> and <MAT>.

If <MAT> and <MAT> are zero (determined at <NUM>) or if l = lmax (determined at <NUM>), then the final bi-prediction may be generated at <NUM> using the refined motion vectors. Otherwise, the counter l is incremented at <NUM>, and the process is iterated, with the L0 and L1 prediction signals <MAT> and <MAT> and the bi-prediction template signal <MAT> being updated (at <NUM>, <NUM>) using the MVs <MAT> and <MAT>.

<FIG> illustrates an example of a DMVR process using an example optical-flow-based motion derivation method for calculating the motion refinements of a DMVR block. As shown in <FIG>, the optical MVs of one DMVR block is identified by iteratively modifying the original MVs based on the optical flow model. Although such a method can provide good motion estimation accuracy, it also introduces a high amount of complexity increase. To reduce the derivation complexity, in one embodiment of disclosure, it is proposed to only apply one iteration for deriving the motion refinement using the proposed motion derivation method, e.g., to only apply the process illustrated at <NUM> through <NUM> to derive the modified MVs of a DMVR block.

The optical-flow-based motion derivation model may be more efficient for small CUs than large CUs, due to the high consistency among the samples' characteristics inside a small block. In some embodiments, it is proposed to enable the proposed optical-flow-based motion derivation for the DMVR blocks when its block size is small (e.g., no larger than a given threshold). Otherwise, the existing block-matching based motion refinement will be used to derive the local motion refinement for the current block (e.g. along with the proposed DMVR latency removal/reduction methods described herein).

Note that various hardware elements of one or more of the described embodiments are referred to as "modules" that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc..

Claim 1:
A video decoding method comprising:
at a first block, refining (<NUM>, <NUM>) a first non-refined motion vector and a second non-refined motion vector to generate a first refined motion vector and a second refined motion vector, wherein refining of the first non-refined motion vector and the second non-refined motion vector is performed using decoder-side motion vector refinement (DMVR);
using one or both of the first non-refined motion vector and the second non-refined motion vector, predicting (<NUM>, <NUM>) motion information of a second block, the second block being a spatial neighbor of the first block;
predicting (<NUM>) the first block with bi-prediction using the first refined motion vector and the second refined motion vector;
determining (<NUM>) a deblocking filter strength for the first block based at least in part on the first non-refined motion vector and the second non-refined motion vector; and
predicting (<NUM>, <NUM>) motion information of a third block using at least one of the first refined motion vector and the second refined motion vector, wherein the third block and the first block are collocated blocks in different pictures.