Patent Publication Number: US-2022232204-A1

Title: Control of memory bandwidth consumption of affine mode in versatile video coding

Description:
CROSS-REFERENCES TO RELATED APPLICATIONS 
     The present application is a continuation of U.S. patent application Ser. No. 16/665,484, filed Oct. 28, 2019, which claims the benefit of U.S. Provisional Patent Application Nos. 62/757,004, filed Nov. 7, 2018; 62/769,875, filed Nov. 20, 2018; and 62/792,195, filed Jan. 14, 2019; each of which are hereby incorporated by reference in their entireties into the present disclosure. 
    
    
     BACKGROUND 
     1. Field 
     One or more aspects of the disclosed subject matter are directed to video encoding or decoding and more specifically to video encoding or decoding with memory bandwidth conservation when an affine motion model is used. 
     2. Description of Related Art 
     The VVC (Versatile Video Coding) is a new video compression standard being developed by the joint video experts team (JVET) jointly established by ISO/IEO MPEG and ITU-T. The VVC standard for single layer coding will be finalized by the end of 2020, with a design goal of being at least 50% more efficient than the previous standard MPEG HEVC/ITU-T H.265 Main-10 profile. 
     Among proposed coding tools to VVC under consideration, the affine motion compensation prediction introduces a more complex motion model for better compression efficiency. In previous standards such as HEVC, only a translational motion model is considered, in which all the sample positions inside a PU (prediction unit) may have a same translational motion vector for motion compensated prediction. However, in the real world, there are many kinds of motion, e.g., zoom in/out, rotation, perspective motions and other irregular motions. The affine motion model supports different motion vectors at different sample positions inside a PU, which effectively captures more complex motion. Different sample positions inside a PU, such as four corner points of the PU, may have different motion vectors as supported by the affine mode. A PU coded in affine mode and affine merge mode may have uni-prediction (list 0 or list 1 prediction) or bi-directional prediction (i.e. list 0 and list 1 bi-prediction). 
     In the current VVC design (see JVET-P2001, “Versatile Video Coding (Draft 7)”), the sub-block size for the affine mode is fixed to 4×4, which creates the 4×4 bi-directional prediction for the worst-case memory bandwidth consumption of motion compensation. In the HEVC, the worst-case memory bandwidth consumption for motion compensation is 8×8 bidirectional prediction, while 8×4 and 4×8 PUs use uni-prediction only. The increased memory bandwidth budget can never catch up with the path of sample rate increase (e.g., HEVC is typically for 4K video at 60 fps, while VVC will be used for 8K video at 60 fps, another factor of 4 increase in terms of sample processing rate). 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       One or more aspects of the disclosed subject matter will be set forth in detail with reference to the drawings, in which: 
       The present application claims the benefit of U.S. Provisional Patent Application Nos. 62/757,004, filed Nov. 7, 2018; 62/769,875, filed Nov. 20, 2018, and 62/792,195, filed Jan. 14, 2019, whose disclosures are hereby incorporated by reference in their entireties into the present disclosure. 
         FIG. 1A  is a flow chart showing an overview of one or more aspects of the disclosed subject matter; 
         FIG. 1B  illustrates an affine motion model according to one or more aspects of the disclosed subject matter; 
         FIG. 1C  depicts a 4-parameter affine motion model used in the VVC according to one or more aspects of the disclosed subject matter; 
         FIG. 1D  depicts a 6-parameter affine motion model used in the VVC according to one or more aspects of the disclosed subject matter; 
         FIG. 2  shows 4 (i.e. 2×2) sub-block motion vectors in a PU coded in affine mode and bi-directional inter prediction; 
         FIG. 3  shows a reference block bounding box of 4 sub-block motion vectors in a PU coded in affine mode and bi-directional inter prediction; 
         FIG. 4  shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (horizontal direction); 
         FIG. 5  shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (vertical direction); 
         FIG. 6  is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode according to one or more aspects of the disclosed subject matter; 
         FIG. 7  is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode according to one or more aspects of the disclosed subject matter; 
         FIG. 8  is an algorithmic flow chart of a method for controlling the sub-block motion vector spread in the affine motion estimation on the encoder side according to one or more aspects of the disclosed subject matter; 
         FIG. 9  is an algorithmic flow chart of a method for determining block size according to one or more aspects of the disclosed subject matter; and 
         FIG. 10  is a block diagram showing an example of circuitry on which any of the preferred embodiments, or any other embodiment, can be implemented. 
     
    
    
     DETAILED DESCRIPTION 
     The description set forth in detail with reference to the drawings, in which like reference numerals refer to like elements or operations throughout. 
       FIG. 1A  is a flow chart showing an overview of one or more aspects of the disclosed subject matter. In step  102 , the reference block bounding box size and at least one threshold are determined. In step  104 , it is determined whether the reference block bounding box size exceeds the at least one threshold. If so, the affine motion compensation is performed in step  106  using a first motion compensation operation. In response to a determination that the reference block bounding size does not exceed the predefined threshold, the affine motion compensation is performed in step  108  using a second affine motion compensation operation that is different from the first motion compensation operation. Either way, the results are output in step  110 . 
       FIG. 1B  shows an affine motion model according to one or more aspects of the disclosed subject matter. Regarding the affine motion model, the origin (0, 0) of the x-y coordinate system can be at the top-left corner point of a picture as illustrated in  FIG. 1B . Similarly, other drawings in the description having an x-y coordinate system can also have the origin (0, 0) of the x-y coordinate at the top-left corner of a picture. Generally, a PU coded in affine mode and affine merge mode can have uni-prediction or bi-directional prediction. The uni-prediction can correspond to a list 0 or list 1 prediction, while the bi-directional prediction can correspond to a list 0 and list 1 bi-prediction. Although various algorithmic descriptions further described herein focus on the uni-prediction mode to explain the algorithm, it should be appreciated that if a PU is coded in bi-directional affine or bi-directional affine merge mode, the process of affine mode and affine merge mode described herein are performed separately for list 0 and list 1 predictions. In the affine motion model, the motion vector {right arrow over (v)}=(v x ,v y ) at a sample position (x,y) inside a PU is defined as follows: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             x 
                           
                           = 
                           
                             
                               a 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               x 
                             
                             + 
                             
                               c 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               y 
                             
                             + 
                             e 
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             y 
                           
                           = 
                           
                             
                               b 
                               ⁢ 
                               x 
                             
                             + 
                             
                               d 
                               ⁢ 
                               y 
                             
                             + 
                             f 
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   1 
                 
               
             
           
         
       
     
     where a, b, c, d, e, f are the affine motion model parameters, which define a 6-parameter affine motion model (see  FIG. 1D ). 
       FIG. 1C  shows a 4-parameter affine motion model according to one or more aspects of the disclosed subject matter. The 4-parameter affine motion model can be a restricted affine motion model because the parameters are restricted. For example, the 4-parameter model can be described with four parameters by restricting a=d and b=−c in Equation 1: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             x 
                           
                           = 
                           
                             
                               a 
                               ⁢ 
                               x 
                             
                             - 
                             
                               b 
                               ⁢ 
                               y 
                             
                             + 
                             e 
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             y 
                           
                           = 
                           
                             
                               b 
                               ⁢ 
                               x 
                             
                             + 
                             
                               a 
                               ⁢ 
                               y 
                             
                             + 
                             f 
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   2 
                 
               
             
           
         
       
     
     In the 4-parameter affine motion model, the model parameters a, b, e, f are determined by signaling two control point vectors at the top-left and top-right corner of a PU.  FIG. 1C  shows two control point vectors {right arrow over (v)}=(v 0x ,v 0y ) at sample position (x 0 ,y 0 ) and {right arrow over (v)}=(v 1x ,v 1y ) at sample position (x 1 ,y 1 ). Accordingly, Equation 2 can be rewritten as: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             x 
                           
                           = 
                           
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         1 
                                         ⁢ 
                                         x 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         x 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       x 
                                       1 
                                     
                                     - 
                                     
                                       x 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   x 
                                   - 
                                   
                                     x 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             - 
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         1 
                                         ⁢ 
                                         y 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         y 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       x 
                                       1 
                                     
                                     - 
                                     
                                       x 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   y 
                                   - 
                                   
                                     y 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             + 
                             
                               v 
                               
                                 0 
                                 ⁢ 
                                 x 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             y 
                           
                           = 
                           
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         1 
                                         ⁢ 
                                         y 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         y 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       x 
                                       1 
                                     
                                     - 
                                     
                                       x 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   x 
                                   - 
                                   
                                     x 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             + 
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         1 
                                         ⁢ 
                                         x 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         x 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       x 
                                       1 
                                     
                                     - 
                                     
                                       x 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   y 
                                   - 
                                   
                                     y 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             + 
                             
                               v 
                               
                                 0 
                                 ⁢ 
                                 y 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   3 
                 
               
             
           
         
       
     
     It should be appreciated that in  FIG. 1C , (x 1 −x 0 ) is equal to the PU width and y 1 =y 0 . Accordingly, the two control point vectors do not have to be at the top-left and top-right corner of a PU to derive the parameters of the 4-parameter affine motion model. As long as the two control points have x 1 ≠x 0  and y 1 =y 0 , Equation 3 is valid. 
       FIG. 1D  shows a 6-parameter affine motion model according to one or more aspects of the disclosed subject matter. The model parameters for the 6-parameter affine motion model can be determined by signaling three control point vectors at the top-left, top-right, and bottom-left corner of a PU. For example, with three control point vectors {right arrow over (v)}=(v 0x ,v 0y ) at sample position (x 0 ,y 0 ), {right arrow over (v)}=(v 1x ,v 1y ) at sample position (x 1 ,y 1 ) and {right arrow over (v)}=(v 2x ,v 2y ) at sample position (x 2 ,y 2 ), Equation 1 can be rewritten as: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             x 
                           
                           = 
                           
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         1 
                                         ⁢ 
                                         x 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         x 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       x 
                                       1 
                                     
                                     - 
                                     
                                       x 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   x 
                                   - 
                                   
                                     x 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             + 
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         2 
                                         ⁢ 
                                         x 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         x 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       y 
                                       2 
                                     
                                     - 
                                     
                                       y 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   y 
                                   - 
                                   
                                     y 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             + 
                             
                               v 
                               
                                 0 
                                 ⁢ 
                                 x 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             y 
                           
                           = 
                           
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         1 
                                         ⁢ 
                                         y 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         y 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       x 
                                       1 
                                     
                                     - 
                                     
                                       x 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   x 
                                   - 
                                   
                                     x 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             + 
                             
                               
                                 
                                   ( 
                                   
                                     
                                       v 
                                       
                                         2 
                                         ⁢ 
                                         y 
                                       
                                     
                                     - 
                                     
                                       v 
                                       
                                         0 
                                         ⁢ 
                                         y 
                                       
                                     
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   
                                     
                                       y 
                                       2 
                                     
                                     - 
                                     
                                       y 
                                       0 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   y 
                                   - 
                                   
                                     y 
                                     0 
                                   
                                 
                                 ) 
                               
                             
                             + 
                             
                               v 
                               
                                 0 
                                 ⁢ 
                                 y 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   4 
                 
               
             
           
         
       
     
     It should be appreciated that in  FIG. 1D , (x 1 −x 0 ) is equal to the PU width, (y 2 −y 0 ) is equal to the PU height, y 1 =y 0 , and x 2 =x 0 . Accordingly, the three control point vectors do not have to be at the top-left, top-right and bottom-left corner of a PU as shown in  FIG. 1D  to derive the parameters of the 6-parameter affine motion model. As long as the three control points have x 1 ≠x 0 , y 2 ≠y 0 , y 1 =y 0  and x 2 =x 0 , Equation 4 is valid. 
     Further, to constrain the memory bandwidth consumption of the affine mode for motion compensation, the motion vectors of a PU coded in affine mode are not derived for each sample in a PU. For example, as shown in  FIG. 1C  and  FIG. 1D , all the samples inside a sub-block (e.g. 4×4 block size) of the PU can share a same motion vector. Deriving the motion vector can be based on a sub-block motion data derivation process of the affine mode where the motion vector is derived at sample position (x,y) chosen for the sub-block and by using Equation 3 or Equation 4, and whether to use Equation 3 or Equation 4 depends on the type of affine motion model. In the current VVC design, the sub-block size is fixed to 4×4 and the sample position chosen for deriving the sub-block motion vector field of a PU coded in affine mode is the center point of each 4×4 sub-block of the PU. 
     The concepts of determining the reference block bounding size will be described.  FIG. 2  shows 4 (i.e. 2×2) sub-block motion vectors in a PU coded in affine mode and bi-directional inter prediction and depicts the geometric relationship of 4 sub-block vectors in a PU coded in affine mode and bi-directional inter prediction. The 4 sub-block vectors (with sub-block size m*n) can be any 4 sub-block vectors within the PU whose locations satisfy the following conditions: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             x 
                             
                               s 
                               ⁢ 
                               1 
                             
                           
                           = 
                           
                             
                               x 
                               
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             m 
                           
                         
                       
                     
                     
                       
                         
                           
                             y 
                             
                               s 
                               ⁢ 
                               1 
                             
                           
                           = 
                           
                             y 
                             
                               s 
                               ⁢ 
                               0 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             x 
                             
                               s 
                               ⁢ 
                               2 
                             
                           
                           = 
                           
                             x 
                             
                               s 
                               ⁢ 
                               0 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             y 
                             
                               s 
                               ⁢ 
                               2 
                             
                           
                           = 
                           
                             
                               y 
                               
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             n 
                           
                         
                       
                     
                     
                       
                         
                           
                             x 
                             
                               s 
                               ⁢ 
                               3 
                             
                           
                           = 
                           
                             
                               x 
                               
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             m 
                           
                         
                       
                     
                     
                       
                         
                           
                             y 
                             
                               s 
                               ⁢ 
                               3 
                             
                           
                           = 
                           
                             
                               y 
                               
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             n 
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   5 
                 
               
             
           
         
       
     
     By substituting Equation 5 into the affine motion model, the 4 sub-block vectors are be derived by 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             
                               x 
                               ⁢ 
                               s 
                               ⁢ 
                               0 
                             
                           
                           = 
                           
                             
                               a 
                               ⁢ 
                               
                                 x 
                                 
                                   s 
                                   ⁢ 
                                   0 
                                 
                               
                             
                             + 
                             
                               c 
                               ⁢ 
                               
                                 y 
                                 
                                   s 
                                   ⁢ 
                                   0 
                                 
                               
                             
                             + 
                             e 
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             
                               y 
                               ⁢ 
                               s 
                               ⁢ 
                               0 
                             
                           
                           = 
                           
                             
                               b 
                               ⁢ 
                               
                                 x 
                                 
                                   s 
                                   ⁢ 
                                   0 
                                 
                               
                             
                             + 
                             
                               d 
                               ⁢ 
                               
                                 y 
                                 
                                   s 
                                   ⁢ 
                                   0 
                                 
                               
                             
                             + 
                             f 
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   6 
                 
               
             
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             
                               x 
                               ⁢ 
                               s 
                               ⁢ 
                               1 
                             
                           
                           = 
                           
                             
                               v 
                               
                                 x 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             
                               a 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               m 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             
                               y 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               s 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               1 
                             
                           
                           = 
                           
                             
                               v 
                               
                                 y 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 s0 
                               
                             
                             + 
                             
                               b 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               m 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   7 
                 
               
             
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             
                               x 
                               ⁢ 
                               s 
                               ⁢ 
                               2 
                             
                           
                           = 
                           
                             
                               v 
                               
                                 x 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             
                               c 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               n 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             
                               y 
                               ⁢ 
                               s 
                               ⁢ 
                               2 
                             
                           
                           = 
                           
                             
                               v 
                               
                                 y 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             
                               d 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               n 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   8 
                 
               
             
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             v 
                             
                               x 
                               ⁢ 
                               s 
                               ⁢ 
                               3 
                             
                           
                           = 
                           
                             
                               v 
                               
                                 x 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             
                               a 
                               ⁢ 
                               m 
                             
                             + 
                             
                               c 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               n 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             v 
                             
                               y 
                               ⁢ 
                               s 
                               ⁢ 
                               3 
                             
                           
                           = 
                           
                             
                               v 
                               
                                 y 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             
                               b 
                               ⁢ 
                               m 
                             
                             + 
                             
                               d 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               n 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   9 
                 
               
             
           
         
       
     
     The parameters of the affine motion model, i.e. (a, b, c d), can be calculated in any suitable way such as using Equation 3 or 4. 
     For motion compensation, reference blocks are loaded around the co-located sub-block locations with the offsets determined by the sub-block motion vectors.  FIG. 3  shows a reference block bounding box of 4 sub-block motion vectors in a PU coded in affine mode and bi-directional inter prediction. Assuming (x 0 ,y 0 ) are the coordinate of the sub-block 0 in  FIG. 3  and all the sub-blocks in the PU have equal size m*n, and f x *f y  are filter taps used in the motion compensation, the coordinates of upper-left and bottom-right locations of the reference blocks are listed in Table 1 below (using Equations 7 to 9): 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 Coordinates of upper-left and bottom-right corners of reference blocks of 
               
               
                 four sub-block vectors 
               
            
           
           
               
               
               
               
            
               
                   
                 Coordinates 
                   
                   
               
               
                   
                 of upper-left  
                   
                   
               
               
                 Sub- 
                 corner of the 
                 Coordinates of upper-left corner of 
                 Coordinates of bottom-right corner of 
               
               
                 block 
                 sub-block 
                 the reference block 
                 the reference block 
               
               
                   
               
            
           
           
               
               
               
               
            
               
                 0 
                 (x 0 , y 0 ) 
                 
                   
                     
                       
                         ( 
                         
                           
                             
                               x 
                               0 
                             
                             + 
                             
                               v 
                               
                                 x 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             - 
                             
                               
                                 f 
                                 x 
                               
                               2 
                             
                             + 
                             1 
                           
                           , 
                             
                           
                             
                               y 
                               0 
                             
                             + 
                             
                               v 
                               
                                 y 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             - 
                             
                               
                                 f 
                                 y 
                               
                               2 
                             
                             + 
                             1 
                           
                         
                         ) 
                       
                     
                   
                 
                 
                   
                     
                       
                         ( 
                         
                           
                             
                               x 
                               0 
                             
                             + 
                             
                               v 
                               
                                 x 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             
                               
                                 f 
                                 x 
                               
                               2 
                             
                             + 
                             m 
                           
                           , 
                           
                             
                               y 
                               0 
                             
                             + 
                             
                               v 
                               
                                 y 
                                 ⁢ 
                                 s 
                                 ⁢ 
                                 0 
                               
                             
                             + 
                             
                               
                                 f 
                                 y 
                               
                               2 
                             
                             + 
                             n 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                   
               
               
                 1 
                 (x 0  + m, y 0 ) 
                 
                   
                     
                       
                         
                           
                             
                               
                                 ( 
                                 
                                   
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           a 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       m 
                                     
                                     + 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           0 
                                         
                                         + 
                                         
                                           v 
                                           
                                             x 
                                             ⁢ 
                                             s 
                                             ⁢ 
                                             0 
                                           
                                         
                                         - 
                                         
                                           
                                             f 
                                             x 
                                           
                                           2 
                                         
                                         + 
                                         1 
                                       
                                       ) 
                                     
                                   
                                   , 
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   bm 
                                   + 
                                   
                                     ( 
                                     
                                       
                                         y 
                                         0 
                                       
                                       + 
                                       
                                         v 
                                         
                                           y 
                                           ⁢ 
                                           s 
                                           ⁢ 
                                           0 
                                         
                                       
                                       - 
                                       
                                         
                                           f 
                                           y 
                                         
                                         2 
                                       
                                       + 
                                       1 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                          
                       
                     
                   
                 
                 
                   
                     
                       
                         
                           
                             
                               
                                 ( 
                                 
                                   
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           a 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       m 
                                     
                                     + 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           0 
                                         
                                         + 
                                         
                                           v 
                                           
                                             x 
                                             ⁢ 
                                             s 
                                             ⁢ 
                                             0 
                                           
                                         
                                         + 
                                         
                                           
                                             f 
                                             x 
                                           
                                           2 
                                         
                                         + 
                                         m 
                                       
                                       ) 
                                     
                                   
                                   , 
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   bm 
                                   + 
                                   
                                     ( 
                                     
                                       
                                         y 
                                         0 
                                       
                                       + 
                                       
                                         v 
                                         
                                           y 
                                           ⁢ 
                                           s 
                                           ⁢ 
                                           0 
                                         
                                       
                                       + 
                                       
                                         
                                           f 
                                           y 
                                         
                                         2 
                                       
                                       + 
                                       n 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                         ⁢ 
                         
                            
                            
                         
                       
                     
                   
                 
               
               
                   
               
               
                 2 
                 (x 0 , y 0  + n) 
                 
                   
                     
                       
                         
                           
                             
                               
                                 ( 
                                 
                                   
                                     cn 
                                     + 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           0 
                                         
                                         + 
                                         
                                           v 
                                           
                                             x 
                                             ⁢ 
                                             s 
                                             ⁢ 
                                             0 
                                           
                                         
                                         - 
                                         
                                           
                                             f 
                                             x 
                                           
                                           2 
                                         
                                         + 
                                         1 
                                       
                                       ) 
                                     
                                   
                                   , 
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                   + 
                                   
                                     ( 
                                     
                                       
                                         y 
                                         0 
                                       
                                       + 
                                       
                                         v 
                                         
                                           y 
                                           ⁢ 
                                           s 
                                           ⁢ 
                                           0 
                                         
                                       
                                       - 
                                       
                                         
                                           f 
                                           y 
                                         
                                         2 
                                       
                                       + 
                                       1 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                          
                       
                     
                   
                 
                 
                   
                     
                       
                         
                           
                             
                               
                                 ( 
                                 
                                   
                                     cn 
                                     + 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           0 
                                         
                                         + 
                                         
                                           v 
                                           
                                             x 
                                             ⁢ 
                                             s 
                                             ⁢ 
                                             0 
                                           
                                         
                                         + 
                                         
                                           
                                             f 
                                             x 
                                           
                                           2 
                                         
                                         + 
                                         m 
                                       
                                       ) 
                                     
                                   
                                   , 
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                   + 
                                   
                                     ( 
                                     
                                       
                                         y 
                                         0 
                                       
                                       + 
                                       
                                         v 
                                         
                                           y 
                                           ⁢ 
                                           s 
                                           ⁢ 
                                           0 
                                         
                                       
                                       + 
                                       
                                         
                                           f 
                                           y 
                                         
                                         2 
                                       
                                       + 
                                       n 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                          
                       
                     
                   
                 
               
               
                   
               
               
                 3 
                 (x 0  + m, y 0  + n) 
                 
                   
                     
                       
                         
                           
                             
                               
                                 ( 
                                 
                                   
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           a 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       m 
                                     
                                     + 
                                     cn 
                                     + 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           0 
                                         
                                         + 
                                         
                                           v 
                                           
                                             x 
                                             ⁢ 
                                             s 
                                             ⁢ 
                                             0 
                                           
                                         
                                         - 
                                         
                                           
                                             f 
                                             x 
                                           
                                           2 
                                         
                                         + 
                                         1 
                                       
                                       ) 
                                     
                                   
                                   , 
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   bm 
                                   + 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                   + 
                                   
                                     ( 
                                     
                                       
                                         y 
                                         0 
                                       
                                       + 
                                       
                                         v 
                                         
                                           y 
                                           ⁢ 
                                           s 
                                           ⁢ 
                                           0 
                                         
                                       
                                       - 
                                       
                                         
                                           f 
                                           y 
                                         
                                         2 
                                       
                                       + 
                                       1 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                          
                       
                     
                   
                 
                 
                   
                     
                       
                         
                           
                             
                               
                                 ( 
                                 
                                   
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           a 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       m 
                                     
                                     + 
                                     cn 
                                     + 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           0 
                                         
                                         + 
                                         
                                           v 
                                           
                                             x 
                                             ⁢ 
                                             s 
                                             ⁢ 
                                             0 
                                           
                                         
                                         + 
                                         
                                           
                                             f 
                                             x 
                                           
                                           2 
                                         
                                         + 
                                         m 
                                       
                                       ) 
                                     
                                   
                                   , 
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   bm 
                                   + 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                   + 
                                   
                                     ( 
                                     
                                       
                                         y 
                                         0 
                                       
                                       + 
                                       
                                         v 
                                         
                                           y 
                                           ⁢ 
                                           s 
                                           ⁢ 
                                           0 
                                         
                                       
                                       + 
                                       
                                         
                                           f 
                                           y 
                                         
                                         2 
                                       
                                       + 
                                       n 
                                     
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                          
                       
                     
                   
                 
               
               
                   
               
            
           
         
       
     
     Based on the coordinates listed in Table 1, the coordinates of upper-left and bottom-right corners of the reference block bounding box in  FIG. 3 , i.e. (xb ul ,yb ul ) and (xb br ,yb br ), are defined as: 
     
       
         
           
             
               
                   
               
               
                 
                   Equation 
                   ⁢ 
                       
                   10 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             x 
                             ⁢ 
                             
                               b 
                               
                                 u 
                                 ⁢ 
                                 l 
                               
                             
                           
                           = 
                           
                             
                               min 
                               ⁡ 
                               ( 
                               
                                 0 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       a 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   m 
                                 
                                 , 
                                 cn 
                                 , 
                                 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     m 
                                   
                                   + 
                                   
                                     c 
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                             + 
                             
                               ( 
                               
                                 
                                   x 
                                   0 
                                 
                                 + 
                                 
                                   v 
                                   
                                     x 
                                     ⁢ 
                                     s 
                                     ⁢ 
                                     0 
                                   
                                 
                                 - 
                                 
                                   
                                     f 
                                     x 
                                   
                                   2 
                                 
                                 + 
                                 1 
                               
                               ) 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             y 
                             ⁢ 
                             
                               b 
                               
                                 u 
                                 ⁢ 
                                 l 
                               
                             
                           
                           = 
                           
                             
                               min 
                               ⁡ 
                               ( 
                               
                                 0 
                                 , 
                                 bm 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       d 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   n 
                                 
                                 , 
                                 
                                   bm 
                                   + 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                             + 
                             
                               ( 
                               
                                 
                                   y 
                                   0 
                                 
                                 + 
                                 
                                   v 
                                   
                                     y 
                                     ⁢ 
                                     s 
                                     ⁢ 
                                     0 
                                   
                                 
                                 - 
                                 
                                   
                                     f 
                                     y 
                                   
                                   2 
                                 
                                 + 
                                 1 
                               
                               ) 
                             
                           
                         
                       
                     
                   
                 
               
               
                    
               
             
           
         
       
       
         
           
             
                 
               
                 
                   Equation 
                   ⁢ 
                       
                   11 
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       x 
                       ⁢ 
                       
                         b 
                         br 
                       
                     
                     = 
                     
                       
                         max 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           
                             
                               ( 
                               
                                 1 
                                 + 
                                 a 
                               
                               ) 
                             
                             ⁢ 
                             m 
                           
                           , 
                           cn 
                           , 
                           
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   a 
                                 
                                 ) 
                               
                               ⁢ 
                               m 
                             
                             + 
                             
                               c 
                               ⁢ 
                               n 
                             
                           
                         
                         ) 
                       
                       + 
                       
                         ( 
                         
                           
                             x 
                             0 
                           
                           + 
                           
                             v 
                             
                               x 
                               ⁢ 
                               s 
                               ⁢ 
                               0 
                             
                           
                           + 
                           
                             
                               f 
                               x 
                             
                             2 
                           
                           + 
                           m 
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   
                     
                       y 
                       ⁢ 
                       
                         b 
                         br 
                       
                     
                     = 
                     
                       
                         max 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           bm 
                           , 
                           
                             
                               ( 
                               
                                 1 
                                 + 
                                 d 
                               
                               ) 
                             
                             ⁢ 
                             n 
                           
                           , 
                           
                             bm 
                             + 
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   d 
                                 
                                 ) 
                               
                               ⁢ 
                               n 
                             
                           
                         
                         ) 
                       
                       + 
                       
                         ( 
                         
                           
                             y 
                             0 
                           
                           + 
                           
                             v 
                             
                               y 
                               ⁢ 
                               s 
                               ⁢ 
                               0 
                             
                           
                           + 
                           
                             
                               f 
                               y 
                             
                             2 
                           
                           + 
                           n 
                         
                         ) 
                       
                     
                   
                 
               
             
           
         
       
     
     where max( ) and min( ) are functions used to return the largest and the smallest value from a set of data, respectively. 
     By using Equations 6 and 7, the width and height of the reference block bounding box, i.e. (bxW4,bxH4) can be computed by: 
     
       
         
           
             
               
                   
               
               
                 
                   Equation 
                   ⁢ 
                       
                   12 
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       
                         
                           
                             bxW 
                             ⁢ 
                             4 
                           
                           = 
                           
                             
                               max 
                               ⁡ 
                               ( 
                               
                                 0 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       a 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   m 
                                 
                                 , 
                                 cn 
                                 , 
                                   
                                 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     m 
                                   
                                   + 
                                   
                                     c 
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁢ 
                             
                               ( 
                               
                                 0 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       a 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   m 
                                 
                                 , 
                                 cn 
                                 , 
                                 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     m 
                                   
                                   + 
                                   
                                     c 
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                           
                           + 
                           
                             f 
                             x 
                           
                           + 
                           m 
                           - 
                           1 
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       
                         
                           
                             bxH 
                             ⁢ 
                             4 
                           
                           = 
                           
                             
                               max 
                               ⁡ 
                               ( 
                               
                                 0 
                                 , 
                                 bm 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       d 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   n 
                                 
                                 , 
                                 
                                   bm 
                                   + 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁡ 
                             ( 
                             
                               0 
                               , 
                               bm 
                               , 
                               
                                 
                                   ( 
                                   
                                     1 
                                     + 
                                     d 
                                   
                                   ) 
                                 
                                 ⁢ 
                                 n 
                               
                               , 
                               
                                 bm 
                                 + 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       d 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   n 
                                 
                               
                             
                             ) 
                           
                           + 
                           
                             f 
                             y 
                           
                           + 
                           n 
                           - 
                           1 
                         
                       
                     
                   
                 
               
             
           
         
       
     
       FIG. 4  shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (horizontal direction). The width and height of the reference block bounding box (bxW h ,bxH h ) for 2 sub-block motion vectors in a PU, which is coded in affine mode and unidirectional inter prediction, can be computed by: 
     
       
         
           
             
               
                   
               
               
                 
                   Equation 
                   ⁢ 
                       
                   13 
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         W 
                         h 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   a 
                                 
                                 ) 
                               
                               ⁢ 
                               m 
                             
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   a 
                                 
                                 ) 
                               
                               ⁢ 
                               m 
                             
                           
                           ) 
                         
                       
                       + 
                       
                         f 
                         x 
                       
                       + 
                       m 
                       - 
                       1 
                     
                   
                 
               
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         H 
                         h 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             bm 
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             bm 
                           
                           ) 
                         
                       
                       + 
                       
                         f 
                         y 
                       
                       + 
                       n 
                       - 
                       1 
                     
                   
                 
               
             
           
         
       
     
     If the PU coded in affine mode uses unidirectional inter prediction, the reference block bounding box can also be drawn in the vertical direction.  FIG. 5  shows a reference block bounding box of 2 sub-block motion vectors in a PU coded in affine mode and unidirectional inter prediction (vertical direction). As shown in  FIG. 5 , in this case, the reference block bounding box size, i.e. (bxW v ,bxH v ), can be computed by (see also Table 1 sub-block 0 and 2): 
     
       
         
           
             
               
                   
               
               
                 
                   Equation 
                   ⁢ 
                       
                   14 
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         W 
                         v 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             cn 
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             cn 
                           
                           ) 
                         
                       
                       + 
                       
                         f 
                         x 
                       
                       + 
                       m 
                       - 
                       1 
                     
                   
                 
               
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         H 
                         v 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   d 
                                 
                                 ) 
                               
                               ⁢ 
                               n 
                             
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   d 
                                 
                                 ) 
                               
                               ⁢ 
                               n 
                             
                           
                           ) 
                         
                       
                       + 
                       
                         f 
                         y 
                       
                       + 
                       n 
                       - 
                       1 
                     
                   
                 
               
             
           
         
       
     
     From Equation 12, Equation 13 and Equation 14, it can be seen that the reference block bounding block size is independent of sub-block locations inside the PU; it purely depends on the parameters of the affine motion model (i.e. a, b, c, d), sub-block size (i.e. m*n) and filter tap lengths (i.e. f x *f y ) used for motion compensation. 
     In the current VVC design (JVET-P2001), the sub-block size used for the affine mode is 4×4 (i.e. m=n=4), and the filter tap used for luma motion compensation of the affine mode is 6×6 (i.e. f x =f y =6). The reference block bounding box sizes for the VVC are defined in Equations 15, 16 and 17. 
     
       
         
           
             
                 
               
                 
                   Equation 
                   ⁢ 
                      
                   15 
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       
                         
                           bxW4 
                           = 
                           
                             
                               max 
                               ⁢ 
                               
                                 ( 
                                 
                                   0 
                                   , 
                                   
                                     4 
                                     ⁢ 
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                   
                                   , 
                                     
                                   
                                     4 
                                     ⁢ 
                                     c 
                                   
                                   , 
                                     
                                   
                                     
                                       4 
                                       ⁢ 
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           a 
                                         
                                         ) 
                                       
                                     
                                     + 
                                     
                                       4 
                                       ⁢ 
                                       c 
                                     
                                   
                                 
                                 ) 
                               
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁢ 
                             
                               ( 
                               
                                 0 
                                 , 
                                 
                                   4 
                                   ⁢ 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       a 
                                     
                                     ) 
                                   
                                 
                                 , 
                                   
                                 
                                   4 
                                   ⁢ 
                                   c 
                                 
                                 , 
                                 
                                   
                                     4 
                                     ⁢ 
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                   
                                   + 
                                   
                                     4 
                                     ⁢ 
                                     c 
                                   
                                 
                               
                               ) 
                             
                           
                           + 
                           9 
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       
                         
                           bxH4 
                           = 
                           
                             
                               max 
                               ⁢ 
                               
                                 ( 
                                 
                                   0 
                                   , 
                                   
                                     4 
                                     ⁢ 
                                     b 
                                   
                                   , 
                                     
                                   
                                     4 
                                     ⁢ 
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                   
                                   , 
                                     
                                   
                                     
                                       4 
                                       ⁢ 
                                       b 
                                     
                                     + 
                                     
                                       4 
                                       ⁢ 
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           d 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                                 ) 
                               
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁢ 
                             
                               ( 
                               
                                 0 
                                 , 
                                 
                                   4 
                                   ⁢ 
                                   b 
                                 
                                 , 
                                   
                                 
                                   4 
                                   ⁢ 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       d 
                                     
                                     ) 
                                   
                                 
                                 , 
                                   
                                 
                                   
                                     4 
                                     ⁢ 
                                     b 
                                   
                                   + 
                                   
                                     4 
                                     ⁢ 
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                   
                                 
                               
                               ) 
                             
                           
                           + 
                           9 
                         
                       
                     
                   
                 
               
             
           
         
       
       
         
           
             
                 
               
                 
                   Equation 
                   ⁢ 
                       
                   16 
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         W 
                         h 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               4 
                               ⁢ 
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   a 
                                 
                                 ) 
                               
                             
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               4 
                               ⁢ 
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   a 
                                 
                                 ) 
                               
                             
                           
                           ) 
                         
                       
                       + 
                       9 
                     
                   
                 
               
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         H 
                         h 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               4 
                               ⁢ 
                               b 
                             
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               4 
                               ⁢ 
                               b 
                             
                           
                           ) 
                         
                       
                       + 
                       9 
                     
                   
                 
               
             
           
         
       
       
         
           
             
                 
               
                 
                   Equation 
                   ⁢ 
                       
                   17 
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         W 
                         v 
                       
                     
                     = 
                     
                       
                         max 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           
                             4 
                             ⁢ 
                             c 
                           
                         
                         ) 
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               4 
                               ⁢ 
                               c 
                             
                           
                           ) 
                         
                       
                       + 
                       9 
                     
                   
                 
               
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         H 
                         v 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               4 
                               ⁢ 
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   d 
                                 
                                 ) 
                               
                             
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               4 
                               ⁢ 
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   d 
                                 
                                 ) 
                               
                             
                           
                           ) 
                         
                       
                       + 
                       9 
                     
                   
                 
               
             
           
         
       
     
     Now that the above concepts of reference block bounding box computation have been explained, various aspects of the disclosed subject matter using some or all of these concepts will be further described herein. 
       FIG. 6  is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode. Based on the CPMVs (control point motion vectors) received for the PU, the decoder computes the reference block bounding box size and switches to the fall-back mode (from the affine mode) if the reference blocking box size exceeds the pre-defined thresholds. In this variant, the reference block bounding box is computed for 2×2 sub-block vectors if the PU uses bidirectional affine mode, and for 2×1 and 1×2 sub-block vectors if the PU uses unidirectional affine mode. In one implementation, the following steps may be applied: 
     1. Based on the CPMVs of the PU, the affine motion model parameters (a, b, c, d, e, j) are computed in step  602 . 
     2. If it is determined in step  604  that the PU uses bi-directional affine mode (i.e. bi-pred), the reference block bounding box size bxW4*bxH4 of 2×2 sub-block vectors is computed in step  606  by: 
     
       
         
           
             { 
             
               
                 
                   
                     
                       
                         
                           bxW4 
                           = 
                           
                             
                               max 
                               ⁢ 
                               
                                 ( 
                                 
                                   0 
                                   , 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     m 
                                   
                                   , 
                                   cn 
                                   , 
                                   
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           a 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       m 
                                     
                                     + 
                                     
                                       c 
                                       ⁢ 
                                       n 
                                     
                                   
                                 
                                 ) 
                               
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁡ 
                             ( 
                             
                               0 
                               , 
                               
                                 
                                   ( 
                                   
                                     1 
                                     + 
                                     a 
                                   
                                   ) 
                                 
                                 ⁢ 
                                 m 
                               
                               , 
                               cn 
                               , 
                               
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       a 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   m 
                                 
                                 + 
                                 
                                   c 
                                   ⁢ 
                                   n 
                                 
                               
                             
                             ) 
                           
                           + 
                           
                             f 
                             x 
                           
                           + 
                           m 
                           - 
                           1 
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       
                         
                           bxH4 
                           = 
                           
                             
                               max 
                               ⁢ 
                               
                                 ( 
                                 
                                   0 
                                   , 
                                   bm 
                                   , 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                   , 
                                   
                                     bm 
                                     + 
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           d 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       n 
                                     
                                   
                                 
                                 ) 
                               
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁢ 
                             
                               ( 
                               
                                 0 
                                 , 
                                 bm 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       d 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   n 
                                 
                                 , 
                                 
                                   bm 
                                   + 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                           
                           + 
                           
                             f 
                             y 
                           
                           + 
                           n 
                           - 
                           1 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     where m*n is the sub-block size, and f x *f y  is the filter tap size used in the luma motion compensation. 
     3. If the PU uses unidirectional affine mode (i.e. uni-pred), the reference block bounding box size bxW h *bxH h  of 2×1 sub-block vectors and the reference block bounding box size bxW v *bxH v  of 1×2 sub-block vectors are computed in step  608  by: 
     
       
         
           
             { 
             
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         W 
                         h 
                       
                     
                     = 
                     
                       
                         max 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   a 
                                 
                                 ) 
                               
                               ⁢ 
                               m 
                             
                           
                           ) 
                         
                       
                       - 
                       
                         min 
                         ⁢ 
                         
                           ( 
                           
                             0 
                             , 
                             
                               
                                 ( 
                                 
                                   1 
                                   + 
                                   a 
                                 
                                 ) 
                               
                               ⁢ 
                               m 
                             
                           
                           ) 
                         
                       
                       + 
                       
                         f 
                         x 
                       
                       + 
                       m 
                       - 
                       1 
                     
                   
                 
               
               
                 
                   
                     
                       bx 
                       ⁢ 
                       
                         H 
                         h 
                       
                     
                     = 
                     
                       
                         max 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           bm 
                         
                         ) 
                       
                       - 
                       
                         min 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           bm 
                         
                         ) 
                       
                       + 
                       
                         f 
                         y 
                       
                       + 
                       n 
                       - 
                       1 
                     
                   
                 
               
             
           
         
       
       
         
           
             { 
             
               
                 
                   
                     
                       bxW 
                       v 
                     
                     = 
                     
                       
                         max 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           cn 
                         
                         ) 
                       
                       - 
                       
                         min 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           cn 
                         
                         ) 
                       
                       + 
                       
                         f 
                         x 
                       
                       + 
                       m 
                       - 
                       1 
                     
                   
                 
               
               
                 
                   
                     
                       bxH 
                       v 
                     
                     = 
                     
                       
                         max 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           
                             
                               ( 
                               
                                 1 
                                 + 
                                 d 
                               
                               ) 
                             
                             ⁢ 
                             n 
                           
                         
                         ) 
                       
                       - 
                       
                         min 
                         ⁡ 
                         ( 
                         
                           0 
                           , 
                           
                             
                               ( 
                               
                                 1 
                                 + 
                                 d 
                               
                               ) 
                             
                             ⁢ 
                             n 
                           
                         
                         ) 
                       
                       + 
                       
                         f 
                         y 
                       
                       + 
                       n 
                       - 
                       1 
                     
                   
                 
               
             
           
         
       
     
     where m*n is the sub-block size, and f x *f y  is the filter tap size used in the luma motion compensation of the affine mode. 
     4. Thresholds Thred b , Thred h  and Thred v  may be set to values defined by: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             T 
                             ⁢ 
                             h 
                             ⁢ 
                             r 
                             ⁢ 
                             e 
                             ⁢ 
                             
                               d 
                               b 
                             
                           
                           = 
                           
                             
                               ( 
                               
                                 
                                   f 
                                   x 
                                 
                                 + 
                                 
                                   2 
                                   ⁢ 
                                   m 
                                 
                                 - 
                                 1 
                                 + 
                                 
                                   δ 
                                   x 
                                 
                               
                               ) 
                             
                             * 
                             
                               ( 
                               
                                 
                                   f 
                                   y 
                                 
                                 + 
                                 
                                   2 
                                   ⁢ 
                                   n 
                                 
                                 - 
                                 1 
                                 + 
                                 
                                   δ 
                                   y 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             T 
                             ⁢ 
                             h 
                             ⁢ 
                             r 
                             ⁢ 
                             e 
                             ⁢ 
                             
                               d 
                               h 
                             
                           
                           = 
                           
                             
                               ( 
                               
                                 
                                   f 
                                   x 
                                 
                                 + 
                                 
                                   2 
                                   ⁢ 
                                   m 
                                 
                                 - 
                                 1 
                                 + 
                                 
                                   δ 
                                   x 
                                 
                               
                               ) 
                             
                             * 
                             
                               ( 
                               
                                 
                                   f 
                                   y 
                                 
                                 + 
                                 n 
                                 - 
                                 1 
                                 + 
                                 
                                   δ 
                                   y 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             T 
                             ⁢ 
                             h 
                             ⁢ 
                             r 
                             ⁢ 
                             e 
                             ⁢ 
                             
                               d 
                               v 
                             
                           
                           = 
                           
                             
                               ( 
                               
                                 
                                   f 
                                   x 
                                 
                                 + 
                                 m 
                                 - 
                                 1 
                                 + 
                                 
                                   δ 
                                   x 
                                 
                               
                               ) 
                             
                             * 
                             
                               ( 
                               
                                 
                                   f 
                                   y 
                                 
                                 + 
                                 
                                   2 
                                   ⁢ 
                                   n 
                                 
                                 - 
                                 1 
                                 + 
                                 
                                   δ 
                                   y 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                       
                   18 
                 
               
             
           
         
       
     
     where δ x *δ y &gt;0 defines the margin for controlling the memory bandwidth consumption. 
     5. If the PU uses bi-directional affine mode (i.e. bi-pred) and it is determined in step  610  that bxW4*bxH4≤Thred b  or if the PU uses unidirectional affine mode (i.e. uni-pred) and it is determined in step  612  that both bxW h *bxH h ≤Thred h , bxW v *bxH v ≤Thred v , the sub-block motion vectors of the PU, {right arrow over (v)}=(v x ,v y ) at a sample position (x,y), are generated in step  614  by using the affine motion model: 
     
       
         
           
             { 
             
               
                 
                   
                     
                       v 
                       x 
                     
                     = 
                     
                       
                         a 
                         ⁢ 
                         x 
                       
                       + 
                       
                         c 
                         ⁢ 
                         y 
                       
                       + 
                       e 
                     
                   
                 
               
               
                 
                   
                     
                       v 
                       y 
                     
                     = 
                     
                       
                         b 
                         ⁢ 
                         x 
                       
                       + 
                       
                         d 
                         ⁢ 
                         y 
                       
                       + 
                       f 
                     
                   
                 
               
             
           
         
       
     
     where (x,y) for a sub-block vector can be the center location of the sub-block. 
     6. Otherwise (fallback mode), if the PU uses bi-directional affine mode (i.e. bi-pred) and bxW4*bxH4&gt;Thred b  or if the PU uses unidirectional affine mode (i.e. uni-pred) and either bxW h *bxH h &gt;Thred h  or bxW v *bxH v &gt;Thred v , the fallback mode is triggered, and in a corresponding one of steps  616  and  618 , the sub-block motion vectors of the PU, {right arrow over (v)}=(v x ,v y ) at a sample position (x,y), are set to a same motion vector. For example, 
     
       
         
           
             { 
             
               
                 
                   
                     
                       v 
                       x 
                     
                     = 
                     
                       
                         a 
                         ⁢ 
                         
                           x 
                           0 
                         
                       
                       + 
                       
                         c 
                         ⁢ 
                         
                           y 
                           0 
                         
                       
                       + 
                       e 
                     
                   
                 
               
               
                 
                   
                     
                       v 
                       y 
                     
                     = 
                     
                       
                         b 
                         ⁢ 
                         
                           x 
                           0 
                         
                       
                       + 
                       
                         d 
                         ⁢ 
                         
                           y 
                           0 
                         
                       
                       + 
                       f 
                     
                   
                 
               
             
           
         
       
     
     where (x 0 ,y 0 ) is the coordinate of the center point of the PU. (x 0 ,y 0 ) can be set to other locations of the PU. For example, if (x 0 ,y 0 ) is set to the coordinate of the top-left corner of the PU, then all the sub-block vectors {right arrow over (v)}=(v x ,v y ) of the PU are actually set to the control point motion vector of the PU at the top-left PU corner location. 
     By setting all the sub-block vectors to a same vector, the reference block bounding box size of 2×2, 2×1 and 1×2 sub-block vectors in the fallback mode is (f x +2m−1)*(f y +2n−1), (f x +2m−1)*(f y +n−1) and (f x +m−1)*(f y +2n−1), respectively, which is guaranteed to be smaller than the pre-defined thresholds Thred b , Thred h  and Thred v . 
     7. The generated sub-block vectors are passed to the motion compensation and the rest of the decoder processing in step  620 . 
     In the affine mode of the current VVC design, the sub-block size is fixed to 4×4 (i.e. m=n=4) and the filter-tap is fixed to 8×8 (i.e. f x =f y =6). If δ x  and δ y  are set to δ x =δ y =2, thresholds Thred b , Thred h  and Thred v  become 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             T 
                             ⁢ 
                             h 
                             ⁢ 
                             r 
                             ⁢ 
                             e 
                             ⁢ 
                             
                               d 
                               b 
                             
                           
                           = 
                             
                           
                             1 
                             ⁢ 
                             5 
                             * 
                             1 
                             ⁢ 
                             5 
                           
                         
                       
                     
                     
                       
                         
                           
                             T 
                             ⁢ 
                             h 
                             ⁢ 
                             r 
                             ⁢ 
                             e 
                             ⁢ 
                             
                               d 
                               h 
                             
                           
                           = 
                             
                           
                             1 
                             ⁢ 
                             5 
                             * 
                             1 
                             ⁢ 
                             1 
                           
                         
                       
                     
                     
                       
                         
                           
                             T 
                             ⁢ 
                             h 
                             ⁢ 
                             r 
                             ⁢ 
                             e 
                             ⁢ 
                             
                               d 
                               v 
                             
                           
                           = 
                             
                           
                             1 
                             ⁢ 
                             1 
                             * 
                             1 
                             ⁢ 
                             5 
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                      
                   19 
                 
               
             
           
         
       
     
     Which means the memory bandwidth consumption of the affine mode controlled by the algorithm described in  FIG. 6  won&#39;t exceed the worst case memory bandwidth consumption of the HEVC, which uses 8-tap interpolation filters for the motion compensation of 8×8 bi-directional PUs and 8×4/4×8 unidirectional PUs. The reference block size for an 8×8 directional PU using 8-tap interpolation filters is 15*15 for both the list 0 and list 1 prediction, and the reference block size for an 8×4/4×8 unidirectional PU is 15*11/11*15 for list 0 or list 1 prediction. Those values match the thresholds set in Equation 19. 
       FIG. 7  is an algorithmic flow chart of a method for controlling memory bandwidth consumption of the affine mode. Based on the CPMVs (control point motion vectors) received for the PU, the decoder computes the reference block bounding box size and switches to the fallback mode (from the affine mode) if the reference blocking box size exceeds the pre-defined thresholds. In this embodiment, the reference block bounding box is always computed for 2×2 sub-block vectors independent of the PU prediction type (unidirectional or bidirectional prediction). In one implementation, the following steps may be applied: 
     1. Based on the CPMVs of the PU, the affine motion model parameters (a, b, c, d, e, are computed in step  702 . 
     2. Compute the reference block bounding box size bxW4*bxH4 of 2×2 sub-block vectors in step  704  by 
     
       
         
           
             { 
             
               
                 
                   
                     
                       
                         
                           
                             bxW 
                             ⁢ 
                             4 
                           
                           = 
                           
                             
                               max 
                               ⁢ 
                               
                                 ( 
                                 
                                   0 
                                   , 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     m 
                                   
                                   , 
                                   cn 
                                   , 
                                   
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           a 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       m 
                                     
                                     + 
                                     
                                       c 
                                       ⁢ 
                                       n 
                                     
                                   
                                 
                                 ) 
                               
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁢ 
                             
                               ( 
                               
                                 0 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       a 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   m 
                                 
                                 , 
                                 cn 
                                 , 
                                 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         a 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     m 
                                   
                                   + 
                                   
                                     c 
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                           
                           + 
                           
                             f 
                             x 
                           
                           + 
                           m 
                           - 
                           1 
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       
                         
                           
                             bxH 
                             ⁢ 
                             4 
                           
                           = 
                           
                             
                               max 
                               ⁢ 
                               
                                 ( 
                                 
                                   0 
                                   , 
                                   bm 
                                   , 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                   , 
                                   
                                     bm 
                                     + 
                                     
                                       
                                         ( 
                                         
                                           1 
                                           + 
                                           d 
                                         
                                         ) 
                                       
                                       ⁢ 
                                       n 
                                     
                                   
                                 
                                 ) 
                               
                             
                             - 
                           
                         
                       
                     
                     
                       
                         
                           
                             min 
                             ⁢ 
                             
                               ( 
                               
                                 0 
                                 , 
                                 bm 
                                 , 
                                 
                                   
                                     ( 
                                     
                                       1 
                                       + 
                                       d 
                                     
                                     ) 
                                   
                                   ⁢ 
                                   n 
                                 
                                 , 
                                 
                                   bm 
                                   + 
                                   
                                     
                                       ( 
                                       
                                         1 
                                         + 
                                         d 
                                       
                                       ) 
                                     
                                     ⁢ 
                                     n 
                                   
                                 
                               
                               ) 
                             
                           
                           + 
                           
                             f 
                             y 
                           
                           + 
                           n 
                           - 
                           1 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     where m*n is the sub-block size, and f x *f y  is the filter tap size used in the luma motion compensation. 
     3. Thresholds Thred b  and Thred u  may be both set to (f x +2m−1+δ x )*(f y +2n−1+δ y ), or set to different values based on prediction type (unidirectional prediction or bi-directional), such as 
     
       
         
           
             { 
             
               
                 
                   
                     
                       T 
                       ⁢ 
                       h 
                       ⁢ 
                       r 
                       ⁢ 
                       e 
                       ⁢ 
                       
                         d 
                         b 
                       
                     
                     = 
                     
                       
                         ( 
                         
                           
                             f 
                             x 
                           
                           + 
                           
                             2 
                             ⁢ 
                             m 
                           
                           - 
                           1 
                           + 
                           
                             δ 
                             x 
                           
                         
                         ) 
                       
                       * 
                       
                         ( 
                         
                           
                             f 
                             y 
                           
                           + 
                           
                             2 
                             ⁢ 
                             n 
                           
                           - 
                           1 
                           + 
                           
                             δ 
                             y 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   
                     
                       T 
                       ⁢ 
                       h 
                       ⁢ 
                       r 
                       ⁢ 
                       e 
                       ⁢ 
                       
                         d 
                         u 
                       
                     
                     = 
                     
                       2 
                       * 
                       
                         ( 
                         
                           
                             f 
                             x 
                           
                           + 
                           
                             2 
                             ⁢ 
                             m 
                           
                           - 
                           1 
                           + 
                           
                             δ 
                             x 
                           
                         
                         ) 
                       
                       * 
                       
                         ( 
                         
                           
                             f 
                             y 
                           
                           + 
                           
                             2 
                             ⁢ 
                             n 
                           
                           - 
                           1 
                           + 
                           
                             δ 
                             y 
                           
                         
                         ) 
                       
                     
                   
                 
               
             
           
         
       
     
     where δ x *δ y &gt;0 defines the margin for controlling the memory bandwidth consumption. 
     4. If it is determined in step  706  the PU uses bi-directional affine mode (i.e. bi-pred) and it is determined in step  708  that bxW4*bxH4≤Thred b  or if the PU uses unidirectional affine mode (i.e. uni-pred) and it is determined in step  710  that bxW4*bxH4≤Thred u , the sub-block motion vectors of the PU, {right arrow over (v)}=(v x ,v y ) at a sample position (x,y), are generated in step  712  by using the affine motion model: 
     
       
         
           
             { 
             
               
                 
                   
                     
                       v 
                       x 
                     
                     = 
                     
                       
                         a 
                         ⁢ 
                         x 
                       
                       + 
                       
                         c 
                         ⁢ 
                         y 
                       
                       + 
                       e 
                     
                   
                 
               
               
                 
                   
                     
                       v 
                       y 
                     
                     = 
                     
                       
                         b 
                         ⁢ 
                         x 
                       
                       + 
                       
                         d 
                         ⁢ 
                         y 
                       
                       + 
                       f 
                     
                   
                 
               
             
           
         
       
     
     where (x,y) for a sub-block vector can be the center location of the sub-block. 
     5. Otherwise (fallback mode), if the PU uses bi-directional affine mode (i.e. bi-pred) and bxW4*bxH4&gt;Thred b  or if the PU uses unidirectional affine mode (i.e. uni-pred) and bxW4*bxH4&gt;Thred u , the fallback mode is triggered in a corresponding one of steps  714  and  716 , and the sub-block motion vectors of the PU, {right arrow over (v)}=(v x ,v y ) at a sample position (x,y), are set to a same motion vector. For example, 
     
       
         
           
             { 
             
               
                 
                   
                     
                       v 
                       x 
                     
                     = 
                     
                       
                         a 
                         ⁢ 
                         
                           x 
                           0 
                         
                       
                       + 
                       
                         c 
                         ⁢ 
                         
                           y 
                           0 
                         
                       
                       + 
                       e 
                     
                   
                 
               
               
                 
                   
                     
                       v 
                       y 
                     
                     = 
                     
                       
                         b 
                         ⁢ 
                         
                           x 
                           0 
                         
                       
                       + 
                       
                         d 
                         ⁢ 
                         
                           y 
                           0 
                         
                       
                       + 
                       f 
                     
                   
                 
               
             
           
         
       
     
     where (x 0 ,y 0 ) is the coordinate of the center point of the PU. (x 0 ,y 0 ) can be set to other locations of the PU. For example, if (x 0 ,y 0 ) is set to the coordinate of the top-left corner of the PU, then all the sub-block vectors {right arrow over (v)}=(v x ,v y ) of the PU are actually set to the control point motion vector of the PU at the top-left PU corner location. 
     By setting all the sub-block vectors to a same vector, the reference block bounding box size of 2×2 sub-block vectors in the fallback mode is (f x +2m−1)*(f y +2n−1), which is guaranteed to be smaller than the pre-defined threshold values of Thred b  and Thred u . 
     6. The generated sub-block vectors are passed to the motion compensation and the rest of the decoder processing in step  718 . 
     It should be appreciated that in the bi-directional affine mode, a PU has both list 0 and list 1 predictions. In the disclosed subject matter, the reference bounding box size bxW4*bxH4 is computed independently for list 0 and list 1 prediction with the respective list 0/list 1 affine motion model parameters (a, b, c, d) of the PU, and the threshold Thred b  is set separately for list 0 and list 1 prediction (though the values of the threshold could be the same). With the memory bandwidth control algorithms described above, the following four combinations are possible for a PU coded in bi-directional affine mode: 1) the regular sub-block motion vector fields are used for both the list 0 and list 1 motion compensation of the PU; 2) the regular sub-block motion vector field is used for list 0 motion compensation but the fallback mode (i.e. a single vector for list 1 prediction of the entire PU) is used for list 1 motion compensation; 3) the fallback mode (i.e. a single vector for list 0 prediction of the entire PU) is used for list 0 motion compensation but the regular sub-block motion vector field is used for list 1 motion compensation; and 4) the fallback mode (i.e. a first single vector for list 0 prediction and a second single vector for list 1 prediction of the entire PU) is used for both list 0 and list 1 motion compensation. 
       FIG. 8  is an algorithmic flow chart of a method for controlling the sub-block motion vector spread in the affine motion estimation on the encoder side. The basic idea is to constrain the sub-block motion vector spread at PU level. The sub-block motion vector spread can be measured by using the reference block bounding box size. During the encoder decision process, the encoder guarantees that the affine mode, if it is selected for a PU, has the sub-block motion vector spread constrained to meet the pre-defined budget of the worst-case memory bandwidth consumption at PU level. During the affine motion estimation in which a set of candidate CPMVs (control point motion vectors) is evaluated for the affine mode of the PU, the following steps may be applied: 
     1. Based on the CPMVs of a candidate point, the affine motion model parameters (a, b, c, d, e, f) are computed in step  802 . 
     2. If the PU uses bi-directional affine mode (i.e. bi-pred), as determined in step  804 , the reference bounding box size bxW4*bxH4 is computed in step  806  by using Equation 12. Otherwise, if the PU uses unidirectional affine mode, the reference bounding box size bxW h *bxH h  and bxW v *bxH v  are computed in step  808  by using Equation 13 and Equation 14, respectively. 
     3. To control the sub-block motion vector spread at PU level, the current candidate point is skipped for cost evaluation in the following cases: 
     a. If, as determined in step  810 , the PU uses bi-directional affine mode (i.e. bi-pred) and bxW4*bxH4&gt;Thred4. 
     b. Or if, as determined in step  812 , the PU uses uni-directional affine mode and either bxW h *bxH h &gt;Thred h  or bxW v *bxH v &gt;Thred v , where Thred4, Thred h  and Thred v  may be pre-defined by using Equation 18. 
     4. Otherwise, the cost of the current candidate point is computed in step  814  by using the affine motion model parameters (a, b, c, d, e, f), the reference block data and the original PU data. 
     5. The cost of the best match point is then updated in step  816 . 
     6. Repeat steps  802  through  816  to loop through all the candidate points for the PU to obtain the estimated CPMVs for the PU that deliver the best rate-distortion cost. 
     The estimated affine CPMVs for the PU, if available, may be further evaluated against the costs of regular motion vectors and intra prediction mode estimated for the PU to decide whether the current PU should be encoded in the affine mode. 
     By using the proposed method, the worst-case memory bandwidth consumption for the affine mode is restricted not only at sub-block level but also at PU level. Note that the sub-block motion vector spread within a PU of affine mode is determined by the affine parameters (a, b, c, d). 
     It should be appreciated that in in step  810  the decision on whether the reference bounding box size exceeds the pre-defined threshold is done independently for the list 0 and list 1 prediction of a PU coded in bi-directional affine mode. 
     The restriction can also be imposed by a bitstream constraint. For example, the bitstream constraint can be specified as follows: 
     A bitstream conforming to the VVC standard shall satisfy the following conditions: 
     1. If the PU uses bi-directional affine mode (i.e. bi-pred), the reference block bounding box size bxW4*bxH4 is less than or equal to Thred4. 
     2. Otherwise, if the PU uses uni-directional affine mode, the reference block bounding box size bxW h *bxH h  and bxW v *bxH v  is less than or equal to Thred h  and hred v , respectively. 
     The implementation depicted in  FIG. 8  just exhibits one example of the embodiment; other variants can be implemented. For example: 
     1. Instead of using the bounding boxes in both the horizontal direction (i.e. bxW h , bxH h  and Thred h ) and the vertical direction (i.e. bxW v , bxH v  and Thred v ) in the case of unidirectional affine mode, the bounding box either in the horizontal or in the vertical direction may be used. 
     2. Instead of computing the threshold and bounding box size for 4 sub-block vectors (2×2 consecutive sub-block vectors) in the case of bi-directional affine mode, and for 2 sub-block vectors (in horizontal and/or vertical direction) in the case of unidirectional affine mode, the threshold and bounding box size may be computed for other numbers of consecutive sub-block vectors with a PU. 
     3. The constraint may be imposed independent of the prediction type (i.e. bi-directional or unidirectional affine mode). In one implementation, the constraint may be simply expressed as a bitstream conforming to the VVC standard shall stratify that the reference block bounding box size bxW4*bxH4 is less than or equal to Thred4. 
     4. Instead of using the fixed sub-block size m*n (e.g. 4×4), the sub-block size may be further made adaptive to impose additional constraints at the sub-block level. 
       FIG. 9  is an algorithmic flow chart of a method for determining block size. In step  902 , the reference block bounding box size and the at least one threshold are determined. In step  904 , the reference block bounding box size and the at least one threshold are compared. If the reference block bounding size exceeds the at least one threshold, then in step  906 , the sub-block size m′*n′ (or as appropriate, m′*n or m*n′, with m′&gt;m, and n′&gt;n) is used for the generation of the affine sub-block motion vector field and for affine motion compensation. Otherwise, in step  908 , the sub-block size m*n is used. Any combination of bi-directional, horizontal unidirectional and vertical unidirectional modes can be used. The results are output in step  910 . 
     For example, the memory bandwidth control algorithm can be modified based on the shape of the PU coded in the affine mode: 
     1. If the PU coded in affine mode uses bi-directional inter prediction and the reference block bounding box size of four sub-block vectors exceeds a pre-defined threshold, i.e. bxW4*bxH4&gt;Thred4, then sub-block size m′*n′ (with m′&gt;m, and n′&gt;n) is used for the generation of the sub-block motion data field for the motion compensation of the PU. Otherwise, sub-block size m*n is used. 
     2. Otherwise, if the PU coded in affine mode uses unidirectional inter prediction, and if the reference block bounding box size of two sub-block vectors exceeds a pre-defined threshold in both horizontal and vertical directions, i.e. bxW h *bxH h &gt;Thred h  and bxW v *bxH v &gt;Thred v , then sub-block size m′*n (with m′&gt;m) is used if PU width is larger than PU height, or sub-block size m*n′ (with n′&gt;n) is used PU width is less than or equal to PU height. 
     Otherwise, sub-block size m*n is used for the generation of the sub-block motion data field for the motion compensation of the PU. 
     In another variation, instead of adaptively selecting the sub-block size based on the size of reference block bounding box, the selection may be based on the width and/or height of the reference block bounding box, and/or based on the DDR burst size and alignments, or based on any combinations of above mentioned parameters (i.e. size, with and height of the reference block bounding box, the DDR burst size and alignments, and etc.). 
     It should be appreciated that in the disclosed matter the bitsteam constraints and/or sub-block size adaptation are done independently for the list 0 and list 1 prediction of a PU coded in bi-directional affine mode. 
     In another variation, the sub-block vectors of affine mode used for motion compensation may be the same as or different from the ones used for (affine) merge/AMVP list derivation (used as spatial neighboring candidates), for de-blocking filter and for storage of temporal motion vectors (TMVPs). For example, the sub-block motion vectors of the affine mode for motion compensation may be generated by the algorithm that adaptively selects sub-block sizes (e.g. 8×8/8×4 adaptively), while the sub-block motion vectors of the affine mode for (affine) merge/AMVP list derivation (used as spatial neighboring candidates), for de-blocking filter and for storage of temporal motion vectors (TMVPs) may be generated by using a fixed sub-block size (e.g. 4×4). In another example, the sub-block motion vectors of the affine mode for motion compensation, for (affine) merge/AMVP list derivation (used as spatial neighboring candidates), for de-blocking filter and for storage of temporal motion vectors (TMVPs) may be generated by the algorithm described herein that adaptively selects sub-block sizes (e.g. 8×8/8×4 adaptively). 
     A hardware description of a computer/device (e.g., the image processing device  1000 ) according to exemplary embodiments, or any other embodiment, which is used to encode and/or decode video is described with reference to  FIG. 10 . In  FIG. 10 , the image processing device  1000  includes a CPU  1002  which performs one or more of the processes described above. The process data and instructions may be stored in memory  1004 , which may be transitory, non-transitory, or a combination of the two. The video (data) to be encoded/decoded can original from any source such as an external memory, a network  1006 , or other location/device. These processes and instructions may also be stored on a storage medium disk  1008  such as a hard drive (HDD) or portable storage medium or may be stored remotely. Further, the claimed advancements are not limited by the form of the computer-readable media on which the instructions of the inventive process are stored. For example, the instructions may be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device with which the image processing device  1000  communicates, such as an image processing device or computer. 
     Further, the claimed advancements may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU  1002  and an operating system such as Microsoft Windows, UNIX, Solaris, LINUX, Apple MAC-OS and other suitable operating system. 
     The image processing device  1000  may be a general-purpose computer or a particular, special-purpose machine. In one embodiment, the image processing device  1000  becomes a particular, special-purpose machine when the processor  1002  is programmed to perform network performance testing. The image processing device may be implemented as an encoder, a decoder, or a device which both encodes and decodes images. The image processing device can be implemented in a mobile phone, a laptop, a tablet, a general purpose computer, a set-top box, a video decoding device such as an Amazon Fire TV Stick or device, a Roku device, a television, a video monitor, a still camera, a video camera, a scanner, a multifunction printer, an automobile display, or any desired device. 
     Alternatively, or additionally, the CPU  1002  may be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize. Further, CPU  1002  may be implemented as multiple processors cooperatively working in parallel to perform the instructions of the inventive processes described above. 
     The image processing device  1000  in  FIG. 10  also includes a network controller  1010 , such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with network  1006 . As can be appreciated, the network  1006  can be a public network, such as the Internet, or a private network such as an LAN or WAN network, or any combination thereof and can also include PSTN or ISDN sub-networks. The network  1006  can also be wired, such as an Ethernet network, or can be wireless such as a cellular network including EDGE, 3G and 4G wireless cellular systems. The wireless network can also be WiFi, Bluetooth, or any other wireless form of communication that is known. 
     The image processing device  1000  further includes a display controller  1012 , such as a graphics card or graphics adaptor for interfacing with display  1014 , such as a monitor. A general purpose I/O interface  1016  interfaces with a keyboard and/or mouse  1018  as well as a touch screen panel  1020  on or separate from display  1014 . General purpose I/O interface also connects to a variety of peripherals  1022  including printers and scanners. 
     A sound controller  1024  is also provided in the image processing device  1000  to interface with speakers/microphone  1026  thereby providing sounds and/or music. 
     The general-purpose storage controller  1028  connects the storage medium disk  1008  with communication bus  1030 , which may be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the image processing device  1000 . A description of the general features and functionality of the display  1014 , keyboard and/or mouse  1018 , as well as the display controller  1012 , storage controller  1028 , network controller  1010 , sound controller  1024 , and general purpose I/O interface  1016  is omitted herein for brevity. 
     The exemplary circuit elements described in the context of the present disclosure may be replaced with other elements and structured differently than the examples provided herein. Moreover, circuitry configured to perform features described herein may be implemented in multiple circuit units (e.g., chips), or the features may be combined in circuitry on a single chipset. For that matter, any hardware and/or software capable of implementing any of the above embodiments, or any other embodiment, can be used instead of, or in addition to, what is disclosed above. 
     The functions and features described herein may also be executed by various distributed components of a system. For example, one or more processors may execute these system functions, wherein the processors are distributed across multiple components communicating in a network. The distributed components may include one or more client and server machines, which may share processing, in addition to various human interface and communication devices (e.g., display monitors, smart phones, tablets, personal digital assistants (PDAs)). The network may be a private network, such as a LAN or WAN, or may be a public network, such as the Internet. Input to the system may be received via direct user input and received remotely either in real-time or as a batch process. Additionally, some implementations may be performed on modules or hardware not identical to those described. Accordingly, other implementations are within the scope that may be claimed. 
     While preferred embodiments have been set forth above, those skilled in the art who have reviewed the present disclosure will readily appreciate that other embodiments can be realized within the scope of the disclosure. For example, disclosures of numerical values and of specific technologies are illustrative rather than limiting. Also, whenever technically feasible, features from different embodiments can be combined, and the order in which operations are performed can be varied. Further, wherever technically feasible, any feature disclosed herein can be used for encoding, decoding or both. The one or more aspects of the disclosed subject matter are not limited to VVC implementations and can be utilized with any video encoding/decoding system. Therefore, the one or more aspects of the disclosed subject matter should be construed as limited only by the appended claims.