Abstract:
A method of video encoding is disclosed which is content adaptive. The encoding method is automatically adjusted to optimize the encoding, the adjusting depending on the content of the pictures being encoded. A system for implementing the method and a non-transitory computer-readable storage medium for storing instructions of the method are also disclosed.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention is generally directed to video, and in particular, to digital video processing. 
       BACKGROUND 
       [0002]    Video encoders compress sequences of video pictures, or frames, by reducing spatial and temporal redundancies. This is done by performing prediction processes in the spatial and/or temporal domains. If the prediction process uses only information in a current picture, it is referred to as intra-prediction, and the picture being encoded is called an I-picture. By contrast, if the prediction process uses correlations between different pictures, it is referred to as inter-prediction. Most encoders support two types of inter-prediction, called P (predicted) prediction and B (bidirectional) prediction. The main difference is that P-prediction predicts the value of a current block based on only one prediction block, while B prediction allows interpolation-based prediction of a current block based on two previously encoded blocks. 
         [0003]    A macroblock (MB) is a block of 16×16 pixels. All macroblocks in an I-picture are intra-predicted, while MBs in a P picture may be either P-inter or intra-predicted (whichever is more efficient). Finally, MBs in a B picture are allowed to be either B-inter, P-inter, or intra-predicted. 
         [0004]    In video compression, a group of pictures (GOP) specifies the order in which intra- and inter-pictures are arranged. The GOP is a group of successive pictures within a coded video stream. Each coded video stream consists of successive GOPs. 
         [0005]    Pictures are encoded based on prediction structures. A prediction structure describes which pictures in a GOP are used to encode a given picture in the GOP and the type of each prediction: I, P, or B. Existing encoding methods use a fixed prediction structure, without taking into account the nature of the picture content. This can result in encoding which is not optimal. 
       SUMMARY OF EMBODIMENTS 
       [0006]    A method of video encoding is disclosed which is content adaptive—that is, the encoding method automatically adjusts one or more aspects of an encoding process in order to optimize the encoding. The adjustments depend on the content of the pictures being encoded. In an embodiment, the aspects adjusted may be a size of a group of pictures and a prediction structure. A system for implementing the method and a non-transitory computer-readable storage medium for storing instructions of the method are also disclosed. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings wherein: 
           [0008]      FIG. 1  shows examples of group of pictures decompositions; 
           [0009]      FIG. 2  shows two examples of a prediction structure; 
           [0010]      FIGS. 3A and 3B  show an example of a method of video encoding; and 
           [0011]      FIG. 4  is a block diagram of an example device or system in which one or more disclosed embodiments may be implemented. 
       
    
    
     DETAILED DESCRIPTION 
       [0012]    A method and system for content adaptive video encoding will now be described in detail. The method includes selecting a group of pictures (GOP) and a prediction structure to optimize the encoding of the pictures, or frames, that make up a moving video image. An optimization metric that may be used to optimize the encoding is rate-distortion (RD) cost. These terms are explained in what follows. 
         [0013]    Rate-distortion cost refers to the combination of bitrate, or number of bits, needed to encode a picture or group of pictures, and image distortion introduced by the encoding. Generally, reducing the number of bits used to encode a picture will tend to result in more distortion—there is less information in the encoded picture. A combination of rate and distortion is therefore needed for a reasonable metric of encoding optimization. Methods of determining distortion are discussed later herein. 
         [0014]      FIG. 1 , not to be considered limiting, shows examples of GOPs and GOP decompositions. Shown are 12 successive pictures in a video sequence. The 12 successive pictures are shown decomposed into two distinct GOP decompositions  110  and  120 . In decomposition  110  the 12 pictures are shown decomposed into four GOPS,  115   a - 115   d  having respective sizes of two pictures, three pictures, three pictures, and four pictures. In decomposition  120  the same 12 pictures are decomposed into three GOPS,  125   a - 125   c  having respective sizes of four pictures, three pictures, and five pictures. The number of GOPs and the number of pictures in each GOP is not fixed. 
         [0015]      FIG. 2 , not to be considered limiting, shows examples of two different prediction structures,  205 A and  205 B, being applied to an example GOP containing nine pictures. In  205 A, only the first picture is encoded as an I picture, while all other pictures are encoded a P pictures, each picture referring to the previous one. The prediction structure  205 A may provide near-optimal encoding when, for example, the pictures being encoded include a rapidly moving object, but may provide suboptimal encoding when the pictures include a slowly moving object. 
         [0016]    By contrast, if the pictures include, for example, a slowly moving object, and therefore exhibit high temporal correlations, the prediction structure  205 B may provide closer to optimal encoding than  205 A. Prediction structure  205 B uses B frames and is referred to as a hierarchical B-structure (HBS). Thus, the prediction structure that results in optimal encoding, as measured by RD cost, depends on the content of the pictures being encoded. 
         [0017]    Therefore, an encoding method that adapts its prediction structures to picture content may provide encoding optimization that is superior to an encoding method with a fixed prediction structure. Furthermore, a content adaptive video encoding method using HBS&#39;s may provide even better encoding. 
         [0018]      FIGS. 3A and 3B  show an embodiment of a method  300  for content adaptive video encoding.  FIG. 3A  conveys an overview of method  300  and  FIG. 3B  shows details of an embodiment of a method of determining an RD cost for a GOP decomposition at  315 . Overall the method may be described as follows. A set of GOP decompositions of a predetermined number of successive pictures is selected. For each GOP decomposition in the set, a GOP decomposition RD cost is determined by determining a GOP RD cost for each GOP in that GOP decomposition. A GOP decomposition from the set having a minimum GOP decomposition RD cost is selected for use in encoding the successive pictures. 
         [0019]    Referring to  FIG. 3A , method  300  is initialized by selecting a set of GOP decompositions from all the possible GOP compositions of a predetermined number of successive pictures  305 . The set may be selected by, for example, selecting a small number of GOP decompositions already known to work well. Alternatively, only GOP decompositions within a defined range of sizes may be selected. In another alternative, known scene change algorithms may be used to guide the selection of GOP sizes. 
         [0020]    Method  300  is further initialized by storing an initial value of a stored GOP decomposition RD cost and an initial value of a stored GOP RD cost  305 . Method  300  is further initialized by setting a quantity TOTAL equal to zero  305 . The quantity TOTAL will be used to sum GOP RD costs to obtain a GOP decomposition RD cost. 
         [0021]    Method  300  is further initialized by defining a set of prediction structures from possible prediction structures for a GOP  305 . The set of prediction structures may include at least one hierarchical B structure. Defining a set of prediction structures may include selecting prediction structures that meet constraints of a decoded picture buffer (DFB) used to store previously decoded pictures. Since decoders may be constrained in terms of memory and processing capabilities, there may be a limit on the size of the DFB. Alternatively, known scene change algorithms may be used to guide the selections of prediction structures for the set of prediction structures. 
         [0022]    Following initialization, a GOP decomposition is selected from the set of GOP decompositions  310 . A GOP decomposition RD cost for the selected GOP decomposition is determined  315 . This GOP decomposition RD cost is compared with the stored GOP decomposition RD cost  320 . If the GOP decomposition RD cost is not less than the stored value the method returns to  310  and a new GOP decomposition is selected. If, on the other hand, the GOP decomposition RD cost is less than the stored value, this new GOP decomposition RD cost is stored  325 , replacing the current stored value. 
         [0023]    The method then checks if there are any remaining GOP decompositions in the set  330 . If there are, the method returns to  310  to select a new GOP decomposition from the set. If there are not, then the stored GOP decomposition RD cost is a minimum, and the GOP decomposition having that minimum is used to encode the successive pictures  335 . 
         [0024]      FIG. 3B  shows details of determining GOP decomposition RD cost at  315  in  FIG. 3A . Overall, determining GOP decomposition RD cost in this method embodiment may be summarized as follows. As described above, a set of prediction structures is defined in initialization  305 . For each GOP in a GOP decomposition, a prediction structure RD cost is determined for each prediction structure in the set. A minimum prediction structure RD cost among all of these determined prediction structure RD costs is chosen as the GOP RD cost for the GOP. A GOP decomposition RD cost is determined by summing GOP RD costs for all GOPs in the GOP decomposition. 
         [0025]    Referring to  FIG. 3B , details of determining GOP decomposition RD cost are as follows. A GOP in a current GOP decomposition is selected  345 . (A current GOP decomposition has been selected at  310  in  FIG. 3A .) A prediction structure is selected from the previously initialized set of prediction structures  350 . An RD cost is determined for each picture in the GOP for the selected prediction structure  355 . 
         [0026]    In an embodiment an RD cost for each picture may be determined by determining a distortion and a bitrate and using the formula: 
         [0000]        RD  cost=distortion+ L *bitrate  Equation (1)
 
         [0000]    where L is a parameter that depends on the picture type and encode parameters such as a quantization parameter. Distortion may be determined by applying a motion estimation process for a current GOP and a current prediction structure. The applied motion estimation process makes it possible to predict the content of every picture in the GOP, by using a set of previously encoded pictures, referred to as reference pictures. This prediction process usually produces predicted pictures close but different from the original picture. The difference between the predicted picture and the original one is called prediction error or prediction distortion. Usually, the higher the prediction distortion, the higher the amount of information that needs to be encoded to represent the original picture. Distortion may be measured by using a metric known in the art, such as sum of absolute differences (SAD), sum of absolute transformed differences, (SATD) or mean absolute difference (MAD). Bitrate may also be determined using known methods. Determination of RD cost for each picture in a GOP may include estimating a bitrate from a distortion using a rate-distortion model, such as a quadratic rate-distortion model. Determining RD cost may be applied to original input pictures rather than reconstructed ones, or to down-sampled versions of a video. 
         [0027]    Returning to  FIG. 3B , the RD costs of all pictures in the GOP are summed to determine a GOP RD cost  360 . This GOP RD cost is compared to a stored GOP RD cost  365 . (The stored GOP RD cost is initialized at  305  in  FIG. 3A .) If the GOP RD cost is not less than the stored GOP RD cost the method returns to  350 , where a new prediction structure is selected. If, on the other hand, the GOP RD cost is less than the stored GOP RD cost, this new GOP RD cost is stored  370 , replacing the current stored value. 
         [0028]    The method then checks if there are any remaining prediction structures  375 . If there are, the method returns to  350  to select a new prediction structure. If there are not, then the stored GOP RD cost contains a minimum RD cost over the prediction structures. The stored GOP RD cost is then added to the amount TOTAL  380 . 
         [0029]    The method then determines whether there are remaining GOPS in the current GOP decomposition  385 . If there are, then the method returns to  345 , and a new GOP is selected. If there are no remaining GOPS, then the sum in TOTAL is GOP decomposition RD cost. This GOP decomposition RD cost is passed to  320  in  FIG. 3A  and the overall method shown in  FIG. 3A  continues. 
         [0030]    To reduce computational complexity, an early termination strategy based on per-picture or per GOP maximum RD costs could be implemented in an embodiment of the method. 
         [0031]    A method as just described may also be described by the following pseudocode: 
         [0000]    
       
         
               
             
           
               
                   
               
             
             
               
                      GOP decomposition and initialize its RD cost to a big value 
               
               
                      RD_Optimal_Decomposition = MAX_RD_COST (for initialization) 
               
               
                      Optimal_Decomposition = one GOP IPPPPP (for initialization) 
               
               
                      // Find the decomposition into a set of GOPs, which leads to the 
               
               
                      minimal RD cost 
               
               
                 For any possible decomposition D of the next 
               
               
                 MAX_NUM_LOOKAHEAD_PICTURES pictures into a set of GOPs (i.e., (GOP i )) 
               
               
                      // Compute the RD cost of decomposition D by accumulating the RD 
               
               
                      cost of its GOPs 
               
               
                      RD_D = 0 
               
               
                      For every GOP i   
               
               
                         // Find the prediction structure for GOP i , which leads to the 
               
               
                         minimal RD cost 
               
               
                         Optimal_Prediction_Strcture = IPPPPP GOP structure (for 
               
               
                         initialization) 
               
               
                           RD_ Optimal_Prediction_Structure = MAX_RD_COST 
               
               
                         For every prediction structure PS of GOP i  meeting the DBP 
               
               
                         constraints 
               
               
                           RD_PS = RD cost of PS 
               
               
                           If (RD_PS &lt; RD_Optimal_Prediction_Structure) 
               
               
                              Optimal_Prediction_Structure = PS 
               
               
                              RD_Optimal_Prediction_Structure = RD_PS 
               
               
                           End 
               
               
                         End 
               
               
                         RD_D = RD_D + RD_Optimal_Prediction_Structure 
               
               
                      End 
               
               
                      // Select D as optimal decomposition if its RD cost is lower than the 
               
               
                    best RD cost so far 
               
               
                         If (RD_D &lt; RD_Optimal_Deptimal_Decomposition = D 
               
               
                      End 
               
               
                  End 
               
               
                   
               
             
          
         
       
     
         [0032]      FIG. 4  is a block diagram of an example device or system  400  in which one or more disclosed embodiments may be implemented. The system  400  may include, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. The system  400  includes a processor  402 , a memory  404 , a storage  406 , one or more input devices  408 , and one or more output devices  410 . The system  400  may also optionally include an input driver  412  and an output driver  414 . It is understood that the system  400  may include additional components not shown in  FIG. 4 . 
         [0033]    The processor  402  may include a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, wherein each processor core may be a CPU or a GPU. The memory  404  may be located on the same die as the processor  402 , or may be located separately from the processor  402 . The memory  404  may include a volatile or non-volatile memory, for example, random access memory (RAM), dynamic RAM, or a cache. Memory  404  may include a decoded picture buffer (DFB) configured to store previously decoded pictures. These stored pictures may be used by processor  402  for encoding to form predictors for subsequent pictures. 
         [0034]    The storage  406  may include a fixed or removable storage, for example, a hard disk drive, a solid state drive, an optical disk, or a flash drive. The input devices  408  may include a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE 802 signals). The input device may also include a source of video information, such as, but not limited to, a video camera, or a video playback device such as, but not limited to, a Blu-Ray player. The output devices  410  may include a display, a speaker, a printer, a haptic feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE 802 signals), or a video display. 
         [0035]    The input driver  412  communicates with the processor  402  and the input devices  408 , and permits the processor  402  to receive input from the input devices  408 . The output driver  414  communicates with the processor  402  and the output devices  410 , and permits the processor  402  to send output to the output devices  410 . It is noted that the input driver  412  and the output driver  414  are optional components, and that the system  400  will operate in the same manner if the input driver  412  and the output driver  414  are not present. 
         [0036]    System  400  may be configured to implement a method of content adaptive video encoding, such as that described above, as follows. Processor  402  may be configured to perform the method. An input device  408  may be configured to supply video information, such as successive pictures, to processor  402 . Memory device  404  may be configured to exchange video information with processor  402  and store video information. 
         [0037]    Processor  402  may obtain a predetermined number of successive pictures supplied by input device  408 . Processor  402  may retrieve a set of GOP decompositions of the successive pictures from memory device  404 . Processor  402  may then proceed to determine, for each GOP decomposition in the set, an RD cost by determining an RD cost for each GOP in that GOP decomposition. Processor  402  may select a GOP decomposition from the set having a minimum RD cost for use in encoding the successive pictures. Processor  402  may be configured to perform all steps of each of the embodiments of a method for content adaptive video encoding as described herein. 
         [0038]    It should be understood that many variations are possible based on the disclosure herein. Although features and elements are described above in particular combinations, each feature or element may be used alone without the other features and elements or in various combinations with or without other features and elements. 
         [0039]    The methods provided may be implemented in a general purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine. Such processors may be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediary data including netlists (such instructions capable of being stored on a computer readable media). The results of such processing may be maskworks that are then used in a semiconductor manufacturing process to manufacture a processor which implements aspects of the present invention. 
         [0040]    The methods or flow charts provided herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable storage medium for execution by a general purpose computer or a processor. Examples of computer-readable storage mediums include a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs).