Patent Publication Number: US-11030777-B2

Title: Adaptive subband coding for lifting transform

Description:
CROSS-REFERENCE TO RELATED APPLICATION(S) 
     This application claims priority under 35 U.S.C. § 119(e) of the U.S. Provisional Patent Application Ser. No. 62/731,588, filed Sep. 14, 2018 and titled, “ADAPTIVE SUBBAND CODING FOR LIFTING TRANSFORM,” which is hereby incorporated by reference in its entirety for all purposes. 
    
    
     FIELD OF THE INVENTION 
     The present invention relates to three dimensional graphics. More specifically, the present invention relates to compression of three dimensional graphics. 
     BACKGROUND OF THE INVENTION 
     A lifting transform, which is a point cloud color compression method adopted in PCC test model TMC13 for compression of CAT1 sequences, is not optimally implemented. 
     SUMMARY OF THE INVENTION 
     Lifting is a transform designed for color compression of point clouds which is adopted in one of the MPEG test models. The performance of lifting is improved herein. All the lifting coefficients are first divided into several subbands based on their assigned weights, which indicate the level of importance of each coefficient. Then, for each subband, a set of three dead-zones are derived for the three color components. The dead-zones of Cb and Cr channels are typically larger than that of Luma channel. In the original lifting scheme, Chroma is not suppressed at all. In contrast, as described herein, the size of the dead-zone is increased for different color components, which means that more quality (and bandwidth) is able to be adaptively provided for luminance coefficients than chrominance coefficients. 
     In one aspect, a method programmed in a non-transitory memory of a device comprises multiplying a residual of a point cloud by a weight to generate a weighted residual, quantizing the weighted residual to generate a quantized level and dividing the quantized level by the weight to generate a reconstructed residual which is used for color compression of the point cloud. The method further comprises dividing a plurality of lifting coefficients into a plurality of subbands and deriving a set of dead-zones for each subband for a set of color components. Dividing the plurality of lifting coefficients into the plurality of subbands is based on an assigned weight of each lifting coefficient. The set of color components includes Cb, Cr and luma channels. The weight is a square root of the weight of a point. If the weight of the point is less than a weight of a less significant weight, then a dead-zone size is increased. The weight is based on how many times a point has been used for prediction and distances to predicted points. 
     In another aspect, an apparatus comprises a non-transitory memory for storing an application, the application for: multiplying a residual of a point cloud by a weight to generate a weighted residual, quantizing the weighted residual to generate a quantized level and dividing the quantized level by the weight to generate a reconstructed residual which is used for color compression of the point cloud and a processor coupled to the memory, the processor configured for processing the application. The application is further for: dividing a plurality of lifting coefficients into a plurality of subbands and deriving a set of dead-zones for each subband for a set of color components. Dividing the plurality of lifting coefficients into the plurality of subbands is based on an assigned weight of each lifting coefficient. The set of color components includes Cb, Cr and luma channels. The weight is a square root of the weight of a point. If the weight of the point is less than a weight of a less significant weight, then a dead-zone size is increased. The weight is based on how many times a point has been used for prediction and distances to predicted points. 
     In another aspect, a system comprises a multiplication module configured for multiplying a residual of a point cloud by a weight to generate a weighted residual, a quantization module configured for quantizing the weighted residual to generate a quantized level and a dividing module configured for dividing the quantized level by the weight to generate a reconstructed residual which is used for color compression of the point cloud. The system further comprises a subband module configured for dividing a plurality of lifting coefficients into a plurality of subbands and a deriving module configured for deriving a set of dead-zones for each subband for a set of color components. Dividing the plurality of lifting coefficients into the plurality of subbands is based on an assigned weight of each lifting coefficient. The set of color components includes Cb, Cr and luma channels. The weight is a square root of the weight of a point. If the weight of the point is less than a weight of a less significant weight, then a dead-zone size is increased. The weight is based on how many times a point has been used for prediction and distances to predicted points. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a diagram of the TMC13 quantization strategy according to some embodiments. 
         FIG. 2  illustrates a diagram of using lifting for TMC13 according to some embodiments. 
         FIG. 3  illustrates a flowchart of implementing a lifting quantization scheme. 
         FIG. 4  illustrates a flowchart of implementing the adaptive subband coding for lifting transform according to some embodiments. 
         FIG. 5  illustrates a block diagram of an exemplary computing device configured to implement the adaptive subband coding method according to some embodiments. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
     Lifting is a transform designed for color compression of point clouds which is adopted in one of the MPEG test models. The performance of lifting is improved herein. All the lifting coefficients (or residuals) are first divided into several subbands based on their assigned weights, which indicate the level of importance of each coefficient. Then, for each subband, a set of three dead-zones are derived for the three color components. The dead-zones of Cb and Cr channels are typically larger than that of Luma channel. In the original lifting scheme, Chroma is not suppressed at all. In contrast, as described herein, the size of the dead-zone is increased for different color components, which means that more quality (and bandwidth) is able to be adaptively provided for luminance coefficients than chrominance coefficients. Then, the point cloud content (or other 3D content) is encoded using the lifting transform. 
       FIG. 1  illustrates a diagram of the TMC13 quantization strategy according to some embodiments. For the fixed-point, a point cloud is quantized using a quantizer with a step-size Q. For floating-point, a point cloud is multiplied by a multiplier and then quantized with step-size Q. If not multiplied by the multiplier, significant color distortion is observed at the output due to reduced precision when a quantizer with fixed point step-size is used. For the multiplier, each point is assigned an influence weight, which depends on how many times the point has been used for prediction and what are the distances to the predicted points. The higher the number of times used for prediction, the larger the influence weight. The smaller the distance to the predicted points, the larger the influence weight. Points in a lower Level of Detail (LOD) usually have larger influence weights since they are usually used more for prediction of the remaining points. As shown in  FIG. 1 , the points are able to be re-ordered based on the LOD. 
       FIG. 2  illustrates a diagram of using lifting for TMC13 according to some embodiments. A lifting residual is multiplied by a residual scale factor, and the result is sent to a quantizer with an integer step-size. The residual scale factor is also multiplied and sent to a quantizer dead-zone, where the result goes to the quantizer with an integer step-size. The quantizer with an integer step-size determines the lifting level. 
       FIG. 3  illustrates a flowchart of implementing a lifting quantization scheme. In the step  300 , a lifting residual (or a block of residuals) is multiplied by a weight (e.g., the square root of a weight of a point, w(p)) to generate a weighted residual. In the step  302 , the weighted residual goes to a quantizer to be quantized to generate a quantized level. In the step  304 , the quantized level is divided by the weight to generate a reconstructed residual. 
       FIG. 4  illustrates a flowchart of implementing the adaptive subband coding for lifting transform according to some embodiments. In the step  400 , a lifting residual (or a block of residuals) is multiplied by a weight (e.g., the square root of a weight of a point, w(p)) to generate a weighted residual. In the step  402 , the weighted residual goes to a quantizer to be quantized to generate a quantized level. At the quantizer, if w(p)&lt;w LSW , then the dead-zone is increased; otherwise, the dead-zone is kept unchanged, where w LSW  is the weight of the less significant weight. For a given percentile α (e.g., α=95%), an objective is to find the less significant weight, w LSW , such that α percent of the weights are smaller than w LSW  and (1−α) percent of the weights are larger than w LSW . In the step  404 , the quantized level is divided by the weight to generate a reconstructed residual, which is used for color compression of a point cloud. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is modified. 
       FIG. 5  illustrates a block diagram of an exemplary computing device configured to implement the adaptive subband coding method according to some embodiments. The computing device  500  is able to be used to acquire, store, compute, process, communicate and/or display information such as images and videos including 3D content. The computing device  500  is able to implement any of the adaptive subband coding aspects. In general, a hardware structure suitable for implementing the computing device  500  includes a network interface  502 , a memory  504 , a processor  506 , I/O device(s)  508 , a bus  510  and a storage device  512 . The choice of processor is not critical as long as a suitable processor with sufficient speed is chosen. The memory  504  is able to be any conventional computer memory known in the art. The storage device  512  is able to include a hard drive, CDROM, CDRW, DVD, DVDRW, High Definition disc/drive, ultra-HD drive, flash memory card or any other storage device. The computing device  500  is able to include one or more network interfaces  502 . An example of a network interface includes a network card connected to an Ethernet or other type of LAN. The I/O device(s)  508  are able to include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface and other devices. Adaptive subband coding application(s)  530  used to implement the adaptive subband coding method are likely to be stored in the storage device  512  and memory  504  and processed as applications are typically processed. More or fewer components shown in  FIG. 5  are able to be included in the computing device  500 . In some embodiments, adaptive subband coding hardware  520  is included. Although the computing device  500  in  FIG. 5  includes applications  530  and hardware  520  for the adaptive subband coding method, the adaptive subband coding method is able to be implemented on a computing device in hardware, firmware, software or any combination thereof. For example, in some embodiments, the adaptive subband coding applications  530  are programmed in a memory and executed using a processor. In another example, in some embodiments, the adaptive subband coding hardware  520  is programmed hardware logic including gates specifically designed to implement the adaptive subband coding method. 
     In some embodiments, the adaptive subband coding application(s)  530  include several applications and/or modules. In some embodiments, modules include one or more sub-modules as well. In some embodiments, fewer or additional modules are able to be included. 
     In some embodiments, the adaptive subband coding hardware  520  includes camera components such as a lens, an image sensor, and/or any other camera components. 
     Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device. 
     To utilize the adaptive subband coding method described herein, a device acquires or receives 3D content and processes and/or sends the content in an optimized manner to enable proper, efficient display of the 3D content. The adaptive subband coding method is able to be implemented with user assistance or automatically without user involvement. 
     In operation, the adaptive subband coding method more efficiently processes 3D content including compressing the data such that much less information is sent. 
     Some Embodiments of Adaptive Subband Coding for Lifting Transform 
     
         
         1. A method programmed in a non-transitory memory of a device comprising: 
       
    
     multiplying a residual of a point cloud by a weight to generate a weighted residual; 
     quantizing the weighted residual to generate a quantized level; and 
     dividing the quantized level by the weight to generate a reconstructed residual which is used for color compression of the point cloud.
     2. The method of clause 1 further comprising:   

     dividing a plurality of lifting coefficients into a plurality of subbands; and 
     deriving a set of dead-zones for each subband for a set of color components.
     3. The method of clause 2 wherein dividing the plurality of lifting coefficients into the plurality of subbands is based on an assigned weight of each lifting coefficient.   4. The method of clause 2 wherein the set of color components includes Cb, Cr and luma channels.   5. The method of clause 1 wherein the weight is a square root of the weight of a point.   6. The method of clause 5 wherein if the weight of the point is less than a weight of a less significant weight, then a dead-zone size is increased.   7. The method of clause 1 wherein the weight is based on how many times a point has been used for prediction and distances to predicted points.   8. An apparatus comprising:   

     a non-transitory memory for storing an application, the application for:
         multiplying a residual of a point cloud by a weight to generate a weighted residual;   quantizing the weighted residual to generate a quantized level; and   dividing the quantized level by the weight to generate a reconstructed residual which is used for color compression of the point cloud; and       

     a processor coupled to the memory, the processor configured for processing the application.
     9. The apparatus of clause 8 wherein the application is further for:   

     dividing a plurality of lifting coefficients into a plurality of subbands; and 
     deriving a set of dead-zones for each subband for a set of color components.
     10. The apparatus of clause 9 wherein dividing the plurality of lifting coefficients into the plurality of subbands is based on an assigned weight of each lifting coefficient.   11. The apparatus of clause 9 wherein the set of color components includes Cb, Cr and luma channels.   12. The apparatus of clause 8 wherein the weight is a square root of the weight of a point.   13. The apparatus of clause 12 wherein if the weight of the point is less than a weight of a less significant weight, then a dead-zone size is increased.   14. The apparatus of clause 8 wherein the weight is based on how many times a point has been used for prediction and distances to predicted points.   15. A system comprising:   

     a multiplication module configured for multiplying a residual of a point cloud by a weight to generate a weighted residual; 
     a quantization module configured for quantizing the weighted residual to generate a quantized level; and 
     a dividing module configured for dividing the quantized level by the weight to generate a reconstructed residual which is used for color compression of the point cloud.
     16. The system of clause 15 further comprising:   

     a subband module configured for dividing a plurality of lifting coefficients into a plurality of subbands; and 
     a deriving module configured for deriving a set of dead-zones for each subband for a set of color components.
     17. The system of clause 16 wherein dividing the plurality of lifting coefficients into the plurality of subbands is based on an assigned weight of each lifting coefficient.   18. The system of clause 16 wherein the set of color components includes Cb, Cr and luma channels.   19. The system of clause 15 wherein the weight is a square root of the weight of a point.   20. The system of clause 19 wherein if the weight of the point is less than a weight of a less significant weight, then a dead-zone size is increased.   21. The system of clause 15 wherein the weight is based on how many times a point has been used for prediction and distances to predicted points.   

     The present invention has been described in terms of specific embodiments incorporating details to facilitate the understanding of principles of construction and operation of the invention. Such reference herein to specific embodiments and details thereof is not intended to limit the scope of the claims appended hereto. It will be readily apparent to one skilled in the art that other various modifications may be made in the embodiment chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.