Abstract:
The use of the three-dimensional DCT as a key compression technology requires development of an entirely new quantizing mechanism. The embodiment described herein uses a Human Visual Model to develop quantizers based on a combination of descriptive characteristics of the video source, enabling independent derivation of said quantizers in both encoder and decoder sides of the compression and playback process.

Description:
CROSS-REFERENCE TO RELATED APPLICATION(S) 
       [0001]    This application is a continuation of U.S. application Ser. No. 14/266,757, filed on Apr. 30, 2014, which is a non-provisional and claims benefit of a U.S. Provisional Application No. 61/818,419, filed on May 1, 2013. 
     
    
     BACKGROUND 
       [0002]    The present invention relates generally to compression of moving video data, and more particularly to the application of quantization of the three-dimensional Discrete Cosine Transform (DCT) representation of moving video data for the purposes of removing visually redundant information. 
       SUMMARY 
       [0003]    In accordance with one aspect of the invention, a method is provided for removal of all subjectively redundant visual information by means of calculating optimal visually-weighed quantizers corresponding to the decorrelating-transformed block decomposition of a sequence of video images. The contrast sensitivity of the human eye to the actual time-varying transform-domain frequency of each transform component is calculated, and the resolution of the transformed data is reduced by the calculated sensitivity. 
         [0004]    A second aspect of the invention applies specifically to use of the DCT as the decorrelating transform. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]      FIG. 1  depicts a prior-art compressor featuring an optimal spatial transform and optimal fixed visual quantizers (JPEG). 
           [0006]      FIG. 2  depicts a prior-art compressor featuring a sub-optimal time-varying transform using sub-optimal quantizers fixed or in-band quantizers (MPEG). 
           [0007]      FIG. 3  depicts a prior-art compressor featuring a sub-optimal time-varying transform and a recursive feedback quantizer calculation to generate in-band quantizers. 
           [0008]      FIG. 4  depicts a prior-art compressor featuring a sub-optimal time-varying transform using sub-optimal quantizers fixed or in-band quantizers (wavelet). 
           [0009]      FIG. 5  depicts a compression system featuring an optimal time-varying transform using configuration parameters to independently generate visually optimal quantizers in compressor and decompressor. 
           [0010]      FIG. 6  describes a typical set of configuration parameters that may be used to generate visually optimal time-varying quantizers. 
           [0011]      FIG. 7  defines a typical time-varying contrast sensitivity function. 
           [0012]      FIG. 8  defines a visually optimal quantizer in terms of visual resolution and the contrast sensitivity function specified in  FIG. 7 . 
           [0013]      FIG. 9  refines the visually optimal quantizer definition of  FIG. 8  with angular data specifications. 
           [0014]      FIG. 10  refines the visually optimal quantizer definition of  FIG. 8  with off-axis visual sensitivity human visual system adjustments. 
           [0015]      FIG. 11  depicts a typical symmetric contrast sensitivity function (without angular or off-axis corrections). 
           [0016]      FIG. 12  depicts typical contrast sensitivity function off-axis visual sensitivity human visual system adjustments. 
           [0017]      FIG. 13  depicts typical eccentric-angle visual sensitivity human visual system adjustments. 
           [0018]      FIG. 14  depicts the location of DC, mixed DC/AC, and AC components within a 3-dimensional DCT block. 
           [0019]      FIG. 15  illustrates the calculation of DC component quantizers, and the contributed DC and AC quantizers of a mixed DC/AC component. 
           [0020]      FIG. 16  illustrates the calculation of a statistically ideal mixed DC/AC quantizer. 
           [0021]      FIG. 17  illustrates the application of a configurable Gibbs ringing compensation factor. 
       
    
    
     DETAILED DESCRIPTION 
       [0022]    It is well established in the literature of the field of video compression that video can be well-modeled as a stationary Markov-1 process. This statistical model predicts the video behavior quite well, with measured correlations over 0.9 in the pixel and line directions. 
         [0023]    It is well-known the Karhunen-Loeve Transform (KLT) perfectly decorrelates Markov-distributed video. This means the basis of the KLT is an independent set of vectors which encode the pixel values of the video sequence. 
         [0024]    It is a further result that many discrete transforms well approximate the KLT for large correlation values. Perhaps the best-known such function is the DCT, although many other functions (DST, WHT, etc.) serve as reasonable approximations to the KLT. 
         [0025]    It is for this reason the DCT is used to decorrelate images in the JPEG standard, after which a uniform quantization factor individually chosen for each DCT component is applied to said component, removing visual information imperceptible to the human eye.  FIG. 1  illustrates the use of a Human Visual System quantizer array in JPEG. An individual frame of digitized video  1010  is transformed via a two-dimensional DCT  1020  and then quantized  1020  to remove imperceptible visual data. An entropy removal process  1040  actually compresses the information. The decompression process follows an equivalent set of steps in reverse, when a data set or data stream containing the compressed data  1210  is decompressed  1110  by reversing said entropy removal process, followed by a de-quantization step  1120 , an inverse DCT step  1130 , and a resulting frame  1140  may be displayed or otherwise processed. A key part of the process is good choice of quantizers  1310  that leverage a Human Visual Model to optimally remove redundant information. The use of a Human Vision Model in terms of a Contrast Sensitivity Function to generate two-dimensional quantizer coefficients is taught by Hwang, et al, and by Watson U.S. Pat. No. 5,629,780. 
         [0026]      FIG. 2  illustrates the use of the DCT in the prior-art MPEG standard. A block-based difference after motion estimation  2015  is taken between reference frame(s)  2010  and an individual frame to be compressed  2005 . Said block-based difference after motion estimation  2015  is transformed using the two-dimensional DCT  2020  and quantized  2030 . The resulting quantized data is compressed via an entropy removal process  2040 , resulting in a compressed data set or stream  2210 . A decompression process can then be executed on said compressed data set or stream  2210 , comprising the reverse entropy removal step  2110 , a de-quantizing step  2120 , an inverse two-dimensional DCT process  2130 , and a block-based summation process  2135  using a previously-decompressed reference frame  2140  to generate an individual frame ready for playback or other processing  2145 . The pre-defined fixed quantizer  2310  utilized in said quantization process  2030  and said de-quantization process  2130  cannot leverage the Human Vision Model, as no such model has been developed to apply directly to the difference between video blocks. 
         [0027]    What is needed is a means of removing subjectively redundant video information from a moving sequence of video. 
         [0028]    Many prior-art techniques are taught under the principle of guiding a design of a quantization matrix to provide optimum visual quality for a given bitrate. These techniques, being applicable to motion compensation-based compression algorithms, require a Human Visual Model-driven feedback loop to converge on the quantizers that will show minimal artifact on reconstruction. The use of this Human Visual Model is again limited to its application in the spatial domain. An example of this teaching is U.S. Pat. No. 8,326,067 by Furbeck, as illustrated in  FIG. 3 . A block-based difference after motion estimation  3015  is taken between reference frame(s)  3010  and an individual frame to be compressed  33005 . Said block-based difference after motion estimation  3015  is transformed using the two-dimensional DCT  3020  and quantized  3030 . The resulting quantized data is compressed via an entropy removal process  3040 , resulting in a compressed data set or stream  3210 . A decompression process can then be executed on said compressed data set or stream  3210 , comprising the reverse entropy removal step  3110 , a de-quantizing step  3120 , an inverse two-dimensional DCT process  3130 , and a block-based summation process  3135  using a previously-decompressed reference frame  3140  to generate an individual frame ready for playback or other processing  3145 . The quantizer  3310  utilized in said quantization process  3030  and said de-quantization process  3130  cannot directly leverage the Human Vision Model, as no such model has been developed to apply directly to the difference between video blocks. Therefore a feedback processing step  3240  communicates to a Human Visual Model  3250  which determines the perceptual error, and feeds back recalculated said quantizers  3310  to be used to re-compress said individual frame to be compressed  3005 . Said feedback processing step  3240  may be based on simple perceptual error minimization, or may minimize compression ratio after entropy removal. 
         [0029]    The wavelet transform is another technique commonly used to perform compression. However, the wavelet does not decorrelate video, and thus optimal quantizers based upon a Human Visual Model cannot be calculated. A teaching by Gu et al, U.S. Pat. No. 7,006,568 attempts to address this issue by segmenting video sequences into similar-characteristic segments and calculating 2-D quantizers for each selected segment, chosen to reduce perceptual error in each subband, as illustrated in  FIG. 4 . A frame to be compressed  4005  is decomposed into its subbands via wavelet decomposition  4020  and quantized  4030 . The resulting quantized data is compressed via an entropy removal process  4040 , resulting in a compressed data set or stream  4210 . A decompression process can then be executed on said compressed data set or stream  4210 , comprising the reverse entropy removal step  4110 , a de-quantizing step  4120 , a subband reconstruction process  4130  to generate an individual frame ready for playback or other processing  4140 . The quantizer  4330  utilized in said quantization process  4030  and said de-quantization process  4130  cannot directly leverage the Human Vision Model, as no such model has been developed to apply directly to the poorly-decorrelated video basis of the wavelet decomposition. This prior-art teaching subdivides the video stream into regions of relatively stable visual performance bounded by scene changes, as calculated by a scene analysis process  4310  acting upon said frame to be compressed  4005  and its previous frame in the motion video sequence  4010 . A visually-weighted analysis process  4320  then calculates said quantizers  4330 . 
         [0030]    The current invention improves the compression process by directly calculating the visually optimal quantizers for 3-D transform vectors by evaluating the basis behavior of the decorrelated transform space under a time-varying Human Visual Model, as represented by a Contrast Sensitivity Function. 
         [0031]    As illustrated in  FIG. 5 , a block comprising a plurality of individual frames of digitized video  5010  is transformed via a three-dimensional DCT  5020  and then quantized  5030  to remove imperceptible visual data. An entropy removal process  5040  actually compresses the information. The decompression process follows an equivalent set of steps in reverse, when a data set or data stream containing the compressed data  5210  is decompressed  5110  by reversing said entropy removal process, followed by a de-quantization step  5120 , an inverse DCT step  5130 , and a resulting block of frames  5140  may be displayed or otherwise processed. Said quantizer process  5030  and said de-quantizer process  5120  use quantizers  5420  generated by a quantizer generation process  5410 . Said quantizer generation process  5410  calculates said quantizers  5420  as a function of four sets of configuration data, the conditions under which viewing is expected to take place, and under which visual reconstruction will have no perceptual error  5310 , the configuration of the video stream  5320 , the quantizer generation algorithm to be used  5330 , and the configuration of the applied decorrelating transform  5340 . 
         [0032]    In the current embodiment, said configuration of video stream  5320  is elaborated in  FIG. 6 . Said configuration of video stream  6010  is comprised of individual configuration items H  6020 , the number of pixels per line within the frame, V  6030 , the number of lines within the frame, R  6040 , the frame rate in frames per second, B  6050 , the number of bits used to represent the luminance value per pixel, and Aspect  6060 , the physical aspect ratio or ratio of physical frame width to physical frame height. 
         [0033]    In the current embodiment, said configuration of viewing conditions  5310  is elaborated in  FIG. 6 . Said configuration of viewing conditions  6110  is comprised of individual configuration items D  6120 , the expected viewing distance in screen heights, and I  6130 , the expected average ambient luminance. 
         [0034]    In the current embodiment, said configuration of block-based decorrelating transform  5340  is elaborated in  FIG. 6 . Said configuration of block-based decorrelating transform  6210  is comprised of individual configuration items N  6220 , the number of pixels per transform block, M  6230 , the number of lines per transform block, L  6240 , the number of frames per transform block, Nindex  6250 , the number of frames per transform block, and Mindex  6260 , the number of frames per transform block. 
         [0035]    In the current embodiment, said configuration of quantizer algorithm  5330  is elaborated in  FIG. 6 . Said configuration of quantizer algorithm  6310  is comprised of individual configuration items visual loss factor,  6320  Mx, mixed DC/AC coefficient algorithm,  6330  Rx, Ry and Rz, correlation in pixel, line and frame directions respectively, and  6340  dBG, Gibbs ringing compensation. 
         [0036]      FIG. 7  defines a typical contrast sensitivity function  7010  CSF(u,w,l,X0,Xmax) in terms of said ( 6130 ) viewing conditions configuration item expected average ambient luminance l  7040 , and additional variables u  7020 , 2-dimensional spatial frequency, w  7030 , temporal frequency, X0  7050 , angle subtended by DCT block, and Xmax  7060 , angle subtended by display surface. 
         [0037]    Luminance quantizers are calculated as in  FIG. 8( a ) . The equation  8010  calculates the quantizer Q  8020  for a particular decorrelating transform component of index n  8030  in the pixel direction, a particular decorrelating transform component of index m  8040  in the line direction and a particular decorrelating transform component of index l  8050  in the frame or time direction, a particular decorrelating transform component of position Mindex  8060  in the pixel direction and a particular decorrelating transform component of position Nindex  8070  in the line direction; given said two-dimensional spatial frequency u ( 7020 ), said temporal frequency w ( 7030 ), of said ( 6130 ) viewing conditions configuration item expected average ambient luminance l ( 7040 ), said angle subtended by DCT block X0 ( 7050 ), and said angle subtended by display surface Xmax ( 7060 ). 
         [0038]    The equation  8110  of  FIG. 8( b )  calculates said temporal frequency of a transform component w ( 7030 ) as a function of said configuration of video stream configuration item frame rate in frames per second R ( 6040 ), said configuration of block-based decorrelating transform configuration item number of frames per transform block L ( 6240 ), and said particular decorrelating transform component of index in the frame or time direction l ( 8050 ). 
         [0039]    The equation  9010  of  FIG. 9( a )  depicts a typical definition of said angle subtended by display surface Xmax ( 7060 ) in terms of said configuration of viewing conditions individual configuration item D the expected viewing distance in screen heights ( 6120 ). The equation  9020  of  FIG. 9( b )  depicts a typical definition of said angle subtended by DCT block X0 ( 7050 ) in terms of said configuration of block-based decorrelating transform individual configuration item the number of pixels per transform block N ( 6220 ) and said configuration of block-based decorrelating transform individual configuration item the number of lines per transform block M ( 6230 ). 
         [0040]    Equation  10010  of  FIG. 10  depicts a preferred process calculating said two-dimensional spatial frequency u ( 7020 ) given said particular decorrelating transform component of index in the pixel direction n ( 8030 ), said particular decorrelating transform component of index in the line direction m ( 8040 ), said particular decorrelating transform component of position in the pixel direction Mindex ( 8060 ) and a particular decorrelating transform component of position in the line direction Nindex ( 8070 ). A human visual system orientation response adjustment is rθ  10020 . A human visual system ex-foveal eccentricity response adjustment is rθ  10030 . 
         [0041]    The two-dimensional map of values assumes by said typical contrast sensitivity function CSF(u,w,l,X0,Xmax) ( 7010 ) for equally-weighted is depicted in  FIG. 11 . The contour map of  FIG. 12( a )  further illustrates the symmetric distribution of said typical contrast sensitivity function CSF(u,w,l,X0,Xmax) ( 7010 ), while The contour map of  FIG. 12( b )  illustrates the application of said human visual system orientation response adjustment rθ ( 10020 ) to better model human visual orientation response. The contour map of  FIG. 13  illustrates the application of said human visual system ex-foveal eccentricity response adjustment rθ ( 10030 ) to better model human visual off-axis response. 
         [0042]    As illustrated in  FIG. 14 , said block  14010  transformed via a three-dimensional DCT ( 5020 ) is comprised a plurality of transform components. Transform component (n=0,m=0,l=0)  14020  is classified as pure DC. Transform components with (n=0)  14030 , with (m=0)  14040 , or with (l=0)  14050  are classified as mixed AC/DC. Component where no (l,m,n) is 0 are classified as pure AC components. 
         [0043]    Said quantizer Q ( 8020 ) gives optimal response for pure AC transform components, but produces sub-optimal results for pure DC or mixed AC/DC components, due to the extreme sensitivity of the human eye to DC levels. Pure DC transform components may be quantized by the value that the variance of the DC component is concentrated over the number of possible levels that can be represented in the reconstructed image, as the human eye is constrained to the capabilities of the display. Equation  15010  of  FIG. 15( a )  defines the pure DC transform quantizer as a function of said configuration of block-based decorrelating transform individual configuration item the number of pixels per transform block N ( 6220 ), said configuration of block-based decorrelating transform individual configuration item number of lines per transform block M ( 6230 ), and said configuration of block-based decorrelating transform individual configuration item number of frames per transform block L ( 6240 ). 
         [0044]    Mixed AC/DC components can be quantized by the minimum quantization step size apportioned over the variance of the DCT basis component. This process requires calculation of the per-component variance for the AC and DC components (i.e., the variance calculation in the number of dimensions in which each AC or DC component resides). Similarly, the value of the independent AC and DC quantizers must be calculated using the Contrast Sensitivity Function limited to the number of dimensions in which the AC or DC component resides. As illustrated in  FIG. 15( b ) , the pseudocode C language program calcQ  15110  defines a quantizer suitable for application to the DC portion of mixed AC/DC components quantizer as a function of said configuration of block-based decorrelating transform individual configuration item the number of pixels per transform block N ( 6220 ), said configuration of block-based decorrelating transform individual configuration item number of lines per transform block M ( 6230 ), and said configuration of block-based decorrelating transform individual configuration item number of frames per transform block L ( 6240 ). Said typical AC/DC component with l=0  14050 , the one-dimensional DC quantizer QDCm,n,0  15210  is calculated from said reduced-dimension calculation of the quantizer calcQ  15110 . 
         [0045]    The two-dimensional AC quantizer QACm,n,0  15220  is calculated directly from said typical generalized Contract Sensitivity Function CSF(u,w,l,X0,Xmax)  7010 . 
         [0046]      FIG. 16  illustrates the process of deriving a statistically optimal quantizer Q m,n,0  16310  from said the one-dimensional DC quantizer QDCm,n,0  15210  and said two-dimensional AC quantizer QACm,n,0  15220 . Said correlation coefficient  6330  Rx is used to generate an autocorrelation matrix Mx  16010 . The convolution of said autocorrelation with the DCT in the x direction returns the variance-concentration matrix Cx  16020 . Said process is understood to apply equally in the y and z directions. 
         [0047]    The maximum visual delta of 1/QACm,n,0  16110  calculated to apply to the variance-concentrated range Cx[m,m]*Cy[n,n]  16120  and 1/QDCm,n,0  16130  calculated to apply to the variance-concentrated range Cz[0,0]  16130  is calculated as 1/min(QACm,n,0,QDCm,n,0)  16210 , and can be applied over the entire range Cx[m,m]*Cy[n,n]*Cz[0,0]  16220 . 
         [0048]    Said statistically optimal quantizer Q m,n,0  16310  may now be calculated following with the C language pseudocode excerpt  16320 . It is to be understood that the process of calculating typical statistically ideal mixed AC/DC coefficients is illustrated in the general sense in  FIG. 15  and  FIG. 16 , with minor changes to the procedure obvious to any experienced practitioner of the art. 
         [0049]    The worst-case degradation in visual quality caused by the Gibbs phenomenon as a result of quantization is illustrated in  FIG. 17 a   . A further adjustment to visual quality is supported by said Gibbs ringing adjustment  6340  dBG, which is interpreted ( FIG. 17 b   ) as illustrated in equation  17010  as a logarithmic factor of the actual reduction factor G  17020 . Said dBG  6340  with a value of 0 represents said quantizer reduction factor G  17020  of 8.985%, which precisely removes the worst-case Gibbs ringing from having visible effect. Gibbs ringing removal is applied to said quantizers  5420  generated by said quantizer generation process  5410  as illustrated in equation  17110  by reduction in magnitude by the factor 1−G (one minus said factor G  17020 ).