Patent Publication Number: US-6907143-B2

Title: Adaptive spatio-temporal filter for human vision system models

Description:
BACKGROUND OF THE INVENTION 
   The present invention relates to video processing, and more particularly to an adaptive spatio-temporal filter for human vision system models used in determining the quality of video service. 
   Video is recorded and transmitted via methods that may create errors, such as lossy compression systems and the like. It is of interest to produce objective measures to predict the human perceptibility of these errors. The perceptibility of these errors at and above a threshold is a function of many aspects or parameters of the errors and the video context in which they occur. Perceptual sensitivity to errors in video as a function of spatial frequency varies with local average luminance, duration of local stimulus (temporal extent), the associated temporal frequency content, area (spatial/angular extent) and local contrast of the original, error-free or reference, video. Likewise perceptual sensitivity to errors in video as a function of temporal frequency varies with local average luminance, temporal extent, area, associated spatial frequency content and local contrast of the original video. 
   Existing spatio-temporal filtering in human vision system (HVS) models is generally designed with a fixed response calibrated to one particular set of parameters i.e., spatial and temporal responses are designed to adhere to the HVS sensitivities at one particular average luminance level. Also most HVS models are based on methods of predicting the threshold of just noticeable differences (JND), such as contrast detection and discrimination thresholds. Since the HVS model component are based on mimicking behavior at threshold, behavior above threshold, i.e., at supra-threshold, is not guaranteed. 
   For example J. Lubin in “A Visual Discrimination Model for Imaging System Design and Evaluation”,  Vision Models for Target Detection and Recognition , ed. Eli Peli, World Scientific Publishing, River Edge, N.J. 1995, pp. 245-283, proposed a model that has a fixed spatial filter designed to match human vision system response at only one luminance level and one duration or temporal extent. There is no mechanism to create temporal frequency roll-off at spatial frequencies of peak sensitivity, so this does not match data from human vision experiments. The model proposed by Lubin has no provision for moving peaks in the spatial or temporal frequency response as a function of luminance or other parameters. Also Lubin&#39;s model is based on units of threshold, or “just noticeable difference (JND)”. The only mechanism which modifies response beyond the fixed spatial and temporal filters is a masking mechanism which is a modified version of J. Foley&#39;s model for predicting contrast discrimination (“Contrast Masking in Human Vision”,  Journal of the Optical Society of America , Vol. 70, No. 12 pp.1458-1471, December 1980). However M. Canon showed (“Perceived Contrast in the Fovea and Periphery”,  Journal of the Optical Society of America , Vol. 2, No. 10 pp.1760-1768, 1985) that Foley&#39;s model is grossly in error when used to predict perceptual contrast at even moderate contrast levels above threshold. In addition Lubin&#39;s proposed model does not account for the non-linearities such as those causing the double spatial frequency and phantom pulse visual illusions. Many other human vision based models, such as that of S. Daly, “The Visible Differences Predictor: an Algorithm for the Assessment of Image Fidelity”,  Digital Images and Human Vision , ed. Andrew B. Watson, MIT Press, Cambridge, Mass. 1993, pp. 162-206, do not account for temporal aspects at all. 
   What is desired is a filter that is designed to match human vision system response over a range of perceptual parameters. 
   BRIEF SUMMARY OF THE INVENTION 
   Accordingly the present invention provides an adaptive spatio-temporal filter for human vision system models that is designed to match human vision response over a range of perceptual parameters. A pair of adaptive, lowpass, spatio-temporal filters are coupled in parallel to receive an input video signal, and the outputs are subtracted to provide a total output having a bandpass frequency response calibrated to a human vision system perceptual parameter. A filter adaptation controller also receives the input video signal and, based on an average luminance, generates coefficients for the respective lowpass spatio-temporal filters. Each lowpass spatio-temporal filter has a temporal IIR filter and a two-dimensional spatial IIR filter in cascade. Each of the temporal and spatial filters are composed of a common building block—a first order, unity DC gain, tunable lowpass filter. Each of the component filters receives its coefficients from the filter adaptation controller. In an alternative embodiment the filter adaptation controller receives the output of one of the lowpass spatio-adaptive filters rather than the input video signal as the basis for generating the filter coefficients. 
   The objects, advantages and other novel features of the present invention are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing. 

   
     BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
       FIG. 1  is a block diagram view of an adaptive spatio-temporal filter for human vision system models according to the present invention. 
       FIG. 2  is a block diagram view of a tunable IIR LP filter for use as a building block in the adaptive spatio-temporal filter according to the present invention. 
       FIG. 3  is a graphic view of the frequency response of the adaptive spatio-temporal filter according to the present invention. 
       FIG. 4  is a graphic view comparing HVS experimental data with the response of the adaptive spatio-temporal filter as a function of spatial frequency according to the present invention. 
       FIG. 5  is a graphic view comparing HVS experimental data with the response of the adaptive spatio-temporal filter as a function of temporal frequency according to the present invention. 
   

   DETAILED DESCRIPTION OF THE INVENTION 
   Referring now to  FIG. 1  an adaptive three-dimensional (3-D) filter  10  is shown which improves on the prior filters for human vision system (HVS) models by combining the bulk of the accuracy of the HVS models with an efficiency comparable to non-HVS models, such as the algorithm based on weighted SNR described by T. Hamada et al ( Picture Quality Assessment System by Three - Layered Bottom - Up Noise Weighting Considering Human Visual Perception , SMPTE Journal, January 1999, pgs. 20-26). The filter  10  has a pair of adaptive 3-D filters  12 ,  14 , a “Center” filter and a “Surround” filter, that receive coefficients from a filter adaptation controller  16  acting as a coefficient generator. An input video signal is applied to both filters  12 ,  14  and the filter adaptation controller  16 . The outputs from the filters  12 ,  14  are input to a differencing circuit  18 . The output from either the Center filter  12  or the Surround filter  14  also may be input to the filter adaptation controller  16 . The filter adaptation controller  16  generates filter coefficients from a perceptual parameter extracted from either the input video signal or the output of one of the adaptive 3-D filters  12 ,  14 . Each filter  12 ,  14  is implemented as a cascade of an infinite impulse response (IIR) temporal lowpass filter (LPF)  20 ,  22  followed by a non-causal two-dimensional (2-D) IIR spatial LPF  24 ,  26 , a filter that uses “future” data, i.e., filters both forward and backward. The differencing circuit  18  may be implemented as a summation circuit  28  and a multiplier circuit  30 . The output of the Center filter  12  is one input to the summation circuit  28 , while the other input is the output of the Surround filter  14  multiplied by −1 in the multiplier circuit  30 . The Center filter  12  has a higher bandwidth than the Surround filter  14 , with the relative filter responses being such that the resulting difference generally has a bandpass response, as shown in FIG.  3 . This bandpass response is first calibrated to the contrast sensitivity response of an HVS at a particular luminance level. 
   The filter adaptation controller  16  is responsible for emulating the HVS adaptation to local and global changes in a perceptual parameter, such luminance, contrast, etc., by controlling the bandwidth of each of the temporal and spatial filters  20 - 26 . Each of these filters is composed of a common building block having a topology suitable for integrated circuit (IC) implementation—a first order, unity DC gain, tunable LPF  32  as shown in FIG.  2 . The temporal filters  20 ,  22  use two of the tunable LPFs  32  in cascade to make a second order LPF. The spatial filters  24 ,  26  use two of the tunable LPFs  32  for each dimension (horizontal and vertical) and in opposite directions (right, left, up, down) to produce a zero-phase, non-causal 2-D IIR filter. The tunable LPF  32  shown is a first order discrete recursive difference equation based IIR filter, although other tunable filters may be used. An input signal X is multiplied by an input gain coefficient b 0  in a multiplier circuit  34 , and the result is input to a summation circuit  36 . The output of the summation circuit  36  is fed back to the input of the summation circuit after being multiplied by a feedback coefficient a 1  in another multiplier circuit  38  and delayed by one sample period by a delay circuit  40 . The sum of the coefficients is one, i.e., b 0 +a 1 =1, and therefore the DC gain also is one regardless of the individual values of b 0  (provided 0&lt;b 0 ≦1) and a 1 . This makes the filter  10  manageable since the DC gain is known when taking the difference between the Surround and Center filters  12 ,  14 . The bandwidth and corresponding pole of the tunable LPF  32  may be raised or lowered by raising or lowering b 0 . 
   The filter adaption controller  16  supplies the tunable LPFs  32  with the appropriate b 0  values at each pixel processed depending on the perceptual parameter, such as local average luminance, whether the filter is spatial or temporal, Center or Surround. The b 0  values may be taken from look-up tables derived from calculating the filters to match experimental data. Alternatively they may be calculated to approximate the entries in this look-up table. The look-up table may be derived by calibrating the filters&#39; coefficients to match experimental data. One method is to apply a linear fit to the calibrated b 0  values at different luminance levels:
 
 b   0 = K   1 + K   2 * Lum   (1)
 
where Lum is the local average luminance in nits (candelas/square meter).
 
   For calibration to HVS response data taken from a range of 0.28 to 91 nits, the following values may be used, assuming for this example sampling rates of 32.0 pixels per degree and 60 samples per second: 
                                      spatial Center filter 24   K1 = 0.67   K2 = 0.00067       temporal Center filter 20   K1 = 0.54575   K2 = 0.00295       spatial Surround filter 26   K1 = 0.225   K2 = 0.00225       temporal Surround filter 22   K1 = 0.4125   K2 = 0.00375                    
The filter adaptation controller  16  determines the Lum value based on either the Surround filter  14  input or output, as shown in FIG.  1 . The method for determining Lum may be optimized for particular applications to balance the requirements of filter stability, number of computational operations and accuracy.
 
   For the Lum value based on the input, X[n], to the entire filter  10  the adaptation times are controlled for accuracy by yet another filter. In each dimension Lum is obtained in the filter adaptation controller  16  by filtering according to the following equation:
 
 Lum[n]=Lb   0 *( X[n]−Lum[n −1])+ Lum[n −1]  (2)
 
where n is the nth sample in a particular dimension and Lb 0  corresponds to b 0  in the HVS model. The non-linearities of the filter  10  are given by substituting the recursive equation for Lum in the equation for the tunable filters  32 : 
                     y   ⁡     [   n   ]       =       ⁢       b0   *     x   ⁡     [   n   ]         +       (     1   -   b0     )     *     y   ⁡     [     n   -   1     ]                       =       ⁢       b0   *     (       x   ⁡     [   n   ]       -     y   ⁡     [     n   -   1     ]         )       +     y   ⁡     [     n   -   1     ]                     =       ⁢         (     K1   +     k2   *   Lum       )     *     (       x   ⁡     [   n   ]       -     y   ⁡     [     n   -   1     ]         )       +     y   ⁡     [     n   -   1     ]                     =       ⁢     (     K1   +     K2   *     (       Lb0   *     (       x   ⁡     [   n   ]       -     Lum   ⁡     [     n   -   1     ]         )       +                                   ⁢       Lum   ⁡     [     n   -   1     ]       )       )     *     (       x   ⁡     [   n   ]       -     y   ⁡     [     n   -   1     ]         )       +     y   ⁡     [     n   -   1     ]                   =       ⁢       K1   *     x   ⁡     [   n   ]         +       (     1   -   K1     )     *     y   ⁡     [     n   -   1     ]         +     K2   *     (     Lb0   *     x   ⁡     [   n   ]       *                           ⁢       (       x   ⁡     [   n   ]       -     y   ⁡     [     n   -   1     ]         )     +       (     1   -   Lb0     )     *     Lum   ⁡     [     n   -   1     ]       *                       ⁢       (       x   ⁡     [   n   ]       -     y   ⁡     [     n   -   1     ]         )     )                 =       ⁢       (     IIR   ⁢           ⁢   LPF   ⁢           ⁢   linear   ⁢           ⁢   response     )     +     (     Input   ,     output   ⁢           ⁢   product                           ⁢     terms   -     non   ⁢     -     ⁢   linearities       )                 (   3   )             
 
Thus the output of each tunable first order IIR LPF  32  is a linear combination of a fixed linear LPF with b 0 =K 1  and a non-linear high pass filter whose gain is controlled by the product of Lb 0  and K 2 . Since K 2  is already determined, Lb 0  is chosen such that the combination of visual illusion non-linearities and adaptation times are optimally matched. For low contrast levels and/or luminance levels the filter  10  output is approximately linear, but at higher contrast and luminance levels the non-linear component becomes more substantial so that the non-linearities are consistent with enhanced perceived brightness.
 
   For speed improvement the Lum filter may be eliminated (Lb 0 =1) and the single adaptive filter response is:
 
 y[n]=K   1 * x[n ]+(1 −K   1 )* y[n −1 ]+K   2 *( x[n ]*( x[n]−y[n −1]))
 
   The Lum value may be based on the output from the Surround filter  14 . In the case where speed is more desirable, no additional filtering is used:
 
 Lum[n −1 ]=y   s   [n −1]
 
Other calculation alternatives for y[n] may be based on this equation by substitution into equation (3).
 
   Stability considerations limit the values of Lb 0  that are possible if y[n] represents an individual filter output of the Surround LPF  14 . Also other perceptual parameters besides local average luminance, such as contrast, etc. may be used by the filter adaptation controller  16  to generate the filter coefficients. 
   Test results show a match between human vision system response and the adaptive spatio-temporal filter response, as shown in  FIGS. 4 and 5  for an average luminance of 20 nits.  FIG. 4  is a plot of modulation sensitivity versus spatial frequency with the human vision system model results (Ms#Hz20 nits) being shown as circles or boxes and the corresponding filter response (H tindex(#),ks ) being shown as lines. Likewise  FIG. 5  is a corresponding plot of modulation sensitivity versus temporal frequency at the same luminance level (20 nits) with the human vision system model results (Ms#cpd20 nits) being shown as circles or diamonds and the corresponding filter response (Hrot sindex(#),kt ) being shown as lines. Also rectification of the output is consistent with brightness enhancement in intermittent light, an effect that increases with increasing average luminance with corresponding peaks at increasing temporal frequencies. The behavior of the adaptive spatio-temporal filter  10  is consistent with data from other human vision experiments including: threshold of perception as in discrimination; supra-threshold temporal contrast perception; perceived spatial frequency including frequency aliasing/doubling; perceived temporal frequency including flickering, etc. perceived velocity in both speed and direction; perceived phantom noise including noise, residual images, extra/missing pulses, etc. neural response including voltage waveforms, etc. and additional sensitivity experiments involving changes in: mean luminance; angular extent or size of target image on retina; orientation including rotational, both of target image pattern an masker; spatial frequency for both target and masker; temporal frequency of both target image pattern and masker; duration or temporal extent; and surround or lateral masking effects. masker; duration or temporal extent; and surround or lateral masking effects. 
   Thus the present invention provides an adaptive spatio-temporal filter for human vision system models that has a pair of spatio-temporal filters in parallel with overlapping bandwidths such that the difference of the outputs produces a band pass response, each filter being composed of common adaptive, unity DC gain, tunable lowpass filters having a linear lowpass response portion and a non-linear high-pass response portion, with the filter coefficients being determined for each pixel processed from a perceptual parameter, such as local average luminance, contrast, etc., to produce a result consistent with experimental results of human vision system models.