Abstract:
A method of processing digital video signals produced by a sensor that are to be presented on a viewfinder, the method involving: a first pair of processing operations for scaling and color interpolation; and a second pair of processing operations for the formation of a color matrix and for white balancing. The operations of at least one, and preferably of both of the pairs of processing operations are executed in a single step. The operation of white balancing is moreover performed only for one frame out of K frame in the frame sequence. The preferential application is in the construction of viewfinders for videocameras and digital still cameras.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    The present invention relates to techniques for processing video signals, such as are used in systems for image acquisition, in particular in the sector of consumer-electronics devices, that acquire images by means of digital sensors, such as CCDs or CMOS sensors, including reduced size color displays, and corresponding computer program.  
           [0003]    2. Description of the Related Art  
           [0004]    In the applications to which reference has been made above, the input data are normally acquired according to a Bayer configuration or pattern (see in this connection U.S. Pat. No. 3,971,065). To obtain the final image it is necessary to apply a reconstruction algorithm, the purpose of which is to exploit as far as possible the information acquired by the sensor to obtain the best possible image. For a general discussion of the problems linked to this approach, useful reference may be made to the work of M. Mancuso and S. Battiato: “An introduction to the digital still camera technology”, ST Journal of System Research, Vol. 2, No. 2, pp. 1-9, December, 2001.  
           [0005]    The block diagram of FIG. 1 represents the typical configuration of the image-acquisition system present, for example, in a digital videocamera or still camera of general production.  
           [0006]    The reference number  10  designates the image sensor, i.e., the physical sensor (for example, CCD/CMOS).  
           [0007]    The corresponding output signal is usually organized according to a Bayer pattern of the type presented in FIG. 2.  
           [0008]    The information contained in this type of image corresponds to a sub-sampling of a color image. In the pattern, each pixel contains information regarding just one color “channel” (red, green, blue). All the color channels regarding a pixel can be reconstructed by means of interpolation of the information regarding the individual channels.  
           [0009]    The reference number  12  in the diagram of FIG. 1 designates a module that performs a pre-processing of the data supplied by the sensor  10  with a view to subsequent treatments. In particular, comprised in this module are a block for white balancing, a block for construction of the color matrix, a block for range correction, etc.  
           [0010]    The block  14  performs the reconstruction of the color image from the Bayer pattern. The corresponding output is represented by an image in which the chromatic information has been reconstructed, i.e., in which each pixel has the information regarding each channel, and the missing information has been reconstructed by the module  14 .  
           [0011]    The corresponding image signal can then be transferred to a module  16  functioning as a compression module (typically according to the JPEG standard) with a view to storage in a memory  18 . The memory in question can be constituted either by the physical medium on which the video signals are stored (videocassettes, CD-ROMs, etc.) or by a memory associated to a display unit (not illustrated).  
           [0012]    In the case of a videocamera or similar device (e.g., digital still camera) the signal at output from the sensor  10  is supplied—in addition to the “main” processing chain illustrated previously—also to an auxiliary processing chain having the function of processing an image designed to be presented on a viewfinder that enables the user to recognize the characteristics of the image that he is photographing.  
           [0013]    For the above purpose, the signal arriving from the sensor  10  is supplied to a module designated by  20  which processes it in such a way as to enable presentation of the signal on a display or viewfinder designated by  22 .  
           [0014]    The corresponding refresh factor must be sufficiently high to cause the image presented on the viewfinder to correspond effectively to the images photographed of filmed and treated by the main processing chain comprising the elements designated by  12  to  18 .  
           [0015]    The viewfinder  22  must therefore present a somewhat high refresh rate (frame rate), but the image presented need not have a very high quality, since it primarily has the purpose of functioning as preview of the final image, interpolated as much as possible.  
           [0016]    At the same time it is important that the image presented on the viewfinder  22  should provide a sufficiently faithful representation of the characteristics of the image proper in terms of color balancing and construction of the corresponding matrix. However, since in the final analysis it is an auxiliary or additional function, it is desirable to proceed so that the function of processing implemented in the module  20  is not excessively burdensome from the computational standpoint or from the standpoint of circuit complexity or both.  
         BRIEF SUMMARY OF THE INVENTION  
         [0017]    The disclosed embodiments of the present invention provide a solution to meet these needs that includes in one embodiment a processing method presenting the characteristics recalled specifically in the ensuing claims, and to the corresponding processing system, which can be implemented both in the form of a dedicated processor and in the form of a general-purpose processor (e.g., DSP) adequately programmed. In this perspective, the invention also relates to the computer-program product that can be directly loaded into the memory of a digital processor and contains portions of software code for implementing the method according to the invention when the product is run on a computer.  
           [0018]    An embodiment of the invention operates on video sequences with Bayer matrix of the color filter-array (CFA) type. The fact of working on original CFA data enables savings in terms of resources, time, and space, which is particularly important for real-time applications.  
           [0019]    The solution according to this embodiment favors rapidity and recourse to a small number of operations, albeit maintaining a good quality of the final image, in particular obtaining a reasonable compromise in terms of quality perceived by the end user (in terms of definition, saturation, and tone of the color, and frame rate) and computational complexity.  
           [0020]    This result is achieved taking into account the fact that the parallel chain for processing and storage of the images (modules  12  to  18  of FIG. 1) aims at obtaining the maximum level of quality with a complete spatial and field resolution. The typical liquid-crystal display (LCD) used in devices of a consumer type for application as viewfinders employ just one fraction (for example, ⅓ or ¼) of the original resolution of the sensor, this applying also to the resolution in terms of depth of field linked to the number of bits per pixel.  
           [0021]    The solution described herein adapts the processing to the final conditions and conditions of observation immediately after the process of acquisition. In particular, instead of applying the scaling function from the Bayer pattern to the RGB color planes—after complete processing, and hence with full resolution—and then performing a scaling to the final display conditions, the solution described herein adapts the entire processing line adequately right from the start.  
           [0022]    Results demonstrate the effectiveness of the solution above all as regards the possibility of achieving considerable savings in terms of frame-rate processing, albeit maintaining a high quality of the final image perceived.  
           [0023]    More specifically, the solution described herein envisages recourse to an implementation of a simple type and hence “light” in terms of computational burden, in particular by combining together the processing operations that can be performed in a single step. In addition, in the presence of data that change slowly, there is envisaged execution of the single processing for a number of successive frames.  
           [0024]    Specifically, in the currently preferred embodiment of the invention, there is envisaged: performing in a single step the function of scaling and of interpolation of the color; evaluating the white balancing taking into account only one frame out of K frames, and not all the frames; moreover the gains are estimated on one part of the sub-sampled image; performing in a single step the operations of white balancing and correction of the chromatic components (color matrixing); implementing the range correction by means of a look-up table (LUT); and improving the quantization, using the improved-grey-scale (IGS) algorithm.  
           [0025]    In accordance with a method for processing digital video signals of the present invention, in which the digital video signals are produced by a sensor, the method includes subjecting the digital video signals to a first pair of processing operations of scaling and color interpolation; and to a second pair of processing operations of formation of a color matrix and white balancing, and the operations of at least one of the first and second pair of processing operations are executed in a single step.  
           [0026]    In accordance with another aspect of the foregoing embodiment, the method includes organizing the digital video signals in sets of subsequent frames, each set comprising a given number of frames, and performing the white balance on just one frame for each set. Ideally, the operation of white balancing is performed only for one frame out of K frames in a frame sequence of the digital video signals. Furthermore, implementing a range correction is provided by means of a look-up table.  
           [0027]    In accordance with another embodiment of the invention, a system for processing digital video signals is provided, the digital video signals produced by a sensor and presented ideally on a display device, such as a viewfinder. The system includes a first set of modules for the execution of operations of scaling and of color interpolation; and a second set of modules for the execution of operations of formation of a color matrix and of white balancing. Ideally, the module of at least one of the first and second set of modules is integrated so as to execute the corresponding operations in a single step.  
           [0028]    In accordance with another aspect of the foregoing embodiment, the system is configured for operating on digital video signals in sets of subsequent frames, each set comprising a given number of frames, ideally K frames, and comprising a module for white balancing that is configured for performing the white balancing on just one frame for each set of K frames in the frame sequence of the digital video signals.  
           [0029]    In accordance with another aspect of the foregoing embodiment, a method is provided that includes a first pair of processing operations that are a scaling operation and a color interpolation operation; and subjecting the digital video signals to a second pair of processing operations that are a color matrix formation operation and a white balancing operation, and the operations of at least one of the first and second pair of processing operations are performed in a signal step.  
           [0030]    In accordance with another embodiment of the invention, a method of processing digital video signals is provided that includes performing in a single step the functions of scaling and of interpolation of color; and performing in a single step the operations of white balancing and color matrixing. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0031]    The invention will now be described, purely by way of non-limiting example, with reference to the annexed drawings, in which:  
         [0032]    [0032]FIGS. 1 and 2 have already been described previously;  
         [0033]    [0033]FIG. 3 illustrates, in the form of a block diagram, an embodiment of the invention; and  
         [0034]    FIGS.  4  to  8  illustrate schematically various processing functions that can be implemented in accordance with the invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0035]    In the block diagram of FIG. 3, the reference  10  once again indicates the sensor that supplies at output starting image data in Bayer format, for example on eight bits.  
         [0036]    The corresponding display, such as a viewfinder (for example, an LCD viewfinder) is again designated by  22 . For immediate reference, it may then be assumed that the complex of the modules indicated in the figure with the reference numbers  24  to  34  correspond basically to the block  20  of FIG. 1.  
         [0037]    More precisely, in the block diagram of FIG. 3, the reference number  22  designates the display control unit (of a known type) designed to drive a display of this sort. In what follows it will be assumed, purely by way of example, that said control unit is designed to operate on twelve bits.  
         [0038]    The reference  24  designates a first block or module designed to execute a function of estimation of the values necessary for automatic white balancing (WB) on the data coming from the sensor  10 .  
         [0039]    Considering that usually there do not occur very significant variations in brightness between successive frames, the module  24  performs the corresponding estimation not on all the images, but rather on just one image out of K images (where K is, for example,  8 ).  
         [0040]    The data corresponding to the video images are then treated in a module  26  that carries out in just one step the functions of scaling (by a scaling factor SF fixed externally) and interpolation.  
         [0041]    In a currently preferred embodiment of the invention, the range of possible variation of the scaling factors is usually quite small, there being envisaged, for example, three possible factors equal to 3, 4 and 5.  
         [0042]    The colors of the image that have undergone scaling are then subjected to correction (by adapting the corresponding spectrum to the sensitivity of the human eye) with a matrixing operation performed in a module  28  according to the coefficients received from the estimation block  24 .  
         [0043]    Since the operation performed by the module  28  is in itself a function of a point type, like the WB estimation, it is possible to combine the two operations in a single step.  
         [0044]    The reference  30  designates the function of range compensation performed (according to criteria in themselves known) in order to adapt the colors to the characteristics of response of the display driven by the unit  22 . The range-compensation function is performed according to of the data contained in a look-up table (LUT) designated by the reference  34 .  
         [0045]    Finally, a module designated by  32  reduces the color planes with an effect that is antagonistic to formation of false boundaries.  
         [0046]    Just to provide an example (of course without thereby limiting in any way the scope of the invention) the data at output from the module  28  are usually organized on 24 bit, this format being preserved also after the range-compensation function has been executed. The module  32 , by reducing the chromatic planes, reduces, instead, the format to 12 bits, a value that is used by the control unit  22 . The final number of bits is not in any way limiting and it can be applied for a reduction of the bits in general.  
         [0047]    Passing on to a more detailed examination of the characteristics of the various blocks represented in FIG. 3, it will be recalled in the first place that the role of the white-balancing function (module  24 ) is to compensate possible dominant hues (casts) linked to problems of illumination of the image. The purpose is to eliminate any possible undesired casts in the case of images taken under conditions of illumination with light different from white light.  
         [0048]    As is illustrated in greater detail in FIG. 4, the solution described herein envisages splitting the Bayer pattern into blocks of four pixels (each containing one pixel R, one pixel B and two pixels G) and considering for processing purposes only one pixel for each channel. This means that one of the two pixels G, i.e., the pixels G designated by X in FIG. 4, is not considered.  
         [0049]    Of course the symbols R, B, and G correspond, respectively, to red, blue and green.  
         [0050]    In a preferred way, the functions outlined in what follows are carried out.  
         [0051]    For each pixel in each image block:  
         [0052]    the channel energies (ces) are calculated, ignoring the clipped components (i.e., the components greater than a pre-set saturation point) which could represent erroneous data in the white-balancing operation, by multiplying the energy of the pixels by the minimum pixel energy (min_pwe) amongst the pixels considered in the block and then by accumulating the channel energies (ces):  
           ces=Acc (related —   pwe.min   —   pwe );  
         [0053]    there are then calculated normalized channel energies (nces) for the pixels of each channel with respect to the maximum energy of the channel (max_ce):  
         
       nce=max 
       — 
       ce/related 
       — 
       ce  
     
         [0054]    next, the channel gains are set equal to the normalized channel energies (nces):  
         gains=related_nce;  
         [0055]    finally, the channel gains are applied to all the channel pixels:  
         pixels_value=pixels_value.related_gain.  
         [0056]    In order to accelerate execution of the procedure, since the illumination is normally distributed uniformly on the image, the treatment is carried out by sub-sampling blocks of four pixels in the vertical and horizontal directions.  
         [0057]    Considering the data pattern, the step of sampling in terms of number of pixels must be equal to (2N).  
         [0058]    Represented in FIG. 5 are the blocks of an image used for gain estimation. Considering the sub-sampling referred to above, between one block and the next other pixels are present which are not considered for the purpose of the specific computation. In particular, between one block and the next (whether in the horizontal direction or in the vertical direction) 2N pixels are discarded.  
         [0059]    The reason for resorting to an even number of pixels is to consider a block having the position of the pixel G above the blue and to the left of the red. Only if an even displacement on the Bayer pattern is considered is the same arrangement of the pixels obtained.  
         [0060]    In addition, since it is reasonable to expect that in the majority of the video sequences considered the conditions of illumination will vary gradually in passing from one frame to another, the channel gains are updated only once in a while, for example, once every K frames.  
         [0061]    Assuming a sequence of M frames of size H×V, if the two techniques of acceleration are combined, only (M/K)×(H×V)/(16×N 2 )] frames are considered.  
         [0062]    In the above formula, K indicates every so many frames that the gains are recalculated, N is the sampling step (which is of the 2N type), and M is the number of frames. As regards the formula, it may be noted that, in the case of absence of sampling we have (H×V)/(2×2) blocks, whereas if the frames are sampled their number is equal to ((H×V)/(2×2))/(2×N×2×N)=(H×V)/(16×N×N), a quantity that is finally to be multiplied by the fraction (M/K) of frames considered.  
         [0063]    The purpose of the function of scaling and color interpolation implemented in the module  26  is to create a colored image starting from the data coming from the sensor  10 .  
         [0064]    During color interpolation a processing aimed at a subset of pixels is performed, thus making it possible to obtain in a single step a scaled image in RGB format.  
         [0065]    In order to treat different display resolutions, different scaling factors are considered. If the scaling factors are varied, it is necessary to change also the size of the kernel of the interpolation operator.  
         [0066]    With reference to a matrix structure of the Bayer type, highlighted in FIG. 6, in the case where the scaling to be carried out is by a factor  3 , from each quadrant Q i  (i=1 . . . 4), a set of three values R G B is obtained, by obtaining from a set of 9 elements a single output pixel and performing a scaling by a factor  3  both in the horizontal and in the vertical directions.  
         [0067]    The above figure highlights all the possible cases of arrangement of the intensity of the chromatic channels in the case of scaling equal to 3.  
         [0068]    The formulae for scaling and simultaneously interpolating the missing channels for the four target pixels (each deriving from a quadrant Q i ) are the following (see FIG. 6):  
         [0069]    For Q1 (the central pixel is red):  
         Blue1=( A 1+ C 1+ A 3+ C 3)/4, Green1=( B 1+ A 2 +C 2 +B 3)/4, Red1 =B 2  
         [0070]    For Q2 (the central pixel is green, case A):  
         Green2=( D 1 +F 1 +D 3 +F 3)/4, Blue2=( E 1 +E 3)/2, Red2=( D 2 +F 2)/2  
         [0071]    For Q3 (the central pixel is green, case B):  
         Green3=( A 1 +C 1 +A 6 +C 6)/4, Blue3=( A 5 +C 5)/2, Red3=( B 4 +B 6)/2  
         [0072]    For Q4 (the central pixel is blue):  
         Red4=( D 4 +F 4 +D 6 +F 6)/4, Green4=( E 4+ D 5 +F 5 +E 6)/4, Blue4= E 5.  
         [0073]    For each of these four cases the result is given by a pixel in the RGB format. FIG. 7 represents the quadrants with the three components of color reconstructed according to the central pixel.  
         [0074]    In actual fact only one pixel per quadrant is supplied at output.  
         [0075]    For the case of scaling by a factor  4  (for example to pass from VGA to 160×20) and the case of scaling by a factor  5  (for example, from VGA to 128×96) basically similar solutions are used, but with kernels of size 5×5.  
         [0076]    Passing to the modules  28  (and  24 ) it will be recalled that the role of the color matrix is to improve color rendering and saturation in the image. In particular, the purpose is to correct the spectral sensitivity of the image sensor in a coordinated way with the chromatic characteristics of the display, taking into account the characteristics of the human eye.  
         [0077]    This result is obtained by manipulating the coefficients of the table of the color matrix to be applied to each RGB colored pixel.  
         [0078]    Reasoning in deliberately simplified terms, but without departing from reality, the operation in question is a simple matrix product between the channels of the input colors of a correction matrix, the coefficients of which are fixed with reference to the specific characteristics of the display.  
         [0079]    The solution illustrated herein combines the correction/white balancing (module  24 ) and the aforesaid matrixing operation (module  28 ) in a single step, multiplying the matrix applied to the module  28  by a diagonal matrix. The elements of a diagonal matrix are simply the channel gains calculated as described previously.  
         [0080]    The execution of this combined operation is illustrated in FIG. 8, where WB Matrix designates precisely the aforesaid diagonal matrix, and Matrixing Matrix designates the matrix the coefficients of which take into account the characteristics of the display. The resulting matrix is designated by Combined Matrix.  
         [0081]    In principle, the values coming from the module  24  should undergo clipping (in the output field) prior to being subjected to matrixing. However, possible errors are directly masked by the subsequent chromatic quantization.  
         [0082]    The function of range correction (compensation) implemented by the module  30  corresponds simply to the application of the values of the look-up table  34 , in a situation in which the i-th element of the table contains the value.  
         255×( i/ 255) i/{overscore (a)}   
         [0083]    As regards quantization (or truncation), reference may be made to a function that applies an improved grey scale (IGS).  
         [0084]    If a simple quantization of a channel from 8 bits to 4 bits is considered, the reduced information thus obtained can generate a certain number of false boundaries and hence a substantially worse image.  
         [0085]    The solution described herein envisages however avoiding a brutal truncation of the values so as to prevent formation of new edges in the image.  
         [0086]    Prior to truncation, added to each value is a factor of pseudo-random noise. In this way, it is possible to obtain that pixels that are close to one another having the same value will end up having slightly different final values.  
         [0087]    The noise is determined by the least significant bits of the sum of the value of the pixels and the noise estimated previously. The tests conducted by the present applicant show that the solution described herein enables images of high quality to be obtained reducing considerably the computational burden and the implementation burden as a whole.  
         [0088]    In summary, the main advantages of the solution described herein are linked to the factors outlined in what follows.  
         [0089]    The gain values for white balancing (WB) are determined on the basis of a sub-sampled sensor output. Preferably, the sub-sampling factor is even. The gain estimation is not performed for each frame, but rather once every K frames. When no new values are calculated, the white-balancing block WB uses the last saved value. The functions of scaling and interpolation of the color are implemented in a single step. Likewise, white balancing WB is applied simultaneously to calculation of the color matrix. In general, executing a redundant processing with respect to the pre-set qualitative objectives is avoided.  
         [0090]    Of course, without prejudice to the principle of the invention, the details of implementation and the embodiments may vary widely with respect to what is described and illustrated herein, without thereby departing from the scope of the present invention. For example, it may be envisaged that only part of the processing operations illustrated herein as being performed in a single step is implemented according to this modality. Accordingly, the invention is not limited except as by the appended claims and the equivalents thereof.  
         [0091]    All of the above U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and/or listed in the Application Data Sheet are incorporated herein by reference, in their entirety.