Abstract:
A method for improving the perception of an image may include performing a main separation of the pixels of the image into two categories, one corresponding to pixels of a flat zone, and the other corresponding to pixels of a textured zone. The method may also include processing the pixels of each category according to a method optimized according to the type of zone. Before the main separation step, a preliminary separation of the pixels may be performed into one category of normal pixels intended for the main separation step, and one category of singular pixels, with the criterion for selecting the singular pixels being adapted to identify pixels that would be wrongly identified as pixels of a textured zone. The singular pixels may then be processed according to a method adapted to their nature.

Description:
FIELD OF THE INVENTION 
       [0001]    The present disclosure relates to image processing for improving visual perception, and, more particularly, to image processing performed within embedded systems, such as digital cameras. 
       BACKGROUND OF THE INVENTION 
       [0002]    Among common processes used to improve the visual perception of an image, noise reduction, sharpness improvement and improvement of the balance of brightness and contrast are included. The quality and the usefulness of each of these processes depend on the properties of the image in the vicinity of each pixel processed. Therefore, some zones of the image may be differentiated so as to select the most appropriate process for each zone. 
         [0003]      FIG. 1  shows an image comprising two types of zones where it is desirable to process each zone differently: a flat zone  10 , such as a zone depicting the sky, and a textured zone  12 , such as a zone depicting leaves. In a flat zone, noise reduction is favored, while in a textured zone, sharpness, brightness, and contrast balancing are favored. Noise reduction techniques suited for textured zones can also be used, such as described in U.S. Patent Application Publication No. 2009/0141978 to Roffet et al. 
         [0004]    Common image processes, i.e. at least the relatively simple ones preferably used in embedded systems, are based on a reference kernel defined around each pixel to be processed.  FIG. 2  represents such a kernel. It is a 5×5-pixel square of the image, in the center of which is a pixel to be processed. This kernel size is common and offers a good complexity compromise. 
         [0005]      FIG. 3  represents a reference kernel  14  processing stream, enabling differentiated processing according to the nature of the zone in which the pixel being processed is located. In block  16 , a criterion representative of the pixel belonging to a flat zone ( 10  in  FIG. 1 ) is calculated from the kernel. If the criterion is not met, the pixel is considered to belong to a textured zone ( 12 ), and the kernel  14  undergoes, in block  18 , a process adapted to the textured zones. If the criterion is met, the kernel undergoes, in block  20 , a process adapted to flat zones. The process in blocks  18  or  20  produces a single pixel Px out , replacing the current pixel in the new image. 
         [0006]    One example of selection criterion can be as follows. As represented in  FIG. 2 , for each pixel i in the reference kernel, the difference d i  between this pixel and the central pixel, which is the pixel to be processed, is calculated. In the case of a 5×5 kernel, 24 differences d 1  to d 24  are obtained, with the zero difference for the central pixel not being counted. The differences are calculated in terms of luminance. 
         [0007]    It is then considered that the pixel to be processed belongs to a flat zone if the number of differences below a luminance threshold is close to the number of pixels in the kernel (e.g. at least equal to 22 in the case of a 5×5 kernel). 
         [0008]    Occasionally, such algorithms do not classify certain pixels correctly. For example, if the pixels in the kernel are all similar except for the central pixel, all the differences have a high value, such that the criterion above classifies the pixel as not belonging to a flat zone, whereas it is manifestly an incorrect pixel in a flat zone. The pixel is thus classified as belonging to a textured zone. The textured zone then undergoes processes such as sharpness improvement, which may result in amplifying the singularity of the wrongly classified pixel, making it even more visible. 
         [0009]    More complex algorithms would enable this specific example to be identified and to potentially correct it. However, it should be recalled that, within the framework of embedded systems, the complexity of the algorithms may need to be limited. Furthermore, only one striking singular case is mentioned above, but there are more subtle cases in which the pixels are wrongly classified and also generate unpleasant visual artifacts. 
         [0010]      FIG. 4  shows a more detailed example of a processing stream  18  of the pixels classified as belonging to a textured zone. A RAM memory is used as a temporary working space and stores parts of the image being processed. A non-volatile memory NVM stores the original images and receives the images after processing. A first process NR includes, for example, an adaptive noise reduction optimized for textured zones, as disclosed in U.S. Patent Application Publication No. 2009/0141978 to Roffet et al. This process works on reference kernels  14  read in the original image, for example, 5×5-pixel kernels containing the pixel to be processed in the center. The pixels to be processed have been identified by the decision step  16  in  FIG. 3 . The process calculates a single pixel from the kernel, with the pixel being stored in the RAM memory. 
         [0011]    The next step SHARP is, for example, an increase in the sharpness. This process also works on a reference kernel,  14 . 2 , which may differ in size to the kernel  14 . The kernel  14 . 2  is read in the RAM memory when the process NR has produced the pixels defining it. The process SHARP calculates a single pixel from the kernel  14 . 2 , with the pixel being written in the RAM memory. 
         [0012]    The next step USM comprises, for example, an increase in the acutance, which includes an increase in the local contrast and an accentuation in the transitions between two adjacent colors. This technique is commonly designated by the acronym USM (UnSharp Mask). Like the previous processes, the USM process works on a reference kernel  14 . 3 , read in the RAM memory when the previous process has produced all the pixels of this kernel. The USM process produces from the kernel  14 . 3  a single pixel defining the final processed image, which is written, for example, in the non-volatile memory NVM. 
         [0013]    It can be seen that each process works on pixels defining an intermediate image produced by the previous process. With three processes, two intermediate images are used, the necessary zones of which are stored in the RAM memory. This RAM memory is generally a fast-access memory so as not to slow down the processing. It is generally not possible to store the intermediate images in the non-volatile memory NVM, as access to such memories is too slow. 
       SUMMARY OF THE INVENTION 
       [0014]    The object is to have a processing method differentiated by zones that reduces or does not generate any visible artifacts, while being less complex. 
         [0015]    The present disclosure discloses an approach including a method for improving the perception of an image comprising performing a main separation of the pixels of the image into two categories, one corresponding to pixels of a flat zone, and the other corresponding to pixels of a textured zone. The method may include processing the pixels of each category according to a method optimized according to the type of zone. Before the main separation step, a preliminary separation of the pixels may be performed into one category of normal pixels intended for the main separation step, and one category of singular pixels, the criterion for selecting the singular pixels being adapted to identify pixels that would be wrongly identified as pixels of a textured zone. The singular pixels may then processed according to a method adapted to their nature. 
         [0016]    According to one embodiment, the preliminary separation may comprise obtaining a reference kernel around the pixel being analyzed, and calculating the difference between each pixel in the reference kernel and the current pixel. The preliminary separation may comprise identifying the current pixel as a singular pixel if any one of the following conditions is met: the number of non-zero positive differences is close to the number of pixels in the kernel; the number of non-zero negative differences is close to the number of pixels in the kernel; the number of non-zero positive differences is close to the number of non-zero negative differences; and the luminance dynamics of the reference kernel are below a threshold. 
         [0017]    According to one embodiment, the processing of each singular pixel may comprise calculating four sums of differences, a first one equal to the sum of the differences corresponding to the north and south pixels in the reference kernel, a second one equal to the sum of the differences corresponding to the east and west pixels, a third one equal to the sum of the differences corresponding to the north-west and south-east pixels, and the fourth one equal to the sum of the differences corresponding to the north-east and south-west pixels. The processing may also include finding the minimum sum, and calculating the current pixel as the mean of pixels located on the axis of the pixels corresponding to the minimum sum. 
         [0018]    It may also be desirable to reduce the memory resources used when an original image, or a zone of the latter, undergoes a series of independent processes, with each producing a pixel calculated using a respective reference kernel. To that end, the reference kernels may all comprise pixels from the original image. 
         [0019]    According to one embodiment, the processes may be performed in parallel and a resulting pixel is produced as a combination of the respective pixels produced by the processes. According to one embodiment, the method may comprise replacing a key pixel in the reference kernel used for a current process with the pixel produced by the previous process. According to another embodiment, the processes may be performed on a textured image zone and comprise an adaptive noise reduction, a sharpness increase, and an improvement of the acutance using the unsharp masks. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0020]    Some embodiments will be explained in the following description, in relation with, but not limited to, the following figures, in which: 
           [0021]      FIG. 1  is a schematic diagram of an image comprising two zones, according to the prior art. 
           [0022]      FIG. 2  is a schematic diagram of a reference kernel for identifying a type of zone, according to the prior art. 
           [0023]      FIG. 3  is a schematic diagram of an example of a differentiated processing stream for processing two zones of an image in a differentiated manner, according to the prior art. 
           [0024]      FIG. 4  is a schematic diagram of a processing stream for processing a textured zone of an image, according to the prior art. 
           [0025]      FIG. 5  is a schematic diagram of a processing stream for processing an image based on a pre-identification of singular pixels, according to the present invention. 
           [0026]      FIG. 6  is a schematic diagram of one use of a reference kernel for processing the singular pixels, according to the present invention. 
           [0027]      FIG. 7A  is a schematic diagram of one embodiment of a processing stream for processing a textured zone, optimized to limit the memory resources, according to the present invention. 
           [0028]      FIG. 7B  is a schematic diagram of a second embodiment of an optimized processing stream, according to the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0029]      FIG. 5  illustrates a differentiated processing stream for processing some zones of an image, wherein singular pixels are pre-selected and are extracted from the main stream to undergo a specific process. The object is to extract from the main stream the pixels likely to be classified by error in a category tending to produce visual artifacts, particularly the category corresponding to a textured zone. An opposite classification error, i.e. comprising classifying a pixel as belonging to a flat zone when it belongs to a textured zone, may not have any noteworthy visual consequences, and the object here is not to identify such pixels. 
         [0030]    In practice, the pixels of the image are separated into three different categories, each one being subjected to an appropriate process. The main stream corresponds, for example, to the one in  FIG. 3 . Thus, in block  16 , the category of the pixel is determined according to a simple typical criterion and, depending on the result, it is directed to a process adapted to textures, in block  18 , or to flat zones, in block  20 . 
         [0031]    A decision or pre-selection step  50  is provided upstream from step  16 . The reference kernels  14  of the original image are compared with a singularity criterion. If the pixel is classified as singular, it is extracted from the normal stream and directed to an appropriate process in block  52 . Like the processes in blocks  18  and  20 , the process in block  52  produces a single pixel Px out  calculated from the reference kernel. It may be understood that, if the pixel is not classified as singular, it is directed to the normal stream, via the separation in block  16 . 
         [0032]    Generally speaking, the singularity criterion is developed to detect pixels that would be wrongly classified in step  16 , particularly pixels belonging to a flat zone which would be classified as belonging to a texture. The following example of a singularity criterion has the advantage of being simple. Like for the criterion for separating flat/textured zones described in relation with  FIG. 2 , the difference d i  between the pixel i and the central pixel is calculated for each pixel i in the reference kernel. These differences may then be used in step  16 , as applicable. 
         [0033]    A pixel is then considered singular if any one of the following conditions is met:
       1. the number of non-zero positive differences is close to the number of pixels in the kernel;   2. the number of non-zero negative differences is close to the number of pixels in the kernel;   3. the number of non-zero positive differences is close to the number of non-zero negative differences; and   4. the luminance dynamics of the reference kernel is of low amplitude.       
 
         [0038]    “Close” means a difference of a few units between the numbers, depending on the size of the kernel. For a 5×5 kernel, the numbers can be considered close if the difference is less than or equal to 2, i.e. on the order of 10% of the number of pixels in the kernel. The difference selected in each condition may be optimized by successive tests or statistical analyses. 
         [0039]    The fourth component can be expressed by d max −d min &lt;L, in which d max  and d min  are respectively the maximum difference and the minimum difference found in the reference kernel, and L is a luminance threshold, for example, on the order of 20% of the maximum dynamics. This component may enable contrasted regular pixels to be excluded from the image, such as light points on a dark surface or holes on a pale surface. Such pixels are appropriately processed in the normal stream, in which they may be classified as belonging to a texture. 
         [0040]    The process performed in block  52  could be the process reserved for flat zones in block  20 , generally a noise reduction by averaging calculated on the pixels in the kernel. A singular pixel could indeed be a pixel distorted by noise. 
         [0041]    However, the abovementioned criterion, particularly its third or fourth component, also enables pixels located on a transition between zones of different brightness to be detected, which would also be detected in block  16  as belonging to a texture. Such a detection in block  16  is not necessarily wrong, since a transition between two texture elements can be considered to belong to the texture. However, better results have been obtained by processing such pixels using directional averaging. 
         [0042]      FIG. 6  shows an example of using the pixels in the reference kernel to perform a simple directional averaging calculation. In the reference kernel, nine pixels are kept: the central pixel and the North N, South S, East E, West W, North-West NW, North-East NE, South-West SW and South-East SE pixels, i.e. a reduced reference kernel of 3×3 pixels is formed. The sum of the differences is calculated for each pair of opposite pixels in the reduced kernel, i.e. d 11 +d 14  (or d W +d E ), d 1 +d 24  (or d NW +d SE ), d 3 +d 22  (or d N +d S ), and d 5 +d 20  (or d NE +d SW ). The outgoing pixel Px out  is then calculated as the mean of the pixels corresponding to the minimum sum of the differences. For example, if the minimum sum is d N +d S , the following is obtained: 
         [0000]      Px out =(Px N +Px S )/2. 
         [0043]    According to one embodiment, the value of the central pixel Px c  is used: 
         [0000]      Px out =(Px N +Px S +Px c )/3. 
         [0000]    According to another embodiment, the mean of two pixels, or of all the pixels, located on the axis of the two pixels corresponding to the minimum sum is calculated. 
         [0044]    As mentioned above in relation with  FIG. 4 , when a process comprises the successive application of several independent algorithms, which is common for processing textured zones, each algorithm typically stores in a volatile memory an intermediate image portion, used by the following algorithm. It may be desirable to try to reduce the size of the volatile memory, particularly in embedded systems. This reduces the surface area of the circuits and power consumption. 
         [0045]      FIG. 7A  illustrates one embodiment of a processing stream enabling the size of the volatile memory to be reduced. The principle underlying this stream is to use reference kernels for the successive algorithms whose pixels are taken from the original image, and to only handle in each step the pixel produced by the previous step. 
         [0046]    For example, in  FIG. 7A , for each pixel to be processed, a reference kernel  14  is read in the memory MEM containing the original image. This kernel is processed typically by the first algorithm NR, such as an adaptive noise reduction algorithm, which produces a first intermediate pixel Px 2 . A reference kernel  14 . 2 ′ is regenerated by taking the kernel  14  and replacing its key pixel, generally the central pixel, with the pixel Px 2 . This kernel  14 . 2 ′ supplies the next algorithm SHARP, for example, a sharpness improvement, which produces a new intermediate pixel Px 3 . A new reference kernel  14 . 3 ′ is regenerated, for the next algorithm USM, by taking the original kernel  14  and by replacing its key pixel with the pixel Px 3 . 
         [0047]    In this way, only the reference kernel  14  is stored in a fast memory, with this kernel being reused by each algorithm in the processing chain. The size of the reference kernel may depend on the algorithm used. Whatever its size, its pixels are all taken from the original image, which makes it easier to manage kernels of different sizes. 
         [0048]    The processing time is furthermore significantly accelerated, as an algorithm in the chain no longer needs to wait for the previous algorithm to produce all the pixels of its reference kernel. This reference kernel is ready as soon as its key pixel is produced by the previous algorithm. 
         [0049]    It may be understood that, from the second algorithm in the chain onwards, the algorithms do not act on the same pixels as in the classic stream in  FIG. 4 . Therefore, a different resulting image is expected. It happens that the resulting image, even if it is indeed different in some cases, provides a satisfactory, or better, visual perception. 
         [0050]      FIG. 7B  illustrates a second embodiment of a processing stream enabling the size of the volatile memory to be reduced. Compared to  FIG. 7A , instead of including the outgoing pixel of an algorithm in the center of the reference kernel for the next algorithm, each algorithm is used in parallel on a kernel fully formed of pixels from the original image, including the key pixel. The pixels produced independently by the algorithms are combined by a process or circuit WAVG to produce the outgoing pixel Px out . This combination may be a weighted sum. As part of a usual process comprising a noise reduction (NR), an increase in sharpness (SHARP), and an improvement in the acutance (USM), respective values of 1, 1/32 and 1/64 are used as an example for the weighting coefficients. The end result is different from the one produced by the streams in  FIGS. 4 and 7A , but the visual perception of the image is also satisfactory.