Abstract:
An image processing apparatus comprises an extractor for extracting a brightness component from image data; a scale converter for obtaining a distribution of the brightness component on relatively large scale; a brightness component improver for improving the distribution of the brightness component of the image data by using the brightness component and an output of the converter; and an image reproducer for reproducing the image data by using an output of the improver as a distribution of a brightness component of a new image, wherein the apparatus further comprises a face detector for detecting a face area from the image data, and the improver adjusts a degree of improvement by using a distribution of a brightness component of the face area being an output of the detector. Thus, it is possible to automatically adjust the degree of improvement of the luminance distribution according to the image data to be processed.

Description:
[0001]    This application claims priority from Japanese Patent Application Nos. 2003-199119 filed on Jul. 18, 2003 and 2003-199122 filed on Jul. 18, 2003 which are hereby incorporated by reference. 
       BACKGROUND OF THE INVENTION 
       [0002]    1. Field of the Invention 
         [0003]    The present invention relates to image processing apparatus and method which improve a distribution of brightness of a digital image. 
         [0004]    2. Related Background Art 
         [0005]    Conventionally, as a method of taking a photograph of appropriate brightness, a method of measuring average luminance of a scene to be taken or photographed and then controlling shutter speed, an aperture value and the like based on the measured average luminance is known. Moreover, an exposure control method based on so-called evaluation photometry that a scene is first divided into predetermined areas, luminance of each of the divided areas is measured, the measured luminance is appropriately weighted with respect to each area, average luminance is then obtained based on the weighted values, and appropriate exposure is thus obtained with respect to the entire scene is known. 
         [0006]    Incidentally, in a scene taken against light (also called a backlight scene) that the brightness of a main subject is remarkably dark as compared with that of its background, the main subject portion on a taken or photographed image is inevitably dark. Therefore, to take a photograph in which appropriate brightness can be secured even in case of a backlight scene, it is necessary at the moment of taking the photograph to preset exposure of a camera so that the subject is taken brightly as compared with a case of taking an average photograph. However, such an exposure correction operation is troublesome for a user, and moreover skill is required to appropriately set the exposure of the camera. Besides, in a digital image process to the image obtained by capturing the backlight scene, a process equivalent to the exposure correction operation can be performed. However, in any case, even if the exposure correction is appropriately performed with respect to the main subject, the background portion of the main subject adversely tends to become bright excessively. 
         [0007]    In order to solve such a problem, it is necessary to obtain an image of appropriate brightness even in the backlight scene or the like that it is generally difficult to appropriately determine the brightness of the image. 
         [0008]    Thus, to achieve this, in analog photographing technique, print of appropriate brightness can be obtained by performing so-called a dodging process in a darkroom. Therefore, it is desirable even in a digital image process to easily achieve a dodging process similar to that in the analog photographing technique. 
         [0009]    For example, a method of achieving the dodging process is proposed by Daniel J. Jobson et al. in “A Multiscale Retinex for Bridging the Gap Between Color Images and the Human Observation of Scenes”, IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 7, July 1997 (hereinafter called prior art 1). In this method, a difference process between a component obtained by logarithmic transforming a digital image and a low-frequency component of the logarithmic-transformed component is performed to darken a bright component and brighten a dark component in a low-frequency area in the digital image, thereby achieving image improvement. 
         [0010]    Besides, other methods of achieving the dodging process in the digital image process are proposed by Reinhard et al. in “Photographic Tone Reproduction for Digital Images”, acm Transactions on Graphics, July 2002, Vol. 21, No. 3 (hereinafter called prior art 2), and Moroney “Local Color Correction Using Non-Linear Masking”, IS&amp;T/SID Eighth Color Imaging Conference (hereinafter called prior art 3). In each of these methods, a dodging-like effect is obtained in the digital image process by using a luminance component of a digital image and a low-frequency component thereof. 
         [0011]    However, in such conventional methods, a degree of improvement in a case where large improvement is necessary with respect to the brightness of the main subject in the digital image to be processed is the same as that in a case where a little improvement is necessary with respect to the brightness of the main subject. Therefore, there is a problem that it is impossible to perform suitable improvement of the luminance distribution for the image data to be processed. This originates mainly in a fact that a purpose concerning how it is necessary to improve the luminance distribution in the image data to be processed is not clear. 
         [0012]    Moreover, conventionally, when the conventional dodging method is applied to a case where the scene obtained by photographing and capturing the digital image to be processed is not the backlight scene, there is a problem that, although an effect of the correction is inferior, a processing time is long and costs for a memory or the like used in the process are unnecessarily high as compared with the case where the exposure correction is normally performed by the digital image process. In addition, even in the backlight scene, the necessity of brightness correction is not high with respect to an image other than the image in which particularly a person&#39;s face has been recorded darkly due to backlight. 
       SUMMARY OF THE INVENTION 
       [0013]    An object of the present invention is to solve the above conventional problems, and furthermore to automatically adjust a degree of improvement of a brightness distribution with respect to image data particularly obtained by photographing a person in a backlight state, thereby enabling to improve the brightness distribution according to the image data to be processed. 
         [0014]    Another object of the present invention is to solve the above conventional problems, and furthermore to automatically adjust a degree of improvement of a luminance distribution particularly with respect to a backlight scene, i.e., image data obtained by photographing a person&#39;s face in a backlight state, thereby enabling to optimally improve a brightness distribution according to the image data to be processed and also enabling to perform a process at relatively low costs with respect to image data obtained by capturing a scene, other than the backlight scene, for which the process effect is predicted to be inferior. 
         [0015]    In order to achieve the above objects, the present invention is characterized by an image processing apparatus comprising: a brightness component extraction means for extracting a brightness component from image data; a scale conversion means for obtaining a distribution of the brightness component on relatively large scale; a brightness component improvement means for improving the distribution of the brightness component of the image data by using the brightness component and an output of the scale conversion means; and an image reproduction means for reproducing the image data by using an output of the brightness component improvement means as a distribution of a brightness component of a new image, wherein the image processing apparatus further comprises a face detection means for detecting a face area from the image data, and the brightness component improvement means adjusts a degree of improvement by using a distribution of a brightness component of the face area being an output of the face detection means. 
         [0016]    Further, the present invention is characterized by an image processing method comprising: an extraction step of extracting a low-frequency brightness component from image data of an input image; a detection step of detecting a face area in the input image; a setting step of setting a parameter according to a distribution of a brightness component of the face area; and a correction step of correcting a brightness component of the image data by using the low-frequency brightness component and the parameter. 
         [0017]    Moreover, the present invention is characterized by a program for achieving, by a computer, an image processing method comprising: an extraction step of extracting a low-frequency brightness component from image data of an input image; a detection step of detecting a face area in the input image; a setting step of setting a parameter according to a distribution of a brightness component of the face area; and a correction step of correcting a brightness component of the image data by using the low-frequency brightness component and the parameter. 
         [0018]    Furthermore, the present invention is characterized by an image processing method comprising: a first brightness component improvement step of improving a brightness component distribution of an input image by using a distribution of a brightness component of the input image on a predetermined scale; a detection step of detecting a predetermined image area by analyzing the image processed in the first brightness component improvement step; a second brightness component improvement step of improving the brightness component distribution of the input image by using a brightness component value of the predetermined image area detected in the detection step; and an output step of outputting the image processed in the second brightness component improvement step. 
         [0019]    Furthermore, the present invention is characterized by a program for achieving, by a computer, an image processing method comprising: a first brightness component improvement step of improving a brightness component distribution of an input image by using a distribution of a brightness component of the input image on a predetermined scale; a detection step of detecting a predetermined image area by analyzing the image processed in the first brightness component improvement step; a second brightness component improvement step of improving the brightness component distribution of the input image by using a brightness component value of the predetermined image area detected in the detection step; and an output step of outputting the image processed in the second brightness component improvement step. 
         [0020]    Furthermore, the present invention is characterized by an image processing apparatus comprising: a luminance extraction means for extracting a luminance component from image data; a scale conversion means for obtaining a distribution of the luminance component on relatively large scale; a luminance improvement means for improving a luminance distribution of the image data; and an image reproduction means for reproducing the image data by using an output of the luminance improvement means as a luminance distribution of a new image, wherein a process effect of the luminance improvement means in which an output of the scale conversion means is used is predicted, when it is predicted that the process effect is high, the luminance improvement means improves the luminance distribution of the image data by using the output of the scale conversion means, and, when it is predicted that the process effect is low, the luminance improvement means improves the luminance distribution of the image data without using the output of the scale conversion means. 
         [0021]    Furthermore, the present invention is characterized by an image processing method comprising: a luminance extraction step of extracting a luminance component from image data; a scale conversion step of obtaining a distribution of the luminance component on relatively large scale; a luminance improvement step of improving a luminance distribution of the image data; and an image reproduction step of reproducing the image data by using an output in the luminance improvement step as a luminance distribution of a new image, wherein a process effect in the luminance improvement step in which an output in the scale conversion step is used is predicted, when it is predicted that the process effect is high, the luminance improvement step improves the luminance distribution of the image data by using the output in the scale conversion step, and, when it is predicted that the process effect is low, the luminance improvement step improves the luminance distribution of the image data without using the output in the scale conversion step. 
         [0022]    Furthermore, the present invention is characterized by a program for achieving, by a computer, an image processing method comprising: a luminance extraction step of extracting a luminance component from image data; a scale conversion step of obtaining a distribution of the luminance component on relatively large scale; a luminance improvement step of improving a luminance distribution of the image data; and an image reproduction step of reproducing the image data by using an output in the luminance improvement step as a luminance distribution of a new image, wherein a process effect in the luminance improvement step in which an output in the scale conversion step is used is predicted, when it is predicted that the process effect is high, the luminance improvement step improves the luminance distribution of the image data by using the output in the scale conversion step, and, when it is predicted that the process effect is low, the luminance improvement step improves the luminance distribution of the image data without using the output in the scale conversion step. 
         [0023]    Other object and features of the present invention will be apparent from the following description in conjunction with the accompanying drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0024]      FIG. 1  is a block diagram showing the structure of an image processing system according to the first embodiment of the present invention; 
           [0025]      FIG. 2  is a flow chart showing an algorithm of an application program in the image processing system; 
           [0026]      FIGS. 3A ,  3 B and  3 C are diagrams for respectively explaining process contents in face detection according to the first embodiment; 
           [0027]      FIG. 4  is a block diagram showing the structure of an image processing system according to the second embodiment of the present invention; and 
           [0028]      FIG. 5  is a flow chart showing an algorithm of an application program in the image processing system. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     First Embodiment 
       [0029]      FIG. 1  is a block diagram showing the structure of an image processing system according to the first embodiment of the present invention. 
         [0030]    In  FIG. 1 , reference numeral  1  denotes an image input means for inputting digital image data (hereinafter, called image data) into the image processing system. For example, the input means is constituted by a digital camera, a scanner or the like. 
         [0031]    Reference numeral  2  denotes a luminance extraction means for extracting a luminance component and a color component from the image data input by the image input means  1 . 
         [0032]    Reference numeral  3  denotes a scale conversion means for obtaining a distribution, on relatively large scale, of the luminance components of the image data output from the luminance extraction means  2 . 
         [0033]    Reference numeral  4  denotes a luminance improvement means for improving a distribution of the luminance components of the image data by use of the luminance component of the image data output from the luminance extraction means  2  and the distribution, on relatively large scale, of the luminance components output from the scale conversion means  3 . 
         [0034]    Reference numeral  5  denotes an image reproduction means for reconstituting the image data by synthesizing the improved luminance component output from the luminance improvement means  4  and the color component output from the luminance extraction means  2  together. 
         [0035]    Reference numeral  6  denotes a parameter adjustment means for adjusting parameters so as to optimize, according to the image data, a degree of improvement to be processed in the luminance improvement means  4  on the basis of the luminance component of the image data output from the luminance extraction means  2 . 
         [0036]    Reference numeral  7  denotes a face detection means for detecting, as a face area, a portion corresponding to a person&#39;s face from the image data output from the image reproduction means  5 . 
         [0037]    The above image processing system can be operated by an application program functioning on a general-purpose computer. Hereinafter, in the present embodiment, the image processing system which is operated mainly by the application program will be explained. 
         [0038]      FIG. 2  shows an algorithm of the application program by which the image processing system according to the present embodiment is operated on the general-purpose computer. 
         [0039]    First, when the application program is activated, a user inputs a file name of the image data, and the image data and the photographing data are read and stored in a storage unit of the computer (step S 101 ). 
         [0040]    Here, the read image data corresponds to a two-dimensional arrangement of M×N pixels (here, M is the number of horizontal pixels and N is the number of vertical pixels) composed of, e.g., eight-bit pixels, and the image data is composed of three planes of R, G and B components. The image data is assumed to be expressed by the R, G and B components as R(x, y), G(x, y) and B(x, y) (here, (x, y) are integers representing a pixel position; 1≦x≦M, and 1≦y≦N). In a case where the image data is compressed by a JPEG (Joint Photographic Experts Group) method or the like, the image data is expanded in accordance with a predetermined expansion (or extraction) method, and the image data composed of the pixels of the R, G and B components is obtained. 
         [0041]    Next, the luminance component is extracted on the basis of each pixel of the R, G and B components composing the image data (step S 102 ). 
         [0042]    With respect to extraction of the luminance component, for example, it is assumed that the pixel components of R, G and B correspond to data in an sRGB color space described in IEC (International Electro-technical Commission) 61966-2-1, and the data is converted into data of CIE (Commission Internationale de i&#39;Eclairage) 1931 XYZ by gamma conversion and a 3×3 matrix operation in accordance with a method described in the IEC 61966-2-1. Here, when the converted data of XYZ are respectively assumed as X(x, y), Y(x, y) and Z(x, y), data of Y(x, y) corresponds to the luminance component to be extracted. When the luminance extraction is achieved by hardware, for example, it can be achieved by a table reference circuit (section of gamma conversion) according to a look-up table and a matrix operation circuit. 
         [0043]    As a method of extracting the luminance component, the above process may be simplified, that is, the luminance component may be extracted only by the matrix operation. Further, conversion from RGB into YCbCr, RGB into L*a*b*, or RGB into HSV may be used. 
         [0044]    Next, the distribution of the luminance component on relatively large scale is obtained from the extracted luminance component (step S 103 ). 
         [0045]    In order to obtain the distribution of the luminance component on relatively large scale (distribution of the low frequency luminance component), for example, as shown in the prior art 1, a convolution operation is performed with respect to the extracted luminance component and a Gaussian function to supply an output. However, in the prior art 1, the convolution operation is performed with respect to each pixel of the R, G and B components of the improved image data instead of the luminance component of the image data. Here, to raise image quality of the improved image data, it is preferable that the convolution operation with the plural Gaussian functions different in the standard deviation is performed to obtain the distribution of the luminance component on plural scales. It should be noted that the above process of obtaining the distribution of the luminance component on large scale is called scale conversion hereinafter. In case of achieving the scale conversion by hardware, e.g., a convolution operation circuit. 
         [0046]    Next, the distribution of the luminance component of the image data is improved using the luminance component of the image data and the distribution of the luminance component to which the scale conversion was performed (step S 104 ). 
         [0047]    As an example of a process, according to a method based on the prior art 1, logarithmic transformation is performed for the luminance component and the distribution of the luminance component to which the scale conversion was performed, and the obtained difference between them is output. Further, a weighted average of difference outputs on different scales is assumed as the improved luminance component. However, since the degree of improvement cannot be adjusted according to an image by this method, a logarithmic transformation output of the luminance component, to which the scale conversion was performed, is multiplied by a coefficient. This coefficient corresponds to a parameter of adjusting the degree of improvement. An output of the luminance component which was improved based on the above-explained process is indicated by the following equation (1). 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       Y 
                       ′ 
                     
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∑ 
                       n 
                     
                      
                     
                       
                         w 
                         n 
                       
                        
                       
                         { 
                         
                           
                             log 
                              
                             
                                 
                             
                              
                             
                               Y 
                                
                               
                                 ( 
                                 
                                   x 
                                   , 
                                   y 
                                 
                                 ) 
                               
                             
                           
                           - 
                           
                             γ 
                             · 
                             
                               log 
                                
                               
                                 [ 
                                 
                                   
                                     
                                       F 
                                       n 
                                     
                                      
                                     
                                       ( 
                                       
                                         x 
                                         , 
                                         y 
                                       
                                       ) 
                                     
                                   
                                   * 
                                   
                                     Y 
                                      
                                     
                                       ( 
                                       
                                         x 
                                         , 
                                         y 
                                       
                                       ) 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                         
                         } 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0048]    Here, Y′(x, y), F n (x, y), w n , n and y are respectively an output of the improved luminance component, a Gaussian function, weight between scales, a parameter representing the scale and a parameter representing the degree of improvement. Further, a symbol “*” denotes the convolution operation. 
         [0049]    It has been known that the weight between scales can be omitted by adjusting a standard deviation of the scale (replaced to a simple average) and it is preferable to return a luminance unit to an original luminance unit by an inverse transformation (exp. operation) rather than a value logarithmic-transformed is output as in the equation (1) as image quality of the improved image data. Therefore, it is preferable that an output represented by the following equation (2) corresponds to the improved luminance component. 
         [0000]        Y ′( x,y )=exp         log  Y ( x,y )−γ·Avg{log [ F   n ( x,y )* Y ( x,y )]}           (2) 
         [0050]    Here, the Avg indicates an average value operation. The following equation (3) may be available instead of the equation (2). 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       Y 
                       ′ 
                     
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         Y 
                          
                         
                           ( 
                           
                             x 
                             , 
                             y 
                           
                           ) 
                         
                       
                       
                         
                           [ 
                           
                             Avg 
                              
                             
                               { 
                               
                                 
                                   
                                     F 
                                     n 
                                   
                                    
                                   
                                     ( 
                                     
                                       x 
                                       , 
                                       y 
                                     
                                     ) 
                                   
                                 
                                 * 
                                 
                                   Y 
                                    
                                   
                                     ( 
                                     
                                       x 
                                       , 
                                       y 
                                     
                                     ) 
                                   
                                 
                               
                               } 
                             
                           
                           ] 
                         
                         γ 
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
         [0051]    Incidentally, the luminance improvement process in the step S 104  is a preprocess which extracts a face area with a high degree of accuracy in a step S 106 , whereby the intensified luminance improvement process is performed here. In the present embodiment, a predetermined value γ=1 is used. 
         [0052]    It should be noted that the average value operation of a scale conversion output on the plural scales is performed by the scale conversion process in the step S 103 , and an average value of the scale conversion output on the plural scales may be the distribution of the luminance component to which the scale conversion was performed. 
         [0053]    In case of achieving this luminance conversion by the hardware, for example, it can be achieved by an average value operation circuit, a circuit for forming the look-up table, a table storage unit, the table reference circuit (section of gamma conversion) and a division circuit. The average value operation circuit may be provided in the scale conversion means. 
         [0054]    Next, the improved luminance component and the color components X(x, y) and Z(x, y) converted in the step S 102  are synthesized to reconstitute the image data (step S 105 ). 
         [0055]    Here, initially, the color component is corrected in accordance with the change of the luminance component so that color of the reconstituted image data is not changed as much as possible. For example, the color components X(x, y) and Z(x, y) are respectively multiplied by a ratio between before and after changing the luminance component Y′(x, y)/Y(x, y). Then, the RGB data is obtained from X, Y and Z data. A process executed here corresponds to inverse conversion of the process in the step S 102 . Therefore, the 3×3 matrix operation and an inverse gamma conversion process are executed to output data of eight-bit for each of the R, G and B components. In case of achieving the reconstitution of the image data by hardware, for example, a multiplication/division circuit, the matrix operation circuit, and the table reference circuit (section of inverse gamma conversion) depending on the look-up table can be used. 
         [0056]    In case of using another method such as a conversion from the RGB into the YCbCr in the step S 102  as a method of extracting the luminance component, it is needless to say that a corresponding inverse conversion process has to be executed in this process. 
         [0057]    Next, the location corresponding to a person&#39;s face is detected as the face area from the image data in which the luminance distribution has been improved (step S 106 ). 
         [0058]    An example of process content for detecting the face area from the image data is shown in  FIGS. 3A to 3C . First, the image data is subjected to a low-pass filtering process of predetermined image size and is simultaneously reduced. Then, areas where the R, G and B components are locally small are extracted as candidates of the image areas of eyes on the basis of the pixel value of the reduced image data, and candidates sets each consisting of the extracted two candidates are made. Next, on the basis of equivalency of sizes, a difference in luminance, angles from the horizontal and the like between the two areas in each candidate set, it is judged whether or not the two areas in the candidate set represent eyes ( FIG. 3A ). Subsequently, with respect to the candidate set judged as eyes, a rectangular area is set from such two locations based on preset parameters concerning predetermined location relations ( FIG. 3B ), and the rectangular area is decided based on the edge in the vicinity of the border of the rectangular area and color information within this area ( FIG. 3C ). More specifically, with respect to the edge, it is judged whether or not edge components exceeding predetermined intensity occupies at a predetermined ratio or more in a predetermined width area in the vicinity of the border. Besides, with respect to the color information, an average value of the pixel values within the rectangular area is calculated, and it is then judged whether or not the calculated average value is in a preset skin color area. When both the edge and the color information satisfy respective conditions, the area in question is judged as the face area, and the position information of the rectangular area judged as the face area is output as the face area. 
         [0059]    In the present invention, the method of detecting the face area from the image data is not limited to the above. That is, for example, a method proposed by Yang et al. in “Detecting Faces In Images: A Survey”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, January 2002 may be applied. 
         [0060]    Next, a parameter for determining a degree of a luminance distribution to be improved is adjusted from the luminance component extracted in the step S 102  and the face area detected in the step S 106  (step S 107 ). 
         [0061]    Hereinafter, an example of a parameter adjusting method will be explained as to a case of performing the luminance conversion by the equation (3) in the step S 104 . 
         [0062]    First, in the luminance component of the image data, an average value of the luminance values within the face area is calculated. Here, it should be noted that matters such as eyes, hairs and the like which are not skin either exist within the rectangular area detected as the face area. Therefore, the average value of only the luminance values of predetermined percentage or more from the highest luminance value is calculated with respect to the pixels within the face area, and the calculated average value is assumed as a representative value of the luminance of the portion corresponding to the skin (hereinafter, called a human skin representative luminance value). On one hand, a predetermined value (hereinafter, called a human skin target luminance value) is preset as a desirable value of the luminance value of the human skin in the image to be improved, a weighted average of the human skin representative luminance value and the human skin target luminance value is obtained based on predetermined weight, and the obtained average value is set as a target luminance value. Here, when it is assumed that the human skin representative luminance value is Y 0  and the target luminance value after the weighted average is obtained is Y 1 , a parameter γ, by which the luminance value Y 0  reaches a predetermined luminance value (Y 0 ≦Y 1 ), is assumed as a parameter representing the degree of improvement. Supposing that an operating result within [ ] described in the denominator of the equation (3) is nearly equal to Y(x, y), the parameter γ can be obtained by the following equation (4). 
         [0000]    
       
         
           
             
               
                 
                   γ 
                   = 
                   
                     1 
                     - 
                     
                       
                         log 
                          
                         
                             
                         
                          
                         
                           Y 
                           1 
                         
                       
                       
                         log 
                          
                         
                             
                         
                          
                         
                           Y 
                           0 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
         [0063]    Then, in a manner similar to that in the step S 104 , the distribution of the luminance component of the image data is improved using the luminance component of the image data, the distribution of the scale-converted luminance component, and the obtained parameter γ (step S 108 ). 
         [0064]    Subsequently, in a manner similar to that in the step S 105 , the improved luminance component and the color components X(x, y) and Z(x, y) converted in the step S 102  are synthesized to reconstitute the image data (step S 109 ). 
         [0065]    In the foregoing example, the method of improving the distribution of brightness in the optimum condition according to the image data to be processed based on the prior art 1 is explained. In the following, a method of improving the distribution of brightness in the optimum condition according to the image data to be processed based on the prior art 2 will be explained. Here, it should be noted that, since a difference between the above operation and the following operation is mainly directed to the process of improving the distribution of the luminance component of the image data to be executed in the step S 105 , this part different from the above example will be mainly explained. 
         [0066]    According to the prior art 2, the degree of generating a halo after performing the luminance conversion by using the luminance components on different scales is evaluated, and the most suitable scale is determined so that deterioration in image quality due to generation of the halo does not occur and then the luminance conversion is to be performed. A case of applying the present invention to the process will be indicated in the following equation (5). 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       Y 
                       ′ 
                     
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       a 
                       · 
                       
                         Y 
                          
                         
                           ( 
                           
                             x 
                             , 
                             y 
                           
                           ) 
                         
                       
                     
                     
                       1 
                       + 
                       
                         e 
                         · 
                         
                           V 
                            
                           
                             ( 
                             
                               x 
                               , 
                               y 
                             
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
         [0067]    Here, the V(x, y) denotes a luminance component selected from the luminance components to which the plural scale conversions were performed, symbol a denotes a parameter of adjusting intensity of the luminance component, and symbol e denotes a parameter of indicating the degree of improvement of the luminance component. Incidentally, it is needless to say that an adjustment method of an improvement parameter in the step S 107  has to be changed in accordance with the change of an adjustment method in the luminance component improvement process. 
         [0068]    In the example explained based on the prior art 2, although the luminance component itself of the image is to be adjusted together with the degree of improvement of the luminance component, it is needless to say that the present invention may be combined with a process of adjusting the luminance component itself of the image. 
         [0069]    In the process of the steps S 104  and S 108 , as a value within [ ] described in the denominator of the equation (3), a bilateral filtering process proposed by Durand et al. in “Fast Bilateral Filtering for the Display of High-Dynamic-Range Images”, acm Transactions on Graphics, July 2002, Vol. 21, No. 3 may be applied to the luminance component. In that case, the bilateral filtering process is previously executed to the luminance component in the scale conversion process in the step S 103 . 
         [0070]    Incidentally, because the process through the steps S 104  to S 106  is the preprocess for obtaining the parameter value in the step S 107 , it is possible to perform the process in the steps S 104  to S 106  to the image data previously reduced in size with respect to the original image data, determine the parameter value in the step S 107 , and then perform the process in and after the step S 108 . By doing so, it is possible to shorten a processing time even when the size of the original image data is remarkably large. 
         [0071]    Besides, in the present embodiment, the eight-bit R, G and B data are assumed as the image data to be processed. However, the present invention is also applicable to a case where optimum eight-bit R, G and B image data are reproduced from 16-bit R, G and B image data. 
         [0072]    According to the present embodiment, particularly, with respect to the image data obtained by photographing a person in a backlight state, it is possible to automatically adjust the degree of improvement of the luminance distribution and also optimally improve the distribution of brightness according to the image data to be processed. 
       Second Embodiment 
       [0073]      FIG. 4  is a block diagram showing the structure of an image processing system according to the second embodiment of the present invention. 
         [0074]    In  FIG. 4 , reference numeral  41  denotes an image input means for inputting digital image data (hereinafter, called image data) into the image processing system. For example, the input means  41  is constituted by a digital camera, a scanner, or the like. Reference numeral  42  denotes a luminance extraction means for extracting a luminance component and a color component from the image data input by the image input means  41 . Reference numeral  43  denotes a backlight detection means for obtaining a degree of backlight of the image data input by the image input means  41  from the luminance component of the image data output from the luminance extraction means  42 . Reference numeral  44  denotes a face detection means for detecting, as a face area, a portion corresponding to a person&#39;s face from the image data input by the image input means  41 . Reference numeral  45  denotes a luminance improvement judgment means for selecting an luminance improvement process method from the degree of the backlight output from the backlight detection means  43  and the detection result of the face detection means  44 , and adjusting, according to the image data, a parameter so as to optimize the degree of improvement processed by a luminance improvement means  47  from the luminance component of the image data output from the luminance extraction means  42 . Reference numeral  46  denotes a scale conversion means for obtaining a distribution, on relatively large scale, of the luminance components of the image data output from the luminance extraction means  42 . Reference numeral  47  denotes the luminance improvement means for improving a distribution of the luminance components of the image data by use of the luminance component of the image data output from the luminance extraction means  42  and the distribution, on relatively large scale, of the luminance components output from the scale conversion means  43 . Reference numeral  48  denotes an image reproduction means for reconstituting the image data by synthesizing the improved luminance component output from the luminance improvement means  47  and the color component output from the luminance extraction means  42  together. 
         [0075]    The above image processing system can be operated by an application program functioning on a general-purpose computer. Hereinafter, in the present embodiment, the image processing system which is operated mainly by the application program will be explained. 
         [0076]      FIG. 5  shows an algorithm of the application program by which the image processing system according to the present embodiment is operated on the general-purpose computer. 
         [0077]    First, when the application program is activated, a user inputs a file name of the image data, and the image data is read and stored in a storage unit of the computer by the image data input means  41  (step S 201 ). 
         [0078]    Here, the read image data corresponds to a two-dimensional arrangement of M×N pixels (here, M is the number of horizontal pixels and N is the number of vertical pixels) composed of, e.g., eight-bit pixels, and the image data is composed of three planes of R (red), G (green) and B (blue) components. The image data is assumed to be expressed by the R, G and B components as R(x, y), G(x, y) and B(x, y) (here, (x, y) are integers representing a pixel position; 1≦x≦M, and 1≦y≦N). In a case where the image data is compressed by a JPEG method or the like, the image data is expanded according to a predetermined expansion (or extraction) method, and the image data composed of the R, G and B component pixels is obtained. 
         [0079]    Next, the luminance component is extracted based on each pixel of the R, G and B components constituting the image data by the luminance extraction means  42  (step S 202 ). 
         [0080]    With respect to extraction of the luminance component, for example, it is assumed that the R, G and B pixel components correspond to data in an sRGB color space described in IEC 61966-2-1, and the data is converted into data of CIE 1931 XYZ by gamma conversion and a 3×3 matrix operation according to a method described in the IEC 61966-2-1. Here, when the converted X, Y and Z data are respectively assumed as X(x, y), Y(x, y) and Z(x, y), the Y(x, y) data corresponds to the luminance component to be extracted. 
         [0081]    As a method of extracting the luminance component, the above process may be simplified, that is, the luminance component may be extracted only by the matrix operation. Further, conversion from RGB into YCbCr, RGB into L*a*b*, or RGB into HSV may be used. 
         [0082]    Hereinafter, an outline of the process in and after a step S 203  will be first explained, and then the process in each step will be explained in detail. 
         [0083]    First, the backlight detection means  43  obtains a degree of backlight in the image data input in the step S 201 , based on the luminance component extracted in the step S 202  (step S 203 ). 
         [0084]    Here, when the obtained degree of backlight is low, the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low, and the flow advances to a step S 206 . 
         [0085]    Next, the face detection means  44  extracts a person&#39;s face area from the image data input in the step S 201  (step S 204 ). 
         [0086]    Here, when the face area of which the size is equal to or larger than a predetermined size is not detected, the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low, and the flow advances to the step S 206 . 
         [0087]    Then, the luminance improvement judgment means  45  obtains a luminance representative value of the face area and a luminance representative value of the entire image, and predicts the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used, on the basis of the luminance representative value of the entire image and the luminance representative value of the face area (step S 205 ). 
         [0088]    Next, the luminance improvement judgment means  45  determines a parameter for determining a degree of the luminance distribution to be improved, according to the predicted result of the process effect (step S 206 ). 
         [0089]    Here, when it is predicted that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low, and the flow advances to the step S 208 . 
         [0090]    Next, the scale conversion means  46  obtains the distribution of the luminance component on relatively large scale from the luminance component extracted in the step S 202  (step S 207 ). 
         [0091]    Then, the luminance improvement means  47  improves the distribution of the luminance component of the image data by using the parameter obtained in the step S 206 , in accordance with the predicted result of the process effect (step S 208 ). 
         [0092]    Subsequently, the image reproduction means  48  synthesizes the improved luminance component and the color component converted in the step S 202  together, and thus reconstitutes the image data (step S 209 ). 
         [0093]    Hereinafter, the process in each of the steps S 203  to S 209  will be explained in detail. 
         [0094]    In the step S 203 , the degree of backlight of the image data input in the step S 201  is obtained based on the extracted luminance component. 
         [0095]    First, a luminance histogram which represents a frequency of the luminance component with respect to each luminance range is formed based on the luminance component of the image data. Here, it is assumed that symbol P 1  denotes a frequency of ranges YT 1  to YT 2  of the predetermined luminance value corresponding to an intermediate portion of the luminance range and symbol P 2  denotes a frequency of a range YT 2  to YTmax of the luminance value brighter than the intermediate portion, a degree of backlight (PB) is obtained by the following equation (6). 
         [0000]        PB=P 2 −kB·P 1  (6) 
         [0000]    where 0&lt;YT1&lt;YT2&lt;YTmax is satisfied, symbol YTmax denotes the maximum value of the luminance range, and symbol kB denotes a constant. 
         [0096]    Here, it should be noted that the equation (6) is based on the assumption that, in a backlight scene, backlight occurs because the area of the brighter portion is large in the image. 
         [0097]    When the obtained degree of backlight is lower than a predetermined value, the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low. 
         [0098]    Incidentally, the method of obtaining the degree of backlight is not limited to the above. That is, as proposed in Japanese Patent Application Laid-Open No. 2000-134467 by the assignee of the present application, it is possible to use, as the degree of backlight, the number of peak areas obtained from a luminance histogram. In that case, for example, if there are the plural peak numbers, the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low. Incidentally, it is needless to say that the degree of backlight may be determined by appropriately incorporating the number of peak areas into the equation (6). 
         [0099]    In the step S 204 , the person&#39;s face area is detected from the image data input in the step S 201 . 
         [0100]    An example of process content for detecting the face area from the image data is shown in  FIGS. 3A to 3C . First, the image data is subjected to a low-pass filtering process of predetermined image size and is simultaneously reduced. Then, areas where the R, G and B components are locally small are extracted as candidates of the image areas of eyes on the basis of the pixel value of the reduced image data, and candidates sets each consisting of the extracted two candidates are made. Next, on the basis of equivalency of sizes, a difference in luminance, angles from the horizontal and the like between the two areas in each candidate set, it is judged whether or not the two areas in the candidate set represent eyes ( FIG. 3A ). Subsequently, with respect to the candidate set judged as eyes, a rectangular area is set from such two locations based on preset parameters concerning predetermined location relations ( FIG. 3B ), and the rectangular area is decided based on the edge in the vicinity of the border of the rectangular area and color information within this area ( FIG. 3C ). More specifically, with respect to the edge, it is judged whether or not edge components exceeding predetermined intensity occupies at a predetermined ratio or more in a predetermined width area in the vicinity of the border. Besides, with respect to the color information, an average value of the pixel values within the rectangular area is calculated, and it is then judged whether or not the calculated average value is in a preset skin color area. When both the edge and the color information satisfy respective conditions, the area in question is judged as the face area, and the position information of the rectangular area judged as the face area is output as the face area. 
         [0101]    Here, when the face area of which the size is equal to or larger than a predetermined size is not detected, the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low. 
         [0102]    In the present invention, the method of detecting the face area from the image data is not limited to the above. That is, for example, the method proposed by Yang et al. in “Detecting Faces In Images: A Survey”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, January 2002 may be applied. 
         [0103]    Subsequently, the process in each of the steps S 207  and S 208  will be explained, and thereafter the process in each of the steps S 205  and S 206  will be explained. 
         [0104]    In the step S 207 , the distribution of the luminance component (i.e., the low-frequency component of the luminance component), on relatively large scale, is obtained from the luminance component extracted in the step S 202 . 
         [0105]    To obtain the distribution of the luminance component on relatively large scale, for example, as shown in the prior art 1, a convolution operation is performed with respect to the extracted luminance component and a Gaussian function to supply an output. However, in the prior art 1, the MAX operation is performed with respect to each pixel of the R, G and B components of the improved image data instead of the luminance component of the image data. Here, to raise image quality of the improved image data, it is preferable that the convolution operation with the plural Gaussian functions different in the standard deviation is performed to obtain the distribution of the luminance component on plural scales. It should be noted that the above process of obtaining the distribution of the luminance component on large scale is called scale conversion hereinafter. 
         [0106]    In the step S 208 , the distribution of the luminance component of the image data is improved using the parameter obtained in the step S 206 , in accordance with the predicted result of the process effect. 
         [0107]    First, a luminance improvement process to be performed when the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is high will be explained. 
         [0108]    As an example of a process, according to the method based on the prior art 1, logarithmic transformation is performed for the luminance component and the distribution of the luminance component to which the scale conversion was performed, and the obtained difference between them is output. Further, a weighted average of difference outputs on different scales is assumed as the improved luminance component. However, since the degree of improvement cannot be adjusted according to the image by this method, a logarithmic transformation output of the luminance component, to which the scale conversion was performed, is multiplied by a coefficient. This coefficient corresponds to the parameter of adjusting the degree of improvement. An output of the luminance component which was improved based on the above-explained process is indicated by the following equation (1). 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       Y 
                       ′ 
                     
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∑ 
                       n 
                     
                      
                     
                       
                         w 
                         n 
                       
                        
                       
                         { 
                         
                           
                             log 
                              
                             
                                 
                             
                              
                             
                               Y 
                                
                               
                                 ( 
                                 
                                   x 
                                   , 
                                   y 
                                 
                                 ) 
                               
                             
                           
                           - 
                           
                             γ 
                             · 
                             
                               log 
                                
                               
                                 [ 
                                 
                                   
                                     
                                       F 
                                       n 
                                     
                                      
                                     
                                       ( 
                                       
                                         x 
                                         , 
                                         y 
                                       
                                       ) 
                                     
                                   
                                   * 
                                   
                                     Y 
                                      
                                     
                                       ( 
                                       
                                         x 
                                         , 
                                         y 
                                       
                                       ) 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                         
                         } 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0109]    Here, Y′(x, y), F n (x, y), w n , n and γ are respectively an output of the improved luminance component, a Gaussian function, weight between scales, a parameter representing the scale and a parameter representing the degree of improvement. Further, a symbol “*” denotes the convolution operation. 
         [0110]    It has been known that the weight between scales can be omitted by adjusting a standard deviation of the scale (replaced to a simple average) and it is preferable to return a luminance unit to an original luminance unit by an inverse transformation (exp. operation) rather than a value logarithmic-transformed is output as in the equation (1) as image quality of the improved image data. Therefore, it is preferable that an output represented by the following equation (2) corresponds to the improved luminance component. 
         [0000]        Y ′( x,y )=exp         log  Y ( x,y )−γ·Avg{log [ F   n ( x,y )* Y ( x,y )]}           (2) 
         [0111]    Here, the Avg indicates an average value operation. 
         [0112]    Incidentally, the following equation (3) may be available instead of the equation (2). 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       Y 
                       ′ 
                     
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       Y 
                        
                       
                         ( 
                         
                           x 
                           , 
                           y 
                         
                         ) 
                       
                     
                     
                       
                         [ 
                         
                           Avg 
                            
                           
                             { 
                             
                               
                                 
                                   F 
                                   n 
                                 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     , 
                                     y 
                                   
                                   ) 
                                 
                               
                               * 
                               
                                 Y 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     , 
                                     y 
                                   
                                   ) 
                                 
                               
                             
                             } 
                           
                         
                         ] 
                       
                       γ 
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
         [0113]    More preferably, in the present embodiment, the luminance distribution is converted through the process of the following equation (7) instead of the equation (3). 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       Y 
                       ′ 
                     
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         Y 
                          
                         
                           ( 
                           
                             x 
                             , 
                             y 
                           
                           ) 
                         
                       
                       
                         γ 
                          
                         
                             
                         
                          
                         0 
                       
                     
                     
                       
                         [ 
                         
                           Avg 
                            
                           
                             { 
                             
                               
                                 
                                   F 
                                   n 
                                 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     , 
                                     y 
                                   
                                   ) 
                                 
                               
                               * 
                               
                                 Y 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     , 
                                     y 
                                   
                                   ) 
                                 
                               
                             
                             } 
                           
                         
                         ] 
                       
                       γ 
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where, γ0 is a parameter for adjusting the luminance component of the entire image. 
         [0114]    Besides, it is possible to calculate the average value of the scale conversion outputs on plural scales in the scale conversion process of the step S 207 , and set the obtained average value as the distribution of the scale-converted luminance component. 
         [0115]    Next, the luminance improvement process to be performed when the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low will be explained. In that case, it is predicted that the improvement equivalent to the result of the above luminance improvement process can be achieved even if ordinary exposure correction is performed in the digital image process. More specifically, γ=0 is set in the equation (7), and the luminance conversion is performed according to the following equation (8). 
         [0000]        Y ′( x,y )= Y ( x,y ) γ0   (8) 
         [0116]    As compared with the equation (7), it is unnecessary in the equation (8) to obtain the distribution on relatively large scale in the scale conversion process. Moreover, in the equation (8), the operation of the dominator and the division both to be performed in the equation (7) are unnecessary. 
         [0117]    Then, the processes of the steps S 205  and S 206  to determine the parameter in the luminance improvement process will be explained in detail. 
         [0118]    In the step S 205 , the luminance representative value of the face area and the luminance representative value of the entire image are obtained. Then, based on the luminance representative value of the entire image and the luminance representative value of the face area, the luminance improvement judgment means  45  predicts the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used. 
         [0119]    More specifically, as the luminance representative value of the entire image, the average value of the luminance values is calculated and set as Y A . On one hand, a luminance value Y AT  being the target of the image to be improved with respect to a luminance representative value Y A  of the entire image is previously stored as a table, and the luminance value Y AT  being the target of the image to be improved with respect to the obtained luminance representative value Y A  of the entire image is obtained by referring to the table. Then, the luminance of the entire image based on, e.g., the equation (8) is corrected, whereby the parameter γ0 for obtaining the luminance value Y AT  being the target of the luminance value Y A  of the entire image can be obtained by the following equation (9). 
         [0000]      γ0=log  Y   A /log  Y   AT   (9) 
         [0120]    Further, in the luminance component of the image data, the average value of the luminance values within the face area is calculated and set as Y f . Here, it should be noted that matters such as eyes, hairs and the like which are not skin either exist within the rectangular area detected as the face area. Therefore, the average value of only the luminance values of predetermined percentage or more from the highest luminance value is calculated with respect to the pixels within the face area, and the calculated average value is assumed as a representative value of the luminance of the portion corresponding to the skin (hereinafter, called a human skin representative luminance value). On one hand, a predetermined value (hereinafter, called a human skin target luminance value) is preset as a desirable value of the luminance value of the human skin in the image to be improved, a weighted average of the human skin representative luminance value and the human skin target luminance value is obtained based on predetermined weight, and the obtained average value is set as a target luminance value. Here, when it is assumed that the human skin representative luminance value is Y f  and the target luminance value after the weighted average is obtained is Y fT , a parameter γ, by which the luminance value Y f  reaches a predetermined luminance value, is assumed as a parameter representing the degree of improvement. In respect that the human skin representative luminance value Y f  is corrected by the previously obtained parameter γ0, a human skin representative luminance value Y′ f  is obtained by the following equation (10). 
         [0000]      Y′ f =Y f   γ0   (10) 
         [0121]    Here, in case of Y′ f &lt;Y fT  (11), it is possible to predict that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is high. Therefore, in case of the inequality (11), the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is high. On the other hand, in case other than the inequality (11), the luminance improvement judgment means  45  predicts that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low. 
         [0122]    Incidentally, a predetermined positive constant C may be added to the above inequality (11), and, in that case, the inequality (11) is modified into Y′ f +C&lt;Y fT  (12). 
         [0123]    When it is difficult to clearly predict whether or not the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is high, the exposure correction of which the processing cost is low may be performed. 
         [0124]    Incidentally, in a case where the human skin representative luminance value is not remarkably low (i.e., dark) as compared with the human skin target luminance value, it is essentially unnecessary to improve the luminance in the face area of the image. In that case, it is needless to say that the luminance improvement judgment means  45  can predict that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low. 
         [0125]    In the step S 206 , the parameter for determining the degree of the luminance distribution to be improved is determined according to the prediction result obtained through the steps S 203  to S 205 . 
         [0126]    In any case, when it is predicted that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is low, the parameter γ0 obtained by the equation (7) is output as it is, and the process advances to the step S 208 . 
         [0127]    Meanwhile, when it is predicted that the effect in the process of the luminance improvement means  47  in which the distribution, on relatively large scale, of the luminance component of the image data being the output of the scale conversion means  46  is used is high, the parameter γ which satisfies that the after-correction human skin representative luminance value Y′f is the target luminance value Y fT  after the weighted average is assumed as the parameter representing the degree of improvement. Supposing that an operating result within [ ] described in the denominator of the equation (7) is nearly equal to Y(x, y) and the numerator thereof is replaced with Y′ f , the parameter γ can be obtained by the following equation (4). 
         [0000]    
       
         
           
             
               
                 
                   γ 
                   = 
                   
                     1 
                     - 
                     
                       
                         log 
                          
                         
                             
                         
                          
                         
                           Y 
                           fT 
                         
                       
                       
                         log 
                          
                         
                             
                         
                          
                         
                           Y 
                           f 
                           ′ 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
         [0128]    Then, the luminance conversion process is performed in the step S 208 , based on the parameter output in the step S 206 . 
         [0129]    In the step S 209 , the luminance component improved in the step S 208  and the color component converted in the step S 202  are synthesized together, and thus the image data is reconstituted. 
         [0130]    That is, the color component is first corrected according to the change of the luminance component so that the color of the after-reconstituted image data does not change as much as possible. For example, a ratio Y′(x, y)/Y(x, y) being a before-change/after-change ratio of the luminance component is multiplied respectively to the color components X(x, y) and Z(x, y), and then the R, G and B data are obtained from the X, Y and Z data. Here, it should be noted that that process to be performed here is equivalent to the inverse conversion (transformation) of the conversion to be performed in the process of the step S 202 . Therefore, the 3×3 matrix operation and the inverse gamma conversion process are performed to output the eight-bit R, G and B data. 
         [0131]    Incidentally, it is needless to say that, when a method of converting RGB into YCbCr is used as the method of extracting the luminance component in the step S 202 , the corresponding inverse conversion process should be performed. 
         [0132]    In the above, the method of optimally improving the brightness distribution according to the image data processed based on the prior art 1 is explained. However, it is needless to say that the present invention is applicable also to the prior arts 2 and 3 in which the luminance component of the digital image and its low-frequency component are used. 
         [0133]    Moreover, in the process of the step S 208 , the bilateral filtering process proposed by Durand et al. in “Fast Bilateral Filtering for the Display of High-Dynamic-Range Images”, acm Transactions on Graphics, July 2002, Vol. 21, No. 3 may be applied to the luminance component as the value of [ ] described in the denominator of the equation (7) In that case, the bilateral filtering process is performed beforehand to the luminance component in the scale conversion process in the step S 207 . 
         [0134]    In the above, the effect in the process in which the distribution, on relatively large scale, of the luminance component of the image data is used is predicted by analyzing the image data. Besides, for example, it is also possible to predict the effect in the process in which the distribution, on relatively large scale, of the luminance component of the image data is used, by using additional information of a digital camera image described in “Exchangeable Image File Format For Digital Still Camera Standard Exif Version 2.2” issued by JEITA (Japan Electronics and Information Technology Industries Association). 
         [0135]    In that case, for example, the additional information of the image data photographed by a digital camera is first analyzed. Then, when it is judged based on such analysis that automatic exposure is not set in an exposure mode, there is a possibility that the user intentionally settled the brightness of the image when photographing it. Thus, it is predicted that the effect in the process in which the distribution, on relatively large scale, of the luminance component of the image data is used is low, whereby it is better not to improve the luminance distribution of the image data. 
         [0136]    Further, when the additional information of the image data photographed by the digital camera is analyzed and it is judged based on the analysis that stroboscopic photographing is performed, the possibility of backlight scene is low. Thus, it is predicted that the effect in the process in which the distribution, on relatively large scale, of the luminance component of the image data is used is low, whereby it is better not to improve the luminance distribution of the image data. 
         [0137]    Moreover, when the additional information of the image data photographed by the digital camera is analyzed and it is judged based on the analysis that a photographing mode is set to a standard mode or a mode except for a person mode, the possibility that the main subject is a person is low. Thus, it is predicted that the effect in the process in which the distribution, on relatively large scale, of the luminance component of the image data is used is low, whereby it is better not to improve the luminance distribution of the image data. 
         [0138]    As above, by analyzing the additional information of the image data photographed by the digital camera, the luminance correction of which the process efficiency is high can be achieved according to need. Incidentally, it is needless to say that the prediction of the effect in the process in which the distribution, on relatively large scale, of the luminance component of the image data is used and which uses the additional information of the image data may be combined with the luminance improvement process for predicting, by analyzing the image data, the effect in the process in which the distribution, on relatively large scale, of the luminance component of the image data is used. 
         [0139]    Moreover, in the above, the eight-bit R, G and B data are assumed as the image data to be processed. However, the present invention is also applicable to a case where optimum eight-bit R, G and B image data are reproduced from 16-bit R, G and B image data. 
         [0140]    As many apparently widely different embodiments of the present invention can be made without departing from the spirit and scope thereof, it is to be understood that the present invention is not limited to the specific embodiments thereof expect as defined in the appended claims.