Abstract:
When an image contains a frame image representing a white frame, highlight portions except the frame are influenced by the white color of the frame, and gradation is not appropriately corrected. Hence, no sufficient dynamic range for gradation can be obtained. To prevent this, a frame recognition section ( 8 ) detects a frame image contained in an input image. A highlight/shadow calculation section ( 6 ) and a white balance calculation section ( 7 ) generate correction information of an image portion other than the detected frame image. An image correction section ( 10 ) corrects gradation of the image portion other than the frame image on the basis of the generated correction information.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    The present invention relates to an image processing apparatus and method and, more particularly, to an image processing apparatus for processing an image containing a frame image and a method therefor.  
           [0003]    2. Description of the Related Art  
           [0004]    For example, when gradation of an image is to be corrected, the entire image is corrected independently of whether a frame image (to be also simply referred to as a “frame” hereinafter) with an impression of, e.g., a frame is present in the image to be processed.  
           [0005]    For this reason, when a white frame is contained in the image to be processed, highlight portions except the frame are influenced by the white color of the frame, and gradation is not appropriately corrected. Hence, no sufficient dynamic range for gradation can be obtained.  
         SUMMARY OF THE INVENTION  
         [0006]    The present invention has been made to solve the above problem, and has as its object to provide an image processing apparatus capable of appropriately processing an image containing a frame image and a method therefor.  
           [0007]    In order to achieve the above object, according to a preferred aspect of the present invention, there is provided an image processing apparatus comprising: detection means for detecting an image area excluding a frame image contained in an input image; generation means for generating correction information of the detected image area; and correction means for correcting the image area on the basis of the generated correction information.  
           [0008]    Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0009]    [0009]FIG. 1 is a view showing the functional blocks (modules) of software according to an embodiment of the present invention;  
         [0010]    [0010]FIG. 2 is a flow chart showing the operation of the first embodiment of the present invention;  
         [0011]    [0011]FIG. 3 is a view for explaining data held by a parameter holding section;  
         [0012]    [0012]FIG. 4 is a flow chart showing details of processing of a frame recognition section;  
         [0013]    [0013]FIGS. 5A to  5 D are views for explaining the criteria for determining whether a pixel partially constructs a frame;  
         [0014]    [0014]FIGS. 6A and 6B are views for explaining data stored in an image information holding section;  
         [0015]    [0015]FIG. 7 is a flow chart showing details of processing of an image identification section;  
         [0016]    [0016]FIGS. 8A to  8 H are views showing details of an image portion identification operation by the image identification section;  
         [0017]    [0017]FIG. 9 is a flow chart showing details of processing of a highlight/shadow calculation section;  
         [0018]    [0018]FIG. 10 is a graph showing a luminance histogram;  
         [0019]    [0019]FIG. 11 is a flow chart showing details of processing of a white balance calculation section;  
         [0020]    [0020]FIG. 12 is a flow chart showing details of processing of an image correction section;  
         [0021]    [0021]FIG. 13 is a graph showing the characteristics of a look-up table prepared by the image correction section;  
         [0022]    [0022]FIGS. 14A and 14B are views showing an image having a frame with gradation;  
         [0023]    [0023]FIG. 15 is a flow chart showing the operation of the second embodiment of the present invention;  
         [0024]    [0024]FIG. 16 is a flow chart showing details of processing of an image identification section;  
         [0025]    [0025]FIGS. 17A to  17 P are views for explaining an image portion detection operation;  
         [0026]    [0026]FIGS. 18A to  18 L are views for explaining an image portion detection operation; and  
         [0027]    [0027]FIG. 19 is a block diagram showing the hardware arrangement of an image processing apparatus according to the present invention. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0028]    Preferred embodiments of the present invention will be described below in detail with reference to the accompanying drawings. An image processing apparatus of the present invention is realized by an apparatus having a hardware arrangement as shown in FIG. 19, for example, a computer apparatus such as a personal computer, or by supplying software (to be described later) to a dedicated computer apparatus.  
         [0029]    Referring to FIG. 19, a CPU  2  of a computer apparatus  100  executes a program stored in a ROM  1  and a storage section  8  using a RAM  3  and the storage section  8  such as a hard disk as a work memory. This program includes at least an operating system (OS) and software (to be described later) for executing processing of the present invention.  
         [0030]    Image data to be processed by the computer apparatus  100  is input from an input device such as a digital still camera  7  through an input interface (I/F)  6  and processed by the CPU  2 . The processed image data is converted by the CPU  2  into a form and format according to an output device and sent to the output device such as a printer  11  through an output I/F  10 . The input image data, output image data, and image data under processing may be stored in the storage section  8  or sent to a monitor  5  such as a CRT or an LCD through a video I/F  4  to display the image, as needed. These processing and operations are instructed by the user through a keyboard as an input device or a mouse as a pointing device connected to a keyboard I/F  9 .  
         [0031]    As the input and output I/Fs  6  and  10 , SCSI or GPIB interfaces as general-purpose interfaces, parallel interfaces such as centronics, and serial interfaces such as RS232, RS422, IEEE1394, or USB (Universal Serial Bus) are used.  
         [0032]    As the storage section  8 , not only a hard disk but also a storage medium such as a magneto-optical disk (MO) or an optical disk including a digital video disk (DVD-RAM) can be used. As the device for inputting image data, a digital video camera, an image scanner, or a film scanner can be used in addition to the digital still camera. Image data can also be input from the above storage medium or through a communication medium. As the device for outputting image data, a printer such as a laser beam printer, an ink-jet printer, or a thermal printer, or a film recorder can be used. Processed image data may be stored in the above storage medium or sent to a communication medium.  
         [0033]    First Embodiment  
         [0034]    [0034]FIG. 1 is a view showing the functional blocks (modules) of software of the first embodiment. FIG. 2 is a flow chart showing the operation of the first embodiment. The operation of this embodiment will be described below in detail in units of functional blocks.  
         [0035]    [Frame Recognition] 
         [0036]    In step S 1 , an input image  1  is read by an image input section  2  and stored in an image buffer  4 . In step S 2 , the image data buffered in the image buffer  4  is checked in units of pixels by a frame recognition section  8 , whose processing is shown in FIG. 4 in detail. It is determined whether a pixel partially constructs a frame (step S 41 ), and the determination result is stored in an image information holding section  9  (step S 42 ) On the basis of determination in step S 43 , processing in steps S 41  and S 42  is repeated for all image data buffered in the image buffer  4 , and then, the flow advances to step S 3 .  
         [0037]    Determination in step S 41  is done by comparing the color of a pixel of interest with the colors of eight pixels (adjacent pixels) adjacent to the pixel of interest. If a condition for recognizing a frame is satisfied, the pixel of interest is marked as part of a frame. If the condition is not satisfied, the pixel of interest is marked not to construct a frame.  
         [0038]    [0038]FIGS. 5A to  5 D are views for explaining the criteria for determining whether a pixel partially constructs a frame. When any one of the following conditions is satisfied, a pixel e of interest is recognized as part of a frame.  
         [0039]    (1) As shown in FIG. 5A, pixels a, b, d, and e have the same color.  
         [0040]    (2) As shown in FIG. 5B, pixels b, c, e, and f have the same color.  
         [0041]    (3) As shown in FIG. 5C, pixels e, f, h, and i have the same color.  
         [0042]    (4) As shown in FIG. 5D, pixels d, e, g, and h have the same color.  
         [0043]    The “same color” in the above conditions may be replaced with, e.g., “colors within a predetermined range”.  
         [0044]    [0044]FIGS. 6A and 6B are views for explaining data stored in the image information holding section  9 . The image information holding section  9  holds data of 1 bit/pixel (FIG. 6B) in correspondence with image data of 8 bits/pixel of each color (FIG. 6A). That is, the image information holding section  9  holds binary data with the same sizes as the vertical and horizontal sizes of an image stored in the image buffer  4 .  
         [0045]    [Image Identification] 
         [0046]    In step S 3 , an image portion, i.e., an image portion other than the frame is identified from the data stored in the image information holding section  9  by an image identification section  11 , whose processing is shown in FIG. 7 in detail (steps S 51  to S 54 ). Information of the upper, lower, left, and right ends of the image portion as the identification result is stored in a parameter holding section  5 . Terms “upper end”, “lower end”, “left end”, and “right end” will be described later in detail. Detection of an image portion except the frame will be described below. However, the frame portion can also be detected in accordance with almost the same procedure.  
         [0047]    [0047]FIGS. 8A to  8 H are views showing details of an image portion identification operation by the image identification section  11 . In step S 51 , the left end of the image is detected. To do this, the image is checked in units of columns from the left to the right. The position of the first column containing a pixel marked not to construct a frame is detected as the left end (FIGS. 8A and 8B).  
         [0048]    In step S 52 , the upper end of the image is detected. The image is checked in units of rows from the upper side to the lower side. The position of the first row containing a pixel marked not to construct a frame is detected as the upper end (FIGS. 8C and 8D).  
         [0049]    In step S 3 , the right end of the image is detected. The image is checked in units of columns from the detected left end to the right. The position of a column on the immediately left side of a column in which all pixels are marked to construct a frame is detected as the right end. When the right end is not detected, i.e., a column in which all pixels are marked to construct a frame is not detected until the right end of the image, the position of the rightmost column of the image is set as the right end (FIGS. 8E and 8F).  
         [0050]    In step S 4 , the lower end of the image is detected. The image is checked in units of rows from the detected upper end to the lower side. The position of a row on the immediately upper side of a row in which all pixels are marked to construct a frame is detected as the lower end. When the lower end is not detected, i.e., a row in which all pixels are marked to construct a frame is not detected until the lower end of the image, the position of the lowermost row of the image is set as the lower end (FIGS. 8G and 8H).  
         [0051]    In the above description, a column or row having a pixel marked not to construct a frame or a column or row in which all pixels are marked to construct a frame is detected. However, in consideration of a case wherein an end of the frame tilts, curves, or undulates, a column or row having at least a predetermined number of pixels marked to construct a frame or at least a predetermined number of consecutive pixels marked to construct a frame may be detected.  
         [0052]    [Calculation of Highlight Point and Shadow Point] 
         [0053]    In step S 4 , a highlight point and a shadow point are calculated by a highlight/shadow calculation section  6 , whose processing is shown in FIG. 9 in detail, on the basis of the information stored in the parameter holding section  5 , and stored in the parameter holding section  5 . More specifically, in step S 1 , image data of the image portion except the frame is read out from the image buffer  4 , and a luminance histogram shown in FIG. 10 is generated. Next, on the basis of the generated histogram, a highlight point LH and a shadow point LS are calculated in steps S 12  and S 13 . The highlight point LH is the minimum luminance value in the highlight area. The shadow point LS is the maximum luminance value in the shadow area.  
         [0054]    In the luminance histogram shown in FIG. 10, since luminances in the highlight area (99% to 100%) are 230 to 255, the highlight point LH is 230. Additionally, since luminances in the shadow area (0% to 1%) are 0 to 14, the shadow point LS is 14.  
         [0055]    [Calculation of White Balances] 
         [0056]    In step S 5 , the white balances and black balances are calculated by a white balance calculation section  7 , whose processing is shown in FIG. 11 in detail, on the basis of the information stored in the parameter holding section  5 , and stored in the parameter holding section  5 . More specifically, in steps S 21  and S 22 , each pixel is read out from the image buffer  4 , the average luminance value (white balance) of pixels with luminances falling between the highlight point LH and a corrected highlight point HP is calculated for each of the R, G, and B colors, and the average luminance value (black balance) of pixels with luminances falling between a corrected shadow point SP and the shadow point LS is calculated for each of the R, G, and B colors.  
         [0057]    Referring to FIG. 10, the average luminance of pixels with luminances falling within the range from LH=230 to HP=245 is calculated as the white balance for each of the R, G, and B colors, and the average luminance of pixels with luminances falling within the range from SP=10 to LS=14 is calculated as the black balance for each of the R, G, and B colors. These results are stored in corresponding registers RH, GH, BH, RS, GS, and BS in the parameter holding section  5  (FIG. 3).  
         [0058]    [Image Correction] 
         [0059]    In step S 6 , gradation of the image is corrected by an image correction section  10 , whose processing is shown in FIG. 12 in detail, on the basis of the information stored in the parameter holding section  5 , and the correction result is written in the image buffer  4 . More specifically, a look-up table for gradation correction is prepared on the basis of the white balances and black balances stored in the parameter holding section  5  (step S 31 ). Image data read out from the image buffer  4  in units of pixels is subjected to gradation correction using the look-up table. The corrected image data are written in the image buffer  4  (step S 32 ).  
         [0060]    [0060]FIG. 13 is a graph showing the characteristics of the look-up table. The look-up table is prepared on the basis of the white balances RH, GH, BH, and white point LH, and the black balances RS, GS, and BS, and black point LS. In the example shown in FIG. 13, the gamma correction level for the highlight portion increases in the order of green, blue, and red. In this way, by emphasizing green and blue with respect to red, so-called color fog of a bluish (fogged with blue) image can be corrected.  
         [0061]    [Image Output] 
         [0062]    Finally, in step S 7 , the image which has undergone gradation correction and buffered in the image buffer  4  is output by an image output section  3  as an output image  12 .  
         [0063]    [Parameter Holding Section] 
         [0064]    [0064]FIG. 3 is a view showing data held by the parameter holding section  5 . In the initial state, appropriate values are stored as the corrected highlight point HP and corrected shadow point SP.  
         [0065]    [Recognition of Frame with Gradation] 
         [0066]    When the following conditions are set for determination in step S 41  in consideration of a frame with gradation as shown in FIG. 14A, the pixel e of interest can be recognized to construct a frame (FIG. 14B). To determine the following conditions, RGB image data is temporarily converted into HSB data or HSL data. This conversion technique is known and a detailed description thereof will be omitted.  
         [0067]    (1) Pixels a, b, d, and e shown in FIG. 5A have the same hue, and the difference between the lightness and saturation has a predetermined value or less.  
         [0068]    (2) Pixels b, c, e, and f shown in FIG. 5B have the same hue, and the difference between the lightness and saturation has a predetermined value or less.  
         [0069]    (3) Pixels e, f, h, and i shown in FIG. 5C have the same hue, and the difference between the lightness and saturation has a predetermined value or less.  
         [0070]    (4) Pixels d, e, g, and h shown in FIG. 5D have the same hue, and the difference between the lightness and saturation has a predetermined value or less.  
         [0071]    Second Embodiment  
         [0072]    In the first embodiment, gradation correction when one image portion (e.g., a photograph) is contained in one image has been described. However, when frame recognition of the present invention is applied, even when a plurality of image portions are contained in one image, gradation correction can be appropriately performed for each image portion. The second embodiment in which, for example, two image portions are recognized, and gradation correction is performed for each of the two recognized image portions will be described below. An image portion detection method to be described below can be applied to detect not only two image portions but also three or more image portions, as a matter of course.  
         [0073]    [0073]FIG. 15 is a flow chart showing the operation of the second embodiment. The operation of the second embodiment will be described below in detail in units of functional blocks.  
         [0074]    [Frame Recognition] 
         [0075]    In step S 61 , an input image  1  is read by an image input section  2  and stored in an image buffer  4 . In step S 62 , the image data buffered in the image buffer  4  is checked in units of pixels by a frame recognition section  8 . It is determined whether a pixel partially constructs a frame (step S 41 ), and the determination result is stored in an image information holding section  9  (step S 42 ). On the basis of determination in step S 43 , processing in steps S 41  and S 42  is repeated for all image data buffered in the image buffer  4 , and then, the flow advances to step S 63 .  
         [0076]    [Image Identification] 
         [0077]    In step S 63 , an image portion, i.e., an image portion other than the frame is identified from the data stored in the image information holding section  9  by an image identification section  11 , whose processing is shown in FIG. 16 in detail (steps S 71  to S 76 ). Information of the upper, lower, left, and right ends of the image portion as the identification result is stored in a parameter holding section  5 .  
         [0078]    The operation of the image identification section  11  will be described in detail. In step S 71 , the left end of the image is detected. To do this, the image is checked in units of columns from the left. The position of a column containing a pixel marked not to construct a frame is detected as the left end. Subsequently, in step S 72 , it is determined whether the left end is detected. If NO in step S 72 , detection is ended. If YES in step S 72 , the flow advances to step S 73 .  
         [0079]    In step S 73 , the upper end of the image is detected. The image is checked in units of rows from a row containing a pixel marked not to construct a frame and located at the uppermost portion of the column at the left end detected in step S 71  to the upper side. A row having at least a predetermined number of consecutive pixels marked to construct a frame is detected. The position of a low immediately below the row is detected as the upper end.  
         [0080]    In step S 74 , the values of the detected left and upper ends are set as the initial values of the right and lower ends of the image. In step S 75 , the right end of the image is detected. The image is checked from the position of the right end initially set in step S 74  to the right in units of columns. A column having at least a predetermined number of consecutive pixels marked to construct a frame is detected. The position of a column immediately on the left side of the column is detected as the right end.  
         [0081]    Instep S 76 , the position of the right end of the image is compared with that of the lower end. Processing advances on the basis of the comparison result.  
         [0082]    (1) When the right end is on the lower left side of the lower end, processing is ended.  
         [0083]    (2) When the right end is on the upper side of the lower end, the flow returns to step S 75 .  
         [0084]    (3) When the lower end is on the left side of the right end, the flow advances to step S 77 .  
         [0085]    In step S 77 , the lower end of the image is detected. The image is checked from the current lower end position to the lower side in units of rows. A row having at least a predetermined number of consecutive pixels marked to construct a frame is detected. The position of a row immediately above the row is detected as the lower end.  
         [0086]    When detection processing shown in FIG. 16 is ended, it is determined in step S 64  whether the upper, lower, left, and right ends of the image are detected, i.e., an image portion is detected. If YES in step S 64 , information representing the upper, lower, left, and right ends of the image portion are stored in the parameter holding section  5 , and the flow advances to step S 65 . If NO in step S 64 , i.e., when detection is ended, the flow advances to step S 69 , and the image which has undergone gradation correction and buffered in the image buffer  4  is output by an image output section  3  as an output image  12 .  
         [0087]    Steps S 65  to S 67  correspond to steps S 4  to S 6  in FIG. 2 and have substantially the same processing contents as described above, and a detailed description thereof will be omitted.  
         [0088]    In step S 68 , information in the area of an image information holding section  9 , which corresponds to the image portion which has undergone gradation correction, is marked again to construct a frame. After the information in the image information holding section  9  is updated, the flow returns to step S 63  to detect the next image portion.  
       EXAMPLE 1 OF IMAGE RECOGNITION  
       [0089]    [0089]FIGS. 17A to  17 P are views for explaining image recognition when one image contains two image portions.  
         [0090]    At the time point of step S 62 , information as shown in FIG. 17A is stored in the image information holding section  9 . Next, in step S 71 , a column containing a pixel determined not to construct a frame is searched for from the left in units of columns to detect the left end of the image (FIG. 17B). In step S 73 , on the right side of the detected left end, a row having at least a predetermined number of consecutive pixels marked to construct a frame is searched for to the upper side in units of rows to detect the upper end of the image (FIGS. 17C and 17D). In step S 74 , the same values as those of the left and upper ends are set as the initial values of the right and lower ends of the image.  
         [0091]    In step S 75 , the right end of the image is detected. The image is checked from the position of the currently set right end of the image to the right in units of columns. The position of a column immediately on the left side of a column having at least a predetermined number of pixels marked to construct a frame is detected as the right end (FIGS. 17E and 17F).  
         [0092]    In step S 76 , the position of the right end is compared with that of the lower end. In the example shown in FIGS. 17A to  17 P, since the lower end is on the left side of the right end, the flow advances to step S 77 . In step S 77 , the lower end of the image is detected. The image is checked from the currently set lower end to the lower side in units of rows. The position of a row immediately above a row having at least a predetermined number of consecutive pixels marked to construct a frame is detected as the lower end (FIGS. 17G and 17H).  
         [0093]    In step S 76 , again, the position of the right end is compared with that of the lower end. In this case, since the right end is on the lower left side of the lower end, the area of an image portion is determined, and the flow advances to step S 64 . Since the image portion is detected, steps S 65  to S 67  are executed on the basis of determination in step S 64 . The detected image portion is subjected to gradation correction. In step S 68 , the information in the image information holding section  9  is updated, and pixels corresponding to an area indicated by a broken line in FIG. 17I, i.e., the image portion which has undergone gradation correction, are marked again to construct a frame.  
         [0094]    In step S 63 , again, another image portion is detected in accordance with the same procedure as described above (FIGS. 17I to  17 P). Since the image portion is detected, steps S 65  to S 67  are executed on the basis of determination in step S 64 . The image portion is subjected to gradation correction. In step S 68 , the information in the image information holding section  9  is updated. After this, the flow returns to step S 63  again. However, since only areas marked to construct frames are stored in the image information holding section  9 , detection is ended on the basis of determination in step S 72 . After determination in step S 64 , an image which has undergone gradation correction is output in step S 69 .  
       EXAMPLE 2 OF FRAME RECOGNITION  
       [0095]    [0095]FIGS. 18A to  18 L are views for explaining image recognition. In FIGS. 18A to  18 L, an image portion represented by data stored in the image information holding section  9  after execution of step S 62  has a U shape due to some reason. The original image portion has, e.g., a rectangular photographic image.  
         [0096]    First, in step S 71 , the left end of the image is detected (FIGS. 18A and 18B). In step S 73 , the upper end of the image is detected (FIGS. 18C and 18D). In step S 75 , the right end of the image is detected (FIGS. 18E and 18F). In step S 76 , the position of the right end is compared with that of the lower end. Since the lower end is on the left side of the right end, the flow advances to step S 77 .  
         [0097]    In step S 77 , the lower end of the image is detected (FIGS. 18G and 18H). In step S 76 , again, the position of the right end and that of the lower end are compared. Since the right end is on the upper side of the lower end, the flow returns to step S 75 .  
         [0098]    In step S 75 , the right end of the image is detected (FIGS. 18J and 18K). In step S 76 , again, the position of the right end is compared with that of the lower end. Since the right end is on the lower left side of the lower end, the image portion is determined, and the flow advances to step S 64 .  
         [0099]    As described above, according to the above-described embodiments, since an image containing a frame image is subjected to gradation correction excluding the frame image, the gradation can be appropriately corrected without any influence of the color or luminance of the frame image. In addition, a frame image with gradation can also be recognized using a similar algorithm. Furthermore, with application of this algorithm, even when an image contains a plurality of images such as photographs separated by frame images, appropriate gradation correction can be performed for each image.  
         [0100]    [Terms: Left End, Upper End, Right End, and Lower End] 
         [0101]    In the above description of embodiments, the left, upper, right, and lower ends of an image are detected. More exactly, the coordinates of positions indicated by hollow bullets in FIGS. 17A to  18 H are detected, and these positions are compared with each other. For example, a right end or lower end means the coordinates of a position where a line corresponding to the right end of an image portion crosses the contour of the entire image or the contour of the image portion.  
         [0102]    As has been described below, according to the present invention, an image containing a frame image can be appropriately processed.  
         [0103]    As many apparently widely different embodiments of the present invention can be made without departing from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments thereof except as defined in the appended claims.