Abstract:
A method and apparatus for interactively and automatically enhancing image data by modifying one or more features of an image. The method includes inputting image data representing an image to be modified, analyzing the image for features to be modified, automatically selecting one or more feature to be modified, determining a first degree of modification using a automatic image enhancement techniques or a stored user preference, reversibly modifying the image based on the selected image feature and the first degree of modification, showing the image before improvement, showing the image after improvement and allowing the user to further modify at least one feature of the image data.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of Invention 
     This invention relates to image processing. 
     2. Description of Related Art 
     Automatic image enhancement (AIE) techniques operate on sampled images and invoke image processing operations on the sampled images by analyzing the sub-sampled images. Automatic image enhancement techniques automatically modify images by adjusting exposure, contrast, sharpness, color balance and/or saturation. After analyzing the sampled image, automatic image enhancement techniques automatically determine conservative corrective actions, to perform on the image. The determined appropriate image processing is then applied to the image. Specifically, automatic image enhancement techniques can be performed in a fully automatic mode, where conservative image processing parameters can be chosen to ensure that the image quality does not get worse. 
     In the automatic image enhancement field, methods are known for modifying image data in various ways. U.S. Pat. No. 5,371,615 to Eschbach, U.S. Pat. No. 5,450,217 to Eschbach et al., U.S. Pat. No. 5,581,370 to Fuss et al., U.S. Pat. No. 5,450,502 to Eschbach et al., and U.S. Pat. No. 5,363,209 to Eschbach et al., each incorporated herein by reference in its entirety, disclose various automatic image enhancement techniques. 
     Alternatively, for high quality publication and printing, manually modifying image features may be preferred for artistic reasons. Specifically, a wide variety of software image enhancement applications are available that allow a skilled user to manually optimize image features so that a particular image may be tailored to match certain qualities desired by the user. 
     Lastly, it is known to use a genetic algorithm to modify an image on a random basis and use the results, as judged by a user, to modify the image modification in a series of modifications that are eventually acceptable to a user. One example of this method is PhotoGenetics, by QBeo Corporation. 
     SUMMARY OF THE INVENTION 
     This invention provides systems and methods that adjust automatic image modifications until a user accepts the results. This invention separately provides systems and methods that allow image modification to be adjusted to accommodate image data that regularly comes from a specific source and requires similar adjustments. 
     This invention separately provides systems and methods for adjusting the amount of user interaction so that the user can be trained or have better control over the output according to their skill level and time available. 
     The systems and methods according to this invention use known automatic image modification methods and systems for users that do not possess either the skill or the time to manually modify features to produce acceptable output. The systems and methods according to this invention add user input and interaction to automatic image enhancement. Alternatively, U.S. patent application Ser. No. 09/487,805, filed Jan. 20, 2000, incorporated herein in its entirety, relates to an automatic method for adjusting color quality. In the &#39;805 application, an image is analyzed for various color problems, such as luminance differences, contrasting colors, etc. A user without the skill to manually modify images, but needing modifications other than the standard automatic image enhancement modifications to an image, can use the systems and methods of this invention to implement increased modifications to an image to produce results the user can accept. 
     In various exemplary embodiments of the systems and methods of this invention, differing numbers of adjustment parameters may be provided based on the user&#39;s skill level or time available. For example, speckle removal can be adjusted based on the size of the anomaly and the size of the comparison area. Accordingly, to remove speckle, the systems and methods of this invention can, for example, generically increase or decrease the speckle removal parameters, or allow the user to adjust the size of anomaly analyzed for and/or the comparison area used. In the increased adjustment parameter mode, the user is given a suggested anomaly size and a suggested comparison size. The user is then allowed to adjust these parameters based on experience and time constraints. The user can be trained using the systems and methods of this invention. The user&#39;s adjustment parameters can be adjusted by the user to fit individual preferences without requiring the user to determine starting values. 
     These and other features and advantages of this invention are described in or are apparent from the following detailed description of various exemplary embodiments of the systems, methods and graphical user interfaces according to this invention. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The invention will be described in relation to the following drawings in which reference numerals refer to like elements, and wherein: 
     FIG. 1 is a block diagram of one exemplary embodiment of a system that modifies images according to this invention; 
     FIG. 2 is a block diagram of one exemplary embodiment of the image modification manager  140  of FIG. 1; 
     FIG. 3 is one exemplary embodiment of a graphical user interface that allows a user to interactively modify the adjustment parameters according to this invention; 
     FIG. 4 is a flowchart outlining one exemplary embodiment of a method for modifying image data according to invention; and 
     FIGS. 5A-5C are a flowchart outlining in greater detail one exemplary embodiment of the step of analyzing the image  500  of FIG.  4 . 
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
     FIG. 1 shows a functional block diagram of one exemplary embodiment of an interactive automatic image enhancement processing system  100  according to this invention. As shown in FIG. 1, the image processing system  100  includes an input/output interface  110  that connects the image processing system  100  to an image data source  200  via a link  201 , a display device  112  via a link  113 , one or more input devices  114  via one or more links  115  and an image data sink  202  via a link  203 . The image processing system  100  also includes a controller  120 , memory  130 , and an image modification manager  140 , all connected via a link  199 . 
     The controller  120  may be implemented as a main program circuit, a timer circuit, or an integrated circuit that calls the image modification manager  140 , arranges the memory  130 , and handles input/output events. Alternately, the system and apparatus may use an external controller (not shown) to handle generic functions, retaining the controller  120  to handle internal functions only. 
     The memory  130  includes one or more queues, stacks, address spaces, registers, files, arrays and buffers. As shown in FIG. 1, the memory  130  can include a initial image memory portion  132 , a modified image memory portion  134  and a degree of modification memory portion  136 . These memory portions  132 - 136  are exemplary embodiments of one way of storing the received image data and are intended to be illustrative and not limiting. 
     The image modification manager  140  modifies the image according to one or more implemented automatic image enhancement techniques, which can include any known or later developed automatic image enhancement techniques, such as those disclosed in the incorporated references. The image modification manager will make intelligent, conservative changes that generally improve image quality according to the standard automatic image enhancement protocol. The degrees and types of modifications can be stored in the degree of modification memory portion  136 . 
     The image data source  200  provides image data to the input/output interface  110  over the link  201 . Once the image data source  200  has transmitted the image data to the input/output interface  110 , the input/output interface  110  stores, under control of the controller  120 , the image in the initial image memory portion  132  of the memory  130 . 
     The controller  120  scans the memory  130  for image data available to be processed. When the controller  120  determines that the initial image memory portion  132  contains such data, the controller  120  copies the data in the initial image memory portion  132  to the modified image memory portion  134 . The controller  120  then invokes the image modification manager  140 . The controller  120  copies the image data for the convenience of the system and so that the user can compare the current image and the modified image. Alternatively, the current image can be modified, losing the ability to compare the current and modified images. It should be appreciated that this is also within the scope of the invention. 
     The image modification manager  140  analyzes the image data in the modified image memory portion  134  to determine a first degree of modification according to the implemented automatic image enhancement. The image modification manager  140  may then modify the image data or not as discussed below. After the image modification manager  140  has completed its analysis and performed any determined modifications, if necessary, of the image data, the degree of modification determined and/or used may be stored in the degree of modification memory portion  136 . Alternatively, the degree of modification may be stored within the image modification manger  140 , or any other appropriate location. 
     Once the image modification manager  140  has automatically enhanced the image, the initial image, the modified image and the degree(s) of modification can be output through the input/output interface  110  to the display device  112 . The user can then choose to make more radical changes to the image using the image modification manager  140  discussed above or any other appropriate circuit or software, such as those discussed below. Once the user has increased or decreased the amount of modification to the image so that it is acceptable to the user, the image is passed via the input/output interface  110  to the image data sink  202 . 
     The interactive automatic image enhancement system  100  can then either save the degree(s) of modification stored in the degree of modification memory portion  136  for future use with similar types of images or as a new set of control parameters for the user. The degrees of modification may include boundaries for automatic image enhancement, additional or modified local control variables, and minimum and maximum allowable modifications. Alternatively, the degree of modification memory portion  136  can be cleared so that the user is initially provided with default modifications to the next image. 
     In various exemplary embodiments, the memory  130  is implemented using static or dynamic RAM. However, the memory  130  can also be implemented using a removable disk, a hard drive, flash memory or the like. Therefore, the memory  130  can be implemented using any appropriate combination of alterable, volatile or non-volatile memory or non-volatile or fixed memory. 
     The image data source  200  can be a digital camera, a scanner, a locally or remotely located computer, or any known or later developed device that is capable of generating electronic image data. Similarly, the image data source  200  can be any suitable device that stores and/or transmits electronic image data, such as a client or a server of a distributed network. The image data source  200  can be integrated with the interactive automatic image enhancement processing system  100 , as in a digital copier having an integrated scanner. 
     The image data sink  202  can be any device or system that is capable of outputting or storing the processed image data generated by the interactive automatic image enhancement processing system  100 . Thus, the image data sink  202  can be a printer, a copier or other image forming device, a hard drive, a facsimile device, a display device, a memory, or the like. 
     When the image data source  200  or image data sink  202  is physically separate from the interactive automatic image enhancement processing system  100 , the links  201  and  203  can be any known or later developed connection device or structure, such as a connection over the public switch telephone number, a local area network, a wide area network, an intranet, an extranet, the Internet, or any other known or later distributed processing network. It should be appreciated that the links  201  and  203  can include both wired and wireless connection structures and devices. 
     It should be appreciated that while the electronic image data can be generated at the time of processing by the interactive automatic enhancement processing system  100 , the electronic image data could have been generated at any time in the past. 
     Moreover, the electronic image data need not have been generated from an original physical document, view or physical embodiment, but could have been created from scratch electronically. The image data source  200  is thus any known or later developed device which is capable of supplying electronic image data over the link  201  to the interactive automatic image enhancement processing system  100 . 
     The input devices  114  can include one or more of a mouse, a keyboard, a track ball, a track pad, a touch screen, or any other known or later developed device that a user can use to supply data and/or control signals to the interactive automatic image enhancement processing system  100 . Similarly, the display device  112  can be a cathode ray tube, a LCD, a printer or any other known or later developed device that is capable of displaying the current and modified images in various graphical user interfaces according to this invention. 
     The links  113  and  115 , while generally comprising a direct cable connection to the input/output interface  110 , can be any known or later developed system or device for connecting the input devices  114  and/or the display devices  115 , respectively, to the input/output interface  110 , such as those devices discussed above with respect to the links  201  and  203 . 
     FIG. 1 further shows the input/output interface  110 , the controller  120 , the image modification manager  140  and memory  130  connected by the link  199 . The link  199 , while generally comprising a data/control bus connecting the input/output interface  110 , the controller  120 , the image modification manager  140  and memory  130 , can be any known or later developed system or device for connecting the devices, such as the links  201  and  203  discussed above. 
     FIG. 2 shows a functional block diagram of one exemplary embodiment of the image modification manger  140  of FIG. 1 according to this invention. As shown in FIG. 2, the exemplary image modification manager  140  can include one or more of a white point/black point correction manager  142 , a gamma manager  144 , a luminance manager  146 , a speckle manager  148 , a sharpness manager  150 , a compression artifact manager  152 , and a saturation manager  154 . The various managers  142 - 154  may use various ones of the automatic image enhancement methods and techniques disclosed in the incorporated references or any other known or later developed automatic image enhancement technique. For example, the sharpness manager  150  can use the automatic image enhancement method disclosed in the incorporated  209  patent to determine the maximum local contrast within the image, determine a filter value that increases local contrasts to a predetermined percentage of the maximum local contrast and applies a filter. 
     The image modification manager  140  does not need to include all of the various managers  142 - 154 . The image modification manager  140  does not need to use all of the various managers  142 - 154  when analyzing a given image. Further, the image modification manager  140  can include other known or later developed automatic image enhancement methods without modifying the scope of the invention. 
     As shown in FIGS. 1 and 2, in various exemplary embodiments, the interactive automatic image enhancement processing system  100  is implemented using a programmed general purpose computer. Alternatively, each of the circuits shown in FIGS. 1 and 2 can be implemented as physically distinct hardware circuits within an ASIC, or using a digital signal processor, a FPGA, a PDL, a PLA, or a PAL, or using discrete logic elements or discrete circuit elements. The particular form of each of the circuits shown in FIGS. 1 and 2 will be a design choice and will be obvious and predictable to those skilled in the art. 
     FIG. 3 is an exemplary embodiment of an interactive automatic image enhancer graphical user interface  300 . The graphical user interface  310  indicates an current image portion  310 , an updated image portion  320  and various image modification portions  330 - 390 . The user views the current image in the current image portion  310 , the updated image in the updated image portion  320 , and the selected image modifiers, including one or more of a white/black point modification portion  330 , a luminance modification portion  340 , a sharpness correction/modification portion  350 , a saturation modification portion  360 , a gamma modification portion  370 , a speckle modification portion  380  and another modification portion  390 . 
     The current image can be the image after an initial modification by automatic image enhancement, the image after additional modifications using the systems and methods of this invention, or the original, unmodified image. 
     The user can then decide to apply the recommended modifications by selecting a modification selection widget  331 - 391  of the corresponding modification portion  330 - 390 . The image can be modified by the image modification manager  140  as discussed above. The user can then select additional modification to the image. The additional modifications can be generated by the appropriate ones of the modification managers  142 - 154  shown in FIG. 2, using increased modifications parameters to modify the image. Thus, a user can forcibly apply more radical changes than automatic image enhancement would normally provide. 
     Once the user finds the modified image acceptable, the user can save the modified image, have the modified image re-analyzed, output the image, or the like. The user can thus make the modified image into the current image and use the modified image as the current image. Thus the exemplary embodiment is meant to be illustrative and not limiting. 
     The embodiment shown in FIG. 3 uses check boxes as the selection widgets  331 ,  341 ,  351 ,  361 ,  371 ,  381  and  391 . However, any known or later developed selection widgets, such as radio buttons, slider bars, toggle buttons, or other graphical interface objects can be used as the selection widgets. A recommendation can be applied immediately upon selecting a corresponding selection widget, or by selecting an “Apply” widget. It should be appreciated that Min/Average/Max, More/Original/Less, or other recommendation control metaphors are usable to select the levels or amounts of further (or lesser) modification to be forcibly applied to the modified image. 
     The exemplary embodiment of the graphical user interface  310  shown in FIG. 3 has the current image portion  310  placed next to the updated image portion  320  in a split screen arrangement. Other arrangements, such as tabs, multiple images or other embodiments may be used to display the image before and after modification. Additionally, only the current image portion  310  or only the modified image portion  320  can be shown and used. 
     In the exemplary embodiment of the graphical user interface  300  shown in FIG. 3, the modification portions  330 - 390  are positioned beneath the current and modified image portions  310  and  320 . It should be appreciated that the modification portions  330 - 390  can be positioned anywhere within the graphical user interface  300 . Similarly, showing the modification portions  330 - 390  is optional. Various modifications to the graphical user interface  300  will be obvious to those skilled in the art. 
     FIG. 3 shows the white/black point correction portion  330 , the luminance modification portion  340 , the sharpness modification portion  350 , the saturation modification portion  360 , the gamma modification portion  370 , the speckle modification portion  380 , and the other modification portion  390  as automatic image enhancement methods as discussed below. The graphical user interface can show all implemented methods, only those methods that were used to create the modified image, or other techniques for organizing and displaying the modification methods for a user to select to modify the image. 
     Additionally, the managers may have differing levels of detail shown that allow a user to modify the image according to their skill level and time constraints. For example, speckle removal can be controlled as simply more or less speckle removal. Alternatively, speckle removal can be controlled by the reference size of the anomaly and the tone difference required to trigger analysis. Additional factors are disclosed in the incorporated automatic image enhancement references. 
     FIG. 4 is a flowchart outlining one exemplary embodiment of a method for interactively and automatically enhancing an input image according to this invention. Beginning in step S 100 , control continues to step S 200 , where image data representing an current image is received. The image data may be converted between formats, and a duplicate image may be created for modification and comparison with the current image. Other image preparation tasks may also be performed. Next, in step S 300 , the image data is analyzed. The image data may be analyzed using the implemented automatic image enhancement techniques. Then, in step S 500 , a user interaction state is invoked to allow the user to increase the amount of image modification applied if the automatic image enhancement techniques have not modified the image enough. These image modification techniques are performed, and repeated as necessary with the user&#39;s input. 
     Once the user has determined that no further image modification techniques are to be forcibly applied to the modified image, control continues to step S 600 , where the processed image data is output. In step S 700 , the method ends. 
     While in the user interaction state S 500 , depending on the particular inputs supplied by the user, control can jump to any one of steps S 510 -S 570 . In particular, in step S 510 , the black point and white point values of the modified image data are revised to further modify (or lesser the modifications to) the image. Control then jumps back to step S 500 . In step S 520 , the sharpness of the modified image is further modified. Control then again jumps back to step S 500 . Similarly, in step S 530 , the luminance of the modified image is further modified. 
     In step S 540 , speckle noise and possible other types of noise can be more aggressively removed. In step S 550 , the saturation of the various colors in the modified image is further modified. In step S 560 , the gray scale/color balance is further modified, while in step S 570 , any other desirable modification techniques can be further applied to the modified image. Control then continues to step S 500  until the user determines the image is acceptable. 
     FIGS. 5A-5C are a flowchart outlining in more detail one exemplary embodiment of a method for modifying the image after automatic image enhancement has modified the image data of step S 300 . The process begins in step S 300  and continues to step S 310 , where the image data is analyzed for spatial modifications. Next, in step S 320 , a determination is made whether to modify the image data. If the image data is to be modified, control continues to step S 322 . Otherwise, control jumps directly to step S 330 . 
     In step S 322 , modifications to the image data can be determined. Then, in step S 324 , the modification weighting, amounts or patterns used can be recorded. Control then continues to step S 330 . 
     In step S 330 , the image data is analyzed for speckle removal. Next, in step S 340 , a determination is made whether to modify the image data. If the image data is to be modified, control continues to step S 342 . Otherwise, control jumps directly to step S 350 . 
     In step S 342 , speckle removal in the image data can be determined. Next, in step S 344 , the speckle removal weighting, amounts or patterns used can be recorded. Control then continues to step S 350 . 
     In step S 350 , the image data is analyzed for luminance modifications. Next, in step S 360 , a determination is made whether to modify the luminance of the image data. If the image data is to be modified, control continues to step S 362 . Otherwise, control jumps directly to step S 370 . 
     In step S 362 , modifications to the luminance of the image data can be determined. Then, in step S 364 , the luminance modification weighting, amounts or patterns used can be recorded. Control then continues to step S 370 . 
     In step S 370 , the image data is analyzed for black point/white point correction. Next, in step S 380 , a determination is made whether to modify the image data. If the image data is to be modified, control continues to step S 382 . Otherwise, control jumps directly to step S 390 . 
     In step S 382 , black point/white point corrections to the image data can be determined. Next, in step S 384 , the black point/white point correction weighting, amounts or patterns used can be recorded. Control then continues to step S 390 . 
     In step S 390 , the image data is analyzed for sharpness modifications. Next, in step S 400 , a determination is made whether to modify the image data. If the image data is to be modified, control continues to step S 402 . Otherwise, control jumps directly to step S 410 . 
     In step S 402 , sharpness modifications to the image data can be determined. Then, in step S 404 , the sharpness modification weighting, amounts or patterns used can be recorded. Control then continues to step S 410 . 
     In step S 410 , the image data is analyzed for other types of image data modifications. Next, in step S 420 , a determination is made whether to modify the other types of image data. If the other types of image data can be to be modified, control continues to step S 422 . Otherwise, control jumps directly to step S 429 . 
     In step S 422 , other types of image data modifications to the other types of image data can be determined. Next, in step S 424  the other types of image data modification weighting, amounts or patterns used can be recorded. Control then continues to step S 429 . 
     In step S 429 , the new weightings for modifying the image data are used to modify the image data. Control then continues to step S 430 , where control is returned to step S 500 . 
     The methods described herein are explained in linear fashion for clarity. It should be appreciated that the process may occur in a multitask, multithreaded or piped manner to enhance the speed and use of resources. Further, not all known methods for analysis and weighing have been discussed herein. Finally, the methods for analysis and weighting discussed herein need not all be included in the analysis and weighting methods described herein. 
     As used herein, the term “enhanced” connotes a condition where one picture condition is deemed better than another picture condition. It should be appreciated that it would always be possible to determine a better image. Similarly, the term “acceptable” connotes a condition where one value is deemed less desirable than another value. Again, it should be appreciated that it would always be possible to determine any different value. Accordingly, these terms are not intended to describe an ultimate or absolute condition. Rather, as recognized by one skilled in the art, these terms are intended to describe a condition that is relative to a desired level of accuracy. 
     In a second exemplary embodiment of the invention, the method described above may have increasing levels of user involvement in the modification of the image data. The user may be offered increasing levels of detail each time the image is modified. This allows the user to be trained by the system so that the user becomes proficient in image analysis and modification. This also allows a user that is experienced to fine-tune an image based on the user&#39;s preferences and time constraints. 
     For example, the initial modification may remove the standard amount of speckles according to automatic image enhancement techniques. The user may then force more speckle removal. As an additional step, the second embodiment may then offer the user the ability to choose the anomaly size, or the tone difference required for speckle analysis. Then, in addition iterations, more options may be offered to the user to fine tune the speckle removal algorithm. 
     By this embodiment, the algorithm may be taught user preferences, and the user may learn about image modification. Thus, when a user next modifies an image, the embodiment may apply the modification parameters applied previously as an initial modification. The modified degrees of modification may include new boundaries for automatic image enhancement, additional or modified local control variables, and minimum and maximum allowable modifications. 
     While this invention has been described in conjunction with the exemplary embodiments outlined above, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the exemplary embodiments of the invention, as set forth above, are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the invention.