Source: http://www.google.com/patents/US6987876?dq=7,117,286
Timestamp: 2017-11-22 13:34:54
Document Index: 178511686

Matched Legal Cases: ['art 102', 'art 102', 'art 102', 'art 102', 'art 102', 'art 102', 'art 102', 'art 102']

Patent US6987876 - System and methods for determining the settings of multiple light sources in ... - Google Patents
Systems and methods where a lighting configuration of a vision system is determined using a controllable lighting system and at least one image evaluation tool. The lighting configuration is usable to obtain a desired inspection image of at least one feature of a workpiece Base images are obtained using...http://www.google.com/patents/US6987876?utm_source=gb-gplus-sharePatent US6987876 - System and methods for determining the settings of multiple light sources in a vision system
Publication number US6987876 B2
Application number US 09/921,886
Also published as US6627863, US20020074480, US20020076096
Publication number 09921886, 921886, US 6987876 B2, US 6987876B2, US-B2-6987876, US6987876 B2, US6987876B2
Inventors Andrew D. Silber, Richard M. Wasserman
Patent Citations (31), Non-Patent Citations (2), Referenced by (20), Classifications (16), Legal Events (4)
System and methods for determining the settings of multiple light sources in a vision system
US 6987876 B2
Systems and methods where a lighting configuration of a vision system is determined using a controllable lighting system and at least one image evaluation tool. The lighting configuration is usable to obtain a desired inspection image of at least one feature of a workpiece Base images are obtained using actual illumination settings of the controllable lighting system. Simulated or synthetic sets image results are generated, based on base images and synthetic lighting configurations. The synthetic lighting configurations include at least one illumination setting which is different from the actual illumination settings used to obtain one or more component base images. A best actual or synthetic lighting configuration is chosen based on the best corresponding set of image results.
However, in some cases, the synthetic image may not accurately represent the image obtained using the lighting settings determined based on the synthetic image. Additionally, in many instances, although linear vision systems have great flexibility on how to light an object, there are no tools to help a user choose the best lighting solution. For example, if the user wishes to automatically create a part program using a computer aided design (CAD) file data, there is generally sufficient CAD information to move the stage to proper coordinates to successfully perform an auto-focus process. However, there may not be enough information from the CAD file data to achieve desired lighting configurations.
The vision system components portion 110 includes a stage 111 having a central transparent portion 112. A part 102 to be imaged using the vision system 100 is placed on the stage 111. The light from a number of different light sources 115–118 passes through a lens system 113 after illuminating the part 102, and possibly before illuminating the part 102, and is gathered by a camera system 114 to generate an image of the part 102. The light sources used to illuminate the part 102 include a stage light 115, a coaxial light 116, and/or a surface light, such as a ring light 117 or a programmable ring light (PRL) 118.
Each of the light sources 115–118, separately, or in combination, constitute a controllable lighting system. It should be appreciated that any one of the various light sources 115–118 described above can include a plurality of different colored light sources. That is, for example, the stage light 115 can include a red light source, a green light source and a blue light source. Each of the red, blue and green light sources of the stage light 115 will be separately driven by the power source 190, and may be considered as a separate light source in various embodiments of the systems and methods according to this invention.
In the vision system 100, a previous setting memory portion 142 stores previous settings for the various light sources 115–118 that were in place prior to the part program executor and/or generator 170 adjusting one or more of the light sources 115–118. A property memory portion 141 stores data identifying the light source or sources that will be adjusted to obtain the desired image quality, data defining the operation mode for operating the various light sources 115–118, data defining the image quality that is to be used as the metric for determining whether the identified light source(s) to be used to illuminate the part 102 need further adjusting, and data defining whether the image data in the regions of interest is to be filtered.
After beginning in step S1000, operation continues to step S2000, where a start-up routine is performed and the program for the vision system in FIG. 1 is initialized. During this step, in various exemplary embodiments, the vision system is operable to control over the light sources, including the height of any programmable ring light (PRL), by manual, semi-automatic or automatic means. In the case of manual control, in various exemplary embodiments a lighting tool is open to facilitate user control of the vision system lighting. A more detailed description of the start-up process will be described with reference to FIG. 5. Next, in step S3000, actual or base image data is acquired and stored, and synthetic image data is generated and stored, in the memory. A more detailed description of the base and synthetic image data acquiring and storing process will be described with reference to FIGS. 7–9. Operation then continues to step S4000.
In step S4000, one or more desired image characteristics are extracted or determined from the base and synthetic images. Here, captured and synthesized images are analyzed to extract one or more characteristics related to features and/or regions of interest of the image that are indicative of the quality of a particular lighting solution i.e., lighting vector or lighting configuration. A more detailed description of the image characteristic extraction process will be described with reference to FIG. 10. Next, in step S5000, the extracted characteristics from the base and synthetic images are analyzed. The analysis process will be described in more detail with reference to FIGS. 11–13. Then, in step S6000, in various exemplary embodiments, either a best image or an adequate image is determined based on the analyzed extracted characteristics from the base and synthetic images. The process for selecting the best image will be described in greater detail with reference to FIG. 14. Next, in step S7000, the lighting configuration of the determined best/adequate image is stored for future use to set the various light sources in the vision system when repeating a substantially similar imaging or inspection operation. Finally, in step S8000, operation ends.
Various exemplary embodiments of the systems and methods according to this invention are described herein as generating a simulated image as the basis for an image result which is evaluated. However, it should be appreciated that the image result may be determined from a variety of data representations not generally recognized as a simulated image. Provided that such data representations are usable to provide one or more image results which are usable to determine whether a particular lighting configuration provides a best or adequate image according to the systems and methods of this invention, such data representations are included in the scope of the terms “simulated image” or “synthetic image” or “synthesized image”, and thus are within the scope of the systems and methods according to this invention. It should be further appreciated that, in various other exemplary embodiments, depending on the image results to be determined, an objective function, an image characteristic extraction, an image metric or another form of image result, may be determined directly from the base images and a governing lighting vector without needing to represent or generate a simulated image as a recognizable intermediate step.
In step S2020, since the CAD file data is available, a determination is made whether to use the CAD file data, based on the convenience of use of the CAD data, for example. If the CAD file data will not be used, then operation jumps to step S2040. Otherwise, if the CAD file data will be used, then operation proceeds to step S2030. In step S2030, one or more of the image capture parameters described below with respect to steps S2040–S2070 are determined or extracted from the CAD file data and information representing the vision system configuration. The one or more of the image capture parameters are then used in the one or more corresponding steps S2040–S2070. Operation then continues to step S2040.
In step S2070, the primary purpose of the current operation is determined and the analysis parameters are set for the critical features and/or regions of interest corresponding to the current operation. In various exemplary embodiments, the primary purpose of the current operation is determined by the type of image analysis tool selected by an operator or an automatic program that is operating the vision system. Furthermore, in various exemplary embodiments, some or all of the analysis parameters are determined based on the placement and “training” of the selected image analysis tool. An exemplary process for setting the analysis parameters when the current operation includes analyzing an edge location will be described in greater detail with reference to FIG. 6. Operation then proceeds to step S2080, where operation returns to step S3000.
In step S2075, a “start-up” confidence value for the edge location is set, if applicable. The confidence value defines whether the labeled edge location is well defined or not and in various exemplary embodiments includes consideration of both fabrication tolerances for the edge position on a part and the clarity and regularity of the edge line in the image. For example, if the expected fabrication tolerance is small, and the expected or observed clarity and regularity of the edge line in the image are good, then the confidence value for the edge location is set at a high level, indicating that the edge location should be a dominant factor in determining a desirable lighting vector. Conversely, if the expected fabrication tolerance is crude, and the expected or observed clarity and regularity of the edge line in the image are poor due to material transparency, rough surface finish, large edge radius, or the like, then the confidence value for the edge location is set at a low level, indicating that the edge location is a poor factor for determining a desirable lighting vector. As explained further below, the confidence value is used according to the systems and methods of this invention to determine whether the associated type of image result, that is, the edge location in this case, is expected to have good reliability as an image result and should be accorded high importance in determining a desirable lighting configuration, or poor reliability as an image result and should be accorded lower importance in determining a desirable lighting configuration. In various exemplary embodiments, the start-up confidence value for the edge location is set by the operator of the vision system by input to an edge tool user interface, or by manually or automatically indicating various points along the edge through a suitable user interface and operating the vision system to determine the quality of a best fit to the points, or based on the CAD data operations of step S2030. Use of the confidence value for the edge location in various exemplary embodiments according the systems and methods of this invention is described in detail further below. Operation the continues with step S2076.
In step 2076 any additional “non-edge” analysis parameters which are useful for the purpose of determining a desirable lighting vector for the current operation are set. For example, surface height determination and surface finish determination are enhanced by proper lighting. In such a case, suitable analysis parameters such as confidence values related to the surface height determination and surface finish determination, as well as other analysis parameters will be apparent to one skilled in the art, may be used according to various embodiments of the systems and methods according to this invention. Operation then proceeds to step S2077, where operation returns to step S2080.
FIG. 7 is a flowchart outlining in greater detail one exemplary embodiment of a method for acquiring and storing the image data to be used in accordance with this invention of step S3000. Beginning in step S3000, operation proceeds to step S3100, where base image data is acquired and stored. Base image data, is, in general, actual images that have been captured and stored using the vision system with a defined lighting vector corresponding to each base image. Then, in step S3200, synthesized image data is generated, or acquired, and stored. The synthesized images can be generated, for example, by combining two or more of the base images to create a synthesized image. Next, in step S3300, a determination is made whether more image data needs to be acquired and stored. It should be appreciated that in various exemplary embodiments, iterations of the acquiring and generating steps S3100–S3200 are governed by the step S3300, in order to gather all the required lighting/image data. However, in other exemplary embodiments, the steps S3100 and S3200 may gather a sufficient or exhaustive set of data in one iteration. It should be appreciated that the step S3300 is not necessary in such other embodiments. If, in step S3300, more image data needs to be acquired and stored, then operation jumps back to step S3100. Otherwise, if no more image data is to be acquired and stored, then operation proceeds to step S3400, where operation returns to step S4000.
FIG. 8 is a flowchart outlining in greater detail one exemplary embodiment of the method for acquiring and storing base image data of step S3100. Beginning in step S3100, operation proceeds to step S3110, where the first or next lighting configuration, also referred to as a lighting vector, is selected as a current lighting configuration. As previously discussed with reference to the vision system shown in FIG. 1, each different lighting configuration includes a different setting for each of the light sources 115–118. In particular, each lighting configuration indicates whether each particular light source is on or off, and, if any light source or portion of the programmable ring light is on, the height of the programmable ring light above the stage 102. Next, in step S3120, a base image is acquired when the part 102 is illuminated according to the current lighting configuration. Then in step S3130, the acquired base image and corresponding lighting configuration is stored in memory. Operation then proceeds to step S3140.
More generally, it should be appreciated that the incorporated 187 application includes alternative embodiments useable for the various base image acquiring and synthetic image acquiring operations of FIGS. 7–9, either separately or combination, as will be apparent to one skilled in the art.
Beginning in step S4000, operation proceeds to step S4010, where a first/next one of the actual and synthetic lighting configurations is selected as the current lighting configuration. Then, in step S4020, one or more parameters for learning and/or running an edge analysis tool are determined or defined. In various exemplary embodiments, these parameters can be determined or defined using a dual area contrast tool such as that shown in FIG. 3, for example, in determining the standard deviations of the pixel intensities of two or more regions of interest near an edge, categorizing a largest and smallest standard deviation, and then setting the scanning direction based on the standard deviation results so that the process is performed from a smooth side of the image to a rough side of the image. In various other exemplary embodiments, step S4020 is analogous to and/or includes one or more of the operations described in relation to the steps S2072–2075 of FIG. 6. Other exemplary parameters, as well as image analysis tools operable to perform various edge tool operations and dual-area contrast tool operations usable for characteristics extraction are also available in commercial vision machines and software, such as the Quick Vision series of vision inspection machines and QVPAK software available from Mitutoyo America Corporation (MAC), located in Aurora, Ill. Operation then proceeds to step S4030.
In step S4030, an edge detection is performed for the selected current lighting configuration and the parameters of steps S4010 and S4020. For example, in various exemplary embodiments, an edge tool is used to analyze the intensity profile of an edge for five separate scan lines across the edge in a “learn” mode of the tool. In step S4030, the edge detection can be performed to analyze and store, or “learn”, various characteristics and/or features of the scan line intensity profiles which characterize the edge, so that the edge tool can be run automatically using the learned parameters. Various exemplary characteristics and/or features of the scan line intensity profiles include the intensity change across the edge, the direction of intensity increase across the edge, the number or proportion of scan lines across the edge that include intensity changes above a threshold value, and the like. In one exemplary embodiment, the mean value of each characteristics or feature is the value learned or stored as the basis for actual “run-time” edge measurements performed later. Operation then proceeds to step S4035.
In step S4040, it is determined whether the edge detection operation of step S4030 detected a valid edge. For example, the characterizations in step S4030 may indicate that the edge so “weak” that an edge measurement based on the image according to the current lighting configuration would be either impossible or too uncertain. If it is determined that a valid edge was not detected, then operation jumps to step S4090. Otherwise, operation proceeds to step S4050.
In step S5170, the position score and shape score can be normalized relative to various rank scores of steps S5120–S5170, and in accordance with the design of the classifier used to determine the best lighting/image. Then, in step S5180, a determination is made as to whether all lighting configurations have been selected as the current configuration. If all lighting configurations have not been selected, then operation returns to step S5110 where the process is repeated for the next lighting configuration. The operation for determining the metrics for all lighting configurations can continuously update a internal list for all ranked images. For example, each time an image under a lighting configuration is ranked, the operation can automatically position the ranked image in sequenced order, i.e., rank from highest to lowest values, as compared to all of the other ranked images. If it is determined in step S5180 that all of the lighting configurations have been selected, the operation returns to step S5200 to apply the classifier.
In step S5215, a determination is made whether the current confidence score indicates that the edge location is well defined. If, in step S5215, the confidence score indicates a well defined edge, operation proceeds to step S5220, where the membership functions to be used by the classifier are set to a “high confidence” configuration appropriate for classifying images including a well-defined edge, and operation continues to step S5230. However, if, in step S5220, the confidence score does not indicate a well defined edge, operation proceeds to step S5225, where the membership functions to be used by the classifier are set to a “low confidence” configuration appropriate for classifying images including a poorly-defined edge and operation continues to step S5230.
In step 85240, membership functions are applied for any remaining metrics/image characteristics to obtain or determine the individual fuzzy classifier values that represent the relative importance of each metric/image feature. FIG. 15 shows various exemplary embodiments of membership functions that are usable in step S5240 to determine the fuzzy classifier values for various different types of scores or ranks. Next, in step S5245, an overall classifier value is determined for the current lighting configuration by combining the individual fuzzy classifier values obtained in steps S5230–5240 using a fuzzy “AND” operation. That is, the lighting configuration that has a set of image results including a close position score or a top edge rank, a close shape score or a top shape rank, a strong edge score, a low standard deviation score in at least one region of interest, and a large value of TN, will be evaluated as corresponding to a favorable lighting configuration. Then, in step S5257, the results of the determined overall classifier value are stored. Operation then continues to step S5255.
The inventors have found that classifiers including a set of most or all of the image results included in FIGS. 12–13 generally approximate or exceed the accuracy and robustness of an expert human vision system operator. However, it should also be appreciated that a classifier can be based on a set of one or more metrics/image features in various exemplary embodiments. For example, when the confidence value is high, a set consisting of the position rank alone may suffice for many applications. Thus, it should be understood that the particular steps outlined in FIG. 13 are illustrative only and should not be considered limiting of the scope of step S5200 of FIG. 11.
In step S6070, a base image is captured using the illumination setting determined in steps S6040–S6060 to illuminate the part 102. Then, in step S6080, one or more characteristics are extracted from the captured base image using the operation of step S4000, and stored. That is, the characteristic extraction process performed in step S6080 is the same characteristic extraction process shown in FIG. 10. Next, in step S6090, a determination is made whether the first light source is at the opposite end of the its illumination range. If, in step S6090, the first light is not at the opposite end of its illumination range, operation proceeds to step S6100. Otherwise, in step S6090, if the first light is at the opposite end of its illumination range, operation jumps to step S6110. In step S6100, the illumination setting of the first light source is changed by a predetermined amount. Operation then jumps back to step S6060. In contrast, in step S6110, a determination is made whether the height of the programmable ring light is at the opposite end of the refined height range. If, in step S6110, the programmable ring light is not at the opposite end of the refined height range, then operation proceeds to step S6120 and the height of the programmable ring light is changed within the refined height range. Operation then jumps back to step S6060. Otherwise, in step S6110, if the programmable ring light is at the opposite end of the refined height range, then operation continues to step S6130.
In step S6130, the classifier is applied to the new series of base images captured and analyzed in steps S6010–S6120. The classifier process used in step S6130 is the same used in step S5200. Next, in step S6140, if it useful in a particular application, the best base image, corresponding to the best lighting configuration determined by the classifier from the new series of base images, is displayed. Then, in step S6150, operation returns to step S7000 where the lighting configuration/vector settings corresponding to the best actual base image are stored for future use.
In FIG. 1, the control system portion 120 is, in various exemplary embodiments, implemented using a programmed general purpose computer. However, the control system portion 120 can also be implemented on a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, a hardwired electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PLD, PLA, FPGA or PAL, or the like. In general, any device, capable of implementing a finite state machine that is in turn capable of implementing the flowcharts shown in FIGS. 4–14 can be used to implement the control system portion 120.
US20050002555 * Apr 28, 2004 Jan 6, 2005 Fanuc Ltd Image processing apparatus
US20050094867 * Oct 30, 2003 May 5, 2005 Jones James D.Jr. Method of inspecting threaded fasteners and a system therefor
JPH0511261A Title not available
JPH0519294A Title not available
JPH1026771A Title not available
JPH04265946A Title not available
JPH05333373A Title not available
JPH09325363A Title not available
JPH10232408A Title not available
JPS61105582A Title not available
1 Leon, F. Puente, "Enhanced imaging by fusion of illumination series", Proc. of the European Symposium on Lasers and Optics in Manufacturing, Munich, FR Germany, Jun. 20, 1997, pp. 1-12.
2 U.S. Appl. No. 09/736,187, filed Dec. 15, 2000, Wasserman.
U.S. Classification 382/152, 250/205
International Classification G06K9/00, G01B11/00, G01J1/32, G06T1/00, G01N21/27, G01N21/88
Cooperative Classification G01N2021/8845, G01N21/8806, G01N2021/8835, G01N21/274, G01J1/32
European Classification G01N21/27E, G01J1/32, G01N21/88K
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SILBER, ANDREW D.;WASSERMAN, RICHARD M.;REEL/FRAME:012060/0199