Abstract:
Multi-modal soldering inspection system comprises means for automatic inspection involving imaging means for acquiring solder joint images, mode-selecting means for selecting auto mode, review mode, or eye mode, means for integrating inspection data obtained in the auto and the review modes, replay means for replaying images acquired with the imaging means to display them, input means for inputting the review data, and storage for storing the inspection data.  
     The multi-modal soldering inspection system works in two or three modes wherein, at first in the auto mode, the system performs automatic inspection of solder joints of a printed circuit board (PCB) and stores addresses and/or images of suspect points tentatively judged as defective in the automatic inspection, next in the review mode, the system reacquires or replays images of the suspect points to display them with markers superimposed thereon so that an operator can carry out the verification by looking at the displayed images, and finally in the eye mode, the system displays a PCB map indicating addresses of points for visual inspection so that an operator can carry out direct visual inspection of the PCB referring to the map and accomplish the total inspection by inputting the visual inspection data.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    The instant invention relates to a multi-modal system for solder joint inspection of a printed circuit board (PCB) packaged with electronic parts. The system works not only in auto mode for automatic inspection but also in review mode for verification of the auto mode data, which can be followed by additional eye mode for direct visual inspection by humans.  
           [0002]    Visual inspection of electronic products has been done with naked eyes or using a stereoscopic microscope, but with advanced miniaturization of parts, human inspection has become increasingly more difficult and more unreliable.  
           [0003]    Hence, demands for the automation have emerged nowadays to raise reliability and to save human cost.  
           [0004]    Many kinds of apparatuses for automatic soldering inspection of electronic parts on PCBs have been developed and already used in assembly lines.  
           [0005]    However, they had still problems mainly in their cost-performance.  
           [0006]    They were expensive as compared with their ability of saving human cost.  
           [0007]    Users often requested perfect performance in automation where correlation between automatic inspection and human discrimination marks a hundred percent coincidence. Makers also often tried to attain it by making long-term, consuming efforts of highly-skilled and high-cost engineers.  
           [0008]    Because improvement of the recognition capability might require from several days to several months the length of which depends upon user&#39;s production environment, the machine prices became more expensive.  
           [0009]    Makers could not neglect the risk of increasing costs.  
           [0010]    In a matter of fact, a human brain recognizes an object in a way quite different from that of a computer.  
           [0011]    There is no causality between them, so a confusion matrix in-between expresses only their statistical correlation.  
           [0012]    From a theoretical point of view, a hundred percent coincidence could never be obtained and fine tuning of computer parameters for recognition intending to attain it will necessarily fail.  
           [0013]    From reasons mentioned above, the embodiments of the present invention intend to solve the problem of recognition in the conventional automatic soldering inspection machines by presenting a system where no perfect coincidence of recognition is demanded.  
         OBJECTS OF THE INVENTIONS  
         [0014]    An object of the present invention is to provide a multi-modal soldering inspection system which works in two modes wherein, in auto mode, the system performs automatic inspection of solder joints of a printed circuit board (PCB) and stores addresses of suspect points which have been tentatively judged as defective in the automatic inspection and, in review mode, the system reacquires images of the suspect points and display them with markers superimposed thereon so that an operator can carry out the verification by looking at the displayed images and accomplish the total inspection by inputting the reviewed data.  
           [0015]    Another object of the present invention is to provide a multi-modal soldering inspection system which works in two modes wherein, in auto mode, the system performs automatic inspection of solder joints of a PCB and stores images and addresses of suspect points which have been tentatively judged as defective in the automatic inspection and, in review mode, the system replays the stored images and display them with markers superimposed thereon so that an operator can carry out the verification by looking at the displayed images and accomplish the total inspection by inputting the reviewed data.  
           [0016]    Yet another object of the present invention is to provide a multi-modal soldering inspection system which works in three modes wherein, in auto mode, the system performs automatic inspection of solder joints of a PCB and stores images and addresses of suspect points which have been tentatively judged as defective in the automatic inspection, in review mode, the system replays images of the suspect points and display them with markers superimposed thereon so that an operator can carry out the verification by looking at the images and input the reviewed data, and, in eye mode, the system displays a PCB map indicating addresses of points for visual inspection so that an operator can carry out direct visual inspection of the PCB referring to the map and accomplish the total inspection by inputting the visual inspection data.  
         SUMMARY OF THE INVENTION  
         [0017]    In accordance with a feature of the present invention, a multi-modal soldering inspection system comprises means for automatic inspection involving imaging means for imaging solder joints, mode-selecting means for selecting auto mode or review mode, means for integrating inspection data obtained in the auto and the review modes, storage for storing the inspection data, display means for displaying images reacquired with the imaging means, and input means for inputting the reviewed data, enabling dual mode inspection consisting of auto mode performing automatic inspection of solder joints of a printed circuit board (PCB) and review mode reacquiring images of the suspect points which has been judged tentatively in the auto mode as fault soldering and displaying them so that an operator can carry out the verification and accomplish the total inspection by inputting the reviewed data.  
           [0018]    In accordance with another feature of the present invention, a multi-modal soldering inspection system comprises means for automatic inspection involving imaging means for imaging solder joints, mode-selecting means for selecting auto mode or review mode, means for integrating inspection data obtained in the auto and the review modes, storage for storing the inspection data, replay means for replaying images stored with the storage and displaying them, and input means for inputting the reviewed data, enabling dual mode inspection consisting of auto mode performing automatic inspection of solder joints of a PCB and storing images of suspect points which has been judged tentatively in the auto mode as fault soldering and review mode replaying the stored images and displaying them so that an operator can carry out the verification and accomplish the total inspection by inputting the reviewed data.  
           [0019]    In accordance with yet another feature of the present invention, a multi-modal soldering inspection system comprises means for automatic inspection involving imaging means for imaging solder joints, mode-selecting means for selecting auto mode, review mode, or eye mode, means for integrating inspection data obtained in the auto and the review modes, storage for storing the inspection data, replay means for replaying images stored with the storage and displaying them, and input means for inputting the reviewed data and the visual inspection data, enabling triple mode inspection consisting of auto mode performing automatic inspection of solder joints of a PCB and storing images of suspect points which has been judged tentatively in the auto mode as fault soldering, review mode replaying the stored images and displaying them so that an operator can carry out the verification, and eye mode displaying a PCB map indicating addresses of points for visual inspection so that an operator can carry out direct visual inspection of the PCB referring to the map and accomplish the total inspection by inputting the visual inspection data. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0020]    [0020]FIG. 1 illustrates schematically optical constitution of an active vision system involved in a multi-modal soldering inspection system of an embodiment of the present invention.  
         [0021]    [0021]FIG. 2 is a diagram showing geometrical relationships between the visual axis of the active vision system and an object for inspection involved in a multi-modal soldering inspection system of an embodiment of the present invention.  
         [0022]    [0022]FIG. 3 is a schematic diagram showing system constitution and horizontally turned poses of a PCB involved in a multi-modal soldering inspection system of an embodiment of the present invention.  
         [0023]    [0023]FIG. 4 is a flow diagram showing steps of teaching involved in a multi-modal soldering inspection system of an embodiment of the present invention.  
         [0024]    [0024]FIG. 5 is a flow diagram showing steps of auto mode inspection involved in a multi-modal soldering inspection system of an embodiment of the present invention.  
         [0025]    [0025]FIG. 6 is a flow diagram showing steps of review mode inspection involved in a multi-modal soldering inspection system of an embodiment of the present invention.  
         [0026]    [0026]FIG. 7 is a schematic diagram showing system constitution involved in a multi-modal soldering inspection system of second embodiment of the present invention.  
         [0027]    [0027]FIG. 8 is a flow diagram showing steps of auto mode inspection and review mode inspection involved in a multi-modal soldering inspection system of second embodiment of the present invention.  
         [0028]    [0028]FIG. 9 is a flow diagram showing steps of auto mode inspection and review mode inspection involved in a multi-modal soldering inspection system of third embodiment of the present invention.  
         [0029]    [0029]FIG. 10 is a flow diagram showing steps of teaching involved in a multi-modal soldering inspection system of fourth embodiment of the present invention.  
         [0030]    [0030]FIG. 11 is a flow diagram showing steps of auto mode inspection, review mode inspection, and eye mode inspection involved in a multi-modal soldering inspection system of fourth embodiment of the present invention.  
         [0031]    [0031]FIG. 12 is a flow diagram showing steps of teaching involved in a multi-modal soldering inspection system of fifth embodiment of the present invention.  
         [0032]    [0032]FIG. 13 is a flow diagram showing steps of auto mode inspection, review mode inspection, and eye mode inspection involved in a multi-modal soldering inspection system of fifth embodiment of the present invention.  
         [0033]    [0033]FIG. 14 illustrates schematically sequential display and multiple-frame display of suspect images in review mode in a multi-modal soldering inspection system of sixth embodiment of the present invention.  
         [0034]    [0034]FIG. 15 is a schematic diagram showing system constitution involved in a multi-modal soldering inspection system of seventh embodiment of the present invention.  
         [0035]    [0035]FIG. 16 is a schematic diagram showing system constitution involved in a review station of eighth embodiment of the present invention.  
         [0036]    [0036]FIG. 17 illustrate schematically a top view, a front view, a side view, and a perspective view of a solder joint produced between an LSI electrode and a printed pad pattern. 
     
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS  
       [0037]    Prior art of an active vision inspection apparatus with mirror motion was disclosed in Japan Patent Application No. Tokuganhei 11-285067 filed by the inventor.  
         [0038]    Following the prior art, the present invention intends to introduce an inspection system for parts-mounted and soldered PCBs wherein angular bird&#39;s eye perspective view images are utilized for automatic or visual recognition.  
         [0039]    Optical constitution of an active vision system involved in embodiments of the present invention is illustrated schematically in FIG. 1.  
         [0040]    Geometry of the visual axis of the active vision system and an object for inspection is depicted in FIG. 2.  
         [0041]    The system constitution and horizontally turned poses of a PCB involved in first embodiment of the present invention are shown schematically in FIG. 3.  
         [0042]    Before explaining first embodiment, the reason why that constitution is adopted for the active vision system of the present invention is to be described.  
         [0043]    A close-up lens or a microscope objective is generally used for obtaining images with resolution higher than squares of several μm per pixel, but the lens focal length is too short for active vision application wherein the viewing axis should be directed at all directions.  
         [0044]    Therefore, embodiments of the present invention are equipped with a telescopic optical system composed of an active objective having long lens focal length and an ocular.  
         [0045]    Objective L 1  shown in FIG. 1 is active.  
         [0046]    Object P o  positions in variable distances.  
         [0047]    Focusing at object P o  is achieved by moving objective L 1  forward or backward along the optical axis.  
         [0048]    Objective L 1  forms aerial figure P a  and ocular L 2  relays it to zoom lens L 3.    
         [0049]    The magnification is adjusted by zooming of zoom lens L 3  and then zoom lens L 3  projects object figure P i  onto imaging element I.  
         [0050]    The optical axis is held horizontal and active mirror M is placed in front of objective L 1.    
         [0051]    Two pivots of revolution (not shown) hold active mirror M so that the extended virtual axes may cross orthogonally each other on the mirror surface.  
         [0052]    Active mirror M is deflected in azimuth and/or inclination to point the optical axis at object P o.    
         [0053]    The geometrical model is to be mentioned in two steps—the first is a step where objective L 1  forms aerial figure P a  and the second is a step where ocular L 2  relays figure P a.    
         [0054]    An active vision system using mirror reflection is optically equivalent to a system with an off-set virtual camera rotating around the centroid of the mirror.  
         [0055]    [0055]FIG. 2 shows an orthogonal coordinate system (x a , y a , z a ) attached to aerial figure P a  whose origin is the center of the image plane of P a  and the z a  axis is the viewing axis.  
         [0056]    As is shown in FIG. 1, the viewing axis is divided into two parts.  
         [0057]    A part from object P o  to mirror M is ‘active’ in direction and accordingly variable in length.  
         [0058]    Another part from mirror M to imaging element I is horizontal and constant in length.  
         [0059]    The origin O w  of a world coordinate system (x w , y w , z w ) is the center of revolution of mirror M.  
         [0060]    The x w  axis is horizontal and vertical to the y w  axis which is parallel to the horizontal optical axis.  
         [0061]    The z w  axis is perpendicular and object plane P is held horizontal.  
         [0062]    Rotation of the normal to mirror M by angle φ about the z w  axis produces azimuth deflection of the viewing axis by angle 2φ.  
         [0063]    Rotation of the normal to mirror M by angle θ about the x w  axis produces inclination deflection of the viewing axis by angle 2θ.  
         [0064]    Assuming that light from object P o  goes straightforward across the origin O w  with no mirror reflection, we image virtual aerial image P a ′ at z a ′=−d that is equivalent to aerial image P a.    
         [0065]    Motion of active mirror M produces rotation of the normal about the respective axes of the world coordinate system as expressed in a virtual coordinate system (x a ′, y a ′, z a ′) which is attached to virtual image P a  (Expression 1);  
         [0066]    Expression 1  
         (           x   a   ′               y   a   ′               z   a   ′           )     =         R   θ          R   ϕ            R   φ          (           x   w               y   w               z   w           )         +     (         0           0           d         )                 R   θ     =     (         1       0       0           0         cos                 2      θ             -   sin                   2      θ             0         sin                 2      θ           cos                 2      θ           )               R   ϕ     =     (         1       0       0           0       1       0           0       0       1         )                 R   φ     =     (           cos                 2      φ             -   sin                   2      φ         0             sin                 2      φ           cos                 2      φ         0           0       0       1         )            
                           
 
         [0067]    (x w , y w , z w ): The world coordinate system  
         [0068]    (x a , y a , z a ): The virtual aerial image coordinate system  
         [0069]    R θ : Rotation matrix about the x w  axis  
         [0070]    R φ : Rotation matrix about the y w  axis  
         [0071]    R φ : Rotation matrix about the z w  axis,  
         [0072]    where R θ , R φ , R φ  are the rotation matrices about the x w  axis, about the y w  axis, and about the z w , axis, respectively.  
         [0073]    As is shown in FIG. 2, the aperture of objective L 1  produces a visual cone around the viewing axis, projecting an elliptic view field onto object plane P.  
         [0074]    Azimuth angle   or inclination angle θ of active mirror M moved so as to point the viewing axis at a world coordinate point (x w , y w , z w ) is given in the following Expression 2.  
         [0075]    Expression 2  
             φ   =                  1   2        arctan                     x   w       y   w                     θ   =                  1   2        arctan                     y   w       z   w                                     
 
         [0076]    Moving active mirror M only in azimuth under fixed inclination forms a circular cone of the viewing axis in the world coordinate system whose vertex is the origin O w  and the axis is the y w  axis.  
         [0077]    As the result, changes in azimuth by angle φ about the z w  axis for a given inclination angle θ draw a conic section on object plane P  
         [0078]    The viewing angle α pointing to object P o  is given in the following Expression 3, where h is the vertical distance from the origin O w  to object plane P and l is the path length from the origin O w  to object P o.    
         [0079]    The locus on P is a hyperbola because P is parallel to the axis of the circular cone.  
         [0080]    Expression 3  
             α   =                arctan              x   w   2     +     y   w   2         h                     z   w     =              h               l   =                    x   w   2     +     y   w   2     +     h   2                                     
 
         [0081]    Expression 4 is the lens formula of objective L 1 , where u is the distance to the object point, v is the distance to the image point, and f is the lens focal length shown in FIG. 1.  
         [0082]    Expression 4  
           1   u     +     1   v       =     1   f                           
 
         [0083]    In accordance with perspective projection, magnification m a  of aerial image P a  or magnification m a ′ of virtual aerial image P a ′ is given in Expression 5.  
         [0084]    Expression 5  
         m   a     =       m   a   ′     =     v   u                             
 
         [0085]    Light from ocular L 2  passes through zoom lens L 3  and hits the surface of imaging element I as is shown in FIG. 1.  
         [0086]    Many reports on calibration of zoom lenses have accorded in a point that the lens model depends upon its special lens constitution including the distance to the image point.  
         [0087]    As for a model of zoom lens L 3 , embodiments of the present invention utilize an empirical function obtained from experiments on magnifications and zooming pulses required therefor.  
         [0088]    In embodiments of the present invention, magnification m a  of aerial image P a  vanes m accordance with the distance to the object point, u.  
         [0089]    To attain constant magnification m i    
         [0090]    which is to be irrespective of the focal length, the zooming magnification m b  is automatically adjusted according to the original magnification m a  (Expression 6).  
         [0091]    Expression 6  
         
       m 
       a 
       m 
       =m 
       l 
       =const.  
     
         [0092]    Thus, the active vision system of the present invention is able to acquire “bird&#39;s eye perspective views” of object P o  by looking down at P o  present on object plane P  
         [0093]    Next, the object pose is to be mentioned.  
         [0094]    When object P o  turns about the axes of an object centered coordinate system (x o , y o , z o ) by respective angles ξ, η, and ζ, rotations about the corresponding axes of the world coordinate system are given in the respective rotation matrices of R ξ , R η , and R ζ  (Expression 7).  
         [0095]    Expression 7  
           (           x   o               y   o               z   o           )     =         R   ξ          R   η            R   ζ          (           x   w               y   w               z   w           )         +     (         0           0             -   l           )         ,     
            R   ξ     =     (         1       0       0           0         cos                 ξ             -   sin                   ξ             0         sin                 ξ           cos                 ξ           )                 R   η     =     (           cos                 η         0         sin                 η             0       1       0               -   sin                   η         0         cos                 η           )                 R   ζ     =     (           cos                 ζ             -   sin                   ζ         0             sin                 ζ           cos                 ζ         0           0       0       1         )            
                           
 
         [0096]    Therefore, the object&#39;s orientation with respect to the view angle of the active vision system is expressed in Expression 8.  
         [0097]    Expression 8 
         (           x   a   ′               y   a   ′               z   a   ′           )     =         R   0          R   ϕ          R   φ          R   ξ   T          R   η   T            R   ζ   T          (           x   o               y   o               z   o           )         +     (         0           0             d   -   l           )                             
 
         [0098]    (x w , y w , z w ): The world coordinate system  
         [0099]    (x o , y o , z o ): The object coordinate system  
         [0100]    R ξ : Rotation matrix about the x w  axis  
         [0101]    R η : Rotation matrix about y w  axis  
         [0102]    R ζ : Rotation matrix about z w  axis  
         [0103]    Expression 8 gives a general relationship between a viewing angle of an active vision system and an object pose. A three-dimensional object exhibits its external feature better or worse depending upon its pose with respect to the viewer.  
         [0104]    In embodiments of the present invention, the stage is equipped with a PCB pose controller able to set a PCB in a pose presenting angular appearances of part electrodes where solder joints may expose themselves best to the vision system.  
         [0105]    For inspection of a PCB mounted with comparatively flat parts such as LSIs or chips, the stage is equipped with a turntable.  
         [0106]    A PCB which is carried in from a manufacturing line is turned with the turntable horizontally about the centroid of the PCB.  
         [0107]    The turn as shown in FIG. 3 is given in ζ rotation of Expression 7. Angles of turn in usual inspection status are ξ=η=0° and ζ=0°, or −135°.  
         [0108]    The turned PCB presents an “angular perspective view” of a part including front and side scenes of the electrode(s).  
         [0109]    Combination of a bird&#39;s eye perspective view with an angular view of an electrode in embodiments of the present invention composes an angular bird&#39; eye perspective view as is given in Expression 8 wherein the top, the front, and the side planes are seen at once (see FIG. 17).  
         [0110]    A study of information theory on polyhedral objects has proved that an image of a cube obtained in a direction viewing most planes with nearly uniform areas gives its maximal morphological information.  
         [0111]    Most electrodes of surface mount parts are cubical in shape.  
         [0112]    Therefore, morphological information of a soldered electrode image is also maximal in an angular bird&#39;s eye perspective view.  
         [0113]    Appearance of the front solder or the side solder reflects quantity, wettability, or a lifted lead and that of the top solder reflects excess solder or abnormal soldering.  
         [0114]    Next, a device for illuminating a PCB during inspection is to be mentioned.  
         [0115]    As the view axis is active and the PCB pose and position are variable, embodiments of the present invention are equipped with white light sources widely distributed over the PCB so as to catch lights reflected spectrally from solder surfaces on the PCB.  
         [0116]    As for teaching PCB information, inspection point addresses are taught utilizing part-mount address data, part shape data, and computer aided design (CAD) data.  
         [0117]    Also are taught the centroid of the PCB and its poses during inspection.  
         [0118]    Next, layout of inspection areas is to be mentioned.  
         [0119]    Embodiments of the present invention are equipped with a program for layout of inspection areas.  
         [0120]    Using part-mount and soldering point addresses of a PCB, pleural soldering points are automatically grouped in an inspection area, which is then enclosed with a rectangular frame.  
         [0121]    The active vision system images the inspection area and displays it on a monitor screen.  
         [0122]    The rectangular frame is metamorphosed into a trapezoid in accordance with Expression 8 for real angular bird&#39;s eye transformation, and then it is superimposed on the displayed real image.  
         [0123]    Watching the screen, an operator manually modifies the trapezoidal area to obtain suitable size and shape.  
         [0124]    Next, evaluating soldering quality is to be mentioned.  
         [0125]    An inspection system of an embodiment of the present invention carries a program for automatically extracting inspection points, a program for orthogonal transformation of perspective images, a program for calculating discriminative parameters used for evaluating part-mount and soldering quality, and an algorithm for evaluation.  
         [0126]    In the teaching mode, an embodiment of the present invention automatically extracts inspection points by processing inspection area image signals.  
         [0127]    In the inspection mode, an embodiment of the present invention morphs perspective images of the inspection points into orthogonal style images and calculates discriminative parameters therefrom.  
         [0128]    Then, an embodiment of the present invention evaluates part-mount and soldering quality by feeding the discriminative parameters into the algorithm for evaluation.  
         [0129]    Based on the evidences mentioned above, constitution and performance of a multi-modal soldering inspection system of an embodiment of the present invention are to be described referring to a schematic diagram showing system constitution (FIG. 3) and schematic illustrations depicting PCB poses with respect to the viewing angle.  
         [0130]    [0130]FIG. 3 demonstrates PCB  1  and electronic part  2  mounted thereon.  
         [0131]    PCB  1  is held with a stage which is equipped with a turntable (not shown). In top views shown below the side view in FIG. 3, PCB  1  denotes an original pose, PCB  1 ′ denotes a pose after horizontal turn by 45°, and PCB  1 ″ denotes a pose after horizontal turn by −135°.  
         [0132]    Active vision system  3  and active mirror  4  are positioned over PCB  1 . Active vision system  3  is composed of imaging device  3   a  and active optical system  3   b.    
         [0133]    Active vision system  3  and active mirror  4  are connected to control system  5  which consists of subunit  6  for active mirror control, subunit  7  for active lens system control, subunit  8  for imaging control, unit  9  for integrated control of subunits  6 ˜ 8 , unit  10  for control of the total system, unit  11  for image processing, and unit  12  for storage.  
         [0134]    Units  9 ,  10 ,  11 , and  12  interact each other via bus  17 .  
         [0135]    Control system  5  is connected to input unit  13 , output unit  14 , communication unit  15 , and display unit  16 .  
         [0136]    Steps of teaching are to be mentioned according to a flow diagram shown in FIG. 4.  
         [0137]    First, an operator teaches the geometry of active optical system  3   b  and the stage (not shown) (ST  11 ) and also the position and the pose of the PCB (ST  12 ). Next, utilizing part mount data, part shape data, or PCB CAD data, he/she teaches the addresses to which parts are to be mounted and the electrode addresses corresponding to soldering inspection points (ST  13 ).  
         [0138]    An inspection system of an embodiment of the present invention automatically gathers several inspection points of a part into an inspection area and encloses it with a rectangular frame (ST  14 ).  
         [0139]    An embodiment then captures an image of the area and displays it on a monitor screen.  
         [0140]    By calculating view-angle dependent perspective transformation, an embodiment of the present invention automatically metamorphoses the rectangular frame into a trapezoid with corresponding angular bird&#39;s eye perspective shape and superimposes it on the real inspection area image (ST  15 ).  
         [0141]    Watching the real inspection point images, the operator manually modifies size and shape of the superimposed trapezoid and then teaches magnification for imaging (ST  16 ).  
         [0142]    After gathering several inspection areas into one view field, an embodiment of the present invention calculates the centroid address of the view field (ST  17 ) and also the gaze control data and the magnification data for the view field (ST  18 ).  
         [0143]    Steps of automatic inspection in auto mode are to be mentioned along a flow diagram of FIG. 5 and steps of verification by watching displayed suspect point images in the following review mode are to be explained along a flow diagram of FIG. 6.  
         [0144]    After the PCB is carried in (ST  21 ) and auto mode is selected, the stage chucks the PCB and turns it by horizontal angle 45° about the centroid (ST  22 ) (see PCB  1 ′ and electronic part  2 ′ shown in FIG. 3).  
         [0145]    An inspection system of an embodiment of the present invention gazes at a view field taught in the teaching steps by moving active mirror  4  to point the optical axis at the field, by translating active objective L 1  to focus thereon, and by zooming zoom lens L 3  to adjust magnification (ST  23 ). Then an embodiment grabs the view field image (ST  24 ), automatically extracts the inspection points within each inspection area (ST  25 ), morphs the perspective image into an orthogonal style (ST  25 ′), calculates the discriminative parameters for each point (ST  25 ″), evaluates part-mount and soldering quality of each point (ST  26 ), and stores the addresses of the view field and suspect points to be reviewed subsequently (ST  27 ).  
         [0146]    When inspection of all inspection areas in the 45° turn pose was finished, the system turns the PCB by horizontal angle −135° (see PCB  1 ″ and electronic part  2 ″ shown in FIG. 3) and performs inspection again over all inspection areas (ST  23 ˜ST  27 ).  
         [0147]    When inspection of all inspection areas is accomplished in the 45° and −135° poses, ST  28  turns into YES. Then an embodiment integrates the evaluation data for every inspection point in both poses (ST  29 ).  
         [0148]    Next, review mode is selected as is shown in FIG. 6.  
         [0149]    The stage turns the PCB by horizontal angle 45° about the centroid (ST  31 ).  
         [0150]    An embodiment of the present invention gazes at a review field enclosing suspect points by moving active mirror  4  to point the optical axis at the field, by translating active objective L 1  to focus thereon, and by zooming zoom lens L 3  to adjust magnification (ST  32 ).  
         [0151]    Then an embodiment grabs the view field image (ST  33 ), displays the suspect point images, and superimposes markers thereon (ST  34 ). An operator performs verification by looking at the displayed suspect point images (ST  35 ) and inputs the judgment (ST  36 ).  
         [0152]    When inspection of all inspection areas in the 45° pose was finished, the system turns the PCB by horizontal angle −135° and performs the review steps again over all review areas (ST  32 ˜ST  36 ). When inspection of all review areas is accomplished in the 45° and −135° poses, ST  37  turns into YES.  
         [0153]    Then an embodiment integrates the review data for every suspect point in both poses (ST  38 ).  
         [0154]    Finally, an embodiment reports the results and carries out the PCB (ST  39 ).  
         [0155]    Constitution of an inspection system of second embodiment of the present invention and a posed PCB are schematically demonstrated in FIG. 7. The figure illustrates soldered electronic parts  2  mounted on PCB  1  and an imaging unit  3  placed upward of the PCB.  
         [0156]    Imaging unit  3  is connected to control system  20  which consists of automatic inspection unit  21 , image replay unit  22 , mode selection unit  23 , and storage  24 .  
         [0157]    Units  21 ˜ 23  and storage  24  communicate each other via bus  25 . Control system  20  is connected with input unit  13 , output unit  14 , communication unit  15 , and display unit  16 .  
         [0158]    Parallel action of auto mode and review mode of second embodiment is to be mentioned in accordance with a flow diagram of FIG. 8.  
         [0159]    After the PCB is carried in (ST  41 ) and auto mode is selected for (i)th PCB, imaging unit  3  grabs images of the PCB and automatic inspection unit  21  performs automatic inspection of the inspection points thereof (ST  42 ), evaluates the package quality (ST  43 ), and stores the address and the image of the view field including suspect points to be reviewed subsequently (ST  44 ).  
         [0160]    After all the points were inspected, the embodiment carries out (i)th PCB (ST  45 ).  
         [0161]    In parallel with the auto mode of (i)th PCB mentioned above, second embodiment simultaneously performs review mode inspection of (i-1)th PCB. The embodiment replays the stored review field images and superimposes markers on suspect point images (ST  46 ).  
         [0162]    An operator performs verification by looking at the replayed suspect point images (ST  47 ) and inputs the judgment (ST  48 ).  
         [0163]    Then the embodiment reports the total inspection results (ST  49 ).  
         [0164]    A multi-modal soldering inspection system of third embodiment of the present invention is to be described.  
         [0165]    Constitution of an inspection system of third embodiment and the position and pose of a PCB are similar to those of first embodiment shown in FIG. 3.  
         [0166]    Because hardware constitution of the active vision system of third embodiment and steps of teaching are also quite similar to those of first embodiment, explanation is to be omitted.  
         [0167]    Third embodiment performs review mode inspection in parallel with auto mode inspection by means of system control unit  10  shown in FIG. 3. The performance is denoted in a flow diagram of FIG. 9.  
         [0168]    After the (i)th PCB is carried in (ST  51 ) and auto mode is selected, the stage chucks the PCB and turns it by horizontal angle 45° about the centroid (ST  52 ).  
         [0169]    An inspection system of third embodiment of the present invention gazes at a view field taught in the teaching steps by moving active mirror  4  to point the optical axis at the field, by translating active objective L 1  to focus on there, and by zooming zoom lens L 3  to adjust magnification (ST  53 ).  
         [0170]    Then the embodiment grabs the view field image (ST  54 ), automatically extracts the inspection points within each inspection area, morphs the perspective image into an orthogonal style, calculates the discriminative parameters for each point (ST  55 ), evaluates part-mount and soldering quality of each point (ST  56 ), and stores the address and the image of the view field including suspect points to be reviewed subsequently (ST  57 ).  
         [0171]    When inspection of all inspection areas in the 45° turn pose was finished, the system turns the PCB by horizontal angle −135° and performs inspection again over all inspection areas (ST  53 ˜ST  57 ). When inspection of all inspection areas is accomplished in the 45° and −135° poses, ST  58  turns into YES.  
         [0172]    Then third embodiment integrates the evaluation data for every inspection point in both poses (ST  59 ) and carries out (i)th PCB (ST  60 ).  
         [0173]    In parallel with the auto mode of (i)th PCB mentioned above, third embodiment simultaneously performs review mode inspection of (i-1)th PCB.  
         [0174]    The embodiment replays the stored review field images and superimposes markers on suspect point images (ST  61 ).  
         [0175]    An operator performs verification by looking at the replayed suspect point images (ST  62 ) and inputs the judgment (ST  63 ). When all the inspection is finished, ST  64  turns into YES and then the embodiment reports the total inspection results of (i-1)th PCB (ST  65 ).  
         [0176]    Next, a multi-modal soldering inspection system of fourth embodiment of the present invention is to be described.  
         [0177]    Constitution of an inspection system of fourth embodiment and the position and pose of a PCB are similar to those of second embodiment shown in FIG. 7.  
         [0178]    Steps of teaching of forth embodiment are denoted in a flow diagram of FIG. 10.  
         [0179]    First, an operator teaches the position and the pose of the PCB (ST  71 ). Next, utilizing part mount data, part shape data, or PCB CAD data, he/she teaches the addresses to which parts are to be mounted and the electrode addresses corresponding to soldering inspection points (ST  72 ). An inspection system of fourth embodiment of the present invention automatically gathers several inspection points of a part into an inspection area and encloses it with a frame (ST  73 ).  
         [0180]    Fourth embodiment then captures an image of the area, displays it on a monitor screen, and superimposes the frame on the real inspection area image (ST  74 ).  
         [0181]    An operator classifies all the inspection points into auto mode inspection points or eye mode inspection points (ST  75 ).  
         [0182]    Inspection points occluded or shaded by tall parts will not present good images for automatic recognition.  
         [0183]    Therefore they are classified into eye mode inspection points.  
         [0184]    On the other hand, inspection points without occlusion or shading will present good images for automatic detection of flaw soldering. Therefore they are classified into auto mode inspection points.  
         [0185]    The classification of inspection points will eliminate time-consuming laborious tuning-up of recognition capability for difficult points and will also save total inspection cost efficiently.  
         [0186]    Then the operator manually modifies size and shape of the superimposed frame (ST  76 ).  
         [0187]    After gathering several inspection areas into one view field, fourth embodiment of the present invention calculates the centroid address of the view field (ST  77 ).  
         [0188]    For eye mode inspection points, the embodiment automatically lays eye inspection markers (ST  80 ), superimposes them on a PCB parts-mount map (ST  81 ).  
         [0189]    The operator corrects location of the markers (ST  82 ).  
         [0190]    Fourth embodiment performs auto mode and review mode at the same time under the control of mode selection unit  23  in FIG. 7.  
         [0191]    The embodiment permits the operator to execute eye mode inspection of a preceding PCB.  
         [0192]    The three-modal inspection is to be mentioned in accordance with a flow diagram of FIG. 11.  
         [0193]    After (i)th PCB is carried in (ST  91 ) and auto mode is selected, imaging unit  3  grabs images of the PCB  1  and automatic inspection unit  21  caries out automatic inspection of the inspection points thereof (ST  92 ), evaluates the package quality (ST  93 ), and stores the address and the image of the view field including suspect points to be reviewed subsequently (ST  94 ). After all the points were inspected, the embodiment carries out (i)th PCB (ST  95 ).  
         [0194]    In parallel with the auto mode of (i)th PCB mentioned above, fourth embodiment simultaneously performs review mode inspection of (i-1)th PCB.  
         [0195]    The embodiment replays the stored review field images and superimposes markers on suspect point images (ST  96 ).  
         [0196]    An operator performs verification by looking at the replayed suspect point images (ST  97 ) and inputs the judgment (ST  98 ).  
         [0197]    Then the embodiment reports the inspection results (ST  99 ).  
         [0198]    Next, the embodiment turns into eye mode of (i-1)th PCB, displays a PCB parts-mount map, and superimposes eye inspection point markers on it (ST  100 ).  
         [0199]    The operator conducts visual inspection of the eye inspection points by referring to the map and inputs the results and/or repairs faulty solder joints (ST  101 ).  
         [0200]    A multi-modal soldering inspection system of fifth embodiment of the present invention is to be mentioned.  
         [0201]    Constitution of an inspection system of fifth embodiment and the position and pose of a PCB are similar to those of first embodiment shown in FIG. 3.  
         [0202]    Steps of teaching are to be mentioned according to a flow diagram shown in FIG. 12  
         [0203]    First, an operator teaches the geometry of active optical system  3   b  and the stage (not shown) (ST  111 ) and also the position and the pose of the PCB (ST  112 ).  
         [0204]    Next, utilizing part mount data, part shape data, or PCB CAD data, he/she teaches the addresses to which parts are to be mounted and the electrode addresses corresponding to soldering inspection points (ST  113 ). An inspection system of fifth embodiment of the present invention automatically gathers several inspection points into an inspection area and encloses it with a rectangular frame (ST  114 ).  
         [0205]    Fifth embodiment then captures an image of the area and displays it on a monitor screen.  
         [0206]    By calculating view-angle dependent perspective transformation, an embodiment of the present invention automatically metamorphoses the rectangular frame into a trapezoid with corresponding angular bird&#39;s eye perspective shape and superimposes it on the real inspection area image (ST  115 ).  
         [0207]    An operator classifies all the inspection points into auto mode inspection points and eye mode inspection points (ST  116 ).  
         [0208]    Inspection points occluded or shaded by tall parts will not present good images for automatic recognition.  
         [0209]    Therefore they are classified into eye mode inspection points.  
         [0210]    On the other hand, inspection points without occlusion or shading will present good images for automatic detection of flaw soldering. Therefore they are classified into auto mode inspection points.  
         [0211]    The classification of inspection points will eliminate time-consuming laborious tuning-up of recognition capability for difficult points and will also save total inspection cost efficiently.  
         [0212]    Then the operator manually modifies size and shape of the superimposed trapezoid and then teaches magnification for imaging (ST  117 ).  
         [0213]    After gathering several inspection areas into one view field, fifth embodiment of the present invention calculates the centroid address of the view field (ST  118 ) and also the gaze control data and the magnification data for the view field (ST  119 ).  
         [0214]    For eye mode inspection points, the embodiment automatically lays eye inspection markers (ST  120 ), superimposes them on a PCB parts-mount map (ST  121 ).  
         [0215]    The operator corrects location of the markers (ST  122 ).  
         [0216]    Fifth embodiment performs auto mode and review mode at the same time under the control of system control unit  10  in FIG. 3.  
         [0217]    The embodiment permits the operator to execute eye mode inspection of a preceding PCB.  
         [0218]    The three-modal inspection is to be mentioned in accordance with a flow diagram of FIG. 13.  
         [0219]    After the (i)th PCB is carried in (ST  131 ) and auto mode is selected, the stage chucks the PCB and turns it by horizontal angle 45° about the centroid (ST  132 ).  
         [0220]    An inspection system of fifth embodiment of the present invention gazes at a view field taught in the teaching steps by moving active mirror  4  to point the optical axis at the field, by translating active objective L 1  to focus on there, and by zooming zoom lens L 3  to adjust magnification (ST  133 ).  
         [0221]    Then the embodiment grabs the view field image (ST  134 ), automatically extracts the inspection points within each inspection area, morphs the perspective image into an orthogonal style, calculates the discriminative parameters for each point (ST  135 ), evaluates part-mount and soldering quality of each point (ST  136 ), and stores the address and the image of the view field including suspect points to be reviewed subsequently (ST  137 ).  
         [0222]    When inspection of all inspection areas in the 45° turn pose was finished, the system turns the PCB by horizontal angle −135° and performs inspection again over all inspection areas (ST  133 ˜ST  137 ). When inspection of all inspection areas is accomplished in the 45° and −135° poses, ST  138  turns into YES.  
         [0223]    Then fifth embodiment integrates the evaluation data for every inspection point in both poses (ST  139 ) and carries out (i)th PCB (ST  140 ).  
         [0224]    In parallel with the auto mode of (i)th PCB mentioned above, fifth embodiment simultaneously performs review mode inspection of (i-1)th PCB.  
         [0225]    The embodiment replays the stored review field images and superimposes markers on suspect point images (ST  141 ).  
         [0226]    An operator performs verification by looking at the replayed suspect point images (ST  142 ) and inputs the judgment (ST  143 ).  
         [0227]    When all the inspection is finished, ST  144  turns into YES and then the embodiment reports the total inspection results of (i)th PCB (ST  145 ).  
         [0228]    Next, the embodiment turns into eye mode of (i-1)th PCB, displays a PCB parts-mount map, and superimposes eye inspection point markers on it (ST  146 ).  
         [0229]    The operator conducts visual inspection of the eye inspection points by referring to the map and inputs the results and/or repairs faulty solder joints (ST  147 ).  
         [0230]    A multi-modal soldering inspection system of sixth embodiment of the present invention is to be described.  
         [0231]    As are depicted in FIG. 14, sixth embodiment, in the review mode inspection, is able to display suspect point images not only in a sequential, one-by-one fashion but also in a multiple-frame fashion demonstrating several images simultaneously.  
         [0232]    An inspector is able to select one-by-one display for precise observation of suspect images or multiple-image display for quick verification.  
         [0233]    A multi-modal soldering inspection system of seventh embodiment of the present invention is to be mentioned.  
         [0234]    As are shown in FIG. 15, seventh embodiment is equipped with PCB import unit  151  for carrying PCBs in, inspection unit  152  for auto mode inspection, PCB export unit  153  for carrying them out, sorter  154  for sorting PCBs with suspect points and/or eye inspection points, and stocker unit  155  for storing them for review or eye mode inspection.  
         [0235]    The embodiment enables in-line auto mode inspection, out-line review mode inspection of stored PCBs having suspect points, and eye mode inspection of stored PCBs having eye inspection points.  
         [0236]    A review station of eighth embodiment of the present invention is to be described.  
         [0237]    Eighth embodiment is a station which is able to be placed anywhere apart from a multi-modal soldering inspection system and enables review mode inspection or eye mode inspection without using the inspection system itself  
         [0238]    As are illustrated in FIG. 16, eighth embodiment is equipped with image replay unit  161 , storage  162 , input unit  13 , output unit  14 , communication unit  15 , and display unit  16 .  
         [0239]    The embodiment is able to receive addresses and images of suspect points of a PCB from a multi-modal soldering inspection system of the present invention and replays the suspect images superimposed with suspect point markers.  
         [0240]    The embodiment enables an operator to perform review mode inspection by watching the replayed images.  
         [0241]    The embodiment is also able to receive eye inspection data from a multi-modal soldering inspection system of the present invention and displays a PCB map superimposed with eye inspection point markers. The embodiment enables an inspector to perform eye mode inspection by referring to the PCB map.