Patent Description:
When using a component mounter (component insertion assembly device) to insert a lead of an electronic component into a through-hole of a circuit board, for example, as disclosed in <CIT>, a lead of an electronic component is inserted into a through-hole of a circuit board by using a suction nozzle or chuck device or the like of a mounting head of the component mounter to pick up the electronic component, moving above the circuit board, positioning the lead tip (lower end) of the electronic component directly above the through-hole of the circuit board, and lowering the electronic component. Here, because the position of the electronic component held by the mounting head shifts for each electronic component, before the electronic component held by the mounting head is moved above the circuit board, it is moved to an imaging position above a component imaging camera, the lead tip of the electric is imaged from below by the camera, and by performing image processing of the image using conventional pattern matching, the center position of the lead tip is recognized, then, based on the recognition result, the position deviation of the electronic component is corrected, and the lead of the electronic component is positioned directly above the through-hole of the circuit board.

<CIT>) proposes to provide an objective, fast and economical method of inspecting objects, such as electronic packages having a plurality of leads protruding therefrom, It discloses an automatic system for the inspection of a plurality of protruding features of an object employing computer vision. The system examines the position of a tip of the feature and compares it with an ideal position to determine if the object is defective or not. The position of the tip is calculated from a cluster of perceived pixels above a certain threshold of light intensity. In one embodiment, mathematic morphological manipulations of the gray scale perceived pixels assists in the analysis of the positions of the protruding features. JPH0545123 (A) proposes to recognize a change in the shape of the fore end of a lead at a high speed, to detect a correct position of the fore end of the lead and to prevent, as another purpose, an error in the direction and the amount of correction of the position of the fore end of the lead on the basis of the degree of reliability of a shape feature amount for recognizing the change in the shape of the fore end even when noise is contained in this amount, regarding the lead of which the fore end is round. It discloses a means of computing the position of the center of a closed area of a binary image, a means of computing the positions of apexes and the circumscribing positions of a rectangle circumscribing the closed area of the binary image, and a means of correcting the position of the center of the closed area of the binary image on the basis of the positions of the apexes and the positions of contacts of the circumscribed rectangle, are provided. The position of the center of the closed area of the binary image is corrected on the basis of the positions of the apexes and the positions of the contacts of the rectangle circumscribed about the closed area. Besides, the position of the fore end of a lead is corrected on the basis of an image feature amount representing a positional slippage of the lead, by using a fuzzy inference, and execution of the correction is enabled in a wide range on the basis of the degree of reliability of a shape feature amount even when noise is contained in the shape feature amount.

If the shape of the lead tip of the electronic component is a stable round shape, the difficulty level of the image processing is not particularly high, but with recent electronic circuit components, shaft-type leads (insertion pins) are formed, but the tip of the lead is a cut surface, so even for normal components, the lead tip shape is not stable, and, as shown in <FIG>, the same type of electronic component is displayed in the image with blurry sections of the lead tip, or as shown in <FIG>, scattered light at the lead tip is displayed in the image (halation), or as shown in <FIG>, the lead tip displays as an elliptical shape, or as shown in <FIG>, a portion of the lead tip does not display, or as shown in figs. (g) and (h), a center portion of the lead tip does not display. Thus, even for the same type of electronic component, depending on the state of the cut surface of the lead tip, the lead tips display as various different shapes, such that it is difficult to recognize lead tips with good accuracy by image processing lead images using conventional pattern matching, meaning that the recognition accuracy of the center position of the lead tip is poor.

Therefore, an object of the present invention is to recognize with good accuracy a center position of a lead tip from an image captured by a camera even when the shape of lead tips is not stable.

To solve the above problems, the present invention is a lead tip position image recognition method for recognizing a center position of a lead tip of an electronic component to be inserted into a through-hole of a circuit board by processing an image of the lead tip (hereinafter referred to as "lead image") captured by a camera using an image recognition device, the lead tip position image recognition method including: a learning process in which an operator specifies the center position of the lead tip in the lead image and learning is performed such that, when the lead image is inputted into the image recognition device, the output is the center position of the lead tip specified by the operator; and a recognizing process of inputting the lead image acquired by imaging the lead tip of the electronic component using the camera into the image recognition device and outputting the center position of the lead tip. Accordingly, even if the shape of the lead tips is not stable, due to the learning process, it is possible to learn the relationship between the shape of the lead tip in the lead image that is inputted and the center position of the lead tip that is to be outputted (training data), and to recognize with good accuracy the center position of the lead tip from an inputted lead image.

In this case, the image recognition device may be configured using a machine learning system such as a neural network.

Also, in the recognizing process, processing may switch to the learning process when an image processing error occurs in which the center position of the lead tip in the lead image cannot be recognized, and learning may be performed such that the output when the lead image for which the image processing error occurred is inputted to the image recognition device is the center position of the lead tip specified by the operator. In this manner, if learning is performed using lead images for which image processing errors occurred, subsequently, even if a similar lead image is inputted to the image recognition device, an image processing error will not occur, and the center position of the lead tip will be recognized with good accuracy, thus, the recognition accuracy of the center position of lead tips can be improved while reducing the frequency of image processing errors.

Also, in the learning process, at least one processing may be performed on the lead image out of rotation, mapping, brightness alteration, or shape alteration to generate multiple images, and learning may be performed such that the output when each of the multiple images are inputted to the image recognition device is the center position of the lead tip specified by the operator. Accordingly, even if there are not many lead images to be used for learning, it is possible to increase the training data (combinations of "input" and "correct output"), such that efficient learning is performed and the recognition accuracy of the center position of lead tips is improved.

Further, in a case of imaging from below an electronic component held by a mounting head of a component mounter using a component imaging camera to perform image recognition of the center position of the lead tip, during an insertion process of inserting the lead of the electronic component by positioning the lead of the electronic component held by the mounting head at the through-hole of the circuit board based on the recognition result of the image recognition device, when a positioning error occurs in which the lead of the electronic component cannot be inserted into the through-hole of the circuit board, processing may switch to the learning process, and learning may be performed such that the output when the lead image for which the positioning error occurred is inputted to the image recognition device is the center position of the lead tip specified by the operator. Accordingly, if learning is performed using lead images for which positioning errors occurred, subsequently, the recognition accuracy of the center position of the lead tip can be improved when a similar lead image is inputted to the image recognition device, and the frequency of positioning errors is reduced.

In general, because high calculating ability is required for learning training data, when performing learning of training data at an image recognition device, it is possible that the calculating ability of the image recognition will be insufficient.

Thus, the learning device that learns the relationship between the lead images inputted to the image recognition device and the center position of the lead tips to be outputted (training data) may be provided separately to the image recognition device, and the results of learning at the learning device may be sent to the image recognition device. Accordingly, because the image recognition device does not have to perform learning of the training data, it is not necessary to raise the calculating ability of the image recognition device to that capable of performing learning of the training data, and learning of the training data and recognition of the center position of the lead tip can be split efficiently between the learning device and the image recognition device.

With the present invention, if the calculating ability of the image recognition device is high, learning of the training data and recognizing of the center position of the lead tip may both be performed in the image recognition device. In this case, the image recognition device may be configured to be switchable between a recognition mode in which the image recognition device receives a lead image and outputs the center position of the lead tip, and a learning mode in which learning is performed of the training data; and a specifying means configured to be used by an operator to specify the center position of the lead tip during the learning mode may be provided, wherein the image recognition device may be configured to, when in the learning mode, perform learning such that the center position of the lead tip in the lead image specified by the operator during the learning mode is the output when the lead image is inputted to the image recognition device.

Descriptions of two embodiments for carrying out the present invention are given below.

A first embodiment of the present invention will be described based on <FIG>. First, the overall configuration of the component mounter is described using <FIG>.

Component mounter <NUM> is provided with head moving device <NUM> that moves mounting head <NUM> that holds electronic component <NUM>, component supply device <NUM> that supplies electronic component <NUM>, conveyor <NUM> that conveys circuit board <NUM>, component imaging camera <NUM> that images from below electronic component <NUM> held by mounting head <NUM>, mark imaging camera <NUM> that images from above reference marks or the like on circuit board <NUM>, and the like. Component imaging camera <NUM> is fixed facing up between component supply device <NUM> and conveyor <NUM>, and lighting device <NUM> that illuminates from below electronic component <NUM> held by mounting head <NUM> is attached to an upper portion of component imaging camera <NUM>. Mark imaging camera <NUM>, on the other hand, is attached facing down on mounting head <NUM>, and is moved as one with mounting head <NUM> by head moving device <NUM>.

Control device <NUM> of component mounter <NUM> is configured from a computer or the like, and performs control such that electronic component <NUM> supplied by component supply device <NUM> is picked up by mounting head <NUM>, moved above circuit board <NUM> by head moving device <NUM>, and inserted into through-hole <NUM> of circuit board <NUM>.

Further, control device <NUM> of component mounter <NUM> is equipped to function as image recognition device <NUM>, and before moving the electronic component <NUM> held by mounting head <NUM> above circuit board <NUM>, moves the electronic component <NUM> to an imaging position above component imaging camera <NUM>, uses camera <NUM> to capture an image from below of the tip (lower end) of lead <NUM> of the electronic component <NUM>, acquires the lead image, inputs the lead image to image recognition device <NUM>, and outputs the center position of the tip of lead <NUM>. Note that, control device <NUM> and image recognition device <NUM> may be configured form separate computers or from one computer.

Image recognition device <NUM> is configured using a machine learning system such as a neural network or a deep learning system, and in a learning mode (learning process), using learning by training (learning with training), learns a relationship (training data) between the shape of the tip of lead <NUM> in a lead image that is inputted, and a center position of the tip of lead <NUM> to be outputted, and then, in a recognition mode (recognition process) recognizes with good accuracy the center position of the tip of the lead <NUM> from an inputted lead image.

Here, learning of training data may be performed before starting production, but in the first embodiment, learning of training is performed every time an image processing error or a positioning error occurs. Here, "image processing error" refers to a failure to recognize the center position of the tip of lead <NUM> in the lead image during the recognition process; "positioning error" refers to failure to insert lead <NUM> of electronic component <NUM> into through-hole <NUM> of circuit board <NUM> in an insertion process in which lead <NUM> of electronic component <NUM> held by mounting head <NUM> is positioned at and inserted into through-hole <NUM> of circuit board <NUM>.

The training data learning method involves displaying a lead image from when an image processing error or a positioning error occurred on display device <NUM> such as liquid crystal display or a CRT, and then an operator specifying the center position of the tip of lead <NUM> in the lead image by operating operation section <NUM> (specifying means) such as a keyboard, mouse, or touch panel while looking at the lead image displayed on display device <NUM>. Accordingly, image recognition device <NUM> learns how to make the output when a lead image for which an image processing error or a positioning error occurred is inputted the center position of the tip of lead <NUM> specified by the operator.

Here, at least one processing may be performed on the lead image out of rotation, mapping, brightness alteration, or shape alteration to generate multiple images, and learning may be performed such that the output when each of the multiple images are inputted to image recognition device <NUM> is the center position of the tip of lead <NUM> specified by the operator. Accordingly, even if there are not many lead images to be used for learning, it is possible to increase the training data (combinations of "input" and "correct output"), such that efficient learning is performed and the recognition accuracy of the center position of the tips of leads <NUM> is improved.

For example, because the directions in which blurs occur at the tip of leads <NUM> are varied, a lead image with a blur is rotated by a specified angle each time, such that multiple lead images with different rotation angles are created, which means that, no matter which direction blurs occur in, it is possible to recognize with good accuracy the center position of the tip of lead <NUM> from the lead image.

In a case in which image recognition device <NUM> is configured using the neural network shown in <FIG>, signals flow to an input layer, an intermediate layer (hidden layer) and an output layer. There may be one intermediate layer, or two or more intermediate layers. A neural network refers to an overall model that has problem-solving ability due to each level of neurons (nodes) that form a network by synapse connections having their weighting changed (synapse connection strength) by learning. In a neural network, neurons of an intermediate layer and an output layer receive stimulus from each of the neurons in the previous layer, and those stimuli are combined with weighting and passed to the neurons of the next layer. Here, the weighting is the importance thing, and by changing the weighting in the training data learning process to adjust the output to an appropriate value, even for "unknown inputs", it is possible to achieve "correct outputs" with high accuracy.

In the first embodiment, in the input layer, for example, a lead image of <NUM> pixels x <NUM> pixels is inputted, and in the output layer where the center position of the tip of lead <NUM> is in the inputted lead image of <NUM> pixels x <NUM> pixels is outputted. For ease of understanding, <FIG> shows an example of an input and output of <NUM> pixels x <NUM> pixels.

The recognition accuracy of the center position of the tip of lead <NUM> can be improved by making the output layer larger than the input layer. For example, if the input layer is <NUM> × <NUM> pixels, the output layer may be <NUM> × <NUM> pixels or <NUM> × <NUM> pixels. This makes it possible to perform position recognition at a sub-pixel level with respect to the inputted image.

As training data, in addition to inputting and outputting lead images of positive samples as in <FIG>, it is possible to add inputs and outputs of portions other than leads <NUM> that are negative samples, as shown in <FIG>, so as to prevent mis-detection of leads <NUM>.

Learning of training data may be performed by learning and memorizing by each type of electronic component <NUM>, or by shared learning and memorizing for multiple component types.

In positioning processing of lead <NUM>, as shown in <FIG>, by performing a search in order using raster scanning or the like of the processing range with respect to the captured image that corresponds to the input layer of the neural network, a region of the captured image that contains lead <NUM> can be detected, and applied to the input layer of the neural network.

Alternatively, instead of raster scanning, blob analysis or the like may be used. For example, by performing binarization of the captured image and then blob analysis or the like, as shown in <FIG>, a candidate region in the captured image that may contain lead <NUM> may be detected and applied to the input layer of the neural network. With blob analysis or the like, a portion other than lead <NUM> may be detected, or the position detection accuracy even when detecting a lead may be low, but it is possible to detect lead <NUM> only with high position detection accuracy using the post-processing of the neural network.

Control device <NUM> of component mounter <NUM>, by cooperating with image recognition device <NUM> and performing the programs of <FIG> and <FIG>, uses mounting head <NUM> of component mounter <NUM> to pick up and hold electronic components <NUM> supplied by component supply device <NUM>, recognizes the center position of the tip of lead <NUM> of the electronic components <NUM>, controls operation of inserting lead <NUM> of the electronic components <NUM> into through-holes <NUM> of circuit board <NUM>, and performs learning of training data each time an image processing error or a positioning error occurs. Processing contents of the programs of <FIG> and <FIG> are described below.

The component mounter control program of <FIG> is for controlling operations from picking up and holding electronic component <NUM> supplied by component supply device <NUM> until inserting lead <NUM> of the electronic component <NUM> into through-hole <NUM> of circuit board <NUM>, and is performed when pickup operation of the electronic component <NUM> is started. When the program is performed, first, in step <NUM>, electronic component <NUM> supplied by component supply device <NUM> is picked up and held by mounting head <NUM> of component mounter <NUM>, and moved to an imaging position above component imaging camera <NUM>. Then, continuing to step <NUM>, the tip (lower end) of lead <NUM> of the electronic component <NUM> is imaged from below by camera <NUM> to acquire a lead image.

Then, continuing to step <NUM>, the lead image is inputted into image recognition device <NUM> and image processing is performed to output the center position of the tip of the lead <NUM>. Then, continuing to step <NUM>, it is determined whether an image processing error occurred (whether the center position of the tip of the lead <NUM> in the lead image was failed to be recognized). If it is determined that an image processing error occurred, continuing to step <NUM>, the learning processing program of <FIG> is performed.

On the other hand, if it is determined in step <NUM> that an image processing error did not occur, continuing to step <NUM>, the electronic component <NUM> held by mounting head <NUM> of component mounter <NUM> is moved above circuit board <NUM> and, based on the center position of the tip of lead <NUM> output from image recognition device <NUM>, the positioning deviation of the electronic component <NUM> is corrected, lead <NUM> of the electronic component <NUM> is positioned directly above through-hole <NUM> of circuit board <NUM>, then mounting head <NUM> is lowered such that lead <NUM> of the electronic component <NUM> is inserted into through-hole <NUM> of circuit board <NUM>.

Then, continuing to step <NUM>, it is determined whether a positioning error occurred (whether lead <NUM> of electronic component <NUM> was failed to be inserted into through-hole <NUM> of circuit board <NUM>), and if it is determined that a positioning error did not occur, the program finishes, but if it is determined that a positioning error occurred, continuing to step <NUM>, the learning processing program of <FIG> is performed.

The learning processing program of <FIG> is performed at step <NUM> of <FIG> when an image processing error or a positioning error occurs. When the program is performed, first, in step <NUM>, the lead image for which the image processing error or the positioning error occurred is displayed on liquid crystal display device <NUM>, and in step <NUM>, processing stands by until an operator specifies the center position of the tip of lead <NUM> in the lead image by operating operation section <NUM> while viewing the lead image displayed on display device <NUM>.

Then, when the operator has specified the center position of the tip of lead <NUM> in the lead image, continuing to step <NUM>, learning is performed such that the output when the lead image for which the image processing error occurred is inputted to image recognition device <NUM> is the center position of the tip of lead <NUM> specified by the operator. This learning processing may be performed by image recognition device <NUM> itself, or by control device <NUM> of component mounter <NUM>. In the latter case, the result (weighting) of the learning performed by control device <NUM> of component mounter <NUM> is sent to image recognition device <NUM>.

According to an embodiment described above, even if the shape of the tips of leads <NUM> of electronic components <NUM> is not stable, it is possible to learn the relationship between the shape of the tip of lead <NUM> in the lead image that is inputted to image recognition device <NUM> and the center position of the tip of lead <NUM> that is to be outputted (training data), and to recognize with good accuracy the center position of the tip of lead <NUM> from the inputted lead image.

Further, in the embodiment above, when an image processing error occurs in which the center position of the tip of lead <NUM> in the lead image cannot be recognized, because learning is performed such that the output when a lead image for which an image processing error occurred is inputted into image recognition device <NUM> is the center position of the tip of the lead specified by an operator, subsequently, if the same lead image is inputted into image recognition device <NUM>, an image processing error does not occur, and the center position of the tip of lead <NUM> can be recognized with good accuracy. Accordingly, it is possible to improve the recognition accuracy of the center position of the tip of lead <NUM> while reducing the frequency of image processing errors.

Further, in the embodiment above, when a positioning error occurs in which lead <NUM> cannot be inserted into through-hole23 of circuit board <NUM>, because learning is performed such that the output when a lead image for which a positioning error occurred is inputted into image recognition device <NUM> is the center position of the tip of the lead specified by an operator, subsequently, it is possible to improve the recognition accuracy of the center position of the tip of lead <NUM> in a case in which the same lead image is inputted into image recognition device <NUM>, and the frequency of positioning errors can be reduced.

In the embodiment above, because there is more than sufficient calculating ability in image recognition device <NUM> or control device <NUM> of component mounter <NUM>, learning of training data and recognition of the center position of the tip of lead <NUM> are both performed in image recognition device <NUM> or control device <NUM> of component mounter <NUM>, but in general, because high calculating ability is required for learning of training data, it is possible that the calculating ability of image recognition device <NUM> or control device <NUM> of component mounter <NUM> will not be sufficient for learning of training data.

Thus, in a second embodiment of the present invention, as shown in <FIG>, learning device <NUM> with high calculating ability is provided outside of component mounter <NUM>, and learning device <NUM> is connected to component mounter <NUM> via a network, and learning device <NUM> learns the relationship between the lead image inputted to image recognition device <NUM> of component mounter <NUM> and the outputted center position of the tip of lead <NUM> (training data), and sends the learning result (weighting) to image recognition device <NUM> of component mounter <NUM>.

In this case, learning device <NUM> may be configured using a production management computer that manages a production line including component mounter <NUM>, or a computer for learning may be provided separately. Also, learning device <NUM> may be connected to multiple component mounters <NUM> via a network, and learning of training data at multiple component mounters <NUM> may be performed at a single learning device <NUM>.

Further, training data (a lead image and the center position of the tip of lead <NUM> specified by an operator) may be created at the component mounter <NUM> side and sent to learning device <NUM>, or only the lead image may be sent to learning device <NUM>, the lead image may be displayed on a display screen of learning device <NUM>, and an operator may specify the center position of the tip of lead <NUM> in the lead image displayed on the display screen of display device <NUM>.

Note that, training data and learning results may be saved in a server of the production line or in a memory device of component mounter <NUM>, the learning data may be sent to another component mounter <NUM> of the mounting line, new lead images may be added to the training data, and learning may be performed again.

The present invention is not limited to the embodiments above and it goes without saying that various embodiments with changes that do not extend beyond the scope of the invention, as defined by the appended claims, are possible, for example, as lead image that is inputted, edge information (outline boundary line, edge gradient, or the like) of lead <NUM> may be used, or training data may be learned by an employee of the manufacturer of the component mounter <NUM> using a learning device, and those learning results (weightings) may be presented to the user of the component mounter <NUM>.

Claim 1:
A lead tip position image recognition method for recognizing a center position of a lead tip of an electronic component (<NUM>) to be inserted into a through-hole (<NUM>) of a circuit board (<NUM>) by processing a lead image which is an image of the lead tip captured by a camera (<NUM>) using an image recognition device (<NUM>), the lead tip position image recognition method characterized by:
a learning process in which an operator specifies the center position of the lead tip in the lead image and learning is performed such that, when the lead image is inputted into the image recognition device (<NUM>), the output is the center position of the lead tip specified by the operator; and
a recognizing process of inputting the lead image acquired by imaging the lead tip of the electronic component (<NUM>) using the camera (<NUM>) into the image recognition device (<NUM>) and outputting the center position of the lead tip.