Patent Description:
In general, in a manufacturing process using surface-mount technology (SMT) on a printed circuit board, a screen printer prints solder paste on the printed circuit board, and a mounter mounts components on the printed circuit board printed with the solder paste.

In addition, an automated optical inspection (AOI) device is used to inspect the mounting state of the components mounted on the printed circuit board. Such devices are known, e.g., from <CIT>. The AOI device inspects whether the components are normally mounted on the printed circuit board without displacement, lifting, or tilting by using a captured image of the printed circuit board.

On the other hand, in a process in which the AOI device generates an image for the printed circuit board, noise may occur in multiple reflections of light irradiated on the printed circuit board or in the process of processing received light by an image sensor. That is, optical noise and signal noise may variously occur, and if the noise occurring in this manner is not reduced, the quality of the captured image of the printed circuit board generated by the AOI device may deteriorate. When the quality of the captured image of the printed circuit board deteriorates, inspection of the mounting state of the components mounted on the printed circuit board using the captured image of the printed circuit board may not be accurately performed.

The present disclosure provides a printed circuit board inspection apparatus that inspects a mounting state of a component by using depth information with reduced noise on the component obtained based on depth information on the component.

The present disclosure provides a computer-readable recording medium that records a program including executable instructions for inspecting a mounting state of a component by using depth information with reduced noise on the component obtained based on depth information on the component.

The present disclosure provides a method of inspecting a mounting state of a component by using depth information with reduced noise obtained based on depth information on the component.

According to one embodiment of the present disclosure, a printed circuit board inspection apparatus according to the subject-matter of claim <NUM> is proposed.

According to one embodiment of the present disclosure, a non-transitory computer-readable recording medium according to the subject-matter of claim <NUM> is proposed.

According to one embodiment of the present disclosure, a computer-implemented method of inspecting a mounting state of a component by a printed circuit board inspection apparatus according to the subject-matter of claim <NUM> is proposed.

Further embodiments of the present disclosure are provided in the dependent claims.

The printed circuit board inspection apparatus according to various embodiments of the present disclosure may process the depth information on the component through the machine-learning-based model, thereby reducing noise from the depth information on the component and inspecting the mounting state of the component mounted on the printed circuit board by using the depth information with reduced noise on the component. The printed circuit board inspection apparatus may remove noise such as an unreceived signal or a peak signal from the depth information on the component by using the machine-learning-based model even though a relatively small number of pieces of image data are obtained to generate the depth information, and may generate the depth information on the component so that the lost shape can be restored using the machine-learning-based model even though a relatively small number of pieces of image data are obtained so that information for generating the depth information is insufficient. In addition, the printed circuit board inspection apparatus may not perform error restoration of the joint shape of the component while correcting the three-dimensional (3D) sharpness of the edges of the component as much as possible, and may detect the shape of an additionally measured foreign material without deteriorating the same.

In this manner, by reducing noise in the depth information on the component and by performing shape restoration on the component mounted on the printed circuit board and a solder paste as closely as possible to the shape of the actual component and solder paste, it is possible to inspect the mounting state of the component more accurately.

Embodiments of the present disclosure are illustrated for describing the technical spirit of the present disclosure. The scope of the claims according to the present disclosure is not limited to the embodiments described below or to the detailed descriptions of these embodiments.

All technical or scientific terms used herein have meanings that are generally understood by a person having ordinary knowledge in the art to which the present disclosure pertains, unless otherwise specified. The terms used herein are selected for only more clear illustration of the present disclosure, and are not intended to limit the scope of claims in accordance with the present disclosure.

The expressions "include", "provided with", "have" and the like used herein should be understood as open-ended terms connoting the possibility of inclusion of other embodiments, unless otherwise mentioned in a phrase or sentence including the expressions.

A singular expression can include meanings of plurality, unless otherwise mentioned, and the same is applied to a singular expression stated in the claims.

The terms "first", "second", etc., used herein are used to identify a plurality of components from one another, and are not intended to limit the order or importance of the relevant components.

The term "unit" used in these embodiments means a software component or hardware component, such as a field-programmable gate array (FPGA) and an application specific integrated circuit (ASIC). However, a "unit" is not limited to software and hardware, it may be configured to be an addressable storage medium or may be configured to run on one or more processors. For example, a "unit" may include components, such as software components, object-oriented software components, class components, and task components, as well as processors, functions, attributes, procedures, subroutines, segments of program codes, drivers, firmware, micro-codes, circuits, data, databases, data structures, tables, arrays, and variables. Functions provided in components and "units" may be combined into a smaller number of components and "units" or further subdivided into additional components and "units.

The expression "based on" used herein is used to describe one or more factors that influence a decision, an action of judgment or an operation described in a phrase or sentence including the relevant expression, and this expression does not exclude additional factor influencing the decision, the action of judgment or the operation.

When a certain component is described as "coupled to" or "connected to" another component, this should be understood as having meaning that the certain component may be coupled or connected directly to the other component or that the certain component may be coupled or connected to the other component via a new intervening component.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, like or relevant components are indicated by like reference numerals. In the following description of embodiments, repeated descriptions of the identical or relevant components will be omitted. However, even if a description of a component is omitted, such a component is not intended to be excluded in an embodiment.

<FIG> illustrates a printed circuit board inspection apparatus according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, a printed circuit board inspection apparatus <NUM> inspects the mounting state of at least one component mounted on a printed circuit board <NUM>. A transport unit <NUM> may move the printed circuit board <NUM> to a predetermined position in order to inspect the mounting state of the components. In addition, when the inspection is completed by the printed circuit board inspection apparatus <NUM>, the transport unit <NUM> may move the printed circuit board <NUM>, which has been inspected, to deviate from the predetermined position, and may move another printed circuit board <NUM> to a predetermined printed circuit board.

According to various embodiments of the present disclosure, the printed circuit board inspection apparatus <NUM> may include a first light source <NUM>, an image sensor <NUM>, and a frame <NUM>. The first light source <NUM> and the image sensor <NUM> may be fixed to the frame <NUM>. The number and arrangement of each of the first light source <NUM>, the image sensor <NUM>, and the frame <NUM> shown in <FIG> are for the purpose of explanation, but are not limited thereto. For example, one first light source <NUM> may be arranged in the position of the image sensor <NUM> shown in <FIG>, and a plurality of image sensors may be arranged in the position of the first light source <NUM> shown in <FIG>. The first light source <NUM> and the image sensor <NUM> may be arranged in various directions and angles through the plurality of frames <NUM>.

In one embodiment, the first light source <NUM> may irradiate, with a pattern of light, the printed circuit board <NUM> moved to a predetermined position to inspect the mounting state of the component. In the case of a plurality of first light sources <NUM>, they may be arranged to have different irradiation directions, different irradiation angles, and the like. In addition, in the case of a plurality of first light sources <NUM>, pitch intervals of the pattern of light irradiated from the first light sources <NUM> may be different from each other. For example, the pattern of light may be light having a pattern having a certain period, which is irradiated to measure a three-dimensional (3D) shape of the printed circuit board <NUM>. The first light source <NUM> may irradiate a pattern of light in which the brightness of the stripes has a sine wave shape, a pattern of light in an on-off form in which bright and dark parts are repeatedly displayed, or a triangular wave-pattern of light having a triangular waveform with a change in brightness. However, this is for illustrative purposes only, and the present disclosure is not limited thereto, and the first light source <NUM> may irradiate light including various types of patterns in which a change in brightness is repeated at a constant period.

In one embodiment, the image sensor <NUM> may receive a pattern of light reflected from the printed circuit board <NUM> and the component mounted on the printed circuit board <NUM>. The image sensor <NUM> may generate image data using the received pattern of light.

<FIG> is a block diagram illustrating a printed circuit board inspection apparatus according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, the printed circuit board inspection apparatus <NUM> may include a first light source <NUM>, an image sensor <NUM>, a memory <NUM>, and a processor <NUM>. In addition, the printed circuit board inspection apparatus <NUM> may further include a communication circuit <NUM>. Each component included in the printed circuit board inspection apparatus <NUM> may be electrically connected to each other to transmit and receive signals and data.

In one embodiment, the printed circuit board inspection apparatus <NUM> may include a plurality of first light sources <NUM>. The first light source <NUM> may irradiate an inspection object (e.g., a printed circuit board) with a pattern of light. For example, the first light source <NUM> may irradiate the entire inspection object with a pattern of light or may irradiate an object (e.g., a component mounted on a printed circuit board) included in the inspection object with a pattern of light. Hereinafter, for convenience of description, although the first light source <NUM> is mainly described as irradiating the component mounted on the printed circuit board with a pattern of light, the present disclosure is not limited thereto. The first light source <NUM> may irradiate, with a pattern of light, the entire printed circuit board to be inspected or one region of the printed circuit board including at least one component mounted on the printed circuit board.

In one embodiment, the first light source <NUM> may include a light source (not shown), a grating (not shown), a grating transport device (not shown), and a projection lens unit (not shown). The grating can convert light irradiated from the light source into a pattern of light. The grating can be transported through a grating transport mechanism, for example a piezo actuator (PZT), to generate phase-shifted pattern of light. The projection lens unit may allow the pattern of light generated by the grating to be irradiated to the component mounted on the printed circuit board, which is an object included in the inspection object. Further, the first light source <NUM> may form a pattern of light through various methods such as liquid crystal display (LCD), digital light processing (DLP), and liquid crystal on silicon (LCOS), and may allow the formed pattern of light to be irradiated to the component mounted on the printed circuit board which is an object included in the inspection object.

In one embodiment, the image sensor <NUM> may receive a pattern of light reflected from the component. For example, the image sensor <NUM> may receive the pattern of light reflected from the component to generate image data on the component. The first image sensor <NUM> may transmit the generated image data on the component to the processor <NUM>.

In one embodiment, the memory <NUM> may store instructions or data related to at least one other component of the printed circuit board inspection apparatus <NUM>. Also, the memory <NUM> may store software and/or programs. For example, the memory <NUM> may include an internal memory or an external memory. The internal memory may include at least one of volatile memory (e.g., DRAM, SRAM or SDRAM), and non-volatile memory (e.g., flash memory, hard drive, or solid state drive (SSD)). The external memory may be functionally or physically connected to the printed circuit board inspection apparatus <NUM> through various interfaces.

In one embodiment, the memory <NUM> may store instructions for operating the processor <NUM>. For example, the memory <NUM> may store instructions that cause the processor <NUM> to control other components of the printed circuit board inspection apparatus <NUM> and to interwork with an external electronic device or a server. The processor <NUM> may control the other components of the printed circuit board inspection apparatus <NUM> based on the instructions stored in the memory <NUM> and may interwork with the external electronic device or the server. Hereinafter, the operation of the printed circuit board inspection apparatus <NUM> will be described mainly with each component of the printed circuit board inspection apparatus <NUM>. Also, instructions for performing an operation by each component may be stored in the memory <NUM>.

In one embodiment, the memory <NUM> stores a machine-learning-based model. The machine-learning-based model receives first depth information on a first object using a pattern of light reflected from the first object among patterns of light irradiated from a plurality of second light sources. For example, the first depth information may include at least one of a shape, color information for each pixel, brightness information, and a height value.

For example, the plurality of second light sources and the plurality of first light sources <NUM> may be the same or different. Although the plurality of second light sources are different from the plurality of first light sources <NUM>, the number of the plurality of second light sources may be the same as the number of the plurality of first light sources <NUM>. Further, even if the plurality of second light sources are included in another printed circuit board inspection apparatus, the arrangement positions of the plurality of second light sources in the other printed circuit board inspection apparatus may correspond to the arrangement positions of the plurality of first light sources in the printed circuit board inspection apparatus <NUM>. In the machine-learning-based model, when the first depth information is input, first depth information with reduced noise is output.

For example, the first depth information generated using a pattern of light reflected from the first object may generate noise in multiple reflections of the pattern of light irradiated on the first object or in the process of processing the received light by the image sensor. For example, the noise may be a portion of the first depth information that does not correspond to the shape of the first object or that is determined not to be related to the first object. In order to improve the quality of the image on the first object, for example, the 3D image on the first object, the machine-learning-based model is trained to output the first depth information with reduced noise. Examples of the machine-learning-based model may include a convolutional neural network (CNN), a generative adversarial network (GAN), and the like. When the first depth information is input to the machine-learning-based model, a detailed method of training the machine-learning-based model to output the first depth information with reduced noise will be described later.

In addition, the machine-learning-based model may be stored in a memory of an external electronic device or server interworking with the printed circuit board inspection apparatus <NUM> by wire or wirelessly. In this case, the printed circuit board inspection apparatus <NUM> may transmit and receive information to and from the external electronic device or server interworked by wire or wirelessly to reduce the noise of the first depth information.

In one embodiment, the processor <NUM> may drive an operating system or an application program to control at least one other component of the printed circuit board inspection apparatus <NUM>, and may perform a variety of data processing, calculation, and the like. For example, the processor <NUM> may include a central processing unit or the like, or may be implemented as a system on chip (SoC).

In one embodiment, the communication circuit <NUM> may communicate with an external electronic device or an external server. For example, the communication circuit <NUM> may establish communication between the printed circuit board inspection apparatus <NUM> and an external electronic device. The communication circuit <NUM> may be connected to a network through wireless communication or wired communication to communicate with an external electronic device or external server. As another example, the communication circuit <NUM> may be connected to an external electronic device in a wired manner to perform communication.

The wireless communication may include, for example, cellular communication (e.g., LTE, LTE advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), etc.). Further, the wireless communication may include short-range wireless communication (e.g., Wi-Fi, light fidelity (Li-Fi), Bluetooth, Bluetooth low power (BLE), Zigbee, near field communication (NFC), etc.).

In one embodiment, the processor <NUM> may generate second depth information on the component by using the pattern of light reflected from the component mounted on the printed circuit board received by the image sensor <NUM>. For example, the processor <NUM> may generate second depth information on the component by using an image of the component generated using the pattern of light reflected from the component generated by the image sensor <NUM>. As another example, the image sensor <NUM> may transmit the received information on the pattern of light to the processor <NUM>, and the processor <NUM> may generate an image of the component and may use the image of the component to generate the second depth information on the component. The processor <NUM> may generate the second depth information on the component by applying an optical triangulation method or a bucket algorithm to the image of the component. However, this is for illustrative purposes only, and the present disclosure is not limited thereto, and the second depth information on the component may be generated through various methods.

In one embodiment, the processor <NUM> may input the second depth information to the machine-learning-based model. For example, when the machine-learning-based model is stored in the memory <NUM>, the processor <NUM> may directly input the second depth information to the machine-learning-based model. As another example, when the machine-learning-based model is stored in an external electronic device or an external server, the processor <NUM> may control the communication circuit <NUM> to transmit the second depth information to the external electronic device or the external server.

In one embodiment, the processor <NUM> may obtain the second depth information with reduced noise from the machine-learning based model. For example, when the machine-learning-based model is stored in the memory <NUM>, the processor <NUM> may input the second depth information with reduced noise directly from the machine-learning-based model. As another example, when the machine-learning based model is stored in the external electronic device or the external server, the processor <NUM> may obtain the second depth information with reduced noise from the external electronic device or the external server through the communication circuit <NUM>.

In one embodiment, the processor <NUM> inspects the mounting state of the component mounted on the printed circuit board by using the first depth information with reduced noise. For example, the processor <NUM> may generate a 3D image of the component using the first depth information with reduced noise. In addition, the processor <NUM> may inspect the mounting state of the component using the generated 3D image of the component. For example, the processor <NUM> may use the 3D image of the component to inspect whether the component is mounted at a predetermined position, whether the component is mounted in a predetermined direction, whether at least a portion of the component is tilted and mounted, whether there is a foreign object in the component, or the like, thereby inspecting the mounting state of the component.

In one embodiment not falling under the scope of the claimed invention, when visibility information about the first object is further input, the machine-learning-based model may output first depth information with reduced noise using the visibility information. For example, the visibility information is information indicating the degree of noise, and the machine-learning-based model may use the visibility information to effectively reduce the noise in the first depth information. A specific method for training the machine-learning-based model to output the first depth information with reduced noise when the first depth information and the visibility information are input to the machine-learning-based model will be described later.

In one embodiment not falling under the scope of the claimed invention, the processor <NUM> may generate visibility information about a component using a pattern of light which is reflected from the component and is received by the image sensor <NUM>. For example, visibility information represents the ratio of the amplitude (Bi(x, y)) of a brightness signal of image data to an average brightness (Ai(x, y)) and tends to generally increase as reflectivity increases. The visibility information (Vi(x, y)) may be represented by Equation <NUM>.

For example, patterns of light may be emitted respective from the plurality of first light sources <NUM> to the printed circuit board in various directions, thereby generating a plurality of pieces of image data of a component by the image sensor <NUM> or the processor <NUM>. The processor <NUM> may extract N brightness degrees (Ii1, Ii2,. , and IiN) at individual positions (i(x, y)) in an X-Y coordinate system from the plurality of pieces of generated image data and may calculate an average brightness (Ai(x, y)) using an amplitude (Bi(x, y)) and an N-bucket algorithm. The processor <NUM> may generate visibility information (Vi(x,y)) using the calculated amplitude (Bi(x, y)) and average brightness (Ai(x, y)). In addition, the processor <NUM> may further input the generated visibility information about the component into the machine-learning-based model.

In one embodiment, the machine-learning-based model may receive a plurality of pieces of depth information about a first object generated using a pattern of light reflected from the first object among patterns of light emitted from the plurality of second light sources. Since each of the plurality of second light sources emits a pattern of light to the first object and the pattern of light emitted by each of the plurality of second light sources is reflected from the first object and is received by the image sensor, a plurality of pieces of depth information about the first object may be generated.

When the plurality of pieces of depth information is input, the machine-learning-based model may generate and output first depth information with reduced noise. A specific method for training the machine-learning-based model to generate and output the first depth information with reduced noise when the plurality of pieces of depth information is input to the machine-learning-based model will be described later. For example, the first depth information is representative depth information about the first object and may be generated based on the plurality of pieces of depth information about the first object.

In one embodiment, the processor <NUM> may generate a plurality of pieces of depth information about a component using a pattern of light which is reflected from the component and is received by the image sensor <NUM>. Since each of the plurality of first light sources emits a pattern of light to the component and the pattern of light emitted by each of the plurality of first light sources is reflected from the component and is received by the image sensor <NUM>, a plurality of pieces of depth information about the component may be generated.

The processor <NUM> may input the plurality of pieces of depth information about the component into the machine-learning-based model. For example, each of the plurality of first light sources <NUM> emits a pattern of light to the component mounted on the printed circuit board, and the image sensor <NUM> may generate a plurality of pieces of image data about the component using the pattern of light reflected from the component. The image sensor <NUM> may transmit the plurality of pieces of image data to the processor <NUM>. The processor <NUM> may generate a plurality of pieces of depth information about the component using the plurality of pieces of image data.

In one embodiment, the processor <NUM> may obtain second depth information with reduced noise from the machine-learning-based model. The second depth information may be generated by the machine-learning-based model based on the plurality of pieces of depth information about the component. For example, the second depth information is representative depth information about the component and may be generated based on the plurality of pieces of depth information about the component.

In one embodiment, the machine-learning-based model may receive a plurality of pieces of image data about a first object generated using a pattern of light reflected from the first object among patterns of light emitted from the plurality of second light sources. When the plurality of pieces of image data is input, the machine-learning-based model may generate and output first depth information with reduced noise. A specific method for training the machine-learning-based model to generate and output the first depth information with reduced noise when the plurality of pieces of image data about the first object is input to the machine-learning-based model will be described later.

In one embodiment, the processor <NUM> may input a plurality of pieces of image data about a component generated using a pattern of light which is reflected from the component and is received by the image sensor <NUM> into the machine-learning-based model. In another example, the processor <NUM> may generate a plurality of pieces of image data about a component using information about a pattern of light which is reflected from the component and is received by the image sensor <NUM> and may input the plurality of pieces of generated image data into the machine-learning-based model.

In one embodiment, the processor <NUM> may obtain first depth information with reduced noise from the machine-learning-based model. Second depth information may be generated by the machine-learning-based model based on the plurality of pieces of image data.

<FIG> is a flowchart illustrating a method of inspecting a mounting state of a component by a printed circuit board inspection apparatus according to various embodiments of the present disclosure.

Although process steps, method steps, algorithms, and the like have been described in a sequential order in the flowchart shown in <FIG>, such processes, methods, and algorithms may be configured to be operated in arbitrary appropriate orders. In other words, the steps of the processes, methods, and algorithms described in various embodiments of the present disclosure need not be performed in the order described in this disclosure. Also, although some steps are described as being performed asynchronously, in other embodiments, some of these steps may be performed simultaneously. Also, the illustration of the process by depiction in the drawings does not mean that the illustrated process excludes other changes and modifications thereto, that any of the illustrated processes or steps thereof is essential to one or more of the various embodiments of the present disclosure, or that the illustrated process is preferred.

In operation <NUM>, the printed circuit board inspection apparatus <NUM> irradiates a component mounted on a printed circuit board with a pattern of light. For example, the processor of the printed circuit board inspection apparatus <NUM> may control a plurality of first light sources such that the pattern of light is irradiated to each of a plurality of components mounted on the printed circuit board to be inspected.

In operation <NUM>, the printed circuit board inspection apparatus <NUM> receives the pattern of light reflected from the component and may generate first depth information on the component using the pattern of light. For example, the first image sensor may generate an image of the component using the pattern of light reflected from the component, and may transmit the generated image of the component to the processor. The processor may generate the first depth information on the component using the image of the component and received from the first image sensor.

In operation <NUM>, the printed circuit board inspection apparatus <NUM> inputs the first depth information to a machine-learning-based model. For example, when the machine-learning-based model is stored in the memory of the printed circuit board inspection apparatus <NUM>, the processor may directly input the first depth information to the machine-learning-based model. As another example, when the machine-learning-based model is stored in an external electronic device or an external server, the processor may control a communication circuit to transmit the first depth information to the external electronic device or the external server.

In operation <NUM>, the printed circuit board inspection apparatus <NUM> obtains the first depth information with reduced noise from the machine-learning based model. For example, when the machine-learning-based model is stored in the memory, the processor may obtain the first depth information with reduced noise directly from the machine-learning-based model. As another example, when the machine-learning-based model is stored in the external electronic device or the external server, the processor may obtain the first depth information with reduced noise from the external electronic device or the external server through the communication circuit.

In operation <NUM>, the printed circuit board inspection apparatus inspects the mounting state of the component using the first depth information with reduced noise. For example, the processor may generate a 3D image of the component using the first depth information with reduced noise. In addition, the processor may inspect the mounting state of the component using the generated 3D image of the component.

<FIG> and <FIG> are conceptual diagrams illustrating a learning method of a machine-learning-based model according to various embodiments of the present disclosure. <FIG> shows an embodiment not falling under the scope of the claimed invention.

Referring to <FIG>, a machine-learning-based model <NUM> is trained to output second depth information with reduced noise <NUM>, using second depth information <NUM> on a second object generated using a pattern of light reflected from the second object among patterns of light irradiated from a plurality of second light sources and third depth information <NUM> on the second object generated using the pattern of light reflected from the second object among the patterns of light irradiated from a plurality of third light sources.

Based on results learned to output the second depth information with reduced noise <NUM>, the machine-learning-based model <NUM> may output first depth information with reduced noise even when the first depth information on the first object different from the second object used for learning is input.

In one embodiment, the second depth information <NUM> and the third depth information <NUM> is input to the machine-learning-based model <NUM> for learning. The number of the plurality of third light sources irradiating the pattern of light used to generate the third depth information <NUM> is larger than the number of a plurality of first light sources and larger than the number of a plurality of second light sources having the same number as the number of the plurality of first light sources. Since the number of the plurality of third light sources is larger than the number of the plurality of second light sources, the number of a plurality of images of the second object used in generating the third depth information <NUM> may be larger than the number of a plurality of images of the second object used in generating the second depth information <NUM>. Since the irradiation direction, irradiation angle, and pitch interval of each of the plurality of third light sources are different from each other, all of the plurality of images of the second object used in generating the third depth information <NUM> may be images of the second object, but they may be different images from each other. Similarly, since the irradiation direction, irradiation angle, and pitch interval of each of the plurality of second light sources are different from each other, all of the plurality of images of the second object used in generating the second depth information <NUM> may be images of the second object, but they may be different images from each other.

In addition, since the number of the plurality of third light sources is larger than the number of the plurality of second light sources, the plurality of third light sources may irradiate the second object with light while having at least one irradiation direction, at least one irradiation angle, and at least one pitch interval which are different from those of the plurality of second light sources. Accordingly, the number of the plurality of images of the second object used in generating the third depth information <NUM> is larger than the number of the plurality of images of the second object used in generating the second depth information <NUM>. As a result, the generated third depth information <NUM> may generate relatively less noise than the second depth information <NUM>. Accordingly, the shape of the object measured through depth information generated using a large number of light sources may be closer to the actual shape of the object compared to the shape of the object measured through depth information generated using a small number of light sources.

In one embodiment, since the third depth information <NUM> generates relatively less noise than the second depth information <NUM>, the third depth information <NUM> may be used as depth information that is a reference in a process in which the machine-learning-based model <NUM> transforms the second depth information <NUM> to reduce noise from the second depth information <NUM> or a process in which the machine-learning-based model <NUM> is trained to output noise from the second depth information <NUM>.

In one embodiment, the machine-learning-based model <NUM> is trained to transform the second depth information <NUM> to converge to the third depth information <NUM>. Hereinafter, for convenience of description, the second depth information <NUM> transformed to converge to the third depth information <NUM> is referred to as transformation depth information. For example, the machine-learning-based model <NUM> may compare the transformation depth information and the third depth information <NUM>. The machine-learning-based model <NUM> may adjust a parameter for transformation of the second depth information <NUM> based on the comparison result. By repeating the above process, the machine-learning-based model <NUM> may determine the parameter for transformation of the second depth information <NUM> such that the second depth information <NUM> converges to the third depth information <NUM>. Through this, the machine-learning-based model <NUM> may be trained to transform the second depth information <NUM> to converge to the third depth information <NUM>. The machine-learning-based model <NUM> may be trained to output the transformation depth information as the second depth information with reduced noise <NUM>. In this manner, the machine-learning-based model <NUM> may be trained to transform the second depth information <NUM> to converge to the third depth information <NUM>, so that the shape of the object can be measured more accurately even when the number of images of an object available in generating depth information is relatively insufficient.

In one embodiment, the machine-learning-based model <NUM> may be trained to detect noise from the second depth information <NUM>. For example, the machine-learning-based model <NUM> may be trained to detect noise from the second depth information <NUM> and to output the second depth information with reduced noise <NUM> by reducing the detected noise.

For example, the machine-learning-based model <NUM> may be trained to detect a first portion which is determined to be noise from the second depth information <NUM>, by comparing the transformation depth information and the second depth information <NUM>. For example, the machine-learning-based model <NUM> may be trained to detect a portion in which the difference between the transformation depth information and the second depth information <NUM> is equal to or larger than a predetermined threshold, as a first portion. The machine-learning-based model <NUM> may be trained to output the second depth information with reduced noise <NUM> by reducing the noise detected from the third depth information <NUM>.

Referring to <FIG> showing an embodiment not falling under the scope of the claimed invention, the machine-learning-based model <NUM> may be trained to output the second depth information with reduced noise <NUM>, using the second depth information <NUM>, the third depth information <NUM>, and visibility information <NUM> on the second object generated using a pattern of light reflected from the second object among patterns of light irradiated from the plurality of second light sources.

The machine-learning-based model <NUM> may output the first depth information with reduced noise even if the first depth information on the first object different from the second object used in learning and visibility information on the first object are input, based on results trained to output the third depth information with reduced noise <NUM>.

In one embodiment not falling under the scope of the claimed invention, the third depth information <NUM>, the fourth depth information <NUM>, and the visibility information <NUM> may be further input to the machine-learning-based model <NUM>. The machine-learning-based model <NUM> may be trained to adjust the transformation depth information by using the visibility information <NUM> to more accurately represent the shape of the second object. For example, the visibility information <NUM> is information indicating the degree of noise occurring in the second depth information <NUM>, which is depth information about the second object, and may indicate whether the third second information <NUM> is a quality measurement value. For example, the machine-learning-based model <NUM> may be trained to determine whether there is a second portion having a preset threshold or greater in the visibility information <NUM>.

For example, when the second portion exists, the machine-learning-based model <NUM> may be trained to determine a portion of the transformation depth information corresponding to the second portion and to adjust the part corresponding to the second portion based on the visibility information <NUM>. The machine-learning-based model <NUM> may be trained to output the adjusted transformation depth information as the second depth information with reduced noise <NUM>.

In another example, when no second portion exists, the machine-learning-based model <NUM> may be trained to determine not to adjust the transformation depth information and to output the transformation depth information as the second depth information with reduced noise <NUM>.

In one embodiment not falling under the scope of the claimed invention, in order to more accurately detect noise, the machine-learning-based model <NUM> may be trained to detect a third portion that is determined to be noise although it is not actually noise from the first part determined to be noise, using the visibility information <NUM>. When the third portion is detected, the machine-learning-based model <NUM> may be trained to exclude the third portion from the first portion and to determine the first portion from which the third portion is excluded to be the noise from the second depth information <NUM>. Also, when the third portion is not detected, the machine-learning-based model <NUM> may be trained to output the second depth information with reduced noise <NUM> by determining the first portion to be the noise in the second depth information <NUM> and reducing the noise determined in the second depth information <NUM>.

Referring to <FIG>, a machine-learning-based model <NUM> may be trained to generate the second depth information <NUM> about the second object using a plurality of pieces of depth information <NUM>, <NUM>, <NUM>, and <NUM> about the second object generated using a pattern of light reflected from the second object among patterns of light emitted from a plurality of second light sources. The machine-learning-based model <NUM> may be trained to output second depth information with reduced noise <NUM> using the generated second depth information <NUM> and third depth information <NUM>. A specific method for training the machine-learning-based model <NUM> to output the second depth information with reduced noise <NUM> using the second depth information <NUM> and the third depth information <NUM> is the same as that illustrated <FIG>, and thus a description thereof is omitted.

In one embodiment, the plurality of pieces of depth information <NUM>, <NUM>, <NUM>, and <NUM> about the second object generated using the pattern of light reflected from the second object among the patterns of light emitted from the plurality of second light sources may be input to the machine-learning-based model <NUM>. The machine-learning-based model <NUM> may be trained to generate the second depth information <NUM>, which is representative depth information about the second object, using the plurality of pieces of depth information <NUM>, <NUM>, <NUM>, and <NUM>.

Further, although not shown, a plurality of pieces of image data about the second object generated using the pattern of light reflected from the second object among the patterns of light emitted from the plurality of second light sources may be input to the machine-learning-based model <NUM>. The machine-learning-based model <NUM> may be trained to generate the second depth information <NUM> about the second object using the plurality of pieces of image data about the second object.

In one embodiment, a plurality of pieces of image data about the second object generated using the pattern of light reflected from the second object among the patterns of light emitted from the plurality of second light sources may be input to the machine-learning-based model <NUM>. The machine-learning-based model <NUM> may be trained to generate the second depth information <NUM> using the plurality of pieces of image data.

In one embodiment, the third depth information <NUM> generated using a pattern of light irradiated from a plurality of third light sources may be generated by the printed circuit board inspection apparatus including the number of plurality of third light sources larger than the number of plurality of second light sources. In addition, the third depth information <NUM> may be generated by the printed circuit board inspection apparatus including the number of plurality of second light sources smaller than the number of plurality of third light sources. In this case, a detailed method of generating the third depth information <NUM> will be described in <FIG>.

<FIG> and <FIG> are conceptual diagrams illustrating the operation of a machine-learning-based model according to various embodiments of the present disclosure. <FIG> shows an embodiment not falling under the scope of the claimed invention.

Referring to <FIG>, first depth information <NUM> on a first object generated using a pattern of light reflected from the first object among patterns of light irradiated from a plurality of second light sources may be input to the machine-learning-based model <NUM>. In addition, when the first depth information <NUM> is input, the machine-learning-based model <NUM> may output first depth information with reduced noise <NUM>.

Hereinafter, for convenience of description, depth information generated using light irradiated from a plurality of third light sources greater than the number of the plurality of second light sources is referred to as reference depth information, and the first depth information <NUM> transformed to converge to the reference depth information by the machine-learning-based model <NUM> is referred to as transformation depth information.

In one embodiment, the machine-learning-based model <NUM> may transform the first depth information <NUM> to converge to the reference depth information. In this case, the machine-learning-based model <NUM> may output the transformation depth information as the first depth information with reduced noise <NUM>.

In one embodiment, the machine-learning-based model <NUM> may detect noise from the first depth information <NUM>. For example, the machine-learning based model <NUM> may detect noise from the first depth information <NUM> and may output the first depth information with reduced noise <NUM> by reducing the detected noise.

For example, the machine-learning-based model <NUM> may detect a first portion determined to be noise by comparing the transformation depth information and the first depth information <NUM>. For example, the machine-learning-based model <NUM> may detect a portion in which the difference between the transformation depth information and the first depth information <NUM> is equal to or larger than a predetermined threshold, as the first portion. The machine-learning-based model <NUM> may output the first depth information with reduced noise <NUM> by reducing the noise detected from the first depth information <NUM>.

Referring to <FIG> showing an embodiment not falling under the scope of the claimed invention, the first depth information <NUM> and visibility information <NUM> about a second object generated using a pattern of light reflected from the second object among the patterns of light emitted from the plurality of second light sources may be input to a machine-learning-based model <NUM>. When the second depth information <NUM> and the visibility information <NUM> are input, the machine-learning-based model <NUM> may output first depth information with reduced noise <NUM> using the visibility information <NUM>.

In one embodiment, the machine-learning-based model <NUM> may determine whether there is a second part having a preset threshold or greater in the visibility information <NUM>. For example, when the second part exists, the machine-learning-based model <NUM> may determine a part of transformation depth information corresponding to the second part and may adjust the part corresponding to the second part based on the visibility information. The machine-learning-based model <NUM> may output the adjusted transformation depth information as second depth information with reduced noise <NUM>.

In another example, when no second part exists, the machine-learning-based model <NUM> may be trained to determine not to adjust the transformation depth information and to output the transformation depth information as the second depth information with reduced noise <NUM>.

In one embodiment not falling under the scope of the claimed invention, the machine-learning-based model <NUM> may detect a third portion that is determined to be noise although it is not actually noise from the first portion determined to be noise, by using the visibility information <NUM>. When the third portion is detected, the machine-learning-based model <NUM> may exclude the third portion from the first portion and may determine the first portion from which the third portion is excluded to be noise in the second depth information <NUM>. In addition, when the third portion is not detected, the machine-learning-based model <NUM> may determine the first portion to be the noise in the seond depth information <NUM>. The machine-learning-based model <NUM> may output the second depth information with reduced noise <NUM> by reducing the noise that is determined in the second depth information <NUM>.

Referring to <FIG>, a plurality of pieces of depth information <NUM>, <NUM>, <NUM>, and <NUM> about the first object generated using the pattern of light reflected from the second object among the pattern lights emitted from the plurality of second light sources may be input to a machine-learning-based model <NUM>. The machine-learning-based model <NUM> may generate the first depth information <NUM>, which is representative depth information about the first object, using the plurality of pieces of depth information <NUM>, <NUM>, <NUM>, and <NUM>. After generating the first depth information <NUM>, the machine-learning-based model <NUM> may output the first depth information with reduced noise <NUM> as described in <FIG>.

Further, although not shown, a plurality of pieces of image data about the first object generated using the pattern of light reflected from the second object among the patterns of light emitted from the plurality of second light sources may be input to the machine-learning-based model <NUM>. The machine-learning-based model <NUM> may generate the first depth information <NUM>, which is representative depth information about the first object, using the plurality of pieces of image data. After generating the first depth information <NUM>, the machine-learning-based model <NUM> may output the first depth information with reduced noise <NUM> as described in <FIG>.

In this manner, even when a relatively small number of pieces of image data are acquired to generate depth information, the printed circuit board inspection apparatus <NUM> may remove noise, such as an unreceived signal or a peak signal, from the depth information on the component by using machine-learning-based models <NUM>, <NUM>, and <NUM>. Also, the printed circuit board inspection apparatus <NUM> may generate the depth information on the component so that the lost shape can be restored using the machine-learning-based models <NUM>, <NUM>, and <NUM> even if a relatively small number of pieces of image data are obtained and thus information for generating the depth information is insufficient. In addition, the printed circuit board inspection apparatus <NUM> may not perform error restoration of the joint shape of the component while correcting the 3D sharpness of the edges of the component as much as possible, and may detect the shape of an additionally measured foreign material without deteriorating the same.

<FIG> is a conceptual diagram illustrating a learning method of a machine-learning-based model according to various embodiments of the present disclosure.

In one embodiment, a machine-learning-based model <NUM> may include CNN, GAN, and the like. Hereinafter, a learning method of a machine-learning-based model will be described, focusing on GAN, which can perform image transformation using U-net. The machine-learning-based model <NUM> may include a generator <NUM> and a separator <NUM>.

In one embodiment, second depth information <NUM> on a second object generated using a pattern of light reflected from the second object among patterns of light irradiated from a plurality of second light sources is input to the generator <NUM>. Third depth information <NUM> on the second object generated using the pattern of light reflected from the second object among the patterns of light irradiated from a plurality of third light sources may be input to the separator <NUM>.

In one embodiment, the generator <NUM> may generate transformed second depth information by transforming the second depth information <NUM> to converge to the third depth information <NUM>. The separator <NUM> may separate the transformed second depth information and the third depth information <NUM> by comparing the transformed second depth information and the third depth information <NUM>. The separator <NUM> may transmit results obtained by separating the transformed second depth information and the third depth information <NUM> to the generator <NUM>. The generator <NUM> may adjust a parameter for transformation of the second depth information <NUM> according to the result received from the separator <NUM>. This process is repeated until the separator <NUM> cannot separate the transformed second depth information and the third depth information <NUM>, so that the generator <NUM> may be trained to generate transformed second depth information by transforming the second depth information <NUM> to converge to the third depth information <NUM>.

Meanwhile, in the generator <NUM>, the second depth information <NUM> and the third depth information <NUM> on any specific component form a pair. In a case in which any of the second depth information <NUM> and the third depth information <NUM> has a poor quality (a case in which depth information of any one channel, such as a shadow area, a saturation area, and an SNR, for each of at least one pixel is significantly lower than a predetermined reference value compared to other channels), the generator <NUM> may additionally perform a refinement operation to exclude the corresponding component data from learning data.

<FIG> is a diagram illustrating a method of acquiring depth information used in training a machine-learning-based model according to various embodiments of the present disclosure.

As described in <FIG>, the third depth information <NUM> may be generated in the printed circuit board inspection apparatus in which the second depth information <NUM> is generated. For example, as illustrated in <FIG>, it is assumed that the number of a plurality of second light sources <NUM> is <NUM> and the number of a plurality of third light sources is <NUM>. In this case, the processor of the printed circuit board inspection apparatus <NUM> may control the plurality of second light sources <NUM> to irradiate the component mounted on the printed circuit board with a pattern of light, may generate the second depth information <NUM> by using the pattern of light reflected from the component, and may then move the plurality of second light sources <NUM> clockwise or counterclockwise. The processor may control the plurality of second light sources <NUM> moved clockwise or counterclockwise so as to irradiate the component mounted on the printed circuit board with the pattern of light, and may generate the third depth information <NUM> by using the pattern of light reflected from the component and the second depth information <NUM>. Meanwhile, in <FIG>, it has been described that the number of the plurality of second light sources <NUM> is <NUM>, but this is for illustrative purposes only, and is not limited thereto. The number of the second light sources <NUM> may be one rather than plural. As such, the third depth information <NUM> may be generated in the printed circuit board inspection apparatus including the plurality of second light sources having a smaller number than the number of the plurality of third light sources.

<FIG> illustrates an image of a component generated using depth information with reduced noise by a printed circuit board inspection apparatus according to various embodiments of the present disclosure.

In one embodiment, the printed circuit board inspection apparatus <NUM> may generate depth information on a component by using a pattern of light reflected from the component among patterns of light irradiated on the component mounted on the printed circuit board, from a plurality of first light sources. In addition, the printed circuit board inspection apparatus <NUM> may generate a 3D image of the component by using the generated depth information. However, noise may occur in multiple reflections of light irradiated on the printed circuit board or in the process of processing the received light by the image sensor. If the generated noise is not reduced, the quality of the 3D image of the component generated by the printed circuit board inspection apparatus <NUM> may be deteriorated, and accurate inspection of the mounting state of the component may not be performed.

In one embodiment, the printed circuit board inspection apparatus <NUM> may reduce noise from the depth information on the component by using a machine-learning-based model, and may generate the 3D image of the component by using the depth information with reduced noise. Since the 3D image generated using the depth information with reduced noise may more accurately display the shape of the component, more accurate inspection of the mounting state of the component can be performed.

Referring to <FIG>, when the printed circuit board inspection apparatus <NUM> generates the 3D image of the component by using the depth information as is without reducing noise, a shape of a connection portion <NUM> (e.g., solder paste or the like) between the component and the printed circuit board may be displayed as an abnormal shape in the 3D image or displayed as having a hole, due to the noise. On the other hand, when the printed circuit board inspection apparatus <NUM> reduces noise in the depth information on the component by using the machine-learning-based model and generates the 3D image of the component by using the depth information with reduced noise, a shape of a connection portion <NUM> (e.g., solder paste or the like) between the component and the printed circuit board may be more accurately displayed in the 3D image.

Referring to <FIG>, when the printed circuit board inspection apparatus <NUM> generates the 3D image of the component by using the depth information as is without reducing noise, a shape of an edge <NUM> of the component may be displayed as an abnormal shape in the 3D image due to the noise. On the other hand, when the printed circuit board inspection apparatus <NUM> reduces noise in the depth information on the component by using the machine-learning-based model and generates the 3D image of the component by using the depth information with reduced noise, a shape of the edge <NUM> of the component may be more accurately displayed in the 3D image.

Referring to <FIG>, when the printed circuit board inspection apparatus <NUM> generates the 3D image of the component by using the generated depth information as is, an internal shape <NUM> of the component may be displayed as an abnormal shape in the 3D image such as having a hole in the component due to the noise. On the other hand, when the printed circuit board inspection apparatus <NUM> reduces noise in the depth information on the part by using the machine-learning based model and generates the 3D image of the component by using the depth information with reduced noise, an internal shape <NUM> of the component may be more accurately displayed in the 3D image.

As described above, the printed circuit board inspection apparatus <NUM> may display the shape of the component more accurately through the 3D image generated using the depth information with reduced noise, thereby performing more accurate inspection of the mounting state of the component.

While the foregoing methods have been described with respect to particular embodiments, these methods may also be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recoding medium includes any kind of data storage devices that can be read by a computer system. Examples of the computer-readable recording medium includes ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device and the like. Also, the computer-readable recoding medium can be distributed to the computer systems which are connected through a network so that the computer-readable codes can be stored and executed in a distribution manner. Further, the functional programs, codes and code segments for implementing the foregoing embodiments can easily be inferred by programmers in the art to which the present disclosure pertains.

Claim 1:
A printed circuit board inspection apparatus (<NUM>) that inspects a mounting state of a component mounted on a printed circuit board (<NUM>) and includes a plurality of first light sources (<NUM>); an image sensor (<NUM>); a memory (<NUM>); and a processor (<NUM>), wherein the plurality of first light sources (<NUM>) is configured to irradiate the component with a pattern of light;
the image sensor (<NUM>) is configured to receive a pattern of light reflected from the component;
the memory (<NUM>) is configured to store a machine-learning-based model (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>);
wherein the processor (<NUM>) is configured to:
generate first depth information on the component by using the pattern of light reflected from the component and received by the image sensor (<NUM>);
input the first depth information into the machine-learning-based model (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>);
obtain first depth information with reduced noise from the machine-learning-based model (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>); and
inspect the mounting state of the component by using the first depth information with reduced noise,
wherein the machine-learning-based model (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is trained to transform second depth information (<NUM>) on an object generated using a plurality of different images of patterns of light reflected from the object among patterns of light irradiated from a plurality of second light sources to converge to third depth information (<NUM>) on the-object generated using a plurality of different images of patterns of light reflected from the object among patterns of light irradiated from a plurality of third light sources,
wherein the machine-learning-based model (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) outputs the first depth information with reduced noise when the first depth information is input based on a training result,
wherein the number of the plurality of second light sources is the same as the number of the plurality of first light sources (<NUM>), and the number of the plurality of third light sources is larger than the number of the plurality of first light sources (<NUM>), and
wherein the number of the plurality of different images used for generating the third depth information (<NUM>) is larger than the number of the plurality of different images used for generating the second depth information (<NUM>).