Patent ID: 12260633

DETAILED DESCRIPTION

It will be appreciated that for simplicity and clarity of illustration, where appropriate, reference numerals have been repeated among the different figures to indicate corresponding or analogous elements. Additionally, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein can be practiced without these specific details. In other instances, methods, procedures, and components have not been described in detail so as not to obscure the related relevant feature being described. The drawings are not necessarily to scale and the proportions of certain parts may be exaggerated to better illustrate details and features. The description is not to be considered as limiting the scope of the embodiments described herein.

Several definitions that apply throughout this disclosure will now be presented.

The term “coupled” is defined as connected, whether directly or indirectly through intervening components, and is not necessarily limited to physical connections. The connection can be such that the objects are permanently connected or releasably connected. The term “comprising” means “including, but not necessarily limited to”; it specifically indicates open-ended inclusion or membership in a so-described combination, group, series, and the like.

FIG.1illustrates a defect detection system100in accordance with an embodiment of the present disclosure.

The system structure and related components in the defect detection system100of the present disclosure will be described below throughFIG.1.

The defect detection system100is connected to a first detection device10, The first detection device10has an optical element for photographing the object20and obtaining the image of the object. The first detection device10performs a preliminary defect detection of the object through the image of the object20. If the first detection device10determines as a result that a certain area (i.e., the first area) on the object has a defect, the image of the area (i.e., the first image) is cut out and transmitted to the defect detection system100.

In some embodiments, the first detection device10is, for example, an automatic optical inspection equipment (AOI) or an automatic visual inspection equipment (AVI). The objects20are, for example, various types of circuit boards. The common defects are, for example, inner and outer layer deviation, finger discoloration, foreign matter on fingers, finger copper exposure, unclear text, ink falling off, gold face discoloration, gold face foreign matter, gold face copper exposure, finger crush injury, anti-welding falling off, gold face crush injury, or soft board dirt. The first image is an image regarded as showing features of defects on the first area on the object and is determined to have defects. The present disclosure is not limited to the above-mentioned cases of automatic optical detection devices, objects, and defects.

In the embodiment, the defect detection system100includes a recheck station110, a server120, and a database130.

The recheck station110is connected to the first detection device10and is used to receive the data of the first image and the first image output from the first detection device10. The data of the first image can be the size of the first image (such as length, width) and production line data of the object20(such as station, material, time, batch, test result, packaging) in the production line.

In some embodiments, the recheck station110also stores the received first image in the recheck station110.

In some embodiments, the recheck station110is interconnected with other equipment in the defect detection system100and acts as a relay station for integrating equipment and data transmission. For example, if the first image is stored in a specific directory in the hard disk of the recheck station110, the specific directory is set as a shared folder, and the shared folder is mounted on the server120, the server120can obtain the first image from the shared folder.

In some embodiments, the recheck station110is a verification repair server, which is not specifically limited in the present disclosure.

The server120is connected between the recheck station110and the database130. The server120stores a detection model in AI. The server120is used to obtain the first image, select the corresponding detection model according to the size of the first image, detect the defect of the first image according to the detection model, and output the detection result to the recheck station110. The recheck station110is also used to display the detection results output by the server120.

In some embodiments, the server120is, for example, a server or a server cluster including multiple servers, which is not specifically limited in the present disclosure.

The database130is used to store the detection results output by the first detection device10and the detection results output by the server120.

In some embodiments, the database130is used to store the data of the first image outputted by the recheck station110.

In some embodiments, the database130may be a local database. The database130is, for example, MySQL or ES (elastic search), which is not specifically limited in the embodiment of the present disclosure.

In some embodiments, the defect detection system100also includes a first software (not shown), which can run in the recheck station110. The first software is used to monitor the data of the first image, enter the recheck station110, store the data of an image in the database130, and store the data of the first image in a queue. The first software is also used to receive the detection result determined by the server120and display the detection result on the display screen of the recheck station110.

In some embodiments, the defect detection system100also includes a second software (not shown), which can run on the server120. The second software is used to take out the data of the first image from the queue and transmit the data of the first image to the server120. The second software is also used to receive the detection results output by the server120and transmit the detection results output by the server120to the first software.

In the embodiment, when the recheck station110is connected to the server120and the database130, data is exchanged through physical lines and various input ports, such as universal serial bus (USB), serial port, FireWire (IEEE 1394 standard), VME bus. Alternatively, the recheck station110can also be implemented through various communication chips, such as BLUETOOTH chip and WI-FI chip, so as to exchange data. The embodiment of the present disclosure is not specifically limited to this.

The following describes a process in which various elements interact in the defect detection system100to perform a defect detection method.

FIG.2is a schematic diagram of the working process of the defect detection system according to the embodiment of the present disclosure.

The user puts the object20into the first detection device10, and the first detection device10takes pictures of the object20and performs the first detection. If it is determined that the object is free of defects, there is no need to further examine the object20. If it is determined that the object20is defective, the first detection device10will acquire the first image of the first area on the object20and transmit the first image to the recheck station110. The recheck station110stores the detection result of the first detection of the object20by the first detection device10and the first image of the object20in a specific directory in the hard disk of the recheck station110. The specific directory is set as a shared folder and mounted on the server120. The first software loaded in the recheck station110monitors the entry of new data (such as the data of a new first image), stores the new data in the database130, and waits in line for the automatic determination of the server120. The second software extracts the data of the first image queued for processing from the database130and transmits the data of the first image to the server120. The server120also takes out the first image from the shared folder, and the server120performs defect detection based on the data of the first image and the first image. The server120transmits the detection result back to the second software, and the second software transmits the detection result of the server120to the first software. The first software transmits the detection result of the server120back to the display software in the recheck station110to display the detection result of the server120in the recheck station110.

FIG.3shows a flow chart of one embodiment of a defect detection method of the present disclosure.

In one embodiment, the defect detection method can be applied to the server120.

As shown inFIG.3, the defect detection method according to the embodiment of the present disclosure includes the following steps:

At block31, obtaining a first image corresponding to a first area on the object, the first area is the area where the object is determined to be defective on the first detection.

The first detection device determines that the area of the defect is not unique, other defects may be apparent and thus the number of first areas may be multiple. Accordingly, there may be number of first images, which is not specifically described in the present disclosure.

At block32, selecting a corresponding detection model according to a size of the first image.

In some embodiments, the selection of a corresponding detection model according to a size of the first image includes: selecting the corresponding detection model according to a proportion of size of the first image.

In some embodiments, the selection of the corresponding detection model according to the proportion of the size of the first image includes:selecting a first detection model when the length of the first image is greater than the width of the first image;selecting a second detection model when the length of the first image is less than the width of the first image in the manner of a landscape image; andselecting a third detection model when the length and width of the first image are equal.

The first detection model, the second detection model, and the third detection model are different.

In the embodiment, the products produced on the production line are diverse, the shapes and structures of objects20are diverse, and the sizes of the first image cut into the first area determined as a defect in the object20by the first detection device10is different. If multiple models are trained according to different products, their applicability is too low and too many models are difficult to manage. Therefore, only three detection models are trained according to the proportions of the size of the first image.

For different products, the size of the components on the product is generally different, and the locations of the components on the product are also different. The components set on the object can be determined, after the first detection device10produces the first image according to the size of each component on the object, the first detection device can distinguish by reference to length and width proportions of the size of the first image, and train three different detection models accordingly, so as to maximize the utility of the model.

In some embodiments, the angle and direction of the first detection device10photographing the object are the same, the length direction of the first image is parallel to the length direction of the image of the object, and the width direction of the object is parallel to the width direction of the image thereof.

FIG.4shows a flow chart of one embodiment of a model training method of the present disclosure.

As shown inFIG.4, the model training method according to the embodiment of the present disclosure includes the following steps:

At block41, obtaining a first network and a second network.

The first network is variational automatic encoder (VAE) network, and the second network is depth residual network (RESNET).

In the embodiment, the VAE network includes an input layer, a coding network, an intermediate hidden variable layer, and a decoding network. The depth RESNET includes a convolution layer, a pooling layer, a hidden layer, and a full connection layer.

The RESNET is ResNet34 network.

At block42, integrating the first network and the second network and constructing dual model training network.

In the embodiment, the hidden layer containing feature vectors of VAE network and the hidden layer containing feature vectors of RESNET are integrated in parallel, to obtain the dual model training network. A Softmax classifier can also be used as the output layer of the network.

In the embodiment, the Resnet34 network is used to reduce the gradient disappearance problem caused by too many layers in model training. With the new application of the VAE network, the mis-determinations causing a low detection rate can be improved. The dual model training network re-determines the features of a flawless image through reinforcement learning, and finally calculates the differences in features between the incoming image and the reconstructed image to more accurately classify the apparent defects in the image.

At block43, obtaining a set of training images.

In the embodiment, the set of the training images include a plurality of labeled training images. The training images are manually labeled training sample images, which are used to train the detection model.

The server is mounted on the recheck station, so that the image captured by the first detection device and the data document about the captured image generated by the first detection device can be obtained, and the document content can be analyzed. Cutting the image according to the size of the first area and marking it as defective or flawless, and saving the marked image (training image) to the specified path are carried out. The cut images are divided into three groups according to the proportion of the size, the length of the training image is greater than the width, the length of the training image is less than the width, and the length of the training image is equal to the width, the three groups of training images are used to train three models respectively to maximize the utility of the model.

At block44, inputting the first training image in the training image set into the dual model training network for training to obtain the first detection model.

The length of the first training image is greater than the width of the first training image.

At block45, inputting the second training image in the training image set into the dual model training network for training to obtain the second detection model.

The length of the second training image is less than the width of the second training image.

At block46, inputting the third training image in the training image set into the dual model training network for training to obtain the third detection model.

The length of the third training image is equal to the width of the third training image.

In the embodiment, the VAE network and RESNET are used for dual model training. The production line data is imported into the local database every day. The online detection model can calculate the accuracy of the model through the data in the database. The user or server can decide whether to update the detection model according to the accuracy of the model.

At block33, detecting defect on the first image according to the detection model.

In the embodiment, the server detects the defect of the first image according to the detection model. If the first image is detected as a defect, the detection result that the object is defective is output. If the first image is detected to be flawless, the detection result that the object is flawless is output.

At block34, outputting a detection result.

In the embodiment, the server outputs the detection result, and displays the detection result. If the detection result is a defect confirmed, the output object is defective. If the detection result is flawless, the output object is flawless.

FIG.5illustrates an electronic device140in accordance with an embodiment of the present disclosure.

The electronic device140can further include, but is not limited to, at least one processor141, a communication bus142, a storage device143, and at least one communication interface60. The processor141, the storage device143and the communication interface142can be connected and communicate with each other through the communication bus142. The processor141may execute the program code of program segment stored in the storage device143to implement blocks31-34in method shown inFIG.3and blocks41-46in method shown inFIG.4.

The block diagram merely shows an example of the electronic device140and does not constitute a limitation to the electronic device140. In other examples, more or less components than those illustrated may be included, or some components may be combined, or different components used. For example, the electronic device140may also include input and output devices, a network access device, a bus, and the like.

The processor141may be a central processing unit (CPU), or may be other general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a Field-Programmable gate array (FPGA) or other programmable logic device, a transistor logic device, a discrete hardware component. The processor141may also be any conventional processor. The processor141is a control center of the electronic device200. The processor141may include one or more chips, such as a Pentium chip, and the processor141may include an AI accelerator, such as a neural processing unit (NPU).

The communication bus142may include a path for transmitting information between various components of the electronic device140(such as the processor141, the storage device143, and the communication interface144).

The storage device143can be used to store program segments. The processor141operates or executes the program segments stored in the storage device143and recalls data stored in the storage device143and implements various functions of the electronic device140. The storage device143may mainly include a storage program area and a storage data area, the storage program area may store an operating system, an application (such as sound playback and image playback) required for at least one function. The storage data area may store data created.

The storage device143may include a RAM, the storage device143may also include non-volatile memory such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), and a Secure Digital (SD) card, a flash card, at least one disk storage device, flash device, or other volatile or non-volatile solid-state storage device.

The storage device143may exist independently and be connected to the processor141through the communication bus142. The storage device143may also be integrated with the processor141.

The communication interface144uses devices such as any transceiver to communicate with other devices or communication networks, such as Ethernet, wireless access network (RAN), wireless local area networks (WLAN).

In one embodiment, the processor141may include one or more CPUs.

In one embodiment, the electronic device may include a plurality of processors. Each of these processors can be a single core processor or a multi-core processor. The processor herein may refer to one or more devices, circuits, and/or processing cores for processing data, such as computer program instructions.

The embodiment of the present disclosure also provides a non-transitory storage medium. The computer instruction is stored in the non-transitory storage medium. When the instruction runs on the electronic device140, the electronic device140can execute the defect detection method provided by the above embodiments. The computer program may be stored in a computer readable storage medium. The steps of the various method embodiments described above may be implemented by a computer program when executed by a processor. The computer program includes a computer program code, which may be in the form of source code, object code form, executable file, or some intermediate form. The computer readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random-access memory (RAM), electrical carrier signals, telecommunications signals, and software distribution media.

The embodiment of the present disclosure also discloses a computer program. When the computer program product runs on the computer, the computer is caused to perform the above related steps to realize the defect detection method in the above embodiments.

The embodiment of the present disclosure also provides a device, which can be a chip, component or module, and the device can include a connected processor and a storage device. The storage device is used to store the computer execution instructions. When the device is running, the processor can execute the computer execution instructions stored in the storage device to enable the chip to execute the defect detection method in the above embodiments.

The electronic device, the computer storage medium, the computer program of the method or the chip provided in this embodiment are used to execute the method provided above. Therefore, the beneficial effects it can achieve can refer to the beneficial effects in the method provided above and will not be repeated here.

For the convenience and simplicity of description, only the division of the above functional modules is illustrated. In practical application, the above functions can be allocated by different functional modules according to needs, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

The device embodiments described above are only schematic. For example, the division of the module or unit is only a logical function division. In actual implementation, there can be another division mode. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. On the other hand, the mutual coupling or direct coupling or communication connection shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical, or other forms.

The unit described as a separate part can be or may not be physically separated, and the part displayed as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed to multiple different places. Some or all of the units can be selected according to the actual needs to achieve the purpose of the embodiment.

Each functional unit in each embodiment of the present disclosure can be integrated into one processing unit, each unit can exist separately, or two or more units can be integrated into one unit. The above integrated units can be realized in the form of hardware or software functional units.

If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. In essence, the technical solution of the embodiment of the present disclosure or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single chip microcomputer, chip) or processor to perform all or part of the steps of the method described in each embodiment of the present disclosure. The storage media includes USB flash disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disc or optical disc and other media that can store program code.

Even though numerous characteristics and advantages of the present technology have been set forth in the foregoing description, together with details of the structure and function of the present disclosure, the disclosure is illustrative only, and changes may be made in the detail, especially in matters of shape, size, and arrangement of the parts within the principles of the present disclosure, up to and including the full extent established by the broad general meaning of the terms used in the claims. It will therefore be appreciated that the exemplary embodiments described above may be modified within the scope of the claims.