# Detecting and Measuring Defects in Wafer Die Using GAN and YOLOv3
## 2 Research Method
### Defect Data Augmentation by GAN and Their Autoannotation  
As proposed by [PERSON] et al. [17], GAN has a wide range of applications, such as fashion, advertising, science, and games. Since the images for defect detection are usually captured in a stable environment, each image is roughly the same regardless of location or color. Therefore, the general traditional data augmentation method is not necessarily applicable. The case company does not have much time for engineers to collect a huge image set, let alone an additional time-consuming annotation work for a large number of image sets. To overcome this difficulty, this study took advantage of the powerful generative capabilities of GAN to create richer types of defects. The basic idea of GAN is shown in Appendix A.1 of Appendix A.  
When we directly input a set of die images into the GAN model, however, we found that its objective function value fluctuated during the iteration and converged only with difficulty. Additionally, we also found that the GAN model could not generate high-resolution pseudo images effectively. It could only generate the approximate outline of the die, but the details could not be identified. Consequently, a strategy of simply generating the particle defects was adopted.  
The detailed process is shown in Figure 3. We used the defect coordinate position, and the information of length and width that the annotator has previously noted in the real image, which helped to cut out the patch that indicates the area of each particle defect. Otsu binarization [18] was then used to eliminate the background in the patch as far as possible, retaining the original appearance of the particle defects, and the defects were attached to a white background image with the same size as the GAN input image. As shown in the bottom left of Figure 3, GAN is composed of two networks: a generator and a discriminator. During the first iteration, the generator generated poor pseudo images and the discriminator distinguished them from real images easily. During the second iteration, the quality of pseudo images generated by the generator was improved, which fooled the underlying discriminator. With the rise of the ability of the discriminator, real images and pseudo images can be recognized, which will also drive the improvement of the generator. The training process of adversarial learning between the two networks was continued and a generative model similar to the real image distribution was created. As the learning object became simpler, the objective function converged rapidly, and generated more realistic pseudo particle defects. Finally, we embedded the pseudo particle defects into the defect-free dies to create a generative pseudo image set.  
Figure 3: The flowchart of GAN-based data argumentation and auto-annotation.  
Although we used GAN for data augmentation to increase the diversity of defects, the annotation files of these pseudo defective images were not generated. In line with the previous literature, an additional manual method was adopted to annotate the pseudo defect images [14; 15]. In order to save time when annotating the pseudo defective images, DIP techniques were used to automatically annotate the pseudo particles as shown in the bottom right of Figure 3. The CCL algorithm [16] scanned the image from left to right and top to bottom. If the gray values between adjacent pixels were found to be similar during scanning, they were labeled with a same index. Each pseudo particle defect would be regarded as a blob, and information of its minimum bounding box could also be registered. Then the XML annotation file of the pseudo defective image could be output, which reduced the time spent in annotating the pseudo defective images.