Defect Transfer GAN: Diverse Defect Synthesis for Data Augmentation
Authors: Ruyu Wang
Sabrina Hoppe
Eduardo Monari
Marco Huber
Venue: ICLR 2022 Submitted (2022)
Keywords: Defect synthesis
Generative Adversarial Networks
Content transfer
Automated visual inspection
Data augmentation

Abstract: Large amounts of data are a common requirement for many deep learning approaches. However, data is not always equally available at large scale for all classes. For example, on highly optimized production lines, defective samples are hardly acquired while non-defective samples come almost for free. The defects however often seem to resemble each other, e.g., scratches on different products may only differ in few characteristics. In this work, we propose to make use of the shared characteristics by transferring a stylized defect-specific content from one type of background product to another. Moreover, the stochastic variations of the shared characteristics are captured, which also allows generating novel defects from random noise. These synthetic defective samples enlarge the dataset and increase the diversity of defects on the target product. Experiments demonstrate that our model is able to disentangle the defect-specific content from the background of an image without pixel-level labels. We present convincing results on images from real industrial production lines. Furthermore, we show consistent gains of using our method to enlarge training sets in classification tasks.

Under review as a conference paper at ICLR 2022
DEFECT TRANSFER GAN: DIVERSE DEFECT SYNTHE- SIS FOR DATA AUGMENTATION
An Anonymous authors
Paper under double-blind review
ABSTRACT
Large amounts of data are a common requirement for many deep learning ap- proaches. However, data is not always equally available large scale for all classes. For example, on highly optimized production lines, defective samples are hardly acquired while non-defective samples come almost for free. The defects however often seem to resemble each other, e.g., scratches on different products may only differ in few characteristics. In this work, we propose to make use of the shared characteristies by transferring stylized defect-specific content from one type of background product to another. Moreover, the stochastic variations the shared characteristics are captured, which also allows generating novel defects from random noise. These synthetic defective samples enlarge the dataset and in- crease the diversity of defects on the target product. Experiments demonstrate that our model is able to disentangle the defect-specific content from the background of an image without pixcl-level labels. We present convincing results on images from real industrial production lines. Also, we show consistent gains of using our method to enlarge training sets in classification tasks.
INTRODUCTION
Automated Visual Inspection (AVI) is vital for quality control in modern production lines. Despite the fact that AVI has been studied for decades, it remains a challenging task with many open research questions await to be answered. One of the main challenges in data-driven AVI is the acquisition of suitable training data. This is for two reasons: First, collecting a vast amount of labelled data is usually labor-intensive and time-consuming. where and what to look for. However, the acquired label information is task-specific and cannot be reused or transferred to a new task most cases. Thus, the tedious labelling process must be repeated for each new product, even if its defect is similar to other products in people's eyes. Second, in real-world scenarios such as highly optimized production lines, a more severe problem emerges: data imbalance. Only very few defective parts produced by design. Moreover, the acquired anomaly images from a single product are lacking diversity and may not capture the full defect distribution. Training robust deep neural network model in such conditions is very challenging.
many cases, even experts are required to identify
Since collecting sufficient real-world defective samples is impractical, algorithms to synthesize re- quired images became focus in research. Image synthesis through Generative Adversarial Net- works (GANs) (Goodfellow et al. 2014) has shown promising performance in recent years. But it also requires large amounts of balanced data which are not available in most industrial use cases, in particular for irregular defect patterns and large variation. Therefore, GANs tend to overfit to the training examples when trained with little data (Karras et al., 2020aa).
In this work, we tackle these issues by exploiting cross cross-domain information: we first define two sets of domains foreground domains and background domains. The foreground domain describes a set of images that contains specific foreground content to be grouped into a distinctive category, and each content has a different style. The background domain instead considered as group of images that shares similar structural appearance over the whole image. For example, we set foreground domains as defect types and background domains as product types while the styles of defects indicate their artistic looks such light heavy strokes. Building upon StarGAN v2 (Choil cet al.) 2020). the concept underlying this work is to transfer and generate foreground contents with variety of styles across different background domains, illustrated in Figure 1

---

Under review as a conference paper at ICLR 2022
Syuthetic Defective Sumples
Refermne Geaeeeeeeeed
Prottet
Applt
Todtec
Deteer
Scaatches
Apply
Output
Spots
Latent aatet-Conder
Ranbon Naise
Apply
Irstact C
Chipat
Figure 1: The underlying concept of DT-GAN to transfer and generate foreground contents with variety of artistic styles (e.g., light / heavy strokes) across different background domains.
The contributions of this work are three-fold: First, we introduce Defect Transfer GAN (DT-GAN), a model that learns transferring existing foreground content and generating novel contents onto dif- ferent backgrounds at the same time. In the real-world scenario, it allows defect inspection networks to learn from a variety of synthetic defective images by composing the foreground defects together with various non-defective images from different products. Second, DT-GAN is able disentangle the foreground defect-specific content and the defect-irrelevan background in a weakly-supprrvied manner. Third, extensive experiments show that our method can generate diverse and real-looking defective samples even for products with only 20 real defective images. These defective images generated by DT-GAN boost the performance defect inspection networks significantly.
RELATED WORK
GANs have shown their power in many computer vision tasks such as image synthesis (Lučić et al,, 2019), style translation (Johnson al. 20166), super-resolution (Ledig et al.) (2177), image impaint- ing (Pathak et al. (2016) and many other applications. To quantify the performance of GANs, visual quality and the diversity of generated images are considered as two of the most important criteria. Recent models address these requirements either by dedicated loss functions (Mao et al.) 2019b
Yang et al., 2019) or architectural design (Brock et al.) 2019). StyleGAN (Karras al.] 2020b), the latest state-of-the-art model in image synthesis, introduces stochastic variation in image gener- ating process by adding per-pixel noise after each convolution. However, non-trivial to adapt the model to transform given input images due to the design of the generator.
In contrast, image-to-image translation methods (Isola et al.] 2017) provide a way to recover the connection between inputs and the generated images while encouraging diversity. For example, et al. 7b) Huang et al. (2018) impose consistent mappings latent space to achieve the goal. Some approaches (Ma et al. [2019] Park al. (019) use reference images guidance generate diverse outputs. Mokady et al. (2020) further extends the translation task from styles to contents. learns identify specific content in given input (e.g., specific pair of glasses) and transfer it to the target image. However, aforementioned methods only consider the translation between two domains and their extension multiple domains is non-trivial.
Surface defect detection is one of the important tasks in real-world industrial manufacturing. It aims at identifying and classifying defects with the help of machine vision. Traditional methods (Ngan et al. 2011) build models upon hand-crafted feature extractors, which are unstable and out- performed by deep learning based models. However, the performance and generalization ability of deep learning approaches are restricted due to limited number of defective samples in real-world scenarios. Data augmentation aims to enrich the training dataset by introducing different kinds of invariance for the model to capture. Several recent works (Niu et al.) 2020: ZZang et al. 2021) have proposed to adopt GANs as a data augmentation method to generate realistic defective sam- ples. Among them, Defect-GAN (Zhang et al. 2021) tries to capture the stochastic variation within defects by mimicking the defacement and restoration processes. However, still learns a deter- ministic mapping between inputs and outputs while DT-GAN achieves multi-mo modality by varying styles. Moreover, our method can generate realistic defects with sophisticated patterns copied from real-world defective samples.
2

---

Under review as a conference paper at ICLR 2022
Image
Shared Layers
Latent Code z
SJaVee pnaaes
Reference image
Image
Shared
(l
Notie
BG
Su(a)
Ca(
Seta
Ge00
Classffer
Classifor
Output image c(x,3.
x1
43
X1
12
6Z
TG-Demain
Specific
Contents
Contents
(a) Generator
(b)
Mapping Network (c) Style- Content Encoder
(d) Discriminator
Figure 2: Overview of all modules in DT-GAN.
3
METHODOLOGY
Our primary aim is to perform unpaired image-to-image translation across multiple foreground do- mains within single model. In our use case, the foreground domains refer to the defect types, which means we want to achieve translations between different types of defects while the back- ground remains unaffected. We assume that there is always adequate amount of normal samples (c.g., non-defective) available, while anomaly samples are rare and hard to acquire.
3.1 PROPOSED FRAMEWORK
Our framework builds StarGAN v2, multimodal image-to-image translation model. Given input image x E X' and an arbitrary domain y E V/ StarGAN v2 generates a domain specific style code in learned style space and outputs image that is stylized to fit the domain of y. Its network architecture consists of four modules: generator, mapping network, a style encoder and a discriminator. We modify and extend all four modules (see Figure 2} and describe the key differences in details as below.
Style-Content Separation. Given a latent code z and a domain y. the mapping network M (ig urre (()) generates style code s M, and domain specific content = (z) in different branches. worth mentioning that M, here denotes an output of M corresponding to the domain This feature allows our method to separately model the structural appearance (i.e. content) and its artistic looks (i.e. style), which is essential because applying different styles to the same content enriches the diversity of outputs. By randomly sampling z from a standard normal distribution and from all available foreground domains, M is able to produce diverse style codes and domain specific contents.
The encoder E Figure 2(c)) extracts the style code s = Ey and the domain specific content = from given image x, which reflect the characteristics of reference images instead of randomly sampled noise.
Foreground/Backgrouud (FG/BG) Disentanglement. The generator G Fiiuue a()) translates an input image x into an output image G(x, S, c) according to given domain specific style code s and content č, which are provided either by the mapping network M when generating from random noise or by the style-content encoder when transferring an existing content from a reference image. To achieve a FG/BG disentanglement, we split the channels of the three-dimensional feature map (i.e.. H x W X C) at the bottle neck of G into two parts. The model is then forced to encode the background into the first channels and the domain specific content é into the latter channels by classification losses discussed in Section 3.2. è can then be replaced with content č from the target domain. The adaptive instance normalization (AdaIN) (Huang & Belongie (017) is then used to inject ğ into č during the decoding process while the background BGo is decoded separately. StarGAN v2 learns FG and BG together which leads to a conditional relationship between both. Our disentanglement and separate encoding break this conditioning and therefore enable our method to freely combine FG and BG as well as learn the full variation FG content. Finally, (G and concatenated together and then fused before output.
3

---

Under review as a conference paper at ICLR 2022
Multi-task discriminator with auxiliary classifiers. The discriminator D FFgure 2(d)) is multi- task discriminator with two auxiliary classifiers: a foreground domain classifier and background domain classifier. This feature strengthens the disentanglement of FG and BG by first ensuring the input image x contains a domain specific content that can be recognized by the foreground domain classifier independent of the background. Later, each branch Dy in the multi-task discriminator D is trained to determine if an image X is real image of its foreground domain or a fake image G(x, S, c) generated by G. Apart from that, one extra branch BGdss attached decide whether the background information of the input images is well preserved.
Content Transfer, Mokady et al. (220)) introduced a concept that model should be able to identity the difference between two domains when one of the domains contains feature that the other does not have. We refer to this concept as 'anchor' and extend to multiple domains (>22 by the FG/BG disentanglement, the multi-task discriminator and the foreground content classifier in D. We treat domain Normal as the anchor domain i.e. set the domain specific content to zero, because a normal image has no domain specific content in our definition. As result, we can now transfer contents between combination of FG and BG domains (see Figure 10).
Compared to StarGAN v2, our method not only models style codes and contents separately but also disentangles the foreground and background of an image in a weakly-supervised manner. These features allow explicit control over output images by combining desired style codes and contents from one of the subnetworks with the input images. Therefore, it leads to higher variance regarding the location, structural pattern and artistic style of defects in the synthetic images of DT-GAN.
3.2 TRAINING OBJECTIVES
Given an image x E X, its original foreground domain y E ) and its background domain p E P. the following objectives are used to train our framework.
Adversarial loss. In the training phase, a noise vector E 2 and target foreground domain ÿ E y are sampled randomly. Both of them are fed M, producing target style code s and target content C as follows: 5, ē = Mj(z). Goal of the training is ensure that 8 and č are sampled from the distribution over styles and contents of the target domain y. The generator G then combines an image x with š and è and learns to generate output image G(x, S, é) that is indistinguishable from real images in the target domain y. We encourage this behavior by using an adversarial loss same in Choi et al. (2020)
Ladv Ex.y log Dy (x)] + Exx.j. (1 (G
where Dy and Dy are the output branches of D that correspond to the source domain y and the target domain y. respectively.
Style-content reconstruction loss. Similar to StarGAN v2, to enforce the generator takes the style code š and the domain specific content è into consideration during the generation process, employ a style-content reconstruction loss
Laty con F.x. T
S (x(G(x, e)] + Ex (x(G((x,,,))1
(2) C). Here, the style-
This objective urges the style-content encoder E to recover and č from G(x,3, content encoder E learns a mapping from an image to its style and content domains, which allows G to synthesize image with given and from reference images test time.
Diversity loss. In order to further boost the diversity of output images from G, we introduce a loss that encourages diversity as follows: for pair of random latent codes z and Z2 we compute S. c Mg(z;) for i E {1, 2} and enforce a different outcome of the generator G for differently mixed style and content input pairs:
Css Ex. 21 [[| S1, 22) G(x, S2, [1]
[| 51 ū, G(x, [1] + F Fx.
+ Em.no Bx. 22 [|| Sm, ē,,) G(x, [1]]
where m, n E (1,2 # n} and o E (1,2). Driven by this term, the generator G is forced to discover meaningful style features and contents that eventually lead to diversity in generated images.
4

---

Under review as a conference paper at ICLR 2022
We ignore the denominator ||z zell1 of the original diversity loss (Mao al. 2019a for stable
training as in StarGAN v2.
Cycle consistency loss. To ensure that the generated image G(x, 5, preserves the domain- invariant properties its input image x, we impose the cycle consistency loss (Zhu et al. 2017a)
Laye Ex...i x|| (, e)11))
where 8, (x) is the extracted style code and domain specific content of the input image x, and y is the original domain x. By learning to reconstruct the input image x with given style code 8 and content è, the generator G is then further encouraged to disentangle the background, the domain specific content and the style code.
Content consistency loss. Besides the cycle consistency loss, we apply another constraint to en- force that the detached domain specific content from is consistent with the one retrieved from E according to
Loon.eye = x... [l - elli] + Ex [l 3, - -
(5)
where è = (x), - (x). (x)( and FGa(G(x,56 8, 2)) are the pop-out domain specific content from input image x and generated image G(x, 8, C), respectively.
Classification losses. We employ two classification losses: the first one is the foreground content classification loss
CPG. = Ex...il log DFG (v/aa] + Ex [I log DFG cls (B) Xfake)
(6)
which aims to ensure that the domain specific content is properly encoded and carries enough infor- mation from the target domain. The second one is the background classification loss
Luc.ch+ = Exm..pl log (BCLas( Xreai) Expr.pl log DBGLets (p)xxaee)
(7)
where p is the corresponding background type of Xteal and Xfake With the help of this objective, the generator G learns to preserve the domain-invariant characteristics of its input image x while dissociating the foreground domain specific part.
Full objective. Our full objective functions can be summarized as
min max Ladv Assy con Laty.oon Ads Lds Aeye Leye+
G.F.E
(8)
Acom.eyc Lemaaye + Ara.cs Lra.es + ABaies LBG.dd
where Aay, Ads Aeye Avom.eye Apaeess and ABG.ets are the hyperparameters for each term.
4 EXPERIMENTS
We evaluated the images generated by DT-GAN through a series of experiments both quantitatively and qualitatively. Finally, demonstrate the benefits of our generated images when being used as data augmentation for a defect classification task on limited data.
performed real industrial dataset: Surface Defect Inspection (SDI) dataset that contains three different kinds of products from production lines and samples from each product are classified into three mutually exclusive classes: Normal, Scrat and Spot. All of the images are grayscale. Detailed statistics of the dataset are summarized in Appendix A Note that only the training set was used GAN training, the test set was left untouched for final evaluation in classifier training. For fair comparison, all images were resized to 128 128 resolution for both GAN training and classifier training, which was also the highest resolution used in the baselines for image generation. For comparison, we also conducted experiments on the widely used MVTee Anomaly Detection dataset (Bergmann et al. 2019) in Appendix E.4
Dataset. All experiments
4.1 DEFECT GENERATION
Baselines. As discussed in Section DT-GAN either the mapping network to randomly generate styles and defects, or it can use the style-content encoder to extract both from reference images. We refer these cases 'latent-guided" and "reference-guided"" respectively.
5

---

Under review as a conference paper at ICLR 2022
Since the two ways of guidance are fundamentally different, evaluated them against two sets of baselines: Our reference-guided image generation was compared to Mokady et al. (2020) and StarGAN v2, because both of them perform reference-guidde translation. Note that Mokady et al. 2020) can only translate between two domains while StarGAN v2 and DT-GAN can achieve multi-domain translation within single model. Images generated through the latent-guided part of DT-GAN were compared to state-of-the-art GANs in image synthesis: BigGAN (Brock et al.) 2019) and StyleGAN v2 (Karras et al. 2020b). We BigGAN to condition defect types during training while StyleGAN v2 was trained unconditionally, All baselines were trained from scratch with the public implementations provided by the authors
4.1.1 QUANTITATIVE EVALUATION
Metries. We employed the commonly used frechet inception distance (FID) (Heusel et al., (017)) to evaluate both the visual quality and the diversity of the generated images. We also report the kernel inception distance (KID) (Binkowski et al. 2018) which is a more stable metric for small sets of images like our SDI dataset. Lower FID and KID scores indicate better performance.
Both scores are shown in Table I We observe that methods like BigGAN and StyleGAN v2, which perform defect synthesis purely based on latent codes, generally provide unsatisfactory results on the SDI dataset, presumably due to the small number of defective samples that were available. These methods then struggle to capture the complex and irregular patterns of defects. We also experimented with augmentation methods for GAN training (Karras et al. [022aa] Zhao et al. 2020) but did not find a consistent improvement (see Appendix E.2 We thus only report the best scores.
Reference-guides synthesis methods like Mokady et al. (2020) and StarGAN v2 seem to generate more realistic images. The scores of StarGAN v2 on a single product are omitted here because generating images with specified background is not possible due to its network design- -the product type changes in output images, which we refer to as 'identity-shift'. As seen in Table 1, our method achieves better scores in all cases. We believe this is due to the fact that our method allows free combination of foreground defects and backgrounds, making the generated images more diverse even with small number of training samples.
Table I: Quantitative comparison of DT-GAN with baseline image synthesis methods using FID and KID. Note that the reported values are not comparable between columns, because they were calculated on different training sets.
FIDJ
KIDJ
Method
A
B
C
All
A
B
C
All
Mokady (2020)
68.69
66.90
36.21
58.63
0.050
0.036
0.030
0.036
StarGAN v2
37.70
0.013
StyleGAN v2
90.10
52.95
138.09
35.34
0,072
0,027
0.186
0,013
BigGAN DiffAug
218.74
134.41
270.89
155.88
0.220
0.121
0.378
0.099
Ours
58.43
36.44
22.68
29.73
0.025
0.013
0.012
0.009
4.1.2
QUALITATIVE EVALUATION
We present a qualitative comparison with the baseline methods in latent-guided image synthesis Figure 3, To make a fair comparison, we trained StyleGAN v2 and BigGAN on each product separately have control on background products. Note however, that images from DT-GAN were always obtained from a single model. We can see that some generated samples from StyleGAN v2 do not contain clear defects, and samples from BigGAN present abnormal grid patterns. Both methods do not take images as inputs but generate synthetic images according to a given latent code which contains information for both FG and BG. This conditioning leads limited diversity in the output images. On the other hand, StarGAN v2 performs translation based on input images but suffers from the same entanglement issue. Thus, it fails to preserve the background, which results in artifacts or identity-shift in its outputs. Our network architecture that disentangles foreground and background seems to mitigate these issues. See Appendix E.4 for more images.
We could not obtain the code of Defect-GAN reproduce their results.

---

Under review as a conference paper at ICLR 2022
StleGAN B
Source
SarGAN v2
Ours
SylGAN v2
BigGAN
Source
SharIAN c2
Ous
(a) Norma -tt-Strrtcchee
(b) Norma 1-to-Soots
Figure 3: Qualitative comparison of latent-guided image synthesis results. In each subfigure: on the left, defective images are fully generated from random noise. On the right, random defects synthesized onto given normal samples. Note that BigGAN* denotes it was trained with DiffAug.
Style/Cunical Mokady al. SurGAN v2
Ours
Souree
Style-Content
Mokady et all StarGA V2
Ours
(a) Norma 1 1-to-Scratchhs
(b) Normal-to-Spol
Figure 4: Qualitative comparison of reference-guider image synthesis results the SDI dataset. Each method transforms the given source images into target foreground domains (e.g.. Scratches) with the styles and contents extracted from the reference images.
Also for reference-guided image synthesis, where used different background and foreground reference images as illustrated in Figure 4 only our method produces high quality images with preserved background from the source and transferred foreground defect from the reference.
Ablation study. We visually demonstrate the effect of each component we added to DT-GAN compared to StarGAN v2 in Figure 5) using the examples of both latent- and sesrennce-uiied image synthesis from Normal to Scratches. The quantitative evaluation can be found in Appendix E.3
Column (a) corresponds to StarGAN v2 and highlights the drawback of entangled FG/BG again (i.e. the identity-shift in the background). We first tackle this problem by modeling the style code and foreground content explicitly and feeding them separately the generator. This leads to better preservation of the background structure in column (b) for the reference-guide guided subnetwork. but not for the latent-guided synthesis on the bottom Figure 5) Thus, add foreground classifier in the discriminator in (c) to ensure the output image contains the desired foreground content (scratch). Similarly, we introduce a background classifier to the discriminator in column (d). Note that the additional product type labels can be acquired automatically from production lines.
For column (e), we add the separate decoders for foreground and background in the generator which are fused only in the end. This enhances the preservation of background characteristics like lighting even more. Imposing an additional penalty foreground content extracted from a normal sam- ple as described in Section 3.1 leads another visual improvement of the foreground edges for diference-guided synthesis in column (f). Finally, inspired by StyleGAN, we incorporate adaptive noise injection to the mapping network, which significantly boosts the performance of our latent- guided image synthesis as shown in column (g).
Styling. We visually demonstrate the effect of style codes in method by randomly sampling those and combining them with fixed reference background and foreground images in Figure 6 where variety of artistic styles can be seen the output columns.

---

Under review as a conference paper at ICLR 2022
Source
Style/ Content
(4)
(b)
(0)
(4)
(6)
Reference Guided
Tatent- Guided
Figure 5: Ablation study. (a) The baseline StarGAN v2. (b) + Style-Content branches. (c) Fore- ground classifier. (d) Background classifier. (e) Separately decoding foreground and background in G. (f) + Anchor foreground domain (e.g. No Normal )). (g) Noise injection in Mapping Network.
Sonree
Content
Kandoma Sampled Styles
Source
Content
Randecs Sampled Styles
(a) Norma to-Scrrttches
(b) Normal-to-Spe
Figure 6: Visual effect of randomly sampled style codes fixed pairs of reference background
(Source) and foreground (Content) images.
4.2 DT-GAN FOR DATA AUGMENTATION
We also evaluated our method data augmentation method for defect classification the SDI dataset. defined one task 'general', where the classifier was trained on images from all products once, while task 'single product' only used the subset of images for product.
Besides, we incrementally varied the amount of real Normal data available for classifier training: 4500, 6600, 12000 and 18600. In the case of defective images, all of them were always used due to the small amount unless otherwise specified. As backbone we used ResNet-50 (He et al. (016a) with ImageNet pretrained weights. For experiments with synthetic data, attached auxiliary domain classifier to the network through a Gradient Reversal Layer (Ganin & Lempitsky 2015).
Table 2: Quantitative comparison of the baseline methods defect classification task at the scale of 12000 images/class. The reported values are the achieved error rates (%) over five runs.
Method
ResNet-50
EfficientNet-b4
No-Aug
21.64+1.2 .24
12.0640.66 6.64
Trad-Aug
12.5840.81
9.33+0.73
Mokady (2020)
11.11+1.19
13.26+1.13
StarGAN v2
13.07+1.30
12.25+0.79
StyleGAN v2
11.55+1.79
11.68+0.76
BigGAN+DiffAug
11.45+0.61
12.06+0.50
Ours
9.940.69
9.14+1.02
8

---

Under review as a conference paper at ICLR 2022
Since the SDI dataset is highly imbalanced, we oversampled the minority classes (Ling al. 1998) unless the data was balanced through synthetic images. Additionally, we always applied traditional data augmentation techniques like random horizontal flips. jittering and lighting (Shorten & Khosh- goftaar 2019) except where noted. All following results were evaluated by the achieved error rates over five runs with different random seeds.
Effectiveness of synthetic data. We first compare classifier performance for no augmentation (No- Aug), traditional data augmentation (Trad-Aug), and combination of traditional augmentation with synthetic images for GAN methods including DT-GAN. We also introduce a stronger back- bone, EfficientNet-bbe (Tan & Lej 2019), to demonstrate that our results are not confined a specific network. Table 2] shows that our method is the only one that improves performance for both back- bones, presumably due to the combination of high visual image quality and diversity in our samples.
Table Experimental results on using different amount of synthetic images generated by DT-GAN to train classifiers. The left-most column stands for number of samples per class to be classified. The training set of the baselines is balanced by oversampling while ours is by synthetic images.
Dataset Size
20A
All
Trad-Aug
Ours
Trad-Aug
Ours
4500
15.55+0.63
14.28+1.25
12.75=0.61
11.04#0.76
6600
16.69:0.76
14.41+3.12
13.07=1.57
10.60:0.48
12000
16.95+1.02
14.22+1.53
12.05=0.81
9.90=0.69
18600
12.
15.36+0.86
12.37=0.32
10.21:0.96
Impact of dataset size. Motivated by the limited availability of data in real-world production sce- narios, we therefore evaluated DT-GAN for data augmentation on subset of the full SDI dataset (All), which only contains 20 defective samples in product A for each defect type (20A). In this case, DT-GAN was also trained on the reduced subset. As shown in Table 3) there is clear improvement when synthetic images from DT-GAN are used as data augmentation, even for the extremely limited data subset. Further results on single product classifiers can be found in Appendix E.
Table 4: Cross-domain effect on single product classifiers trained with reference-guided synthetic images the scale of 12000 images/class.
Trad-Aug
VA
vB
vC
vABC
13.81+2.36
11.8142.65
12.72+2.87
11.99*1.63
11.0923.49
6.80+1.64
6.40+1.34
6.60+1.52
6.59+1.34
5.60#1.34
C
16.57=3.20
13.1442.81
11.23+0.80
14.85+1.73
11.4240.96
Cross-domain effect. We hypothesized that limited data can be counteracted by transferring de- fects across multiple background products, if there are at least some defects that occur on multiple products (See Appendix I for further discussion). We tested this approach by comparing the per- formance of classifiers trained synthetic images with defects from specific source (vA, vB, vC) to classifiers trained on images with defects from all products (vABC). As we can see in Table 4 the best performances reached by the models that take over defects from other products. We interpret this as support for our hypothesis and its practical usefulness.
5 CONCLUSION
We propose a novel method, DT-GAN, which allows diverse defect synthesis both by generating from randomly sampled noise and by following the guidance of given reference images. Due to explicit style-content separation and FG/BG disentanglement, DT-GAN achieves higher image fidelity, better variance in defects and full control over background and foreground while being sample-efficient. We demonstrated the feasibility and benefits of DT-GAN on a real industrial defect classification task and the results show our method provides consistent gains even with limited data and boosts the performance of classifiers compared to state-of- tff-the-art image synthesis methods. For future investigation, we aim to represent defects more explicitly (c.g., localization) to improve the explainability of the model and also enhance the model transferability unseen products.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We aim for full reproducibility by publishing the source code and dataset with the final version of the paper. Besides, we provide descriptions of the training details in Appendix B, the evaluation setup in Appendix and the network architecture Appendix D
REFERENCES
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad com- prehensive real-world dataset for unsupervised anomaly detection. In 2019 IEEE/CVE Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 9584-9592, 2019. doi:
10.1109/CVPR.2209.00999
Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and A. Gretton. Demystifying MMD GANs. ArXiv, 01s/1801.0140) 2018.
Andrew Brock, Jeff Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, abs/1 1t//1800.110 11096, 2019.
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster r- cnn for object detection in the wild. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3339-3348, 2018
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Ma- chine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 1180-1189, Lille, France, 07-09 Jul 2015. PMLR. URL htthttt::// pictps://proceedinng htt.://iiinn.i..oorrrss//////. mimin155htmll
mlr
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In NeurIPS, 2014.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogni- tion. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016a.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks, 2016b. URL /nttp://arxiv. wtt.:o///////....00.02033 cite arrivil603.05027Commmos ECCV 2016 camera-ready.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NIPS, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Xun Huang, Ming-Yu Liu, Serge J. Belongie, and J. Kautz. Multimodal unsupervised image-to- image translation. ArXiv, abs/1804.0472. 2018.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 5967-5976, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution, 2016.
10

---

Under review as a conference paper at ICLR 2022
Tero Karras, Miik Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020b.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report University of Toronto, Toronto, Ontario, 2009.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro
Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, el. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, 2017.
Charles Ling, Charles X. Ling, and Chenghui Li. Data mining for direct marketing: Problems and solutions. In In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining (KDD-98, pp. 73-79. AAAI Press, 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Mario Lučić, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. Kamalika Chaudhuri and Ruslan Salakhut- dinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4183-4192 PMLR, 09-15 Jun 2019. URL m.l.ppres. v v97/lucic1 9a html
https: http:::://ppocceeddie
Liqian Ma, Xu Jia, Stamatios Georgoulis, Tinne Tuytelaars, and Luc Van Gool. Exemplar guided unsupervised image-to-image translation with semantic consistency. In ICLR, 2019.
Qi Mao, Hsin- Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming-Hsuan Hsuan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1429-1437 2019a.
Qi Mao, Hsin- Ying Lee, Hung- Yu Tseng, Siwei Ma, and Hiugn Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In CVPR, 2019b.
Ron Mokady, Sagie Benaim, Lior Wolf, and Amit Bermano. Masked based unsupervised content transfer. In International Conference on Learning Representations, 2020. URL https://
@pentevvee..ett/ooruue http://frummidddB..o -d=--ed--BB----- 91BtvH
Henry Y. T. Ngan, Grantham K. H. Pang, and Nelson H. C. Yung. Review article: Automated fabric defect detection-a review. Image Vision Comput., 29(7):442-458 June 2011. 1. ISSN 0262- 8856. doi: (0.1016/jiimviiss 1.0.00000000....... Utthhttps: //oi: wtt.://...g0000.11111...... 1016/1 imavis.
2011 02 002)
Shuanlong Niu, Bin Li, Xinggang Wang, and Hui Lin. Defect image sample generation with gan for improving defect recognition. IEEE Transactions Automation Science and Engineering, 17(3)_611-1622. 2020. doi: 10.10997ASE.220022977444
Taesung Park, Ming- Yu Liu, Ting- Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. CVPR, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zach DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue. Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536-2544, 2016.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv: (BMo:060904444 2016.
11

---

Under review as a conference paper at ICLR 2022
Connor Shorten and T. Khoshgoftaar. A survey image data augmentation for deep learning.
Journal of Big Data, 6:1-48, 2019.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- lernational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. #Iess/v97/tanI9a.15
PMLR, 09-15 Jun 2019. URL https::/pprcceednggs..
Dingdong Yang. Seunghoon Hong, Yunseok Jang, Tiangchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. In ICLR, 2019.
Gongjie Zhang, Kaiwen Cui, Tzu-Yi Yi Hung, and Shijian Lu. Defect-gan: High-fidelity defect syn- thesis for automated defect inspection. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 2524-2534, January 2021.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training, 2020.
Jun- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. In NIPS, 2017b.
12

---

Under review as a conference paper at ICLR 2022
A THE SURFACE DEFECT INSPECTION DATASET
The Surface Defect Inspection (SDI) dataset consists of 20,414 images at 128 x 128 resolution.
contains three background domains A, product B and product C, each can be further classified into three foreground domains-Normee example images of the SDI dataset. To be noticed that the dataset is highly imbalanced between normal and defective samples but also between different products as shown in (Table 5) This more challenging task when training deep neural networks like GANs and downstream classifiers.
Scrat ches and Spots. Figure shows
only
For each foreground and background domains, we randomly select 50 images for joint valida- tion/test set, which is then further split into separate sets in the ratio of 3:7, and use all remaining images as training sets for GAN and classifier training. We present the distribution of the training set when training DT-GAN in Table 6 Note that the normal samples used in GAN training are only a subset of all available samples in No Normal and we keep the rest of them for generating defective samples at test time. For classifier training, show the statistics in Table 7, where the number of normal samples involved classifier training increase incrementally. The validation set used select the best model during classifier training while the test is left untouched until the final evaluation. Both of the validation and test set are inaccessible by DT-GAN.
Table 6: The training set for DT-GAN and the baseline image synthesis methods.
Table 5: Distribution of the full SDI dataset, Overview
Overview
A
B
B
C
Normal 6250
6250
6250
Scratches
340
167
121
Normal
700
700
700
Spots 108
670
258
Scratches
290
117
71
Spots
58
620
208
Table 7: The training, validation and test set for classifier training, where N increases incrementally-1500 1500, 2200, 4000 and 6200.
Train
Validation
Test
C
B
Normal N
N
N
12
18
15
38
32
35
Scratches
290
117
71
14
16
15
36
34
35
Spots
58
620
208
14
16
15
36
34
35
B TRAINING DETAILS
DT-GAN. We follow the training scheme as described in StarGAN v2 with minor modifications. To fit the model single Nvidia GTX TITAN X, the batch size reduced four while the model is still trained for 100,000 iterations. The training time is about three and a half days on the dedicated GPU with the modified network architectur and loss functions mentioned in Section 3 in PyTorch
Paszke et al. 2017). We set AAy = I, As = II, Xaye = Auamaye = 1, Act. and ABG for the SDI dataset. other design choices remain the same in StarGAN v2.
Classifiers. We train all the classifiers that use ResNet-50 as backbone for 100 epochs with the SGD optimizer (Ruder) 2016) and batch size 256. The initial learning rate 0.001, momentum is 0.9 and weight decay is 1e-4. A learning rate scheduler is set to reduce the learning rate by factor of 0.1 when the validation loss stops decreasing for epochs. The same setting also applies to EfficientNet-b4, except the batch size is reduced to 128. Although DT-GAN can synthesize realistic defective samples, we notice that there still exists a domain gap between the generated samples and the real samples. To explore the full potential of the generated samples, attach auxiliary
We based implementation
code from StarGAN v2: hatp::/githua.commllovaassssnnn
13

---

Under review as a conference paper at ICLR 2022
(a) Normal
(b) Scratches
(c) Spots
Figure 7: Overview of the SDI dataset.
source classifier to distinguish between synthetic and real samples. Then, this classifier connected to the backbone (e.g. ResNet-50) through Gradient Reversal Layer. With the help of the Gradient Reversal Layer, the backbone is forced to extract the shared features between synthetic and real samples, which ensures all training samples effectively learned.
We design a two-layer perceptron that connects to the average pooling layer in ResNet-50 as shown Figuee 8 Note that the usual fully connected layer after the average pooling in 5et remains the same and is not affected by the extra branch we added. Inspired by Chen et al. (2018). a three- layer perceptron is used for EfficientNet-bb b4 instead shown in Figure 9 Its layers initialized with a random normal distribution, where the standard deviation is set to 0.01 for the first two layers and 0.05 for the output layer. The biases for all layers are set O.
14

---

Under review as a conference paper at ICLR 2022
PC unit Logistic
FC 5 unii Logistic
ResNet-50
EllicianNNa-b4
1024 uiit Rel II
uit Lagieic
CRL
1924 noir Swics
1024 unit Swrish
2 unit Logistic
Figure 8: ResNet-50 with GRL.
Figure 9: EfficientNet--4 with GRL.
C EVALUATION SETUP
Generated samples from DT-GAN. DT-GAN requires images as input for generating synthetic data. At test time, we translated each Normal image in the SDI dataset into four defective images: two with Scratches. and two with Spots. s. The translations were performed by two subnetworks: by the mapping network . using random noise (latent-guided and by the style-content encoder E using a reference image ("rfference-guided) each defective foreground domain. Similarly, we also randomly sampled one reference image from the training set for each defective foreground domain. The corresponding style codes and defect contents were then produced by the two subnetworks respectively and fed to the generator for target image generation.
We first randomly sampled one latent code for
We conducted classification experiments separately on images generated from the two subnetworks and mixture set of both (i.e. 50% from each subnetwork). Experiments show consistent gains of using synthetic images generated from DT-GAN (Table 8). We observe that the latent-guided synthetic images in general perform better than the reference-guided one, while the mixture set provides more stable results with regard to the standard deviation. Presumably the mixture set benefits from the combination of samples from reference-guides synthesis, which are well aligned with the original defect distribution, and the samples from latent- latent-guided synthesis, i.c. from random noise, which adds novel but plausible defects the dataset. In the main text, we report the results of the mixture set for all experiments, including the quantitative evaluation of DT-GAN.
Table 8: Classification results with regard to the synthetic images generated from the two subnet- works and the mixture set.
Dataset Size
All
Trad-Aug
Latent
Reference
Mix
4500
12.75+0.61
10.72+0.96
11.48+0.88
11. 1..40.76
6600
13.07+1.57
10.34+1.86
11.55#1.64
10. 10.60#0.48
12000
12.05+0.81
9,90 1.26
10.40=0.99 99
9.90=0.69
18600
12.37+0.32
11.04:1.26
12.12+0.75
10.21=0.96
Frechét inception distance (FID) and Kernel inception distance (KID). We used the feature vectors from the last average pooling layer of the ImageNet pretrained Inception- V3 to calculate both scores. For each test image from the Normal domain, translated it into synthetic defective image each defect domain. The style codes and contents for the translation were acquired two ways: by randomly sampling from the standard normal distribution and by randomly sampling reference image from the train set of defect domain. To calculate the FID and KID score, we generated 4000 defective samples per product per defect domain for each way of guidance, and formed the mixture set by randomly sampling 2000 images per product per defect domain from each way. The reported FID and KID scores were then computed between the defective images in the training set and the mixture set of synthetic defective images. The same procedure was applied when computing scores on single product subsets the SDI dataset. For example, for product A, we calculated the scores between the defective image of product A in the training set and the mixture set of synthetic defective images of product A.
D NETWORK ARCHITECTURE
In this section, we provide the architectural details of all four modules in DT-GAN.
15

---

Under review as a conference paper at ICLR 2022
Table 9: Generator architecture.
(a) Encoder
Layer
Resample
Norm
Output Shape
Image x
128 x 128 x 3
Conv
128 x 128
ResBlk
AvgPool
IN
64 x 64 x 256
ResBIk
AvgPool
IN
32 x 32 x 512
ResBIk
AvgPool
IN
16 16 x 512
ResBlk
IN
16 16 x 512
ResBlk
IN
16
512
(h) Background Decoder
(c) Foreground Decoder
Layer
Resample
Norm
Output Shape
Layer
Resample
Norm
Output Shape
Input
16
16
448
Input
16
16
64
ResBlk
IN
16 x 16 448
ResBlk
AdaIN
16 x 16
64
ResBlk
IN
16
512
ResBlk
AdaIN
16 16 256
ResBIk
16
ResBIk
AdaIN
16 x 256
ResBlk
Upsample
IN
32 x 32 512
ResBlk
Upsample
AdaIN
32 x 32 x 256
ResBlk Upsample
IN
64 x 64
256
ResBlk
Upsample
AdaIN
64 64 x 128
ResBlk
Upsample
IN
128 x 128 x 448
ResBlk
Upsample
AdaIN
128 x 128 x 64
(d) Fusion
Layer
Resample
Norm
Output Shape
Input
128 x 128 x (448 64)
Conv 1x1
128 x 128 x
Table 10: Mapping network architecture.
(a) Shared Layers
Layer
Activation
Output Shape
Latent z
16
Linear
ReI.U
512
Linear
ReL.U
512
Linear
ReLU
512
Linear
ReLU
512
(b) Style Code
(c) Content
Layer
Activation
Output Shape
Layer
Resample
Activation
Noise
Output Shape
Input
512
Input
512
Linear
ReLU
512
Reshape
1
Linear
ReLU
512
ResBlk
Upsample
IN
True
2 2 x 512
Linear
ReLU
512
ResBlk
Upsample
IN
True
4 x 512
Linear
64
ResBlk
Upsample
IN
True
x 8 x 256
ResBlk
Upsample
True
16
x 128
Conv
True
x 64
Generator (Tblle 9). For the SDI dataset, the encoder part the generator consists of three down- sampling blocks and two intermediate blocks (Table 9 (a)), all of them are pre-activation residual units (He et al. 2016b). Then the encoded feature map split channel-wise into background (Ta- ble 9 (b)) and foreground (Table (c)). Both of them are then carried through separate decoders. We use the instance normalization (IN) and the adaptive instance normalization (AdaIN) as indicated. The style code is injected into all AdaIN layers to modulate the affine transformations. Note that
16

---

Under review as a conference paper at ICLR 2022
Table 11: Style-content encoder and discriminator architectures.
(a) Shared Layers
Layer
Resample
Norm
Ouiput Shape
Input
128 x 128 x 3
Conv 1xl
128 x 128 x 64
ResBIk
AvgPool
64 x 64 256
ResBIk
AvgPool
32 32 512
ResBlk
AvgPool
16 x 16 512
(b) Style Code Discriminator and BG Classifier
(c) Content FG Classifier
Layer
Resample
Norm Output Shape
Layer
Resample
Norm Output Shape
Input
x 16 512
Input
16 16 x 512
ResBlk
AvgPool
8 x 512
LReLU
16 16 512
ResBlk
AvgPool
x x 512
Conv 1x1 *K
x 16 64 *K
LRcLU
x 4 x 512
Conv 4x4
x 1 x 512
LReLU
1x 5
Reshape
512
Linear *K
D
AdaIN is only used in the foreground decoder. The outputs of both decoders are only fused in the end (Table (d)).
Mapping Network (Table 10 The mapping network consists of four shared linear layers Tabll 10 (a)) and two separate branches: one for generating style codes (Table 10(b) and one for contents Table c)). Each of them is further divided into K output branches, where K denotes the number of domains. The dimension of the input, the output style code and the output content is set to 16, 64, and 16 x 16 x 64, respectively. The latent code is sampled from the standard normal distribution. Note that we apply per-pixel noise after each convolution in the content branch, which we have observed to increase the diversity of generated defects significantly (cf. Figure (g)).
Style-Content Encoder Tablle II.. The style-content encoder consists of a CNN Table 11 (a)) with two branches (aalle 11 (b) and (c)) in the mapping network. Each branch has K outputs, where K' is the number of domains. Three pre-activation residual blocks are shared among two branches, followed by specific structure for each branch. The output dimension in Table 11 is set to 64, which denotes the dimension of the style code.
Discriminator Table 11).
The discriminator is a multi-task discriminator with two auxiliary clas- sifiers for the foreground content and the background. The structure is almost identical to the style- content encoder, except is set parallel to final linear layer in Table (b) and provides the logits for background classification. The foreground classifier instead acts on top of the output in Table 11 (c) and four more pre-activation residual layers are applied to encode the content into logits for foreground content classification.
for real/fake classification. The background classifier acts in
E ADDITIONAL RESULTS
E.1 ADDITIONAL RESULTS ON THE SDI DATASET
We provide additional reference-guided image synthesis results on the SDI dataset in Figure 10, We demonstrate all the possible transfers among all foreground domains. Both style codes and contents are extracted from the reference images. To be noted that DT-GAN can append and remove foreground defects not only onto Normal 1 samples but also defective samples. For example, in the fifth column of Figure 10 the original seratch in the source image is removed and only the defects from the reference images are presented in the output images.
Besides, we present additional evaluations showing the effectiveness of our synthetic data according to Table 3. As in Table 12 the synthetic images from DDT-GAN also boost the performance in
17

---

Under review as a conference paper at ICLR 2022
Normal
Scratcites
Source
Reference
Spots
Serateles
Nomaa
Figure 10: reference-guidee image synthesis results on the SDI dataset. The first row and the first column are the real images sampled from the dataset, while the rest are synthetic images generated by the proposed DT-GAN. Our model provides translations between different foreground domains (Normal, Scratches and Spots) with styles and contents extracted from reference images while the backgrounds from source images well preserved.
single product classifiers, where the classifiers were trained on the subset of images for one product (A, B, C) instead of the full dataset (ABC).
we assumed that the data-insufficiency problem can be mitigated by transferring defects across multiple background products. To examine if this assumption holds, we compared the performance of classifiers trained on synthetic images with defects from a specific source (vA, vB, vC) to classifiers trained on images with defects from all products (vABC). The results on the cross-domain effect with regard to different sizes of the training set are shown in Table 13 We again notice that using our synthetic data is beneficial. Moreover, in most cases the performance is further improved by exploiting cross-domain information (i.e. by transferring defects from other products). We interpret this as support for our assumption and the practical usefulness of method in the real-world scenario. The case of cross-domain image synthesis when the desired combination is not presented in the training set covered in the study on the MVTec Anomaly Detection dataset (Bergmann et al. 2019) (see Appendix E.4).
As discussed in Section 4.2
18

---

Under review as a conference paper at ICLR 2022
Table 12: Quantitative results for DT-GAN as data augmentation method to train general and single product classifiers. The left-most column indicates the number of samples per class, including all images from the training set plus increasing amounts of synthetic images. In the first row, 20A refers to the case of 20 real defective samples for product A, while All refers to the full training set.
Dataset Size
B
ABC
Trad-Aug
Ours
Trad-Aug
Ours
Trad-Aug
Ours
Trad-Aug
Ours
4500
35.09+2.62 27.64=3.12
7.8+1.48
5.6=1.67
15.24+1 .90 13.14+1.70 15.55+0.63 14,28+1.22
6600
39.6442.28 27.64=1.65 8.8+1.64
6.2+1.64
15.81+1.73
12.38=1.65
16.69+0.76 76 14.41=3.12
12000
34.18+4.39
28.55=7.32
5.8+0.45
5.6+1.14
16.19+1.17
10.86+1.28
16.95+1.05
14.22=1.55
18600
39.4547.06 32.55+5.04 7.240.84
5.2+1.10
14.8640.85
13.14:2.06
16. 12+2.12 19 15.3640.86
All
Dataset Size
B
ABC
Trad-Aug
Ours
Trad-Aug
Ours
Trad-Aug
Ours
Trad-Aug
Ours
4500
16.00=1.04
10.18:1.75
8.79=0.45
5.60:1.51
17.1346.62 14.09:2.27
12.75+0.61 11.04:0.76
14.90+1.38 10.54=1.22
7.60=1.51
6.80=3.11
15.2342.33 11.42:0
13.07=1.57
10.60=0.48
12000
13.8142.36
6.72#1.65
6.80#1.64
4.60=0
16.5743.20
13.90:2.57
12.05+0.81
9.9020.69
18600
13.63#2.22
10.54#2.45
6.80&1.79
4.99
15.6240.85
11.61#1.24 12.3740.32 10.21=0.96
effect single product classifiers trained with reference-guien guided synthetic images at all scales. Note that here A, B and C stand for products in the SDI dataset while vA, vB, vC and vABC indicate the defects are copied from which reference set.
Table 13:
Dataset Size
A
Trad-Aug
vA
vB
vC
VABC
4500
16.00=1.04
12.90+2.61
13.08+1.65
14.90+2.46
15.27=3.49
6600
14.90+1.38
13.99+1.88
11.26*1.04
14.36a4.04
16.00=2.85
12000
13.81+2.36
11.81+2.65
12.72+2.87
11.99:1.63
11.09#3.49
18600
13.63#2.22
12.72+5.22
14.36#3.83
14.18
13.81
Dataset Size
B
Trad-Aug
vA
vB
VC
VABC
4500
8.79#0.45
7.80+2.15
5.60=1.14
10.19+0.84
6.79+1.30
6600
1.60+1.51
6.80=1.65
7.80=1.10
8.00+2.34
6.00=1.411
12000
6.80+1.64
6.40+1.34
6.60=1.52
6.59+1.34
5.60#1.34
18600
6.80+1.79
6.1911.78
4.40=1.14
6.60+1.95
5.99#1.58
Dataset Size
Trad-Aug
vB
VC
VABC
4500
17.1414.6
14.85+0.52
16.76+2.58
13.90+1.98
12.00+1.59
6600
15.23+2.33
13.14 1.24
13.90=2.29
14.28+1.34
12.57+1.57
12000
16.57+3.20
13.14+2.81
11.23+0.80
14.85
11.42+0.96
18600
15.62+0.85
13.71#1.73
15.99=6.75
12.57+3.26
12.95+2.98
E.2 ADDITIONAL FID AND KID RESULTS THE SDI DATASET
We provide additional results in the case of training GANs with augmentation methods in Table 14 Augmentation methods like ADA Karras et al. 2020a) or DiffAug (Zhao et al. 2020) are proposed to adapt GAN training to limited data. We applied these augmentation methods to StyleGAN v2 and BigGAN, because these state-of-art image synthesis methods not optimized for small dataset. However, incorporating the augmentation methods in training GANs on the SDI dataset is not always beneficial. The performance of StyleGAN v2 is largely degraded when using ADA, potentially due
19

---

Under review as a conference paper at ICLR 2022
to the conflict between augmentation methods and the decentralized location of defects in the SDI dataset, defects can occur anywhere on the surface. This is in contrast to datasets that were used to evaluate the aforementioned augmentation methods in GANs, where the objects
centralized (e.g., ImageNet (Deng et al. 2009), Cifar (Krizhevsky & Hinton, 2009)) and their attributes (e.g., beard, eye glasses in CelebA (Liu et al. 2013)) only occur in specific images parts.
Table 14: Quantitative comparison of DTT-GAN with baseline image synthesis methods using FID and KID. Note that the reported values not comparable between columns, because they calculated on different training sets. The scores of StarGAN v2 on single products are omitted because generating images with specified background is not possible due its network design.
FIDL
KID
Method
A
B
C
All
A
All
Mokady et al.
2020
68.69
66.90
36.21
58.63
0.050
0.036
0.030
0,036
StarGAN v2
37.70
0,013
StyleGAN v2
90.10
52.95
138.09
35.34
0.072
0.027
0.186
0,013
StyleGAN v2 + ADA
149.66
42.75
135.69
76.16
0.138
0.019
0.191
0.055
BigGAN
235.66
192.89
193.61
151,43
0.248
0.199
0.276
0.115
BigGAN DiffAug
218.74
134.41
270.89
155.88
0.220
0.121
0.378
0,099
Ours
58.43
36.44
22.68
29.73
0.025
0.013
0.012
0.009
E.3 ABLATION STUDY WITH REGARD TO FID AND KID SCORES
We report the FID and KID scores of the ablation study in Table 15 We notice that both subnetworks show positive correlation to each modification except for structural change as in (a) and (e). Among the two subnetworks, the reference-guided subnetwork outperforms the latent-guided one in the beginning, which is due to the fact that transferring existing contents is casier than generating them from random noise. This effect is also observed in Figure 5| However, the performance of the latent- guided subnetwork improves significantly after applying per- per-pixel noise injection. The subnetwork can now output non-deterministie foreground contents even for a fixed input vector which results in better visual quality and higher diversity of generated defects. In the main text, the scores of the mixture set are reported.
Table 15: Ablation study with regard to FID and KID scores.
FID4
KIDI Reference
Latent
Reference
Mix
Latent
Mix
(a) Baseline StarGAN v2
37.73
37.99
37.70
0.013
0,013
0,013
(b) Style-Content branches
43.90
32.61
33.36
0.017
0.011
0.011
(c) Foreground classifier
37.14
32.34
27.69
0.014
0.011
0.008
(d) Background classifier
34.12
32.50
30.23
0.011
0.011
0.010
(e) Separately decoding
foreground and background in G
48.52
38.11
34.79
0.017
0.015
0,011
(1) Anchor foreground domain (e.g. Normal)
43.66
37.45
32.15
0.019
0.015
0.011
(g) Noise injection Mapping Network
33.05
34.42
29.73
0.009
0.011
0.009
E.4 ADDITIONAL RESULTS ON THE MVTEC ANOMALY DETECTION DATASET
The MVTee Anomaly Detection dataset (Bergmann et al. 2019) contains 15 different object and texture categories for anomaly detection. The dataset is formed of non-defective image for training and both non-defective and defective images with various kinds of defects for testing. The pixel-level annotations of all defective images are also provided. It is worth noting that the MVTec Anomaly Detection dataset is relatively small scale in number of images, where the number of training images is ranging from 60 to 391. Moreover, the number of defective images for each defect category in the test set is varying only from to 30, which is relatively limited considering the sophisticated pattern of defects.
20

---

Under review as a conference paper at ICLR 2022
We conducted image synthesis experiments subset of MVTee Anomaly Detection dataset, where we selected four texture categories: Carpet, Leather, Wood and Tile for our targeted sce- nario i.. surface defects. Furthermore, aggregated some of the original defect types defined in the MVTee Anomaly Detection dataset into s scrat ccratches and spots 8 according to their visual ap- pearance. We then simply added the subset of the MVTèc Anomaly Detection dataset to the training set together with the SDI dataset for training DT-GAN, Details of the resulting dataset are shown in Table 17 Note that the small scale of available data posts major challenge for training generative models.
Quantitative Evaluation. We present additional quantitative results on the subset of the MVTec Anomaly Detection dataset in Table 16 following the same evaluation setup as described in Ap- pendix C As shown in Table 16 our method achieves the best scores in Carpet and Wood, which supports our claim that DT-GAN generates synthetic images with higher fidelity and more diverse defect. However, we also observe that StyleGAN v2 seems to outperform our method in Leather and Tile.
Please note that FID and KID are not optimized to evaluate such a small dataset, there the results should only be interpreted together with the qualitative results.
Note the we again omit the FID and KID of StarGAN v2 because it not cable of generating images for specified product due to the 'identtt-shift',, which is also explained in detail in the qualitative evaluation.
Table 16: Quantitative comparison of DT-GAN with baseline image synthesis methods using FID and KID. Note that the reported values are not comparable between columns, because they were calculated on different training sets,
FIDJ
KIDJ
Method
Carpet
Leather
Tile
Wood
Carpet
Leather
Tile
Wood
Mokady (2020)
41.87
60.26
275.12
81.71
0.04
0.03
0.29
0.04
StarGAN v2
StyleGAN v2
51.37
51.60
225.96
140.01
0.05
0.03
0.23
0.12
BigGAN + DiffAAg
34.47
101.70
391.54
113.32
0.03
0.07
0.42
0.07
Ours
22.79
86.13
321.35
75.83
0.01
0.07
0.36
0.03
Qualitative Evaluation. For qualitative results, we again discuss the 'latent-guided" and 'reference- guided' synthesis separately.
We present the Hatent-guiden image synthesis results of StyleGAN v2 in Figure 11 and Figure 12 and BigGAN in Figure 13 and Figure 14 The results are acquired by training one model for each product and then generating 16 images from randomly sampled latent codes from each of them. As pointed out in Section 4.1.2 both methods not adapt well small dataset. They suffer from model collapsing and show signs of overfitting by generating images similar to the training data. For example, StyleGAN v2 generates images either with no clear defect identical to the training set (e.g.. Leather in Figure 11 and Product B in Figure 12). The overfitting we observe here also explains the better FID and KID scores in Table 16 For Tile, we can see clear signs of mode collapse in the generated Tile images of StyleGAN v2. Similarly, BigGAN produces images with single mode and abnormal patterns (e.g., grid structure and gray edges). Unlike StyleGAN v2 and BigGAN, StarGAN v2 and our method both require images as input (i.e. Source). Therefore, we randomly sampled two Normal images and applied eight defects which are generated from randomly sampled latent codes to each of them. As seen in Figure 15) Figurre 16) StarGAN v2 fails to preserve the background from the given input images due to the highly entangled FG and BG. Also it fails to generates legit and diverse defects without separately modeling the style and the content. In contrast to aforementioned methods, our DT-GAN produces images with higher fidelity and more diversity in defect patterns shown in Figure 17 and Figure 18 We believe this again prove the importance of style-content separation and FGBBG disentanglement, which we introduce in Section 3.1
(000)) are shown in Figure 19 and Figure 20 while the results of StarGAN v2 are in Figure 21 and Figure 22 We can observe
For 'reference-guiie -gccdedd image synthesis, the results of Mokady et al.
21

---

Under review as a conference paper at ICLR 2022
clear shift in color in all the outputs from Mokady et al. 20220). can only transfer content between two domains. In order to perform translation from nen-defeetivee sample to a defective one, trained model for each type of defect and for each product. This sums up to be 13 models (Scrate chess ches and Spot.s s for categories and Scratches only for Tile). The results from the intended use within background domain marked in red in both Figure 19 Figure 20 We still show the images that we feed in images from other background domains. As expected, the model then fails to preserve the background of given source images and introduce artifacts to the outputs. Similarly, StarGAN v2 does not preserve the background from the input images. Without style-content separation and FG/BG disentanglement, we observe that StarGAN v2 encodes the background characteristics together with the foreground content of the reference images, which results identity-shits its output images. Moreover, the output images either show no clear defect contain abnormal patterns which sabotage the fidelity. On the contrary, our method can faithfully transfer the foreground content of reference images across given background of different products as shown in Figure 23 and Figure 24 which demonstrate the effectiveness of the style-content separation and FG/BG disentanglement we introduced in Section 3.1
Moreover, Mokady al. (2020)
be found the diagonal and
is also worth noting that our method can perform cross cross-domain image synthesis even the desired combination not presented in the training We demonstrate this product Tile, which only has images with Scratches but no Spots. As shown in Figure 18 and Figure 24 DT-GAN can generated spots one given Tile images. However, this kind of transformation is most useful when the desired combination is reasonable for the downstream applications.
Limitation and Future Work. We have demonstrated the feasibility of the proposed DT-GAN by incorporating more products from the MVTec Anomaly Detection dataset in our training procedure. Intensive experiments have shown that the generated images from DT-GAN yielded better results compared to the baseline image synthesis methods. However, we noticed that despite the diverse patterns of the generated defects, DTTGAN tends to apply the styles learned from the SDI dataset also to the samples from the MV'Tec Anomaly Detection dataset. For example, we can observe some "halo" effects in Leather and Wood in Figure 18 and some of the generated scratches in Figure 17 and Figure 23 are rather weakly pronounced. We hypothesize this can be counteracted by explicitly localizing the defect and enforcing the model to learn conditional relationships between 'styles' and different backgrounds. We aim to address these issues in future work.
22

---

Under review as a conference paper at ICLR 2022
Overview of our formation of the MVTec Anomaly Detection sub-dataset. The first column represents the original defect types in the MVTec Anomaly Detection dataset while the first row stands for the defect types in our targeted scenario. We list the ID of samples we took from MVTec Anomaly Detection dataset and show the number of samples in row Sum.
Table 17:
Carpet
Scratches
Spots
Color
011, 012, 014, 016, 017 018
000, 003, 004, 007, 015, 018
Thread
Hole
000-016
Sum
24
23
(b) Leather
Scratches
Spots
Color
001, 003, 005, 007, 009, 011, 013, 015, 018 000 0018
000, 002, 006, 008, 010, 012, 014
Cut
Fold
- 006, 009 016
Glue
003, 0099 010, 016, 017
000 - 002, 005 - 009, 011 - 015, 018
Poke
000 017
Sum
48
39
(c) Tile
Scratches
Spots
Crack
000 016
Sum
17
0
(d) Wood
Scratches
Spots
Color
003, 005
Scratch
001 - 006, 008 010, 013 016, 018 - 020
000 - 016
Hole
000 - 004, 006 009
Combined
008
001, 002, 009
Sum
19
12
23

---

Under review as a conference paper at ICLR 2022
Randomly Sampled Defects (Scratches)
Product A
Product B
Product
Carpet
Leather
Tile
Wood
Figure 11: Latent- satent-guided image synthesis results of StyleGAN v2 on the SDI dataset and the MVTTec
AD dataset. We train model for each product and generate 16 Scratchheeeeeeeeee ches images from randomly sampled latent codes.
24

---

Under review as a conference paper at ICLR 2022
Randomly Sampled Defects (Spots)
Product
Product
Product
Carpet
Leather
Wood
Figure 12: Latent-guided image synthesis results of StyleGAN v2 on the SDI dataset and the MV'Tec
AD dataset. We train model for each product and generate 16 Spots images from randomly sampled latent codes.
25

---

Under review as a conference paper at ICLR 2022
Randomly Sampled Defects (Scratches)
Product A
Product B
Product
Carpet
Leather
Tile
Wood
Figure 13: Latent-guided image synthesis results of BigGAN with DiffAug on the SDI dataset and the MVTec AD dataset. We train a model for each product and generate 16 Scratches images from randomly sampled latent codes.
26

---

Under review as a conference paper at ICLR 2022
Randomly Sampled Defects (Spots)
Product
Produuc B
Product
Carpet
Leather
Wood
Figure 14: Latent-guided image synthesis results of BigGAN with DiffAug on the SDI dataset and the MVTec AD dataset. We train a model for each product and generate 16 Spots images from randomly sampled latent codes.
27

---

Under review as a conference paper at ICLR 2022
Source
Randomly Sampled Defects (Scratches)
Product A
Product B
Product C
Carpet
Leather
Tile
Wood
Figure 15: Latent-guided image synthesis results of StarGAN v2 on the SDI dataset and the MVTTe AD dataset. The model is trained joint set of aforementioned datasets and performs translation from Normal to Scratches. Note that without the style-content separation and the FG/BG disentanglement, StarGAN v2 not only fails to preserve the background from the given Source image but also fail to generates legit defects.
28

---

Under review as a conference paper at ICLR 2022
Source
Randomly Sampled Defects (Spots)
Product A
Product
Product C
Carpet
Leather
Tile
Wood
Figure 16: Latent-guided image synthesis results of StarGAN v2 on the SDI dataset and the MVTec AD dataset. The model trained on a joint set of aforementioned datasets and performs translation from Normal to Spots. s. Note that without the style-content separation and the FG/BG disentan- glement, StarGAN not only fails preserve the background from the given Source image but also fail to generates legit defects.
29

---

Under review as a conference paper at ICLR 2022
Source
Randomly Sampled Defects (Seratches)
Product
Product B
Product C
Carpet
Leather
Tile
Wood
Figure 17: Latent-guided image synthesis results of DT-GAN on the SDI dataset and the MVTec AD dataset. The model is trained on joint set of aforementioned datasets and performs translation from Normal 1 to Sc Scratches. Note the our model takes input Source images as background and only synthesizes the foreground defects from randomly sampled latent code compared to StyleGAN v2 and BigGAN.
30

---

Under review as a conference paper at ICLR 2022
Source
Randomly Sampled Defects (Spots)
Product A
Product
Product C
Carpet
Leather
Tile
Wood
Figure 18: Latent- guided image synthesis results of DT-GAN on the SDI dataset and the MVTec AD dataset. The model is trained joint set of aforementioned datasets and performs translation from Normal to Spots. Note the our model takes input Source images as background and only synthesizes the foreground defects from randomly sampled latent code compared to StyleGAN v2 and BigGAN.
31

---

Under review as a conference paper at ICLR 2022
Normal
Source
Reference
Scratches
Figure 19: ifterence-guidedd image synthesis results of Mokady et al. (2020) on the SDI dataset and the MVTec AD dataset. We train model for each product and each defect type. Then we translate Normal images to Scratches by taking the Source as background and applying the foreground defect from Reference to it.
32

---

Under review as a conference paper at ICLR 2022
Normal
Source
Reference
Spots
Figure 20: ffterence-guidddd image synthesis results of Mokady et al. (2020) on the SDI dataset and
the MVTec AD dataset. We train model for each product and each defect type. Then we translate Normal 1 images to Spots by taking the Source as background and applying the foreground defect from Reference to
33

---

Under review as a conference paper at ICLR 2022
Normal
Source
Reference
Scratches
Figure 21: Reference-guidee image synthesis results StarGAN v2 on the SDI dataset and the MV'Tec AD dataset. The model trained on joint set of aforementioned datasets and performs translation from Normal to Scratches by taking the Source background and applying the foreground defect from Reference to Note that without the style-content separation and the FG/BG disen- tanglement, StarGAN v2 not only fails to preserve the background from the given Source image but also fail to generates legit defects.
34

---

Under review as a conference paper at ICLR 2022
Normal
Source
Reference
Spots
Figure 22: Refereecce-guded image synthesis results of StarGAN v2 on the SDI dataset and the MVTec AD dataset. The model trained on joint set of aforementioned datasets and performs translation from Normal to Spot by taking the Source as background and applying the fore- ground defect from Reference it. Note that without the style-content separation and the FG/BG disentanglement, StarGAN v2 not only fails to preserve the background from the given Source image but also fail to generates legit defects.
35

---

Under review as a conference paper at ICLR 2022
Normal
Source
Reference
Scratches
Figure 23: Reference-guided image synthesis results DT-GAN on the SDI dataset and the MVTec AD dataset. The model is trained joint set of aforementioned datasets and performs translation from Normal to Scrat ches by taking the Source as background and applying the foreground defect from Reference to it.
36

---

Under review as a conference paper at ICLR 2022
Normal
Source
Reference
Spots
Figure 24: Reference-guidee image synthesis results of DT-GAN on the SDI dataset and the MV'Te AD dataset. The model is trained joint set of aforementioned datasets and performs translation from Normal to Spots s by taking the Source as background and applying the foreground defect from Reference to
37


Reviews:
[{"review_id": "qIdfvFcpB_B", "reviewer": "Reviewer_GYiR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n+ The paper addresses an interesting and important application. \n\n+ The technical implementation details are well documented. The appendix provides great details on the experimental protocol and additional results. The authors promise to publish the code and dataset so this could be a nice contribution to the community. \n\n+ The classification accuracy was higher when using the proposed data synthesis approach. \n\nWeakness: \n-\tI have a hard time to find anything that is technically novel. All the training losses are inherent from StartGANv2 or CycleGAN. The main difference lies in the foreground/background separation, but there was not ablation study to validate this. \n\n-\tOnly results on a *closed* dataset is reported. I feel that this paper would be a lot more better suited for conferences with a focus on applications. I am not sure what’s the take-away lesson on the technical components from this paper.  ", "summary_of_the_paper": " This paper introduces Defect Transfer GAN to transfer or generate different types of defects such as stretches or spots (foreground) and apply them onto different product images (background). The method builds upon StarGANv2 and add the cycle/content consistency loss and classification loss between foreground and background. Experiments were reported on the Surface Defect Inspection dataset. Results show that the proposed method achieved better FID and KID scores. The paper shows that training on the synthesized dataset performs better than traditional data augmentation.", "summary_of_the_review": "I think this paper tackles an interesting applications. The results are thorough and solid. My main concern of this paper is that the technical novelty is limited. I am thus leaning negative about this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hUR3Ve1wiDR", "reviewer": "Reviewer_swLC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe paper is well writen and easy to follow, the proposed methods seems to mitigate the data collection and labelling costs of the defect inspection problem widely available in production industry.\n\nWeakness:\nFrom technical part, the model is basically a tailored StarGAN v2 tuned for the defect generation task, the novelty and contribution of methodology is limited, particularly for ICLR 2022.\n\nThe employed SDI dataset is not publicly available and it's difficult for researchers to evaluate and compare the performances.\n\nWhile StyleGAN v2 and BigGAN are involved for comparison, how do you train them for the synthesis? By fintuning? or training from scratch?\n\nOnly scratch and spots are involved, the number of categories for defects are too limited and it's not convincing to support the claim. Much more defects and products need to be tested and evaluated.\n", "summary_of_the_paper": "\nThe paper prposes a StarGAN based model to distangle the defect foreground, background, transfer the style of foreground and then synthesize the defected image for different products. The quality of the synthesized defect images are evaluated using FID, KID. The approach is also appled to augment defect images to improve the performance of defect classification.", "summary_of_the_review": "\nThe work presented in the paper can be regarded as an application of StarGAN v2, while the paper is well writen and easy to follow, the novelty and contribution of methodology is limited, particularly for ICLR 2022.\n\nAs only scratch and spots are involved, the number of categories for defects are too limited and it's not convincing to support the claim. Much more defects and products need to be tested and evaluated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZELkxiiGt2n", "reviewer": "Reviewer_NBex", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "> It is hard to understand what is the style and what is the content in the targeted application. Even though the method is proposed to tackle the defect recognition problem; while style and content are not clearly defined in such a context. This confuses the concept of style and content throughout the draft.\n\n> The technical novelty looks weak: Most of components are similar to the StarGAN v2. While authors insist that they added few more modules such as background/foreground classifier, however this extension is rather trivial. Method-wise it is hard to find the improvement. In the application-side there could be, however it was not clearly explained why such style/content separation and bg/fg classification are important for the targeted application.\n\n> Experiments are somehow limited: Only 1 dataset is involved for the experiment. Furthermore, the presentation for the results is rather weak. Figures 3 through 6 are not clearly explained. They are not the natural images and it is hard to empathize that the images obtained is operating well for the given scenario. I think authors need to better explain such a subtle improvement.", "summary_of_the_paper": "Authors proposed the GAN method that can translate the defect between images. The obtained results from the proposed GAN is able to disentangle the defect-specific content from the background images in the weakly-supervised manner. By using the proposed GAN method, authors augmented the training data. The accuracy gain obtained using the data augmentation shows consistent and significant gap to the conventional method without the data augmentation.", "summary_of_the_review": "Due to the limited presentation, experiments and technical novelty, I am on the borderline for this draft yet. However, I could go towards the accept, if authors could made effective rebuttal to my comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P_hVP2UiEa", "reviewer": "Reviewer_Ks5k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- Applying GANs to defect synthesis is interesting.\n\n- The results of reference-guided defect transfer are promising.\n\n**Weaknesses**\n\n- My main concern is the technical novelty. From the task perspective, using GANs for data augmentation is not new, and using GANs for style/content transfer is also not new. This paper just applies it to a new area. From the technique perspective, most of the architectures are borrowed from StarGAN v2. Some loss terms are added to improve the performance, but all losses seem to be trivial. Despite the good results, I cannot see many insights from this work (*e.g.*, how can this work guide other researchers?)\n\n- The proposed approach is only evaluated on the Surface Defect Inspection (SDI) dataset. The superiority cannot be fully verified.", "summary_of_the_paper": "This work applies GAN for data augmentation to enhance defect classification. Concretely, based on StarGAN v2 structure, the proposed framework is able to encode the foreground and the background separately, enabling diverse defect synthesis by either transferring the defect from a reference sample or synthesizing from randomly sampled noises.", "summary_of_the_review": "To be honest, I do not think this work is suitable for ICLR as most audiences may not get interested. I strongly recommend the authors submit this work to a conference/journal in the industry field.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AE6PccE97Px", "reviewer": "Reviewer_GkV1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Detailed comments are as follows.\n\n1. The proposed method is reasonable, but the technical novelty is only moderate. The GAN formulation described in the paper heavily relies on StarGAN v2, with additional addition of the content transfer strategy introduced in Mokady et al., 2020.  \n\n2. The presentation needs significant improvements. For example, the contents/writing of Section 3 are way too similar to StarGAN v2. This could negatively affect the evaluation of the technical contributions of this work.\n\n3. The experimental results are not convincing. The experiment is tested on only the SDI dataset, and the comparisons do not include existing SOTA techniques for defect classification. It is not clear whether the claimed main advantage of increasing the diversity of defects on each specific target product would still be valid for the case that defects of different products vary significantly. Also, it would be more insightful to include the experiment of defect localization (say, over the MVTec AD dataset) to further justify the usefulness of the proposed approach.\n\n4. The ablation study on page 7 should be made more comprehensive (e.g., with explicit quantitative results), rather than with qualitative examples as in Figure 5.\n\n5. The paragraph of cross-domain effect (with respect to Table 4) needs to be discussed in more detail. Is the defect transferring still effective for the case that defects of different products vary significantly?\n", "summary_of_the_paper": "This paper describes a StarGAN-based approach to generating synthetic images of various defects based on the underlying training data. The proposed method considers two different types of domains, e.g., the foreground domain as defect types and the background domain as product types, and the derived synthetic images are included in the training data to improve the performance of defect classification.", "summary_of_the_review": "The technical novelty of the proposed method can be improved, and the experiments lack comparisons with SOTA techniques for defect classification. This work is not ready for publication yet.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a GAN based method for synthesizing various types of defects as foreground on different product images (background). The method builds upon StarGANv2, and adds the cycle/content consistency loss and classification loss between foreground and background. While the paper considers an important problem/application, the reviewers found it lacking sufficient novelty for publication. The paper will be more suited for publication at an application oriented venue.
Decision: Reject