## 6 Discussion and Implications  
Technical images (such as images of PCBs and solders) demonstrate a pre-defined design and low variability, so the detailed information about their design is crucial for the detection of defects. In unsupervised learning, unlabeled samples are used to extract information about possible patterns in data. Transfer learning technique is utilized to re-use knowledge learned by a network in some other but related area. In our experiment, we used VGG16 as a feature extractor in several ways (A0, A1, and A2), using it either alone or with the RotNet fine-tuning, to compare how network would learn the salient features from the data and use them for learning of normal patterns in data. Unsupervised training should extract intrinsic image features, without domain-specific knowledge. The patterns learned are based on features from pre-trained ImageNet. We assume that the space of features, extracted by ImageNet, are rather comprehensive. The approach A0 analyzes whether we could use the features from ImageNet, without fine-tuning. The approaches A1 and A2 use RotNet fine-tuning upon transfer learning to find a subspace of broad in scope ImageNet features that more efficiently represents the features of our image set. The difference between approaches A1 and A2 is that the former one exploits more generic features of the image, and the latter one uses more specific information after fine-tuning. The comparison of signatures of normal samples and sample to analyze is based on assumption that the transformation of input image into the signature possesses the property of continuity in feature space. In the consequence, two alike images (having nearly the same characteristics) will be mapped to feature vectors in the same small vicinity. However, if an image contains a defect, its signature should be located far from the cluster of the defect-free samples. In case of solder dataset DS, the MCC-A1 crossed the plot of MCC-A0 (pretrained VGG16 neural network without RotNet fine-tuning) when the number of training samples was above 600 (Figure 7, Tables 1 and 2). MCC-A2 is below both plots. We can conclude that at some stage the fine-tuning facilitates better extraction of common normal patterns from the images. For PCB dataset (Figure 8, Tables 3 and 4), the behavior of MCC-A1 and MCC-A2 is the same; the first one is above the second. On the contrary to Figure 7, there is no crossing of MCC-A0 and MCC-A1, i.e., the quality of defect detection for fine-tuned network is  
Figure 8: PCB dataset: MCC for A0, A1, and A2. worse than for untuned one. It could be explained by considering the number of samples in the DS and PCB sets. In case of PCB dataset, a very small number of training samples and rather large number of parameters of layers FC1 and FC2 to be trained causes the following problem. The back-propagation algorithm could not reach a global optimum for thousands of trainable parameters, due to insufficient number of training samples. It does not learn much as a small number of input parameters cannot be used for optimization of a much larger number of network-tunable parameters. As soon as the number of training samples grows, the results of the optimization keep getting better. The more normal cases are used for training, the more successful fine-tuning becomes, and at some stage MCC-A0 goes under MCC-A1, as Figure 7 demonstrates. In both cases MCC-A1 is above MCC-A2. The similar observation was also made in [35]. The reason is, the RotNet fine-tuning creates artificial labels and trains the network to learn and to extract the features related to these artificial classes. As the result, the earlier located fully-connected layer FC1 (used in model A1) extracts more generic features of the dataset images. The next layer FC2 (used in A2) is tuned to extract features associated to artificially introduced class labels (related to transformations of rotation), and not to the properties of the dataset itself. This observation could explain the manifested difference in behavior of MCC-A1 and MCC-A2.