# SDD-CNN: Small Data-Driven Convolution Neural Networks for Subtle Roller Defect Inspection
## 3 Materials and Methods
### CNN Architectures
#### 3.3.2 Inception v3  
[PERSON] et al. introduced new design principles and optimization ideas based on GoogLeNet and proposed Inception v3. The paper points out that the representational bottleneck caused by severe compression should be avoided in the early stage of a network. In addition, higher dimensional representations are more suitable for processing locally within a network. Accordingly, Inception v3 optimizes and improves Inception v1 from the following three aspects: (1) factorize convolutions with a large filter size, including the use of a multilayer perceptron to replace a \(5\times 5\) filter and the special factorization of a \(3\times 3\) filter into an asymmetric structure of \(3\times 1\) and \(1\times 3\); (2) introduce auxiliary classifiers with a regularization effect to accelerate the convergence of the network; (3) parallel a convolutional layer with _stride_ = 2 and a pooling layer to achieve more efficient grid size reduction.  
Noteworthily, Inception v3 proposed a mechanism to regularize the classifier layer by estimating the marginalized effect of label-dropout during training. For each training example \(x\), the model computes the probability of each label \(k\)\(\in\)[\(1\dots\)\(K\)]:  
\[p(k|x)=\frac{\exp(z_{k})}{\sum_{i=1}^{K}\exp(z_{i})} \tag{3}\]  
where \(z_{i}\) represents the logits or unnormalized log-probabilities. Considering the ground-truth distribution over labels \(q(k\,\)\(\,\)\(\,\)\(\,\)\(x)\) for this training example, the loss can be defined as the cross entropy:  
\[l=-\sum_{k=1}^{K}\log(p(k))q(k) \tag{4}\]  
Minimizing this is equivalent to maximizing the expected log-likelihood of a label. Additionally, the cross-entropy loss is differentiable with respect to \(z_{k}\) and the gradient has a simple form:  
\[\frac{\partial l}{\partial z_{k}}=p(k)-q(k) \tag{5}\]  
Consider the case of a single ground-truth label y, so that \(q(y)\) = 1 and \(q(k)\) = 0 for all \(k\) = \(y\). For a particular example \(x\) with label y, the log-likelihood is maximized for \(q(k)\) = \(\delta_{k,y}\), where \(\delta_{k,y}\) is Dirac delta, which equals 1 for \(k\) = \(y\) and 0 otherwise. This maximum is not achievable for finite \(z_{k}\), but is approached if \(z_{y}\) >> \(z_{k}\) for all \(k\) \(\
eq\)\(y\). This, however, can cause over-fitting and encourage the differences between the largest logit and all others to become large, which reduces the ability of the model to adapt. In order to encourage the model to be less confident about its predictions, a distribution over labels \(u(k)\), independent of the training example \(x\), and a smoothing parameter \(\epsilon\) are introduced to replace the \(q(k\,\)\(\,\)\(\,\)\(x)\) with  
\[q^{\prime}(k\,\)\(\,\)\(\,\)\(\,\)\(x)=(1-\epsilon)\delta_{k,y}+\epsilon u(k) \tag{6}\]  
In this paper, considering that the labels of the roller dataset have been normalized by the LD method, the uniform distribution \(u(k)\) = \(1/K\) is used so that  
\[q^{\prime}(k)=(1-\epsilon)\delta_{k,y}+\frac{\epsilon}{K} \tag{7}\]  
where \(K\) = 11 classes and \(\epsilon\) = 0.1.