# Reverse Knowledge Distillation with Two Teachers for Industrial Defect Detection
## 3 Methods
### Teacher Model  
Currently, many distillation network architectures focus on distillation from a single teacher to a single student for anomaly detection and localization [15]. However, in the human learning process, a student does not learn from only one teacher; instead, he may learn from different teachers or different sources of information. Multiple teacher models provide multiple interpretations of the student model's task by providing multiple streams of information, and the student model can use the teacher model's views of the target task to improve the model's performance. Thus, an ensemble of multiple teacher network predictions typically yields better performance than a single teacher network. Therefore, two teachers' distillation networks are adopted.  
Due to the different teachers needed, ResNet34 and Wide_ResNet50 pretrained on ImageNet are used in our framework [41; 42]. They are enabled to extract rich features from images. In them, the last block (conv5_x) is removed and the output feature maps are extracted by the remaining blocks (conv2_x, conv3_x, and conv4_x). In Wide_ResNet50, they are named E11, E12, and E13, respectively. Similarly, they are known as E21, E22, and E23 in ResNet34, respectively. Finally, we fuse E11 and E21, E12 and E22, and E13 and 23, respectively, and then use them as intermediate feature layers of the network.  
Since it has only one student decoding network for calculating the loss, it needs to fuse the two teachers' feature channels and use a simple concatenation operation. For knowledge transfer, cosine similarity is used as our loss. The loss function is defined in Equation (1). In training, the size of the input image is 224 \(\times\) 224 \(\times\) 3, so the size of the first layer (E11, E21) feature map is 64 \(\times\) 64 \(\times\) 320, the second layer (E12, E22) feature  
Figure 1: Overview of our proposed framework for anomaly detection and localization. A contains A1, A2, and A3. Similarly, A2 and A3 are the same as A1. In the same way, B and C have the same properties as A. Our model consists of two different pretrained teachers encoder E, a multiple attentional feature fusion module, and a student decoder D. A multiscale channel attention module (Mscam) is used to fuse features in different layers from E and ensemble high-level and low-level features from Mscam by multiscale feature (MFF) block. Iterative attentional feature fusion (iAFF) is adopted to fuse features from two MFF and map it onto a compact feature vector by a one-class embedding (OCE) block. By minimizing the similarity loss between the two teacher models, student D learns to imitate the behavior of concatenated features in training. The anomaly map is generated by fusing the multiscale similarity maps of the student decoder.  
map is 32 \(\times\) 32 \(\times\) 640, and the third layer (E13, E23) feature map is 16 \(\times\) 16 \(\times\) 1280. After fusion, the feature maps of each layer are 64 \(\times\) 64 \(\times\) 320, 32 \(\times\) 32 \(\times\) 640, and 16 \(\times\) 16 \(\times\) 1280, respectively.  
In order to obtain a two-dimensional anomaly map \(M^{k}\), the formula is calculated as follows.  
\[M^{k}(h,w)=1-\frac{f_{T}^{k}(h,w)\cdot f_{S}^{k}(h,w)}{\|f_{T}^{k}(h,w)\|\cdot \|f_{S}^{k}(h,w)\|} \tag{1}\]  
where \(f_{T}^{k}\) and \(f_{S}^{k}\) represent the feature vectors extracted from the teacher network and the student network, respectively. The \(h\) and \(w\) are defined as the height and width of the feature map, and k denotes the number of feature layers.  
In the training process, the loss function \(L\) is represented as follows.  
\[L=\sum_{k=1}^{3}\left\{\frac{1}{H_{k}W_{k}}\sum_{h=1}^{H_{k}}\sum_{w=1}^{W_{k} }M^{k}(h,w)\right\} \tag{2}\]  
where \(h\), \(w\) and \(k\) have the same meaning as in Equation (1).