# Improving Deep Mutual Learning via Knowledge Distillation
## 2 Materials and Methods
### DML and KD  
In this section, we are the first to combine both DML [11] and KD [8]. In Deep Convolutional neural network, Given the N samples of training data \(D\ =\{x_{i}\}_{i=1}^{N}\) from a collection of C classes, with \(Y\ =\{y_{i}\}_{i=1}^{N}\) as a label set and \(y_{i}\in\{1,\,2,\,3,\ldots,\,\text{C}\}\). The idea of this DML is to use two simpler networks to learn together as a knowledge transfer technique between one another without using a pre-trained teacher to improve the classification accuracy of the two or more networks. In addition to using cross-entropy loss as a supervised learning loss to predict the correct labels for training instances, this techniquealso uses KL-divergence loss to equalize the estimated probability between pairs and measure [11] the match of predictions \(p_{1}\) and \(p_{2}\). The supervised learning loss formula [18] that we use is:  
\[L_{s}\ =\ -\sum_{i=1}^{N}y_{i}\ \log(p(x_{i}))+(1-y_{i})\log(1-p(x_{i})) \tag{1}\]  
where \(p\) is the probability of the class and \(y_{i}\) is the corresponding class label. Figure 1 shows that both \(G_{1}\) and \(G_{2}\) networks produce outputs in the softmax layer in the form of logit \(z^{c}\). So, the probability of class C from sample \(x_{i}\) is defined as:  
\[p^{c}(x_{i})=\frac{e^{(z^{c})}}{\sum\limits_{n=1}^{M}e^{(z^{c})}} \tag{2}\]  
Then, the KL divergence formula is used to calculate the mimicry loss to denote \(p_{1}\) and \(p_{2}\) predictions of each network, which is defined as:  
\[L_{kl}(p_{2}\ \|\ p_{1})\ =\ \sum_{N=1}^{N}\sum_{c=1}^{C}p_{2}^{c}(x_{i})\log \frac{p_{2}^{c}(x_{i})}{p_{1}^{c}(x_{i})} \tag{3}\]  
Figure 1: Deep mutual learning (DML) diagram [11] that the network \(G_{1}\) and \(G_{2}\) trained simultaneously together, and using a KLD-based.  
Thus, the entire loss function for the \(G_{1}\) network using the formula can be calculated as:  
\[L_{\text{G1}}=\ L_{\text{s1}}+\ \lambda L_{kl}(p_{2}\parallel p_{1}) \tag{4}\]  
where \(\lambda=1\), is a weight factor. Similarly, for network \(G_{2}\) that the entire loss function can be calculated as:  
\[L_{\text{G2}}=\ L_{\text{s2}}+\ \lambda L_{kl}(p_{1}\parallel p_{2}) \tag{5}\]  
Knowledge distillation in [8] is one of the most popular knowledge transfer methods today, and it uses a teacher-student framework. The basic idea of this method is that a pre-trained teacher network (i.e., a cumbersome network or the biggest network) using certain hyperparameters are then used to train an untrained student network (i.e., small network) for the purpose of transferring knowledge. This process uses a distillation knowledge equation where a temperature (\(T\)) is involved and can be varied to obtain a soft probability output from a class C image which can be calculated as:  
\[P^{c}(x_{i})\ =\ \frac{e^{(x_{i}^{c}/T)}}{\sum\limits_{c=1}^{C}e^{(x_{i}^{c}/T)}} \tag{6}\]  
Suppose the teacher network is marked as \(G_{t}\) and the student network is \(G_{s}\), then the distillation loss can be defined as:  
\[L_{kld}(P_{t},P_{s})\ =\ -\frac{1}{N}\sum\limits_{n=1}^{N}\sum\limits_{c=1}^{C}P _{t}^{c}(x_{i})\log\ P_{s}^{c}(x_{i}) \tag{7}\]  
As a result, the student loss function contained in Figure 2 is minimized during the training process based on (6) and (7) as:  
\[L_{\text{G}_{s}}=\ L_{\text{s}}(\text{G}_{s},\text{D},\text{Y})+\ \lambda L_{kld}(P_{t},P_{s}) \tag{8}\]  
where \(\lambda\) is a balancing value between the two losses. The main purpose of the teacher-student framework is to force the student output probability to imitate or match the pre-trained teacher network's probability output.