# Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
## 3 Proposed Methods
### Our Proposal  
Since we experimentally discovered that, by eliminating the translator and inserting the whole transfer learning process within a single cycle, we were able to eliminate the problem highlighted in the [PERSON] model, we can thus try to put the translator back, but this time, by adding a new loss function which in Figure 2 is called Xent (2). We do this to ensure that the translated features are, this time, as a function of the classification problem and not independently of this. Formally, the new cross-entropy loss introduced is defined in Equation (3) where we can see that the input values cl (fs) directly link the student's features to the classification layer. Compared to the solution analyzed in the previous section, we want to remove the limitation that the student and the teacher must have a layer with the same dimensions, but this time adding a new loss function. Thanks to the elimination of this constraint, in our proposed model, all types of networks, i.e., teachers, can teach any other type of network, i.e., students. In particular, we propose a _translator_ layer plus a cross-entropy loss function to overcome this limitation. The translator can be represented by a simple layer or even \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Name** & **Baseline** & **KD** & **Romero** & **T1S** & **T1S + Aug.** \\ \hline Cifar10 & & & & & \\ \hline Lenet5 & 65.43 & 64.84 & 62.11 & 64.12 & **65.49** \\ AlexNet & 72.75 & 70.93 & 70.38 & 73.74 & **73.99** \\ MobileNetV2 & 74.82 & 63.22 & 64.92 & 77.53 & **81.60** \\ Resnet18 & 78.76 & 82.64 & 82.35 & 80.81 & **83.52** \\ \hline Cifar100 & & & & & \\ \hline Lenet5 & 28.34 & 32.62 & 29.72 & **33.86** & 33.65 \\ AlexNet & 35.86 & 36.93 & 37.96 & **40.49** & 38.12 \\ MobileNetV2 & 31.31 & 29.03 & 31.26 & **43.25** & 39.47 \\ Resnet18 & 51.82 & 46.42 & 46.03 & **60.77** & 54.18 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of the baseline test accuracies on the Cifar10 and Cifar100 datasets with the knowledge distillation (KD) [1], Romero approach [2], with one teacher (T1S) and one teacher with augmentation (T1S + Aug.), after 200 epochs, using two Resnet50 as teachers (one for Cifar10 and one for Cifar100). The student is of the same size as the feature layer adapted to that of the teacher. by another network that transforms the dimensions of the teacher's features layer into the dimensions of the student's features layer, as we can see in Figure 2.  
More formally, the cross-entropy loss indicates how far the model predictions are from the supervised labels (\(y_{i}\)). The cross-entropy loss is defined as  
\[\mathcal{L}_{xent}(y,x)=-\frac{1}{m}\sum_{i}^{m}\log\Biggl{(}\frac{e^{x_{i}y_{i }}}{\sum_{j}^{m}e^{x_{i}y_{j}}}\Biggr{)} \tag{1}\]  
which computes the loss between predicted classes' probabilities \(x\) (in Figure 2, \(x\) is the output of the layer called Classify) and target \(y\) (in Figure 2, \(y\) is the target). In the training process, we update the model weights at each batch of size \(m\). In Equation (1), \(n\) is the number of classes, so \(x_{i,j}\) is the probability of the \(j\)-th class of the \(i\)-th prediction and \(x_{i,y_{i}}\) is the probability corresponding to the \(y_{i}\)-th class of the \(i\)-th prediction. The MSE loss is used to minimize the distance between the teacher's features and the student's features. The general formula of this last loss function is defined as follows:  
\[\mathcal{L}_{MSE}(y,x)=\frac{1}{m}\sum_{i}^{m}(y_{i}-x_{i})^{2} \tag{2}\]  
where \(y=f_{l_{1}}\) is the student's features (\(l_{1}\) is the name of the features layer) while \(x=f_{t}\) is the teacher's features. Combining together the two losses defined in Equations (1) and (2), we obtain the loss proposed for our model. For the case where only one teacher is used, the loss function is the following:  
\[ \begin{split}\mathcal{L}(y,f_{t},f_{l_{1}})=\mathcal{L}_{MSE}(f_ {t},f_{l_{1}})+\mathcal{L}_{xent}(y,cl(f_{t}))+\\ +\mathcal{L}_{xent}(y,l_{0}(f_{l_{1}}))\end{split}