# Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
## 3 Proposed Methods
### More than One Teacher  
From what we saw above with the introduction of our solution, we realized that it is always possible to take advantage of a single teacher. Now, we ask ourselves whether  
Figure 2: Proposed approach with one teacher. We can distinguish three different networks: the teacher, the translator, and the student. The loss function is composed by the mean squared error (MSE) between the student and translator plus the first cross-entropy (XENT (1)) of the student’s feature and classify layer flow, plus the second cross-entropy (XENT (2)) of the translator’s feature and classify layer flow. it is possible to generalize this concept to more teachers and understand under what conditions the introduction of other teachers in the transfer learning process continues to bring advantages. In the multi-teacher scenario, the final loss function uses the student's last layers \(l_{i}\) that are closest to the output layer \(l_{0}\) or the classification layer. In particular, the teacher \(t_{1}\) will use the layer \(l_{1}\), the teacher \(t_{2}\) will use the layer \(l_{2}\), etc. As we did in the case of a single teacher \(t_{1}\) in which we take their features layer \(f_{t_{1}}\) and transfer it to the student through the use of the MSE loss with the student's layer \(l_{1}\), generalizing this process for each teacher \(t_{i}\), we transfer their knowledge to the student via a new MSE loss between the features' layer \(f_{t_{i}}\) and the student layer \(l_{i}\) (see Equation (5) for details). In addition to minimizing the distance between the features' layers through the use of an MSE loss, as we did in our proposal in the case of a single teacher, we need a new additional \(\mathcal{L}_{xent}(y,x)\) (defined in Equation (1)) for each new teacher introduced. As shown in Figure 3, the features layer of teacher 2, called \(f_{t_{2}}\), is the same size as the student's layer \(l_{2}\) but not necessarily the same size as the student's layer \(l_{1}\). This observation forces us to transform the size of \(f_{t_{2}}\) into the size of \(l_{1}\) before we can calculate the XENT3 loss shown in Figure 3. To perform this last step, the existing layer \(l_{1}\) is used so the \(\mathcal{L}_{xent3}(y,x)\) input values will be \(x=l_{0}(l_{1}(f_{t_{2}}))\). This is best described in Equation (4). All this reasoning can be generalized to the case of a student with T teachers, as better described in Equation (5). Below, we formalize the loss function in the case in which we only have two teachers:  
\[ \begin{split}\mathcal{L}(y,f_{t_{1}},f_{t_{1}},f_{t_{2}},f_{t_{2} })=\mathcal{L}_{MSE}(f_{t_{1}},f_{t_{1}})+\mathcal{L}_{xent}(y,l_{0}(f_{t_{1}} ))+\\ \mathcal{L}_{MSE}(f_{t_{2}},f_{t_{2}})+\mathcal{L}_{xent}(y,l_{0} (l_{1}(f_{t_{2}})))+\\ +\mathcal{L}_{xent}(y,l_{0}(f_{t_{1}}))\end{split}  \tag{4}\]  
Figure 3: Proposed approach with two teachers. We can distinguish four different networks: the teacher, the translator 1, the translator 2, and the student. The loss functions MSE(1), XENT(1), and XENT(2) are composed as in Figure 2, however, here we also add MSE(2) between the features of Translator2 and the student’s layer Features2 and the third cross-entropy XENT(3) using the features of Translator2 and the student’s classification layer. where \(f_{t_{1}}\) and \(f_{t_{2}}\) are the translated features of the two teachers, \(f_{t_{1}},f_{t_{2}}\) are the student features into which we want to transfer the learning of the teachers, \(l_{0}\) it the last layer function or classification layer, and \(y\) is the labels. We can generalize the formula in Equation (4) for T-teachers using the function composition \(\circ\) as \(g\circ f(x)=g(f(x))\). Consequently, the loss function of our model in the case where exactly T teachers and one student are used is the following:  
\[ \begin{split}\mathcal{L}(y,f_{t},f_{t})=\mathcal{L}_{xent}(y,l _{0}(f_{l_{0}}))+\\ +\sum_{i=1}^{T}\left[\mathcal{L}_{MSE}(f_{t_{i}},f_{l_{i}})+ \mathcal{L}_{xent}(y,l_{0}\circ l_{1}\circ...\circ l_{i-1}(f_{t_{i}}))\right] \end{split}