## 3 Proposed Method
### OtO Method  
The multi-stage learning of the final output of the teacher network stimulates the learning potential of the student network; however, the performance improvement of the student network is limited because the teacher network only delivers the probabilistic information available at the final exit. The student network receiving information is a multi-exit branch network, and the gap between the learning ability of the pre-branch  
Figure 2: Overall framework diagram of the MSKD. Each of these exits possesses a distillation loss from the teacher network and classification loss from the true labels.  
student network and that of the teacher network is too large. This gap affects learning to a certain extent. Therefore, the teacher network is also constructed as a multi-exit architecture network and the teaching style is changed to one-to-one instruction such that the teacher network is targeted to guide the student network through different stages of learning. The overall framework of our proposed OtO method is shown in Figure 3, and the supervisory information consists of two parts: (1) the true label and (2) the branch network of teachers corresponding to that stage. The loss of the \(k\)-th exit of the student network in this method can be expressed as:  
\[\begin{split} Loss^{k}_{\text{{o}to}}&=Loss^{k}_{ \text{{c}la}}+Loss^{k}_{dis^{\prime}}\\ &=\sum_{i=1}^{N}\Bigl{(}(1-\alpha)L_{CE}\Bigl{(}p^{k}_{s}(x_{i},1),y_{i}\Bigr{)}+\alpha\tau^{2}D_{KL}\Bigl{(}p^{k}_{s}(x_{i},\tau),p^{k}_{t}(x_{i},\tau)\Bigr{)}\Bigr{)}\end{split} \tag{8}\]  
where \(p^{k}_{t}(x_{i},\tau)\) denotes the output of the probability distribution of the \(k\)-th exit of the teacher network after softening. As can be seen with the above description, the main difference between Equation (5) and Equation (8) is the source of guidance of the teacher network, one is the final output of the teacher network with the specific formula \(p_{t}(x_{i},\tau)\) in Equation (5) and the other corresponds to the output of the teacher branch network with the specific formula \(p^{k}_{t}(x_{i},\tau)\) in Equation (8). Therefore, the overall loss of the student network can be expressed as:  
\[\begin{split} Loss_{\text{{o}to}}&=\sum_{k=1}^{K} Loss^{k}_{\text{{o}to}}\\ &=\sum_{k=1}^{K}\sum_{i=1}^{N}\Bigl{(}(1-\alpha)L_{CE}\Bigl{(}p^{ k}_{s}(x_{i},1),y_{i}\Bigr{)}+\alpha\tau^{2}D_{KL}\Bigl{(}p^{k}_{s}(x_{i}, \tau),p^{k}_{t}(x_{i},\tau)\Bigr{)}\Bigr{)}\end{split} \tag{9}\]  
Figure 3: Overall framework of the OtO method. Each of these exits has two types of losses: a distillation loss from the branch network corresponding to the teacher network and classification loss from the true label.  
When each exit of the student network corresponds to an exit of the teacher network for one-to-one learning, the difference in the learning capacities between the teacher and student networks becomes small. We can then seek commonalities and differences in the training process with students sharing parameters in the backbone of the network and varying independent learning in the branches. Additionally, information is shared from different stages and relative differences are maintained. The same ensemble approach as that of MSKD is used to integrate multi-stage learning outcomes as the final learning outcome during testing.