## 2 General Structure of the Methodology
### Multi-Source Data Input Module
#### 2.2.2 Optimization of the Lightweight Model Based on Knowledge Distillation  
In the knowledge distillation process, the teacher model not only provides traditional label information but also conveys richer knowledge to the student model through intermediate-layer feature representations. Soft labels, which represent the probabilities of each category in the Softmax output layer, are introduced. These soft labels encompass both the true label information and additional implicit knowledge beyond the actual labels. Although the output of the Softmax layer contains substantial information--providing the probability values for the true labels as well as indicating which negative labels are closer to the true labels--when the entropy of the probability distribution is low, the entropy of the negative labels' distribution also becomes small. In such cases, the hard label values may approach zero, rendering them ineffective as soft labels. To address this, a temperature coefficient (\(T\)) is typically introduced in the Softmax function, enhancing the influence of negative label information and facilitating the transfer of more comprehensive knowledge. The knowledge distillation process can be divided into the following two stages:  
Figure 3: Flowchart of transfer learning. Figure 2: Lightweight rail surface condition identification method based on knowledge distillation and transfer learning. * Teacher model training phase. In this phase, the teacher model is trained until it achieves satisfactory performance, after which its training parameters are solidified. The teacher model then generates high-fidelity knowledge outputs, including soft labels and intermediate layer features, which are subsequently transferred to the student model during the distillation process. * Student model training phase. In this phase, the student model optimizes its training process by utilizing the soft labels generated by the teacher model. The performance of both the teacher and student models is evaluated using the following distillation loss function when aligning the respective logical units: \[L_{KD}(p(u,\ T),\ p(z,\ T))=\sum_{i=0}^{k}-p_{i}(u_{i},\ T)log(p_{i}(z_{i},\ T))\] (1) where \(T\) is the temperature coefficient used to control the softening of the output probabilities, \(u\) and \(z\) are the logical units of the output of the teacher and student models, respectively, \(u_{i}\) and \(z_{i}\) are the values of the logical units of the \(i\)-th class of the teacher and student models, \(p_{i}\) is the class probability of the \(i\)-th class, and \(k\) denotes the category. When \(T\) is positive infinity, the distillation loss at backpropagation is optimized as follows: \[\frac{\partial L_{KD}(p(u,\ T),\ p(z,\ T))}{\partial z_{i}}=\frac{p_{i}(z_{i}, \ T)-p_{i}(u_{i},\ T)}{T}\approx\frac{1}{T}\Bigg{(}\frac{1+\big{(}\frac{z_{i} }{T}\big{)}}{k+\sum_{j=0}^{k}\frac{z_{i}}{T}}-\frac{1+\big{(}\frac{u_{i}}{T} \big{)}}{k+\sum_{j=0}^{k}\frac{u_{j}}{T}}\Bigg{)}\] (2) Assuming that \(z_{i}\) is 0-mean, there is the following: \[\frac{\partial L_{KD}(p(u,\ T),\ p(z,\ T))}{\partial z_{i}}\approx\frac{1}{ kT^{2}}(z_{i}-u_{i})\] (3)  
In addition to training with the soft labels generated by the teacher model, incorporating hard labels further enhances the learning process. This is achieved by balancing the contributions of two distinct objective functions. The first objective function represents the cross-entropy loss between the teacher model's predictions at a higher temperature \(T\) and the student model's predictions at the same elevated \(T\). The second objective function corresponds to the cross-entropy loss between the student model's predictions and the ground-truth hard labels, computed at \(T=1\).