# Research and Application of Edge Computing and Deep Learning in a Recommender System
## 3 Recommendation Models Based on Knowledge Distillation and Edge Computing
### Knowledge Distillation  
Knowledge distillation operates within a teacher-student framework, where the practice structure is typically a well-trained model serving as a source of knowledge for learning. The student model gains the teacher model's knowledge through distillation training, with a potential slight performance loss traded for transferring knowledge from complex teacher models to a simpler student model. Currently, most instances of knowledge distillation focus solely on a single teacher model, overlooking the possibility that a single student model can be supervised by multiple teacher models [20; 21]. Alternatively, recognizing the equal significance of multiple teacher models renders it impractical to acquire more valid knowledge based on inherent distinctions in these models. To enhance model accuracy, the knowledge distillation model, typically involving a single teacher model, was expanded to include multiple teacher models. Consequently, a student model can now be trained by several teacher models, enabling the acquisition of valid knowledge aligned with the interior differences among the models. In this study, the teacher model comprised state-of-the-art models such as DeepFM, the DCN, and xDeepFM, while a user-defined neural network model formed the student model. Within this framework, the student model can autonomously learn the importance of different teacher models for a given data sample and integrate this knowledge based on learned diverse degrees of importance. This approach significantly contributes to enhancing the learning capability of the student model [22]. The primary structure of this teacher-student model is presented in Figure 1 below:1. Different teacher models (quantity: n) were trained. These models were then used as samples in the training set to produce soft labels, \(\widetilde{y}^{Ti}\), accordingly. In order to figure out the soft labels, \(\widetilde{y}^{Ti}\), \(Z(Z=(z_{1},z_{2}\ldots z_{t}))\) was output through logits from the teacher model and then divided by the \(T\) parameter; the outcome obtained was subjected to \(Softmax\) at last. Here, \(T\) is a temperature coefficient. The higher the \(T\) parameter, the gentler the corresponding distribution probability tends to be. 2. The number of datasets was input into the student model, which was followed via operating it in the same way as the teacher model and obtaining an output of logits. Subsequently, relevant calculations were divided into two steps: First, via division by the same \(T\) parameter as that of the teacher model, \(\widetilde{y}^{Ti}=Softmax\left(\frac{z_{T}^{T}}{T}\right)\), \(Softmax\)-based computing was conducted, obtaining an output of soft predictions, \(\widetilde{y}^{S}\), and this output was then compared with soft labels of the teacher model. Second, after \(Softmax\)-based calculations, the predicted values were acquired and then compared with Ture labels. 3. For the corresponding loss function, \(KDLoss\) was selected, and it was formed via the combination of a relative entropy loss function and a cross entropy loss function [23]. Here, Kullback-Leibler divergence was used to measure the asymmetry of differences in probability distributions A and B, while cross entropy represents a difference value between the predicted sample label and the true sample label. A combination of these loss functions is beneficial to better reveal both differences in and differential values of the predicted samples and the true samples. \[KDLoss=\sum_{i=1}^{n}w_{i}\cdot KL(\widetilde{y}^{T_{i}},\widetilde{y}^{S}) \cdot\lambda\cdot T^{2}+CE(label,y^{S})\cdot(1-\lambda)\] (1) where \(KL\) stands for relative entropy, \(CE\) stands for cross entropy, and \(w_{i}\) stands for the weight of the teacher model, \(i\). Under the circumstance that coefficient \(\lambda\) is 0, \(KDLoss=CE(label,y^{S})\), no knowledge distillation is performed for the hybrid loss function that adopts a neural network of the cross entropy loss function. 4.