# Knowledge Distillation-Based GPS Spoofing Detection for Small UAV
## 2 Related Work
### Knowledge Distillation  
With the rapid development of the deep neural network (DNN), there are many advanced applications providing smart and intelligent assistance in many fields. With the compact deployment of AI and ML in computation-limited devices, IoT and CPSs are becoming indispensable to human activities. Advanced communication methods also accelerate cloud computing, fog computing, and edge computing [23] to be deployed ubiquitously to provide computational services. However, these computational services require many computational resources and lack the flexibility to provide instant support when communication quality is low. There are many reasons to deploy lightweight DNNs on computation-limited devices. Transfer learning (TL) is becoming a popular way to make lightweight DNNs feasible for computation-limited devices. Among the many technologies of TL, knowledge distillation has many compact and computation-limited applications. KD has many explorations on knowledge categories, training schemes, training architectures, distillation algorithms, and performance comparisons [23]. The development speeds up the compact applications deployment in IoT and CPSs. In deployment, the most important part is distilling knowledge, which determines the quality of the networks after distillation. To explore this mechanism, a review mechanism in knowledge distillation uses an attention-based fusion module, a hierarchical context loss function, and a residual learning framework to improve the learning process of the review mechanism. The results show that the mechanism can improve the performance of transferred models on classification, object detection, and instance segmentation tasks [24]. Conventional approaches of KD focus on the transfer of instances and ignore the correlational congruence of KD. A generalized kernel-based Taylor series expansion to capture the correlation between instances in transfer learning was proposed in [25], which outperforms the conventional methods in the accuracy of the identification rate. However, accuracy improves with the sacrifice of storage. Among the many methods of KD, the most popular model is the teacher-student (TS), which has two different networks: the teacher network and the student network. In KD processing, the student network imitates the input and output of teacher networks, so it does not need additional training materials and preserves private information in some special scenarios. Some research studies have investigated the research of knowledge distillation with the student-teacher framework on vision tasks. Their comprehensive survey explained the mechanisms of knowledge distillation [26]. Due to the different structures, there is a knowledge gap between student and teacher models in knowledge distillation. The gap causes the student models to underfit the teachers' models. A spherical knowledge distillation is proposed to eliminate the gap and significantly improve the students' performance [27]. There are also some applications of TS in different fields. To improve the observation accuracy and reduce the computation load, [28] focuses on human position predictions with transfer learning. With less input, the trained student models achieve similar abilities as the teacher models. Ref. [29] leverages TS to deploy lightweight deep neural networks to UAVs and provide good vision computation servicesfor UAV delivery. To deploy deep reinforcement learning in a large-scale UAV network, [30] transfers leader knowledge to local UAVs with TS to reduce the overhead for computation. To prevent direct contact with private information, TS transfers the complex neural network to lightweight networks with the consideration of preserving privacy. The new lightweight network is deployed in UAV networking to assist UAVs in recognizing targets [31]. The TS model of KD has the advantage of deploying complex neural networks to compact devices, which can obtain different sophisticated functions from the complex network and help computation-limited platforms achieve intelligent functions.