## 3 On the Prospect of TinyML/DL
### Knowledge Distillation--KD  
The approach known as knowledge distillation was first introduced by [PERSON] et al. [77] and further generalized by [PERSON] et al. [78; 79], and it is a method that aims to compress neural network models. Knowledge distillation involves training a smaller neural network, known as the \"student\", to imitate a larger network, the \"teacher\", by minimizing a loss function shared by both models. Despite the large model being trained already, the smaller one can utilize the larger one's knowledge to learn raw data online, correcting itself with a loss function when it makes misleading predictions. This approach allows for the real-time training of small and energy-efficient models [80], but it necessitates the student model to depend on a pre-trained model, referred to as the teacher.  
Conventional Transfer Learning  
The biggest challenge in static TinyML is adapting the model to a new task. For example, if a model is well trained on recognizing images of cats and suddenly captures an image of a dog, it will probably misclassify the dog as a cat, since it does not have much knowledge to understand patterns out of the learned category (concept drift [66]). In order to address the problem, techniques such as _transfer learning (TL)_ are introduced. The TL technique uses a pre-trained model on a large dataset, e.g., Imagenet [81], and fine-tunes some of its last layers while freezing the earlier ones, when deployed to a tiny device, with the aim of learning new tasks. However, fine-tuning the last layers (mainly the last fully connected layers since the cost of retraining more layers is prohibitive in tiny devices) may cause imperceptible improvement or no improvement at all, since in this way, we affect only the weights of the classifier and we do not enter the main core of the network. For instance, inspired by the way a convolution neural network (CNN) works, in which every layer has a specific task to learn, retraining only the last fully connected layer will probably be ineffective, since we modify the results of the training procedure and not the procedure itself in order to embody the weights of the new task before the classification outcome. There are works, like [82], that try to retrain more than the last layer but it has a trade-off regarding energy consumption. We further describe transfer learning techniques in Section 3.2.2.  
Inference-based TinyML/DL has paved the way for ML/DL applications in healthcare, industry, everyday life, etc. For instance, [PERSON] et al. designed a prototype system to deploy ML inference tasks on microcontroller-powered edge devices for predicting blood pressure-related metrics [83], while [PERSON] et al. developed a real-time application on low-powered devices to support nurses' work with elderly individuals [84]. Regarding industry, an efficient TinyML model [85] is also used in industries, in order to detect anomalies through IoT devices related to production process. Furthermore, the use of TinyML models in detecting pavement anomalies through intelligent vehicles [86], gas leakage [87], coughs [88], falls [89] and smart agriculture [90] are explored. There are also hardware-based implementations, like the one in [91], in which an ultra-low-power IoT monitoring device is presented, showcasing the integration of Bluetooth and NFC connectivity for monitoring asset activity and so on.