## 3 On the Prospect of TinyML/DL
### Knowledge Distillation--KD
#### 3.2.2 On-Device Real-Time Learning (Reformable TinyML)  
The majority of the existing literature focuses on TinyML solutions that are based on offline settings and only support neural network inference on low-powered and ultra-low-powered devices like MCUs (see Table 3). On the contrary, the training phase of a network is very costly in terms of both energy and storage. In this section, we introduce methods/techniques that pave the way for online on-device training. Continual (or lifelong) learning (CL), unlike conventional transfer learning, allows for real-time online training using raw data. In this way, the model can learn new tasks without forgetting the knowledge gained from the original task. In the literature, CL methodologies are divided into three main categories [92]: _architecture-, regularization-,_ and _rehearsal-_ based methodologies [19]. Architecture-based approaches adapt layers and activation functions to prevent forgetting, depending on the deployed device, while regularization-based approaches reinforce existing knowledge through penalties in the loss function or (_knowledge distillation_) [93]. With the term _rehearsal or native rehearsal_[94; 95; 96], we refer to a learning technique in which for every training batch, a randomly chosen subset of training data is saved to an external storage and re-forwarded for training, replacing another random subset of the incoming batch patterns, mainly aiming at handling _catastrophic forgetting_ (a phenomenon in which acquired knowledge tends to vanish, due to repeatedly learning new incoming data). Despite being the most suitable solution for combating catastrophic forgetting, this method raises data privacy concerns as sensitive raw data must be stored externally from the edge device. The key is to identify the optimal solution by considering the memory requirements and the specific data that need to be stored for accurate outcomes. We can say that there is a trade-off between the precision of saved information and non-harmful forgetting, known as the _stability/plasticity dilemma_[7]. Thus, the sub-category of the rehearsal-based approaches which alleviates this trade-off is the one called _latent replay_[97]. _Latent replay_-based approaches store activation outputs of past data at some random intermediate layer, instead of the data themselves, reducing the storage requirements compared to those of conventional rehearsal-based approaches. One of the first latent replay- and gradient-based approaches, designed for MCUs in the literature, is the one described in [98]. Combining fully 32-bit precision continual learning, the multi-core parallelization capability of a 22 nn Risc-v-based microprocessor, namely _VEGA_, and a quantization strategy, [PERSON] et al. provided a platform (QLR-CL) that outperformed a low-power STM32 L4 microcontroller, and specifically, it is 65x faster and 37x more energy-efficient than the aforementioned microcontroller. Additionally, experiments were carried out using both 8-bit and 7-bit quantized memory, resulting in a small decrease of at least 0.26% and 5% in accuracy, respectively, compared to the 32-bit implementations. Generally, tested on Core50 dataset, this method achieves up to 77% accuracy, while it reduces the memory footprint (about 64 MB), paving the way for more online adaptive strategies for TinyML to emerge. [PERSON] et al., in [99], questioned the efficacy of the backpropagation algorithm on online on-device training in MCUs due to the fact that there is a need of gradient computation, which further means that both enough computation power and memory storage are required. Instead, they proposed the Forward-Forward (FF) algorithm, in which the forward pass and the backward pass procedures are replaced by two forward ones that contain dissenting type of data (positive and negative ones, respectively).