## 3 On the Prospect of TinyML/DL
### Knowledge Distillation--KD
#### 3.2.2 On-Device Real-Time Learning (Reformable TinyML)
The goal is to either increase (FF on positive data) or decrease (FF on negative data) the sum of the squares of the activities in every layer, namely _goodness_, and based on this metric, to reconfigure the weights of the aforementioned layers. In [100], [PERSON] et al. present an improved variation of FF, called \(\mu\)-FF, with the aim of further reducing the memory and computation requirements of the above heuristic by separating the method into two _blocks_: the first one functions as a feature extractor, for the better understanding of non-linear data, while in the second phase of the algorithm, the training procedure is executed, using Mean Square Error (MSE) as the loss function along with a ridge regularization term. Experiments were conducting in an STM32 MCU, using the X-cube-AI tool for quantization and the Fashion-Mnist dataset, and it was shown that the MCU performance when running \(\mu\)-FF online on-device is comparable with the corresponding one when the backpropagation algorithm is deployed in the same MCU in an offline setting. Although this heuristic is an attractive means for resource-starving devices, it has a low-speed convergence rate, and in general, its performance is inferior to the conventional back-propagation algorithm. However, dynamic pruning ([67]), as firstly introduced in Section 3.2.1, can enhance online training as it is based on the \"iterate and regrow\" technique. Elements (weights or neurons) are dynamically removed and re-added in order for the model to keep its composition (otherwise, it is going to progressively lose all of its connections after a few epochs). The main idea behind this method is that in a fully connected network, the connections of neurons that are going to be removed in a layer are close to zero in weight, and hence, they are of less importance. For example, [PERSON] et al. [101], inspired by network science, proposed a scheme of random close-to-zero weight connection removal, and in following a preferential attachment, the retrieval of the connection completes after every epoch, following a power law distribution. The basic idea using the power law distribution is to create scale-free networks, which are constructed to reinforce the high-influence connections/nodes in the network and hence retain the high performance of the network. Furthermore, the works outlined in [102; 103] take this idea a step further by sparsifying a fully connected neural network, e.g., an MLP, dynamically either by creating an original scale-free network or modifying the number of removable connections following dynamic functions during training, focusing on lessening the training time of the network while achieving better accuracy than the initial network in some cases. In this section, the papers that follow are based on the fact that the knowledge being transmitted, or the weights of a pre-trained network, is information that contributes to the model generalization [104], but the models have to be able to adapt to new circumstances in order to retain high performance. For instance, in the work presented in [105], a transfer learning (TL) kind of scheme is proposed. In contrast with conventional TL, TinyOL does not retrain the last fully connected layerm but it functions as an extra layer connected to the last layer of the already pre-trained model. The weights in TinyOL are adapted to new data through online stochastic gradient descent (SGD), while it runs in RAM, with pre-trained weights frozen once TinyOL starts being fine-tuned. Furthermore, [106] presents a scheme that combines transfer learning with K-nearest neighbors, enabling training on embedded systems and IoT units using minimal data. In this way, networks can evolve over time, enhancing their performance by implementing independent local training.