## 3 On the Prospect of TinyML/DL
### Taxonomy of TinyML/DL Models  
#### 3.2.1 Static Model Inference  
Low-powered devices, such as sensors and MCUs, have been utilized in the literature to perform efficient machine learning tasks. They have wide-ranging applications in areas like everyday life, security, surveillance, industrial monitoring, smart agriculture, etc. [1]. The field of on-edge data processing using ML algorithms (TinyML) is a new and emerging area. With its fast and real-time performance, it also enhances devices' capabilities and privacy and reduces communication costs between them and a server. Of course, conventional ML techniques cannot be deployed on the aforementioned devices because these devices lack the computational and memory resources required for an ML/DL algorithm to run efficiently. Consequently, multiple methods are recommended to optimize the inference phase of an ML model by developing compact models, as outlined in the following sections. Pruning  
In this section, pruning aims to reduce the memory requirements of the model by reducing its trainable parameters--hyperparameters. Thus, pruning is bisected into two main categories: _static_, in which the model completes training and is then pruned, \begin{table}
\begin{tabular}{c c c c} \hline
**Framework/Platform** & **Architecture** & **Purpose** & **Capabilities** \\ \hline TensorFlow Federated & CFL & smart-grid applications & open source \\ (by Google) & CFL & finance & open source \\ \hline FATE & CFL & healthcare application & not open source \\ \hline BrainTorrent [53] & DFL & academic usage & open source, developer-friendly API \\ \hline Scatterbrained [11] & DFL & academic usage, & open source, \\  & & suitable for IoT networks, & scalability, \\ FedML [61] & CFL & benchmarking & reliability \\ \hline FedStellar [62] & \begin{tabular}{c} CFL, \\ Semi-Decentralized FL, \\ DFL \\ \end{tabular} & \begin{tabular}{c} addresses heterogeneous \\ FL-based problems \\ \end{tabular} & \begin{tabular}{c} easy customization, \\ deployment of complex network \\ topologies, \\ secure communication, \\ enough storage for \\ FL models \\ \end{tabular} \\ \hline FL-SEC [11] & \begin{tabular}{c} DFL, \\ Blockchain \\ \end{tabular} & \
\begin{tabular}{c} orientation of \\ Artificial Internet \\ of Things (AIoT) \\ \end{tabular} &
\begin{tabular}{c} not open source, \\ defense in \\ poisoning attacks, \\ communication reduction \\ \end{tabular} \\ \hline \end{tabular}
\end{table} 
Table 2: Frameworks used in federated learning. and _dynamic--dynamic sparsity during training_--in which the model is both pruned and regrown iteratively (the latter one will be further analyzed in Section 3.2.2. Despite the phase in which the pruning is conducted (static or dynamic), we can define other categorizations of pruning as follows [67]: structured pruning (like channel pruning), which can easily be used in conjunction with any other optimization method, and unstructured pruning, which provides the network with the largest size reduction possible but makes it more vulnerable to performance decrease [68]. According to the work in [69], there is a smaller sub-network whose randomly initialized parameters make it capable of replacing the initial fully connected network and therefore undergoes effective training while the redundant information of the network can be ignored. This technique is called _Lottery Ticket Hypothesis_, and the chosen sub-networks are called _winning tickets_. This pruning technique effectively reduces the trainable parameters of feedforward neural networks (such as MLPs and CNNs) by up to 90%. As a result, memory requirements are reduced, enabling deployment on resource-limited devices for inference tasks without any loss in accuracy. In fact, in some cases, it can even improve the initial results. Similarly, the technique of _Depth Pruning_, described in [70], involves discarding layers after training to create a sub-network. An auxiliary network is then used as the head of the pruned network, resulting in a marginal decrease in accuracy for an already pre-trained model.