###### Abstract
We refrained from setting hard \"performance\" thresholds such as number of citations or \(h\)-index as paper selection criteria, because we are well aware of how citation inflation efforts distort the scientific value of an article. Our survey overlaps to a small extent with some earlier surveys; the kind/extent of overlap and our discrepancy is described in the rest of this paragraph. Table 1 lists published surveys most relevant to our work: [PERSON] et al. (2015) comprehensively presented the possibilities of deploying distributed deep learning and deep reinforcement learning in wireless networks, emphasizing distributed computing, e.g., Hadoop. In this survey, the model was mainly split, and every device trains on a part of the whole model, instead of collaboratively training the same model by sharing model parameters (the case we focus on). In parallel, [PERSON] et al. (2016) presented a detailed review about machine learning optimization in order to fit in low-power devices (e.g., embedded systems, MCUs, etc. ); nevertheless, they mainly focused on machine learning instead of deep learning approaches. [PERSON] et al. (2016) pointed out the necessity of shifting toward decentralized learning techniques and the need to construct suitable learning models for edge devices; however,they solemnified the network infrastructural perspective without referring to model optimization. Despite the benefits of working on edge devices, there are pivotal concerns about both communication safety and data adequacy among the participating entities, as analyzed by [PERSON] et al. in [PERSON] et al. (2014). In this work, they address _trust_ issues, regarding node cooperation in fully distributed environments, using blockchain-based methods; however, swarm nodes need more than enough computational power in order to meet both the high-energy and memory expectations of blockchain methodology. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Title** & **Year** & **Purpose** \\ \hline Distributed Machine Learning & & \\ for Wireless Communication & 2021 & it focuses on wireless networks \\ Networks: Techniques, Architectures, & & \\ and Applications [15] & & \\ \hline An Overview of Machine & & \\ Learning within Embedded & 2021 & it emphasizes machine learning \\ and Mobile Devices—Optimizations & & for embedded devices \\ and Applications [1] & & \\ \hline Demystifying Swarm Learning: & & the main goal of this work \\ A New Paradigm of Blockchain-based & 2022 & was to tackle the imbalanced \\ Decentralized Federated Learning [14] & & training dataset problem by \\  & & leveraging a blockchain-based \\  & & infrastructure \\ \hline Decentralized learning & 2021 & it focuses on distributed FL \\ in federated deployment environments: & & \\ A system-level survey [16] & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Reporting the most relative existing surveys. Figure 1: Screening process [17] of relevant academic works. The present survey aimed to contribute the following:  
* The current advancements of the amalgamation of three wide areas, TinyML, decentralized federated learning, and Swarm Learning, for which their amalgamation revolutionize both the viability and autonomy of resource-starving decentralized environments;
* The incorporation of new up-to-date entries, published in the recent literature, focusing on cutting-edge cross-device decentralized federated learning methodologies, e.g., on-device reformable TinyML, peer-to-peer communication among tiny devices, blockchain contributions to communication safety, etc. ;
* The exploration of current challenges that have yet to be fully addressed in the deployment of deep neural networks on resource-starving devices;
* The presentation of challenges concerning the _ethical concerns_ that this rapidly increasing technological era (e.g., Generative AI models) faces. The organization of this paper is as follows: The introduction of decentralized federated learning and TinyML/DL is covered in Sections 2 and 3, respectively.