###### Abstract
In CFL, nodes usually form a star-shaped network, with the server being in the center of the network, so all nodes have access to it. In DFL, although nodes usually connect only with their one-hop neighbors, so that they form a ring-shaped network (ad hoc or peer-to-peer networks), there are also works that have implemented d-ring or mesh [10] topology under DFL settings. FL has also been proven to mitigate the lack of data availability of these devices during training, since they do not have to own all of the data required for training in order to reach convergence. Although there are a lot of approaches [11] regarding CFL, they do not actually meet the needs of TinyML environments, like IoTs, due to their requirement of a coordinator server. The energy required by the edge nodes to send their parameters to the server when embarking on the learning process may cause the depletion of their total energy, and/or _communication_ overhead/flooding, since every node is obliged to communicate with the server. Moreover, the fact that the training process still takes place outside of the end devices hinders the ability of the devices to be independent. Despite the fact that FL was first introduced in order to handle the problem of privacy concerns, since nodes do not have to send their collected data over the network, only some of their parameters, as shown in [12], there is a possibility of reconstructing the initial data sent to the server by aggregating their total gradients. Thus, the literature has focused on DFL, which is a scheme a) that operates without a central orchestration; b) in which every node shares information with its neighbor nodes, which are wirelessly connected; c) that has unsolved problems like the broadcast storm problem [13] in ad hoc networks; and d) that lacks trustworthiness, since in ad hoc networks, there are often attacks that resource-starving devices cannot recognize/handle (due to their lack of energy/memory needed to always authenticate their collaborators). Moreover, _Swarm Learning_ (SL) [14] has emerged, which successfully combines DFL methodology in wireless networks with blockchain-based protocols for retaining trustworthiness and/or security. thiness in the network. SL practices are not amenable to being deployed to end devices, since they are also computationally expensive. The emergence of the aforementioned distributed deep learning technologies and the progress in the development of products running over distributed environments create the pressing need to describe the most significant advances in this research landscape. In light of this need, the main question that our survey aspires to answer is the following: What are the most recent and important solutions for addressing deep learning challenges in resource-starving environments that can be encountered in (a) highly distributed architectures running either over wireless ad hoc networks or over conventional peer-to-peer networks and (b) miniaturized devices, e.g., microcontrollers and sensors? Since security is always an important issue in distributed architectures, a second question that this survey will adequately cover is about What has been proposed for guaranteeing privacy and security in distributed deep learning? The anatomy of this survey is the following: we aimed to provide answers to the aforementioned questions by drawing on academic papers, published in well-respected academic databases and digital libraries of publishers, e.g., Google Scholar, IEEEXplore, ACM DL, Scopus, and MDPI (see Figure 1), and whose publication age is no larger than five years. We ensured that the surveyed articles appeared in ACM/IEEE Transactions and Journals, and/or in journals with a significantly large impact factor, or in premium quality conferences such as NeurIPS, IJCNN, ICDM, and so on. We also included some yet unpublished articles (e.g., from arxiv.org), but only very recent articles for which (some of) the authors are prominent scientists or when we felt that the article is going to be a significant one.