# Towards a Decentralized Collaborative Framework for Scalable Edge AI
## 6 Experimental Evaluation
### Efficacy of the Proposed Approach  
We extend the FLASH framework with the MDD module to aid the clients in discovering and distilling the models. We also use FL as the most common baseline for decentralized learning methods. To have a fair comparison, we run the training for FL in an ideal scenario to produce its best results. To this end, we use the uniform (or homogeneous) setting (**U**) for Federated Learning in which devices have the same hardware configurations and are available throughout the training. We aim to demonstrate that different clients in the network can learn within their own groups at their own pace (i.e., FL group vs. individual clients' group, as in Figure 2). Then, the MDD service will prove to be effective if discovering and distilling the models for the individual clients or groups via the MDD service results in improvements of the models produced by other parties. In this case, the individual clients request the FL groups' model and use it as their initial model before training. Figure 3: The quality impact of heterogeneity. We find that the values are zero in heterogeneous cases when the models diverge. The values are greater than one when an experiment results in better model accuracy than the baseline with the default configurations. Experimental setting:Similarly to Figure 2, the experiments involve a small group of 10 independent parties individually conducting local training on their own data (**IND**) for a variable number of epochs and a large group consisting of the remaining clients training a global model via FL for 500 rounds (**FL**). We contrast the model quality of these two groups with the case when the proposed architecture is used (**MDD**). The 10 individual learners will request a global model trained by the FL group from the discovery service, and then each of the 10 learners will distill the FL model with their own model. The result plots show the test accuracy averaged by testing a model on the 10 individual clients (**IND**). The x-axis shows the local epochs used for training the model using the isolated 10 individual clients' datasets. Therefore, this number would be 0 in the FL case. Traditional AI models:First, we conduct the experiments using 10 clients in an individual group and 9990 clients in an FL group to train using logistic regression as the model on synthetic dataset distributed in non-IID fashion over all the clients in both groups. As shown in Figure 4, training the individual clients for a few epochs attains high average accuracy (**IND**) due to the simplicity of the LR model. The FL model (**FL**) shows lower accuracy when tested on the test datasets of the 10 individual parties because of the large population size, which confirms with observations about FL in non-IID settings [19]. However, when the 10 clients request and distill the FL model via (**MDD**), their average test accuracy improves by 12 accuracy points. Deep Neural Network (DNN) models:Next, we leverage DNN models to conduct experiments in a device heterogeneous setting. Specifically, both Recurrent Neural Network (RNN) and Convolution Neural Network (CNN) models are trained by over 10 clients who learn individually, and 3990 and 803 clients belong to the FL group. The FEMNIST and REDDIT datasets, used to train the CNN and RNN models, respectively, are distributed in a non-IID fashion among all the clients in both groups. Next, in Figure 5a, we observe that training for the model locally for 100 epochs does improve the model for the individual parties (**IND**). The FL model (**FL**) shows good accuracy when tested on the 10 individual clients. This might be due to the CNN models' ability to generalize well on samples of the Feminist dataset (which consists of easy-to-classify image classes of digits and letters). Moreover, the model is trained on many samples from 3390 clients in the FL group.