# FLIP: A New Approach for Easing the Use of Federated Learning
## 4 V&V: Validation and Verification of the Platform
### Objectives Verification  
The main objective defined for this work is to create a platform that allows the use of FL in a simple way, so researchers could work with clinical data and machine learning algorithms without having to worry about the architecture and deployment of the system. To verify it, a specific study is defined consisting of the following steps. * Step 1: setting the main objective of the study, in charge of defining the main objective of the experiment, that is, if it is possible to claim that this solution is easy to use. * Step 2: setting the study context, in charge of defining the guidelines about how the study is conducted. * Step 3: defining the main tools for the study, responsible for identifying the means to be applied during the study. * Step 4: the implementation of the study and the collection of results focused on both, conducting the analysis according to the previous guidelines and the tools to be used and then collecting the information from the users. \begin{table}
\begin{tabular}{c c c c c c} \hline
**Server** & **Loss LR** & **Loss NN** & **Loss NN FL** & **Time Local vs. Federated** & **Max RAM Local vs. Federated** \\ \hline MacBook PRO & 0.09145 & 0.08537 & 0.07672 & +45 s & +21,636 kb \\ Unix Server 1 & 0.10235 & 0.09229 & 0.08824 & +31 s & 9992 kb \\ Unix Server 2 & 0.09332 & 0.083167 & 0.07666 & +15 s & +8640 kb \\ \hline \end{tabular}
\end{table} 
Table 3: Results of the verification experiments. Step 5: analysis of the results and drawing of conclusions, in charge of discussing whether the main objective was achieved according to the results gathered from the study. With the aim of not only checking if the platform is usable, but also comparing its level of usability with an existing solution, in Step 2 we carry out a comparative study. To do so, we first select the solution to be compared with, that is, Flower. We choose this one since it can be considered as the basis of our platform as previously explained and due to the fact that it also provides the main capabilities needed for a real federated learning implementation. Regarding the tools to be used, for the analysis of user acceptance in terms of ease of use, we perform a SUS (System Usability Scale) test, since it can be considered the most widely adopted regulated questionnaire for the appraisal of perceived usability [24]. Regarding the usability test implementation, it is divided into two different parts: first, the users are asked to complete a set of tasks in the Flower scenario and then to answer a specific SUS to provide their opinion about their experience. Once this first step is completed, a second round is conducted following the same approach, focused on the use of our platform. For both rounds, users are provided with a list of instructions with the different steps that a machine learning expert should follow in a real scenario. These steps are included in Table 4.  
Along with the standard SUS questions, an additional set of four initial questions is also included at the beginning of the test. They are related to the level of studies and the background knowledge about data science, federated learning, and computer system administration that helped us to understand the profiles of the involved participants. The test was finally performed by a total of 14 users, which is enough for a sample of usability tests according to the literature [25]. Figure 5 shows the results for these initial questions:  
Figure 4: NN model performance vs. server capacities radar chart. According to these initial results, the participants have a high educational level (at least graduate), half of them have a data scientist profile, and a total of 11 have basic or advanced knowledge about federated learning and computer system administration. Once provided with this initial information, users are asked to complete a SUS questionnaire for each of the solutions tested. For comparative analysis and presentation of results, we rely on the System Usability Scale Analysis Toolkit provided by [26].