# Robust Federated Learning for Mitigating Advanced Persistent Threats in Cyber-Physical Systems
## 4 Experimental Results
### Results  
In order to study the effect of FL on constructing an accurate global model for APT detection, we first analyzed the results for the MLP, CNN, and 1D CNN in ten different scenarios. In each scenario, a model was trained on only one of the datasets listed in Table 3. The trained model was then evaluated on the test subset, which contained 20 percent of the samples from each class. Figure 3 shows the obtained results using a heatmap for each algorithm. The datasets used for training the model in each experiment are indicated on the vertical axis, whereas the APT classes are specified along the horizontal axis. The color spectrum on the right side of each box indicates the range of colors used to differentiate between smaller and larger values. Figure 3 illustrates that the MLP was not able to detect APT classes with acceptable accuracy when trained on a subset of data with only a few APT classes included. Given that the majority of samples were non-APT malware, the model exhibited better performance in detecting malware. Among the datasets used for training, datasets six to eight resulted in better overall performance across all classes. Comparing the spectrum of colors in Figure 3b with the other panels in this figure, it can be concluded that the 1D CNN generally outperformed the MLP and DAE when trained on partial data. Nevertheless, this model seemed to exhibit poorer performance in detecting samples of the APT28 class than its competitors. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Parameter** & **1D CNN** & **DAE** & **MLP** \\ \hline Epochs & 500 & 100 & 10 \\ Batch size & 64 & 256 & 32 \\ No. hidden layers & 2 & 2 & 1 \\ No. neurons & 64, 64 & 128, 256 & 128, 64 \\ Learning rate & 0.001 & 0.001 & 0.002 \\ Noise factor & - & 0.5 & - \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Parameter settings used for edge local models. \begin{table}
\begin{tabular}{l l c} \hline \hline
**Dataset** & **Classes** & **No. Samples** \\ \hline Dataset 1 & Malware, Transparent Tribe, Winnti Group, Carbanak, Desert & 7710 \\  & Falcon, Violin Panda & \\ Dataset 2 & Malware, APT29, Lazarus Group, Mirage, Sandwork, Violin Panda & 7523 \\  & Malware, Patchwork, Transparent Tribe, Carbanak, APT28, & 8043 \\  & Volatile Cedar & \\ Dataset 4 & Malware, Carbanak, APT28, Mirage, Shiqiang, Violin Panda & 7441 \\  & Malware, Winnti Group, Carbanak, APT30, Volatile Cedar, & 7569 \\  & Violin Panda & \\ Dataset 6 & Malware, Patchwork, Winnti Group, Lazarus Group, Mirage, & 7921 \\  & Volatile Cedar & \\ Dataset 7 & Malware, Hurricane Panda, Transparent Tribe, APT29, & 7952 \\  & APT30, Shiqiang & \\ Dataset 8 & Malware, Patchwork, Transparent Tribe, APT29, APT28, & 8141 \\  & Lazarus Group & \\ Dataset 9 & Malware, Transparent Tribe, Winnti Group, Mirage, Shiqiang, & 7658 \\  & Violin Panda & \\ Dataset 10 & Malware, Carbanak, APT30, Lazarus Group, Mirage, Desert Falcon & 7506 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Scenarios used to create training datasets for each experiment. Each FL node uses only one of these datasets. Interestingly, the 1D CNN's performance was more consistent across different experiments and scenarios. On the other hand, the DAE exhibited poorer detection performance than the MLP and 1D CNN, as the color spectrum shown in Figure 3c is generally darker than those in Figure 3a,b. Regardless, the achieved results showed less variance than those of the MLP. Although using a more advanced structure with more hidden layers could probably boost the performance of the MLP, 1D CNN, and DAE, we relied on simple structures to study the effect of using FL on boosting detection performance. Figure 3: Success rates of detecting different APT classes without using FL. In each experiment, the indicated model is trained on only one dataset and evaluated on the test subset. (**a–c**) show the results of MLP, 1D-CNN, and DAE, respectively.