# Towards Federated Learning with Byzantine-Robust Client Weighting
## 3 Experimental Evaluation
### Experimental Results  
The Shakespeare experiments without any attackers are shown in Figure 3, and the executions with attackers are shown in Figure 4. The results of the MNIST experiments are similar to the results of the Shakespeare experiments and are relegated to Appendix B.  
Figure 4: Accuracy by round under Byzantine attacks for the Shakespeare experiments. Curves correspond to preprocessing procedures, and columns correspond to different aggregation methods. In the two rows of the experiment, the Byzantine clients perform a model negation attack with 1 and 10% attackers, respectively.  
Figure 3: Accuracy by round without any attackers for the Shakespeare experiments. Curves correspond to preprocessing procedures, and columns correspond to different aggregation methods.  
Figure 3 shows that using truncation (dashed orange curve, labeled \"Truncate\") remains comparable to the properly weighted mean estimators, where no preprocessing is done (solid blue curve, labeled \"Passthrough\"), whereas ignoring clients' sample sizes and weighing the clients uniformly (dotted green curve, labeled \"Ignore\") is sub-optimal. This effect is pronounced when the unweighted median is used (middle column, dotted green curve), since it is generally very far from the mean with our unbalanced partition.  
From Figure 4, we observe that even with a single attacker performing a trivial attack (first row), using the weights directly without preprocessing (solid blue curve) is devastating, whereas when our truncation-based preprocessing method is used in conjunction with robust mean aggregations (i.e., median and trimmed mean) (dashed orange curve, two last columns), convergence remains stable even when there are \(\alpha\) (=10%) attackers (second row). In contrast, the same cannot be said for the regular mean aggregator, as can be seen by the sub-optimal accuracy (2 nd row) and an occasional accuracy drop (1 st row) in the leftmost column (the drops can be explained by the fact that in each round we randomly select clients for training, and so the byzantine clients have varying effects across different rounds). We note that our method may be slightly less accurate in some cases when ignoring reported client sample sizes and weighing them uniformly (dotted green curve, second row, middle column). This is expected because we allow Byzantine clients to potentially get close to the \(\alpha^{*}\)-proportion (50%, in this case) of the weight. However, our method is significantly closer to the optimal solution when there are no or only a few attackers (see Figure 3). Moreover, when used with robust mean aggregation methods, it maintains their robustness properties.  
The results from the first experiment, running without any attackers (Figure 3), demonstrate that ignoring client sample size results in reduced accuracy, especially when median aggregation is used, whereas truncating according to our procedure is significantly better and is on par with properly using all weights. These results highlight the imperativeness of using sample size weights when performing server aggregations.  
Whereas Figure 3 shows that truncation-based preprocessing performs on par with that of taking all weights into consideration when all clients are honest, Figure 4 demonstrates that the results are very different when there is an attack. In this case, we see that when even a single attacker reports a highly exaggerated sample size and the server relies on all the values of \(N\), the performance of all aggregation methods, including robust median and trimmed mean, quickly degrades.  
In contrast, in our experiments, robustness is maintained when truncation-based preprocessing is used in conjunction with robust mean aggregations such as median or trimmed mean, even when Byzantine clients attain the maximally supported proportion, \(\alpha=10\%\).