# ZenoPS: A Distributed Learning System Integrating Communication Efficiency and Security
## 6 Experiments
### Empirical Results  
The result of ZenoPS in the synchronous mode is shown in Figure 3. We can see that, when there are no Byzantine attacks, all algorithms have similar performance. When there are Byzantine attacks, using a Zeno validator, ZenoPS converged almost as well as using averaging without any attack. However, the trimmed mean had relatively bad results under sign-flipping attacks, and both the trimmed mean and Phocas failed under the random attacks rescaled to 1. In some cases, where the attacks were relatively weak, such as the sign-flipping attacks rescaled by 6 and the random attacks rescaled by 8, Phocas also had good convergence, which is an option that is cheaper than Zeno. However, in general, using Zeno as the validator provided the best performance under all kinds of attacks. Figure 3: ZenoPS in the synchronous mode with various attacks and validators. In each experiment, we launched 1 server, 1 validator, and 16 workers. In each iteration, each worker had a 50% probability of being active. (**a**) No attacks. (**b**) Sign-flipping attack rescaled by 6. (**c**) Random attack rescaled to 1. (**d**) Random attack rescaled by 8. Furthermore, we show that ZenoPS is capable of using multiple servers and validators. Different servers are responsible for different blocks of the model parameters. The updates sent to the servers will be evenly hashed to the validators, so that different validators are assigned a similar workload. Figure 4 shows that, by using multiple validators, ZenoPS has a similar performance compared to the case of using a single validator. Note that the multiple validators send updates to the server independently and asynchronously. Such asynchronicity causes additional noise compared to the case where there is only \(1\) Zeno validator. Furthermore, without changing the hyperparameter \(b=3\) for the trimmed mean and Phocas validators, adding more validators results in more potentially harmful updates being dropped. Hence, the trimmed mean and Phocas validators have slightly better results in Figure 4 than those in Figure 3c. The result of ZenoPS in the asynchronous mode is shown in Figure 5. When there are no Byzantine attacks, using Zeno as the validator provides good convergence similar to FedAsync. Adding Byzantine attacks makes FedAsync performs much worse. Using Zeno as the validator, ZenoPS converges as well as the cases where there are no attacks. We also show the results where two servers and two validators are used in Figure 6.  
Figure 4: ZenoPS in the synchronous mode with multiple validators. In each experiment, we launched 2 servers, 2 validators, and 16 workers. In each iteration, all workers were active. (**a**) No attacks. (**b**) Random attack rescaled to 1. Figure 5: ZenoPS in the asynchronous mode with various attacks and validators. In each experiment, we launched 1 server, 1 validator, and 8 workers. In each communication round, each worker had a 50% probability of dropping the entire communication. (**a**) No attacks. (**b**) Sign-flipping attack rescaled by 2. (**c**) Random attack rescaled to 1. (**d**) Random attack rescaled by 8. As shown in Figure 7, we tested the sensitivity to the hyperparameter \(\rho\), with fixed \(\epsilon=0\). We varied \(\rho\) in \(\{0.1,-0.01,-0.001,0,0.001,0.01,0.1\}\). In most cases, the Zeno validator was insensitive to \(\rho\) and converged to the same value. The only exception was the case of \(\rho=0.1\), where the threshold was too large and made the convergence extremely slow. Thus, while a grid search needs to be performed for hyperparameter tuning in practice, for \(\rho\) we recommend using a negative value close to 0. Although the convergence analysis shows that a larger \(\rho\) yields smaller error bounds, it will also decrease the number of approved updates. As shown in Figure 7, when \(\rho=0.1\), only 16% of the updates passed the validation. By decreasing the threshold, the approval rate approached the ideal value of 80% (20% of the updates are Byzantine).