SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In a similar context of reducing the total running time overhead, some asynchronizd FL (AFL) methods have been proposed. We shall clarify that our wait-free local training protocol is vastly different from existing AFL methods, from the aspects of both the motivation and the communication-computation design. Specifically, AFL seeks to tackle the challenge of slow client in a heterogeneous FL system by reducing idle time, while in contrast, our method adopts a joint communication-computation design, thereby eliminating idle time. In particular, the usual idle waiting time has been replaced by training time for the personalized model in our proposed method. Since our method fully utilizes the idle waiting time at the local side with personalized model training, we did not consider incorporating the conventional AFL set-up into our framework.
To better demonstrate the outperformance of our wait-free (WF) local training protocol, we evaluated the performance of two state-of-the-art AFL methods (FedAsync [57] and FedBuff [58]) combined with the compute-and-wait PFL+ system using Ditto [15] (the PFL system that trains generic and personalized models simultaneously); see Tab. 9. The speedup factors are computed with respect to the performance of "Vanilla Ditto", where a larger speedup factor signifies better efficiency. The results demonstrate the outperformance of our proposed WF protocol in reducing the total running time overhead.
Methods	3 epochs	5 epochs
40%	80%	40%	80%
Speedup
Vanilla Ditto	1×	1×	1×	1×
Ditto w/ Our WF	1.97×	1.38×	2.87×	1.93×
Ditto w/ FedAsync	1.36×	1.15×	1.29×	1.22×
Ditto w/ FedBuff	1.40×	1.26×	1.23×	1.40×
Table 9: Communication cost comparison of various methods for personalized model accuracies on CIFAR-10 to reach target accuracy (40%/80%) with non-IID setting  $\alpha = 0.1$ . The speedup factors are with respect to the performance of the Vanilla Ditto with the compute-and-wait protocol.
F Discussion on the applicability to the partial client participation scheme
In this work, we consider a full client participation scheme. However, in general, we may want the option that an edge device/client could be not selected to participate in training for numerous consecutive communication rounds, especially in the case of an FL system with a large number of clients. In this subsection, we discuss how our wait-free local training protocol should be extended/modified.
For an unselected client, the client should not be continuously training its personalized model using the "stale" GM model weights, which could result in model overfitting on limited data, and model drift/divergence.
To address this issue, there are two straightforward solutions:
- The first solution is to adopt a broadcasting scheme with regular latest GM broadcasts by the server. Such a scheme would ensure that every client receives the latest GM weights, thereby alleviating the negative impact of stale GM weights for unselected clients, and avoiding potential client drift. In real-world network connections, the bandwidth of the downlink connection from the server to the clients is typically larger than the corresponding bandwidth of the uplink connection. Hence, such design would be practical.
- The second solution is to set an upper limit on the number of local training steps. The basic underlying idea is to terminate the local PM training after a certain number of local training steps. This could be an alternative option to address the issue of limited communication for cross-device FL scenarios. It should be noted that such design would still reduce the overall total training time, since the PM training is still performed within the idle waiting time in conventional FL.
We give further experiment results (see below) for partial participation with two typical participation rates ( $c = 0.1$  and  $0.2$ ) on non-IID CIFAR-10 with  $\alpha = 1$  in Tab. 10.
Methods	c=0.1	c=0.2
GM	PM	GM	PM
FedAvg	83.16	78.25	83.84	79.03
FedProx	82.97	78.23	83.75	79.44
FedRoD	84.22	89.31	84.90	89.68
Ours	84.35	89.56	85.11	90.37
Table 10: Sensitivity analysis for hyper-parameter  ${\lambda }_{\mathrm{G}}$  on non-IID CIFAR-10.
These results show that our method is applicable for partial participation, and even outperforms our baselines.