SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: ATP with different model architectures, accuracy (mean  $\pm$  s.d. %) on target clients
Method	Shallow CNN on CIFAR-10	ResNet-50 on CIFAR-100
Feature shift	Label shift	Hybrid shift	Avg. Rank	Feature shift	Label shift	Hybrid shift	Avg. Rank
No adaptation	64.39 ± 0.18	69.33 ± 0.37	61.99 ± 0.47	7.3	45.31 ± 0.30	51.63 ± 0.15	40.01 ± 0.17	7.3
BN-Adapt	66.46 ± 0.22	54.99 ± 0.38	50.40 ± 0.43	7.0	47.75 ± 0.29	34.85 ± 0.26	30.31 ± 0.09	7.3
SHOT	65.60 ± 0.18	49.98 ± 0.29	45.95 ± 0.47	9.0	45.42 ± 0.30	31.06 ± 0.32	27.44 ± 0.14	9.3
Tent	65.61 ± 0.24	50.12 ± 0.25	45.91 ± 0.49	8.7	45.91 ± 0.46	31.34 ± 0.11	27.93 ± 0.31	8.3
T3A	64.31 ± 0.27	66.96 ± 0.43	59.65 ± 0.58	8.3	45.31 ± 0.30	51.42 ± 0.15	39.89 ± 0.20	7.7
MEMO	65.89 ± 0.31	71.95 ± 0.25	64.17 ± 0.47	5.3	48.42 ± 0.14	55.19 ± 0.28	42.53 ± 0.20	3.7
EM	61.74 ± 0.25	76.28 ± 0.29	67.54 ± 0.41	5.0	43.00 ± 0.31	59.34 ± 0.15	44.82 ± 0.27	5.0
BBSE	56.92 ± 0.53	75.99 ± 0.44	66.64 ± 0.53	6.3	37.26 ± 0.64	56.97 ± 0.20	40.09 ± 0.51	7.0
Surgical	64.45 ± 0.12	73.75 ± 0.42	65.67 ± 0.44	5.7	45.18 ± 0.38	54.83 ± 0.26	42.50 ± 0.33	6.7
ATP-batch	66.90 ± 0.05	76.23 ± 0.32	68.88 ± 0.35	2.3	48.35 ± 0.45	58.06 ± 0.53	46.82 ± 0.32	2.7
ATP-online	67.13 ± 0.17	78.56 ± 0.32	71.52 ± 0.51	1.0	49.08 ± 0.26	61.86 ± 0.25	49.51 ± 0.23	1.0
From Table 5, we observe that under the new model architecture (and the new dataset), the performance of ATP is highly similar to the results of the ResNet-18 + CIFAR10 experiment in Table 1 in Subsection 6.1. ATP, in all three scenarios, can handle various types of distribution shifts and surpass baseline methods. This suggests that ATP is compatible with multiple model architectures.
C.3 Robustness to global model
In this subsection, we design experiments to answer the following question: is ATP robust to the choice of global model? Specifically, we have three sub-questions:
- Is ATP robust to the parameter of global model? (C.3.1)
- Is ATP robust to the algorithm to train global model? (C.3.2)
C.3.1 Robustness to the parameter of global model (online updated global model)
In the main text, we primarily focused on the scenario where the global model remains fixed. However, in practical FL systems, the global model may also undergo continuous online updates. Therefore, after obtaining the adaptation rates through ATP training, the global model might have been further updated for several rounds. This raises a question: Are the "outdated" adaptation rates still effective after several rounds of updates to the global model?
Table 6: Accuracy  $(\%)$ , ATP can learn adaptation rates that generalize to global models with different numbers of communication rounds under hybrid shift on CIFAR-10
Method	200 + 0 Rounds	+10 Rounds	+20 Rounds	+50 Rounds	+100 Rounds
No adaptation	63.68 ± 0.24	63.88 ± 0.20	64.03 ± 0.13	64.30 ± 0.08	64.56 ± 0.11
ATP-batch	73.05 ± 0.35	73.20 ± 0.40	73.25 ± 0.37	73.47 ± 0.48	73.61 ± 0.28
ATP-online	75.37 ± 0.22	75.61 ± 0.23	75.69 ± 0.20	75.80 ± 0.15	75.83 ± 0.28
We design experiment to apply the "outdated" adaptation rates to the global model that has undergone additional updates for several rounds, to see if they can still improve the test-time accuracy of the global model. Specifically, we optimize the adaptation rates  $\alpha$  with  $\boldsymbol{w}_G^T$  where  $T = 200$ , but test the adaptation rates with  $\boldsymbol{w}_G^{T + \Delta T}$  with  $\Delta T = 10, 20, 50, 100$  rounds. We use the same setting of hybrid shift on CIFAR-10 experiments. As shown in Table 6, while further optimizing the global model can marginally improve the accuracy, both ATP-batch and ATP-online can effectively enhance the test-time accuracy through personalization, even when  $\alpha$  is trained using an outdated version of the global model.
C.3.2 Robustness to the algorithm to train global model
In the main text, we used FedAvg [31] to train the global model. However, in real-world FL systems, other FL algorithms may be employed for training the global model, considering stability optimization or fairness. Therefore, we aim to investigate whether ATP can also be applied to other commonly used FL algorithms.
Table 7: Accuracy  $(\%)$ , ATP enhances different global models under hybrid shift on CIFAR-10
Method	FedAvg	FedProx (μ = 0.01)	q-FFL (q = 1)
No adaptation	63.68 ± 0.24	63.77 ± 0.25	63.87 ± 0.23
ATP-batch	73.05 ± 0.35	72.95 ± 0.33	73.15 ± 0.21
ATP-online	75.37 ± 0.22	75.51 ± 0.19	75.79 ± 0.15
In particular, we use FedProx [23], an FL algorithm designed to handle heterogeneous setting, and  $q$ -FFL [24], an FL algorithm enhancing performance fairness among participating clients. For all global model, we use the same setting of hybrid shift on CIFAR-10 experiments. As shown in Table 7, both ATP-batch and ATP-online can consistently improve the test-time accuracy across different FL algorithms to train global models.
C.4 Convergence and generalization
In Section 5, Appendix B.2 and B.3, we theoretically show that ATP has good convergence and generalization guarantees. In this section, we visualize the training and testing loss curves to verify the fast convergence and superior generalization of ATP under different cohort size  $C$ . The results are shown in Figure 8.
(a)  $C = 240$
(b)  $C = 120$
(d)  $C = 30$
Figure 8: Loss curves of ATP under different cohort size  $C$
(c)  $C = 60$
(e)  $C = 15$
Convergence Under full participation ( $C = 240$ ), both the training and testing loss converge stably and fast, indicating the reliable convergence of ATP. With partial participation, as the cohort size decreases ( $C = 120, 60, 30, 15$ ), the training loss curve exhibits greater fluctuations, primarily due to sampling different subsets of clients in each communication round. However, the testing loss curve still converge stably with similar speed, indicating that ATP is robust to partial participation.
Generalization Under full participation ( $C = 240$ ), the training and testing loss curves decrease synchronously without any overfitting. This implies that our algorithm exhibits excellent generalization. Similar observations can be made for partial participation ( $C = 120, 60, 30, 15$ ). Additionally, it is worth noting that the test loss is lower than the train loss, which may seem counterintuitive. This is primarily due to the use of different corruptions between the testing and source clients. The accuracy of clients varies significantly under different corruptions, as evidenced by the fluctuations in the training curve when  $C = 15$ . However, we can still analyze the generalization performance by comparing the trends of the two curves.
C.5 Toy example for negative adaptation rate (RQ2)
In Section 6.2, we notice that ATP learns negative adaptation rates for running means and variance under label shift. In this subsection, we use a toy example to show why negative adaptation rate can improve the model performance under label distribution shift.
We consider a binary classification problem with input  $x \in \mathbb{R}$  and binary output  $y \in \{-1, +1\}$ , where  $-1$  is the negative class and  $+1$  is the positive class. Let the feature for negative samples  $(x|y = -1) \sim \mathcal{N}(-1, 0.8^2)$  and for positive samples  $(x|y = +1) \sim \mathcal{N}(+1, 0.8^2)$ . Let the label distribution  $\operatorname*{Pr}(y = 1) = \frac{1}{2}$  for training set, and  $\operatorname*{Pr}(y = 1) = \frac{5}{6}$  for testing set. Therefore, for the training distribution, we have
$$
\mathbb {E} x = \Pr (y = - 1) \mathbb {E} (x | y = - 1) + \Pr (y = + 1) \mathbb {E} (x | y = + 1) = 0
$$
$$
V a r (x) = \mathbb {E} [ V a r (x | y) ] + V a r (\mathbb {E} [ x | y ]) = 1. 6 4
$$
We consider a simple network with only one BN layer, with both normalization and affine transformation (as a linear classifier). There are four modules, each is a scalar: running mean  $\mu$ , running variance  $\sigma^2$ , weight  $\gamma$ , bias  $\beta$ .
(a) Train, Acc=0.89
(d)  $\alpha = 0$ $\mathrm{Acc} = 0.89$
Figure 9: Adapting batch norm running statistics under label shift.
(b)  $\alpha = 1$ $\mathrm{Acc} = 0.73$
(c)  $\alpha = 0.5$ , Acc=0.83
(e)  $\alpha = -0.5$ , Acc=0.92
Training During training, given enough training data, we have  $\mu_{train} = \mathbb{E}x = 0$  and  $\sigma_{train}^2 = 1.64$ . Figure 9a shows the histogram of  $z = \frac{x - \mu}{\sigma}$ , i.e., the intermediate feature after normalization before the transformation. By comparing the histograms of  $z$  of two classes, we notice that the optimal decision boundary is  $z = 0$ , which indicate that  $\beta_{train} = 0$  and  $\gamma_{train} > 0$ . We store the corresponding  $\mu_{train}, \sigma_{train}^2, \gamma_{train}, \beta_{train}$ , and only update running statistics  $\mu_{train}, \sigma_{train}^2$  during testing.
Testing without updating running statistics  $(\alpha = 0)$  Figure 9d shows the testing result when we do not update the running statistics, i.e.,  $\alpha = 0$ . Since two conditional feature distributions are symmetric, the accuracy will not change.
Testing with  $\alpha > 0$  Positive adaptation rates align the intermediate feature distribution. When we use  $\alpha = 1$ , the distribution of  $z$  will be centralized. As shown in Figure 9b, such alignment greatly reduces the accuracy. Similar result is also observed with any positive  $\alpha$ , e.g.,  $\alpha = 0.5$  in Figure 9c.
Testing with  $\alpha < 0$  While  $\alpha = 0$  has stable accuracy under label shift, by comparing the histograms of  $z$  of two classes in Figure 9d, we notice that  $z = 0$  is not the optimal decision boundary anymore, because there are less negative samples than positive samples. By using negative adaptation rate  $\alpha < 0$ , the normalization layer can further "disalign" the intermediate feature, which can further improve the accuracy, as shown in Figure 9e.