SlowGuess/NeurIPS_2023_OCR huggingface dataset:
between the average confidence and the percentage of correct samples across all groups. ECE values range from 0 to 1, where 0 signifies perfect calibration and 1 indicates complete miscalibration. MCE, on the other hand, is akin to ECE but focuses on the largest gap for any group rather than the weighted average.
Table 11 and Table 12 summarize the ECE and MCE results for varying non-IID degrees in CIFAR-100 and CIFAR-10 datasets, with Dirichlet parameters represented as  $\dot{\alpha} = [5,0.5,0.1]$ . Typically, ECE and MCE values tend to increase as clients possess more non-IID data, which is a result of overfitting each client's local data. In the CIFAR-10 dataset, all methods that incorporate MetaVD exhibit a decline in ECE and MCE values as  $\dot{\alpha}$  decreases, in contrast to standard FL or PFL methods that show an increase in these values. MetaVD effectively mitigates overfitting to local data and enhances calibration by leveraging the heterogeneity present in the training data. Consequently, methods employing MetaVD consistently achieve the lowest ECE and MCE values in almost all scenarios.
Examining a reliability diagram can provide a visual comparison of uncertainty calibration results, as it illustrates the relationship between prediction probabilities and true labels. Ideally, if a model predicts a specific class with a certain probability, the actual label should correspond to that confidence level, placing the point on the reliability diagram's diagonal line. However, if the prediction probability and the true label do not align, the point would be below or above the diagonal line. In this context, ECE represents the average distance of each point from the diagonal line, while MCE indicates the maximum distance of any point from the diagonal line, offering a comprehensive understanding of the model's calibration performance.
Figure 4 and Figure 5 present the reliability diagrams for CIFAR-100 and CIFAR-10 with  $\dot{\alpha} = 0.5$  respectively. These figures allow us to compare the calibration performance of FedAvg, Reptile, MAML, and PerFedAvg, both with and without the integration of MetaVD. Remarkably, in most instances, employing MetaVD leads to improved calibration, drawing the reliability diagrams closer to the identity function. This supports our findings that MetaVD significantly improves the uncertainty calibration of the models as well.
J Additional Results on Multi-domain Datasets
Hetrogenity
Method	CelebA + CIFAR-100 dataset
α = 5.0	α = 0.5	α = 0.1
Test (%)	OOD (%)	Δ	Test (%)	OOD (%)	Δ	Test (%)	OOD (%)	Δ
FedAvg [1]	43.42	44.45	+1.03	43.65	43.45	-0.20	40.46	41.00	+0.54
Reptile [22]	50.40	49.07	-1.33	48.92	48.93	+0.01	43.41	42.16	-1.25
MAML [21]	49.97	48.40	-1.57	47.39	48.51	+1.12	44.80	44.79	-0.01
PerFedAvg (HF-MAML) [23]	50.61	49.61	-1.00	49.21	50.91	+1.70	44.20	44.13	-0.07
FedAvg+MetaVD (ours)	48.08	48.88	+0.80	48.23	48.62	+0.39	42.98	45.19	+2.21
Reptile+MetaVD (ours)	53.04	53.97	+0.93	52.26	54.75	+2.49	45.83	47.05	+1.22
MAML+MetaVD (ours)	52.82	53.47	+0.65	51.34	52.82	+1.48	46.83	48.33	+1.50
PerFedAvg+MetaVD (ours)	53.14	54.26	+1.12	51.18	53.06	+1.88	46.65	47.11	+0.46
Table 13: Classification accuracies with different (non-IID) heterogeneity degrees of  $\dot{\alpha} = [5.0, 0.5, 0.1]$  in multi-domain datasets (a); CelebA + CIFAR-100.
Hetrogenity Method	CIFAR-100 + FEMNIST dataset
\( \dot{\alpha} = 5.0 \)	\( \dot{\alpha} = 0.5 \)	\( \dot{\alpha} = 0.1 \)
Test (%)	OOD (%)	\( \Delta \)	Test (%)	OOD (%)	\( \Delta \)	Test (%)	OOD (%)	\( \Delta \)
FedAvg [1]	63.90	53.57	-10.33	64.02	52.88	-11.14	62.13	49.65	-12.48
Reptile [22]	66.47	57.60	-8.87	66.13	56.22	-9.91	64.10	55.81	-8.29
MAML [21]	66.86	53.25	-13.61	66.56	55.72	-10.84	64.56	56.21	-8.35
PerFedAvg (HF-MAML) [23]	67.47	57.15	-10.33	66.57	55.24	-11.33	65.16	56.70	-8.46
FedAvg+MetaVD (ours)	65.91	57.59	-8.32	65.58	56.85	-8.73	63.26	52.78	-10.48
Reptile+MetaVD (ours)	68.61	60.95	-7.66	68.35	59.07	-9.28	65.29	58.11	-7.18
MAML+MetaVD (ours)	68.41	58.86	-9.55	68.24	61.21	-7.03	65.46	58.07	-7.39
PerFedAvg+MetaVD (ours)	68.27	58.72	-9.55	67.93	61.32	-6.61	66.81	59.13	-7.68
Table 14: Classification accuracies with different (non-IID) heterogeneity degrees of  $\dot{\alpha} = [5.0,0.5,0.1]$  in multi-domain datasets (b); CIFAR-100 + FEMNIST.
Unlike existing federated learning algorithms that usually assume a single-domain approach where only one dataset is used in the experiment. In this section, we further evaluate the performance of our method on real-world non-IID FL experiments by introducing multi-domain datasets in which we assume each client can have data
Hetrogenity Method	CelebA + CIFAR-100 + FEMNIST dataset
\( \dot{\alpha} = 5.0 \)	\( \dot{\alpha} = 0.5 \)	\( \dot{\alpha} = 0.1 \)
Test (%)	OOD (%)	\( \Delta \)	Test (%)	OOD (%)	\( \Delta \)	Test (%)	OOD (%)	\( \Delta \)
FedAvg [1]	63.57	53.10	-10.47	63.73	55.55	-8.18	61.86	52.98	-8.88
Reptile [22]	65.93	57.98	-7.95	66.98	57.12	-9.86	63.89	52.33	-11.56
MAML [21]	67.70	59.06	-8.64	67.08	58.15	-8.93	65.67	54.41	-11.26
PerFedAvg (HF-MAML) [23]	67.39	59.17	-8.22	67.20	57.03	-10.17	65.08	54.54	-10.54
FedAvg+MetaVD (ours)	63.62	55.87	-7.75	65.93	58.58	-7.35	64.59	55.61	-8.98
Reptile+MetaVD (ours)	67.63	60.95	-6.68	68.78	61.59	-7.19	65.18	56.46	-8.72
MAML+MetaVD (ours)	69.42	61.52	-7.90	68.81	61.60	-7.21	66.56	56.54	-10.02
PerFedAvg+MetaVD (ours)	67.83	60.74	-7.09	68.05	61.17	-6.88	65.11	55.64	-9.47
Table 15: Classification accuracies with different (non-IID) heterogeneity degrees of  $\dot{\alpha} = [5.0, 0.5, 0.1]$  in multi-domain datasets (d); CelebA + CIFAR-100 + FEMNIST.
from different domains. Multi-domain learning [87-89] aims to leverage all available training data across different domains to enhance the performance of the model. However, directly utilizing data from different domains typically results in a suboptimal global model. In the context of federated learning, it becomes even more critical to apply multi-domain learning effectively. We use three different FL datasets to construct the multi-domain task distributions: CelebA, FEMNIST, and CIFAR-100. The non-IID heterogeneous environment has been also assumed in the field of multi-domain learning. We utilize the Dirichlet sampling technique  $(\dot{\alpha} = [5.0, 0.5, 0.1])$  to sample each client's local CIFAR-100 data.
Table 13, 14, and 15 illustrate the classification accuracies for Test and OOD clients with varying degrees of heterogeneity. Employing MetaVD leads to significantly improved prediction accuracy, with larger improvements in OOD accuracy compared to the improvement in Test accuracy, across all multi-domain settings and degrees of heterogeneity. The results indicate that MetaVD can improve the versatility and performance of FL algorithms when applied to a broad range of multi-domain datasets.
K Additional Results on Model Compression
Heterogeneity
Method	CIFAR-10 dataset	α = 0.1
α = 5.0	α = 0.5	Test (%) OOD (%) Sparsity (%)	α = 0.1
Test (%)	OOD (%)	Sparsity (%)	Test (%)	OOD (%)	Sparsity (%)
FedAvg+SNIP	81.03	80.46	70.0	78.53	79.30	70.0	71.67	73.13	70.0
Reptile+SNIP	83.21	82.10	70.0	81.43	81.72	70.0	74.56	69.57	70.0
MAML+SNIP	83.35	81.59	70.0	81.47	81.84	70.0	72.51	69.27	70.0
PerFedAvg+SNIP	83.33	83.49	70.0	80.97	82.47	70.0	76.27	75.90	70.0
Reptile+MetaVD	84.70	84.28	0	83.20	83.40	0	76.51	82.07	0
MAML+MetaVD	83.93	84.95	0	81.32	81.81	0	77.27	79.05	0
PerFedAvg+MetaVD	83.88	83.96	0	81.06	81.47	0	76.06	81.77	0
Reptile+MetaVD+DP	84.19	84.46	77.12	82.67	82.79	79.25	75.85	80.52	76.18
MAML+MetaVD+DP	84.37	84.95	76.76	82.65	83.64	78.85	77.77	80.74	72.47
PerFedAvg+MetaVD+DP	84.04	83.69	78.55	82.64	83.52	79.32	77.10	83.24	72.87
Table 16: Results of study on model compression with different (non-IID) heterogeneity strengths of  $\dot{\alpha} = 5.0$ ,  $\dot{\alpha} = 0.5$ , and  $\dot{\alpha} = 0.1$  in the CIFAR-10 dataset. MetaVD+DP does not communicate the model parameters when the dropout rate is larger than 0.9.
Federated Learning optimization involves frequent communication of model parameters between devices and the central server, which can be slow and may raise privacy concerns. Therefore, it is crucial to minimize communication costs by reducing both the size of exchanged model parameters and the number of communication rounds. MetaVD also has the benefit of compressing the model parameters needed for each client device. To explore the compression capabilities of MetaVD, we conducted experiments on CIFAR-10 and CIFAR-100 datasets, comparing our method to the baseline Pruning algorithm SNIP [101]. For the implementation of the SNIP algorithm in the context of Federated Learning, we referred to the work of Jiang et al. (2022) [102]; we let SNIP prune the original model to the target sparsity right after the first round.
Table 16 and Table 17 demonstrate the results of Test $(\%)$ , OOD $(\%)$ , and Sparsity $(\%)$  percentage of the models with or without MetaVD+DP. Sparsity represents the ratio of zero-valued model parameters in the personalized layer. The DP refers to the process of dropping communication of weights in the FL algorithm, where weights with a dropout rate greater than 0.9 are dropped after 150 rounds. We have set these 150 rounds to allow MetaVD
Heterogeneity
Method	CIFAR-100 dataset
\(\dot{\alpha}=5.0\)	\(\dot{\alpha}=0.5\)	\(\dot{\alpha}=0.1\)
Test (%)	OOD (%)	Sparsity (%)	Test (%)	OOD (%)	Sparsity (%)	Test (%)	OOD (%)	Sparsity (%)
FedAvg+SNIP	43.76	44.11	70.0	42.53	41.54	70.0	37.80	40.34	70.0
Reptile+SNIP	49.54	49.61	70.0	48.38	48.62	70.0	42.03	43.68	70.0
MAML+SNIP	51.06	49.56	70.0	48.16	46.99	70.0	42.03	42.49	70.0
PerFedAvg+SNIP	50.66	48.85	70.0	48.71	48.46	70.0	43.23	43.09	70.0
Reptile+MetaVD	53.71	54.50	0	52.06	51.50	0	44.56	44.62	0
MAML+MetaVD	52.40	51.78	0	50.21	49.75	0	43.84	48.17	0
PerFedAvg+MetaVD	51.67	51.70	0	50.02	48.70	0	43.31	46.19	0
Reptile+MetaVD+DP	52.56	55.03	65.83	50.94	50.85	57.26	45.13	45.19	75.72
MAML+MetaVD+DP	52.31	52.29	74.61	50.30	50.88	77.88	45.75	45.78	72.26
PerFedAvg+MetaVD+DP	51.68	52.56	70.08	51.64	51.47	68.88	45.16	46.45	55.18
Table 17: Results of study on model compression with different (non-IID) heterogeneity strengths of  $\dot{\alpha} = 5.0$ ,  $\dot{\alpha} = 0.5$ , and  $\dot{\alpha} = 0.1$  in the CIFAR-100 dataset. MetaVD+DP does not communicate the model parameters when the dropout rate is larger than 0.9.
to adequately learn the client-specific dropout rates. The target sparsity of the SNIP algorithm was set to 0.7. Note that, in our MetaVD+DP setting, even though we dropped weights with a dropout rate greater than 0.9, the actual sparsity values are observed to be lower and tend to be around  $70\%$ , as indicated in the tables.
Overall, most models removed over  $70\%$  of their parameters in the personalized layer without losing much performance. In CIFAR-10, as heterogeneity increased, the performance gap between SNIP and MetaVD+DP tended to increase. For instance, in CIFAR-10 with  $\dot{\alpha} = 5.0$ , the OOD performance of SNIP model and MAML+MetaVD+DP model were 81.59 and 84.95, respectively, whereas in CIFAR-10 with  $\dot{\alpha} = 0.1$ , they were 69.27 and 80.74 respectively, demonstrating an increased difference in performance. This was similar in the case of Reptile+MetaVD+DP and PerFedAvg+MetaVD+DP. In CIFAR-100 dataset, the performance gap, according to the increase in heterogeneity, was not noticeable. Surprisingly, dropping model parameters could lead to performance improvements in some cases. For example, in CIFAR-100 with  $\dot{\alpha} = 0.5$ , the PerFedAvg+MetaVD+DP model's OOD performance increased from 48.70 to 51.47. This experiment demonstrates that the MetaVD method can decrease the communication cost in Federated Learning (FL) by compressing model parameters. This emphasizes the efficiency of MetaVD, this approach can be applied to numerous FL algorithms without significantly increasing communication costs. It also boosts the generalization performance for new, unseen clients. These aspects make our approach a significant addition to the field of Federated Learning.
(a) FedAvg
(b) FedAvg + MetaVD
Accuracy
(c) Reptile
(d) Reptile + MetaVD
(e) MAML
(f) MAML + MetaVD
(g) PerFedAvg
Confidence
Figure 4: Reliability diagrams for (a) FedAvg, (b) FedAvg+MetaVD, (c) Reptile, (d) Reptile+MetaVD, (e) MAML, (f) MAML+MetaVD, (g) PerFedAvg and (h) PerFedAvg+MetaVD in CIFAR-100 ( $\dot{\alpha} = 0.5$ ).
(h) PerFedAvg + MetaVD
(g) PerFedAvg
Confidence
Figure 5: Reliability diagrams for (a) FedAvg, (b) FedAvg+MetaVD, (c) Reptile, (d) Reptile+MetaVD, (e) MAML, (f) MAML+MetaVD, (g) PerFedAvg and (h) PerFedAvg+MetaVD in CIFAR-10 ( $\dot{\alpha} = 0.5$ ).