SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To validate the MetaVD approach, we conducted extensive experiments in various scenarios following the FL benchmark research [18], including different degrees of non-IID and client participation rates.
2We adapted this rule from [33]. In general, the mode of the product of  $M$  Gaussians,  $\prod_{m = 1}^{M}\mathcal{N}(\mu^m,\sigma^m)$  simplifies to  $\mu^{\mathrm{agg}} = \sum_{m = 1}^{M}r^{m}\mu^{m}$  where  $r^m = ((\sigma^m)^2)^{-1} / \sum_{m = 1}^{M}((\sigma^m)^2)^{-1}$  [67]. Interestingly, Eq.(5) is equivalent to maximizing the logarithm of the product of the weighted posteriors [68]. In the heterogeneous data, the product rule can achieve a smaller aggregation error than the mixture of Gaussian posteriors.
Hetrogenity
Method	CIFAR-100 dataset	CIFAR-10 dataset
α = 5.0	α = 0.5	α = 0.1
Test (%)	OOD (%)	Δ	Test (%)	OOD (%)	Δ	Test (%)	OOD (%)	Δ
FedAvg [1]	42.35	43.08	+0.73	41.92	41.96	+0.04	71.65	71.57	-0.08
FedAvg+FT [18]	41.49	42.45	+0.96	40.99	39.83	-1.16	69.62	68.38	-1.24
FedProx [73]	42.23	44.11	+1.88	42.03	40.51	-1.52	72.27	73.75	+1.48
FedBE [31]	45.17	45.43	+0.26	44.29	44.23	-0.06	70.23	69.19	-1.04
pFedGP [34]	42.69	43.07	+0.38	42.44	42.53	+0.09	71.94	76.83	+4.89
Reptile [22]	47.87	47.73	-0.14	46.13	45.94	-0.19	73.93	76.36	+2.43
MAML [21]	48.30	49.14	+0.84	46.33	46.65	+0.32	76.06	74.89	-1.17
PerFedAvg (HF-MAML) [23]	48.19	47.35	-0.84	46.22	46.36	+0.14	75.42	79.56	+4.14
FedAvg+MetaVD (ours)	47.82	50.26	+2.44	47.54	47.55	+0.01	76.87	76.25	-0.62
Reptile+MetaVD (ours)	53.71	54.50	+0.79	52.06	51.50	-0.56	76.51	82.07	+5.56
MAML+MetaVD (ours)	52.40	51.78	-0.62	50.21	49.75	-0.46	77.27	79.05	+1.78
PerFedAvg+MetaVD (ours)	51.67	51.70	+0.03	50.02	48.70	-1.32	76.06	81.77	+5.71
Table 1: Classification accuracies with different (non-IID) heterogeneity degrees of  $\dot{\alpha} = [5.0, 0.5]$  in CIFAR-100 and  $\dot{\alpha} = 0.1$  in CIFAR-10. The higher score, the better.
We used multiple FL datasets [72], including CIFAR-10, CIFAR-100, FEMINIST, and CelebA. To evaluate the effectiveness of the hypernetwork, we also performed an ablation study comparing MetaVD to regular VD. Additionally, we assessed the uncertainty calibration and model compression ability of MetaVD. Finally, we test MetaVD with multi-domain datasets.
Baselines. We compare our method with standard FL methods such as FedAvg [1] and FedProx [73], meta-learning PFL algorithms like Reptile [22], MAML [21], and PerFedAvg [23], and Bayesian FL methods including FedBE [31] and pFedGP [34]. To ensure consistency, we employ the widely-used CNN model for FL [3, 74, 25] across all baselines. Additionally, "fine-tuning" (FT) refers to performing few-shot adaptation steps with FedAvg before evaluation. An overview of baselines can be found in Appendix D.
Implementation. For reproducibility, we run experiments in a containerized environment that simulates FL communication with clients only on a server. We test  $T = 1000$  of total FL rounds, following the conventions in [18]. One baseline model can be run on a single GPU. All experiments are run on a cluster of 32 NVIDIA GTX 1080 GPUs. MetaVD's hypernetwork consists of an embedding layer of dimension  $(1 + M / 4)$ , followed by three fully connected NNs with a Reaky ReLU activation and an exponential activation for the dropout logit output. The predicted dropout variable is then applied to the global weight of other baselines. In our study, we apply MetaVD to only one fully connected layer before the output layer [75-78], which leads to significant performance improvements in all experiments. See Appendix E for implementation details. Our code is available at https://github.com/insujeon/MetaVD.
4.1 Generalization on Non-IID Settings
Datasets and training. To evaluate the generalization capabilities of the models under non-IID data conditions, we perform tests on both CIFAR-10 and CIFAR-100 datasets with varying degrees of heterogeneity. We follow the similar evaluation protocols of the pFL-Bench [18], using Dirichlet allocation to partition each dataset into 130 clients with different Dirichlet parameters, denoted as  $\dot{\alpha} = [5,0.5,0.1]$ . As shown in Figure 2, the class labels and data size per client are heterogeneous across clients. A smaller  $\dot{\alpha}$  represents a higher degree of heterogeneity. To evaluate
Figure 2: Visualization of client's data distribution in different non-IID degrees  $(\dot{\alpha} = [0.5,0.1])$
the test accuracy and generalization performance of the baselines on new clients, we randomly select 30 out of 130 clients as out-of-distribution (OOD) clients, which are not involved in the training phase. See Appendix G for more details.
Results. Table 1 shows the weighted average classification accuracy for participating (Test) and non-participating (OOD) clients on CIFAR-10 and CIFAR-100 datasets with varying non-IID degrees. The generalization gap, denoted by  $\Delta$ , represents the difference between OOD and Test accuracy. As shown in Table 1, PFL methods such as Reptile, MAML, and PerFedAvg generally outperform non-PFL methods such as FedAvg and FedProx. While the Bayesian ensemble approach, FedBE, improves on FedAvg, it still lags behind PFL methods in OOD accuracy. Additionally, pFedGP exhibits suboptimal performance in the presence of heterogeneous distribution of data among clients. As  $\dot{\alpha}$  changes from 5.0 to 0.5, the test and OOD accuracy of all models decreases as the degree of non-IID increases. When combined with MetaVD, all baselines show significant performance improvements, regardless of whether they are FL or PFL algorithms (e.g., Reptile enhances from 47.87 to 53.71 adapting MetaVD). These results demonstrate the adaptability and effectiveness of MetaVD in mitigating model overfitting and handling non-IID client data in FL contexts.
Settings. To evaluate the capability of the hypernetwork in MetaVD, we perform an ablation study by comparing MetaVD with naive VD [46] and EnsembleVD [79] approaches on the CIFAR-100 dataset. The naive (global) VD model maintains a global dropout parameter shared by all clients; the dropout parameter is treated as a global model parameter, as in FedAvg. EnsembleVD maintains  $M$  independent dropout parameters for all clients. The client-specific dropout parameter can be stored in each client analogous to the partial FL [80, 81, 5]. In contrast, MetaVD utilizes a hypernetwork to learn the personal dropout rate across all clients. Bayesian posterior aggregation rules are applied in all model based on dropout rates to update the global model parameter [82, 83].
Results. Table 2 outlines the results of the ablation study; MetaVD's hypernetwork-based posterior modeling outperforms all other baselines. The dropout rates in baselines such as VD or EnsembleVD could not fully learn the independent dropout variables well due to restricted client participation. On the other hand, both the hypernetwork and the global parameter converge well in MetaVD. This observation demonstrates that
Method	CIFAR-100 dataset
Test (%)	ODD (%)	Δ
Reptile [22]	47.87	47.73	-0.14
Reptile+VD [46]	50.20	49.28	-0.92
Reptile+EnsembleVD [79]	52.49	52.36	-0.13
Reptile+MetaVD (ours)	53.71	54.50	+0.79
MetaVD's hypernetwork provides a more data-efficient approach to learning client-specific model uncertainty compared to other baselines. Further ablation studies performed on the FEMNIST dataset are shown in Appendix H.
4.3 Uncertainty Calibration
Table 2: MetaVD ablation study in CIFAR-100.
CIFAR-100 dataset
Method	ECE (%)	MCE (%)
FedAvg [1]	0.60	36.79
FedAvg+FT [18]	0.69	45.04
FedProx [73]	0.67	39.69
FedBE [31]	0.50	34.66
Reptile [22]	0.77	50.52
MAML [21]	0.75	46.57
PerFedAvg (HF-MAML) [23]	0.69	45.27
FedAvg+MetaVD (ours)	0.39	25.27
Reptile+MetaVD (ours)	0.57	42.40
MAML+MetaVD (ours)	0.52	37.26
PerFedAvg+MetaVD (ours)	0.43	30.20
Table 3: Uncertainty calibration scores (ECE and MCE) measured on the OOD client in CIFAR-100  $(\dot{\alpha} = 0.1)$ . The lower, the better.
Figure 3: Reliability diagrams for (a) Reptile, (b) MAML, (c) Reptile+MetaVD, and (d) MAML+MetaVD in CIFAR-100.
Settings. Identifying any potential bias in the model's prediction is important to avoid serious consequences, especially when the model is used to make important decisions [84-86]. In the FL
environment, where customers have limited non-IID data, it is even more critical to properly calibrate the prediction model. Therefore, we investigate whether the proposed MetaVD approach can also improve the calibration measures for FL baselines. The Expected Calibration Error (ECE) measures the expected deviation between a model's predicted probability and the actual positive class frequency, while the Maximum Calibration Error (MCE) measures the maximum difference. These calibration metrics are commonly used to evaluate the reliability of probabilistic predictions.
Results. Table 3 summarizes the ECE and MCE values tested with the OOD clients in the CIFAR-100 dataset and shows that the meta-learning based PFL algorithms (e.g., Reptile, MAML, and PerFedAvg) tend to have higher ECE and MCE values than the conventional FL algorithms (e.g., FedAvg, FedProx, and FedBE). This means that PFL baselines achieve high classification accuracy, but their probability predictions are more likely to be biased. This may be a byproduct of the additional optimization-based adaptation steps of meta-learning with limited local client data. On the other hand, our MetaVD approach significantly reduces both ECE and MCE values for all meta-learning-based methods, indicating that MetaVD effectively mitigates overfitting and reduces bias on the OOD clients. Figure 3 shows the reliability diagrams that visualize the model calibration [84-86] in CIFAR-100 ( $\dot{\alpha} = 0.5$ ). They plot the expected sample accuracy as a function of confidence. If the model is perfectly calibrated, then the diagram becomes the identity function. Any deviation from a perfect diagonal represents miscalibration. While Reptile and MAML tend to be overconfident in their predictions, Reptile+MetaVD comes fairly close to the desired diagonal function. Remarkably, in most instances, adapting MetaVD leads to improved model calibration.
4.4 Client Participation
Sparsity
Method	FEMNIST dataset
s = 0.2	s = 0.1	s = 0.05
Test (%)	OOD (%)	Δ	Test (%)	OOD (%)	Δ	Test (%)	OOD (%)	Δ
FedAvg [1]	88.08	85.29	-2.79	88.13	84.70	-3.43	88.06	86.22	-1.84
FedAvg+FT [18]	88.33	86.37	-1.96	87.85	86.95	-0.90	87.66	87.11	-0.55
Reptile [22]	88.55	86.52	-2.03	88.39	87.20	-1.19	87.86	88.22	+0.36
FedAvg+MetaVD (ours)	88.81	85.28	-3.53	88.69	85.71	-2.98	88.66	86.21	-2.45
Reptile+MetaVD (ours)	89.90	89.04	-0.86	89.86	88.63	-1.23	89.43	88.71	-0.72
Table 4: Results of classification accuracies with different participant client rates of  $s = 0.2$ ,  $s = 0.1$ , and  $s = 0.05$  in the FEMNIST dataset. We report the results of participating clients (Test) and non-participating clients (OOD). The higher the better.
Settings. In real-world FL scenarios, such as intermittent connections between clients and servers or limited client device performance, numerous clients may be unable to participate in each FL round. This is important for cross-device FL with a large number of clients or resource-limited clients. In this experiment, we evaluated the performance of the methods under different levels of client participation in each FL round. We experimented with 200 clients in the FEMNIST dataset. For each FL round, we randomly selected 40, 20, and 10 clients to participate during training, with participating client rates  $s$  of 0.2, 0.1, and 0.05, respectively. To measure OOD accuracy, we excluded 40 preselected clients from the selection of 200 clients so that they do not participate in the entire training.
Results. The overall classification results with different participant client rates  $s$  are summarized in table 4. Reptile performs better than FedAvg. The effect of data heterogeneity on performance degradation becomes more severe as more clients participate in training. The decrease in test accuracy as the participating client rate  $s$  becomes smaller is due to having less training data as fewer clients participate in each round. Meanwhile, Reptile+MetaVD outperforms the other baselines. Interestingly, the performance drop for Reptile+MetaVD is not as significant as for Reptile, showing that MetaVD can adapt well to FL scenarios with smaller participant sizes.
4.5 Muti-domain Datasets
Settings. Existing federated learning algorithms usually assume a single-domain approach, where only one dataset is used in the experiment. Multi-domain learning [87-89] aims to utilize all available training data across different domains to improve the performance of the model. In this section, we
Method	2 Domains (a)	2 Domains (b)	2 Domains (c)	3 Domains (d)
Test (%)	OOD (%)	Test (%)	OOD (%)	Test (%)	OOD (%)	Test (%)	OOD (%)
FedAvg [1]	43.65	43.45	64.02	52.88	86.81	81.01	63.73	55.55
Reptile [22]	48.92	48.93	66.13	56.22	87.50	83.05	66.98	57.12
MAML [21]	47.39	48.51	66.56	55.72	88.57	84.52	67.08	58.15
PerFedAvg (HF-MAML) [23]	49.21	50.91	66.57	55.24	87.94	83.75	67.20	57.03
FedAvg+MetaVD (ours)	48.23	48.62	65.58	56.85	87.38	82.67	65.93	58.58
Reptile+MetaVD (ours)	52.26	54.75	68.35	59.07	88.63	85.03	68.78	61.59
MAML+MetaVD (ours)	51.34	52.82	68.24	61.21	88.59	85.02	68.81	61.60
PerFedAvg+MetaVD (ours)	51.18	53.06	67.93	61.32	88.27	85.32	68.05	61.17
further evaluate the performance of PFL algorithms on a multi-domain FL dataset, where we assume that each client can have data from different domains. We use three different FL datasets to construct the multi-domain task distributions: FEMNIST, CIFAR-100, and CelebA. To sample each client's local data, we use the Dirichlet sampling technique  $(\dot{\alpha} = 0.5)$  used in the §4.1.
Results. Table 5 shows the classification accuracies for the Test and OOD clients when using a combination of two or three datasets. The meta-learning based PFL algorithm consistently outperforms FedAvg in terms of classification accuracy. In addition, when MetaVD is applied to either the FL or PFL algorithms, it significantly improves prediction accuracy in all multi-domain settings. Notably, MetaVD shows greater improvements in OOD accuracy compared to the improvement in test accuracy. Overall, these results demonstrate the versatility of MetaVD in improving robustness and generalization even in multi-domain FL datasets.
4.6 Model Compression
Settings. Federated learning optimization requires frequent communication of model parameters between devices and the central server, which can be slow and may raise privacy concerns. Therefore, minimizing communication costs by reducing the size of model parameters is an important issue in FL. To explore the compression capabilities of MetaVD, we performed an additional experiment on the CIFAR-10 dataset. The sign DP indicates that each model parameter is dropped during the FL communication.
We used the thresholding technique to drop the model parameter; Any parameter with a dropout rate greater than 0.8 was dropped during the FL rounds.
Table 5: Classification accuracies with multi-domain datasets. (a) CelebA + CIFAR-100, (b) CIFAR-100 + FEMNIST, (c) CelebA + FEMNIST, (d) CelebA + CIFAR-100 + FEMNIST.
CIFAR-10 dataset (α = 0.5)
Method	Test (%)	OOD (%)	Sparsity(%)
Reptile+MetaVD	83.20	83.40	0
MAML+MetaVD	81.32	81.81	0
PerFedAvg+MetaVD	81.06	81.47	0
Reptile+MetaVD+DP	81.40	80.98	80.06
MAML+MetaVD+DP	81.48	81.73	79.49
PerFedAvg+MetaVD+DP	82.43	82.19	78.20
Table 6: Results of model compression. MetaVD+DP does not communicate the model parameters whose dropout rates are larger than 0.8.
Results. Table 6 shows the test and OOD accuracy results, as well as the sparsity  $(\%)$  in the CIFAR-100 dataset. The sparsity represents the proportion of zero-value model parameters in the personalized layer. A higher sparsity percentage indicates more parameter pruning or elimination performed on the weight. In our experiment, MetaVD was able to prune about  $80\%$  of the weights in the personalized layer. In addition, when we dropped the communication of the parameters between the client and the server using the dropout thresholding technique, MetaVD still showed relatively good performance. In the case of Reptile+MetaVD+DP, the performance decreased by about  $2\%$  while using only  $20\%$  of the weights. On the other hand, MAML+MetaVD+DP and PerFedAvg+MetaVD+DP show an improvement in performance of about  $1\%$ . This shows that MetaVD can compress the model parameters required in the personalized layer, reducing the communication cost in FL without sacrificing much performance. Appendix K shows more experiments on model compression results on the CIFAR-100 dataset with different non-IID settings.
In this study, we presented a new novel Bayesian personalized federated learning (PFL) approach called meta-variational dropout (MetaVD). MetaVD utilizes a hypernetwork that predicts the dropout rates for each independent NN parameter, which enables effective model personalization and adaptation in federated learning (FL) with non-IID and limited data scenarios. In addition, MetaVD is the first approach to exploit variational dropout uncertainty in posterior aggregation in PFL. MetaVD's dropout posterior modeling provides a principled Bayesian aggregation strategy to consolidate local models into a global model, thereby improving training convergence. MetaVD is also a generic approach that is compatible with any other existing meta-learning-based PFL algorithms to avoid model overfitting. In addition, MetaVD's ability to compress model parameters can be used to reduce communication costs. A potential limitation of our approach is that it may increase the complexity of the model by introducing an additional hypernetwork. However, the hypernetwork is kept on the server, and we have verified that applying MetaVD to just one last layer before the output layer yields significant performance improvements in all experiments. Experimentally, MetaVD's performance has been validated on several PFL benchmarks, including CIFAR-10, CIFAR-100, FEMINIST, and CelebA, as well as multi-domain datasets. It demonstrates superior classification accuracy and uncertainty calibration, especially for out-of-distribution (OOD) clients. Overall, the experimental results show MetaVD to be a highly versatile approach capable of addressing many challenges in FL.
This work was supported by Center for Applied Research in Artificial Intelligence(CARAI) grant funded by Defense Acquisition Program Administration(DAPA) and Agency for Defense Development(ADD) (UD190031RD), Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No.2019-0-01082, SW StarLab and No.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)), Basic Science Research Program through National Research Foundation of Korea (NRF) funded by the Korea government (MSIT) (NRF-2020R1A2B5B03095585). Gunhee Kim is the corresponding author.
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[2] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with Non-IID data. arXiv preprint arXiv:1806.00582, 2018.
[3] Jaehoon Oh, Sangmook Kim, and Se-Young Yun. FedBABU: Towards enhanced representation for federated image classification. arXiv preprint arXiv:2106.06042, 2021.
[4] Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tianjian Chen, Han Yu, and Qiang Yang. FedVision: An online visual object detection platform powered by federated learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13172-13179, 2020.
[5] Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. PartialFed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 34:23309-23320, 2021.
[6] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloe Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.
[7] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018.
[8] Khalil Muhammad, Qinqin Wang, Diarmuid O'Reilly-Morgan, Elias Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. FedFast: Going beyond average for faster training of federated recommender systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1234–1242, 2020.
A The Global Posterior Decomposition in Bayesian FL
In Bayesian estimation, an alternative to Maximum Likelihood Estimation (MLE) is the inference or estimation of the posterior distribution of the parameters given all the data, denoted as  $p(w|\mathcal{D}\equiv \mathcal{D}^1\cup \dots \cup \mathcal{D}^M)$ . This posterior distribution is proportional to the product of the likelihoods and a prior,  $p(w|\mathcal{D})\propto p(w)\prod_{m = 1}^{M}p(\mathcal{D}^{m}|w)$ . In the case of a uniform prior, the modes of the global posterior coincide with the MLE solutions. This establishes an equivalence between the inference of the posterior mode and optimization. Under the uniform prior, any global posterior distribution that exists decomposes into a product of local posteriors:
$$
P (\theta | \mathcal {D}) \propto \prod_ {m = 1} ^ {M} p \left(\theta | \mathcal {D} ^ {m}\right) \tag {6}
$$
This proposition is well discussed in the recent Bayesian FL work [33]. In the FL context, the global model can be computed in the server by multiplicatively aggregating the local models adapted to each client. However, in practice, aggregating the posterior with the overfitted local models into the global one is difficult due to the heterogeneity among clients' data and the permutation invariance property of the NN architecture [90]. Thus, frequent communications between the server and the client are still demanded in the FL. The challenge of inferencing the local and global model and improving communication efficiency remains an active research area for real federated applications.
B The Motivation of Hierarchical Prior in MetaVD
In Section 3.2, we adopt the hierarchical prior [64, 65, 91] proposed in [47] for several reasons. The first reason is simply that it is a well-posed Bayesian prior that can avoid a degenerate posterior problem of the conventional VD prior [46, 61, 62]. The second reason is that we want a sparse prior to reduce communication costs by compressing the model; the hierarchical prior has been proven effective for parameter pruning. Although we briefly mentioned the KL divergence term as  $\mathrm{KL}(q(w^m;\phi)\| p(w^m)) = \sum_{k = 1}^{K}0.5\log (1 + (\alpha_k^m)^{-1})$  in the manuscript for readability, here we provide more detailed descriptions of applying the hierarchical prior.
Under the hierarchical prior assumption, we consider the joint prior and joint posterior distributions [92]. The joint prior,  $p(w^{m}, \gamma^{m}) = p(w^{m}|\gamma^{m})p(\gamma^{m})$ , is defined as a combination of a zero-mean Gaussian distribution,  $p(w^{m}|\gamma^{m}) = \mathcal{N}(w^{m}|0, \gamma^{m})$ , and a uniform hyper-prior,  $p(\gamma^{m}) = \mathcal{U}(\gamma^{m}|a, b)$ , over the variance. Then, we define a (conditional) joint variational posterior,  $q(w^{m}, \gamma^{m}|\phi) = q(w^{m}|\phi)q(\gamma^{m})$ , comprising the (conditional) dropout posterior  $q(w^{m}; \phi = (\theta, \phi, e^{m}))$  defined in Section 3.2 of the manuscript and an additional Dirac delta distribution,  $q(\gamma^{m})$ , to approximate the true (joint) posterior  $p(w^{m}, \gamma^{m}|\mathcal{D}^{m})$  (given the client's dataset  $\mathcal{D}^{m}$ ).
ELBO. With the hierarchical prior  $p(w^{m},\gamma^{m}) = p(w^{m}|\gamma^{m})p(\gamma^{m})$  and the (conditional) joint posterior  $q(w^{m},\gamma^{m}|\phi) = q(w^{m}|\phi)q(\gamma^{m})$  of the MetaVD, we can derive the local objective for each client as follows: