SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Insu Jeon* Minui Hong† Junhyeog Yun* Gunhee Kim†
Seoul National University, Seoul, South Korea
\*{insuj3on,antemrdm}@gmail.com
$\dagger$  {alsdml123,gunhee}@snu.ac.kr
Federated Learning (FL) aims to train a global inference model from remotely distributed clients, gaining popularity due to its benefit of improving data privacy. However, traditional FL often faces challenges in practical applications, including model overfitting and divergent local models due to limited and non-IID data among clients. To address these issues, we introduce a novel Bayesian meta-learning approach called meta-variational dropout (MetaVD). MetaVD learns to predict client-dependent dropout rates via a shared hypernetwork, enabling effective model personalization of FL algorithms in limited non-IID data settings. We also emphasize the posterior adaptation view of meta-learning and the posterior aggregation view of Bayesian FL via the conditional dropout posterior. We conducted extensive experiments on various sparse and non-IID FL datasets. MetaVD demonstrated excellent classification accuracy and uncertainty calibration performance, especially for out-of-distribution (OOD) clients. MetaVD compresses the local model parameters needed for each client, mitigating model overfitting and reducing communication costs. Code is available at https://github.com/insujeon/MetaVD.
Federated learning (FL) aims at training a global model from distributed clients without sharing or collecting their sensitive raw data. Thanks to its privacy-preserving aspect of FL [1], it is increasingly popular to be applied to various applications such as image classification [2, 3], object detection [4, 5], keyboard suggestion [6, 7], recommendation [8, 9], and healthcare [10, 11]. The conventional FL algorithm could achieve convergence when the data from different clients is independently and identically distributed (IID) [12-14]. However, due to the differences in preferences, locations, and usage habits of clients, the private data in FL are usually non-IID. When the data distributions of the clients vary, the local model learned from each client can diverge, and thus learning an optimal global model could fail [15, 16]. Furthermore, the scale of client data may not be sufficient to train a local model with large parameters, causing model overfitting and poor generalization [17, 18].
To overcome the challenge caused by non-IID data, personalized federated learning (PFL) has emerged [15, 16]. In the PFL, each client is allowed to have its own personalized model trained on each client's local data while still participating in the global model training. There are many branches of PFL, such as those based on multitask learning [19, 20], meta-learning [21-26], and transfer learning [27-30]. Although these approaches improve training convergence in non-IID data settings, they may still experience model overfitting with limited client data. Recently, the Bayesian learning paradigm has also been introduced in FL to address the overfitting by incorporating uncertainty in the model parameters [31-37]. However, they may also struggle with divergent local models when the data from different clients exhibit significant statistical variability. Motivated by these challenges, we aim to address the issues of FL with the non-IID and limited client data simultaneously.
In this paper, we present Meta-Variational Dropout (MetaVD), a novel Bayesian meta-learning approach [38-42] developed for PFL. MetaVD learns to predict client-dependent dropout rates
via a hypernetwork [25, 43-45]. This mechanism facilitates a data-efficient estimation of client-specific posterior simply by modulating a shared global NN parameter across all clients. The adaptation of MetaVD to conventional FL algorithms (e.g., FedAvg [1], Reptile [22], MAML [21], or PerFedAvg [23]) allows flexible model personalization across a variety of non-IID client distributions. MetaVD also incorporates a unique model aggregation strategy based on client-specific dropout uncertainty, providing a principled Bayesian way to consolidate local models into a global model. This strategy significantly enhances the convergence of the FL algorithm on non-IID data. In addition, MetaVD inherits the merit of Variational Dropout (VD) [46, 47], which facilitates model compression. This not only improves model generalization for out-of-data (OOD) clients but also reduces communication costs for parameter exchange. We performed an extensive analysis on MetaVD covering a wide range of FL scenarios [18], including different scales, non-IID degrees, partitions, and multi-domain settings [48]. In all of these experiments, MetaVD achieves excellent results compared to other state-of-the-art baselines.
A standard approach to FL (e.g., FedAvg [1]) iterates between the local training on the client devices and global optimization at the server. Given  $M$  clients, each of which has a data set  $\mathcal{D}^m = \{(x_i^m,y_i^m)\}_{i = 1}^{|\mathcal{D}^m|}$ , the FL problem can be formulated as follows:
$$
\text {S e r v e r :} \min  _ {w} \mathcal {J} (w) = \sum_ {m = 1} ^ {M} g ^ {m} \mathcal {J} ^ {m} (w), \quad \text {C l i e n t :} \mathcal {J} ^ {m} (w) = \frac {1}{| \mathcal {D} ^ {m} |} \sum_ {i} \ell \left(x _ {i} ^ {m}, y _ {i} ^ {m}; w\right). \tag {1}
$$
$w$  denotes the model parameter, and the global learning objective  $\mathcal{J}(w)$  at the server is a weighted average of the local objectives  $\mathcal{J}^m(w)$  over  $M$  clients. The weight  $g^m$  is proportional to the size of the local dataset (e.g.,  $|\mathcal{D}^m| / |\mathcal{D}|$ ). The local loss in a client device is usually defined as the empirical negative log-likelihood on the  $m$ -th client's dataset  $\mathcal{D}^m$  (i.e.,  $\ell(\mathcal{D}^m; w) = -\log p(y^m | x^m, w)$ ).
The local training is carried out in parallel fully (or partly) in each client device, with multiple Stochastic Gradient Descent (SGD) [49] epochs to get the fine-tuned local parameter  $w^{m}$ . Then, the aggregation step computes the global parameter in the server by taking the weighted average of the local parameters (i.e.,  $\bar{w} \gets \sum_{m=1}^{M} g^{m} w^{m}$ ). The global model parameter  $\bar{w}$  is then used as the initial parameter for each client in the next round of local training. FL aims to train models on large distributed datasets by only exchanging model parameters (e.g.,  $\bar{w}$  and  $w^{m}$ ) between server and local devices, thereby minimizing privacy leakage of client data.
Challenges in FL. There are many challenges to real-world FL applications. (1) Heterogeneity of client data. The original FL algorithm converges well when the client data is IID [12-14]. However, the client data often has different characteristics (e.g., classes or tasks follow non-IID). The  $w^{m}$  would drift away from each other, causing the  $\bar{w}$  to be suboptimal [15, 16]. (2) Sparse participation. In practice, the total number of clients  $M$  can be extremely large, while communication between the server and the clients can be intermittent or unreliable. This creates the challenge of inconsistent training due to a small subset of participating clients in each round of communication [18]. (3) Poor generalization due to limited data. When the training data available on each local device is limited, the local model can easily overfit, resulting in poor generalization to unseen clients [17, 18]. (4) Communication cost. FL optimization requires frequent communication between local devices and the central server to exchange model parameters. This process is slow and could introduce additional privacy concerns. Therefore, reducing model size is also an important area of research [50].
Bayesian FL. While conventional FL methods use a point estimate of the parameter as in Eq.1, recent work [31-33] has incorporated probability distributions over the model parameters. In these Bayesian FL approaches, the client device first estimates the local posterior from its data, and then the server aggregates the partially updated local posteriors into a global posterior. Based on FedAvg, a Gaussian distribution is used to represent each parameter in FedAG [32], and a posterior aggregation strategy using the MCMC technique is proposed in FedPA [33]. FedBE [31] uses a Bayesian ensemble global model with Gaussian or Dirichlet distributions. These methods improve prediction confidence and model convergence. However, FedAG [32] and FedBE [31] assume a global Gaussian posterior, but only consider point estimates of local parameters. FedPA [33] uses global and local Gaussian posteriors, but only maintains the global posterior mean on the server, making it difficult to track local uncertainties. In addition, the performance of previous methods can still be compromised by the
statistical discrepancies in the client data, as they are not tailored to the heterogeneous FL scenario. Recently, Bayesian PFL methods have also been introduced for non-IID data [34-37, 51, 52]. For example, pFedGP [34] learns a common Gaussian kernel for all clients and infers personalized classifiers for each client. pFedBayes [35] assumes an independent posterior model for each device while learning a shared global prior. These approaches achieve a substantial predictive performance improvement in non-IID data senerios compared to the previous Bayesian FL approaches. However, their approach still has limitations, such as the high computational cost of inverting a large kernel matrix in pFedGP [34] and imposing strong constraints in pFedBayes [35]. Their model aggregation rule does not account for parameter uncertainty directly. In addition, probabilistic modeling typically requires additional parameters to approximate the density, which can increase communication costs.
3 Approach: Meta-Variational Dropout
We propose a new Bayesian PFL approach, called Meta-Variational Dropout (MetaVD), which can simultaneously handle model personalization, regularization, and compression. MetaVD also exploits the uncertainties in model aggregation, thereby improving the training convergence on non-IID data. In addition, MetaVD is a universal approach that is compatible with various existing FL algorithms.
3.1 Variational Inference for FL
Instead of approximating the local posterior using MCMC in Bayesian FL [31-33], we can utilize the (amortized) Variational Inference (VI) framework of Bayesian meta-learning [38-42] that is originally developed for few-shot multi-task learning [53-55]. Considering each  $m$ -th client in FL as an individual task, an evidence lower-bound (ELBO)  $\mathcal{L}_{\mathrm{ELBO}}$  over all the  $M$  datasets is defined as
$$
\max  _ {\phi} \mathcal {L} _ {\mathrm {E L B O}} (\phi) = \sum_ {m = 1} ^ {M} g ^ {m} \left\{\mathbb {E} _ {q \left(w ^ {m}; \phi\right)} \left[ \log p \left(y ^ {m} \mid x ^ {m}, w ^ {m}\right) \right] - \mathrm {K L} \left(q \left(w ^ {m}; \phi\right) \mid \mid p \left(w ^ {m}\right)\right) \right\}. \tag {2}
$$
Here,  $p(y^{m}|x^{m},w^{m})$  represents a likelihood model constructed using a neural network (NN) on each  $m$ -th client's data [56-59], and  $w^{m}$  is a client-specific NN parameter.  $q(w^{m};\phi)$  is a variational posterior (or probabilistic) distribution over the  $w^{m}$ , which is also characterized by  $\phi$ .  $g^{m}$  is the local weight as defined in Eq.(1).  $p(w^{m})$  is a prior distribution that acts as a regularizer for the  $q(w^{m};\phi)$ . The local ELBO defined on each  $m$ -th client in Eq.(2) makes tradeoffs between the expected log-likelihood on its local dataset  $\mathcal{D}^m$  and the KL divergence with a prior. In fact, maximizing the  $\mathcal{L}_{\mathrm{ELBO}}(\phi)$  with respect to the variational parameter  $\phi$  is equivalent to minimizing  $\sum_{t = 1}^{M}g^{m}\mathrm{KL}(q(w^{m};\phi)||p(w^{m}|\mathcal{D}^{m}))$ . In theory,  $q(w^{m};\phi)$  is trained to approximate the true local posterior distribution over the  $w^{m}$  (i.e.,  $q(w^{m};\phi)\approx p(w^{m}|\mathcal{D}^{m})$ ) [56-59].
A straightforward Bayesian PFL approach might be utilizing a separate local posterior model  $q(w^{m};\phi)$  for each client device (e.g.,  $\phi = (\phi^{1},\dots ,\phi^{M})$ ). However, only a sparse subset of clients can participate in each FL round due to the communication availability of edge devices. Moreover, the number of clients  $M$  can be extremely large in practice, and some clients might only have small datasets. Thus, learning the variational parameter  $\phi^m$  independently for each device is difficult. As an alternative, we introduce a new hypernetwork-based [25, 43-45] conditional dropout posterior modeling approach that can be data-efficiently trained across multiple clients in FL.
3.2 Meta-Variational Dropout
Posterior model. To promote efficient model personalization and reduce overfitting in Bayesian FL, we define the posterior model in Eq.2 based on a Variational Dropout (VD) technique that multiplies continuous Gaussian noise to the NN parameters during training to prevent overfitting [46, 47, 60-62]. MetaVD extends the VD posterior by employing a global hypernetwork that learns to predict client-specific dropout rates (or personal model structure). In MetaVD, the variational posterior distribution for each  $m$ -th client's parameter,  $q(w^{m};\phi)$  in Eq.2, can be constructed as
$$
q \left(w ^ {m}; \phi = (\theta , \psi , e ^ {m})\right) = \prod_ {k = 1} ^ {K} \mathcal {N} \left(w _ {k} ^ {m} \mid \theta_ {k}, \alpha_ {k} ^ {m} \theta_ {k} ^ {2}\right) \text {w h e r e} \alpha^ {m} = h _ {\psi} \left(e ^ {m}\right). \tag {3}
$$
Figure 1: Overview of the Meta-Variational Dropout algorithm. (a) The server's hypernetwork predicts client-specific dropout rates from client embedding,  $e^{m}$ . The global parameters,  $\theta$ , and dropout variables,  $\alpha^{m}$ , are sent to each  $m$ -th client. (b) A few-shot local adaptation is performed on each client's device in parallel; then, the updated parameters are sent back to the server. (c) The server then aggregates those parameters to update its variational parameters  $\theta$ ,  $\psi$ , and  $e$ .
The variational parameter  $\phi$  is characterized by three distinct parameters  $(\theta, \psi, e)$ . The  $\theta$  is a global NN model parameter kept at the server (with no client index  $m$ ), where  $K$  is the size of the model parameter. The posterior distribution over the  $m$ -th client's weight  $w^{m}$  is described as a product of independent Gaussian distributions. In the product term in Eq.(3), each  $k$ -th factor describes a conditional Gaussian noise multiplication to the global NN parameter (e.g.,  $w^{m} = \theta * \epsilon^{m}$  where  $\epsilon^{m} \sim \mathcal{N}(\vec{1}, \alpha^{m})$  and  $\vec{1}$  is an  $K$ -dimensional all-one vector). The  $\alpha_{k}^{m}$  represents the client-specific dropout variable on each  $k$ -th index of NN parameter  $\theta_{k}$ . The  $h_{\psi}$  indicates a hypernetwork parameterized by  $\psi$  [43-45, 45] predicts the client-specific dropout rate  $\alpha^{m}$ . The  $e^{m}$  is a learnable client embedding used as an input to the  $h_{\psi}$ . In essence, MetaVD is a technique that modulates a global NN parameter with Gaussian noises. By predicting the client-specific dropout variable via a hypernetwork, a single NN can be reconfigured for various different clients. Since learning a hypernetwork across multiple local clients is more efficient than learning all the local posteriors independently, this approach can mitigate the sparse client participation and limited data issues in FL.
Prior model. To optimize the posterior model  $q(w^{m};\phi)$ , we need to specify the KL divergence term and the prior model  $p(w^{m})$  in Eq.(2). Prior modeling in MetaVD requires two criteria: (i) the KL divergence terms must be independent of the NN parameter  $\theta$  to ensure the lower-bound assumption in VD [46, 47, 61, 62], and (ii) all clients must share the same prior model to support the multiplicative posterior aggregation rule in Bayesian FL. We adopt the hierarchical prior [64, 65, 47] discussed in [47] because of its straightforward analytic KL term derivation and proven effectiveness in network regularization and sparsification. Under the hierarchical prior assumption, the KL divergence term in Eq.(2) is simplified to  $\mathrm{KL}(q(w^m;\phi)||p(w^m)) = \sum_{k = 1}^{K}0.5\log (1 + (\alpha_k^m)^{-1})$ ; see Appendix B for a more detailed derivation. This KL term is independent of the global NN parameter  $\theta$  and efficiently regularizes the dropout variable. The same hierarchical prior is applied to all  $M$  clients.
Client-side objective. Initially, the hypernetwork in the server approximates the dropout variable  $\alpha^m$  for each  $m$ -th client. The global parameter  $\theta$  and  $\alpha^m$  are transmitted to each client device. Then, the posterior model in a  $m$ -th client is trained on local data using the following ELBO objective:
$$
\max  _ {\theta , \alpha^ {m}} \mathcal {L} _ {\mathrm {E L B O}} ^ {m} (\theta , \alpha) = \frac {1}{| \mathcal {D} ^ {m} |} \sum_ {i} \log p \left(y _ {i} ^ {m} \mid x _ {i} ^ {m}, f (\epsilon ; \theta , \alpha^ {m})\right) - \sum_ {k = 1} ^ {K} 0. 5 \log \left(1 + \left(\alpha_ {k} ^ {m}\right) ^ {- 1}\right), \tag {4}
$$
which maximizes  $\mathcal{L}_{\mathrm{ELBO}}^m$  on the client dataset  $\mathcal{D}^m = \{(x_i^m, y_i^m)\}_{i=1}^{|\mathcal{D}^m|}$  with respect to the variational parameters (i.e.,  $\theta, \alpha^m$ ). The optimization can be done by the stochastic gradient variational Bayes (SGVB). [56-58], which reparameterizes the random weight variable  $w^m$  using a differentiable transformation as  $w_k^m = f(\epsilon_k; \theta_k, \alpha_k^m) = \theta_k + \sqrt{\alpha_k^m} \theta_k \epsilon_k$ , with a random IID noise  $\epsilon_k \sim \mathcal{N}(0, 1)$ . The intermediate weight  $w_k^m$  is now differentiable with respect to  $\theta_k$ , hence  $\alpha_k^m$ , and can be optimized via the SGD [49]. The analytical KL term acts as a regularization for  $\alpha_k^m$ .
Server-side objective. Once a local posterior adaptation is done in each client device, the updated parameters  $\theta_{*}^{m}$  and  $\alpha_{*}^{m}$  are returned to the server, then the server updates the variational parameters
The dropout rate is  $p = \alpha /(1 + \alpha) \in [0, 1]$  [63, 46]. We refer to  $\alpha$  as the dropout variable for simplicity.
Algorithm 1 MetaVD algorithm with MAML and Reptile variant for FL
Input: # of communication round  $R$ , # of client  $N$ , server learning rate  $\eta$ , client learning rate  $\gamma$ , inner learning rate  $l$ , local steps  $E$ , inner steps  $I$ , KL divergence parameter  $\beta$ . Initialize a global parameter  $\theta$ , hypernetwork  $h_{\psi}(\cdot)$  and  $e$ .  
for  $r = 1, \dots, R$  do  
    ▷ FL Rounds  
    Sample  $M$  clients from  $1, \dots, N$  clients  
    for  $m = 1, \dots, M$  do  
        Set  $\theta^m = \theta$  and  $\alpha^m = h_{\psi}(e^m)$   
        Send  $\theta^m$  and  $\alpha^m$  to the  $m$ -th client  
         $\theta_{*}^{m}, \alpha_{*}^{m} \gets \text{LOCALADAPTATION}(\theta^{m}, \alpha^{m})$   
    Compute global param  $\theta_{*}^{\mathrm{agg}}$  using  $\theta_{*}^{m}, \alpha_{*}^{m}$  and Eq.(5)  
     $\Delta \theta \gets \theta_{*}^{\mathrm{agg}} - \theta$ $\Delta \alpha^m \gets \alpha_{*}^m - \alpha^m$ $\theta \gets \theta + \eta \Delta \theta$ $\psi \gets \psi + \eta \frac{1}{M} \Sigma_m g_m((\nabla_{\psi} \alpha^m)^T \Delta \alpha^m)$   
    for  $m = 1, \dots, M$  do  
         $e^m \gets e^m + \eta (\nabla_{e^m} \psi)^T (\nabla_{\psi} \alpha^m)^T \Delta \alpha^m$
procedure LOCALADAPTATION_MAML  $(\theta ,\alpha)$    
for Each local step  $i$  from 1 to  $E$  do Set  $\theta_i^\prime = \theta_i$  and  $\alpha_{i}^{\prime} = \alpha_{i}$  Sample dataset  $D_{\mathrm{tr}}^{m}$  and  $D_{\mathrm{val}}^{m}$  from  $D^{m}$    
for Each inner step  $j$  from 1 to  $I$  do  $\theta_i^\prime \gets \theta_i^\prime -l\nabla_{\theta_i^\prime}\mathcal{L}_{ELBO}^m (\theta_i^\prime ,\alpha_i^\prime ;D_{\mathrm{tr}}^m))$ $\theta_{i}\gets \theta_{i} - \gamma \nabla_{\theta_{i}}\mathcal{L}_{ELBO}^{m}(\theta_{i}^{\prime},\alpha_{i}^{\prime};D_{\mathrm{val}}^{m})$ $\alpha_{i}\gets \alpha_{i} - \gamma \nabla_{\alpha_{i}}\mathcal{L}_{ELBO}^{m}(\theta_{i}^{\prime},\alpha_{i}^{\prime};D_{\mathrm{val}}^{m})$    
Set  $\theta_{*} = \theta_{i}$  and  $\alpha_{*} = \alpha_{i}$    
Send  $\theta_{*}$  and  $\alpha_{*}$  to the server   
procedure LOCALADAPTATION_REPTILE(0,  $\alpha$    
for Each local step  $i$  from 1 to  $E$  do  $\theta_i\gets \theta_i - \gamma \nabla_{\theta_i}\mathcal{L}_{ELBO}^m (\theta_i,\alpha_i;D^m))$ $\alpha_{i}\gets \alpha_{i} - \gamma \nabla_{\alpha_{i}}\mathcal{L}_{ELBO}^{m}(\theta_{i},\alpha_{i};D^{m})$    
Set  $\theta_{*} = \theta_{i}$  and  $\alpha_{*} = \alpha_{i}$    
Send  $\theta_{*}$  and  $\alpha_{*}$  to the server
(e.g.,  $\theta, \psi, e^{m}$ ) using the updated and local parameters. To update the global NN parameter  $\theta$ , we first assume the Bayesian posterior aggregation rule:  $p(w|\mathcal{D}) \propto \prod_{m=1}^{M} p(w^{m}|\mathcal{D}^{m})$  [31-33, 66]. Since each local posterior in Eq.(3) is a Gaussian (dropout) distribution, their product is also Gaussian:  $\mathcal{N}(w|\theta_{*}^{\mathrm{agg}}, \cdot) \approx \prod_{m=1}^{M} \mathcal{N}(w^{m}|\theta_{*}^{m}, \alpha_{*}^{m}(\theta_{*}^{m})^{2})$ . This gives us an exact aggregation rule to compute the maximum a posterior (MAP) solution of the  $\theta_{*}^{\mathrm{agg}}$  as follows:
$$
\theta_ {*} ^ {\text {a g g}} = \frac {1}{M} \sum_ {m} r ^ {m} \theta_ {*} ^ {m} \text {w h e r e} r ^ {m} = \frac {g ^ {m} \left(\alpha_ {*} ^ {m} \left(\theta_ {*} ^ {m}\right) ^ {2}\right) ^ {- 1}}{\sum_ {m} g ^ {m} \left(\alpha_ {*} ^ {m} \left(\theta_ {*} ^ {m}\right) ^ {2}\right) ^ {- 1}}. \tag {5}
$$
Note that Eq.(5) has the intuitive interpretation that the aggregation weight  $r^m$  is inversely proportional to its corresponding dropout variable (or noise variance)  $\alpha_*^m$ . Thus, parameters with high uncertainty have correspondingly less influence on the global mean prediction. In this way, we can fully exploit the uncertainty of the model parameters in the aggregation. To fully update each global NN parameter  $\theta_k$ , we follow the similar parameter update rule of FedAvg, except that the Bayesian aggregation Eq.(5) is utilized as in Algo.1. For the parameter of hypernetwork  $\psi$ , a more general update rule is used following [25]. We compute the changes in the updated dropout for each client  $\Delta \alpha^m$  as described in Algo.1, then an approximated gradient of the hypernetwork parameter is computed using the chain rule as  $\nabla_{\psi} \mathcal{L}_{\mathrm{ELBO}}^m(\alpha^m) = (\nabla_{\psi} \alpha^m)^T \Delta \alpha^m$ , where  $\nabla_{\psi} \alpha^m$  is the gradient of the output of hypernetwork with respect to  $\psi$ .  $\Delta \alpha^m$  is an approximation of the vector-jacobian product, which we are inspired by the work of [25] and [69]. The gradient for  $\nabla_{e^m} \mathcal{L}_{\mathrm{ELBO}}^m(\alpha^m)$  can be derived using the similar chain rule. The detailed update rules for each parameter are in Algo.1.
Combination with other meta-learning algorithms. Since the KL regularization in Eq.(4) is independent of the global (or initial) parameter  $\theta$ , MetaVD can be combined with several existing meta-learning based PFL algorithms (e.g., Reptile [22, 70], MAML [21, 55], PerFedAvg [23, 71]). For example, MAML [21, 55] requires some internal update steps using a subsampled dataset to compute the second-order gradient for  $\theta$ . Reptile [22, 70] uses only a first-order gradient computation as described in Algo.1, which is similar to FedAvg except for the addition of the learning rate  $\eta$ . Each local adaptation step is performed using the  $\mathcal{L}_{\mathrm{ELBO}}^m$  in Eq.(4) and the local data set  $D^{m}$ . Unlike conventional meta-learning based PFL algorithms, which maintain only one global initialization parameter, MetaVD allows to change the mode of the initialization parameters for each client.