SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mohammad Mahdi Rahimi Hasnain Irshad Bhatti Younghyun Park
Humaira Kousar Jaekyun Moon
{mahi,hasnain,dnffkf369,humairakousar32}@kaist.ac.kr
jmoon@kaist.edu
Federated Learning (FL) is a decentralized machine learning paradigm that enables collaborative model training across dispersed nodes without having to force individual nodes to share data. However, its broad adoption is hindered by the high communication costs of transmitting a large number of model parameters. This paper presents EvoFed, a novel approach that integrates Evolutionary Strategies (ES) with FL to address these challenges. EvoFed employs a concept of 'fitness-based information sharing', deviating significantly from the conventional model-based FL. Rather than exchanging the actual updated model parameters, each node transmits a distance-based similarity measure between the locally updated model and each member of the noise-perturbed model population. Each node, as well as the server, generates an identical population set of perturbed models in a completely synchronized fashion using the same random seeds. With properly chosen noise variance and population size, perturbed models can be combined to closely reflect the actual model updated using the local dataset, allowing the transmitted similarity measures (or fitness values) to carry nearly the complete information about the model parameters. As the population size is typically much smaller than the number of model parameters, the savings in communication load is large. The server aggregates these fitness values and is able to update the global model. This global fitness vector is then disseminated back to the nodes, each of which applies the same update to be synchronized to the global model. Our analysis shows that EvoFed converges, and our experimental results validate that at the cost of increased local processing loads, EvoFed achieves performance comparable to FedAvg while reducing overall communication requirements drastically in various practical settings.
Federated Learning (FL) provides a decentralized machine learning framework that enables model training across many devices, known as clients, without needing to collect and process sensitive client data on the centralized server [1]. The typical FL process begins with each client downloading an identical initialized model from a central server, performing model updates with local data, and then uploading the updated local model for the next communication round. Subsequently, the server combines the uploaded models to refine the global model, typically using a technique like FedAvg [2]. This iterative cycle repeats for a fixed number of rounds, ensuring collaborative model improvement across clients.
Although FL provides notable benefits, such as a certain level of privacy preservation and the utilization of diverse data sources, one of the major challenges associated with FL is the significant
communication overhead involved in transmitting model updates between clients and the server, especially when dealing with models that have a large number of parameters.
Various strategies have been developed to mitigate the communication burden in FL. These techniques can be broadly classified into three categories: i) Compressing updates: sparsification [1], structured updates [3], and quantization [4, 5] reduce the size of transmitted model updates, ii) Local computation: performing multiple local epochs at the clients [6] lessens the frequency of communication with the server, and iii) Advanced aggregation methods and client selection: MOCHA [7] enhances the efficacy of update aggregation and [8, 9] reduce communication by only selecting a subset of clients to participate in each training round.
Existing FL techniques primarily rely on transmitting gradient signals or model updates, which are computed through backpropagation (BP). On the other hand, Evolutionary Strategies (ES) [10-12] update model parameters by utilizing fitness values obtained from evaluating a population of models. This approach eliminates the need for a gradient signal, as depicted in Fig. 1. Recent advances in neuroevolution have shown promise in supervised settings and competitive performance with reinforcement learning in control tasks [13-15, 11, 16-18]. In this context, ES offers a distinct advantage compared to traditional BP methods. This paradigm shift opens up the possibility of searching for novel solutions beyond the gradient
Figure 1: ES follows an iterative ask, evaluate, and tell approach for optimization. The strategy starts by generating candidate solutions of a base model(ask), which are then assessed using a fitness measure (evaluate). The base model is updated towards a better solution using the evaluation results (tell).
based methods. However, it is critical to note that ES, despite its potential, still lags behind gradient-based methods in certain problems like supervised learning. Fig. 2(a) reveals the performance gap between ES and BP. This emphasizes the need for a balanced approach that would leverage the strengths of both ES and gradient-based methods to achieve optimal results.
Our proposed method operates on the premise of incorporating high-quality gradient signals into the evaluation process of ES. The process can be formalized as follows:
Given a base model, denoted by  $\theta$ , we instantiate a population  $P$  comprised of  $N$  model samples. Each individual sample,  $\theta^i$ , is derived by adding random perturbations to  $\theta$ . Unlike traditional ES, where the fitness of  $\theta^i$  corresponds to its performance on the task, we instead assess the fitness of each sample  $\theta^i$  by measuring its similarity to  $\theta'$ , the model parameters updated through gradient descent steps. This operation effectively exploits the gradient signal to construct a fitness vector, a process we term Population-Based Gradient Encoding (PBGE). Fig. 2(b) shows the results of a representative experimental setting where PBGE is able to follow BP closely on the FMNIST dataset while maintaining an effective compression rate of over  $98.8\%$ . In particular, this method significantly outperforms sparsification strategies at equivalent compression rates.
In the context of FL, the gradient signal can be encoded into a fitness vector for the population of models and communicated to the server. For global synchronization, a well-established approach like FedAvg would involve reconstruction and aggregation of the clients' models and sharing of the aggregated model with the clients. However, by utilizing shared random seeds, we can ensure uniformity in the generated populations between clients and the server. This allows us
to transmit and aggregate only the small fitness vectors without reconstructing the models, reinforcing communication efficiency. In addition, there is an advantage in implementing extra privacy measures. When encryption is desired, the required overhead would be much smaller with the fitness vectors than with the model parameter vectors because of the size difference. A brief overview of EvoFed is shown in Fig. 3 (the detailed methodology is provided in Section 4).
(a) Estimation with ES
(b) The proposed method
Figure 2: Test accuracy on FM-NIST dataset comparing (a) ES against BP, and (b) PBGE with BP and Sparse with  $98.8\%$  compression.
Figure 3: Overview of the proposed EvoFed: (1) Using the shared random seed, each client generates a population of perturbations around the local model. Each client also performs a gradient update of the local model using the local data as in conventional FL. (2) Each client evaluates the fitness of each perturbed model with respect to the updated model. The fitness values are communicated to the server. (3) The server aggregates the fitness values. Clients update their local models using broadcast aggregated fitness.
Contributions. To summarize, our main contribution is to introduce a novel concept of Population-Based Gradient Encoding (PBGE), which allows an accurate representation of the large local gradient vector using a relatively small fitness vector, significantly reducing the communication burden as well as the encryption overhead in FL environments. We also propose and verify the EvoFed framework that integrates PBGE into federated learning based on the exchange and aggregation of the fitness vectors between clients and the server.
Our approach achieves similar performance to FedAvg on both FMNIST and CIFAR-10 datasets. The advantage is that our scheme achieves very high compression, exceeding  $98.8\%$  on FMNIST and  $99.7\%$  on CIFAR-10 in key practical settings. Significantly, our EvoFed model also outperforms traditional compression methods, offering superior results at similar compression rates. The price we pay is the overhead of generating a population of perturbed models and computing similarity measures. This overhead could be substantial in the form of increased local processing time depending on the size of the population, but the client computational load can be traded with local memory space by employing parallel processing. In addition, as discussed later, reduce population size without affecting performance.
Federated Learning. Several techniques have been proposed to alleviate the communication overhead in FL, such as model compression [1, 19, 20], distillation techniques [21-25], and client update subsampling [26, 27]. However, these approaches often come with trade-offs, including increased convergence time, lower accuracy, or additional computational requirements. More importantly, they still require the exchange of model parameters, which requires substantial communication bandwidth. Furthermore, [28] shows that the gradient sparsity of all participants has a negative impact on global convergence and communication complexity in FL.
Evolutionary Strategies. ES are black-box optimization algorithms inspired by biological evolution [29]. ES algorithms iteratively refine a population of solutions based on fitness evaluations. Natural Evolution Strategies (NES) is a specific variant of ES [30, 12, 31-35]. Within the NES framework, the distribution  $p_{\psi}(\theta)$ , parameterized by  $\psi$ , is adopted to represent the population and maximize the average objective value  $\mathbb{E}_{\theta \sim p_{\psi}}[f(\theta)]$  via stochastic gradient ascent, where  $f(\theta)$  is the fitness function. NES algorithms leverage a score function estimator as in [10]. Our method follows the guidelines provided by [36], where the parameter distribution  $p_{\psi}$  is a factored Gaussian. Accordingly, we can represent  $\mathbb{E}_{\theta \sim p_{\psi}}[f(\theta)]$  using the mean parameter vector  $\theta$ , such that  $\mathbb{E}_{\theta \sim p_{\psi}}[f(\theta)] = \mathbb{E}_{\epsilon \sim \mathcal{N}(0,I)}[f(\theta + \sigma \epsilon)]$ . Given a differentiable function estimator, the well-known conversion procedure allows optimization over  $\theta$  to be rewritten as (see, for example, [37, 11, 38])
$$
\nabla_ {\theta} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, I)} [ f (\theta + \sigma \epsilon) ] = \frac {1}{\sigma} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, I)} \{f (\theta + \sigma \epsilon) \epsilon \}. \tag {1}
$$
The strategy of sharing seeds for random number generators to synchronize parallel processes and maintain consistency is a well-established practice in areas ranging from parallel simulations to cryptographic protocols [39]. The appeal of shared random seeds stems from their ability to offer deterministic randomness, ensuring that multiple entities, operating independently, can produce synchronized and identical random outputs. Within the realm of ES, this concept was effectively harnessed by [36] to address the challenge of scalability. We also utilize the shared random seeds to
Figure 4: Illustration of one update step of typical ES and the proposed Population-Based Gradient Encoding (PBGE) strategy. Left: using loss value  $L(\theta)$  directly for the fitness  $f(\theta)$ , as in existing ES, leads to an inherently noisy estimate of the gradient signal  $\nabla_{\theta} \mathbb{E}_{\epsilon}[f(\theta + \sigma \epsilon)]$  due to the noisy loss surface. Right: PBGE defines fitness as the distance to the updated model  $\theta'$  obtained through BP (i.e.,  $f(\theta) = -||\theta - \theta'||_2^2$ ). This enables the reliable estimate of  $\theta'$  on the convex surface. By sampling a sufficient number of perturbations  $\epsilon_i$ , a decent gradient signal can be obtained, which aligns to the true gradient signal  $\nabla_{\theta} L(\theta)$  computed from the BP.
scale down the communication load in a distributed setting. However, the key difference is that the work of [36] distributes the perturbations into multiple workers dealing with a single global objective, whereas our method generates identical perturbations across all clients, with each having a distinct objective. An important consequence is that with our method, each client only needs to communicate  $N$  fitness values to update the model, instead of  $MN$  values in [36] with  $M$  denoting the number of nodes, enabling scalability regardless of the number of clients.
Federated Learning and Evolutionary Strategies. Several studies have explored the optimization of FL using ES. For instance, [40] introduced an evolutionary approach for network architecture search (NAS) in real-time FL, which minimizes local payload while optimizing model performance. Sparse evolutionary training (SET) [41] substitutes fully connected layers in neural networks with sparse layers to decrease the number of model parameters. Furthermore, [42] presented the SET algorithm that optimizes neural networks in FL via a bi-objective method to maximize accuracy performance while minimizing communication overhead. Additionally, [43] introduces the MOEA/D framework [44] to the environment of FL and FLEA [45] utilized Evolutionary Algorithms in FL setup at the client-level to evolve models.
While these studies have made significant contributions, the present work establishes a unique way of using ES as a method to reduce communication overhead in FL by transmitting fitness values instead of model parameters. In particular, the new gradient-driven fitness function essentially separates our work from the traditional utilization of ES for compression.
3 Population-Based Gradient Encoding
This work focuses on the process of encoding gradient information through an identical population of models generated at both ends of some network link. Population distribution is a zero mean isotropic multivariate Gaussian with fixed covariance  $\sigma^2 I$ . We also adopt 'mirrored sampling' [46, 47] for variance reduction where the Gaussian noise vector  $\epsilon$  is instantiated with pairs of perturbations  $\epsilon, -\epsilon$ .
Given the reference point  $\theta' = \theta - \eta \nabla L(\theta)$  where  $\eta$  is the learning rate in the BP-based gradient update and  $\nabla L(\theta)$  represents the gradient derived from the data, we define the fitness function  $f(\theta)$  to measure the similarity between the model parameters  $\theta$  and  $\theta'$ :  $f(\theta) = -||\theta - \theta'||_2^2$ . This choice ensures that the gradient of the expectation,  $\nabla_{\theta} \mathbb{E}_{\epsilon \sim \mathcal{N}(0,I)}[f(\theta + \sigma \epsilon)]$ , aligns with the actual gradient  $\nabla L(\theta)$ , effectively encoding the gradient information in the fitness values:
$$
\nabla_ {\theta} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, I)} [ - | | (\theta + \sigma \epsilon) - \theta^ {\prime} | | _ {2} ^ {2} ] = - \nabla_ {\theta} | | \theta - \theta^ {\prime} | | _ {2} ^ {2} = 2 (\theta - \theta^ {\prime}) \tag {2}
$$
where the first equality simply follows from the assumption that  $\epsilon$  is zero-mean. The visual representation of this process is illustrated in Fig. 4. Eq. 2 gives  $\theta' = \theta - \frac{1}{2} \nabla_{\theta} \mathbb{E}_{\epsilon}[-||(\theta + \sigma \epsilon) - \theta'||_2^2]$ , and comparing with the BP operation  $\theta' = \theta - \eta \nabla L(\theta)$ , we have
$$
\eta \nabla_ {\theta} L (\theta) = \frac {1}{2} \nabla_ {\theta} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, I)} [ - | | (\theta + \sigma \epsilon) - \theta^ {\prime} | | _ {2} ^ {2} ].
$$
Now also utilizing Eq. 1, we write
$$
\eta \nabla_ {\theta} L (\theta) = \frac {1}{2 \sigma} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, I)} \left\{f (\theta + \sigma \epsilon) \epsilon \right\} \tag {3}
$$
Figure 5: Partition-based Model Parameter Compression. Illustration of dividing model parameters  $\theta$  into  $K$  partitions and compressing them individually using PBGE.
where  $f$  is the specific distance-based fitness function defined above. Approximating the expectation by sampling, we further write
$$
\eta \nabla_ {\theta} L (\theta) \approx \frac {1}{2 N \sigma} \sum_ {i = 1} ^ {N} f \left(\theta + \sigma \epsilon_ {i}\right) \epsilon_ {i} \tag {4}
$$
which shows how the gradient update information can be encoded using the fitness values  $f(\theta + \sigma \epsilon_{i})$  corresponding to the perturbations  $\epsilon_{i}$ . Consequently, this removes the need to evaluate each perturbation with the dataset, as is done in existing ES, and the entire update process is encoded with low-cost distance measurement.
Finally, the update on  $\theta$  itself based on the fitness values is naturally given by
$$
\theta^ {\prime} \approx \theta + \frac {1}{2 N \sigma} \sum_ {i = 1} ^ {N} f \left(\theta + \sigma \epsilon_ {i}\right) \epsilon_ {i} \tag {5}
$$
allowing a remote model update based on the transmitted  $f(\theta + \sigma \epsilon_{i})$  values with the shared knowledge of the  $\epsilon_{i}$  values.
The implemented algorithm consistently executes three steps: (i) Compute the target  $\theta^{\prime}$  through gradient descent, (ii) Implement perturbations to the model parameters and assess the perturbed parameters by computing their Euclidean distance to  $\theta^{\prime}$ , and (iii) Utilize the assessment results and encode the gradient with the fitness measures.
Partitioning. In the context of PBGE, the fitness function indicates the model distance, not the model performance on client data. This feature enables a unique partition-based approach for handling a large number of model parameters. Here, the parameters, flattened into a vector  $\theta$ , are split into  $K$  partitions:  $\theta[1], \theta[2], \dots, \theta[K]$ . Each partition is then effectively encoded individually using PBGE, which is advantageous when working with large models, as storing a large population of those may be burdensome for clients with limited memory. Partitioning allows us to compute  $K$  fitness values per each perturbation, providing  $K$  times more reference points to encode the gradient information given a population size. Hence, even with a small population size (requiring less memory), we achieve robust performance as validated by the empirical results provided in Supplementary Materials. Essentially, partitioning provides a tradeoff of memory with communication as more fitness values now need to be communicated per each perturbation. This partitioning process is visualized in Fig. 5. Notably, this partitioning technique can be used with any model architecture, regardless of its specific design, providing a practical and efficient means of compressing large models.
4 EvoFed
EvoFed operates on the principle that the evolutionary update step depends only on the fitness values given the perturbation samples. In the FL context, we can leverage this characteristic to devise an accurate yet communication-efficient strategy for model updates. This section provides a detailed exposition of our methodology, breaking it down into stages for clarity. This iterative process, as outlined in Algorithm 1, aims to gradually converge the model parameters at all nodes to an optimal solution while minimizing data transmission during each update. An overall view of our proposed methodology is depicted in Fig. 3.
Contour of local loss  $\mathrm{L_j}(\theta_t)$
Contour of aggregated fitness  $F_{t}$
Figure 6: Local Updates and Fitness Evaluation (left) and Server/Client-side Update (right). Left: Client performs BP on local data  $\theta_{t}$  to obtain  $\theta_{t,j}^{\prime}$ , evaluates fitness  $f_{t,j}^{i}$  by distance measure (e.g. L2) with  $\theta_{t}^{i}$ , and uploads  $\mathbf{f}_{t,j}$  to the server. Right: After obtaining the aggregated fitness  $\mathbf{F}_t$ , all nodes update the baseline model  $\theta_{t}$  according to Eq. 9.
4.1 Initial Setup
The initialization of the server and clients in EvoFed begins with the same baseline model, denoted by  $\theta_0$ . A key assumption is that the server and all clients share the same seed (e.g., via broadcasting) for identical random population generation. This approach ensures consistent baseline models and populations across all nodes.
In this step, a population of candidate solutions (in this case, models) is generated by adding random perturbations to the current best solution (the baseline model). The generation of an  $i$ -th member of the model population can be formulated as follows:
$$
\theta_ {t} ^ {i} = \theta_ {t} + \mathcal {N} (0, \sigma I) \tag {6}
$$
Here,  $\mathcal{N}(0,\sigma I)$  represents the perturbations sampled from a multivariate normal distribution with zero mean and a shared covariance matrix  $\sigma I$ . We also denote the population at each node at any given time  $t$  as  $\mathbf{P}_t = \{\theta_t^1,\theta_t^2,\dots,\theta_t^N\}$ , where  $\theta_t^i$  represents the  $i$ -th model in the population generated by adding the  $i$ -th perturbation to the baseline model parameters, and  $N$  is the size of the population.
4.2 Local Updates and Fitness Evaluation
Each client node begins by executing BP on its local dataset, using the baseline model,  $\theta_t$ , resulting in an updated model,  $\theta_t'$ . Following this, the fitness of each member  $\theta_t^i$  in the local population,  $\mathbf{P}_t$ , is evaluated. This evaluation is done by measuring the similarity between  $\theta_t'$  and  $\theta_t^i$ . The L2 norm or Euclidean distance serves as the measure of this similarity. The fitness of  $\theta_t^i$  is represented as  $f(\theta_t^i)$ :
$$
f \left(\theta_ {t} ^ {i}\right) = - \left| \left| \theta_ {t} ^ {\prime} - \theta_ {t} ^ {i} \right| \right| _ {2} ^ {2} \tag {7}
$$
The process of local update and fitness evaluation is illustrated in Fig. 6. The fitness values are the only information that needs to be communicated among nodes. The fitness vectors are significantly smaller in size compared to the model parameter vectors, which helps reduce the communication overhead. Hence, each client sends a fitness vector,  $\mathbf{f}_t = \{f(\theta_t^1), f(\theta_t^2), \dots, f(\theta_t^N)\}$  corresponding to all population members, to the server.
4.3 Server-side Aggregation and Update
The server's responsibility is to aggregate the fitness values reported by all client nodes, forming a global fitness vector  $\mathbf{F}_t$  comprising  $N$  elements, with  $\mathbf{F}_t^i$  representing the fitness value of the  $i^{th}$  member of the population. Each client's contribution to  $\mathbf{F}_t$  is weighted by their respective batch size  $b_j$ , giving a larger influence to clients with potentially more accurate local updates.
The global fitness vector  $\mathbf{F}_t$  is computed as follows:
$$
\mathbf {F} _ {t} = \frac {1}{\sum_ {j = 1} ^ {M} b _ {j}} \sum_ {j = 1} ^ {M} b _ {j} \mathbf {f} _ {t, j} \tag {8}
$$
where  $M$  is the total number of clients and  $\mathbf{f}_{t,j}$  is the fitness value vector from client  $j$  at time  $t$ . After the aggregation, the server broadcasts  $\mathbf{F}_t$  so that each client updates the baseline model  $\theta_t$ .