WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average

## Abstract

Abstract The performance of deep neural networks is enhanced by ensemble methods, which average the output of several models. However, this comes at an increased cost at inference. Weight averaging methods aim at balancing the generalization of ensembling and the inference speed of a single model by averaging the parameters of an ensemble of models. Yet, naive averaging results in poor performance as models converge to different loss basins, and aligning the models to improve the performance of the average is challenging. Alternatively, inspired by distributed training, methods like DART and PAPA have been proposed to train several models in parallel such that they will end up in the same basin, resulting in good averaging accuracy. However, these methods either compromise ensembling accuracy or demand significant communication between models during training. In this paper, we introduce WASH, a novel distributed method for training model ensembles for weight averaging that achieves state-of-the-art image classification accuracy. WASH maintains models within the same basin by randomly shuffling a small percentage of weights during training, resulting in diverse models and lower communication costs compared to standard parameter averaging methods.

## 1 Introduction

In order to enhance the accuracy of a given class of models, aggregating the answers of multiple instances trained in parallel can be done via model *ensembling*. This can lead to significant improvements in modern deep learning models (), increasing generalization ability. However, this comes at the cost of evaluating multiple instances of a given model at inference. This increases both the required memory and computations, resources which can be critical for on-device inference (). To resolve this issue, the population of models can be fused into a single model to obtain both the generalization improvements of ensembling and the inference cost of a single model. Since independent models can be linearly connectable (), a simple technique is to average the weights of the different models to obtain a fused model ().

There are however limits to this method. For models that are too dissimilar, the performance of the averaged model may not be better than chance (). To mitigate this, the ensemble can either use a pre-trained network as a starting point () or ensure that models share part of their optimization path (). Nevertheless, diminishing too much the ensemble diversity comes at the cost of its performance (see Fig. 6 of ()), revealing a tradeoff between model diversity and weight averageability. Inspired by distributed training, techniques like DART () and PAPA () have been proposed to train a population of models in parallel on heterogeneous data while communicating to balance this tradeoff. DART, similarly to LocalSGD (), averages all the models regularly to avoid models diverging. PAPA controls the diversity of the models more finely, by pushing them towards the averaged parameters using an Exponential Moving Average (EMA) like EASGD (), achieving better performances. Notably, they show that training a population in such a way results in models that generalize better than a single model trained with the same compute as the entire population, demonstrating the potential of these distributed approaches. However, existing methods require a regular computation of the average model using an all-reduce operation, either to remove periodically any diversity in the population () or in the case of PAPA, to compute an EMA of the average. This results in a high communication cost during the parallel training of the population of models (), hindering the scalability of these approaches as the population size increases ().

We propose in this paper a novel distributed method to train a population of models in parallel while keeping their weights within the same basin. It requires a fraction of the communication cost of PAPA but displays greater model diversity during training, increasing the final averaging accuracy. Our main idea is to shuffle parameters between models during training, forcing them to learn using the others’ parameters. We refer to ’parameter shuffling’ as the following idea. A permutation is randomly chosen, and models will communicate peer-to-peer their parameters following the permutation.
The use of a permutation is distinct from the notion of weight permutation of () which is inside one model. We denote our method, which achieves Weight Averaging using parameter SHuffling, as WASH, and represent it schematically in Fig. [1](#S1.F1).

Figure: Figure 1: Representation of training with WASH. A population of models is being trained separately. (1) After each training step, (2) a small percentage of parameters are permuted between models. (3) At the end of the training, the model weights are averaged, resulting in a high-performance model.
Refer to caption: /html/2405.17517/assets/x1.png

#### Contributions.

Our work makes the following contributions: (1) We propose a novel method for the training of a population of models that can be weight averaged, which we refer to as WASH (Weight Averaging using parameter SHuffling).
By shuffling a small number of parameters between models during training, the resulting population can be weight-averaged into a high-performance model for a fraction of the communication volume of methods such as PAPA. (2) We find that WASH provides state-of-the-art results on image classification tasks, resulting in models with performances at the level of ensembling methods while only requiring a single network at inference time. (3) We provide experiments to better understand the improvement provided by WASH, notably on how WASH reduces the distance between models in the population implicitly while maintaining diversity. (4) We perform different ablations on our method, showing the impact of the shuffling. (5) Our code is made available at
github.com/fournierlouis/WASH.

## 2 Related work

#### Ensemble and weight averaging.

By combining predictions from multiple models, ensemble methods significantly improve a predictive system’s ability to make accurate generalizations (; ), while reducing the variance of estimator ().
This variance reduction is especially effective when errors are uncorrelated and models show diversity, meaning they do not fail simultaneously on the same instances (; ). However, ensembles require additional passes through each model for inference, leading to increased computational costs. This cost can become prohibitive with a large number of models. As a remedy under some constraints, models can be averaged together to remove the computational burden during inference. Averaging the weights of models was first explored in
simple linear () and convex scenarios (; ). In deep learning, () establish that weight averaging is a first-order approximation of the ensemble when models are close in the weight space. Notably, a simple averaging of multiple points along the SGD trajectory leads to better generalization. Following mode connectivity (; ) and the observation that many optima of independent models are connectable, (; ) propose
learning simplexes in parameter space with a regularization penalty to encourage diversity in weight space, and (; ) propose to train multiple model branches with different last-layer initialization and hyperparameters concurrently. These models are later averaged to enhance generalization and reduce the inference cost. However, for these models to be amenable to weight averaging, they need to begin with the same pre-trained initialization () which can diminish diversity among the models. To alleviate this issue, neuron alignment techniques (; ; ; ) match the units of multiple networks to make them amenable to weight averaging, however, they rarely work in practical scenarios (), often obtaining performance below the individual models. DART () and Branch-Train-Merge (BTM) () propose a three-phase training pipeline. The process begins with an initial shared training phase, followed by the parallelized training of multiple models, each diversified through different data domains or different data augmentations. Finally, these models are merged into a single model. They find that iterative refinement of the last 2 stages enhances the overall optimization trajectory and improves generalization. To enhance diversity among models, PAPA () proposes to rather gradually adjust model weights towards the population average throughout the training process, beginning from random initialization. However, these approaches can result in substantial communication costs during training. Conversely, WASH tackles high communication costs by permuting only a small fraction of parameters among models during training, while ensuring the branches remain amenable to weight averaging at the end.

#### Distributed and federated learning.

In distributed training of deep learning models, the trade-off between communications and model performance is a core concern (; ), and finding methods efficiently alleviating some of its communication cost is a recurrent theme in different areas of research (; ). For instance, communication overhead being a key concern of decentralized optimization, it has been shown in this literature that to train models in a data-parallel setting with a limited communication budget, a key metric to observe is the average distance to the consensus (; ; ; ; ). The techniques discussed earlier to train a population of models for weight averaging are similar to methods in the LocalSGD (; ) and Federated Learning (; ; ) literature. The training in DART and BTM is similar to the LocalSGD training, where models are averaged regularly after several steps of computations. PAPA, which uses an EMA of the averaged model to gradually move the models towards consensus, is similar to methods like EASGD () or SlowMo (). Only averaging a population at the end of training like in BTM was also proposed for LocalSGD (), and cross-gradient aggregation () can be seen as a way to shuffle gradients locally. Federated learning also uses techniques discussed previously for model merging (; ; ). Finally, our method can be thought of as training a global model where each local model picks randomly from a subset of parameters when shuffled. This can be linked to Bayesian learning () in particular for federated learning (; ), or to federated subnetwork training (; ).

Figure: Algorithm 1 Training with WASH

## 3 Parameter shuffling in an ensemble for weight averaging

#### Motivation of our training procedure.

We aim to balance the benefits of model ensembling with the computational efficiency of using a single model at inference via weight averaging. In other words, our objective is to produce a single model resulting from the ensembling. A set of $N$ model parameters $\{\theta_{n}\}_{n\leq N}\subset\mathbb{R}^{d}$ are trained in parallel on the same dataset, with a different data order and possibly different data augmentations and regularizations. To avoid divergence among the models, PAPA applies an EMA every $T$ training steps and produces the following update

$$ $\tilde{\theta}_{n}\leftarrow\alpha\theta_{n}+(1-\alpha)\bar{\theta}\,,$ (1) $$

where $\bar{\theta}\triangleq\frac{1}{N}\sum_{n=1}^{N}\theta_{n}$ represents the average of the model weights, also referred to as the consensus, and $\alpha\in]0,1[$ is weighted depending on the learning rate. Despite its benefits, this method has drawbacks, including the need for synchronized global communication across all models, which can be inefficient, and the potential reduction in model diversity due to the consensus constraint, which may reduce model expressivity. Indeed, we observe that after each update

$$ $\sum_{n}\lVert\tilde{\theta}_{n}-\bar{\theta}\rVert^{2}=\alpha^{2}\sum_{n}\lVert\theta_{n}-\bar{\theta}\rVert^{2}<\sum_{n}\lVert{\theta}_{n}-\bar{\theta}\rVert^{2}\,,$ (2) $$

which shows that the EMA step of methods like PAPA directly reduces the distance of the models to the consensus, hindering their diversity.

#### Proposed method: WASH.

To address these challenges, we propose the following stochastic parameter shuffling step instead of the EMA, defined for any individual parameter $\theta^{i}_{n}\in\mathbb{R}$ of a model $\theta_{n}=[\theta^{i}_{n}]_{i=1}^{d}$ by

$$ $\hat{\theta}^{i}_{n}\leftarrow\begin{cases}\theta^{i}_{\pi_{i}(n)}&\text{with probability }p,\\ \theta^{i}_{n}&\text{otherwise},\end{cases}$ (3) $$

where $\pi_{i}$ denotes a random permutation of the indices $\{1,...,N\}$, chosen uniformly at each iteration for each parameter index $i\in\{1,...,d\}$, and independently from the Bernoulli variable of Eq. ([3](#S3.E3)). Notably, this parameter shuffling reduces in expectation to

$$ $\mathbb{E}[\hat{\theta}_{n}]=(1-p)\theta_{n}+p\bar{\theta}\,.$ (4) $$

Thus, WASH aligns, in expectation, with the EMA of Eq. ([1](#S3.E1)) for $p=(1-\alpha)$. The expected number of parameters communicated by each model at each step is thus $p\times d$ while for PAPA, each model communicating all of its parameters every $T$ steps, this amounts to $\frac{d}{T}$. Thus, $p\ll\frac{1}{T}$ results in a significantly reduced communication overhead favorable to WASH. However, the model diversity is higher, as WASH preserves the consensus distance, as shown by

$$ $\sum_{n}\lVert\hat{\theta}_{n}-\bar{\theta}\rVert^{2}=\sum_{n}\sum_{i}(\hat{\theta}^{i}_{n}-\bar{\theta}^{i})^{2}=\sum_{i}\sum_{n}({\theta}^{i}_{n}-\bar{\theta}^{i})^{2}=\sum_{n}\lVert{\theta}_{n}-\bar{\theta}\rVert^{2}\,.$ (5) $$

#### Layer-wise adaptation via WASH.

Recognizing that different network layers may require varying levels of adaptation due to their roles and dynamics, we introduce a layer-specific probability adjustment. Assuming $L$ layers in the network, we set for each layer $l$ (where $0\leq l<L$)

$$ $p_{l}=p\left(1-\frac{l}{L-1}\right)\,,$ (6) $$

where $p$ is a base probability. In other words, the parameters of the first layer have a shuffling probability of $p$, while the final layer’s parameters are never shuffled. This adaptation ensures that deeper layers, which are typically slower to train and more sensitive to the input features, undergo fewer permutations than the more generalizable early layers. This strategy not only preserves the specificity required by the initial layers but also further halves the overall communication overhead.

#### Full procedure.

Alg. [1](#alg1) presents the training of a population of $N$ models using WASH. Starting from the same initialization, our training procedure alternates between local gradient computations and shuffling communications. At inference, we simply average the weights of the models, obtaining a single model with parameters $\bar{\theta}$. Note that techniques like REPAIR () or activation alignment () could be incorporated to improve the alignment of the models, but we found them to be unnecessary to obtain high accuracy and kept our evaluation framework minimal for the sake of simplicity.

## 4 Experiments

#### Training methods.

We showcase the capacities of WASH for training a population of neural networks on standard image classification tasks. As a Baseline, we consider a population trained separately, with each model working on a different dataset order and different data augmentations and regularization (if they are used). This is the same baseline as (), only starting from the same initialization, but we found that this change brought no significant impact on performance.
We also compare WASH to PAPA () on the same tasks (with PAPA however using models with a different initialization), to show our improvement despite requiring a fraction of the communication cost. We do not provide comparisons to DART () or the variants of PAPA as their performances are generally inferior (). We also propose a variant of WASH named WASH+Opt that also permutes the optimizer state associated with the parameter shuffled (in our case, the momentum of SGD), doubling the volume of communications. We do not permute or recompute the running statistics of BatchNorm layers for simplicity.

**Table 1: Communication volume and inference costs of four training techniques. The baseline Ensemble is trained separately but requires a linearly increasing inference cost. In our experiments, we fix the base probability of WASH and WASH+Opt to be equal to $0.001$ or $0.05$ when training on CIFAR-10/100 or ImageNet, resulting in a reduction of communication volume over PAPA.**
|  | Communication volume |  |  |
| --- | --- | --- | --- |
| Technique | CIFAR-10/100 | ImageNet | Inference cost |
| Ensemble | 0 0 | 0 0 | $N$ |
| PAPA | $1$ | $1$ | $1$ |
| WASH | $\nicefrac{{1}}{{200}}$ | $\nicefrac{{1}}{{4}}$ | $1$ |
| WASH+Opt | $\nicefrac{{1}}{{100}}$ | $\nicefrac{{1}}{{2}}$ | $1$ |

#### Communication cost.

Training with PAPA requires computing an all-reduce operation on all of the models’ parameters every $T=10$ training steps. In comparison, WASH requires, in expectation, a shuffling of $p/2$ of the parameters of the models at every training step. Thus, by keeping a base probability $p\leq 0.2$, WASH results in a more communication-efficient training. In practice, $p$ will be equal in our experiments to $0.001$ or $0.05$, ensuring a communication volume reduction of $200$ or $4$.

#### Evaluation strategy.

After training, the population of models obtained can be evaluated in three separate ways. As a baseline, the performance of the population can be evaluated as an Ensemble, averaging the predictions of the models. The parameters of the models can be averaged to obtain a single model, which we refer to as Averaged. This is equivalent to UniformSoup in () or AvgSoup in () for example. More elaborate averaging methods have been proposed, such as GreedySoup (), which averages an increasing number of models (in order of validation accuracy) until the averaging does not improve accuracy. We report the accuracy of the Ensemble and Averaged model for all training techniques, as well as the GreedySoup accuracy of the Baseline. Like (), we find that the GreedySoup accuracy corresponds to the accuracy of a single model for the Baseline and that the Averaged model accuracy outperforms the GreedySoup model for the other techniques, and thus chose not to report it. We summarize in Tab. [1](#S4.T1) the communication volume and inference cost necessary for training a separate Ensemble of models, or training with PAPA, WASH, or WASH+Opt.

**Table 2: Ensemble and Averaged Model accuracy for a heterogeneous population of models; trained with varying data augmentations and regularizations. We compare models trained separately (Baseline), with PAPA, or our method WASH and its variant WASH+Opt. We also report the GreedySoup accuracy for the Baseline models. The best Ensemble (black) and Averaged (blue) accuracy are reported in bold. Except on CIFAR-10, WASH and in particular WASH+Opt provide the best performance for the final Averaged Model, with performances comparable to the Ensemble of models for a fraction of the inference cost.**
| Method | Baseline (trained separately) | PAPA | WASH (ours) | WASH+Opt (ours) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Config | #N | Ensemble | Averaged | GreedySoup | Ensemble | Averaged | Ensemble | Averaged | Ensemble | Averaged |
| CIFAR-10 |  |  |  |  |  |  |  |  |  |  |
| VGG-16 | 3 | 95.98$\pm$.42 | 10.00$\pm$.00 | 95.26$\pm$.05 | 96.12$\pm$.34 | 96.13$\pm$.24 | 95.89$\pm$.23 | 95.97$\pm$.24 | 95.91$\pm$.36 | 95.85$\pm$.27 |
|  | 5 | 96.28$\pm$.40 | 10.00$\pm$.00 | 95.42$\pm$.10 | 96.24$\pm$.17 | 96.21$\pm$.13 | 96.15$\pm$.10 | 96.20$\pm$.10 | 96.00$\pm$.21 | 96.04$\pm$.14 |
|  | 10 | 96.47$\pm$.07 | 10.00$\pm$.00 | 95.39$\pm$.24 | 96.32$\pm$.13 | 96.31$\pm$.13 | 96.27$\pm$.10 | 96.18$\pm$.13 | 96.14$\pm$.08 | 96.20$\pm$.05 |
| ResNet18 | 3 | 97.15$\pm$.28 | 10.17$\pm$.29 | 96.62$\pm$.38 | 97.33$\pm$.05 | 97.24$\pm$.05 | 97.21$\pm$.19 | 97.19$\pm$.17 | 97.22$\pm$.07 | 97.25$\pm$.14 |
|  | 5 | 97.33$\pm$.08 | 10.09$\pm$.16 | 96.61$\pm$.03 | 97.35$\pm$.12 | 97.31$\pm$.06 | 97.21$\pm$.10 | 97.25$\pm$.12 | 97.18$\pm$.09 | 97.16$\pm$.07 |
|  | 10 | 97.59$\pm$.01 | 9.26$\pm$1.28 | 96.79$\pm$.14 | 97.39$\pm$.13 | 97.34$\pm$.06 | 97.30$\pm$.10 | 97.28$\pm$.04 | 97.20$\pm$.13 | 97.16$\pm$.13 |
| CIFAR-100 |  |  |  |  |  |  |  |  |  |  |
| VGG-16 | 3 | 80.36$\pm$.15 | 1.00$\pm$.00 | 77.92$\pm$.22 | 78.89$\pm$.10 | 78.77$\pm$.16 | 79.10$\pm$.88 | 79.05$\pm$.68 | 79.15$\pm$.61 | 79.15$\pm$.41 |
|  | 5 | 81.32$\pm$.56 | 1.00$\pm$.00 | 77.81$\pm$.25 | 79.51$\pm$.38 | 79.24$\pm$.43 | 79.65$\pm$.27 | 79.39$\pm$.21 | 79.75$\pm$.21 | 79.71$\pm$.20 |
|  | 10 | 82.24$\pm$.15 | 1.00$\pm$.00 | 77.83$\pm$.65 | 79.95$\pm$.11 | 79.64$\pm$.13 | 80.05$\pm$.18 | 79.70$\pm$.25 | 80.03$\pm$.11 | 79.76$\pm$.13 |
| ResNet18 | 3 | 82.84$\pm$.48 | 1.00$\pm$.01 | 80.06$\pm$1.5 | 81.58$\pm$.12 | 81.53$\pm$.13 | 81.91$\pm$.34 | 81.90$\pm$.36 | 81.99$\pm$.06 | 82.08$\pm$.09 |
|  | 5 | 83.72$\pm$.49 | 1.00$\pm$.00 | 80.72$\pm$.52 | 82.09$\pm$.30 | 82.01$\pm$.34 | 82.16$\pm$.42 | 81.97$\pm$.28 | 82.35$\pm$.17 | 82.17$\pm$.15 |
|  | 10 | 84.18$\pm$.20 | 1.00$\pm$.00 | 80.61$\pm$.43 | 82.32$\pm$.09 | 82.15$\pm$.14 | 82.43$\pm$.32 | 82.31$\pm$.38 | 82.42$\pm$.31 | 82.18$\pm$.22 |
| ImageNet |  |  |  |  |  |  |  |  |  |  |
| ResNet50 | 3 | 76.16$\pm$.28 | 0.10$\pm$.00 | 74.15$\pm$.11 | 75.62$\pm$.15 | * | 74.39$\pm$.14 | 74.34$\pm$.18 | 74.30$\pm$.22 | 74.18$\pm$.26 |
|  | 5 | 76.68$\pm$.06 | 0.10$\pm$.00 | 74.47$\pm$.06 | 75.80$\pm$.21 | * | 74.63$\pm$.11 | 74.59$\pm$.07 | 74.44$\pm$.21 | 74.39$\pm$.21 |

### 4.1 Main experiments

#### Experimental setup.

We showcase the performance of WASH for training neural networks on image classification tasks on the CIFAR-10, CIFAR-100 (), and ImageNet () datasets. We use the same training framework as () for a fair comparison. We train a population of $N$ models for $N\in\{3,5,10\}$, on the ResNet-18, 50 and VGG-16 architectures. 2% of the training data is kept as validation for computing the GreedySoup. Like (), we consider one framework with heterogeneous models, learning with different data augmentations and regularizations, and one homogeneous setting with no data augmentations, where the only difference between the models’ training is the dataset shuffling. Details are presented in the Appendix. Models are trained with SGD with momentum, a weight decay of $10^{-4}$, and a cosine annealing scheduler with starting and minimum learning rates $0.1$ and $10^{-4}$. For CIFAR-10/100, we train over 300 epochs with a batch size of 64, and 90 epochs with a batch size of 256 for ImageNet. For WASH and WASH-Opt we initialize the models with the same parameters and choose $p$ with cross-validation to be equal to $0.001$ or $0.05$ when training on CIFAR-10/100 or ImageNet. We do not require any alignment technique such as REPAIR.

#### Main results.

Tab. [2](#S4.T2) and Tab. [3](#S4.T3) correspond respectively to the heterogeneous and homogeneous settings. We report the test accuracies as the average of 3 runs for the Ensemble of models, the Averaged model, and the GreedySoup for the Baseline (equivalent to the best model). Consistent with the findings of (), we find that networks trained separately have a high Ensemble accuracy, but perform as random when averaged. On CIFAR-10/100, methods like PAPA and WASH result in lower Ensemble accuracy but almost no difference between the Ensemble and Averaged accuracies. In general, WASH and WASH+Opt outperform PAPA, despite requiring a lower communication volume. On ImageNet, our parallelization procedure resulted in a slightly lower Baseline accuracy and we were not able to reproduce PAPA’s baseline due to our distributed constraint. The WASH Averaged model reaches a high accuracy, like previously.
Both of our methods reduce the gap with accuracies of the baseline Ensemble, indicating that WASH hinders less the diversity of the population of models while maintaining weight averagability. However, a gap still remains, which may be inherent to models being in the same basin. WASH and WASH+Opt have very similar results, with the simpler WASH being better on the homogeneous case and WASH+Opt being better on the heterogeneous case.

**Table 3: Ensemble and Averaged Model accuracy for a homogeneous population of models. We compare models trained separately (Baseline), with PAPA, or our methods WASH and WASH+Opt. The best Ensemble (black) and Averaged (blue) accuracy are reported in bold. We observe the same results in this setting, with WASH in particular reaching close to the Ensemble performance.**
| Method | Baseline (trained separately) | PAPA | WASH (ours) | WASH+Opt (ours) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Config | #N | Ensemble | Averaged | GreedySoup | Ensemble | Averaged | Ensemble | Averaged | Ensemble | Averaged |
| CIFAR-10 |  |  |  |  |  |  |  |  |  |  |
| VGG-16 | 3 | 94.93$\pm$.06 | 10.00$\pm$.00 | 93.60$\pm$.41 | 94.38$\pm$.14 | 94.34$\pm$.18 | 94.41$\pm$.23 | 94.58$\pm$.17 | 94.45$\pm$.05 | 94.47$\pm$.02 |
|  | 5 | 95.29$\pm$.05 | 10.00$\pm$.00 | 93.82$\pm$.30 | 94.55$\pm$.12 | 94.58$\pm$.12 | 94.72$\pm$.08 | 94.70$\pm$.17 | 94.63$\pm$.11 | 94.68$\pm$.14 |
|  | 10 | 95.23$\pm$.06 | 10.00$\pm$.00 | 93.82$\pm$.06 | 94.79$\pm$.18 | 94.78$\pm$.20 | 94.66$\pm$.03 | 94.54$\pm$.07 | 94.71$\pm$.07 | 94.61$\pm$.13 |
| ResNet18 | 3 | 96.14$\pm$.10 | 10.00$\pm$.00 | 95.42$\pm$.27 | 95.89$\pm$.04 | 95.89$\pm$.06 | 95.77$\pm$.12 | 95.77$\pm$.17 | 95.85$\pm$.04 | 95.87$\pm$.10 |
|  | 5 | 96.19$\pm$.16 | 10.00$\pm$.00 | 95.31$\pm$.09 | 95.99$\pm$.08 | 95.99$\pm$.08 | 95.96$\pm$.08 | 95.98$\pm$.05 | 95.94$\pm$.12 | 95.98$\pm$.12 |
|  | 10 | 96.34$\pm$.02 | 10.00$\pm$.00 | 95.26$\pm$.11 | 96.10$\pm$.25 | 96.11$\pm$.24 | 96.08$\pm$.07 | 96.12$\pm$.09 | 96.07$\pm$.07 | 96.08$\pm$.14 |
| CIFAR-100 |  |  |  |  |  |  |  |  |  |  |
| VGG-16 | 3 | 77.63$\pm$.24 | 1.00$\pm$.00 | 73.76$\pm$.35 | 75.10$\pm$.11 | 75.09$\pm$.16 | 76.30$\pm$.37 | 76.04$\pm$.58 | 76.04$\pm$.03 | 75.96$\pm$.18 |
|  | 5 | 78.52$\pm$.10 | 1.00$\pm$.00 | 73.76$\pm$.18 | 75.56$\pm$.16 | 75.55$\pm$.14 | 76.63$\pm$.27 | 76.48$\pm$.23 | 76.64$\pm$.15 | 76.13$\pm$.18 |
|  | 10 | 79.26$\pm$.06 | 1.00$\pm$.00 | 73.99$\pm$.26 | 76.24$\pm$.44 | 76.26$\pm$.43 | 77.06$\pm$.12 | 76.43$\pm$.18 | 76.72$\pm$.15 | 75.94$\pm$.26 |
| ResNet18 | 3 | 79.54$\pm$.17 | 1.00$\pm$.00 | 76.84$\pm$.54 | 77.83$\pm$.26 | 77.86$\pm$.30 | 78.90$\pm$.17 | 78.76$\pm$.25 | 78.66$\pm$.08 | 78.56$\pm$.21 |
|  | 5 | 80.11$\pm$.23 | 1.00$\pm$.00 | 76.83$\pm$.45 | 77.94$\pm$.16 | 77.92$\pm$.19 | 79.24$\pm$.32 | 79.09$\pm$.43 | 79.32$\pm$.19 | 79.19$\pm$.15 |
|  | 10 | 80.55$\pm$.13 | 1.00$\pm$.00 | 76.80$\pm$.41 | 78.40$\pm$.15 | 78.44$\pm$.22 | 79.65$\pm$.17 | 79.43$\pm$.16 | 79.34$\pm$.34 | 79.19$\pm$.45 |

Figure: Figure 2: Average distance to the consensus (i.e. the averaged model) during training for a heterogeneous population of $5$ models trained on CIFAR-100, either separately, with PAPA, PAPA-all, or our method WASH. Starting at consensus, models initially diverge from each other before converging back during convergence, mainly due to weight decay. Models trained with WASH have a smaller distance to consensus than ones trained separately; allowing them to be averaged with no performance loss. By training with PAPA-all (i.e. averaging to a single model every few epochs), models are not able to reach the same diversity as WASH between these averaging steps. Finally, the EMA of PAPA has a strong pulling effect towards consensus, resulting in a similar distance as PAPA-all. The jitter in the curve is due to the immediate distance reduction caused by the EMA steps.
Refer to caption: /html/2405.17517/assets/x2.png

Figure: Figure 3: 2D optimization example. We train 2 points with SGD on a simple loss function with 2 local and 1 global minima (upwards and downwards triangle). The two models are trained from two different starting points (plus signs). If the points are trained separately (yellow), they converge to their closest local minimum (yellow circles). By training with PAPA (blue), the points reach a consensus but then converge to one of the local minima (blue circles). By training with WASH (red), the shuffling (seen by the horizontal and vertical lines in the trajectory) allows more diversity in the optimization path, and the points both reach the global minimum (red circles).
Refer to caption: /html/2405.17517/assets/x3.png

### 4.2 Why do shuffling parameters help?

In this section, we propose to explain the improvement provided by our parameter shuffling over previous mechanisms such as BTM, DART, or PAPA, that focus on the averaging of parameters. We first show that models trained with WASH have a smaller distance to consensus than models trained separately. Then, we argue that despite this, WASH is a weak perturbation on the training of the models and that it incites diversity in the models.

#### Reducing distance to consensus.

To better analyze the diversity of the models trained with WASH, we propose to report the distance of the models to the consensus (the averaged model) during training, as a proxy of the diversity metric. (; ) showed that the difference between the Ensemble and the Averaged models depends on the distance between models. We present in Fig. [2](#S4.F2) the average distance of the models to the consensus, for models trained separately, with PAPA, PAPA-all, or with WASH. PAPA-all is a variant of PAPA functionally identical to DART. The idea is to average the weights every few epochs before letting the models diversify again.
We observe that WASH results in a consistently lower distance to consensus than the baseline, despite explicitly leaving the distance to consensus unchanged during the shuffling step, and shuffling only a small number of parameters. Thus, the smaller distance at the end of the training explains why the averaging of the parameters does not cause a decrease in performance. By comparison, PAPA-all (i.e. DART) results in alternating phases where models diversify before being averaged, and we observe that the models are not able to reach the diversity of WASH. Similarly, the EMA of PAPA has a strong pulling effect and results in average in a similar diversity as PAPA-all. Thus, we find that models trained with WASH have a higher diversity than models trained with PAPA or PAPA-all, while being close enough that averaging them does not cause a loss of performance. More generally, we show in Fig. [6](#S5.F6) of the Appendix that various interpolations of models trained by WASH result in a similar performance, showcasing that they all reside in the same loss basin.

#### Encouraging diversity.

WASH can be viewed as a weak perturbation on the models: the parameter shuffling affects more weakly the models than parameter averaging or the EMA of PAPA as only a few parameters are affected at a time, and the consensus distance is unaffected. Furthermore, the shuffling of parameters increases the diversity of the trajectories seen by the models. We showcase this with a toy example, by training jointly two points with SGD on a 2D loss function with 2 local minima and 1 global minimum, either training them separately, with PAPA, or with WASH. We represent the trajectories corresponding to each method in Fig. [3](#S4.F3). Training separately the two points makes them converge into a separate local minimum (i.e. a different basin). Training with PAPA allows the two points to reach a consensus, however they converge together into a local minimum. In contrast, by training with WASH, we show that both points reach the global minimum, as the shuffling allows for a greater diversity of points to optimize with. We provide more details in the Appendix.

### 4.3 Ablations

We present in this section ablations to better understand the effect of the parameter shuffling, varying the layer-wise probability adaptation, the base probability value, and the shuffling period.

Figure: Figure 4: Average distance to the consensus for different layer-wise adaptations of WASH, for different slices of the model’s parameters. Keeping the probability constant across layers ensures the lowest distance to consensus for the first quarters. Surprisingly, in the last quarter of parameters, despite initially starting with a higher distance to consensus, the ‘decreasing probability’ shows a lower distance to consensus later in training; despite shuffling being less frequent than the other schedules. The ‘increasing probability’ showcases how early layers are sensible to shuffling.
Refer to caption: /html/2405.17517/assets/x4.png

Figure: (a) Ensemble and Averaged accuracy for varying base probability values. We observe a phase transition as the base probability augments between a phase where permuting does not improve the averaged model accuracy and a phase where the ensemble accuracy is equal to the averaged model accuracy. Between the phases, the ensemble accuracy diminishes.
Refer to caption: /html/2405.17517/assets/x5.png

#### Layer-wise adaptation variations.

For WASH, we found that a probability decrease with depth provided the best results. We show in Tab. [4](#S5.T4) of the Appendix the performances for alternatives, with probability either staying constant or increasing with depth. We find lower performances for both alternatives.
In Fig. [4](#S4.F4), we report the models’ distances to consensus for all three schedules. More precisely, we provide the distances for different slices of the models’ parameters, indicating the effect of the shuffling depending on the depth. Shuffling equally all layers results in the lowest distance to the consensus as predicted, except in the last quarter of parameters. Here, surprisingly, our base ‘decreasing’ schedule shows a lower distance to consensus despite shuffling less frequently. We also observe a particularly strong effect of the shuffling for early layers, as the distance is more emphasized in the first quarter between the ‘increasing’ curve and the others.

#### Base probability variation.

We present in Fig. [5(a)](#S4.F5.sf1) the Ensemble and Averaged for different values of $p$, the base shuffling probability of the first layer. Rather than a smooth increase of the Averaged model accuracy, we observe a phase transition between a phase where the Averaged model accuracy is not improved by the shuffling and a sudden increase in the accuracy where it reaches the accuracy of the Ensemble. Just before the transition, the Ensemble accuracy is decreased, before increasing again back to its previous performance. The accuracy decreases only slightly even when increasing the shuffling probability to $1$, indicating the resilience of the models even to heavy shuffling.

#### Shuffling is beneficial at every step.

Finally, we propose to show the impact of the parameter shuffling at different steps of the training by varying the epoch at which the shuffling either starts or stops. In Fig. [5(b)](#S4.F5.sf2), we show that there is no improvement by having a warmup or slowdown period in parameter shuffling, indicating that all phases of the training are improved by WASH. Furthermore, stopping parameter shuffling early results in a much smaller loss of Averaged accuracy compared to starting late. In other words, shuffling at the start of training before models converge is more impactful as models may still reside in different loss basins.

## Conclusion

We proposed a novel distributed training method, WASH, aimed at training a population of models in parallel. These models are averaged at the end of training to obtain a highly performing model with accuracies close to the ensemble accuracy for a fraction of the inference cost. Our method requires a fraction of the communication cost of similarly performing techniques while obtaining state-of-the-art results for our weight-averaged models. We show that our novel parameter shuffling does not explicitly reduce the distance between models while increasing the diversity of the optimization paths seen by the population. Still, we observe that the distance between our models is smaller than training them separately, allowing them to be averaged at the end of training.

## Acknowledgements

This work was supported by Project ANR-21-CE23-0030 ADONIS, EMERG-ADONIS from Alliance SU, and Sorbonne Center for Artificial Intelligence (SCAI) of Sorbonne University (IDEX SUPER 11-IDEX-0004). This work was granted access to the AI resources of IDRIS under the allocations 2023-A0151014526 made by GENCI. Eugene Belilovsky also acknowledges the FRQNT New Scholar grant.

## 5 Appendix

#### 2D optimization example

The loss function we consider is a heavily simplified version of the Ackley function. With a minima in $(x_{m},y_{m})$ defined by

$$ $g(x,y,x_{m},y_{m},\lambda)=\exp{(-\lambda\sqrt{0.5((x-x_{m})^{2}+(y-y_{m})^{2}})}\,,$ (7) $$

the function we consider in our example is

$$ $f(x,y)=-10g(x,y,10,10,0.1)-5g(x,y,8,3,0.3)-5g(x,y,3,8,0.3)\,.$ (8) $$

This function has a 2 local minima in $(3,8)$ and $(8,3)$ and a global minimum in $(10,10)$. In all three cases, the starting points are $(0,5)$ and $(5,0)$. We compute SGD by first computing the exact gradient of the function and then adding Gaussian noise to the gradient. The learning rate is $0.1$ and we optimize for $1000$ steps. For PAPA, we consider $\alpha=0.99$. For WASH, the shuffling probability is equal for both coordinates and equal to $0.01$.

#### Interpolation heatmap

Here, we propose to display a heatmap showing the accuracy of more varied interpolations between $5$ models trained separately, with WASH, or WASH+Opt. We observe how WASH and WASH+Opt trained models converge to the same loss bassin, and that a large number of possible interpolations result in a high accuracy. The heatmaps are presented in Fig. [6](#S5.F6).

Figure: (a) Accuracy heatmap of the Baseline. The interpolated models’ performance is equal to random ones.
Refer to caption: /html/2405.17517/assets/x7.png

#### Layer-wise adaptation variants performance

We showcase in Tab.[4](#S5.T4) the performance of the three variants of layer-wise adaptations of WASH.

**Table 4: Test accuracies of WASH with variants of the shuffling probability per depth. Trained with a population of $5$ models on CIFAR-100 with a ResNet-18. The results show that permuting the first layers is more important than the later layers. Still, a constant probability across layers does not decrease WASH’s performance much.**
| Proba. at layer | Technique |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | to | L-1 | Ensemble | Averaged | GreedySoup | Best model | Worst model |
| $10^{-3}$ | $\searrow$ | 0 | 82.22$\pm$.38 | 82.15$\pm$.22 | 81.94$\pm$ 0.25 | 80.89$\pm$.03 | 78.80$\pm$.77 |
| $10^{-3}$ | $\rightarrow$ | $10^{-3}$ | 82.04$\pm$.19 | 81.94$\pm$.15 | 81.69$\pm$.23 | 80.60$\pm$.16 | 78.67$\pm$.89 |
| 0 | $\nearrow$ | $10^{-3}$ | 81.75$\pm$.35 | 81.37$\pm$.10 | 81.14$\pm$.20 | 80.08$\pm$.40 | 78.55$\pm$.70 |

#### Augmentations and regularization used

We follow the same data augmentations and regularizations used in [] for a fair comparison. We use Mixup (random draw from {0, 0.5, 1.0} for CIFAR-10/100 or from {0, 0.2} for ImageNet), Label smoothing (random draw from {0, 0.05, 0.1} for CIFAR-10/100 or from {0, 0.1} for ImageNet), CutMix (random draw from {0, 0.5, 1.0} for CIFAR-10/100 or from {0, 1.0} for ImageNet) and Random Erasing (random draw from {0, 0.15, 0.35} for CIFAR-10/100 or from {0, 0.35} for ImageNet).

For our experiments, we required a single A100 GPU for up to 14 hours to train up to a population of 10 models, and up to 40 hours for a population of 20 models. Similarly, we required 16 A100 GPUs to train in parallel a population of 5 models on ImageNet.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: I think Tab 1 is only a conceptual illustration of the communication efficiency of WASH. In real world, do the authors think that the randomness in WASH will make it harder for really efficient hardware implementation?
soundness: 3
strengths: 1. The paper is clearly written and the related work section is thorough and very informative.
2. Besides the main results in Tab 2 & 3, the authors also provided other informative empirical results to show why the proposed method is effective and superior to PAPA, like the observation of consensus distances and the 2D example.
summary: This work proposed a shuffling-based ensemble training algorithm that enables immediate weight averaging with good generalization ability without the need of further finetuning. The proposed method is motivated by the fact that a previous method called PAPA will gradually reduce the distance of the models to their consensus, thus hidering their diversity. In contrast, the proposed shuffling operation preserves the consensus distance and thus potentially improves model diversity and performance. Moreover, it also requires less communication and thus improves communication efficiency.
weaknesses: 1. The authors mainly compared the proposed method with PAPA. I understand that maybe this work is a close follow-up on PAPA and thus PAPA is the main competitor. But I checked the results in the PAPA paper and found that PAPA itself can be inferior to DART. So I think it would be better to include DART in the comparisons or at least discuss why not including it.

2. Advantages over PAPA are marginal most of times. The gaps are only significant with larger models and larger datasets.

3. While there are reproduction issues with PAPA on ImageNet, I think it would be better to also include those results in Table 3 and faithfully explain how the results went wrong, considering that there is still spare space for the main text.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Addressing the weakness mentioned above.

* The choice of a linear schedule for the shuffle probability is rather arbitrary. The ablation in the appendix examines a constant vs increasing vs decaying schedule and we see that the decaying schedule gives an improvement. Are there experiments with other schedules? I wonder if p could be tuned per layer based on the variance of the weights across the ensemble members.

* Why do the imagenet experiments require so much larger p than the cifar experiments? Is there an intuitive reason for this? Should we expect that as models and data scale, p will get closer to 1?

* What is the distance metric used in Figure 2?
soundness: 3
strengths: The paper proposes a very interesting and novel approach for keeping the ensemble members near each other in weight space, while keeping the communication cost low.

The method clearly works: There is a clear improvement in communication cost without a significant negative impact on performance. The image classification benchmarks show improvement over PAPA for multiple architectures.

The paper is very well presented: The paper also dives into the intuition of why WASH works. It explains the dynamics of the networks through a 2d toy example. It also showcases that WASH has a better diversity by examining the distances of the individual models from the consensus throughout training.

The experiments are thorough: The paper ablates three design choices: the layerwise adaptation of the shuffle probability, the magnitude of the shuffle probability and the idea of using a warmup period instead of enabling shuffling from iteration 1.
summary: The paper proposes a novel method, wash, for training deep ensembles with two key selling points:
* The individual models remain in the same weight basin, meaning that robust predictions can be made by averaging the weights of the individual members and executing a single forward pass.
* Wash requires a fraction of the communication cost incurred in each training step compared to similar approaches.

WASH works by shuffling a small percentage of the weights between the ensemble members in each training iteration. The result is that the members become robust to this augmentation and by the end of training, the weights remain near each other (in the same basin).

Experimental results show that WASH matches and often exceeds the performance of PAPA at a much lower communication cost. The models are evaluated on image classification benchmarks: Cifar10, Cifar100 and ImageNet, across three architectures: VGG16, ResNet18 and ResNet50.
weaknesses: I was disappointed that the experiments did not detail the impact of the reduced communication cost. The work is primarily motivated by the reduction in communication cost, but we don't see experiments showcasing the overall compute reduction during training or the improvement in training time. It is impressive that the communication cost is reduced by a factor of 1/2-1/200, but it is unclear whether this results in compute savings or faster training.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Here are my questions and further concerns:

- I understand that the homogeneous setup is described as not using data augmentation, as noted in line 315, which states "homogeneous setting with no data augmentations." However, the performance of the single model presented in Table 3, specifically the performance of GreedySoup—given that it achieves the best validation metric for a single model in a from-scratch training scenario—appears excessively high. Could it be that basic data augmentation techniques, such as random cropping with random horizontal flipping, were applied instead of no augmentation at all?

- According to Table 2, the accuracy of the single ResNet50 model on ImageNet is approximately 74%. Given that there are no modifications to the ResNet50 architecture and that training data augmentation was applied, this result seems quite unusual. For example, the V1 ResNet50 checkpoint from torchvision achieves around 76% when trained for 90 epochs with a batch size of 256 and without a complex training recipe. Since this paper employs advanced augmentation techniques—such as mixup, label smoothing, cutmix, and random erasing—we would expect the performance to exceed 76%. In fact, the PAPA paper (Jolicoeur-Martineau et al., 2024) demonstrates that a single ResNet50 model achieves an accuracy of 76.8% on ImageNet under these conditions. Therefore, there are concerns about the appropriateness of the experiments conducted, in addition to the earlier mentioned issues regarding the no-data-augmentation in the homogeneous setup.

- Is there a specific reason why the ImageNet results are absent from the homogeneous setup? Currently, the ImageNet results are only presented in Table 2 and not in Table 3.

- Figures 2 and 3 do not need such a large allocation of space, as they lack small elements that require close examination. As they stand, they occupy excessive space.

- The conclusion section is missing a section number; it should be labeled as "5. Conclusion" rather than just "Conclusion."

- How statistically significant is the claim in Appendix B.3 that WASH outperforms PAPA in terms of ECE for the CIFAR-100 scenario? Additionally, since Table 4 mentions the "optimal temperature," it appears that the ECE was computed after applying temperature scaling. How does the ECE compare without temperature scaling? Also, it might be worth considering NLL as an additional metric for evaluating predictive uncertainty.

- The results in Appendix C.5 suggest that using UniformSoup (i.e., averaging) yields better performance than GreedySoup for the multiple soup ingredients obtained with WASH. Do you have any educated guesses as to why this might be the case? Why could selecting only the weight-averaged ingredients based on validation metrics result in worse performance?
soundness: 2
strengths: - The concept behind the proposed WASH algorithm is straightforward and intuitive. Moreover, it incorporates various engineering considerations to improve practical performance, such as adjusting shuffling probability based on depth.
- Sections 1 and 2 provide a comprehensive overview of existing work on ensembling and weight-averaging in the fields of machine learning and deep learning.
- Figure 1 and Algorithm 1 present the proposed WASH algorithm in a clear and easily understandable way.
summary: This work proposes the WASH algorithm for weight-averaging multiple models trained in parallel under a from-scratch training scenario (i.e., each model starts with different random initializations). Naively trained models from these different initializations converge to different basins on the loss landscape, and thus they cannot be weight-averaged. The WASH algorithm addresses this issue by employing a simple strategy that mixes subset of weights between models during training, enabling them to converge into the same basin and thus be effectively weight-averaged.
weaknesses: - It seems the authors argue that the core of the WASH algorithm is communication efficiency. However, the current manuscript only discusses the theoretical communication volume presented in Table 1. According to main tables (Tables 2 and 3), the WASH algorithm does not show a significant performance advantage over the existing PAPA algorithm. To demonstrate the utility of the WASH algorithm, it would be helpful to present actual communication costs alongside the performance metrics. For instance, providing the wall-clock time required for training could enhance this analysis.

- The main tables (Tables 2 and 3) presents the results of weight-averaging for independently trained models, including UniformSoup (labeled as "Averaged") and GreedySoup. These results are redundant, as none of the combinations achieve effective weight-averaging in such cases. Furthermore, GreedySoup merely reflects the performance of the single model with the best validation metric. Notably, the GreedySoup algorithm relies on the assumption that fine-tuned weights, derived from the same pre-trained model, lie within a single basin suitable for weight-averaging—an assumption valid in transfer learning contexts but not applicable to the from-scratch training scenario in Table 3.

- The ultimate goal of the WASH algorithm is not only to facilitate weight-averaging of multiple models trained in parallel through parameter shuffling but also to ensure that the resulting weight-averaged model achieves superior performance. Therefore, it is crucial to compare the WASH algorithm with other high-performance single-model approaches, particularly those based on weight-averaging (such as single-trajectory strategies for from-scratch training, including Stochastic Weight Averaging by Izmailov et al. (2018) and the Lookaround optimizer by Zhang et al. (2023)). However, the only baseline currently presented is PAPA, along with the fundamental baselines of a single model representing a performance lower bound and an ensemble model (commonly referred to as deep ensembles) representing a performance upper bound.

- Recent trends have significantly shifted from from-scratch training to transfer learning approaches that involve fine-tuning pre-trained models. For instance, the model soup paper referenced in this study focused on fine-tuning both supervised and CLIP pre-trained vision models for ImageNet in 2022, yielding extensive experimental findings. Since this work is exclusively centered on a from-scratch training scenario, the lack of fine-tuning experiments with contemporary pre-trained models makes the results less compelling in 2024.

---

confidence: 3
contribution: 1
initial_rating: 5
presentation: 4
questions: Please refer to the weaknesses section.
soundness: 3
strengths: * I believe this paper addresses an impactful problem; model averaging with minimal communication cost is promising for democratizing ensemble methods in real-world tasks.
* The proposed method is convincing and intuitive. Moreover, the paper provides empirical analyses that reveal how the method works, which helps the reader understand it.
* The paper is generally well-written and easy to follow.
summary: This paper proposes WASH, a weight averaging method for obtaining a weight-averaged model with minimal communication cost during the training procedure. The key idea is to randomly shuffle a small fraction of the parameters. Experimental results demonstrate that the proposed method achieves state-of-the-art classification results with a reduced communication burden.
weaknesses: I do not find any major flaws in this paper, except in the experimental section. I would like to focus on the experiments on the ImageNet dataset, as I believe one of the main advantages of these model averaging methods is that they can be easily applied to large data regimes and large-scale models compared with probabilistic methods such as Bayesian neural networks.

* The accuracy of the proposed method is significantly lower than the PAPA baseline by 1.23 pp (75.80% for the PAPA ensemble vs. 74.39% for the WASH ensemble method).
* The paper only provides results with small-sized models (e.g., ResNet-50 and smaller). Including results with modern neural networks such as Vision Transformers and ConvNeXt would strengthen the paper.
* I wouldn't say the communication improvement is significant, as WASH still uses between 1/2 and 1/4 of the communication volume compared with PAPA. Providing scenarios or real-world numbers would improve the paper. For example, when we train ResNet-50 on ImageNet, how much communication bandwidth and training time can we save? Do we have to shuffle the models every step to achieve high accuracies?
* The performances of the proposed method, PAPA, and the deep ensemble baseline on ImageNet-1K are all lower than I expected. I anticipated the performance would be higher than 76% even without utilizing ensemble methods [1]. If we utilize modern data augmentations, the accuracy should be about 80% [2].
* It is a minor point, but I consider the performance difference between WASH and WASH+Opt as a weakness. Providing the results for one method and placing the others in the appendix may improve clarity.
* Also, solely providing an image classification task might not be sufficient to demonstrate the effectiveness of the proposed method.

I enjoyed reading about the method and the analyses. However, the performance improvements in terms of accuracy as well as communication cost are not groundbreaking when we consider the high standards of the conference.



[1] https://pytorch.org/vision/0.18/models/generated/torchvision.models.resnet50.html
[2] Wightman, Ross, Hugo Touvron, and Hervé Jégou. "Resnet strikes back: An improved training procedure in timm." *arXiv preprint arXiv:2110.00476* (2021).


---

**Aftre rebuttal.** I believe the proposed method has the potential for impact, but I am not fully convinced by the experimental results or their reliability as presented in the current manuscript.

Meta-review: This paper proposes an intuitive method to enhance diversity while maintaining the same basin by shuffling a small percentage of weights for ensemble learning. The primary advantage highlighted is improved communication efficiency. Although the overall scores show some divergence, the reviewers' concerns are consistent: while the basic idea is generally accepted, doubts remain about the experiments. These include unconvincing results on ImageNet, lack of comparisons with stronger baselines, and unclear advantages in computational time. The discussions on this paper have been thorough and ultimately converge toward rejection. We hope the reviewers’ comments and discussions will help the authors further refine and improve their method.

Decision: Reject