Linear Weight Interpolation Leads to Transient Performance Gains
Authors: Gaurav Iyer
Gintare Karolina Dziugaite
David Rolnick
Venue: Accepted by TMLR ()

Abstract: We train copies of a neural network on different sets of SGD noise and find that linearly interpolating their weights can, remarkably, produce networks that perform significantly better than the original networks. However, such interpolated networks consistently end up in unfavorable regions of the optimization landscape: with further training, their performance fails to improve or degrades, effectively undoing the performance gained from the interpolation. We identify two quantities that impact an interpolated network's performance and relate our observations to linear mode connectivity. Finally, we investigate this phenomenon from the lens of example importance and find that performance improves and degrades almost exclusively on the harder subsets of the training data, while performance is stable on the easier subsets. Our work represents a step towards a better understanding of neural network loss landscapes and weight interpolation in deep learning.

Published in Transactions on Machine Learning Research (09/2024)
Linear Weight Interpolation Leads to Transient Performance Gains
Gaurav Iyer
gourav. nourao-iyer@mill..oo quebec
McGill University
Mila Quebec Al Institute
Gintare Karolina Dziugaite
gkdz@ggoggle. com
Google DeepMind
McGill University
Mila Quebec Al Institute
David Rolnick
drotnick@mmila. quebec
Mclill University
Mila Quebec AI Instibute
Reviewed on OpenReview: https: // openreview. net/ forum? id- XGAdBXIFcj
Abstract
We Irain copies of a neural network on different. sets of SGD noise and find that linearly interpolating their weights can, remarkably, produce networks that perform significantly better than the original networks. However, such interpolated networks consistently end up in unfavorable regions of the optimization landscape: with further training, their per- formance fails to improve degrades, effectively undoing the performance gained from the interpolation. We identify two quantities that impact interpolated network's perfor- mance and relate our observations to linear mode connectivity. Finally, we investigate this phenomenon from the lens of example importance and find that performance improves and degrades almost exclusively on the harder subsets of the training data, while performance stable on the casier subsets. Our work represents a step towards a better understanding of neural network loss landscapes and weight interpolation in deep learning.
1 Introduction
Linear interpolations of neural network weights are of considerable interest in modern deep learning, for both
theoretical and practical purposes (Singh & Jaggi 2020 Li et al. 2023 Neyshabur et al. 2020). They have
aided our understanding of training dynamics and loss landscapes
FFranlle et al.] 2020 Sharma et al. 2024
Paul et al. 2022a and can improve performance at convergence (Tzmailov al. 2018 Matena & Raffel 2022 Wortsman et al. 2022].
Much of the prior work on this topic has focused on linear interpolations of network weights during late-stage training or at convergence (Wortsman et al.) 2022), or interpolating SGD iterates of network along single training trajectory (Zhang et al. 2019)). However, our understanding of the properties of such networks and their optimization remains limited in more general settings. The observations and investigations presented here attempt to address this gap and improve understanding of how linear interpolations evolve in the context of SGD noise throughout the training process.
We train copies networks on different sets of SGD noise (i.e data order and augmentation) on standard vision tasks. More specifically, we consider a network initialization that is trained for a small number of iterations k after which its copics A and are trained on different sets of SGD noise for s epochs.
With this context, our main findings and contributions summarized follows:

---

Published in Transactions on Machine Learning Research (09/2024)
ResNet20 CIFAR-10
ResNet56 CIFAR-100
0.88
0.625
0.86
0.600
Test Accuracy
0.82
0.550
0.80
0.525
0.76
Child Network
0.500
0.475
0.74
Averaged Netwark
Point Interpolation
10
12
14
16
18
20
22
24
26
28
30
10
12
14
16
18
20 Epoch
22
24
26
28
30
Epoch
Figure We train two copics of network on different sets of SGD noise, then average their weights and continue training on the resulting network. We find that test accuracy shoots up upon interpolation, but then precipitously drops and improvement stalls. The network initialization was trained for k = 10 epochs before being cloned into "child networks" A and B. A and B were then trained on different SGD trajectories for 10 epochs, before being averaged and trained further for 10 epochs.
We show that the performance of the network derived by interpolating and (referred to as the interpolant henceforth) varies significantly throughout the optimization process. Specifically, given display better accuracy than A or B the training and test set, if the value
k, the interpolant of s is small enough. Later in training (i.e. as becomes larger), interpolant performance becomes significantly worse.
We attempt to train interpolants derived early in training and find that the improvement in per- formance seemingly transient while they display better immediate performance, they either fail to improve or degrade over the next few epochs of training. Empirically, we find that standard learning rate schedule adjustments effectively provide no speed-up in optimization. We hypothesize that first-order optimization methods may be unsuitable for training interpolants and maintaining the accuracy boost.
We also note that trained interpolated networks perform just well standard networks when trained to convergence by interpolating carly, one can obtain models that perform better only early in training.
We analyze this phenomenon through the lens of example importance for good generalization, and find that "important" examples (those with higher EL2N scores as defined in Paul et al. (2021)) contribute significantly to the boost and subsequent drop in performance. Furthermore, there is nearly no drop in performance on the less important examples there can no increase performance
this case since networks perfectly learn these examples early in training. We hypothesize that our observations occur due different subpopulations of the data becoming stable to SGD noise at different points in training, noted by Paul et al. (2021)
2
Related Work
Network weight interpolation has attracted significant interest in recent work. Izmailov et al. (2018) intro- duccs Stochastic Weight Averaging, where averaging SGD itcrates along the same trajectory lead to better generalization. In similar vein, Zhang al. (2019h introduces the Lookahead optimizer, which interpo- lates set of "fast weights" to update the actual weights of the network, leading to better training stability
2

---

Published in Transactions on Machine Learning Research (09/2024)
and generalization. Wortsman et al. (2022) show that networks fine-tuned with different hyperparameter configurations often lie in the same loss basin, and that averaging them can lead to better robustness and performance. [lharco et al. (2023) introduce the notion of "task vectors", which capture task-specific direc- tions in weight space. Furthermore, they can be arithmotically combined with fine-tuned network weights and with each other to improve and/or degrade performance on specific tasks. Several other works support these observations (Matena & Raffel 2022 Li et al.) 2022; McMahan et al. 2017]. In contrast to these settings, our work focuses on interpolants obtained in the earlier stages of training.
Our work is largely based on the framework provided by Frankle et al. 2020), which introduced linear mode connectivity networks trained from the same initialization on different sets of SGD noise (after some short period of shared training) converge to the same linearly connected minimum.. We build upon their work by investigating interpolants earlier in training, instead of at convergence. [Mizadeh et al.) (2021) observe that networks starting with the same initialization are connected by linear, low-loss paths in the context of continual and multitask learning. Zhou et al. (2023) show that this notion of linear mode connectivity exlends to layer-wise feature maps as well. Permulation symmelries have been shown to align networks that they become linearly mode connected (Eutezari et al. [0222] Ainsworth et al. 2023 Sharma ct al. 2024). Garipov et al. (0118) and Draxler et al. (2018) show that networks trained from different initializatinns can be connected by non-linear low-loss paths.
Paul et al. 2021 propose the GraNd and EL2N scores to identify important examples early in training across different network architectures and training configurations. We make use of the EL2N score to analyze observations made in this work and relate it to example importance. Toneva et al. (2018) shows that deep networks learn "casy" data earlier in training and rarely forget this subset. Furthermore, these examples do not contribute significantly to final generalization performance. Our experiments support these findings we find that interpolated network performance is most variable on examples that are learned late in training. Baldock et al.] (2021) introduces the notion of effective prediction depth to empirically measure example importance and correlate it to the accuracy and speed of learning a given example.
3
Interpolating Weights Early in Training
3.1 Linear Mode Connectivity and Instability Analysis
We say that two neural networks are linearly connected if all networks along the linear path connecting the two trained networks in weight space have loss no larger than the end points (Frankle et al. 2020). The authors empirically demonstrate this through instability analysis, where two copies of a network are trained on different samples of SGD noise to convergence. If the two converged networks are linearly connected, the original network is said to be stable to SGD noise.
In our work, we cannot use instability directly as we will soon see, we often encounter interpolants that perform better than the networks they were derived from, and therefore taking maximum over performance errors will not suit our purposes. Instead, difference (increase or decrease) from the average error of the two networks that we are interpolating between. Unlike (Frankle et al.] 2020), we track the interpolant's performance throughout training rather than only at convergence.
consider similar metric but pay attention to the maximum
3.2 Interpolation Performance Throughout Training
Understanding the effect of linear interpolation early in training helps improve our understanding of linear mode connectivity since it is well-known that the early, noisy stage of training heavily impact training dynamics and performance at later stages (Goyal et al.) 2018 Jastrzebski et al. 2020: Gilmer et al. 2022 Paul et al. 20226 Fort et al. 2020). Our analysis can also help tease apart the properties and structure of neural network loss landscapes at different stages in the training process.
We start with a network that randomly initialized or trained for k stops. Then, two copics of this network, A and B, are trained on different sets of SGD noise for s epochs. Experimental details for the training setup can be found in Appendix A We will show the results of our experiment on CIFAR-10 (Krizhevsky 2009)
3

---

Published in Transactions on Machine Learning Research (09/2024)
with ResNet20 networks (He al. 2016) here (specifically Fig. in this context) additional results different datasets and network architectures can be found in Appendix E and onward.
iters
iters
0.9
0.8
0.7
0.6
0.4
0.3
0.2
10
20
30
40
50
60
70
80
20
30
40
50
60
70
80
k
400 iters
k 750 iters
0.90
Test Accuracy
0.80
0.7
0.70
0.6
0.65
0.5
0.80
3906 iters (10 epochs)
0
0.5
0.775
0.75
0.750
20
40
60
80
100
120
140
160
(in epochs)
Figure 2: Progression of lest. performance of ResNet20 networks resulting from linear weight interpolation of child networks (a and 1, where a is the interpolation weight) trained on different sets of SGD noise on CIFAR-10. The network initialization is trained for & steps before being cloned into child networks, after which they are trained on different SGD trajectorics for s epochs. As increases, interpolants perform just as well or better than the child networks for an increasing number of epochs before their performance degrades. Eventually, interpolants perform well all points in training until convergence, making the network "stable to SGD noise" à la Frankle al. (2020). The vertical dotted black lines represent learning rale drops by a factor of 0.1, following the standard training scheme for this task. Note that the interpolants are not trained further after interpolation their performance is simply recorded while child networks are trained on different sets of SGD noise. We repeat this experiment on 3 network initializations and provide error bars.
We observe that interpolant performance depends on both and s. When = 0, i.e. copies of the network are made at initialization without any training, interpolants perform extremely poorly. However, when the value of k is increased slightly, we make two observations:
(a) the interpolants slightly outperform networks A and B early in training, and
(b) the rate at which interpolant performance degrades becomes smaller.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Furthermore, as k scales, both these observations become more dominant - interpolants outperform the original networks for a longer portion of the training period, and their performance degrades more slowly. Note that the interpolants are not trained further- their performance is simply noted while networks A and B continue training on separate SGD trajectories.
Once & is sufficiently large, interpolant performance improves by more or less the same amount regardless of the exact value of s. This is important from a practical perspective we can potentially improve the performance of a network by simply training two copies on different sets of SGD noise for a small number of steps and interpolating their weights. Based on the scale and nature of the task, this may be preferable to training single network for multiple epochs. It can be reasoned that the performance of an interpolated network drops as s increases since networks A and B grow further apart in L2 distance in weight space when trained through different SGD trajectories. The same observation applies to other notions of distance as well (see Appendix G of Frankle et al. (2020)).
Observation (b) is partly captured by Frankle el al. (2020) they show that instability to SGD noise i.e. the performance gap between networks A and B and their interpolants at convergence gradually decreases as k increases. When k becomes sufficiently large, the interpolated weights perform better than approximately as well as networks A and B at all stages of training, making the network "stable to SGD noise". Intuitively, an increase in k may correspond to gradually determining the "basin fate" of the network (Fort et al. 2020). explaining the decrease in the rate at which performance degrades. However, we believe that the picture is still far from clear and does not explain observation (a) at all.
One can also observe that in the later stages of training, interpolants can do just as well but do not outperform networks A and B we address an aspect of this in Section
4 Training Linear Interpolants
From practical perspective, our findings so far lead to natural question: if lincarly interpolating the weights of networks trained on different sets of SGD noise leads to better performance (given appropriate values of & and s), can we efficiently leverage this to speed up optimization?
We address this by looking at the simplest case two copies of a network are made after epochs, trained on different sets of SGD noise, and then interpolated to produce a network with better performance. The rosulting interpolant is then trained further to convergence. From the previous section, one may observe that the degradation or improvement in performance is most significant for 0.5 will use this setting in our further experiments, though our observations generally hold for other reasonable values of a (i.e. not very close to 0 or 1).
We train ResNet-20 (He et al. 2016 networks on CIFAR-10 Krizhevsky 2009) with different values of & and and evaluate their test performance with focus on how performance changes at and after averaging network weights (see Fig. 3 and Fig. g In general, we find that while interpolating weights improves performance, it stagnates or degrades immediately over the next few epochs of further training. The subfigure corresponding to k = 1 and 8 = 1 is an exception to this rule but is not useful in pratice, since gains in performance very early in training are usually "easy" to achieve.
Simply dropping the learning rate after interpolation removes this observed stagnation /roo in performance, but negatively impacts the performance at convergence, which may be undesirable (see Fig. 12 As a compromise, we try warming up the learning rate after interpolation instcad and find that this does not cause any noticeable difference in the originally observed behavior performance continues degrade when the learning rate is dropped and gradually increased. These results can be observed in Fig. 4
These results suggest that networks obtained by weight interpolation consistently end up in regions of the loss landscape that are locally "better", but are not naturally amenable to further training through SGD the degradation in the improved performance heavily implies that training is stalled as the interpolated network escapes its present position in the loss landscape. Furthermore, this phenomenon does not seem to be specific to a particular model architecture or choice of hyperparameters (such as the learning rate schedule), hinting that what we observe is something more fundamental to the optimization process.
5

---

Published in Transactions on Machine Learning Research (09/2024)
10
0.8
Innerppllnnt Performance Avcroga Performance of Child Nots)
0.7
Test Accuraty
0.5
0.3
-2
0.2
Initialization Child Network A
Average Interpolant
Epoch
Epoch
Figure 3: (a) Zoomed-in view of test performance of child networks and interpolant before and after weight averaging, for and = 5 epochs. We observe rise in test accuracy followed by immediate drop over the next few epochs. In addition to training the interpolant, reference. (b) Difference between interpolant and child network performance, for various (sufficiently large) values of k. We consistently observe that the interpolant performs significantly better than the child networks at the point of interpolation, but there is effectively no difference after 1-2 epochs. See Fig. for variations.
also train the child networks independently
Our results so far raise two key questions:
Given sufficiently large values for tt and s, why does linear interpolation of network weights trained from the same initialization on different sets of SGD noise lead to networks that perform consistently and significantly better? Answering this can potentially lead to finding efficient heuristic neuristic/method /method to ascertain appropriate values for h and s.
Why interpolants derived in this manner not naturally amenable further training through SGD? What are the properties of interpolants and their immediate positions in weight space or the loss landscape that cause this, and if and how can this be prevented or mitigated?
5 Example Importance
We now consider the following questions: Are there specific examples the training set that are impacted more hcavily than others by weight interpolation? If so, is the same of examples that contributo significantly to the improvement and subsequent drop in performance?
Frankle al. 2020 note that networks become stable SGD noise early but at different points in training for different datasets during the training process. Furthermore, Paul et al. (2021) introduce metrics (sce Section 5.1 to measure example importance for training and achieving good generalization. The authors show that data subsets differentiated by these metrics become stable to SGD noise at significantly different points in training in general, the more important the example, the later in training stability is achieved with respect to it. Motivated by this, we explore how performance changes and after interpolation on subsets of data of similar importance according to this metric.

---

Published in Transactions on Machine Learning Research (09/2024)
Warmup for 10 iters
Warmup for 100 iters
Warmup for 1000 iters
0.850
0.825
0.800
Text Acturacy
0.775
0.750
0.725
0.700
0.675
10
15
10
15
Epoch
Epoch
Epoch
Network
NetworkB
Point Interpolation
Performance after Interpolation Linea
Performance after interpolation
Figure 4: Zoomed-in view of test performance of child networks before interpolation and the segght-veraged network at, and after interpolation, for variety of k 5 and 5 epochs note that the plot begins after the first k epochs. Test performance of the averaged network continues dropping even when linear warmup is employed after interpolation. The learning rate is warmed up over t iterations, where the learning rate is x I at iteration i, and is the original learning rate.
5.1 EL2N Score
Introduced by Paul et. al. (2021), the Error L2-Norm (EL2N) score of a training example is defined as the expected norm the difference between its label as predicted by the model and its ground truth label, where the expectation computed some carly epoch t over multiple trained network initializations. The authors demonstrate that the higher-scoring examples are more difficult to learn but are also more important for generalization removing these difficult examples from the training set had the biggest effect the final generalization of the model.
5.2 Interpolant Performance Improves and Degrades on Important Examples
We maintain the experiment setup from the previous section, where child networks (ResNet20) are trained on different sets of SGD noise, averaged, and then trained further. In Fig. 5 we track the performance of the child networks and the averaged network across equal splits of CIFAR-10'S training set. The data is split according to the EL2N scores of the training examples computed at epoch 10 of training cach split consists of 12500 examples, with Split 1 and Split containing examples with the lowest and highest FL.2N scores respectively.
As expected and shown by previous work, more important examples (as defined above) are indeed learned later in training. We also observe that primarily on Splits 3 and that the network's performance improves and subsequently drops. While the former may seem obvious (these are the only splits where there is space for performance to improve), the latter is not linearly interpolating the weights causes drop in performance almost exclusively the harder examples in dataset. Furthermore, the network maintains stellar performance the casier examples when trained further.
Our observations emphasize the connection between interpolated network performance and stability to SGD noise on subpopulations of data differentiated by example importance interpolants derived from networks trained on scparate SGD noise trajectorics perform better on data subpopulations that are still unstable SGD noise. On the other hand, once stability to SGD noise is reached on specific subpopulation, there is degradation in performance from weight interpolation.

---

Published in Transactions on Machine Learning Research (09/2024)
Split 1
Split
Split
Split
Subpopulation Accuracy
10 Epach
15
20
0
5
10 Epoch
15
20
15
20
10 Epoch
15
20
Epoch
Shared
Network A
Network
Point of Interpolation
Averaged Network
Figure 5: Performance of ResNet20 network on splits of CIFAR-10's training set differentiated by EL2N score, where Split 1 contains the least important examples, and Split 4 contains the most important ones. The rise and subsequent drop in performance due to interpolation is most noticeable in Split 3 and Split 4, which the network performs relatively poorly on. As noted in prior work (Paul et al. 2021), the network is likely lo be unstable to SGD noise with respect to Splits and 4 at such an early point in training, providing more evidence for the connection between our observations and stability to SGD noise.
6 Conclusion and Discussion
We have empirically demonstrated that when copies of a neural network are trained on different sets of SGD noise, the network resulting from a linear interpolation of their weights can perform better than the trained copies. We frame this observation picture of how networks bocome stable to SGD noise. Interestingly, when such interpolated network is further trained, performance fails to improve degrades over the next few epochs. We also demonstrate that strategies such as dropping warming up the learning rate do not remedy this observation linearly interpolated networks consistently end up in brittle arcas of the loss landscape that make the network perform better in the short term but impede further optimization. We analyze this phenomenon from the lens of example importance and empirically demonstrate that the observed improvement and degradation performance occur specifically in data subsets that are more important for generalization, providing further evidence for the connection between our observations and differences in stability to SGD noise examples differentiated by example imporlance.
extension of linear mode connectivity, offering a complete
We conclude by noting some limitations and directions for future work. The biggest question raised by our work is: Why does a simple linear interpolation of network weights trained on different sets SGD noise consistently show better performance, and how can we effectively leverage this improvement? While we connect our observations to prior work, we do not comprchensively answer this question. The strategics we explore mitigate the drop in performance when training the interpolants are based on previously used optimization techniques and tricks; due to the empirical nature of our study, cannot conclude whether alternative optimization method exists that could robustly optimize the interpolants, exploiting the drop in error to speed up training. It is also important to note that the current observations only apply to the standard vision tasks, and it is an empirical question whether they would extend to other domains.
Overall, our work provides insight into the behavior network weight interpolations during the carly stages of training and the same time, raises new questions address. We believe that this will help us better understand neural network loss landscapes and weight interpolation in the future.
8

---

Published in Transactions on Machine Learning Research (09/2024)
Acknowledgments
This research was supported by an NSERC Discovery grant and the Canada CIFAR AI Chairs program. The authors acknowledge material support from NVIDIA and Intel the form of computational resources and are grateful for technical support from the Mila IDT team in maintaining the Mila Compute Cluster. The authors would also like to thank Devin Kwok for helpful feedback and discussions in the early stages of this work, and Amr Khalifa for feedback draft.
References
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging Models Modulo Permutation Symmetries. In International Conference on Learning Representations, 2023. URL https:
//openreview.o wtt.:/orrrriiii
Robert Baldock, Hartmut, Maennel, and Behnam Neyshabur. Deep Learning Through the Lens of Example Difficulty. Advances in Neural Information Processing Systems, 34:10876-10889, 2021.
Luke N. Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey, CINIC-10 is not ImageNet CIFAR-10, 2018.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially No Barriers in Neural Network Energy Landscape. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1309-133 1018. PMLR, 10-15 Jul 2018. URL httpps:/pproceedinggs mlr. press/ prwss/v80/saaeee v80/ 17800/raxler180 htmll
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks. In International Conference on Learning Representations, 2022. URL https:://ppenrevieww wwt.:eeeeeetoooooiiddiiiiaiiaeee gytemkL
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepidch Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geomelry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861 5861, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Represenlations, 2019. URL httpps://oennrevier
net/fforum2iddxI1--BBBB..
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear Mode Connectivity and The Lottery Ticket Hypothesis. In International Conference on Machine Learning, pp. 3259-3269, PML.R, 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew G Wilson. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. Advances Neural Information Processing Systems, 31, 2018
Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Edward Dahl, Zachary Nado, and Orhan Firat. A Loss Curvature Perspective on Training Insta- bilities of Deep Learning Models. In International Conference on Learning Representations, 2022. URL https://openrevieet Dett 36vUs
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEFE Conference on Computer Vision and Patlern. Recognition, pp. 770-778, 2016.

---

Published in Transactions on Machine Learning Research (09/2024)
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidtt, Hannanch Hajishirzi, and Ali Farhadi. Editing Models with Task Arithmetic. In International Conference on Learning Represenlations. 2023. URL
jrj
Pavel Iamailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. A Averaging Weights Leads To Wider Optima and Better Generalization. arXiv preprint arXiv:1803.032. 054077 2018.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho", and Krzysztof Geras*. The Break-Even Point on Optimization Trajectorics of Deep Neural Networks. In International Conference on Learning Represenlations, 2020. URL http:::t/////oeereviie... http://///pppprrreeeeooooo/rrtrr wtt.:/runnteftoitittteoeo
r1g67C4KWB
Alex Krizhevsky, Learning Multiple Layers of Features From Tiny Images, PhD thesis, University of Toronto, 2009.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettle- moyer. Branch-Trese Teth---raii----e Merge: Embarrassingly Parallel Training of Expert Language Models. arXiv preprint arXiv:2208. 03306, 2022.
Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable Weight Averaging: Effi cient Training by Optimizing Historical Solutions. In The Fleventh International Conference on Learning Representations, 2023. URL http:://ppenreeeeeeeett/tttorr//
Michael S Matena and Colin A Raffel. Merging Models with Fisher- Averaging. Advances in Neural Information Processing Systems, 35:17703-17716 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication Eifficien Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 20-22 Apr 2017. URL https://proceee wwweeeddiinnmmm......mmmmmemee.m
Scyed Iman Mirzadch, Mchrdad Farajtabar, Dilan Gorur, Razvan Pascamu, and Hassan Ghasemzadch. Lin- ear Mode Connectivity in Multitask and Continual Learning. In International Conference on Learning Representations, 2021. URL htte:sss//ppaaaeereeeeee////oodd
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What Is Being Transferred In Transfer Learning? Advances in neural information processing systems, 33:512-523, 2020.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep Learning on a Data Diet: Finding Important Examples Early in Training, In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL https://openrerier pee/foormmiddUJjpp-----
Mansheej Paul, Feng Chen, Brett W Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dz- iugaite, Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? arxiv preprint STT::2210000444 2022a.
Mansheej Paul, Brett Larsen, Surya Ganguli, Jonathan Frankle, and Gintare Karolina Dziugaite. Lottery lickets on data diet: Finding initializations with sparse trainable networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 18918. Curran Associates, Inc., 2022b. URL paper
nwwrsss.eerprccoo cc/
wwnferenceeeecceeooooooeceeeeeee pdf
Ekansh Sharma, Devin Kwok, Tom Denton, Daniel M. Roy, David Rolnick, and Gintare Karolina Dziugaite. Simultancous Linear Connectivity of Neural Networks Modulo Permutation, 2024.
Sidak Pal Singh and Martin Jaggi. Model Fusion via Optimal Transport. Advances in Neural Information Processing Systems, 33:22045-220555 2020.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J Gordon. An Empirical Study of Example Forgetting During Deep Neural Network Learning. arXiv preprint Xie:18112 05159, 2018.
Mitchell Wortsman, Gabriel Ihharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Topes, Ari S Mor- cos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple line-tuned models iniproves accuracy without increasing inference time. In Interna- tional conference on machine learning, pp. 23965- 23998. PMLR, 2022.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead Optimizer: Steps Forward, 1 Step Back. Advances in Neural Information Processing Systems, 32, 2019.
Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going Beyond Linear Mode Connectivity: The Layerwise Lincar Feature Connectivity. arXiv preprint orXiv:2307.0800 08.888 2023.
A Experimental Setup
For all experiments, networks were trained with batch size of 128 for a total of 160 epochs. A stepwise learning rale schedule was employed with initial learning rate of 0.1 the learning rate is reduced by factor of 10 at epoch 80 and epoch 120. When linear warmup is employed over iterations, the learning rate is x at iteration where I is the initial learning rate. All datasets were also augmented with random crops to a size of 32x32 pixels after padding pixels to each border in the original image, with a constant value of 0, and horizontal image flips with a probability of 0.5.
B Results using LayerNorm
Relative to BatchNorm, LayerNorm does not enable large learning rates as a consequence, we cannot make use of the same training sctup. In order to draw fair comparison between the two cases, we lincarly warm up the learning rale to 0.1 over 100 iterations. We attempt to keep the number warmup iterations as small as possible since warmup is known to induce linear mode connectivity onset more quickly Frankle & Carbin (2019) Frankle et al. (2020). We note that the performance gain due interpolation and drop after interpolation is smaller compared to the BatchNorm case we leave more complete analysis of this for future work.
11

---

Published in Transactions on Machine Learning Research (09/2024)
iters
0.7
0.5
0.4
0.2
80
60
Thst Accuracy
mmm
0.6
apochs
Aloba
atcbe
3lsha 0.5
alsha 0.75
alsha - 1.0
(in epochs)
Figure Progression of test performance of networks resulting from lincar weight interpolation of child networks.
Split
Split
Split 3
Split 4
1.0
Subpopulation Accuracy
0.8
0.6
0.4
0.2
0
5
10
15
20
0
5
10
15
20
0
5
10 Epoch
15
20
0
5
10
15
20
Epoch
Epoch
Epoch
Shared pre pre-training
Network
Network B
Point of Interpolation
Averaged Network
Figure 7: Performance of a ResNet20 network on splits of CIFAR-10's training set differentiated by EL2N
score.
12

---

Published in Transactions on Machine Learning Research (09/2024)
k=1,s=1
k=1,s=5
==
0.7
0.8
0.6
0.6
0.5
0.5
0.5
Network
0.4
Network
0.4
Averaged Network
0.3
4
10
10
15
10
15
20
5,
5,
k 5,
10
0.8
Test Accuracy
0.75
0.65
0.7
0.65
0.60
10
k 10,
k 10, 5
10, s 10
0.80
0.775
0.80
0.750
0.75
0.725
0.70
10
10
15
Epoch
Figure 8: Test performance of child networks before interpolation and the weight-averaged network at, and after interpolation, for variety of k and s values in epochs.
13

---

Published in Transactions on Machine Learning Research (09/2024)
C
Relative Change in Accuracy and Error on Data Splits
Split
By Interpolation
After Interpolation
Accuracy
Error
A Accuracy
A Error
Split
0.00285
-0.98367
-0.0049
inf
Split 0.03166
-0.94480
-0.04067
27.21314
Split 0.13364
-0.7386
-0.14097
3.6074
Split 0.18621
-0.14679
-0.17065
0.19244
Table 1: Relative change (denoted by A) in accuracy and error due weight interpolation and just after weight interpolation, data splits based on difficulty.. The former is computed using the difference ween interpolant eccuracy/error and the mean accuracy /error of the child networks. The latter is computed using the difference between interpolant accuracy /error just after interpolation, and after training it for one epoch. The results are averaged over 5 runs. Note that A Error After Interpolation for Split is reported as inf since the interpolant achieves perfect accuracy (and consequently zero error) before being trained in nearly runs considered.
D Additional Results for ResNet20 + CIFAR-10
k =,s=1
k= 5=
k =0
0.8
0.7
Network
Network
Averaged Network
2
10
0.0
2.5
5.0
7.5
10.0
12.5
15.0
10
15
20
5,
Test Accuracy
0.8
0.5
2
10
0.0
2.5
5.0
7.5
10.0
12.5
15.0
10
15
20
k
10,
10,
k 10, s
10
0.6
0.0
12.5
15.0
Epoch
Figure 9: Zoomed-in view of lest performance of child networks before interpolation and the weight-averaged network at, and after interpolation, for a variety & and values in epochs. We consistently observe a rise test accuracy followed by an immediate pause improvement or drop over the next few epochs. Interestingly, the magnitude of improvement
to be independent of the exact values of k and
14

---

Published in Transactions on Machine Learning Research (09/2024)
k- 1,5
1,
1,5
0.8
0.8
0.7
0.7
0.7
0.6
Network:
0.6
Network
0.5
Averaged Network
0.5
k.
5,
0.825
0.825
Test Accuracy
0.800
0.800
0.800
0.775
0.775
0.775
0.750
0,750
0.750
0.725
0.725
0.725
2
10
10
15
10
15
20
10,
10,
10,
0.82
0.82
0.80
0.80
0.80
0.75
0.78
0.78
0.76
2
10
15
5
10
20
Epoch
The sctup remains the same as in Figure 9 except no augmentation is used i.c. SGD noise consists of only the randomness due to data orders. Our observations are still characterized by a rise and subsequent fall in performance removing augmentations from the training setup only makes this behavior more noticeable.
Figure 10:
Network
0.6
Network
0.6
Aweraged
2
10
10
15
15
20
k 5,
5,
5,
10
0.825
0.85
0.85
Test Accuracy
0.800
0.80
0.775
0.80
0.75
0.750
0.725
0.70
10,
10,
10,
10
0.850
0.86
0.800
0.82
0.80
0.80
0.775
10
15
15 20
Epoch
Figure 11: The setup remains the same in Figure 9 except child networks trained on non-overlapping, equally sized random splits of the training data. To be specific, the initialized network is trained on the entire train set in the & phase, the child networks are trained on splits in the phase, and the resulting interpolant trained on the entire dataset. Once again, our observations largely remain the same.
15

---

Published in Transactions on Machine Learning Research (09/2024)
drop Epoch 20, immediately after interpolation
LR drop Epoch 80
0.90
0.88
Test Accuracy
m WW
0.84
0.82
Network
0.80
Network
Averaged Network
Point Interpolation
20
40
60
80
100
120
20
40
60
80
100
120
Epoch
Epoch
Figure 12: Dropping learning rate carly training removes the performance degradation observed Fig. but affects final performance significantly.
wwwmmmw
0.00
MM Mmmm
Toan Acu/0Cc
0.85
Accuracy
0.75
0.79
20
40
50
70
10
20
30
70
M
Talt kcuu0
Teat Jocuracy
0.60
0.75
Ran
/000
Trah AcuraCt
0.65
Accuracy
mwm
3.60
ChidiA
0.70
Interpolant:
20
30
40
30
60
70
20
20
30
40
50
30
Epoch
Eparh
Figure 13: Extended training runs of interpolants with constant learning rate note that training accuracy non-monotonically yet consistently increases as training progresses, implying that the drop in performance after interpolation is not due to overfitting.
16

---

Published in Transactions on Machine Learning Research (09/2024)
E VGG-16 + CIFAR-10
Rers
100 iters
0.5
10
20
30
60
10
20
30
40
50
60
70
80
400 iters
750
0.2
Test Accuracy
0.6
0.2
0.5
10
60
70
80
0.500
0.850
alsha
alsha 03
0.800
aloba
alcha
0.775
160
epochs)
Figure 14: Progression of test performance of networks resulting from linear weight interpolation of child networks.
k ==51
1s=5
Network
Network
0.4
Averaged Network
10
0.0
2.5
5.0
7.5
10.0
12.5
15.0
10
15
20
5.5
Test Accuracy
0.8
0.0
7.5
10.0
12.5
15.0
20
k 10.
=1
k=
10.
k 10,
10
0.6
Epoch
Figure 15: Test performance of child networks before interpolation and the weight-averaged network al, and after interpolation, for a variety of k and s values in epochs.
17

---

Published in Transactions on Machine Learning Research (09/2024)
F ResNet56 + CIFAR-100
0.5
0.5
0.3
0.2
0.0
0.0
70
opochs
Test Accurayy
0.8
0.50
0.45
0.1
0.40
epochs
0.70
alsha 0
alsha 0.25
0.55
alche 0.75
aloba 05
aloha 1.0
(in apochs)
Figure 16: Progression of test performance of networks resulting from linear weight interpolation of child networks.
k= 5,5
3,
3, 10
Network
Network
Aweraged Network
10.0
15.0
20
k 5,5
5=5,s=5
5.
0.6
Test lest Accuracy
0.4
0.2
10
0.0
2.5
5.0
7.5
10.0
12.5
15.0
10
20
10
0.6
0.0
7.5
10.0
12.5
15.0
20
Epoch
Figure 17: Test performance of child networks before interpolation and the network at, and after interpolation, for varioty of and s values epochs.
18

---

Published in Transactions on Machine Learning Research (09/2024)
G ResNet20 + CINIC-10
iters
100 iters
0.7
2.7
0.6
0.
1.0
20
30
60
50
60
70
80
10
20
50
40
50
60
70
80
0.75
0.70
Test Accuracy
0.40
0.55
0.4
0.50
10
30
50
70
80
20
50
60
70
80
0.750
0.72
0.700
alsha
alsha 03
0.675
aloha
alcha
160
epochs)
Figure 18: Progression of test performance of networks resulting from linear weight interpolation of child networks on CINIC-10 (Darlow et al. 2018 with a ResNet20.
k=1s=1
k k=1s=5
== ==1s== 10
0.6
Network A
0.5
Averaged Network
20
Test Accuracy
0.7
0.5
0.4
10
0.0
2.5
5.0
7.5
10.0
12.5
15.0
10
15
20
k
10, 5
==
10.
10, s
10
0.6
0.4
12.5
15.0
20
Epoch
Figure 19: Test performance of child networks before interpolation and the weight-averaged network at, and after interpolation, for a varioty of k and s values in opochs on CINIC-10 with a RcsNet20.
19


Reviews:
[{"review_id": "mtEkvpV44k", "reviewer": "Reviewer_CxPM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an extremely well written and cohesive paper. Despite my lack of background knowledge in this area, I feel after reading it that I understand the problem, its significance, and at least some of the unresolved questions in this area.\n\nTaking the authors at their word about what work has been done previously, it appears that they are the first to investigate \"early\" interpolation of networks, including with the two post-interpolation training variants: continuing to train the constituent networks or training the interpolated network itself. Their findings are non-obvious (to me at least) and raise interesting questions about where interpolant networks lie in the loss landscape relative to their constituent networks.\n\nWhile this is an empirical paper with no theoretical results, the authors do a very good job of relating their findings to previously observed phenomena, offering compelling (if necessarily speculative) explanations of the interpolant networks' behavior.\n\nThe experiments are clear and appropriate for investigating the questions of interest, and the findings are certainly of interest for understanding neural network loss landscapes and convergence behavior.\n\nMy one quibble is that I'm slightly confused by the first \"key question\" on page 6: \"Why does linear interpolation of network weights trained from the same initialization on different sets of SGD noise lead to networks that perform consistently and significantly better? How does this depend on quantities $k$ and $s$?\" Looking at Figure 2, it seems that if $k$ isn't large enough, then the interpolant networks only perform better for a very small number of additional epochs $s$, and their performance continues to degrade in $s$. It seems like this claim, which is repeated in several places, needs to be appropriately qualified or at least clarified. I think the authors do this elsewhere (e.g. \"given appropriate values of $k$ and $s$\"), but it would be easy for a reader to lose track of this caveat.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the behavior of *interpolant* neural networks, which are defined as networks derived by linearly interpolating the weights from two constituent networks A and B. The interpolant network may then continue to be trained as a single network, or its constituent networks can continue to be trained separately.\n\nThe authors produce a collection of empirical findings, including that interpolant networks can outperform their constituent networks but that this performance can degrade with subsequent additional training after the point of interpolation. This behavior appears to be sensitive both to the timing of when interpolation occurs and to how much additional training takes place, and it varies depending on how \"hard\" samples are to learn. The authors relate these observations to previously described phenomena of linear mode connectivity and stability to SGD noise.\n\nAs a caveat to this review, I have no expertise in neural network interpolation and am unfamiliar with all but one of the works cited.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Some clarification of the above point, either in the paper or in the comments (if I've possibly misunderstood something).\n- Figure 2 caption: $\\alpha$ should be the interpolation weight, not the timestep.\n- End of section 3: observations (a) and (b) aren't actually labeled as such.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ndJqO7MbPi", "reviewer": "Reviewer_uKDG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The authors present interesting results pertaining to model averaging and the stability of trained models to SGD noise, both topics which have received significant interest in the community served by TMLR.\n- The experimental results are clean and convincing, and support the main results of the paper. The authors also perform several ablations on their results, investigating the impact of learning rate schedules.\n- The writing of the article is clear.\n- The authors put their work in the context of recent work on interpolating networks, and on example difficulty etc (although I should note that I'm not an expert in this literature!)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider vision tasks and analyse the performance of neural networks whose weights are obtained by interpolating the weights of two trained networks. Specifically, the authors train a network for a small number of SGD iterations $k$, after which two copies of the network are trained independently on the same data set with different realisations of SGD noise (i.e. data order and data augmentations) for $s$ epochs. This work thus builds on the earlier work by Frankle et al. (ICML 2020) who studied interpolants at convergence. \n\nHere, the authors show that early in training, interpolated networks outperform the network from which the interpolants were cloned from, but deteriorates later in training. Interestingly, they show that the both the initial boost in performance and the later deterioration depend on the same examples. They identify these examples using the Error L2-Norm score (EL2N) introduced by Paul et al. (NeurIPS '21). The EL2N score is the expected difference in $\\ell_2$ norm between model prediction and input label averaged over multiple networks after training for a few epochs $t$. Here, the authors show that interpolant performance improves and degrades on inputs with large EL2N score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the manuscript is ready for publication in its current form.", "broader_impact_concerns": "I do not think the work raises ethical implications that should be addressed separately.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UYJCUywbJM", "reviewer": "Reviewer_8TwW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The result is clean and well-explained. The observed phenomenon is novel and can have interesting empirical and theoretical implications\n\nWeakness: I feel that the result and its analysis are a little too shallow and straight forward and can be made much deeper. For example, the authors should (1) conduct experiments to show why there is a gain and why there is a drop? To me (and perhaps most of the audience), it appears that the drop is due to simple overfitting, is this true or not? The authors should offer an answer empirically\n\nBesides, the gain on split 3 is much more significant on split 4. There should also be an explanation\n\nLastly, I think comparing in the raw accuracies is misleading. The authors should report the **relative** change in the **prediction error**, which makes it possible and reasonable to compare between split 1 and other splits. It might well be the case that split 1 is more affected in this metric", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work empirically shows that that linear interpolation leads to a performance gain immediately after merging, and this gain disappears if one finetunes the merged model", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness section", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers were anonymous in recommending this paper for acceptance. Since this paper is on a topic that is near and dear to me, I gave it a full read too. I think it provides a very simple additional insight into what I think may be an important phenomenon - namely, that SGD doesn't necessarily find the best solution (in terms of either train or test loss) and that a very simple operation (averaging weights) can find better solutions. But interestingly, and "novely", this paper shoes that such gains go away if you train with SGD some more. The paper summarizes it best: "These results suggest that networks obtained by weight interpolation consistently end up in regions of the loss landscape that are locally “better”, but are not naturally amenable to further training through SGD". This is an interesting result, if a bit of a bite-sized one. The paper includessSome additional analysis into which datapoints are impacted by this phenomenon, which I think are also interesting though somewhat unsurprising. The paper is written clearly and is easy to understand.

I think the only weakness of the paper is its lack of depth (noted by one reviewer). Most notably, the experiments are in what is now a relatively "toy"/old-school setting - training ResNets on CIFAR-10. While there are certainly some applications of ML that involve training similarly small models on similarly small datasets from scratch, the setup is a pretty far cry from current standard practice (which mainly involves fine-tuning much larger models). I think some of the results in the paper likely carry over to more realistic settings, but it does make me wonder how much of the results are specific to their setting. Heaviest on my mind is whether data augmentation has anything to do with it, because if the datapoints are differently stochastically augmented in each of the child networks after "splitting" training, then they actually aren't seeing the exact same data. Showing what happens when there's no augmentation and/or when the data is actually truly different would be interesting. The latter setting is most similar to e.g. federated learning/branch-train-merge, so would be especially interesting to know. I looked to the appendix to see how thorough/"heavy" the augmentation was, and it says "augmented with random
crops to a size of 32×32 pixels" - this is a little confusing because CIFAR-10 images are 32x32, I assume you mean you did some zooming first before cropping? Totally separately, it also makes me wonder optimizers that specifically attempt to influence the "sharpness" of the solutions would impact this (because, in a cartoon sense, SGD will only find a solution at the "edge" of a flat basin; interpolating will move into the middle of it; sharpness-aware minimization perhaps would also move to the middle of the basin)?

I don't necessarily think the paper needs to go into all of the questions I mentioned above (though I would ask that you clarify the specific kind of augmentation used). I think the paper would be considerably stronger even with a small amount of additional experiments (e.g. with no augmentation, or with completely different data) and would be quite a lot stronger if it considered more experimental settings (say, language models, or sharpness-aware minimization). I think these experiments are optional for the purpose of acceptance, but I nevertheless encourage the authors to consider them to amplify the paper's impact/reach.
Decision: Accept (poster)