Revisiting Mode Connectivity in Neural Networks with Bezier Surface
Authors: Jie Ren
Pin-Yu Chen
Ren Wang
Venue: ICLR 2025 Poster (2025)
Keywords: mode connectivity
Bézier surfaces
loss landscape
deep learning
Primary area: interpretability and explainable AI

Abstract: Understanding the loss landscapes of neural networks (NNs) is critical for optimizing model performance. Previous research has identified the phenomenon of mode connectivity on curves, where two well-trained NNs can be connected by a continuous path in parameter space where the path maintains nearly constant loss. In this work, we extend the concept of mode connectivity to explore connectivity on surfaces, significantly broadening its applicability and unlocking new opportunities. While initial attempts to connect models via linear surfaces in parameter space were unsuccessful, we propose a novel optimization technique that consistently discovers Bézier surfaces with low-loss and high-accuracy connecting multiple NNs in a nonlinear manner. We further demonstrate that even without optimization, mode connectivity exists in certain cases of Bézier surfaces, where the models are carefully selected and combined linearly. This approach provides a deeper and more comprehensive understanding of the loss landscape and offers a novel way to identify models with enhanced performance for model averaging and output ensembling. We demonstrate the effectiveness of our method on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets using VGG16, ResNet18, and ViT architectures.

Published as a conference paper at ICLR 2025
REVISITING MODE CONNECTIVITY IN NEURAL NET- WORKS WITH BEZIER SURFACE
Jie Ren ? Pin-Yu Chen², Ren Wang
Illinois Institute of Technology, ?University of Wisconsin-Madison, IBB Research
ABSTRACT
Understanding the loss landscapes of neural networks (NNs) is critical for op- timizing model performance. Previous research has identified the phenomenon of mode connectivity on curves, where two well-trained NNs can be connected by a continuous path in parameter space where the path maintains nearly con- stant loss. In this work, extend the concept of mode connectivity to explore connectivity on surfaces, significantly broadening its applicability and unlock- ing new opportunities. While initial attempts to connect models via linear faces in parameter space were unsuccessful, we propose a novel optimization technique that consistently discovers Bézier surfaces with low-loss and high- accuracy connecting multiple NNs in a nonlinear manner. We further demon- strate that even without optimization, mode connectivity exists in certain cases of Bézier surfaces, where the models are carefully selected and combined linearly. This approach provides deeper and more comprehensive understanding of the loss landscape and offers a novel way to identify models with enhanced perfor- mance for model averaging and output ensembling. We demonstrate the effec- tiveness of our method on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets using VGG16, ResNet18, and VIT architectures. The codes are available at attps://qithut Httmm/MMMMLGGGop///eerreee
TIML
INTRODUCTION
Understanding the loss landscapes of deep neural networks (DNNs) is crucial for understanding the training process in deep learning (Keskar al. 2017). One of the most compelling phenomena observed within these landscapes is mode connectivity trained models (modes) in a model's pa- rameter space can be connected by paths of low loss (Garipov et al. 2018 Draxler et al., 2018). This phenomenon reveals that neural networks, even when initialized differently and converging to dis- tinct optima, may share underlying structural relationships that allow for smooth transitions between them in parameter space. Understanding how these optima are connected can reveal key information about network's ' performance across different tasks and datasets (Wortsman et al.] (022). well as its stability and susceptibility to adversarial attacks (Zhao et al. 2020; Wang et al. 2024 2023).
One widely studied form of mode connectivity linear mode connectivity, which describes a more specific scenario where the linear interpolation between two trained neural networks maintains roughly constant loss (Frankle et al. 2020). Importantly, the discovery of mode connectivity has led many applications such exploring the safety alignment of large language models (Peng al., 2024). where visualizing the safety landscape reveals how finetuning can compromise safety, and improved model robustness by finding paths in flat regions of the loss landscape (Zhao al. [0200] Wang et al. 2024. Tatro et al. (2020), making models less sensitive to adversarial examples. Addi- tionally, it enhances optimization efficiency by aiding escape from local minima and supports class incremental learning by enabling smooth connections between tasks without forgetting (Wortsman
al. 2022 Wen et 2023).
Despite the insights gained from mode connectivity, most existing studies focus on curve-based approaches that are inherently limited. These methods rely on information derived from just two endpoints-twe neural networks with fixed -retrrrttn-ssriiiiiig the exploration to a one- dimensional path. This limitation prevents the capture of richer structural information that exists
"Corresponding Author (Ren Wang: rwang74@iit.edu))

---

Published as a conference paper at ICLR 2025
beyond these endpoints and confines the search space narrow region within the parameter space. The core motivation of this work is to extend mode connectivity from curves to surfaces, thereby enabling broader and deeper exploration of the parameter space. By constructing surfaces instead of simple curves, we can integrate information from multiple models simultaneously, allowing for a more comprehensive search for optimal solutions. Surface connectivity offers a higher-dimensional perspective on the loss landscape, facilitating the discovery of models that exhibit better gener- alization and performance. In this paper, we present a novel learning framework that leverages Bézier surfaces in nonlinear manner to explore mode connectivity based on Bézier surfaces (See Figure ! for the loss landscape of Bézier surfaces before and after applying our method), offering deeper insights into neural network loss landscapes and opening up new avenues in model opti- mization. Additionally, We investigate the conditions under which the surface spanned by linear combinations of models exhibits mode connectivity. Finally, we introduce practical applications of our Bézier surface-based mode connectivity approach, including model averaging and output en- sembling, demonstrating its potential to enhance performance in these tasks.
Loss Surface
sown
300M
(b)
Figure 1: (a) The loss values the Bézier surface high before applying method; (b) The loss values on the Bézier surface are consistently low after applying our method. Each point on the surface represents the loss obtained by model. The Bézier surface is based on two parametric directions.
Contributions This paper makes the following key contributions:
Bézier Surface-Based Mode Connectivity Framework and Algorithm: This paper in- troduces a novel framework that extends mode connectivity from curves to surfaces using nonlinear Bézier surfaces. It also presents efficient algorithm that constructs these Bézier surfaces to connect independently trained models, systematically optimizing the surface to maintain low loss and high accuracy across the entire parameter space. This approach provides a more comprehensive exploration the neural network loss landscape.
Investigation of Linear Surface Connectivity: The paper examines when surfaces formed by linear combinations of neural network models exhibit mode connectivity properties, identifying key conditions for this to occur. This provides a foundation for understanding the limits of linear surface-based approaches.
Applications of Bézier Surface Mode Connectivity: (1) Model Averaging: The Bézier surface framework provides new view for averaging models across the parameter space, leading to improved model performance; (2) Output Ensembling: The approach enhances model ensembling by connecting multiple models via Bézier surfaces, creating more effective ensemble of models.
2 RELATED WORK
Neural Network Loss Landscape and Mode Connectivity. Understanding the loss landscapes of deep neural networks is critical for analyzing their optimization and generalization behaviors. Despite the non-convex nature of neural network loss functions, overparameterized networks of- ten converge to solutions with similar performance across different training runs. This observa- tion has prompted investigations into the geometry of these loss landscapes, with particular focus
2

---

Published as a conference paper at ICLR 2025
on the sharpness and flatness of minima. Models trained with large batch sizes tend converge to sharper minima, associated with poorer generalization compared to flatter minima found using smaller batches, emphasizing the relationship between the geometry of the minima and generaliza- tion abilities (Keskar et al. 2017). Mode connectivity explores the pathways in the loss landscape that connect different minima. Studies have shown that simple, low-loss paths often through weight space interpolations can connect independently trained models, indicating that minima are not iso- lated but reside within a connected manifold of low-loss regions (Garipov et al. 2018 Draxler al.) 2018). Techniques like Stochastic Weight Averaging (SWA) build on this concept by averaging weights along the trajectory of stochastic gradient descent (SGD) to find flatter minima that general- ize better (Izmailov et al., 2018). Furthermore, regularization techniques such as dropout and batch normalization have been shown to promote smoother landscapes, which facilitate mode connectiv- ity (Fort& & Jastrzebski] 2019). Mode connectivity has also been applied in various tasks, including transfer learning (Wortsman et al., 2022). robustness (Wang et al. 2024), safety alignment of large language models (Peng et al. (022)), and class incremental learning (CIL) (Wen et al. (023). For
instance, it has been explored to study and enhance l, robustness and to fine-tune pre-traing trained mod- els for improved performance on downstream tasks (Wortsman et al.] [022] Zhao al.) 2020 Wang
iet al.) 2024 2023). Additionally, mode connectivity has been utilized as a post-processing step after each CIL phase optimize continual learning performance (Wen et al. 2023). Several previous
works explored the extensions of mode connectivity. Some focused on identifying specific patterns
within loss surfaces of multiple neural networks (Skorokhodov & Burtsev 2019 Czarnecki et al.
2019). Others explored low-loss spaces using multiple simplices (Benton et al. (202]), which rely on localized, piecewise-
approximations, or modeled the loss landscape as collection of high- dimensional wedges (Fort et al.] (219)), which formulate the problem as a linear interpolation of manifolds and face challenges in scalability and interpretability. In contrast, our method focuses on nonlinear, Bézier surface-based mode connectivity, offering global and smooth mapping, enabling efficient optimization, and enhancing visualization. While one study examined geometry across multiple loss subspaces, its focus was pairwise mode connectivity and did not extend to con- structing surfaces with provable low-loss properties (Chen & Saidi). Despite these advancements, most works focus on curve-based approaches, which restrict exploration to narrow regions of the parameter space. Our method uniquely emphasizes the exploration of mode connectivity through surfaces, providing insights beyond the scope of these prior works.
Linear Mode Connectivity. While mode connectivity typi- cally involves non-linear paths between minima, intriguing possibility is the existence of linear paths that connect different minima without encountering high-loss barriers. Linear Mode Connectivity (LMC) refers to special case of mode connec- tivity where the low-loss path between two trained models is a linear interpolation (Frankle et al.) 2020). which exists when models share initial weights or tramiig data ordering. Fur- ther theoretical work has demonstrated that overparameteriza- tion neural networks makes linear low-loss paths more com- mon (Kuditipudi et al.) 2019), and techniques such as weight permutation alignment allow for LMC between independently trained models by reordering neurons to match across mod- els (Entezari et al. 2022). These findings have practical impli- cations for model merging and model ensembling, where lin- ear combinations of weights can result in averaging or ensem- bles that outperform traditional methods (Rame et al.) 2022] Ainsworth et al., 2023). While these works primarily focus on linear combinations between two models, we extend the explo- ration to settings involving more than two models, generalizing the idea of linear mode connectivity.
(a) Bézier curve
(b) Bézier surface
3 BÉZIER CURVE AND BÉZIER SURFACE
An illustration of Bézier curve and Bézier surface. The Bézier surface is based on two parametric di- rections u E [0. 1] and v E [0, 1]. with control points arranged in a grid.
Figure 2:
Bézier Curve. A Bézier curve is defined using a parametric equation where the parameter & varies between O and 1. The
3

---

Published as a conference paper at ICLR 2025
general form of a Bézier curve of degree is given by:
(tt) Σ
1 ( ((1-)))-PP
11
n!!
(I)
i! n 1)!
1=0
where P; represent the control points. When = 2, equation reduces to the most commonly used quadratic Bézier curve with three control points (Po. PI, and P2):
(t) (1 1) 26(1 P1 + P2
which is illustrated in Figure 2 (a). In the context of mode connectivity, P; are neural networks with the same architecture, and Btt) denotes curve in the parameter space.
Bézier Surface. A Bézier surface is an extension of the Bézier curve from two parametric directions, [0, 1] and u E [0, 1}, with control points arranged in grid. The surface is defined as:
B(u, v) = ΣΣ
(u
==
where Pt. are control points, and Bin and () are the Bernstein polynomial:
(u,(u) = 12 (") u0(1 u)" Bym(())
m
v -)) w)
(4)
The Bézier surface can be expanded by first summing over and then over i, yielding:
B(u,v) Σ Bjmm(v) 3=0
(v)
Σ i=0
Bi,
(5)
When u and equal to or 1, we get the four corner control points that always lie on the Bézier surface: B(0, 0) Poo, (00,) 1) Pom. B(1,0) = Pno: B(1,1) Prm. The Bézier surface is illustrated Figure 2 (b). Here, we use 9 control points by setting n = m = 2. Four them placed at the corners. The remaining five points do not lie directly points outside of the surface). Together, these nine models generate the entire parameter surface by varying and Similar to the Bézier curve case, P.j are neural networks in the context of mode connectivity (We use O,j to represent neural networks in the following sections), with Buu, v) denoting a low-loss surface in the parameter space. In this paper, our focus on the surface scenario.
the surface (see the red
4 MODE CONNECTIVITY ON SURFACES
Exploring mode connectivity through curves has already yielded valuable insights into the structure of the loss landscape. However, examining the landscape using surfaces provides even deeper in- sights, offering more comprehensive understanding of how models traverse these spaces. Unlike curves, surfaces enable the exploration of a broader parameter space and incorporate information from more models, which lead to the development of more advanced techniques applicable across different domains. In this section, we outline our approach to constructing them and discuss the advantages of using surfaces for mode connectivity. The central question we seek to answer is:
How can we find surface that encompasses models with both low training loss and high test accuracy?
4.1 A PILOT EXPLORATION IN LINEAR SETTINGS
To answer this question, we begin with pilot exploration in the simplest case, where the surface in the parameter space is spanned by a linear combination of three models. Consider these three models, 0,, 02, and 03, where the coefficients satisfy aj + 02 + az = I and a, can vary. These coefficients form surface parameterized by:
0(a1, 03)
0 + a2 O2 + a3 Oz
One can see that given fixed group of (a) a2 aa), & is fixed model on the surface. Therefore, there is no optimization or learnable variables involved in the surface generation.
4

---

Published as a conference paper at ICLR 2025
In experiments, train these three models indepen- dently with different initializations. This experiment is con- ducted the CIFARIO dataset using VGGI6 architecture. By varying o1; 022, and (:3, we generate a surface through the linear interpolation of these three model parameters. We then evaluate the accuracy of the models on this surface and present the results in Figure 3
Model (0. 9.
From Figure 3 it is evident that the surface generated by lin- ear combinations of models does not achieve low loss and high accuracy. In most cases, linear combinations of dif- ferent models fail create surface with these desirable properties. This raises the question: if simple linear combi- nations are inadequate, can we learn such a surface through nonlinear methods?
Model (1, 0,
Model
(0, 0)
tuples to the models surfice
Figure 3: Each point on the triangu- lar surface represents the accuracy of model obtained through lincar combi- nation of the three corner models pa- rameter space. The accuracies are low on the parameter surface, which is defined by the simple linear combination of three distinct models trained with different ini- tializations the parameter space.
4.2 MODE CONNECTIVITY ON A BÉZIER SURFACE
Here, we propose a novel optimization method that ad- dresses the surface construction question in nonlinear man- ner. Our approach leverages Bézier surfaces in equation (3) to connect multiple models, consistently yielding a smooth surface with low loss and high accuracy.
Connecting Procedure. We begin by explaining how the models are connected on the given sur- face") We redefine equation 3 from the perspective of model combinations:
(o v)
B B B.)
jm
1} are the learnable control points, except for 0oo. 0om. OnD. and 0um, which are weights of four independently trained neural networks. These four control points have fixed parameters and will always be on the surface. We will use the four control points as anchors of the surface, and the remaining control points are free to be optimized.
where = (Oig.i E ...1,,....11.
E 1,, (0.1,2,-...mm11),
Learning Objective. Next, we define the objective of the proposed surface optimization problem. The core idea is to learn Bézier surface where all points exhibit low-loss and high-accuracy prop- erties. Notice that the Bézier surface is defined by control points, with the four corner control points fixed, as they already demonstrate acceptable performance. By optimizing the remaining trainable control points 6, we can enhance the performance across the surface. We introduce our loss i(() on the entire surface (o(u, v) over uniform distribution below:
((0) - 100[(o] J. L(cbolu vo) v) du du B(g.r)) [L(ss v))],
Here, we specify the loss function L as task specific loss on the surface, we use cross-entropy loss to calculate different between the output of the network and the target classes for our following description. And distribution (u, v) on (u,e) u) E [0, 1 defined as:
1000
00j Du
|| I.
abba
x Oá0
Juv qu,(u, v) =
x
du du
(8)
dv
du
do
where Yue represents the normalized density of points on the Bézier surface, weighted by the gradi- ent of the parameterized surface. This density accounts for the varying distribution of points across the surface, ensuring the loss integral reflects the true geometric properties of the surface. Direct
We remark that our method can he easily generalized to more types of nonlinear surfaces. We choose Bézier surface because of its simplicity.
5

---

Published as a conference paper at ICLR 2025
computation of que is intractable for stochastic gradient- gradient-based optimization because it relies on the gradients of the parameterization Φø(u, u), where ( v) depends on learned parameters O.
Then the numerator of {() is the surface integral of the loss L(de) on the surface, and the denomi- nator is the normalizing constant of the uniform distribution on the surface defined by do(s)).
We aim to minimize this loss with respect to O. However, due to the computational intractability of directly optimizing this loss, we propose a more computationally feasible loss below:
(10)=/o (()) = f. /. L(do(u, d)) du ((w.,)))((001))) (L uu, v))]
(9)
where U(00 11²) represents the uniform distribution over [0, 1|2 This yields an expectation of L(do (u, 2)) with respect to a uniform distribution on (u, v) E [0. 1J2 while {(0) is an expecta- tion with respect to a uniform distribution on the surface. The two losses align, for instance, when (0(-:)) defines a bilinear surface with four corner control points and linear parametrization in u and v. To minimize (()). the intuitive way is sample ū, ē from the uniform distribution ([0, 1]²) and make gradient step for f with respect to the loss L(do ū)). This way, we obtain unbiased estimates of the gradients of E(0), as
VoL (i, 8)) ((..0)-0(001133) (oL(00 w)) Vol(O).
(10)
We repeat these updates until convergence. To achieve this in more efficient way, employ three-phase optimization process in the Learning Algorithm below.
Learning Algorithm. Minimizing equation necessitates updating all control models simulta- neously, which complicates the training process and is inefficient if done from scratch. Therefore, found that ensuring the edge curves formed by control points achieve low loss and high accuracy can help shape the surface, resulting in the entire surface that also exhibits low loss and high accuracy properties. To improve efficiency, we adopt an "outer-to-inner" optimization strategy, progressing from curves to the surface. The idea is that optimizing the surface becomes significantly easier once the key curves achieve acceptable performance. Therefore, we first focus on optimizing the two par- allel curves at the surface edges when u = 0 and 1 while v changes. (Phase 1), and then update the entire surface by refiingg all the trainable control points (Phase 2). The explanation of each phase is provided below, with the algorithm outlined in Algorithm I We also provide detailed algorithm in the Appendix.
Phase I: In this phase, we focus on optimizing the control points along the horizontal curves at the surface edges, parameterized by u when u = O and u = 1. This optimization performed iteratively over Ej epochs. As seen in equation (5). only the control points 0,r and 0nj Ū E (1,.... m 1))
contribute to the updates when u = 0 and u = 1, while the corner points remain fixed. Phase
corresponds to lines 2-4 in Algorithm
Phase In this phase, we optimize all control points for Eg epochs, except the fixed corner control points, by uniformly sampling over and and minimizing the expected loss across the entire surface. This process is repeated until convergence, producing a smooth Bézier surface that connects the corner points while minimizing the loss. Phase 2 corresponds lines 5-7 in Algorithm [
To demonstrate the effectiveness of our method, we present results using the VGG16 architecture on the CIFAR-10 dataset when n m = 2 (with control points for the surface). Figures 4 and 5 illustrate the loss and accuracy surfaces before and after training, respectively. Unlike Figure D here we show loss and accuracy values on the surface in 2D. Once the optimization is complete, all values are fixed, spanning the surface along w and v. Each point in the figures represents a loss
or accuracy value obtained by a model on the surface, evaluated on the test data. As shown, our method successfully optimizes the surfaces, transforming them from high loss (Figure 4 (a)) to low loss (Figure 4 (b)) and from low accuracy (Figure 5 (a)) to high accuracy (Figure (b)). These re- sults confirm that our method can effectively identify surfaces with low loss and high accuracy. One can also see from these two figures that the valleys the training loss surface correspond closely to the peaks in test accuracy, enabling the efficient selection of optimal models without extensive test set evaluations. Further discussions and additional results across various datasets and model archi- tectures provided in the Experimental Section. We further propose a more efficient method by updating only specific layers of the model, significantly reducing computational overhead. Details of this layer-specific optimization approach can be found in the Appendix.

---

Published as a conference paper at ICLR 2025
0.
0.8
0001244
Pirmeer
4.0014
0401354
622
02
000024
00
3.0
22
0.4
0.6
0.0
3.0
0. 5.2
0.4
2.5
0.0
3.0
Parametter
Parameter
Loss surface hefore training
(b) Loss surface after training
Figure 4: The losses on the Bézier surface decrease significantly after training. Each point represents a loss value evaluated on the test data. There are nine control points (n m = 2). Loss values at (u, v) = (0, 0), (0, 1), (1,0), (1 correspond to four corner control points Poo. O2o. Ouz, and 0z2.
Algorithm 1: Bezier Surface Mode Connectivity Algorithm (Summary)
Input: Initial weights of the four corner control points Qoo. Ano- 00m. and Am number of total epochs E = E₁ + Ez, number of random samples k. batch size B
Output: Optimized Bezier surface control points
Initialize control points 0
for epoch from to E do
Sample B training data in each batch and sample (O, 1) for times
Update curves (0, v) and s (1, v) from sampled points by minimizing equation
for epoch from Eg do
Sample B training data in each batch and sample u, UO, 1) for times
Update full Bézier surface de (u, v) from sampled points by minimizing equation
Accoracy
Accuracy Surface
Existence of Linear Sur- face Mode Connectivity. While we have shown that linear combinations of models often fail to produce a desirable sur- face, we have identified special cases where they can succeed. Specifically, when n = 4 and any two pairs of these four mod- els satisfy the linear mode connectivity property, it is possible to construct a Bézier surface where ery model exhibits low loss and high accuracy. By selecting 9oo. 9.o. 00m- and 9pm from training trajectory, we en- sure that the pairs of these models meet the linear mode connectivity condition. The interior control points O, are then obtained via linear interpolation of the four corner points, with O:y calculated as a weighted combination O,, = (1 - t) s)00 + t(1 s) + (1 t)s@om. + ts0nm. where t and i. The models the surface thus pure linear combinations of the four initial models. We demonstrate that this surface exhibits low loss and high accuracy using VGG16 on the CIFAR-10 dataset, shown in Figure 6 We select the four corner points from epochs 220,
Ed
(a) Accuracy surface before training
(b) Accuracy surface after training
Figure 5: The accuracies on the Bézier surface increase significantly after training. Each point in the figures represents an accuracy value obtained by a model on the surface, evaluated on the test data. There are nine control points (n m 2). Accuracy values (u, v) (0, 0). (0. 1), (1, 0), (1,1) are the values obtained by four corner control points Oon. O20, Oo2, and O22.

---

Published as a conference paper at ICLR 2025
200, 180, and 160, then combine them using the process outlined above. As result, the surface demonstrates similarly low loss and high accuracy, comparable to that of the four corner points.
4.3 APPLICATIONS OF MODE CONNECTIVITY WITH BÉZIER SURFACE
While this paper primar- ily focuses introduc- ing a surface mode con- nectivity method plore the neural network loss landscape, we also aim to demonstrate its potential for various ap- plications. In this sec- tion, we explore two key applications of our proposed Bézier surface- based mode connectivity framework: model aver- aging and output ensem- bling. By extending mode connectivity from curves generalization.
1000
(a) Loss values on the surface
(b) Accuracy values on the surface
Figure 6: The Bézier surface linearly spanned by the four corner points shows similarly low loss and high accuracy, matching the performance of the corner points. The four corner points are VGG16 models selected from epochs 220, 200, 180, and 160 during the training CIFAR-10 dataset.
surfaces, we enable more effective methods for model optimization and
Model Merging. The concept of model soups, first introduced by Wortsman et al. (Wortsman) al. 2022), involves averaging the weights of multiple fine-tuned models improve accuracy with- out increasing inference time. Unlike traditional ensemble methods, which aggregate the outputs of multiple models, model soups merge the weights of independently fine-tuned models, leveraging the observation that models trained on the same dataset task often reside in similar low-loss basins of the error landscape. This principle aligns with findings in linear mode connectivity, which sug- gests that multiple neural networks can be connected by paths of non-increasing loss between them (Garipov et al., 2018 Draxler et al. 2018).
Building on these ideas, our proposed Bézier surface connectivity framework acts model merg- ing method that enables a more thorough exploration of the model parameter space by averaging across surfaces rather than linear paths. Unlike conventional model merging methods such as Lion (ct al. (2024), which operate under the constraint that models lie within a single basin, our method allows for connecting and merging models across basins. This enables wider utilization of di- verse models. Since each control point lies in the parameter space and represents a model, surface- based averaging allows performing models that outperform those created by traditional weight averaging techniques. As seen in equation (6), each point on the surface represents model merged with different weights. In our implementation. we select the point on the surface that yields the best performance (Model Merging Accuracy). and compare it with the average accuracy of the four corner control models (Avg Initial Model Accuracy). As shown in Table we report the accuracy of the best pest-performing model on the surface. Our experimental results show that model merging through Bézier surfaces consistently outperforms the component models, achieving better generalization and accuracy. Ad- ditionally, Figure (b). you can see highland on the accuracy surface, indicating regions with higher accuracy than the four initial corner models. We remark that model merging can also be conducted in our linear surface mode connectivity setting.
connect more than models, leading to the discovery of new, high-
Output Ensembling. Ensembling techniques traditionally combine the predictions of multiple models to improve generalization. However, the use of simple linear averaging in these methods limits the potential to fully explore the parameter space. Recent advancements mode connectivity research (Wortsman et al. 2021 Rame et al., 2022) have demonstrated that models can be con- nected by paths of constant or non -oncirceeasing loss in parameter space, which leads to more robust ensembling strategies. Our Bézier surface-basee method expands upon these findings by connect- ing up to four independent models, allowing for a broader and more nuanced exploration of the parameter space. Unlike linear mode connectivity, which only considers paths between two models,
8

---

Published as a conference paper at ICLR 2025
surface-based connectivity explores multidimennionall relationships, resulting in models that gener- alize better on different tasks and datasets.
In approach, perform model ensembling by combining the output probabilities of different models positioned on the Bézier surface. To do this, we uniformly sample values and v at regular intervals (defaulting 0.1 in experiments) across the surface, creating diverse set of models. By averaging the outputs of these sampled models, we generate ensemble predictions. We found that this ensembling strategy consistently led to improved accuracy, surpassing the performance of the best individual model on the Bézier surface.
5 EXPERIMENTS
5.1 EXPERIMENT SETTINGS
The main goal of this section is to further demonstrate that the methods proposed in Section 4 successfully identify low-loss, high-accuracy surfaces connecting given modes across various ar- chitectures and datasets. In particular, we evaluate our method on three different datasets includ- ing CIFAR-10 (Krizhevsky & Hinton| 2009), CIFAR- 100 (Krizhevsky & Hinton 2009), and Tiny- Imagenet (Le & Yang, 2015) using ResNet18 (He et al.) (016), VGG16 (Simonyan & Zisserman 2015), and ViT model architectures. In this paper, we primarily use CIFAR-IO 10 and VGG16 as the default dataset and architecture for presenting loss and accuracy surface plots. By default, we set 72 m = unless otherwise specified.
5.2 MODE CONNECTIVITY ON BÉZIER SURFACE
0.
(a) CIFAR- ResNet18
CIFAR-100 VGG16
(c) Tmageetm VGG16
Figure 7: Our Bézier surface approach can effectively identify low-loss surfaces after training on various datasets and architectures. The loss values similarly small across the surfaces.
Results More Datasets & Architectures. We first evaluate the effectiveness of our Bézier surface-based mode connectivity method by examining the loss and accuracy surfaces after training on variety of datasets and architectures. The surfaces depicted in Figure illustrate our method's ability to identify low-loss landscapes, enhancing optimization in diverse settings. In Table I we present results on more diverse datasets and architectures, and show comparisons of Model Merging Accuracy, Average Surface Accuracy, and Average Corner Accuracy achieved by our method. The Avg Corner Accuracy refers to the average accuracy of the four corner control models, while the Avg Surface Accuracy represents the average accuracy of the sampled models the surfaces. The Model Merging Accuracy denotes the highest accuracy found on the Bézier surface. The results demonstrate that method consistently identifies low-loss surfaces (surface mode connectivity) across different settings. In the Appendix, we conducted additional experiments on surfaces with more control points. The results show that even with increased complexity, Bézier surfaces consis- tently maintain low training loss and high test accuracy properties across all configurations.
Linear Surface Mode Connectivity. We have shown the result of applying this approach to the VGG16 architecture CIFAR-10 in Figure 6 Here, we repeat the experiment using ResNet18 on CIFAR- 100. We select four corner points from epochs 220, 200, 180, and 160 during the training. Similarly, found that the linear combinations of models create surface with desirable properties,

---

Published as a conference paper at ICLR 2025
Table Comparisons of Model Merging Accuracy, Average Surface Accuracy, and Average Comee Accuracy obtained by our method across different datasets and architectures. The Avg Corner Accu- racy represents the average accuracy of the four corner control models. The Avg Surface Accuracy is the average accuracy of the models on the surfaces (by sampling). The Model Merging Accuracy the highest accuracy obtained on the Bézier surface. The results show that (1) our method can always find low-loss surfaces in different settings; (2) model merging through the Bézier surface can discover models with improved performance.
DATASET
MODEL
Model Merging Accuracy
Avg Surface Accuracy
Avg Corner Accuracy
CIFAR-I0
ResNer18
90.9%
89.3%
90.1%
CIFAR-10
VGG16
90.7%
89.8%
89.6%
CIFAR-10
VIT
70.2%
65,4%
70.0%
CIFAR-100
VGG16
70.8%
69.3%
70.7%
ImageemageNer
VGG16
51.1%
47.4%
50.7%
demonstrating high accuracy throughout, as shown in Figure B[). The loss surface is shown in the Appendix. Across both experiments, observed that the central region of the surface typically exhibited slightly lower loss and higher accuracy compared to the corner models, further supporting the utility of linear surface mode connectivity in certain settings.
5.3 MODEL MERGING AND ENSEMBLING BASED ON SURFACE MODE CONNECTIVITY
We further explore model merging and output ensembling to highlight the potential applications of surface mode connectivity.
Model Merging. For model Merging, we obtain the optimal model on the surface evaluate model merging. Jabbls[demoostrall the boost in validation accuracy for models obtained from the surface using our Bezier surface as model merging method compared points. We obtained the model with high accuracy. You may also check the validation accuracy surface after training shown in Figure 5 showing that there's s plateau with higher accuracy than four models on the surface. The central high accuracy region in Figure b)) also illustrates that linear surface mode connectivity can be utilized for model merging without the need for optimization.
the four corner control
Output Ensembling. For model ensembling, we sampled values of and v uniformly between O and at intervals of 0.1, generating diverse models along the Bézier surface. By averaging the output probabilities from these sampled models, we created an ensemble that consistently improved accuracy. In our experiments, we apply the Bézier surface- based model ensembling method on the CIFAR-10 dataset using VGG16 models, which results in notable accuracy improvements. Specif- ically, the average accuracy of the four independently trained models (corner points) was 89.96% while the best-performing model on the Bézier surface achieved 90.70%. After applying our ensem- bling technique. the accuracy further increased to 92%, demonstrating the effectiivnnes of surface- based model ensembling in enhancing generalization and outperforming both individual models and traditional linear interpolation methods.
CONCLUSION
In this paper, we extend the concept of mode connectivity from curves to surfaces, offering broader exploration neural network loss landscapes. By leveraging Bézier surfaces, we demonstrate the ability to connect multiple models with low-loss and high-accuracy regions, enhancing both model merging and optimization. Our approach not only uncovers deeper insights into the structure of the parameter space but also improves model averaging and ensembling performance. We validate the effectiveness of this method across various datasets and architectures, including CIFAR-10, CIFAR- 100, and Imay--ma
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work is supported by the NSF under Grants 2246157 and 2319243. We are thankful for the computational resources made available through NSF ACCESS and Argonne Leadership Computing Facility.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Merging models mod- ulo permutation symmetries. In International Conference on Learning Representations, 2023.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pp. 769-779. PMLLR, 2021.
Ting Chen and Tristan Luca Saidi. Examining the geometry of neural mode connecting loss sub- spaces.
Wojeiech Marian Czarnecki, Simon Osindero, Razvan Pascanu, and Max Jaderberg. A deep neural network's loss surface contains every low-dimensional pattern. arXiv preprint arXiv: 191.00999,,, 2019.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309 1318. PMLR, 2018.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation in- variance in linear mode connectivity of neural networks. In International Conference Learning Representations, 2022.
Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. Advances in Neural Information Processing Systems, 32, 2019.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Large-scale model ensembling through low-rank connections. In International Conference on Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pp. 3259-3266 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson Loss surfaces, mode connectivity, and fast ensembling of dons. Advances Neural Information Processing Systems, pp. 8803-8812, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Izmailov, AG Wilson. D Podoprikhin, Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 20/8, pp. 876-885, 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Torono, 2009.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems, 32, 2019.

---

Published as a conference paper at ICLR 2025
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Kai Lion, Lorenzo Noci, Thomas Hofmann, and Gregor Bachmann. How good is single basin? In International Conference on Artificial Intelligence and Statistics, pp. 4015-4023, PMLR, 2024.
Yun Peng, Pin- Yu Chen, Matthew Hull, and Duen Horng Chau, Navigating the safety land- scape: Measuring risks in finetuning large language models. arXiv preprint arXiv:2405. 17374, 2024.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamoniy, Patrick Gallinari, and Matthicu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821-10836, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Ivan Skorokhodov and Mikhail Burtsev. Loss landscape sightseeing with multi-point optimization. arXiv preprint (0t0:::911008666 2019.
Norman Tatro, Pin-Yo Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Op- timizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300- 15311, 2020.
Ren Wang, Yuxuan Li, and Sijia Liu. Exploring diversified adversarial robustness in neural networks via robust mode connectivity. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Ren Wang, Yuxuan Li, and Alfred Hero. Deep adversarial defense against multilevel-Ip attacks.
IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2024.
Haitao Wen, Haoyang Cheng, Heqian Qiu, Lanxiao Wang, Lili Pan, and Hongliang Li. Optimiz- ing mode connectivity for class incremental learning. In International Conference Machine Learning, pp. 36940-36957. PMLR, 2023.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Ari S Morcos, Ali Farhadi. Yair Carmon, and Simon Kornblith. Learning model ensembles with lower variance predictions. In Advances in Neural Information Processing Systems, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp.
PMLR, 2022.
Pu Zhao, Pin-Yu Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In International Conference on Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ALGORITHM DETAILS
We enclose the full algorithm below
Algorithm 2:
Bezier Surface Mode Connectivity Algorithm
Input:
Initial weights Oon- Oum, Om.. and Onn (fixed four end control points), number of epochs E1, E2, with epoch E + E2, learning rate 7, number of random samples
k. training dataset Do. batch size B
Output: Optimized Bezier surface control points
Initialize control points O,g with linear interpolation for i.j E {0, ..... n2 do (i, + (0, 0), (0, m)) (n,0), 0), (n, m) then Let t 1 and
Initialize 0g using linear interpolation as:
0 (1 (1 0)0 + t(l 3)0nn (1 t)s0om ts00mm
Define Quo, 0om. Pno. and prm as fixed endpoints for epoch in I to E1 do for each data batch Da Do do
for v in sampled U/ ~ UO,,7 1) do
10
11
(0, B, (v) Oor B1, (v) + 00m Bm. (v) B2(1, on Bo. (v) + On B (v) + Onm Bm (v)
for each x E D2 do
compute loss ((0) x/4 E., (Leash v)): Ltask (B2(1, (1,1))); at for each learnable Ozj, n0 Compute gradients V0.j- 30, Update O2 Oig 7/
15
17
for epoch in + E1 + Eg do for each data batch D, Do do
for v in sampled U; UO,, 1) and u in sampled u, ~ Uniform(O, 1) do
19
B(u, v) Σt EØ= O.g Bi,
20
21
for each x E D2 do compute loss T- E Trask (B(u,,
23
Compute gradients V0:j 50,
at for each learnable 0:j
24
Update O.j t Oiş Vo
25
empty sampled-points
A.2
LINEAR SURFACE MODE CONNECTIVITY
We construct a Bézier surface where the control points are the learnable parameters, denoted as Oij- The intermediate control points the surface calculated linear combinations of four corner models ooo. Ono. 0om, and O femm as follows:
Ozj (1 () 8) 000 1(1 s)) (1 l)s0om l. 1m
where f = and j This results in surface composed of linear combinations of the four n initial models.
We first applied this approach with the ResNet18 architecture the CIFAR-100 dataset, selecting corner points from epochs 220, 200, 180, and 160. As shown in Figure g the surface generated through these linear combinations exhibited low loss and high accuracy, comparable the perfor-
13

---

Published as a conference paper at ICLR 2025
mance of the corner models. This success occurs because the corner models happen to be linearly connectable for every pair of them.
Surface
Surface
1.
303100
3090445
26-
3031115
46-
3091802
10.00
3090/a
300489
60
0.2 0. 0.3
1.3
0.0
02
0.4
Paramoser
Parameter
(a) Loss values the Bézier surface. The face lincarly spans the four corner points, show- ing similarly low loss, matching the performance of the corner points. The corner points are ResNet18 models from epochs 220, 200, 180, and 160 during training CIFAR-100.
(b) Accuracy values the Bézier surface. This surface also spans the four corner points, show- ing high accuracy across epochs, matching the performance of the individual models.
Figure 8: Comparison of loss and accuracy surfaces on the Bézier surface spanned by four ResNet18 models, selected from epochs 220, 200, 180, and 160 during training on CIFAR- Both surfaces show consistency, with low loss and high accuracy distributed evenly across the surface, indicating that the corner models generalize well across the parameter space.
A.3 EXISTENCE OF LINEAR SURFACE MODE CONNECTIVITY
While linear combinations of models often fail to produce a desirable surface, we have identified special where success possible. Specifically, when n and each pair among these four models satisfies the linear mode connectivity property, a Bézier surface can be constructed in linear manner where every model exhibits low loss and high accuracy. By selecting 800. 6nD 90m and 6 pm from a training trajectory, the pairs meet the linear mode connectivity condition. The interior control points f-y are obtained through linear interpolation of the four corner points:
iij = (1 s)600 + t(1 s)0na + (1 -0)00m + ts8
where / = and _, The models on the surface are thus linear combinations of the initial models.
The Bézier surface S(u, v) can be expressed as:
S(u, v) Coo(u, Poo + Coz(u, Po + Czo(u, P2 + Cz2(u, (u, P2
where the coefficients (u, v) are combinations of the basis polynomials and the relationships between control points.
Computational Definition of the Basis Polynomials: The Bernstein polynomials Bi,(u) are defined as:
(u) (1)
u' (1)
u)m-i i 1......,
Similarly, the polynomials Bym (w) are defined as:
BB. (v) (m) oi - -m) m) == 1,.....mm
m
14

---

Published as a conference paper at ICLR 2025
In our case, we have 2, so:
Be,((u) = (1 - u)?: B1,2(u) = 2u(1 - u). (o,2(1) = v)?
B2.2(u) = u² (2) =
B1.((1) = 2n(1 - uv),
Coefficients for Each Corner Control Point: The coefficients Ciyl (u. v) are expressed in terms of the Bernstein polynomials:
1 + B1,2(u) Bo,2(1) + (1.2(1)B1,2()))
Coo(u, v) (b,2(u) (), + Bo.z(u)B1,22)
o) = B0.2(u)222(0 + 2 Bo.2(w) + ( (122 + (2,()))
C20(u,0) B2,(()) (1,2(u) + B1,) + (u)
1 Cza(u, v) (22(())2222 2 B1,z(u) B22(v) + (1,2(u) + (2)
One can verify that Coo v) + Coz(u, + C2o(u, v) + C22(u, =
These coefficients sum to one, ensuring that S(u, v) remains convex combination of the comner points. We demonstrate this surface's effectiveness using VGG16 on the CIFAR-IO dataset, achiev- ing low loss and high accuracy, comparable to that of the four corner points, shown in Figure
Relationships Among the Coefficients Although the coefficients Coo(u, e), Ca, v), C20 (u, u), and C22 (u, v) are functions of the parameters and v, they exhibit inherent relationships due to the properties of the Bernstein polynomials and the construction of the Bézier surface.
Sum-to-One Property: For all u, E [0, 1), the coefficients satisfy:
Coo(u, v) + Co2 v) + C2o(u, v) + C22(u, v) 1.
This ensures that S(u, v) convex combination the corner control points, and surface lies within their convex hull.
Symmetry Relations: The coefficients exhibit symmetry properties due to the symmetrical nature of the Bernstein polynomials:
Coo v) C22 u, v). Cuz (u, v) Czo(u, 1 v)
C2o v) Co2(1 u, v),
O22 (u, v) = Coo - v)).
These relationships reflect the inherent symmetry in the Bézier surface construction.
Interdependence of Coefficients: Any coefficient be expressed in terms of the others using the sum-to-one property:
Cy v1 Σ Cu(u,v).
(k.I)
(A.1)
This highlights the interconnectivity among the coefficients.
15

---

Published as a conference paper at ICLR 2025
Proportional Relationships: The ratios of certain coefficients are related through the pa- rameters u and v. For example:
Coo (u, v) (1 u) v)
2
C22 (U, v)
1100
Similarly, the ratio between Co2 and C20 is:
Coz(u, v))
(1 u)o
C2o(u, v)
uu1 v)
These proportional relationships demonstrate how changes a and affect the relative influence of each corner control point on the surface.
A.4 EXPERIMENTS SURFACES WITH MORE CONTROL POINTS
To further evaluate the scalability of our method, we conducted additional experiments on Bézier surfaces using 3x3, 4x4, and 5x5 control points the CIFAR-I0 dataset. These experiments were performed using a simple convolutional neural network (CNN) architecture, consisting of three con- volutional layers with kernel size and padding followed by max-pooling layers for downsam- pling. The fully connected layers include 356-dimensional hidden layer and final output layer that matches the number of classes. Dropout is applied after the convolutional layers mitigate overfitting.
These configurations allow test our method's ability to construct low-loss surfaces in increas- ingly complex parameter spaces, Table 2 summarizes the results of these experiments.
Table 2: Accuracy achieved with Bézier surfaces of different configurations on CIFAR-10.
Control Points
Avg Acc of Corners ( (%)
Highest Acc (%)
Avg Acc for Surface ( (%)
3x3
80.2
82.0
79.7
4x4
80.2
82.4
80.0
5x5
80.2
82.9
80.3
The results Table Lindicate that increasing the number control points leads to better ficxibility of the Bézier surface, resulting in better performance. Specifically, the highest accuracy achieved on the sampled points progressively increases as the number of control points grows from 3x3 to 5x5.
These findings demonstrate the robustness and scalability of approach, showing its ability to capture low-training loss and high-test-accuracy regions in the parameter space, even under more complex configurations. This strengthens the potential for method to extend larger models and datasets in future work.
A.5 MODEL ENSEMBLING COMPARISONS
To illustrate the advantages of our method for model ensembling, we conducted additional experi- ments comparing surface-based ensembling with traditional four-corner ensembling.
Jableissummariess the results our experiments on CIFAR- 10 with two architectures, VGG16 and R8 The results highlight the advantage of surface-based ensembling in capturing additional diversity and achieving superior accuracy compared to four-corner ensembling.
Table 3: Comparison four-corner ensembling and surface-bar baaface-based ensembling on CIFAR-10. 10.
Model
Four-Corner Ensemble (%) Surface Ensemble (%)
VGG16
90.4
92.0
ResNet18
90.1
92.7
The results demonstrate that surface-basee based ensembling consistently outperforms four-corner ensem- bling. By exploring the parameter space more comprehensively. Bézier surfaces capture diverse
16

---

Published as a conference paper at ICLR 2025
model predictions, leading to improved accuracy during model ensembling. This advantage is par- ticularly significant in scenarios where comnrr models reside in different basins, as traditional meth- ods fail to navigate the non-linear loss landscape effectively.
These findings reinforce the applicability and robustness of our method for model ensembling, show- casing its potential in diverse configurations and datasets.
A.6 LAYER-SPECIER OPTIMIZATION FOR EFFICIENCY
To improve computational efficiency, we conducted experiments where only a subset of the model's layers were optimized instead of the entire network. Specifically, we updated only the last convo- lutional layer and the fully connected layers in the Multi-Layer Perceptron (MLP) module. This approach significantly reduced computational overhead while maintaining competitive accuracy.
Due to time constraints, these experiments were validated on simple convolutional neural network (CNN) architecture. The network consisted of three convolutional layers (kernel size 3, padding 1), max poxopoooing for downsampling, and 256-dimensional fully connected hidden layer. The final layer matched the number of classes. The experiments were performed on a single NVIDIA 4090 GPU sampling 80 points per batch the surface. The results summarized in Table
Table 4: Comparison of accuracy and efficiency for layer-specific optimization CIFAR-10.
Updated Layers
Avg Acc of Corners (%)
Highest Ace (%)
Avg Surface Acc (%)
Time (26 Epochs)
All layers
80.2
82.0
79.7
47 min
Last conv + MLP
80.2
80.3
70,3
15 min
RESULTS AND TRADE-OFFS
The results demonstrate that optimizing only the last convolutional and MLP layers yields significant efficiency gains, reducing runtime by over 68% with a single NVIDIA 4090 GPU, while incurring moderate drop in surface accuracy. These findings highlight the potential of layer-specific optimiza- tion scalable approach for larger models and datasets.
B EFFICIENT EVALUATION OF LOSS AND ACCURACY SURFACES
B. USING BATCH-LEVEI APPROXIMATION FOR TRAINING LOsS SURFACE
During training, 80 points are sampled per batch, covering most of the regions on the surface. Evalu- ating the losses for just the last few batches provides reliable approximation of the full training loss on the surface. This observation suggests that smaller subset of batches can effectively represent the overall surface, significantly reducing computation costs without sacrificing accuracy.
In our experiment CIFAR-10 with the VGG16 architecture, evaluated the training loss surface using a subset data equivalent to four epochs. The results indicate strong consistency with evalua- tions conducted on the full training dataset. Specifically, when evaluated on the full training dataset, the valley of the loss surface was located at the (u, v) = (0.9, 0.2). On the subset evaluation, this shifted slightly (0.9, 0.1). Both points, however, lie within the same low-loss region, underscoring the robustness of this approximation. Similarly, the peak of the loss surface consistently remained (u, v) (0.4, 1.0) for both evaluations, as shown Figure 9
B.2 TEST ACCURACY AND ITS APPROXIMATION
The grid search for the accuracy surface occurs during the inference phase and is thus far less computationally intensive than training. To further optimize efficiency. a subset of the test dataset can be used for evaluation. In our experiment, using CIFAR-10 with the VGG16 architecture, the accuracy surface evaluated on subset of the test dataset showed a valley consistently located at (,, (0.4, 0), matching the results obtained from the full test dataset evaluation. The peak shifted slightly, from (u, v) = (0.9, 0.2) to (u, v) (0.9, 0.1), but both points fall within the same high-accuracy region, demonstrating the surface's structural robustness.
17

---

Published as a conference paper at ICLR 2025
Accuracy
Accuracy
90-12
Permeete
2.0
2.4
$1.41
$540
2.
6-4 0.6
0.6
1/3
$455
200
0.
022
0.6
5477
Parameter
(a) Accuracy surface evaluated on subset the test dataset. The valley at (0.4, 0), and the peak shifts slightly compared to the full dataset.
(b) Accuracy surface evaluated on the full dataset. The valley is at (0.4, 0), and the val- ley remains same for both subset and full test datasets.
Figure 9:
Comparison of accuracy surfaces evaluated on a subset (left) and the full test dataset (right) using CIFAR-10 with the VGG16 architecture. Both surfaces show consistent regions of high accuracy, demonstrating the reliability subset evaluation.
18


Reviews:
[{"review_id": "hk51hI7I7n", "reviewer": "Reviewer_LRVF", "summary": "This paper investigates connecting multiple models in parameter space through constructing appropriate surfaces. It is well-known that a simple linear hyperplane does not suffice and non-linear methods are needed. To that end, the authors propose using Bézier surfaces, where four points are used to represent the model parameters and nine other points in the parametrization are subsequently optimized such that uniformly sampled surface points also have low loss. The authors show that they can construct surfaces exhibit low loss everywhere, thus succesfully connecting multiple models with a single surface. They further show that the best point on the surface outperforms all the individual models and can thus be used to merge several models.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is very well-written and clearly explains the problem as well as the techniques that aim to solve them. I like that the proposed method simply consisting of Bézier curves remains rather simple.\n2. The experiments performed show quite convincingly that the proposed method succeeds in connecting multiple minima. The authors also investigate a variety of architectures, making the results stronger.", "weaknesses": "1. There seem to be quite a few related works missing that also explore the construction of surfaces to connect multiple minima [1, 2, 3, 4, 5]. The authors definitely need to add the listed papers to the related works and clearly articulate how theirs is different and what advantages it provides. \n2. For model merging, the authors do not seem to compare against any other method? It would be interesting to understand whether this technique allows one to leverage the diversity from all the points (that were obtained using different inits and shuffling). Standard merging always needs to be careful to end up in the same basin, and thus diversity of the points seems naturally reduced. Similarly for the output ensembling experiments, the obvious baseline of solely ensembling the four end points is missing. Does the surface really provide diversity beyond those four points? This is currently unclear with the provided experimental results.\n3. I think taking the best performing point on the entire surface is (1) a bit an unfair comparison and (2) very expensive to do as a dense grid of models needs to be evaluated on the test set. I think it would be more appropriate and efficient to compare against some sort of “mean” value on the surface. Does a Bézier curve admit a natural “centroid”? If yes, how does that one perform compared to the individual models? \n4. Another related work for model merging is [6] which explored how a given ensemble can be constructed within the same convex region, and thus also allowing to average weights while still profiting from diversity. It would be interesting to understand which approach works better. \n\n\n[1] Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling, Benton et al., 2021\n\n[2] Large Scale Structure of Neural Network Loss Landscapes\n\n[3] Loss landscape sightseeing with multi-point optimization, Skorokhodov et al., 2019\n\n[4]  A deep neural network’s loss surface contains every low-dimensional pattern, Czarnecki et al., 2019\n\n[5] Examining the geometry of neural mode connecting loss subspaces, Chen et al. \n\n[6| How good is a single basin? Lion et al., 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aQSzOgfYk8", "reviewer": "Reviewer_b47E", "summary": "This paper explores extending the concept of \"mode connectivity\" in neural networks from one-dimensional paths to two-dimensional surfaces using Bézier surfaces. Traditionally, mode connectivity demonstrates that two trained models can be connected by a low-loss path in parameter space. Here, the authors introduce a novel method to connect multiple models on a smooth, low-loss surface, broadening the potential for optimization and generalization. They detail an algorithm that constructs and optimizes Bézier surfaces to maintain low loss and high accuracy across various architectures (VGG16, ResNet18, ViT) and datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet). The study finds that nonlinear surfaces outperform simple linear interpolations, especially for model averaging and ensembling applications, ultimately enhancing performance in tasks like model merging and ensemble accuracy.", "questions": "1. **Ambiguity in the Central Question About \"Both Low Loss and High Accuracy\" (Line 195)**.\nThe central question on line 195 could benefit from greater specificity regarding \"low loss\" and \"high accuracy.\" Are the authors referring to training loss and testing accuracy? Given the generalization gap, distinguishing training and testing here would provide meaningful context. If both metrics are from the same set (either training or testing), the statement may be redundant, as low loss often correlates with high accuracy on that set. Specifying if this is about generalization (low training loss translating to high testing accuracy) could substantiate the relevance of this question.\n\n2. **Scalability Concerns for Optimization of Many Parameters (θ) in Equation 6**.\nEquation 6 implies a potentially extensive optimization of numerous control points (θ values) across the Bézier surface. This approach seems computationally heavy, especially for large models with millions of parameters. Could the authors discuss the scalability of this optimization? Is there any strategy to reduce the computational load or parameterize this approach efficiently to make it viable for larger architectures?\n\n3. **Justification for Selecting Models from Specific Epochs (Figure 6)**.\nFigure 6 shows models chosen from epochs 220, 200, 180, and 160. However, it’s unclear why these specific epochs were selected or why only a single training trajectory was used. Would models from other epochs, or from different training trajectories, produce similar results? Providing a rationale for these choices or showing comparative results could help validate the generalizability of this selection process.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and well-organized. It is easy to read.\n2. The visualizations and plots are also very clear, facilitating the understanding.", "weaknesses": "1. **Premature Claim of \"Low-Loss Curve\" in Line 161 (Bottom of Page 3)**.\nIn line 161, the paper refers to a \"low-loss curve\" as if it were already established, though at this point, neither the method nor specific criteria for determining a \"low-loss\" property have been introduced. Could the authors clarify what they mean by \"low-loss\" here and either postpone this claim until it is better supported or define it explicitly at the outset? Additionally, grounding this concept with a preliminary explanation or notation would improve clarity.\n\n\n2. **Rationale for Defining $q_{uv}$ in Equation 8 and Substitution with Uniform Distribution**.\nThe definition of $q_{uv}$​ in Equation 8 lacks an explanation of its theoretical motivation and why it can be replaced by a uniform distribution for practical purposes. What are the specific benefits of this choice, and how does this approximation impact the accuracy or reliability of the surface mode connectivity in experiments? A deeper rationale for this formulation would clarify its role in the model's performance.\n\n\n3. **Lack of Distance Quantification Between Corner Control Points in Loss Landscape Visualizations**.\nThe visualizations of the loss landscapes do not quantify or highlight the parameter distances between corner points (control points). If these control points represent very similar models with minor parameter variations, the diversity of the parameter space explored may be limited, especially when these models are trained under comparable conditions. How would the approach fare with intentionally diverse model initializations, varying training settings, or other augmentations? Such differences could test the robustness of the surface connectivity under broader training conditions.\n\n4. **Limited Impact of Experiments and Marginal Gaps in Results (e.g., Table 1)**.\nThe experimental evaluation relies primarily on relatively small datasets like CIFAR-10, CIFAR-100, and Tiny-ImageNet, which may limit the generalizability of the findings to larger, more complex datasets. Additionally, Table 1 shows only marginal improvements between the baseline and the model merging or ensembling results. Could the authors address how these findings might scale to larger datasets and discuss the significance of these marginal gaps, particularly given the computational overhead involved in the proposed approach? Expanding on the implications for practical, large-scale applications would enhance the impact of these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ztjAH3pYHb", "reviewer": "Reviewer_3hmG", "summary": "The paper explores the concept of mode connectivity in neural network loss landscapes, expanding it from traditional curve-based connections to surface-based connections. This approach offers a comprehensive way to merge models, enabling applications such as model averaging and output ensembling.", "questions": "See the above weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tExtending mode connectivity from curves to Bézier surfaces is a significant topic.\n2.\tThe proposed method is sound.\n3.\tWriting is good to follow.\n4.\tThe figure illustration is good in this paper", "weaknesses": "1.\tOnly evaluate the performance on small datasets. Large datasets like image-net should be included.\n2.\tLack of theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)