Do Deep Neural Network Solutions Form a Star Domain?
Authors: Ankit Sonthalia
Alexander Rubinstein
Ehsan Abbasnejad
Seong Joon Oh
Venue: ICLR 2025 Poster (2025)
Keywords: mode connectivity
loss landscapes
neural networks
parameter space
star domain
Primary area: optimization
TL;DR: We conjecture that SGD solution sets for DNNs are star-domains modulo permutation invariances.

Abstract: It has recently been conjectured that neural network solution sets reachable via stochastic gradient descent (SGD) are convex, considering permutation invariances. This means that a linear path can connect two independent solutions with low loss, given the weights of one of the models are appropriately permuted. However, current methods to test this theory often require very wide networks to succeed. In this work, we conjecture that more generally, the SGD solution set is a star domain that contains a star model that is linearly connected to all the other solutions via paths with low loss values, modulo permutations. We propose the Starlight algorithm that finds a star model of a given learning task. We validate our claim by showing that this star model is linearly connected with other independently found solutions. As an additional benefit of our study, we demonstrate better uncertainty estimates on Bayesian Model Averaging over the obtained star domain. Further, we demonstrate star models as potential substitutes for model ensembles.

Published as a conference paper at ICLR 2025
Do DEEP NEURAL NETWORK SOLUTIONS FORM A STAR DOMAIN?
Ankit Sonthalia Alexander Rubinstein'
Ehsan Abbasnejad
Scong Joon Oh
1 Tübingen AT Center, Universität Tübingen
2 University of Adelaide 3 Monash University
ABSTRACT
It has recently been conjectured that neural network solution sets reachable via stochastic gradient descent (SGD) are convex, considering permutation invariances (Entezari et al. 2021). This means that linear path can connect two independent solutions with low loss, given the weights of one of the models are appropriately permuted. However, current methods to test this theory often require very wide networks succeed (Ainsworth et al. 2022 Benzing et al. 2022). In this work, we conjecture that more generally, the SGD solution set a star domain that contains star model that is linearly connected to all the other solutions via paths with low loss values, modulo permutations. We propose the Starlight algorithm that finds star model of a given learning task. We validate our claim by showing that this star model is linearly connected with other independently found solutions. As an additional benefit of our study, we demonstrate better uncertainty estimates on Bayesian Model Averaging over the obtained star domain. Further, we demonstrate star models as potential substitutes for model ensembles. Our code available at https///giibbb.commaasssttalaall
INTRODUCTION
The learning problem for neural network is inherently characterized by non-convex loss landscape, leading to multiple possible solutions rather than a singular one. Efforts to comprehend this landscape and the of solutions have been ongoing.
A significant early discovery in this area (Garipov et al. 2018) demonstrated that almost any two independent solutions could be connected through a simple low-loss curve. While this finding highlighted the vastness of the solution set, other research has focused on its complexity. For instance, permutation symmetries allow neuron positions in different layers be jointly swapped without changing the function represented by the neural network (Brea et al., 2019 Singh & Jaggi 2020 Ainsworth et al. 2022 Guerrero Peña et al.) 2023). Entezari et al. 2021 proposed that when accounting for these symmetries, the solution set found by stochastic gradient descent (SGD) essentially becomes convex, i.e., any pair of independent solutions can connected through low-loss line segment after appropriate permutation is applied to one of the models. Notably, Sharma et al. (2024) investigate the stronger property of simultaneous linear connectivity, wherein permuting given model linearly connects it to several other models. However, recent works like Ainsworth et al.] (2022) study convexity in the context of the formulation in Entezari et al. (202T). Our work therefore refers to their conjecture as the "convexity conjecture" (Conjecture while acknowledging that other, stronger forms of convexity can be formulated.
The convexity conjecture has faced challenges. Subsequent studies (Juneja et 2022 Benzing et 2022] Ainsworth et al.) [2022] Altintas et al. 2023 Guerrero Peña et al.) (023) revealed that even after algorithms, two distinct solutions in the parameter space might still be separated by a high loss barrier (Frankle et al. 2020; Entezari et al. 202T) upon performing linear interpolation. These studies attribute this discrepancy to various factors, including network
the application of permutation-finding
depth and width, dataset complexity (Ainsworth et al. 2022) and high learning rates (Altintas et al.)
"Corresponding author; anki it. sonthal lauuni -tuebingen....

---

Published as a conference paper at ICLR 2025
2023). Theoretical investigation (Entezari et al. 2021 Ferbach et al. 2024 suggests that in general, the conjecture needs wide networks to hold.
In response to these findings, research introduces the star domain conjecture. We propose that more generally, solutions in deep neural networks (DNNs) form a star domain rather than a convex set, modulo permutation symmetries. A star domain is set A with at least known as a star point, ag E A that connected to every other element in A. A convex set is specific instance of star domain. The star domain conjecture thus proposes that in cases where convexity (Entezari et al. 202T) does not hold, a weaker form of convexity (i.e. star-shaped connectivity) still exists.
special element,
high loss
The star domain conjec- ture is still a stronger as- sertion than mode connec- tivity (Garipov al. (0018) which states that any two models Oa and On can be connected through possi- bly non- con-iinear path in the so- lution space. As special case, this path could be as simple
01
05
0
02
04
0g
0
03
piece-wise lin- car path comprising a third point Oe such that (0,,0e) and (0g, Oc) are linearly connected. In contrast, our conjecture implies that all pairs of solutions are interconnected via shared third solution, the star point, which common to all solution pairs: 30c such that v0 OB E S, (0,, 0c) and (0B, Oc) are linearly connected, where S is the solution set.
Convexity conjecture (Entezari et al.) holds only for wide networks.
Star domain conjecture (ours) holds for networks.
low loss
We substantiate our star domain conjecture with empirical evidence by introducing the Starlight algorithm to identify candidate star model for given learning task. Starlight finds model that is linearly connected with finite set of independent solutions. We demonstrate that these star model candidates have low loss barriers with arbitrary set of solutions that were not used in constructing the star model candidates. This provides strong evidence that there exist star models that are linearly connected with other solutions.
In addition to validating the conjecture, our research delves into the distinctive characteristics of star models. We find that sampling from the star domain for Bayesian Model Averaging (BMA) leads to better uncertainty estimates than ensembles. Additionally, we demonstrate star models as a possible substitute to model ensembles, with lower inference time and memory footprint. These differences highlight the potential advantages of star models in various neural network applications.
We summarise contributions:
1. The star domain conjecture for characterizing connectivity in neural network solution sets.
2. The Starlight algorithm for identifying star model for gradient-based basedd learning task.
3. Analysis of practical benefits shown by the star models.
RELATED WORK
We introduce the relevant development of findings toward the understanding of DNN solution sets.
Mode Connectivity. Garipov et al. (0018) and Draxler et al. (2018) concurrently discovered mode connectivity. Goommae et al. (2018) soon followed, showing non non-linear connectivity even between networks obtained using different training schemes. Kuditipudi et al. (2019) explained mode connec- tivity via dropout stability and noise stability. [enton et al. (2021) went on show that there exist not only simple paths, but also volumes of low loss, connecting several DNN solutions. These works focus general, non-linear connectivity, while study stricter condition, viz., linear connectivity.
2

---

Published as a conference paper at ICLR 2025
Linear Mode Connectivity (LMC). Frankle et al. (2020) were the first to study LMC. Later, Entezari
ct al. (2021) proposed that SGD solution sets are convex modulo permutations, while Singh Jaggi (2020); Ainsworth et al. 2022); Guerrero Peña et al. (2023) introduced "re-basin' methods, L.e. methods for bringing different solutions into the same basin. Recent work (Ainsworth et al. 2022
Altintas et al. 2023, Benzing al.) 2022) also noted failure cases for LMC, while |eerbach al.
(2024) theoretically investigated convexity for sufficiently wide nets. Our analysis builds upon these findings and reveals evidence for weaker property, viz., star-shaped connectivity, in cases where convexity does not hold.
Star-shaped connectivity (SSC). Zhou et al. (2019) show that SGD follows an epochwise star-convex path but they focus on the optimization trajectory rather than the structure of the loss landscape. Annesi al. (023) provide valuable insights for SSC the loss landscape for the simple case of the negative spherical perceptron. In contrast, we consider more complex models and learning tasks, and propose novel verification method for SSC. [in et al. (2024) explore star-shaped connectivity to finitely many solutions. Our work, in contrast, additionally considers permutation invariances (Ainsworth al. 2022 Entezari et al. 2021) and provides evidence that star models trained this way might be connected to infiitely many other solutions.
Practical Applications. Mode connectivity has found applications in model fusion Garipov et al.
2018: Singh & Jaggi 2020),, adversarial robustness (Zhao al. 2019: Wang al. 2023), continual learning (Mirzadeh et al. 2020 Wen et al.) 2023). and federated learning (Wang et al., 2019) Ainsworth al.] (022)- In contrast, work focuses understanding the surface of the loss
landscape. However, we also explore potential applications, e.g., Bayesian Model Averaging.
3
THE STAR DOMAIN CONJECTURE
In this section, we introduce our star domain conjecture before discussing its practical applications in Bayesian model averaging and model ensembling in Section
3.1 BACKGROUND: THE CONVEXITY CONJECTURE
Here, we formally state the convexity conjecture, starting with basic notations. A neural network is function f.(.) parameterized by f E O, where θ is the parameter space. Given dataset D, formulate non-negative loss L(A) = L (0; D) 2 0 and minimize L (0) to find a solution in e. The solution set is S {0 (0) x 0).
The loss barrier was first defined by Frankle et al. 2020). We use the formulation in Entezari et al. 2021), i.e., the barrier between 0, 0g E θ is B (0 0B) max 0.1] (0 A, 0B), where
E.(0,, C ((1
0^ + 0n) ((1 - (0A) + (On))
the differenne between the loss value t, and the linear interpolation of the losses at the end end-points. Two solutions A,, 0g E θ are said to be linearly mode-connected, or LMC (Frankle et al. 2020 when their loss barrier approximately zero: B (0 (A) 0
B(8 0p)
C(9A)
The convexity conjecture is constructed upon parameter space where the permutation symmetries are factored out. permutation invariance (Brea et al.] (019)) can be formulated as equivalence relation ~ between
C(0
two points A 0g in the parameter space such that O ~ 0g if and only if there exists permutation of the parameters such that (0)A) Og and the functions represented by them identical: for (x) = fag (x) for all æ. Given two points 0A and 0p, we look for the permutation of 0g that
connects it to A (or vice versa) with low loss barrier as possible (Ainsworth al. 2022 Entezari
et al. 2021 Guerrero Peña et al., 2023). A winning permutation (Entezari et al.) 202 for models 0, and 0, is defined as
π
= arg min B (0(4)).0g) REPs,
9 0
where Po = ( 0} is the set of all punetion-preserving permutations of
3

---

Published as a conference paper at ICLR 2025
Conjecture 1. Convexity Conjecture (Entezari et al. 2021 Let S be the set of GGGrreechable solutions for a deep neural network f(0) trained for a certain task. Let A 0g E be two solutions. Then, there exists a minimum width h such that if (() is wider than h, then with high probability, (0B) such that O and 0g are highly likely be linearly
Og can be permuted obtain 0B- 0n~
mode-connected, i.e., B (0B.00 0A) 0.
We refer to this as the (quasi-) convexity conjecture, because, by definition, a convex set is precisely a set where the line segment between any two elements is included in the set. The conjecture provides a geometric intuition that the solution set is generally convex, modulo permutations.
Theoretical results only validate the conjecture in limited settings, given sufficiently wide networks (Entezari et al., 2021 Ferbach et al., 2024). Empirical validations exhibit mixed accounts. Ainsworth et al. (2222) notably achieve zero barrier between two ResNet-20-32 models trained on CIFAR- 10,
but there remains a loss barrier between narrower models, even after weight matching. They further report network depth and dataset complexity as aggravating factors. Benzing et al. (2022) provide permutation algorithm. While fully connected
interesting insights using their activation-matehingg networks (FCNs) live in the same loss valley even at initialization, convolutional nets (CNNs) are usually not connected even after considering permutation invariances. Guerrero Peña et al. (2023) introduce Sinkhorn re-basin, differentiable permutation-finet -inding approach; however, even with two-layer NNs, the barrier between CIFAR-10 10 models, albeit low, remains non-zero. For CNN architectures like VGG, the barrier is substantially high. Altintas et al. (2023) show that aggravating factors for LMC include the Adam optimizer (Kingma & Ba 2017). absence of warmup, and task complexity.
Hence, in cases where strong evidence for the convexity conjecture is absent, it is important to consider other possible topologies for general DNN solution sets. To this end, we propose the star domain conjecture for characterizing DNN solution sets that do not enjoy convexity (Entezari et al.) 2021 modulo permutations.
3.2
THE STAR DOMAIN CONJECTURE
We propose weaker form of convexity for characterizing DNN solution sets. We argue that DNN solution sets are generally star domains, modulo cmnction-preservivi permutations. While/Annes et al. (2023) demonstrate this property for simple spherical negative perceptrons (without permutations), we argue that it holds for even deeper, more complex nets after considering permutation invariances.
We start with the necessary definitions to make formal description of the conjecture. A set A C R is star domain if there exists an element
E A such that for any other element E A and vt such that 0 < < 1, we have that (1 t) ag + t- a E A, i.e., all points on the line segment between an and lie in A. We call such dg a star point. In the context of the parameter space, we wwwddddooaappeeeeeee solution set as a star model.
refer to the star point of
A "solution" in our work refers to global minima theoretically reachable by SGD, unless stated otherwise (please also refer to Appendix [B.4).
Conjecture 2. Star Domain Conjecture. Consider neural network f X + Y, where X' C TR.. Y C IR. Assume that f is parameterized by 0 E R. Define C RT, the solution set of f as the set of parameters such that every 0 E S minimizes a given loss L over given dataset (((,,,1);
D= =((x 31)s; x4 E X.M E V, SiSN), Define B(0,, 02) as the loss burrier encountered upon linearty interpolating between 01 and 0z
Then, for any E > O, there exists a network width E such that if f() is wider than h, then S is a star domain modulo permutation symmetries, up tolerance of e. Specifically, there exists star model f* E 5 such that for any other solution f E S, there exists a funntiionpressrriing permutation T8 (8) such that B(0,6 8*) <.
A convex set is special case of a star domain, where all the elements are star points. Thus, according to the convexity conjecture (Entezari et al. 2021), every member of a DNN solution set is a star model, given sufficient width. In contrast, the central thesis of our work is that as network width increases, star domainness (where some models are star models) arises before convexity (where all models are star models, in agreement with the convexity conjecture).
4

---

Published as a conference paper at ICLR 2025
Our conjecture concerns deep neural networks, where theoretical validation remains difficult. How- ever, for the simpler case of two-layer neural networks with nne-dimensional inputs, we state Theorem heer.. We defer the proof of this theorem to Appendix [A] Additionally, in Appendix A.2 we empirically validate our theoretical result under simple settings.
Assumption 1. Consider a two-layer linear network fe(z) = 02T, parameterized by = (01,0) ④ where 01, 02 R is the network width. Assume that the inputs are bounded [xc| S 1. Define [(0; (I 3:) == 27 (0 022;- y to be the MSE loss. Let S be the solution with unit norms |0, ll2 ||02122
Given solutions & S, define the barrier as ( 8) == max C((1 t) + t0) - () tC(O).
that satisfy (())
rE
Let À be the uniform distribution over the solution set S, which is well-defined as the solution set measurable. We define the probability
P(m,e) e) Poi (V0 permutation of neurons s.1. B(Ø, x (0)) s).
Theorem 1. Given Assumption and an arbitrary e > O, P(m, E) + 1 as m + oo.
Theorem I guaranttess the existence of star models for two-layer linear networks, under the constraints laid out in Assumption [ This result implies that for fixed tolerance level m increases, more and more models in the solution set become star models. Eventually, virtually all solutions will be star models, implying the convexity of the solution set (in agreement with the convexity conjecture).
3.3 FINDING A STAR MODEL
We provide empirical evidence for the star domain conjecture via two steps. First, we present a method for finding model. Second, we verify that the model found is indeed star model: it has a low loss barrier with an arbitrary solution in S. Here, we focus on the first step.
We consider necessary condition for model 0*: given an arbitrary of models Z {0,, ., } C S, 0* has to be connected to all of them, modulo permutation invariances.
We present recipe for finding such 0*
We first obtain a finite set Z = {01, 02. Ox) } of models, independently trained with different random seeds controlling the initialization, batch composition, and augmentation. We then formulate a loss function that, for fixed Z, encourages low loss barriers between and some permuted versions of {01, .,....,, By. The objective may be expressed
8% arg min 0
Σ (O.TT
N
O.
where
is the winning permutation defined in Section 3.1 that permutes O,, without changing
-.
the represented function, and while minimizing the loss barrier against 0. To solve this optimization problem, we propose to minimize the expected loss on the linear interpolation between the model in question 9 and each source model On, after permutations. We modify the training objective as Oz arg ming (0) where
Σz(0) = N 1 c ((--))0++ (1-t)-++t-
0, (0)) dt
(4)
This expresses the expected loss on the of line segments between 0 and o, (Ox). where each
source model 0, ~ Unif( ()) is chosen at random and then each point on the line segment is sampled as E Unif(O, 1]. The optimization problem eq. (4) involves computational challenges. Resolving the continuous integral over is non-trivial for complex learning problems. Furthermore, π -. assumes access to the winning permutation. However, the winning permutation depends on which is constantly changing over the course of optimization. We introduce the following solutions.
5

---

Published as a conference paper at ICLR 2025
Train loss barrier
0.4
Train loss barrier
w
regular regular
regular- regular
2
0.2
star regu-rr
star regular
10 20
30
40
50
0
10
20
30
40
50
Number source models
Number of source models
Figure 1: Starness of star model source models. We plot the loss barriers B(0", 0,,) between star models 0* and heldout models 0,, E H at different numbers of source models Z used for learning the star model 0* (orange points). The heldout is disjoint with the source models: We provide reference point given by the loss barrier between two regular solutions B0 0g) for A.. 0B E S (blue plot). The error bars indicate one standard deviation across five held-out models |H] 5. Incorporating more source models |2| enables finding better star model with a lower loss barrier against an arbitrary solution.
2
Monte-Carlo opti- mization scheme. Instead of estimating (z) precisely at every iteration, rely on a Monte- Carlo estimation scheme, inspired by the parameter-curve fitting method by
Algorithm Starlight: Training Star Model.
Input. dataset D {(1; )))== source models Z {01, 0g.................. ... ON initial model 9o. learning rate X, number of batches m, number of steps K. Set 0 0g.
Output. 0
for k = to
if (k: 1) mod m == 0
for n = 1 to N do Step 1. Update On
Garipov al.) (2018).
(0n) 8n -0
At iteration k 2
we sample (2())
endfor
uniformly from Z and {A) from Unif(0, 1] Hence, we obtain single point on the manifold, calculate the cross-entropy loss at this point, and subsequently the gradients for updating 0.
endif
Step 2. Sample O, ~ Zn) t Unif(0, 0, 1]. and a batch from D.
Step 3. Compute loss [([1 t) 0 + 0,, B).
Step 4. Compute gradients v +- Ve[((1 t) 0,).
Step 5.
Update 0 + 0 A(1 t)
endfor
Finding optimal permutations. We perform weight matching (Ainsworth et al. 2022). i.e. we seek a permutation n,, that maximizes the dot product n,, (f.),, for each 0,, E Z. This procedure aligns each source model 6, with the candidate star model 0. This operation is performed at the beginning of every epoch instead every iteration, speeding up the optimization process significantly.
Algorithm describes the detailed procedure. Once we find & that has a low expected loss Σ (0) on the linear paths to finite set of source models Z, we may verify if this 0 is likewise linearly connected with an arbitrary solution 0 $ Z.
3.4 EMPIRICAL EVIDENCE
We introduced Starlight to find candidate star model. Now, we propose method to verify the model found in Section 3.3 is star model by checking its linear connection to arbitrary solution 0N+1 & Z, i.e., not part of the set source models used for finding the star model. We refer to such models as held-out solutions H that are disjoint from the source models: H Z 0.
We describe our main findings with reference to ResNet-18 18 (He et al. 2016) models trained on CIFAR (Krizhevsky al. (002)) using SGD, using 50 souree models and held-out models. We present results for additional architectures (e.g.. VGG (Simonyan & Zisserman 2015) and DenseNet (Huang al. 2017)), ) large-scale dataset (ImageNet-1k (Deng et al. 2009)) and settings (for instance, Adam

---

Published as a conference paper at ICLR 2025
Table 1: Empirically verifying the star domain conjecture. "Regular loss" and "Star loss" indicate training losses for regular models in Z and star models 0*, respectively. "Star-regular" refers to the barrier B (0", 0n) between a star model and one of the heldout models in H. For comparison, "Regular-regula"" is the loss barrier B (4A, 0p) between two arbitrary models. We report values to one standard deviation over several runs, except for ImageNet. In cach casc, star models exhibit significantly lower loss barriers with other models, than the corresponding average loss barrier between two regular models.
Dataset
Architecture
Regular loss
Star loss
Regular- regular
Star-regular
CIFAR-10
ResNet-18
0.001 = 0.000
0.001 = 0.000
0.383 + 0.056
0.078 + 0.007
CIFAR-10
ResNet-18 (Adam)
0.001 = 0.000
0.015 = 0.000
1.368 + 0.551
0.335 + 0.022
CIFAR-10
VGG11
0.003 = 0.000
0.022 = 0.000
0.515 = 0.034
0.131 0.005
CIFAR-10
VGG19
0.001 = 0.000
0.059 = 0.000
1.281
0.153
0.336 0.078
CIFAR-10
DenseNet
0.001 = 0.000
0.157 = 0.000
4.634 = 0.727
1.729 0.409
CIFAR-100
ResNet-18
0.004 = 0.001
0.005 = 0.000
2.905 = 0.047
0.756 + 0.049
CIFAR-100
DenseNet
0.006 = 0.000
0.635 = 0.000
6.920 + 0.216
3.735 + 0.180
ImageNet-Ik
ResNet-18
0.711
1.380
5.948
2.794
star 100 dource
source
star
heldout
CIFARIO
0.4
0.4
100
heldout
source
heldout
0.2
0.2
source
heldout
source
heldout
0.0
0.0
star
source
star
heldout
96
96
CIFAR 100
star
Source
star
heldout
100
100
2
2
source
heldout
Source
heldout
source
heldout
source
heldout
80
80
star
source
star
heldout
60
Train loss
Train accuracy
Figure 2: Loss barriers for star models. We interpolate between a star model 6* and regular models that are trained with SGD. There are two types of regular models, depending on whether they are used for finding the star model: source models Z are used, and heldout models H are not. Along the interpolation, we visualize the loss barrier by plotting the loss and accuracy values (orange curves). For these curves, / = O corresponds to the star model 0*, For reference, we plot the interpolation between two arbitrary regular models (blue curves). The error bands correspond to one standard deviation.
(Kingma & Ba 2017)) in Table and Appendix E Likewise, our empirical findings are built upon the training loss and accuracy, but confirm that they also transfer over to test loss and accuracy in Appendix D.I We largely use standard recipes to train the models in our experiments, with the exception of star models where we additionally incorporate the steps in Algorithm [ We further We summarize our observations below.
describe our experimental setup in Appendix B
Convexity conjecture does not hold. In Figure 2, we show loss barriers between two independently trained solutions (blue "regular-regular" curves). We observe that the loss increases and accuracy drops significantly around 0.5, even after applying the algorithm (Ainsworth al. 2022) to
find the winning permutation. We present another piece of evidence that the convexity conjecture does not hold for thin ResNets, reconfirming the findings of Ainsworth al. (2022).
Star model has low loss barriers with other solutions. In Figure 2 we show the training losses and accuracies along linear paths between the candidate star model 0- and other types of solutions (either source models Z or held-out models H). They are indicated with red curves. As reference, we always plot the confidence interval of loss and accuracy values along the line segments between two regular solutions (blue curves). We observe that, for the source models in Z, star-to-regular connections enjoy essentially zero loss barriers, in contrast with regular-to- to-regular connections, which remain significantly higher at 0.381, for 10. This demonstrates that it is possible to find model 6* connected to |Z] = 50 models simultancously. The same is true for the line segments between the star model 0* and held-out model picked from H 5 models; although the barrier
7

---

Published as a conference paper at ICLR 2025
0.4
40
star- rreuularrr
far-regular
0.36
34
22
0.2
y 0.05x+0.30
y 0.31x
0.32
4x
8x
28
0.0
0.0
0.4
0.8
1.2
0.9
1.0
1.1
regular-regular
regular-regular Depth
Width
Figure 3: "Starness" vs. model width and depth. For starness model width (left), we vary the width of a WideResNet (depth 22) from 1x to 8x. For starness vs. model depth, we vary the depth of WideResNet (width 1x) from 22 to 40 layers. For each depth-width combination, the loss barriers B(0", 0%) between star models l and heldout models 6% E H on the y-axis. As reference point, we plot the barrier between two regular solutions A00,,, 0B), on the x-axis. The points are annotated with the corresponding widths or depths. Star models consistently enjoy better linear connections with regular models, than do the regular models amongst each other.
plot
between the star model and the heldout model non-zero, remains as low as 0.077 compared to 0.381 for the regular-to-regule case.
greater number of source models enhances "starness". Our star model is constructed from the set of source models Z. We question whether greater [2] induces greater "starness" of the solution found by Starlight. In Figure I we plot the loss barrier against the number of source models 50 used to construct the star model. For statistical significance,
2 S |Z| < statistics between two regular, independently trained models in with error bars indicating one standard deviation. We observe that the loss barriers between these star models and the held-out models decrease as / increases. The decreasing trend has not saturated after |Z| = 50. We stopped there because of computational limits. However, including more source models likely to enhance connectivity between the obtained star model and the other solutions even further.
include loss barrier
Effect of model width and depth. Prior work stresses the importance of model width and depth (Ainsworth et al. 2022 Entezari et al. (202T) in determining loss barriers between two solutions. We investigate the effect of model width and depth for residual nets. Specifically, we consider WideRes- Nets (Zagoruyoo & Komodakis (017) of widths 1x, 2x, 4x, and 8x that of a normal ResNet (depth 22). We also consider ResNets of depths 22, 28, 34, and 40. We compare the barriers achieved by "regular-regular" and "star-regular" pairs for each case Figure 3. Our investigation confirms existing reports of decreasing loss barriers as model width increases. We observe significantly lower star-regular barriers than regular-regular barriers for models of identical widths (e.g., roughly 0.004 compared to 0.012 at width 8x). In fact, is possible to fit linear regression line to the observed barrier values, wherein the star-regular barriers about third of the regular-regulat barriers at any given width (Figure 3 left). We draw similar conclusions from varying depth (Figure 3 right), although the change barriers the varying width case.
change the model depth is not quite pronounced as it is for
While both the convexity conjecture and the star domain conjecture involve solution sets obtained through SGD, we also investigate the impact of using the Adam optimizer. Specifically, train 15 regular models, with [// = 5 and (Z = 10). We then train star model and evaluate its barriers with the models in the held-out set. Results can be found in Table [] (second row). We observe that Adam-traing regular solutions have higher loss barrier between them (1.368) compared to SGD-trained regular solutions (0.383). Likewise, the barrier between the star model and regular models also increases from 0.078 for SGD solutions to 0.335 for Adam solutions. While both reguular-regulann and "star-regular" connections suffer with this change of optimizer, "star-regular" connections still fare significantly better than "regular-regular" connections. This finding suggests that Adam solutions are also highly likely enjoy star-shaped connectivity.
Effect of optimizer.
Caveats. Despite the promising observations above, our star domain conjecture lacks theoretical verification, except for the simple case of two-layer linear networks (Appendix AA). Hence, for deep neural networks, it remains a conjecture. From the empirical perspective, loss barriers between the star model and other solutions often yield values that are significantly greater than zero. However,
8

---

Published as a conference paper at ICLR 2025
Table 2: Model fusion performances. "Regular" indicates single models; Ensemble indicates vanilla average of the probability vectors across the member models; "Star" indicates model found using Starlight using the regular models as the source set Z. ResNet18 has been used throughout. We show one standard deviation for the error bars. In addition, we report the accuracy of the best member the ensemble ("Best of n²) and the aceuracy of the best star model ("Best of 3"). Star models perform better than single, regular models but use only fraction of the compute required by the ensemble at test time.
Dataset
#Models
Regular
Best of
Ensemble
Star
Best of
2
95.21 + 0.03
95.24
95.76
95.30 + 0.16
95.43
CIFAR-10
5
95.07 + 0.14
95.24
96.02
95.17 +: 0.15
95.27
50
95.13 + 0.16
95.44
96.27
95.32 + 0.20
95.54
2
77.32 + 0.16
77.49
79.58
77.96 + 0.24
78.14
CIFAR-100
5
77.36 + 0.21
77.68
80.38
78.12 + 0.02
78.15
50
77.33
+ 0.28
77.94
81.30
78.38 + 0.10
78.48
Train / test complexity
(11)/((1)
(ww)/O(m)
((1)/O(1)
Probabiiity AUROC
0.92
Entroyy AURoIC
0.92
star
0.016
ensemble
ensemble
0.86
0.86
0.001
ensemble
2
10
12
14
16
18
2
4
10
12
14
16
18
12
16
(a) AUROC (Max. Probability)
(b) AUROC (Entropy)
(c) ECE
Figure 4: Bayesian model averaging. The star model was trained using 50 source models. The x-axis denotes the number of models sampled from the star domain for Bayesian model averaging or from the set of source models.
emphasize that this paper is focused on providing the lower bound in evidence supporting the star domain conjecture. Considering a larger number of source models for the star model construction, improving Starlight, and developing better algorithm for finding the winning permutations will potentially contribute to the discovery of better star models in the solution set.
Conclusion. Our experimental results confirm existing reports that the convexity conjecture requires very wide networks to hold, and has otherwise several failure cases for which we propose relaxed version. viz.. the star domain conjecture. We obtain strong empirical evidence that the star model found through Starlight is likely to be a true star model. Our analysis thus sheds further light on solution set geometry for narrower and deeper networks, as well as for complex learning tasks where the convexity conjecture struggles. We invite the community to expand upon our findings and converge toward a more accurate understanding of the loss landscape.
4 PRACTICAL APPLICATIONS
The star domain conjecture introduces novel dichotomy of solution types: "star" and "non-star" models. Most solutions are non-star and lack linear connections with other solutions. However, in Section 3 have presented strong evidence for the existence of star models. In this section, examine the properties and potential benefits of star models in practice. 4. star models and the surrounding star domain provide a better posterior for Bayesian Model Averaging. In Section
whether
we propose star models as a practical alternative to model ensembling.
4.1 BAYESIAN MODEL AVERAGING
Bayesian model averaging (BMA) enhances uncertainty estimation by averaging predictions from the posterior of models in the parameter space. Posterior families in the literature range from simple Gaussian (Blundell et al. 2015) and Bernoulli (Gal & Ghahramani 2016) distributions more

---

Published as a conference paper at ICLR 2025
complex geometries like splines (Garipov al. 2018) and simplices (Benton et al. 2021). Here, examine if the star domain provides a good posterior family for BMA-based uncertainty estimation.
Setup. The posterior of interest is the collection of line segments between the star model 0 and other solutions (0l..... }r that are independently found. Similarly to Starlight, we sample first from the model index of {1.... N} uniformly and then sample from the line segment Unif[O. 1]. As in standard BMA, we consider set models sampled from the posterior and the post-softmax average of their outputs. We ResNet-18 models trained on CIFAR-IO. As a baseline, we present the BMA for the independent solutions {0,,..- }
Evaluation. We assess the predictive uncertainty of the BMA-based confidence estimates. For the ranking metric, we use the area under ROC curve (AUROC), considering both max- maxxprobabilliy and entropy-based confidence measures. We also show results based on expected calibration error (ECE).
Results. Figure 4 shows uncertainty quantification at different numbers of posterior samples from 2 19. BMA using the star domain posterior consistently exhibits better AUROC values than baseline deep-ensemble estimates. However, ECE is worse than that of the deep ensemble. The star domain posterior provides avenues for more precisely ranked uncertainty estimates, albeit absolute-value uncertainty quantification may not be precise.
Conclusion. Our proposed star domain posterior offers better uncertainty estimates than the deep ensemble baseline in rank-based predictive uncertainty evaluation.
4.2 POTENTIAL USAGE IN MODEL FUSION
Given a fixed amount of training data, a popular approach to maximize model generalizability is ensembling, i.e., fusing predictions from multiple independent models. This basic approach suffers from computational complexities during both training and inference. Every input has to be processed by individual member models test time. Storing multiple models also leads to higher memory footprint, scaling linearly with the number of ensemble members.
Starlight can also be understood as a method for aggregating multiple source models (0..... 0y) into a single model 6*. From a computational perspective, star models reduce the necessary time and storage complexity during inference. We investigate whether the star models provide an enhanced generalization compared to the individual models.
Setup. We slightly modify the training objective of Starlight to align with a better generalization capability of the star model. We add a cross- term L (0) so that Lioa Z) = (z) (0).
where (0) is the original optimization objective for the star model discovery in eq. 44
Evaluation. We evaluate test accuracies for models trained with varying numbers of source models ((Z)) and compare them to ensembles using the same source models.
Results, Results in Table show that star models consistently outperform regular models (78.4% vs. 77.3%) for CIFAR- 100 with / 50). While less accurate than ensembles over Z, star models require only fraction of the compute during inference.
Conclusion. "Stamnes"" of a solution may enhance generalization. In scenarios where test-time inference costs are critical, star models could be a promising alternative vanilla ensembles.
CONCLUSION
This paper proposes novel understanding of SGD loss landscapes. The traditional picture before Garipov et al. (2018) was one of extreme non-convexity, in contrast with the current picture of near- perfect convexity in a canonical, modulo-perrutationns space (Entezari et al., 202T) for extremely wide nets. Our claim becomes relevant when narrower and deeper nets, complex datasets, and different optimization schemes are considered. We propose a weaker form of convexity in these cases, i.e., the solution set is star domain modulo permutations. Our empirical findings support this hypothesis. We propose the Starlight algorithm to find candidate "star models" and verify that they indeed linearly connected to other solutions. In addition the empirical evidence for the star domain conjecture, we present potential use cases for star models in practice, including uncertainty estimation through Bayesian model averaging, and model fusion.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 18099A The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Alexander Rubinstein. The authors would also like to thank Arnas Uselis and Bálint Muesányi for helpful insights.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging Models mod- ulo Permutation Symmetries. The Eleventh International Conference on Learning Representations, September 2022. URL https:://ppereeeeeeeee htwp://wwwnn/oouuuueaaaamaa [ttp:/mmdddommmmmmmmlm
net/
Gul Sena Altintas, Gregor Bachmann, Lorenzo Noci, and Thomas Hofmann. Disentangling Lin- ear Mode-Conneetivity, December 2023. URL arXiv:2312.09833 [cs].
09832
Brandon Livio Annesi, Clarissa Lauditi, Carlo Lucibello, Enrico M. Malatesta, Gabriele Perugini, Fabrizio Pittorino, and Luca Saglietti. Star-Shaped Space of Solutions of the Spherical Negative Per- ceptron. Physical Review Letters, 13
November 2023. doi: 10. R0.1100 227301. URL httth: aps org/ http:////..//.1.11///////ar
137.1
Publisher: American Physical Society.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling. In Proceedings of the 38th International Conference on Machine Learning, pp. 769-779. PMLR, July 2021. URL https ISSN: 2640-3498.
//proceedings ml prepaaasssss...sessse.e........e
press /tt.//11/////// http1////eeettneeeeeeee html
Frederik Benzing. Simon Schug, Robert Meier, Johannes Von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger. Random initialisations performing above chance and how to find them. OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), November 2022. URL httlhttpss..
/ttpp:://ppenzevieett wwwp///wwpwww.eppp///pppp hate://oooommmmdom__eee -iddddddHs5sbbb qFI
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Network. In Proceedings of the 32nd International Conference on Machine Learn-
ing, pp. 1613-1622. PMLR, June 2015. URL URL httpthtts///prcceeeeeee blunde 1indeel115 15. html
mlr. pa.r.press/v33]
ISSN: 1938-7228.
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise permutation saddles, connected by equal-loss valleys across the loss landscape, July 2019. URL
org/ abs /1907, 02911 arXiv:1907.0299 0000009001111 [cs,
stat].
Djork-Amt Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), February 2016. URL http::/arxiv..rrt
abs/ /1ee/151111 07289 1XXi:1511.002200 [cs].
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially No Barriers in Neural Network Energy Landscape. In Proceedings of the 35th International Conference on Machine Learning, pp. 1309 1318. PMLR, July 2018. URL CRLLtttpp://ppoceediii
mlr
Treas/v80/Graxleet
html
ISSN: 2640-3498.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The Role of Permutation Invari- ance in Linear Mode Connectivity of Neural Networks. International Conference on Learning Rep- resentations, October 2021. URL URI.inttss://ppeereeee net. d
d=dN iyttmk11,
11

---

Published as a conference paper at ICLR 2025
Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving Linear Mode Connectivity of Neural Networks via Optimal Transport. In Proceedings of The 27th International Conference Artificial Intelligence and Statistics, pp. 3853-3861. PMLR, April 2024. URL ISSN: 2640-3498.
ngs ml wtw.ssssl......ss.ee.p...
(eta8//febbccc22e2 htm
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. In Proceedings of the 37th International
Conference on Machine Learning, pp. 3259-3269. PMLR, November 2020. URL https //proceedings .arssggsmmli: mlr. pwwss/viiie.eeeee Frankle20a. html
ISSN: 2640-3498.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1050-1059. PMLR, June 2016. URL https:pppppoccccdnnggggooo/ mlr...m.r...... press/ preess
v48/ hte/gallc html,
ISSN: 1938-7228.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018, URL https::/ nips http://pprrr...iisssccppppppprre files /ttp:///pppeee..eaaa//e/////000 /hahhhhhh///
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Using Mode Connectivity for Loss Landscape Analysis, June 2018. URL http:://arxiv.or//aattt
1806606977 arXiv:1806.06977 [cs, stat].
Fidel A. Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit Sinkhorn differentiation. In 2023 IEEE/CVF Conference Computer Vision and Patten Recognition (CVPR), pp. 20237-20244 Vancouver, BC, Canada, June 2023. IEEE. ISBN 9798350301298. doi: 10.100//VVRS5222922223000 wwwp://////.eeeeeeeeeeeee 11203774171
URL WRLp:Etpp://eeexoorrr
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 20/6 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8851-11 doi: 10.1109/C 2016.90. URL
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely Con- nected Convolutional Networks. In 2017 IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 2261-2269, July 2017. doi: 10. 10.1109//2R URL
https::/Aeeexploxz: ieee . org/ /cccmett/ ISSN: 1063-6919.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, pp. 448 456. PMLR, June 2015. URL Cress/v37/Loffit
mlr
html
ISSN: 1938-7228.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear Connectivity Reveals Generalization Strategies. September 2022. URL httphttss//ppeeeeeeee.eeee
Eorum? 2 id=hy 6M0J6
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017.
URL http: org/ abs/1 412.44466 6980, arXiv:1412.6683 [cs].
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Con- volutional Neural Networks. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL htt //tp:::://pppppp/ee/ wwweeesssiiipe nips. cc
fites/ppprr///e////ee www.rt-bbbaattt..a.......aaa.... html
2012 2o12/hash/c39822dbb2 wtw.:333666666866633333333333
Rohith Kuditipudi, Xiang Wang. Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets. In Advances in Neural Information Processing Systems, volume 32. NeurIPS, 2019. URL [tttpps// www.eiiininnnnnnnnnneeeeme httppsscccpppppe /ttpur/ppp.c/ppppp///////////// 4
httml
12

---

Published as a conference paper at ICLR 2025
Guillaume Leclere, Andrew Ilyas, Logan Engstrom. Sung Min Park, Hadi Salman, and Aleksander
Mędry. FFCV: Accelerating Training by Removing Data Bottlenecks. In 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1201 12011-12020, Vancouver, BC, Canada, June 2023. IEFE. ISBN 9798350301 298. doi: 10. (0.109/9VVP552222222...13 URL htt.hhtttss//eeexxlllreeee
leee wttp:////mmee/////3333333333
Zhanran Lin, Puheng Li, and Lei Wu. Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity, April 2024. URL arXiv:2404.06391 [cs, stat].
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. URL http://arxix: httpv/v/.rggabb//111111 05101 1eX.0:1711.05100 [es].
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear Mode Connectivity in Multitask and Continual Learning. International Conference on Learning Representations, Octoher 2020. URL wal.https:/// http::///ppeeevviie ew. net. forumm ?id=
Fmg_ Tmg_F
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Rigg-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. URL htw.::ttp////mmm http:tttss//prrccddinngsssss nwwr.gg........ppp..oe.. wwtpwpppppppppeeccccc files/
Abstract html
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for Activation Functions. ICLR 2018 Workshop, February 2018. URL
Ekansh Sharma, Devin Kwok, Tom Denton, Daniel M. Roy, David Rolnick, and Gintare Karolina Dziugaite. Simultaneous linear connectivity of neural networks modulo permutation. CoRR, January 2024.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition, April 2015. URL htrp:////mppppeeeeeeeeeooooooroe
net. wtt.eneooooddddssll d= 1=6s11scCvha
1556 urXiv:1409.11 1556
[es].
Sidak Pal Singh and Martin Jaggi. Model Fusion via Optimal Transport. In Advances in Neural Information Processing Systems, volume 33, pp. 22045-22055. Curran Asso- ciates, Inc., 2020. URL httts:://pocceeeinnnn neurips cct.ccc///o cc/pappr//220//aahh 02229889155844443ccee28885556333
html
Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing Mode Connectivity via Neuron Alignment. In Advances in Neural Information Processing Systems, volume 33, pp. 15300-15311. Curran Asso- ciates, Inc., 2020. URL htttps://poocceeiinssseooooo neurips cc cc/papper/222//hahh [Bcac222292att99)eee166666118 Abst html
Neha Verma and Maha Elbayad. Merging Text Transformer Models from Different Initializations. Transactions on Machine Learning Research, May 2024. ISSN 2835 8856. URL https
net http://ww.....t.oo..oo.o...attn.
20of (S2Fprof
Chengkun Wang, Wenzhao Zheng, Junlong Li, Jie Zhou, and Jiwen Lu. Deep Factorized Met- ric Learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7672-7682, Vancouver, BC, Canada, June 2023. IEEE. ISBN 9798350301 298. doi: 10.1109/CVPR52229..02 200.:799..222.07744. 10205348/
.aaeeeerrg////eee URL ALL htt.ttts//eeeexxooreeeee hot.://roe...e...roooooooootttt
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun. Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated Learning with Matched Averaging. September 2019. URL /tttpss:/openreviiiiii
net/ffore l
13

---

Published as a conference paper at ICLR 2025
Haitao Wen, Haoyang Cheng, Heqian Qiu, Lanxiao Wang, Lili Pan, and Hongliang Li. Opti-
mizing Mode Connectivity for Class Incremental Learning. In Proceedings of the 40th In- ternational Conference Machine Learning, pp. -6950- PMLR, July 2023. URL
press wwwpssssvp000 http///2222/2222222 html ISSN: 2640-3498.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks, June 2017. URL
http:
/22xdv..rr//ab/11055.000 07146 arXiw:1605.07146 [es].
Pu Zhao, Pun-Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness. September 2019. URL
httpps://peenrriee..eeeee http:///www.......oo.oooooooootn
Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang, and Vahid Tarokh. Sgd converges to global minimum in deep learning via star-convex path. ICLR, 2019. URL httpt::/oppereeiiiiiii
Bet/padfidd-BIIcciRee
14

---

Published as a conference paper at ICLR 2025
A SPECIAL CASE STUDY: L-LAYER LINEAR NETWORKS
Here, analyse the case of an L-layer linear network, focusing L = 2. This relatively simpler case enables us to perform theoretical analysis that would have been significantly more challenging for deep neural networks.
Our theoretical and empirical analyses suggest that the geometry of the solution set for 2-layer linear networks is approximately star-shaped. More precisely, the solution set of such network will have increasingly many star models as the width increases; eventually, at sufficient width, almost all models models, naturally implying convexity of the solution set. As promised Section 3.2 we share our theoretical and empirical findings in detail below.
A.1 THEORETICAL VERIFICATION
We re-state and prove our claim for 2-layer linear networks (Theorem
Assumption 2. (re-stating Assumption 7 Consider a two-layer linear network fo(æ) = 0 02æ, parameterized by 0 (01, 02) θ where 01, 02 R is the network width. Assume that the inputs are bounded |z| I 1. Define L(0; (x;, y:), == 2 ( 02x; y/3² to be the MSE loss.
Let S be the solution solutions o, 0 E S, we define the barrier as (( == max (1
with unit norms ||0 ||2 ||02||2
that satisfy L(0) 0. Given two
:c(0.1)
Let X be the uniform distribution over the solution set S, which is well-defined as the solution set is measurable. We define the probability
P(m,c) c) == Poi.br (4B E S, permutation of neurons x s.t. B( x()) S
(5)
Theorem 2. (re-stating Theorem m + DC.
Given Assumption 2 and an arbitrary e > 0, P(m, c) - as
Proof. Observe that the output the dot product 0, 8g is scalar C. Considering the MSE loss defined in terms of the dot product. [,Crz; y²)²² ", the solution is unique due to strong convexity. We may thus write down the solution set as {(01, 02) 02 = C} where C minimises the MSE loss.
Now, define
Q(m, e) == Poroers (VO S, 3 permutation of neurons s.t. (01T(02) + x(0))02 201 e)
(6)
and prove the lemma below:
Lemma 1. Assume the conditions of Assumption 2 Then for any > 0, m E N, we have that
e(m,e) 2 e).,e).
follows that it sufficient to prove: Q(m,c) c) - 1 as m + 3o for all <>
Lemma Consider the solution set S = {(0,, 62) 1on Φ2 E RM oI Φ2
C, | |
1}. where |C| < 1. Let X be the Lebesgue measure confined to S C Rm x RM Let > 0. Then,
P Porrd
sup 0,,02:0 C==C RES.
min
|o m(22) + m(1)) Φ2 -2 se +1
as 00, where Sm the set of permutations on (1,..., ,..
Proving the lemmas will conclude the proof.
Now, let us prove the lemmas.
Lemma 1. Assume the conditions of Assumption 2 Then for any e and m E N, we have that P(m,e) 6) 2 (m,,)) e).
15

---

Published as a conference paper at ICLR 2025
Proof. Fix the width m and tolerance 0. We show that for all solutions 0 E S, there exists permutation x such that
| x(22) + + x(1) Φ2 2012 <,,
then o a star model with tolerance e. Specifically, we will prove that if equation 7 holds, then
Bi 0) <.
This implies that the event of & satisfying equation iis subset of the event of ø being a star model, proving that P(m, c) 2 (m,,.).
Let x be an input and y be the corresponding label. Since & and T(0) are solutions, L(0,z) [(T(O), = O. We are interested in the linear interpolation between ø = (Φ1, 62) and π(θ)
(m(01), (02)) in the space R. IR.". For any E [0, 1], consider:
((1- 1)01 + (T()))) ((1 1)02 - (T())) = (1 - () Φ2r + 1111 ()[ x(0) + T(01) x2Φ) + (0(T)(=(((2)) (02)) (1 - 21 + 2(") L(1 (00 (02)2 ()]]=(())) +00)) ()22))
= (0(0,
where we used y = o1 hz = (0i)) T(0)))((B)).
Define the loss interpolation
(((d,((0),x) = (()) y/2
Then,
(B2) + TT(2) x22)) 2
=
= =1 - t)² (0]=(0))=++(0)) 02z -2y]*
= (211 - 1)3
Φ2 - -2C] 2 ||:||
I |ot (02) (0,)) 02 2012
(since 0 st 1 and |||||| s 1)
<e (by assumption).
Thus, for all E [0, 1].
L((0,0) Ezex (L,(0,0', <c Now, consider the function B'),0'):
B(,,0') = max (Le(s) 0') - -(1 ()11)) (C((')) tE(0.1]
max Le(0, 0') (since L(Ø) = L(0') 0) rE[0.1]
<.
Therefore, B(o,0) 8') < c, which implies that 6 is a star model with tolerance e.
Lemma Consider the solution set {(01, 62) dn 02 E R of Φ2
C, | ||22112
1}, where |C| < 1. Let X be the uniform distribution confined to S C R. x Th. Let e > 0. Then,
P
Por
max min |o | (T22) T -21² < (8,, TES,
+1
as 0o, where Sm is the set of permutations on (1,.... mm.
Proof. We may sample Φ1, $2 pA(01, 62) by sampling from pa ps(0 )x (02()))))) Note that px(m1) = Unif(]|@n112 = 1} and px px((2(01)) = Unif(]]eol1e = and of 02 C).
Then, each element th: (for E (1.... , follows the distribution defined by the PDF: p(óx) C,(11 - 0f " for some constant C, and d; E [-1, p(o1x) converges in distribution to N(, 1 As such, given arbitrary 8 > 0, may choose such that the CDF of di m uniformly bounded away from the CDF of (0, m by 5. Note that given samples ou from px (01). may sort them (d10(1):" ".. (1)(m)) with some suitable permutation E Sm. This lets build
16

---

Published as a conference paper at ICLR 2025
the empirical CDF defined by (x) == n E, Lez Given two samples @1 px(Ø1) and 0 (01), we may sort them with permutations a and respectively, to ensure:
(la())) ((l))ll2 S (|(()1) F + ([7()))) Fx s =
(9)
3
with high chance 1 - 8. Taking क = o- T, we have
[1 (02) + = =01) 02 - -20 S x022) of + (001) 12 -f
$ |0 x(02) =(01) π x(22) ((1))=((0)))))))))
(011) ((22) cl²
+ ((0:)T 02 - oT + 1 02 Cl
$ x(22 - =(0))
=(0)))=((2)) x(22)|
(011) 82 - 02
I 11 x((1)) 1(0)) 21 11
E + <e
3 3
with high chance 1 - 8.
This proves that
[ (02) x(11) -20²² sc +1
P Pa
min mT
as m + co. This result can easily be extended to the statement in the lemma using the compactness of the unit sphere.
A.2 EMPIRICAL VERIFICATION
In the theoretical analysis above, we hypothesized for two-layer linear networks that network width gets larger, the proportion of star models in the solution set increases. To verify this, we performed small-scale analysis using 2-layer linear networks. First, we provide empirical evidence towards Lemma Z in Appendix A.2.1 Then, in Appendix A.22. we consider a full-fledged learning problem and show that the proportion of star models increases as network width increases.
A.2.1 DATA-FREE VERIFICATION
We provide empirical evidence towards Lemma by considering vectors confined to the surface of unit sphere.
Setup. We consider samples of o (01. 62) and 0 (01, 02) where Φ1, d2, 01 01 E R", and || = ||>2||2 ||0 = ||02]|2 1, oj Φ2 = 0, = C. Specifically, we consider C = 0.5. The sampling is carefully carried out reflect the conditions assumed in Lemma different values of e and compute (mm, e), Le., the number of samples & out of No for which the condition in Lemma 2 holds, as m increases. We set No = N₂ = 100. The width m ranges from 21 to 2n while we consider 50 values of e between 0.01 and 0.1 (inclusive). Note that we use the same 0 samples for evaluating all & samples.
We consider
We observe that as width (marked along the x-axis) increases, Qm,, e) increases well. Before reach Q(m, e) = 1, we (almost) always pass through an intermediate region where 0 < Q(m, e) < 1, thus confirming that Q(m,e) slowly grows 1 as width increases. Thus, we successfully validate Lemma 2
Results and conclusion.
We present the results in Figure 5
A.2.2 VERIFICATION OF MAIN THEOREM
In the previous section, we validated Lemma 2 for the case where the input dimensionality d = 1. Here, we further validate the statement of Theorem 2 for the case where d = 2, using a full-fledged machine learning task. Note that our experimental setup slightly differs from the conditions of Theorem (e.g. binary cross-entropy loss instead of MSE, and d = 2 instead of d = 1). This is on purpose: aim to show that theoretical result holds even after relaxing the conditions therein.
17

---

Published as a conference paper at ICLR 2025
star-domainness
0.01
0.8
gasiton (0.0 (0.01 0.10)
0.04
0.05
0.6
0.06
0.07
0.08
0.2
0.09
0.0
16
32 Width
64
128
256
512
1024
convexity
Figure 5: Verification of our theoretical result in Lemma We compute Q(m, at different widths m and tolerances c. The y-axis denotes the tolerances, ranging from 0.01 at the top to 0.1 at the bottom. The x-axis denotes the widths m. The color bar encodes different values of ((m, e), ranging from 0 to 1. For fixed as width m increases and (m,,)) e) goes from 0 to 1, there almost always exists transitionary period where 0 Q(m, e) L.e,, some, but not all samples ø satisfy the star model criterion. We label these transition phases under "star-domainness" in the figure. This observation further confirms our theoretical result.
Setup. We consider two-layer linear networks trained using binary cross-entropy loss. We 2D binary classification dataset consisting of samples from class-conditional Gaussians. Specifically, D (2,, 9:); 24 (M,, Σ), x IR2, y e0, 1. We train / 100 copies this model. using 100 random seeds. Next, we fix each model ø E Z and compute its barriers with each of the other models 0, E Z Φb. define an empirical star model 6* E Z a model that has its loss barriers with all the other models in Z bounded below c: ((0", π (0:))
for all 0 Z l*, We test networks with
widths 2, 4, 8, 32. Our focus lies the proportion of empirical star models with tolerance as m increases. An increasing trend in this proportion w.r.t. increasing m would confirm our theoretical result.
Results. For each width m, we report the proportion (out of 100) of empirical star models, as defined above. We observe that the number star models at given tolerance increases as m increases (c.g.. at e = 10-6, there are no star models for m = 2 while 21 out of 100 models are "star" at m = 8). At m 32, e 10 10-3 all models become star models, i.e., convexity achieved. We present the quantitative results in Table 3
==
10-5 5
=
10
=
10
m 2
0.00
0.01
0.05
0.12
m 4
0.03
0.08
0.25
0.40
m 8
0.21
0.38
0.60
0.72
m 32
0.86
0.97
1.00
1.00
Table 3: Verifying Theorem 2 We fix different thresholds and at each given width, compute the proportion of star models at tolerance of e. Given the same e, we observe that this proportion increases with the width m.
18

---

Published as a conference paper at ICLR 2025
Conclusion. We confirm through experiments that the network width m increases, the number of star models in the solution set increases.
A.2.3 VERIFICATION FOR NON-LINEAR NETWORKS
Here, we present additional results in support of Theorem for non-linear two-layer neural networks. We describe them below.
Motivation and setup. Our aim is to verify that the proportion of star models in the solution set increases as network width increases. The experimental setup is similar to that in Appendix [A.2.2] where we train two-layer neural networks on the task of binary classification. The key difference this time is that we introduce a non-linearity (ELU) between the two layers. At each network width m, we train 100 models and for each of these models, we measure the maximum barrier upon linear interpolation with any other model.
Results and conclusion. We present our results in Table We notice that the proportion star models at any given width and tolerance is lower compared to the completely linear case; however, this proportion still increases (for example, at tolerance 0.93 for = 32) as width increases. Once again, confirm that our overall conclusion stands consistent with the claim Theorem I, even after introducing a non-linearity between the layers.
Te - 6, it goes from 0.00 at m 2 to
10
c=
10-6
c=
10
== 10
2
0.00
0.00
0.00
0.00
m = 4
0.00
0.00
0.00
0.04
8
0.00
0.00
0.00
0.71
m = 32
0.93
1.00
1.00
1.00
Table Verifying Theorem 2 for non-linear networks. We fix different thresholds and each given width, compute the proportion of star models at a tolerance of e. Given the same e, we observe that this proportion increases with the width m.
A.3 DIRECTIONS FOR FUTURE WORK
A.3.1 EXTENSION THEORETICAL RESULTS
Our theoretical result in Appendix A. is established for a two-layer linear network under simplifying assumptions (Assumption 2) The key ingredient in the proof is the existence of closed form characterization of the solution set, i.e., 01 02 = C. Extending this result to more complex cases and proving that star-shaped connectivity arises at much smaller widths than convexity (as shown in our empirical results) would be interesting.
However, deriving theoretical results for full-fiedged neural networks is notoriously difficult and, in many cases, infeasible due to their complexity. Theorem 2, proven for the simple yet abundantly challenging case of 2-layer linear network represents our best attempt providing a theoretical foundation for our findings.
The most closely related work to our paper the highly influential work by Entezari et al. (021), who also tackled the simplified setup of 2-layer neural network with a ReLU activation in between (Theorem 1. in their paper). However, they approach the problem differently and consider random initializations within hypercubes, instead of solutions to a learning problem. Thus, their proof avoids the problem of characterising the solution set a two-layer NN that contains a non-linearity between the layers. This underscores the broader challenge of developing theoretical proofs related to DNN solution sets. In our work, we take a different route by focusing on actual solutions to an optimization problem, although this necessitates dropping the non-linearity. We believe this puts the complexity of our theoretical results on a comparable footing with Entezari et al. (2021) while still offering a different perspective.
To further address the gap in theoretical results, we provide extensive empirical evidence covering three different architectures, three datasets including ImageNet, and ablation studies in Appendix C
19

---

Published as a conference paper at ICLR 2025
6
5
4
3
2
regular-regular star-regular
10
15
20
25
30
35
40
45
50
Number source models
Figure 6: Transformer results. Star-regular barrier steadily decreases as we use more source models train the model.
Our experiments indicate significant evidence towards our theoretical result even when the restrictions of Assumption are lifted. We hope that these observations serve as inspiration for other researchers to seek out more rigorous theoretical verification in future work.
and Appendix D
A.3.2 EXTENSION OF EMPIRICAL RESULTS: TRANSFORMERS
As discussed in Appendix A.3.I our study provides extensive empirical results. However, there always room to include other settings. For example, transformers exhibit unique permutation symmetries: the latent symmetries within each attention head and the order of the heads themselves. However, there limited precedent the literature for empirically investigating transformer-spectie
permutation symmetries. Prominent studies, including Entezari al. (2021 Ainsworth al. (2022),
Tatro et al. (2000) ; Singh & Jaggi 2020), largely exclude transformers from their analyses. Given this, transformers lie outside our primary scope in this study. Considering the current practical relevance of transformers, we believe a dedicated study on the star-domainness of transformer loss landscapes a promising direction for future research.
In the current work, we instead present small-scale results using a minimal transformer architecture. Since the main focus of our work is star-dommisnsss rather than exploring permutation symmetries in transformers, we conduct our experiments without permutations.
Setup. We use a minimal VIT model with the MNIST dataset. We train 50 source models and hold out additional models for barrier evaluation. Each regular model is trained using the AdamW (Loshchilov & Hutter 2019) optimizer with learning rate of 1e 3 (weight decay 5e 3) for 100 epochs. Next, we train star models using more and more source models (10, 20, 30, 40 and 50). Finally, we compare the regular-regular barriers with the star-regular barriers.
Results and conclusion. We present results in Table 5 and consistent with that in the main paper: the star-regular barrier decreases as we increase the number of source models, and the final stau-regular barrier of 322 is significantly lower than the regular-regular reference point 5, 5.687. This preliminary experiment thus confirms evidence for the star domain conjecture in vision transformers.
and Figure Our observation here remains
Possible future work. After completing this study, we encountered the recent work of Verma & Elbayad (2024), which investigates transformer loss landscapes using data-driven permutation algorithm. Adapting their method to our Algorithm could yield valuable insights. We leave this direction to future work.
20

---

Published as a conference paper at ICLR 2025
Table 5: Transformer results. The star model was trained using 50 source models. Our results stay consistent with those in the main paper (Table
Regular loss
Star loss
Regular-regular
Star-regular
0.001 + 0.001
0.099
5.687 1.167
2.312 + 0.708
B IMPLEMENTATION DETAILS
In this section, we describe the setup for replicating our experimental results.
B.1 MODEL TRAINING
Our model training hyperparameters largely reflect standard practices, but we describe them here for completeness. We used NVIDIA A100 GPUs for most of our experiments. All experiments were performed on single GPUs.
ResNet18 on CIEAR. For ResNet18 models trained on CIFAR-10 and CIFAR-100, we use a batch size of 128. We normalize the data using ImageNet statistics. For data augmentation, we apply padding to the image its horizontal mirror, and then randomly crop out 32 x2 region. We train for 200 epochs using SGD with momentum 0.9 and weight decay of 54 The initial learning rate is 0.1 and follows cosine decay schedule to reach by the end of training. Star models and regular models trained using otherwise identical hyperparameters, except that the star models the training objective described in Algorithm| []] The differences between different models in Z and come from the random seed the beginning of the training process. We use the following implementation for Each regular
the ResNet: model took roughly 30 minutes to train, while the star model ([Z) 50) took roughly 6 hours to train.
DenseNet-40-12 on CIFAR. Our DenseNet models use largely the same training settings as ResNet18. We highlight the differences here. DenseNet uses batch size of 64. The weight decay factor is 1e - 4, and the models are trained for 300 epochs. The learning rate, ini- tially 0.1, multiplied with 0.1 epochs 150 and 225. We use the following implementation: https://github..
com http://github.coomoaa same training recipe. Each regular model was trained for roughly 3.5 hours. Training the star model took approximately 7 hours.
Star models follow the
VGGs on CIFAR. 150. Other settings are identical to those used for ResNet18. We use the following implementation: Star models follow
The initial learning rate is set to 0.05 and is multiplied by 0.1 at epochs 100 and
https://githb..c cwwc same recipe as regular models. It took roughly minutes to train each source model, and 35 minutes to train star model.
ResNets on ImageNet. For ImageNet, we use batch size of 256. Models are trained for 100 epochs, using SGD with a learning rate of 0.1 which is multiplied by 0.1 at epochs 30, 60, 90. The weight decay factor is le -4 4. We use the ResNet18 implementation included in PyTorch (Paszke) ct al, 2019). We leverage the open-source library FFCV (Leclere et al.) 2023) to speed up our experiments. For data augmentation, we resize the image its horizontal mirror to 256 x 256 and randomly crop out a 224 224 region. Each source model took roughly 13 hours to train, while a star model required about 2 days.
Weight matching. We weight matching (WM) (Ainsworth et al. 2022 Our implementation leverages an open-source Python package called "rebasin": https://pppi.og/pppjee/rhhssissi
Total compute. We estimate to have spent approximately 50 70 days of NVIDIA A100 compute hours for the experiments in this paper (not including experiments that did not make it into the paper).
21

---

Published as a conference paper at ICLR 2025
B.2 CALCULATION LOSS BARRIERS
We use the definition in ? to calculate loss barriers between any given pair of models (eq. 1 Since the definition in eq. ① involves an infinite search space for the maximum, we sample finite T = {ti, .1. ... tt of equi-spaced points and compute the maximum as
max L () -) + + ((1 ((0a) + (8BB) RET
(10)
Our sampling of equidistant points is consistent with prior work Ainsworth et al. (2022); ; Guer rero Peña et al. (2023). The size of 'I' itself varies in prior work. Because of the scale of our experiments, we use |T| 11, including the end-points. In Appendix C.1 we show that this size is sufficient for obtaining statistically significant results.
B.3 HANDLING BATCU NORMALIZATION
Batch normalization (Toffe & Szegedy 2015) is integral to efficient DNN training. describe the so-called "variance collapse" problem that leads to degradation interpolated models. As a solution, we follow Ainsworth et al. (2022) and recalculate the batch statistics for each interpolated model, by performing one forward pass through the entire training set before performing evaluation.
B.4 NOTE ON DEFINING SOLUTIONS
In general, "solution" in this paper refers the global minima of the objective function. Below, we justify this definition.
Theory. In Theorem T we ensure global minima by assuming precisely zero loss. In Conjecture we also assume global minima that are theoretically reachable by SGD.
Practice. experiments involving DNNs, it is tricky to determine whether global minima were reached. However, we offer the following assurances.
For our CIFARI0 and CIFAR100 experiments. we reach extremely low training loss values (for example, training loss the order of 10-3 3 for C- models). These results strongly suggest that the obtained models are very close to the global minima of the objective function.
For our ImageNet experiments, the loss values are not exactly zero. However, the loss at the actual global minimum of the ImageNet classification task is an undetermined quantity because of inherent label noise. Nonetheless, our models reach state-of state-of-the-art accuracy for the given dataset and architecture (> 70% validation accuracy for ResNet18). Given this, we believe that our models serve as reasonable and sufficiently reliable proxies for global minima.
C STATISTICAL SIGNIFICANCE OF OUR RESULTS
In this section, we validate our choices concerning the reporting of our results and demonstrate that findings statistically significant.
C. SAMPLE SIZES FOR INTERPOLATION
An essential part of our experimental setup involves computing the loss barrier between two given networks. To achieve this, we selected a set equally spaced points between / = and = I and evaluated the interpolated models at these points. This process is computationally intensive and becomes slower as the number interpolation points increases. Throughout this study, we used the points t 0.0, 0.1, ... 1.0. In this section, demonstrate that the number of interpolation points we used does not negatively impact the significance our results. To this end, we conducted ablation study CIFAR ReReeNet18 models, varying the number of interpolation points for
22

---

Published as a conference paper at ICLR 2025
computing the loss barrier. We present the comparison in Table 7 As we increased the number of interpolation points from 11 to 51, we observed a decrease 0.007 in the "regular-regular" barrier and increase of 0.004 in the "star-regular" barrier. For reference, these differences are less than the standard deviations in the corresponding observations, and are thus statistically insignificant for final conclusions.
C.2 SIZE OF HELD-OUT SET
To reduce noise in our results, we compare barriers after computing them for several model pairs. The size of the held-out set H is usually 5, and sometimes even 3. Here, we confirm that this is a large enough sample size and that considering larger set of held-out models does not change our results fundamentally. In particular, we vary the number of held-out models [ and source models [Z] and obtain the corresponding mean barrier values well as standard deviations. First, we set both H| and Z to 3. Then, we set H = [2] = 5 and finally, H = |Z| = 15. In each case, interpolate all held-out models with all source models. Hence, in the last case, perform 225 "regular-regular" barrier computations and 15 "star-regular" barrier computations. We present the results in Table 8 We observe that the average "regular-regular" loss barrier between two arbitrary models remains larger than 0.37 throughout, with a standard deviation close to 0.05. In contrast, the average "star-regular" barrier remains lower than 0.1, with a standard deviation lower than 0.02. None of the observed metrics or our conclusions change significantly when increasing the number of samples. This observation provides confidence that our practice setting the number of held-out models to 3 or 5 provides reliable estimates while also being computationally cheaper.
C.3 MAXIMUM AND MINIMUM BARRIERS
Throughout the study, we consider mean values of "regular-regula"" and "star-regular" loss barriers for comparison. Here, additionally compare maximum and minimum barrier values for each model pair and confirm that the same trend holds, i.e., "star-regular" barriers are lower than eregular-regular barriers. We present the results in Table We observe that the sample size increases, the minimum barrier values go down, while the maximum barrier values go up for both "regular-regular" and "star-regular" pairs. But the minimum barrier obtained by "regular-regular" pairs is still 0.25, which is sig significantly higher than even the maximum "star-regular" barrier, i.e., 0.117. This observation confirms that the model, on average, exhibits better linear connectivity with other arbitrary models, than even the most "connected" arbitrary models exhibit between each other.
D ABLATIONS
D.1 TEST METRICS
The main discourse around mode connectivity (Garipov et al. 20188) as well convexity (?) is built around training loss and accuracy. Therefore, our empirical observations support of the star domain conjecture also primarily (Section 4. require our conclusions to hold for the test set. Therefore, we also examine the veracity of our claims with respect test loss and accuracy. We present our representative findings in Table 6 Similarly to the training set results (Table []. we observe that the "star-regular" test loss barriers consistently lower than "regular-regula" test loss barriers. Notably, for VGG11, our star model achieves a zero test loss barrier with regular models, in comparison to a barrier of 0.24 between two regular models. Overall, our observations indicate that the star domain conjecture holds for both training and test losses.
the training set. However, applications like Bayesian model averaging
D.2 CONNECTIVITY TO ADAM SOLUTIONS
Throughout the study, we have largely focused on the SGD-trained solutions and the star model built from SGD-trained source models. Here, we examine the connectivity between SGD solutions, SGD-induced star models and Adam-trained solutions. Figure shows the loss landscape across different types of solutions. We observe, as before, the loss barrier between SGD solutions and our star model 8* is nearly non-existent, while the SGD solutions are generally not linearly connected. Our star model 0* shows less connectivity with Adam solutions (the curve between "adam" and
23

---

Published as a conference paper at ICLR 2025
Table 6: Empirically verifying the star domain conjecture. "Regular loss" and "Star loss" indicate lest losses for regular models in Z and star models 0*, respectively. "Star-regular" refers to the barrier B (0", 9%)) between a star model and one of the heldout models in H. For comparison, "Regulat-regular" is the test loss barrier B (8 0B) between two arbitrary models. We report values up to one standard deviation over several runs, except for ImageNet. In each case, star models exhibit significantly lower loss barriers with other models, than the corresponding average loss barrier between two regular models. This trend is consistent with our observation for training losses in Table
Dataset
Architecture
Regular loss
Star loss
Regular-regular
Star-regular
CIFAR-10
ResNet-18
0.181 0.005
0.222
0.336 0.057
0.035 0.007
CIFAR-10
ResNet-18 (ADAM)
0.334 0.010
0.299
1.099 0.516
0.168 0.015
CIFAR-10
VGG11
0.421 + 0.011
0.456
0.242 0.036
0.000 + 0.000
CIFAR-I0 10
VGG19
0.444 = 0.019
0.395
0.903
0.150
0.117 = 0.067
CIFAR-10
DenseNet
0.269 1 0.012
0.290
4.405 + 0.730
1.612 + 0.408
CIFAR-100
ResNet-18
0.925 + 0.007
1.216
2.306
0.039
0.447 + 0.051
CIFAR-I00 100
DenseNet
1.312 0.019
1.115
5.613 0.209
3.005 = 0.176
ImageNet-Ik
ResNet-18
1.203
1.634
5.477
2.548
Table 7: Different sample sizes for computing loss barriers. We report the "regular-regular" and "star-regular" loss barriers computed using eq. (IO), with different sizes of the set T of interpolation points. Using 51 interpolation points instead of 11 does not lead significant change in the computed loss barriers.
Sample size
Regular-regular
Star-regular
11
0.376
0.055
0.090 + 0.007
21
0.369
0.053
0.089 + 0.006
51
0.369 0.053
0.094 = 0.008
Table Computing average loss barriers over different sizes of the sets of source models 7 and heldout models H. We compute minimum, average and maximum "star-regular" barriers ("SR-min", "SR-avg." " and "SR-max." respectively) over varying numbers of heldout models H. For comparison, we compute minimum, average and maximum "regular-regular" barriers ("RR-min"" "RR-avg." and "RR-max." respectively). Increasing the number of held-out models from to 15 does not significantly change the observed trend.
Sample size
RR-min.
SR-min.
RR-avg.
SR-avg.
RR-max.
SR-max.
0.300
0.082
0.376 = 0.055
0.090
0.007
0.445
0.095
0.292
0.069
0.381 = 0.059
0.077 + 0.007
0.530
0.084
15
0.255
0.071
0.382 = 0.046
0.093 = 0.014
0.540
0.117
24

---

Published as a conference paper at ICLR 2025
2.0
Training loss
1.0
0.0
star
sgd
sgd
adam
adam
star
Figure 7: Training loss landscape across SGD models, Adam models, and SGD-induced star models. We plot test loss across different types solutions in S. Our star model 0* ("star" in the plot) is constructed from set of SGD-trained models Z. We note that the star model is well-connected with SGD solutions. There remains a loss barrier between the star model and Adam solutions, but significantly lower than the barrier among Adam solutions.
"star-sgd") than with SGD solutions. However, note that the loss barrier is significantly lower than for the linear interpolations between pairs of Adam solutions (the curve between "adam" and "adam"). Based on this observation, we conclude that while the scope of our conjecture remains within SGD-trained solutions, there are hints that our star model shows enhanced connectivity with other types of solution subsets.
D.3 COMPARISON WITH SINKHORN-REBASIN
Guerrero Peña et al. (2233) introduced novel permutation finding algorithm viz., Sinkhorn-rebasin
aimed at reducing the loss barriers between two arbitrary models. When using the data-free setting,
the method Guerrero Peña al. (2023) can be considered differentiable form of weight matching Ainsworth et al. (2022). The authors notably demonstrate that their method performs better than weight matching on average, albeit it only partially eliminates loss barriers between two given models. However, warkkhon--ebbastee does not yet support networks with skip connections, making it unsuitable for our experiments involving ResNets and DenseNets. Additionally, Sinkhorn- requires hyperparameter tuning (such as optimizer and learning rate), which could introduce confounding factors into our experiments.
The primary objective of this paper is to investigate the empirical validity of our star domain conjecture. We find weight matching sufficient for this purpose. However, our empirical verification is based upon comparing "regular-regular" barriers, i.e., loss barriers between two arbitrary solutions, and "star-regular" barriers, iee,, loss barriers between the star model and other arbitrary solutions. is important verify how much Sinkhom--rebain can further improve these Pregular-regulit loss barriers. To this end, we compare "star-regular" barriers regularrregular" barriers after applying Sinkhorn-rebasin (SH) instead of Weight Matching. We use VGG19 models with batch normalization, trained on CIFAR-10, 10, for this purpose. First, we perform a hyperparameter search on the learning rate for the permutation-findie finding algorithm Guerrero Peña al. (2023). using Cra distance (as described in Guerrero Peña et al. (2023)) as the optimization objective. Our search space is the 0.01, 0.1, I.O, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 80.0, 100.0, 120.0, 150.0, 200.0. We observe the best "regular-regular" loss barrier between two pretrained models from Guerrero Peña et al. (2023). learning rate of 150.0. This loss barrier forms the reference point "regular-regular comparison. Next, we train star model using our own source models, and then compute the "star- regular" barrier with one of the pre-trained models from Guerrero Peña al. (2023). The results are presented in Figure
our
The "regular-regular" barriers obtained in our experiment are comparable to those reported in Figure 6 of Guerrero Peña et al. (2023). We observe that in this particular case, Sinkhom--rebasn exhibits higher loss barriers than weight matching, although might be possible to reduce this barrier further with a different set of hyperparameters. Nevertheless. "star-regular" barriers remain lower than "regular- regularr" barriers in both cases. Future work may look more closely into the effect of using different permutation algorithms.
25

---

Published as a conference paper at ICLR 2025
1.2
star-WM
1.00
regular-WM
1.0
star-SH
0.95
regular-SH
0.8
0.90
train costs
0.85
0.6
train
0.80
0.4
0.75
0.2
0.70
0.0
0.65
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0,4
0.6
0.8
1.0
1.2
0.90
1.0
0.85
test costs
0.80
0.8
test
0.75
0.6
0.70
0.65
0.4
0.60
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 8: Effect of using Sinkhorn-rebasin instead of weight matching. We compare "star-regular" barriers against "regular-regular" barriers after fixing the permutation algorithm used match the weights of the two models being interpolated. star- WM and regular-Wil WM refer to the barriers after applying weight matching Ainsworth et al. (2022). Similarly, star-SH and regular-Si refer to barriers after applying kinkhom-rebasiie Guerrero Peña et al. (003). While vanilla WM outperforms SH in this case, better hyperparameter choice may eventually cause SH to outperform WM. Nevertheless, the findings this investigation further support our conclusion that models are well-connected to other regular models, in comparison to how well the regular models are connected amongst themselves.
D.4 DIFFERENT SAMPLING SCHEMES FOR MONTE- MONTE-CARLO PPTIMIZATION
While training our models using Starlight (Algorithm I sample the interpolation factor from On 1]. However, empirical results show that loss barriers are typically achieved close to the center of the interpolation line 0.5). This raises the question: would Starlight be more efficient if the sampling scheme placed more weight toward the center of the interpolation line? To test this, we run ablations with different sampling schemes.
Uniform (used in main paper): Unif(O, 1].
Beta: ~ Beta(2, 2). We sample more around t = 0.5.
Constant 0.5: /. 0.5.
26

---

Published as a conference paper at ICLR 2025
Table 9: Different sampling schemes for Starlight. We report the "star-regular" and "regular-regular" barriers for each sampling case; averages were computed over three different runs using different random seeds. While sampling from a beta distribution "Beta" performs slightly better, the difference is too small to be statistically significant.
Sampling scheme
Regular loss
Star loss
Regular regular
Star-regulat
Unif[0, 1] Beta 2)
0.001 0.000
0.001 + 0.000
0.383 1 0.056
0.084 1 0.012
do
0.001 = 0.000
do
0.069 + 0.010
ConstantO..5
do
0.082 = 0.004
- do
0.018 + 0.009
We show the results in Table 9 obtain star models with identical training loss (0.001). Beta achieves a slightly better loss barrier (0.069) than Uniform (0.084), while the difference is not particularly significant in this case. Constant sampling obtains a much worse star model in terms training loss (0.08). However, the finding with Beta suggests that Starlight can potentially benefit from better sampling schemes for in future work.
We observe that both the Uniform and Beta sampling schemes
D.5 EFFECT OF PERTURBATIONS
While the main point of our work is to show that a star model exists, it would also be interesting to find out whether other models in the vicinity of the discovered star model, also exhibit star-model-like properties. To test this, we slightly perturbed the CEFAR1O-ReNNet star model and evaluated its "starness". Specifically, we sampled points 0. l + e from the surface of a sphere of radius around 0* We consider three samples for each perturbation radius, and calculate barriers with three held-out models. Results can be found in Table TO. Indeed, perturbations with 16 are still good star model candidates.
Table 10: Barriers after perturbation. Even after perturbation, star-regular barriers remain substan- tially lower than the regular-regular barrier of 0.38, indicating that there exist several star models instead of just one.
E
Star loss
Star-regular barrier
21
0.001 + 0.000
0.078 + 0.009
22
0.001 + 0.000
0.080 + 0.009
23
0.002 + 0.000
0.086 + 0.007
24
0.012 + 0.001
0.120 + 0.013
D.6 MODIFIED STARLIGHT: SEGRLRE BARRIERS
Here, we investigate the starness star models trained with the modified objective in Section 4.2 We consider ResNet18 models trained CIFAR-10 and CIFAR-100, reporting the mean of 3 x 9 star-regular barriers for each case. Results can be found in Table IT Similar to the trend shown in Figure [ the star-regular barriers steadily decrease (e.g., from 0.31 to 0.12 for CIFAR-10) with an increasing number of source models, and remain significantly lower than the average regular-regular barrier (e.g., 0.38 for CIFAR-10).
27

---

Published as a conference paper at ICLR 2025
Table 11: Star model barriers with the modified objective, after the addition cross-entropy term to the loss. The barriers follow the decreasing trend shown in Figure [ and for 50 source models, they become significantly lower than the average eegularregular barrier.
Dataset
#source
Star-regular barrier
CIFAR-10
0.31 0.05
CIFAR-10 10
5
0.25 + 0.05
CIFAR-10 10
50
0.12 + 0.02
CIFAR-100
2
2.70 + 0.06
CIFAR-100
5
2.10 = 0.10
CIFAR- 100
50
0.76 = 0.09
D.7 ROLE OF ACTIVATION FUNCTION
Our experiments largely make of networks with the ReLU activation, homogeneous function. To test the influence of ReLU's homogeneity on our results, we conducted minimal experiments using two non-homogeneous activation functions: ELU (Clevert et al. 20T6) and Swish (Ramachandran et al. 2018). Results are presented in Table 12 Even in this minimal setup, we observe that star- regular barriers are significantly lower than regular-regular barriers (for instance, 0.224 0.497 for ELU). We expect the star-regular barriers to get better with more source models. Our results indicate that star-domainness can emerge even for non- non-homogeneous activation functions.
Table 12: Barriers with different activation functions. We used 3 held-out models and 10 source models each case. We observe lower star-regular than regular-regulat barriers even in this minimal setup using non-homogeneous activation functions. This indicates that "starness" may not be consequence of the homogeneous property of the function. As a reference point, we report the star-regular barrier for ReL.U star models trained using 10 source models.
Activation function
Regular loss
Star loss
Regular-regular
Star-regular
ReLU
0.001 +'2 0.001
0.001
0.383 + 0.056
0.19
ELU
0.007 + 0.001
0.068
0.497 + 0.174
0.224 + 0.021
Swish (SiLU)
0.001 + 0.000
0.009
0.347 + 0.046
0.208 + 0.019
E ADDITIONAL RESULTS
Our main results are reported in Figure and Table in the main paper. For the sake of completeness, we report interpolation plots for the rest of our experiments in this section. We include plots for DenseNet (Figure(). VGG (Figure 10), and ImageNet-1k 1k (Figure I1).
28

---

Published as a conference paper at ICLR 2025
CIFAR CIFARIO
100
100
50
50
10
10
100
100
50
50
star source Z
star 0* heldout H
star ("" source star
heldout
Train loss
Train accuracy
Figure 9: Loss barriers for DenseNett40-12 star models on CIFAR. We plot training loss and
accuracy curves obtained upon interpolation between star-regular and regular-regular models pairs.
Star-regular loss barriers continue to lower than regular-regular barriers, as observed in Figure
100
100
VGG11
0.25
0.25
90
90
0.00
0.00
80
80
100
100
VGG19
80
80
star 0* source Z
star 0* heldout H
star 0" source Z
star 0" heldout H
Train loss
Train accuracy
Figure 10: Loss barriers for VGG star models trained on CIFAR-10. Training loss and accuracy curves obtained upon interpolation between star-regular and regular- regular model pairs. We observe the same trend as in Figure
100
100
5.0
5.0
50
2.5
2.5
0.0
star 0* source Z
star 0*
heldout H
star 0* source Z
star 0* heldout H
Train loss
Train accuracy
Figure 11: Loss barriers for ResNet18 star models trained on ImageNet-Ik:: training loss and accuracy curves obtained upon interpolation between star-regular and regular-regular model pairs. While ImageNet models struggle to achieve LMC, star-regular barriers still fare much better than regular-regular barriers.
29


Reviews:
[{"review_id": "lBJIbo5Tcs", "reviewer": "Reviewer_H3sm", "summary": "The paper proposes a weaker version of the conjecture that the minima recovered by SGD are convex up to permutations, proposing instead that this set is star-shaped up to permutations. A small theoretical motivation is given for shallow linear networks, but the main evidence is empirical, where the authors propose an algorithm to find the star model at the center of the start shape from a ensemble of trained networks. They then observe that the star-shaped conjecture is approximately true for networks where the convex conjecture is not. \n\nFinally two alternatives are proposed to ensembling: sampling from the star shape, or taking the star model. Both seem to be promising.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, and the conjecture makes sense and is easy to understand. The experiments are quite convincing, though one could obviously alway try larger/deeper networks.", "weaknesses": "The authors do not propose any intuition why star-shape is the right choice to fix the convex up to permutation conjecture. In the theoretical results, can star-shapedness be guaranteed for significantly smaller widths than convexity?\n\nStar-shape is significantly weaker than convexity (especially in high dimension, I would guess), so it is not completely surprising that one can reach a lower barrier. On the other hand switching from convexity to star shape makes this conjecture much harder to check and compute, since one has to find the star model at the center, and the experiments suggest that one needs a very large amount of models to find this star model accurately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfWbcHiaOe", "reviewer": "Reviewer_eHk4", "summary": "# \n\nThe paper raises a conjecture if the trained neural networks, after taking into account for the permutation symmetries, form a star domain, i.e. there exists a neural network that is linearly connected to all the other networks. \n\nAuthors propose an algorithm to find the model that is star model by utilizing permutation matching algorithms and by defining a loss that minimizes the error barrier between the star model and other networks in the set.\n\nAuthors empirically validate the conjecture by showing that one can find a star model that is linearly connected to all the networks. \n\nAuthors show that model found by Starlight leads to mild improvement in generalization than individual models but still is a bit worse than model ensembling", "questions": "I might be missing it but are there any experiments where Starlight algorithm finds a model where permutations fail to reduce the error barrier?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Paper is easy to follow and give sufficient background.\n- Introduced the star-conjecture that is stronger than mode connectivity as in [1] but weaker than weak linear connectivity at defined in [2]", "weaknesses": "- Relationships with other notions of linear connectivity is not very well discussed.\n- If I understand correctly, the experiments only conducted when there is weak linear connectivity modulo permutation holds since weak linear connectivity implies the star conjecture:\n    - If weak linear connectivity modulo permutation holds, one can pick any model $\\theta$ from the set $Z = \\{\\theta_1, \\dots, \\theta_N\\}$ and find permutation $P_1, \\dots, P_N$ that linearly connects the model $\\theta_i$ to $theta$. This should lead to star connectivity\n- Results for Starlight over model ensembling are not very extensive,\n\nMinor comments:\n\n- the bibliography needs many fixes. Specially because authors cite the arxiv version of the paper but should cite the publications", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wpn0LbnIFJ", "reviewer": "Reviewer_3x6L", "summary": "This paper conjectures that SGD solution set of neural networks is a star domain. \nThey have proved their conjecture for a toy model of neural networks, namely shallow linear neural networks with just one-dimensional input data . They also propose a Starlight algorithm to find a star model of a given task. It is mentioned that the benefit of having star domain for network solution is in Bayesian Model Averaging. Authors also supported their claims by proving empirical studies.", "questions": "1- This questions is related to the title and the whole paper `DO DEEP NEURAL NETWORK SOLUTIONS FORM A STAR DOMAIN?'\nDo authors by 'solution' mean the global minima of objective function? If yes, how can authors make sure that global minima are reached in practice by SGD (objective functions are highly non-convex), let say in practical settings like limited width of the neural network? This question also concerns your empirical observations. \nAlso `The learning problem for a neural network is inherently characterized by a non-convex loss landscape, leading to multiple possible solutions rather than a singular one.' As far as I am aware, the first and main concern in optimization of NN is non-global optima solutions. \nAnd could you please elaborate what is the role of SGD here? Why not other optimization methods? \n\n2-Theorem 1 is targeting a toy model of neural networks (shallow, linear, ...), could you please elaborate how this theory under restrictive Assumption 1 can support your Conjecture? Could authors also elaborate on how Assumption 1 makes sense in practice? \n\n3-Line 236: How can you reach all those solutions, while it is computationally too expensive? In fact, there are infinitely many solutions for neural networks, it kinda seems too far to extract a set of solutions. How large that set need to be to leave you a star point? \n\n4-Line 267: It is not clear for me how authors handle finding optimal permutations? Could you please elaborate more?  \n\n5-Theorem 1 is valid for $m\\to \\infty$. How authors can handle this in practice? \n\n6-Line 502, it is stated that this approach is useful to decrease computational expenses, but I am not sure how it can happened following your proposed setting in 236 and solving (3). \n\n7-Better to clarify about importance of start shaped models sooner in text regarding your comments in section 4.2.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is written clearly at some parts but I believe the writing can still be improved. \nFor example, it was not clear at the beginning of paper why such a star model property is important in NN, but authors explain this later \nin their section 4. \nThe main contribution of the paper seems to be interesting, but I am still not convinced about their details.", "weaknesses": "I think beyond their Theorem 1 that is proved for a really toy model of neural networks under strict assumptions, there is no theoretical support over their claimed conjecture. \nI also believe there are confusing points in the paper that need to be clarified, please see my question list below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9qtzxtmBZn", "reviewer": "Reviewer_gkBc", "summary": "The paper presents a conjecture regarding the nature of solution sets for deep neural networks (DNNs) trained via stochastic gradient descent (SGD). It introduces the concept of a \"star domain,\" proposing that these solution sets can be characterized by a special \"star model\" that connects to all other solutions in a low-loss path, modulo permutation invariances. The paper proposes the ``Starlight\" algorithm that finds a star model of a given learning task and confirms existing reports that the convexity conjecture requires very wide networks to hold based on experimental results.", "questions": "-  Theorem 1 is established for an oversimplified two-layer linear network. Can the author at least show results with two-layer network with nonlinear activations? Otherwise, the theory is too weak to support the star domain conjecture.\n\n-  Follow-up questions : How do the authors envision extending the theoretical framework of the star domain conjecture to encompass a broader range of neural network architectures and training methods? What specific conditions or assumptions must hold for the star domain conjecture to be valid in practical applications?\n\n-  About Empirical Validation: how do the authors plan to validate the star domain conjecture across different types of neural network architectures (e.g., transformers, recurrent networks)? Can the authors provide additional insights or results regarding the performance of the Starlight algorithm in high-dimensional parameter spaces?\n\n- About algorithm complexity: Given the complexity of the Starlight algorithm, what steps have the authors taken to ensure its computational efficiency and ease of implementation in practical settings? Are there potential optimizations or modifications to the Starlight algorithm that could enhance its performance, especially in large-scale applications?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduction of the star domain conjecture adds a novel perspective to existing theories regarding the connectivity of neural network solutions. The Starlight algorithm is well-defined and offers practical means to identify star models, contributing to the field of model ensemble methods. The paper includes thorough empirical validation across various architectures and datasets, strengthening the claims made.", "weaknesses": "While the empirical results are promising, the theoretical underpinnings of the star domain conjecture could be elaborated further. The authors acknowledge challenges in validating the conjecture theoretically but could provide more insights into potential avenues for future theoretical exploration.\n\nThe experiment results primarily focus on specific settings (e.g., ResNet, CIFAR-10). Broader validation across different tasks and architectures could enhance the generalizability of the findings.\n\nThe algorithm's complexity may pose challenges for reproducibility. Detailed discussions on implementation and computational overhead would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)