Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks
Authors: Xin-Chun Li
Jin-Lin Tang
Bo Zhang
Lan Li
De-Chuan Zhan
Venue: NeurIPS 2024 poster (2024)
Keywords: Loss Landscape
Asymmetric Valley
Model Fusion
Perturbation Analysis
Primary area: optimization_for_deep_networks

Abstract: Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks
Bo Zhang Lan Li De-Chuan Zhan School of Artificial Intelligence, Nanjing University, Nanjing, China National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China flixc, tangil, zhangb, 1i1)@lamda, cn, zhandcenju. edu. en
Xin-Chun
Tang Jin-Lni Jii-LLin
Abstract
Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests additional asymmetry of the valley beyond the flat and sharp ones. yet without thoroughly examining its causes implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for ID visualization. Our major observation shows that the degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates
with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.
Introduction
The massive number of parameters and complex structure deep neural networks (DNNs) have catalyzed extensive research to mine their underlying mechanics [55] 34
42
622
71). Visualizing
and exploring the loss surfaces of DNNs is the most intuitive way 43 18 which has ignited
many interesting findings, such as the monotonic linear interpolation 22 19 66 and linear mode
connectivity 22 14
21
65
Loss landscape visualization has also been applied to show the understand the effectiveness of Batch Normalization [3 61]
optimization trajectory 53
33
35
BERT 23, deep ensemble
17
30
21]
and so on.
i.e., the shape of the valleys they reside in, is very popular research topic. The concept of flat minima originally proposed by 28). who defines the size of the connected region around the minima where the loss remains relatively unchanged flatness. Subsequent studies debate whether the flat or sharp minima could reflect the generalization ability [38 (33 36) 57
Perturbation analysis around the local minima of DNNs
63
41 B). The previous works constrain the valley shape to be symmetric, while recent work points out that all DNN valleys are flat or sharp, and there also exist asymmetric valleys 25 which has not been systematically studied as far as we know.
[6] 12
This paper in-depth analyzes the factors that may affect the valley symmetry of DNNs. Previous work's s analysis of valley shape primarily utilizes the ID interpolation of 0, + Xe, where O5 represents the minima solution and denotes a random noise. As shown in Fig. [ we believe that the valley symmetry depends both on the convergence solution and noise, with each of them being influenced by some factors. The most significant innovation our research is considering the effect of noise direction on valley visualization, as previous work has simply taken the Gaussian noise.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(A) Rew Notse
(8) Filter Naron-cciled Noise
(C) Nermescaled Noise
Spreusee Ore de (-14)
Akz
Vally So
The illustration of investi- gated factors that could affect the valley symmetry. The matters a lot.
Figure 1:
Figure The illustration of different visualization methods for 1D visualization. The norm-scaled noise unifies the mag- nitude of various noise without changing their directions.
Specifically, we start by comprehensively and carefully determining the visualization method plot the valley shape, which could influence the conclusion significantly [31 43, 41]. We finally take the Norm-Scaled (NS) visualization method [31] that normalizes the noise and further scales to ||0y|| for better determining the plot range of X, where the direction of raw noise is not changed. Then, after investigating common noise directions and 6 special ones, we conclude: the degree of sign consistency between the noise and the convergence solution should be a determining factor for asymmetry. This phenomenon is basically insensitive to the utilized datasets. Next, we focus on the impact of the network architecture with or without Batch Normalization (BN) [32,, indicating that the BN initialization also impacts the valley symmetry. Finally, different hyperparameters lead to solutions with various valley widths but show no asymmetry consistently.
Aside from empirical observations, theoretical insights for our interesting findings are provided. We first declare that adding sign-consistent noise to parameters may have larger probability of keeping activating the neurons or keeping the overwhelming score in classification tasks. Then, we show
that the trace of the Hessian matrix along the sign-consistent direction is smaller, implying a flatter
region. The above findings inspire applications in the fields of model fusion 47 68 33 21. This
paper first explains why model aggregation based on pre-trained models often leads to performance
improvements, i.e., the success of model soups 688 and then proposes constraining the sign of DNN
parameters in federated learning [54 73 50 to facilitate aggregation.
Our novel contributions directions that have not been studied yer; (2) proposing that the flat region could be expanded along the direction that has a higher sign consistency with the convergence solution; (3) pointing out the influence of BN and its initialization on valley symmetry, (4) presenting theoretical insights to explain our interesting finding; (5) explaining and inspiring effective algorithmss in model fusion.
be summarized (1) exploring the valley shape under different noise
2 Related Works
Exploring the Valley Shape of DNNs. The valley around a minima solution has originally been viewed as flat or sharp (28). and the large batch size training may lead to sharp minima with poor generalization (38. However, [12] declares that two solutions that are scale-invariant in performance may lie in regions with significantly different flatness. 43 verifies the findings of [12] by applying
varying weight decay to small and large batch training, leading to results contrary to (8] With the
filter-normalized plots, 43 again observes that sharpness correlates well with generalization error. Later work also shows that the sharpness calculation should be dependent on the parameter scale [41] or the dataset [3]. Whether valley width reflects the generalization is still inconclusive. The proposal of asymmetric valley (55 further throws this debate into limbo, as valleys around minima could be flat on one side but sharp the other side, which makes it more difficult to define flatness. This paper thoroughly studies the causes and implications of the unexplained asymmetry phenomenon.
Exploiting the Valley Shape of DNNs. Exploring the valley shape of DNNs could help us better understand the inherent principles of DNNs. The work (43 utilizes 2D surface plots to show that the residual connections in ResNet 27 could prevent the explosion of non-convexity when networks get deep, and [T] attributes the success of BN to its effects of making landscape significantly more smooth. The asymmetric valley (25] provides a sounding explanation for the intriguing phenomenon

---

in stochastic weight averaging 33 Additionally, studying the valley shape could also benefit the proposal of effective optimization algorithms, e.g., the entropy- -bsed SGD (6), and the (adaptive) sharpness-aware minimization [66 41 Penalizing the gradient could also lead to solutions around flat regions 77.. We also apply the findings this paper to the area of model fusion.
Model Fusion. Directly averaging two independent models may encounter a barrier due to training randomness or permutation invariance [4 [2 2 51 24]. However, if the two models are generated from separate training of a common pre-trained model, the model fusion may perform better than individual models, i.e., the model soups [88 68 A recent work [70] finds that resolving sign conflicts when merging multiple task-specific models is neccessary. which is most related to our current work. We will explain the success of model soups based on the relation between the asymmetrii valley and the sign consistency of model parameters. Popular federated learning algorithms also take the parameter averaging process to fuse the individual models updated on isolated clients [54 52).
A huge challenge is the Non-Independent and Identical Data distributions of data islands (Non- I.I.D data) 73 29), which could make local models too diverged to merge. Multiple regularization
methods are proposed to align parameters before model fusion [46 45 37]. We propose effective regularization method that focuses on the sign of parameters, which is inspired by our interesting findings.
3 Basic Notations and Preliminaries
Our major tool is plotting the ID error curve of DNNs following the formula O, + de (Fig. (A)). 0, denotes the converged model, and denotes noise vector sampled from a specific distribution. More about the visualization of DNN loss landscape could be found in [4] 49).
3.1 Previous Studies: Exploring Oy with Fixed
e
The previous studies focus studying the valley shape under different fy, and aim to mine the shape's 's relation to generalization. To mitigate the influence of parameter scales and make the visualization fairly comparable between different 0y. [43 proposes the filter normalization method to
properly visualize the loss landscape. The processing of the noise is e² + |Teaa]
[ where i is the index of layer and is the index of filter. This way normalizes each filter in the noise to have the same norm of the corresponding filter in the converged point 0y. Further, [41] proposes a proper definition of sharpness (i.e., adaptive sharpness) based on the filter normalization, extending it to all parameters and formally defining: To, diag (concat (I] [||
"ar,,,/lll.....l"""""" Iw)))),
where f with 1 S S m denotes the j-th convolution filter in O, and ny is the number of parameters it owns. wb with s j S q denotes the j-th parameter that not included in any filters. I is a vector with all values as one. Then, the adaptive noise To is utilized to study the sharpness of 0y.
3.2 Our Study: Exploring and À with Fixed 0,
Different from the previous studies, we aim to explore the valley shape of a fixed 9y under different and X. First, the direction of ID interpolation in previous works is limited to the Gaussian noise, while we study the impact of different noise types. Second, setting À positive or negative could obtain a valley with different flatness, i.e., the asymmetric valley (25). Hence, under the fixed 0y. we do not need to rectify the noise direction -ilter-wisely. We take the visualization way used in [31], which only normalizes the noise and then re- re-scales it to the norm of 8,, i.e., c + |d|| 110 rll. The utilized Norm-Scaled (NS) noise is shown in Fig. (C). Compared with Filter NS noise (Fig. (B)). this way does not change the direction of the noise and shows the original valley shape along the direction e. Additionally, to plot 1D error curves in the same figure, we fix À in the range of [-,, Another scale factor s is added to control the visualized width. Overall, we use the following way to plot ID error curves: 0f +
T 11 where À E [-1, and we set 1.0 by default. The Frobenius norm is used. Notably, we utilize the NS noise by default and use the Filter NS noise when comparing the valley shape under different converged points, e.g., the studying of BN initialization in Sect. 4.2.
3

---

-II 3)
c- 01-2.
18 c- (..00.1)
NS BECHGGH G12, 21
15 (A.5)
1S
Test Error
2x62
0.0
Test Erer
1.0
<m 185
<r K1
Cr KiSgiif
Cr Sigiti
Er (ngiii
0.5
with
0.8
0.0
1.0-1.0
1.0
0.0
10.10
0.0
1:0.100
1.0
0.0
X.
X.
)
Figure 3: The valleys under 7 common noise types. The second row shows the results of replacing the sign of noise with that of y, leading to asymmetric valleys. (VGG16 with BN on CIFARIO)
4
Experimental Findings
This section presents the major findings about factors that respectively impact the noise and converged points. Experimental details and more verification results are in Appendix and B
4.1 Factors that Affect 6
We train VGGI6 64 with BN 32 on CIFARIO 39 for 200 epochs and obtain the converged model.
For the given f we plot the 1D error curves under the following 7 common noise types: (1) G(O, 1): Gaussian noise with mean as and std as 1; (2) U (-1, 1): uniform noise in the range [-1, (3) {-1.0, }}: uniform noise with values only in 11, 0, and 1; (4) G(1,1): Gaussian noise with mean as 1 and std as 1; (5) UO, 1): uniform noise the range [o. 1]: (6) {0. 1): uniform noise with values only in 0 and 1: (7) (1): constant noise with all values as 1. For each plot, we set E {0.2, 2.0) and A e [-1,1] to show curves under various levels of width. The results are shown in the first row of Fig. 3 We observe that the valley shapes along these 7 noise directions are almost symmetric, except that the last four noise directions show slight asymmetry when s = 0.2.
Then, fantastic idea motivates us to change the sign of the noise. The detail of this motivation is provided in Appendix A.44 Specifically, we use the following method replace the sign of e with that of fy: c < |<| sign( ff), where| |- returns the absolute value element- wisely and sign()) returns 1 or -1 based on whether the element positive or negative. The corresponding results of the common noises become completely asymmetric, which are plotted in the second row of Fig. 3 Furthermore, the valleys all follow the tendency that the positive direction is flat while the negative direction is sharp. Hence, we propose our major finding: the sign consistency beiween noise and converged model determines the asymmetry and the valley is flatter along the noise direction with larger sign consistency. The finding is formulated as: L(0, + an) < L(0 - an). where 17 = |el sign(O,) denotes the sign-consistent noise, and a > 0 is a constant. L(-) is the loss function, which could be the prediction error or cross-entropy loss. The following three experimental studies could further verify this interesting finding.
The Manual Construction of Noise Direction. We element- weement-wise sample the noise from G(O, 1). and then manually change its elements' sign with a given ratio r (0.0, 0.1...., 1.0). For example, r= 0.5 means that we sample 50% elements in the noise and change their sign to the same as 0y- Then, we plot the average test error the positive interpolations and negative interpolations, i.e., Ex [0r ANS(:)) with À E [0. 1] and X [-1, 0], respectively. Fig. plots the average test errors on two groups of networks and datasets. The test errors of positive and negative directions nearly equal when 0%. As becomes larger, the average test error of the positive direction monotonically decreases while the negative one increases, implying that the valley shape becomes more and more asymmetric.
The Investigation of 6 Special Noise Directions. We then investigate several special noise directions including (1) the initialization before training. i.e., 61 6u: (2) the converged model itself, i.e., c2 fy; (3) ea sign(0,): ): (4) 64 s0 μu): (5) 65 = sgp(0;) ); (6) es sgp(0, -1)). Here, y denotes the mean value for each parameter group, e.g., the mean value of "convl 1. weight"". sgp(-) returns I or based on whether the element is positive or not. The visualization results are provided in
Fig. 5 The first four directions lead to asymmetry, while the last two do not. First, the elements in e2

---

CIFAR1O VGG168N
Reerage Eiror
<
Test Error
0.4-
02
0.5
0.0
SVHN ResNet20
Average Error
0.0
2=signi0; -x)
t=sgp(0)
t=3gp00 -m
04-
Test Error
0.5
Sign Change Ratio
0.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
A
Figure 4: The impacts of manually constructed Gaus- sian noise with different levels of sign consistency.
Figure 5: The valley shape under 6 special noise types. (VGG16 with BN on CIFARIO)
ALL
CLF
FEAT
LAYER1
CONV
BN
1.0
0.0
1.0
Ure
Ere
EYo
0.5-
0.0
-1.0
1.0:1.0
0.0
1.0-1.0
0.0
1.0-1.0
1.0-1.0
10:1.0
Figure 6: Verification results on ImageNet with pre-trained ResNeXt101.
and E3 surely have the same sign with 0y, which leads to an asymmetric valley. Because the mean values of most parameters are near zero, 64 performs likely é3. 6n and eG only have the same sign with the positive parameters Oy, and applies zero to negative parameters in 0,, which shows no asymmetry. The most interesting result is the lu. whose elements may be centered around zero according to the Kaiming initialization [26). However, the BN initialization asymmetric, which leads to asymmetric curves (Sect. 4.2.1) The results of VGG11 without BN on CIFAR100 [9] show no asymmetry when e = 0 (Appendix
Fig. 17
The Finding Holds for ImageNet and Various Parameter Groups. We extend the findings to large-scale datasets and apply noise only specific parameter groups. Specifically, we use the pre-trained models (e.g., Re11eXt101 (69] downloaded from "torchvision" Because these models are pre-trained on ImageNet [00, we could directly use them to verify our findings without additional training. Multiple parameter groups are considered as follows: (1) "ALL" denotes the whole parameters; (2) "CLF" denotes the weights in the final classifier layer; (3) "FEAT" denotes the weights in the layers aside from the final classifier; (4) "LAYER1" denotes parameters in the first several blocks; (5) "CONY" denotes all convolution parameters; (6) "BN" denotes all of the BN parameters. As shown in Fig. ( applying sign-consistent noise could lead asymmetric valleys. Notably, this holds for both the metrics of CE loss and prediction error.
4.2 Factors that Affect 0f
Then we focus studying the effects of BN and its initialization, then present the results under various hyperparameters.
htte:://gyvocaa..ggggusss////bts html
5

---

BN dee ghagghhll
Blweegit - 185, 10
BN BMaeighht 600 0.1)
Parameter Distrib Juii
Parameter Disribution
cur degia
Gu Vight
Dilen whigg
1.2
80
10
Tese Ertr
NS Notse
-a use a2s use Paramalaa meats Hy
Hus tae u.s sas
Parematter miaer Ap
10
Frlee MS Notse
TIst EIstETTO
116
64
1.:01
0.0
1.:10
3.0
Figure 8: The interpolation between two models trained with batch size as 32 and 2048. (VGGI6 with BN on CIFARIO)
Figure 7: The impact of BN and its initializa- tion on the valley symmetry. (VGG16 with BN on CIFAR10)
LR=0.1
LR=0.003
BS-32
BS-2048
WD-1e-3
WD=1e-5
1.0
Test Errr
Fo Pia a
1020
0.5-
bas
0.0
|elnsign80)
Signled
1.0
Test Error
net it wane Crk 2-22
0.5
0.0
-2.0
0.0
1.0
-1.0
0.0
-14
0.0
1.0
-.00
0.0
1.0
1.0 -1.0
A
A
Figure The impact of various hyperparameters on valley symmetry. (VGG16 with BN on CIFAR1O)
4.2.1 BN and Initialization
The default initialization method of BN is setting w as ones and b as zeros [32 9.. Hence, the w, i.., the "BN.weight" may be asymmetric after convergence. We take three ways initialize the BN weights, including (1) the elements are all ones; (2) the elements are sampled from U(0, 1); (3) the values sampled from G(O, 0.1). We train three models based these three types of BN initialization. In Fig. [7] the first row shows the initial and converged parameter distribution of a specific BN weight. The traditional initialization leads to converged BN weights with all positive values, which are nearly centered around 0.2. The uniform initialization between and 1 also leads to positive converged weights. The symmetric Gaussian initialization leads converged values symmetric around 0. Then, we plot the valley shapes under the noise direction c E {0, 1}. Because this part involves comparison among different convergence points, we plot the results by both the NS noise and Filter NS noise. As vividly displayed in Fig. / the first two initialization ways encounter obvious asymmetry while the Gaussian initialization shows nearly perfect symmetry. If we carefully analyze the sign consistency ratio of them, we could easily explain this phenomenon. The noise direction {0, 1} has larger overlap with the first two initialization methods because the converged BN weights are all positive, while it has a lower overlap with the initialization from G(0, 0.1). This implies that the traditional BN initialization will lead to nearly all positive converged BN weights, which may influence the valley symmetry.
4.2.2 Hyperparameters
38 shows that the batch size could influence the valley width, and further 43 advocates that weight decay could also play role in the valley width. Different from them, we aim to study whether these hyperparameters influence the valley symmetry. We train VGG16 networks on CIFAR10 with various hyperparameters. We use the SGD optimizer with momentum value
Previously,

---

Duto
AW-LD
Aw-0d
A--00
A=-44 2=-02
A-2p
A-G2
A-Q4
A-C4
J-02
Aw12
Figure 10: The leftmost shows a digit sample from aaklearn.digit and others show the pattern of w x sign w). = 0.0 shows the learned classification weight w.
of 0.9. The default learning rate (LR) is 0.03, batch size (BS) is 256, and weight decay (WD) is 0.0005. Then, we will correspondingly set the LR in (0.1, 0.003). BS in (32, 2048), and WD in (0.001, 1.000,0.00000 0.00001). The curves are in Fig. 9 The first row applies the G 1) noise, while the second row changes its sign to the converged models' sign. Obviously, different hyperparameters may lead to valleys with various widths, while the valleys are all symmetric. The asymmetry valleys in the second row again verify the previous findings in Sect. 4.1
Then, we explore the interpolation between two solutions under different hyperparameters, which are studied in [38 43 22 We take the batch size of 32 and 2048 as an example and denote the converged solution as f, and f-- The test error curve of (1 - A)0 f, XØ5; is plotted, with À [-1,2]. Aside from the interpolation curve, we also plot the parameter distributions of f,, and 8 The parameters are divided into five groups, including "BN Weight", "CLF Weight". "CLFF Bias", "Other Weight", and "Other Bias". "CLF" denotes the last classification layer, and "Other" denotes other layers aside from the BN layers and the last classification layer. To simplify the figures, we only plot the mean and standard deviation of the parameters, denoted Up and op. Fig. [hsows the parameter distributions and the interpolation curve. The interpolation curve shows that the small batch training (i.c., À = 0.0) lies in a sharper and nearly symmetric valley, while the large batch training (i.e., X = 1.0) lies in flatter but asymmetric valley. Small batch training (i.e., 0f) leads to parameters with smaller mean and std values. This is because the utilized weight decay is 0.0005, which makes the parameter scale smaller due to longer training [43. Then, we explain the different results of valley symmetry. The interpolation formula could be re-written f,, X(0 0,,) and f/2 (2 (f() -0,, ). which respectively shows the 1D interpolation centered around f and 0z2- If we let e = 0z2 - ff. then we could plot the sign consistency ratio (i.e., how many parameters have the same sign) of 0 and c, and f,, and c. The results are in Fig. 8 where we provide the values of the five parameter groups. Obviously, 0z is more consistent in the sign values, which shows flatter region towards the positive direction. In contrast, the sign consistency ratio of 0,, is smaller, which only shows slight asymmetry.
5 Theoretical Insights to Explain the Finding
This section provides theoretical insights from the aspects of ReLU activation and softmax function to explain the interesting phenomenon. The forward process of DNNs contains amounts of calculation of h c.g., the fully connected and convolution layer. W denotes the weight matrix and h is hidden representation. A special case is the final classification layer with the softmax function. Given f R&, the ground- truth label y [C',, and the weight matrix W E RCxd the softened probability vector is p softmax h.. The cross coos--y (CE) loss function is L(W)- = -lo Py. The gradient of w, is Iw, c- = y} - h,, with c [C] and [--} being the indication function. This implies that the update direction of W lies in the subspace spanned by hidden representations, which also holds for intermediate layers and convolution layers [60]. After adequate training steps, the parameters that activate the ReLU function or correspond to the ground-truth label should correlate well with their inputs. We show a demo classification weight learned on the "sklearn. digits"" H Fig. 10 shows the pattern change of w X sign(w) w) with X E 1100 1.0]. Clearly, the weight under À 0.0 correlates well with the input sample (i.c., Digit O in Fig. I0). Setting À > 0.0 will almost keep the pattern, while À= < 0.0 destroys it significantly. That is, high score for the target class, while setting À < 0.0 may decrease the score.
À sign( w) with À 0.0 may keep providing
For the ReLU activation, it also holds that + À sign(w) will have a higher probability of keeping activating the neurons when À 0.0. To simplify the analysis, we assume the learned w equals a h+ + 5, where a is a constant and 8 is random Gaussian vector. Then we could easily verify that
https://scikit 1earn.--eeerrrrrr org/

---

(w + À * sign( "h will have higher probability of keeping activating neurons under positive À than the negative one. The details and simulation results are in Appendix [D.2] If the neuron outputs are only simply scaled by a factor, it will not affect the relative scores of the final classification, For example, the inequation of wIth h > we h will not change if h is scaled by a positive factor, while does not hold for h whose values are not activated, i.e., h 0.
Then we provide further analysis via analyzing the Hessian matrix of the softmax weights. Specif- ically, the Hessian of L(W)= - log Py w.r.t W is H = (diag(p) - pp") @ hh? where @ denotes the Kronecker product. The trace of H is tr( (H) tr(diag(p) pp") tr(hh"). The first part could be calculated as Σe Po (1- Pe), where e is the class index. According to the above analysis, adding sign-consistent noise to "y could enlarge the score of W h, which may make the Py larger and Poey smaller [48]. That is, the predicted probability vector tends to be a one-hot vector when adding sign-consistent noise, and Σ. Po(1 (p)) will be near zero. Hence, the trace of the Hessian matrix is smaller along the sign-consistent direction. Since softmax is convex, and the eigenvalues of the Hessian are all positive, smaller trace means smaller eigenvalues, which makes the loss curve flatter. The empirical observation can be found in Appendix D.3
Applications to Model Fusion
6.1 Explaining the Success of Model Soups
Commonly, the interpolation of two independently found DNN solutions may encounter barrier 22
65
Surprisingly, if these two models are updated from the same pre-trained model, then the barrier
will disappear, and the linear interpolation brings a positive improvement [58] 68). The common
explanation follows that the pre-trained model may possess less instability when compared to the random initialization models (20). We guess that the sign consistency ratio may influence the
parameter fusion performance of the two models. Perhaps, the sign of model parameters updated based on the pre-trained model remain nearly unchanged during the process of fine-tuning.
We experimentally verify our guess on two datasets, i.e., training VGG16BN with completely random initialization CIFAR10, and training ResNet18 with pre-trained initialization from PyTorch on Flowers [59]. The datasets are first uniformly split into two partitions, and then two models with corresponding initialization are separately trained or fine-tuned for 50 epochs. The checkpoints in the {1,2, 3, 5, 10, 20, 30, 50)50)-tt -th epoch are stored. For checkpoints in the specific epoch, we denote the two models on the two partitions as Oa and Op, respectively. The interpolation accuracy of A + 20g on the test set is plotted in Fig. [| The interpolation curve of VGGI6BN on CIFARIO indeed encounters a significant barrier, especially when the epoch is larger, e.g., E = 50. In contrast, the interpolation surpasses the individual models on Flowers, which is attributed to the pre-trained ResNet18. As an explanation, we calculate the sign consistency ratio between 0.1 and 0,, 0g and 0,. and A and 0g. denoted as "SSR-IA", "SSR-IB", and "SSR-AB", respectively. Oy means the initialization model. We also plot the gap of model interpolation and individual models when À = 0.5, i.e., Acc(0.50) a + 0.50,) shows that the sign consistency ratio could almost perfectly reflect the tendency of the interpolation gap. Notably, the sign consistency ratio of models on Flowers is higher than 0.95, which means that fine-tuning the pre-trained model does not change the parameter signs a lot, which facilitates the following parameter interpolation. For better illustration and fair comparison, pre-trained ResNet18 with a random initialized ResNet18, and we find the plots shown in Fig. TT tend to be like the results of the random initialized VGGI6BN. The previous work (70) points out that disagreement on the sign of given parameter's values across models is a major source of interference during model merging. Although our finding is similar to the previous work, the motivation and specific explanation differs a lot.
0.5(Acc A) + Ac)(Op)) The right of Fig. [ clearly
also replace the
6.2 Regularizing the Sign Change in Federated Learning
Traditional machine learning models will encounter the challenges posed by "isolated data islands". e.g., the Non-I.I.D. data (54 29). FedAvg (54), as the most standard federated learning (FL. method, utilizes the parameter server architecture 44,, and fuses collaborative models from local nodes without centralizing users' data. Specifically, the local clients receive the global model from the
server and update it respectively their devices using private data, and the server periodically
8

---

CIFARIO VOGLGBN
CIFARLO VOCL6BN
Flowers Resheti8 (No Pretrain)
Flowors Res (No Protran))
Perfor www.eraneeoooo Cap
Test Accuncy
Same Sin Nabo
100
05
1.0
26 50
Inte Pterpplation:
Epoch
Interpoiation: A
Epoch
Flowers Pro-Trained RecNet18
Flowerr Pro-Trained RecNor18
0:06
Test Accuracy
0.7
Some Sign Fatto
0.03
0.-
23R-A
-0.0
0.0
0.5 Interpolation
10
20
30
50
Epoch
Figure 11: The models fine-tuned from pre-trained model have a higher sign consistency ratio. The top two sub-figures do not utilize pre-trained models, while the last one utilizes pre-trained ResNet18.
Table I: Aggregation performance comparisons of FedSign with several popular FL algorithms.
Dir. a
FedAvg
FedProx
MOON
FedDyn
FedPAN
FedSign
10.0
81.53 + 0.17
81.84
82.44
80.45
81.92
82.59 + 0.09
C0
1.0
80.54 a 0.11
80.42
80.12
79.78
80.30
80.76 0.14
0.5
77.69 a 0.21
78.12
76.77
78.02
77.78
78.41 a 0.35
10.0
75.74 + 0.24
76.58
76.25
76.37
76.84
77.05 + 0.14
CINIC-10
1.0
72.19 # 0.15
72.25
72.06
73.12
73.46
74.59 0.20
0.5
68.24 + 0.32
69.11
68.55
69.01
70.14
70.63 = 0.16
averages these models for multiple communication rounds. That is, FedAvg takes the simple parameter averaging fuse local models. However, due the Non-L.L.D data and permutation invariance of DNNs [1 67 72), the local models could become too diverged to effectively merge. Numerous efforts are paid regularize the local models so as not to go too far away from the global model, such as the proximal term proposed by FedProx [46]. the contrastive regularization proposed by MOON 45 the dynamic regularization of FedDyn []] and the position-ayy aware neurons proposed
by FedPAN [51] Inspired by our finding, we propose to regularize the sign change when updating
local models, i.e.,
k = 7 (sgp(0))o sgp((0t)o
where / denotes the communication round and k is the index of client. L is the common cross- entropy loss of the k-th client, and 0; is the global model received from the server. 0% +- O, is the local model to be updated, and the loss could regularize the sign of 05 to be close to that of 0, Because obtaining the sign of parameters not a continuous function, we therefore apply a sigmoid function to parameters an approximation. We name this method FedSign and list its pseudo-code in the Appendix C Compared with other regularization methods, our proposed FedSign is rell-motivated because of our finding that interpolating sign-consistent models may lead to flatter loss region.
Experimental studies verified 1IFAR10 39 and CINIC10 g that are commonly utilized in previous works 72 [51- Decentralizing the training data of these datasets by a Dirichlet distribution could simulate the Non-L.I.D scenes as in real-world FL. The Dirichlet alpha a is utilized control the Non-I.I.D, level, with smaller representing more rigorous heterogeneity between clients' data, We set a {10.0, 1.0, 0.5) respectively. The number of clients is 100, and the total communication round is 200. During each round, a random set of 10% clients participate in FL and every client takes 5 epochs update on their individual data. After all communication rounds, we evaluate the model performance
global test set on the server (i.e., the original test set of corresponding datasets). We select our hyperparameter from (0.001,0.00 0.1} and report the best results. For a = 10.0, i.e., a relatively I.I.D. scene, smaller is better. In contrast, = 0.1 or 7 = 0.01 will be more proper for a 0.5. The performance comparison results are listed in Tab. D For FeaAvg and our proposed FedSign, we rerun the experimental studies five times and list the standard deviation of accuracies,

---

showing that the accuracy doesn't 't fluctuate very much, FedSign could surpass the compared methods, which shows the positive effects of regularizing the sign change FL.
Limitations and Future Works
Although provide theoretical insights to explain the interesting phenomenon, no formal proofs are provided to show the conditions and scopes that lead to asymmetric valleys. Additionally, this phenomenon is only investigated in the image classification tasks. Future research includes providing formal theoretical foundations for our findings and verifying them on more tasks. According to the analysis in Sect. 5 advocate that this phenomenon is more likely to be applicable DNNs that contain both the ReLU and softmax.
8 Conclusion
We explore and exploit the asymmetric valley of DNNs via numerous experimental studies and theoretical analyses. We systematically examine various factors influencing valley symmetry, high- lighting the significant role of sign consistency between noise direction and the converged model. The findings offer valuable insights into practical implications, enhancing the understanding of model fusion. novel regularization method proposed for better model averaging in federated learning.
Acknowledgements
This work is partially supported by National Science and Technology Major Project (20227D01 14805), NSFC (62376118 18, 62006112 62250069, 61921006), Collaborative Innova- tion Center of Novel Software Technology and Industrialization. Professor De-Chuan Zhan is the corresponding author.
References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In ICLR, 2021.
[2] Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023.
[3] Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. In International Conference on Machine Learning, pages 840-902, 2023.
[4] Stephen C. Ashmore and Michael S. Gashler. A method for finding similarity between multi- layer perceptrons by forward bipartite alignment. In International Joint Conference on Neural Networks, pages 1-7, 2015.
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision, 2014. [6] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. The 5th International Conference on Learning Representations, 2017.
[7] Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the robustness of convolutional neural networks internal architecture and weight perturbations. CoRR, abs/1 0ttp///03.0225.... 2017.
[8] Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not imagenet or CIFAR-10. CoRR, abs/18 0.03505, 2018.
[9] Jim Davis and Logan Frank. Revisiting batch norm initialization. In Computer Vision ECCV
2022 17th European Conference, pages 212 228, 2022.
[10] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
10

---

[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4171-4186, 2019.
[12] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, pages 1019-1028, 2017.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR,
abs/2010.11929 2020.
[14) Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference on Machine Learning, pages 1308-1317, 2018.
[15] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In The Tenth International Conference on Learning Representations, 2022.
[16] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware min- imization for efficiently improving generalization. In The 9th International Conference on Learning Representationss, 2021.
[17] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. CoRR, (bss/991..0257, 2019.
[18] Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. In Advances in Neural Information Processing Systems 32, pages 6706-6714, 2019.
[19] Jonathan Frankle. Revisiting "qualitatively characterizing neural network optimization prob- lems". CoRR, abs/2012.06888, 2020.
[20] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference Machine Learning, pages 3259-3269, 2020.
[21] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems 31, pages 8803-8812 12, 2018.
[22] Ian J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization problems. In 3rd International Conference on Learning Representations, 2015.
[23] Yaru Hao, Li Dong. Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages
4141-4150, 2019.
[24] SUN Haoran, WANG Yang, LIU Haipeng, and QIAN Biao. Fine-grained cross-modal fusion based refinement for text-to-image synthesis. Chinese Journal of Electronics, 32(6):1329-133 2023.
[25] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549-2560, 2019.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 20/5 IEEE International Conference on Computer Vision, pages 1026-1034, 2015.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[28] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1::144, 1997.
[29] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. The non-iid data quagmire of decentralized machine learning. In ICML, pages 4387-4398, 2020.
[30] Gao Huang, Yixuan Li, Gcoff Pleiss, Zhuang Liu, John E. Hoperoft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In 5th International Conference on Learning Representations, 2017.
[31] Daniel Jiwoong Im, Michael Tao, and Kristin Branson. An empirical analysis of the optimization of deep network loss surfaces. CoRR, 1b1/1612.0 2016.

---

[32] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, pages 448 456, 2015.
[33] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, pages 876-885, 2018.
[34] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31, pages 8580-8589, 2018.
[35] Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof J. Geras. The break-even point on optimization trajecto- ries of deep neural networks. In 8th International Conference on Learning Representations. 2020.
[36] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In The 8th International Conference on Learning Representations, 2020.
[37] Sai Praneeth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank J. Reddi, Sebastian U, Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, pages
5132-5143 5143, 2020.
[38] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima.
In The 5th International Conference on Learning Representations, 2017.
[39] Alex Krizhevsky. Learning multiple layers of features from tiny images, 2012.
[40] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6))84-99)) 2017.
[41] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. Proceedings of the 38th International Conference on Machine Learning. pages 5905-5914, 2021.
[42] Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems 32, pages 8570-8581, 2019.
[43] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31, pages 6391-6401, 2018.
[44] Mu Li, Li Zhou, Zichao Yang, Aaron Li, Fei Xia, David G Andersen, and Alexander Smola. Parameter server for distributed machine learning. In NeurlPS, volume 6, page 2, 2013.
[45] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In CVPR,
pages 10713-10722, 2021.
[46] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys, 2020.
[47] Weishi Li, Yong Peng. Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. CoRR, abs/2309. 0102239911698.. 15698. 2023.
[48] Xin-Chun Li, Wen-Shu Fan, Shaoming Song, Yinchuan Li, Bingshuai Li, Yunfeng Shao, and De-Chuan Zhan. Asymmetric temperature scaling makes larger networks teach well again. In Advances in Neural information Processing Systems 35, 2022.
[49] Xin-Chun Li, Lan Li, and De-Chuan Zhan. Visualizing, rethinking, and mining the loss landscape of deep neural networks. CoRR, abs/2405. 12493, 2024.
[50] Xin-Chun Li. Shaoming Song, Yinchuan Li, Bingshuai Li, Yunfeng Shao, Yang Yang, and De-Chuan Zhan. MAP: model aggregation and personalization in federated learning with incomplete classes. CoRR, abs/2404.09232, 2024.
[51] Xin-Chun Li, Yi-Chu Xu, Shaoming Song, Bingshuai Li, Yinchuan Li, Yunfeng Shao, and De-Chuan Zhan. Federated learning with position-aware neurons. In IEEE Conference on Computer Vision and Pattern Recognition, pages 10082100911, 2022.
[52] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Nearal Information Processing Systems 33,
2020.
12

---

[53]
Eliana Lorch. Visualizing deep network training trajectories with pea. 2016.
[54]
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguira y Arcas. Communication--cfiiicie learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pages 1273-1282,
2017.
[55]
Guido Montúfar, Razvan Pascanu, KyungHyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems
27, pages 2924-2932, 2014.
[56] Yuval Netzer, Tiejie Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[57] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems 30, pages 5947-5956, 2017.
[58] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems 33, 2020.
[59] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large num- ber of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008.
[60] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In 9th International Conference on Learning Representations, 2021.
[61] Shibani Santurkar, Dimitris Tsipras, Andrew llyas, and Aleksander Madry. How does batch normalization help optimization? In Advances in Neural Information Processing Systems 31, pages 2488-2498, 2018.
[62] LYU Shen-Huan, CHEN Yi-He. and ZHOU Zhi-Hua. A region-based analysis for the feature concatenation in deep forests. Chinese Journal of Electronics, 1166):1072-1080 2022. Hai Shu and Hongtu Zhu. Sensitivity analysis of deep neural networks. In The Thirt-Third
[63]
AAAI Conference on Artificial Intelligence, pages 4943-4950, 2019.
[64]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, 2015.
[65] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Advances in Neural Information Processing Systems 33, 2020.
[66] Tiffany J. Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? In International Conference on Machine Learning, pages
2022.
[67] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In 8th International Conference on Learning Representations, 2020.
[68] Mitchell Wortsman, Gabriel Ilharco. Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes. Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022.
pages
[69] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5987-5995, 2017.
[70] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems 36, 2023.
[71] YAN Yan, MA Hongzhong, and LI Zhendong. An improved grasshopper optimization algorithm for global optimization. Chinese Journal of Electronics, 30(3):551-45 2021.
[72] Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and
Xiang Chen. Fed2: Feature-aligned federated learning. In KDD, pages 2066-2074, 2021.
[73] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vīkas Chandra. Federated learning with non-iid data. CoRR, abs/ hts/1806.00582.. 2018
[74] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pages 26982-26992, 2022.
13

---

A Experimental Details
In this section, we list the datasets, networks, and training details that are utilized in the body. Finally, we provide the details that motivate us to change the sign of the noise.
A.1 Dataset
The utilized datasets include "skiearn.digits" SVHN (56). CIFARIO/100 39 CINICIO 8 Flowers [59], Food101 [5], and ImageNet [10]. We detail these datasets as follows.
"sklearn.digits" contains 1797 samples of 10 digits, with each sample being a x im- age. We use this to provide simple code demo and provide theoretical verification as in Appendix [D.3]
SVHN [56) is the Street View House Number dataset which contains 10 numbers to classify, The raw set contains 73,257 samples for training and 26,032 samples for evaluation. The
image size 32 x 32.
CIFAR10 and CIFAR100 (39 subsets of the Tiny Images dataset and respectively have 10/100 classes to classify, They consist of 50,000 training images and 10,000 test images.
The image size is 32 32.
CINIC10 8 is combination of CIFAR10 and ImageNet 44), which contains 10 classes. It contains 90,000 samples for training, validation, and testing, respectively. We do not use the validation set. The image size is 32 32.
Flowers [5 consists of 102 fine-grained flower categories, where we select 2,000 images for training and 2,000 images for testing.
Food101 consists of 101 food categories, with 101, 000 images. For each class, 250 manually reviewed test images are provided as well as 750 training images. All images were rescaled to have a maximum side length of 512 pixels.
ImageNet IT0] consists 1000 image categories for classification. This dataset is utilized to pre-train models as listed in "torchvision". Due to the large amounts of data, we only select 5,000 images in the "val" partition plot the curves corresponding figures.
A.2 Network Details
We utilize VGG (64). ResNet (27). ResNeXt 69 AlexNet
40
VIT 13 in this paper. We detail
their architectures follows:
VGG contains series of networks with various layers. The paper of VGG 64 presents VGG11, VGG13, VGG16, and VGG19. We follow their architectures and report the configuration of VGGI as an example: 64, M. 128, M, 256, 256, M. 512, 512, M, 512, 512, M. "M" denotes the max-poo max-pooling layer. VGG1 contains 8 convolution blocks and three fully-connected layers in 64]. The VGG architecture could use BatchNorm [32 not, which clearly declared in the body. If BN is used, it will be added after each convolution layer and before the ReLU activation function.
ResNet introduces residual connections to plain neural networks. We take the CIFAR versions used in the paper 27 for CIFAR10/100 and CINICIO, i.e., ResNet20 with the basic block. For Flowers, we use pre-trained ResNet18 from Py ResNet commonly 32. which is added before ReLU activation.
uses BatchNorm
AlexNet 40
consists of five convolutional layers followed by three fully connected layers.
ResNeXt 669 introduces the group convolution to ResNet and we utilize the pre-trained version downloaded from "torchvision".
VIT [13 follows the transformer architecture for image classification tasks. It divides an image into a sequence of fixed-size patches, processes these patches linearly, and then feeds them into a transformer encoder to capture the global context the image. In our paper, we
https://scikktt learn, http:///////////////o//oo//o/aaa dtw.eettt..odddddddeeeee... load_digits.
html
14

---

take 12 layers in the transformer encoder and use 8 heads in each multi-head self- atffnattenion block. We set the embedding dimension as 128 to reduce the computation burden.
A.3 Training Details
We provide the training details for obtaining converged models. We investigate the following pairs of datasets and networks. For the series of Fig. 3 and Fig. [5] we train VGG16BN CIFARIO, ResNet20 on SVHN, VGG11 with no BN on CIFAR 100, AlexNet on Food101, and VIT on F0od101. For VGG16BN ResNet20. VGGI1 with no BN, and AlexNet. we use the SGD optimizer with momentum of 0.9. We set the learning rate as 0.03. We use a cosine annealing way to decay the learning rate across 200 training epochs. The default weight decay is 0.0005, and the default batch size is 256. For training VIT, we use the Adam W optimizer with a learning rate of 0.0001. The batch size is 256, and the weight decay is 0.0005. We also take the cosine annealing way decay the learning rate.
The equation of the BN layer X w X-mx + b, where X R Cxd denotes the feature map with "/Vx*1 C' channels, and mx E RC and Vx RC are channel-wise mean and variance values of the feature map. In practice, w, b E RC are learnable parameters, while mx and Vx are running statistics that are calculated during the forward pass. When interpolating f de with BN layers, we should clear these running statistics after interpolating model parameters and feed the interpolated model to the dataset for another forward pass to calculate proper data distributions. The forward- againn process is also utilized in previous works 33 68
For Fig. 7 we only change the initialization method of BN layers and keep the other hyperparameters not changed. For the series of Fig.8) we only change specific hyperparameter including the learning rate, batch size, or weight decay. Specifically, the learning rate is varied in {0.1, 0.003), and the batch size is varied in (32, 2048), and the weight decay is varied in {0.001, 0.000013.
A.4 Motivation of Changing the Sign of Noise
The asymmetric valley is initially proposed by (25). while it does not propose the inherent principles behind the phenomenon. only points out that adding asymmetric noise (e-g.. < {0, to DNNs may result in an asymmetric valley. The symmetric noise around zero may not show such patterns (e.g., é {-1,0, 0,1,0,1).. 13). This inspires us to plot valleys along three types of symmetric noise directions and four types of asymmetric noise directions, and the results are shown in Fig. 3 Indeed, the last four types of noise are not symmetric around zero and show slight asymmetric valleys. However, this is not so obvious. Notably, the utilized network in Fig. 3 has BN layers, i.e.. VGG16 with BN.
Then, we try to apply e {0, 1} to DNNs without BN, i.e., VGGI1 without BN. The valleys become symmetric as shown in the first row of Fig. 13 Hence, this makes us consider the effect of BN layers. Fortunately, we find that the traditional BN initialization will initialize the values in "BN.weif weight" as 1, and the converged BN weights are all positive. The initial findings could be summarized as:
If DNNs have BN, and the parameters are perturbed by noise with symmetric values around zero, the valleys are symmetric. This is shown as the top first three plots in Fig.
If DNNs have BN, and the parameters are perturbed by noise with asymmetric values around zero, the valleys are slightly asymmetric. This is shown as the top last four plots in Fig. 3
If DNNs do not have BN, and the parameters are perturbed by noise with symmetric values around zero, the valleys are symmetric. This is shown as the top first three plots in Fig. [3
If DNNs do not have BN, and the parameters are perturbed by noise with asymmetric values around zero, the valleys are symmetric. This is shown as the top last four plots in Fig. 13
That is, only the second case shows slightly asymmetric valleys, where the noise (e.g., c E {0. 1)) has large sign consistency with the BN weights (e.g., 0.0). As shown in Sect. 4.2. if we replace the initialization of BN weights with a random Gaussian initialization, then the plotted valley becomes symmetric because the converged BN weights are symmetric around zero again (Fig. 7
That is, the converged "BN "BN.weight weight" are all positive values under the common BN initialization. If perturb them by asymmetric noise (c.g.. e (0, 1), e G(L, e (O, 1), ere e {1}) that has
15

---

Text Error
3.0
R5e- OZL
15 PANIL TI
WS OCL
50 E O%
1.0
0.6
1.0 1.0
2.
16:10
44
10 1.0
01
1.0
24
20-
1.0
2.2
1.
Figure 12: The valleys under 7 common noise types. The second row shows the results of replacing the sign of noise with that of yy leading asymmetric valleys. (ResNet20 SVHN)
ELl
Hi
1G--U D
IC
Tost DostEner
Toct Erour
Figure 13: The valleys under 7 common noise types. The second row shows the results of replacing the sign noise with that of 0,. leading to asymmetric valleys. (VGG11 without BN on CIFARI00)
a larger sign consistency with weight, the plotted valleys are asymmetric. This provides an explanation for the asymmetric valley found by (25).
From the above summary, we guess that the sign consistency between parameters and the noise may lead asymmetric valleys. Hence, fantastic idea motivates us to change the sign of the noise the same as 0, which leads to obvious asymmetric valleys (the bottom row of Fig. 3 and Fig. 13)
B More Experimental Studies
In this section, we provide more experimental verification results make our work more solid. We list the supplemented results by the corresponding experimental studies in the body.
B.1 More Results as Fig.
shows the valley shape under 7 common noise types. The results are plotted via training VGGT6 with BN CIFARIO. We list more results verify the finding is common across various networks and architectures, including: (1) ResNet20 on SVHN (Fig. 12 (2) VGG11 without BN on CIFARI00 (Fig. (3). Obviously, these results indeed similar, which verifies again that the sign consistency ratio matters a lot in the valley symmetry.
Fig.
We also extend the findings to the large-scale dataset and popular network architectures including: (1) AlexNet on Food101 (Fig. 14:: (2) ViT on Food101 (Fig. [5).. Applying noise with higher sign
G2.
(
Took Biror
T0ch Eroot
0.5
2.
2.
20
Figure 14: The valleys under 7 common noise types. The second row shows the results of replacing the sign of noise with that of fy, leading to asymmetric valleys. (AlexNet Food101)
16

---

DE WHINK TI
M5 OOL
ISS COA
Thut Eiror
302t Ervor
-10
0.6
1.0 1.0
10
10.10
11
10.10
91
10-110
2.0
10-10
10.11.
93
30
Figure 13: The valleys under 7 common noise types. The second row shows the results of replacing the sign of noise with that of 0y, leading asymmetric valleys. (ViT 100d
1.0
rudo
zusignoi
rusignir p0
zwsspoko
Test Error
0.5
-10
Figure 16: The valley shape under 6 special noise types. (ResNer20 on SVHN)
consistency also leads asymmetric valleys. However, the large models are relatively stable some extent, and the asymmetry is not as obvious as the results on the previous datasets and networks.
B.2 More Results Fig. 5
Fig. 5 shows the valley shape under 6 special noise types. The results are plotted via training VGG16 with BN on CIFARIO. We list more results verify the finding is nearly common across various networks and architectures, including: (1) ResNet20 on SVHN (Fig. 16): (2) VGGI1 without BN on CIFAR100 (Fig. 17). Obviously, these results are indeed similar. An exceptional case is the first sub-figure in Fig. 17 Lee., using the initialization as noise leads to a symmetric valley, while the valley in Fig. 5 and Fig. 16 is asymmetric. This is because of the initialization of BN parameters, where the
latter two utilize BN weights as all ones. However, Fig. 17 takes VGG11 without BN layers, which shows no asymmetry.
We also extend the findings to the large-scale dataset and popular network architectures including: (1) AlexNet Food101 (Fig. 18); (2) VIT Food101 (Fig. 19p. Similar to the previous results, although the asymmetry is not as obvious as the results on the previous datasets and networks, the curves still show slight asymmetry when the sign consistency is high.
B.3 More Results as Fig.
Fig. 6 shows the valley shape investigated on ImageNet with pre-trained ResNeXt101. We also provide similar results with the pre-trained ResNet50 as Fig. 20
redkgrok-p0
Zesgpi83
Test Enor
1.0
0.3
-1.0
1.0
0.0
-1.0
1.0
0.5
-10
1.0
0.0
Figure 17: The valley shape under 6 special noise types. (VGG11 without BN on (IFAR100)
17

---

Test ErErrr
0.5
-10
0.0
3.0
-10
0.0
100
-10
0.0
2.0
-10
0.0
110
0.
1.0
-10
0.0
Figure 18: The valley shape under 6 special noise types. (AlexNet on Food101)
1.0
Test Emor
0.5
0.3
1.0
1.2
0,0
-10
3.0
0.4
1.0
1,2
0.0
Figure 19: The valley shape under 6 special noise types. (ViT Food101)
B.4 More Results as Fig. 7
Fig. 7 shows the impact BN and its initialization on the valley symmetry. The results are plotted via training VGG16 with BN CIFAR10. We list more results verify the finding is common across various networks and architectures, including (1) ResNer20 on SVHN (Fig. 2T (2) ResNer20
CIFAR100 (Fig. 22). Obviously, these results indeed similar, which shows that the original BN initialization may lead to an asymmetric valley. Replacing the original initialization with the symmetric Gaussian distribution could lead symmetric valleys.
B.5 More Results as Fig.
Fig. shows the impact of various hyperparameters valley symmetry. The results are plotted via training VGG16 with BN on 10. We list more results to verify the finding is common across various architectures, i.e., ResNet20 on CIFARIO (Fig. 23 This also verifies that the hyperparameters have less impact on the valley symmetry.
B.6 More Results Fig.
Fig. 8 studies the interpolation between two models that trained under different hyperparameters, e.g., learning rate (LR), batch size (BS), and weight decay (WD). The body only shows the impact of batch size on training VGG16BN. Then, we list more results to verify the finding is common
across various architectures, including: (1) the impact of learning rate on training VGG16BN on
CIFARIO (Fig. 24 (2) the impact of weight decay on training VGGI6BN on CIFARIO (Fig 25 (3)
1.0
0.5
0.0
(-Ict-one))
Lar
CE
Cotor
-1.0
0.0
1.0.1.0
0.0
1.0-1.0
0.0
1.5-1.
06
1.0-1.0
00
10-1.00
0.0
1.0
A
A
X
Figure 20: Verification results on ImageNet with pre-trained ResNet50.
18

---

RN #.
BN W.ieigght-- 120, 11
BN weight GIO,
Wileeghht-3. 1.0
BN BN.weight 010,
BN BN.eeght 610, 0.11
Histoyram
Wi-t
Ve:al Comerped
Histagrem
Intal Conwarged
Ietal Consarped
intal Conwerged
Zenvarged
Convarged
0.
0.5
1/
0.0
05
10
0.
0.5
0.0
05
10
00
0.5
10
0.0
05
1.0
Testt Enror
NS Nole
11,
(0 (0.10,511
0t 11 1.10
11
11,
Filter NS Ross
10:00 Error
(8.3)
1.10
1010
25
11
1.0
1.0
Figure 21: The impact of BN and its initial- ization on the valley symmetry. (ResNet20 on SVHN)
Figure 22: The impact of BN and its initializa- tion the valley symmetry. (ResNet20 on CI- FARI00)
LR-0.1
LR-0.003
B5-32
BS-2048
WD-1e-3
WD-1e55
wold
Text
jetsignota
Fa
Fel
20
2.0
3.0
1.0
60
10
3.0
9.
20
3.0 1.0
10 1.0
0.
Figure 23: The impact of various hyperparameters on valley symmetry. (ResNer20 on CIFAR10)
the impact of batch size training ResNet20 on CIFARIO (Fig. 26). From these figures, could again observe that the parameter scale influences the valley width, while the sign consistency ratio matters to the valley symmetry.
Additionally, we could further obtain the following conclusion: a larger learning rate (e.g., 0.1), a smaller batch size (e.g., 32), or larger weight decay (e.g.. 0.001) could lead to better performances, which are shown in these figures with a lower test error when compared the opposite hyperparameter. However, their parameter scales are relatively smaller than opposite ones, making their valley width sharper. And commonly, a larger parameter scale of H,, will let the sign of f 8,, conform to 8 more, which leads to flatter region along the positive direction of f/2-
Figure 24: The interpolation between two models trained with learning rate as 0.1 and 0.003. (VGG16 with BN on CIFAR10)
Figure 25: The interpolation between two models trained with weight decay as 0.001 and 0.00001. (VGG16 with BN on CIFARIO)
Figure 26: The interpolation between two models trained with batch size as 32 and 2048. (ResNet20 on CIFAR10)
19

---

7: end for Algorithm FedSign
ServerProcedure:
1: for global round 20,,2..... do
2: St sample max(Q: K, clients
3: for k S do
4: 05-
ClientProcedure(k; 0,)
5: end for
6: O2+1 - [5:] 0
ClientProcedure(k; 0c):
1: 0 +0
2: for local epoch e 1,2....
3: for each batch with B samples from D do
4: Calculate the loss as Eq. update 0, using, e.g. SGD with momentum
5: end for
6: end for
7: Return: the updated model 0f
C Pseudo Code of FedSign
We provide pseudo code of the application federated learning, i.e., FedSign. Our proposed FedSign takes a novel regularization method to limit the change of local models' signs, whose goal is for better model fusion the server. The pseudo-code is listed in Algo. [ Here, T denotes the number of communication rounds, K is the total number of clients, Q is a client participating ratio in each round, E is the local update epochs of participated clients, B denotes the batch size, and D denotes the private data of the k-th client. In our experimental studies, we set T 200, 100, Q= 10%. E 5, and B 64. After all communication rounds, we obtain the final aggregated model 0T+1. and then we test its accuracy on the global test set to evaluate the aggregation performance of FedSign.
D Detailed Theoretical Analysis and Verification
Our major finding is formulated as: I(G, an) < I[(f--a)) an)), where = -|]] sign(fy) denotes the dign-consistent noise, and
a constant. I.-) the loss function. This finding holds for several settings, including (1) the cases when L(-) is the prediction error or cross-entropy loss, which are verified Fig. 6 and Fig. 20 (2) the cases of applying noise to whole parameters, only to softmax classification layer, or other layers, which are verified in Fig. and Fig. 20
We present theoretical analysis from several possible aspects and finally attribute this finding the properties of ReLU activation and softmax classification.
D.1 Gradient Analysis
First, we guess that the sign(O,) may have correlation to the gradient. Specifically, if ((,)( then given very smaller A. we may have the following relationship:
L(O, + XV L(0,) L(0, XVo,
where the first and third terms denote the gradient ascent step and gradient descent step, respectively. we set e Vo, L()r) and X ε [-,,1]. the plotted curve may be asymmetric. However, we frustratingly find that sign(0,) are almost orthogonal to the Vey On one hand, the converged model oy has very small gradient values, and elements in Vo, C are nearly zero. On the other hand, adding sign(O,) to Oy cannot obtain lower emror than O, itself, which is not the same as the gradient descent direction. This implies that the inherent properties of them are not the same. Hence, we try other possible explanations.
20

---

D.2 ReLU Activation
The asymmetry may be related to the inherent asymmetry of the ReLU activation function. According the analysis in Sect. the gradient of parameters lies in the subspace spanned by the corresponding inputs. Hence, the converged parameters may correlate with their inputs to some extent. Fig. shows The learned classification weight
the pattern of learned classification weight on the "sklear..digi"" w shows pattern of "0" as shown in the figure. However, it is hard for us to provide concrete expression about the correlation of weight with its corresponding inputs, even for the simplest softmax classification layer. Hence, we assume that the learned w equals to a. h 5, where is a constant and a random Gaussian vector. h denotes the hidden representation. And then we simulate the distribution of (w- À sign( (w))² by the following Python code 1 Specifically, we set h Rt as a vector sampled from the distribution of C(O, 1), and set a = 0.1. We take 8 as Gaussian vector sampled from the distribution of G(O, 1). Then we sample N 10000 groups of ē and plot the distribution of (w + À sign(w))² hh under À с { {-1.0, -0.5, 0.0, 0.5. 1.0). The distributions are shown in Fig. 27 Obviously, with an increasing X, the distribution shifted right. In other words, negative À may decrease the value of (w+ À sign( (w) h. In ReLU activation, the negative values are not activated, making the loss error change a lot.
Listing 1: Simulate ReLU
I import numpy as np
2 from matplotlib import pyplot as plt
d 512
0.1
6 mp. rapdomn randn (d)
7 lambs = mp mp. linspaee (-1.0, 1.0, 5)
plt. figure (figsize= 3))
for lamb in lambs:
11
vs []
for in range (10000):
= up random. randn (d) 0. h va. append (np do + lamb (2.0 (w 0.0) 1.0), h))
hist (va, bins=500)
16 plp.p..llee legend (( ["mmmb=aar (aabb((3 (}" format (lamb) for lamb in lambs])
show
Aside from the simulation demo, we also provide the hidden activations for ResNet20 trained SVHN. The results are shown in Fig. 28 We plot the activation confusion matrix of 0 + (elsign(()) with À e [1100, 1.0]. For each then compare the features with the original features extracted by 0, and plot the activation confusion matrix. The sum of diagonal values represents the outputs that the original model and the interpolated model commonly activate or do not activate. The value in "[]" shows the sum of diagonal values. Obviously, the value larger when À than that of X -a, with a E (0.2, 0.4, 0.6, 0.8, 1.0).
obtain the hidden features extracted by 0, + |elsign(0)) (0,) We
To conclude. for the ReLU activation, adding sign-consistent noise to parameters will have a higher probability of activating the neurons. If the neuron outputs only simply scaled by factor, it will not affect the relative scores of the final classification. For example, the inequation of w/ w will change if h is scaled by positive factor, while it does not hold for whose values activated, i.e., h O.
D.3 Softmax Function
Given e Rd, the ground-trun truth label E [C], and the weight matrix W RCxd the softened probability vector is p (o't) The cross-entropy (CE) loss function is L(W) log Py
The gradient of w. is Sw. = -(I(c= y) - hp),, with c [C] and I(-} being the indication function. Specifically, the Hessian L(W)- = -log w.r.t W is H = (diag(y) -pppp"" -pp") hh? where & denotes the Kronecker product. The trace H (r(H) (rdiia(p)) - -pp") * tr trhh"") The first part could be caleulated as P1 pc), where is the class index.
21

---

Before ReLU
80
A=-1.0
0=50.5
60
A-0.0
0-0.5
A-1.0
-50
200
After Re
A--1.0
A--0.5
A-0.0
0.0.5
A=10
20-
Figure 27: The distribution of (w À (w)n(w))> with 0.1 x h 6. J and are sampled from G(O, 1).
1w
10 5963
2.-06.00.53
Am (0040000000 10.007
2--00 10.7653
0. 10.9911
Aun 00.7681
A (9.764)
1.0 1.00 10.0933
Oriy. Nol
0.255
0.219
0.100
0.202
0.158
0.136
0.209
Ong
Nst Aet.
Act.
Not
Aet Art.
Azt.
Figure 28: The activation confusion matrix of 0y + x |e|signn(O)) with A [-1.0,1.00)
We use the "ikgits dataset and train a softmax classification weight via the "LogisticRegres sion" classifier. The demo code is listed in Python code 2 When the training finishes, we obtain the converged weight W. Then, we perturb it via the Gaussian noise c and the sign-consistens Gaussian noise [<| sign W). respectively. Given X [-1, 1], calculate several metrics: (1) the test enror; (2) the cross-entropy loss; (3) the average trace of Px diag(p) -pp" i.e.,
(4) the trace of the Hessian matrix, i.c., tr HHx) (5) the coefficient of the first-order approximation L w.r.p. X, i.e., c" gA. where dA denotes the gradient w.r.p. W + A/c] * sign(W): (6) the coefficient of the second-order approximation of w.r.p. X, i.e., e HA where HA denotes the Hessian matrix w.r.p. W A/el sign(W) The above metrics are shown in Fig. 29 The test error and cross-entroee cross-entropy loss in the second row show asymmetry, because the second row takes the sogn-conisttet noise. More specifically, with positive X the average trace of PA is smaller than under a negative À -a. Therefore, the trace of H smaller than that of H Hx--a- A smaller trace of Hessian means flatter loss region.
The analysis from ReLU and softmax explain the observed phenomenon in this paper, i.e., the sign-consistent noise leads to valley asymmetry and the positive direction is flatter. However, the theoretical insights are formal proofs, which are future works.
Error
CE Loss
E2[tr(P,]]
tr(M))
e M2e
0.075
300
0.050
200
25-
0.025
100
0.0
0-
0.05
1.0
0.10
400
0.2
200
0.5
20
0.05
200
0.0
0-
-1.0
0.0
1.0
-1.5
0.0
1.0
-1.0
0.0
1.0
-1.0
3.
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
A
Figure 29: Several metrics calculated on simple softmax classification demo.
22

---

Listing 2: Softmax Demo
I import numpy as up
2 from
import softmax
from matplotlib import pyplot plt
from sklearn datasets S import load_digits
5 from sklearn linear_moder import LogisticRegression
6 from Aalearn.metrile import 1og_loss
def grad_hessian X, 2, P):
N, X.shape
C len (ap. unique (Y))
OY np dia. (np. ones ones(C)), (N, 1, 1))
Id np tile tile
XX (X[: None, :] x[: :, Nonel) reshape (M, D) PIP (P[: None, :] (Id P(:, ;, Nonel)) reshape (N, C)
15
g -1.0 up dot ((OY
P).T, x) /
H np d XX) /
H Heshapp (C, C, D, D) transpose (0, 2, 1, 3) reshape (C * D,
D)
return 8,
x, Y = load maad_digited (return ( X_y=True)
model LogisticRegression (max_iter=500 =it_intercept=Falss) model.fit(C Y)
25 w model.coet.
24 noise_W mp. randdan (*W. shape)
sign_noise__
up abs (noise_w) (2.0 * (W 0.0) 1)
for noise in [noise_W, sign_noise_w] 28 data = []
for lamb in mp. 1ppsppppccee11.0, 1.0, 21):
Wn - - lamb noise
up. upt (X, Nn.T)
32
softmax (2, axis-1)
8, H
head_hessiandee (W, x, Y, z, P)
trp = np. sum
(1 P), axis=1) mean ()
trh np num (np diag
np sum (g Wn)
37
sd = dot (Wn reshape (-1), np. (H, Wn. reshape (-1)))
loss log loss ty_try y-pred=P)
= np. (up argmax axis=1) != Y)
data appaad
trh, fd, sd, loss, errl)
41
data up aprarayydaat) titles = ["GiniP", "Tr(H),, "n"Tg", "a"THn", "Loss", "Error"] plt Siiuur(ffigizeeee for in rang
42
2))
plt. subplot 6, + 1)
46
pit. plt.plot i))
plt. wwtle p show
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly list the paper's scope and contributions. The contributions are summarized in Sect. [
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations and future works presented Sect.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification: We do not provide striet theoretical proof because it is complex for DNNs. We only provide theoretical insights explain the observations our paper, which does not need too many assumptions and complete proof. The theoretical insights are provided in Sect. [aadd Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: We report our experimental details and the phenomena in our paper are easy to reproduce. The details are provided in Appendix A A demo code to reproduce our finding is provided in Appendix [D3 and Code 2 A pseudo-code for our application to federated learning is also provided in Algo.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
25

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide core codes and demo code to reproduce the observed phenomena in our paper. We do not provide codes with external links. The demo code in Code [ and Code
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips.co/ cwtpublcc/guides..ep/pepcppp httpc//guuudee/oooSuuubbiiiee SrlodeSubmissionoolll.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings and details are provided in the Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Our paper presents interesting phenomena about DNNs, which do not need eπrrr bars. However, provide multiple groups of experimental observations verify that the phenomena are common across various conditions. For example, multiple groups of experimental studies provided to support our finding, e.g., the plots in Appendix
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: The experimental studies in paper do not need too much computation budget, which could reproduced on mainstream devices.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httph::t////// httpar/ppp.......b..//.tccaaiaaa
Answer: [Yes]
Justification:
Our work satisfies the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
27

---

Our work has no societal impact because we focus on studying the basic properties of DNNs.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The dataset and pre-trained used in the work are publicly available, and were unable to find the license for the dataset we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
28

---

The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ comddatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "F8R1qehsb2", "reviewer": "Reviewer_MUA2", "summary": "This paper investigates the characteristics of the asymmetric valley in deep neural networks (DNNs) for classification. The authors perform a perturbation analysis around the local minima, considering the direction of the injected noise. The asymmetric valley demonstrates that DNNs exhibit smaller fluctuations with sign-aligned noise, while opposite noise dramatically increases the test error. Other interesting findings include: (1) Batch normalization initialization and hyperparameter choices affect the occurrence of asymmetry. (2)The ReLU activation and softmax function might explain the phenomenon. The authors also leverage the findings to explain the success of model fusion and propose a sign-regularized method for federated learning.", "questions": "See weaknesses.", "limitations": "The discussion doesn't include transformer architecture.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-structured and clearly presents its interesting findings. The authors start with empirical findings and further apply their discoveries to practical applications, including the model fusion trick and federated learning.\n2.\tThe theoretical insights regarding the presence of ReLU and softmax are convincing.\n3.\tThe proposed direction-aligned noise method is simple and effective, potentially inspiring further research in areas like transfer learning and federated learning.", "weaknesses": "1.\tIn Section 6.1, the authors claim that pre-training is important for the success of model fusion. However, the compared models have different architectures (VGG16BN vs. ResNet18) and are trained on different datasets (CIFAR10 vs. Flowers). This experimental setting weakens the credibility of the conclusion.\n2.\tRegarding the conclusion of Fig. 8, “The interpolation curve shows that the small batch training (i.e., λ = 0.0) lies in a sharper and nearly symmetric valley, while the large batch training (i.e., λ = 1.0) lies in a flatter but asymmetric valley,” I find this problematic. Fig. 9 already shows that the valley is asymmetric for BS=32, while Fig. 8 concludes that BS=32 is symmetric. For the possible explanation for the lower part of Fig. 8, my understanding is that BS=2048 has a more biased value distribution, while BS=32 is more zero-centered. Therefore, the same strength of interpolated noise affects BS=32 more. Besides, the degree of asymmetry is not well-defined in the paper, the conclusion based on this comparison might be problematic.\n3.\tThe conclusion from Fig. 8 also conflicts with the statement in related work that “large batch size training may lead to sharp minima with poor generalization.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and finding our paper's strengths, e.g., \"**the paper is well-structured and clear**\", \"**the theoretical insights are convincing**\", and \"**the proposed method is simple and effective**\".\n\n########\n\nQ1: The different architectures in Section 6.1.\n\nA1: As stated in lines 263-265, the pre-trained model could make the traditionally encountered barrier disappear when linearly interpolating models [55, 64].\nThe previous works **have already shown** that pre-training is important for the success of model fusion [55, 64].\nHence, our focus in Section 6.1 is not to support this conclusion further.\nInstead, we would like to **show the correlation between sign consistency and the model interpolation performance**.\nUsing different DNN architectures could support the correlation convincingly.\n\nAccording to the reviewer's suggestion, we also **replace the pre-trained resnet18 with a random initialized one**, and we find the plots tend to be like the left part of Fig. 11 (i.e., becoming similar to the random initialized VGG16BN).\nWe will **add these plots for better illustration in the later version**.\nThese three groups of plots could better verify the correlation found in Section 6.1.\n\n########\n\nQ2: The valley flatness of BS 32 and BS 2024 in Fig. 8 and Fig. 9.\n\nA2: To answer this question, we must declare two points of view.\n\n(1) **The valley width is closely related to the method of visualization**.\n\nThis point of view has been stated multiple times in our paper, e.g., the lines 60-69, the lines 95-99, and the lines 198-200.\nIndeed, large batch training may lead to poor generalization compared to small batch training.\nHowever, whether the large batch training leads to sharp minima is **still inconclusive**.\n\nFirst, the minima width in the visualization depends on both the parameter scale of $\\theta_f$ and $\\epsilon$ (e.g., previous works [37, 42] found) and the direction of $\\epsilon$ (e.g., our paper found).\nFor example, the plot in Fig. 8 adds $\\epsilon=\\theta_{f_2} - \\theta_{f_1}$ to the BS 32 solution $\\theta_{f_1}$, while Fig. 9 adds the norm-scaled $\\epsilon \\sim G(0, 1)$ to the BS 32 solution $\\theta_{f_1}$.\nDifferent noises have different parameter scales and directions, which makes the visualized shape inconsistent.\n\nSecond, this phenomenon **could also be found** in the previous work [42], i.e., the Fig.2 vs. Fig.3 in [42].\nIn other words, **the inconsistency of valley width in Fig.8 and Fig.9 is rational**.\n\n(2) **Our work does not focus on the valley width but on the valley symmetry**.\n\nWe have also stated this point of view in lines 27-30, 69-70, 94-95, 106-107, 198-200, and 205-207, etc. The **inconsistency of valley width has been explained in [42]**. We find that the **valley symmetry between Fig.8 and Fig. 9 also differs a lot**, which is because of the sign consistency ratio as shown in the upper-right of Fig. 8.\n\nTo conclude, perturbed by norm-scaled noise, BS 32 indeed lies on a flatter minima (Fig.3 in [42] and Fig.9 in our paper).\nHowever, perturbed by the same noise $\\epsilon=\\theta_{f_2} - \\theta_{f_1}$, BS 32 lies on a sharper minima (Fig.2 in [42] and Fig.8 in our paper).\nThis implies that the valley width is closely related to the visualization method, which further **makes the correlation between valley width and generalization performance not so clear**.\nOur work does not focus on this debate and aims to study the reasons behind valley asymmetry.\n\n########\n\nQ3: The conclusion from Fig. 8 also conflicts with the statement in related work that “large batch size training may lead to sharp minima with poor generalization.”\n\nA3: Indeed, large batch training may lead to poor generalization compared to small batch training.\nHowever, whether the large batch training leads to sharp minima is **still inconclusive**, which is stated in lines 26-27.\nAdditionally, the \"conflict\" **has already been explained** in [42] by proposing a proper filter-norm scaled visualization method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BtxJl2bFma", "reviewer": "Reviewer_Jp1U", "summary": "This paper explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. The major contribution is the observation that the degree of sign consistency between the perturbation and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function are provided to explain the asymmetry of DNN valleys. Imposing sign alignment in federated learning is proposed for model parameter alignment.", "questions": "1. Fig.1 is not clear enough.\n2. There are two \"then\"s in line 177.", "limitations": "Lack of theoretical analysis and the discussion on the implications of valley asymmetry to local search based optimization methods.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The observation that the degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry is reasonable, and supported by the theoretical explaination based on ReLU activation and softmax function.", "weaknesses": "1. The asymmetry of DNN valleys is not unexpected, and has been studied in [24]. \n2. The observation that sign consistency between the noise and the convergence point affects the valley asymmetry, taking the ReLU activation and softmax function into account, is intuitive to a large degree and hence deos not emerge as a significant contribution. \n3. Lack of theoretical analysis, such as the bound of degree of asymmetry caused by sign consistency.\n4. Although the application of sign consistency in federated learning is provided, the readers may be more interested in the implications of valley asymmetry to local search based optimization methods, such as SGD, SAM etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and **finding our observation reasonable**.\n\n########\n\nQ1: The asymmetry is not unexpected and has been studied in [24].\n\nA1: Surely, our work is majorly motivated by the proposal of asymmetry valley in [24], and we have declared this relation in lines 28-29 and 67-70.\nWe have also declared that \"**the asymmetric valley is initially proposed by [24], while it does not propose the inherent principles behind the phenomenon**\" in Appendix A.4.\nAppendix A.4 also **in-detail presents our motivation for changing the sign of noise to explain the asymmetry valley**, which has not been studied in previous works.\nCompared with [24], we have the following additional contributions:\n\n(1) we provide **both empirical explanations and theoretical insights for the asymmetry valley**;\n\n(2) we provide **a novel concept named sign consistency ratio**;\n\n(3) we provide **detailed analysis for the DNNs with BN layers and show the initialization matters**;  \n\n(4) we also provide **specific applications to model fusion**, including model soups and federated learning.\n\nAdditionally, the reviewer QEpC has recognized that \"**to my knowledge, the question of minima asymmetry was not systematically investigated after its first appearance in [24], so I consider the contributions of this study novel and significant**\".\n\n########\n\nQ2: The sign consistency is intuitive and does not emerge as a significant contribution.\n\nA2: We don't agree with the reviewer's view.\nWhen reviewers think that this explanation is intuitive and reasonable, they recognize the comprehensibility and acceptability of our work.\n\nHowever, they do not know the process of discovering this explanation required a lot of experimental studies, such as the process in Appendix A.4 and the extensive visualization results in the paper.\nThese experiments include validation results on multiple data sets, various DNNs, different initialization methods, and even different DNN blocks on ImageNet.\n\nThe authors advocate that **summarizing a reasonable explanation that is easy to understand and accepted by the community through abundant experimental results in various scenarios is a significant contribution to the research area**.\n\n########\n\nQ3: Lack of theoretical analysis.\n\nA3: The lack of strict theoretical proof is indeed our weakness, but we provide some theoretical insights to support our findings.\n\nThe reviewer MUA2 thinks that \"**the theoretical insights are convincing**\". \n\nThe reviewer QEpC also declares that \"**although theoretical support is mostly intuitive, with no rigorous proofs (the authors mention that in the Limitations section, which is commendable), it paves the way for future work that will address this limitation**\".\n\nWe will try our best to provide formal theoretical results in future work.\n\n########\n\nQ4: The application to local-search based optimization methods, such as SGD and SAM.\n\nA4: Surely, the traditional works focus on searching for a wider minima that has better generalization performance, which has been stated in lines 76-79.\nHowever, these previous works majorly assume the width of minima is well correlated with the generalization performance, **which is not inclusive until now**.\n\nHence, we provide **a novel application area** (i.e., model fusion) and apply our findings to federated learning.\nAdditionally, our application to federated learning is well-motivated, which **is also recognized by the reviewer MUA2 and the reviewer QEpC**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pXnTDFBz9S", "reviewer": "Reviewer_QEpC", "summary": "In this work, the phenomenon of asymmetric valleys in deep neural networks (DNNs) minima, first observed and described in [1], is systematically examined. By studying different types of (random) directions around a given minimum, the authors discovered that the degree of sign consistency between the chosen direction and the minimum point is a key factor for asymmetry. Besides thorough empirical evaluation, theoretical reasoning is provided to explain this phenomenon. The findings are also practically applied in model fusion.\n\n[1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.", "questions": "1. The Investigation of 6 Special Noise Directions:  \nEmpirically, noise directions that have the same sign with the positive parameters and applies zero to negative parameters ($\\epsilon_5$ and $\\epsilon_6$) do not show asymmetry. I wonder, why so? Do the authors have any intuition on that?  \nTo me, that’s a little counterintuitive, since these directions specifically target the sensitive BN.weight parameters (according to the results of Sec. 4.2.1), don’t they? Also, does the same lack of asymmetry is exhibited by the negative-sign noise directions?\n2. The Finding Holds for ImageNet and Various Parameter Groups:  \n    a) What type of noise is applied here?  \n    b) It is confusing that BN parameters seem like the most “robust to asymmetry” (according to Fig. 6) in light of other experimental results of this paper, underlining the specific role of BN parameters in asymmetry. Even the original work [1] claimed that “directions on BN parameters are more asymmetric”. Could the authors elaborate more on that?\n3. Line 186: centered around 0.5?\n4. Section 4.2.1:  \nIs the noise applied to all parameters or to BN parameters only? If to all, it is indeed remarkable how the initial distribution of BN weights, which take a negligible part in total model parameters, may influence the asymmetry of certain directions (getting back to Question 2b)!\n5. Section 5: Did the authors train a plain linear classifier or a neural network in the sklearn.digits experiment?\n6. Section 6.2:  \nEq. (1): I might be wrong, but it seems that the regularizer is maximized when the signs of $\\theta_t$ and $\\theta_t^k$ are aligned, which is undesired, isn’t it?  \nThe experiments here require stds, as the results presented in Table 1 could lack statistical significance.  \nAlso, what are the hyperparameters of the baselines, how were they chosen? Were they validated just as $\\gamma$ hyperparameter in FedSign? \n7. I suppose that [2], where sign disagreement is also taken into account in model merging, could be a valuable related work.\n\n[1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.\n\n[2] Yadav, Prateek, et al. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems 37 (2023).", "limitations": "The authors discuss the limitations in Section 7.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "All in all, I like this work. \n\nGenerally better loss landscape understanding is important for modern deep learning, and this work takes another valuable step in this direction. To my knowledge, the question of minima asymmetry was not systematically investigated after its first appearance in [1], so I consider the contributions of this study novel and significant.\n\nThe paper is well structured. The narrative is easy to follow, the authors motivate their experiments (“Then, a fantastic idea motivates us to change the sign of the noise” – liked that) and consequently validate their conjectures. \n\nThe claims are clearly formulated and well supported empirically. Empirical evaluation is solid, involving multiple datasets, architectures, and hyperparameter options. The authors tackled many important questions concerning the study of asymmetric values and the sign consistency conjecture. Although theoretical support is mostly intuitive, with no rigorous proofs (the authors mention that in the Limitations section, which is commendable), it paves the way for future work that will address this limitation. Demonstrated practical implications of the insights of this work for model fusion are also important and require further investigation.\n\n[1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.", "weaknesses": "I see no concrete substantial weaknesses in this work. I list some of my questions and suggestions in the following section. \n\nTo me, the main shortcoming is weak theoretical support of the claims made in this work. However, the authors acknowledge this, and I hope that future work will address this disadvantage in some way. \n\nAlso, the experiment in Sec. 6.2 about practical implication in federated learning appears to me a little hastily conducted. Some more details would be appreciated (see specific questions below).\n\nThe text could be slightly polished to eliminate typos and small inaccuracies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and finding our paper \"**well structured and easy to follow**\" and stating \"**the claims are clearly formulated and well supported empirically**\".\nAlso, the authors thank you for **your recognition that the contributions are novel and significant when compared to the previous work [24]**.\n\n########\n\nQ1: The Investigation of 6 Special Noise Directions.\n\nA1: A great question!\nOur paper's major observation is that **the sign consistency between DNN parameters and noise determines the symmetry**.\nThrough your question, the authors guess that the level of asymmetry may **be also related to the property of perturbed parameters, e.g., the total number and distribution of the parameters**.\n\nTo answer Q1, the special noise $\\epsilon_5$ and $\\epsilon_6$ does not show asymmetry because it only affects the positive parameters.\n\nIf the parameters are all positive, they may be relatively stable when adding or subtracting some specific values.\nOtherwise, if the distribution of parameters is more disordered (e.g., containing various scopes of positive and negative values), the parameters may be more sensitive to sign-consistent perturbation.\nWe investigate the negative-sign noise directions and find similar results.\n\nThis guess may also provide explanations for BN layers, i.e., Q2.\n\n########\n\nQ2: The Finding Holds for ImageNet and Various Parameter Groups.\n\nA2: First, we use Gaussian noise.\nSecond, compared with other groups of parameters, **BN parameters (especially the BN.weight) have a simpler parameter distribution**.\nAccording to the first question's guess (e.g., A1), BN parameters tend to be less sensitive to sign-consistent noise.\n\nThis observation does not contradict the findings in [24].\n\nFirst, they find that adding random noise directions on BN parameters is more asymmetric, which is due to the possible larger sign consistency.\n\nSecond, adding totally sign-consistent noise to BN parameters indeed leads to asymmetry valleys (e.g., Fig.6 and Fig.7), but the asymmetry tendency is less obvious.\n\nThird, applying totally sign-consistent noise to other parameters (e.g., a complex parameter distribution) leads to obvious asymmetry valleys.\n\nAs an initial guess, **the significance level of asymmetry has the following rank: A > B > C > D**. \n\nA denotes a complex parameter distribution perturbed by sign-consistent noise. (100% sign consistency)\n\nB denotes a simpler parameter distribution perturbed by sign-consistent noise. (100% sign consistency, but the parameter distribution is simpler)\n\nC denotes a simpler parameter distribution perturbed by random noise. (a larger probability of > 50% sign consistency)\n\nD denotes a complex parameter distribution perturbed by random noise. (about 50% sign consistency)\n\nThe previous work [24] found **C > D** (i.e., directions on BN parameters are more asymmetric), while our paper found **A > D** and **B > C** (i.e., sign-consistent noise leads to asymmetry).\n\nYour question helps us guess that **A > B**, which derives the entire rank.\n\nIn summary, your great question brings us a novel guess about the influence of parameter distribution on valley symmetry. This guess should be checked further. Thanks for your insights.\n\n########\n\n\nQ3: Line 186.\n\nA3: We are sorry that this is a mistake. The distribution is centered around 0.5, while the ones under the initialization of U(0, 1) are centered around 0.2.\n\n########\n\nQ4: Section 4.2.1.\n\nA4: We only apply noise sampled from \\{0, 1\\} to BN parameters, which conforms to the experimental setting in [24].\n\nWe also have some experimental studies that apply noise to the whole parameters when considering the influence of BN, and please see the details in Appendix A.4.\nSpecifically, we compare the valley symmetry of DNNs with or without BN layers.\n\n########\n\nQ5: Section 5: Did the authors train a plain linear classifier or a neural network in the sklearn.digits experiment?\n\nA5: As we stated in Section 5 (line 260), the demo code is provided in Appendix D.3. The demo code is listed in Python code 2 (Page 23).\n\nThe code trains a plain linear classifier as a simple illustration.\n\n########\n\nQ6: Section 6.2\n\nA6: Thanks for your correction! The design of this loss follows the idea of Negative Log Likelihood, where we miss a minus. We will correct this formula in the future version.\n\nWe rerun the experimental studies five times and list the std. of accuracies for a portion part in Table. 1. The experimental results are as follows. The accuracy doesn't fluctuate very much.\n\n|Dir.$\\alpha$|FedAvg|FedSign|\n|-|-|-|\n|CIFAR-10 10.0|81.53 (0.17)|82.59 (0.09)|\n|CIFAR-10 1.0|80.54 (0.11)|80.76 (0.14)|\n|CIFAR-10 0.5|77.69 (0.21)|78.41 (0.35)|\n\nAll results will be added in the future version.\n\nFedAvg does not have additional hyper-parameters. FedProx, MOON, and FedDyn have a regularization coefficient like $\\gamma$, and we search for them in the scope of \\{0.1, 0.01, 0.001\\}. The best results are reported. The hyper-parameters of FedPAN are set according to the original paper.\n\n########\n\nQ7: A valuable related work.\n\nA7: Thanks for your recommendation! This is indeed related to our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks for all the reviewers' efforts in reviewing our paper!

We are delighted that the three reviewers found our strengths.

The reviewer MUA2 advocates "**the paper is well-structured and clear**", "**the theoretical insights are convincing**", and "**the proposed method is simple and effective**".

The reviewer Jp1U **finds our observation reasonable**.

The reviewer QEpC thinks our paper is "**well structured and easy to follow**" and states "**the claims are clearly formulated and well supported empirically**". Additionally, the reviewer QEpC recognizes that **the contributions are novel and significant when compared to the previous work [24]**.

However, the reviewer Jp1U has some concerns about our paper, and we list the brief responses correspondingly.

########

1. The asymmetry of DNN valleys has been studied in [24].

We have listed our novel contributions in the paper and also provide them in the detailed responses to reviewer Jp1U.

The reviewer QEpC recognizes that **the contributions are novel and significant when compared to the previous work [24]**.

########

2. The observation of sign consistency seems intuitive.

We don't agree with the reviewer's view. The authors advocate that **summarizing a reasonable explanation that is easy to understand and accepted by the community through abundant experimental results in various scenarios is a significant contribution to the research area**.

########

3. Lack of theoretical analysis.

We have provided some theoretical insights to support our findings.

We have pointed out this weakness in the Limitations section.

The reviewer MUA2 thinks that "**the theoretical insights are convincing**". 

The reviewer QEpC also declares that "**although theoretical support is mostly intuitive, with no rigorous proofs (the authors mention that in the Limitations section, which is commendable), it paves the way for future work that will address this limitation**".

########

4. The implications of valley asymmetry to local search based optimization methods.

**A novel application area** (i.e., model fusion) is a contribution of our paper **instead of a weakness**.

Meta-review: The paper shows under what conditions a minima of the loss surface is asymmetric along a given direction, i.e. when it significantly deviates from a locally symmetric (usually parabolic) shape.

Two reviewers voted for recommendation (7 and 5 scores), while one voted for rejection (score of 4). The discussion phase has not led to reviewers reconciling their opinions.

The discover condition under which a minima is asymmetric seems rather natural and simple, which I see as a strength of the paper.

The applications to model fusion and federated learning are intriguing, as also highlighted by QEpC and underscore the relevance of the studied concepts. For example, whether or not the minima is asymmetric along the interpolation line has a meaningful impact on how easy or hard it is to fuse models (”fine-tuning the pre-trained model does not change the parameter signs a lot, which facilitates the following parameter interpolation”)

Jp1U who voted for rejection remarked that the paper lacks theoretical results (e.g. lack of bounds of the degree of asymmetry). I think that while the paper does emphasizes empirical results and intuitions, it is not in itself a reason to reject the paper.

All in all, I think the paper, while borderline, does clear the bar for acceptance and it is my pleasure to recommend it for this year's program.
Decision: Accept (poster)