Title: Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Loss Landscape

Asymmetric Valley

Model Fusion

Perturbation Analysis
Abstract: Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

Abstract

Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.

1 Introduction

The massive number of parameters and complex structure of deep neural networks (DNNs) have catalyzed extensive research to mine their underlying mechanics [55, 34, 42, 62, 71]. Visualizing and exploring the loss surfaces of DNNs is the most intuitive way [43, 18], which has ignited many interesting findings, such as the monotonic linear interpolation [22, 19, 66] and linear mode connectivity [22, 14, 21, 15, 65, 2]. Loss landscape visualization has also been applied to show the optimization trajectory [53, 33, 35], understand the effectiveness of Batch Normalization [32, 61], BERT [11, 23], deep ensemble [17, 30, 21], and so on.

Perturbation analysis around the local minima of DNNs [7, 63], i.e., the shape of the valleys they reside in, is a very popular research topic. The concept of flat minima was originally proposed by [28], who defines the size of the connected region around the minima where the loss remains relatively unchanged as flatness. Subsequent studies debate whether the flat or sharp minima could reflect the generalization ability [38, 43, 36, 57, 16, 12, 41, 3]. The previous works constrain the valley shape to be symmetric, while recent work points out that not all DNN valleys are flat or sharp, and there also exist asymmetric valleys [25], which has not been systematically studied as far as we know .

This paper in-depth analyzes the factors that may affect the valley symmetry of DNNs. Previous work's analysis of valley shape primarily utilizes the 1D interpolation of θ f + λglyph[epsilon1] , where θ f represents the minima solution and glyph[epsilon1] denotes a random noise. As shown in Fig. 1, we believe that the valley symmetry depends both on the convergence solution and noise, with each of them being influenced by some factors. The most significant innovation in our research is considering the effect of noise direction on valley visualization, as previous work has simply taken the Gaussian noise .

Noise:

𝜖𝜖

𝜽𝜽

𝒇𝒇

+

𝝀𝝀𝝀𝝀

,

𝝀𝝀 ∈

[

-𝟏𝟏

,

𝟏𝟏

]

Dataset

Hyper-parameters

Architecture

Initialization

Magnitude

Direction

Converged Model:

𝜃𝜃

𝑓𝑓

𝜃𝜃

𝑓𝑓 -𝜖𝜖

𝜃𝜃

𝑓𝑓

+

𝜖𝜖

𝜃𝜃

𝑓𝑓

𝜃𝜃

𝑓𝑓 -𝜖𝜖

𝜃𝜃

𝑓𝑓

+

𝜖𝜖

𝜃𝜃

𝑓𝑓

Valley Symmetry

Error

Error

𝜃𝜃

𝑓𝑓

s = 0.5

s = 1.0

s = 2.0

𝜃𝜃

𝑓𝑓

+

𝜆𝜆 ∗

s

∗

𝜖𝜖1

𝜖𝜖1

𝜃𝜃

𝑓𝑓

𝜃𝜃

𝑓𝑓

+

𝜆𝜆 ∗

s

∗

𝜖𝜖2

𝜖𝜖2

𝜃𝜃

𝑓𝑓

𝜖𝜖1

𝜖𝜖2

𝜖𝜖2

𝜖𝜖1

𝜃𝜃

𝑓𝑓

+

𝜆𝜆𝜖𝜖2

𝜃𝜃

𝑓𝑓

+

𝜆𝜆𝜖𝜖1

𝜃𝜃

𝑓𝑓

𝜖𝜖2

𝜖𝜖1

𝜃𝜃

𝑓𝑓

+

𝜆𝜆𝑇𝑇

𝜃𝜃

𝑓𝑓

𝜖𝜖2

𝜃𝜃

𝑓𝑓

+

𝜆𝜆𝑇𝑇

𝜃𝜃

𝑓𝑓

𝜖𝜖1

𝜃𝜃

𝑓𝑓

(A) Raw Noise

(B) Filter Norm-scaled Noise

(C) Norm-scaled Noise

Specifically, we start by comprehensively and carefully determining the visualization method to plot the valley shape, which could influence the conclusion significantly [31, 43, 41]. We finally take the Norm-Scaled (NS) visualization method [31] that normalizes the noise glyph[epsilon1] and further scales it to ∣∣ θ f ∣∣ for better determining the plot range of λ , where the direction of raw noise is not changed. Then, after investigating 7 common noise directions and 6 special ones, we conclude: the degree of sign consistency between the noise and the convergence solution should be a determining factor for asymmetry . This phenomenon is basically insensitive to the utilized datasets. Next, we focus on the impact of the network architecture with or without Batch Normalization (BN) [32], indicating that the BN initialization also impacts the valley symmetry. Finally, different hyperparameters lead to solutions with various valley widths but show no asymmetry consistently.

Aside from empirical observations, theoretical insights for our interesting findings are provided. We first declare that adding sign-consistent noise to parameters may have a larger probability of keeping activating the neurons or keeping the overwhelming score in classification tasks. Then, we show that the trace of the Hessian matrix along the sign-consistent direction is smaller, implying a flatter region. The above findings inspire applications in the fields of model fusion [47, 68, 33, 2]. This paper first explains why model aggregation based on pre-trained models often leads to performance improvements, i.e., the success of model soups [68], and then proposes constraining the sign of DNN parameters in federated learning [54, 73, 50] to facilitate aggregation.

Our novel contributions can be summarized as (1) exploring the valley shape under different noise directions that have not been studied yet ; (2) proposing that the flat region could be expanded along the direction that has a higher sign consistency with the convergence solution ; (3) pointing out the influence of BN and its initialization on valley symmetry ; (4) presenting theoretical insights to explain our interesting finding ; (5) explaining and inspiring effective algorithms in model fusion .

2 Related Works

Exploring the Valley Shape of DNNs . The valley around a minima solution has originally been viewed as flat or sharp [28], and the large batch size training may lead to sharp minima with poor generalization [38]. However, [12] declares that two solutions that are scale-invariant in performance may lie in regions with significantly different flatness. [43] verifies the findings of [12] by applying varying weight decay to small and large batch training, leading to results contrary to [38]. With the filter-normalized plots, [43] again observes that sharpness correlates well with generalization error. Later work also shows that the sharpness calculation should be dependent on the parameter scale [41] or the dataset [3]. Whether valley width reflects the generalization is still inconclusive. The proposal of asymmetric valley [25] further throws this debate into limbo, as valleys around minima could be flat on one side but sharp on the other side, which makes it more difficult to define flatness. This paper thoroughly studies the causes and implications of the unexplained asymmetry phenomenon .

Exploiting the Valley Shape of DNNs . Exploring the valley shape of DNNs could help us better understand the inherent principles of DNNs. The work [43] utilizes 2D surface plots to show that the residual connections in ResNet [27] could prevent the explosion of non-convexity when networks get deep, and [61] attributes the success of BN to its effects of making landscape significantly more smooth. The asymmetric valley [25] provides a sounding explanation for the intriguing phenomenon

in stochastic weight averaging [33]. Additionally, studying the valley shape could also benefit the proposal of effective optimization algorithms, e.g., the entropy-based SGD [6], and the (adaptive) sharpness-aware minimization [16, 41]. Penalizing the gradient could also lead to solutions around flat regions [74]. We also apply the findings in this paper to the area of model fusion .

Model Fusion . Directly averaging two independent models may encounter a barrier due to training randomness or permutation invariance [4, 22, 2, 51, 24]. However, if the two models are generated from separate training of a common pre-trained model, the model fusion may perform better than individual models, i.e., the model soups [58, 68]. A recent work [70] finds that resolving sign conflicts when merging multiple task-specific models is neccessary, which is most related to our current work. We will explain the success of model soups based on the relation between the asymmetric valley and the sign consistency of model parameters . Popular federated learning algorithms also take the parameter averaging process to fuse the individual models updated on isolated clients [54, 52]. A huge challenge is the Non-Independent and Identical Data distributions of data islands (NonI.I.D. data) [73, 29], which could make local models too diverged to merge. Multiple regularization methods are proposed to align parameters before model fusion [46, 1, 45, 37]. We propose an effective regularization method that focuses on the sign of parameters, which is inspired by our interesting findings .

3 Basic Notations and Preliminaries

Our major tool is plotting the 1D error curve of DNNs following the formula θ f + λglyph[epsilon1] (Fig. 2 (A)). θ f denotes the converged model, and glyph[epsilon1] denotes a noise vector sampled from a specific distribution. More about the visualization of DNN loss landscape could be found in [43, 49].

3.1 Previous Studies: Exploring θ f with Fixed glyph[epsilon1]

The previous studies focus on studying the valley shape under different θ f , and aim to mine the shape's relation to generalization . To mitigate the influence of parameter scales and make the visualization fairly comparable between different θ f , [43] proposes the filter normalization method to properly visualize the loss landscape. The processing of the noise is glyph[epsilon1] i,j ← glyph[epsilon1] i,j ∣∣ glyph[epsilon1] i,j ∣∣ ∣∣ θ i,j f ∣∣ , where i is the index of layer and j is the index of filter. This way normalizes each filter in the noise glyph[epsilon1] to have the same norm of the corresponding filter in the converged point θ f . Further, [41] proposes a proper definition of sharpness (i.e., adaptive sharpness) based on the filter normalization, extending it to all parameters and formally defining: T θ f = diag ( concat (∣∣ f 1 ∣∣ 2 I n 1 , . . . , ∣∣ f m ∣∣ 2 I n m , ∣ w 1 ∣ , . . . , ∣ w q ∣)) , where f j with 1 ≤ j ≤ m denotes the j -th convolution filter in θ f and n j is the number of parameters it owns. w j with 1 ≤ j ≤ q denotes the j -th parameter that is not included in any filters. I is a vector with all values as one. Then, the adaptive noise T θ f glyph[epsilon1] is utilized to study the sharpness of θ f .

3.2 Our Study: Exploring glyph[epsilon1] and λ with Fixed θ f

Different from the previous studies, we aim to explore the valley shape of a fixed θ f under different glyph[epsilon1] and λ . First, the direction of 1D interpolation in previous works is limited to the Gaussian noise, while we study the impact of different noise types. Second, setting λ positive or negative could obtain a valley with different flatness, i.e., the asymmetric valley [25]. Hence, under the fixed θ f , we do not need to rectify the noise direction filter-wisely. We take the visualization way used in [31], which only normalizes the noise and then re-scales it to the norm of θ f , i.e., glyph[epsilon1] ← glyph[epsilon1] ∣∣ glyph[epsilon1] ∣∣ ∣∣ θ f ∣∣ . The utilized Norm-Scaled (NS) noise is shown in Fig. 2 (C). Compared with Filter NS noise (Fig. 2 (B)), this way does not change the direction of the noise and shows the original valley shape along the direction glyph[epsilon1] . Additionally, to plot 1D error curves in the same figure, we fix λ in the range of [-1 , 1 ] . Another scale factor s is added to control the visualized width. Overall, we use the following way to plot 1D error curves: θ f + λ ∗ s ∗ glyph[epsilon1] ∣∣ glyph[epsilon1] ∣∣ ∣∣ θ f ∣∣ , where λ ∈ [-1 , 1 ] , and we set s = 1 . 0 by default. The Frobenius norm is used. Notably, we utilize the NS noise by default and use the Filter NS noise when comparing the valley shape under different converged points, e.g., the studying of BN initialization in Sect. 4.2.1.

0.0

0.5

1.0

Test Error

NS

ε

s=0.2

s=1.0

s=2.0

NS

ε

∼U(-1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{-1, 0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼G(1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼U(0, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼{1}

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

4 Experimental Findings

This section presents the major findings about factors that respectively impact the noise and converged points. Experimental details and more verification results are in Appendix A and B.

4.1 Factors that Affect glyph[epsilon1]

We train VGG16 [64] with BN [32] on CIFAR10 [39] for 200 epochs and obtain the converged model. For the given θ f , we plot the 1D error curves under the following 7 common noise types : (1) G ( 0 , 1 ) : Gaussian noise with mean as 0 and std as 1 ; (2) U (-1 , 1 ) : uniform noise in the range [-1 , 1 ] ; (3) {-1 , 0 , 1 } : uniform noise with values only in -1 , 0 , and 1 ; (4) G ( 1 , 1 ) : Gaussian noise with mean as 1 and std as 1 ; (5) U ( 0 , 1 ) : uniform noise in the range [ 0 , 1 ] ; (6) { 0 , 1 } : uniform noise with values only in 0 and 1 ; (7) { 1 } : constant noise with all values as 1 . For each plot, we set s ∈ { 0 . 2 , 1 . 0 , 2 . 0 } and λ ∈ [-1 , 1 ] to show curves under various levels of width. The results are shown in the first row of Fig. 3. We observe that the valley shapes along these 7 noise directions are almost symmetric, except that the last four noise directions show slight asymmetry when s = 0 . 2 .

Then, a fantastic idea motivates us to change the sign of the noise. The detail of this motivation is provided in Appendix A.4. Specifically, we use the following method to replace the sign of glyph[epsilon1] with that of θ f : glyph[epsilon1] ←∣ glyph[epsilon1] ∣ ∗ sign ( θ f ) , where ∣ ⋅ ∣ returns the absolute value element-wisely and sign (⋅) returns 1 or -1 based on whether the element is positive or negative. The corresponding results of the 7 common noises become completely asymmetric, which are plotted in the second row of Fig. 3. Furthermore, the valleys all follow the tendency that the positive direction is flat while the negative direction is sharp. Hence, we propose our major finding: the sign consistency between noise and converged model determines the asymmetry and the valley is flatter along the noise direction with a larger sign consistency . The finding is formulated as: L ( θ f + aη ) < L ( θ f -aη ) , where η = ∣ glyph[epsilon1] ∣ ∗ sign ( θ f ) denotes the sign-consistent noise, and a > 0 is a constant. L (⋅) is the loss function, which could be the prediction error or cross-entropy loss. The following three experimental studies could further verify this interesting finding.

The Manual Construction of Noise Direction . We element-wise sample the noise glyph[epsilon1] from G ( 0 , 1 ) , and then manually change its elements' sign with a given ratio r ∈ { 0 . 0 , 0 . 1 , . . . , 1 . 0 } . For example, r = 0 . 5 means that we sample 50% elements in the noise and change their sign to the same as θ f . Then, we plot the average test error of the positive interpolations and negative interpolations, i.e., E λ [ Error ( θ f + λ NS ( glyph[epsilon1] ))] with λ ∈ [ 0 , 1 ] and λ ∈ [-1 , 0 ] , respectively. Fig. 4 plots the average test errors on two groups of networks and datasets. The test errors of positive and negative directions are nearly equal when r = 0% . As r becomes larger, the average test error of the positive direction monotonically decreases while the negative one increases, implying that the valley shape becomes more and more asymmetric.

The Investigation of 6 Special Noise Directions . Wethen investigate several special noise directions including (1) the initialization before training, i.e., glyph[epsilon1] 1 = θ 0 ; (2) the converged model itself, i.e., glyph[epsilon1] 2 = θ f ; (3) glyph[epsilon1] 3 = sign ( θ f ) ; (4) glyph[epsilon1] 4 = sign ( θ f -µ ) ; (5) glyph[epsilon1] 5 = sgp ( θ f ) ; (6) glyph[epsilon1] 6 = sgp ( θ f -µ ) . Here, µ denotes the mean value for each parameter group, e.g., the mean value of 'conv1.weight'. sgp (⋅) returns 1 or 0 based on whether the element is positive or not. The visualization results are provided in Fig. 5. The first four directions lead to asymmetry, while the last two do not. First, the elements in glyph[epsilon1] 2

0.0

0.2

0.4

0.6

0.8

Average Error

CIFAR10 VGG16BN

Negative:

λ

∼ [-1, 0]

Positive:

λ

∼ [0, 1]

0%

20%

40%

60%

80%

100%

Sign Change Ratio

0.0

0.2

0.4

0.6

0.8

Average Error

SVHN ResNet20

Negative:

λ

∼ [-1, 0]

Positive:

λ

∼ [0, 1]

0.0

0.5

1.0

Test Error

ε

=

θ

0

s=0.2

s=1.0

s=2.0

ε

=

θf

s=0.2

s=1.0

s=2.0

ε

=sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

=sign(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

0.0

0.5

1.0

ALL

Error

CE Loss

CLF

Error

CE Loss

FEAT

Error

CE Loss

LAYER1

Error

CE Loss

CONV

Error

CE Loss

BN

Error

0

4

8

CE Loss

-1.0

0.0

1.0

λ

0.0

0.5

1.0

ε

←|

ε

|*sign(

θf

)

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

0

4

8

CE Loss

and glyph[epsilon1] 3 surely have the same sign with θ f , which leads to an asymmetric valley. Because the mean values of most parameters are near zero, glyph[epsilon1] 4 performs likely as glyph[epsilon1] 3 . glyph[epsilon1] 5 and glyph[epsilon1] 6 only have the same sign with the positive parameters in θ f , and applies zero to negative parameters in θ f , which shows no asymmetry. The most interesting result is the glyph[epsilon1] 1 = θ 0 , whose elements may be centered around zero according to the Kaiming initialization [26]. However, the BN initialization is asymmetric, which leads to asymmetric curves (Sect. 4.2.1). The results of VGG11 without BN on CIFAR100 [39] show no asymmetry when glyph[epsilon1] = θ 0 (Appendix B, Fig. 17).

The Finding Holds for ImageNet and Various Parameter Groups . We extend the findings to large-scale datasets and apply noise only to specific parameter groups. Specifically, we use the pre-trained models (e.g., ResNeXt101 [69]) downloaded from 'torchvision' 1 . Because these models are pre-trained on ImageNet [10], we could directly use them to verify our findings without additional training. Multiple parameter groups are considered as follows: (1) 'ALL' denotes the whole parameters; (2) 'CLF' denotes the weights in the final classifier layer; (3) 'FEAT' denotes the weights in the layers aside from the final classifier; (4) 'LAYER1' denotes parameters in the first several blocks; (5) 'CONV' denotes all convolution parameters; (6) 'BN' denotes all of the BN parameters. As shown in Fig. 6, applying sign-consistent noise could lead to asymmetric valleys. Notably, this holds for both the metrics of CE loss and prediction error.

4.2 Factors that Affect θ f

Then we focus on studying the effects of BN and its initialization, then present the results under various hyperparameters.

0.0

0.5

1.0

0

100

200

Histogram

BN.weight=1.0

Initial

Converged

0.0

0.5

1.0

0

20

40

BN.weight ∼ U(0, 1)

Initial

Converged

0.0

0.5

1.0

0

20

40

BN.weight ∼ G(0, 0.1)

Initial

Converged

0.0

0.5

1.0

Test Error

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

NS Noise

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

-1.0

0.0

1.0

λ

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

-1.0

0.0

1.0

λ

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

Filter-NS Noise

-0.25

0.00

0.25

0.50

Parameter mean:

μp

0.00

0.25

0.50

0.75

1.00

Parameter std.:

σp

Parameter Distribution of

θf

1

BN Weight

CLF Weight

Other Weight

-0.25

0.00

0.25

0.50

Parameter mean:

μp

Parameter Distribution of

θf

2

BN Weight

CLF Weight

Other Weight

BN Weight

CLF Weight

Other Weight

Sign Consistency Ratio

θf

1

vs

.

θf

2 -

θf

1

θf

2

vs

.

θf

2 -

θf

1

0.0

0.2

0.4

0.6

0.8

1.0

-1.0

0.0

0.5

1.0

2.0

λ

0.0

0.5

1.0

Test Error

Interpolation of BS 32 and BS 2048

θf

1

,

θf

1 +

λ

(

θf

2 -

θf

1

)

θf

2

,

θf

2

+(

λ

-1)(

θf

2 -

θf

1

)

0.0

0.5

1.0

Test Error

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

LR=0.003

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

BS=32

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

BS=2048

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

WD=1e-3

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

WD=1e-5

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

4.2.1 BN and Initialization

The default initialization method of BN is setting w as ones and b as zeros [32, 9]. Hence, the w , i.e., the 'BN.weight', may be asymmetric after convergence. We take three ways to initialize the BN weights, including (1) the elements are all ones; (2) the elements are sampled from U ( 0 , 1 ) ; (3) the values sampled from G ( 0 , 0 . 1 ) . We train three models based on these three types of BN initialization. In Fig. 7, the first row shows the initial and converged parameter distribution of a specific BN weight. The traditional initialization leads to converged BN weights with all positive values, which are nearly centered around 0 . 2 . The uniform initialization between 0 and 1 also leads to positive converged weights. The symmetric Gaussian initialization leads to converged values symmetric around 0 . Then, we plot the valley shapes under the noise direction glyph[epsilon1] ∈ { 0 , 1 } . Because this part involves a comparison among different convergence points, we plot the results by both the NS noise and Filter NS noise. As vividly displayed in Fig. 7, the first two initialization ways encounter obvious asymmetry while the Gaussian initialization shows nearly perfect symmetry. If we carefully analyze the sign consistency ratio of them, we could easily explain this phenomenon. The noise direction glyph[epsilon1] ∈ { 0 , 1 } has a larger overlap with the first two initialization methods because the converged BN weights are all positive, while it has a lower overlap with the initialization from G ( 0 , 0 . 1 ) . This implies that the traditional BN initialization will lead to nearly all positive converged BN weights, which may influence the valley symmetry .

4.2.2 Hyperparameters

Previously, [38] shows that the batch size could influence the valley width, and further [43] advocates that weight decay could also play a role in the valley width. Different from them, we aim to study whether these hyperparameters influence the valley symmetry. We train VGG16 networks on CIFAR10 with various hyperparameters. We use the SGD optimizer with a momentum value

Digit 0

λ

=-1.0

λ

=-0.8

λ

=-0.6

λ

=-0.4

λ

=-0.2

λ

=0.0

λ

=0.2

λ

=0.4

λ

=0.6

λ

=0.8

λ

=1.0

of 0.9. The default learning rate (LR) is 0.03, batch size (BS) is 256, and weight decay (WD) is 0 . 0005 . Then, we will correspondingly set the LR in { 0 . 1 , 0 . 003 } , BS in { 32 , 2048 } , and WD in { 0 . 001 , 0 . 00001 } . The curves are in Fig. 9. The first row applies the G ( 0 , 1 ) noise, while the second row changes its sign to the converged models' sign. Obviously, different hyperparameters may lead to valleys with various widths, while the valleys are all symmetric. The asymmetry valleys in the second row again verify the previous findings in Sect. 4.1.

Then, we explore the interpolation between two solutions under different hyperparameters, which are studied in [38, 43, 22]. We take the batch size of 32 and 2048 as an example and denote the converged solution as θ f 1 and θ f 2 . The test error curve of ( 1 -λ ) θ f 1 + λθ f 2 is plotted, with λ ∈ [-1 , 2 ] . Aside from the interpolation curve, we also plot the parameter distributions of θ f 1 and θ f 2 . The parameters are divided into five groups, including 'BN Weight', 'CLF Weight', 'CLF Bias', 'Other Weight', and 'Other Bias'. 'CLF' denotes the last classification layer, and 'Other' denotes other layers aside from the BN layers and the last classification layer. To simplify the figures, we only plot the mean and standard deviation of the parameters, denoted as µ p and σ p . Fig. 8 shows the parameter distributions and the interpolation curve. The interpolation curve shows that the small batch training (i.e., λ = 0 . 0 ) lies in a sharper and nearly symmetric valley, while the large batch training (i.e., λ = 1 . 0 ) lies in a flatter but asymmetric valley. Small batch training (i.e., θ f 1 ) leads to parameters with smaller mean and std values. This is because the utilized weight decay is 0 . 0005 , which makes the parameter scale smaller due to longer training [43]. Then, we explain the different results of valley symmetry. The interpolation formula could be re-written as θ f 1 + λ ( θ f 2 -θ f 1 ) and θ f 2 +( λ -1 )( θ f 2 -θ f 1 ) , which respectively shows the 1D interpolation centered around θ f 1 and θ f 2 . If we let glyph[epsilon1] = θ f 2 -θ f 1 , then we could plot the sign consistency ratio (i.e., how many parameters have the same sign) of θ f 1 and glyph[epsilon1] , and θ f 2 and glyph[epsilon1] . The results are in Fig. 8, where we provide the values of the five parameter groups. Obviously, θ f 2 is more consistent in the sign values, which shows a flatter region towards the positive direction. In contrast, the sign consistency ratio of θ f 1 is smaller, which only shows slight asymmetry.

5 Theoretical Insights to Explain the Finding

This section provides theoretical insights from the aspects of ReLU activation and softmax function to explain the interesting phenomenon. The forward process of DNNs contains amounts of calculation of W T h , e.g., the fully connected and convolution layer. W denotes the weight matrix and h is a hidden representation. A special case is the final classification layer with the softmax function. Given h ∈ R d , the ground-truth label y ∈ [ C ] , and the weight matrix W ∈ R C × d , the softened probability vector is p = softmax ( Wh ) . The cross-entropy (CE) loss function is L ( W ) = -log p y . The gradient of w c is g w c = -( I { c = y } -p c ) h , with c ∈ [ C ] and I {⋅} being the indication function. This implies that the update direction of W lies in the subspace spanned by hidden representations, which also holds for intermediate layers and convolution layers [60]. After adequate training steps, the parameters that activate the ReLU function or correspond to the ground-truth label should correlate well with their inputs. We show a demo classification weight learned on the 'sklearn.digits' 2 . Fig. 10 shows the pattern change of w + λ ∗ sign ( w ) with λ ∈ [-1 . 0 , 1 . 0 ] . Clearly, the weight under λ = 0 . 0 correlates well with the input sample (i.e., Digit 0 in Fig. 10). Setting λ > 0 . 0 will almost keep the pattern, while λ < 0 . 0 destroys it significantly. That is, w + λ ∗ sign ( w ) with λ > 0 . 0 may keep providing a high score for the target class, while setting λ < 0 . 0 may decrease the score.

For the ReLU activation, it also holds that w + λ ∗ sign ( w ) will have a higher probability of keeping activating the neurons when λ > 0 . 0 . To simplify the analysis, we assume the learned w equals a ∗ h + δ , where a is a constant and δ is a random Gaussian vector. Then we could easily verify that

( w + λ ∗ sign ( w )) T h will have a higher probability of keeping activating neurons under a positive λ than the negative one. The details and simulation results are in Appendix D.2. If the neuron outputs are only simply scaled by a factor, it will not affect the relative scores of the final classification. For example, the inequation of w T 1 h > w T 2 h will not change if h is scaled by a positive factor, while it does not hold for h whose values are not activated, i.e., h = 0 .

Then we provide a further analysis via analyzing the Hessian matrix of the softmax weights. Specifically, the Hessian of L ( W ) = -log p y w.r.t. W is H = ( diag ( p ) -pp T ) ⊗ hh T , where ⊗ denotes the Kronecker product. The trace of H is tr ( H ) = tr ( diag ( p ) -pp T ) ∗ tr ( hh T ) . The first part could be calculated as ∑ c p c ( 1 -p c ) , where c is the class index. According to the above analysis, adding sign-consistent noise to w y could enlarge the score of w T y h , which may make the p y larger and p c ≠ y smaller [48]. That is, the predicted probability vector tends to be a one-hot vector when adding sign-consistent noise, and ∑ c p c ( 1 -p c ) will be near zero. Hence, the trace of the Hessian matrix is smaller along the sign-consistent direction. Since softmax is convex, and the eigenvalues of the Hessian are all positive, a smaller trace means smaller eigenvalues, which makes the loss curve flatter. The empirical observation can be found in Appendix D.3.

6 Applications to Model Fusion

6.1 Explaining the Success of Model Soups

Commonly, the interpolation of two independently found DNN solutions may encounter a barrier [22, 65]. Surprisingly, if these two models are updated from the same pre-trained model, then the barrier will disappear, and the linear interpolation brings a positive improvement [58, 68]. The common explanation follows that the pre-trained model may possess less instability when compared to the random initialization models [20]. We guess that the sign consistency ratio may influence the parameter fusion performance of the two models. Perhaps, the sign of model parameters updated based on the pre-trained model remain nearly unchanged during the process of fine-tuning .

We experimentally verify our guess on two datasets, i.e., training VGG16BN with completely random initialization on CIFAR10, and training ResNet18 with pre-trained initialization from PyTorch on Flowers [59]. The datasets are first uniformly split into two partitions, and then two models with corresponding initialization are separately trained or fine-tuned for 50 epochs. The checkpoints in the { 1 , 2 , 3 , 5 , 10 , 20 , 30 , 50 } -th epoch are stored. For checkpoints in the specific epoch, we denote the two models on the two partitions as θ A and θ B , respectively. The interpolation accuracy of ( 1 -λ ) θ A + λθ B on the test set is plotted in Fig. 11. The interpolation curve of VGG16BN on CIFAR10 indeed encounters a significant barrier, especially when the epoch is larger, e.g., E = 50 . In contrast, the interpolation surpasses the individual models on Flowers, which is attributed to the pre-trained ResNet18. As an explanation, we calculate the sign consistency ratio between θ A and θ I , θ B and θ I , and θ A and θ B , denoted as 'SSR-IA', 'SSR-IB', and 'SSR-AB', respectively. θ I means the initialization model. We also plot the gap of model interpolation and individual models when λ = 0 . 5 , i.e., Acc ( 0 . 5 θ A + 0 . 5 θ B ) -0 . 5 ( Acc ( θ A ) + Acc ( θ B )) . The right of Fig. 11 clearly shows that the sign consistency ratio could almost perfectly reflect the tendency of the interpolation gap. Notably, the sign consistency ratio of models on Flowers is higher than 0 . 95 , which means that fine-tuning the pre-trained model does not change the parameter signs a lot, which facilitates the following parameter interpolation . For better illustration and fair comparison, we also replace the pre-trained ResNet18 with a random initialized ResNet18, and we find the plots shown in Fig. 11 tend to be like the results of the random initialized VGG16BN. The previous work [70] points out that disagreement on the sign of a given parameter's values across models is a major source of interference during model merging. Although our finding is similar to the previous work, the motivation and specific explanation differs a lot.

6.2 Regularizing the Sign Change in Federated Learning

Traditional machine learning models will encounter the challenges posed by 'isolated data islands', e.g., the Non-I.I.D. data [54, 29]. FedAvg [54], as the most standard federated learning (FL) method, utilizes the parameter server architecture [44], and fuses collaborative models from local nodes without centralizing users' data. Specifically, the local clients receive the global model from the server and update it respectively on their devices using private data, and the server periodically

0.0

0.5

1.0

Interpolation:

λ

0.1

0.3

0.5

0.7

0.9

Test Accuracy

CIFAR10 VGG16BN

E=1

E=2

E=3

E=5

E=10

E=30

E=50

1

2

3

5

10

20

30

50

0.8

0.9

1.0

Same Sign Ratio

CIFAR10 VGG16BN

SSR-IA

SSR-IB

SSR-AB

-0.06

-0.03

0.00

0.03

0.06

Performance Gap

0.0

0.5

1.0

Interpolation:

λ

0.0

0.1

0.2

0.3

0.4

0.5

Test Accuracy

Flowers ResNet18 (No Pretrain)

E=1

E=2

E=3

E=5

E=10

E=30

E=50

1

2

3

5

10

20

30

50

0.90

0.95

1.00

Same Sign Ratio

Flowers ResNet18 (No Pretrain)

SSR-IA

SSR-IB

SSR-AB

-0.06

-0.03

0.00

0.03

0.06

Performance Gap

0.0

0.5

1.0

Interpolation:

λ

0.1

0.3

0.5

0.7

0.9

Test Accuracy

E=1

E=2

E=3

E=5

E=10

E=30

E=50

1

2

3

5

10

20

30

50

0.8

0.9

1.0

Same Sign Ratio

SSR-IA

SSR-IB

SSR-AB

-0.06

-0.03

0.00

0.03

0.06

Performance Gap

±

±

averages these models for multiple communication rounds. That is, FedAvg takes the simple parameter averaging to fuse local models. However, due to the Non-I.I.D. data and permutation invariance of DNNs [51, 67, 72], the local models could become too diverged to effectively merge. Numerous efforts are paid to regularize the local models so as not to go too far away from the global model, such as the proximal term proposed by FedProx [46], the contrastive regularization proposed by MOON [45], the dynamic regularization of FedDyn [1], and the position-aware neurons proposed by FedPAN [51]. Inspired by our finding, we propose to regularize the sign change when updating local models, i.e.,

where t denotes the communication round and k is the index of client. L k ce is the common crossentropy loss of the k -th client, and θ t is the global model received from the server. θ k t ← θ t is the local model to be updated, and the loss could regularize the sign of θ k t to be close to that of θ t . Because obtaining the sign of parameters is not a continuous function, we therefore apply a sigmoid function to parameters as an approximation. We name this method FedSign and list its pseudo-code in the Appendix C. Compared with other regularization methods, our proposed FedSign is well-motivated because of our finding that interpolating sign-consistent models may lead to a flatter loss region.

Experimental studies are verified on CIFAR10 [39] and CINIC10 [8] that are commonly utilized in previous works [72, 51]. Decentralizing the training data of these datasets by a Dirichlet distribution could simulate the Non-I.I.D. scenes as in real-world FL. The Dirichlet alpha α is utilized to control the Non-I.I.D. level, with a smaller α representing a more rigorous heterogeneity between clients' data. We set α ∈ { 10 . 0 , 1 . 0 , 0 . 5 } respectively. The number of clients is 100 , and the total communication round is 200 . During each round, a random set of 10% clients participate in FL and every client takes 5 epochs update on their individual data. After all communication rounds, we evaluate the model performance on a global test set on the server (i.e., the original test set of corresponding datasets). We select our hyperparameter γ from { 0 . 001 , 0 . 01 , 0 . 1 } and report the best results. For α = 10 . 0 , i.e., a relatively I.I.D. scene, a smaller γ is better. In contrast, γ = 0 . 1 or γ = 0 . 01 will be more proper for α = 0 . 5 . The performance comparison results are listed in Tab. 1. For FeaAvg and our proposed FedSign, we rerun the experimental studies five times and list the standard deviation of accuracies,

showing that the accuracy doesn't fluctuate very much. FedSign could surpass the compared methods, which shows the positive effects of regularizing the sign change in FL.

7 Limitations and Future Works

Although we provide theoretical insights to explain the interesting phenomenon, no formal proofs are provided to show the conditions and scopes that lead to asymmetric valleys. Additionally, this phenomenon is only investigated in the image classification tasks. Future research includes providing formal theoretical foundations for our findings and verifying them on more tasks. According to the analysis in Sect. 5, we advocate that this phenomenon is more likely to be applicable to DNNs that contain both the ReLU and softmax.

8 Conclusion

We explore and exploit the asymmetric valley of DNNs via numerous experimental studies and theoretical analyses. We systematically examine various factors influencing valley symmetry, highlighting the significant role of sign consistency between noise direction and the converged model. The findings offer valuable insights into practical implications, enhancing the understanding of model fusion. A novel regularization method is proposed for better model averaging in federated learning.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In ICLR , 2021.

Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations , 2023.

Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. In International Conference on Machine Learning , pages 840-902, 2023.

Stephen C. Ashmore and Michael S. Gashler. A method for finding similarity between multilayer perceptrons by forward bipartite alignment. In International Joint Conference on Neural Networks , pages 1-7, 2015.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In The 5th International Conference on Learning Representations , 2017.

Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the robustness of convolutional neural networks to internal architecture and weight perturbations. CoRR , abs/1703.08245, 2017.

Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not imagenet or CIFAR-10. CoRR , abs/1810.03505, 2018.

Jim Davis and Logan Frank. Revisiting batch norm initialization. In Computer Vision - ECCV 2022 - 17th European Conference , pages 212-228, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4171-4186, 2019.

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning , pages 1019-1028, 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR , abs/2010.11929, 2020.

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference on Machine Learning , pages 1308-1317, 2018.

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In The Tenth International Conference on Learning Representations , 2022.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In The 9th International Conference on Learning Representations , 2021.

Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. CoRR , abs/1912.02757, 2019.

Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. In Advances in Neural Information Processing Systems 32 , pages 6706-6714, 2019.

Jonathan Frankle. Revisiting "qualitatively characterizing neural network optimization problems". CoRR , abs/2012.06898, 2020.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning , pages 3259-3269, 2020.

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems 31 , pages 8803-8812, 2018.

Ian J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization problems. In 3rd International Conference on Learning Representations , 2015.

Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , pages 4141-4150, 2019.

SUN Haoran, WANG Yang, LIU Haipeng, and QIAN Biao. Fine-grained cross-modal fusion based refinement for text-to-image synthesis. Chinese Journal of Electronics , 32(6):1329-1340, 2023.

Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32 , pages 2549-2560, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision , pages 1026-1034, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation , 9(1):1-42, 1997.

Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. The non-iid data quagmire of decentralized machine learning. In ICML , pages 4387-4398, 2020.

Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In 5th International Conference on Learning Representations , 2017.

Daniel Jiwoong Im, Michael Tao, and Kristin Branson. An empirical analysis of the optimization of deep network loss surfaces. CoRR , abs/1612.04010, 2016.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning , pages 448-456, 2015.

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence , pages 876-885, 2018.

Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31 , pages 8580-8589, 2018.

Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof J. Geras. The break-even point on optimization trajectories of deep neural networks. In 8th International Conference on Learning Representations , 2020.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In The 8th International Conference on Learning Representations , 2020.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning , pages 5132-5143, 2020.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In The 5th International Conference on Learning Representations , 2017.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2012.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: adaptive sharpnessaware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning , pages 5905-5914, 2021.

Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems 32 , pages 8570-8581, 2019.

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31 , pages 6391-6401, 2018.

Mu Li, Li Zhou, Zichao Yang, Aaron Li, Fei Xia, David G Andersen, and Alexander Smola. Parameter server for distributed machine learning. In NeurIPS , volume 6, page 2, 2013.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In CVPR , pages 10713-10722, 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys , 2020.

Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. CoRR , abs/2309.15698, 2023.

Xin-Chun Li, Wen-Shu Fan, Shaoming Song, Yinchuan Li, Bingshuai Li, Yunfeng Shao, and De-Chuan Zhan. Asymmetric temperature scaling makes larger networks teach well again. In Advances in Neural Information Processing Systems 35 , 2022.

Xin-Chun Li, Lan Li, and De-Chuan Zhan. Visualizing, rethinking, and mining the loss landscape of deep neural networks. CoRR , abs/2405.12493, 2024.

Xin-Chun Li, Shaoming Song, Yinchuan Li, Bingshuai Li, Yunfeng Shao, Yang Yang, and De-Chuan Zhan. MAP: model aggregation and personalization in federated learning with incomplete classes. CoRR , abs/2404.09232, 2024.

Xin-Chun Li, Yi-Chu Xu, Shaoming Song, Bingshuai Li, Yinchuan Li, Yunfeng Shao, and De-Chuan Zhan. Federated learning with position-aware neurons. In IEEE Conference on Computer Vision and Pattern Recognition , pages 10082-10091, 2022.

Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems 33 , 2020.

Eliana Lorch. Visualizing deep network training trajectories with pca. 2016.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , pages 1273-1282, 2017.

Guido Montúfar, Razvan Pascanu, KyungHyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems 27 , pages 2924-2932, 2014.

Yuval Netzer, Tiejie Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems 30 , pages 5947-5956, 2017.

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems 33 , 2020.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729, 2008.

Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In 9th International Conference on Learning Representations , 2021.

Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Advances in Neural Information Processing Systems 31 , pages 2488-2498, 2018.

LYU Shen-Huan, CHEN Yi-He, and ZHOU Zhi-Hua. A region-based analysis for the feature concatenation in deep forests. Chinese Journal of Electronics , 31(6):1072-1080, 2022.

Hai Shu and Hongtu Zhu. Sensitivity analysis of deep neural networks. In The Thirty-Third AAAI Conference on Artificial Intelligence , pages 4943-4950, 2019.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations , 2015.

Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Advances in Neural Information Processing Systems 33 , 2020.

Tiffany J. Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? In International Conference on Machine Learning , pages 22325-22341, 2022.

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In 8th International Conference on Learning Representations , 2020.

Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning , pages 23965-23998, 2022.

Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition , pages 5987-5995, 2017.

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems 36 , 2023.

YANYan, MA Hongzhong, and LI Zhendong. An improved grasshopper optimization algorithm for global optimization. Chinese Journal of Electronics , 30(3):451-459, 2021.

Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. Fed2: Feature-aligned federated learning. In KDD , pages 2066-2074, 2021.

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. CoRR , abs/1806.00582, 2018.

Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning , pages 26982-26992, 2022.

A Experimental Details

In this section, we list the datasets, networks, and training details that are utilized in the body. Finally, we provide the details that motivate us to change the sign of the noise.

A.1 Dataset

The utilized datasets include 'sklearn.digits' 3 , SVHN [56], CIFAR10/100 [39], CINIC10 [8], Flowers [59], Food101 [5], and ImageNet [10]. We detail these datasets as follows.

' sklearn.digits ' contains 1797 samples of 10 digits, with each sample being a 8 × 8 image. We use this to provide a simple code demo and provide theoretical verification as in Appendix D.3.

SVHN [56] is the Street View House Number dataset which contains 10 numbers to classify. The raw set contains 73,257 samples for training and 26,032 samples for evaluation. The image size is 32 × 32 .

CINIC10 [8] is a combination of CIFAR10 and ImageNet [40], which contains 10 classes. It contains 90,000 samples for training, validation, and testing, respectively. We do not use the validation set. The image size is 32 × 32 .

CIFAR10 and CIFAR100 [39] are subsets of the Tiny Images dataset and respectively have 10/100 classes to classify. They consist of 50,000 training images and 10,000 test images. The image size is 32 × 32 .

Flowers [59] consists of 102 fine-grained flower categories, where we select 2,000 images for training and 2,000 images for testing.

Food101 [5] consists of 101 food categories, with 101, 000 images. For each class, 250 manually reviewed test images are provided as well as 750 training images. All images were rescaled to have a maximum side length of 512 pixels.

ImageNet [10] consists 1000 image categories for classification. This dataset is utilized to pre-train models as listed in 'torchvision'. Due to the large amounts of data, we only select 5,000 images in the 'val' partition to plot the curves in corresponding figures.

A.2 Network Details

We utilize VGG [64], ResNet [27], ResNeXt [69], AlexNet [40], ViT [13] in this paper. We detail their architectures as follows:

VGG contains a series of networks with various layers. The paper of VGG [64] presents VGG11, VGG13, VGG16, and VGG19. We follow their architectures and report the configuration of VGG11 as an example: 64, M, 128, M, 256, 256, M, 512, 512, M, 512, 512, M. 'M' denotes the max-pooling layer. VGG11 contains 8 convolution blocks and three fully-connected layers in [64]. The VGG architecture could use BatchNorm [32] or not, which is clearly declared in the body. If BN is used, it will be added after each convolution layer and before the ReLU activation function.

ResNet introduces residual connections to plain neural networks. We take the CIFAR versions used in the paper [27] for CIFAR10/100 and CINIC10, i.e., ResNet20 with the basic block. For Flowers, we use pre-trained ResNet18 from PyTorch. ResNet commonly uses BatchNorm [32], which is added before ReLU activation.

AlexNet [40] consists of five convolutional layers followed by three fully connected layers.

ResNeXt [69] introduces the group convolution to ResNet and we utilize the pre-trained version downloaded from 'torchvision'.

ViT [13] follows the transformer architecture for image classification tasks. It divides an image into a sequence of fixed-size patches, processes these patches linearly, and then feeds them into a transformer encoder to capture the global context of the image. In our paper, we

take 12 layers in the transformer encoder and use 8 heads in each multi-head self-attention block. We set the embedding dimension as 128 to reduce the computation burden.

A.3 Training Details

We provide the training details for obtaining converged models. We investigate the following pairs of datasets and networks. For the series of Fig. 3 and Fig. 5, we train VGG16BN on CIFAR10, ResNet20 on SVHN, VGG11 with no BN on CIFAR100, AlexNet on Food101, and ViT on Food101. For VGG16BN, ResNet20, VGG11 with no BN, and AlexNet, we use the SGD optimizer with a momentum of 0.9. We set the learning rate as 0.03. We use a cosine annealing way to decay the learning rate across 200 training epochs. The default weight decay is 0.0005, and the default batch size is 256. For training ViT, we use the AdamW optimizer with a learning rate of 0.0001. The batch size is 256, and the weight decay is 0.0005. We also take the cosine annealing way to decay the learning rate.

The equation of the BN layer is X = w X -m X √ v X + η + b , where X ∈ R C × d denotes the feature map with C channels, and m X ∈ R C and v X ∈ R C are channel-wise mean and variance values of the feature map. In practice, w , b ∈ R C are learnable parameters, while m X and v X are running statistics that are calculated during the forward pass. When interpolating θ f + λglyph[epsilon1] with BN layers, we should clear these running statistics after interpolating model parameters and feed the interpolated model to the dataset for another forward pass to calculate proper data distributions. The forward-again process is also utilized in previous works [33, 68].

For Fig. 7, we only change the initialization method of BN layers and keep the other hyperparameters not changed. For the series of Fig. 8, we only change a specific hyperparameter including the learning rate, batch size, or weight decay. Specifically, the learning rate is varied in { 0 . 1 , 0 . 003 } , and the batch size is varied in { 32 , 2048 } , and the weight decay is varied in { 0 . 001 , 0 . 00001 } .

A.4 Motivation of Changing the Sign of Noise

The asymmetric valley is initially proposed by [25], while it does not propose the inherent principles behind the phenomenon. It only points out that adding asymmetric noise (e.g., glyph[epsilon1] ∼ { 0 , 1 } ) to DNNs may result in an asymmetric valley. The symmetric noise around zero may not show such patterns (e.g., glyph[epsilon1] ∼ {-1 , 0 , 1 } ). This inspires us to plot valleys along three types of symmetric noise directions and four types of asymmetric noise directions, and the results are shown in Fig. 3. Indeed, the last four types of noise are not symmetric around zero and show slight asymmetric valleys. However, this is not so obvious. Notably, the utilized network in Fig. 3 has BN layers, i.e., VGG16 with BN.

Then, we try to apply glyph[epsilon1] ∼ { 0 , 1 } to DNNs without BN, i.e., VGG11 without BN. The valleys become symmetric as shown in the first row of Fig. 13. Hence, this makes us consider the effect of BN layers. Fortunately, we find that the traditional BN initialization will initialize the values in 'BN.weight' as 1, and the converged BN weights are all positive. The initial findings could be summarized as:

If DNNs have BN, and the parameters are perturbed by noise with symmetric values around zero, the valleys are symmetric. This is shown as the top first three plots in Fig. 3.

If DNNs have BN, and the parameters are perturbed by noise with asymmetric values around zero, the valleys are slightly asymmetric. This is shown as the top last four plots in Fig. 3.

If DNNs do not have BN, and the parameters are perturbed by noise with symmetric values around zero, the valleys are symmetric. This is shown as the top first three plots in Fig. 13.

If DNNs do not have BN, and the parameters are perturbed by noise with asymmetric values around zero, the valleys are symmetric. This is shown as the top last four plots in Fig. 13.

That is, only the second case shows slightly asymmetric valleys, where the noise (e.g., glyph[epsilon1] ∈ { 0 , 1 } ) has a large sign consistency with the BN weights (e.g., > 0 . 0 ). As shown in Sect. 4.2.1, if we replace the initialization of BN weights with a random Gaussian initialization, then the plotted valley becomes symmetric because the converged BN weights are symmetric around zero again (Fig. 7).

That is, the converged 'BN.weight' are all positive values under the common BN initialization. If we perturb them by asymmetric noise (e.g., glyph[epsilon1] ∈ { 0 , 1 } , glyph[epsilon1] ∈ G ( 1 , 1 ) , glyph[epsilon1] ∈ U ( 0 , 1 ) , or glyph[epsilon1] ∈ { 1 } ) that has

0.0

0.5

1.0

Test Error

NS

ε

s=0.2

s=1.0

s=2.0

NS

ε

∼U(-1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{-1, 0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼G(1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼U(0, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼{1}

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

0.3

0.5

1.0

Test Error

NS

ε

s=0.2

s=1.0

s=2.0

NS

ε

∼U(-1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{-1, 0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼G(1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼U(0, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼{1}

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.3

0.5

1.0

Test Error

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

a larger sign consistency with 'BN.weight', the plotted valleys are asymmetric. This provides an explanation for the asymmetric valley found by [25].

From the above summary, we guess that the sign consistency between parameters and the noise may lead to asymmetric valleys. Hence, a fantastic idea motivates us to change the sign of the noise to the same as θ , which leads to obvious asymmetric valleys (the bottom row of Fig. 3 and Fig. 13).

B More Experimental Studies

In this section, we provide more experimental verification results to make our work more solid. We list the supplemented results by the corresponding experimental studies in the body.

B.1 More Results as Fig. 3

Fig. 3 shows the valley shape under 7 common noise types. The results are plotted via training VGG16 with BN on CIFAR10. We list more results to verify the finding is common across various networks and architectures, including: (1) ResNet20 on SVHN (Fig. 12); (2) VGG11 without BN on CIFAR100 (Fig. 13). Obviously, these results are indeed similar, which verifies again that the sign consistency ratio matters a lot in the valley symmetry.

We also extend the findings to the large-scale dataset and popular network architectures including: (1) AlexNet on Food101 (Fig. 14); (2) ViT on Food101 (Fig. 15). Applying noise with a higher sign

0.3

0.5

1.0

Test Error

NS

ε

s=0.2

s=1.0

s=2.0

NS

ε

∼U(-1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{-1, 0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼G(1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼U(0, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼{1}

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.3

0.5

1.0

Test Error

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

0.3

0.5

1.0

Test Error

NS

ε

s=0.2

s=1.0

s=2.0

NS

ε

∼U(-1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{-1, 0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼G(1, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼U(0, 1)

s=0.2

s=1.0

s=2.0

NS

ε

∼{0, 1}

s=0.2

s=1.0

s=2.0

NS

ε

∼{1}

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.3

0.5

1.0

Test Error

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

=

θ

0

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=

θf

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

consistency also leads to asymmetric valleys. However, the large models are relatively stable to some extent, and the asymmetry is not as obvious as the results on the previous datasets and networks.

B.2 More Results as Fig. 5

Fig. 5 shows the valley shape under 6 special noise types. The results are plotted via training VGG16 with BN on CIFAR10. We list more results to verify the finding is nearly common across various networks and architectures, including: (1) ResNet20 on SVHN (Fig. 16); (2) VGG11 without BN on CIFAR100 (Fig. 17). Obviously, these results are indeed similar. An exceptional case is the first sub-figure in Fig. 17, i.e., using the initialization as noise leads to a symmetric valley, while the valley in Fig. 5 and Fig. 16 is asymmetric. This is because of the initialization of BN parameters, where the latter two utilize BN weights as all ones. However, Fig. 17 takes VGG11 without BN layers, which shows no asymmetry.

We also extend the findings to the large-scale dataset and popular network architectures including: (1) AlexNet on Food101 (Fig. 18); (2) ViT on Food101 (Fig. 19). Similar to the previous results, although the asymmetry is not as obvious as the results on the previous datasets and networks, the curves still show slight asymmetry when the sign consistency is high.

B.3 More Results as Fig. 6

Fig. 6 shows the valley shape investigated on ImageNet with pre-trained ResNeXt101. We also provide similar results with the pre-trained ResNet50 as in Fig. 20.

-1.0

0.0

1.0

λ

0.3

0.5

1.0

Test Error

ε

=

θ

0

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=

θf

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.3

0.5

1.0

Test Error

ε

=

θ

0

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=

θf

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

0.3

0.5

1.0

Test Error

ε

=

θ

0

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=

θf

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sign(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

)

s=0.2

s=1.0

s=2.0

-1.0

0.0

1.0

λ

ε

=sgp(

θf

-

μ

)

s=0.2

s=1.0

s=2.0

B.4 More Results as Fig. 7

Fig. 7 shows the impact of BN and its initialization on the valley symmetry. The results are plotted via training VGG16 with BN on CIFAR10. We list more results to verify the finding is common across various networks and architectures, including (1) ResNet20 on SVHN (Fig. 21); (2) ResNet20 on CIFAR100 (Fig. 22). Obviously, these results are indeed similar, which shows that the original BN initialization may lead to an asymmetric valley. Replacing the original initialization with the symmetric Gaussian distribution could lead to symmetric valleys.

B.5 More Results as Fig. 9

Fig. 9 shows the impact of various hyperparameters on valley symmetry. The results are plotted via training VGG16 with BN on CIFAR10. We list more results to verify the finding is common across various architectures, i.e., ResNet20 on CIFAR10 (Fig. 23). This also verifies that the hyperparameters have less impact on the valley symmetry.

B.6 More Results as Fig. 8

Fig. 8 studies the interpolation between two models that trained under different hyperparameters, e.g., learning rate (LR), batch size (BS), and weight decay (WD). The body only shows the impact of batch size on training VGG16BN. Then, we list more results to verify the finding is common across various architectures, including: (1) the impact of learning rate on training VGG16BN on CIFAR10 (Fig. 24); (2) the impact of weight decay on training VGG16BN on CIFAR10 (Fig. 25); (3)

0.0

0.5

1.0

ALL

Error

CE Loss

CLF

Error

CE Loss

FEAT

Error

CE Loss

LAYER1

Error

CE Loss

CONV

Error

CE Loss

BN

Error

0

4

8

CE Loss

-1.0

0.0

1.0

λ

0.0

0.5

1.0

ε

←|

ε

|*sign(

θf

)

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

CE Loss

-1.0

0.0

1.0

λ

Error

0

4

8

CE Loss

0.0

0.5

1.0

0

10

20

30

Histogram

BN.weight=1.0

Initial

Converged

0.0

0.5

1.0

0

2

4

6

BN.weight ∼ U(0, 1)

Initial

Converged

0.0

0.5

1.0

0

2

4

6

BN.weight ∼ G(0, 0.1)

Initial

Converged

0.0

0.5

1.0

Test Error

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

NS Noise

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

-1.0

0.0

1.0

λ

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

-1.0

0.0

1.0

λ

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

Filter-NS Noise

0.0

0.5

1.0

0

10

20

30

Histogram

BN.weight=1.0

Initial

Converged

0.0

0.5

1.0

0

5

10

BN.weight ∼ U(0, 1)

Initial

Converged

0.0

0.5

1.0

0

2

4

6

BN.weight ∼ G(0, 0.1)

Initial

Converged

0.3

0.5

1.0

Test Error

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

NS Noise

-1.0

0.0

1.0

λ

0.3

0.5

1.0

Test Error

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

-1.0

0.0

1.0

λ

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

-1.0

0.0

1.0

λ

ε

∼ {0, 1}, s=0.1

ε

∼ {0, 1}, s=1.0

Filter-NS Noise

0.0

0.5

1.0

Test Error

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

LR=0.003

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

BS=32

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

BS=2048

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

WD=1e-3

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

WD=1e-5

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

0.0

0.5

1.0

Test Error

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

-1.0

0.0

1.0

λ

ε

←|

ε

|*sign(

θf

)

ε

∼ G(0, 1), s=0.5

ε

∼ G(0, 1), s=1.0

ε

∼ G(0, 1), s=2.0

the impact of batch size on training ResNet20 on CIFAR10 (Fig. 26). From these figures, we could again observe that the parameter scale influences the valley width, while the sign consistency ratio matters to the valley symmetry.

Additionally, we could further obtain the following conclusion: a larger learning rate (e.g., 0.1), a smaller batch size (e.g., 32), or a larger weight decay (e.g., 0.001) could lead to better performances, which are shown in these figures with a lower test error when compared the opposite hyperparameter. However, their parameter scales are relatively smaller than opposite ones, making their valley width sharper. And commonly, a larger parameter scale of θ f 1 will let the sign of θ f 1 -θ f 2 conform to θ f 1 more, which leads to a flatter region along the positive direction of θ f 2 .

0.00

0.25

0.50

0.75

Parameter mean:

μp

0.00

0.05

0.10

Parameter std.:

σp

Parameter Distribution of

θf

1

BN Weight

CLF Weight

Other Weight

0.00

0.25

0.50

0.75

Parameter mean:

μp

Parameter Distribution of

θf

2

BN Weight

CLF Weight

Other Weight

BN Weight

CLF Weight

Other Weight

Sign Consistency Ratio

θf

1

vs

.

θf

2 -

θf

1

θf

2

vs

.

θf

2 -

θf

1

0.0

0.2

0.4

0.6

0.8

1.0

-1.0

0.0

0.5

1.0

2.0

0.0

0.5

1.0

Test Error

Interpolation of LR 0.1 and LR 0.003

θf

1

,

θf

1 +

λ

(

θf

2 -

θf

1

)

θf

2

,

θf

2

+(

λ

-1)(

θf

2 -

θf

1

)

λ

0.0

0.5

1.0

Parameter mean:

μp

0.0

0.1

0.2

0.3

0.4

Parameter std.:

σp

Parameter Distribution of

θf

1

BN Weight

CLF Weight

Other Weight

0.0

0.5

1.0

Parameter mean:

μp

Parameter Distribution of

θf

2

BN Weight

CLF Weight

Other Weight

BN Weight

CLF Weight

Other Weight

Sign Consistency Ratio

θf

1

vs

.

θf

2 -

θf

1

θf

2

vs

.

θf

2 -

θf

1

0.0

0.2

0.4

0.6

0.8

1.0

-1.0

0.0

0.5

1.0

2.0

λ

0.0

0.5

1.0

Test Error

Interpolation of WD 1e-3 and WD 1e-5

θf

1

,

θf

1 +

λ

(

θf

2 -

θf

1

)

θf

2

,

θf

2

+(

λ

-1)(

θf

2 -

θf

1

)

0.0

0.5

1.0

Parameter mean:

μp

0.0

0.5

1.0

1.5

Parameter std.:

σp

Parameter Distribution of

θf

1

BN Weight

CLF Weight

Other Weight

0.0

0.5

1.0

Parameter mean:

μp

Parameter Distribution of

θf

2

BN Weight

CLF Weight

Other Weight

BN Weight

CLF Weight

Other Weight

Sign Consistency Ratio

θf

1

vs

.

θf

2 -

θf

1

θf

2

vs

.

θf

2 -

θf

1

0.0

0.2

0.4

0.6

0.8

1.0

-1.0

0.0

0.5

1.0

2.0

0.0

0.5

1.0

Test Error

Interpolation of BS 32 and BS 2048

θf

1

,

θf

1 +

λ

(

θf

2 -

θf

1

)

θf

2

,

θf

2

+(

λ

-1)(

θf

2 -

θf

1

)

λ

Algorithm 1 FedSign

ServerProcedure :

1: for global round t = 0 , 1 , 2 , . . . , T do

3: for k ∈ S t do

2: S t ← sample max ( Q ⋅ K, 1 ) clients

4: ˆ θ k t ← ClientProcedure( k , θ t )

6: θ t + 1 ←∑ ∣ S t ∣ k = 1 1 ∣ S t ∣ ˆ θ k t

5: end for

7: end for

ClientProcedure ( k , θ t ):

1: θ k t ← θ t 2: for local epoch e 1 , 2 , . . . , E do

4: Calculate the loss as in Eq. 1, update θ t using, e.g., SGD with momentum

= 3: for each batch with B samples from D k do

5: end for

6: end for

7: Return : the updated model ˆ θ k t

C Pseudo Code of FedSign

We provide a pseudo code of the application to federated learning, i.e., FedSign. Our proposed FedSign takes a novel regularization method to limit the change of local models' signs, whose goal is for better model fusion on the server. The pseudo-code is listed as in Algo. 1. Here, T denotes the number of communication rounds, K is the total number of clients, Q is a client participating ratio in each round, E is the local update epochs of participated clients, B denotes the batch size, and D k denotes the private data of the k -th client. In our experimental studies, we set T = 200 , K = 100 , Q = 10% , E = 5 , and B = 64 . After all communication rounds, we obtain the final aggregated model θ T + 1 , and then we test its accuracy on the global test set to evaluate the aggregation performance of FedSign.

D Detailed Theoretical Analysis and Verification

Our major finding is formulated as: L ( θ f + aη ) < L ( θ f -aη ) , where η = ∣ glyph[epsilon1] ∣ ∗ sign ( θ f ) denotes the sign-consistent noise, and a > 0 is a constant. L (⋅) is the loss function. This finding holds for several settings, including (1) the cases when L (⋅) is the prediction error or cross-entropy loss, which are verified in Fig. 6 and Fig. 20; (2) the cases of applying noise to whole parameters, only to softmax classification layer, or other layers, which are verified in Fig. 6 and Fig. 20.

We present theoretical analysis from several possible aspects and finally attribute this finding to the properties of ReLU activation and softmax classification.

D.1 Gradient Analysis

First, we guess that the sign ( θ f ) may have a correlation to the gradient. Specifically, if glyph[epsilon1] = ∇ θ f L( θ f ) , then given a very smaller λ , we may have the following relationship:

where the first and third terms denote the gradient ascent step and gradient descent step, respectively. If we set glyph[epsilon1] = ∇ θ f L( θ f ) and λ ∈ [-1 , 1 ] , the plotted curve may be asymmetric. However, we frustratingly find that sign ( θ f ) are almost orthogonal to the ∇ θ f L . On one hand, the converged model θ f has very small gradient values, and elements in ∇ θ f L are nearly zero. On the other hand, adding sign ( θ f ) to θ f cannot obtain a lower error than θ f itself, which is not the same as the gradient descent direction. This implies that the inherent properties of them are not the same. Hence, we try other possible explanations.

D.2 ReLU Activation

The asymmetry may be related to the inherent asymmetry of the ReLU activation function. According to the analysis in Sect. 5, the gradient of parameters lies in the subspace spanned by the corresponding inputs. Hence, the converged parameters may correlate with their inputs to some extent. Fig. 10 shows the pattern of learned classification weight on the 'sklearn.digits'. The learned classification weight w shows a pattern of '0' as shown in the figure. However, it is hard for us to provide a concrete expression about the correlation of weight with its corresponding inputs, even for the simplest softmax classification layer . Hence, we assume that the learned w equals to a ∗ h + δ , where a is a constant and δ is a random Gaussian vector. h denotes the hidden representation. And then we simulate the distribution of ( w + λ ∗ sign ( w )) T h by the following Python code 1. Specifically, we set h ∈ R d as a vector sampled from the distribution of G ( 0 , 1 ) , and set a = 0 . 1 . We take δ as a Gaussian vector sampled from the distribution of G ( 0 , 1 ) . Then we sample N = 10000 groups of δ and plot the distribution of ( w + λ ∗ sign ( w )) T h under λ ∈ {-1 . 0 , -0 . 5 , 0 . 0 , 0 . 5 , 1 . 0 } . The distributions are shown in Fig. 27. Obviously, with an increasing λ , the distribution is shifted right. In other words, a negative λ may decrease the value of ( w + λ ∗ sign ( w )) T h . In ReLU activation, the negative values are not activated, making the loss error change a lot.

Aside from the simulation demo, we also provide the hidden activations for ResNet20 trained on SVHN. The results are shown in Fig. 28. We plot the activation confusion matrix of θ f + λ ∗∣ glyph[epsilon1] ∣ sign ( θ f ) with λ ∈ [-1 . 0 , 1 . 0 ] . For each λ , we obtain the hidden features extracted by θ f + λ ∗ ∣ glyph[epsilon1] ∣ sign ( θ f ) . We then compare the features with the original features extracted by θ f and plot the activation confusion matrix. The sum of diagonal values represents the outputs that the original model and the interpolated model commonly activate or do not activate. The value in '[]' shows the sum of diagonal values. Obviously, the value is larger when λ = a than that of λ = -a , with a ∈ { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } .

To conclude, for the ReLU activation, adding sign-consistent noise to parameters will have a higher probability of activating the neurons . If the neuron outputs are only simply scaled by a factor, it will not affect the relative scores of the final classification. For example, the inequation of w T 1 h > w T 2 h will not change if h is scaled by a positive factor, while it does not hold for h whose values are not activated, i.e., h = 0 .

D.3 Softmax Function

Given h ∈ R d , the ground-truth label y ∈ [ C ] , and the weight matrix W ∈ R C × d , the softened probability vector is p = softmax ( Wh ) . The cross-entropy (CE) loss function is L ( W ) = -log p y . The gradient of w c is g w c = -( I { c = y } -p c ) h , with c ∈ [ C ] and I {⋅} being the indication function. Specifically, the Hessian of L ( W ) = -log p y w.r.t. W is H = ( diag ( p ) -pp T ) ⊗ hh T , where ⊗ denotes the Kronecker product. The trace of H is tr ( H ) = tr ( diag ( p ) -pp T ) ∗ tr ( hh T ) . The first part could be calculated as ∑ c p c ( 1 -p c ) , where c is the class index.

-50

0

50

100

150

200

250

0

20

40

60

80

Before ReLU

λ

=-1.0

λ

=-0.5

λ

=0.0

λ

=0.5

λ

=1.0

-50

0

50

100

150

200

250

0

20

40

60

After ReLU

λ

=-1.0

λ

=-0.5

λ

=0.0

λ

=0.5

λ

=1.0

Not Act.

Act.

Orig. Not Act.

Orig. Act.

0.146

0.271

0.162

0.420

λ

=-1.0 [0.567]

Not Act.

Act.

0.185

0.232

0.172

0.411

λ

=-0.8 [0.596]

Not Act.

Act.

0.219

0.199

0.163

0.419

λ

=-0.6 [0.638]

Not Act.

Act.

0.259

0.158

0.140

0.443

λ

=-0.4 [0.702]

Not Act.

Act.

0.299

0.118

0.116

0.467

λ

=-0.2 [0.766]

Not Act.

Act.

0.326

0.091

0.118

0.464

λ

=0.2 [0.791]

Not Act.

Act.

0.321

0.097

0.136

0.447

λ

=0.4 [0.768]

Not Act.

Act.

0.314

0.104

0.153

0.430

λ

=0.6 [0.744]

Not Act.

Act.

0.302

0.116

0.169

0.413

λ

=0.8 [0.715]

Not Act.

Act.

0.289

0.129

0.178

0.405

λ

=1.0 [0.693]

We use the 'sklearn.digits' dataset and train a softmax classification weight via the 'LogisticRegression' classifier. The demo code is listed in Python code 2. When the training finishes, we obtain the converged weight W . Then, we perturb it via the Gaussian noise glyph[epsilon1] and the sign-consistent Gaussian noise ∣ glyph[epsilon1] ∣ ∗ sign ( W ) , respectively. Given a λ ∈ [-1 , 1 ] , we calculate several metrics: (1) the test error; (2) the cross-entropy loss; (3) the average trace of P λ = diag ( p )-pp T , i.e., E x [ tr ( P λ )] ; (4) the trace of the Hessian matrix, i.e., tr ( H λ ) ; (5) the coefficient of the first-order approximation of L w.r.p. λ , i.e., glyph[epsilon1] T g λ , where g λ denotes the gradient w.r.p. W + λ ∣ glyph[epsilon1] ∣ ∗ sign ( W ) ; (6) the coefficient of the second-order approximation of L w.r.p. λ , i.e., glyph[epsilon1] T H λ glyph[epsilon1] , where H λ denotes the Hessian matrix w.r.p. W + λ ∣ glyph[epsilon1] ∣ ∗ sign ( W ) . The above metrics are shown in Fig. 29. The test error and cross-entropy loss in the second row show asymmetry, because the second row takes the sign-consistent noise. More specifically, with a positive λ = a , the average trace of P λ is smaller than under a negative λ = -a . Therefore, the trace of H λ = a is smaller than that of H λ =-a . A smaller trace of Hessian means a flatter loss region.

The analysis from ReLU and softmax explain the observed phenomenon in this paper, i.e., the sign-consistent noise leads to valley asymmetry and the positive direction is flatter. However, the theoretical insights are not formal proofs, which are future works.

0.0

0.5

Error

0

10

20

CE Loss

0.025

0.050

0.075

Ex

[tr(

Pλ

)]

100

200

300

tr(

Hλ

)

0

25

50

ε T gλ

0.05

0.10

0.15

ε T Hλε

-1.0

0.0

1.0

λ

0.0

0.5

1.0

ε

←|

ε

|*sign(

θf

)

-1.0

0.0

1.0

λ

0

20

-1.0

0.0

1.0

λ

0.05

0.10

-1.0

0.0

1.0

λ

200

400

-1.0

0.0

1.0

λ

0

100

-1.0

0.0

1.0

λ

0.1

0.2

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly list the paper's scope and contributions. The contributions are summarized in Sect. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations and future works are presented in Sect. 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not provide strict theoretical proof because it is complex for DNNs. We only provide theoretical insights to explain the observations in our paper, which does not need too many assumptions and complete proof. The theoretical insights are provided in Sect. 5 and Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We report our experimental details and the phenomena in our paper are easy to reproduce. The details are provided in Appendix A. A demo code to reproduce our finding is provided in Appendix D.3 and Code 2. A pseudo-code for our application to federated learning is also provided in Algo. 1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide core codes and a demo code to reproduce the observed phenomena in our paper. We do not provide codes with external links. The demo code is in Code 1 and Code 2.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental settings and details are provided in the Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our paper presents interesting phenomena about DNNs, which do not need error bars. However, we provide multiple groups of experimental observations to verify that the phenomena are common across various conditions. For example, multiple groups of experimental studies are provided to support our finding, e.g., the plots in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The experimental studies in our paper do not need too much computation budget, which could be reproduced on mainstream devices.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work satisfies the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work has no societal impact because we focus on studying the basic properties of DNNs.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The dataset and pre-trained used in the work are publicly available, and we were unable to find the license for the dataset we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the phenomenon of asymmetric valleys in deep neural networks (DNNs) minima, first observed and described in [1], is systematically examined. By studying different types of (random) directions around a given minimum, the authors discovered that the degree of sign consistency between the chosen direction and the minimum point is a key factor for asymmetry. Besides thorough empirical evaluation, theoretical reasoning is provided to explain this phenomenon. The findings are also practically applied in model fusion.\n\n[1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.\n\nSTRENGTHS:\nAll in all, I like this work. \n\nGenerally better loss landscape understanding is important for modern deep learning, and this work takes another valuable step in this direction. To my knowledge, the question of minima asymmetry was not systematically investigated after its first appearance in [1], so I consider the contributions of this study novel and significant.\n\nThe paper is well structured. The narrative is easy to follow, the authors motivate their experiments (“Then, a fantastic idea motivates us to change the sign of the noise” – liked that) and consequently validate their conjectures. \n\nThe claims are clearly formulated and well supported empirically. Empirical evaluation is solid, involving multiple datasets, architectures, and hyperparameter options. The authors tackled many important questions concerning the study of asymmetric values and the sign consistency conjecture. Although theoretical support is mostly intuitive, with no rigorous proofs (the authors mention that in the Limitations section, which is commendable), it paves the way for future work that will address this limitation. Demonstrated practical implications of the insights of this work for model fusion are also important and require further investigation.\n\n[1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.\n\nWEAKNESSES:\nI see no concrete substantial weaknesses in this work. I list some of my questions and suggestions in the following section. \n\nTo me, the main shortcoming is weak theoretical support of the claims made in this work. However, the authors acknowledge this, and I hope that future work will address this disadvantage in some way. \n\nAlso, the experiment in Sec. 6.2 about practical implication in federated learning appears to me a little hastily conducted. Some more details would be appreciated (see specific questions below).\n\nThe text could be slightly polished to eliminate typos and small inaccuracies.\n\nQUESTIONS:\n1. The Investigation of 6 Special Noise Directions:  \nEmpirically, noise directions that have the same sign with the positive parameters and applies zero to negative parameters ($\\epsilon_5$ and $\\epsilon_6$) do not show asymmetry. I wonder, why so? Do the authors have any intuition on that?  \nTo me, that’s a little counterintuitive, since these directions specifically target the sensitive BN.weight parameters (according to the results of Sec. 4.2.1), don’t they? Also, does the same lack of asymmetry is exhibited by the negative-sign noise directions?\n2. The Finding Holds for ImageNet and Various Parameter Groups:  \n    a) What type of noise is applied here?  \n    b) It is confusing that BN parameters seem like the most “robust to asymmetry” (according to Fig. 6) in light of other experimental results of this paper, underlining the specific role of BN parameters in asymmetry. Even the original work [1] claimed that “directions on BN parameters are more asymmetric”. Could the authors elaborate more on that?\n3. Line 186: centered around 0.5?\n4. Section 4.2.1:  \nIs the noise applied to all parameters or to BN parameters only? If to all, it is indeed remarkable how the initial distribution of BN weights, which take a negligible part in total model parameters, may influence the asymmetry of certain directions (getting back to Question 2b)!\n5. Section 5: Did the authors train a plain linear classifier or a neural network in the sklearn.digits experiment?\n6. Section 6.2:  \nEq. (1): I might be wrong, but it seems that the regularizer is maximized when the signs of $\\theta_t$ and $\\theta_t^k$ are aligned, which is undesired, isn’t it?  \nThe experiments here require stds, as the results presented in Table 1 could lack statistical significance.  \nAlso, what are the hyperparameters of the baselines, how were they chosen? Were they validated just as $\\gamma$ hyperparameter in FedSign? \n7. I suppose that [2], where sign disagreement is also taken into account in model merging, could be a valuable related work.\n\n[1] Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32, pages 2549–2560, 2019.\n\n[2] Yadav, Prateek, et al. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems 37 (2023).\n\nLIMITATIONS:\nThe authors discuss the limitations in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. The major contribution is the observation that the degree of sign consistency between the perturbation and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function are provided to explain the asymmetry of DNN valleys. Imposing sign alignment in federated learning is proposed for model parameter alignment.\n\nSTRENGTHS:\nThe observation that the degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry is reasonable, and supported by the theoretical explaination based on ReLU activation and softmax function.\n\nWEAKNESSES:\n1. The asymmetry of DNN valleys is not unexpected, and has been studied in [24]. \n2. The observation that sign consistency between the noise and the convergence point affects the valley asymmetry, taking the ReLU activation and softmax function into account, is intuitive to a large degree and hence deos not emerge as a significant contribution. \n3. Lack of theoretical analysis, such as the bound of degree of asymmetry caused by sign consistency.\n4. Although the application of sign consistency in federated learning is provided, the readers may be more interested in the implications of valley asymmetry to local search based optimization methods, such as SGD, SAM etc.\n\nQUESTIONS:\n1. Fig.1 is not clear enough.\n2. There are two \"then\"s in line 177.\n\nLIMITATIONS:\nLack of theoretical analysis and the discussion on the implications of valley asymmetry to local search based optimization methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the characteristics of the asymmetric valley in deep neural networks (DNNs) for classification. The authors perform a perturbation analysis around the local minima, considering the direction of the injected noise. The asymmetric valley demonstrates that DNNs exhibit smaller fluctuations with sign-aligned noise, while opposite noise dramatically increases the test error. Other interesting findings include: (1) Batch normalization initialization and hyperparameter choices affect the occurrence of asymmetry. (2)The ReLU activation and softmax function might explain the phenomenon. The authors also leverage the findings to explain the success of model fusion and propose a sign-regularized method for federated learning.\n\nSTRENGTHS:\n1.\tThe paper is well-structured and clearly presents its interesting findings. The authors start with empirical findings and further apply their discoveries to practical applications, including the model fusion trick and federated learning.\n2.\tThe theoretical insights regarding the presence of ReLU and softmax are convincing.\n3.\tThe proposed direction-aligned noise method is simple and effective, potentially inspiring further research in areas like transfer learning and federated learning.\n\nWEAKNESSES:\n1.\tIn Section 6.1, the authors claim that pre-training is important for the success of model fusion. However, the compared models have different architectures (VGG16BN vs. ResNet18) and are trained on different datasets (CIFAR10 vs. Flowers). This experimental setting weakens the credibility of the conclusion.\n2.\tRegarding the conclusion of Fig. 8, “The interpolation curve shows that the small batch training (i.e., λ = 0.0) lies in a sharper and nearly symmetric valley, while the large batch training (i.e., λ = 1.0) lies in a flatter but asymmetric valley,” I find this problematic. Fig. 9 already shows that the valley is asymmetric for BS=32, while Fig. 8 concludes that BS=32 is symmetric. For the possible explanation for the lower part of Fig. 8, my understanding is that BS=2048 has a more biased value distribution, while BS=32 is more zero-centered. Therefore, the same strength of interpolated noise affects BS=32 more. Besides, the degree of asymmetry is not well-defined in the paper, the conclusion based on this comparison might be problematic.\n3.\tThe conclusion from Fig. 8 also conflicts with the statement in related work that “large batch size training may lead to sharp minima with poor generalization.”\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe discussion doesn't include transformer architecture.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}