Title: Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: ReLU

deep learning

optimization

parameterization

normalization

neural network

training dynamics
Abstract: We introduce a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. Our proposed analysis reveals a critical instability in common neural network parameterizations and normalizations during stochastic optimization, which impedes fast convergence and hurts generalization performance. Addressing this, we propose Geometric Parameterization (GmP), a novel neural network parameterization technique that effectively separates the radial and angular components of weights in the hyperspherical coordinate system. We show theoretically that GmP resolves the aforementioned instability issue. We report empirical results on various models and benchmarks to verify GmP's advantages of optimization stability, convergence speed and generalization performance.

Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks

Abstract

We introduce a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. Our proposed analysis reveals a critical instability in common neural network parameterizations and normalizations during stochastic optimization, which impedes fast convergence and hurts generalization performance. Addressing this, we propose Geometric Parameterization (GmP) , a novel neural network parameterization technique that effectively separates the radial and angular components of weights in the hyperspherical coordinate system. We show theoretically that GmP resolves the aforementioned instability issue. We report empirical results on various models and benchmarks to verify GmP's advantages of optimization stability, convergence speed and generalization performance.

1 Introduction

In standard neural networks, each neuron performs an affine transformation on its input x ∈ R n followed by an element-wise nonlinear activation function g :

where the affine transformation is parameterized by a weight vector w ∈ R n and a bias scalar b ∈ R . Rectified Linear Unit (ReLU) [17] is arguably the most popular activation function in modern deep learning architectures due to its simplicity and effectiveness:

We introduce a novel concept for ReLU networks, called characteristic activation boundary , which is defined as the set of input locations with zero pre-activations. By definition, such boundaries separate the active and inactive regions of ReLU neurons in the input space, which play a critical role in ReLU networks since they serve as the fundamental building blocks for the decision boundaries.

In this work, we analyze the evolution dynamics of the characteristic activation boundaries in ReLU networks. Our novel analysis identifies a critical instability in many common parameterizations and normalizations that operate in the Cartisian coordinate, including Standard Parameterization, Weight Normalization [56], Batch Normalization [24] and Layer Normalization [2]. We show theoretically that this issue destabilizes the evolution of the characteristic boundaries in the presence of stochastic gradient noise, and empirically that it impedes fast convergence and hurts generalization performance. To address this, we introduce a novel neural network parameterization, named Geometric Parameterization (GmP) . As opposed to traditional parameterizations and normalizations which operate in the Cartesian coordinate, GmP operates in the hyperspherical coordinate which automatically decouples the radial and angular components of the weights. Our theoretical results show that GmP stabilizes the evolution of characteristic activation boundaries during stochastic optimization. Our empirical

studies confirm the efficacy of GmP on various models and benchmarks. We report notable empirical improvements in optimization stability, convergence speed, and generalization performance, which validate our hypotheses and theoretical results.

2 Neural Characteristic Activation Analysis for ReLU Networks

This section defines characteristic activation boundary and its geometric connection to ReLU features, which serve as the basics of the proposed analysis for understanding neural network training dynamics.

2.1 Preliminary and Terminology

Standard Parameterization (SP) refers to the weight-bias parameterization as defined in Eq (1).

Weight Normalization (WN) [56] is a reparameterization technique that decouples the length l and the direction v / ∥ v ∥ 2 of w in a standard ReLU unit (1):

WNmakes the length l and the direction v / ∥ v ∥ 2 of the weight vector independent of each other in the Cartesian coordinate system, which is effective in improving the conditioning of the gradients and thus speeding up optimization.

Batch Normalization (BN) [24] is a widely-used normalization layer in modern deep learning architectures such as ResNet [22], which is effective in accelerating and stabilizing stochastic optimization of neural networks [29]. BN standardizes the pre-activation using the empirical mean and variance estimated by mini-batch statistics:

where γ, β ∈ R are two free parameters to be learned from data, which adjusts the output of the BN layer as needed to increase its expressiveness.

2.2 ReLU Characteristic Activation Boundary

Noticing that the ReLU activation function (2) is active for positive arguments s > 0 and inactive for negative arguments s < 0 , we introduce a novel concept called characteristic activation boundary (CAB) at the cut-off point s = 0 (i.e., the point with zero pre-activations), which will play a central role in our proposed characteristic activation analysis.

Definition 2.1 (CAB) . The characteristic activation boundary (CAB) for a ReLU unit is defined by the set of input locations with zero pre-activations:

ACABis an ( n -1) -dimensional hyperplane that separates the active and inactive regions of a ReLU unit in the input space R n . Fig 1(a) visualizes a CAB in R 2 .

Definition 2.2 (Spatial location) . The spatial location of a CAB is defined as

which is a point on the corresponding CAB since w T ϕ + b = 0 . Not that the vector that goes from the origin to ϕ specifies the shortest path between the origin and the CAB. Therefore, each spatial location uniquely determines a CAB. Fig 1(a) visualizes the spatial location of a CAB in R 2 . CABs play a critical role in ReLU networks, since they effectively specify the locations of ReLU features (i.e., non-linearities) which serve as the building blocks for the decision boundaries.

x

1

x

2

|

λ

|

ϕ

θ

B

0

(a)

B

,

ϕ

∈

R

2

.

x

1

x

2

Original feature

SP after perturbation

WN after perturbation

GmP after perturbation

(b)

δ

= 0

.

0001

.

x

1

x

2

(c)

δ

= 0

.

001

.

x

1

x

2

(d)

δ

= 0

.

01

.

x

1

x

2

(e)

δ

= 0

.

1

.

2.3 Instability of Characteristic Activation Boundary During Stochastic Optimization

In the presence of stochastic gradient noise, we identify an instability issue in the evolution of CABs under common neural network parameterizations and normalizations.

Proposition 2.3 (Instability of SP) . A perturbation ε to the weight w under SP (1) can result in an arbitrarily large change in the angular direction of the CAB if w has a similar magnitude to ε 1 .

Proof. The change in the angular direction of the CAB under SP is given by

Since w has a similar magnitude to ε , in the extreme case one can construct a small perturbation ε = -(1 + δ ) w with an infinitesimal δ . Then, the angular change is ⟨ w , w + ε ⟩ = ⟨ w , -δ w ⟩ = π , which rotates the CAB by 180°.

In general, the angle ⟨ w , w + ε ⟩ can take arbitrary values in [0 , π ] even for a small perturbation ε . This indicates that CABs are vulnerable to small perturbations when w has a small norm 1 . This has the implication that even a small gradient noise could destabilize the evolution of ReLU features during stochastic optimization and thus destroy the learning signal for the decision boundaries of the network. Such instability could prevent practitioners from using larger learning rates [18].

It might be tempting to think that WN does not suffer from this issue as it decouples the length and the direction of w as in Eq (3). However, we show that WN can also be vulnerable to small perturbations.

Proposition 2.4 (Instability of WN) . A perturbation ( ε , ε ′ ) to ( v ,l) under WN (3) can result in an arbitrarily large change in the angular direction of the CAB if v has a similar magnitude to ε .

Proof. The change in the angular direction of the CAB under WN is given by

which has an identical form to Eq (7). Therefore, in the extreme case one similarly construct a small perturbation ε = -(1 + δ ) v with an infinitesimal δ , which rotates the CAB by 180°.

Furthermore, BN can also suffer from this instability issue as it can be viewed as a generalized version of weight normalization.

Proposition 2.5 (Instability of BN) . Without loss of generality, assume that the input x has zero mean. A perturbation ε to the weight w under BN (4) can result in an arbitrarily large change in the angular direction of the CAB if w has a similar magnitude to ε .

Proof. Since ˆ E x [ w T x ] = w T ˆ E x [ x ] = 0 by assumption, Eq (4) can be re-written as

where the norm ∥ w ∥ ˜ Σ is with respect to the data covariance matrix ˆ Σ = ˆ Var [ x ] estimated by minibatch statistics. It can be seen that Eq (9) has the same form as WN (3) except that WN fixes ˆ Σ = I . Therefore, the instability argument for WN in the proof of Proposition 2.4 also holds for BN. It is worth noting that the same proof technique can be used to show instability for Layer Normalization (LN) [2], which is another normalization technique widely used in Transformers [61], since BN and LN have identical functional form (4) except that the expectation and variance operators in LN are applied to the feature axis rather than the batch axis.

Figs 1(b)-1(e) simulate the evolution behaviors of the CABs under SP and WN in R 2 , showing that even a small perturbation δ of magnitude 10 -3 can drastically change the spatial locations of the CABs. More generally, this instability issue exists in many neural network parameterization and normalization techniques that operate in the Cartesian coordinate, since the fundamental issue is that the change in angular direction of the CAB always has the same unstable form as in Eq (7).

3 Geometric Parameterization

This section introduces a novel Geometric Parameterization (GmP) and demonstrates its nice theoretical property, eliminating the instability issue in common parameterizations and normalizations.

3.1 Characteristic Activation Boundary in the Hyperspherical Coordinate System

In a high dimensional input space, most data points live in a thin shell since the volume of a high dimensional space concentrates near its surface [4]. Intuitively, the spatial locations of CABs should be close to the thin shell where most data points live, since this spatial affinity between CABs and data points will introduce ReLU features (non-linearities) at suitable locations in the input space to separate different inputs x by assigning them different activation values. The use of hyperspherical coordinate enables us to explicitly control the locations of such non-linearities.

Definition 3.1. The spatial location of a CAB in the hyperspherical coordinate system is given by

where the radius λ corresponds to b / ∥ w ∥ 2 in SP, and the unit directional vector u ( θ ) corresponds to w / ∥ w ∥ 2 in SP and is determined by the angle θ = [ θ 1 , · · · , θ n -1 ] T :

u ( θ ) ∈ S n -1 is a unit directional vector on the unit hypersphere S n -1 := { x ∈ R n : ∥ x ∥ 2 = 1 } . Definition 3.2. The CAB in the hyperspherical coordinate system is given by n

Geometrically speaking, the angle θ controls the direction of a CAB, while the radius λ controls the distance between the origin and the CAB. Calculating the pre-activation of a ReLU unit for an input x is equivalent to projecting x onto the unit vector u ( θ ) and then adding the radius λ to the signed norm of the projected vector. From this perspective, it is clear that a CAB is a set of inputs whose projections over u ( θ ) have signed norm -λ . For this reason, we refer to this radial-angular decomposition in the hyperspherical coordinate system as Geometric Parameterization (GmP).

3.2 Geometric Parameterization for ReLU Networks

Definition 3.3 (GmP) . A ReLU unit under geometric parameterization (GmP) is given by

where r, λ, θ are three learnable parameters. The radial and angular parameters λ and θ = [ θ 1 , · · · , θ n -1 ] specify the spatial location of the CAB, while the scaling parameter r controls the scale of the activation. These parameters have n +1 degrees of freedom in total (same as SP).

Remark 3.4 (Computational cost) . Let n = fan-in and m = fan-out for a layer. GmP needs to compute 2 n -2 more scalars [sin( θ 1 ) , · · · , sin( θ n -1 ) , cos( θ 1 ) , · · · , cos( θ n -1 )] than SP for each of the m neurons, which incur an extra cost of O ( mn ) for all neurons in a layer. However, since the cost of computing the affine transformation for each layer is also O ( mn ) , the total computational cost of GmP remains O ( mn ) for each layer, which is identical to SP.

Remark 3.5 (Layer-size independent parameter initialization) . Unlike existing neural network parameterizations which are sensitive to initialization, GmP can work with less carefully chosen initialization schemes independent of the width of the layer, thanks to an invariant property of the hyperspherical coordinate system. To see this, we first consider the distribution of the angular direction of the CAB under SP. Under popular initialization methods such as the Glorot initialization [16] and He initialization [21], each element in the initial weight vector w under SP is independently and identically sampled from a zero mean Gaussian distribution with a layer-size dependent variance. Interestingly, this always induces a uniform distribution over the unit n -sphere for the direction of the CAB, no matter what variance value is used in that Gaussian distribution. This allows us to initialize the angular parameter θ uniformly at random by sampling from the von Mises-Fisher distribution [63, 65]. The parameter λ is initialized to zero due to its connection λ = b / ∥ w ∥ 2 to SP and the common practice to set b = 0 at initialization. The scaling parameter r is initialized to one, based on the intuition that the scale roughly corresponds to the total variance of the weights w in SP. We highlight that none of the parameters λ , θ , and r in GmP requires layer-size dependent initialization.

Remark 3.6 (Internal covariate shift) . One implicit assumption of our characteristic activation analysis is that the input distribution to a neuron always centers around the origin during training. This assumption automatically holds for one-hidden-layer networks since the training data can be centered during data pre-processing. However, this assumption is not necessarily satisfied for multihidden-layer networks, since the inputs to an intermediate layer are transformed by the weights and squashed by the activation function in the previous layer. We propose a simple technique called Input Mean Normalization (IMN), which is a parameter-free layer that centers the input to each intermediate layer using the empirical mean estimated by mini-batch statistics:

It might be tempting to think that IMN is similar to Mean-only Batch Normalization (MBN) [56]. However, we emphasize that MBN is unable to address the internal covariate shift problem as it is applied to pre-activations rather than post-activations.

3.3 Theoretical Analysis

Section 2.3 identified an instability in the evolution of CABs under common parameterizations and normalizations due to a fundamental issue in the Cartesian coordinate system. In contrast, CABs under GmP is much more stable under perturbation, since the radius λ and angle θ of the spatial location ϕ are automatically disentangled in the hyperspherical coordinate system. This means that small perturbations to the parameters in GmP will only cause small changes in the spatial location of the CAB. Below, we show that under a small perturbation, the change in the angular direction of a CAB under GmP is bounded by the magnitude of the perturbation.

Theorem 3.7. With an infinitesimal perturbation ε := [ ε 1 , · · · , ε n -1 ] T to the angular parameter θ , the change in the angular direction u ( θ ) ∈ S n -1 ( n ≥ 2) of a CAB under GmP is bounded by

x

0

z

ϕ

|

λ

|

(a)

λ <

0

, θ

= 0

.

x

0

z

ϕ

|

λ

|

(b)

λ >

0

, θ

=

π

.

0

2000

4000

6000

8000

10000

Train step

t

2

16

2

12

2

8

2

4

2 0

2 4

2 8

2 12

2 16

i

|

i

,

t

+1

i

,

t

|

SP

WN

BN

GmP

(c) Change in

B

and

ϕ

at each train step (lr=0.1).

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

y

Test RMSE: 1.224

Training data

Ground-truth

Prediction before training

Prediction after training

Training (beginning --> end)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

x

(d) SP (lr=0.01).

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

y

Test RMSE: 1.200

Training (beginning --> end)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

x

(e) WN (lr=0.01).

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

y

Test RMSE: 1.219

Training (beginning --> end)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

x

(f) BN (lr=0.01).

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

y

Test RMSE: 0.496

Training (beginning --> end)

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

x

The proof of Theorem 3.7 can be found in Appendix B, which is based on an elegant idea from differential geometry that the angle ⟨ u ( θ ) , u ( θ + ε ) ⟩ is simply the norm ∥ ε ∥ M θ of the perturbation with respect to the metric tensor M θ for the hyperspherical coordinate, which is bounded by ∥ ε ∥ 2 . This implies that optimizing the geometric parameters in GmP directly translates into a smooth evolution of the spatial locations of CABs even in the presence of stochastic gradient noise. Looking back at Figs 1(b)-1(e), we can see that CABs under GmP gradually moves away from its original spatial location as we increase δ , which is in sharp contrast to the unstable evolution of CABs under other parameterizations.

4 Empirical Evaluation

This section contains empirical evaluation of GmP with neural network architectures of different sizes on both illustrative demonstrations and more challenging machine learning classification and regression benchmarks. A more detailed setup for each experiment can be found in Appendix C 2 .

x

x

x

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

2

(a) SP (lr=0.01).

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

x

2

(b) WN (lr=0.01).

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

x

2

(c) BN (lr=0.01).

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

x

2

(d) GmP (lr=0.01).

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

2

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

x

2

(f) WN(lr=0.1).

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

x

2

3

2

1

0

1

2

3

x

1

3

2

1

0

1

2

3

x

2

0

20

40

60

80

100

Train step

t

0

30

60

90

120

150

180

i

|

i

,

t

+1

i

,

t

|

SP

WN

BN

GmP

(i) Change in the angular direction

θ

(0°-180°) of the weights at each train step (lr=0.1).

4

3

2

1

0

1

2

3

4

x

1

4

3

2

1

0

1

2

3

4

2

Test Acc: 0.800

Classification decision boundary

Spatial location of characteristic

activation boudary

(j) SP (lr=0.01).

4

3

2

1

0

1

2

3

4

x

1

4

3

2

1

0

1

2

3

4

x

2

Test Acc: 0.810

(k) WN (lr=0.01).

4

3

2

1

0

1

2

3

4

x

1

4

3

2

1

0

1

2

3

4

x

2

Test Acc: 0.875

(l) BN (lr=0.01).

4

3

2

1

0

1

2

3

4

x

1

4

3

2

1

0

1

2

3

4

x

2

Test Acc: 0.885

4.1 Illustrative Experiments

This section verifies the validity of the hypotheses of our proposed characteristic activation analysis on two illustrative experiments aided with visualization, and demonstrates that the improved stability under GmP is beneficial for neural network optimization and generalization.

4.1.1 1D Levy Regression

In Fig 2, we train a one-hidden-layer network with 100 ReLU units under SP, WN, BN and GmP on the 1D Levy regression dataset using Adam [28]. As shown in Figs 2(a)-2(b), both the CAB B and its spatial location ϕ reduce to the same point in R , which will be referred to as the characteristic activation point. The angle θ of the characteristic activation point can only take two values 0 or π corresponding to the two directions on the real line. To use GmP in 1D, θ is initialized to 0 or π uniformly at random and fixed throughout training. Clearly, GmP significantly improves the stability of the evolution of the characteristic activation point and allows us to use a 10 × large learning rate as selected by cross validation. Fig 2(c) shows that the maximum change max i | ∆ ϕ i,t | = max i | ϕ i,t +1 -ϕ i,t | under GmP is always smaller than one throughout training at each train step t , which allows small but consistent updates to be accumulated. In contract, the changes under other parameterizations can be up to 2 16 at some steps. Such abrupt and huge changes of spatial locations make the evolution of the spatial locations inconsistent. Consequently, it is much harder for optimizers to allocate those activations to the suitable locations during training. In addition, many activations are allocated to the regions that are far away from the data region and cannot be seen in Figs 2(d)-2(f), and these activations become completely useless. The stable evolution of the characteristic point under GmP leads to improved optimization stability and generalization performance (i.e., the best test RMSE) on this task, as shown in Figs 2(d)-2(g).

4.1.2 2D Banana Classification

In Fig 3, we train a one-hidden-layer network with 10 ReLU units under SP, WN, BN and GmP on the 2D Banana classification dataset using Adam. Figs 3(a)-3(h) show that GmP allows us to use a 10 × larger learning rate (as selected by cross validation) while maintaining a smooth evolution of the characteristic activation boundary. Fig 3(i) shows that GmP is the only method that guarantees stable updates for the angular directions of the CAB during training with a large learning rate: under GmP, the maximum change max i | ∆ θ i,t | = max i | θ i,t +1 -θ i,t | at each train step t remains low throughout training, while under other parameterizations the change can be up to 180° at some steps. This verifies the hypothesis in our proposed characteristic activation analysis. Figs 3(j)-3(m) show that under GmP, the spatial locations of CABs move towards different directions during training and spread over all training data points in different regions, which forms a classification decision boundary with a reasonable shape that achieves the best generalization performance (i.e., the highest test accuracy) among all compared methods.

4.2 Machine Learning Benchmarks

This section evaluate GmP on common machine learning regression and classification benchmarks with a variety of neural network architectures, demonstrating its broad applicability.

4.2.1 UCI Regression with MLP

We evaluate GmP on 7 regression problems from the UCI dataset [11]. We train an MLP with one hidden layer and 100 hidden units for 10 different random 80/20 train/test splits. We use the Adam optimizer [28] with cross-validation. We find that the optimal learning rate is 0 . 1 for GmP and 0 . 01 for all the other methods. Table 1 shows that for in most cases, GmP consistently achieves the best test RMSE on all benchmarks, significantly outperforming other methods.

0

10

20

30

40

50

60

Epoch

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Top5 training accuracy

SP

WN

WN+MBN

BN

GmP

GmP+IMN

(a) Top-5 training accuracy.

0

10

20

30

40

50

60

Epoch

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Top5 validation accuracy

(b) Top-5 validation accuracy.

4.2.2 ImageNet32 Classification with VGG

We evaluate GmP with a medium-sized convolutional neural network VGG-6 [58] on ImageNet32 [8], which contains all 1.3M images and 1,000 categories from ImageNet (ILSVRC 2012) [10], but with the images resized to 32 × 32 . We follow the experimental setup for optimization and data augmentation as in [8]. This provides insights into how the batch size and intermediate normalization layer affect the empirical convergence speed and generalization performance of different parameterizations. We use cross-validation and find that the optimal initial learning rate is 0 . 1 for GmP and 0 . 01 for all the other methods. Table 2 shows that GmP+IMN consistently achieves the best top-1 and top-5 validation accuracy for all batch sizes considered. Furthermore, the improvement of GmP+IMN over other methods gets larger as the batch size increases, highlighting the robustness and scalability of GmP with large batch sizes. In addition to achieving the best performance, Fig 4 shows that GmP+IMN (the green curve) also converges significantly faster than other compared methods: its top-5 validation accuracy converges within 25 epochs, which is 10 epochs earlier than the second best method BN. The ablation study GmP vs GmP+IMN shows that IMN significantly improves the performance of GmP, which is expected since it addresses the problem of covariate shifts between hidden layers. Notably, Wide ResNet (WRN 28-2) [68] trained with BN and a batch size of 500 only achieved 43 . 08% top-1 validation accuracy as reported in [8], underperforming VGG-6 trained with GmP+IMN ( 43 . 62% as shown in Table 2). This reveals the significance of better parameterizations: even a small non-residual network like VGG-6 with GmP+IMN can outperform large, wide residual networks like WRN 28-2 .

4.2.3 ImageNet Classification with ResNet

We evaluate GmP with a large residual neural network, ResNet-18 [22], on the full ImageNet (ILSVRC 2012) dataset [10], which consists of 1,281,167 training images and 50,000 validation images that contain objects from 1,000 categories. The size of the images ranges from 75 × 56 to 4288 × 2848 . We follow the experimental setup for optimization and data augmentation as in [22]. Specifically, we use the SGD optimizer with momentum 0.9 and reduce the learning rate by 0 . 1 at epochs 30, 60 and 80. All models are trained for 90 epochs. We use a

batch size of 256 for all methods. We use cross-validation and find that the optimal initial learning

rate is 0 . 1 for all compared methods. We employ random horizontal flip, random resizing (256-480) with preserved aspect ratio, random crop (224), and color augmentation for data augmentation during training [31]. To address the internal covariate shift problem, we employ IMN for GmP. Following [56], MBN is used for WN. The result of SP is not reported as BN is used in ResNet by default. Table 3 reports the single-center-crop top-1 and top-5 validation accuracy for all compared methods, which shows that GmP+IMN significantly outperforms BN and WN+MBN in terms of both top-1 and top-5 validation accuracy. This demonstrates that our method is useful for improving large-scale residual network training.

5 Related Work

5.1 Neural Network Training Dynamics

Neural Tangent Kernels (NTKs) [25, 36] show that wide networks evolve like linear models during training, while Neural Network Gaussian Processes (NNGPs) [49, 35] provide insights into how wide neural networks generalize. [15] studies the evolution of the Hessian spectrum of neural networks during training. [3] investigates the curvatures of different principle components around the optimum of a regularized linear autoencoder. [1, 32, 70] analyze the training dynamics with natural gradient descent (NGD); see Appendix D for a discussion of the connection between GmP and NGD. [45] investigates the effectiveness of stochastic gradient descents for neural network training. [23, 60, 57, 71, 6, 37, 72, 38, 50, 54, 33, 62, 30] study the effects of Adam, BN and weight decay on training dynamics. [64, 26] study the failure cases in joint training of deep ensembles. [7, 53, 34, 5, 43] investigate the training dynamics of distributed optimization with different client subsampling schemes. [48, 52, 13, 14, 19, 20, 12, 55] pay special attention to analyzing the ReLU activations in neural networks. In contrast, our proposed characteristic activation analysis studies the evolution of the characteristic activation boundaries during stochastic optimization.

5.2 Neural Network Parameterization and Normalization

In addition to SP, WN [56] and BN [24], there are many other neural network normalization and parameterization techniques. Instead of normalizing the batch axis as in BN, Layer Normalization (LN) [2] operates on the feature axis, which is preferred for small batches or variable-length inputs such as text [61]. Other variants of BN include Switchable Normalization [44] and Instance Enhancement Batch Normalization [39]. There are also normalization techniques designed for specific applications. For instance, Instance Normalization [59] and Group Normalization [66] are special cases of LN designed for CNNs, while Spectral Normalization [47, 69] is specifically designed for GANs and transformers. All these methods operate in the Cartesian coordinate. The instability argument for BN also holds for these methods since they have a similar form to BN except that they normalize the input tensors along different axes. Besides, there is also a line of research on learning orthogonal rotation matrices for the weight vector [42, 40, 41, 51]. In contrast, our proposed Geometric Parameterization (GmP) overcomes the instability issue by operating in the hyperspherical coordinate.

6 Conclusion

We presented a novel characteristic activation analysis for understanding the training dynamics of ReLU networks, which exploits special activation values to characterize ReLU units. Using the proposed analysis, we identified a critical instability in common neural network parameterizations and normalization techniques that operate in the Cartesian coordinate. Addressing this, we proposed a new parameterization called Geometric Parameterization (GmP) which operates in the hyperspherical coordinate. We demonstrated the theoretical advantages of GmP for ReLU networks. We performed empirical evaluations to verify our analysis, showing its improved training stability, convergence speed and generalization performance on a variety of tasks with different neural network architectures. More broadly, we believe that the general idea behind our proposed analysis of viewing neural network parameters from a different perspective has the potential to reveal new research directions for the study of neural network optimization and training dynamics. Limitations and potential future work directions are discussed in Appendix E.

Acknowledgments and Disclosure of Funding

We thank Isaac Reid and Ross Viljoen for helpful feedback and discussions. WC acknowledges funding via a Cambridge Trust Scholarship (supported by the Cambridge Trust) and a Cambridge University Engineering Department Studentship (under grant G105682 NMZR/089 supported by Huawei R&D UK). HG acknowledges generous support from Huawei R&D UK.

Part of this work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service ( www.csd3.cam.ac.uk ), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/T022159/1), and DiRAC funding from the Science and Technology Facilities Council ( www.dirac.ac.uk ).

References

Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation , 10(2):251276, 1998.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Xuchan Bao, James Lucas, Sushant Sachdeva, and Roger B Grosse. Regularized linear autoencoders recover the principal components, eventually. Advances in Neural Information Processing Systems , 33:6971-6981, 2020.

Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of Data Science . Cambridge University Press, 2020.

Wenlin Chen, Samuel Horváth, and Peter Richtárik. Optimal client sampling for federated learning. Transactions on Machine Learning Research , 2022.

Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243 , 2020.

Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 , 2017.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Yonatan Dukler, Quanquan Gu, and Guido Montúfar. Optimization theory for relu neural networks trained with normalization layers. In International conference on machine learning , pages 2751-2760. PMLR, 2020.

Konstantin Eckle and Johannes Schmidt-Hieber. A comparison of deep networks with relu activation function and linear spline-type methods. Neural Networks , 110:232-242, 2019.

Dennis Maximilian Elbrächter, Julius Berner, and Philipp Grohs. How degenerate is the parametrization of neural networks with the relu activation function? Advances in neural information processing systems , 32, 2019.

Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning , pages 2232-2241. PMLR, 2019.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 315-323. JMLR Workshop and Conference Proceedings, 2011.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016.

Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning , pages 2596-2604. PMLR, 2019.

Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems , 32, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems , 30, 2017.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela van der Schaar. Joint training of deep ensembles fails due to learner collusion. Advances in Neural Information Processing Systems , 36, 2024.

Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, and Shuicheng Yan. Deep learning with s-shaped rectified linear activation units. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of length-direction decoupling in non-convex optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 806-815. PMLR, 2019.

Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning , 2023.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems , 32, 2019.

Antoine Labatie, Dominic Masters, Zach Eaton-Rosen, and Carlo Luschi. Proxy-normalizing activations to match batch normalization while removing batch dependence. Advances in Neural Information Processing Systems , 34:16990-17006, 2021.

Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learning via guided participant selection. In 15th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 21) , pages 19-35, 2021.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 , 2017.

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems , 32, 2019.

Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. arXiv preprint arXiv:1910.07454 , 2019.

Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems , 33:14544-14555, 2020.

Senwei Liang, Zhongzhan Huang, Mingfu Liang, and Haizhao Yang. Instance enhancement batch normalization: An adaptive regulator of batch noise. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 4819-4827, 2020.

Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg, Liam Paull, Li Xiong, Le Song, and Adrian Weller. Orthogonal over-parameterized training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7251-7260, 2021.

Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.06243 , 2023.

Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. Advances in neural information processing systems , 30, 2017.

Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 1739-1748. IEEE, 2022.

Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learningto-normalize via switchable normalization. arXiv preprint arXiv:1806.10779 , 2018.

Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning , pages 3325-3334. PMLR, 2018.

Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml , volume 30, page 3. Atlanta, GA, 2013.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 , 2018.

Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems , 27, 2014.

Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012.

Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. The Journal of Machine Learning Research , 21(1):10197-10260, 2020.

Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems , 36:79320-79362, 2023.

Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference on machine learning , pages 2847-2854. PMLR, 2017.

Monica Ribero and Haris Vikalo. Communication-efficient federated learning via optimal client sampling. arXiv preprint arXiv:2007.15197 , 2020.

Simon Roburin, Yann Dubois de Mont-Marin, Andrei Bursuc, Renaud Marlet, Patrick Perez, and Mathieu Aubry. A spherical analysis of adam with batch normalization, 2021.

David Rolnick and Konrad Kording. Reverse-engineering deep relu networks. In International conference on machine learning , pages 8178-8187. PMLR, 2020.

Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems , 29, 2016.

Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems , 31, 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 , 2016.

Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350 , 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics: Learning dynamics of normalized neural network using sgd and weight decay. Advances in Neural Information Processing Systems , 34:6380-6391, 2021.

Geoffrey S Watson. Distributions on the circle and sphere. Journal of Applied Probability , 19(A):265-280, 1982.

Andrew Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan Iliescu, Mikel Lujan, and Gavin Brown. To ensemble or not ensemble: When does end-to-end training fail? In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part III , pages 109-123. Springer, 2021.

Andrew TA Wood. Simulation of the von mises fisher distribution. Communications in statistics-simulation and computation , 23(1):157-164, 1994.

Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV) , pages 3-19, 2018.

Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems , 34:17084-17097, 2021.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Jason Ramapuram, Dan Busbridge, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. $\sigma$reparam: Stable transformer training with spectral reparametrization, 2023.

Guodong Zhang, James Martens, and Roger B Grosse. Fast convergence of natural gradient descent for over-parameterized neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281 , 2018.

Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321 , 2019.

x

0

z

ϕ

|

λ

|

(a)

λ <

0

, θ

= 0

.

x

0

z

ϕ

|

λ

|

(b)

λ >

0

, θ

=

π

.

x

0

z

ϕ

|

λ

|

(c)

λ >

0

, θ

= 0

.

x

0

z

ϕ

|

λ

|

(d)

λ <

0

, θ

=

π

.

x

1

x

2

|

λ

|

ϕ

θ

0

(a)

λ <

0

, θ

∈

[0

, π

)

.

x

1

x

2

|

λ

|

ϕ

θ

0

(b)

λ >

0

, θ

∈

[

π,

2

π

)

.

x

1

x

2

|

λ

|

ϕ

θ

0

(c)

λ >

0

, θ

∈

[0

, π

)

.

x

1

x

2

|

λ

|

ϕ

θ

0

(d)

λ <

0

, θ

∈

[

π,

2

π

)

.

A More Visualizations of ReLU Characteristic Activation Boundaries

Figs 5 and 6 respectively visualize the characteristic activation boundaries in the input spaces R and R 2 under different conditions of the radius and angle parameters.

B Proof of Theorem 3.7

Proof. Since u has unit length, the change in the angular direction ⟨ u ( θ ) , u ( θ + ε ) ⟩ under infinitesimal perturbation ε is equal to the arc length δs between the two points u ( θ ) and u ( θ + ε ) . By the generalized Pythagorean theorem, the arc length is given by

where M θ is the metric tensor for the hyperspherical coordinate system. Therefore, the change in the angular direction ⟨ u ( θ ) , u ( θ + ε ) ⟩ under infinitesimal perturbation ε is simply the norm of ε with respect to the metric tensor M θ for the hyperspherical coordinate system:

Hence, we need to work out a formula for calculating M θ .

Let x = [ x 1 , · · · , x n ] T ∈ R n ( n ≥ 2 ) be an input in the Cartesian coordinate system, where the metric tensor is the Kronecker delta m ′ ij = δ ij . For the Geometric Parameterization of the unit hypersphere S n -1 , we have

The metric tensor M θ for the Geometric Parameterization of S n -1 is the pullback of the Euclidean metric in R n :

̸

We discuss the diagonal ( a = b ) and off-diagonal ( a = b ) elements separately:

̸

∂u

Without loss of generality, we assume 0 < a < b < n . Then, we have

On the other hand, by recursively collecting like terms and using the identity that sin 2 ( θ q ) + cos 2 ( θ q ) = 1 , ∀ q , we have

̸

This shows that m ab = 0 for a = b and hence M θ is a diagonal matrix.

· a = b : Following a similar argument as above, we can obtain the diagonal elements of M θ :

On the other hand, by Eq (22), we have

Hence, it follows that

and m 11 = 1 .

Therefore, the metric tensor for the hyperspherical coordinate is a diagonal matrix

Finally, the change in the angular direction of a unit vector u ( θ ) under a perturbation ε to θ is given by the norm of ε with respect to the tensor matrix M θ :

Furthermore, since -1 ≤ sin( θ j ) ≤ 1 , it follows that 0 ≤ m i,i = ∏ i -1 j =1 sin 2 ( θ j ) ≤ 1 for all i . Therefore, Eq (27) is bounded by

This completes the proof.

C Detailed Experimental Setups

C.1 UCI Regression with MLP

We train an MLP with one hidden layer and 100 hidden units for 10 different random 80/20 train/test splits. We use the Adam optimizer [28] with full-batch training. We use cross-validation to select the learning rate for each compared method from the set { 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 , 0 . 1 , 0 . 3 } . We find that the optimal initial learning rate is 0 . 1 for GmP and 0 . 01 for all the other compared methods. We report test root mean squared error (RMSE). All models are trained on a single NVIDIA GeForce RTX 2080 Ti.

C.2 ImageNet Classification with ResNet

We train a ResNet-18 [22] on the ImageNet (ILSVRC 2012) dataset [10], which consists of 1,281,167 training images and 50,000 validation images that contain objects from 1,000 categories. The size of the images ranges from 75 × 56 to 4288 × 2848 . We follow the experimental setup for optimization and data augmentation as in [22]. We use the SGD optimizer with momentum 0.9, which turns out to be better than Adam for image classification tasks [22]. We reduce the learning rate by 0 . 1 at epochs 30, 60 and 80. All models are trained for 90 epochs. We use a batch size of 256 for all methods. We use cross-validation to select the learning rate for each compared method from the set { 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 , 0 . 1 , 0 . 3 } . We find that the optimal initial learning rate is 0 . 1 for all compared methods. We employ random horizontal flip, random resizing (256-480) with preserved aspect ratio, random crop (224), and color augmentation for data augmentation during training [31]. To address the internal covariate shift problem, we employ Input Mean Normalization (IMN) for GmP. Following [56], Mean-only Batch Normalization (MBN) is used for WN. We report singlecenter-crop top-1 and top-5 validation accuracy. The result of SP is not reported as BN is the default normalization for ResNet. All models are trained on a single NVIDIA A100 (80GB).

C.3 Ablation Study: ImageNet32 Classification with VGG

To maintain a manageable computational cost for the ablation study, we train a VGG-6 [58] on ImageNet32 [8], which contains all 1.3M images and 1,000 categories from ImageNet (ILSVRC 2012) [10], but with images resized to 32 × 32 . We follow the experimental setup for optimization and data augmentation as in [8]. We use the SGD optimizer with momentum 0.9, which turns out to be better than Adam for image classification tasks [22]. We reduce the learning rate by 0 . 1 at epochs 30, 60 and 80. All models are trained for 90 epochs. We train the model using three common batch sizes { 256 , 512 , 1024 } for all methods. We use cross-validation to select the learning rate for each compared method from the set { 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 , 0 . 1 , 0 . 3 } . We find that the optimal initial learning rate is 0 . 1 for GmP and 0 . 01 all the other methods. We employ random horizontal flips for data augmentation during training. We conduct an ablation study to explore the effects of Input Mean Normalization (IMN) for GmP and Mean-only Batch Normalization (MBN) for WN in deep networks. We report top-1 and top-5 validation accuracy. All models are trained on a single NVIDIA GeForce RTX 2080 Ti.

D Connections to Natural Gradient Descent

The parameter space of a neural network can be thought of as a Riemannian manifold M , for which the neural network parameterization specifies the coordinate for M . In this paper, we reveal that standard parameterization is vulnerable to small perturbations of the parameters (e.g., SGD noise) whereas our proposed Geometric Parameterization is much more robust against perturbations.

One thing to note is that the full version of natural gradient descent is invariant to neural network parameterizations since it is defined in an abstract form without any specific parameterization/coordinate. In the abstract form, we let the loss function be l ( θ ) = -log p ( Y | X,θ ) where θ are the abstract parameters and ( X,Y ) is an abstract data point. The fisher matrix F ( θ ) = E p ( X,Y ) [ ∇ l ( θ ) ∇ l ( θ ) T ] is the metric tensor for M , where p ( X,Y ) is the abstract data distribution. A single step of the full version of natural gradient descent is given by

where h : M → R is any differentiable function and η t is the learning rate. Note that the exponential map Exp θ t maps the update F ( θ t ) -1 ∇ h ( θ t ) from the tangent space T θ t M back to the manifold M , which results in an exact update invariant to the parameterization/coordinate. However, it is usually infeasible to calculate the exponential map since it requires solving a linear system of size dim ( M ) , which is the total number of parameters in the neural network. In practice, the most commonly used version of natural gradient descent is its first-order approximation given by

which is a second-order optimization method that is invariant up to first-order transformation of the parameterization/coordinate. Since our proposed Geometric Parameterization is a nonlinear transformation of standard parameterization, it will still behave differently under the first order approximation of the full version of natural gradient descent.

E Limitations and Future Work

Beyond ReLU activation. This work analyzed ReLU networks due to their wide adoption. However, the proposed characteristic activation analysis is general and can be directly applied to other ReLUlike activation functions such as Leaky ReLU [46], ELU [9] and SReLU [27], because the definition of spatial locations is essentially any breakpoints (i.e., the sources of non-linearity) in a piecewise activation function. The analysis of smooth activations such as Sigmoid and Tanh is left for future work.

Beyond single-hidden-layer networks. This work only performed characteristic activation analysis for single-hidden-layer ReLU networks and proposed a practical workaround to address the covariate shift issue between hidden layers by using input mean normalization (IMN). For future work, this analysis needs to be generalized to examine training dynamics in multiple-hidden-layer networks to understand the theoretical behaviors of deep networks. One potential difficulty is that the characteristic activation boundaries of multiple-hidden-layer networks become piecewise linear partitions of the input space, which are less straightforward to analyze. A possible solution would be to consider how the assignment of each data point to the partition evolves during training, similar to how we track the characteristic activation boundaries.

Sparse activation and large-width limiting behavior. Interestingly, the product structure of Geometric Parameterization (GmP) as shown in Eq (11) indicates that it could lead to sparse activations or even vanished gradients. Empirically, we find that the activations and gradients of GmP will not vanish during training if the GmP parameters are initialized from the von Mises-Fisher distribution (i.e., uniformly distributed on the hypersphere) as discussed in Remark 3.5. For future work, it would be interesting to investigate the large-width limiting behavior of GmP (e.g., similar to NTK [25] or µ P [67]) to provide theoretical guarantees of the sparsity patterns in GmP activations.

Efficient representation of hidden neurons. Directly representing the hidden neurons rather than the parameters in the hyperspherical coordinate could be a more efficient representation. However, it is unclear how to train such neural networks since gradient-based optimization is no longer applicable. One approach would be to use rejection sampling to allocate the hidden neurons on the hypersphere, but it is inefficient and suffers from curse of dimensionality. For future work, it would be interesting to find a more efficient learning algorithm that directly allocates hidden neurons on the hypersphere.

Combining GmP with existing techniques. We leave the investigation of the theoretical properties and empirical performance of combinations of GmP and existing normalization (e.g., BN or LN) and regularization techniques (e.g., weight decay) for future work. It would also be interesting to examine the performance of GmP with other neural network architectures (e.g., transformers) on large datasets from different domains (e.g., NLP) under future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claims made in the paper, which match the theoretical and experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations and potential future work directions are discussed in Appendix E. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proof of each Proposition is in the main paper following the statement. The proof of Theorem 3.7 can be found in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The implementation details of GmP are fully disclosed in Section 3.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is available at https://github.com/Wenlin-Chen/ geometric-parameterization . All datasets used are publicly available common ML benchmarks and are properly cited in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental setting/details are fully disclosed in Section 4 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All main experimental results in Section 4.2 are accompanied by error bars or confidence intervals.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources are disclosed in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research is conducted with the NeurIPS Code of Ethics in every respect. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

This is a technical ML paper focusing on the theoretical properties of the training dynamics of ReLU networks. There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a technical ML paper focusing on the theoretical properties of the training dynamics of ReLU networks. The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets used are publicly available common ML benchmarks and are properly cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies and attempts to solve the problem of unstable activation boundaries in ReLU neural networks. The authors define a relu unit in terms of its activation boundary which is the set of inputs that cause the preactivation to be zero. The point on this hyperplane closest to the origin (termed a CAB) is used to track each neuron's activation boundary. The authors show that standard parameterization, weight norm and batch norm all suffer from an instability in the direction of the activation boundary when the weights are small since the direction of the weight can change sign. To address this, they propose to operate in hyperspherical coordinates where the angles $\\theta_1, ... , \\theta_{N-1}$ and radius $\\lambda$ are used to define the hyperplane instead. Taking gradient steps in this geometric parameterization maintains bounded changes in the angles under small learning rate due to a nice metric property of this coordinate system. The\n\nSTRENGTHS:\nThis paper identifies an issue with ReLU neurons that can lead to a training instability at large learning rates and attempts to solve the problem with an elegant parameterization motivated by switching to a spherical coordinate system. The computational cost is of the same order as the original matrix computation, so it is still efficient to compute, unlike other approaches such as natural gradient descent. Next, the parameterization is size independent which is a nice property not enjoyed by SP networks. Further, the authors provide many experiments which show that the locations of the activation boundaries are more stable under their proposed parameterization, leading to a speed up in optimization.\n\nWEAKNESSES:\nThere are some assumptions in the analysis that I was unsure about (such as worst case direction of the perturbations. If instead perturbations were randomly oriented with respect to $w$, does it change the conclusions?). The authors also mention that other normalization solutions suffer from similar problems (like Layernorm etc) but this would be useful to demonstrate, perhaps in the Appendix.\n\nQUESTIONS:\n1. Some of the stability analysis is performed under the worst case assumption that $\\epsilon \\propto w$. Wouldn't this be unlikely to happen for high dimensional inputs? The main instability is caused when the vectors $w$ change sign/direction under a single step of GD. Could this issue also be solved by initializing the weights in SP to have larger variance relative to the learning rate? \n2. Have the authors considered what the structure of the gradients look like in this parameterization? Naively it seems like some of the gradient entries would have very small magnitude in high dimension since they involve products of bounded trigonometric functions. Is there a \"limiting\" large width description in this parameterization like for NTK parameterization or $\\mu$P?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper introduces a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. The authors figure out the instability in common neural network parameterization and normalization during stochastic optimization. To address this, the authors propose Geometric Parameterization (GmP), which parameterizes the unit vectors by hyperspherical coordinate representation rather than scaling. The authors conduct experiments to verify the effectiveness of the method.

STRENGTHS:
This paper proposes a parameterization method based on spherical coordinate transformation. It is novel to analyze neural networks from a geometric perspective, and the method solves the problem of  instability during stochastic optimization. 

Besides, this paper is easy to follow writhing, generally well presented.

WEAKNESSES:
Lack discussions on other activation functions.

QUESTIONS:
Have the authors explored how GmP performs on other activation functions? Does it also perform better? 

Suppose we use GmP on the hidden neurons rather than the weights, like scaling operations projecting $x$ onto a hypersphere. In other words, we use $\theta_1,\cdots,\theta_n$ to represent $x$ by spherical coordinate system. Can the authors give some analysis theoretically or experimentally?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary.\nAuthors consider training  neural nets with Adam after a change of coordinate to spherical coordinate system. They show that in spherical coordinate system the direction of the half spaces which specifies the activeness of the Relus in stable with respect to small changes in the angles, hence the optimization is supposed to be easier. The perform some experiments to see the advantage of using this coordinate system for optimization in practice.\n\nSTRENGTHS:\nThis work goes against the conventional practive in deep learning which is running first order method in the standard coordinate system, rather it change to spherical coordinates. They further discuss some advantages of this system for stability of the stability of the activation regions. They conduct various experiments to show this advantage empirically. I think this is an interesting idea; the contribution of the current paper should be assessed in their experiments section, where they show the advantage of this change of coordinates in training. However in the theoretical part there are major concerns that I will mention in the following.\n\nWEAKNESSES:\nFirst the proof of Theorem 3.7. is wrong; I think what the authors meant in the argument of the Theorem is $\\|u(\\theta) - u(\\theta + \\epsilon)\\|$ rather than their dot product (the dot product claim is obviously wrong since if epsilon goes to zero right hand side goes to zero while left hand side is constant). even with $\\|u(\\theta) - u(\\theta + \\epsilon)\\|$ instead, the proof in appendix is wrong; the correct formula of length given metric $M_\\theta$ on a manifold is: \\|u(\\theta) - u(\\theta + \\epsilon)\\| = \\int_{0}^1 \\sqrt{\\epsilon^\\top M_{\\theta + t\\epsilon} \\epsilon}. The proof can be corrected by estimataing $M_{\\theta + t\\epsilon}$ with $M_\\theta$. Though the final argument is expected to be true, perhaps with different constants. \n\nSecond, the paper is highly overclaimed about tjhe theoretical results. The theoretical claim of the paper is only that the normal vector of half space change smoothly if one parameterize in spherical coordinates despite normal coordinates, independent of the training algorithm, while the authors are claiming at the beginning that they analyze the behavior of training algorithms in this case and analyze its advantage over normal coordinates which is not the case. \n\nminor comments:\nLine 155 -> typo IMH instead of IMN\n\nFigure 2, (d), ..., (g): how is the fact that activations are more spread out related to the smooth evolution of spatial locations? (the plot with the yellow lines), and these are in 1 and 2D as far as I see, do you have a version of these plots in higher dim? (since that is the authors claim)\n\nQUESTIONS:\n-optimizing in the normal coordinate system with relu activation is known to create sparse features, which can help with generalization, and I am not sure if happens if one optimizes in spherical coordinates. Does the authors see sparsity in their training as well, and if not how do they assess this fact with their claim that spherical coordinate has better generalization?\n\n-In particular authors show in experiments that they can pick a larger learning rate in spherical coordinates, but this fact solely cannot be an indicator of better generalization (even if two algorithms A and B run in the same coordinate system and A picks larger learning rate than B it is still not clear which one generalize better). Can the authors further elaborate on their claim about superiority of spherical coordinates for the generalization of the final network?\n\nLIMITATIONS:\nNo theoretical justification for superiority of their method for optimization/generalization.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn standard neural networks each neuron, and activation function g performs the following operation on the input $x\\in\\mathbb{R}^{n}$:\n$$z=g\\left(w^{t}x+b\\right)$$\nWhen usually g in the ReLU activation. \n\nIn this work the authors identify a critical instability in many common NN settings, which theoretically destabilizes the evolution of the characteristic boundaries, and empirically impedes fast convergence and hurts generalization performance. To address this, they introduce a novel NN parameterization, named Geometric Parameterization (GmP) which operates in the hyper spherical coordinate. The authors show that this parameterization, both theoretically and empirically, improves optimization stability, convergence speed, and generalization performance.\n\nSTRENGTHS:\nThe main strength of this paper (besides the actual methods introduced) is that the results are clear and concise. As a reader, it is relatively easy for me, based on the empirical results, to understand how these methods perform compared to other common methods. I will now explain each dimension.\n\n**Originality**: I think that this paper introduces novel ideas into neural networks (NN). The use of a different coordinate system is not something that I have seen before, and it opens the door to many interesting new approaches to NN.\n\n**Quality**: There is not much to say here, as I feel this is a quality paper. All claims and lemmas are proved as needed.\n\n**Clarity**: The clarity of this paper is good. This mainly stems from the fact that the paper separates the explanation of the problem from the explanation of the solution. Both parts are described using theoretical explanations and empirical/graphical examples.\n\n**Significance**: It is hard to estimate significance, as usually only time will tell how significant the research is, but I feel that this paper will be significant. I believe that the main significance lies not necessarily in the methods introduced, but rather in the idea behind them. The idea of a different perspective on the learned parameters has the potential to lead to more unique and potentially beneficial views.\n\nWEAKNESSES:\nI will start with a somewhat nit-picking weakness. The paper uses $\\mathcal{B}$ and ϕ in two different contexts. ϕ is used in Definition 2.2 as a vector and in Definition 3.1 as a function. $\\mathcal{B}$ is used in Definition 2.2 as a hyperplane and in Definition 3.2 as a function. This isn't accidental, as there is a strong connection between the definitions, but I believe that using slight differences in the notations would be better.\n\nAnother weakness is that the article hasn't provided an empirical case where GmP isn't the best option. It is possible that GmP is always the best choice (and if so, great), but usually, there are cases where new techniques aren't the best options. It would have been interesting to see a scenario where GmP isn't the best option and (perhaps under future work) understand why.\n\nQUESTIONS:\nI have few technical questions regarding section B – proof of theorem 3.7:\n\t\n1. It is not clear to me why equation 16 is true. I would suggest adding an explanation or a reference to one.\n\t\n2. The analyze of the case where a≠b. It isn't immediately clear from equation 18 why \" sum of terms that are either zero or with alternating signs\". Maybe it is simple calculus, but I suggest adding additional step to show this claim.\n\nLIMITATIONS:\nYes, the authors do address the limitations. The main limitation is the input mean normalization that being address in appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-20"}