Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks
Authors: Wenlin Chen
Hong Ge
Venue: NeurIPS 2024 poster (2024)
Keywords: ReLU
deep learning
optimization
parameterization
normalization
neural network
training dynamics
Primary area: optimization_for_deep_networks
TL;DR: We identify a training instability issue for common neural network parameterizations and normalizations and propose a novel Geometric Parameterization to fix it.

Abstract: We introduce a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. Our proposed analysis reveals a critical instability in common neural network parameterizations and normalizations during stochastic optimization, which impedes fast convergence and hurts generalization performance. Addressing this, we propose Geometric Parameterization (GmP), a novel neural network parameterization technique that effectively separates the radial and angular components of weights in the hyperspherical coordinate system. We show theoretically that GmP resolves the aforementioned instability issue. We report empirical results on various models and benchmarks to verify GmP's advantages of optimization stability, convergence speed and generalization performance.

Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks
Wenlin Chen University of Cambridge MPI for Intelligent Systems vc337@cam ac.uk .uk
Hong Ge University of Cambridge hg3440cam. ac uk
Abstract
We introduce a novel approach for analyzing the training dynamics of ReLU net- works by examining the characteristic activation boundaries of individual ReLU neurons. Our proposed analysis reveals critical instability in common neural net- work parameterizations and normalizations during stochastic optimization, which impedes fast convergence and hurts generalization performance. Addressing this, we propose Geometric Parameterization (GmP), novel neural network parame- terization technique that effectively separates the radial and angular components weights in the hyperspherical coordinate system. We show theoretically that GmP resolves the aforementioned instability issue. We report empirical results on various models and benchmarks to verify GmP's advantages of optimization stability, convergence speed and generalization performance.
Introduction
In standard neural networks, each neuron performs an affine transformation on its input x E IR" followed by an element-wise nonlinear activation function
z = g(w" x +b),
where the affine transformation is parameterized by weight vector E R and a bias scalar E IR. Rectified Linear Unit (ReLU) [17] is arguably the most popular activation function in modern deep learning architectures due to its simplicity and effectiveness:
g(s) (s)UU(s) max(0, s).
We introduce novel concept for ReLU networks, called characteristic activation boundary, which is defined as the set of input locations with zero pre-activations. By definition, such boundaries separate the active and inactive regions of ReLU neurons in the input space, which play critical role in ReLU networks since they serve as the fundamental building blocks for the decision boundaries.
In this work, analyze the evolution dynamics of the characteristic activation boundaries in ReLU networks. Our novel analysis identifies a critical instability in many common parameterizations and normalizations that operate in the Cartisian coordinate, including Standard Parameterization. Weight Normalization [56], Batch Normalization [24] and Layer Normalization (2). We show theoretically that this issue destabilizes the evolution of the characteristic boundaries in the presence of stochastic gradient noise, and empirically that it impedes fast convergence and hurts generalization performance. To address this, introduce novel neural network parameterization, named Geometric Parameteri- zation (GmP). As opposed to traditional parameterizations and normalizations which operate in the Cartesian coordinate, GmP operates in the hyperspherical coordinate which automatically decouples the radial and angular components of the weights. Our thcoretical results show that GmP stabilizes the evolution of characteristic activation boundaries during stochastic optimization. Our empirical
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

studies confirm the efficacy of GmP on various models and benchmarks. We report notable empirical improvements in optimization stability, convergence speed, and generalization performance, which validate our hypotheses and theoretical results.
Neural Characteristic Activation Analysis for ReLU Networks
This section defines characteristic activation boundary and its geometric connection to ReLU features, which serve the basics of the proposed analysis for understanding neural network training dynamics.
2.1 Preliminary and Terminology
Standard Parameterization (SP) refers to the weight-bias parameterization as defined in Eq [I
Weight Normalization (WN) [56 is reparameterization technique that decouples the length and the direction Y/1 vll2 of standard ReLU unit (T):
(((1T)) Y |
2 == ReLU
+b
(3)
WN makes the length and the direction v/[Mvl2 of the weight vector independent of each other Cartesian coordinate system, which is effective in improving the conditioning of the gradients and thus speeding up optimization.
Batch Normalization (BN) [4 is widely-used normalization layer in modern deep learning architectures such as ResNet 22 which effective in accelerating and stabilizing stochastic
optimization of neural networks 29 BN standardizes the pre-activation using the empirical mean
and variance estimated by mini-batch statistics:
BN(w" x +b) = W x Ex/wTT
+ 3,
(4)
Varx /wT x +b]
where 7, 3 E R are two free parameters to be learned from data, which adjusts the output of the BN layer as needed to increase its expressiveness.
2.2 ReLU Characteristic Activation Boundary
Noticing that the ReLU activation function (2) is active for positive arguments > 0 and inactive for negative arguments < O, we introduce a novel concept called characteristic activation boundary (CAB) the cut-off point s 0 (i.e., the point with zero pre- pre-activations), which will play a central role in our proposed characteristic activation analysis.
Definition 2.1 (CAB). The characteristic activation boundary (CAB) for a ReLU unit is defined by the set of input locations with zero pre- are-activations:
((w, b) {x IR": w x x
(5)
CAB an (n dimeenssona hyperplane that separates the active and inactive regions of ReLU unit the input space r. Fig ((a visualizes CAB in R².
Definition 2.2 (Spatial location). The spatial location CAB defined as
Φ(w,b) b) = w w
w
b
W
[
|w/22
which is a point on the corresponding CAB since w & +b= 0. Not that the vector that goes from the origin to & specifies the shortest path between the origin and the CAB. Therefore, each spatial location uniquely determines CAB. Fig ((a) visualizes the spatial location of CAB in R22 CABs play critical role in ReLU networks, since they effectively specify the locations of ReL.U features
(i.c., non- non-linearities, which serve as the building blocks for the decision boundaries.

---

12
Ony festure
(a)
R
(b) 6 0.0001.
(c) 0.001.
(d) 0.01.
(c) 0.1.
Figure 1: (a) Characteristic activation boundary (CAB) B (brown solid line) and spatial location
Φ -uu(0) (()) of a ReLU unit ReLU(((00)* x +X) = ReLU(cos(O)zz + sin(0) + A) for inputs x E F The CAB forms a line in R which acts as boundary separating inputs into two regions. Green arrows denote the active region, and red arrows denote the inactive region. (b)-(e) Stability of the CAB of ReLU unit in R² under small perturbations lines denote characteristic activation boundaries B, and colored dotted lines connect the origin and spatial locations of B. Smaller changes between the perturbed and original boundaries imply higher stability. GmP is most stable against perturbations.
51 the parameters. Solid
2.3 Instability of Characteristic Activation Boundary During Stochastic Optimization
In the presence of stochastic gradient noise. we identify an instability issue in the evolution of CABs under common neural network parameterizations and normalizations.
Proposition 2.3 (Instability of SP). perturbation to the weight w under SP I can result in an
arbitrarily large change in the angular direction of the CAB if w has similar magnitude to e!
Proof. The change in the angular direction of the CAB under SP is given by
w | w E||2
+
+ E arccos
Since w has similar magnitude to e, in the extreme case can construct small perturbation = -(1 + 5) w with an infinitesimal d. Then, the angular change is (w, w + = (w, -6 n which rotates the CAB by 180"
In general, the angle (w, w + ε) can take arbitrary values in [0, π| even for a small perturbation e. This indicates that CABs are vulnerable to small perturbations when w has small norm!l This has the implication that even a small gradient noise could destabilize the evolution of ReLU features during stochastic optimization and thus destroy the learning signal for the decision boundaries of the network. Such instability could prevent practitioners from using larger learning rates (18]
It might be tempting to think that WN does not suffer from this issue as it decouples the length and the direction of in Eq 33. However, show that WN can also be vulnerable to small perturbations.
Proposition 2.4 (Instability of WN). A perturbation (E, e²) (v.l) under WN B result in
arbitrarily large change in the angular direction of the CAB if v has a similar magnitude to e.
Proof. The change in the angular direction of the CAB under WN is given by
( Ivvlll
(1+
v v+ |
arccos ( (vv(v== vv E)
(8)
11v||21lV +
which has identical form to Eg 77. Therefore, in the extreme case similarly construct a small perturbation & -(1 + 5) v with an infinitesimal 5, which rotates the CAB by 180".
Furthermore, BN can also suffer from this instability issue as it can be viewed as generalized version of weight normalization.
|| is supposed to be small during training as large weights would lead to overfitting, numerical instability and even divergence [18] (e.g., the popular weight decay method explicitly regularizes ||w||a to be close to zero).

---

Proposition 2.5 (Instability of BN). Without loss of generality, assume that the input x has zero mean, A perturbation to the weight under BN 4 can result in an arbitrarily large change the angular direction of the CAB if w has a similar magnitude to e.
Proof. Since Ex(wTx x] w [x[xx[] = O by assumption, Eq 44) can be re- re-written as
BN(W x +b) = w x
=
w
T
x
+
W w
w |>
where the norm |/ with respect to the data covariance matrix Σ Var[x] estimated by mini-
batch statistics. It can be seen that Eq ( has the same form WN 33) except that WN fixes E = I. Therefore, the instability argument for WN in the proof Proposition 2.4 also holds for BN. worth noting that the same proof technique can be used to show instability for Layer Normalization (LN) 2]. which another normalization technique widely used in Transformers [61], since BN and LN have identical functional form (4) except that the expectation and variance operators in LN applicd to the feature axis rather than the batch axis.
Figs l(b) ((e) simulate the evolution behaviors of the CABs under SP and WN in R.? showing that even a small perturbation á of magnitude 10-3 can drastically change the spatial locations of the CABs. More generally, this instability issue exists in many neural network parameterization and normalization techniques that operate in the Cartesian coordinate, since the fundamental issue is that the change in angular direction the CAB always has the same unstable form as in Eq
3 Geometric Parameterization
This section introduces a novel Geometric Parameterization (GmP) and demonstrates its nice theoret- ical property, eliminating the instability issue in common parameterizations and normalizations.
3.1 Characteristic Activation Boundary in the Hyperspherical Coordinate System
In a high dimensional input space, most data points live in a thin shell since the volume of a high dimensional space concentrates near its surface [4. Intuitively, the spatial locations of CABs should be close to the thin shell where most data points live, since this spatial affinity between CA and data points will introduce ReI.U features (non-linearitie) at suitable locations in the input space to separate different inputs by assigning them different activation values. The use of hyperspherical coordinate enables to explicitly control the locations of such non-lincsiities
Definition 3.1. The spatial location of CAB in the hyperspherical coordinate system given by
Φ(>, 0) -À (O),
(10)
where the radius corresponds to l/ll in SP, and the unit directional vector u(f) corresponds to w//wwlaa in SP and is determined by the angle 0 [01, O 1n=
cos 0
sin cos (02)
sin(0; (in) cos (Os)
u
(11)
sin(O1) sin sin(0,) sin (6 (0n 2) cos(O (in(()) (01) sin (02) sin (z sin 1in
u(0) E 55 is unit directional vector on the unit hypersphere 5 {x E IR" |x||| 1}. Definition 3.2. The CAB in the hyperspherical coordinate system given by
B(A, 0) {x E R" :(0)" x + 0}.
(12)
Geometrically speaking, the angle controls the direction of a CAB, while the radius À controls the distance between the origin and the CAB. Calculating the pre-activation of ReLU unit for an input x is equivalent to projecting x onto the unit vector u0) and then adding the radius to signed norm of the projected vector. From this perspective, it is clear that a CAB is set of inputs whose projections over u()) have signed norm -. For this reason, we refer this radial-angula, decomposition in the hyperspherical coordinate system Geometric Parameterization (GmP).

---

3.2 Geometric Parameterization for ReLU Networks
Definition 3.3 (GmP), A ReLU unit under geometric parameterization (GmP) is given by
2 = (GlL(u(O)'++>)
(13)
where A, 0 are three learnable parameters. The radial and angular parameters À and 0 = [0,,-.. 0n-1] specify the spatial location of the CAB, while the scaling parameter r controls the scale of the activation. These parameters have n + 1 degrees of freedom in total (same as SP).
Remark 3.4 (Computational cost). Let n- and m= out for a layer. GmP needs to compute c0s-0n than SP for each of the m neurons, which incur an extra cost of (mm)) for all neurons in layer. However, since the cost of computing the affine transformation for each layer is also O (mre), the total computational cost of GmP remains (mm) for each layer, which is identical to SP.
2n - 2 more scalars [sin(O)), sin sin(0,.1), cos(O)),
Remark 3.5 (Layer-size independent parameter initialization). Unlike existing neural network parameterizations which are sensitive to initialization, GmP can work with less carefully chosen initialization schemes independent of the width of the layer, thanks to an invariant property of the hyperspherical coordinate system. To see this, we first consider the distribution of the angular direction of the CAB under SP. Under popular initialization methods such the Glorot initialization [6]] and He initialization (21), each element in the initial weight vector w under SP is independently and identically sampled from a zero mean Gaussian distribution with layer-size dependent variance. Interestingly, this always induces a uniform distribution over the unit nt-phere for the direction of the CAB, no matter what variance value is used in that Gaussian distribution. This allows to initialize the angular parameter 0 uniformly at random by sampling from the von Mises-Fisher distribution zero due to connection À 1/lw//l SP and the
63 65]. common practice to set 6 = 0 at initialization. The scaling parameter is initialized to one. based on the intuition that the scale roughly corresponds to the total variance of the weights w in SP. We highlight that none of the parameters X, 0, and in GmP requires layer-size dependent initialization.
The parameter À is initialized
Remark 3.6 (Internal covariate shift). One implicit assumption of our characteristic activation analysis is that the input distribution to neuron always centers around the origin during training. This assumption automatically holds for one- mee-idden-llaee networks since the training data can be centered during data pre- pre-proeessing However, this assumption is not necessarily satisfied for multi- hidden-layer networks, since the inputs to intermediate layer are transformed by the weights and squashed by the activation function in the previous layer. We propose a simple technique called Input Mean Normalization (IMN), which is a parameter-free layer that centers the input to each intermediate layer using the empirical mean estimated by mini-batch statistics:
2 ReLU(((()) (x-xxx)) + A).
(14)
might be tempting to think that IMN is similar to Mean-only Batch Normalization (MBN) [56].
However, we emphasize that MBN is unable to address the internal covariate shift problem as it is applied to pre-activations rather than post-activations.
3.3 Theoretical Analysis
identified an instability in the evolution of CABs under common parameterizations and normalizations due to fundamental issue in the Cartesian coordinate system. In contrast, CABs under GmP is much more stable under perturbation, since the radius À and angle of the spatial location @ are automatically disentangled in the hyperspherical coordinate system. This means that small perturbations to the parameters in GmP will only cause small changes in the spatial location of the CAB. Below, show that under small perturbation, the change in the angular direction of CAB under GmP is bounded by the magnitude of the perturbation.
Section 2.3
Theorem 3.7. With an infinitesimal perturbation E == [E1,... "En-1 to the angular parameter O, the change in the angular direction u(O) E S (n 2 2) of CAB under GmP is bounded by
(u()), u +e)) = arccos (u(0)² u(O + +E))
of
sin ((0))
of |ell2- (15)
2
5

---

x
0
0
2000
000
6000
50C0
10005
(a) < 0,8
(b) A > 0,0
(c) Change in B and & at each train step (Ir=0.1).
Prscctioe aftrree oer
(d) SP (lr0).01).
(e) WN (Ir=0.01).
(1) BN (lr=0.01).
(g) GmP (Ir=0.1).
Figure 2: (a)-(b) Characteristic activation point B (intersection of brown solid lines and the x-axis) and spatial location Φ = (0) of ReL.U unit = = Rel + X) (blue solid lines) for inputs E RR. Green arrows denote active regions, and red arrows denote inactive regions. (c) Evolution dynamies of the characteristic points B in a one-hidden-layer network with 100 ReLU units for a ID Levy regression problem under SP, WN, BN and GmP during training. SP stands for standard parameterization, WN stands for weight normalization, BN stands for batch normalization, and GmP stands for geometric parameterization. Smaller values are better as they indicate higher stability of the evolution of the characteristic points during training. The y-axis row illustrates the experimental setup. including the network's 's predictions at initialization and after training. and the training data and the ground-truth function (Levy). Bottom row: the evolution of the characteristic activation point for the 100 ReLU units during training. Each horizontal bar shows the spatial location spectrum for a chosen optimization step, moving from the bottom (at initialization) to the top (after training with Adam). More spread of the spatial locations covers the data better and adds more useful non-linean non-linearities to the model, making prediction more accurate. Regression accuracy measured by root mean squared error (RMSE) on separate test set. Smaller RMSE values are better. We use gross-validation to select the learning rate for each method. The optimal learning rate for SP, WN, and BN is lower than that for GmP, since their training becomes unstable with higher learning rates, as shown in (c).
logg scale. (d)-(g): The top
The proof of Theorem 3.7 can be found in Appendix B which is based on an elegant idea from differential geometry that the angle (u(#), u(O + =)) simply the norm || of the perturbation with respect to the metric tensor M, for the hyperspherical coordinate, which bounded by ||=||2- This implies that optimizing the geometric parameters in GmP directly translates into a smooth evolution of the spatial locations of CABs even the presence of stochastic gradient noise. Looking we can see that CABs under GmP gradually moves away from its original spatial location as we increase 6, which is in sharp contrast to the unstable evolution CABs under other parameterizations,
back at Figs] ((b) ((e)
4 Empirical Evaluation
This section contains empirical evaluation of GmP with neural network architectures of different sizes on both illustrative demonstrations and more challenging machine learning classification and regression benchmarks. A more detailed setup for each experiment can be found in Appendix
"Our code is available wttcppps//gettab....///elllllaaa httppp//w..ettccccoooooao..a.aae

---

(a) SP (I=)).1).
(b) WN (I=0000)).
(c) BN (l==0.01).
(d) GmP (I==0.01).
(c) SP (Ir=0.1).
(1) WN (Ir-0.1).
(g) BN (Ir=0.1).
(h) GmP (Ir=0.1).
180
max [0 - 0..l
SP
150
120
WN
BN
GmP
20
100
Train step
(i) Change in the angular direction 0 (0*-180") of the weights at each train step (I=00.1).
Taas Acc: 0,800
Test Acc: 0.810
Test Acc: 0.875
Test Acc. 0.885
SP (I==0.01).
WN (Ir=0001).
() BN (I==0.01).
(m) GmP (Ir=0.1).
Figure 3: Performance of finle-hiddden-lyer neural network with 10 ReLU units the 2D Banana classification dataset under SP, WN, BN and GmP trained using Adam. SP stands for standard parameterization, WN stands for weight normalization, BN stands for batch normalization, and GmP stands for geometric parameterization. (a)-(h): Trajectories of the spatial locations of the 10 ReLU units during training. Each color depicts one ReLU unit. Smoother evolution means higher training stability. The evolution under GmP is stable, so we can use a 10x larger learning rate. (i): Evolution dynamies of the angular direction of CABs. Smaller values are better as they indicate higher robustness against stochastic gradient noise. (i)-(m): Network predictions after training. Black bold lines depict the classification boundary between two classes. Classification accuracy is measured on separate test set. Higher accuracy values are better. The red stars show the spatial locations of 10 ReLU units. Intuitively speaking, more evenly spread out red stars are better for classification accuracy, they provide more useful non-linearity.

---

Table 1: Test RMSE for MLPs trained on 7 UCI benchmarks.
Benchmark
Boston
Concrete
Energy
Naval
Power
Wine
Yocht
SP
9.370
0.145
5.472
0.144
0.808 0.274
0.002 0.000
4.065 0.029
0.623 0.008
0.063
WN
0.156
5,952
2.063
0.789
0.832
0.076
BN
8.469
0.158
0.160
1.648 0.302
0.001
0.028
0.622 0.011
0.005
Gma
3.087
0.144
5.153 0.098
0.474 = 0.013
0.003 0.001
4.022 :: 0.025
0.618 = 0 0.008
0.584 0.046
4.1 Illustrative Experiments
This section verifies the validity of the hypotheses of our proposed characteristic activation analysis on two illustrative experiments aided with visualization, and demonstrates that the improved stability under GmP is beneficial for neural network optimization and generalization.
4.1.1 ID Levy Regression
In Fig 2 we train one-hidden-layer network with 100 ReLU units under SP, WN, BN and GmP on the 1DI LivLevy regression dataset using Adam [28]. As shown in Figs 2(a) 2 both the CAB B and its spatial location é reduce to the same point in IR, which will be referred to as the characteristic activation point. The angle 0 of the characteristic activation point can only take two values 0 or corresponding
the two directions on the real line. To use GmP in ID, 0 is initialized 0 or и uniformly at random and fixed throughout training. Clearly, GmP significantly improves the stability of the evolution of the characteristic activation point and allows us to use a 10x large learning rate as selected by cross validation. Fig [(c) shows that the maximum change max; | = max; [0s,++1 - Φ under GmP is always smaller than one throughout training at each train step 1, which allows small but consistent updates to be accumulated. In contract, the changes under other parameterizations can up to 216 some steps. Such abrupt and huge changes of spatial locations make the evolution of the spatial locations inconsistent. Consequently, it is much harder for optimizers to allocate those activations to the suitable locations during training. In addition, many activations are allocated to the regions that are far away from the data region and cannot be seen in Figs 2 2(D) and these activations become completely useless. The stable evolution of the characteristic point under GmP leads to improved optimization stability and generalization performance (i.e., the best test RMSE) on this task, as shown in Figs 2(d) (gg
4.1.2 2D Banana Classification
In Fig 3 we train a one-hidden-layer network with 10 ReI.U units under SP, WN, BN and GmP on the 2D Banana classification dataset using Adam. Figs 3(a) 3())3(hh))) show that GmP allows use 10x larger learning rate (as selected by cross validation) while maintaining a smooth evolution of the characteristic activation boundary. Fig 3(i shows that GmP is the only method that guarantees stable updates for the angular directions of the CAB during training with a large learning rate: under GmP, the maximum change max; | max; 10:,+1 0z,l each train step remains low throughout training, while under other parameterizaaions the change can be up to 180 at some steps. This verifies the hypothesis in our proposed characteristic activation analysis. Figs 3() 3(m) show that under GmP, the spatial locations of CABs move towards different directions during training and spread over all training data points in different regions, which forms a classification decision boundary with reasonable shape that achieves the best generalization performance (i.e., the highest. test accuracy) among all compared methods.
4.2 Machine Learning Benchmarks
This section evaluate GmP on common machine learning regression and classification benchmarks with variety of neural network architectures, demonstrating its broad applicability.
4.2.1 UCI Regression with MLP
We evaluate GmP on 7 regression problems from the UCI dataset [1 We train an MLP with one hidden layer and 100 hidden units for 10 different random 80/20 train/test splits. We use the Adam optimizer 28 with cross- validaaaii,..... We find that the optimal learning rate is 0.1 for GmP and 0.01 for all the other methods. Table[]shows that for in most cases, GmP consistently achieves the best test RMSE on all benchmarks, significantly outperforming other methods.
8

---

Table 2: Top-1 and top-5 validation accuracy (%) for VGG-6 trained on ImageNet32.
Metric
Top-1 validation accuracy
Top-5 validation accuracy
Batch size
256
512
1024
1024
SP
38.31 + 0.13
36.99
+ 0.11
35.02 + 0.03
62.48 + 0.14
60.71 : 0.18
58.14 + 0.39
WN
39.13 + 0.10
37.92
0.12
36.17 + 0.03
63.28 + 0.02
61.93 + 0.09
60.16 + 0.18
WN+MBN
42.22
0.01
40.96
0.02
39.33
+ 0.07
66.04 = 0.07
65.08
- 0.03
63.32
BN
42.79
0.03
41.90
0.19
41.39
- 0.02
67.17 0.08
66.50
0.25
65.89 4.000.00
GmP
40.76
41.65
0.09
41.29
= 0.08
06.08
69.49
GmP+IMN
43.14 +
43.62 + 0.08
42.70
+ 0.15
67. + 0.05
67.76 0.09
66.98 +
105 105tcaiigg. 3200500
0.5
0.5
WN+MBN
BN
Gmp
04-
GmP+IMN
30
40
Epoch
(a) Top-5 training accuracy.
(b) Top-5 validation accuracy.
Figure 4: Convergence speed for VGG-6 trained on the ImageNet32 dataset with batch size 1024.
4.2.2 ImageNet32 Classification with VGG
We evaluate GmP with medium-sized convolutional neural network VGG-6 (58 on ImageNet32 8), which contains all 1.3M images and 1,000 categories from ImageNet (ILSVRC 2012) [10,, but with the images resized to 32 32. We follow the experimental setup for optimization and data augmentation as in [8). This provides insights into how the batch size and intermediate normalization layer affect the empirical convergence speed and generalization performance of different parame- e
terizations. We use cross-validation and find that the optimal initial learning rate is 0.1 for GmP and 0.01 for all the other methods. Table 2 shows that GmP+IMN consistently achieves the best top-1 and top-5 validation accuracy for all batch sizes considered. Furthermore, the improvement of GmP+IMN over other methods gets larger the batch size increases, highlighting the robustness and scalability of GmP with large batch sizes. addition to achieving the best performance, Fig 4 shows that GmP+IMN (the green curve) also converges significantly faster than other compared methods: its top-5 validation accuracy converges within 25 epochs, which is 10 epochs earlier than the second best method BN. The ablation study GmP GmP+IMN shows that IMN significantly improves the performance of GmP, which is expected since it addresses the problem of covariate shifts between hidden layers. Notably, Wide ResNet (WRN 28-2) (68 trained with BN and a batch size of 500 only achieved 43. 43.08% top-1 validation accuracy as reported in [8) underperforming VGG-6 trained with GmP+IMN (43.62% as shown in Table This reveals the significance of better parameterizations: even a small non-residual network like VGG-6 with GmP+IMN can outperform large, wide residual networks like WRN 28-2.
4.2.3 ImageNet Classification with ResNet
We evaluate GmP with a large residual neural network, ResNet-18 (22. on the full ImageNet (ILSVRC 2012) dataset (10), which consists of 1,281,167 training images and 50.000 validation images that contain objects from 1,000 categories. The size of the images ranges from 75 x 56 to 4288 x 2848. We follow the experimental setup for optimization and data augmentation as in (22]. Specifically, we use the SGD optimizer with momentum 0.9 and reduce the learning rate by 0.1 epochs 30, 60 and 80. All models are trained for 90 epochs. We use
Table 3: Single-center-crop validation accuracy (%) for ResNet-18 trained on ImageNet (ILSVRC 2012).
Metric
Top
Top valid. ace.
WN+MBN BN
66.37 0.16
86.69 + 0.11
66.85 1 0.05
86.92 1 0.02
GmP+IMN
67.24 0.21
87.19 0.15
batch size of 256 for all methods. We use cross vaass-validation and find that the optimal initial learning

---

rate is 0.1 for all compared methods. We employ random horizontal flip, random resizing (256-480) with preserved aspect ratio, random crop (224). and color augmentation for data augmentation during training [1I. To address the internal covariate shift problem, we employ IMN for GmP. Following [56], MBN is used for WN. The result of SP is not reported as BN is used in ResNet by default. Table 3 reports the single-center-crop top-1 and top-5 validation accuracy for all compared methods, which shows that GmP+IMN significantly outperforms BN and WN+MBN in terms of both top-1 and top-5 validation accuracy. This demonstrates that our method is useful for improving large-scale residual network training.
5 Related Work
5.1 Neural Network Training Dynamics
show that wide networks evolve like linear models during training, while Neural Network Gaussian Processes (NNGPs) (49 35 provide insights into how wide neural networks generalize. [15] studies the evolution of the Hessian spectrum of neural networks during training.
Neural Tangent Kernels (NTKs) 25
36
3 investigates the curvatures of different principle components around the optimum of a regularized linear autoencoder. 11 32 70] analyze the training dynamics with natural gradient descent (NGD); see Appendix D for discussion of the connection between GmP and NGD. 45 investigates the effectiveness of stochastic gradient descents for neural network
training. 23 60 57
33 62 30 study the effects of Adam. BN and weight 26 study the failure cases in joint training of deep ensembles. investigate the training dynamics of distributed optimization with different client subsampling schemes. 48, 52
37 72
38
50
decay on training dynamics. 64
773
20 12 55 pay special attention to analyzing the ReLU activations in neural networks. contrast, our proposed characteristic activation analysis studies the evolution of the characteristic activation boundaries during stochastic optimization.
1
14
19
5.2 Neural Network Parameterization and Normalization
In addition SP, WN 56 and BN (24), there are many other neural network normalization and pa- rameterization techniques. Instead of normalizing the batch axis as in BN, Layer Normalization (LN) 2 operates on the feature axis, which is preferred for small batches or variable-length inputs such as text [61]. Other variants of BN include Switchable Normalization [4] and Instance Enhancement Batch Normalization [39] There are also normalization techniques designed for specific applications. For instance, Instance Normalization [59] and Group Normalization [66] are special cases of LN designed for CNNs, while Spectral Normalization 47 69 is specifically designed for GANs and transformers. All these methods operate in the Cartesian coordinate. The instability argument for BN also holds for these methods since they have a similar form to BN except that they normalize the input tensors along different axes. Besides, there is also a line of research on learning orthogonal rotation matrices for the weight vector [42 40 41 51 In contrast, our proposed Geometric Parameterization (GmP) overcomes the instability issue by operating in the hyperspherical coordinate.
Conclusion
We presented novel characteristic activation analysis for understanding the training dynamics of ReI.U networks, which exploits special activation values to characterize ReLU units. Using the proposed analysis, we identified a critical instability in common neural network parameterizations and normalization techniques that operate in the Cartesian coordinate. Addressing this, we proposed a new parameterization called Geometric Parameterization (GmP) which operates in the hyperspherical coordinate. We demonstrated the theoretical advantages of GmP for ReLU networks. We performed empirical evaluations to verify our analysis, showing its improved training stability, convergence speed and generalization performance on variety of tasks with different neural network architectures. More broadly, we believe that the general idea behind our proposed analysis of viewing neural network parameters from a different perspective has the potential to reveal new research directions for the study of neural network optimization and training dynamics. Limitations and potential future work directions are discussed in Appendix E
10

---

Acknowledgments and Disclosure of Funding
We thank Isaac Reid and Ross Viljoen for helpful feedback and discussions. WC acknowledges funding via a Cambridge Trust Scholarship (supported by the Cambridge Trust) and Cambridge University Engineering Department Studentship (under grant G105682 NMZR/089 supported by Huawei R&D UK). HG acknowledges generous support from Huawei R&D UK.
Part of this work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (www. cam. ac. uk), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant APP/T02119//1 and DiRAC funding from the Science and Technology Facilities Council (www. dirac ac ac.uk) uk).
References
[1] Shun-Ichi Amari Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276, 1998.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint (000::607.060 2016.
[3] Xuchan Bao, James Lucas, Sushant Sachdeva, and Roger B Grosse. Regularized linear au- toencoders recover the principal components, eventually. Advances in Neural Information Processing Systems, 33:6971 1-6981,222210 2020.
[4] Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of Data Science. Cambridge University Press, 2020.
[5] Wenlin Chen, Samuel Horváth, and Peter Richtárik. Optimal client sampling for federated learning. Transactions on Machine Learning Research, 2022.
[6] Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece. Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[7] Yae Jee Cho, Jianyu Wang. and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choiee selection strategies. arXiv preprint arXiv:2010.01243 2020.
[8] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprini /00.:0//77008/// 2017.
[9] Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: /51 1.07289, 2015.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[11] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[12] Yonatan Dukler, Quanquan Gu, and Guido Montúfar. Optimization theory for relu neural networks trained with normalization layers. In International conference on machine learning, pages 2751-2760. PMLR, 2020.
[13] Konstantin Eckle and Johannes Schmidt-Hieber. A comparison of deep networks with relu activation function and linear spline-type methods. Neural Networks, 110:232-244 2019.
[14] Dennis Maximilian Elbrächter, Julius Berner, and Philipp Grohs. How degenerate is the parametrization of neural networks with the relu activation function? Advances neural information processing systems, 32, 2019.
[15] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232-2241. PMLR, 2019.
[16] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
11

---

[17] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings, 2011.
[18] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
[19] Boris Hanin and David Roinick. Complexity of linear regions in deep networks. In International Conference on Machine Learning, pages 2596-2604. PMLR, 2019.
[20] Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems, 32, 2019.
[21] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[23] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems, 30, 2017.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
[25] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[26] Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela der Schaar. Joint training of deep ensembles fails due to learner collusion. Advances in Neural Information Processing Systems, 36, 2024.
[27] Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, and Shuicheng Yan. Deep learning with s-shaped rectified linear activation units. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[28] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[29] Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of length-direction decoupling in non-convex optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 806-815. PMLR, 2019.
[30] Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023.
[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[32] Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
[33] Antoine Labatie, Dominic Masters, Zach Faton-Rosen and Carlo Luschi. Proxy- noryy-nommalizing activations to match batch normalization while removing batch dependence. Advances in Neural Information Processing Systems, 34: 16990- 2021.
[34] Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learning via guided participant selection. In 15th (USENIX) Symposiun on Operating Systems Design and Implementation ((OSDI) 2/), pages 19-35, 2021.
[35] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sick-Dic Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 2017.
12

---

[36] Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
[37]
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. arXiv preprint 1rXiv:1910.07454 2019.
[38] Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33:1 11:1144444455555 14555, 2020.
[39] Senwei Liang, Zhongzhan Huang, Mingfu Liang, and Haizhao Yang. Instance enhancement batch normalization: An adaptive regulator of batch noise. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 34, pages 4819 1819-488 4827, 2020.
[40] Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg. Liam Paull, Li Xiong, Le Song, and Adrian Weller. Orthogonal over- peeererrratte training. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 7251-7260, 2021.
[41] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Hco. Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint 1eXiv:231100224: 2023.
[42]
Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. Advances in neural information processing systems, 30, 2017.
[43] Bing Luo, Wenli Xiao, Shigiang Wang, Jianwei Huang, and Leandros Tassiulas. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM IEEE, 2022.
Conference on Computer Communications, pages 1739-1746
[44] Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning- to-normalize via switchable normalization. arXiv preprint arXiv:1806. 10779, 2018.
[45] Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrizer learning. In International Conference on Machine Learning, pages 333444334. PMLR, 2018.
[46] Andrew Maas, Awni Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Atlanta, GA, 2013.
[47] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 1802.05957, 2018
[48] Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014.
[49] Radford M Neal. Bayesian learning for neural networks, volume 1 118. Springer Science & Business Media, 2012.
[50] Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. The Journal of Machine Learning Research, 21()):0017-10260, 2020.
[51] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng. Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320-79362. 2023.
[52] Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Nohl-Dickstein. On the expressive power of deep neural networks. In international conference on machine learning, pages 2847-2854. PMLR, 2017.
[53] Monica Ribero and Haris Vikalo. Communication--fficiiee federated learning via optimal client sampling. arXiv preprint arXiv: 15197, 2020.
[54] Simon Roburin, Yann Dubois de Mont-Marin, Andrei Bursuc, Renaud Marlet, Patrick Perez, and Mathieu Aubry. spherical analysis of adam with batch normalization, 2021.
[55] David Rolnick and Konrad Kording. Reverse-engineering deep relu networks. International conference machine learning, pages 8178-8187 PMLR, 2020.
13

---

[56] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances neural information processing systems, 29, 2016.
[57] Shibani Santurkar, Dimitris Tsipras. Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems, 31, 2018.
[58] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1e.
2014.
[59] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: /607.0802,, 2016.
[60] Twan Van Laarhoven. 1.2 regularization versus batch and weight normalization. arXiv preprint
urXiv:1706.05350, 2017.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics: Learning dynamies of normalized neural network using sgd and weight decay. Advances in Neural Information Processing Systems, 14:6380-6391, 2021.
[63] Geoffrey S Watson. Distributions on the circle and sphere. Journal of Applied Probability, 19(A):265-280, 1982.
[64] Andrew Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan Iliescu, Mikel Lujan, and Gavin Brown. To ensemble or not ensemble: When does end-to-end training fail? In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part III, pages 109-123. Springer, 2021.
[65] Andrew TA Wood. Simulation of the von mises fisher distribution. Communications in maaistics-simulatioon and computation, 23(1)1157-164, 1994.
[66] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018
[67] Ge Yang. Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097, 2021.
[68] Sergey Zagoruyko and Nikos Komodakis. 1605::605.07144 146, 2016.
Wide residual networks.
arXiv preprint
[69] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Jason Ramapuram, Dan Busbridge, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. SisigmaSreparam: Stable transformer training with spectral reparametrization, 2023.
[70] Guodong Zhang, James Martens, and Roger B Grosse. Fast convergence of natural gradient descent for perr-parametrrize neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[71] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810. 12281, 2018.
[72] Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321, 2019.
14

---

x
(a) A < 0,0 0. (b) 1
(c) 0 0.
(d) À < 0,0 n.
Figure 5: Visualization of characteristic activation boundaries (brown solid lines) and spatial locations (0) of ReLU unit z ReLU(u(OOx X) (blue solid lines) for inputs æ R. arrows denote active regions and red arrows denote inactive regions.
Green
22
31
|
Ø
A S
A
31
21
(a) À < 0, E (),
(b) A 0,0 E n, 2m). (c) > 0,0 e [0. z). (d) À < 0, E 2m).
Figure Visualization of characteristic activation boundaries (brown solid lines) and spatial locations A (f) of a ReLU unit = = ReLU(u(B)) x A) = ReLU(cos((P) (C))(()))) + sin( + X) for inputs
x E R.2. Green arrows denote active regions and red arrows denote inactive regions.
A
More Visualizations of ReLU Characteristic Activation Boundaries
Figs and respectively visualize the characteristic activation boundaries in the input spaces R and R under different conditions of the radius and angle parameters.
B Proof of Theorem 3.7
Proof. Since u has unit length, the change in the angular direction (u(0), u(O + +)) under infinitesi- mal perturbation is equal to the length 8s between the two points u(0) and u(O + e). By the generalized Pythagorean theorem, the are length given by
58
Σ mis8:e = ve Mae = =
(16)
where Mg is the metric tensor for the hyperspherical coordinate system. Therefore, the change the angular direction (u(0), u(O E)) under infinitesimal perturbation is simply the norm of with respect to the metric tensor M, for the hyperspherical coordinate system:
(u u(O +)) arccos (u(0) u(O e)) = ||
(17)
Hence, we need to work out a formula for calculating Mg.
Let [_,,... F.. IT Rd (n be input in the Cartesian coordinate system, where the metric tensor is the Kronecker delta m/ = diz- For the Geometric Parameterization of the unit
hypersphere 5=-1, we have
(.) cos(O,) II
sin)) k=1
0 < n,
(18)
n
U, (0)
sin(O
k=1
15

---

The metric tensor Mg for the Geometric Parameterization of 5 is the pullback of the Euclidean metric in I:":
Ou; ouj
Ou, Ou,
mab
maj 00, 00b
(19)
do, 00,
i=1
We discuss the diagonal b) and dff-oiaannal (a * b) elements separately:
of b: First, we note that
Dus 0 for 0 < i q. For g sis n, we have 7
Ou:
okg cos cos((x)) sin(Ok)
dig
00n(Ox)) cos(O;)
II sin(Os)
S <
00g
k=1
Oun
(0,) sin(Og)
(in(
a0,
(20)
Without loss of generality, we assume 0 < < <<< n. Then, we have
72
Ou: Ou,
1
Ou, Ou;
ob
00, 00,
00, 00,
1
Sra cos(Ou)
Ou:
cos(O,) cos(Os)
(O:)
(On(
sin (0x)
sin(Ok)
a0,
sin(Oa) sin(Os) k=1
i=b
k=1
cos
cos(
Σ cos(O:)
II (0s) k=l
Ou:
cos
sin 20x)
sin (0a)
00,
sin sin(Ob)
i=b
k=1
c0s)
b-1
sin((b) cos(O.)
II sin? (0ks
sin
cos(Os)
(0s)Op)
n-
cos (0s))
II sin 2 (0xc)
+
II sin (0xx) k=1
sin(Os)
sin(Os)
b=b+1
cos( (0s)
n
n II h=1
sin? (0c)
+
(0.)
II sin (0x)
sin 2
(0x))
sin(Oa) sin (0,)
k=1
1=b+1
(21)
On the other hand, by recursively collecting like terms and using the identity that sin sin)(0,) cos2 (0,,) = 1, Vg. we have
(0:)
II
sin ((n))
+
II sin (0x)
i=0+1
k=1
2
1-1
n-2 2
Σ
cos (0.)
II kml
sin 2(xx)
II
sin 2 (0ww) (cos?(On. + sin?(0n-1))
Σ 2 ((s)
II
2 (0%)
II sin 2(xx)
(22)
Amml
A=
n-3 3
n 3
Σ (0,))
II sin (0x)
+
((n)
b=++1
sin 2 (0k).
This shows that mab 0 for + h and hence Mg is diagonal matrix.
16

---

b: Following similar argument as above, we can obtain the diagonal elements of Mg:
Oui
Ou: 2
Mpa
00a
==
cos)
(0a) sin sin²(On)
sin (0x) +
(0,)
II (Ox)
sin (in)
sin (0n)
k=1
ima
k=1
k=1
cos²(Oa)
sin (0x)
Σ cos? (0:)
II (On)
+ II sin (0+))
sin (fa)
(23)
On the other hand, by Eq 22 we have
6-1
n-1 II
cos ( (0.)
II
sin
sOn)
sin 3 (0x)
II km
sin (0x).
(24)
Hence, it follows that
cos (Ba)
Moa
II
sin? (0x)
s
sin
sin? 2(xx)
k=1
k=1
-Oin)(0a)
sin (0x) + cos)((.))
sin (0x)
k=
k=1
(25)
a-1
(sinn((0)) + cos (os))
II
sin (0k))
a-1
II
sin (0x),
2 nn
and mi1
Therefore, the metric tensor for the hyperspherical coordinate a diagonal matrix
sin (01)
sin (01) sin 3
Mg
(26)
sin (0.)
II sin (8,))
Finally, the change in the angular direction of a unit vector u(O) under a perturbation to 0 given by the norm of e with respect to the tensor matrix Mg:
VET
maaf
sin (i,2(0,) E7.
(27)
Furthermore, since -1 sin(O,) < 1, follows that S ma, Therefore, Eq (27p is bounded by
11;=1 sin)((,) S for all i.
n-1 e of
cf +
sin (0,,
2 = ||e||2.
1=2
This completes the proof.
17

---

C Detailed Experimental Setups
C.1 UCI Regression with MLP
We train MLP with one hidden layer and 100 hidden units for 10 different random 80/20 train/test splits. We use the Adam optimizer [28] with full-batch training. We use cross-validation to select the learning rate for each compared method from the set {0.001, 0.003, 0.01, 0.03, 0.1, 0.3). We find that the optimal initial learning rate is 0.1 for GmP and 0.01 for all the other compared methods. We report test root mean squared error (RMSE). All models are trained on single NVIDIA GeForce RTX 2080 Ti.
C.2 ImageNet Classification with ResNet
We train ResNet-18 18 on the ImageNet (ILSVRC 2012) dataset [0] which consists of 1, 1,281,167
training images and 50,000 validation images that contain objects from 1,000 categories. The size of the images ranges from 75 56 4288 x 2848. We follow the experimental setup for optimization and data augmentation as in [22] We use the SGD optimizer with momentum 0.9, which turns out to better than Adam for image classification tasks [22]. We reduce the learning rate by 0.1 at epochs 30, 60 and 80. All models are trained for 90 epochs. We use batch size of 256 for all methods. We use cross-validation to select the learning rate for each compared method from the set {0.001, 0.003, 0.01, 0.03, 0.1, 0.3). We find that the optimal initial learning rate is 0.1 for all compared methods. We employ random horizontal flip, random resizing (256-480) with preserved aspect ratio, random crop (224), and color augmentation for data augmentation during training (3I). To address the internal covariate shift problem, we employ Input Mean Normalization (IMN) for GmP. Following [56). Mean-only Batch Normalization (MBN) is used for WN. We report single- center-crop top-1 and top-5 validation accuracy. The result of SP is not reported as BN is the default normalization for ResNet. All models are trained on single NVIDIA A A100 (80GB).
C.3 Ablation Study: ImageNet32 Classification with VGG
To maintain a manageable computational cost for the ablation study, we train a VGG-6 588 on ImageNet32 [8), which contains all 1.3M images and 1,000 categories from ImageNet (ILSVRC 2012) [0), but with images resized to 32 x 32. We follow the experimental setup for optimization and data augmentation as in [ We use the SGD optimizer with momentum 0.9, which turns out to be better than Adam for image classification tasks [22]. We reduce the learning rate by 0.1 at epochs 30, 60 and 80. All models are trained for 90 epochs. We train the model using three common batch sizes (256, 512, 1024] for all methods. We use cross-validation to select the learning rate for each compared method from the set {0.001, 0.003, 0.01, 0.03, 0.1, 0.3). We find that the optimal initial learning rate is 0.1 for GmP and 0.01 all the other methods. We employ random horizontal flips for data augmentation during training. We conduct an ablation study to explore the effects of Input Mean Normalization (IMN) for GmP and Mean-only Batch Normalization (MBN) for WN in deep networks. We report top-1 and top-5 validation accuracy. All models are trained on a single NVIDIA GeForce RTX 2080 Ti.
D Connections to Natural Gradient Descent
The parameter space of neural network can be thought of as Riemannian manifold M, for which the neural network parameterization specifies the coordinate for M. In this paper, we reveal that standard parameterization is vulnerable to small perturbations of the parameters (e.g.. SGD noise) whereas our proposed Geometric Parameterization is much more robust against perturbations.
One thing to note is that the full version of natural gradient descent is invariant to neural network pa- rameterizations since is defined in an abstract form without any specific pprrameterzzationcoordnaa In the abstract form, parameters and (X, Y) is an abstract data point. The fisher matrix F(0) = Ex(xx))|VIC (VI(0) [V(())V(())] is the metric tensor for M, where p(X, Y) is the abstract data distribution. A single step of the full version of natural gradient descent is given by
let the loss function be l(0)
log p( X, 0) where 0 are the abstract
4141 t 0e T) Expo, (0(00)-'/((0
(28)
18

---

where h M + R is any differentiable function and Th is the learning rate. Note that the exponential map Expp, maps the update F(0.) Vh((,)) from the tangent space To, M back the manifold M, However, it is usually
which results in an exact update invariant to the parameterization/ooordaaa infeasible to calculate the exponential map since requires solving linear system of size dim( (M). which is the total number of parameters in the neural network. In practice, the most commonly used version of natural gradient descent is its first-order approximation given by
O 02 - " F(O)) VV((0))
(29)
which is second-order optimization method that is invariant up to first-order transformation of the parameterization//oordiee Since our proposed Geometric Parametcrization is a nonlinear transformation of standard parameterization, it will still behave differently under the first order approximation the full version of natural gradient descent.
E Limitations and Future Work
Beyond ReLU activation. This work analyzed ReLU networks due their wide adoption. However, the proposed characteristic activation analysis is general and can be directly applied to other ReLU- like activation functions such as Leaky ReLU [46], ELU [9] and SReLU [27]. because the definition of spatial locations is essentially any breakpoints (i.e., the sources of non- con-linearity) in a piecewise activation function. The analysis of smooth activations such as Sigmoid and Tanh is left for future work.
Beyond singlehhidden-laye networks. This work only performed characteristic activation analysis for single-hidden-layer ReLU networks and proposed a practical workaround to address the covariate shift issue between hidden layers by using input mean normalization (IMN), For future work, this analysis needs to be generalized to examine training dynamics in muliple-hiddenn-ayer networks to understand the theoretical behaviors of deep networks. One potential difficulty is that the characteristic activation boundaries of multiple-hideen- input space, which are less straightforward to analyze. possible solution would be consider how the assignment of each data point to the partition evolves during training, similar to how we track the characteristic activation boundaries.
networks become piecewise linear partitions of the
Sparse activation and large-width limiting behavior. Interestingly, the product structure of Geo- metric Parameterization (GmP) as shown in Eq (11) indicates that it could lead to sparse activations or even vanished gradients. Empirically, we find that the activations and gradients of GmP will not vanish during training if the GmP parameters initialized from the von Mises-Fisher distribution (i.e., uniformly distributed on the hypersphere) as discussed in Remark (.5] For future work, it would be interesting to investigate the farge-width limiting behavior of GmP (e.g., similar to NTK 25 or uP (77) to provide theoretical guarantees of the sparsity patterns in GmP activations.
Efficient representation of hidden neurons. Directly representing the hidden neurons rather than the parameters in the hyperspherical coordinate could he a more efficient representation. However, unclear how train such neural networks since gradient-based based optimization is no longer applicable. One approach would be to use rejection sampling to allocate the hidden neurons on the hypersphere, but it is inefficient and suffers from curse of dimensionality. For future work, it would be interesting to find a more efficient learning algorithm that directly allocates hidden neurons on the hypersphere.
Combining GmP with existing techniques. We leave the investigation of the theoretical properties and empirical performance of combinations of GmP and existing normalization (c.g.. BN or LN) and regularization techniques (c.g., weight decay) for future work. It would also be interesting to examine the performance of GmP with other neural network architectures (c.g., transformers) on large datasets from different domains (e.g., NLP) under future work.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made in the paper, which match the theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations and potential future work directions are discussed in Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
The proof of each Proposition is in the main paper following the statement.
The proof of Theorem 3.7 can be found in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The implementation details of GmP are fully disclosed in Section 3
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

reproducible, they Answer: [Yes]
Justification: Our code is available at https://github. com/Wenlin-Ches
geometric- www.rriic-pprrmmeeriaattete
All datasets used are publicly available common ML benchmarks and are properly cited in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm cc/
public/guide Saaiii//giies/CCCoo httss//oodSummissioooooo..
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pubb httppss.ccpublli/guuueeee Sttpeee/s/oooSSeeeiseeiiiiii
ssioiiooiiiiioiociicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental setting/details are fully disclosed in Section 4 and Appendix C Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All main experimental results in Section 4.2 are accompanied by error bars or confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are disclosed in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neuripp: http:///wwwrpp.oooo//oc/cccccaaa
Answer: [Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is a technical ML paper focusing on the theoretical properties of the training dynamics of ReLU networks. There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This is technical ML paper focusing the theoretical properties of the training dynamics of ReLU networks. The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets used are publicly available common ML benchmarks and properly cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "PB2oi0iFg8", "reviewer": "Reviewer_D8e2", "summary": "In standard neural networks each neuron, and activation function g performs the following operation on the input $x\\in\\mathbb{R}^{n}$:\n$$z=g\\left(w^{t}x+b\\right)$$\nWhen usually g in the ReLU activation. \n\nIn this work the authors identify a critical instability in many common NN settings, which theoretically destabilizes the evolution of the characteristic boundaries, and empirically impedes fast convergence and hurts generalization performance. To address this, they introduce a novel NN parameterization, named Geometric Parameterization (GmP) which operates in the hyper spherical coordinate. The authors show that this parameterization, both theoretically and empirically, improves optimization stability, convergence speed, and generalization performance.", "questions": "I have few technical questions regarding section B – proof of theorem 3.7:\n\t\n1. It is not clear to me why equation 16 is true. I would suggest adding an explanation or a reference to one.\n\t\n2. The analyze of the case where a≠b. It isn't immediately clear from equation 18 why \" sum of terms that are either zero or with alternating signs\". Maybe it is simple calculus, but I suggest adding additional step to show this claim.", "limitations": "Yes, the authors do address the limitations. The main limitation is the input mean normalization that being address in appendix E.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The main strength of this paper (besides the actual methods introduced) is that the results are clear and concise. As a reader, it is relatively easy for me, based on the empirical results, to understand how these methods perform compared to other common methods. I will now explain each dimension.\n\n**Originality**: I think that this paper introduces novel ideas into neural networks (NN). The use of a different coordinate system is not something that I have seen before, and it opens the door to many interesting new approaches to NN.\n\n**Quality**: There is not much to say here, as I feel this is a quality paper. All claims and lemmas are proved as needed.\n\n**Clarity**: The clarity of this paper is good. This mainly stems from the fact that the paper separates the explanation of the problem from the explanation of the solution. Both parts are described using theoretical explanations and empirical/graphical examples.\n\n**Significance**: It is hard to estimate significance, as usually only time will tell how significant the research is, but I feel that this paper will be significant. I believe that the main significance lies not necessarily in the methods introduced, but rather in the idea behind them. The idea of a different perspective on the learned parameters has the potential to lead to more unique and potentially beneficial views.", "weaknesses": "I will start with a somewhat nit-picking weakness. The paper uses $\\mathcal{B}$ and ϕ in two different contexts. ϕ is used in Definition 2.2 as a vector and in Definition 3.1 as a function. $\\mathcal{B}$ is used in Definition 2.2 as a hyperplane and in Definition 3.2 as a function. This isn't accidental, as there is a strong connection between the definitions, but I believe that using slight differences in the notations would be better.\n\nAnother weakness is that the article hasn't provided an empirical case where GmP isn't the best option. It is possible that GmP is always the best choice (and if so, great), but usually, there are cases where new techniques aren't the best options. It would have been interesting to see a scenario where GmP isn't the best option and (perhaps under future work) understand why.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our work! We are encouraged by your positive comments on the originality, quality, clarity, and significance of our paper. Below, we address your concerns point by point.\n\n> Explain the notations $\\mathcal{B}$ and $\\boldsymbol{\\phi}$ in different contexts.\n\nWe would like to clarify that $\\boldsymbol{\\phi}$ is a vector and $\\mathcal{B}$ is a hyperplane throughout the paper. Taking $\\phi$ as an example, the vector $\\boldsymbol{\\phi}$ in Definitions 2.1 and 3.1 refers to the same vector in different coordinate systems (i.e., Cartesian and hyperspherical coordinate systems, respectively). We write $\\boldsymbol{\\phi}$ as a function of $\\lambda$ and $\\boldsymbol{\\theta}$ in Definition 3.1 because we want to emphasize that we have changed to a different coordinate system. In the camera-ready version, we will add a sentence to clarify that Definitions 2.1 and 3.1 define the same vector $\\boldsymbol{\\phi}$ under different coordinate systems, and we will explicitly write $\\boldsymbol{\\phi}(\\mathbf{w},b)$ as a function of $\\mathbf{w}$ and $b$ in Definition 2.1 in order to make the contrast between the two definitions clearer. We will also similarly clarify $\\mathcal{B}$ in the camera-ready version.\n\n> Discuss empirical cases where GmP isn't the best option and (perhaps under future work) understand why.\n\nWe find that GmP is almost always the best option for MLPs in all cases we have seen. For CNNs, we observe that the benefit of GmP becomes less significant as the network gets more and more overparameterized. We suspect that GmP may not be the best option when the network gets extremely overparameterized. We agree with the reviewer that it would be interesting to understand this behavior under future work, which we are currently working on. In addition, as discussed in Appendix E, it would also be interesting to see whether GmP is the best option for Transformers under future work. \n\n> Explain why Equation 16 is true.\n\nFirstly, note that arc length $=$ angle $\\times$ radius. In our case, since the radius of a unit vector is one, the angle between the two unit vectors $\\mathbf{u}(\\boldsymbol{\\theta})$ and $\\mathbf{u}(\\boldsymbol{\\theta}+\\boldsymbol{\\varepsilon})$ is equal to the arc length $\\delta s$ between the two points. By the generalized Pythagorean theorem, the arc length between two points with a small change $\\delta\\boldsymbol{\\theta}$ is given by:\n$$\\delta s=\\sqrt{\\sum_{i,j}m_{ij}\\delta\\theta_i\\delta\\theta_j}=\\lVert \\delta\\boldsymbol{\\theta}\\rVert_M=\\lVert \\boldsymbol{\\varepsilon}\\rVert_M$$\nIn the camera-ready version, we will add this explanation to the proof of Theorem 3.7 in Appendix B.\n\n> Explain why the sum of terms in Equation 18 cancels out in the case where $a\\not=b$.\n\nWe first write Equation 17 in a more compact form:\n$$u_i(\\boldsymbol{\\theta})=\\cos\\theta_i\\prod_{k=1}^{i-1}\\sin\\theta_k,\\quad 0<i<n$$\n$$u_n(\\boldsymbol{\\theta})=\\prod_{k=1}^{n-1}\\sin\\theta_k$$\nIt can be seen that $\\frac{\\partial u_i}{\\partial\\theta_q}=0$ for $0<i<q$. For $q\\leq i \\leq n$, we have\n$$\\frac{\\partial u_i}{\\partial\\theta_q}=-\\delta_{iq}\\prod_{k=1}^{i}\\sin\\theta_k+\\cos\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin\\theta_k\\right)\\left(\\sum_{k=1}^{i-1}\\frac{\\delta_{kq}\\cos\\theta_k}{\\sin\\theta_k}\\right),\\quad q\\leq i<n$$\n$$\\frac{\\partial u_n}{\\partial\\theta_q}=\\frac{\\cos\\theta_q}{\\sin\\theta_q}\\prod_{k=1}^{n-1}\\sin\\theta_k$$\nWithout loss of generality, we assume $a<b$. From Equation 18, we have\n$$m_{ab}=\\sum_{i=1}^n\\frac{\\partial u_i}{\\partial\\theta_a}\\frac{\\partial u_i}{\\partial\\theta_b}=\\sum_{i=b}^n\\frac{\\partial u_i}{\\partial\\theta_a}\\frac{\\partial u_i}{\\partial\\theta_b}=\\sum_{i=b}^{n-1}\\cos\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin\\theta_k\\right)\\left(\\sum_{k=1}^{i-1}\\frac{\\delta_{ka}\\cos\\theta_k}{\\sin\\theta_k}\\right)\\frac{\\partial u_i}{\\partial\\theta_b}+\\frac{\\cos\\theta_a\\cos\\theta_b}{\\sin\\theta_a\\sin\\theta_b}\\prod_{k=1}^{n-1}\\sin^2\\theta_k$$\n$$=\\frac{\\cos\\theta_a}{\\sin\\theta_a}\\left(\\sum_{i=b}^{n-1}\\cos\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin\\theta_k\\right)\\frac{\\partial u_i}{\\partial\\theta_b}+\\frac{\\cos\\theta_b}{\\sin\\theta_b}\\prod_{k=1}^{n-1}\\sin^2\\theta_k\\right)$$\n$$=\\frac{\\cos\\theta_a}{\\sin\\theta_a}\\left(-\\sin\\theta_b\\cos\\theta_b\\prod_{k=1}^{b-1}\\sin^2\\theta_k+\\frac{\\cos\\theta_b}{\\sin\\theta_b}\\sum_{i=b+1}^{n-1}\\cos^2\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin^2\\theta_k\\right)+\\frac{\\cos\\theta_b}{\\sin\\theta_b}\\prod_{k=1}^{n-1}\\sin^2\\theta_k\\right)$$\n$$=\\frac{\\cos\\theta_a\\cos\\theta_b}{\\sin\\theta_a\\sin\\theta_b}\\left(-\\prod_{k=1}^b\\sin^2\\theta_k+\\sum_{i=b+1}^{n-1}\\cos^2\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin^2\\theta_k\\right)+\\prod_{k=1}^{n-1}\\sin^2\\theta_k\\right)$$\nOn the other hand, by recursively collecting like terms and applying $\\sin^2\\theta_q+\\cos^2\\theta_q=1$, we have\n$$\\sum_{i=b+1}^{n-1}\\cos^2\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin^2\\theta_k\\right)+\\prod_{k=1}^{n-1}\\sin^2\\theta_k=\\sum_{i=b+1}^{n-2}\\cos^2\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin^2\\theta_k\\right)+\\prod_{k=1}^{n-2}\\sin^2\\theta_k$$\n$$=\\sum_{i=b+1}^{n-3}\\cos^2\\theta_i\\left(\\prod_{k=1}^{i-1}\\sin^2\\theta_k\\right)+\\prod_{k=1}^{n-3}\\sin^2\\theta_k=\\cdots=\\prod_{k=1}^b\\sin^2\\theta_k$$\nThis shows that $m_{ab}=0$ for $a\\not=b$. While this derivation only involves collecting like terms, we realized that it might not be that straightforward as it involves many steps. We thank the reviewer for pointing this out. In the camera-ready version, we will add this derivation to the proof of Theorem 3.7 in Appendix B.\n\nWe hope that this has sufficiently addressed all your concerns. We thank the reviewer again for their insightful questions and for helping us improve this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cfBPMRBLyw", "reviewer": "Reviewer_ywDS", "summary": "Summary.\nAuthors consider training  neural nets with Adam after a change of coordinate to spherical coordinate system. They show that in spherical coordinate system the direction of the half spaces which specifies the activeness of the Relus in stable with respect to small changes in the angles, hence the optimization is supposed to be easier. The perform some experiments to see the advantage of using this coordinate system for optimization in practice.", "questions": "-optimizing in the normal coordinate system with relu activation is known to create sparse features, which can help with generalization, and I am not sure if happens if one optimizes in spherical coordinates. Does the authors see sparsity in their training as well, and if not how do they assess this fact with their claim that spherical coordinate has better generalization?\n\n-In particular authors show in experiments that they can pick a larger learning rate in spherical coordinates, but this fact solely cannot be an indicator of better generalization (even if two algorithms A and B run in the same coordinate system and A picks larger learning rate than B it is still not clear which one generalize better). Can the authors further elaborate on their claim about superiority of spherical coordinates for the generalization of the final network?", "limitations": "No theoretical justification for superiority of their method for optimization/generalization.", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "This work goes against the conventional practive in deep learning which is running first order method in the standard coordinate system, rather it change to spherical coordinates. They further discuss some advantages of this system for stability of the stability of the activation regions. They conduct various experiments to show this advantage empirically. I think this is an interesting idea; the contribution of the current paper should be assessed in their experiments section, where they show the advantage of this change of coordinates in training. However in the theoretical part there are major concerns that I will mention in the following.", "weaknesses": "First the proof of Theorem 3.7. is wrong; I think what the authors meant in the argument of the Theorem is $\\|u(\\theta) - u(\\theta + \\epsilon)\\|$ rather than their dot product (the dot product claim is obviously wrong since if epsilon goes to zero right hand side goes to zero while left hand side is constant). even with $\\|u(\\theta) - u(\\theta + \\epsilon)\\|$ instead, the proof in appendix is wrong; the correct formula of length given metric $M_\\theta$ on a manifold is: \\|u(\\theta) - u(\\theta + \\epsilon)\\| = \\int_{0}^1 \\sqrt{\\epsilon^\\top M_{\\theta + t\\epsilon} \\epsilon}. The proof can be corrected by estimataing $M_{\\theta + t\\epsilon}$ with $M_\\theta$. Though the final argument is expected to be true, perhaps with different constants. \n\nSecond, the paper is highly overclaimed about tjhe theoretical results. The theoretical claim of the paper is only that the normal vector of half space change smoothly if one parameterize in spherical coordinates despite normal coordinates, independent of the training algorithm, while the authors are claiming at the beginning that they analyze the behavior of training algorithms in this case and analyze its advantage over normal coordinates which is not the case. \n\nminor comments:\nLine 155 -> typo IMH instead of IMN\n\nFigure 2, (d), ..., (g): how is the fact that activations are more spread out related to the smooth evolution of spatial locations? (the plot with the yellow lines), and these are in 1 and 2D as far as I see, do you have a version of these plots in higher dim? (since that is the authors claim)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback! We believe that your major concern regarding Theorem 3.7 is due to a misunderstanding of our notation. Below, we first address your major concern and then respond to your other comments point by point.\n\n> Clarification of the argument of Theorem 3.7\n\nWe believe that there is a misunderstanding of our notation. We would like to clarify that the argument of Theorem 3.7 is regarding the **angle** between the two unit vectors $\\mathbf{u}(\\boldsymbol{\\theta})$ and $\\mathbf{u}(\\boldsymbol{\\theta}+\\boldsymbol{\\varepsilon})$, rather than the dot product or distance between those two vectors. Therefore, what Theorem 3.7 states is that the angle between $\\mathbf{u}(\\boldsymbol{\\theta})$ and $\\mathbf{u}(\\boldsymbol{\\theta}+\\boldsymbol{\\varepsilon})$ is bounded by $\\lVert\\boldsymbol{\\varepsilon}\\rVert_2$, which goes to zero as $\\boldsymbol{\\varepsilon}\\to 0$. Appendix B along with our response to Reviewer D8e2 provides a detailed proof using techniques from differential geometry, which shows that Theorem 3.7 is indeed correct.\n\nWe regret that our notation $\\left<\\mathbf{u}(\\boldsymbol{\\theta}),\\mathbf{u}(\\boldsymbol{\\theta}+\\boldsymbol{\\varepsilon})\\right>$ could be misleading as we realized that it can be used to denote dot product in some cases. We thank the reviewer for pointing this out. In the camera-ready version, we will explicitly write down the definition of our notation as follows to avoid any confusion or ambiguity:\n$$\\left<\\mathbf{u}(\\boldsymbol{\\theta}),\\mathbf{u}(\\boldsymbol{\\theta}+\\boldsymbol{\\varepsilon})\\right>\\equiv\\text{arccos}(\\mathbf{u}(\\boldsymbol{\\theta})^\\text{T}\\mathbf{u}(\\boldsymbol{\\theta}+\\boldsymbol{\\varepsilon}))=\\sqrt{\\varepsilon_1^2+\\sum_{i=2}^{n-1}\\left(\\prod_{j=1}^{i-1}\\sin^2(\\theta_j)\\right)\\varepsilon_i^2}\\leq\\lVert\\boldsymbol{\\varepsilon}\\rVert_2$$\n\n> Clarification of claims about theoretical results\n\nBelow, we clarify the contributions of our work and our corresponding claims. \n1. The first contribution of this work is a novel characteristic activation analysis for ReLU networks, which reveals that traditional NN parameterizations and normalizations in the Cartesian coordinate destablize the evolution of the characteristic boundaries during training. This contribution correpsonds to our claim about analyzing the behavior of traditional NN parameterizations and normalizations during training. We do not claim analyzing NN training algorithms.\n2. The second contribution of this work is the novel geometric parameterization (GmP), which makes a change of coordinate to train NNs in the hyperspherical coordinate. We provide both theoretical guarantees and empirical results to showcase the advantage of GmP in terms of stability, convergence speed, and generalization. This contribution corresponds to our claim about the advantage of training NNs in the hyperspherical coordinate.\n\nWe will make our claims more precise as above in the camera-ready version.\n\n> How is the fact that activations are more spread out related to the smooth evolution of spatial locations? \n\nThe more spread-out activations of GmP is one of the positive consequences of smooth evolution of the spatial locations. Figures 2g shows that smooth evolution of spatial locations in GmP allows small and consistent changes to be accumulated, resulting in more spread-out activations eventually. In contrast, Figure 2c shows that the change of spatial locations under other parameterizations can be up to $2^{16}$. Such abrupt and huge changes of spatial locations make the evolution of the spatial locations inconsistent. Consequently, it is much harder for optimizers to allocate those activations to the suitable locations during training. In addition, we would like to point out that some of the activations are even allocated to the regions that are far away from the data region and cannot be seen in Figure 2d-2f, and those activations become completely useless. We will add a discussion on this in the camera-ready version.\n\n> Do you have a version of these plots in higher dim?\n\nIt is very hard to visualize the trajectories of characteristic points/boundaries in more than 2D. However, our Theorem 3.7 guarantees smooth evoluation under GmP in any dimensions in theory. We use the plots in 1D and 2D act as proofs of concepts in practice, since those are the only cases that can be visualized.\n\n> Do the authors see sparsity in their training as well?\n\nThanks for this insightful question! Yes, empirically, we find that GmP also creates sparse features. This can be explained by Equation 11, which shows that GmP can easily create sparse features due to the product structure. We conjecture that this is one of the reasons why GmP provides better generalization and are currently working on a follow-up project to investigate this sparsity behavior and its connection to generalization. We will add a discussion on this in the camera-ready version.\n\n> A larger learning rate in spherical coordinates solely cannot be an indicator of better generalization. Can the authors further elaborate on their claim about superiority of spherical coordinates for the generalization of the final network?\n\nWe do not claim a larger learning rate is an indicator of better generalization. What we claim is that a larger learning rate makes convergence faster, as shown in Figure 4. Our claim about GmP's better generalization is empirically supported by GmP's better test performance on a variety of ML tasks with different NN architectures. As mentioned above, we are currently investigating why GmP results in better generalization from a theoretical perspective as a follow-up work (e.g., sparsity and large width limiting behavior like NTK and $\\mu$P). We will include this clarification in the camera-ready version.\n\nWe thank the reviewer again for their insightful questions and for helping us improve this work. We hope that our response has resolved their concerns and that the reviewer will reconsider their rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCJ41TNteB", "reviewer": "Reviewer_x1Bf", "summary": "This paper introduces a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. The authors figure out the instability in common neural network parameterization and normalization during stochastic optimization. To address this, the authors propose Geometric Parameterization (GmP), which parameterizes the unit vectors by hyperspherical coordinate representation rather than scaling. The authors conduct experiments to verify the effectiveness of the method.", "questions": "Have the authors explored how GmP performs on other activation functions? Does it also perform better? \n\nSuppose we use GmP on the hidden neurons rather than the weights, like scaling operations projecting $x$ onto a hypersphere. In other words, we use $\\theta_1,\\cdots,\\theta_n$ to represent $x$ by spherical coordinate system. Can the authors give some analysis theoretically or experimentally?", "limitations": "N/A", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper proposes a parameterization method based on spherical coordinate transformation. It is novel to analyze neural networks from a geometric perspective, and the method solves the problem of  instability during stochastic optimization. \n\nBesides, this paper is easy to follow writhing, generally well presented.", "weaknesses": "Lack discussions on other activation functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our paper! We will address your comments point by point below.\n\n> Discuss GmP on other activation functions.\n\nOur characteristic activation analysis is a very general idea that can be applied to a broader family of activation functions beyond ReLU (e.g., Leaky ReLU, ELU, SReLU, etc) with exactly the same argument. This is because our definition of spatial locations is essentially any breakpoints in a piecewise activation function, since those breakpoints are the sources of nonlinearity. We will add this explanation to the camera-ready version. We leave the characteristic activation analysis for smooth activations for future work since we specifically focus on ReLU-like activations as indicated in the title of our paper.\n\n> What if we use GmP on the hidden neurons rather than the weights?\n\nThank you for this insightful question! We did consider directly representing the hidden neurons in the hyperspherical coordinate, since this is conceptually a more efficient representation. However, we find that it is very difficult to train such neural networks since gradient-based optimization is no longer applicable. We tried to use rejection sampling to allocate the hidden neurons on the hypersphere, but it is inefficient and suffers from curse of dimensionality in high dimensional space. With that being said, we do think that this could potentially be a promising direction and are currently working on a more efficient learning algorithm for allocating hidden neurons on the hypersphere as a follow-up project. We will add a disucssion of this in the camera-ready version.\n\nWe thank the reviewer again for their encouraging and insightful comments which have helped us improve this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kJdYHDUNUz", "reviewer": "Reviewer_FLhQ", "summary": "This paper identifies and attempts to solve the problem of unstable activation boundaries in ReLU neural networks. The authors define a relu unit in terms of its activation boundary which is the set of inputs that cause the preactivation to be zero. The point on this hyperplane closest to the origin (termed a CAB) is used to track each neuron's activation boundary. The authors show that standard parameterization, weight norm and batch norm all suffer from an instability in the direction of the activation boundary when the weights are small since the direction of the weight can change sign. To address this, they propose to operate in hyperspherical coordinates where the angles $\\theta_1, ... , \\theta_{N-1}$ and radius $\\lambda$ are used to define the hyperplane instead. Taking gradient steps in this geometric parameterization maintains bounded changes in the angles under small learning rate due to a nice metric property of this coordinate system. The", "questions": "1. Some of the stability analysis is performed under the worst case assumption that $\\epsilon \\propto w$. Wouldn't this be unlikely to happen for high dimensional inputs? The main instability is caused when the vectors $w$ change sign/direction under a single step of GD. Could this issue also be solved by initializing the weights in SP to have larger variance relative to the learning rate? \n2. Have the authors considered what the structure of the gradients look like in this parameterization? Naively it seems like some of the gradient entries would have very small magnitude in high dimension since they involve products of bounded trigonometric functions. Is there a \"limiting\" large width description in this parameterization like for NTK parameterization or $\\mu$P?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper identifies an issue with ReLU neurons that can lead to a training instability at large learning rates and attempts to solve the problem with an elegant parameterization motivated by switching to a spherical coordinate system. The computational cost is of the same order as the original matrix computation, so it is still efficient to compute, unlike other approaches such as natural gradient descent. Next, the parameterization is size independent which is a nice property not enjoyed by SP networks. Further, the authors provide many experiments which show that the locations of the activation boundaries are more stable under their proposed parameterization, leading to a speed up in optimization.", "weaknesses": "There are some assumptions in the analysis that I was unsure about (such as worst case direction of the perturbations. If instead perturbations were randomly oriented with respect to $w$, does it change the conclusions?). The authors also mention that other normalization solutions suffer from similar problems (like Layernorm etc) but this would be useful to demonstrate, perhaps in the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments on our paper! We will address your comments point by point below.\n\n> It would be useful to demonstrate that other normalization solutions suffer from similar problems (like Layernorm etc), perhaps in the Appendix.\n\nThank you for your suggestion. In the camera-ready version, we will add derivations for these to the Appendix. Here, we take Layernorm as an example and illustrate why it suffers from similar problems. The key of the proof for Layernorm is to realize that the only difference bewteen Layernorm and BN is that they normalize the input tensors along different axes (i.e., BN normalizes inputs across the batch axis and Layernorm normalizes inputs across the feature axis). Therefore, the proof for Layernorm is the same as the proof for BN, except that the expectation and variance operators are now computed along the feature axis.\n\n> If perturbations $\\varepsilon$ were randomly oriented with respect to $w$, does it change the conclusions of the analysis? What about high dimensional inputs?\n\nNo, the conclusion does not change. This is because the angle between $\\varepsilon$ and $w$ can only be between 0 and 180 degrees in any dimensional space. Therefore, even if $\\varepsilon$ was randomly oriented with respect to $w$ in high dimensional space, there would still be a 50% chance that the angle between them is greater than 90 degrees, in which case the gradient update will be unstable as the direction of the characteristic boundaries/spatial locations is significantly changed.\n\n> Could this instability issue also be solved by initializing the weights in SP to have larger variance relative to the learning rate?\n\nUnfortunately, no. This is because SP is extremely sensitive to the variance of the initialization distribution as discussed in Remark 3.5 of the paper. If we did not strictly follow the variance as suggested in Glorot/He initialization and used a larger variance instead, the final performance of SP would be very poor.\n\n> Have the authors considered what the structure of the gradients look like in this parameterization?\n\nThank you for this insightful question! Empirically, we find that the structure of the activations and their gradients under GmP do exhibit some sparsity pattern, and we conjecture that this is one of the reasons why GmP has good generalization. Interestingly, even with the product structure and some sparsity pattern, we find that empirically the activations and gradients will not vanish during training if we initialize the GmP parameters from the von Mises–Fisher distribution (i.e., uniformly distributed on the hypersphere) as discussed in Remark 3.5. We conjecture that GmP may have a large width limiting behavior similar to NTK or $\\mu$P, which we are currently investigating in a follow-up project. We will add a discussion on this to the camera-ready version.\n\nWe thank the reviewer again for their positive and insightful comments which have helped us improve this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable and insightful comments on our paper and for helping us improve this work. We appreciate that three reviewers support the acceptance of our paper and highlight the novelty, quality, clarity and significance of our proposed characteristic activation analysis and geometric parameterization.

We have addressed each reviewer's concerns separately below their respective review. In particular, we believe that Reviewer ywDS's major concern regarding Theorem 3.7 is due to a misunderstanding of our notation, which we have clarified in our rebuttal to their review. We hope that our responses have sufficiently addressed all reviewers' concerns.

Meta-review: This paper highlights instability phenomena that occur at activation boundaries in ReLU neural networks when weights used in standard parameterizations and normalizations are perturbed. To address these issues, the authors use a hyperspherical coordinate system and demonstrate how it may promote the stability of optimization processes and improves generalization.

The reviewers appreciated the motivating problem presented in the paper, instability phenomena, and found the proposed parameterization to be both interesting and potentially beneficial in practice. Finally, some of the reviewers praised the writing and felt the paper was easy to follow.
Decision: Accept (poster)