Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks

Authors: wenlin  chen
hong  ge
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:44:14

Abstract: We introduce a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. Our proposed analysis reveals a critical instability in common neural network parameterizations and normalizations during stochastic optimization, which impedes fast convergence and hurts generalization performance. Addressing this, we propose Geometric Parameterization (GmP), a novel neural network parameterization technique that effectively separates the radial and angular components of weights in the hyperspherical coordinate system. We show theoretically that GmP resolves the aforementioned instability issue. We report empirical results on various models and benchmarks to verify GmP's advantages of optimization stability, convergence speed and generalization performance.

Main paper: 
 introduction: 
 In standard neural networks, each neuron performs an affine transformation on its input x ∈ R n followed by an element-wise nonlinear activation function g:EQUATIONwhere the affine transformation is parameterized by a weight vector w ∈ R n and a bias scalar b ∈ R.Rectified Linear Unit (ReLU) [17] is arguably the most popular activation function in modern deep learning architectures due to its simplicity and effectiveness:EQUATIONWe introduce a novel concept for ReLU networks, called characteristic activation boundary, which is defined as the set of input locations with zero pre-activations. By definition, such boundaries separate the active and inactive regions of ReLU neurons in the input space, which play a critical role in ReLU networks since they serve as the fundamental building blocks for the decision boundaries.In this work, we analyze the evolution dynamics of the characteristic activation boundaries in ReLU networks. Our novel analysis identifies a critical instability in many common parameterizations and normalizations that operate in the Cartisian coordinate, including Standard Parameterization, Weight Normalization [56] , Batch Normalization [24] and Layer Normalization [2] . We show theoretically that this issue destabilizes the evolution of the characteristic boundaries in the presence of stochastic gradient noise, and empirically that it impedes fast convergence and hurts generalization performance.To address this, we introduce a novel neural network parameterization, named Geometric Parameterization (GmP). As opposed to traditional parameterizations and normalizations which operate in the Cartesian coordinate, GmP operates in the hyperspherical coordinate which automatically decouples the radial and angular components of the weights. Our theoretical results show that GmP stabilizes the evolution of characteristic activation boundaries during stochastic optimization. Our empirical studies confirm the efficacy of GmP on various models and benchmarks. We report notable empirical improvements in optimization stability, convergence speed, and generalization performance, which validate our hypotheses and theoretical results. 
neural characteristic activation analysis for relu networks: 
 This section defines characteristic activation boundary and its geometric connection to ReLU features, which serve as the basics of the proposed analysis for understanding neural network training dynamics.Standard Parameterization (SP) refers to the weight-bias parameterization as defined in Eq (1) .Weight Normalization (WN) [56] is a reparameterization technique that decouples the length l and the direction v /∥v∥2 of w in a standard ReLU unit (1) :EQUATIONWN makes the length l and the direction v /∥v∥2 of the weight vector independent of each other in the Cartesian coordinate system, which is effective in improving the conditioning of the gradients and thus speeding up optimization.Batch Normalization (BN) [24] is a widely-used normalization layer in modern deep learning architectures such as ResNet [22] , which is effective in accelerating and stabilizing stochastic optimization of neural networks [29] . BN standardizes the pre-activation using the empirical mean and variance estimated by mini-batch statistics:EQUATIONwhere γ, β ∈ R are two free parameters to be learned from data, which adjusts the output of the BN layer as needed to increase its expressiveness.Noticing that the ReLU activation function ( 2) is active for positive arguments s > 0 and inactive for negative arguments s < 0, we introduce a novel concept called characteristic activation boundary (CAB) at the cut-off point s = 0 (i.e., the point with zero pre-activations), which will play a central role in our proposed characteristic activation analysis.Definition 2.1 (CAB). The characteristic activation boundary (CAB) for a ReLU unit is defined by the set of input locations with zero pre-activations:EQUATIONA CAB is an (n -1)-dimensional hyperplane that separates the active and inactive regions of a ReLU unit in the input space R n . Definition 2.2 (Spatial location). The spatial location of a CAB is defined asEQUATIONwhich is a point on the corresponding CAB since w T ϕ +b = 0. Not that the vector that goes from the origin to ϕ specifies the shortest path between the origin and the CAB. Therefore, each spatial location uniquely determines a CAB. Fig 1(a) visualizes the spatial location of a CAB in R 2 . CABs play a critical role in ReLU networks, since they effectively specify the locations of ReLU features (i.e., non-linearities) which serve as the building blocks for the decision boundaries. Smaller changes between the perturbed and original boundaries imply higher stability. GmP is most stable against perturbations.In the presence of stochastic gradient noise, we identify an instability issue in the evolution of CABs under common neural network parameterizations and normalizations. Proposition 2.3 (Instability of SP). A perturbation ε to the weight w under SP (1) can result in an arbitrarily large change in the angular direction of the CAB if w has a similar magnitude to ε1 .Proof. The change in the angular direction of the CAB under SP is given byEQUATIONSince w has a similar magnitude to ε, in the extreme case one can construct a small perturbation ε = -(1 + δ) w with an infinitesimal δ. Then, the angular change is ⟨w, w + ε⟩ = ⟨w, -δ w⟩ = π, which rotates the CAB by 180°.In general, the angle ⟨w, w + ε⟩ can take arbitrary values in [0, π] even for a small perturbation ε. This indicates that CABs are vulnerable to small perturbations when w has a small norm 1 . This has the implication that even a small gradient noise could destabilize the evolution of ReLU features during stochastic optimization and thus destroy the learning signal for the decision boundaries of the network. Such instability could prevent practitioners from using larger learning rates [18] .It might be tempting to think that WN does not suffer from this issue as it decouples the length and the direction of w as in Eq (3). However, we show that WN can also be vulnerable to small perturbations. Proposition 2.4 (Instability of WN). A perturbation (ε, ε ′ ) to (v,l) under WN (3) can result in an arbitrarily large change in the angular direction of the CAB if v has a similar magnitude to ε.Proof. The change in the angular direction of the CAB under WN is given byEQUATIONwhich has an identical form to Eq (7) . Therefore, in the extreme case one similarly construct a small perturbation ε = -(1 + δ) v with an infinitesimal δ, which rotates the CAB by 180°.Furthermore, BN can also suffer from this instability issue as it can be viewed as a generalized version of weight normalization.Proposition 2.5 (Instability of BN). Without loss of generality, assume that the input x has zero mean. A perturbation ε to the weight w under BN (4) can result in an arbitrarily large change in the angular direction of the CAB if w has a similar magnitude to ε.Proof. Since Êx [w T x] = w T Êx [x] = 0 by assumption, Eq (4) can be re-written asEQUATIONwhere the norm ∥w∥ Σ is with respect to the data covariance matrix Σ = Var[x] estimated by minibatch statistics. It can be seen that Eq (9) has the same form as WN (3) except that WN fixes Σ = I. Therefore, the instability argument for WN in the proof of Proposition 2.4 also holds for BN. It is worth noting that the same proof technique can be used to show instability for Layer Normalization (LN) [2] , which is another normalization technique widely used in Transformers [61] , since BN and LN have identical functional form (4) except that the expectation and variance operators in LN are applied to the feature axis rather than the batch axis.Figs 1(b)-1(e) simulate the evolution behaviors of the CABs under SP and WN in R 2 , showing that even a small perturbation δ of magnitude 10 -3 can drastically change the spatial locations of the CABs. More generally, this instability issue exists in many neural network parameterization and normalization techniques that operate in the Cartesian coordinate, since the fundamental issue is that the change in angular direction of the CAB always has the same unstable form as in Eq (7) . 
geometric parameterization: 
 This section introduces a novel Geometric Parameterization (GmP) and demonstrates its nice theoretical property, eliminating the instability issue in common parameterizations and normalizations.In a high dimensional input space, most data points live in a thin shell since the volume of a high dimensional space concentrates near its surface [4] . Intuitively, the spatial locations of CABs should be close to the thin shell where most data points live, since this spatial affinity between CABs and data points will introduce ReLU features (non-linearities) at suitable locations in the input space to separate different inputs x by assigning them different activation values. The use of hyperspherical coordinate enables us to explicitly control the locations of such non-linearities. Definition 3.1. The spatial location of a CAB in the hyperspherical coordinate system is given by ϕ(λ, θ) = -λ u(θ), (10) where the radius λ corresponds to b /∥w∥2 in SP, and the unit directional vector u(θ) corresponds to w /∥w∥2 in SP and is determined by the angle θEQUATION)u(θ) ∈ S n-1 is a unit directional vector on the unit hypersphere S n-1 := {x ∈ R n : ∥x∥ 2 = 1}. Definition 3.2. The CAB in the hyperspherical coordinate system is given byEQUATION)Geometrically speaking, the angle θ controls the direction of a CAB, while the radius λ controls the distance between the origin and the CAB. Calculating the pre-activation of a ReLU unit for an input x is equivalent to projecting x onto the unit vector u(θ) and then adding the radius λ to the signed norm of the projected vector. From this perspective, it is clear that a CAB is a set of inputs whose projections over u(θ) have signed norm -λ. For this reason, we refer to this radial-angular decomposition in the hyperspherical coordinate system as Geometric Parameterization (GmP).Definition 3.3 (GmP). A ReLU unit under geometric parameterization (GmP) is given byEQUATIONwhere r, λ, θ are three learnable parameters. The radial and angular parameters λ and θ = [θ 1 , • • • , θ n-1 ] specify the spatial location of the CAB, while the scaling parameter r controls the scale of the activation. These parameters have n + 1 degrees of freedom in total (same as SP).Remark 3.4 (Computational cost). Let n=fan-in and m=fan-out for a layer. GmP needs to compute 2n -2 more scalars [sin(θ 1 ), • • • , sin(θ n-1 ), cos(θ 1 ), • • • , cos(θ n-1 )] than SP for each of the m neurons, which incur an extra cost of O(mn) for all neurons in a layer. However, since the cost of computing the affine transformation for each layer is also O(mn), the total computational cost of GmP remains O(mn) for each layer, which is identical to SP.Remark 3.5 (Layer-size independent parameter initialization). Unlike existing neural network parameterizations which are sensitive to initialization, GmP can work with less carefully chosen initialization schemes independent of the width of the layer, thanks to an invariant property of the hyperspherical coordinate system. To see this, we first consider the distribution of the angular direction of the CAB under SP. Under popular initialization methods such as the Glorot initialization [16] and He initialization [21] , each element in the initial weight vector w under SP is independently and identically sampled from a zero mean Gaussian distribution with a layer-size dependent variance. Interestingly, this always induces a uniform distribution over the unit n-sphere for the direction of the CAB, no matter what variance value is used in that Gaussian distribution. This allows us to initialize the angular parameter θ uniformly at random by sampling from the von Mises-Fisher distribution [63, 65] . The parameter λ is initialized to zero due to its connection λ = b /∥w∥2 to SP and the common practice to set b = 0 at initialization. The scaling parameter r is initialized to one, based on the intuition that the scale roughly corresponds to the total variance of the weights w in SP. We highlight that none of the parameters λ, θ, and r in GmP requires layer-size dependent initialization.Remark 3.6 (Internal covariate shift). One implicit assumption of our characteristic activation analysis is that the input distribution to a neuron always centers around the origin during training. This assumption automatically holds for one-hidden-layer networks since the training data can be centered during data pre-processing. However, this assumption is not necessarily satisfied for multihidden-layer networks, since the inputs to an intermediate layer are transformed by the weights and squashed by the activation function in the previous layer. We propose a simple technique called Input Mean Normalization (IMN), which is a parameter-free layer that centers the input to each intermediate layer using the empirical mean estimated by mini-batch statistics:EQUATIONIt might be tempting to think that IMN is similar to Mean-only Batch Normalization (MBN) [56] . However, we emphasize that MBN is unable to address the internal covariate shift problem as it is applied to pre-activations rather than post-activations.Section 2.3 identified an instability in the evolution of CABs under common parameterizations and normalizations due to a fundamental issue in the Cartesian coordinate system. In contrast, CABs under GmP is much more stable under perturbation, since the radius λ and angle θ of the spatial location ϕ are automatically disentangled in the hyperspherical coordinate system. This means that small perturbations to the parameters in GmP will only cause small changes in the spatial location of the CAB. Below, we show that under a small perturbation, the change in the angular direction of a CAB under GmP is bounded by the magnitude of the perturbation.Theorem 3.7. With an infinitesimal perturbation ε:= [ε 1 , • • • , ε n-1 ]T to the angular parameter θ, the change in the angular direction u(θ) ∈ S n-1 (n ≥ 2) of a CAB under GmP is bounded by We use cross-validation to select the learning rate for each method. The optimal learning rate for SP, WN, and BN is lower than that for GmP, since their training becomes unstable with higher learning rates, as shown in (c).EQUATION) x 0 z ϕ |λ| (a) λ < 0, θ = 0. x 0 z ϕ |λ| (b) λ > 0, θ = π.The proof of Theorem 3.7 can be found in Appendix B, which is based on an elegant idea from differential geometry that the angle ⟨u(θ), u(θ + ε)⟩ is simply the norm ∥ε∥ M θ of the perturbation with respect to the metric tensor M θ for the hyperspherical coordinate, which is bounded by ∥ε∥ 2 . This implies that optimizing the geometric parameters in GmP directly translates into a smooth evolution of the spatial locations of CABs even in the presence of stochastic gradient noise. Looking back at Figs 1(b)-1(e), we can see that CABs under GmP gradually moves away from its original spatial location as we increase δ, which is in sharp contrast to the unstable evolution of CABs under other parameterizations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [Yes] Justification: Our code is available at https://github.com/Wenlin-Chen/ geometric-parameterization. All datasets used are publicly available common ML benchmarks and are properly cited in the paper. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
empirical evaluation: 
 This section contains empirical evaluation of GmP with neural network architectures of different sizes on both illustrative demonstrations and more challenging machine learning classification and regression benchmarks. A more detailed setup for each experiment can be found in Appendix C2 . This section verifies the validity of the hypotheses of our proposed characteristic activation analysis on two illustrative experiments aided with visualization, and demonstrates that the improved stability under GmP is beneficial for neural network optimization and generalization.In max i |∆ϕ i,t | = max i |ϕ i,t+1 -ϕ i,t| under GmP is always smaller than one throughout training at each train step t, which allows small but consistent updates to be accumulated. In contract, the changes under other parameterizations can be up to 2 16 at some steps. Such abrupt and huge changes of spatial locations make the evolution of the spatial locations inconsistent. Consequently, it is much harder for optimizers to allocate those activations to the suitable locations during training. In addition, many activations are allocated to the regions that are far away from the data region and cannot be seen in This section evaluate GmP on common machine learning regression and classification benchmarks with a variety of neural network architectures, demonstrating its broad applicability.We evaluate GmP on 7 regression problems from the UCI dataset [11] . We train an MLP with one hidden layer and 100 hidden units for 10 different random 80/20 train/test splits. We use the Adam optimizer [28] with cross-validation. We find that the optimal learning rate is 0.1 for GmP and 0.01 for all the other methods. Table 1 shows that for in most cases, GmP consistently achieves the best test RMSE on all benchmarks, significantly outperforming other methods. We evaluate GmP with a medium-sized convolutional neural network VGG-6 [58] on ImageNet32 [8] , which contains all 1.3M images and 1,000 categories from ImageNet (ILSVRC 2012) [10] , but with the images resized to 32 × 32. We follow the experimental setup for optimization and data augmentation as in [8] . This provides insights into how the batch size and intermediate normalization layer affect the empirical convergence speed and generalization performance of different parameterizations. We use cross-validation and find that the optimal initial learning rate is 0.1 for GmP and 0.01 for all the other methods. Table 2 shows that GmP+IMN consistently achieves the best top-1 and top-5 validation accuracy for all batch sizes considered. Furthermore, the improvement of GmP+IMN over other methods gets larger as the batch size increases, highlighting the robustness and scalability of GmP with large batch sizes. In addition to achieving the best performance, Fig 4 shows that GmP+IMN (the green curve) also converges significantly faster than other compared methods: its top-5 validation accuracy converges within 25 epochs, which is 10 epochs earlier than the second best method BN. The ablation study GmP vs GmP+IMN shows that IMN significantly improves the performance of GmP, which is expected since it addresses the problem of covariate shifts between hidden layers. Notably, Wide ResNet (WRN 28-2) [68] trained with BN and a batch size of 500 only achieved 43.08% top-1 validation accuracy as reported in [8] , underperforming VGG-6 trained with GmP+IMN (43.62% as shown in Table 2 ). This reveals the significance of better parameterizations: even a small non-residual network like VGG-6 with GmP+IMN can outperform large, wide residual networks like WRN 28-2. We evaluate GmP with a large residual neural network, ResNet-18 [22] , on the full ImageNet (ILSVRC 2012) dataset [10] , which consists of 1,281,167 training images and 50,000 validation images that contain objects from 1,000 categories. The size of the images ranges from 75 × 56 to 4288 × 2848. We follow the experimental setup for optimization and data augmentation as in [22] . Specifically, we use the SGD optimizer with momentum 0.9 and reduce the learning rate by 0.1 at epochs 30, 60 and 80. All models are trained for 90 epochs. We use a batch size of 256 for all methods. We use cross-validation and find that the optimal initial learning rate is 0.1 for all compared methods. We employ random horizontal flip, random resizing (256-480) with preserved aspect ratio, random crop (224), and color augmentation for data augmentation during training [31] . To address the internal covariate shift problem, we employ IMN for GmP. Following [56] , MBN is used for WN. The result of SP is not reported as BN is used in ResNet by default. Table 3 reports the single-center-crop top-1 and top-5 validation accuracy for all compared methods, which shows that GmP+IMN significantly outperforms BN and WN+MBN in terms of both top-1 and top-5 validation accuracy. This demonstrates that our method is useful for improving large-scale residual network training.5 Related Work 
neural network training dynamics: 
 Neural Tangent Kernels (NTKs) [25, 36] show that wide networks evolve like linear models during training, while Neural Network Gaussian Processes (NNGPs) [49, 35] provide insights into how wide neural networks generalize. [15] studies the evolution of the Hessian spectrum of neural networks during training. [3] investigates the curvatures of different principle components around the optimum of a regularized linear autoencoder. [1, 32, 70] analyze the training dynamics with natural gradient descent (NGD); see Appendix D for a discussion of the connection between GmP and NGD. [45] investigates the effectiveness of stochastic gradient descents for neural network training. [23, 60, 57, 71, 6, 37, 72, 38, 50, 54, 33, 62, 30] study the effects of Adam, BN and weight decay on training dynamics. [64, 26] study the failure cases in joint training of deep ensembles. [7, 53, 34, 5, 43] investigate the training dynamics of distributed optimization with different client subsampling schemes. [48, 52, 13, 14, 19, 20, 12, 55] pay special attention to analyzing the ReLU activations in neural networks. In contrast, our proposed characteristic activation analysis studies the evolution of the characteristic activation boundaries during stochastic optimization.In addition to SP, WN [56] and BN [24] , there are many other neural network normalization and parameterization techniques. Instead of normalizing the batch axis as in BN, Layer Normalization (LN) [2] operates on the feature axis, which is preferred for small batches or variable-length inputs such as text [61] . Other variants of BN include Switchable Normalization [44] and Instance Enhancement Batch Normalization [39] . There are also normalization techniques designed for specific applications. For instance, Instance Normalization [59] and Group Normalization [66] are special cases of LN designed for CNNs, while Spectral Normalization [47, 69] is specifically designed for GANs and transformers. All these methods operate in the Cartesian coordinate. The instability argument for BN also holds for these methods since they have a similar form to BN except that they normalize the input tensors along different axes. Besides, there is also a line of research on learning orthogonal rotation matrices for the weight vector [42, 40, 41, 51] . In contrast, our proposed Geometric Parameterization (GmP) overcomes the instability issue by operating in the hyperspherical coordinate. 
conclusion: 
 We presented a novel characteristic activation analysis for understanding the training dynamics of ReLU networks, which exploits special activation values to characterize ReLU units. Using the proposed analysis, we identified a critical instability in common neural network parameterizations and normalization techniques that operate in the Cartesian coordinate. Addressing this, we proposed a new parameterization called Geometric Parameterization (GmP) which operates in the hyperspherical coordinate. We demonstrated the theoretical advantages of GmP for ReLU networks. We performed empirical evaluations to verify our analysis, showing its improved training stability, convergence speed and generalization performance on a variety of tasks with different neural network architectures. More broadly, we believe that the general idea behind our proposed analysis of viewing neural network parameters from a different perspective has the potential to reveal new research directions for the study of neural network optimization and training dynamics. Limitations and potential future work directions are discussed in Appendix E. 1 x 2 |λ| ϕ θ 0 (a) λ < 0, θ ∈ [0, π). x 1 x 2 |λ| ϕ θ 0 (b) λ > 0, θ ∈ [π, 2π). x 1 x 2 |λ| ϕ θ 0 (c) λ > 0, θ ∈ [0, π). x 1 x 2 |λ| ϕ θ 0 (d) λ < 0, θ ∈ [π, 2π). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This is a technical ML paper focusing on the theoretical properties of the training dynamics of ReLU networks. The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All datasets used are publicly available common ML benchmarks and are properly cited in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Proof. Since u has unit length, the change in the angular direction ⟨u(θ), u(θ + ε)⟩ under infinitesimal perturbation ε is equal to the arc length δs between the two points u(θ) and u(θ + ε). By the generalized Pythagorean theorem, the arc length is given byEQUATIONwhere M θ is the metric tensor for the hyperspherical coordinate system. Therefore, the change in the angular direction ⟨u(θ), u(θ + ε)⟩ under infinitesimal perturbation ε is simply the norm of ε with respect to the metric tensor M θ for the hyperspherical coordinate system:EQUATION)Hence, we need to work out a formula for calculating M θ .Let x = [x 1 , • • • , x n ] T ∈ R n (n ≥ 2) be an input in the Cartesian coordinate system, where the metric tensor is the Kronecker delta m ′ ij = δ ij . For the Geometric Parameterization of the unit hypersphere S n-1 , we haveEQUATION)The metric tensor M θ for the Geometric Parameterization of S n-1 is the pullback of the Euclidean metric in R n :EQUATION)We discuss the diagonal (a = b) and off-diagonal (a ̸ = b) elements separately:• a ̸ = b: First, we note that ∂ui ∂θq = 0 for 0 < i < q. For q ≤ i ≤ n, we have∂u i ∂θ q = -δ iq i k=1 sin(θ k ) + cos(θ i ) i-1 k=1 sin(θ k ) i-1 k=1 δ kq cos(θ k ) sin(θ k ) , q ≤ i < n, ∂u n ∂θ q = cos(θ q ) sin(θ q ) n-1 k=1 sin(θ k ).(20) Without loss of generality, we assume 0 < a < b < n. Then, we haveEQUATIONOn the other hand, by recursively collecting like terms and using the identity that sin 2 (θ q ) + cos 2 (θ q ) = 1, ∀q, we haveEQUATIONThis shows that m ab = 0 for a ̸ = b and hence M θ is a diagonal matrix.• a = b: Following a similar argument as above, we can obtain the diagonal elements of M θ :EQUATIONOn the other hand, by Eq (22) , we haveEQUATIONHence, it follows thatEQUATIONand m 11 = 1.Therefore, the metric tensor for the hyperspherical coordinate is a diagonal matrixEQUATIONFinally, the change in the angular direction of a unit vector u(θ) under a perturbation ε to θ is given by the norm of ε with respect to the tensor matrix M θ :EQUATIONFurthermore, since -1 ≤ sin(θ j ) ≤ 1, it follows that 0 ≤ m i,i = i-1 j=1 sin 2 (θ j ) ≤ 1 for all i. Therefore, Eq (27) is bounded byε 2 1 + n-1 i=2   i-1 j=1 sin 2 (θ j )   ε 2 i ≤ ε 2 1 + n-1 i=2 ε 2 i = ∥ε∥ 2 .This completes the proof.where h : M → R is any differentiable function and η t is the learning rate. Note that the exponential map Exp θt maps the update F (θ t ) -1 ∇h(θ t ) from the tangent space T θt M back to the manifold M , which results in an exact update invariant to the parameterization/coordinate. However, it is usually infeasible to calculate the exponential map since it requires solving a linear system of size dim(M ), which is the total number of parameters in the neural network. In practice, the most commonly used version of natural gradient descent is its first-order approximation given byEQUATIONwhich is a second-order optimization method that is invariant up to first-order transformation of the parameterization/coordinate. Since our proposed Geometric Parameterization is a nonlinear transformation of standard parameterization, it will still behave differently under the first order approximation of the full version of natural gradient descent.Beyond ReLU activation. This work analyzed ReLU networks due to their wide adoption. However, the proposed characteristic activation analysis is general and can be directly applied to other ReLUlike activation functions such as Leaky ReLU [46] , ELU [9] and SReLU [27] , because the definition of spatial locations is essentially any breakpoints (i.e., the sources of non-linearity) in a piecewise activation function. The analysis of smooth activations such as Sigmoid and Tanh is left for future work.Beyond single-hidden-layer networks. This work only performed characteristic activation analysis for single-hidden-layer ReLU networks and proposed a practical workaround to address the covariate shift issue between hidden layers by using input mean normalization (IMN). For future work, this analysis needs to be generalized to examine training dynamics in multiple-hidden-layer networks to understand the theoretical behaviors of deep networks. One potential difficulty is that the characteristic activation boundaries of multiple-hidden-layer networks become piecewise linear partitions of the input space, which are less straightforward to analyze. A possible solution would be to consider how the assignment of each data point to the partition evolves during training, similar to how we track the characteristic activation boundaries.Sparse activation and large-width limiting behavior. Interestingly, the product structure of Geometric Parameterization (GmP) as shown in Eq (11) indicates that it could lead to sparse activations or even vanished gradients. Empirically, we find that the activations and gradients of GmP will not vanish during training if the GmP parameters are initialized from the von Mises-Fisher distribution (i.e., uniformly distributed on the hypersphere) as discussed in Remark 3.5. For future work, it would be interesting to investigate the large-width limiting behavior of GmP (e.g., similar to NTK [25] or µP [67] ) to provide theoretical guarantees of the sparsity patterns in GmP activations.Efficient representation of hidden neurons. Directly representing the hidden neurons rather than the parameters in the hyperspherical coordinate could be a more efficient representation. However, it is unclear how to train such neural networks since gradient-based optimization is no longer applicable. One approach would be to use rejection sampling to allocate the hidden neurons on the hypersphere, but it is inefficient and suffers from curse of dimensionality. For future work, it would be interesting to find a more efficient learning algorithm that directly allocates hidden neurons on the hypersphere.Combining GmP with existing techniques. We leave the investigation of the theoretical properties and empirical performance of combinations of GmP and existing normalization (e.g., BN or LN) and regularization techniques (e.g., weight decay) for future work. It would also be interesting to examine the performance of GmP with other neural network architectures (e.g., transformers) on large datasets from different domains (e.g., NLP) under future work.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.38th Conference on Neural Information Processing Systems (NeurIPS 2024).∥w∥2 is supposed to be small during training as large weights would lead to overfitting, numerical instability and even divergence[18] (e.g., the popular weight decay method explicitly regularizes ∥w∥2 to be close to zero).Our code is available at https://github.com/Wenlin-Chen/geometric-parameterization. 


Decision: This paper highlights instability phenomena that occur at activation boundaries in ReLU neural networks when weights used in standard parameterizations and normalizations are perturbed. To address these issues, the authors use a hyperspherical coordinate system and demonstrate how it may promote the stability of optimization processes and improves generalization.

The reviewers appreciated the motivating problem presented in the paper, instability phenomena, and found the proposed parameterization to be both interesting and potentially beneficial in practice. Finally, some of the reviewers praised the writing and felt the paper was easy to follow.

Reviews: review: paper_summary: paper_summary: In standard neural networks each neuron, and activation function g performs the following operation on the input $x\in\mathbb{R}^{n}$:
$$z=g\left(w^{t}x+b\right)$$
When usually g in the ReLU activation. 

In this work the authors identify a critical instability in many common NN settings, which theoretically destabilizes the evolution of the characteristic boundaries, and empirically impedes fast convergence and hurts generalization performance. To address this, they introduce a novel NN parameterization, named Geometric Parameterization (GmP) which operates in the hyper spherical coordinate. The authors show that this parameterization, both theoretically and empirically, improves optimization stability, convergence speed, and generalization performance.
strength_weakness: strength_weakness: I will start with a somewhat nit-picking weakness. The paper uses $\mathcal{B}$ and ϕ in two different contexts. ϕ is used in Definition 2.2 as a vector and in Definition 3.1 as a function. $\mathcal{B}$ is used in Definition 2.2 as a hyperplane and in Definition 3.2 as a function. This isn't accidental, as there is a strong connection between the definitions, but I believe that using slight differences in the notations would be better.

Another weakness is that the article hasn't provided an empirical case where GmP isn't the best option. It is possible that GmP is always the best choice (and if so, great), but usually, there are cases where new techniques aren't the best options. It would have been interesting to see a scenario where GmP isn't the best option and (perhaps under future work) understand why.
questions: questions: I have few technical questions regarding section B – proof of theorem 3.7:
	
1. It is not clear to me why equation 16 is true. I would suggest adding an explanation or a reference to one.
	
2. The analyze of the case where a≠b. It isn't immediately clear from equation 18 why " sum of terms that are either zero or with alternating signs". Maybe it is simple calculus, but I suggest adding additional step to show this claim.
limitations: limitations: Yes, the authors do address the limitations. The main limitation is the input mean normalization that being address in appendix E.
paper_summary: paper_summary: Summary.
Authors consider training  neural nets with Adam after a change of coordinate to spherical coordinate system. They show that in spherical coordinate system the direction of the half spaces which specifies the activeness of the Relus in stable with respect to small changes in the angles, hence the optimization is supposed to be easier. The perform some experiments to see the advantage of using this coordinate system for optimization in practice.
strength_weakness: strength_weakness: First the proof of Theorem 3.7. is wrong; I think what the authors meant in the argument of the Theorem is $\|u(\theta) - u(\theta + \epsilon)\|$ rather than their dot product (the dot product claim is obviously wrong since if epsilon goes to zero right hand side goes to zero while left hand side is constant). even with $\|u(\theta) - u(\theta + \epsilon)\|$ instead, the proof in appendix is wrong; the correct formula of length given metric $M_\theta$ on a manifold is: \|u(\theta) - u(\theta + \epsilon)\| = \int_{0}^1 \sqrt{\epsilon^\top M_{\theta + t\epsilon} \epsilon}. The proof can be corrected by estimataing $M_{\theta + t\epsilon}$ with $M_\theta$. Though the final argument is expected to be true, perhaps with different constants. 

Second, the paper is highly overclaimed about tjhe theoretical results. The theoretical claim of the paper is only that the normal vector of half space change smoothly if one parameterize in spherical coordinates despite normal coordinates, independent of the training algorithm, while the authors are claiming at the beginning that they analyze the behavior of training algorithms in this case and analyze its advantage over normal coordinates which is not the case. 

minor comments:
Line 155 -> typo IMH instead of IMN

Figure 2, (d), ..., (g): how is the fact that activations are more spread out related to the smooth evolution of spatial locations? (the plot with the yellow lines), and these are in 1 and 2D as far as I see, do you have a version of these plots in higher dim? (since that is the authors claim)
questions: questions: -optimizing in the normal coordinate system with relu activation is known to create sparse features, which can help with generalization, and I am not sure if happens if one optimizes in spherical coordinates. Does the authors see sparsity in their training as well, and if not how do they assess this fact with their claim that spherical coordinate has better generalization?

-In particular authors show in experiments that they can pick a larger learning rate in spherical coordinates, but this fact solely cannot be an indicator of better generalization (even if two algorithms A and B run in the same coordinate system and A picks larger learning rate than B it is still not clear which one generalize better). Can the authors further elaborate on their claim about superiority of spherical coordinates for the generalization of the final network?
limitations: limitations: No theoretical justification for superiority of their method for optimization/generalization.
paper_summary: paper_summary: This paper introduces a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual ReLU neurons. The authors figure out the instability in common neural network parameterization and normalization during stochastic optimization. To address this, the authors propose Geometric Parameterization (GmP), which parameterizes the unit vectors by hyperspherical coordinate representation rather than scaling. The authors conduct experiments to verify the effectiveness of the method.
strength_weakness: strength_weakness: Lack discussions on other activation functions.
questions: questions: Have the authors explored how GmP performs on other activation functions? Does it also perform better? 

Suppose we use GmP on the hidden neurons rather than the weights, like scaling operations projecting $x$ onto a hypersphere. In other words, we use $\theta_1,\cdots,\theta_n$ to represent $x$ by spherical coordinate system. Can the authors give some analysis theoretically or experimentally?
limitations: limitations: N/A
paper_summary: paper_summary: This paper identifies and attempts to solve the problem of unstable activation boundaries in ReLU neural networks. The authors define a relu unit in terms of its activation boundary which is the set of inputs that cause the preactivation to be zero. The point on this hyperplane closest to the origin (termed a CAB) is used to track each neuron's activation boundary. The authors show that standard parameterization, weight norm and batch norm all suffer from an instability in the direction of the activation boundary when the weights are small since the direction of the weight can change sign. To address this, they propose to operate in hyperspherical coordinates where the angles $\theta_1, ... , \theta_{N-1}$ and radius $\lambda$ are used to define the hyperplane instead. Taking gradient steps in this geometric parameterization maintains bounded changes in the angles under small learning rate due to a nice metric property of this coordinate system. The
strength_weakness: strength_weakness: There are some assumptions in the analysis that I was unsure about (such as worst case direction of the perturbations. If instead perturbations were randomly oriented with respect to $w$, does it change the conclusions?). The authors also mention that other normalization solutions suffer from similar problems (like Layernorm etc) but this would be useful to demonstrate, perhaps in the Appendix.
questions: questions: 1. Some of the stability analysis is performed under the worst case assumption that $\epsilon \propto w$. Wouldn't this be unlikely to happen for high dimensional inputs? The main instability is caused when the vectors $w$ change sign/direction under a single step of GD. Could this issue also be solved by initializing the weights in SP to have larger variance relative to the learning rate? 
2. Have the authors considered what the structure of the gradients look like in this parameterization? Naively it seems like some of the gradient entries would have very small magnitude in high dimension since they involve products of bounded trigonometric functions. Is there a "limiting" large width description in this parameterization like for NTK parameterization or $\mu$P?
limitations: limitations: N/A

Comments: title: Has our latest response addressed your remaining concerns?
Additional theoretical results regarding the calculation of the length on the manifold requested by the reviewer
Response to the addition comments from the reviewer
Response
Thank you for your feedback! Please consider our rebuttal and let us know if your concerns have been addressed.
comment: We thank the reviewer once again for their effort in the reviewing process. As there are only a few hours left in the discussion period, we would like to ask if **our latest response regarding the calculation of length on manifold** has addressed the reviewer’s concerns. If so, we kindly invite the reviewer to consider raising their rating. If any concerns remain, we are happy to further discuss and clarify them here.
We would like to add that even if under the definition of length on the manifold mentioned by the reviewer, our argument would still hold true. We demonstrate this below.

First, note that we have shown in the proof of Theorem 3.7 in Appendix B that 
$$\sqrt{\epsilon^T M_{\theta+t\epsilon}\epsilon}=\sqrt{\varepsilon_1^2+\sum_{i=2}^{n-1}\left(\prod_{j=1}^{i-1}\sin^2(\theta_j+t\epsilon)\right)\varepsilon_i^2}\leq\sqrt{\varepsilon_1^2+\sum_{i=2}^{n-1}\varepsilon_i^2}=\lVert\boldsymbol{\varepsilon}\rVert_2$$
Then, using the definition of length on the manifold provided by the reviewer, we have
$$d(u(\theta),u(\theta+\epsilon))\equiv \int_0^1 \sqrt{\epsilon^T M_{\theta+t\epsilon}\epsilon}dt\leq\int_0^1 \lVert\boldsymbol{\varepsilon}\rVert_2 dt = \lVert\boldsymbol{\varepsilon}\rVert_2.$$

Therefore, this distance/length on the manifold would go to zero as epsilon goes to zero. Essentially, Theorem 3.7 is a differential form of the statement, whereas this is an integral form of the statement. We thank the reviewer for this suggestion and will add a corollary for this results under Theorem 3.7 in our paper.
Thank you for your response to our rebuttal. We believe that there is still a misunderstanding. Below, we respond to your new comments point by point and hope that this will sufficiently address your remaining concerns.

> ...do you have a reference for the equation you are claiming in your new response?

The first part of the equation that we are claiming in our new response is
$$\left<\mathbf{u}(\boldsymbol{\theta}),\mathbf{u}(\boldsymbol{\theta}+\boldsymbol{\varepsilon})\right>\equiv\text{arccos}(\mathbf{u}(\boldsymbol{\theta})^\text{T}\mathbf{u}(\boldsymbol{\theta}+\boldsymbol{\varepsilon}))$$
This means that the angle between two unit vectors is given by the arc cosine of the dot product between the two vectors. This is a basic definition from geometry: since the dot product between two vectors x and y is defined as
$$x^Ty=||x||||y||\cos\left<x,y\right>$$
it follows that the angle between those two vectors is given by
$$\left<x,y\right>\equiv\text{arccos}\left(\frac{x^Ty}{||x||||y||}\right)$$
In our case, since we are dealing with unit vectors whose norms are one, the denominator becomes one. This shows why the first part is right.

The second part of the equation that we are claiming in our new response is
$$\text{arccos}(\mathbf{u}(\boldsymbol{\theta})^\text{T}\mathbf{u}(\boldsymbol{\theta}+\boldsymbol{\varepsilon}))=\sqrt{\varepsilon_1^2+\sum_{i=2}^{n-1}\left(\prod_{j=1}^{i-1}\sin^2(\theta_j)\right)\varepsilon_i^2}\leq\lVert\boldsymbol{\varepsilon}\rVert_2$$
This means that the change in the angles between the two unit vectors goes to zero as epsilon goes to zero. The complete proof of this statement can be found in the Appendix B of our paper. Additional details can be found in our response to Reviewer D8e2, which will be included in the final version of the paper. This is the novel part and is one of the main theoretical contributions of our paper.

>  ...the correct formula which I think the authors are interested in using here is the definition of length on manifold

In our latest response below, we have shown that even if under the definition of the length on manifold mentioned by the reviewer, our statement would still hold true. In our paper, we showed a differential version of this statement rather than the integral version suggested by the reviewer. We thank the reviewer for their suggestions and will add this result regarding the length on manifold under Theorem 3.7 in our paper as a corollary.

First, note that we have shown in the proof of Theorem in Appendix B that 
$$\sqrt{\epsilon^T M_{\theta+t\epsilon}\epsilon}=\sqrt{\varepsilon_1^2+\sum_{i=2}^{n-1}\left(\prod_{j=1}^{i-1}\sin^2(\theta_j+t\epsilon)\right)\varepsilon_i^2}\leq\sqrt{\varepsilon_1^2+\sum_{i=2}^{n-1}\varepsilon_i^2}=\lVert\boldsymbol{\varepsilon}\rVert_2$$
Then, using the definition of length on the manifold provided by the reviewer, we have
$$d(u(\theta),u(\theta+\epsilon))\equiv \int_0^1 \sqrt{\epsilon^T M_{\theta+t\epsilon}\epsilon}dt\leq\int_0^1 \lVert\boldsymbol{\varepsilon}\rVert_2 dt = \lVert\boldsymbol{\varepsilon}\rVert_2.$$

Therefore, this distance on the manifold would also go to zero as epsilon goes to zero. 

> ...where do you exactly analyze the behavior of Relu neural networks during training?

Sorry for the confusion. In our paper, we do not claim to analyze the behavior of ReLU networks during training. Our claim in the paper is *"we analyze the evolution dynamics of the characteristic activation boundaries in ReLU networks."*, which can be found in Line 22 of our paper. Nevertheless, we thank the reviewer for helping us making our claims more precise and will make sure that this claim is consistently and clearly stated in the final version of the paper.

> ...what you mean by including "convergence speed, and generalization" analysis as a part of your theoretical contribution.

Sorry again for the confusion, but we do not include convergence speed and generalization as part of our theoretical contribution. In our abstract and introduction section, we only claim *"We show theoretically that GmP resolves the aforementioned instability issue."* (Lines 8-9) and *"Our theoretical results show that GmP stabilizes the evolution of characteristic activation boundaries"* (Lines 31-32). As promised in our rebuttal, we will make our claims more precise to clearly state that our theoretical contribution only includes the identification and solution to the instability issue, and the convergence speed and generalization are only supported by empirical evidence. We thank the reviewer for helping us making our claims more precise.

We hope that our response has addressed all your concerns and that the reviewer will reconsider their rating.
Thank you for your additional response.

The proof of Theorem 3.7 is still wrong, do you have a reference for the equation you are claiming in your new response? like I said before the correct formula which I think the authors are interested in using here is the definition of length on manifold which is given by $d(u(\theta), u(\theta + \epsilon)) = \int_{0}^1 \sqrt{\epsilon^\top M_{\theta + t\epsilon} \epsilon}.$ 

Regarding your response to clarify the contributions in your response above, for the first point, I don't understand the meaning of your sentence "analyzing the behavior of traditional NN parameterizations and normalizations during training." where do you exactly analyze the behavior of Relu neural networks during training? other than that, I acknowledge your argument regarding instability of relu in the normal coordinates, which is INDEPENDENT of the training algorithm. 

Regarding your second point of contribution, see my response in the above paragraph. Also again I don't understand what you mean by including "convergence speed, and generalization" analysis as a part of your theoretical contribution. Where do you exactly show theoretical arguments about "convergence speed" or "generalization" of training algorithms?
Thank you for the detailed response. As a result I increased my rating by 1
I thank the authors for their detailed responses to my questions. I will maintain my positive score.
We thank the reviewer once again for their effort in the reviewing process. As there are only a few working days left in the discussion period, we would like to ask if our response has addressed the reviewer’s concerns. If so, we kindly invite the reviewer to consider raising their rating. If any concerns remain, we are happy to discuss and clarify them further here.