Weight-balancing fixes and flows for deep learning
Authors: Lawrence K. Saul
Venue: Accepted by TMLR ()

Abstract: Feedforward neural networks with homogeneous activation functions possess an internal symmetry: the functions they compute do not change when the incoming and outgoing weights at any hidden unit are rescaled by reciprocal positive values. This paper makes two contributions to our understanding of these networks. The first is to describe a simple procedure, or {\it fix}, for balancing the weights in these networks: this procedure computes multiplicative rescaling factors---one at each hidden unit---that rebalance the weights of these networks without changing the end-to-end functions that they compute. Specifically, given an initial network with arbitrary weights, the procedure determines the functionally equivalent network whose weight matrix is of minimal $\ell_{p,q}$-norm; the weights at each hidden unit are said to be balanced when this norm is stationary with respect to rescaling transformations. The optimal rescaling factors are computed in an iterative fashion via simple multiplicative updates, and the updates are notable in that (a) they do not require the tuning of learning rates, (b) they operate in parallel on the rescaling factors at all hidden units, and (c) they converge monotonically to a global minimizer of the $\ell_{p,q}$-norm. The paper's second contribution is to analyze the optimization landscape for learning in these networks. We suppose that the network's loss function consists of two terms---one that is invariant to rescaling transformations, measuring predictive accuracy, and another (a regularizer) that breaks this invariance, penalizing large weights. We show how to derive a weight-balancing {\it flow} such that the regularizer remains minimal with respect to rescaling transformations as the weights descend in the loss function. These dynamics reduce to an ordinary gradient flow for $\ell_2$-norm regularization, but not otherwise. In this way our analysis suggests a canonical pairing of alternative flows and regularizers.

Published in Transactions on Machine Learning Research (0pptember/2023)
Weight- Fixes and Flows for Deep Learning
Lawrence K. Saul
Lsaut@Bladiromnnnkillal..oo
Flatiron Institute, Center for Computational Mathematics 162 Fifth Avenue, New York, NY 10010
Reviewed on OpenReview: https: opw.:evieepeeveeeeeeemmee net/ forum? id= uallyX
Abstract
Feedforward neural networks with homogeneous activation functions possess an internal symmntry: the functions they compute do not change when the incoming and outgoing weights any hidden unit are rescaled by reciprocal positive values. This paper makes two contributions to understanding of these networks. The first is to describe simple procedure, or fiz, for balancing the weights in these networks: this procedure computes multiplicative rescaling factors-one one at each hidden unit-that rebalance the weights of these networks without changing the end-to-end functions that they compute. Specifically, given an initial network with arbitrary weights, the procodure determines the functionally equivalent network whose weight matrix of minimal -ormm; the weights each hidden unit are said to be balanced when this norm is stationary with respect to rescaling trans- formations. The optimal rescaling factors are computed in iterative fashion via simple multiplicative updates, and the updates are notable in that (a) they do not require the tun- ing of learning rates, (b) they operate in parallel on the rescaling factors at all hidden units, and (c) they converge monotonically to a global minimizer of the norm.. The paper's second contribution is to analyze the optimization landscape for learning in these networks. We suppose that the network's loss function consists of two terms one that is invariant to rescaling transformations, measuring predictive accuracy, and another (a regularizer) that breaks this invariance, penalizing large weights. We show how to derive a weight-balaneing flow such that the regularizer remains minimal with respect to rescaling transformations as the weights descend in the loss function. These dynamics reduce to an ordinary gradient flow for æ--ormm regularization, but not otherwise. In this way our analysis suggests canonical pairing of alternative flows and regularizers.
1
Introduction
Many recent studies of deep learning have focused the important role of symmetries (Bronstein et al. 2021 Kunin et al. 2021 Tanaka. & Kunin 2021 Gluch & Urbanke 2021 Armenta & Jodoin 2021). In large part these studies were inspired by the role that symmetries play in our understanding of the phys- ical world (Anderson 1972 Zee 2016). Of particular interest are symmetries that arise when a model is formulated or expressed in terms more paramcters than its essential degrees of freedom. In physics these symmetries arise from so-called gauge degrees of freedom (Gross) 1992), while deep learning they are present many popular models of overparamcterized networks.
One such model in machine learning is feedforward network with rectified linear hidden units. Such network is specified by the values of its weights, but the function it computes does not change when the incoming and outgoing weights at any hidden unit are inverscly rescaled by some positive value (Glorot et al.] 2011;; see Fig. [ This particular symmetry of deep learning has already led to many important findings. For example, it known that this symmetry gives rise to a conservation law: at each hidden unit, there certain balance of its incoming and outgoing weights that does not change when these networks are trained by gradient flow (i.e., gradient descent the limit of an infinitesimally small learning rate) (Du) et al. 2018 From this conservation law follows another important. observation: if the weights are initially

---

Published in Transactions on Machine Learning Research (ttrembbr/22023
max(O, +
max(0, w.x b)
W1
W
(W)
a a
22
W/2
22
2
aw3
W2
wd
W
awd
a artw,
Fd
Figure 1: A rectified linear unit (RcLU) has the same effect if its incoming weights w and bias b are rescaled by some factor a=0 while its outgoing weights w are rescaled by the inverse factor a a- When a > 1, the rescaling increases the magnitudes of incoming weights and decreases the magnitudes of outgoing weights; when the effects are reversed.
balanced across layers, then they remain so during training, key condition for proving certain convergence results (Arora el al.] 2019). It is also possible, by analyzing the synaptic flows across adjacent layers, lo devise more powerful pruning algorithms (Tanaka et al. 2020j. Finally, a number of authors have proposed more sophisticatod forms of learning that are invariant to these rescaling transformations (Neyshabur al.
2015a; Meng al. 2019 that break this invariance in purposeful ways (Badrinarayanan al. 2015
Stock et al. 2019
Armenta et al. 2021 Zhao al. 2022
These latter works highlight important distinction: though rescaling transformations do not change the functions computed by these networks, they may affect other aspects of learning. For example, unbalanced networks may take longer to converge and/or converge to altogether different solutions. Tt is therefore important to understand the different criteria that can be used to break this invariance.
1.1 Inspiration from physics
To proceed, we consider how similar ideas have been developed to study the physical world. The simplest gauge symmetry arises in classical electrodynamics (Jackson 2002). The gauge degrees of freedom in this setting may be fized to highlight certain physics of interest. Two well-known choices are the Coulomb gauge, which minimizes the volume integral of the squared vector potential (Gubarev et al., (201), and the Lorenz gauge, which simplifies certain wave equations. It natural to ask if there analogous fixes for deep neural networks fixes, for instance, that minimize the norms of weights simplify the dynamics of learning.
With these goals in mind, this paper investigates family of norm-minimizing rescaling transformations for feedforward networks with homogeneous activation functions. These transformation are designed to minimize the entri-wise norm (with p, 2211) of the network's 's weight matrix W, defined as
W
Σ
Σ
W. (P))))
(1)
without changing the end-to-end function that the network computes (Neyhaabr et al. 2015a). Here Wep is the incoming weight at unit from unit j. A particular transformation is specified by set of multiplicative rescaling factors, one at each hidden unit of the network. The first main contribution of this paper is to show how to obtain rescaling factors that minimize the norm in eq. 1 More concretely, we derive simple multiplicative updates to compute these factors (Algorithm D and we prove that these updates converge monotonically to fixed points that represent minimum-nore norm solutions (Theorem D. Notably, these multiplicative updates parameter-free in that they do not require the setting tuning of learning rates.
2

---

Published in Transactions on Machine Learning Research (epteember20233
1.2 From fixes to flows
Norm-mininizing rescaling transformations provide one way to balance the incoming and oulgoing weights at each hidden unit in the network. We also analyze the optimization landscape in networks with rescaling symmetrics and consider how the balancedness of weights can be preserved during learning- thaing--hattii is, in the course of adapting the network's weights and biases.
This analysis yields an interesting counterpart to Noether's theorem for physical systems (Noether 1918)
When a physical system is described by a Lagrangian, its dynamics are specified by a least action principle, and Noether's theorem gives recipe to deduce the conserved quantities that are generated by its symmetries. The theorem has also been applied to deduce conserved quantities in deep learning (Kunin et al. 2021 Tanaka & Kunin 2021 Gluch & Urbanke 2021 The networks we study in this paper have a symnetrr group of rescaling transformations, but here we have sought to solve a sort of inverse problem. Specifically, in our case, the conserved quantities are specified by the choice of regularizer, and instead we have needed a recipe to deduce a compatible dynamics for lcarning-the that is, one in which the desired quantitics are conservod.
It is easy to see how this inverse problem arises. Let E(e) measure the training error of network in terms of its weights and biases, denoted collectively by O = (W,b). Also, let R(W) denote a regularizer that penalizes large weights in some way. When this regularizer is minimized with respect to rescaling transformations, it induces a particular balance of the incoriin and outgoing weights at each hidden unit of the network; see Fig. [ This minimality is expressed by the zeros of derivatives with respect to rescaling factors at each hidden unit, and these zeros are the quantities that we seek to conserve for a weight-balancing flow. Thus we start from a regularizer R(W) and seek a dynamics that not only conserves these zeros, but also descends in a loss function of the form L(O)= = E(O) XR(W) for any A20.
The second main contribution of this paper is to give a recipe for such dynamics namely, to derive weight- balancing flows such that regularizer R(W) remains minimal with respect to rescaling transformations throughout the course of learning. This recipe is given by Theorem 4 and it has an especially simple form
d W
ŌR
Oc
dt ow.y - Wiy aw
(2)
Additionally, in Theorem 5 we prove that these flows descend under fairly general conditions in the loss function, (()). The dynamics in eq. (2) roduces to ordinary gradient flow, namely W (aC/8W)) when the weights are regularized by their fy-norm (i.e., taking R(W)- = E,, W3), but it suggests much richer set of flows for other regularizers, including but not limited to those based on the fp.-- in eq.
The paper is organized as follows. Section 2 presents the multiplicative updates for minimizing the f in eq. 1 with respect to rescaling transformations. Section 3 derives the weight- balancing flows in eq. 2 Finally, section 4 relates these ideas to previous work, and section 5 discusses directions for future research.
2 Weight balancing
In this section we show how to balance the weights in feedforward networks with homogeneous activation functions without changing the end-to-end function that these networks compute. Section 2.1 describes the symmetry group of rescaling transformations in these networks, and section 2.2 presents multiplicative updates to optimize over the elements of this group. Finally, section 2.3 demonstrates empirically that these updates converge quickly in practice. We refer the reader to appendix for a formal proof of convergence.
2.1 Preliminaries
Our interest lies in feodforward networks that parameterize vector-valued functions f Rdd - R". The following notation will be useful. We denote the indices of a network's input, hidden, and output units, respectively, by I, H. and O, and we order the units so that I = dl....,d;, H. = (d+1...., n-r), and O = (n-r+I,....,1 n) where n is the total number of units. Let x E dd denote an input to the network and fxx) E Re its corresponding output. The mapping x + f fxx is determined by the network's weight matrix W ppnn, biases be E R², and activation function g: + R at each non-input unit. The mapping x f(x)
3

---

Published in Transactions on Machine Learning Research (epttember/2223)
can be computed by the feedforward procedure that sequentially activates all of the units in the network- that is, setting h; = 2; for the input units, propagating h; = Wiyhy + by) to the remaining units, and setting fx) ha+s-r from the ith output unit. Since the network feodforward, its weight matrix is strictly lower triangular, and many of its lower triangular entries are also equal to zero (e.g., between input units). We also assume that the output units are unconnected (with Way =0 if j O and i>j) and that the same activation function is used at every hidden unit.
A rescaling symmetry arises at each hidden unit when its activation function is positive homogeneous of degree one Neyshabur et al. 2015a; Dinh et al. 2017)-that is, when it satisfics g(az) = ag(z) for all a 0; see Fig. 1 In this paper we focus on networks of rectified linear hidden units (ReL.s)) with the activation function g(2) max(0,2) 2) at all i H, but this property is also satisfied by linear units (Saxe al. 2013).
leaky ReLUs (He ct al. 2015), and maxout units (Goodfellow et al. 2013).
Our results in this paper apply generally to leedforward networks (i.e., those represented by directed graphs) with homogeneous activation functions. For obvious reasons, the case of layered architectures is of special interest, and in such architectures, the weight matrix will have many zeros indicating the absence of weights between non-adjacent layers. However, we will not assume this structure in what follows. Our only assump- tion is that each network there are no vacuous hidden units or equivalently, that every hidden unit of the network plays some role in the computation of mapping x f(x). For example, a minimal requirement is that for each i H. there exists some j <i such that Way # and also some j>i such that Wa+i #0.
Next we consider the symmetry group of rescaling transformations in these networks. Fach such transfor- mation is specified by a set of rescaling factors, one for each hidden unit. We use
A= = (a E R" 10 if E H. a= otherwise)
(3)
to denote the set of these transformations. Then under particular rescaling, represented by some E A, the network's s weights and biases are transformed multiplicatively as
Wi < Waj(a,/a,),
(4)
b. < b.a..
(5)
It may seem redundant in eq. 33 to introduce rescaling factors at non-hidden units only to constrain them to be equal to one. With this notation, however, we can express the transformations in eqs. 5 without
distinguishing between the network's different types of units. As shorthand, we write (W', b') ~ -W,b) and
W W to denote when these paramcters are equivalent via eqs. 5) up to some rescaling.
2.2 Multiplicative updates
The first goal of this paper is to solve the following problem. Given a feedforward network with weights Wo and biases by, we seek the functionally equivalent network, with (W, (Wo. such that W has the smallest. possible norm in eq. ① We provide an iterative solution to this problem in Algorithm | which computes W and b via sequence of rescaling transformations. Each rescaling transformation is in turn implemented by multiplicative update of the form in eqs. 4 5p. For each update, the key step is to compute the rescaling factor a, at each hidden unit i E H. from a ratio comparing the magnitudes of its ingoing and outgoing weights. In this section, we describe the basic ingredients of this algorithm.
To begin, we define two intermediate quantities that play important roles in both the multiplicative updates of this section and the weight- balantt-bbniingg flows of section 3.3 The first of these is the per-unit regularizer
#
(W) = Σ WoP)
(6)
which we use to denote the local contribution to the norm in eq. 1 from the incoming weights at the 7th unit in the network. The second of these quantities is the stochastic matria
W
( = Σx W
(7)
4

---

Published in Transactions on Machine Learning Research (Cpteember22223)
whose columns sum to one. Note that these quantitics depend the values of and q in the fp.g -.orm, but to avoid an excess of notation, we will not explicitly indicate this dependence.
We now explain the roles played by these quantities in Algorithm I In this algorithm, the key step per update is to compute the rescaling factor as at each hidden unit the network. In terms the above quantities, this rescaling factor takes the especially simple form
Σj Ps( (W) J (W) (W(
(W)
max((p.)
a.
(8)
Intuitively, eq. S shows that the multiplicative updates have a fixed point (where a; 1 for all i) when the per-unit-regularizer in eq- 6 is stationary distribution of the stochastic matrix in eq. 77) Finally, we note that many of the computations in eqs. c can be easily parallelized, and also that the updates in eqs. 3
can be applicd in parallel after computing the rescaling factors eq. (8)
We derive the form of these rescaling factors (and particularly, the curious value of their outer exponent) Our first main result contained in the following theorem:
appendix A
Theorem (Convergence of Multiplicative Updates). For all Wo W R and bo E R", the melttiplccative updates in Algorithm converge to a global minómizer of the entry- wise p.g- nnrm
argmin | such that (W, (Wo, W.b
without changing the function computed by the network. Also, the intermediate solutions from these updates yield monotonically decreasing values for these norms.
A proof of this theorem can also be found in appendix A Note that Theorem [ provides stronger guarantees than exist for the full problem of learning via back-propagated gradients. In particular, these weight- balancing updates converge monotonically to a global minimizer, and they do not require the tuning of hyperparameters such as learning rates. One might hope, therefore, that such fixes could piggyback on top of existing learning algorithms without much extra cost, and this has indeed served as motivation for many previous studics of symmetry and symmetry-breakinn
in deep learning (Ncyshabur et al. 2015a Badrinarayanan et al. 2015 Meng et al. 2019 Stock et al. 2019: Armenta et al. 2021 Zhao et al. 2022
2.3 Demonstration of convergence
Theorem Istaess that the multiplicative updates in Algorithm| converge asymptotically to a global minimum of the norrm. But how fast do they converge in practice? Fig. plots the convergence of the multiplicative updates in Algorithm D for different values of and g and for three randomly initialized networks with differing numbers of hidden layers but the same overall numbers of input (200), hidden (3750), and output (10) units. units, 000-2000-1000-5002255
From shallowest to deepest, the networks had 200-2500-1256 200-1000-750-750--2 10000-77007700-00-500----------0 units. The networks were initialized with zero-valued biases and zero-mean Gaussian random weights whose variances were inverscly proportional the fan-in at cach unit (He et al., 2015). The panels in the figure plot the ratio | as function of the number of multiplicative updates, where |Wollp. and ||W]pp respectively the norms, defined in eq. ( of the initial and updated weight matrix. Results are shown for several values for and q.
units, and
As expected, the updates take longer to converge in deeper networks (where imbalances must propagate through more layers), but in general a high degree of convergence is obtained for a modest number of iterations. The panels show that conventionally initialized networks are (i) far from minimal as measured by the norm of their weights and (ii) easily rebalanced by sequence of rescaling transformations. Finally note that the results in Fig. 2 did not depend sensitively on the value of the random seed.
5

---

Published in Transactions on Machine Learning Research (September/2223)
Algorithm 1 Given network with weights Wo and biases bo, this procedure returns a functionally equivalent network whose rescaled weights W and biases b minimize the norm [| in eq. ① up to some tolerance 8 5>0 0. The set H contains the indices of the network's hidden units. The rescaled weights and biases are computed via a sequence of multiplicative updates of the form in eqs. 5
procedure (W,b) b) = MINNORMFIX(WW) (Wo, bo, H, P. 9, 6)
(W, b, a) f (Wo, bo. 1) repeat
D. Initialize.
for all (i.j) do
Wayl
Tiy (W) f Σ. Wall
Compute stochaslic malria.
for all i do
Pi - E, [W w))))
Compute per-unäl regularizers.
for all i E H do
Σ, (s()) (W) p:(W)
p..7
a; +-
b Compute rescaling factors.
for all (i,j) do
Wij Wiy(a,/a;)
Rescale weights.
for all i do b, - b,a, 6, a; 1 until (ax;(6)) (6:) < 8
b Rescale biases.
D Ieraae unlil convergence.
3 Learning and regularization
The rescaling symmetry in Fig. also has important consoquences for learning in ReLU networks Kunin
et al. 202T] Gluch & Urbanke 2021 Armenta & Jodoin, 2021: Neyshabur et al. 2015a Meng el al. 2019
Badrinarayanan et al., 2015 Armenta et al. 2021 Zhao et al. 2022p. The goal of learning is to discover the weights and biases that minimize the network's s loss function on data set of training examples. In this section examine the conditions for learning under which some norm regularizer of the weight matrix (e.g,, the normm remains minimal with respect to rescaling maansooraatios. Equivalently, these are the conditions for lcarning under which the incoming and outgoing weights at cach hidden unit remain balanced with respect to this norm. Our interest lies mainly in the following question: are there conditions such that the
procedure of the last section can be one and done at the outset of learning? To answer this question, we must understand whether learning and weight-balancing are complementary procedures or whether they are operating in some way at cross-purposes (e.g., the former undoing the latter).
Our ultimate goal in this section is to derive the weight-bblanccin flows in eq. Section 3.1 reviews the basic properties of gradient flow, while section 3.2 derives the more general flows for weight-balancing, Our main results, stated in Theorems ! and 5 are that these flows descend in a regularized loss function while preserving the minimality of the regularizer with respect to rescaling transformations. Finally, section 3.3 analyzes the specific forms and properties of these flows for regularizers based on the e in eq.
3.1 Gradient flow
There are many forms of learning in deep networks. Perhaps the simplest analyze is gradient flow (Elkabetz) & Cohen 2021). ). in which the network's parameters adapted in continuous time along the negative gradient of the loss function. Gradient flows can be derived for any differentiable loss function, and they are widely used to study the behavior of gradient descent in the limit of small learning rates.
There are two properties of gradient flow in neural networks that we will need to generalize for the minimality- preserving flows eq. ( The first is the property of nescent-namely, that the loss function of a network decreases over time. This simple to demonstrate for gradient flow. Let O (W, b) denote the weights and biases of the network, and suppose that the network is trained to minimize loss function L(O). Then
6

---

Published in Transactions on Machine Learning Research SSeptember2222)
0)-9] (1,
(p. g) (1,2)
(p.g) (1,
7-ayer layer 3-laee
7-aayer
7-ayer
5 S-layer
ayyyr 3-ayer
0.9
3-ayer
0.9
W p.q
0.8
Wg
p
0.4
0.7
0.2
0.6
30
40
10
30
40
(p:gg (2, 1)
(p: g) (2,
(p.g)
(2,
7-ayer Saayyrr 3-aaer
7-layer S S-layer
7-ayer
0.9
0.95
S-ayer
3-ayer
3-ayer
W p.9
0.8
0.6
Wo p.
0.7
0.85
0.4
0.6
0.8
0.2
0.5
0.75
20
30
10
20
20
30
(p:gg
(3, 1)
(p. g)
(3,
(p.gg
(3,3) 3)
1
7-layer 5-layer l-aayer
7-layer
7-layer
0.9
S-layer
5-ayer
0.8
3 3-layer
3-ayer
W B-
0.8
0.6
W.
p-
0.6
0.5
20
30
40
10
20
20
30
updates
updates
updates
Figure 2: Convergence of reight-balancing multiplicative updates in networks with differing mumbers of hidden layers but the same overall numbers units. The updates minimize the -orm the weight matrix, and each panel shows the results for a different set of values for and q. See text for details.
under gradient flow, non-stationary points of the loss function, we see that
dC OL de Oc
ac
0.
(10)
dt 0 dt
ae
From eq. 110 it is also clear that the fixed points of gradient flow correspond to stationary points of the loss function. In the next section, we will see how these properties are preserved in more general flows.
Another important property of gradient flow emerges in foodforward networks with homogencous activation functions. This property is consequence of the rescaling symmetry in Fig. [ when such network is trained by gradient flow, their rescaling symmetries give rise to conservation laws, one at each hidden unit of the
network (Du et al.] 2018 Kunin et al.) 2021 Bronstein et al. 2021 Gluch & Urbanke, 2021 Armenta & Jodoin 2021b. These conservation laws do not depend on the details of the network's loss function, requiring only that it is also invarian under the symmetry group of rescaling transformations. In particular, for any such loss function, it is known that the quantity
A:
Σ W
Σ 3
W
(11)
is conserved under gradient flow at each hidden unit i H. The connection between symmetries and conservation laws is well known in physical systems from Nocther's 's theorem (Noether 1918). but it is worth noting that the dynamics of gradient flow were not historically derived from a Lagrangian.
Typical regularizers break this invariance, but possible construct those that Neyshabur al. 201

---

Published in Transactions on Machine Learning Research (cpttember2202))
Our next step is to derive the conserved quantities in eq. 111 as special case of more general framework. For now, we wish simply to emphasize the logic that has been used to derive many conservation laws in neural networks: to start, one assumes that the network is trained by gradient flow in its weights and biases, and then, inspired by Nocther's s theorem in physical systems, one derives the conserved quantitics that are implied by the network's s symmetriess. In the next section, we shall flip this script on its head.
3.2
flows
In this section we investigale a larger lamily of flows for leedforward networks with homogeneous activation functions. We assume that the loss functions for these networks take a standard form with two competing terms one that measures the network's predictive accuracy on the training data, and another that serves regularizer. For example, let vt) denote training set of T labeled examples, and let C(y, f(x)) denote the cost when network's actual output f(x) is evaluated against some reference output y. We use
E(O) - - Clyi, f(x:))
C(y,,
(12)
to denote the error obtained by averaging these costs over the training set. Likewise, we assume that the network's overall loss function is given by
(( E + AR(W),
(13)
where R(W) is regularizer that penalizes large weights and the constant À 2 0 determines the amount of regularization. Note that while the error in eq. [12) may depend in a complicated way on the network's weights and biases, it is necessarily invariant to the rescaling transformations of these parameters in eqs. - 5
On the other hand, typical regularizers are not invariant to rescaling transformations of the weights, and these are the sorts of regularizers that we consider here.
When regularizer penalizes large weights, it is not only useful to prevent overfitting: in a feedforward network with rescaling symmetrics, it also induces natural way mcasure if (and how well) the weights are balanced. Consider the effect of rescaling transformation on the weights at any hidden unit i E H. A rescaling transformation with < redistributes the magnitudes of these weights in a forward direction (from incoming weights to outgoing weights), while a rescaling transformation with a; > will redistribute the magnitudes of weights in a backward direction (from outgoing to incoming weights). We formalize these ideas with the following definition so that we can speak interchangeably of balanceduess and slalionarity:
Definition 2 (Weight balancodness). In feedforward network with homogencous activation functions, we suy that the weights are balanced with respect to the regularizer R(W) if the regularizer is stalionary with respect to infinitesimal instances of the rescaling trensformations of the form in eg.
In section 2 we examined the minimality of the norm in eq. 1 with respect to rescaling transformations. The main ideas of this section, however, do not depend on that specific choice of norm, and therefore in what follows we state them in the more general terms of the above definition. To bogin, we describe more precisely what it means for the regularizer R(W), or indeed any differentiable function of the weight matrix, to be stationary at some point with respect to infinitesimal rescaling transformations.
Lemma 3 (Stationarity). Let K(W) be a differentiable function of the weight matria in feedforward net- work with homogeneous activation functions. Then K. is slationary with respect to rescaling trunsformations at W if and only if
OKC 0 Σ We ow;;
OK Wj aW;))
for all i H.
(14)
Proof. Consider rescaling transformation of the form a; = 1 + da; for all E H. where da, denotes an infunttesiall variation. Under such a transformation, the weights of the network are rescaled via eq. (4) as
The regularizer penalizes large weights, but not large blases, as only the former cause the outputs of the network depend sensitively on the imputs the network.
8

---

Published in Transactions on Machine Learning Research (September/2223)
Wy + 50;)/(1 (1 + day); thus to lowest order, their change is given by oW,, Wiy(da; (aad), Now we can work out (again to lowest order) the change &K = K(W (W) K(W) that results from this variation:
8K = Σ
Ok aw,, 5
Σ fj
Oc
W (ōa; (bū;) ) = Σ oa; Σ
aic aw.;
We
aw, jr W5s)
OK
(15)
aw, 5j
The lemma follows casily from this result. cq. 14] is satisfied, then is clear from eq. 15) that 5K. 0, proving one direction of implication; likewise if åÆ 0 for arbitrary infinitesimal variations dai, then each coefficient of da; in eq. 15 must vanish, proving the other direction.
Regularizers were originally introduced to avoid overfitting (Goodfellow et al. 2016 but it is now widely appreciated that they also serve other purposes. It has been observed, for example, that regularizers can help to learn better models on the training data (Krizhevsky et al. 2012). suggesting that smaller weights in deep networks lead to better behaved gradients. Likewise, it has been observed that highly unbalanced weights lead to much slower training in ReLU networks; the reason is that partial derivatives such as aL/aW,, scale inversely as the weights under rescaling transformations (Ncyshabur et al., 2015a Dinh et al., 2017]. More generally, it has been argued (van Laarhoven 2017) that "by decreasing the scale of the weights, weight. decay increases the effective learning rate" and that "if no regularization is used the weights can grow unbounded, and the effective learning rate goes to 0." All of the above observations suggest role for learning algorithms that actively balance the weights of feedforward networks with homogeneous activation functions. Such a algorithms can in turn be derived from the weight-balaneing flows of the following theorem.
Thcorem 4 (Weight-
(Weight-balancing balancing flows). Let ((O) denote the regularized loss eq. (13). and consider a network whose weights are inilially balanced with respect to the vegularizer R(W) and whose biases al hidden units are initialized at zero and never adapted. Then the weights will remain balanced with respect to R(W) if they evolve as
d OR dt aW )
OC
Wis
- Wij ow
(16)
The theorem shows how to construct canonical, Before proving the theorem, we emphasize the precondition that the biases of all hidden (but not output) units are frozen at zero: i.e., b, = b = 0 for all E H. As we shall see, this requiremen arises because we defined balancedness with respect to a regularizer R(W) that penalizes large weights but not large biases.
flow for any differentiable regularizer.
Proof. By Lemma 3 the weights are balanced with respect to the regularizer R(W) if they satisfy the stationarity conditions (substituting R for KC) in eq. (1).. We prove the theorem by showing that the zeros in these stationarity conditions are conserved quantities analogous to those in eq. 11 As shorthand, let
OR Wi ow,; W5 ow,
OR
Q ===
(17)
denote the imbalance of incoming and outgoing weights at a particular hidden unit. To prove that Q, is conserved, we must show that Q = 0 when the weights of the network evolve under the flow in eq. 16) from an initially balanced state. It follows from eq. (16) that
dQ, = Σ( Wis Oc
OC WH ow,;
(18)
dt
aw,,
13 the network's s loss function L(O) consists of two terms, empirical error E(O) and a regu- larizer R R(W). Substituting these terms into 18 we obtain
From eq.
dt - dQ. -Q=---
-- Wy aW
3g
W
ac aw
Σ
A
OR - W aw W
OR ow,
(19)
TThere other reasons besides weight--allaccnng learn ReLU networks with zero biases. It has been noted that
biases necessary to learn intenssity-cuuivarii representatious of scnsory inputs [Hinton al. 2011 Mohan et 2020):
these representations which the hidden-layer activations scale proportion the intensity visual anditory signals. Such net works also have certain margin-maximian properties when they are trained by gradient flow (Tyy & Ti 2020
9

---

Published in Transactions on Machine Learning Research (eeptember/2023)
Consider the left sum in eq. 19 If the hidden units do not have biases, then {()) is invariant under (and
hence stationary with respect to) rescaling transformations of the weights alone, and the first term vanishes
by Lemma 3 (substituting & for K.). Now consider the right sum in eq. (19 we see from eq. 17 that this term is proportional to Qi itself, so that
dQ; -QQ; dt
(20)
Finally, we observe that = at time 4=0 the weights are initially balanced. In this case, the only solution eq. 220 is the trivial one with Q Q,=Q,=0 =0 for all time, thus proving the theorem.
The calculation in this proof yields another insight. From eq. 20 we see that Q, decays exponentially zero if the weights are not initialized in balanced state. The decay is caused by the regularizer, effect that Tanaka & Kunin (2021) describe as the Nocther lcarning dynamics. However, the decay may be slow for the typically small values of the hyperparameter >> 0 used in practice. Taken logether, the fixes and flows in this paper can be viewed as a way of balancing the weights at the outset and throughout the entire course of learning, rather than relying on the asymptotic effects of regularization to do so in the limit.
A weight-balancing flow only useful insofar as it also reduces the network's s loss function. The weight- balancing flows in eq. (16) do not strictly follow the gradient of the loss function in eq. (13 Nevertheless, our final theorem shows that under relatively mild conditions these flows have the same property of descent.
Theorem 5 (Balanced descent). Let Wij = log WWiyl- Then the weight-balancing flow in eq. 16] descends everywhere that the loss function is not stationary with respect to w and the regularizer has a positive definite Hessian with respect to w:
dC
Oc dw
R²R
< 0 wherever
and
Owdw T > 0.
(21)
dt
Before proving the theorem, we emphasize that eq. 21 refers to the Hessian of the regularizer R(W), not the Hessian of the network's overall loss function L (W). As we shall see, the former very well behaved
for typical regularizers, while the latter (about which one can say very littl) depends in a complicated way the network's fit to the training data.
Proof. The property of descent emerges for these flows in similar way for other generalized flows (Wibisono et al.] 2016 Tanaka & Kunin 2021j. We begin by observing that the flow in eq. 16
takes a simpler form in terms of the variable w. Since Wij = log Wiyl, we have equivalently that W 2015
and differentiating we find /W.//Ow; Wiz. We can use the chain rule to differentiate in eq- respect to w instead of W. In this way we find (in vector notation) that
16 with
d (RR) dw
Oc
(22)
Ow
This equation specifics implicitly how the weights evolve in time through the form of the regularizer, R R(W). To derive an explicit form for this evolution, we differentiate through the left side of the equation:
d OR dw
82R du
(23)
dt
aw ow
Note how the Hessian of the regularizer appears in this equation; as shorthand in what follows we denote 8º this Hessian by H(w)= aw aw . Now suppose that H(w) is positive definite (hence also invertible) at the
current values of the weights. Then combining eqs. 22 23
we see that
dw dt
OC
(24)
Ow
In sum, we have shown that if H(w) is positive definite, then the same dynamics as eq. 24). Thus can also interpret these dynamics generalized gradient flow in
flow in eq. [6] has the
10

---

Published in Transactions on Machine Learning Research (eeptember/2023)
which the weights reparameterized in terms of Waj log [Wa] and the gradient is preconditioned by the inverse Hessian H (w). Now suppose further that the gradient 0L/ /ww does not vanish. Then we have
dC ac du dt
ac
dc
= dw di
H (w) 8w
dw < 0.
(25)
This suffices to prove the theorem, but further intuition may be gained by comparing the argument in eq. 25 to the analogous one for gradient flow in eq. (10). The main differences are the appearance of the inverse Hessian preconditioner (via the regularizer) and the change of variables (from Waj to wij).
3.3 Flows for f..g- norm regularization
In the last section derived weight-balanciie flows for any regularizer R(W). The derivation general, assuming only that the regularizer was differentiable. In this section we investigate the flow in eq. (16) for regularizers based on the norm in eq. 11. Specifically, we consider regularizers of the form
R(W) q Σ
Σ W W.p
(26)
The flows induced by these regularizers have several interesting properties, which briefly summarize. Proofs and further results can be found in appendix
To build intuition, we begin by noting some special cases of interest. The weight-balancing flow in eq. 16
reduces to familiar forms for the simplest choices of and in eq. (26). For example, when p= 2, this flow simplifies (for nonzero weights) to Wij = a.."
BC which can be approximated by gradient descent
OC
Wi Wi 17 aw
(27)
with a small learning rate r) >>>0 0. On the other hand, when p == 1, the flow eq. 16 simplifies to W., - -,,,,,,, which can be approximatod by the exponentiated gradient descent aW,,
dc Wi + We exp (-1) (-n- sign(W.;) aw
(28)
Similar updates based on exponentiated gradients have been studied in many different contexts (Kivinen & Warmuth 1997 Arora et al. 2012 Bernstein et al. 2020 With these special cases in mind, we now state the main result of this section.
Theorem (Almost everywhere descent). If the R(W) is based on the lp.- as in eq. 266,, then the beiight-bllnccig flow in eq. 16 decreases the regulorized loss in every open orthant of weight space.
Theorem ① establishes the property of descent for nnight-balancing flows based on f rogularization. The proof of this theorem requires two steps first, to compute the Hessian for this regularizer, and second, to show that it is positive definite in every open orthant of the weight space. These steps are presented in appendix B
4 Related work
The results in this paper build on earlier work. Most relevant to section is the work on ENorm Stock et al. 2019 an elegant procedure, also based on multiplicative updates, to minimize the no-mor of the weights ReLU networks. The updates for ENorm are derived from block coordinate descent of eq. (26) with respect rescaling transformations for the special case y= the blocks are formed by grouping hidden units in the same layer. The updates in this paper are derived in different way from so-called auxiliary function,
11

---

Published in Transactions on Machine Learning Research (September/2023)
approach used in algorithms for nonnegative matrix factorization (Lee & Seung 2000) and Expectation- Maximization (Dempster et al. 1977). This approach (see appendix [A] has two notable features. First, it leads to updates that are applied in parallel at all of the network's hidden units, and thus unlike those derived from block coordinate descent, they are not specialized to layered networks whose structure suggests a natural partition (i.e., blocking) of the hidden units. Second, it yields closed-form updates to minimize the p..-normm despite the more complicatod gradients that arise when p This more general case is of interest because the maz-norm emerges from eq. in the limit 40000 Neyshabur et al. 2015a).
Our results in section 3 also build on previous work. Most relevant here is the work by Tanaka & Kunin (2021)
that analyzes symmetry-breaking in deep learning via the contimuous-time dynamics of damped Lagrangian
systems. Their work gives an elegant and broadly unifying treatment for overparameterized networks with general symmetry groups. It builds in turn on the highly inftuential work of Wibisono et al.] (2016). who
introduced Bregman Lagrangians to analyze accelerated methods for optimization. (Taaka & Kunin (2021) stated that "a future direction to extend our analysis to the case of rescale symmetry of the homogencous activation functions such as ReLU." In this paper we have not made use of Bregman Lagrangians; instead we have given proof of Theorem 4 stripped down to its bare essentials. To do so, derived the flows of
eq. (16 in a bottom-up fashion, starting from the requirement that balanced weights should remain balanced, rather than top-down fashion, starting from a time-dependent Lagrangian and deriving its Euler-Lagrange equations. In the latter approach, the Bregman kinetic energy plays the role of the regularizer, breaking the rescaling symmetry of the Lagrangian and undoing the conservation law for the Noether charge Qi in eq. [17. We hope that some readers will appreciate the different path we have taken to obtain these results.
Many other studies have also investigated the relationship between rescaling transformations and learning. In seminal paper, Neyshabur et al. (2015a) showed that stochastic gradient descent (SGD) performs poorly in highly unbalanced networks, and in its place, they proposed PathSGD, a rescaling- invariant procedure that approximates steepest descent with respect to a special path-based regularizer. Notably, this regularizer has the distinguishing property that it computes the minimum value of a max-norm regularizer, where the minimum is performed over all networks equivalent up lo rescaling (Neyshabur el al.] 2015b). PathSGD was followed by other formulations of rescaling invariant learning. For example, [adrinarayanan et al. (2015) fixed the rescaling degrees of froedom by constraining weight vectors to have unit norm, while Meng et al.) 2019) showed how to perform SGD in the vector space of paths (as opposed to weights). where the rescaling- invariant value of path is given by the product of its weights. Also, Stock et al. (2019) trained networks by interweaving updates from SGD and ENorm with fn-norm regularization; interestingly, the networks in these experiments generalized better on test data. This improvement suggests to interweave the more general updates section 2, minimizing the formm with learning rules based on their corresponding flows in section 3. Our study lays the foundation for this further exploration.
Even more recent work has suggested that learning can be accelerated by certain rescaling transformations. For instance, Armenta et al. [2021 showed that the magnitudes of backpropagated gradients in ReLU net- works are increased on average by randomly rescaling their weights-a process they call neural teleportation. More generally, Zhao et al. (2022) explored how to choose symmetry group transformations that purposefully increase the norms of gradients for learning. Because these gradients are computed with respect to training examples, this approach can be viewed as a data-driven procedure for manipulating the optimization land- scape via symmetry group transformations. Our approach differs from the above by aiming to minimize the norms of network's weights rather than to maximize the norms of its gradients. We note that the latter are unbounded above with respect to the (non-compact) group of rescaling transformations, and therefore one must be careful to identify the regime in which they serve as a reliable proxy for rates of convergence.
5 Discussion
Many aspects of deep learning are not fully understood. In this paper we have shown that further un- derstanding may be gained from the symmotries of multilayer networks and the analogies they suggest to physical systems. We have leveraged this understanding in two ways. First, we derived simple multiplicative updates that minimize the l norm of the weight matrix over the equivalence class of networks related by
12

---

Published in Transactions on Machine Learning Research (eeptember/2023)
rescaling transformations. Second, we derived weight-balancing flows that preserve the minimality of any (differentiable) regularizer over the course of learning.
There are many questions deserving of further investigation. Most obviously, weight-balanccing flows can be discretized to yield learning rules other than gradient descent. One important question is how to combine such learning rules with accelerated gradient-bae or adaptive learning schedules (Kinggma & Ba 2015 [uchh et al. 2010 Tieleman & Hinton 2018), that are used in all large-scale experiments. Another is how Bright-balancing relates to (or may be incorporated with) schemes such as batch normalization (Loffe & Szegedy 2015), weight normalization (Salimans & Kingma, 2016), and layer normalization (Ba et al. 2016). It will require mix of empirical and thcoretical investigation to undersland the interplay of these methods (van Laarhoven
methods, such as those involving momentum (Polyak] 1964)
2017). In this work, we have not relied heavily on the machinery of Bregman Lagrangians (Wibisono et al., 2016 Tanaka & Kunin 2021 but it is clear that they provide a powerful framework for further progress.
Other potential benefits of weight-balancing are suggested in the more familiar setting of matrix factoriza- tion (Horn& & Johnson] 2012). The basic problem of factorization is underdetermined: any matrix can be written in infinite number of ways the product of two more other matrices. But there are certain canonical factorizations of large matrices, such as the singular value decomposition, that reveal a wealth of information (Eckart & Young, 1936). It. is natural to ask whether the functions computed by multilayer net- works can be represented in a similarly canonical way, and so, whether such representations might suggest more effective strategies for pruning, compressing, or otherwise approximating their weight matrices.
Finally note that there are many possible criteria for weight-balancing besides minimizing the -oorm of the weight matrix. It would be interesting to study other regularizers and their corresponding flows from Theorem 4 We believe that the present work can provide a template for these further investigations and also that such investigations will reveal a similarly rich mathematical structure.
Acknowledgements
The author is grateful to the reviewers and action editor (N. Cohen) for many helpful suggestions. He has also benefited from discussions of these ideas with many colleagues, including A. Barnett, A. Bictti, J. Bruna, M. Eickenberg, R. Gower, B. Larsen, E. Simoncelli, S. Villar, N. Wadia, A. Wibisono, A. Wilson, and R. Yu.
References
P. W. Anderson. More is different. Science, 177(404))3393 3960339666-6 1972.
M. Armenta and P.-. Jodoin. The representation theory of neural networks. Mathematics, 9(24). 2021.
M. A. Armenta, T. Judge, N. Painchaud, Y. Skandarani, C. Lemaire, G. G. Sanchez, P. Spino, and M. Jodoin. Neural teleportation, 2021. ar 2xXiv:2012.01118.
S. Arora, E. Hazan, and S. Kale. The multiplicative weights update method: a meta-algorithm and appli- cations. Theory of Compuling, 8(1):121 164, 2012.
S. Arora, N. Cohen, N. Golowich, and W. Hu. A convergence analysis of gradient descent for deep linear neural networks. In Proceedings of the 8th International Conference on Learning Representations, 2019.
J. Ba, K. Lei, J. Ryan, and G. F. Hinton. Layer normalization, 2016. arxiv:1607.064550
V. Badrinarayanan, B. Mishra, and R. Cipolla. Understanding symmetries in deep networks. In Proceedings of the 8th NeurIPS Workshop Optimization for Machine Learning, 2015.
J. Bernstein, J. Zhao, M. Meister, M.-Y. Liu, A. Anandkumar, and Y. Yue. Learning compositional functions via multiplicative weight updates. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems 33, pp. 13330, 2020.
S. Boyd and I. Vandenberghe. Conver Optimization. Cambridge University Press, 2004.
13

---

Published in Transactions on Machine Learning Research (epttember/2223)
M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličkovié. Geometric deep learning: grids, groups, graphs, geodesics, and gauges, 2021. arxiv:2104.33478.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39:1399: 38, 1977.
L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp minima can generalize for deep nets. In D. Precup and Y. W. Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, pp. 1019-1028. 2017.
S. S. Du, W. Hu, and J. D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Bengio, H. M Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (ods.), Advances in Neural Information Processing Systems 31, pp. 382-393, 2018.
J. C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Proceedings of the 23rd Conference Learning Theory, pp.
2010.
C. Eckart and G. Young. The approximation of one matrix by another lower rank. Psychometrika, 1(3): 211-218, 1936.
O. Elkabetz and N. Cohen. Continuous vs. discrete optimization of deep neural networks. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan (eds.), Advances in Newral Information Processing Systems 34, pp. 4947-4960, 2021.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In G. Gordon, D. Dunson, and M. Dudík (eds.), Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 315 323, 2011.
G. Gluch and R. Urbanke. Noether: the more things change, the more they stay the same, 2021. arXiv:2104.05508.
I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning, MIT Press, 2016.
J. Goodfellow, D. Warde Warde-Farley, M. Mirza, A. C. Courville, and Y. Bengio. Maxout networks. In Proceed- ings of the 30th International Conference on Machine Learning, pp. 1319-1327, 2013.
D. J. Gross. Gauge theory -asttppstttt present, and future? Chinese Journal of Physics, 30:955-977 972, 1992.
F. V. Gubarev, L. Stodolsky, and V. I. Zakharov. On the significance of the vector potential squared. Physical Review Lellers, 86(11):2220 2222, 2001.
Gunawardana and W. Byrne. Convergence theorems for generalized alternating minimization procedures. Journal of Machine Learning Research, 6:2049-2073, 2005.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision, pp. 1026-1034, 2015.
G. E. Hinton, A. Krizhevsky, and D. Wang. Transforming auto-encoders. In Proceedings of the Interna- tional Conference on Artificial Neural Networks, pp. 44 44-51, 2011.
R. A. Horn and C. R. Johnson. Matriz Analysis. Cambridge University Press, 2012.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. R. Bach and D. M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 448 148-456,44 2015.
J. D. Jackson. From Lorenz Coulomb and other explicit gauge transformations. American Journal of Physics, 70:917-922 1221, 2002.
14

---

Published in Transactions on Machine Learning Research (eeptember/2233)
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun (eds.). Proceedings of the 3rd International Conference on Learning Regresenlations, 2015.
J. Kivinen and M. K. Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, (32(1):1-63) 1997.
A. Krizhevsky, I. Sutskever, and G. F. Hinton. Imagenet classification with deep convolutional neural net- works. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1106-1114, 2012.
D. Kumin, J. Begastuy--rrñaa S. Ganguli, D. L. K. Yamins, and H. Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In Proceedings of the 9th International Conference on Learning Representations, 2021.
D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401: 788-791, 1999.
D. D. Lee and H. S. Seung. Algorithms for non- negatveee matrix factorization. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems 13, pp. 556-562. MIT Press, 2000.
K. Lyu and J. Li. Gradient descent maximizes the margin of homogencous neural networks. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Q. Meng, S. Zheng, H. Zhang, W. Chen, Q. Ye, Z.-M. Ma, N. Y, and T.-Y. Liu. G-SGD: Optimizing ReLU neural networks in its positively scale- invariant space. In Proceedings of the 7th International Conference on Learning Representations, 2019.
R. R. Meyer. Sufficient conditions for the convergence of monotonic mathematical programming algorithms. Journal of Computer and System Sciences, 12(1):108-12) 121, 1976.
S. Mohan, Z. Kadkhodaie, F. Simoncelli, and Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In Proceedings of the 8th International Conference on Learning Representations, 2020.
B. Neyshabur, R. Salakhutdinov, and N. Srebro. Path-SGD: Path-normalized optimization in deep neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neurul Information Processing Systems 28, pp. 2422 2430, 2015a.
B. Neyshabur, R. Tomioka, and N. Srebro. form-based capacity control in neural networks. In Proceedings of the 28th Conference on Learning Theory, pp. 1376-1401, 2015b.
E. Noether. Invariante variattionsprobleme. Nachrichten von der Gesellschaft der Wissenschaften Göttin- gen, Wathemnatichh-Plyssalllll Klasse, pp. 235-257, 1918.
B. T. Polyak. Some methods of speoding up the convergence of itcration methods. USSR Computational Mathemalics and Mathemalical Physics, 4(5):1 17, 1964.
T. Salimans and D. P. Kingma. Weight normalization: a simple reparamcterization to accelerate training deep neural networks. In D. D Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 901-909. Curran Associates, Inc., 2016.
L. K. Saul, F. Sha, and D. D. Lee. Statistical signal processing with nonnegativity constraints. In Proceedings of the 8th European Conference on Speech Commanication and Technology, pp. 1001-1004, 2003.
A. M. Saxe, J. L. McClelland, and S. Ganguil. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Y. Bengio and Y. LeCun (eds.), Proceedings of the 2nd International Conference on Learning Representations, 2013.
F. Sha, Y. Lin, K. Saul, and D. D. Lee. Multiplicative updates for nonnegative quadratic programming. Neural Compulation, 19:2004-2033) 2007.
15

---

Published in Transactions on Machine Learning Research (September/2233)
B. K. Sriperumbudur and G. R. G. Lanckriet. A proof of convergence of the concave-convex procedure using zangwill's theory. Newral Compulation, 24:1391-1400 2012.
P. Stock, B. Graham, R. Gribonval, and H. Jégon. Equi-normalization of neural networks. In Proceedings of the 7th International Conference on Learning Representations, 2019.
H. Tanaka and D. Kunin. Noether's learning dynamics: Role of symmetry breaking in neural networks. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vanghan (eds.), Advances in Neural Information. Processing Systems 34, pp. 25646 25660, 2021.
Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.); Aduances in Newral Information Processing Systems 39, pp. 6377-6389, 2020.
T. Tieleman and G. E. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a rumning average of its recent magnitude. COURSERA: Newral networks for machine learning. 4(2):26-33 31, 2018.
T. van Laarhoven. L2 regularization versus batch and weight normalization, 2017. 1xxiv:1706.04340
A. Wibisono, A. C. Wilson, and M. I. Jordan. A variational perspective on accelcrated methods in optimiza- tion. Proceedings of the National Academy of Sciences USA, (3747)))7731-E77555 2016.
C. F. Wu. On the convergence properties of the EM algorithm. Annals of Statistics, 11(1):95 103, 1983.
A. L. Yuille and A. Rangarajan. The concave-conver procedure. Neural Compulalion, 15:915 55:91--936, 2003.
W. J. Zangwill. Nonlinear programming: A unified approach. Prentice Hall, 1969.
A. Zee. Fearful Symmelry: The Search for Beauly in Modern Physics. Princeton University Press, 2016.
B. Zhao, N. Dehmamy, R. Walters, and R. Yu. Symmetry teleportation for accelerated optimization. In
S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Infor- mation Processing Systems 35, 2022.
A Convergence of multiplicative updates
In this appendix we prove that the multiplicative updates in Algorithm 1 minimize the lp.g norm in eq. The proof has many technicalities, so we begin by developing further intuitions in section A. which analyzes the fixed points of these updates, and section A.2 which relates the updates to those considered in previous work. Finally, scction A.3 gives a formal proof of convergence.
A.1 Analysis of fixed points
It is clear that Algorithm | can only converge to fixed points of the multiplicative updates. We begin with an elementary observation of where these fixed points occur.
Proposition (Fixed Points). If weight matria W a fixed point of the maltippliaative updates in Algorithm 7 then all of the rescaling factors in eqs. are equal to unily: i.e., 04 =1 for all i.
Proof. We proceed by induction. Note that a₁ because the network must have at least one input unit and the rescaling factors at all non-hidden units are fixed to one. Let i> and suppose that a, =1 for all <i. The 5th unit in the werw is either hidden (i E H) or not hidden (i g H). If the latter, then as before we have trivially that a, =. If the former, then there must exist some j such that Wy 0 (since otherwise the unit would be unaffected by the network's inputs, a condition that we exclude). Note that the weight Wy is multiplicatively rescaled eq. ( by the ratio a./ay. Since Wiy # 0, fixed point occurs only when a= =a and since aj =1 by the inductive hepothesis, it follows that a; =1
16

---

Published in Transactions on Machine Learning Research (cpttember2023)
Next consider what the fixed points of Algorithm signify. Recall that the updates were introduced to minimize the norm in eq. (1.. The next lemma shows that the fixed points correspond to global minima.
Lemma (Global Optinality). If weight matric W is fixed point of the mulliplicativee updotes in Algorithm 7 then there exists no weight matris W W such that [ < [ lp.g-
Proof. Suppose that W' W with W (ay(as/a,) for some rescaling transformation a E A, and consider how | depends on the rescaling factors of this transformation. This dependence is captured (up to a multiplicative constant) by the continuous function F: + R, where
F(a) - E ( ((sl/(a)))))) ([
(29)
It instructive to examine the dependence of this function on the logarithms of the scaling factors, log a. rather than the scaling factors themselves. Doing so, we find:
F(a) = : exp log Σ e loga; an tin Wass ))
(30)
It follows from basic composition laws of convex functions [Boyd Vandenberghe 2004 that || W is itself a convex function of the vector loga = (log a1, log o2,,.., 0n), and this turn implies that any stationary point of F is global minimizer of F. To locate such point, we examine where the partial derivatives of F vanish. Starting from eq. 29 tedious but straightforward calculation gives
OF 1 (I W. (ibP((a///s)))) - Σ 0a; ag
8-1 Σ W W ((a//aa))))))) (WyiP(as//a))
(31)
Now suppose that W is a fixed point of the multiplicative updates. Then by the previous proposition, it must be the case that the right hand side of eq. (8) is equal to unity. This occurs when the denominator and numerator of eq. are equal, or equivalently when
E EINoP) = ((EW)))) W.
(32)
Eq. 132 can be viewed as balancing condition that holds at fixed points of the multiplicative updates: it equates a sum over incoming weights (on the left) with a sum over outgoing weights (on the right). Comparing the last two equations, we see another consequence of this balance; it implics that gradient of
F in eq. (31b vanishes when a 1, where R is the vector of all ones. But this implies that global minimum of F(a) is obtained when is the identily transformation. In this case, by definition there exists a' E A such that F(a') < F(1), equivalently, there exists W W such that ||'lpp < || J
To summarize, we have not yet shown that the updates in Algorithm I converge. But we have shown that if they do converge, they yield functionally equivalent network whose weights minimize the norm in eq. (1)
A.2 Relation to ENorm
Consider the special case of the norm in eq. 1 when p=g. This special case leads to many simplifications. When p=g, for example, the partial derivative in eq. 31 reduces to
aF 1 Σ Da, a;
Wa)P(a//a)
Σ Iissl((a,/aa))
(33)
17

---

Published in Transactions on Machine Learning Research (eeptember/2223)
This expression is sufficiently simple to pursue strategy of coordinate descent, minimizing F(a) with respect to one rescaling factor at time. In particular, consider the rescaling factor d; at some hidden unit (i E H), and suppose that aj = for all j + Then the partial derivative in eq. (33 vanishes when
04 Σ, W Wiyp
(34)
Eqq (34) is the basis for the ENorm algorithm (Stock et al. 2019 which uses multiplicative updates with rescaling factors, as in eqs. 5 to minimize the fer--norm of the weight matrix. The updates for ENorm are derived from block coordinate descent of eq- (29 for the special case p= the blocks are composed of hidden units in the same layer of a deep network (for which the partial derivatives in eq. (33) decouple). As its name suggests, the procedure aims to equalize the p-norms of incoming and outgoing weights.
When p=q, the astute reader may have noticed mismatch in the outer exponents of eqs. and 34 0 the
former is 1 for the multiplicative updates in this paper, while the latter is 1 for ENorm. The origin of this 4p discrepancy, by factor of two, will be further explained in section A.3 Át a high level, the discrepancy arises because ENorm is based on block coordinate descent whereas the updates in this paper are applied in parallel at all the hidden units of the network. The larger exponent of ENorm corresponds to a more aggressive update, but one that only applied to the rescaling factors in one layer of hidden units. The smaller exponent in our approach needed to provide a similar guarantee of monotonic convergence when all the rescaling factors are updated in parallel. Compared to previous work, our results can be viewed extending the reach of parallelizable multiplicative updates in two directions- first, to the larger family of feedforward (but not strictly layered) networks, and second, to the larger lamily of norms when p*4-
2p
A.3 Proof of convergence
Theorem is proved by showing that the multiplicative updates in Algorithm ! satisfy the preconditions for Meyer's convergence theorem (Meyer) 1976). Meyer's result itself builds on the convergence theory of Zangwill 1969).. Our tools are similar to those that have been used to derive the convergence of other
multiplicatiie updates with nonnegativity constraints (Lee & Seung. 1999 Saul et al. 2003: Sha el al. 2007)
as well as more general iterative procedures in statistical learning (Dempster et al. 1977 Wu. 1983
Yuille
& Rangarajan 2003
Gunawardana & Byrne 2005 Sriperumbudur & Lanckrict 2012).
We prove the theorem with the aid of two additional lemmas. The first lemma considers the effect of single rescaling transformation in the bagght-balaniig procedure of Algorithm [ This lemma is at the heart of the algorithm: it shows that each multiplicative update reduces the entry-wise mormm of the weight matrix.
Lemma 9 (Monotone Improvement). Suppose WW,, and specifically, suppose Wy = W.y where a E A is the veclor of rescaling factors whose elements for i E H are given by eq. Then | p.g < W lp and this inequality is strict unless wW
Proof. Recall the function F A + R from eq. 29 and recall in particular that F(a) = -- [
p.g
if W = W.y(as/az). To prove the lemma, we must show equivalently that F(a) S (1) when a E A is given by eq. (8) and also that this inequality is strict unless 1 The proof based on constructing a so-called auxiliary function, in the derivations of the EM algorithm (Dempster et al.] 1977), nonnegative
matrix factorization (Lee & Seung, 2000), and the convex-concave procedure (Yuille & Rangarajan 2003
Specifically, we seek an auxiliary function G : A + R with the following three properties:
(i) F(a) < G(a) for all a E A.
(ii) F(1) G(1) where 1 is the vector of all ones.
(iii) G : A + R has a unique global minimizer at a E A given by eq.
Suppose, for instance, that we can construct a function with these properties. Then at the minimizer a E A given by eq.
{8) we have at once that
F(a) & G(a) S G(1) F(1),
(35)
18

---

Published in Transactions on Machine Learning Research (epttember/2223)
where the inequalities in eq. (35 follow from properties (i) and (iii) of the auxiliary function and the equality follows from property (ii). This proves the first part of the lemma. Now if a + 1 in eqs. ( and (35). then G(a) < G(1) (because G has a unique global minimizer) and by extension F(a) < F(1). On the other hand, if a1 then W is a fixed point of the multiplicative updates, and this proves the second part of the lemma.
The more challenging part of the proof is to construct an auxiliary function with these properties. As shorthand, let max(p, q), and consider the function
G(a)= 20 ()((W) (at +aja)) + max (0,1 F(I).
(36)
We claim that eq. (36) satisfies the three properties listed above, and we verify them in order of difficulty.
First, we verify property (ii): comparing eqs. (29 and 36
see by simple substitution that F(1) = G(1).
Next we verify property (iii). We begin by observing that E, |W 0 and Σ, |W for all i E H; these conditions are necessary (as mentioned in section 2.1 to ensure that each hidden unit lies on some directed path from the network's inputs to outputs. It follows that the coefficients of oz and aj 2r in eq. (3) are strictly positive, and hence G(a) is strongly conver in the variable (ai, aui.,,,..,, a.)) By setting aG/oa =0 0 for all i E H, we obtain the solution in eq- (8] and by strong convexily this solution must correspond to a unique global minimizer. This verifies property (iii) of the auxiliary function, and it also accounts for the peculiar exponent of + where r = max(p,q), q), that appears in the multiplicative update.
Finally, we verify property (i) of the auxiliary function. To do this, we must work out separately the cases for p < q and p> q. These regimes require different arguments to establish the lower-bounding property of the auxiliary function. We also remind the reader of the definitions in eqs.
Case 1: p To verify property (i) in this case we must show that F(a) S &G(a) when páq. Starting from eq. 29 for F(a), we can derive the auxiliary function in eq. 36 via two simple incqualities:
F(a) = 1 9 Σ Σ W WerPlos P(ar/aa))
(37)
#
= 1 / > (((W)
Σ Tiy (W)
(38)
< 1
"/
E (w) Σ Tap (a)(((aa))))),
(39)
S 2n Σ (i( Fag (W) (a." + az24
(40)
= G(a) for p psq.
(41)
In eq. 33 we have used Jensen's incquality, exploiting the fact that Fig is a stochastic matrix, and in
eq. 44) we have appealed to the inequality of arithmetic and geometric means. Finally, eq. (41) follows
upon resubstituting eq. 7 into the previous line. This verifies property (i) for the case p $ q-
Case 2: p2 9. Now we must show that F(a) I G(a) when p2g. q. Here we exploit the fact that concave function (e.g., x²) lies below any one of its tangents. It follows that
$
F(a)
1 Σ
Σ
W Woy1 WiPP((s/aa) P(aa/a;))))
(42)
9
< F(1)
+ p ij
!
Σ
Σ Wisl
Woll (las/a,)) -1)
(43)
k
F(1) + 1
pel (W( May (W) (att/aa;)) -1).
(44)
p
ij
s F(1) (1 1 9)
+ 2p Σ p: (W) T.. (W) (0?" + a, 2p)
(45)
G(a) for p > q-
(46)
19

---

Published in Transactions on Machine Learning Research (eeptember/2023)
In sum, the auxiliary function G(a) is derived by identifying those parts of [W1]3, that can be bounded by elementary inequalities. In doing so, we find that different bounds are required for the cases p<q and p>> g. and these differences account for the exponent r = max(p,g) q) that appears in the multiplicative updates. ]
Lemmaa 9 shows that each multiplicative update in Algorithm decreases [ unless W is itself fixed point. The lemma also rules out oscillations between distinct global minima. But this by itself not enough to prove that the updales converge. To do this, must. also show that none of the rescaling factors increase without bound. This the content of the next lemma.
Lemma 10 (Compactness of Sublevel Scts). Let C = (W ~ W. P. S l[lll denote the sublevel set of weight matrices whose pp.--ormm does not exceed that of W. This set is compact.
Proof. Let F:A- - R be the function, as defined in eq. 29 that computes the change in the bp.0 -norm of the weight matrix after rescaling transformation. To prove the lemma, must show equivalently that the sublevel set given by F1 fa A F(a) s F(1)} is compact. It follows from the continuity of F that its sublevel sets are closed; thus it remains only to show that F1 is bounded. At high level, this boundedness will follow from the fact that the network has finite depth; similar result has been obtained for the special case p= (Soock et al. 2019). In particular, suppose a E F1 with F(a) I F(1). Then Wiz * 0, it must be the case that
04 (qF(1)) a W
(47)
Im denote an m-step path through the network that starts at some input unit (ju I), passes through the 2th hidden unit after k steps (so that jk i); ends at some output unit (jm E O), and traverses only nonzero weights W 15 * in the process. Note that there must exist at least one such path the 7th hidden unit contributes in some way to the function computed by the network. Since ajo =1 and aj. =a follows that
because otherwise the itth term of the sum in eq.
(29 would by itself exceed F(1). Let jo +
k
(QF(1))
0g
Ojs
ajo
aje Oj
II
W W
(48)
where the inequality follows from eq.
47 Likewise, since ajn. =1 and dz az it follows that
m
m
(1F
ajm
(49)
aj
aje-1
Wish-l
t=k+1
t=
Eqs. 48 49
provide upper and lower bounds on a; for all a E F1. Thus Fi is closed and bounded, hence
compact.
The previous two lemmas establish the necessary conditions for Meyer's monotone convergence theo- Meyer 1976.. Armed with these lemmas, we can now prove Theorem
rem
Proof of Theorem 1 Let Wo denote an initial weight matrix. From Lemma 10 it follows that the set C= [W/W W Wo. Wlpg s [Wollpg) compact. From Lemma g it follows that each multiplicative update yields weight matrix W C whose -oorm is less than equal to the previous one (with equality occurring only when W is both global minimizer and a fixed point of the updates). Finally, from eq. (8) we note that the multiplicative coefficients are continuous function of the weights from which they are derived. The procedure in Algorithm | therefore satisfies the preconditions of compactness, strict monotonicity, and continuity for Meyer's monotone convergence theorem (Meyer 1976) in the setting where fixed points occur at (and only at) global minima of || C.
20

---

Published in Transactions on Machine Learning Research (September/2023 22023
B Weight-balancing flow for lp.-- -prmm regularization
In this appendix we prove Theorem that the weight-balancing flow for the regularizer R(W) in eq. 26 descends in the network's 's regularized loss function. As a preliminary step, we remind the reader of the per-unit regularizer (W) defined in eq. (G) and the stochastic matrix with elements Tyy (W) defined in
eq. (7). From these definitions, it is also a straightforward exercise to verify that Wa aW
aR
(W)p:
and this identity is useful in what follows.
Proof of Theorem C Consider the regularizer R(W in eq. (66 as function of the variable Wij = log W 8-R
To prove the result, we show that the Hessian H(w) uwww in Thcorem 5 is positive definite whenever weight Wij is equal to zero, equivalently when every waj finite. The proof requires two steps first, to compute the Hessian, and second, to show that it is positive definite. For the first step, begin by evaluating the time-derivative in the flow of eq. (6]. Here we find, by repeated differentiation, that
d di
Wi
OR d di.
Tiy (W))
(50)
aW
Tmij Wikh Pi(W) + Tay (W) aWa
Op: Wek
Σ
Σ
(51)
aw sk
Wen
Wa
pra (W)
- - (W))
(W) + g (W) Σ Tak (Ws)p
(52)
Wine
Wik
&
=m)((W)p (pars + (a-p) Σ TW)
(53)
We now read off the nonzero elements of (w) from the relation between eqs. 23 and 53 any nonzero matrix of the same size W, and let Vec(V) denote the flattened representation of V as vector. Then from eqs. (23 and 53
Let V be
we have
() H(W)) VV)
p
Σ
THp Tiy (W)P: V3 + (q-p)
Σ ijk
Tey (W) ViyVi
(54)
if
Σ (: (W) (vo Σ Ta Vis
+ + 2 (W) (E)(())
(55)
0 for all V+
(56)
The strict inequality in the last line is justified by the following observations: (i) both ny (W) and (W) are strictly positive when no weights are exactly equal to zero; (ii) the left term in eq. (55) only vanishes when V has constant rows; (iii) the right term only vanishes when for all rows Σj m. mey (W),,, = 0. Since that one term
is nonzero, however, it is not possible to satisfy both of these conditions simultaneously, and /or/orro the other must be strictly positive. Finally, since Vec(V) H(w) Vec(V) 0 for any nonzero V, we conclude that H(w) is positive definite in every open orthant of the weight space. The result then follows from Theorem 5
21


Reviews:
[{"review_id": "UXbcFMc1Mr", "reviewer": "Reviewer_LFZ9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\nThis paper provides some interesting ideas about the use of gauge transformations in feedforward neural networks. In particular, the examination of rescaling symmetry and how to use them to invoke a minimum norm solution to the weights of a network are an interesting direction of research. Furthermore, the insight that weight-balancing gradient flows if initialized properly remain balanced throughout training is useful insight and motivates the development of regularizers that effectively use the rescaling symmetry. \n\n**Presentation Weaknesses**\n\nThis paper has 21 pages of main content which is significantly larger than the 12-page submission of a typical TMLR paper. This would be acceptable if there were 21 pages worth of content. This in my humble opinion is not the case. The paper is too verbose and expository and the amount of information does not justify its length. Specifically, all proofs could easily be placed in the appendix and instead could be replaced with proof sketches if necessary. Secondly, there is really no need to provide proof for corollaries---certainly not in the main text---as these should be straightforward applications of the main theorems.\n\nI also find the organization of the paper unnecessarily confusing in section 2. I'm not sure why the proof of Theorem 2.1 does not immediately follow. The authors claim to build intuition but they also provide analysis of the fixed points and it takes a much longer time to reach the proof. Either put the proof in the appendix if its not important and provide a proof sketch or provide a more linear path in section 2. This arbitrary suspense is not necessary.\n\n**Technical Weaknesses**\nThere are several technical shortcomings in this paper. I will list them in no specific order but all of them are equally important unless specified otherwise.\n\n- There are two claims about the weight balancing flows that need more empirical evidence. The first is that if the weights are initialized in a balanced state they remain balanced throughout. The other main claim is that the regularize converges exponentially fast in cases where the weights are not initialized in a balanced state. Please provide empirical evidence for both these claims on the same type of networks used in Fig 2.\n\n- A central assumption ins Theorem 3.4 (which is not stated in the theorem statement) is that the Hessian is positive definite and thus invertible (i.e. eqn 48). In general, this is not true. Nothing suggests that the Hessian matrix will be positive definite and thus I find it misleading to follow the conclusions of this theory as this is not a general statement at all. For instance, one could invoke strong convexity to guarantee the Hessian is positive definite but I do not think you do this. So at present the theorem statement and the assumptions in the proof do not match and that needs to be updated. This is also true for Corollary 3.5\n\n- One of the limited motivations of this work is the speed of learning achieved after rescaling transformations. For instance, Stock et. al 2019 motivate their work through minimum norm solutions that learn faster and also generalize better and they show this empirically. This work borrows the same motivations but does not follow through with empirical evidence. Specifically, these claims are not empirically corroborated for the particular type of Multiplicative weight update presented in the paper.\n\n- The paper also leans heavily on symmetry and conservation laws. This is used to motivate the gradient flow through a conservation law perspective but this is not explored at all. What is the point of bringing up conservation laws if you wont to use them for something?\n\n- Prop 2.2 proof is not complete proof. The inductive hypothesis and induction step could be written more clearly. Furthermore, I think this proof of the proposition could be generalized. It appears that all you need is the ratio $a_i/a_j$ to be constant $\\forall i,j$ not that $a_i=a_j=1$. I think the reason this proof goes through is by construction that $a_j$ for all non-hidden units is set to 1. But I'm not 100% convinced that there cannot be a fixed point that is not $a_i=1$ but instead has the same ratio constant. Please correct me if I'm wrong here by providing a mathematical argument.\n\n- Updates in eqn 8-10 can be parallelized but it seems that this must happen one layer at a time in a layer-wise fashion. Otherwise, a hidden layer below and its rescaling factor can affect the current layer's rescaling factor. This means that the overall algorithm is bottlenecked by the depth of the network, not width, and is thus not fully parallelizable. Is this the correct interpretation?\n\n\n\n**Minor**\n- Typo: fA's (right above Proof by Lemma 3.2 on Page 16)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper makes two contributions to our understanding of feedforward neural networks with homogeneous activation functions. The first is a procedure for \"gauge-fixing\" - computing multiplicative rescaling factors at each hidden unit that rebalance the weights of these networks without altering the functions they compute. This is achieved by determining the functionally equivalent network whose weight matrix is of minimal $l_{p,q}$-norm. The paper provides an algorithm to compute these multiplicative weight updates which is also a contribution. Notably, these rescaling factors are computed through an iterative process via simple multiplicative updates which do not require learning rate tuning, operate in parallel, and converge monotonically to a global minimizer of the $l_{p,q}$-norm.\n\nThe second contribution is an analysis of the optimization landscape for learning in these networks in the vein of gradient flows. The study outlines a weight-balancing flow where the regularizer remains minimal with respect to rescaling transformations as the weights descend in the loss function. This process deviates from ordinary gradient flow for $l_2$-norm regularization, suggesting a more nuanced connection between alternative flows and regularizers. These results build on the existing understanding of deep learning networks, highlighting the significant role of rescaling symmetries and their influence on weight balancing and learning optimization, and providing insights into analogous gauge-fixing conditions for neural networks similar to those found in physical sciences.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main requested changes that I have are the following:\n\n- Please provide empirical evidence that the weights remain balanced when they are initialized in a balanced state.\n- Please provide empirical evidence of exponential convergence when weights are not initialized in a balanced state.\n- Please increase the quality of the proof in Prop 2.2\n- Please consider shortening the paper to 12 pages and including main proofs in the appendix and providing proof sketches if necessary.\n- Please provide empirical evidence of the speed of learning after initializing weights with Algorithm 1 in a head-to-head comparison with Stock et. al 2019.\n- Please update the theorem statement in 3.4 to accurately include the proper assumptions on the Hessian.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4TI2HzmYWq", "reviewer": "Reviewer_zCJg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The paper is pleasant to read. \n* The results of this paper are novel to the best of my knowledge. \n(Weight-balancing algorithm and the gauge-fixing flow in particular)\n* The paper is theoretically well grounded, and how it is different than the relevant work is well explained. \n* The norm minimization algorithm is shown to work well numerically for deep networks at initialization. \n\nAs the authors noted, the proposed flow converges to $Q_i \\to 0$ exponentially even if $Q_i$'s are not initialized at zero. \nIt is true that this convergence can take a long time for small $\\lambda$, whereas the norm minimization algorithm is converging quickly \nin numerics. Overall the paper proposes a novel initialization (random init + weight-balancing algorithm) and a novel training paradigm with theoretical guarantees. At convergence, the weights that are initialized balanced would remain balanced under gradient flow as we know from the literature. This paper proposes another flow that preserves in addition the minimum $\\ell_{p,q}$ norm. This might bring better inductive biases than gradient flow and better generalization guarantess. For the latter case, as the authors noted, an interesting case is the limit $q \\to \\infty$.\n\n**Weaknesses:**\n* The paper is too long and unfortunately at times repetetive. I think the first paragraph of Section 3 was discussed before, again on page 12, we are reminded that $\\| \\|_{p,q} $denotes the \\ell_{p,q}$-norm, at the top of page 16, there is yet a similar related work discussion for the line of work around building on Noether's theorems for deep learning, and again at the end there is a further related works section. \nThe discussion also reads somehow verbose (I do not see how the third paragraph is related). \n* Section 2.4 is relation to previous work once again. It is helpful to see how the formula simplifies for the special case $p=q$, however the text therein reads repetetive. Bottom of page 13 is again comparison to related work! It would really help to reader to have one related works section at the beginning (or at the end), and limit the discussion elsewhere that are specific to theorems to one-two sentences. \n* The paper would benefit greatly from numerical experiments, similar to Figure 1 that compares gradient flow vs gauge-flow on the regularized objective for some simple tasks. However, this work builds up the theoretical foundation and I don't think these experiments are necessary for publication at TMLR. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies gauge symmetries of deep feedforward neural networks with positively homogenous activation functions. First the authors propose an algorithm for finding the minimum $\\ell_{p,q}$ norm weight matrix while preserving the network function by iteratively rescaling the incoming and outgoing weights. The algorithm enjoys convergence guarantees as shown in the paper. Secondly, the authors propose a flow that preserves the regularization term $\\mathcal{R}(W)$ which is suggested to be minimized initially at random initialization. The proposed gauge-fixing flow also decreases the loss as proven in the paper, hence it is a valid flow. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am not sure if gauge-symmetries are a property of overparameterization as suggested by the very first paragraph of the paper. I'd suggest removing this. \n\nI got confused for some time about the unusual indexing of hidden neurons. The weight matrix is in fact not only lower-triangular but also even in the lower-triangular entries there are many zeros (for example, $[d] \\times [d]$ are not linked as the input nodes are not connected). It would be good to clarify this early on. \n\nQuestion to authors: would it not be better to use $ W' \\leftarrow^a W $ instead of $ W' \\sim^a W $ ? The first implies that $W' $ comes after updating $W$ whereas the second feels like the order does not matter. \n\n**Minor feedback/questions**\n\nTheorem 2.1, I believe argmin should not include biases $b$.\n\nProposition 2.2 [proof], how is it converse statement trivial? Could the authors elaborate on this please? \n\nLemma 2.4 [proof, (iii)], I believe $G(1)$ is maximizer here. Also below Eq. (18). \n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U1cXG1CpbZ", "reviewer": "Reviewer_h4c9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The authors do a good job of motivating the proposed developments with relevant literature, i.e. the approached problems seem actual and the proposed solutions relevant.\n- To the best of my knowledge, the presented results are novel.\n- Besides some minor concerns, the presentation is very clear.\n\n### Weaknesses\n\nIn the current state, the paper seems like two independent studies both of which are incomplete. Indeed, each of the two presented results can be proposed separately without any loss of content. This by itself is a major concern indicating that the paper requires a major revision.\n\nFurthermore, the developments of the first part of the paper (Section 2) are incremental compared to [2]. Indeed, the main development of this section can be summarized as an extension of the result of [2] from $l_{p,p}$ norms to $l_{p,q}$ norms. Also, [2] provides an extensive empirical study demonstrating the practical benefits of the proposed algorithm, while the practical benefits of the proposed extension are not demonstrated.\n\nThe benefits of the second part (Section 3) are unclear. The authors briefly discuss potential discretization schemes of the proposed dynamics, but these discussions are limited to future work discussion, without any empirical studies of the proposed schemes. From the theoretical perspective, there are no immediate implications, e.g. no novel analysis tools are proposed (as done in [1]).\n\nOther comments:\n\n- The last paragraph of Section 1 (page 2). It is not clear what the “speed” of the network means, as well as the “eventual outcomes of learning”.\n- Formulation of Theorem 2.1. I suggest mentioning that the updates of Algorithm 1 preserve the functional form. In general, it is clear from the previous discussion, but it would improve the clarity of the statement, which is the main result.\n- Proof of Proposition 2.2. Typo on the second line of the proof.\n- Page 14. Typo in the first row. Typo in the first row of the second paragraph.\n- The role of Definition 3.1 is not clear. Indeed, it seems like it’s simply a change of the word “stationary” for “balanced”.\n- Theorem 3.3. The formulation “biases are frozen” is not suitable for a theorem statement.\n- Page 21. It is not clear what “boosted gradients” mean.\n\n[1] Tanaka, H., & Kunin, D. (2021). Noether’s learning dynamics: Role of symmetry breaking in neural networks. *Advances in Neural Information Processing Systems*, *34*, 25646-25660.  \n[2] Stock, P., Graham, B., Gribonval, R., & Jégou, H. (2019). Equi-normalization of neural networks. *arXiv preprint arXiv:1902.10416*.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the rescaling invariance of feed-forward neural networks. Indeed, for instance, for ReLU non-linearity, one can obtain a functionally equivalent configuration of weights by taking a single neuron and multiplying all the input weights by some scalar $a$ and dividing all the output weights by $a$ (see Fig. 1 of the paper). The authors break this invariance by introducing regularization and looking for the weights with the minimal regularizer (norm) value not changing the outputs of the network.\n\nThe main contributions of the paper are the algorithm that finds the configuration of weights with the minimal norm preserving the functional form (Algorithm 1 and Theorem 2.1), and the continuous-time dynamic that minimizes the loss for some class of regularizers while preserving the “minimal regularizer value” property on the way (Theorems 3.3 and 3.4).\n\nIn more detail, the proposed Algorithm 1 operates as follows. For the given configuration of parameters, it iteratively rescales the weights and biases of fully connected layers preserving the functional form (i.e. the outputs for all possible inputs) and minimizing the total norm of the weights. The authors then prove that the algorithm converges and also demonstrate it empirically for several randomly initialized MLPs (multilayer perceptrons). Thus, Algorithm 1 allows fixing the gauge, i.e. choosing the symmetry with the minimal norm.\n\nIn Theorems 3.3 and 3.4, the authors propose a continuous time dynamic of weights that minimizes the loss and converges to the minimal gauge in terms of the regularizer. In particular, if the gauge is chosen to be optimal from the beginning (e.g. via applying Algorithm 1 to the randomly initialized network) it will remain optimal throughout the “training”. However, the authors do not discuss immediate practical benefits and do not study the proposed dynamics empirically.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To summarize, the presented study looks anticlimactic. As mentioned before, the two main contributions are independent of each other. Furthermore, each of them does not provide an immediate application or solve a theoretical problem. \n\nThroughout the paper, the authors discuss potential applications of their work (e.g. setting $q \\to \\infty$ in the norm). I believe that the paper would benefit from an empirical study of these discussed applications. For instance, one might demonstrate the benefits of the proposed dynamic (in Section 3) where the initial gauge is fixed using the algorithm from Section 2. This would both tie the parts of the paper and motivate the developments.", "broader_impact_concerns": "The paper does not bring broad impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the rebuttal, reviewers generally agreed that the contribution of this manuscript, theoretical in nature, is sound and of interest to at least part of the TMLR community.  The main criticisms brought forth are as follows:

(1) lacking empirical evaluation and practical utility of the established theory; and

(2) overly long presentation which may burden readers unnecessarily.

The authors argue that per the TMLR guidelines, point (1) should not be an impediment for publication $-$ a statement which I (as well as most reviewers) accept.  With regards to point (2), I would like to ask the authors to follow a practice which in my opinion has become the standard in ML $-$ differentiation between primary content placed in paper body, and secondary content located in appendixes.  The latter for example may include complete proofs, with short proof sketches appearing instead in paper body.  Following a skim through the text, I agree with some of the reviewers that there is no reason for the main content to exceed the 12 page mark of TMLR.

Overall, given that point (1) is dismissed and point (2) can be treated easily, I am recommending to accept the paper subject to a minor revision.
Decision: Accept (poster)