Improving Deep Learning Speed and Performance through Synaptic Neural Balance

Authors: antonios  alexos
ian  domingo
pierre  baldi
Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-08 23:36:43

Abstract: We present experiments and their corresponding theory, demonstrating that synaptic neural balancing can significantly enhance deep learning speed, accuracy, and generalization, particularly on non-traditional compute paradigms. Given an additive cost function (regularizer) of the synaptic weights, a neuron is in balance if the total cost of its incoming weights equals that of its outgoing weights. For various networks, activation functions, and regularizers, neurons can be balanced using scaling operations without altering their functionality, associated with a strictly convex optimization problem.  In our simulations, we systematically observe that: (1) Fully balancing before training results in better performance as compared to several other training approaches; (2) Interleaving partial (layer-wise) balancing and stochastic gradient descent steps during training results in faster learning convergence and better overall accuracy (with $L_1 $ balancing converging faster than $L_2$ balancing; and (3) When given limited training data, neural balanced models outperform plain or regularized models. and this is true both for both feedforward and recurrent networks. These balancing operations are entirely local, making them viable for biological or neuromorphic systems. This positions synaptic neural balancing as a promising approach for leveraging the unique characteristics of emerging AI accelerators, advancing the efficiency and sustainability of machine learning.

Main paper: 
 introduction: 
 Broadly speaking, neural balance refers to the idea of achieving or keeping a certain equilibrium in a neural network during training or after training, whereby such equilibrium may facilitate better information flow, or lower energy expenditure Shwartz-Ziv [2022] . As such, there are different notions of neural balance including, for example, the notion of balance between excitation and inhibition in biological neural networks [Froemke, 2015 , Field et al., 2020 , Howes and Shatalina, 2022 , Kim and Lee, 2022 , Shirani and Choi, 2023 ]. Here we develop the concept of synaptic neural balance which refers to any systematic relationship between the input and output synaptic weights of individual neurons, or layers of neurons. Specifically, we consider the case where the cost of the input weights is equal to the cost of the output weights, where the cost is defined by some regularizer. One of the most basic examples of such a relationship, described below, is when the sum of the squares of the input weights of a neuron is equal to the sum of the squares of its output weights. In this work, we briefly describe the theory of synaptic neural balance and demonstrate its applications to deep learning regularization. We now describe the base case of synaptic neural balance.Base Case: Consider a neuron with a ReLU activation function inside a network trained to minimize a regularized error function E = E + R, where E is the data-dependent error (typically the negative log-likelihood of the data) and R is the regularizer (typically L 2 regularizer). If we multiply the incoming weights of the neuron by some λ > 0 (including the bias) and divide the outgoing weights of the neuron by the same λ, it is easy to see that this scaling operation does not affect in any way the contribution of the neuron to the rest of the network. Thus, the error E which depends only on the input-output function of the network is unchanged. However, the value of the L 2 regularizer changes continuously with λ, and the corresponding contribution is given by: i∈INEQUATION)where IN and OU T denote the set of incoming and outgoing weights respectively, A = i∈IN w 2 i , and B = i∈OU T w 2 i . When λ moves away from 1, the contribution increases in one direction and decreases in the other. In the direction where it decreases, we can solve for the value λ * associated with the mimimal cost. Without taking derivatives, we note that the product of the two terms on the right-hand side of Equation 1 is equal to AB and does not depend on λ. Thus, the minimum is achieved when these two terms are equal, which yields: (λ * ) 4 = B/A for the optimal λ * . The corresponding new set of weights, v i = λ * w i for the input weights and v i = w i /λ * for the outgoing weights, must be balanced: i∈IN v 2 i = i∈OU T v 2 i . This is because the optimal scaling factor for the optimal synaptic weights can only be λ * = 1. Thus, we can define two operations that can be applied to the incoming and outgoing weights of a neuron: scaling and balancing. In between, we can also consider favorable scaling, or partial balancing, where λ is chosen to reduce the cost without necessarily minimizing it.There have been isolated previous studies of this kind of synaptic balance [Du et al., 2018 , Stock et al., 2022] under special conditions. For instance, in Du et al. [2018] , it is shown that if a deep network is initialized in a balanced state with respect to the sum of squares metric, and if training progresses with an infinitesimal learning rate, then balance is preserved throughout training. However, using an infinitesimal learning rate is not practical. Furthermore, there are many intriguing questions that can be raised. For instance: Why does balance occur? Does it occur only with ReLU neurons? Does it occur only with L 2 regularizers? Does it occur only in fully connected feedforward architectures? Does it occur only at the end of training? What happens if we iteratively balance neurons at random in a large network? And can partial or full balancing, before or during learning, be used as an effective regularization technique? All these questions, but the last one, are addressed by the theory of synaptic neural balance that we have developed and briefly describe in the next section. The last question, on using balancing as a learning regularizer, is the main topic of this paper and is addressed by the experiments presented in the following sections. Unless otherwise specified, throughout the paper, terms like "balancing" or "neural balancing" refer to "synaptic neural balancing". 
the theory of synaptic neural balance: 
 We present a brief summary of the main point of the theory. The complete theory is described in the Appendix with the detailed proofs of all the theorems. The first key point is that the base case described in the Introduction, can be extended in three main directions in terms of the activation functions, the regularizers, and the network architectures.Theorem: (Balance and Regularizer Minimization) Consider a neural network with BiLU activation functions in all the hidden units and overall error function of the form:EQUATIONwhere each function g w (w) is continuously differentiable, depends on the magnitude |w| alone, and grows monotonically from g w (0) = 0 to g w (+∞) = +∞. For any setting of the weights W and any hidden unit i in the network and any λ > 0 we can multiply the incoming weights of i by λ and the outgoing weights of i by 1/λ without changing the overall error E. Then, for any neuron, there exists at least one optimal value λ * that minimizes R(W ). Any optimal value must be a solution of the consistency equation:EQUATIONOnce the weights are rebalanced accordingly, the new weights must satisfy the generalized balance equation:EQUATIONIn particular, if g w (w) = |w| p for all the incoming and outgoing weights of neuron i, then the optimal value λ * is unique and equal to:λ * = w∈OU T (i) |w| p w∈IN (i) |w| p 1/2p = ||OU T (i)|| p ||IN (i)|| p 1/2(5)The decrease ∆R ≥ 0 in the value of the L p regularizer R = w |w| p is given by:∆R = w∈IN (i)|w| p 1/2 -w∈OU T (i)|w| p 1/2 2 (6)After balancing neuron i, its new weights satisfy the generalized L p balance equation:EQUATIONProof: The proof is given in the Appendix. We use the optimal value λ * , which we proved how to find in the Appendix, for our experiments in the next Section.Network Architectures: It is easy to see that the reasoning behind the base case can be applied to any BiLU neurons inside any architecture, such a fully connected feedforward, locally connected feedforward, or recurrent. Again this is because scaling does not change the effect of the neuron on the rest of the network and therefore we can always scale the neuron in a way that minimizes a particular cost function or regularizer. It is even possible to train a network with a certain regularizer and balance it with respect to a different regularizer. This brings us to the main result of the theory which is related to balancing algorithms. Imagine that we have a neural network containing BiLU (e.g. ReLU) neurons, with a fixed set of weights W . These could be the weights before learning has started, during learning (i.e. at a particular epoch), or after learning has finished. Imagine that we start balancing the BiLU neurons one after the other, in some regular order or, more generally, even in a stochastic order. Balancing the weights of a neuron may break the balance of another neuron. So while the value of the regularizer always decreases after each balancing operation, it is not clear what happens to the weights of the network, whether they converge to a stable value, and if so whether this value is unique. The main theorem of the theory is the proof that indeed not only the regularizer converges, but the weights themselves must converge and, most interestingly, they must converge to a unique point, which depends only on the initial set of weights W . The limit does not depend on the order in which the balancing operations are applied.  
experiments and results: 
 In our experiments, we train and compare various neural network architectures using full neural balancing, partial balancing, and L 1 or L 2 regularization. The term "plain" is used to refer to training of neural networks without balancing or regularizers. Full balance is obtained by iteratively balancing all BiLU neurons in the network until convergence is achieved. Partial balance is implemented by balancing the neurons in a layer-wise fashion, starting from the input layer and moving towards the output layer or vice-versa (no significant differences are observed). Due to the gradual nature of partial balance, the periodicity of the balancing operation is key to its implementation. In partial balance, the balancing operation can be performed up to once per epoch. Through the use of partial balancing during training, it has been observed that the ratio of the norms of a neuron's output to input weights tends to equalize, irrespective of the periodicity of epochs that we perform partial balancing operations. We have also observed that partial balancing helps the network converge faster and achieve a balanced state as is expected in a fully-trained network, same is in full balancing. The balancing operations for each neuron in each layer take place in parallel so they do not impose a bottleneck during training. Our results suggest that neural balancing is effective in training various types of neural networks with limited data. Furthermore, this approach proves beneficial in reducing overfitting and enhancing generalization in data-scarce environments.To ensure reproducibility and fairness, experiments comparing training methodologies use the same range of seeds, learning rates, and train/test splits. Every experiment was run with 8 different seeds and the result reported is the average of them. A more detailed description of our experimental setup can be found in the Appendix. The roadmap of our experiments is organized as follows: first, we present experiments with the full dataset on both FCNs and RNNs. Then, we move onto data-scarce environments, amplifying the complexity of the experiments. For every experiment we deploy FCNs and RNNs ranging from smaller to larger sizes. The term FCN refers to Feedforward-layered networks with full connectivity between the layers.In table 1, we assess the use of the full balancing operation before the commencement of training.Compared to a standard initialization, the application of full balancing results in faster convergence, and higher overall accuracy when using the same model architecture, hyperparameters, and training methodologies. Partial balancing at every epoch after a full balance results in the least change due to the fundamentally similar nature of the full balancing operation to the partial balancing operation, hence its omission from the plots. Repeated partial balancing results in the same outcome weights when using the same seed, albeit, over time since those weights aren't balanced from the start. Larger model sizes tend to exhibit a stronger correlation between the use of neural balancing, and the model's rate of convergence. These observations are especially exhibited in the normally trained models, where the use of full balancing at the start results in much faster convergence, as well as a higher final accuracy achieved by the model. table 1 displays the comparison between full balancing and no balancing performed on an FCN on MNIST before training. Each square in the grid represents a combination of a model size and a training methodology, where in every case, neural balancing results in an improvement in the rate of convergence and model accuracy.We test all forms of neural balancing on the MNIST handwritten digit dataset exclusively through FCNs. To fully capture the regularization capability of neural balancing, we test on a range of model architectures. From table 2, we observe that neural partial balancing results in faster convergence and test accuracy across all model sizes.We continue our assessment of neural balancing with experiments performed on the RNN architecture. We train a 3-layered RNN on the IMDB sentiment analysis dataset, once again assessing full neural balancing with a 'plain', and regularized models. table 3 demonstrates that when full balancing is performed before training, the model has a better final accuracy when compared to equivalent, non-balanced methodlologies.In data-scarce environments, models employing neural partial balancing techniques demonstrate accelerated convergence compared to unmodified models. These experiments are executed by stratifying samples equally according to their class labels to maintain a balanced distribution of classes within the training data. We observe that neural balancing results in higher accuracy and faster convergence, which could be attributed to better performance in data-scarce environments.Further, we continue this study with an assessment of neural balancing on an RNN trained with a fraction of the original IMDB dataset. fig. 1 Summing up our experiments we observe the following quantitative results. In FCNs, Neural Balance yields a notable improvement in model performance and convergence speed. Specifically, this method results in a 3-5% performance increase over plain models, and more than a 1% improvement over optimally L1-regularized models. Additionally, L1 neural balancing facilitates convergence at a rate 1.5 to 10 times faster, contingent on model size. When trained on limited datasets (1% of the full data), L1 neural balancing enhances performance by 3-10% compared to plain models, and by 1-5% relative to models regularized with L1 and L2 techniques. Moreover, it achieves up to a 10-fold increase in convergence speed, depending on model size. In RNNs, L1 neural balancing contributes to a 2-5% increase in convergence speed, with the application of L2 neural balancing leading to a more than 15% acceleration in convergence when training on 5% of the data. These findings underscore the efficacy of L1 neural balancing in optimizing both performance and training efficiency across different model architectures. We have extended our experiments due to page limits in the Appendix. 
conclusions: 
 Synaptic balancing provides a novel approach to regularization that is supported by an underlying theory. Synaptic balancing is very general in the sense that it can be applied with all usual cost functions, including all L p cost functions. Synaptic balancing can be carried in full or in partial manner, due to the convexity connection provided by the main theorem. It can be applied at any time during the learning process: at the start of learning, at the end of learning, or during learning, by alternating balancing steps with stochastic gradient steps. Given, neural balance has some limitations; as mentioned earlier it can be applied only to neurons with specific activation functions (BiLU or slightly more general activation functions as shown in the Appendix). Another limitation is that it cannot be applied to neurons in Convolution layers due to the nature of the convolution operation with the kernels. Simulations show that these approaches can improve learning in terms of speed (fewer epochs), accuracy or generalization abilities. Thus, in short, balancing is a novel effective approach to regularization that can be added to the list of tools available to regularize networks, like dropout, and other regularization tools. Finally, a neuron can balance its weights independently of all other neurons in the network. The knowledge required for balancing is entirely local and available at each neuron. In this sense, balancing is a natural algorithm for distributed asynchronous architectures and physical neural systems, and as such it may find applications in neuromorphic chip designs or brain studies.A Appendix / supplemental material B AppendixHere we detail the additional theory, datasets, models, and training procedures used in the experiments in the main paper, separated into subsections which correspond to that of the main paper. We also included some supplemental experiments that are not present in the main paper.In order to ensure that our results are reproducible, when we compare training methodologies, we do so using a sample size of 8 different, and random, seeds per methodology, with those seeds being shared with the other training methodologies. We train all of our models on a server equipped with 8 Nvidia RTX A6000 Ada Generation graphics cards, with 384 GB of total memory, run on CUDA version 12.4. 
 
 Appendix: In our experiments, we annotate 2 different kinds of neural balancing operations: L1 Neural Balancing, and L2 Neural Balancing. The names represent the norms used when balancing the input and output weights, with the L1 norm being used for L1 Neural Balancing, and the L2 norm being used for L2 Neural Balancing.To validate our initial hypothesis, which is that the balancing operation results in the equalization of the norms of the input and output weights for every neuron in a neural network, we observe the ratio between the aforementioned norms during training. We do this through a toy network trained on a simple 2-dimensional dataset for a binary classification task, where the limited number of layers and 'neurons' allow us to measure weights without the computational intensity attributed to accessing values from a large network. We compare the use of full balancing with partial balancing during training. Both methodologies result in the optimal factor λ * calculated during balancing to converge to 1, confirming that the norms of the input and output weights for each neuron equalize through the use of balancing. fig. 2 contains partial balancing performed every epoch on a 5-neuron toy model trained on a 2-dimensional concentric circle toy dataset showing that the input and output weight norms equalize for each neuron. To contextualize the rate of convergence of the norms from the partial balancing toy experiment, we measure the input and output norms of each neuron after a full-balance has been performed on the network. While the full-balance guarantees that the input and output norms of each neuron will always be close to each other, since full balancing is performed until that requirement is met, it remains useful as a benchmark for the rate of convergence of partial-balancing. fig. 3 the rate of convergence of the input and output norms, doing so almost immediately, due to the methodology of full balancing. fig. 2 demonstrates the efficacy of partial-balancing, resulting in a rapid, and computationally less expensive method of 'balancing' neurons.In the main paper, we assess the use of the full balancing operation before the start of training to demonstrate its efficacy at increasing the rate of convergence and overall test accuracy of various model architectures and training styles. Partial balancing at every epoch after a full balance results in the least change due to the fundamentally similar nature of the full balancing operation to the partial balancing operation, hence its omission from the plots. Repeated partial balancing results in wthe same outcome weights when using the same seed, albeit, over time since those weights aren't balanced from the start. In these experiments, we use fully connected neural networks in a few sizes to demonstrate the range of the balancing operation. Full balance before training is shown to increase the rate of convergence, as well as the overall accuracy obtainable during training. To assess full neural balance before training, we performed a full balancing operation on the neurons of the model after the initialization of the model's weights, and before the commencement of training.In the main paper, we assess the use of the partial balancing operation during training to demonstrate its efficacy at increasing the rate of convergence and overall test accuracy of various model architectures and training styles. As included in the main paper in section 4.2, we supplement our tabular results in fig. 6 with plots that delineate the positive impact of partial and full neural balance as performed through the balancing operation during/before training. Following the line of inquiry Accuracy for FCNs of varying sizes on MNIST. We observe that L1 partial balancing outperforms the other training methodologies on all model sizes on the performance of neural balancing on FCNs trained on MNIST, we assess its performance on FashionMNIST using the same model architectures. We use FCNs of various sizes, and perform a partial balance on the model at every epoch, identically to the MNIST experiments. We observed similar results on performance and convergence on FashionMNIST. Regardless of the size of the model, or the methodology used to train said model, neural balancing significantly increases the rate of convergence, as well as its overall test accuracy. In the main paper, we assess the use of the partial balancing operation during training to demonstrate its efficacy at increasing the rate of convergence and overall test accuracy of a recurrent neural network architecture, comparing various training styles in the process. For these experiments, we use the IMDB sentiment analysis dataset. The IMDB dataset is a collection of positively/negatively labeled text containing movie reviews from the popular movie review website IMDB. We use a recurrent neural network with 3 hidden layers to demonstrate the efficacy of the partial balancing operation. In this section, we generalize the basic example of the introduction from the standpoint of the activation functions. In particular, we consider homogeneous activation functions (defined below). The importance of homogeneity has been previously identified in somewhat different contexts Neyshabur et al. [2015b] . Intuitively, homogeneity is a form of linearity with respect to weight scaling and thus it is useful to motivate the concept of homogeneous activation functions by looking at other notions of linearity for activation functions. This will also be useful for Section C.5 where even more general classes of activation functions are considered. Proof. Obviously linear activation functions are additively linear. Conversely, if f is additively linear, the following three properties are true:(1) One must have: f (nx) = nf (x) and f (x/n) = f (x)/n for any x ∈ R and any n ∈ N. As a result, f (n/m) = nf (1)/m for any integers n and m (m ̸ = 0).(2) Furthermore, f (0 + 0) = f (0) + f (0) which implies: f (0) = 0.(3) And thus f(x -x) = f (x) + f (-x) = 0, which in turn implies that f (-x) = -f (x).From these properties, it is easy to see that f must be continuous, with f (x) = xf (1), and thus f must be linear.Definition C.3. A neuronal activation function f : R → R is multiplicative if and only if f (xy) = f (x)(f (y) for any real numbers x and y.Proposition C.4. The class of continuous multiplicative activation functions is exactly equal to the class of functions comprising the functions: f (x) = 0 for every x, f (x) = 1 for every x, and all the even and odd functions satisfying f (x) = x c for x ≥ 0, where c is any constant in R.Proof. It is easy to check the functions described in the proposition are multiplicative. Conversely, assume f is multiplicative. For both x = 0 and x = 1, we must havef (x) = f (xx) = f (x)f (x)and thus f (0) is either 0 or 1, and similarly for f (1). If f (1) = 0, then for any x we must have f (x) = 0 because:f (x) = f (1x) = f (1)f (x) = 0. Likewise, if f (0) = 1, then for any x we must have f (x) = 1 because: 1 = f (0) = f (0x) = f (0)f (x) = f (x).Thus, in the rest of the proof, we can assume that f (0) = 0 and f (1) = 1. By induction, it is easy to see that for any x ≥ 0 we must have:f (x n ) = f (x) n and f (x 1/n ) = (f (x)) 1/n for any integer (positive or negative). As a result, for any x ∈ R and any integers n and m we must have: f (x n/m ) = f (x) n/m . By continuity this implies that for any x ≥ 0 and any r ∈ R, we must have: f (x r ) = f (x) r . Now there is some constant c such that: f (e) = e c . And thus, for any x > 0, f (x) = f (e log x ) = [f (e)] log x = e c log x = x c . To address negative values of x, note that we must have f [(-1)(-1 = f (1) = 1f (-1) 2 . Thus, f (-1) is either equal to 1 or to -1. Since for any x > 0 we have f (-x) = f (-1)f (x), we see that if f (-1) = 1 the function must be even (f (-x) = f (x) = x c ), and if f (-1) = -1 the function must be odd (f (-x) = -f (x)).We will return to multiplicative activation function in a later section.Definition C.5. A neuronal activation function f : R → R is linearly scalable if and only if f (λx) = λf (x) for every λ ∈ R.Proposition C.6. The class of linearly scalable activation functions is exactly equal to the class of linear activation functions, i.e., activation functions of the form f (x) = ax.Proof. Obviously, linear activation functions are linearly scalable. For the converse, if f is linearly multiplicative we must have f (λx) = λf (x) = xf (λ) for any x and any λ. By taking λ = 1, we get f (x) = f (1)x and thus f is linear.Thus the concepts of linearly additive or linearly scalable activation function are of limited interest since both of them are equivalent to the concept of linear activation function. A more interesting class is obtained if we consider linearly scalable activation functions, where the scaling factor λ is constrained to be positive (λ > 0), also called homogeneous functions.Definition C.7. (Homogeneous) A neuronal activation function f : R → R is homogeneous if and only if: f (λx) = λf (x) for every λ ∈ R with λ > 0.Remark C.8. Note that if f is homogeneous, f (λ0) = λf (0) = f (0) for any λ > 0 and thus f (0) = 0. Thus it makes no difference in the definition of homogeneous if we set λ ≥ 0 instead of λ > 0). Remark C.9. Clearly, linear activation functions are homogeneous. However, there exists also homogeneous functions that are non-linear, such as ReLU or leaky ReLU activation functions.We now provide a full characterization of the class of homogeneous activation functions.We first define a new class of activation functions, corresponding to bilinear units (BiLU), consisting of two half-lines meeting at the origin. This class contains all the linear functions, as well as the ReLU and leaky ReLU functions, and many other functions. Definition C.10. (BiLU) A neuronal activation function f : R → R is bilinear (BiLU) if and only if f (x) = ax when x < 0, and f (x) = bx when x ≥ 0, for some fixed parameters a and b in R.These include linear units (a = b), ReLU units (a = 0, b = 1), leaky ReLU (a = ϵ; b = 1) units, and symmetric linear units (a = -b), all of which can also be viewed as special cases of piece-wise linear units f (0x) = 0f (x) = f (0) = 0; (2)f (x) = f (1x) = f (1)x for any positive x; and (3)f (x) = f (-u) = f (-1)u = -f (-1)x for any negative x. Thus f is in BiLU with a = -f (-1) and b = f (1).In Appendix A, we provide a simple proof that networks of BiLU neurons, even with a single hidden layer, have universal approximation properties. In the next two sections, we introduce two fundamental neuronal operations, scaling and balancing, that can be applied to the incoming and outgoing synaptic weights of neurons with BiLU activation functions.Definition C.12. (Scaling) For any BiLU neuron i in network and any λ > 0, we let S λ (i) denote the synaptic scaling operation by which the incoming connection weights of neuron i are multiplied by λ and the outgoing connection weights of neuron i are divided by λ.Note that because of the homogeneous property the scaling operation does not change how neuron i affects the rest of the network. In particular, the input-output function of the overall network remains unchanged after scaling neuron i bt any λ > 0. Note also that scaling always preserves the sign of the synaptic weights to which it is applied, and the scaling operation can never convert a non-zero synaptic weight into a zero synaptic weight, or vice versa.As usual, the bias is treated here as an additional synaptic weight emanating from a unit clamped to the value one. Thus scaling is applied to the bias. Proposition C.13. (Commutativity of Scaling) Scaling operations applied to any pair of BiLU neurons i and j in a neural network commute: S λ (i)S µ (j) = S µ (j)S λ (i), in the sense that the resulting network weights are the same, regardless of the order in which the scaling operations are applied. Furthermore, for any BiLU neuron i:S λ (i)S µ (i) = S µ (i)S λ (i) = S λµ (i).This is obvious. As a result, any set I of BiLU neurons in a network can be scaled simultaneously or in any sequential order while leading to the same final configuration of synaptic weights. If we denote by 1, 2, . . . , n the neurons in I, we can for instance write: i∈I S λi (i) = σ(i)∈I S λ σ(i) (σ(i)) for any permutation σ of the neurons. Likewise, we can collapse operations applied to the same neuron.For instance, we can write: S 5 (1)S 2 (2)S 3 (1)S 4 (2) = S 15 (1)S 8 (2) = S 8 (2)S 15 (1) Definition C.14. (Coordinated Scaling) For any set I of BiLU neurons in a network and any λ > 0, we let S λ (I) denote the synaptic scaling operation by which all the neurons in I are scaled by the same λ.Definition C.15. (Balancing) Given a BiLU neuron in a network, the balancing operation B(i) is a particular scaling operation B(i) = S λ * (i), where the scaling factor λ * is chosen to optimize a particular cost function, or regularizer, asociated with the incoming and outgoing weights of neuron i.For now, we can imagine that this cost function is the usual L 2 (least squares) regularizer, but in the next section, we will consider more general classes of regularizers and study the corresponding optimization process. For the L 2 regularizer, as shown in the next section, this optimization process results in a unique value of λ * such that sum of the squares of the incoming weights is equal to the sum of the squares of the outgoing weights, hence the term "balance". Note that obviously B(B(i)) = B(i) and that, as a special case of scaling operation, the balancing operation does not change how neuron i contributes to the rest of the network, and thus it leaves the overall input-output function of the network unchanged.Unlike scaling operations, balancing operations in general do not commute as balancing operations (they still commute as scaling operations). Thus, in general, B(i)B(j) ̸ = B(j)B(i). This is because if neuron i is connected to neuron j, balancing i will change the connection between i and j, and, in turn, this will change the value of the optimal scaling constant for neuron j and vice versa. However, if there are no non-zero connections between neuron i and neuron j then the balancing operations commute since each balancing operation will modify a different, non-overlapping, set of weights. Definition C.16. (Disjoint neurons) Two neurons i and j in a neural network are said to be disjoint if there are no non-zero connections between i and j.Thus in this case B(i)B(j) = S λ * (i)S µ * (j) = S µ * (j)S λ * (i) = B(j)B(i). This can be extended to disjoint sets of neurons. Definition C.17. (Disjoint Set of Neurons) A set I of neurons is said to be disjoint if for any pair i and j of neurons in I there are no non-zero connections between i and j.For example, in a layered feedforward network, all the neurons in a layer form a disjoint set, as long as there are no intra-layer connections or, more precisely, no non-zero intra-layer connections. All the neurons in a disjoint set can be balanced in any order resulting in the same final set of synaptic weights. Thus we have: Proposition C.18. If we index by 1, 2, . . . , n the neurons in a disjoint set I of BiLU neurons in a network, we have:i∈I B(i) = i∈I S λ * i (i) = σ(i)∈I S λ * σ(i) (σ(i)) = σ(i)∈I B(σ(i))for any permutation σ of the neurons.Finally, we can define the coordinated balancing of any set I of BiLU neurons (disjoint or not disjoint). Definition C.19. (Coordinated Balancing) Given any set I of BiLU neurons (disjoint or not disjoint) in a network, the coordinated balacing of these neurons, written as B λ * (I), corresponds to coordinated scaling all the neurons in I by the same factor λ * , Where λ * minimizes the cost functions of all the weights, incoming and outgoing, associated with all the neurons in I. Remark C.20. While balancing corresponds to a full optimization of the scaling operation, it is also possible to carry a partial optimization of the scaling operation by choosing a scaling factor that reduces the corresponding contribution to the regularizer without minimizing it.In this section, we generalize the kinds of regularizer to which the notion of neuronal synaptic balance can be applied, beyond the usual L 2 regularizer and derive the corresponding balance equations.Thus we consider a network (feedforward or recurrent) where the hidden units are BiLU units. The visible units can be partitioned into input units and output units. For any hidden unit i, if we multiply all its incoming weights IN (i) by some λ > 0 and all its outgoing weights OU T (i) by 1/λ the overall function computed by the network remains unchanged due to the BiLU homogeneity property. In particular, if there is an error function that depends uniquely on the input-output function being computed, this error remains unchanged by the introduction of the multiplier λ. However, if there is also a regularizer R for the weights, its value is affected by λ and one can ask what is the optimal value of λ with respect to the regularizer, and what are the properties of the resulting weights. This approach can be applied to any regularizer. For most practical purposes, we can assume that the regularizer is continuous in the weights (hence in λ) and lower-bounded. Without any loss of generality, we can assume that it is lower-bounded by zero. If we want the minimum value to achieved by some λ > 0, we need to add some mild condition that prevents the minimal value to be approached as λ → 0 ) , or as λ → +∞. For instance, it is enough if there is an interval [a, b] with 0 < a < b where R achieves a minimal value R min and R ≥ R min in the intervals (0, a] and [b, +∞). Additional (mild) conditions must be imposed if one wants the optimal value of λ to be unique, or computable in closed form (see Theorems below). Finally, we want to be able to apply the balancing approach Thus, we consider overall regularized error functions, where the regularizer is very general, as long as it has an additive form with respect to the individual weights:EQUATIONwhere W denotes all the weights in the network and E(W ) is typically the negative log-likelihood (LMS error in regression tasks, or cross-entropy error in classification tasks). We assume that the g w are continuous, and lower-bounded by 0. To ensure the existence and uniqueness of minimum during the balancing of any neuron, We will assume that each function g w depends only on the magnitude |w| of the corresponding weight, and that g w is monotonically increasing from 0 to +∞ (g w (0) = 0 and lim x→+∞ g w (x) = +∞). Clearly, L 2 , L 1 and more generally all L p regularizers are special cases where, for p > 0, L p regularization is defined by: R(W ) = w |w| p .When indicated, we may require also that the functions g w be continuously differentiable, except perhaps at the origin in order to be able to differentiate the regularizer with respect to the λ's and derive closed form conditions for the corresponding optima. This is satisfied by all forms of L p regularization, for p > 0.Remark C.21. Often one introduces scalar multiplicative hyperparameters to balance the effect of E and R, for instance in the form: E = E + βR. These cases are included in the framework above: multipliers like β can easily be absorbed into the functions g w above.Theorem C.22. (General Balance Equation). Consider a neural network with BiLU activation functions in all the hidden units and overall error function of the form:EQUATIONwhere each function g w (w) is continuous, depends on the magnitude |w| alone, and grows monotonically from g w (0) = 0 to g w (+∞) = +∞. For any setting of the weights W and any hidden unit i in the network and any λ > 0 we can multiply the incoming weights of i by λ and the outgoing weights of i by 1/λ without changing the overall error E. Furthermore, there exists a unique value λ * where the corresponding weights v (v = λ * w for incoming weights, v = w/λ * for the outgoing weights) achieve the balance equation:EQUATIONProof. Under the assumptions of the theorem, E is unchanged under the rescaling of the incoming and outgoing weights of unit i due to the homogeneity property of BiLUs. Without any loss of generality, let us assume that at the beginning: w∈IN (i) g w (w) < w∈OU T (i) g w (w). As we increase λ from 1 to +∞, by the assumptions on the functions g w , the term w∈IN (i) g w (λw) increases continuously from its initial value to +∞, whereas the term w∈OU T (i) g w )w/λ) decreases continuously from its initial value to 0. Thus, there is a unique value λ * where the balance is realized. If at the beginning w∈IN (i) g w (w) > w∈OU T (i) g w (w), then the same argument is applied by decreasing λ from 1 to 0.Remark C.23. For simplicity, here and in other sections, we state the results in terms of a network of BiLU units. However, the same principles can be applied to networks where only a subset of neurons are in the BiLU class, simply by applying scaling and balancing operations to only those neurons. Furthermore, not all BiLU neurons need to have the same BiLU activation functios. For instance, the results still hold for a mixed network containing both ReLU and linear units. Remark C.24. In the setting of Theorem C.22, the balance equations do not necessarily minimize the corresponding regularization term. This is addressed in the next theorem. Remark C.25. Finally, zero weights (w = 0) can be ignored entirely as they play no role in scaling or balancing. Furthermore, if all the incoming or outgoing weights of a hidden unit were to be zero, it could be removed entirely from the network Theorem C.26. (Balance and Regularizer Minimization) We now consider the same setting as in Theorem C.22, but in addition we assume that the functions g w are continuously differentiable, except perhaps at the origin. Then, for any neuron, there exists at least one optimal value λ * that minimizes R(W ). Any optimal value must be a solution of the consistency equation:λ 2 w∈IN (i) wg ′ w (λw) = w∈OU T (i) wg ′ w (w/λ) (11)Once the weights are rebalanced accordingly, the new weights must satisfy the generalized balance equation:EQUATIONIn particular, if g w (w) = |w| p for all the incoming and outgoing weights of neuron i, then the optimal value λ * is unique and equal to:λ * = w∈OU T (i) |w| p w∈IN (i) |w| p 1/2p = ||OU T (i)|| p ||IN (i)|| p 1/2 (13)The decrease ∆R ≥ 0 in the value of the L p regularizer R = w |w| p is given by:∆R = w∈IN (i) |w| p 1/2 - w∈OU T (i) |w| p 1/2 2 (14)After balancing neuron i, its new weights satisfy the generalized L p balance equation:w∈IN (i) |w| p = w∈OU T (i) |w| p (15)Proof. Due to the additivity of the regularizer, the only component of the regularizer that depends on λ has the form:R(λ) = w∈IN (i) g w (λw) + w∈OU T (i) g w (w/λ) (16)Because of the properties of the functions g w , R λ is continously differentiable and strictly bounded below by 0. So it must have a minimum, as a function of λ where its derivative is zero. Its derivative with respect to λ has the form:EQUATIONSetting the derivative to zero, gives:λ 2 w∈IN (i) wg ′ w (λw) = w∈OU T (i) wg ′ w (w/λ) (18)Assuming that the left-hand side is non-zero, which is generally the case, the optimal value for λ must satisfy:λ = w∈OU T (i) wg ′ w (w/λ) w∈IN (i) wg ′ w (λw) 1/2 (19)If the regularizing function is the same for all the incoming and outgoing weights (g w = g), then the optimal value λ must satisfy:λ = w∈OU T (i) wg ′ (w/λ) w∈IN (i) wg ′ (λw) 1/2 (20)In particular, if g(w) = |w| p then g(w) is differentiable except possibly at 0 and g ′ (w) = s(w)p|w| p-1 , where s(w) denotes the sign of the weight w. Substituting in Equation 20, the optimal rescaling λ must satisfy:EQUATIONAt the optimum, no further balancing is possible, and thus λ * = 1. Equation 18 yields immediately the generalized balance equation to be satisfied at the optimum:EQUATIONIn the case of L P regularization, it is easy to check by applying Equation 22, or by direct calculation that:w∈IN (i) |λ * w| p = w∈OU T (i) |w/λ * | p (23)which is the generalized balance equation. Thus after balancing neuron, the weights of neuron i satisfy the L p balance (Equation 15). The change in the value of the regularizer is given by:∆R = w∈IN (i) |w| p + w∈OU T (i) |w| p - w∈IN (i) |λ * w| p - w∈OU T (i) |w/λ * | p (24)By substituting λ * by its explicit value given by Equation 21and collecting terms gives Equation 14.Remark C.27. The monotonicity of the functions g w is not needed to prove the first part of Theorem C.26. It is only needed to prove uniqueness of λ * in the L p cases.Remark C.28. Note that the same approach applies to the case where there are multiple additive regularizers. For instance with both L 2 and L 1 regularization, in this case the function f has the form: g w (w) = αw 2 + β|w|. Generalized balance still applies. It also applies to the case where different regularizers are applied in different disconnected portions of the network.Remark C.29. The balancing of a single BiLU neuron has little to do with the number of connections. It applies equally to fully connected neurons, or to sparsely connected neurons.So far we have generalized ReLU activation functions to BiLU activation functions in the context of scaling and balancing operations with positive scaling factors. While in the following sections we will continue to work with BiLU activation functions, in this section we show that the scaling and balancing operations can be extended even further to other activation functions. The section can be skipped if one prefers to progress towards the main results on stochastic balancing.Given a neuron with activation function f (x), during scaling instead of multiplying and dividing by λ > 0, we could multiply the incoming weights by a function g(λ) and divide the outgoing weights by a function h(λ), as long as the activation function f satisfies:EQUATION) for every x ∈ R to ensure that the contribution of the neuron to the rest of the network remains unchanged. Note that if the activation function f satisfies Equation 25, so does the activation function -f . In Equation 25, λ does not have to be positive-we will simply assume that λ belongs to some open (potentially infinite) interval (a, b). Furthermore, the functions g and h cannot be zero for λ ∈ (a, b) since they are used for scaling. It is reasonable to assume that the functions g and h are continuous, and thus they must have a constant sign as λ varies over (a, b). Now, taking x = 0 gives f (0) = h(λ)f (0) for every λ ∈ (a, b), and thus either f (0) = 0 or h(λ) = 1 for every λ ∈ (a, b). The latter is not interesting and thus we can assume that the activation function f satisfies f (0) = 0. Taking x = 1 gives f (g(λ)) = h(λ)f (1) for every λ in (a, b). For simplicity, let us assume that f (x) = 1. Then, we have: f (g(λ)) = h(λ) for every λ. Substituting in Equation 25 yields:f (g(λ)x) = f (g(λ))f (x)(26) for every x ∈ R and every λ ∈ (a, b). This relation is essentially the same as the relation that defines multiplicative activation functions over the corresponding domain (see Proposition C.4), and thus we can identify a key family of solutions using power functions. Note that we can define a new parameter µ = g(λ), where µ ranges also over some positive or negative interval I over which:f (µx) = f (µ)f (x).Let us assume that λ > 0, g(λ) = λ and h(λ) = λ c for some c ∈ R. Then the activation function must satisfy the equation:f (λx) = λ c f (x)(27) for any x ∈ R and any λ > 0. Note that if f (x) = x c we get a multiplicative activation function. More generally, these functions are characterized by the following proposition. Proposition C.30. The set of activation functions f satisfying f (λx) = λ c f (x) for any x ∈ R and any λ > 0 consist of the functions of the form:EQUATION)where c ∈ R, C = f (1) ∈ R, and D = f (-1) ∈ R. We call these bi-power units (BiPU). If, in addition, we want f to be continuous at 0, we must have either c > 0, or c = 0 with C = D.Given the general shape, these activations functions can be called BiPU (Bi-Power-Units). Note that in the general case where c > 0, C and D do not need to be equal. In particular, one of them can be equal to zero, and the other one can be different from zero giving rise to "rectified power units" (Figure 13 ).Proof. By taking x = 1, we get f (λ) = f (1)λ c for any λ > 0. Let f (1) = C. Then we see that for any x > 0 we must have: f (x) = Cx c . In addition, for every λ > 0 we must have: In all cases, it is easy to check directly that the resulting functions satisfy the functional equation given by Equation 27.f (λ0) = f (0) = λ c f (0). So if c = 0, then f (x) = C = f (1) for x ≥ 0. If c ̸ = 0, then f (0) = 0. InA BiPU neuron can be scaled by multiplying its incoming weight by λ > 0 and dividing its outgoing weights by 1/λ c . This will not change the role of the corresponding unit in the network, and thus it will not change the input-output function of the network.As in the case of BiLU neurons, we balance a multiplicative neuron by asking what is the optimal scaling factor λ that optimizes a particular regularizer. For simplicity, here we assume that the regularizer is in the L p class. Then we are interested in the value of λ > 0 that minimizes the function:EQUATIONA simple calculation shows that the optimal value of λ is given by:EQUATIONThus after balancing the weights, the neuron must satisfy the balance equation:EQUATIONin the new weights w.So far, we have focused on balancing individual neurons. In the next two sections, we look at balancing across all the units of a network. We first look at what happens to network balance when a network is trained by gradient descent and then at what happens to network balance when individual neurons are balanced iteratively in a regular or stochastic manner.A natural question is whether gradient descent (or stochastic gradient descent) applied to a network of BiLU neurons, with or without a regularizer, converges to a balanced state of the network, where all the BiLU neurons are balanced. So we first consider the case where there is no regularizer (E = E).The results in Du et al. [2018] may suggest that gradient descent may converge to a balanced state. In particular, they write that for any neuron i:EQUATIONThus the gradient flow exactly preserves the difference between the L 2 cost of the incoming and outgoing weights or, in other words, the derivative of the L 2 balance deficit is zero. Thus if one were to start from a balanced state and use an infinitesimally small learning rate one ought to stay in a balanced state at all times.However, it must be noted that this result was derived for the L 2 metric only, and thus would not cover other L p forms of balance. Furthermore, it requires an infinitesimally small learning rate. In practice, when any standard learning rate is applied, we find that gradient descent does not converge to a balanced state (Figure 1 ). However, things are different when a regularizer term is included in the error functions as described in the following theorem.Theorem C.31. Gradient descent in a network of BiLU units with error function E = E + R where R has the properties described in Theorem C.26 (including all L p ) must converge to a balanced state, where every BiLU neuron is balanced.Proof. By contradiction, suppose that gradient descent converges to a state that is unbalanced and where the gradient with respect to all the weights is zero. Then there is at least one unbalanced neuron in the network. We can then multiply the incoming weights of such a neuron by λ and the outgoing weights by 1/λ as in the previous section without changing the value of E. Since the neuron is not in balance, we can move λ infinitesimally so as to reduce R, and hence E. But this contradicts the fact that the gradient is zero.Remark C.32. In practice, in the case of stochastic gradient descent applied to E + R, at the end of learning the algorithm may hover around a balanced state. If the state reached by the stochastic gradient descent procedure is not approximately balanced, then learning ought to continue. In other words, the degree of balance could be used to monitor whether learning has converged or not. Balance is a necessary, but not sufficient, condition for being at the optimum.Remark C.33. If early stopping is being used to control overfitting, there is no reason for the stopping state to be balanced. However, the balancing algorithms described in the next section could be used to balance this state.In this section, we look at balancing algorithms where, starting from an initial weight configuration W , the BiLU neurons of a network are balanced iteratively according to some deterministic or stochastic schedule that periodically visits all the neurons. We can also include algorithms where neurons are partitioned into groups (e.g. neuronal layers) and neurons in each group are balanced together.The most interesting algorithm is when the BiLU neurons of a network are iteratively balanced in a purely stochastic manner. This algorithm is particularly attractive from the standpoint of physically implemented neural networks because the balancing algorithm is local and the updates occur randomly without the need for any kind of central coordination. As we shall see in the following section, the random local operations remarkably lead to a unique form of global order. The proof for the stochastic case extends immediately to the deterministic case, where the BiLU neurons are updated in a deterministic fashion, for instance by repeatedly cycling through them according to some fixed order.It is also possible to partition the BiLU neurons into non-overlapping subsets of neurons, and then balance each subset, especially when the neurons in each subset are disjoint of each other. In this case, one can balance all the neurons in a given subset, and repeat this subset-balancing operation subset-by-subset, again in a deterministic or stochastic manner. Because the BiLU neurons in each subset are disjoint, it does not matter whether the neurons in a given subset are updated synchronously or sequentially (and in which order). Since the neurons are balanced independently of each other, this can be called independent subset balancing. For example, in a layered feedforward network with no lateral connections, each layer corresponds to a subset of disjoint neurons. The incoming and outgoing connections of each neuron are distinct from the incoming and outgoing connections of any other neuron in the layer, and thus the balancing operation of any neuron in the layer does not interfere with the balancing operation of any other neuron in the same layer. So this corresponds to independent layer balancing, As a side note, balancing a layer h, may disrupt the balance of layer h + 1. However, balancing layer h and h + 2 (or any other layer further apart) can be done without interference of the balancing processes. This suggests also an alternating balancing scheme, where one alternatively balances all the odd-numbered layers, and all the evenly-numbered layers.Yet another variation is when the neurons in a disjoint subset are tied to each other in the sense that they must all share the same scaling factor λ. In this case, balancing the subset requires finding the optimal λ for the entire subset, as opposed to finding the optimal λ for each neuron in the subset.Since the neurons are balanced in a coordinated or tied fashion, this can be called coordinated or tied subset balancing. For example, tied layer balancing must use the same λ for all the neurons in a given layer. It is easy to see that this approach leads to layer synaptic balance which has the form (for an L p regularizer):i w∈IN (i)EQUATIONwhere i runs over all the neurons in the layer. This does not necessarily imply that each neuron in the layer is individually balanced. Thus neuronal balance for every neuron in a layer implies layer balance, but the converse is not true. Independent layer balancing will lead to layer balance. Coordinated layer balancing will lead to layer balance, but not necessarily to neuronal balance of each neuron in the layer. Layer-wise balancing, independent or tied, can be applied to all the layers and in deterministic (e.g. sequential) or stochastic manner. Again the proof given in the next section for the basic stochastic algorithm can easily be applied to these cases (see also Appendix B).Suppose that two connections share the same weight so that we must have: w ij = w kl at all times. In general, when the balancing algorithm is applied to neuron i or j, the weight w ij will change and the same change must be applied to w kl . The latter may disrupt the balance of neuron k or l. Furthermore, this may not lead to a decrease in the overall value of the regularizer R.The case of convolutional networks is somewhat special, since all the incoming weights of the neurons sharing the same convolutional kernel are shared. However, in general, the outgoing weights are not shared. Furthermore, certain operations like max-pooling are not homogeneous. So if one trains a CNN with E alone, or even with E + R, one should not expect any kind of balance to emerge in the convolution units. However, all the other BiLU units in the network should become balanced by the same argument used for gradient descent above. The balancing algorithm applied to individual neurons, or the independent layer balancing algorithm, will not balance individual neurons sharing the same convolution kernel. The only balancing algorithm that could lead to some convolution layer balance, but not to individual neuronal balance, is the coordinated layer balancing, where the same λ is used for all the neurons in the same convolution layer, provided that their activation functions are BiLU functions.We can now study the convergence properties of balancing algorithms.We now consider the basic stochastic balancing algorithm, where BiLU neurons are iteratively and stochastically balanced. It is essential to note that balancing a neuron j may break the balance of another neuron i to which j is connected. Thus convergence of iterated balancing is not obvious.There are three key questions to be addressed for the basic stochastic algorithm, as well as all the other balancing variations. First, does the value of the regularizer converges to a finite value? Second, do the weights themselves converge to fixed finite values representing a balanced state for the entire network? And third, if the weights converge, do they always converge to the same values, irrespective of the order in which the units are being balanced? In other words, given an initial state W for the network, is there a unique corresponding balanced state, with the same input-output functionalities?C.8.1 Notation and Key QuestionsFor simplicity, we use a continuous time notation. After a certain time t each neuron has been balanced a certain number of times. While the balancing operations are not commutative as balancing operations, they are commutative as scaling operations. Thus we can reorder the scaling operations and group them neuron by neuron so that, for instance, neuron i has been scaled by the sequence of scaling operations:EQUATIONwhere n it corresponds to the count of the last update of neuron i prior to time t, and:EQUATIONFor the input and output units, we can consider that their balancing coefficients λ * are always equal to 1 (at all times) and therefore Λ i (t) = 1 for any visible unit i.Thus, we first want to know if R converges. Second, we want to know if the weights converge. This question can be split into two sub-questions: (1) Do the balancing factors λ * n (i) converge to a limit as time goes to infinity. Even if the λ * n (i)'s converge to a limit, this does not imply that the weights of the network converge to a limit. After a time t, the weight w ij (t) between neuron j and neuron i has the value w ij Λ i (t)/Λ j (t), where w ij = w ij (0) is the value of the weight at the start of the stochastic balancing algorithm. Thus: (2) Do the quantities Λ i (t) converge to finite values, different from 0? And third, if the weights converge to finite values different from 0, are these values unique or not, i.e. do they depend on the details of the stochastic updates or not? These questions are answered by the following main theorem.. . Let W denote the initial weights. When the neuronal stochastic balancing algorithm is applied throughout the network so that every neuron is visited from time to time, then E(W ) remains unchanged but R(W ) must converge to some finite value that is less or equal to the initial value, strictly less if the initial weights are not balanced. In addition, for every neuron i, λ * i (t) → 1 and Λ i (t) → Λ i as t → ∞, where Λ i is finite and Λ i > 0 for every i. As a result, the weights themselves must converge to a limit W ′ which is globally balanced, with E(W ) = E(W ′ ) and R(W ) ≥ R(W ′ ), and with equality if only if W is already balanced. Finally, W ′ is unique as it corresponds to the solution of a strictly convex optimization problem in the variables L ij = log(Λ i /Λ j ) with linear constraints of the form π L ij = 0 along any path π joining an input unit to an output unit and along any directed cycle (for recurrent networks). Stochastic balancing projects to stochastic trajectories in the linear manifold that run from the origin to the unique optimal configuration. Proof. Each individual balancing operation leaves E(W ) unchanged because the BiLU neurons are homogeneous. Furthermore, each balancing operation reduces the regularization error R(W ), or leaves it unchanged. Since the regularizer is lower-bounded by zero, the value of the regularizer must approach a limit as the stochastic updates are being applied.For the second question, when neuron i is balanced at some step, we know that the regularizer R decreases by:∆R = w∈IN (i) |w| p 1/2 - w∈OU T (i) |w| p 1/2 2 (36)If the convergence were to occur in a finite number of steps, then the coefficients λ * i (t) must become equal and constant to 1 and the result is obvious. So we can focus on the case where the convergence does not occur in a finite number of steps (indeed this is the main scenario, as we shall see at the end of the proof). Since ∆R → 0, we must have:EQUATIONBut from the expression for λ * (Equation 21), this implies that for every i, λ * n (i) → 1 as time increases (n → ∞). This alone is not sufficient to prove that Λ i (t) converges for every i as t → ∞. However, it is easy to see that Λ i (t) cannot contain a sub-sequence that approaches 0 or ∞ (Figure 14 ). Furthermore, not only ∆R converges to 0, but the series ∆R is convergent. This shows that, for every i, ∆ i (t) must converge to a finite, non-zero value ∆ i . Therefore all the weights must converge to fixed values given by w ij (0)Λ i /Λ j .ȿ 1 (t)= 1 ȿ 2 (t) ȿ 3 (t) ȿ 4 (t) ȿ 5 (t)= 1 ȿ 2 (t)/ȿ 1 (t) ȿ 3 (t)/ȿ 2 (t) ȿ 4 (t)/ȿ 3 (t) ȿ 5 (t)/ȿ 4 (t)Figure 14 : A path with three hidden BiLU units connecting one input unit to one output unit. During the application of the stochastic balancing algorithm, at time t each unit i has a cumulative scaling factor Λ i (t), and each directed edge from unit j to unit i has a scaling factor M ij (t) = Λ i (t)/Λ j (t).The λ i (t) must remain within a finite closed interval away from 0 and infinity. To see this, imagine for instance that there is a subsequence of Λ 3 (t) that approaches 0. Then there must be a corresponding subsequence of Λ 4 (t) that approaches 0, or else the contribution of the weight w 43 Λ 4 (t)/Λ 3 (t) to the regularizer would go to infinity. But then, as we reach the output layer, the contribution of the last weight w 54 Λ 5 (t)/Λ 4 (t) to the regularizer goes to infinity because Λ 5 (t) is fixed to 1 and cannot compensate for the small values of Λ 4 (t). And similarly, if there is a subsequence of Λ 3 (t) going to infinity, we obtain a contradiction by propagating its effect towards the input layer.Finally, we prove that given an initial set of weights W , the final balanced state is unique and independent of the order of the balancing operations. The coefficients Λ i corresponding to a globally balanced state must be solutions of the following optimization problem:EQUATIONunder the simple constraints: Λ i > 0 for all the BiLU hidden units, and Λ i = 1 for all the visible (input and output) units. In this form, the problem is not convex. Introducing new variables M j = 1/Λ j is not sufficient to render the problem convex. Using variables M ij = Λ i /Λ j is better, but still problematic for 0 < p ≤ 1. However, let us instead introduce the new variables L ij = log(Λ i /Λ j ). These are well defined since we know that Λ i /Λ j > 0. The objective now becomes:EQUATIONThis objective is strictly convex in the variables L ij , as a sum of strictly convex functions (exponentials). However, to show that it is a convex optimization problem we need to study the constraintsΛ1 Λ2 Λ3 Λ4 Λ5 Λ2/Λ1 Λ3/Λ2 Λ4/Λ3 Λ5/Λ4Figure 15 : A path with five units. After the stochastic balancing algorithm has converged, each unit i has a scaling factor Λ i , and each directed edge from unit j to unit i has a scaling factorM ij = Λ i /Λ j .The products of the M ij 's along the path is given by: Λ2Λ1 Λ3 Λ2 Λ4 Λ3 Λ5 Λ4 = Λ5 Λ1 . Accordingly, if we sum the variables L ij = log M ij along the directed path, we get L 21 + L 32 + L 43 + L 54 = log Λ 5 -log Λ 1 .In particular, if unit 1 is an input unit and unit 5 is an output unit, we must have Λ 1 = Λ 5 = 1 and thus: L 21 + L 32 + L 43 + L 54 = 0. Likewise, in the case of a directed cycle where unit 1 and unit 5 are the same, we must have: on the variables L ij . In particular, from the set of Λ i 's it is easy to construct a unique set of L ij . However what about the converse? Definition C.35. A set of real numbers L ij , one per connection of a given neural architecture, is self-consistent if and only if there is a unique corresponding set of numbers Λ i > 0 (one per unit) such that: Λ i = 1 for all visible units and L ij = log Λ i /Λ j for every directed connection from a unit j to a unit i.L 21 + L 32 + L 43 + L 54 + L 15 = 0. Λ1 Λ6 Λ2 Λ3 Λ4 Λ7 Λ5 Λ2/Λ1 Λ3/Λ2 Λ4/Λ3 Λ7/Λ4 Λ6/Λ5 Λ7/Λ6Remark C.36. This definition depends on the graph of connections, but not on the original values of the synaptic weights. Every balanced state is associated with a self-consistent set of L ij , but not every self-consistent set of L ij is associated with a balanced state.Proposition C.37. A set L ij associated with a neural architecture is self-consistent if and only if π L ij = 0 where π is any directed path connecting an input unit to an output unit or any directed cycle (for recurrent networks).Remark C.38. Thus the constraints associated with being a self-consistent configuration of L ij ' s are all linear. This resulting linear manifold L depends only on the architecture, i.e., the graph of connections, but not on the actual weight values. The strictly convex function R(L ij ) depends on the actual weights W . Different sets of weights W produce different convex functions over the same linear manifold. If E denotes the total number of connections, then obviously dim L ≤ E. In order to infer all the Λ i , there must exist at least one constrained path going through each node i. Thus, in a layered feedforward network, the dimension of L is given by: dim L = E -M , where here M denotes the size of the largest layer. Let us assume that the first path assigns a multiplier Λ i to unit i and the second path assigns a multiplier Λ ′ i to the same unit. By assumption we must have: α L ij + β L ij = 0 for the first path, and γ L ij + δ L ij = 0. But α + δ and γ + β are also paths from the input layer to the output layer and therefore:α L ij + δ L ij = 0 and γ L ij + β L ij = 0. As a result, α L ij = log Λ i = γ L ij = Λ ′ i .Therefore the assignment of the multiplier Λ i must be consistent across different paths going through unit i.Remark C.39. One could coalesce all the input units and all output units into a single unit, in which case a path from an input unit to and output unit becomes also a directed cycle. In this representation, the constraints are that the sum of the L ij must be zero along any directed cycle. In general, it is not necessary to write a constraint for every path from input units to output units. It is sufficient to select a representative set of paths such that every unit appears in at least one path.Proof. If we look at any directed path π from unit i to unit j, it is easy to see that we must have:EQUATIONThis is illustrated in Figures 15 and 16 . Thus along any directed path that connects any input unit to any output unit, we must have π L ij = 0. In addition, for recurrent neural networks, if π is a directed cycle we must also have: π L ij = 0. Thus in short we only need to add linear constraints of the form: π L ij = 0. Any unit is situated on a path from an input unit to an output unit. Along that path, it is easy to assign a value Λ i to each unit by simple propagation starting from the input unit which has a multiplier equal to 1. When the propagation terminates in the output unit, it terminates consistently because the output unit has a multiplier equal to 1 and, by assumption, the sum of the multipliers along the path must be zero. So we can derive scaling values Λ i from the variables L ij . Finally, we need to show that there are no clashes, i.e. that it is not possible for two different propagation paths to assign different multiplier values to the same unit i. The reason for this is illustrated in Figure 17 .We can now complete the proof Theorem C.34. Given a neural network of BiLUs with a set of weights W , we can consider the problem of minimizing the regularizer R(L ij over the self-admissible configuration L ij . For any p > 0, the L p regularizer is strictly convex and the space of self-admissible configurations is linear and hence convex. Thus this is a strictly convex optimization problem that has a unique solution (Figure 18 ). Note that the minimization is carried over self-consistent configurations, which in general are not associated with balanced states. However, the configuration of the weights associated with the optimum set of L ij (point A in Figure 18 ) must be balanced. To see this, imagine (p > 0), over the linear (hence convex) manifold of self-consistent configurations defined by the linear constraints of the form π L ij = 0, where π runs over input-output paths. The regularizer function depends on the weights. The linear manifold depends only on the architecture, i.e., the graph of connections. This is a strictly convex optimization problem with a unique solution associated with the point A. At A the corresponding weights must be balanced, or else a self-consistent configuration of lower cost could be found by balancing any non-balanced neuron. Finally, any other self-consistent configuration B cannot correspond to a balanced state of the network, since there must exist balancing moves that further reduce the regularizer cost (see main text). Stochastic balancing produces random paths from the origin, where L ij= log M ij = 0, to the unique optimum point A.that one of the BiLU units-unit i in the network is not balanced. Then we can balance it using a multiplier λ * i and replace Λ i by Λ ′ i = Λ i λ * . It is easy to check that the new configuration including Λ ′ i is self-consistent. Thus, by balancing unit i, we are able to reach a new self-consistent configuration with a lower value of R which contradicts the fact that we are at the global minimum of the strictly convex optimization problem.We know that the stochastic balancing algorithm always converges to a balanced state. We need to show that it cannot converge to any other balanced state, and in fact that the global optimum is the only balanced state. By contradiction, suppose it converges to a different balanced state associated with the coordinates (L B ij ) (point B in Figure 18 ). Because of the self-consistency, this point is also associated with a unique set of (Λ B i ) coordinates. The cost function is continuous and differentiable in both the L ij 's and the Λ i 's coordinates. If we look at the negative gradient of the regularizer, it is non-zero and therefore it must have at least one non-zero component ∂R/∂Λ i along one of the Λ i coordinates. This implies that by scaling the corresponding unit i in the network, the regularizer can be further reduced, and by balancing unit i the balancing algorithm will reach a new point (C in Figure 18 ) with lower regularizer cost. This contradicts the assumption that B was associated with a balanced stated. Thus, given an initial set of weights W , the stochastic balancing algorithm must always converge to the same and unique optimal balanced state W * associated with the self-consistent point A. A particular stochastic schedule corresponds to a random path within the linear manifold from the origin (at time zero all the multipliers are equal to 1, and therefore for any i and any j: M ij = 1 and L ij = 0) to the unique optimum point A.Remark C.40. From the proof, it is clear that the same result holds also for any deterministic balancing schedule, as well as for tied and non-tied subset balancing, e.g., for layer-wise balancing for each path π from an input unit to an output unit, going sequentially through the units 0, 1, . . . , n, where 0 corresponds to the input unit, and n corresponds to the output unit of the path. The set of paths in the constraints must cover all the units in the network.Proof. Let us assume that there is a consistent set of multipliers Λ 0 , . . . , Λ n associated with the coefficients L ii-1 = log M ii-1 along the path π, with Λ 0 = Λ n = 1. Since M ii-1 = Λ i /Λ ci-1i-1 , we can derive the multipliers Λ i iteratively by propagating information from the input unit to the output unit, in the form:EQUATIONUsing the boundary conditions Λ 0 = Λ n = 1 gives the formula in Theorem C.42. The same arguments given for BiLU units can be used to complete the proof.Remark C.43. Note that if all the units have the same exponent c associated with the scaling of their activation functions, then the linear constraints have the simplified form: i∈π c n+1-i L ii-1 = 0 (44) 



Reviews: review: title: title: A technique to reparameterize the weights of a neural network, with experiments on MNIST and the IMDB sentiment analysis dataset
main_review: main_review: The paper argues that a neural network trains faster and to better accuracy when the weights of a neural network satisfy a condition called “synaptic neural balancing”.

More technically, the paper proves that, in a neural network with leaky ReLU activation functions, for each neuron (or unit), there exists a functionally equivalent configuration of weights that satisfies the so-called “synaptic neural balance” condition. Synaptic neural balance of a neuron means that the Lp norm of incoming weights is equal to the Lp norm of outgoing weights.
The proof of the theoretical result is straightforward.

Then, the paper goes on with experiments on MNIST and the IMDB sentiment analysis dataset to support the claim that neural network training is faster and more efficient when the synaptic neural balance condition is enforced.

The paper is well written and reads well.



**On the novelty**

Although I don’t know the literature on the subject very well, questions related to how to initialize and reparametrize the weights optimally to achieve faster and better training have been studied for a very long time. The method presented in this work seems rather natural, so I’m a little skeptical about the novelty of this work. In particular, I am surprised that I did not see any occurrence of the word “reparametrization” in this manuscript. There is a broad literature on the effect of reparametrization of neural networks on optimization and generalization – see for example these two papers:

Salimans, Tim, and Durk P. Kingma. "Weight normalization: A simple reparameterization to accelerate training of deep neural networks." Advances in neural information processing systems 29 (2016).

Dinh, Laurent, et al. "Sharp minima can generalize for deep nets." International Conference on Machine Learning. PMLR, 2017.



**On the effectiveness of the method**

The thesis of the manuscript does make sense, intuitively. However, the experiments performed on MNIST and the IMDB sentiment analysis dataset, are very limited in scope, so it seems difficult to extrapolate from this study only. If the synaptic neural balancing method proved to be really effective, it would have implications not just for neuromorphic computing, but also more immediately for current deep learning models trained on GPUs. The fact that the technique is not used in SOTA deep learning makes me think that the benefits (if any) of the method are not worth the burden of making the training scheme more complicated by “rebalancing” the weights.

So the question is whether there is any advantage of this synaptic neural balancing method for neuromorphic computing specifically, compared to GPU-based neural networks (?)



**Other comments**

I am not favorable to using the term “bilinear” (in “BiLU activation function”), because “bilinear” has a very specific meaning in linear algebra, different from the one used here. I’d suggest more simply “leaky ReLU with amplification”, or “generalized leaky ReLU”, or something along these lines.
Also, the term “BiLU”, first mentioned at line 72, does not seem to be defined in the main text. A pointer to Appendix C1 (where it is defined) seems to be missing.

The notation g_w(w) in Eq 2 can be a little confusing. As I understand it, the two occurrences of w refer to two different things here: the first one refers to the *specific weight*, and the second one refers to the *value* of that weight. Maybe adding a one-sentence explanation would be helpful for the reader.
title: title: Interesting theory but application to emerging accelerators unclear
main_review: main_review: The submission describes the theory of neural balance, the idea that the total incoming and outgoing cost of a neuron should be equal.
While this property is normally not fulfilled in neural networks, the authors propose a regularization scheme that seeks to ensure neural balance holds. By construction, the theory is restricted to activations that are linear in both directions from zero, e.g. ReLU, a potential drawback given the popularity of ELU, SELU, or GELU functions which are slightly non-bi-linear.

The theory is interesting and the regularization shows some merit in experiments. Most experiments are conducted on standard fully connected networks (FCN) on MNIST, one experiment is done with a RNN on IMDB. The FCN MNIST experiments show that L1 regularization appears to give the strongest results, while the RNN experiment only shows marginal improvement.

Neural balance appears to yield stronger improvements in the data starved regime, leading to improvements in MNIST classification of up to 8.4%. However, the experimental section here suffers from a lack of baselines or other methods to compare against (other regularizations, dropout, data augmentation). It is not entirely clear to me if the balancing process itself incurs additional compute. If so, then this should be acknowledged and results should be compared against compute-matched baselines (e.g. augmentation). If the regularization terms are indeed just added to the overall loss function, then no (significant) additional compute cost is incurred. 

The theory could be made stronger. For example, one could consider information theoretic or other perspectives to explain why and how neural-balanced networks are desirable.

My main point of contention, however, is if this submission fits within the scope of the workshop. How would, for instance, an analog accelerator benefit from neural balance? The authors should make this point quite clear. If for instance, neural balanced weight matrices are easier to quantize or more noise robust, then I see how this submission would make sense.
But otherwise the relationship to emerging AI accelerators is very unclear to me; I believe the submission should go to a different workshop or conference.