Correlations Are Ruining Your Gradient Descent

## Abstract

Abstract Herein the topics of (natural) gradient descent, data decorrelation, and approximate methods for backpropagation are brought into a common discussion.
Natural gradient descent illuminates how gradient vectors, pointing at directions of steepest descent, can be improved by considering the local curvature of loss landscapes.
We extend this perspective and show that to fully solve the problem illuminated by natural gradients in neural networks, one must recognise that correlations in the data at any linear transformation, including node responses at every layer of a neural network, cause a non-orthonormal relationship between the model’s parameters.
To solve this requires a method for decorrelating inputs at each individual layer of a neural network.
We describe a range of methods which have been proposed for decorrelation and whitening of node output, and expand on these to provide a novel method specifically useful for distributed computing and computational neuroscience.
Implementing decorrelation within multi-layer neural networks, we can show that not only is training via backpropagation sped up significantly but also existing approximations of backpropagation, which have failed catastrophically in the past, benefit significantly in their accuracy and convergence speed.
This has the potential to provide a route forward for approximate gradient descent methods which have previously been discarded, training approaches for analogue and neuromorphic hardware, and potentially insights as to the efficacy and utility of decorrelation processes in the brain.

## 1 Introduction

The method of gradient descent is as popular as it is intuitive.
This method, of stepping in the direction of steepest descent of a function, is applied successfully across the engineering sciences as well as in the modern AI revolution to find (local) optima of arbitrary functions.
Alternative optimization methods have proven largely unsuccessful in being generally applied to continuous functions of arbitrary form, with second-order methods being largely brittle when applied to non-convex functions.
Nonetheless, methods for speeding-up optimizations are not only interesting but have huge potential economic and environmental impact.

In 1998 it was proposed that there might be a perspective beyond typical gradient descent, called natural gradient descent [Amari, ], which might overcome some elements of skew and scale in the updates produced by gradient descent.
Natural gradient descent has since been explored at the edges of the field of optimization, and deep neural network training [Bernacchia et al., , Martens and Grosse, , Desjardins et al., , Heskes, ], with greater stability than traditional second order methods [Dauphin et al., ].
However, it is (as a principle) less widely understood, less applied, and less intuitive than it could be.

Simultaneous to this line of development, the principles behind learning in natural biological systems and potential algorithms for learning in distributed systems have been under investigation [Lillicrap et al., ].
From these fields have sprung a whole range of approximate methods for gradient descent which promise to explain how learning might occur in brains or how it might be enabled in analogue hardware (neuromorphic, analogue, or otherwise).

We aim to bring together these lines of research, contributing to each individually while also providing a common space for impact.
Specifically,

- 1.
We demonstrate that correlations in data (between features) at the input and hidden layers of deep networks are one half of natural gradients, and that they contribute to a non-orthonormal basis in parameters,
- 2.
We explore and expand the efficacy of methods for removing correlations from deep neural networks to better align gradient descent with natural gradients, and
- 3.
We show that decorrelating mechanisms not only speed up learning by backpropagation, but that decorrelation can also enable alternatives approximations to gradient descent.

This paper is organised in an unconventional format due to the multiple sub-fields within which it is embedded and to which it contributes.
Therefore, one should consider each of the coming sections as descriptions of a particular contribution, insight, or result embedded within a larger narrative.
The titles of each section represent their core contribution to this narrative.

The intention of this work is to bring insight to those who wish for more efficient gradient descent and excitement to those interested in approximate gradient descent methods whether for explanation of learning in biological systems or implementation in physical/neuromorphic systems.

## 2 Data correlations can cause parameters to enter a non-orthonormal relation

Here we address the first of our goals.
We describe gradient descent, its relation to natural gradients, and demonstrate the often ignored aspect of input correlations impacting parameter orthonormality.

### 2.1 Gradient Descent

Consider the case in which we have a dataset which provides input and output pairs $(\bm{x},\bm{y})\in\mathcal{D}$, and we wish for some parameterised function, $\bm{z}=\bm{f}_{\bm{\theta}}(\bm{x})$, with parameters $\bm{\theta}$ to produce a mapping relating these.
This mapping would be optimal if it minimised a loss function $\mathcal{L}(\bm{\theta})=\frac{1}{|\mathcal{D}|}\sum_{(\bm{x},\bm{y})\in
\mathcal{D}}\ell(\bm{f}_{\bm{\theta}}(\bm{x}),\bm{y})$, where the sample-wise loss function, $\ell$, can be a squared-error loss for regression, the negative log-likelihood for classification, or any other desired cost.

The problem which we wish to solve in general is to minimise our loss function and find the optimal set of parameters, effectively to find ${\operatorname{argmin}}_{\bm{\theta}}\,\mathcal{L}(\bm{\theta})$.
However, finding this minimum directly is challenging for most interesting problems.
Gradient descent proposes a first-order optimisation process by which we identify an update direction (not directly the optimized value) for our parameters based upon a linearization of our loss function.
This is often formulated as taking a ‘small step’ in the direction steepest descent of a function, in its gradient direction, such that

$$ $\delta{\bm{\theta}}^{\text{GD}}=-\eta\nabla\mathcal{L}(\bm{\theta}),$ $$

where $\eta$ is the step size.

However, this supposition of taking a ’small step’ in the gradient direction hides a specific assumption.
In fact, it is equivalent to minimising a linear approximation (1st order Taylor expansion) of our loss function with an added penalization based upon the change in our parameters.
Specifically, this is the optimum solution of

$$ $\delta{\bm{\theta}}^{\text{GD}}=\underset{\delta\bm{\theta}}{\operatorname{ argmin}}\,\mathcal{L}(\bm{\theta})+\nabla\mathcal{L}(\bm{\theta})^{\top}\delta \bm{\theta}+\frac{1}{2\eta}{\delta\bm{\theta}}^{\top}\delta\bm{\theta}.$ $$

where, ${\delta\bm{\theta}}^{\top}\delta\bm{\theta}=||\delta\bm{\theta}||^{2}_{2}=\sum
_{i}\delta\bm{\theta}_{i}^{2}$.
Note, one can derive gradient descent by simply finding the minimum of this optimisation function (where the derivative with respect to $\delta\bm{\theta}$ is zero).
Thus, gradient descent assumes that it is sensible to measure and limit the distance of our parameter update in terms of the squared (Euclidean) norm of the parameter change.
Natural gradients supposes that this is not the best choice.

### 2.2 Natural Gradient Descent

Figure: Figure 1: Left is shown a loss function landscape (in parameter space) which is a well-conditioned convex loss. In red is shown a model being optimised and the gradient descent direction (grey arrow) points toward the minimum of the function. Right, the loss function is modified to include a positive semi-definite scaling matrix ($\mathcal{L}=\sum_{(\bm{x},\bm{y})\in\mathcal{X}}(\bm{W}\bm{x}-\bm{y})^{\top} \bm{A}(\bm{W}\bm{x}-\bm{y})$.) which causes a skew and stretch of the loss landscape. Gradient descent no longer points toward the minimum but instead in the direction of steepest descent (a detour), whereas natural gradient descent (orange arrow) points directly once more at the loss function minimum.
Refer to caption: x1.png

Amari [], proposed the concept of natural gradients, and put forth the Riemann metric as the sensible alternative to a regular Euclidean measurement of distance.
Specifically, rather than define squared distance with the Euclidean metric ($|\delta\bm{\theta}|^{\text{Euclidean}^{2}}=\sum_{i}\delta\bm{\theta}_{i}^{2}$), you can instead use a general metric instead,

$$ $|\delta\bm{\theta}|^{\text{Riemannian}}=\sum_{i}\sum_{j}g_{ij}\delta\bm{\theta }_{i}\delta\bm{\theta}_{j}=\delta{\bm{\theta}}^{\top}\bm{G}(\bm{\theta})\delta \bm{\theta}$ $$

where $\bm{G}$ is the Riemann metric matrix.
Note, that the values of the matrix, $\bm{G}$, are a function of $\bm{\theta}$ and are thus not static but depend upon $\bm{\theta}$.
We can now make use of our Riemannian metric in place of the previous Euclidean metric such that,

$$ $\delta\bm{\theta}^{\text{NGD}}=\underset{\delta\bm{\theta}}{\operatorname{ argmin}}\,\mathcal{L}(\bm{\theta})+\nabla\mathcal{L}(\bm{\theta})^{\top}\delta \bm{\theta}+\frac{1}{2\eta}{\delta\bm{\theta}}^{\top}\bm{G}\delta\bm{\theta}$ $$

and thus, by finding the minimum of this optimisation (by the first derivative test), we can arrive at a neat formulation of natural gradient descent,

$$ ${\delta\bm{\theta}}^{\text{NGD}}=-\eta\bm{G}^{-1}\nabla\mathcal{L}(\bm{\theta}).$ $$

Natural gradient descent aims to ensure that it is not the steepest direction of descent which is taken, but instead the direction which undoes any skew in the loss function to arrive at a more direct descent toward (local) minima.
This difference is illustrated in Figure 1.

One question remains, how might one arrive at a form for the Riemann metric matrix, $\bm{G}$?

#### The loss distance as a penalty

Natural gradient approaches tend to find the form of this matrix by redefining models as probabilistic and thereafter forming a connection to the Fischer information matrix and information geometry [Amari, , Martens, ].
This unfortunately both obfuscates the intuition for natural gradients and disentangles it from deterministic (point) models with arbitrary losses.

We choose instead to describe this optimiztion in the deterministic regime and provide intuition of its impact, in a manner similar to that of Heskes [].
Suppose that, instead of penalizing the Euclidean distance of our parameter change, that we instead penalize the distance travelled in loss space.
Mathematically, we are supposing that

$$ $\delta\bm{\theta}^{\top}\bm{G}\delta\bm{\theta}:\approx\frac{1}{|\mathcal{D}|} \sum_{(\bm{x},\bm{y})\in\mathcal{D}}(\ell(\bm{f}_{\bm{\theta}+\delta\bm{\theta }}(\bm{x}),\bm{y})-\ell(\bm{f}_{\bm{\theta}}(\bm{x}),\bm{y}))^{2}.$ $$

Note, we here measure distance in terms of individual loss samples as otherwise one loss sample’s value can increase while another’s decreases - i.e. the total loss change can be degenerate to changes in the sample-wise losses.

It can be shown that by expanding this term with its Taylor series, see Appendix A, one arrives at
$\bm{G}=\left\langle\nabla_{\bm{\theta}}\ell^{\top}\nabla_{\bm{\theta}}\ell
\right\rangle_{x}$, bringing us to the same solution as found in general for the natural gradients learning rule update, where

$$ ${\delta\bm{\theta}}^{\text{NG}}=-{\eta}\left\langle\nabla_{\bm{\theta}}\ell^{ \top}\nabla_{\bm{\theta}}\ell\right\rangle^{-1}_{x}\nabla_{\theta}\mathcal{L}.$ $$

This intuition, that natural gradients can be viewed as a method for taking a step of optimization while regularizing the size/direction of the step in terms of loss difference, is a perspective which we believe is more interpretable while arriving at an equivalent solution.

### 2.3 Natural Gradient Descent for a DNN

Moving beyond the simple case of regression, we zoom into this problem when applied to multilayer neural networks.
Defining a feed-forward neural network as

$$ $\bm{x}_{i}=\phi(\bm{h}_{i})=\phi(\bm{W}_{i}\bm{x}_{i-1})$ $$

for $i\in[1...L]$, where $L$ is the number of layers in our network, $\phi$ is a non-linear activation function (which we shall assume is a fixed continous transfer function for all layers), $\bm{W}_{i}$ is a matrix of parameters for layer $i$, $\bm{x}_{0}$ is our input data, and $\bm{x}_{L}$ is our model output.
We ignore biases for now as a simplification of our derivation.

Martens and Grosse [] (as well as Desjardins et al. [] and Bernacchia et al. []) provided a derivation for this quantity.
We demonstrate this derivation in Appendix B, and present its conclusions and assumptions here in short.
Note that a first assumption is made here, that the natural gradient update can be computed for each layer independently, rather than for the whole network.
This approximation not only works in practice [Desjardins et al., ] but is also fully theoretically justified for linear networks [Bernacchia et al., ].
Taking only a single layer of a network, one may determine that

$$ $\bm{G}_{\bm{W}_{i}}=\langle\nabla_{\theta_{\bm{W}_{i}}}\ell^{\top}\,\nabla_{ \bm{\theta}_{\bm{W}_{i}}}\ell\rangle_{\bm{x}_{0}}=\left\langle\text{Vec}\left( \frac{\partial\ell}{\partial\bm{h}_{i}}\bm{x}_{i-1}^{\top}\right)^{\top}\text{ Vec}\left(\frac{\partial\ell}{\partial\bm{h}_{i}}\bm{x}_{i-1}^{\top}\right) \right\rangle_{\bm{x}_{0}}\\$ $$

where the update is computed for a (flattened) vectorised set of parameters, indicated by the $\text{Vec}()$ function.

After inversion, multiplication by the gradient, and reorganisation using the Kronecker mixed-product rule (with the additional assumption that the gradient signal is independent of the activation data distribution) one arrives at

$$ ${\delta\bm{\theta}}_{\bm{W}_{i}}=-{\eta}\text{Vec}\left(\left\langle\frac{ \partial\ell}{\partial\bm{h}_{i}}^{\top}\frac{\partial\ell}{\partial\bm{h}_{i} }\right\rangle^{-1}_{\bm{y}}\left\langle\frac{\partial\ell}{\partial\bm{h}_{i} }\bm{x}_{i-1}^{\top}\right\rangle_{\bm{x}_{i-1}}\left\langle\bm{x}_{i-1}^{\top }\bm{x}_{i-1}\right\rangle^{-1}_{\bm{x}_{i-1}}\right).\\$ $$

Examining the terms of this update, we can see that the original gradient term (as you would compute by backpropagation) is in the middle of this equation.
From the left it is multiplied the inverse correlation of the gradient vectors - tackling any skews in loss landscape as we visualized above.
From the right the gradient descent update is multiplied by the inverse data correlation - specifically the data which acts as an input to this particular layer of the network.
The presence of the inverse of the data correlation term is not only surprising but also an underappreciated aspect of natural gradients.

Note that we here refer to these outer product terms as correlations, despite the fact that a true correlation would require a centering of data (as would a covariance) and normalization.
This is for ease of discussion.

### 2.4 The issue with data correlations

Figure: Figure 2: Left, the gradient descent vector is shown for a linear regression problem, though now the input data $\bm{x}$ has a correlation structure ($\langle\bm{x}\bm{x}^{\top}\rangle=\bm{\Sigma}$. Right, if the correlation matrix is integrated into the parameters, the alignment with direction of steepest descent in this landscape is returned.
Refer to caption: x2.png

In Figure 2, we visualise the impact of data correlations within a linear regression problem.
Note that, as for Figure 1, we are visualising a loss landscape for a simple linear regression problem (equivalent to a single layer network).
As is clear, when our data itself has some correlation structure present, gradient descent once again points off-axis such that it would take a detour during optimization.

It is ultimately rather trivial to show why data correlations ruin gradient descent.
If you consider a new set of data with some correlation structure, $\langle\bm{x}\bm{x}^{\top}\rangle=\bm{\Sigma}$, you may equally (supposing well conditioned data) write this data as a whitened dataset multiplied by the matrix square root of the correlation matrix, $\bm{x}=\bm{\Sigma}^{1/2}\bar{\bm{x}}$ where $\langle\bar{\bm{x}}\bar{\bm{x}}^{\top}\rangle=\bm{I}$.
As such, we can look at the output of a linear model of our inputs, $\hat{\bm{z}}=\bm{W}\bm{x}=(\bm{W}\bm{\Sigma}^{1/2})\bar{\bm{x}}$, as a model in which this additional correlation matrix is a matrix by which our parameters are being brought into a non-orthonormal relationship.
Thus, if we compute gradient descent with respect to our parameters, $\bm{W}$, without accounting for the correlation inducing matrix, $\bm{\Sigma}^{1/2}$, we have updated parameters which are no longer orthonormal, compare Figure 2 right.

This perspective, that input correlations at every layer of a deep neural network cause a non-orthonormal relationship between parameters, is our first major contribution.
In the work which follows, we focus upon undoing data correlations and investigate how this impacts learning in neural networks.

## 3 Simple decorrelation mechanisms can rid networks of data correlations

Above we have shown that data correlations impact the relationship between parameters at linear transformations.
Thereby, the direction of gradient descent is skewed.
We now go on to show how this can be corrected for.

There are multiple potential routes for correction of the gradient descent direction.
One may measure correlation structure and directly invert and apply this inversion to the gradient updates to move toward a natural gradient descent update rule.
However, the data being fed into one’s parameters is still correlated and therefore continues to contribute to a non-orthonormal basis.

Alternatively, we here describe methods for modifying neural network models such that the neural outputs at each layer are decorrelated via some operation, no matter whether we are doing inference or training.
Regular gradient descent in such a case is now closer in its update to natural gradients and furthermore our parameters can now relate in an orthogoal basis.

Note that hereafter we make use of gradient descent to update models and attempt to remove data correlations within our models.
Notably, we do not attempt to remove gradient-correlations, the better known aspect of natural gradients.
Thus performance is potentially left available via that route.

### Existing methods for decorrelation

Data correlations can be removed from every layer of a deep neural network via a number of methods.
These correlations must be undone continuously to keep up with changing correlation structures within deep neural networks while they are trained.
Two main approaches exist to tackle the issue of removing data correlations: measurement of the correlation and inversion (via matrix decomposition) or a direct, continually updated, estimate of a matrix which can achieve a decorrelated outcome.

A number of works exist for directly taking the inverse-square-root of the correlation matrix of some data.
Desjardins et al. [] did so by measuring the correlation matrix of data at every layer of a neural network (at pre-defined checkpoints) and thereafter carried out a matrix decomposition and an inverse (as did Luo []).
Bernacchia et al. [] did this at a minibatch-level, measuring correlations within a mini-batch and inverting these individually.
Batch-normalization has also been extended to whitening and decorrelation for deep neural network training in a similar fashion [Huang et al., , ], and this principle has been extended to much deeper networks, though also while stepping away from the theoretical framing of natural gradients.
These works went so far as to apply decorrelation methods to extremely deep networks (101-layer) networks [Huang et al., , ].

Few methods have considered iteratively computing a decorrelation matrix directly (i.e. without inversion or matrix decomposition) [Ahmad et al., , Dalm et al., ].
The methods which do exist, optimise a matrix for decorrelation alongside the regular weight matrices in a neural network by construction of an appropriate loss function capturing how data should be modified for correlation reduction.
These methods, however, were thus far not theoretically linked to natural gradients.

From computational neuroscience however, a number of methods for dynamic and recurrent removal of data correlations were proposed from the perspective of competitive learning and inhibitory control [Földiák, , Pehlevan et al., , Oja, , Vogels et al., ].
These methods, in contrast to all the previously described approaches, were however only applied in single layer networks, mostly in unsupervised contexts.

Here we present a method which bridges across all of the above, allowing fast and stable decorrelation while having a formulation within a recurrent system of dynamics.

### A note on decorrelation vs whitening

In this work we limit our emphasis on whitening and focus more upon decorrelation.
The reason for this is two-fold. First, the off-diagonal elements of the decorrelation matrices cause the greatest impact in skewing gradient descent.
The diagonal elements simply scale up/down the gradient vector and this can be trivially dealt with by modern optimizers (e.g. with the Adam optimizer [Kingma and Ba, ]).
Second, for data in which features are often zero or extremely sparse, normalizing for unit variance can result in extremely large valued activations (distributions with extremely long tails) or unstable updates in the stochastic updating regime.

### A novel decorrelation mechanism

Figure: Figure 3: Left, existing methods for decorrelation through a system of dynamics (example of Földiák [], representative of Pehlevan et al. [], Vogels et al. [] and more) reduce activity correlations in a manner which is affected by the present scale (eigenvalues) of the decorrelating matrix. Right, existing methods also reduce the scale of all activity in a layer while decorrelating - tending toward the trivial solution of zero activity. The approach proposed in this work solves both of these issues.
Refer to caption: extracted/5894907/Figures/DecorrelationStats.png

To move beyond existing decorrelation methods, we propose a method of decorrelation within a neural network with the following properties:

- 1.
Learns to decorrelate consistently, regardless of the scale of the decorrelation matrix
- 2.
Ensures that decorrelation does not reduce net activity in a layer
- 3.
Allows formulation of decorrelation as a distributed dynamical system

We propose that, at all layers of a neural network, one may construct a linear decorrelating transformation of the data, such that $\bar{\bm{x}}_{i}=\bm{M}_{i}\bm{x}_{i}$ where $\bar{\bm{x}}_{i}$ is intended to be a decorrelated form of the data ${\bm{x}}_{i}$.
This means that a network now has one additional linear transformation per layer and the full network’s computation is now written

$$ $\bm{x}_{i}=\phi(\bm{h}_{i})=\phi(\bm{W}_{i}\bar{\bm{x}}_{i-1})=\phi(\bm{W}_{i} \bm{M}_{i-1}\bm{x}_{i-1})$ $$

where all elements are as described previously, with the addition of a square decorrelating matrix $\bm{M}_{i}$ at every layer.

In order to learn this decorrelating matrix, one may update the decorrelating matrix in an iterative fashion such that

$$ $\bm{M}_{i}\leftarrow g_{i}(\bm{M}_{i}-\eta_{M}\langle\bar{\bm{x}}_{i}\bar{\bm{ x}}_{i}^{\top}\rangle\bm{M}_{i})$ $$

where $\eta_{M}$ is a learning rate for this update, and $g_{i}$ is a scalar gain which ensures that each of the layer’s decorrelated outputs has the same norm as it had prior to decorrelation, $g_{i}=||\bm{x}_{i}^{2}||_{F}/||\bar{\bm{x}}_{i}^{2}||_{F}$.
This update minimises a loss capturing the correlations in $\bar{\bm{x}}$ in the style of Ahmad et al. [].
Updates can be carried out in a stochastic fashion by simply measuring the correlations in $\bar{x}$ within a minibatch for iterative updating, see Appendix D for the full pseudocode for such updating.
Note that one can further improve upon the level of decorrelation by de-meaning the input data prior to this decorrelation process, such that $\hat{\bm{x}_{i}}=\bm{M}_{i}(\bm{x}_{i}-\bm{\mu}_{i})$ where $\bm{\mu}_{i}$ is a unit-wise learned mean or batch-wise computed mean.
We find this to further improve performance in practice.

One might enquire as to why this method is useful or interesting.
Examining Figure 3, this decorrelation rule is effective at reducing the level of correlation at a network layer regardless of the existing scale (eigenvalues) of the decorrelating matrix - a problem faced by the existing learning rules of Földiák [], but also by the similar rules proposed by Pehlevan et al. [], Oja [], Vogels et al. [].
Furthermore, this rule, via the (layer-wise) gain factors $g_{i}$, ensures that the scale (norm) of activities at any given layer remain at the existing scale, rather than reducing, see Figure 3 right.
This ensures that decorrelating data does not tend towards the trivial decorrelation solution in which unit activities tend to zero.

Finally, aside from each of these benefits, this system can be easily converted to a system of recurrent dynamics with local-information, see Appendix C.
This is particularly a benefit if one wished to implement this method in a neuromorphic device or to link it to biological nervous systems.

## 4 Approximate methods for gradient descent work significantly better when coupled with decorrelation

The efficacy of decorrelation rules for improving the convergence speed of backpropagation trained algorithms is significant, as demonstrated in existing work [Huang et al., , , Dalm et al., ].
Rather than focus upon scaling this method to networks of significant depth (e.g. ResNet architectures as investigated by Huang et al. [] and Dalm et al. []) we focus upon shallower networks and show how removal of data correlations enables existing approximate methods for backpropagation.

Figure: Figure 4: Adding a decorrelation mechanism at every layer of a neural network can massively speed up training convergence speed. Results are shown for backpropagation, feedback alignment, and node perturbation train upon the CIFAR10 in four hidden-layer dense networks, and for backpropagation and feedback alignment upon the CIFAR100 and TinyImageNet classification tasks in five hidden-layer convolutional neural networks. All learning methods were combined with the Adam optimizer [Kingma and Ba, ] and the categorical cross entropy loss. Network architectures and training hyperparameters are described in detail in Appendix E. Envelopes show the max and min accuracy levels across five randomly seeded networks. Parameters were selected based upon an independent grid search over decorrelation learning rates and forward learning rates for maximum validation set performance.
Refer to caption: x3.png

It has been found that many existing ‘biologically plausible’ learning rules (i.e. ones which substitute backpropagation of error for alternative methods of gradient assignment which can be considered more plausible for distributed networks such as the brain) do not effectively scale to deeper networks and harder tasks [Bartunov et al., ].
Recently, Dalm et al. [] made use of the decorrelation rule proposed by Ahmad et al. [] in order to enable training of multi-layer neural networks via the node-perturbation algorithm.
In doing so, they did not draw a direct relation to the theories of natural gradients but were aided inadvertently by this effect.

Here we show that not only does decorrelation improve training of multi-layer neural networks with the node-perturbation algorithm (as has already been shown by Dalm et al. []) but also significantly improves multi-layer network training when combined with implementation of the feedback alignment algorithm.
Networks parameters in pseudocode, as well as the hyperparameters which were used for training are presented in Appendix E and an example of the computational pseudocode provided in Appendix D.
Code to reproduce these simulations is available at [https://github.com/nasiryahm/CorrelationsRuinGD](https://github.com/nasiryahm/CorrelationsRuinGD).
Note that all simulations shown below are presented based upon a parameter grid search which was carried out individually for each credit assignment method and best parameters selected for each curve based upon a validation set.

Figure 4 shows the test-set performances of fully-connected and convolutional neural network models trained under various conditions and upon various tasks.
As can be seen, backpropagation (BP) when coupled with decorrelation benefits from significant increases in generalization peformance and training speed in dense networks and smaller convolutional networks.
Similarly, node perturbation (NP) is massively sped up, though suffers from a generally lower accuracy in this training regime.

Most significantly, the accuracy achieved by feedback alignment (FA) is increased far above what was previously achievable, even surpassing the accuracy of backpropagation in a dense network.
For convolutional networks, the inclusion of decorrelation has a significant impact on peak performance and increases the speed of learning by orders of magnitude.

## 5 Discussion

### 5.1 Decorrelation enables approximations of gradient descent

The success of decorrelation in enabling learning with the node perturbation and feedback alignment algorithms indicates that, via removal of cross-correlation structure between nodes in layers of a neural network, minimisation of the loss function is made much ‘easier’.
These two algorithms are by no means the only algorithms which might be enabled by a decorrelation mechanism, and if one is interested in training of neural networks upon exotic hardware, one might consider combining such a distributed decorrelation mechanism with a number of alternative learning algorithms such as direct feedback alignment [Lillicrap et al., , Nøkland, ], surrogate gradient learning Neftci et al. [] and more.

### 5.2 Decorrelation in biological systems

Yet another parallel stream of this work which has engaged with concepts of decorrelation and whitening is that which has taken place in the context of biological neural networks, both on a computational and experimental level.

In the neurosciences, decorrelation has been presented as an explanation of the filtering which happens at multiple levels in nervous systems. The center-surround processing which takes places at the earliest stage of visual processing (at retinal ganglion cells), for example, has been proposed as an aid for decorrelation of visual input to the brain [Pitkow and Meister, ]. At a more general scale, inhibitory plasticity appears to be learned in a manner which also leads to spatial decorrelation [He and Cline, ], something which has also been modelled [Vogels et al., ].

Beyond experimental work, simulation work has also shown that via decorrelation of unit activities, competitive learning can be established to learn subspaces in a distributed and ‘local’ fashion [Bell and Sejnowski, , Földiák, , Oja, ], something which can also be extended for the reproduction of early visual cortical receptive fields [Zylberberg et al., ].

Though much of this work is framed by work in computational neuroscience, the strict decorrelation which we employ to overcome the issues inherent to gradient descent is far from the kind of decorrelation observed in real neural systems.
Neural data suggests that nearby neurons can have correlated activities - a feature which may be required for redundancy and robustness to cell death.
Furthermore, the decorrelation mechanisms which have been observed are often learned in a slow manner and on many different scales depending upon the brain area under consideration.
All in all, though we point toward a promising method by which local and distributed learning can be enabled, there remain significant questions about how much of this analysis might be directly mappable to neural systems.
Finally, a more detailed model of neuronal interaction, for example in the style proposed by Lipshutz and Simoncelli [], would be more appropriate for providing a detailed proposal on the role of different neuron types in decorrelation.

### 5.3 Decorrelation and overfitting

Wadia et al. [] pointed out that the restriction to whitened data or second-order optimization methods can restrict the space of generalization unless properly regularized.
This issue is one to keep in mind when developing methods which speed up training of models significantly and generalization performance should be a point of concern.
However, as also described by Wadia et al. [], it may be that a regularized form of whitening/decorrelation would in fact be optimal for training for maximum generalization performance.

We find that our decorrelation approach has a combination of benefits: increased convergence speed along with increased generalization performance.
One core difference of our approach is a focus upon decorrelation over whitening.
Whitening focuses upon not only decorrelation but a rescaling of all unit activities for a response variance of one.
We speculate that this causes a rescaling of not only ‘signal’ inherent in network activities, but also ‘noise’, and that noise is inherently of a smaller scale than signal in trained multi-layer networks.
Thus we propose that decorrelation might be a better tradeoff, where signal correlations are removed without rescaling them and thus the scale of noise (or training-data specific signal variations) is not increased and used for overfitting.

## 6 Conclusion

Here we present an integration of a range of research directions including natural gradient descent, decorrelation and whitening, as well as approximate or alternative methods for gradient descent.
Notably we illustrate and describe how one component of natural gradients is often overlooked and can be framed as correlations in feature data (at all layers of a neural network) bringing parameters into a non-orthonormal relationship.
We show that data correlation in neural networks can be removed, in a similar fashion to neural competition, through an additional transformation at each layer of a network.
Removing these correlations allows much faster training of neural networks with backpropagation but also magnitudes faster training of neural networks via alternative approximate algorithms.
These results and insights together suggest that failures of ‘biologically-plausible’ learning approaches and learning rules for distributed computing could be overcome through decorrelation and the return of parameters to an orthogonal basis.

## Appendix A Loss distance the approriate Riemann metric for natural gradients

In the main text, we propose that one may find the appropriate form of the Riemann metric for gradient descent by supposing that the metric measured should tend to the total summed sample-wise loss distance

$$ $\delta\theta^{\top}G\delta\theta\rightarrow\frac{1}{|\mathcal{D}|}\sum_{(x_{0} ,y)\in\mathcal{D}}(\ell(f(\theta+\delta\theta,x_{0}),y)-\ell(f(\theta,x_{0}),y ))^{2}.$ $$

By expanding the loss function with it’s Taylor series,

$$ $\displaystyle(\ell(f(\theta+\delta\theta,x_{0}),y)-\ell(f(\theta,x_{0}),y))^{2}$ $\displaystyle=(\ell+\nabla_{\theta}\ell{\delta\theta}+{\delta\theta}^{\top} \nabla^{2}_{\theta}\ell{\delta\theta}+...-\;\ell)^{2}$ (1) $\displaystyle=(\nabla_{\theta}\ell{\delta\theta}+{\delta\theta}\nabla^{2}_{ \theta}\ell{\delta\theta}+...)^{2}$ (2) $\displaystyle={\delta\theta}^{\top}\nabla_{\theta}\ell^{\top}\nabla_{\theta} \ell{\delta\theta}+2(\nabla_{\theta}\ell{\delta\theta}\times{\delta\theta}^{ \top}\nabla^{2}_{\theta}\ell{\delta\theta})+...$ (3) $$

As the change in parameters tend to small values (or equivalently as $\lim_{\eta\rightarrow 0}$), this term can be approximated by its first term.
This first term, when used as the measure, is thus a simple equivalence to the Riemann metric

$$ ${\delta\theta}^{\top}G\delta\theta={\delta\theta}^{\top}\left\langle\nabla_{ \theta}\ell^{\top}\nabla_{\theta}\ell\right\rangle_{x_{0}\in\mathcal{D}}{ \delta\theta}.$ $$

## Appendix B Derivation for the Riemann metric for a deep neural network

Defining our model as

$$ $x_{i}=\phi(h_{i})=\phi(W_{i}x_{i-1})$ $$

for $i\in[1...L]$, where $L$ is the number of layers in our network, $\phi$ is a non-linear activation function (which we shall assume is a fixed continuous transfer function for all layers), $W_{i}$ is a matrix of parameters for layer $i$, $x_{0}$ is our input data, and $x_{L}$ is our model output.
We ignore biases for now as a simplification of our derivation.

First let us begin by collecting all parameter matrices into a single vector of parameters, such that $\theta=\text{Vec}(W_{1},W_{2},...W_{L})^{\top}$ where $\text{Vec}(\cdot)$ indicates the flattening of a tensor into a vector.
Next we can define, the derivative of our loss with respect to the parameters,

$$ $\nabla_{\theta}\mathcal{L}=\left(\frac{\partial\mathcal{L}}{\partial\text{Vec} (W_{1},W_{2},..W_{L})}\right)=\left\langle\text{Vec}\left(\frac{\partial\ell}{ \partial h_{1}}x_{0}^{\top},\frac{\partial\ell}{\partial h_{2}}x_{1}^{\top},.. .\frac{\partial\ell}{\partial h_{L}}x_{L-1}^{\top}\right)\right\rangle_{x_{0} \in\mathcal{D}}$ $$

where we have replaced the derivative terms with the layer computation as can be calculated by backpropagation.
We can now compute more straightforwardly the outer product of this gradient vector with itself, such that

$$ $\displaystyle G(\theta)$ $\displaystyle=\left\langle\nabla_{\theta}\ell^{\top}\nabla_{\theta}\ell\right \rangle_{x_{0}}$ $\displaystyle=\left\langle\text{Vec}\left(\frac{\partial\ell}{\partial h_{1}}x _{0}^{\top},\frac{\partial\ell}{\partial h_{2}}x_{1}^{\top},...\frac{\partial \ell}{\partial h_{L}}x_{L-1}^{\top}\right)^{\top}\text{Vec}\left(\frac{ \partial\ell}{\partial h_{1}}x_{0}^{\top},\frac{\partial\ell}{\partial h_{2}}x _{1}^{\top},...\frac{\partial\ell}{\partial h_{L}}x_{L-1}^{\top}\right)\right \rangle_{x_{0}}$ $$

This is now a matrix of shape $G(\theta)\in\mathbb{R}^{M\times M}$ assuming $\theta\in\mathbb{R}^{M}$.

Rather than making explicit how to calculate the entire matrix, is is more fruitful to break down this computation into the separate blocks of this term, such that the $(i,j)$-block of our matrix is computed

$$ $G_{ij}=\left\langle\text{Vec}\left(\frac{\partial\ell}{\partial h_{j}}x_{j-1}^ {\top}\right)^{\top}\text{Vec}\left(\frac{\partial\ell}{\partial h_{i}}x_{i-1} ^{\top}\right)\right\rangle_{x_{0}}\\ =\left\langle\left(x_{j-1}\otimes\frac{\partial\ell}{\partial h_{j}}\right)^{ \top}\left(x_{i-1}\otimes\frac{\partial\ell}{\partial h_{i}}\right)\right \rangle_{x_{0}}\\$ $$

where $\otimes$ represents the Kronecker (Zehfuss) product. The mixed-product property of the Kronecker product allows us to also re-formulate this as

$$ $G_{ij}=\left\langle\left(x_{j-1}^{\top}x_{i-1}\right)\otimes\left(\frac{ \partial\ell}{\partial h_{j}}^{\top}\frac{\partial\ell}{\partial h_{i}}\right) \right\rangle_{x_{0}}=\left\langle x_{j-1}^{\top}x_{i-1}\right\rangle_{x_{0}} \otimes\left\langle\frac{\partial\ell}{\partial h_{j}}^{\top}\frac{\partial \ell}{\partial h_{i}}\right\rangle_{y}.$ $$

Note that we here made a separation of the expectation values based upon the fact that the loss value depends entirely on label or target output, $y$, and the network activities depend entirely on the value of the inputs, $x_{0}$. We assume, as Bernacchia et al. [] do, that these can therefore be separately averaged.

There are now two possible ways to proceed, each requiring an alternative assumption. First, one may suppose that we decide to optimise only a single weight matrix of the network at a time, which would allow us to ignore all but the diagonal blocks of our matrix, $G$.
Bernacchia et al. [] alternatively arrived at a sole consideration of the (inverse) diagonal blocks of $G$ by assuming a linear network (i.e. that $\phi(x)=x$), and showing that in such a case, the matrix $G$ is singular and that it’s pseudo-inverse can be taken using only the diagonal blocks.

Regardless, if we take a single block at a time, and suppose that we are only updating single weight matrices, we can now formulate the natural gradients learning rule as

$$ $\displaystyle{\delta\theta}_{W_{i}}$ $\displaystyle=-{\eta}\langle\nabla_{\theta_{W_{i}}}\ell^{\top}\,\nabla_{\theta _{W_{i}}}\ell\rangle_{x_{0}}^{-1}\nabla_{\theta_{W_{i}}}\mathcal{L}$ (5) $\displaystyle=-{\eta}\left(\left\langle x_{i-1}^{\top}x_{i-1}\right\rangle_{x_ {0}}\otimes\left\langle\frac{\partial\ell}{\partial h_{i}}^{\top}\frac{ \partial\ell}{\partial h_{i}}\right\rangle_{y}\right)^{-1}\nabla_{\theta_{W_{i }}}\mathcal{L}$ (6) $\displaystyle=-{\eta}\left(\left\langle x_{i-1}^{\top}x_{i-1}\right\rangle^{-1 }_{x_{0}}\otimes\left\langle\frac{\partial\ell}{\partial h_{i}}^{\top}\frac{ \partial\ell}{\partial h_{i}}\right\rangle^{-1}_{y}\right)\text{Vec}\left \langle\frac{\partial\ell}{\partial h_{i}}x_{i-1}^{\top}\right\rangle_{x_{0}}$ (7) $\displaystyle=-{\eta}\text{Vec}\left(\left\langle\frac{\partial\ell}{\partial h _{i}}^{\top}\frac{\partial\ell}{\partial h_{i}}\right\rangle^{-1}_{y}\left \langle\frac{\partial\ell}{\partial h_{i}}x_{i-1}^{\top}\right\rangle_{x_{0}} \left\langle x_{i-1}^{\top}x_{i-1}\right\rangle^{-1}_{x_{0}}\right)$ (8) $$

Thus, we arrive at an expression which allows us to better interpret the impact of computing the natural gradient. We can appreciate that the natural gradient formulation thus achieves two things:
- The left matrix multiplication removes any skew and mis-scaling of the loss function with respect to the hidden activities, dealing with the problem that we classically associate with a skewed loss landscape
- The right matrix multiplication removes the impact of any correlation structure in our input data. Correlation structure in our input-data is equivalent to our parameters living in a non-orthonormal basis set and thus affects the speed and accuracy of training!

## Appendix C Our decorrelation mechanism as a recurrent system of dynamics

One may describe our proposed decorrelation mechanism as a system of lateral dynamics where $\frac{d\bar{\bm{x}}}{dt}=\bar{\bm{x}}-{\bm{M}}^{-1}\bm{x}$.
This system can be shown to precisely arrive at the exact solution as outlined in the above text, $\bar{\bm{x}}=\bm{M}\bm{x}$.
Furthermore, if one wished to update the parameters of this dynamic decorrelation setup in stochastic manner (single sample mini-batches), then one may apply the Sherman-Morrison formula to show that $\bm{M}^{-1}$ should be updated with

$$ $\displaystyle\bm{M}^{-1}\leftarrow(g(\bm{I}-\eta_{M}\bar{\bm{x}}\bar{\bm{x}}^{ \top})\bm{M})^{-1}$ $\displaystyle=g^{-1}\bm{M}^{-1}(I+\frac{\eta_{M}\bar{\bm{x}}\bar{\bm{x}}^{\top }}{1-\eta_{M}\bar{\bm{x}}^{\top}\bar{\bm{x}}})$ $\displaystyle\approx{g}^{-1}\bm{M}^{-1}(I+\eta_{M}\bar{\bm{x}}\bar{\bm{x}}^{ \top})$ $\displaystyle\approx{g}^{-1}(\bm{M}^{-1}+\eta_{M}(\bm{M}^{-1}\bar{\bm{x}})\bar {\bm{x}}^{\top}).$ $$

Assuming that the total decorrelating signal to each individual unit can be locally measured ($\bm{M}^{-1}\bar{\bm{x}}$) then this system of dynamics can also be updated in a local fashion by each node.

## Appendix D Training pseudocode

Figure: Algorithm 1 (Approximate) SGD with Decorrelation in a Multi-Layer Neural Network

Algorithm 1 describes pseudocode for the full computation and updating of a network which is carried out in this work.
This is shown assuming the presentation of a single sample with its corresponding network update, though is in-practice used with a mini-batch of size 256 by default.
The algorithm is shown for the case of backpropagation, though it is compatible with alternative credit assignment methods such as feedback alignment or node perturbation.
Furthermore, the pseudocode shows the forward parameters as being updated by SGD, though in practice these are updated with the Adam [Kingma and Ba, ] optimizer.

## Appendix E Architectures and Hyperparameters

**Table 1: The neural network architectures for the experiments in Figure 4. For Convolutional and Pooling layers, shapes are organised as ‘Kernel Width $\times$ Kernel Height $\times$ Output Channels (Stride, Padding)’.**
| Network | Layer Types | Layer size |
| --- | --- | --- |
| Dense Architecture | Input | 32$\times$32$\times$3 |
| FC | 1000 |  |
| FC | 1000 |  |
| FC | 1000 |  |
| FC | 1000 |  |
|  | FC | 10 |
| Convolutional Architecture (CIFAR10/100) | Input | 32$\times$32$\times$3 |
| Conv | 3$\times$3$\times$32, (1, 0) |  |
| Conv | 3$\times$3$\times$32, (1, 0) |  |
| MaxPool | 2$\times$2 (2, 0) |  |
| Conv | 3$\times$3$\times$64, (1, 0) |  |
|  | Conv | 3$\times$3$\times$64, (1, 0) |
|  | MaxPool | 2$\times$2 (2, 0) |
|  | FC | 1000 |
|  | FC | 10 (or 100 for CIFAR100) |
| Convolutional Architecture (Tiny ImageNet) | Input | 56$\times$56$\times$3 |
| Conv | 3$\times$3$\times$32, (1, 0) |  |
| Conv | 3$\times$3$\times$32, (1, 0) |  |
| MaxPool | 2$\times$2 (2, 0) |  |
| Conv | 3$\times$3$\times$64, (1, 0) |  |
|  | Conv | 3$\times$3$\times$64, (1, 0) |
|  | MaxPool | 2$\times$2 (2, 0) |
|  | FC | 1000 |
|  | FC | 200 |

The specific networks trained for demonstration are of two types: a fully connected network architecture and a convolutional network architecture. The structures are show in Table 1. Two datasets are used for training and testing. These include the CIFAR10 and CIFAR100 datasets Krizhevsky [] and the Tiny ImageNet dataset derived from the ILSVRC (ImageNet) dataset Russakovsky et al. [].

The CIFAR10(100) dataset is composed of a total training set of 50,000 samples of images (split into either 10 or 100 classes) and 10,000 test images.
The TinyImageNet dataset is a sub-sampling of the ILSVRC (ImageNet) dataset, composed of 100,000 training images and 10,000 test images of 200 unique classes.
Images from the ILSVRC dataset have been downsampled to 64$\times$64$\times$3 pixels and during our training and testing we further crop these images to 56$\times$56$\times$3 pixels. Cropping is carried out randomly during training and in a center-crop for testing.

**Table 2: The learning rates as selected for simulations shown in Figure 4.**
| Training Algorithm | Hyper Param | CIFAR 10 (Dense) | CIFAR 10 (Conv) | CIFAR 100 (Conv) | Tiny ImageNet (Conv) |
| --- | --- | --- | --- | --- | --- |
| BP | Adam LR | 1e-4 | 1e-3 | 1e-3 | 1e-3 |
| BP + Decorrelation | Adam LR | 1e-4 | 1e-3 | 1e-3 | 1e-3 |
| Decor LR | 1e-5 | 1e-5 | 1e-5 | 1e-5 |  |
| FA | Adam LR | 1e-4 | 1e-4 | 1e-4 | 1e-5 |
| FA + Decorrelation | Adam LR | 1e-4 | 1e-4 | 1e-4 | 1e-5 |
| Decor LR | 1e-6 | 1e-5 | 1e-5 | 1e-5 |  |
| NP | Adam LR | 1e-5 | - | - | - |
| NP + Decorrelation | Adam LR | 1e-4 | - | - | - |
| Decor LR | 1e-6 | - | - | - |  |

The training hyper-parameters were largely fixed across simulations, with a fixed (mini)batch size of 256, and Adam optimiser parameters of $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e-8$. Aside from these parameters, the learning rates for each simulation curve in Figure 4 were individually optimised. A single run of each simulation (curve), with a 10,000 sample validation set extracted from the training data, was run across a range of learning rates for both the forward and decorrelation optimisation independently. Learning rates were tested from the set $[1e-2,1e-3,1e-4,1e-5,1e-6,1e-7]$ for the forward learning rates, with and without decorrelation learning (again with learning rates tested from this set).
Best parameters were selected for each simulation and thereafter the final results plots created based upon the full training and test sets and with five randomly seeded network models for each curve (see min and max performance as the envelopes shown in Figure 4.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - What is the computational cost of the proposed decorrelation method?
-  What are the implications of having a recurrent formulation for computational efficiency and implementation?
- There are second order methods which seem to have gotten some traction such as shampoo [1, 3] and K-FAC [2]. The statement in the first paragraph against 2nd order methods needs more nuance.

[1] Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp. 1842–1850. PMLR, 2018.
[2] Yi Ren and Donald Goldfarb. Tensor normal training for deep learning models. Advances in Neural Information Processing Systems, 34:26040–26052, 2021.
[3] Vyas, N., Morwani, D., Zhao, R., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. (2024). SOAP: Improving and Stabilizing Shampoo using Adam. Preprint at arXiv, https://doi.org/10.48550/arXiv.2409.11321 https://doi.org/10.48550/arXiv.2409.11321.
soundness: 4
strengths: - The focus on the input correlation part of natural gradient descent is novel, as is the proposed decorrelation mechanism.
- The observation that this improves performance for BP alternatives is very significant, since these methods have the potential to be more efficient for training
- The narrative exposition was easy to follow, and the concepts are explained very clearly
- The (almost) empirical evaluation supports the conclusions of the paper
- connections to biology is interesting
summary: Starting from natural gradient descent, the authors show that correlations in data can cause non-orthonormal relationship between the model's parameters. To mitigate this, they propose a decorrelation mechanism that uses only local information. They demonstrate that this mechanism improves BP accuracy, but also notably improves results for alternate training methods such as feedback alignment, and node perturbation.
weaknesses: - The discussion of methods from computational neuroscience is very cursory, and would benefit from more details such as forms of specific learning rules
- The recurrence aspect of the decorrelation rule could be discussed in more depth in the main text.
- It's not clear if the proposed decorrelation method only has a recurrent formulation, or can be efficiently implemented without recurrence as well.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: To better align this work with the standards and expectations of an ICLR paper, the authors need to sharpen their message and clearly articulate the novel contributions of each section. Specifically, the authors may consider addressing the following questions:
1. How does the consideration of Natural Gradient Descent and the dynamics in curved parameter space contribute to the understanding or derivation of the incremental learning rule? Can this rule be directly derived from the principles of the Natural Gradient theorem?
2. What differentiates the proposed learning rule from existing frameworks for decorrelating neural activity, such as Recursive Least Squares? If there are differences, what are the specific theoretical or practical advantages of the proposed rule?
3. Could the authors provide more extensive numerical analyses to demonstrate whether the benefits of their approach extend beyond faster learning to improved generalization? While Natural Gradient theory primarily explains accelerated convergence, do the authors observe any substantial gains in generalization performance?

Lastly, I want to note that I enjoyed reading the paper and its broad review, which connects the concept of natural gradients to deep network training. However, this discussion currently feels more suited to an opinion piece.
soundness: 3
strengths: • The initial sections of the paper provide an insightful overview of Natural Gradient Descent and offer a compelling perspective on how Amari’s theoretical framework can be contextualized within deep learning. While typical applications of Natural Gradients focus on Bayesian inference, where the Fisher information naturally defines the metric, the authors present an innovative interpretation that broadens its relevance.

• The simulations included in the study convincingly demonstrate that implementing decorrelation at each layer significantly enhances training efficiency, particularly within the Feedback Alignment framework.

• The paper offers a comprehensive overview of the impact of data correlations on training dynamics, which serves to enrich the reader’s understanding of this often underexplored aspect of deep learning optimization.
summary: In this work, the authors address the challenge of data correlations impacting neural activity and the learning process within neural networks. They take a comprehensive approach, beginning with an analysis that connects data correlations to the non-Euclidean geometry of the parameter space. Specifically, they demonstrate that correlations in input data lead to longer paths through parameter space during learning. This observation is rooted in the theory of Natural Gradient Descent, as originally developed by Amari in the late 90's, where parameter space is considered curved and defined locally by the inverse Fisher information matrix. The authors provide evidence that input correlations influence these longer paths.
Next, they propose an incremental learning rule capable of decorrelating inputs within each neural network layer. This rule facilitates a linear transformation to reduce input correlations progressively. The authors demonstrate that employing such a decorrelation mechanism accelerates learning and potentially improves generalization across several learning paradigms, including standard backpropagation, Node Perturbation, and Feedback Alignment. The concluding part of the paper discusses the implications of removing correlations in biological and artificial settings.
weaknesses: The primary issue with this paper is its limited coherence and lack of substantial innovation. The manuscript presents small incremental contributions across several topics and attempts to integrate them into a unified framework. This approach results in a paper that straddles the line between a review and an opinion piece, which does not align well with the expectations for an ICLR submission. Below, I outline the specific areas where the paper falls short in terms of novelty:
1. **Review of Natural Gradient Descent**: The first section provides a summary of Natural Gradient Descent, highlighting that input correlations can be inferred from the derivation of the metric tensor. However, this observation is neither surprising nor particularly useful, as it merely reiterates that input correlations are intrinsic to Natural Gradient theory. Moreover, the concept of learning dynamics in curved parameter spaces is not applied or built upon in subsequent sections, limiting the relevance of this part of the paper to the overall narrative.

2. **Derivation of a Local Learning Rule**: The second section presents a derivation for a local learning rule intended to decorrelate inputs to neural network layers. While the authors point out the novelty of their incremental rule (line 340), this rule closely resembles existing methods such as recursive least squares (e.g., Sussillo and Abbott, 2009), which also stem from the goal of decorrelating neural activity. The resemblance is even clearer when studying the derivation of the proposed decorrelation learning rule in Appendix C. Notably, the proposed rule introduces a novel scalar rescaling that maintains the norm. However, this addition is minor, and the overall innovation is minimal.

3. **Numerical Experiments on Layer-wise Decorrelation**: The third section includes numerical results showing the advantages of input decorrelation for training deep networks. Prior work has already demonstrated that input decorrelation accelerates learning in frameworks such as backpropagation and Node Perturbation. Consequently, the main novelty here is the improved performance observed in the Feedback Alignment framework. However, the paper does not provide sufficient explanation or analysis to elucidate why decorrelation significantly enhances Feedback Alignment, nor does the performance match standard backpropagation. This omission limits the impact of this result, rendering it another incremental contribution.

Collectively, these points suggest that the novelty presented in this work does not meet the standards expected for ICLR.

Finally, the discussion section of the paper, while extensive, lacks coherence and leans heavily on speculation. For instance, the portion discussing biological plausibility is unclear in its message. Although the authors correctly note that decorrelation is observed in various neural circuits, the connection between these observations and the mechanisms or theories presented in the paper remains ambiguous. Additionally, the claim that decorrelation improves generalization (line 479) is inadequately supported by the data. The training curves in Figure 4 do not appear to reach convergence, as indicated by the non-saturating test accuracy. This raises doubts about the robustness of the claimed generalization improvements.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: - typo optimization 153
- Assumption on 182 that NGD independent layers, how serious is this?
- 163, should the expectations on G not be over y too?  
- 251 skewed is being used non-technically? 
- 260 type orthogonal 
- 264 citation missing, gradient correlations
- What is the connection between natural gradients and the approximate gradient methods. Why in particular should the approximate methods benefit from natural gradients?
- The discussion of the orthonormal basis of parameters (page 5) induced by data correlations does not make sense to me. Are the parameters already not orthornormal?
soundness: 3
strengths: I believe this is an original synthesis of concepts resulting in an original and significant modification to approximate gradient methods. 
This paper is extremely well written, and generally of high clarity and quality throughout.
summary: This submission didactically draws together approximate gradient methods, natural gradient descent, and decorrelation methods, to ultimately improve approximate gradient methods. More specifically the paper shows how natural gradients defined with respect to changes in the loss function highlight the importance of data correlation in determining the alignment between GD and natural gradient updates. With this insight in hand, approximate gradient descent methods are shown to be improved by decorrelating the network activations.
weaknesses: - It is not clear that the alignment to natural gradients is what is underlying the improved performance. 
- No assessment of how successful the decorrelation updates are in aligning regular updates to the natural updates. How big is the contribution of gradient correlations? 
- While the improvements in FA and NP are striking, the experimental results are not finely tuned. It is possible with GD to obtain better results than those shown, and it is unknown if FA and NP will match this improvement. 
- To verify decorrelated FA scales, harder experiments would strengthen the paper (e.g those in Bartunov 2018. Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures for ANNs.)
- Given biological relevance, showing application and relevance to recurrent networks would improve the paper.

Meta-review: This paper investigates how correlations in data and internal activations affect the geometry of parameter updates in neural networks, extending the idea of natural gradients and linking it to the need for decorrelating transformations at every layer. While the narrative is well-structured, the work falls short on both novelty and empirical validation. Overall, the paper is a well-written synthesis rather than a substantive advancement in the field, and lacks the strong justification, new methods, or comprehensive empirical evidence needed for acceptance.

Given the variance in the reviews, the AC read the paper in entirety before making a decision.

Decision: Reject