Constrained Parameter Inference as a Principle for Learning
Authors: Nasir Ahmad
Ellen Schrader
Marcel van Gerven
Venue: Accepted by TMLR ()

Abstract: Learning in neural networks is often framed as a problem in which targeted error signals are directly propagated to parameters and used to produce updates that induce more optimal network behaviour. Backpropagation of error (BP) is an example of such an approach and has proven to be a highly successful application of stochastic gradient descent to deep neural networks. We propose constrained parameter inference (COPI) as a new principle for learning. The COPI approach assumes that learning can be set up in a manner where parameters infer their own values based upon observations of their local neuron activities. We find that this estimation of network parameters is possible under the constraints of decorrelated neural inputs and top-down perturbations of neural states for credit assignment. We show that the decorrelation required for COPI allows learning at extremely high learning rates, competitive with that of adaptive optimizers, as used by BP. We further demonstrate that COPI affords a new approach to feature analysis and network compression. Finally, we argue that COPI may shed new light on learning in biological networks given the evidence for decorrelation in the brain.

Ellen Schrader Published in Transactions on Machine Learning Research (01/2023)
Constrained Parameter Inference as Principle for Learning
Nasir Ahmad
n.ahmad@ddonderr..vu.
Department of Artificial Intelligence, Donders Institute, Redboud University
eascruuerr&donders..w..l
Department of Artificial Intelligence, Donders Institute, Radboud University
Marcel van Gerven
m.vangerreem@ddmdeess....m
Department of Artificial Intelligence, Donders Institute, Radboud University
Reviewed OpenReview: https: oppnrrvvepeeneveeuuu net/ formm? id= CUDAbTT1QC
Abstract
Learning in neural networks often framed as a problem in which targeted error signals are directly propagated parameters and used to produce updates that induce more optimal network behaviour. Backpropagation of error (BP) is example of such approach and has proven to be highly successful application of stochastic gradient descent to deep neural nelworks. We propase constrained parameter inference (COPI) as new principle for learning. The COPI approach assumes that learning can be set up in manner where parameters infer their own values based upon observations of their local ncuron activitics. We find that this estimation of network parameters possible under the constraints of decorrelated neural inputs and top-down perturbations of neural states for credit assignment. We show that the decorrelation required for COPI allows learning extremely high learning rates, competitive with that of adaptive optimizers, used by BP. We further demonstrate that COPT affords new approach to feature analysis and network compression. Finally, we argue that COPI may shed new light on learning in biological networks given the evidence for docorrclation in the brain.
1 Introduction
Learning can be defined as the ability of natural and artificial systems adapt changing circunstances based on their experience. In biological and artificial neural networks this requires updating of the parameters that govern the network dynamics (Richards et al. 2019h.
A principled way of implementing learning in artificial neural networks is through the backpropagation of error (BP) algorithm (Linnainmaal 1970 Werbos 1974). BP is a gradient-based method which uses reverse-mode automatic differentiation to compute the gradients that are needed for individual parameter updating [Baydin et al.) 2018). This approach relics on the repeated application of forward and backward passes through the network. In the forward (inference) pass, network activity is propagated forward to compute network outputs. In the backward (learning) pass, the loss gradient associated with the network outputs is propagated in the reverse direction for parameter updating.
While effective, BP makes use of the Iransmission of gradients using biologically implausible non-local opera- tions, and multiple separated network passes (Grossberg) 1987 Crick 1989 Lillirap et al. 2020). Alternative approaches, such as Hebbian lcarning and subspace methods circumvent this problem yet are restricted unsupervised learning and do not afford (deep) credit assignment (Brea & Gerstner 2016 Pehlevan et al. 2015

---

Published in Transactions on Machine Learning Research (01/2023)
Here, propose constrained parameter inference (COPI) as new principle for learning. COPI uses information that can be made locally available at the level of individual parameters whose values are being inferred under certain constraints. Note that COPI is distinct from methods that rely on measuring gradients through activity differences (see the NGRAD hypothesis (Lillierap et al., 2020)), in that no difference in activity needs to be computed determine parameter updates. Specifically, by constructing a mixed network activity state in the BP case a simple summation of the forward and backward passes for output units parameters can infer their own optimal values by observation of node activities alone. This is distinct to many proposed biologically plausible methods which require parameters to measure differences in some or across time between two phases (Bengio. 2014 Scellier & Bengio 2017: Ernoult et al. 2020) W Whittington & Bogacz 2017] Sacramento et al. 2018 Payeur et. al. 2021). Thus, COPI provides a framework which might in future enable online continuous
activity, either physically using separate compartments/signet
learning where parameter updates are based upon single state measurements.
Furthermore, the COPI algorithm is not tied to any particular credit assignment method. In this sense it assumes that eredit can be assigned to units (by user's method of choice) and simply describes how parameters should update their values given network state observation. Credit assignment is integrated into COPI by top-down perturbations. The form of the perturbation is preciscly what determincs which credit-assigument algorithm is being used for learning within the system, whether based on backpropagation, feedback alignment (Lillicrap et al. 2016 Nøkland 2016), target propagation (Bengio 2014 Ahmad et al. 2020) or otherwise. Thus, COP1 does not address credit assignment as such but rather proposes a general approach for learning based upon a single mixed-state regime.
In the following, demonstrate that COPT provides powerful framework for learning which is at least effective backpropagation of error while having the potential to rely local operations only. Moreover, as will be shown, COPI allows for efficient linear approximations that facilitate feature visualisation and network compression. Hence, it also provides benefits in terms of interpretable and efficient deep learning. This has direct implications for modern machine lcarning as COPI can be used as a replacement for the parameter-updatinn step in backpropagation applications across a wide range of settings.
2 Methods
In this section, we develop the constrained parameter inference (COPI) approach and describe its use for parameter estimation in feedforward neural networks.
2.1 Deep neural networks
Let us consider deep neural networks consisting of L layers. The input-output transformation in layer / is given by
1/1 = f(a) = f(Wa))
with output Vi, activation function f. activation 0; Wia, input ot and weight matrix Wi E KiXix where K₁ indicates the number of units in layer 1. As usual, the input a layer / >>> is given by the output of the previous layer, that is, 21 yt-1. Learning in deep neural networks amounts to determining for each layer in the network a weight update Aw, such that the update rule
Wi Wt + 7/w,
converges towards those weights that minimize a loss é for some dataset. D given suitable learning rate > 0. Locally, the optimum by gradient descent (GD) is to take a step in the direction of the negative expected gradient of the loss. That is,
Rd Wi
E Vw.
where, in practice, the expectation is taken under the empirical distribution.
2.2 Constrained parameter inference in feedforward systems
Here, we develop an alternative approach and relate directly to both stochastic gradient descent and local parameter inference. Note that the key transformation in deep feedforward neural network is carried out
2

---

Published in Transactions on Machine Learning Research (01/2023)
by weight matrix given by
ai = Wa
Suppose we know the target activation for this transformation. This can be expressed as an alternative transformation
2 Wi
for some desired weight matrix Wi- Ideally, we would like to use a learning algorithm which guarantees convergence of the current weight matrix to the desired weight matrix. A straightforward proposal is to carry docay from the current weight values the desired weight values, such that the weight update is of the form
Aw; = E [ - Wi] = Wi - Wi.
(1)
Of course, the key goal is to achieve this weight update without making use of the (unknown) desired weights. How to achieve this, is described in the following sections.
2.3 Learning the forward weights
Let us rewrite the desired weight matrix in the following way;
Wi = Wi E [
EE [ 1
= E [Wi ma
E [maar]]]
= E [ E [xyar]]]
with E [ara]]] the (sample) autocorrelation matrix. We here assume this matrix to be invertible, though this condition is later shown to unnecessary. If we plug this back into Eq. (1 then we obtain
A w: E [| E|zz, Wi
(2)
allowing the weight update to be expressed in terms target outputs, 21, rather than (unknown) desired weights. This is an expression the least-squares optimization algorithm.
Let assume for the moment that the inputs Iı are distributed such that they have zero covariance and unit variance, i.e, the inputs are whitened. This implies that the autocorrelation matrix is given by the identity matrix, that is, E [nme]]] = T. In this case, Eq.
2 reduces to the simple update rule
Aw; = E [ W.
In practice, may be unrcasonable (and perhaps even undesirable) assume perfoctly whitenod input data. A more realistic and achievable scenario one in which make the less restrictive assumption that the data is decorrelated rather than whitened. This implies that the autocorrelation matrix is diagonal, and that E [,,;]] diag (E [=?]] with x7 the vector of squared elements of Right-multiplinng both sides of Eq. by this expression, and assuming that E [ is indeed diagonal, obtain
Aw, diag E
[xf]
[2 Wi diag E [x:])
This matrix multiplication amounts to a rescaling of the columns of Aw,, and thereby a relative scaling of their learning rates. This finally leads to our constrained parameter inference (COPI) learning rule
copi W
E [ W, diag (E [=f])
(3)
which solely composed of Hebbian correlational learning term and weight decay term. COPI receives its name from the fact that there are two constraints play. First, the availability of a target or 'desired' state 2 for each layer and, second, the requirement that the inputs xą decorrelated.
3

---

Published in Transactions on Machine Learning Research (01/2023)
2.4 Input decorrelation
We did not yet address how to ensure that the inputs lo each layer are decorrelated. To this end, we introduce new decorrelation method which transforms the potentially correlation-rich outputs VI-1 of layer into dccorrelated inputs 2j to the following layer using the transformation
It RMn-1:
where R is a decorrelating 'lateral' weight matrix.
We set out to reduce the correlation in the output data x by measurement of its correlation and inducing shift toward lower correlation. In particular, consider desired change in given sample of the form
x + 21 77 (E [ددد]] diag (E [=+])) 21.
where the expectations could be taken over the empirical distribution. We can shift this decorrelating transform from the output activities to the decorrelating matrix Rg. To do so, consider substituting 21 = R such that we may write
17 (E [2|د]] diag (E [=+])) 21
Riynt11 17 (E. [maत]] diag (E [afl)) Riy--
+ [R ( |لمم|| diag (E [a+] () Ri)m
We converge to the optimal decorrelating matrix using an update Rí < Rí + m Rg
copi
where
A copi
= E [ ] diag E
2 R
(E [x diag EE
x Ri)
(4)
with = Rag. Note that this decorrelating transform can also be derived rigorously as a gradient descent method, see Appendix
The update can be made more local and therefore more biologically plausible by an approximation, which we explore here. To make the information locally available for the update of the decorrelation, we assume that this correlation reduction be carried out in either direction with the target of correlation reduction and source being exchangeable. This assumption allows us to arrive at more biologically plausible learning rule given by
hio-copi = [E [ - R2 diag (E [=f)) Ri
(5)
with Qi RØa. Equation 5 has exactly the same form
previous COPI decorrelation rule (3) for learning the forward weights though now acting to update its weights in the same manner as the COPI forward learning rule using target states qı These target states are now the total amount of decorrelating signal being provided to a given unit. In effect, the lateral weights are also constantly inferring corrolational structure within the activities of a layer of units but, given the negatively-signed update, they update their values to reduce correlation instead.
This rule more biologically plausible than the theoretically derived decorrelation rule since an update of lateral weight ray connecting unit j to unit i relies qezt rather than æ,9₁- That is, update local to unit only requires access to its own target state rather than the target state of other units. Furthermore, the weight decay term is now scaled based upon the pre-synaptic activity, which is propagated via the synaptic connection, rather than the post-synaptic activity. As this information is available to the post-synaptic unit and synaptic connection, respectively, it can be used for parameter updating. Both the theoretically derived rule and more biologically plausible formulation consistently reduce corrclation in the output states at. as shown in Appendix A
2.5 Error signals
Equation expresses learning of forward weights in terms of target states 2l. However, without access targets for each layer of deep neural network model, one may wonder how this learning approach could be
4

---

Published in Transactions on Machine Learning Research (01/2023)
applied in the first place. To this end, we assume that the target states can be expressed
a. + aδi
with dj an error signal which perturbs the neuron's 's state in a desired direction and a a gain term controlling the strength of the perturbation. Note that if we can directly access the 61, then the optimal weight change to push fulure responses in the desired direction ój is simply 6Gæ; However, here we assume that the weight- modification process cannot directly access 6,, but only 'sees' the perturbed activities = + adj. In this case, as explained above, COPI is necessary to produce correct weight changes and push future responses towards these target activities.
Different error signals can be used to induce effective perturbations. According stochastic gradient descent (SGD), the optimal perturbation is given by
dl daj
as this guarantees that the neuronal states are driven the direction of locally-decreasingg loss. The error D. signal at the output layer is given by 8% = dar diag(f"(at)) øyr." Starting from 07.4 the layer-specific perturbation can be computed via backpropagation by propagating the error signal from output to input da, = (())(("
according to ا7ne 8ar41 gagd with Bes+1
da,
While gradient-basee error signals provide a gold standard for the optimal perturbation, we may want to replace this by more biologically-plausbble credit assignment methods. These methods typically use the same error signal εy å 8% for the output layer but propagate the error signal in the input direction using different proposal mechanisms. As an illustration of such an alternative error signal, we consider feedback alignment (FA) (Lillicrap al.) 2016). which supposes that the perturbation from the previous layer can be propagated through fixed random top-down weights B1+1 instead of the transposed weights (W1+1 R way to address the so-called weight transport problem Grossberg 1987). J. Hence, in FA, the layer-wise perturbations are propagated by
ofth = (a))
In our analyses, we will restrict ourselves to comparing error signals provided by backpropagation and feedback alignment only. Note, however, that other credit assignment methods such as direct feodback alignment (Nokland] 2016) target propagation and its variations (Bengio 2014 Dalm al. 2023) can be
seamlessly integrated in our setup if desired.
2.6 Stochastic COPI
Stochastic COP1 replaces the expectations over the empirical distributions in Eqs. (3) and by single data points, analogous to stochastic gradient descent (SGD). COPI training single data points proceeds by computing the stochastic weight updates. For all COPT implementations, the forward weight updates are given by App = _ - W diag (xf) with target states ž a4 ad, given some suitable error signal &, The decorrelating lateral weight updates are given by A - (nnد) diag (1+) Ri) with qi Raa-- In practice, as usual, we train on minibatches instead of individual data points. See Algorithm for a pseudo-algorithn which uses gradient-based error signals and quadratic loss.
For comparison against SGD, it is instructive to consider (stochastic) COPT applied to single-layer neural network. Recall that the SGD update of single-layer network is given by A x We can manipulate this expression in order to relate SGD to COPI as follows:
dl
sgd W
T.'
da
dl
x ax
da
(a + jsgd)
xT - W (xx))
(6)
5

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm Constrained Parameter Inference
procedure CÖPT(network; data)
network consists of randomly initialized forward weight matrices Wi and lateral weight matrices Ri with SISL
dala consists of N inpul-output pains y")
Parameters: learning rates via and 1ws gain term o; number epochs; batch size
2:
for each epoch do for each batch = {(v, y²) ))) c data do for layer I from to T. do
3:
4:
Forward pass Decorrelate the imput data Compute activation
5:
21 Ray-1
6:
01 Wizi
7:
91 = (ax)
> Compute output
end for
9:
8 = lry.. v²112
> Compute loss Backward pass
for layer from to do
11:
81
de if = L else ó: daz.
0ar+1 8141 Dai
Compute learning signal
12:
end for
for layer from L to do
Update parameters
14:
Wi + Wi + qw ((a + add))a - Wi diag (af))
b. Update forward weights
15:
R t- R na (In (Rez)) diag (x+) Rt)
b Update lateral weights
16:
end for
17: end for
18: end for
19:
return network
20:
end procedure
This update looks similar to the (stochastic) COPI update Apri = (a + a W Wdiag (z²). The koy diffcrence, however, is that, in contrast to SGD, COPI ensures that the inputs are decorrelated, as realized by the COPI update Ap Therefore, the weight decay lerm is unaffected by sample-wise input correlatros. The weight decay term for COPI is Hebbian in nature since the update of wiy relies on [Wdiag (x )is Wig which is local to the synapse. In contrast, SGD is non-Hebbian in nature since it relies on [W (xx" ())s En Wa which is not local to the synapse.
2.7 Linear approximation of non-linear transformations
An additional benefit of the decorrelating properties of COPI is that it allows for the efficient approximation of one more non-linear transformations by linear matrix. As will be shown later, this has applications in interpretable and efficient deep learning.
Let us consider a neural nelwork as a non-linear transformation y = (x). We are interested in computing its linear approximation, given by y = Br. Suppose we have access to empirical data = [M)) and Y ()(,).... (N)] such that (n)) = f (x(")) is the n-th input-output pair. The standard ordinary least squares solution for B is given by
B = YX (XXT)-1
However, this requires the computation of a matrix inverse, which can be prohibitive for large matrices. Networks trained using COPI can instead make use of the property that inputs decorrelated by con- struction. This implies that (XXT)= = diag (I/z,æI, .... 1/aM*T)) & with am the mth row X. This allows us to compute transformation from such decorrelated inputs
B B=YX-c.
Hence, in networks with decorrelated layer-wise inputs or activities, such as produced by COPI, lincar approximations of transformations can be efficiently computed without computing inverses. We will use this property in Section 3.2 for both feature visualization and network compression.

---

Published in Transactions on Machine Learning Research (01/2023)
Note that this is ultimately the mechanism by which COPI also operates, though in sample/batch- manner with a changing output distribution ion due to fluctuating learning signals. Specifically, consider the
COPI algorithm at its fixed point, such that Aw W: = E | Wi diag (IE [ = 0. Under this condition, we can re-arrange to obtain
copt
Wi = = [at]]
diag (E [=,])
which is equivalent to our above formulation though under the assumption of fixed desired output 2t.
3
Results
In the following, we analyse both the convergence properties of COPI and the benefits of the decorrelated reprosentations at every network layer.
3.1
COPI performance standard benchmarks
A) MNIST
B) CIFAR-10
1.00
0.9
test
10
0.99
0.8
Accuracy
(decort)
500
0.98
0.7
copi (bp)
copi (f))
500
0.97
500
0.96
1000
1000
10-1
500
500
10-2
10
Loss
10-3
Los
10
10-2
10-5
10
40
100
0
20
60
80
100
Epochs
Epochs
Figure 1: COPI BP performance standard computer vision classification tasks. A) Train/test accuracy and loss of seven-layer (sce graphical depiction), fully-connected,, feedforward deep neural network model trained and tested the handwritten MNIST dataset. B) Train/test accuracy of five-layer, fully-connected, feedforward deep neural network model trained and tested on the CIFAR-10 dataset. All networks were run with five random seeds and envelopes show standard deviation across these networks.
To validate COPI as a principle for learning, we compared it against backpropagation by training fully- connected deep feedforward neural networks trained the MNIST handwritten digit dataset LeCun et al. 2010) and the CIFAR-10 image dataset (Krizhevsky & Hinton 2009).
COPI simulated together with both gradient-based (g75d) and feedback alignment- based (8(h) perturba- tions with loss function composed of the quadratic loss between network outputs and the one-hot encoded labels as 'desired' output. To clarify the benefits of the COPI decorrelation process, we additionally trained networks in which the forward weights W, were updated by BP using vanilla SGD and lateral weights Rd were introduced and trained by the COPI decorrelating algorithm, labelled 'BP (decorr): Furthermore, we ob- tained basclines with backpropagation alone (no decorrclation) combined with the Adam optimizer (Kingma & Ba 2014 Figure shows the results of these simulations using learning parameters as described
Appendix B

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 1 shows that gradient-based COPI learning is extremely effective. During training, COPI achieves higher accuracy and lower loss than even an adaptive optimization approach (Adam) on the more challenging CIFAR-10 dataset. When BP is combined with decorrelation, training loss remains consistently lower for COPI across both datasets. We can only attribute this benefit to the explicit difference in the forward COPT and BP rules, where COPT relies on the built-in assumption that the inpuls have a decorrelated form. During testing, we observe that COPI slightly outperforms BP (adam) in terms of accuracy on CIFAR-10 and performs consistently better than BP with decorrelation on both datasets. The performance of the more biologically plausible BIO-COPI variant is close to identical to that of regular COPI. COPI learning using feedback alignment was also feasible albeit less effective, consistent with literature (Nokland 2016). This demonstrates that different error signals can easily be plugged into the COPI framework when desired. Note further that results generalize to different nerk depths, as shown in Appendix C as well as to other loss functions, as shown in Appendix D for the cross-entropy loss.
Table 1: Peak performance (accuracy) measures of COPI vs BP for the results presented in Figure Also provided in brackets is the mean epoch at which the networks reached 99% peak performance. Method
Peak Performance + Standard Dev. (Mean # Epochs to 99% of Peak)
MNIST
CIFAR-10
train
test
train
test
bp (adam)
1.0 (6)
0.9838 + 0.0004 (5)
0.9998 + 0.0001 (53)
0.5619 + 0.0023 (36)
bp (decorr)
1.0 (8)
0.9812 + 0.0009 (3)
1.0 0.0 (8)
0.5616 + 0.0047 (8)
copi (bp)
1.0
0.9834 + 0.0007 (3)
+.0
0.5729 + 0.0016 (10)
cupi (la)
0.9740 1 0.0010 (4)
1.0 1 (13)
0.5207 1 0.0022 (6)
bio-copi (bp)
0.0
0.9835 1 0.0009 (3)
1.0 (8)
0.5730 + 0.0040 (9)
Arguably, the largest gain is obtained in terms of convergence speed when using COPI as a learning mech- anism. In general, we find that (BIO-)COPI and decorrelating BP (which uses the COPI mechanism for decorrelation) are able to learn much faster than conventional BP with adaptive optimizer (Adam). As can be scen in Table models employing docorrelation reach close to peak performance (within 99% of peak performance) much more rapidly. This is not due to the used learning rate since performance drops at higher learning rates when using Adam.
3.2 Decorrelation for feature analysis and network compression
The COPI algorithm's requirement for decorrelation at every network layer is not only a restriction but also proves beneficial in a number of ways. We explore the decorrelation, as well as the analyses and computations that it enables.
The proposed decorrelation method produces a representation similar to existing whitening methods, such as ZCA (Bell & Sejnowski 1997). Figure provides a visualisation of a randomly selected set of data samples from the MNIST dataset. From top to bottom are shown: the unprocessed samples, samples processed by the first decorrelating layer of a network trained on MNIST with the COPT algorithm (see MNIST networks described in Figure [ and finally a visualisation of samples transformed by a ZCA transform computed on the whole training set. As can be seen, there is qualitative similarity between the COPI- and ZCA-processed data samples. Remaining differences are attributed to the fact that COPI does not scale the individual elements of these samples (pixels) for unit variance, i.c. whitening, but instead produces decorrelation alone in distributed fashion.
Beyond the input transformation, COPI allows for visualisation of features deeper in the network by ex- ploiting decorrelated layer-wise inputs. That is, we may use the decorrelated input training datasct X and corresponding unit activations A to form a linear approximation A = B,X of the receptive field of units deep in the network using the procedure described in Section 2.7 Here, the i-th row of B provides an average, linear, feature response for unit in layer / of the network.
Figure CC shows such extracted features from a random selection of 100 units from the second, fourth and sixth layer and 10 units from the seventh layer of a network trained using COPT on MNIST. These results
8

---

Published in Transactions on Machine Learning Research (01/2023)
A) Input Data Processing
Network Compression
MNIST Trained Networks
CIFAR-10 Trained Networks
1.00
1.0
9
tran
MNIST
0.9
test
Perforance
0.95
0.8
COP/
0.90
0.85
ZCA
0.4
6
3
2
(Compressed)
[Comoressed) Network Depth
Laser-Wise Feature Analysis
7
F
3
7
P
Z
Z
C D
T
R
<
R
R
6
1 = 2
4
6
7
Figure 2: The effect and utility of decorrelation within COPI network layers for feature readout and network compression. A) Visualisation of the MNIST dataset (top), the decorrelating transformation produced by the COPI algorithm (middle), and, the whiteming transformation produced by ZCA whitening (bottom). B) The decorrelated layer-wise inputs of COPI-trained networks could also be used to efficiently infer linear mappings between distant layers of the network. This allows the removal of network layers and replacement with an inferred, linear approximation of those intermediate layers. Plotted are the performances of the layer MNIST trained COPT networks (left) and five-layer CIFAR-10 trained COPT Networks (right) from Figure [ This network compression process is repeated for five, randomly seeded networks and error bars show standard deviation across these repeats. Layers are removed from the output layer backwards. Here, the left-most bars (seven/fivy layers for unmodified networks, which are provided for comparison. C) Using the decorrelated network inputs of the COPI networks (see middle row in A), the decorrelated inputs, 2, from the entire training data could be (pixel-wise) correlated with the network's layer-wise outputs, (node-wise). This produced linear approximation of the units' preferred features from different network layers.
correspond to the initial
are from a single network used to produce the results shown in Figure 2A. This allows us to observe an increasingly digit-oriented feature preference for units as we go deeper into the network, as well as the presence of digit/ digit/anti-die autti-iiit-selective units.
This same mochanism of producing lincar approximation receptive ficlds also provides computationally efficient method to approxinate the transformation produced by multiple layers of COPI-trained network with linear matrix. That is, may employ the COPI principle to infer a linear matrix Bi approximating the transformation across multiple network layers such that A, ~ B where X is the decorrelated output of some layer A < l. This approach allowed us to rapidly convert MNIST and CIFAR-10 trained networks into networks consisting of any smaller number of layers, effectively providing a straightforward approach for network compression.
Figure 2 shows the performance impact of such conversions for the seven-layer network trained on MNIST depicted in Figure TA (MNIST trained) and the five-layer network trained CIFAR-10 shown in Figure 13.

---

Published in Transactions on Machine Learning Research (01/2023)
Note that this approximation is done in single step using the network's response to the training data and does not require any retraining. Network performance is shown to stay relatively high despite the approximation and removal of layers. In fact, for CIFAR-10, we even see that this approximation returns some small gain in test-set performance. Note that layers are removed sequentially from the end of the network and, as can be seen, there is a significant drop in performance when the all layer of each network are approximated, indicating that the transformation in the first layer is crucial for achieving high performance levels. This is consistent with the change in performance when retraining networks consisting of a smaller number of layers, as shown in Appendix C
4 Discussion
In this paper, we introduced constrained parameter inference as a new approach to learning in feedforward neural networks. We derived an effective local learning rule and showed that, under the right conditions, individual weights can infer their own values. The locality required the removal of confounding influences between unit activities within every layer of the neural network, and to this end, we derived an efficient decorrelation rulc. We further assumed crror signals were available to perturb unit activations towards more desirable states from which the system could learn.
The resulting algorithm allowed us to effectively train deep feedforward neural networks, where performance is competitive with that of backpropagation for both gradient-based and feedback alignment signals. Further- more, our setup enables much higher effective learning rates than are possible than with vanilla BP and thus allows us to learn at speeds exceeding those possible even using adaptive optimizers. This may contribute to reducing carbon footprint when training large network models (Strubell et al.) 2019). The algorithm also allows for more interpretable deep learning via the visualisation of deep decorrelated features (Rudin) 2019 Ras et al. 2022 and could contribute to efficient deep learning it facilitates network compression (Wang, 2021). Going forward, it is important to expand the tasks to which COPI is applied and investigate its application to a broader class of network architectures. For example, COPI is in principle compatible with other net components such as convolutional layers, but requires careful consideration as for how to carry out decorrelation in an optimal manner.
From theoretical standpoint, COPI relates unsupervised methods for subspace learning Oja 1982;
Földiák & Young 1998 Pehlevan et al. 2015). In particular, the form of the learning rule we propose bears resemblance to Oja's rule (Oja 1982), though it focuses on inference of paramcters in the face of perturbations instead of latent factor extraction. See Appendix E for comparison.
Aside from unsupervised methods, the inference of parameters based upon input and output activities has been previously proposed to overcome the woight-transport problem (Akrout et al. 2019 Ahmad et al. 2021 Guerguiev et al. 2019). In particular, these methods attempt to learn the leedback connectivity required for backpropagation via random stimulation of units and a process of weight inference. Our method similarly attempts to carry out weight inference, but does so without random stimulation and with the purpose of learning of the forward model through combination with top-down perturbations.
It is also interesting to note that our decorrelating mechanism captures some of the key elements of batch normalization (loffe & Szegedy 2015 Huang et al.] 2018). First, vanilla batch-normmllizttoon makes use of demeaning, a natural outcome of our decorrelation. Furthermore, whitening of batches has been recently shown to be an extremely effective batch-wise processing stage, yielding state-of-the-art performance on a number of challenging classification tasks (Huang et al. 2018), and reduction of covariance between hidden unit activities has been found to be a generalisation encouraging regularizer (Cogswell et al., 2015). However, and is instead fixed component of the
unlike all of these methods, method is not network architecture, learned over the course of the whole datasct and integrated as a network component.
COPI may also shed light on learning and information processing in biological systems. There is both experimental and theoretical evidence that input decorrelation is a feature of neural processing through a number of mechanisms including inhibition, tuning curvos, attention, and eye movements (Franke et al.,
2017
Bell & Sejnowski 1997 Pitkow & Meister
2012 Segal et al., 2015 Vogels et al. 2011 Abbasi-As)
et al. 2016 Cohen& & Maunsell 2009 Dodds et al. 2019 Graham et al. 2006 In particular, center-surround
10

---

Published in Transactions on Machine Learning Research (01/2023)
filters of the LGN appear to produce a form of whitening. Whitening also appears to be key for sparse coding of visual inputs (King et al. 2013). To what extent there is decorrelation between all units projecting to a neuron is of course questionable, though COPI has the potential for modification to account for global or local correlations. The more biologically plausible decorrelation rule described in Section 2.4 suggests how the decorrelation rule here might operate in a local fashion.
Beyond this, inhibitory and excitatory balance (Denève & Machens 2016) has been formulated in a fash- ion which can be viewed as encouraging decorrelation. Learning rules which capture /inbitrrri balance, such as the one by Vogels et al. (201I), rely on correlative inhibition between units, which in turn reduce the inter-unit covariance. Such detailed balance has been observed across cortical areas and so it does not seem unreasonable to consider this as a method to encourage decorrelation of not just the input but also downstream 'hidden' layers of neural circuits.
When considering biological plausibility, the current work assumes that error signals are available and do not interfere with ongoing network activily. This means that we rely on two-phase credit assignment process. For fully online implementation, the error machinery should be integrated into single mixed pass, which is area for future exploration.
We conclude that constrained parameter inference allows for efficient and effective training of deep feedfor- ward neural networks while also providing a promising route towards biologically plausible deep learning.
References
Reza Abbasi-Asl, Cengiz Pehlevan, Bin Yu, and Dmitri Chklovskii. Do retinal ganglion cells project natural scenes to their principal subspace and whiten them? In 2016 50th Asilomar Conference on Signals, Systems and Computers, pp. 1641-1645, November 2016.
Nasir Ahmad, Marcel A J van Gerven, and Luca Ambrogioni. GAIT-prop: A biologically plausible learning rule derived from backpropagation of error. Advances in Neural Information Processing Systems, 33, December 2020.
Nasir Ahmad, Luca Ambrogioni, and Marcol A van Gerven. Overcoming the weight transport problem via teite--iminn-dependeee weight inference. Neurons, Behavior, Data Analysis, and Theory, 5(3):1 20, August 2021.
Mohamed Akrout, Collin Wilson, Peter C Humphreys, Timothy Lillicra,, and Douglas Tweed. Deep learning without weight transport. ArXiv:1904,0539. April 2019.
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1-43, January 2018.
Anthony J Bell and Terrence Sejnowski. The "independent components" of natural scenes are edge filters. Vision Research, (723))33327 3338, December 1997.
Yoshua Bengio. How aulo-encoders could provide credit assignment. in deep networks via larget propagation. ArXiv, July 2014.
Johanni Brea and Wulfram Gerstner. Does computational neuroscience need new synaptic learning paradigms? Current Opinion in Behavioral Sciences, 11:61 66, October 2016.
Michael Cogswell, Faruk Ahmed, Ross B Girshick, C L Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. International Conference on Learning Representations, May 2015.
Marlene R Cohen and John H R. Maunsell. Atlention improves performance primarily by reducing interneu- ronal correlations. Nature Neuroscience, 15411)11504-1600, December 2009.
Francis Crick. The recent excitement about neural networks. Nature, 337:129 13722, 1989.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Sander Dalm, Nasir Ahmad, Luca Ambrogioni, and Marcel A J van Gerven. Gradient-adjusted incremental target propagation provides effective credit assignment in deep neural networks. Transactions on Machine Learning Research, pp. 1 12, 2023.
Sophie Denève and Christian K Machens. Elficient codes and balanced networks. Nature Neuroscience, 19 (3):375-382, -82, March 2016.
Eric Movoy Dodds, Jesse Alexander Livezoy, and Michael Robert DeWeese. Spatial whitcning in the retina may necessary for V1 to learn a sparse representation of natural scenes. BioRrio, pp. 776799, September 2019.
Maxence Ernoult, Julie Grollier, Damien Querlioz, Yoshua Bengio, and Benjamin Scellier. Equilibrium propagation with continual weight updates. ArXiv, April 2020.
Peter Földiák. Forming sparse representations by local anti-Hebbian learning. Biological Cybernetics, 64(2): 165-170, December 1990.
Peter Földiák and Malcolm P Young. Sparse coding in the primate cortex. The Handbook of Brain Theory and Newral Networks. MIT Press, October 1998.
Katrin Franke, Philipp Berens, Timm Schubert, Matthias Bethge, Thomas Euler, and Tom Baden. Inhibition decorrelates visual feature representations in the inner retina. Nature, 542((644):433944M 4444444494 February 2017.
Daniel J Graham, Damon Chandler, and David J Field. Can the theory of "whitening" explain the center-surround properties of retinal ganglion cell receptive fields? Vision Research, 46(18):2901 2991199111111 September 2006.
Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11(1):23 63, January 1987. ISSN 0364-0213.
Jordan Guerguiev, Konrad P Kording, and Blake A Richards. Spike-based causal inference for weight aligument. ArXiv, October 2019.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 191-800, April 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing
internal covariate shift. ArXio, pp. 1-11, 2015.
Paul D King, Joel Zylberberg, and Michael R DeWeese. Tnhibitory interneurons decorrelate excitatory cells to drive sparse code formation in spiking model of V1. Journal of Newroscience, 33(1)):5475 5485, March 2013.
Diederik P Kingma and Jimmy Ba. Adam: A mothod for stochastic optimization. ArXiv, Docember 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, July 2009.
Yann LeCun, Corinna Cortes, and Christopher C Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yarn.leeun..omm www.:ewwwww...//eeeeeeee /mnist, 2, 2010.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7:13276, November 2016.
Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-344 346,, June 2020.
Seppo Linnainmaa. The representation of the cumulative rounding error of algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Unin. Helsinki, pp- 6 6-7, 1970.
12

---

Published in Transactions on Machine Learning Research (01/2023)
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advonces in Neural Information. Processing Systems, 29, December 2016.
Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology. 15(3):267-273) November 1982.
Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A Richards, and Richard Naud. Burst- dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature Neuroscience, 24(7): 1010-1019, July 2021.
Cengiz Pehlevan, Tao Hu, and Dmitri B Chklovskii. A Hebbian /antin/aniiiiaaa Mettin neural network for linear subspace learning: A derivation from multidimensional scaling of strcaming data. ArXiv, March 2015.
Xaq Pitkow and Markus Meister. Decorrelation and efficient coding by retinal ganglion cells. Nature Newroscience, 15(4):628 635, March 2012.
Gabrielle Ras, Ning Xie, Marcel van Gerven, and Derek Doran. Explainable deep learning: field guide for the uninitiated. Journal of Artificial Intelligence Research, 73:329 397, January 2022.
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Chris- tensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W Lindsay, Kenneth D Miller, Richard Naud, Christopher C Pack, Panayiota Poirazi, Picter Roelfsema, João Sacramento, Andrew Saxe, Ben- jamin Scellier, Anna C Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denís Therien, and Konrad P Kording. deep learning framework for neuroscience. Nature Newroscience, 22((1):1761 1770, October 2019.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use inter- pretable models instead. Nature Machine Intelligence, 155):206 215, May 2019.
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. Advances in Neural Information Processing Systems, 31: 8721 8732, December 2018.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-basod models and backpropagation. Fronliers in Compulational Neuroscience, 11:24, May 2017.
Irina Yonit Segal, Chen Giladi, Michael Gedalin, Michele Rucci, Mor Ben-Tov, Yam Kushinsky, Alik Moke- ichev, and Ronen Segev. Decorrelation of retinal response to natural scenes by fixational eye movements. Proceedings of the National Academy of Sciences U.S.A., 112(10):3110 3115, March 2015.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Enorgy and policy considcrations for deep learning in NLP. ArXiv, 2019.
Tim P Vogels, Henning Sprekeler, Friedemann Zenke, Claudia Clopath, and Wulfram Gerstner. Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks. Science, 334 16092))1569-157,, December 2011.
Shiqiang Wang. Efficient deep learning. Nature Computational Science, 1(3):181-182 March 2021. ISSN 26628457.
Paul Werbos. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University, Cambridge, MA, 1974.
James C R. Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Compulation, 29(5):1225 12222222.. May 2017.
13

---

Published in Transactions on Machine Learning Research (01/2023)
A COPI decorrelation as gradient descent
In the main text, we provided description of the custom learning rule for decorrelation which forms part of the COPI learning approach. Here we expand upon this description and frame the same derivation in terms of gradient descent upon a specific loss function.
The COPI learning algorithms require a docorrclated imput, meaning that our decorrelation method should minimise the ofl-diagonal values of B[xa"), where æ represents the (vector) input data any given layer and the expectation is taken empirically over a whole dataset. To this end, we can define element-wise quadratic loss function 4, representing the total undesirable correlation induced by unit, indexed with respect to all other units, indexed j, within a single sample such that:
4 4= == 2 8:
Σ (Z,2,)²
The derivative of this expression can then be taken with respect to unit i, in order to identify how to modify the activity of unit z; in order to reduce this loss. Specifically,
Ø4,
E (weerz) Tj. s#1
Or:
showing that, via stochastic gradient descent, we can produce greater decorrelation by computing the product between unit actívities and removing a unit-activity proportional measure from each unit, 24 <- I; 77 8 where 7 would be learning rate. Vectorizing this stochastic gradient descent across all units allows us to write an update for our data z such that
x- (xx=" diag (x²)) x
with learning rate v and diag(-) as used the main text, indicates constructing a square matrix of zeros with the given values upon the diagonal. Finally, as in the main text, we can assume that constructed from some transformation, transformation matrix, R, where
Ry, such that can recast this update in terms of the decorrclating
Ry - Ry-n - "xx=""xx"" diag(=")) Ry
Ry R [R-1] (xx" - diag(x²)) R] y.
providing equivalent to our derived update rule for decorrelation Acpi -1 (xx" diag(=?)) R, introduced in the main text.
One may ask why we constructed the specific decorrelation rule described above, rather than using an alter- native existing rulc. For that matter, one may ask why we chose to take the derivative of our decorrolation loss with respect to unit activities, I, when deriving this rule instead of directly with respect to the pa- rameters of the transformation matrix, R. First, the used derivation allowed the production of a simple, Hebbian-like update and allowed us to formulate, admittedly by approximation, similar and elegant learning rules for forward and lateral weight matrices. This important a promising start in order work toward methods for online and local learning of these transformations. Second, on more rigorous note, the learning rule we propose produces reductions in inter-unit correlations which are not affected by the scale (eigenvalues) of the matrix R. This is a property that is induced by our choice of taking the derivative of our decorrelation loss with respect to the unit activities, x, rather than the matrix elements R. Note that we can take the derivative of our above loss with respect a single element of our decorrelating matrix Raj in the following manner,
04 04 dx:
Σ (IaT)) 3:
OR, dr, dRij
However, reducing correlations by taking the full derivative with respect to the elements R, or via al- ternative existing methods which have been proposed for decorrelation through simple anti-Hebbian learn- ing Földiáá 1990 Pecllvvan al. 2015), result in reduction in correlation which is affected by the scale of the matrix R.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Reduction in z after single update
4R
anti-Hebbian at/aR
1.0
2.0
4.0
10.0
50.0
100.0
500.0
1000.0
Scaling of R (inverse scaling of y)
Figure 3: The reduction in correlation in a a toy-dataset when leveraging various decorrelation rules. To produce this plot, a dataset was randomly sampled from a multi-variant gaussian distribution and an initial decorrclating matrix R also samplod. This data, with samples y, is processed by the decorrclating matrix R to form outputs, x. A set of methods were then used to compute a single update to the decorrelating matrix, R, and the magnitude of reduction in the loss function (mean of loss l {= || - diag(=>)) over all datapoints in the dataset) was computed and plotted here. Various scalings were then applied to the matrix R and imput data which maintained the output data such that = (cR)) and the process of computing the efficacy of different learning rules repeated. As shown, only the proposed COPT decorrelating method produces a consistent reduction in the loss function, regardless of the scale of matrix R or the input data y. is for this reason that this rule is desirable when applied conjunction with other learning rules which require a relative scaling. The y-scale of this plot is omitted since its scale is arbitrarily dependent upon the initial sampling of data, and this plot is intended to be illustrative.
We can show this effect empirically in set of simple simulations measuring the magnitude of correlation reduction induced by various learning rules, see Figure X In order to produce these results, we first construct dataset by randomly sampling from 100-dimensional multivariate normal distribution with randomly sampled (arbitrary) covariance matrix. We further initialised a matrix R E I 10000000 composed of the identity function, I, plus random noise added to every element drawn from [-0.1,0.1] uniform distribution. This matrix, R, is used to process the input data, y, in order to attempt to produce decorrelated output x = Ry as in the mothods of the main part of this paper. In order to simulate an alternative scaling of the matrix R without affecting the output data distribution, we simulate a rescaling of R by removing a constant factor from the input data and scaling R by this factor, x = (cR)(y/e) With this setup, we could then demonstrate how various methods for learning the matrix R (with various scalings applied) reduce the loss function,
N Σ
(n(n) (z(m)) Σ
N
T - diag (z/(n) ) 2
C
(())
where n indexes the N samples in the empirical datasct. In Figure 3 is compared to the derivative of this loss with respect to the elements of matrix R, 0l./RRi above, and also against a simple hebi-a learning rule Földiáá 1990 - -(xxT" - diag(x²)).
the COPT lcarning rule for docorrclation
Pehlevan et al. 2015). where A
anti-hecbhian
As can be seen, the propose COPI learning rule is the only decorrelating learning rule which reduces the loss function by a consistent amount given some output distribution for x, regardless of the relative scaling of the decorrclating matrix and the input data y. Having such a decorrelation method, free from a learning rate interference through the scale of matrix R or the unused pre-decorrelation variable y, is crucial for the COPT learning system since the forward learning rule and decorrelating learning rules interact and must be
15

---

Published in Transactions on Machine Learning Research (01/2023)
balanced in their speed of learning to avoid runaway correlations affecting the efficacy of the forward learning
rule.
B Learning setup and parameters
Note that for all simulations which made use of decorrelation (all COPI networks and BP with decorrelation), the networks were first. trained for single epoch with only the decorrelation rule active. This allowed the network to reach decorrelated state (the desired state for this learning) before forward weight updating started. All hidden layers used the leaky rectified lincar unit (leaky-ReI.U) activation function. Training was carried out in mini-batches of size 50 for all simulations (stochastic updates computed within these mini-batches are averaged during application). Network execution and training is described in Algorithm [ Learning rate parameters are described in Table 2 All code used to produce the results this paper is
available at: https://gitt httpss//githbbcomm hateh/mb..oo/aaairyaaam///// / (ConstrainedParaantttffee
nasiryahm/
Table 2: Parameters for CIFAR-10 and MNIST trained networks (cf. Figure
Parameter
BP (Adam)
COPI (with BP/FA gradients)
BP (with decorr)
LeakyReLU negative slope
0.1
Learning rate f'w
0.0001
0.0001
Learning rate yj
0.0001
0.0001
Gain parameter
1.0
1000.0
Adam parameter 31
Adam parameter 92
0.999
Adam parameter
10-8
C Training networks of various depths
(adam)
capi (bp)
Final Performance (after 100 epochs of training)
1.000
0.975
0.950
0.925
0.900
0.875
0.850
Number of layers in trained network
Figure 4: Train and test accuracy of networks of various depths trained by COPI (BP) vs BP with the Adam optimiser, Performance of networks ranging from one to seven layers trained and tested using the MNIST handwritten digit set. The networks are composed such that each hidden layer (where present) is composed of 500 units, input layer size 784, and output layer of size 10.
The main text explored networks with fixed depths. Here we present additional results in which we trained and tested networks between one and seven layers on the MNIST handwritten digit set. These networks were each trained for 100 epochs and their performance measured after training. Parameters used for training followed the setup described in Appendix B
16

---

Published in Transactions on Machine Learning Research (01/2023)
As can be observed, the networks of extremely shallow depth (one and two layer), have measurably lower final performance compared to training by BP with the Adam optimizer. We found in our experimentation that this reflects the relative difficulty of decorrelating the input layer of our network, and the importance of the features in the first layer of the network. similar effect the importance of the first layer of the network for performance, can be observed the main text. Figure 2 network with a linear layer, we observed significant drop in performance when layers (including the first layer) were removed.
where, when approximating a deep
COPI with categorical cross-entropy loss
In the main text, we explored a quadratic loss in the mathematical derivation and simulation. However, this does not imply that COPT only be applied in the application of the quadratic loss function, in fact any arbitrary loss function applied to the outputs can be used to produce gradient-based target.
In particular, take any arbitrary loss which is solely a function of the outputs of network, t= f(yL). By dł
default, we would compute the derivative with respect to the outputs of the network as dyz, This arbitrary formulation differs from a quadratic loss with target, l,, since quadratic loss is proportional to yr. lr. However, possible to reformulate an arbitrary loss function computed on the outputs in terms of a target in the following manner
df df
+ (yız. y1.)
dyc dyL
dl
yL
dyr.
ti
where is target formulated for this layer.
10
1.00
(adam)
X
0.20
bp (decorr)
500
0.95
copl (bp)
Accuracy
0.15
copi (fa)
500
0.90
LOSS
train
0.10
test
500
0.05
500
0.00
0.75
784
0
20
40
50
70
90
100
20
40
50
70
90
100
Epochs
Epochs
Figure 5: The performance, measured by training and test accuracy/Ioog lloss, of five-layer fully- connected feedforward neural network architecture trained using a categorical cross-entropy loss. Plotted are various learning approaches combining decorrelation, our algorithu (COPI), and standard stochastic gradient descent by backpropagation of error (BP). All results are shown for training on the MNIST handwritten digit classification task. All networks were run with five random seeds and envelopes show standard deviation across these networks. Results for BIO-COPI not shown given its near identical performance to regular COPI.
We made use of such a harge-formmultiin approach (though in terms the derivative of the output hidden state az) to train the same network architecture used to train the networks Figure T with a categorical cross-entropy loss. This again demonstrates a favorable performance and convergence speed when training networks using COPI, though these sinulations have not been thoroughly optimized by parameter search.
17

---

Published in Transactions on Machine Learning Research (01/2023)
E
Relation between COPI and Oja's rule
Let us consider the (BIO-)COPI update for single synaptic weights, given by
copi Way
= 2, - wxyaz =
Σ WifRes
2j W2sz + 6,0r
A popi
(giag reszz)
Σ (-ris)a 2j - (-rig)a))
=
The first term both expressions Hebbian update which relies on the states of the pre-synaptic units zj and post-synaptic units 2; or 4 only. The second term in both expressions takes the form of weight decay. This functional form is similar to Oja's rule Oja 1982). which states:
A Agja mai
y: mayy?
Σ mijai Jj miy0ž
for y= = Mæ. COPI differs from Oja's rule in that the forward weight update has an additional term 6,z and the weight decay for both the forward and lateral weights depends on the (squared) pre-synaptic rather than post-synaptic firing rates. The functional impact of the difference in the weight decay scaling (by post- vs pre-synaptic firing rates), is that Oja's rule scales weight decay in order to normalize the scale of the weights which target post-synaptic neuron. By comparison, COPI scales the weight decay such as best infer the scale which the weights should take in order to reproduce the post-synaptic neuron activity given the observed pre-synaptic neuron activity.
18


Reviews:
[{"review_id": "cLU6LzuQwn", "reviewer": "Reviewer_hXBa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is clearly written, the derived learning algorithm is quite simple and yet effective. There are nice and clearly drawn bio-plausibility relationships throughout the manuscript regarding e.g., the local learning rules, or lateral weight updates to decorrelate layer-wise inputs.\n\nWeaknesses: I believe the major weakness that requires the most attention is the coverage of experimental simulations. Currently, not all aspects of the theoretically discussed and presented components of the learning algorithm are evaluated in the manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel learning algorithm to infer parameters of an artificial neural network. In this framework, learning can be established by updating the forward weight values based on observed local neuron activities, under the constraint that layer-wise neural inputs should be decorrelated via learned lateral weights (hence the name \"constrained parameter inference\": COPI). Proposed algorithm is also agnostic to the used credit assignment method to compute the layer-wise learning signal, where more bio-plausible local methods can be explored besides a standard gradient-based error signal. Experiments are performed with fully-connected networks on the MNIST and CIFAR-10 tasks, and COPI is compared against standard backpropagation based parameter optimization. Further contributions on using the outcome of the decorrelation constraint for network compression are discussed.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) It is not really clear to what extent the introduction of target propagation (TP) is relevant in the manuscript. Since it is introduced as a compatible credit assignment approach here, can the authors add in simulations (Fig 1 & Table 1) and show how well it works when one simulates COPI together with TP-based perturbations?\n\n2) Independent of the network compression experiments from Sec 3.2, can the authors demonstrate how important is the layer-wise depth of the FC network to learning with COPI from scratch? The networks in Fig 1, especially for MNIST, can be perhaps simulated with level-wise shallower architectures for this experiment.\n\n3) Authors' proposal in Appendix D remains empirically untouched. How well does this approximation perform?\n\n4) One of the interesting contributions is the network compression capability that arises from the produced linear approximations of receptive fields. However without addressing Appendix F, it is not possible to grasp how does this really work in Section 3.2. I believe a restructuring of the manuscript to address this would be helpful.\n\n5) I found the algorithmic summary in Appendix A quite useful. I think moving Algorithm 1 to the main manuscript might be beneficial in depicting how subsections of Sec 2 come together in forming the learning algorithm that is mainly proposed in the manuscript.", "broader_impact_concerns": "No specific concern.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ynZ7f38DN3", "reviewer": "Reviewer_ZrgT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The method is new, to my knowledge. The problem of plausible learning methods  is important.\n\n\nWeaknesses:\n- The method seems to make some requirement that put its purpose into question: if the required information described in the paper is provided, then traditional methods become local Hebbian-like learning rules, making the proposed method seemingly redundant?\n\n- Secondarily, it is not clear that the experiments are fair, because IIUC they oppose a method COPI with weight decay against BP without weight decay.\n\nIn more detail:\n\nAuthors describe COPI as a local rule, that (by assuming decorrelation) can learn directly from local activities and desired/target activities at any node.\n\nBut the way these desired activities are obtained is by providing a desired *direction*, or *increment*, much like the error gradient\n(over the weights) of BP, or the surrogate gradient of FB alignment. This desired direction of change is called little-delta in the text, in accordance with existing literature on gradient descent.\n\nBut if you have such a little-delta (desired direction of change in activity), then plain gradient learning is already purely local and\nindeed \"Hebbian\", with the delta rather than the activities: deltaW = x * little-delta.\n\nWhat, then, is gained at all by using the more complicated COPI ?\n\nIIUC the specific form of COPI, and especially its assumption of decorrelated inputs, is only necessary because it tries to express\nlearning as a function of raw activities and desired activities themselves, rather than the desired delta. Indeed this seems to be\nexactly what the \"Correspondence to stochastic gradient descent\" section says: the W(xx^T) term only pops up when we rewrite the plain\ngradient update, dW = delta * x, as a function of actual activities a (and desired activities a + delta) instead.\n\nBut then the method assumes that we already have access to the deltas! So what have we gained by using the added complexity and assumptions of COPI?\n\nExperiments are interesting, but difficult to interpret:\n\nThe authors claim the decorrelation layer improves performance across the board. But BP with Adam seems to outperform BP with decorrelation in terms of test error/accuracy in Figure 1a and (eventually) 1b? I may be missing something.\n\nAuthors observe that COPI beats BP (when both have similarly trained decorrelation layers) and attribute this to COPI having \" the built-in\nassumption that the inputs have a decorrelated form\".\n\nBut IIUC COPI includes a weight decay term. Was there a weight decay in the BP version?  If there is no weight decay in the BP experiments, this might be a source  of the difference in results. This is particularly acute since at least some of\nthe failure of BP in the figures seems to be overfitting (test error increasing), and also considering the large size of the networks. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel method for learning in neural networks. This method can learn directly and locally at each synapse from the activities (and desired activities) of the neurons, assuming that 1) desired activities (or desired directions of change in activities) are available for each neuron, and 2) inputs are uncorrelated. The method essentially performs Hebbian learning between inputs and desired outputs, with a weight decay term modulated by the variance of each input.\n\nThe authors propose a method to learn decorrelating layers that decorrelate the output of each main layer, facilitating assumption 2. \n\nVarious experiments suggest that the method is competitive with backpropagation with Adam.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please explain what exactly COPI brings, when  it requires access to desired directions of changes in activities that already allow for local, Hebbian-like learning.\n\n- In experiments, include a weight decay component in BP, comparable to that of COPI, and report results.\n\n- Last paragraph on p.6: please clarify the statement that decorrelation improves performance also for BP, which seems to conflict with my understanding of Figure 1a and (in the long run) 1b. \n\n- I don't understand the visualization part (second to last paragraph of p.  8). Why does decorrelation help here ? How\nexactly is Figure 2b generated?", "broader_impact_concerns": "I do not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lEfuiEvZPf", "reviewer": "Reviewer_UPjV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think that the topic of efficient bio-plausible rules for decorrelation in deep networks is important and I find the model $y=f(WRx)$ to be a simple and elegant solution. The plausible update of R which avoids some problem is therefore interesting as well, although I would have liked to know what plausibility it resolves specifically.\n\nI find the relevant on the learning rate of the decorrelated BP and COPI to be very promising and I would love to see a bit more analysis on that. Here are suggestion for control experiments:  \n\n(a) what if the learning rate parameter $\\eta$ is optimized individually for classical BP and decorrelated BP, would a higher learning rate be more beneficial for BP?  (just reporting the next power of ten to prove it is not that easy to make faster would be sufficient)\n\n(b) I understand that the authors report the result of classical BP as the case with $y=f(WRx)$ with $R$ optimized with SGD. What if $f(Wx)$, is the learning rate better? is the final accuracy on CIFAR better?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes a learning rule in a variant of a classical fully connected multi-layer perceptron. The key difference in the network architecture is to add a matrix $R$ prior to each matrix multiplication with the weights $W$ leading to an effective network activity defined by: $f(WRx)$ where $x$ in the input for this layer. Then the learning rule is defined so that $Rx$ generates a decorrelated (i.e. whitened) version of $x$ while $W$ minimizes the loss function (this defines the BP with decorrelation algorithm reported in the results). To further exhibit a variant of the weights updates for $R$ and $W$ which avoid is \"more biologically plausible\", the COPI learning rule replaces the covariance $\\mathbb{E}[x x^T]$ with its diagonal $diag(x^2)$. The learning rule are then compared on MNIST and CIFAR with many fully connected layers, a strong finding is that decorrelated BP and COPI learn much faster than regular BP for the same performance level.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\nFor the technical description in the main text: the equations and the text is understandable and clear although I believe it could be written in a lighter fashion. More problematic, I find that the general story of the paper is sometimes detached from the mathematical content and could be improved in three ways:  \n\n(1) I find the general framing of the paper around \"constrained parameter optimization\" to be quite miss leading because I thought it would perform some kind of constrained optimization but the technical part of the paper is not described in this way.  \n\n(2) I also think that the paper would from a clearer definition of the meaning of \"biologically plausible\" learning rule which is solved in this paper with a clear statement about why is that problematic for biology: concretely which terms in BP or BP with decorrelation are problematic and how does COPI solves it? (BIOCOPI is also defined in the appendix, but what is the difference? and why does that matter?)  \n\n(3) Unfortunately I did not understand the whole discussion about network compression. I believe that this relates usually to weights pruning or something like that but I did not see how is that relevant for the theoretical method and the experiments. Also I did not understand the figure in the main text about \"removed layers\". What does that mean in the experiments and why is that interesting?  \n\n\nFor the figures reporting the accuracy I would recommend to zoom in in the relevant range (for MNIST 85% - 100%) because this is where everything is happening and I would have liked to see more clearly those learning curves.  \n\n\nAlso I cannot see the filters in Figure 2B because there are too small.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the revised version of the manuscript can be accepted for publication.
There was some discussion on restrictive assumptions of the work. It was noted that the proposed algorithm is useful if the plasticity rule does not have access of the plasticity process to desired directions of activity changes and a paragraph which clarifies this was added in a final revision as demanded. As this was clarified, I recommend acceptance.
Decision: Accept (poster)