SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lyndon R. Duong<sup>1,2</sup>
Eero P. Simoncelli<sup>1,2</sup>
Dmitri B. Chklovskii<sup>1,3</sup>
David Lipshutz<sup>1</sup>
<sup>1</sup> Center for Computational Neuroscience, Flatiron Institute
$^{2}$  Center for Neural Science, New York University
<sup>3</sup> Neuroscience Institute, NYU Langone Medical School
{lyndon.duong, eero.simoncelli}@nyu.edu {dchklovskii, dlipshutz}@flatironinstitute.org
Neurons in early sensory areas rapidly adapt to changing sensory statistics, both by normalizing the variance of their individual responses and by reducing correlations between their responses. Together, these transformations may be viewed as an adaptive form of statistical whitening. Existing mechanistic models of adaptive whitening exclusively use either synaptic plasticity or gain modulation as the biological substrate for adaptation; however, on their own, each of these models has significant limitations. In this work, we unify these approaches in a normative multi-timescale mechanistic model that adaptively whitens its responses with complementary computational roles for synaptic plasticity and gain modulation. Gains are modified on a fast timescale to adapt to the current statistical context, whereas synapses are modified on a slow timescale to match structural properties of the input statistics that are invariant across contexts. Our model is derived from a novel multi-timescale whitening objective that factorizes the inverse whitening matrix into basis vectors, which correspond to synaptic weights, and a diagonal matrix, which corresponds to neuronal gains. We test our model on synthetic and natural datasets and find that the synapses learn optimal configurations over long timescales that enable adaptive whitening on short timescales using gain modulation.
Individual neurons in early sensory areas rapidly adapt to changing sensory statistics by normalizing the variance of their responses [1-3]. At the population level, neurons also adapt by reducing correlations between their responses [4; 5]. These adjustments enable the neurons to maximize the information that they transmit by utilizing their entire dynamic range and reducing redundancies in their representations [6-9]. A natural normative interpretation of these transformations is adaptive whitening, a context-dependent linear transformation of the sensory inputs yielding responses that have unit variance and are uncorrelated.
Decorrelation of the neural responses requires coordination between neurons and the neural mechanisms underlying such coordination are not known. Since neurons communicate via synaptic connections, it is perhaps unsurprising that most existing mechanistic models of adaptive whitening decorrelate neural responses by modifying the strength of these connections [10-16]. However, long-term synaptic plasticity is generally associated with long-term learning and memory [17], and thus may not be a suitable biological substrate for adaptive whitening (though short-term synaptic
Figure 1: Adaptive whitening circuit, illustrated with  $N = 2$  primary neurons and  $K = 2$  interneurons. Left: Dashed ellipses representing the covariance matrices of 2D stimuli s drawn from different statistical contexts. Center: Primary neurons (shaded blue circles) receive feedforward stimulus inputs (shaded purple circles), s, and recurrent weighted inputs,  $-\mathbf{W}\mathbf{n}$ , from the interneurons (teal circles), producing responses r. The interneurons receive weighted inputs,  $\mathbf{z} = \mathbf{W}^{\top}\mathbf{r}$ , from the primary neurons, which are then multiplied elementwise by gains g to generate their outputs,  $\mathbf{n} = \mathbf{g} \circ \mathbf{z}$ . The gains g are modulated at a fast timescale to adaptively whiten within a specific stimulus context. Concurrently, the synaptic weights are optimized at a slower timescale to learn structural properties of the inputs across contexts. Right: Dashed unit circles representing the whitened circuit responses r in each statistical context.
plasticity has been reported [18]). On the other hand, there is extensive neuroscience literature on rapid and reversible gain modulation [19-26]. Motivated by this, Duong et al. [27] proposed a mechanistic model of adaptive whitening in a neural circuit with fixed synaptic connections that adapts exclusively by modifying the gains of interneurons that mediate communication between the primary neurons. They demonstrate that an appropriate choice of the fixed synaptic weights can both accelerate adaptation and significantly reduce the number of interneurons that the circuit requires. However, it remains unclear how the circuit learns such an optimal synaptic configuration, which would seem to require synaptic plasticity.
In this study, we combine the learning and adaptation of synapses and gains, respectively, in a unified mechanistic neural circuit model that adaptively whitens its inputs over multiple timescales (Fig. 1). Our main contributions are as follows:
1. We introduce a novel adaptive whitening objective in which the (inverse) whitening matrix is factorized into a synaptic weight matrix that is optimized across contexts and a diagonal (gain) matrix that is optimized within each statistical context.
2. With this objective, we derive a multi-timescale online algorithm for adaptive whitening that can be implemented in a neural circuit comprised of primary neurons and an auxiliary population of interneurons with slow synaptic plasticity and fast gain modulation (Fig. 1).
3. We test our algorithm on synthetic and natural datasets, and demonstrate that the synapses learn optimal configurations over long timescales that enable the circuit to adaptively whiten its responses on short timescales exclusively using gain modulation.
Beyond the biological setting, multi-timescale learning and adaptation may also prove important in machine learning systems. For example, Mohan et al. [28] introduced "gain-tuning", in which the gains of channels in a deep denoising neural network (with pre-trained synaptic weights) are adjusted to improve performance on samples with out-of-distribution noise corruption or signal properties. The normative multi-timescale framework developed here offers a new approach to continual learning and test-time adaptation problems such as this.
2 Adaptive symmetric whitening
Consider a neural population with  $N$  primary neurons (Fig. 1). The stimulus inputs to the primary neurons are represented by a random  $N$ -dimensional vector  $\mathbf{s}$  whose distribution  $p(\mathbf{s}|c)$  depends
on a latent context variable  $c$ . The stimulus inputs  $s$  can be inputs to peripheral sensory neurons (e.g., the rates at which photons are absorbed by  $N$  cones) or inputs to neurons in an early sensory area (e.g., glomerulus inputs to  $N$  mitral cells in the olfactory bulb). Context variables can include location (e.g., a forest or a meadow) or time (e.g., season or time of day). For simplicity, we assume the context-dependent inputs are centered; that is,  $\mathbb{E}_{\mathbf{s} \sim p(\mathbf{s} | c)}[\mathbf{s}] = \mathbf{0}$ , where  $\mathbb{E}_{\mathbf{s} \sim p(\mathbf{s} | c)}[\cdot]$  denotes the expectation over the conditional distribution  $p(\mathbf{s} | c)$  and  $\mathbf{0}$  denotes the vector of zeros. See Appx. A for a consolidated list of notation used throughout this work.
The goal of adaptive whitening is to linearly transform the inputs  $\mathbf{s}$  so that, conditioned on the context variable  $c$ , the  $N$ -dimensional neural responses  $\mathbf{r}$  have identity covariance matrix; that is,
$$
\mathbf {r} = \mathbf {F} _ {c} \mathbf {s} \quad \text {s u c h t h a t} \quad \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {s} | c)} \left[ \mathbf {r r} ^ {\top} \right] = \mathbf {I} _ {N},
$$
where  $\mathbf{F}_c$  is a context-dependent  $N\times N$  whitening matrix. Whitening is not a unique transformation—left multiplication of the whitening matrix  $\mathbf{F}_c$  by any  $N\times N$  orthogonal matrix results in another whitening matrix. We focus on symmetric whitening, also referred to as Zero-phase Components Analysis (ZCA) whitening or Mahalanobis whitening, in which the whitening matrix for context  $c$  is uniquely defined as
$$
\mathbf {F} _ {c} = \mathbf {C} _ {s s} ^ {- 1 / 2} (c), \quad \mathbf {C} _ {s s} (c) := \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {s} | c)} \left[ \mathbf {s s} ^ {\top} \right], \tag {1}
$$
where we assume  $\mathbf{C}_{ss}(c)$  is positive definite for all contexts  $c$ . This is the unique whitening transformation that minimizes the mean-squared difference between the inputs and the outputs [29].
To derive an algorithm that learns the symmetric whitening matrix  $\mathbf{F}_c$ , we express  $\mathbf{F}_c$  as the solution to an appropriate optimization problem, which is similar to the optimization problem in [15, top of page 6]. For a context  $c$ , we can write the inverse symmetric whitening matrix  $\mathbf{M}_c \coloneqq \mathbf{F}_c^{-1}$  as the unique optimal solution to the minimization problem
$$
\mathbf {M} _ {c} = \underset {\mathbf {M} \in \mathbb {S} _ {+ +} ^ {N}} {\arg \min } f _ {c} (\mathbf {M}), \quad f _ {c} (\mathbf {M}) := \operatorname {T r} \left(\mathbf {M} ^ {- 1} \mathbf {C} _ {s s} (c) + \mathbf {M}\right), \tag {2}
$$
where  $\mathbb{S}_{++}^{N}$  denotes the set of  $N\times N$  positive definite matrices. This follows from the fact that  $f_{c}(\mathbf{M})$  is strictly convex with its unique minimum achieved at  $\mathbf{M}_c$ , where  $f_{c}(\mathbf{M}_{c}) = 2\mathrm{Tr}(\mathbf{M}_{c})$  (Appx. B.1). Existing recurrent neural circuit models of adaptive whitening solve the minimization problem in Eq. 2 by choosing a matrix factorization of  $\mathbf{M}_c$  and then optimizing the components [12; 13; 15; 27].
3 Adaptive whitening in neural circuits: a matrix factorization perspective
Here, we review two adaptive whitening objectives, which we then unify into a single objective that adaptively whitens responses across multiple timescales.
3.1 Objective for adaptive whitening via synaptic plasticity
Pehlevan and Chklovskii [12] proposed a recurrent neural circuit model that whitens neural responses by adjusting the synaptic weights between the  $N$  primary neurons and  $K \geq N$  auxiliary interneurons according to a Hebbian update rule. Their circuit can be derived by factorizing the context-dependent matrix  $\mathbf{M}_c$  into a symmetric product  $\mathbf{M}_c = \mathbf{W}_c\mathbf{W}_c^\top$  for some context-dependent  $N \times K$  matrix  $\mathbf{W}_c$  [15]. Substituting this factorization into Eq. 2 results in the synaptic plasticity objective in Table 1. In the recurrent circuit implementation,  $\mathbf{W}_c^\top$  denotes the feedforward weight matrix of synapses connecting primary neurons to interneurons and the matrix  $-\mathbf{W}_c$  denotes the feedback weight matrix of synapses connecting interneurons to primary neurons. Importantly, under this formulation, the circuit may reconfigure both the synaptic connections and synaptic strengths each time the context  $c$  changes, which runs counter to the prevailing view that synaptic plasticity implements long-term learning and memory [17].
3.2 Objective for adaptive whitening via gain modulation
Duong et al. [27] proposed a neural circuit model with fixed synapses that whitens the  $N$  primary responses by adjusting the multiplicative gains in a set of  $K$  auxiliary interneurons. To derive a neural
For technical purposes, we extend the definition of  $f_{c}$  to all  $\mathbb{S}^N$  by setting  $f_{c}(\mathbf{M}) = \infty$  if  $\mathbf{M}\not\in\mathbb{S}_{++}^{N}$ .
Table 1: Factorizations of the inverse whitening matrix  ${\mathbf{M}}_{c}$  and objectives for adaptive whitening circuits.
Model	Matrix factorization	Objective
Synaptic plasticity [12]	WcWcT	minwfc(WW)
Gain modulation [27]	IN+Wfixdiag(gc)WfixT	mingfc(IN+Wfixdiag(g)WfixT)
Multi-timescale (ours)	αIN+Wdiag(gc)WT	minwEc~p(c) [mingfc(αIN+Wdiag(g)WT)]
circuit with gain modulation, they considered a novel diagonalization of the inverse whitening matrix,  $\mathbf{M}_c = \mathbf{I}_N + \mathbf{W}_{\mathrm{fix}}\mathrm{diag}(\mathbf{g}_c)\mathbf{W}_{\mathrm{fix}}^\top$ , where  $\mathbf{W}_{\mathrm{fix}}$  is an arbitrary, but fixed  $N\times K$  matrix of synaptic weights (with  $K\geq K_N\coloneqq N(N + 1) / 2$ ) and  $\mathbf{g}_c$  is an adaptive, context-dependent real-valued  $K$ -dimensional vector of gains. Note that unlike the conventional eigen-decomposition, the number of elements along the diagonal matrix is significantly larger than the dimensionality of the input space. Substituting this factorization into Eq. 2 results in the gain modulation objective in Table 1. As in the synaptic plasticity model,  $\mathbf{W}_{\mathrm{fix}}^{\top}$  denotes the weight matrix of synapses connecting primary neurons to interneurons while  $-\mathbf{W}_{\mathrm{fix}}$  connects interneurons to primary neurons. In contrast to the synaptic plasticity model, the interneuron outputs are modulated by context-dependent multiplicative gains,  $\mathbf{g}_c$ , that are adaptively adjusted to whiten the circuit responses.
Duong et al. [27] demonstrate that an appropriate choice of the fixed synaptic weight matrix can both accelerate adaptation and significantly reduce the number of interneurons in the circuit. In particular, the gain modulation circuit can whiten any input distribution provided the gains vector  $\mathbf{g}_c$  has dimension  $K \geq K_N$  (the number of degrees of freedom in an  $N \times N$  symmetric covariance matrix). However, in practice, the circuit need only adapt to input distributions corresponding to natural input statistics [7; 30-32]. For example, the statistics of natural images are approximately translation-invariant, which significantly reduces the degrees of freedom in their covariance matrices, from  $\mathcal{O}(N^2)$  to  $\mathcal{O}(N)$ . Therefore, while the space of all possible correlation structures is  $K_N$ -dimensional, the set of natural statistics likely has far fewer degrees of freedom and an optimal selection of the weight matrix  $\mathbf{W}_{\mathrm{fix}}$  can potentially offer dramatic reductions in the number of interneurons  $K$  required to adapt. As an example, Duong et al. [27] specify a weight matrix for performing "local" whitening with  $\mathcal{O}(N)$  interneurons when the input correlations are spatially-localized (e.g., as in natural images). However, they do not prescribe a method for learning a (synaptic) weight matrix that is optimal across the set of natural input statistics.
3.3 Unified objective for adaptive whitening via synaptic plasticity and gain modulation
We unify and generalize the two disparate adaptive whitening approaches [12; 27] in a single multitemscales nested objective in which gains  $\mathbf{g}$  are optimized within each context and synaptic weights  $\mathbf{W}$  are optimized across contexts. In particular, we optimize, with respect to  $\mathbf{W}$ , the expectation of the objective from [27] (for some fixed  $K\geq 1$ ) over the distribution of contexts  $p(c)$ :
$$
\min  _ {\mathbf {W} \in \mathbb {R} ^ {N \times K}} \mathbb {E} _ {c \sim p (c)} \left[ \min  _ {\mathbf {g} \in \mathbb {R} ^ {K}} f _ {c} \left(\alpha \mathbf {I} _ {N} + \mathbf {W} \operatorname {d i a g} (\mathbf {g}) \mathbf {W} ^ {\top}\right) \right], \tag {3}
$$
where we have also generalized the objective from [27] by including a fixed multiplicative factor  $\alpha \geq 0$  in front of the identity matrix  $\mathbf{I}_N$ , and we have relaxed the requirement that  $K \geq K_N$ .
What is an optimal solution of Eq. 3? Since the convex function  $f_{c}$  is uniquely minimized at  $\mathbf{M}_c$ , a sufficient condition for the optimality of a synaptic weight matrix  $\mathbf{W}$  is that for each context  $c$ , there is a gain vector  $\mathbf{g}_c$  such that  $\alpha \mathbf{I}_N + \mathbf{W} \mathrm{diag}(\mathbf{g}_c) \mathbf{W}^\top = \mathbf{M}_c$ . Importantly, under such a synaptic configuration, the function  $f_{c}$  can attain its minimum exclusively by adjusting the gains vector  $\mathbf{g}$ . In the space of covariance matrices, we can express the statement as
$$
\mathbf {C} _ {s s} (c) \in \mathbb {F} (\mathbf {W}) := \left\{\left[ \alpha \mathbf {I} _ {N} + \mathbf {W} \operatorname {d i a g} (\mathbf {g}) \mathbf {W} ^ {\top} \right] ^ {2}: \mathbf {g} \in \mathbb {R} ^ {K} \right\} \cap \mathbb {S} _ {+ +} ^ {N} \quad \text {f o r e v e r y c o n t e x t} c,
$$
where  $\mathbb{F}(\mathbf{W})$  contains the set of covariance matrices that can be whitened with fixed synapses  $\mathbf{W}$  and adaptive gains  $\mathbf{g}$ . Fig. 2 provides an intuitive Venn diagram comparing a non-optimal synaptic configuration  $\mathbf{W}_0$  and an optimal synaptic configuration  $\mathbf{W}_T$ .
Figure 2: Illustration of multi-timescale learning in the space of covariance matrices. Orange and purple regions (identical on the left and right) respectively represent the cone of all positive definite matrices  $\mathbb{S}_{++}^N$ , and the subset of naturally-occurring covariance matrices  $\{\mathbf{C}_{ss}(c)\}$ . Blue regions represent the set of covariance matrices that can be whitened with adaptive gains for a particular synaptic weight matrix. On each side, the yellow circle denotes a naturally-occurring input covariance matrix  $\mathbf{C}_{ss}(c)$  and the dotted white curve illustrates the trajectory of covariance matrices the circuit is adapted to whiten as the gains are modulated (with fixed synapses, note the dotted white curve remains in the blue region). Left: With initial synaptic weights  $\mathbf{W}_0$  the circuit cannot whiten some natural input distributions exclusively via gain modulation, i.e.,  $\{\mathbf{C}_{ss}(c)\} \not\subset \mathbb{F}(\mathbf{W}_0)$ . Right: After learning optimal synaptic weights  $\mathbf{W}_T$ , the circuit can match any naturally-occurring covariance matrix using gain modulation, i.e.,  $\{\mathbf{C}_{ss}(c)\} \subset \mathbb{F}(\mathbf{W}_T)$ .
4 Multi-timescale adaptive whitening algorithm and circuit implementation
In this section, we derive an online algorithm for optimizing the multi-timescale objective in Eq. 3, then map the algorithm onto a neural circuit with fast gain modulation and slow synaptic plasticity. Direct optimization of Eq. 3 results in an offline (or full batch) algorithm that requires the network to have access to the covariance matrices  $\mathbf{C}_{ss}(c)$ , Appx. C. Therefore, to derive an online algorithm that includes neural dynamics, we first add neural responses  $\mathbf{r}$  to the objective, which introduces a third timescale to the objective. We then derive a multi-timescale gradient-based algorithm for optimizing the objective.
Adding neural responses to the objective. First, observe that we can write  $f_{c}(\mathbf{M})$ , for  $\mathbf{M} \in \mathbb{S}_{++}^{N}$  in terms of the neural responses  $\mathbf{r}$ :
$$
f _ {c} (\mathbf {M}) = \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {s} | c)} \left[ \max  _ {\mathbf {r} \in \mathbb {R} ^ {N}} \operatorname {T r} \left(2 \mathbf {r s} ^ {\top} - \mathbf {M r r} ^ {\top} + \mathbf {M}\right) \right]. \tag {4}
$$
To see this, maximize over  $\mathbf{r}$  to obtain  $\mathbf{r} = \mathbf{M}^{-1}\mathbf{s}$  and then use the definition of  $\mathbf{C}_{ss}(c)$  from Eq. 1 (Appx. B.2). Substituting this expression for  $f_{c}$ , with  $\mathbf{M} = \alpha \mathbf{I}_N + \mathbf{W}\mathrm{diag}(\mathbf{g})\mathbf{W}^\top$ , into Eq. 3, dropping the constant term  $\alpha \mathbf{I}_N$  term and using the cyclic property of the trace operator results in the following objective with 3 nested optimizations (Appx. B.2):
$$
\min  _ {\mathbf {W} \in \mathbb {R} ^ {N} \times K} \mathbb {E} _ {c \sim p (c)} \left[ \min  _ {\mathbf {g} \in \mathbb {R} ^ {K}} \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {s} | c)} \left[ \max  _ {\mathbf {r} \in \mathbb {R} ^ {N}} \ell (\mathbf {W}, \mathbf {g}, \mathbf {r}, \mathbf {s}) \right] \right], \tag {5}
$$
$$
\text {w h e r e} \quad \ell (\mathbf {W}, \mathbf {g}, \mathbf {r}, \mathbf {s}) := 2 \mathbf {r} ^ {\top} \mathbf {s} - \alpha \| \mathbf {r} \| ^ {2} - \sum_ {i = 1} ^ {K} g _ {i} \left[ \left(\mathbf {w} _ {i} ^ {\top} \mathbf {r}\right) ^ {2} - \| \mathbf {w} _ {i} \| ^ {2} \right].
$$
The inner-most optimization over  $\mathbf{r}$  corresponds to neural responses and will lead to recurrent neural dynamics. The outer 2 optimizations correspond to the optimizations over the gains  $\mathbf{g}$  and synaptic weights  $\mathbf{W}$  from Eq. 3.
To solve Eq. 5 in the online setting, we assume there is a timescale separation between neural dynamics and the gain/weight updates. This allows us to perform the optimization over  $\mathbf{r}$  before optimizing  $\mathbf{g}$  and  $\mathbf{W}$  concurrently. This is biologically sensible: neural responses (e.g., action potential firing) operate on a much faster timescale than gain modulation and synaptic plasticity [25; 33].
Recurrent neural dynamics. At each iteration, the circuit receives a stimulus  $\mathbf{s}$ . We maximize  $\ell(\mathbf{W}, \mathbf{g}, \mathbf{r}, \mathbf{s})$  with respect to  $\mathbf{r}$  by iterating the following gradient-ascent steps that correspond to repeated timesteps of the recurrent circuit (Fig. 1) until the responses equilibrate:
$$
\mathbf {r} \leftarrow \mathbf {r} + \eta_ {r} \left(\mathbf {s} - \sum_ {i = 1} ^ {K} n _ {i} \mathbf {w} _ {i} - \alpha \mathbf {r}\right), \tag {6}
$$
where  $\eta_r > 0$  is a small constant,  $z_i = \mathbf{w}_i^\top \mathbf{r}$  denotes the weighted input to the  $i^{\mathrm{th}}$  interneuron,  $n_i = g_i z_i$  denotes the gain-modulated output of the  $i^{\mathrm{th}}$  interneuron. For each  $i$ , synaptic weights,  $\mathbf{w}_i$ , connect the primary neurons to the  $i^{\mathrm{th}}$  interneuron and symmetric weights,  $-\mathbf{w}_i$ , connect the  $i^{\mathrm{th}}$  interneuron to the primary neurons. From Eq. 6, we see that the neural responses are driven by feedforward stimulus inputs  $\mathbf{s}$ , recurrent weighted feedback from the interneurons  $-\mathbf{W}\mathbf{n}$ , and a leak term  $-\alpha \mathbf{r}$ .
Fast gain modulation and slow synaptic plasticity. After the neural activities equilibrate, we minimize  $\ell (\mathbf{W},\mathbf{g},\mathbf{r},\mathbf{s})$  by taking concurrent gradient-descent steps
$$
\Delta g _ {i} = \eta_ {g} \left(z _ {i} ^ {2} - \| \mathbf {w} _ {i} \| ^ {2}\right) \tag {7}
$$
$$
\Delta \mathbf {w} _ {i} = \eta_ {w} \left(\mathbf {r} n _ {i} - \mathbf {w} _ {i} g _ {i}\right), \tag {8}
$$
where  $\eta_{g}$  and  $\eta_w$  are the respective learning rates for the gains and synaptic weights. By choosing  $\eta_g\gg \eta_w$ , we can ensure that the gains are updated at a faster timescale than the synaptic weights.
The update to the  $i^{\mathrm{th}}$  interneuron's gain  $g_{i}$  depends on the difference between the online estimate of the variance of its input,  $z_i^2$ , and the squared-norm of the  $i^{\mathrm{th}}$  synaptic weight vector,  $\| \mathbf{w}_i\|^2$ , quantities that are both locally available to the  $i^{\mathrm{th}}$  interneuron. Using the fact that  $z_{i} = \mathbf{w}_{i}^{\top}\mathbf{r}$ , we can rewrite the gain update as  $\Delta g_{i} = \eta_{g}[\mathbf{w}_{i}^{\top}(\mathbf{rr}^{\top} - \mathbf{I}_{N})\mathbf{w}_{i}]$ . From this expression, we see that the gains equilibrate when the marginal variance of the responses along the direction  $\mathbf{w}_i$  is 1, for  $i = 1,\dots ,K$ .
The update to the  $(i,j)^{\mathrm{th}}$  synaptic weight  $w_{ij}$  is proportional to the difference between  $r_i n_j$  and  $w_{ij} g_j$ , which depends only on variables that are available in the pre- and postsynaptic neurons. Since  $r_i n_j$  is the product of the pre- and postsynaptic activities, we refer to this update as Hebbian. In Appx. E.2, we decouple the feedforward weights  $\mathbf{w}_i^\top$  and feedback weights  $-\mathbf{w}_i$  and provide conditions under which the symmetry asymptotically holds.