SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we aim to understand the factors that may degrade the quality of the gradient estimate as a result of the approximation  $\delta_t^l\approx \mathrm{d}_{h_t}L_t^\top$  which introduces bias in the gradient estimate. In the LRU, neural activities, and thus error signals, are processed temporally through dynamics similar to the one of Equation 1. When the norm of each  $\lambda_{i}$  approaches one, neural activity preserves past information, and correspondingly, error signals backpropagated over time contain more information about the future. This suggests that our approximation becomes less accurate as the distribution of  $|\lambda_i|$  narrows around 1, since it discards future error information. Moreover,  $\delta$  worsens as it is backpropagated through more layers. At each layer, backpropagation mixes errors from the next layer and the future state of the layer. Since we neglect future information, only part of the error signal is backpropagated, resulting in a less accurate approximation. We delve deeper into these two approximation sources in a memory task in Section 4.1.
In the following, we analyze the properties of our proposed online learning method empirically and explore how independent recurrent modules aid learning of long-range dependencies. To this end, we first conduct an extensive analysis on the copy task [40], a well-established test bed to study temporal credit assignment in recurrent neural networks. Comparisons to truncated online versions of BPTT and to an extension of SnAp-1 to deep recurrent networks, reveal that independent recurrent modules are generally beneficial for learning long-range dependencies, as they excel in both the online and the offline setting. Finally, we evaluate our method on three tasks of the Long Range Arena [24]: a sequential version of CIFAR [41], LISTOPS [42] and IMDB [43], scaling online learning to sequence lengths of over 4000 time steps and to deep recurrent neural networks. For additional experimental details and hyperparameter configurations, we refer to Appendix B.
4.1 Understanding the approximations behind online learning in networks of LRUs
In this first series of experiments, we investigate the approximation introduced by our online algorithm in detail. We recall that our learning rule only approximates the full gradient when the network has two recurrent layers or more, as it ignores the temporal component of backpropagated error signals. Therefore, we expect that the learning signal becomes less accurate as we increase network depth and shift the eigenvalue distribution towards a norm of 1. To explore the effects of our approximation in a controlled setting, we consider a copy task [40, 10, 15, 18] in which the network observes a length-20 sequence of 7-bit patterns that it must recall when presented with an output token. Intuitively, temporal credit assignment is critical in this task to, among other things, convert forgetting neurons  $|\lambda| \ll 1$  into perfect memory units  $(|\lambda| \approx 1)$  that can solve the task. To ensure that these perfect memory units are scarce at the beginning of training, necessitating learning to create some, we initialize  $\lambda$  uniformly at random in the complex unit disk and set the number of recurrent units per layer to  $N = 64$ . The default architecture used in this section has four layers.
Figure 2: Impact of depth of the network (left), eigenvalues of the network (middle), and type of approximation on the quality of online learning (right). The cosine similarity measures the alignment between the estimated gradient (with our learning rule for all panels, and with spatial backpropagation (Spat.) and 1-step truncated backpropagation (Trunc.) for panels E and F) and the true gradient, computed with backpropagation-through-time (BP). It is computed per layer and then averaged across layers to make a quantitative comparison possible. The encoder and decoder alignments do not influence this metric. This task is solved (100% accuracy) for losses lower than 0.05 and 70% accuracy roughly corresponds to a loss of 0.5. See Section 4.1 for more details.
As remarked in Section 3.2, the updates prescribed by our learning rule match the gradient exactly for all parameters in the last LRU layer. We confirm that empirically for a network of one layer in Figure 2.A. While the approximation quality deteriorates with increasing depth (Fig. 2.A), alignment remains high, noticeably better than for all baseline online learning rules (Fig. 2.E). Moreover, despite alignment decreasing with depth, performance enhances significantly (Fig. 2.B). BPTT exhibits similar improvements with depth, suggesting that this is likely due to the enhanced capabilities of the model. Our rule can learn useful hierarchical temporal representations online whereas baseline methods, 1-step truncated and spatial backpropagation, which ignore most of temporal dependencies, fail (c.f. Fig. 2.F). Additionally, we found that, despite its bias, our learning rule can decrease the loss to 0 when training a 4 layers network on a simpler memory task for long enough.
Exact error terms are backpropagated through recurrent units by weighting the current prediction error by 1 and the future error by  $\lambda$ . In order to maintain an online algorithm, we ignore this dependency on the future. We expect alignment with the gradient to decrease as the distribution of  $|\lambda|$  shifts towards 1. To test that, we initialize the  $\lambda$  uniformly at random in the complex ring of radius  $[|\lambda|_{\min}, 1]$ . Interestingly, alterations in the initial eigenvalue distribution only slightly affect estimation quality in the beginning of training (Fig. 2.C). The key factor seems to be the degradation associated with learning progress, rather than degradation due to larger eigenvalues. Smaller initial  $|\lambda|_{\min}$  values slow down training, as more perfect memory neurons have to be recruited, but all initializations eventually lead to solving the task (Fig. 2.D).
4.2 Independent recurrent modules improve online learning performance
After dissecting the learning dynamics of our algorithm, we next show the importance of the independence of recurrent modules for online gradient-based learning. To this end, we compare linear recurrent units to densely-connected linear recurrent layers which do not have independent recurrent modules. To make the comparison fair, we ensure all models have the same number of parameters and recurrent neurons. In addition to the baselines we considered in the previous section, we include an extended version of the SnAp-1 algorithm that combines spatially backpropagated errors with the SnAp-1 sensitivity approximation. This algorithm reduces to ours when applied to networks of independent recurrent modules. Therefore, it enables us to isolate the impact of the independent recurrent module design on online gradient-based learning. We report final training losses in Table 1 and refer the reader to Appendix B for experimental details.
Table 1: Comparison of final training losses of different online learning algorithms on the copy task of Section 4.2. The independent recurrent modules design improves online learning performance. Performance greatly degrades when the LRU is replaced with a dense recurrent matrix (Linear RNN). Comparison with the SnAp-1 algorithm applied to the GRU architecture highlights that online learning of multilayer networks is difficult without element-wise recurrence. Results are averaged over 5 seeds.
Layer Number layers	LRU 4	Linear RNN 4	GRU 1	GRU 4
SPATIAL	4.66 × 10-1	6.20 × 10-1	6.26 × 10-1	6.55 × 10-1
TRUNCATED	2.62 × 10-1	5.81 × 10-1	6.20 × 10-1	6.49 × 10-1
OURS/SNAP-1	8.44 × 10-3	5.82 × 10-1	3.16 × 10-1	3.27 × 10-1
BPTT	7.59 × 10-6	1.07 × 10-4	2.61 × 10-1	1.94 × 10-1
Table 2: Test accuracy on three tasks of the LRA benchmark [24] for spatial backpropagation, 1-step truncated backpropagation, our algorithm, and full backpropagation through-time. While we always use per time step local losses during training, we accumulate logits over the sequence during inference. We report the mean and std. for three seeds each.
Method	SCIFAR	IMDB	LISTOPS
SPATIAL	58.20 ± 0.70	83.50 ± 0.20	32.02 ± 0.27
TRUNC.	60.01 ± 1.26	84.04 ± 0.47	31.88 ± 0.59
OURS	79.59 ± 1.01	86.48 ± 0.41	37.62 ± 0.68
BPTT	83.40 ± 1.54	87.69 ± 0.39	39.75 ± 0.17
Table 3: Test accuracy of a linear RNN on the CIFAR task. Instead of our learning rule, we apply the SnAp-1 learning rule extended to the multilayer case, as described in Section 4.2.
SCIFAR
50.63 ± 0.23
50.53 ± 0.43
63.71 ± 0.33
65.23 ± 0.56
Our findings confirm the benefits of independent recurrent modules for online learning, in particular for multi-layer networks. To demonstrate that, we first compare our algorithm on the LRU architecture with the SnAp-1 algorithm applied to an equivalent linear recurrent network. The diagonal approximation of the sensitivity tensor in SnAp-1 introduces an additional bias when learning linear RNNs. We found that this additional bias hurts performance: when moving from offline BPTT to online training, the performance drop is significantly higher for linear RNNs. Interestingly, sensitivity approximation does not bring any performance gain, in this setting, compared to the cruder approximations that are 1-step truncated BPTT and spatial backpropagation.
Additionally, we run experiments on another RNN architecture, the GRU [44], to better understand the impact of depth in online and offline RNN training, and to confirm the importance of element-wise recurrence for online learning. In the single layer case, consistent with Menick et al. [15], we find that the SnAp-1 approximation performs competitively with offline BPTT. However, it suffers from depth in contrast to BPTT that benefits from it. This result highlights the importance of depth in this memory task, as well as the difficulty learning over depth poses for existing online learning algorithms.
4.3 Scaling online learning to the long-range arena benchmark
While the approximations typically employed to enable online learning prohibit scaling to tasks with extended temporal structure, the results from our previous section have demonstrated the potential of independent linear units for online learning of long-range dependencies. We therefore move to tasks from the challenging long-range arena benchmark [24] specifically designed to evaluate this ability. Transformers excel in almost any benchmark today. However, they perform surprisingly subpar in this setting [45] in which deep state-space models [20] and LRUs [23] achieve impressive results.
We run experiments on three tasks, sequential CIFAR, IMDB and ListOps. In sequential CIFAR, the network receives the  $32 \times 32$  image as a pixel sequence and has to perform a classification task. In line with Orvieto et al. [23], we use the colored version of sCIFAR instead of the grayscale version
originally proposed. In the IMDB task, the network is given a text encode in bytes of length at most 4000, and has to perform binary classification. In LISTOPS, the input is a sequence of numbers, brackets and operators like MAX which the model needs to evaluate to determine a classification target in the range from 1 to 10. We do not use the three other tasks of the LRA benchmark: the performance gap between different models is usually small in the RETRIEVAL task (c.f. [23]) and, in our preliminary experiments, we could not reach above chance performance in the PATHFINDER tasks with BPTT and the modifications we made to make the loss causal, as described in the next paragraph.
In order to make the sequence models employed on this benchmark compatible with the causality requirement in online learning, we remove the time pooling operation during training and consider a local loss term at every time step instead. During inference, we then evaluate our models using the average of the last layer logits in time which respects causality. Moreover, we replace batch normalization with layer normalization to avoid sending batch statistics backwards in time and consider smaller models to lower the computational burden for online learning. For further experimental details, please refer to Appendix B.
We report results comparing online learning to spatial backpropagation, truncated BPTT and the BPTT upper bound in Table 2. Our online learning algorithm outperforms other online learning approximations, significantly reducing the gap towards BPTT. As in the last section, replacing the LRU with a linear RNN layer in the CIFAR experiment leads to worth online learning performance, c.f. Table 3, providing further evidence for the effectiveness of independent recurrent modules for capturing long-term dependencies.
We have demonstrated that long-range dependencies can be learned online, allowing recurrent neural networks to reach strong performance on a set of tasks from the long-range arena benchmark. Moreover, a detailed analysis of a memory problem revealed that our method significantly outperforms both spatial (online) backpropagation as well as prior approaches based on approximate real-time recurrent learning, coming close to full backpropagation-through-time. These findings may inform the design of new neuromorphic hardware with on-chip learning capabilities, an application where approximate real-time recurrent learning is garnering significant attention [46-48].
While most prior related work focused on developing generic gradient approximation schemes, we asked which architecture would simplify online gradient computations. In high-level terms, our philosophy draws from seminal work on long short-term memory networks [LSTMs; 40] or neural Turing machines [49], which established the importance of architecture design for the success of gradient descent. We build on this insight, moving to the harder setting of online learning. This led us to consider networks built of recurrent independent modules: decoupled units with low-dimensional state vectors, for which exact real-time recurrent learning is cheap. Importantly, this design underlies recent models such as deep linear recurrent units [23] and members of the HiPPO family [21, 22] which achieve strong performance in a wide array of challenging problems, including language modeling at scale [50] and the long-range arena benchmark [22].
We conclude by noting that modularity, the overarching principle behind our approach, is at the very heart of the influential columnar hypothesis in neuroscience [51]. This hypothesis states that the architecture of the neocortex is modular, with the cortical column as an elementary (or canonical, [52]) building block one level of abstraction above neurons. We thus speculate that modularity could be a key neural network design principle discovered by evolution, that considerably simplifies the temporal credit assignment problem. This is inline with our finding that a modular architecture enables learning complicated temporal dependencies through simple local temporal credit assignment mechanisms, letting spatial backpropagation take care of assigning credit over the network hierarchy. We stress this point because numerous biological implementations and alternatives for spatial backpropagation have been proposed [e.g., 53-62], while essentially none exist yet for backpropagation-through-time [63]. Our findings provide a starting point for understanding how the brain deals with the fundamental problem of learning the temporal structure behind its sensory inputs.
All five authors thank Angelika Steger for her support and overarching guidance. João Sacramento thanks Timothy Lillicrap and Greg Wayne for an inspiring early discussion on searching architectures that can learn through limited forms of backpropagation-through-time, and Owen He and Johannes von Oswald for valuable discussions on deep state space models. This research was supported by an Ambizione grant (PZ00P3_186027) from the Swiss National Science Foundation and an ETH Research Grant (ETH-23 21-1). Robert Meier and Asier Mujika were supported by grant no. CRSII5_173721 of the Swiss National Science Foundation.
[1] Blake A. Richards, Timothy P. Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lindsay, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiotia Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording. A deep learning framework for neuroscience. Nature Neuroscience, 22(11): 1761-1770, 2019.
[2] David E. Rumelhart and James L. McClelland. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, pages 318-362. 1986.
[3] Seppo Linnainmaa. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2):146-160, 1976.
[4] Paul Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550-1560, 1990.
[5] Charlotte Frenkel, David Bol, and Giacomo Indiveri. Bottom-up and top-down neural processing systems design: neuromorphic intelligence as the convergence of natural and artificial intelligence. arXiv preprint arXiv:2106.01288, 2021.
[6] Surya Ganguli, Dongsung Huh, and Haim Sompolinsky. Memory traces in dynamical systems. Proceedings of the National Academy of Sciences, 105(48):18970-18975, 2008.
[7] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-280, 1989.
[8] R. E. Wengert. A simple automatic derivative evaluation program. Communications of the ACM, 7(8):463-464, 1964.
[9] Coretin Tallec and Yann Ollivier. Unbiased online recurrent optimization. In International Conference on Learning Representations, 2018.
[10] Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random Kronecker factors. In Advances in Neural Information Processing Systems, volume 31, 2018.
[11] Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal Kronecker-sum approximation of real time recurrent learning. In International Conference on Machine Learning, 2019.
[12] James M. Murray. Local online learning in recurrent networks with random feedback. eLife, 8: e43299, 2019.
[13] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11(1):3625, 2020.
[53] Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2015.
[54] Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1):13276, 2016.
[55] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11, 2017.
[56] Pieter R. Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 19(3):166-180, 2018.
[57] James C. R. Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in Cognitive Sciences, 23(3):235-250, 2019.
[58] Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
[59] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature Neuroscience, 24(7):1010-1019, 2021.
[60] Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi, Johannes von Oswald, and João Sacramento. The least-control principle for local learning at equilibrium. In Advances in Neural Information Processing Systems, volume 35, 2022.
[61] Axel Laborieux and Friedemann Zenke. Holomorphic equilibrium propagation computes exact gradients through finite size oscillations. In Advances in Neural Information Processing Systems, volume 35, 2022.
[62] Walter Senn, Dominik Dold, Akos F. Kungl, Benjamin Ellenberger, Jakob Jordan, Yoshua Bengio, João Sacramento, and Mihai A. Petrovici. A neuronal least-action principle for real-time learning in cortical circuits. bioRxiv 2023.03.25.534198, 2023.
[63] Timothy P. Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current Opinion in Neurobiology, 55:82-89, 2019.
A Network architecture and algorithm
A.1 The linear recurrent unit (LRU)
In Section 2.2, we used a simplified version of the LRU for brevity. Here, we review the more detailed formulation of Orvieto et al. [23]. The LRU is defined as follows:
$$
h _ {t + 1} = \lambda \odot h _ {t} + \gamma \odot B x _ {t} \tag {8}
$$
$$
y _ {t} = \operatorname {R e} \left[ C h _ {t} \right] + D x _ {t}. \tag {9}
$$
The only difference with Equation 1 is the inclusion of the normalization factor  $\gamma \in \mathbb{R}^N$ . Its purpose is to ensure that all units maintain a comparable magnitude. Without normalization, hidden states whose  $\lambda$  is close to 1 can blow up. We initialize  $\gamma$  with
$$
\gamma = \sqrt {1 - | \lambda | ^ {2}}, \tag {10}
$$
and later update it with our learning algorithm.
We use an exponential parametrization for  $\lambda$ :
$$
\lambda := \exp \left(- \exp \left(\nu^ {\log}\right) + i \exp \left(\theta^ {\log}\right)\right). \tag {11}
$$
This parametrization ensures that the norm of  $\lambda$ , equal to  $\exp(-\exp(\nu^{\log}))$ , remains below 1, guaranteeing stability of the network dynamics. This feature provides an advantage to the LRU compared to the linear RNN, as observed in Section 4.2. By representing  $\theta$  as  $\exp(\theta^{\log})$ , we achieve finer tuning of  $\theta$  around 0. In the following, we focus on computing gradients with respect to  $\lambda$ . However, in our simulations, we derive the update for  $\nu^{\log}$  and  $\theta^{\log}$  from these gradients through the chain rule, and apply them to update the corresponding parameters. Similarly, we optimize the logarithm of  $\gamma$ .
A.2 Complete derivation of our algorithm
We provide a more detailed derivation of our online learning rule for a single layer of LRUs, as described in Section 3.1. Recall that we define  $e_t$  to be the non-zero terms of the sensitivity  $\mathrm{d}_{\theta} h_t$  of the state  $h_t$  with respect to the parameters  $\theta$ , with  $\theta = \{\lambda, \gamma, B\}$  the parameters of the LRU impacting the recurrent dynamics. We show that the sensitivities evolve according to
$$
e _ {t + 1} ^ {\lambda} = \lambda \odot e _ {t} ^ {\lambda} + h _ {t} \tag {12}
$$
$$
e _ {t + 1} ^ {\gamma} = \lambda \odot e _ {t} ^ {\gamma} + B x _ {t + 1} \tag {13}
$$
$$
e _ {t + 1} ^ {B} = \operatorname {d i a g} (\lambda) e _ {t} ^ {B} + \gamma x _ {t + 1} ^ {\top} \tag {14}
$$
and that gradient-following parameter updates can be calculated through
$$
\Delta \lambda \propto \sum_ {t = 1} ^ {T} \delta_ {t} \odot e _ {t} ^ {\lambda} \tag {15}
$$
$$
\Delta \gamma \propto \sum_ {t = 1} ^ {T} \operatorname {R e} \left[ \delta_ {t} \odot e _ {t} ^ {\gamma} \right] \tag {16}
$$
$$
\Delta B \propto \sum_ {t = 1} ^ {T} \operatorname {d i a g} \left(\delta_ {t}\right) e _ {t} ^ {B}, \tag {17}
$$
with  $\delta_t = \mathrm{d}_{h_t}L_t$ . These updates are the ones used in our simulations. We now derive the updates per coordinate, and the vectorized form can be straightforwardly obtained from there.
Update for  $\lambda$ . We have
$$
e _ {t + 1, i} ^ {\lambda} = \frac {\mathrm {d} h _ {t + 1 , i}}{\mathrm {d} \lambda_ {i}} = \frac {\mathrm {d}}{\mathrm {d} \lambda_ {i}} \left[ \lambda_ {i} h _ {t, i} \right] + 0 = \lambda_ {i} e _ {t, i} ^ {\lambda} + h _ {t, i}. \tag {18}
$$