Title: Online learning of long-range dependencies
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning

linear recurrent units

temporal credit assignment

biologically-plausible learning

local learning rules

neuromorphic computing
Abstract: Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation by either not being scalable or failing to learn long-range dependencies. Here we present a high-performance online learning algorithm that merely doubles the memory and computational requirements of a single inference pass. We achieve this by leveraging independent recurrent modules in multi-layer networks, an architectural motif that has recently been shown to be particularly powerful. Experiments on synthetic memory problems and on the challenging long-range arena benchmark suite reveal that our algorithm performs competitively, establishing a new standard for what can be achieved through online learning. This ability to learn long-range dependencies offers a new perspective on learning in the brain and opens a promising avenue in neuromorphic computing.

Online learning of long-range dependencies

Abstract

Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation by either not being scalable or failing to learn long-range dependencies. Here we present a high-performance online learning algorithm that merely doubles the memory and computational requirements of a single inference pass. We achieve this by leveraging independent recurrent modules in multi-layer networks, an architectural motif that has recently been shown to be particularly powerful. Experiments on synthetic memory problems and on the challenging long-range arena benchmark suite reveal that our algorithm performs competitively, establishing a new standard for what can be achieved through online learning. This ability to learn long-range dependencies offers a new perspective on learning in the brain and opens a promising avenue in neuromorphic computing.

1 Introduction

How can the connections between neurons in a neural network be adjusted to improve behavior? This question, known as the credit assignment problem, is central in both neuroscience [1] and machine learning [2], owing to its fundamental importance for elucidating learning mechanisms in the brain and constructing intelligent artificial systems. However, the complex and nonlinear nature of neural network processing makes the precise allocation of credit an intricate task.

Deep learning provides a compelling solution to the credit assignment problem via gradient descent, which refines network parameters along the locally most promising direction. For networks processing temporal sequences, gradient computation is made possible by backpropagation-through-time [BPTT; 2-4]. BPTT stores and revisits neural activity in reverse-time order to understand how infinitesimal changes to neural activity, and thus to network parameters, would have impacted the objective function. One important drawback of this algorithm is its requirement to store the entire activity trajectory in memory, which constrains the sequence length for exact gradient computation, impairing the learning of long-term interactions. This constraint becomes a critical bottleneck when working within memory-limited systems, such as neuromorphic hardware [5] and presumably the brain [6].

Alternatives to BPTT for gradient computation do exist. One such approach, forward-mode differentiation [7, 8], involves computing gradients online as the input sequence is processed, by keeping track of the sensitivity of neural activity with respect to each of the network parameters. This marks a qualitative departure from BPTT, as it prepares for all potential future trajectories simultaneously; by contrast, BPTT focuses on improving the activity of a past trajectory. Importantly, the memory footprint of this approach does not depend on sequence length. Still, it remains intractable for real-world applications and likely infeasible in the brain due to its cubic memory scaling and quartic computational complexity in the number of neurons. Recent research focused on approximation

strategies to make online gradient estimation more tractable for general-purpose recurrent networks [9-18]. Our work takes a fundamentally different approach: instead of tailoring the learning algorithm to the neural network architecture, we fix the learning algorithm and seek an architecture that makes it tractable.

We build upon recent advances in linear state space models, a class of recurrent neural networks (RNNs) employing linear recurrent blocks [19-23]. These blocks are stacked and interconnected through nonlinear networks. The key insights from this line of research are that linear recurrent connections simplify temporal credit assignment and enable parallel temporal processing, while nonlinearities between recurrent blocks ensure that network expressiveness remains comparable to that of densely-connected nonlinear RNNs. Much, if not all, of the state-of-the-art performance of those models on long-range temporal tasks [24] can be maintained by transitioning from real-valued to complex-valued neural activities [21-23], and restricting the recurrent connectivity matrix to be diagonal. Recurrent neurons within a given layer are now independent of each other. This greatly improves the tractability of online gradient estimation, as the recurrent parameters of a given neuron do not impact other neurons. We leverage this property to achieve exact online gradient computation within a single layer with as little as twice the memory and compute requirements needed for inference. Further, we demonstrate how this leads to improved gradient estimation compared to existing online learning algorithms when recurrent layers are stacked.

This paper is organized as follows. We start by briefly reviewing existing gradient-based online learning methods in Section 2.1. Next, we introduce the concept of independent recurrent modules, showing how some of the recent high-performance models mentioned above fit in this framework in Section 2.2. Deriving our learning rule requires complex differentiation; we give a concise overview of those tools in Section 2.3. In Section 3, we detail our online learning algorithm that combines exact differentiation within a layer of recurrent independent modules with spatial backpropagation across layers. Finally, in Section 4, we analyze our algorithm and relevant baselines on a synthetic copy task and show that it can learn sequential tasks with sequence lengths up to over 4000 steps.

2 Background

2.1 Online gradient-based RNN learning

We study gradient-based learning of recurrent neural networks, which process input data x 1 , . . . , x T sequentially while maintaining an internal (hidden) state h t . The objective of learning is to minimize a cumulative loss L ( θ ) = ∑ T t =1 L t ( θ ) which measures performance on a task at hand as a function of network parameters θ . The standard algorithm for computing the gradient ∇ L ( θ ) is the offline backpropagation-through-time method, which requires storing the entire input x 1: T , loss L 1: T and internal activity h 1: T sequences, and then revisiting them proceeding backwards in time. Here, we focus on online algorithms which carry the information needed to compute or estimate ∇ L t ( θ ) forward in time. This enables simultaneous processing of inputs and learning for RNNs, without storing past data and network states. In principle, online algorithms can learn arbitrarily long temporal dependencies as well as seamlessly handle sequences of arbitrary length T .

The classical alternative to BPTT for forward-in-time gradient computation is known as real-time recurrent learning [RTRL; 7] in the context of RNNs 2 , a method which has its roots in control theory [26]. While RTRL enables online gradient-based learning, it requires storing d θ h t in memory and updating it as the RNN processes its inputs. The size of this auxiliary variable is O ( n 3 ) , where n = | h | is the number of hidden units in the RNN. For comparison, the memory requirements of BPTT are O ( nT ) . In practice, this precludes the usage of RTRL for all but the smallest of models.

There has been much effort in developing memory-efficient alternatives to RTRL. We now briefly discuss these prior efforts while referring to a recent review by Marschall et al. [14] for a more detailed treatment. We divide prior work into three broad categories. One class of algorithms relies on neglecting terms in d θ h t to reduce its size, thereby creating a biased gradient estimator. Typically this requires introducing crude approximations, which allow going from O ( n 3 ) to a tractable O ( n 2 ) size. Despite such approximations, in many cases, performance still holds in non-trivial tasks [13, 15, 16]. At the end of this spectrum sits instantaneous (spatial) backpropagation, which neglects all temporal

dependencies in the hidden state when approximating the gradient. A second class of algorithms relies on stochastic estimation; this allows retaining unbiased gradient estimates, at the expense of introducing variance [9-11, 17, 18]. Finally, a third class of methods introduces gradient models [critics; 27-29] to produce gradient estimates online. The critics themselves are then either trained separately offline, making such methods hybrid on/offline; or fully online, using temporal difference techniques [30]. We note that despite their widespread use in reinforcement learning, it is not yet well understood whether temporal difference methods can reliably and efficiently improve the performance of a gradient critic on real-world RNN learning problems.

As noted by Irie et al. [31], the RTRL literature mostly focuses on single-layer recurrent networks and remains scarce for deeper networks. Recently, Javed et al. [32] developed a greedy learning algorithm where a growing network is progressively frozen and trained one layer at a time. Existing approximations such as [13, 15] do not prescribe how to learn the parameters of remote layers, and the multi-layer case is not considered in the respective papers. Introducing a powerful RTRL algorithm that scales to networks of multiple layers is the main algorithmic contribution of this paper. This property is of great empirical relevance given the power of depth to learn the temporal structure [e.g. 19, 23].

2.2 Linear recurrent units and independent recurrent modules

Instead of developing approximate, general-purpose forward-mode differentiation methods, our goal shifts towards seeking an expressive architecture allowing exact, tractable online gradient calculation. We propose that networks with linear recurrent units [LRU; 23] and, more generally networks with independent recurrent modules, are particularly well-suited for this purpose.

A linear recurrent unit, depicted in Figure 1, is defined as

with ⊙ the element-wise product. Here, x t ∈ R H represents the input received by the LRU at time t , h t ∈ C N denotes its internal state, and y t ∈ R H its output. The parameters of the unit include λ ∈ C N , B ∈ C N × H , C ∈ C H × N and D ∈ R H × H . The version of the LRU we use in our experiments includes an element-wise normalization factor for the input Bx and uses an exponential parametrization of λ for network stability. We omit these details in the main text for conciseness; see Appendix A.1 for more details.

LRUs differ from traditional recurrent layers in deep learning: they have linear neural dynamics, a complex-valued state h t , and a diagonal connectivity pattern. Orvieto et al. [23] found that the absence of temporal linearity in networks that stack those units through nonlinear connections (see Fig. 1) does not alter expressivity and eases gradient-based learning, a notoriously difficult process for nonlinear RNNs [33, 34]. In addition, the diagonal structure of the recurrence matrix provides several benefits over fully connected ones. First, it affords an easy way to control the eigenvalues of the Jacobian of the system and thus ensure that neural dynamics remain stable. Due to the linearity, it also enables processing the input sequence in parallel [35], significantly accelerating the training of such models on modern computers [22]. Importantly, despite its diagonal parametrization, the LRU remains functionally equivalent to a linear recurrent layer with dense recurrence matrix A , as A can be approximated as accurately as needed by a complex-diagonalizable matrix.

Each complex neuron in an LRU is an independent recurrent module , meaning its current state does not impact the dynamics of other modules. This property greatly simplifies online credit assignment (see Section 3). We focus on this specific architecture due to its simplicity and great empirical performance, but our theoretical insights also apply to networks of independent recurrent modules with low-dimensional state vectors per module.

2.3 A primer on complex differentiation

The use of complex-valued networks, such as the LRU, and hence complex differentiation remains relatively scarce. In the following, we provide a concise review of the tools of complex differentiation integral to the derivation of our online learning rule. We use f and g to denote complex-valued functions that take the complex variable z as input.

linear

encoder

layer norm

IRM

independent

recurrent

modules

skip connection

# classes

GLU

linear

decoder

time

time

layer

layer

spatial processing

temporal processing

neural activities

backpropagated errors

neural activities

state sensitivities

independent recurrent module

Example of IRM:

the linear recurrent unit (LRU)

x number of layers

The Wirtinger derivatives of f are defined through

Using them for complex differentiation allows using similar calculus rules as for real functions. Note that we use the row convention for derivatives, that is d z f is a row vector of size | z | . The following formula holds in general d z f = d ¯ z ¯ f .

The complex derivative of a complex function is similar to a 2 × 2 real-valued matrix as both d z f ∈ C and d ¯ z f ∈ C are necessary to characterize it. Yet, there exists a subclass of functions, called holomorphic functions for which it can be reduced to a 2 dimensional real-valued vector, leading to a more compact representation of derivatives. A continuous function f is holomorphic if it satisfies the Cauchy-Riemann equations

Any affine function, as well as the composition of two holomorphic functions, are themselves holomorphic.

The chain rule of complex differentiation, crucial for automatic differentiation, is

When either f or g is holomorphic, the second term vanishes as d g f = 0 or d z ¯ g = d z g = 0 . When f is a real-valued function, it can be optimized through gradient descent by iteratively updating its input variable z through ∆ z ∝ -d Re[ z ] f ⊤ -i d Im[ z ] f ⊤ = -2d ¯ z f ⊤ .

3 Online learning of networks of independent recurrent modules

Based on the foundations laid down in the preceding section we now derive our online gradient-based learning algorithm for multi-layer networks of independent recurrent modules. We first focus on a single layer, demonstrating that exact forward-mode differentiation is tractable. This insight then guides the derivation of our rule for multi-layer networks.

3.1 Single-layer networks

We focus on parameters θ that influence the hidden states; computing the gradient of any other parameter does not require temporal credit assignment. For the LRU, we have θ = { λ, B } . Recall that L ( θ ) = ∑ T t =1 L t ( y t ( θ )) denotes the loss function that measures how good the outputs y 1: T of a network parametrized by θ are. Its derivative d θ L ( θ ) can be calculated using forward-mode (complex) differentiation:

As mentioned in Section 2.3, the last equality holds as h t +1 is a holomorphic function of h t and of θ , hence h t is a holomorphic function of θ by recursive composition, and L only directly depends on h t through L t . The term δ t := d h t L ⊤ that is here equal to d h t L ⊤ t can easily be computed by spatial backpropagation, as the output y t at time t only depends on the current hidden state h t . We are left with computing the sensitivities d θ h t of the states to the parameters.

Independent recurrent modules do not influence each other. The parameters θ i that directly influence the state h t,i of module i never impact the state h t ′ ,j of another module. As a consequence, the number of non-zero entries of the sensitivity d θ h t grows linearly with the size of θ whenever the number of recurrent neurons within each module is fixed. Applying this to the LRU, d θ h t is entirely characterized by e λ t := (d λ i h t,i ) i and e B t := (d B ji h t,j ) ji . Differentiating Equation 1 using the product rule gives the corresponding updates:

with 1 a vector of size | h | filled with ones. More detail on the derivation of Equation 6 and on how to efficiently simulate this update are given in Appendix A.2. Keeping track of those quantities only requires considering an additional hidden state of size | θ | . 3 Finally, the λ and B updates can be obtained by following the gradient, as calculated in Equation 5:

Interestingly, all the e -updates are local to the neuron or synapse in question, and no approximations were required to accomplish this. This feature makes the algorithm particularly promising for neuroscience and neuromorphic engineering, where localized computation is highly desirable. The parameter update for λ and B is also fully local, as it combines a parameter-specific sensitivity, sometimes considered as an eligibility trace [13], and a postsynaptic error term.

The idea that element-wise recurrence simplifies RTRL precedes our work. It can be found in early work by Mozer [38] and Gori et al. [39], and has been revisited recently [32, 31]. In this paper, we extend this insight to complex numbers and thus do not lose expressivity, unlike previous work. We also note that some approximations to RTRL such as e-prop [13] or SnAp-1 [15] end up being exact when applied to networks with independent recurrent modules.

3.2 Multi-layer networks

The derivation in the last section presumes that the loss L only directly depends on h t through L t . This assumption no longer holds when layers are stacked, which is crucial to the expressiveness of the model. In the following, we explain how we can extend our rule to the multilayer case. Let us consider layer l of the network where we aim to compute the gradient of the loss L with respect to its parameters θ l . The sensitivity d θ l h l t can be computed as before, assuming independent recurrent modules, as θ l does not influence the behavior of the inputs it receives from previous layers. Hence, we are left with computing δ l t = d h l t L ⊤ . The simplification δ l t = d h l t L ⊤ t we made in the previous section still holds for the last layer but is violated for the other layers. This is because L , taken as function of the hidden states h l of layer l , now has an internal memory through the subsequent recurrent layers. The hidden state h l t at time t will thus directly affect all future losses L t ′ for t ′ ≥ t .

As a consequence, one has to resort to backpropagation-through-time to compute d h l t L ⊤ exactly, which breaks causality and rules out the possibility of learning online. To circumvent this issue, we approximate the error signal each layer receives by δ l t ≈ d h l t L ⊤ t so that it can be computed instantaneously with spatial backpropagation. We emphasize that the only source of approximation of this algorithm is the one above. Given that there is no approximation for the last layer, we will always compute the exact gradient for that layer.

We summarize our learning rule in Figure 1. It prescribes augmenting the hidden state of each recurrent layer l with the sensitivity e l . For each input/output sample ( x t , y target t ) , we first update the full entire hidden state ( h t , e t ) using the previous one ( h t -1 , e t -1 ) and current input x t . We then spatially backpropagate the error signal obtained at the last layer by comparing the prediction of the network to its desired value y target t . Finally, we combine e l t and δ l t available at each layer using Equation 7 to compute the current update. So far, we have only described how to update parameters that directly influence the hidden neurons of the recurrent layer. We update the rest of the parameters with the gradient estimate obtained with spatial backpropagation. Importantly, the size of the extended hidden state is upper bounded by the number of neurons plus the number of parameters and is in practice much smaller.

Next, we aim to understand the factors that may degrade the quality of the gradient estimate as a result of the approximation δ l t ≈ d h t L ⊤ t which introduces bias in the gradient estimate. In the LRU, neural activities, and thus error signals, are processed temporally through dynamics similar to the one of Equation 1. When the norm of each λ i approaches one, neural activity preserves past information, and correspondingly, error signals backpropagated over time contain more information about the future. This suggests that our approximation becomes less accurate as the distribution of | λ i | narrows around 1, since it discards future error information. Moreover, δ worsens as it is backpropagated through more layers. At each layer, backpropagation mixes errors from the next layer and the future state of the layer. Since we neglect future information, only part of the error signal is backpropagated, resulting in a less accurate approximation. We delve deeper into these two approximation sources in a memory task in Section 4.1.

4 Experiments

In the following, we analyze the properties of our proposed online learning method empirically and explore how independent recurrent modules aid learning of long-range dependencies. To this end, we first conduct an extensive analysis on the copy task [40], a well-established test bed to study temporal credit assignment in recurrent neural networks. Comparisons to truncated online versions of BPTT and to an extension of SnAp-1 to deep recurrent networks, reveal that independent recurrent modules are generally beneficial for learning long-range dependencies, as they excel in both the online and the offline setting. Finally, we evaluate our method on three tasks of the Long Range Arena [24]: a sequential version of CIFAR [41], LISTOPS [42] and IMDB [43], scaling online learning to sequence lengths of over 4000 time steps and to deep recurrent neural networks. For additional experimental details and hyperparameter configurations, we refer to Appendix B.

4.1 Understanding the approximations behind online learning in networks of LRUs

In this first series of experiments, we investigate the approximation introduced by our online algorithm in detail. We recall that our learning rule only approximates the full gradient when the network has two recurrent layers or more, as it ignores the temporal component of backpropagated error signals. Therefore, we expect that the learning signal becomes less accurate as we increase network depth and shift the eigenvalue distribution towards a norm of 1 . To explore the effects of our approximation in a controlled setting, we consider a copy task [40, 10, 15, 18] in which the network observes a length20 sequence of 7 -bit patterns that it must recall when presented with an output token. Intuitively, temporal credit assignment is critical in this task to, among other things, convert forgetting neurons ( | λ |≪ 1 ) into perfect memory units ( | λ | ≈ 1 ) that can solve the task. To ensure that these perfect memory units are scarce at the beginning of training, necessitating learning to create some, we initialize λ uniformly at random in the complex unit disk and set the number of recurrent units per layer to N = 64 . The default architecture used in this section has four layers.

C

A

B

D

E

F

As remarked in Section 3.2, the updates prescribed by our learning rule match the gradient exactly for all parameters in the last LRU layer. We confirm that empirically for a network of one layer in Figure 2.A. While the approximation quality deteriorates with increasing depth (Fig. 2.A), alignment remains high, noticeably better than for all baseline online learning rules (Fig. 2.E). Moreover, despite alignment decreasing with depth, performance enhances significantly (Fig. 2.B). BPTT exhibits similar improvements with depth, suggesting that this is likely due to the enhanced capabilities of the model. Our rule can learn useful hierarchical temporal representations online whereas baseline methods, 1-step truncated and spatial backpropagation, which ignore most of temporal dependencies, fail (c.f. Fig. 2.F). Additionally, we found that, despite its bias, our learning rule can decrease the loss to 0 when training a 4 layers network on a simpler memory task for long enough.

Exact error terms are backpropagated through recurrent units by weighting the current prediction error by 1 and the future error by λ . In order to maintain an online algorithm, we ignore this dependency on the future. We expect alignment with the gradient to decrease as the distribution of | λ | shifts towards 1 . To test that, we initialize the λ uniformly at random in the complex ring of radius [ | λ | min , 1] . Interestingly, alterations in the initial eigenvalue distribution only slightly affect estimation quality in the beginning of training (Fig. 2.C). The key factor seems to be the degradation associated with learning progress, rather than degradation due to larger eigenvalues. Smaller initial | λ | min values slow down training, as more perfect memory neurons have to be recruited, but all initializations eventually lead to solving the task (Fig. 2.D).

4.2 Independent recurrent modules improve online learning performance

After dissecting the learning dynamics of our algorithm, we next show the importance of the independence of recurrent modules for online gradient-based learning. To this end, we compare linear recurrent units to densely-connected linear recurrent layers which do not have independent recurrent modules. To make the comparison fair, we ensure all models have the same number of parameters and recurrent neurons. In addition to the baselines we considered in the previous section, we include an extended version of the SnAp-1 algorithm that combines spatially backpropagated errors with the SnAp-1 sensitivity approximation. This algorithm reduces to ours when applied to networks of independent recurrent modules. Therefore, it enables us to isolate the impact of the independent recurrent module design on online gradient-based learning. We report final training losses in Table 1 and refer the reader to Appendix B for experimental details.

Our findings confirm the benefits of independent recurrent modules for online learning, in particular for multi-layer networks. To demonstrate that, we first compare our algorithm on the LRU architecture with the SnAp-1 algorithm applied to an equivalent linear recurrent network. The diagonal approximation of the sensitivity tensor in SnAp-1 introduces an additional bias when learning linear RNNs. We found that this additional bias hurts performance: when moving from offline BPTT to online training, the performance drop is significantly higher for linear RNNs. Interestingly, sensitivity approximation does not bring any performance gain, in this setting, compared to the cruder approximations that are 1-step truncated BPTT and spatial backpropagation.

Additionally, we run experiments on another RNN architecture, the GRU [44], to better understand the impact of depth in online and offline RNN training, and to confirm the importance of element-wise recurrence for online learning. In the single layer case, consistent with Menick et al. [15], we find that the SnAp-1 approximation performs competitively with offline BPTT. However, it suffers from depth in contrast to BPTT that benefits from it. This result highlights the importance of depth in this memory task, as well as the difficulty learning over depth poses for existing online learning algorithms.

4.3 Scaling online learning to the long-range arena benchmark

While the approximations typically employed to enable online learning prohibit scaling to tasks with extended temporal structure, the results from our previous section have demonstrated the potential of independent linear units for online learning of long-range dependencies. We therefore move to tasks from the challenging long-range arena benchmark [24] specifically designed to evaluate this ability. Transformers excel in almost any benchmark today. However, they perform surprisingly subpar in this setting [45] in which deep state-space models [20] and LRUs [23] achieve impressive results.

We run experiments on three tasks, sequential CIFAR, IMDB and LISTOPS. In sequential CIFAR, the network receives the 32 × 32 image as a pixel sequence and has to perform a classification task. In line with Orvieto et al. [23], we use the colored version of SCIFAR instead of the grayscale version

originally proposed. In the IMDB task, the network is given a text encode in bytes of length at most 4000 , and has to perform binary classification. In LISTOPS, the input is a sequence of numbers, brackets and operators like MAX which the model needs to evaluate to determine a classification target in the range from 1 to 10 . We do not use the three other tasks of the LRA benchmark: the performance gap between different models is usually small in the RETRIEVAL task (c.f. [23]) and, in our preliminary experiments, we could not reach above chance performance in the PATHFINDER tasks with BPTT and the modifications we made to make the loss causal, as described in the next paragraph.

In order to make the sequence models employed on this benchmark compatible with the causality requirement in online learning, we remove the time pooling operation during training and consider a local loss term at every time step instead. During inference, we then evaluate our models using the average of the last layer logits in time which respects causality. Moreover, we replace batch normalization with layer normalization to avoid sending batch statistics backwards in time and consider smaller models to lower the computational burden for online learning. For further experimental details, please refer to Appendix B.

We report results comparing online learning to spatial backpropagation, truncated BPTT and the BPTT upper bound in Table 2. Our online learning algorithm outperforms other online learning approximations, significantly reducing the gap towards BPTT. As in the last section, replacing the LRU with a linear RNN layer in the CIFAR experiment leads to worth online learning performance, c.f. Table 3, providing further evidence for the effectiveness of independent recurrent modules for capturing long-term dependencies.

5 Discussion

We have demonstrated that long-range dependencies can be learned online, allowing recurrent neural networks to reach strong performance on a set of tasks from the long-range arena benchmark. Moreover, a detailed analysis of a memory problem revealed that our method significantly outperforms both spatial (online) backpropagation as well as prior approaches based on approximate real-time recurrent learning, coming close to full backpropagation-through-time. These findings may inform the design of new neuromorphic hardware with on-chip learning capabilities, an application where approximate real-time recurrent learning is garnering significant attention [46-48].

While most prior related work focused on developing generic gradient approximation schemes, we asked which architecture would simplify online gradient computations. In high-level terms, our philosophy draws from seminal work on long short-term memory networks [LSTMs; 40] or neural Turing machines [49], which established the importance of architecture design for the success of gradient descent. We build on this insight, moving to the harder setting of online learning. This led us to consider networks built of recurrent independent modules: decoupled units with low-dimensional state vectors, for which exact real-time recurrent learning is cheap. Importantly, this design underlies recent models such as deep linear recurrent units [23] and members of the HiPPO family [21, 22] which achieve strong performance in a wide array of challenging problems, including language modeling at scale [50] and the long-range arena benchmark [22].

We conclude by noting that modularity, the overarching principle behind our approach, is at the very heart of the influential columnar hypothesis in neuroscience [51]. This hypothesis states that the architecture of the neocortex is modular, with the cortical column as an elementary (or canonical, [52]) building block one level of abstraction above neurons. We thus speculate that modularity could be a key neural network design principle discovered by evolution, that considerably simplifies the temporal credit assignment problem. This is inline with our finding that a modular architecture enables learning complicated temporal dependencies through simple local temporal credit assignment mechanisms, letting spatial backpropagation take care of assigning credit over the network hierarchy. We stress this point because numerous biological implementations and alternatives for spatial backpropagation have been proposed [e.g., 53-62], while essentially none exist yet for backpropagation-through-time [63]. Our findings provide a starting point for understanding how the brain deals with the fundamental problem of learning the temporal structure behind its sensory inputs.

Acknowledgments and Disclosure of Funding

All five authors thank Angelika Steger for her support and overarching guidance. João Sacramento thanks Timothy Lillicrap and Greg Wayne for an inspiring early discussion on searching architectures that can learn through limited forms of backpropagation-through-time, and Owen He and Johannes von Oswald for valuable discussions on deep state space models. This research was supported by an Ambizione grant (PZ00P3_186027) from the Swiss National Science Foundation and an ETH Research Grant (ETH-23 21-1). Robert Meier and Asier Mujika were supported by grant no. CRSII5_173721 of the Swiss National Science Foundation.

References

Blake A. Richards, Timothy P. Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lindsay, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording. A deep learning framework for neuroscience. Nature Neuroscience , 22(11): 1761-1770, 2019.

David E. Rumelhart and James L. McClelland. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations , pages 318-362. 1986.

Seppo Linnainmaa. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics , 16(2):146-160, 1976.

Paul Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78(10):1550-1560, 1990.

Charlotte Frenkel, David Bol, and Giacomo Indiveri. Bottom-up and top-down neural processing systems design: neuromorphic intelligence as the convergence of natural and artificial intelligence. arXiv preprint arXiv:2106.01288 , 2021.

Surya Ganguli, Dongsung Huh, and Haim Sompolinsky. Memory traces in dynamical systems. Proceedings of the National Academy of Sciences , 105(48):18970-18975, 2008.

Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation , 1(2):270-280, 1989.

R. E. Wengert. A simple automatic derivative evaluation program. Communications of the ACM , 7(8):463-464, 1964.

Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. In International Conference on Learning Representations , 2018.

Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random Kronecker factors. In Advances in Neural Information Processing Systems , volume 31, 2018.

Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal Kronecker-sum approximation of real time recurrent learning. In International Conference on Machine Learning , 2019.

James M. Murray. Local online learning in recurrent networks with random feedback. eLife , 8: e43299, 2019.

Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications , 11(1):3625, 2020.

Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. The Journal of Machine Learning Research , 21(1):135:5320-135:5353, 2020.

Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, and Alex Graves. A practical sparse approximation for real time recurrent learning. In International Conference on Learning Representations , 2021.

Friedemann Zenke and Tim P. Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation , 33(4):899-925, 2021.

Atılım Güne¸ s Baydin, Barak A. Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation. arXiv preprint arXiv:2202.08587 , 2022.

David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt. Learning by directional gradient descent. In International Conference on Learning Representations , 2022.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems , volume 33, 2020.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations , 2022.

Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured states spaces. In Advances in Neural Information Processing Systems , volume 35, 2022.

Jimmy T.H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In International Conference on Learning Representations , 2023.

Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349 , 2023.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 , 2020.

Andreas Griewank and Andrea Walther. Evaluating Derivatives . Society for Industrial and Applied Mathematics, 2008.

L. McBride and K. Narendra. Optimization of time-varying systems. IEEE Transactions on Automatic Control , 10(3):289-294, 1965.

Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In International Conference on Machine Learning , 2017.

Gregory D. Wayne. Self-modeling neural systems . PhD thesis, Columbia University, 2013.

J. Schmidhuber. Recurrent networks adjusted by adaptive critics. In IEEE/INNS International Joint Conference on Neural Networks , volume 1, 1990.

Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning , 3:9-44, 1988.

Kazuki Irie, Anand Gopalakrishnan, and Jürgen Schmidhuber. Exploring the promise and limits of real-time recurrent learning. arXiv preprint arXiv:2305.19044 , 2023.

Khurram Javed, Haseeb Shah, Richard S Sutton, and Martha White. Scalable real-time recurrent learning using columnar-constructive networks. Journal of Machine Learning Research , 24, 2023.

Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5(2), 1994.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning , 2013.

Guy E. Blelloch. Prefix sums and their applications, 1990.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning , 2017.

Michael C Mozer. A focused backpropagation algorithm for temporal pattern recognition. Complex Systems , 3, 1989.

Marco Gori, Yoshua Bengio, and Renato De Mori. BPS: a learning algorithm for capturing the dynamic nature of speech. In International Joint Conference on Neural Networks , 1989.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9 (8):1735-1780, 1997.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.

Nikita Nangia and Samuel R Bowman. ListOps: A diagnostic dataset for latent tree learning. arXiv preprint arXiv:1804.06028 , 2018.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , 2011.

Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation , 2014.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35:16344-16359, 2022.

Friedemann Zenke and Emre O. Neftci. Brain-inspired learning on neuromorphic substrates. Proceedings of the IEEE , 109(5):935-950, May 2021.

Charlotte Frenkel and Giacomo Indiveri. ReckOn: a 28nm sub-mm2 task-agnostic spiking recurrent neural network processor enabling on-chip learning over second-long timescales. In 2022 IEEE International Solid- State Circuits Conference (ISSCC) , volume 65, February 2022.

Yigit Demirag, Charlotte Frenkel, Melika Payvand, and Giacomo Indiveri. Online training of spiking recurrent neural networks with phase-change memory synapses. arXiv preprint arXiv:2108.01804 , 2021.

Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. arXiv preprint arXiv:1410.5401 , 2014.

Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: towards language modeling with state space models. arXiv preprint arXiv:2212.14052 , 2023.

Vernon B. Mountcastle. Modality and topographic properties of single neurons of cat's somatic sensory cortex. Journal of Neurophysiology , 20(4):408-434, 1957.

Rodney J. Douglas, Kevan A.C. Martin, and David Whitteridge. A canonical microcircuit for neocortex. Neural Computation , 1(4):480-488, 1989.

Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , 2015.

Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications , 7(1):13276, 2016.

Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience , 11, 2017.

Pieter R. Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience , 19(3):166-180, 2018.

James C. R. Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in Cognitive Sciences , 23(3):235-250, 2019.

Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience , 21(6):335-346, 2020.

Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature Neuroscience , 24(7):1010-1019, 2021.

Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi, Johannes von Oswald, and João Sacramento. The least-control principle for local learning at equilibrium. In Advances in Neural Information Processing Systems , volume 35, 2022.

Axel Laborieux and Friedemann Zenke. Holomorphic equilibrium propagation computes exact gradients through finite size oscillations. In Advances in Neural Information Processing Systems , volume 35, 2022.

Walter Senn, Dominik Dold, Akos F. Kungl, Benjamin Ellenberger, Jakob Jordan, Yoshua Bengio, João Sacramento, and Mihai A. Petrovici. A neuronal least-action principle for realtime learning in cortical circuits. bioRxiv 2023.03.25.534198 , 2023.

Timothy P. Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current Opinion in Neurobiology , 55:82-89, 2019.

A Network architecture and algorithm

A.1 The linear recurrent unit (LRU)

In Section 2.2, we used a simplified version of the LRU for brevity. Here, we review the more detailed formulation of Orvieto et al. [23]. The LRU is defined as follows:

The only difference with Equation 1 is the inclusion of the normalization factor γ ∈ R N . Its purpose is to ensure that all units maintain a comparable magnitude. Without normalization, hidden states whose λ is close to 1 can blow up. We initialize γ with

and later update it with our learning algorithm.

We use an exponential parametrization for λ :

This parametrization ensures that the norm of λ , equal to exp( -exp( ν log )) , remains below 1, guaranteeing stability of the network dynamics. This feature provides an advantage to the LRU compared to the linear RNN, as observed in Section 4.2. By representing θ as exp( θ log ) , we achieve finer tuning of θ around 0. In the following, we focus on computing gradients with respect to λ . However, in our simulations, we derive the update for ν log and θ log from these gradients through the chain rule, and apply them to update the corresponding parameters. Similarly, we optimize the logarithm of γ .

A.2 Complete derivation of our algorithm

We provide a more detailed derivation of our online learning rule for a single layer of LRUs, as described in Section 3.1. Recall that we define e t to be the non-zero terms of the sensitivity d θ h t of the state h t with respect to the parameters θ , with θ = { λ, γ, B } the parameters of the LRU impacting the recurrent dynamics. We show that the sensitivities evolve according to

and that gradient-following parameter updates can be calculated through

with δ t = d h t L t . These updates are the ones used in our simulations. We now derive the updates per coordinate, and the vectorized form can be straightforwardly obtained from there.

Update for λ . We have

The first equality uses the definition of e λ t +1 , the second the recurrent update of h t as in Equation 8 and the fact that γ ⊙ Bx t +1 does not depend on λ , and the third is the complex product rule. The

gradient of the loss with respect to λ i is then equal to

In the first line, we applied forward-mode complex differentiation (combined with h t being an holomorphic function of λ ) as in Equation 5. In the second, we used that h i is independent of λ j for i = j .

̸

Update for γ . The derivation for the recurrent update of e γ being very similar to the one for λ , we omit it. The fact that γ is a real variable does not change anything to this derivation. However, this makes the gradient computation slightly different. One can act as if γ was a complex variable γ C i and get

Now, looking at the real part of the previous equation, we have

Note that the factor 2 will also indirectly appear in the update of λ , although it is not yet there in the gradient, as we have a real derivative here, when it is a complex one above.

Update for B . We have

and

Note that the update for e B is more general than the one of Equation 6 in the main text in which γ = 1 .

B Experimental details

We base our implementation on the S5 [22] code base 4 . All networks are trained with the AdamW optimizer, with a linear learning rate warm up, followed by a one-cycle cosine learning rate decay. Following common practice, we do not apply weight decay for λ , γ , and use a smaller learning rate for those parameters (global learning rate times a learning rate factor). By default, we initialize λ uniformly at random in the complex disk and θ uniformly at random in [0 , 2 π ] .

B.1 Copy task experimental details and hyperparameters

We report the hyperparameters we used and scanned over for the copy task in Tables 4 and 5. In Section 4.1, we take the default configuration reported in the LRU paper [23] for backpropagationthrough-time and apply it to the different methods we consider. We use 25 epochs, with a linear warmup of 5 epochs. We tuned the learning rate for each method independently in the comparison of Figure 2.E and F and in the one of Table 1. For the 1-layer GRU architecture, following Menick et al. [15], we add an extra readout layer: a 1-hidden layer MLP with hidden dimension 1072 processes the hidden state to generate the output.

B.2 Experimental details and hyperparameters for SCIFAR, IMDB and LISTOPS

For all experiments, we first ran a manual coarse-grained hyperparameter tuning to identify the most important parameters, and then ran the grid search described in 6. The final hyperparameters were each evaluated on 3 fresh seeds for the results reported in Table 2. The training time for our online learning rule on a single Nvidia RTX3090 GPU for SCIFAR, IMDB and LISTOPS was respectively 36 , 10 and 40 hours.

For the comparison with linear RNNs, we kept the number N of hidden neurons fixed compared to the one used the LRU (c.f. Table 6), but changed the model size H to 294 in order to obtain the same number of parameters (our Linear RNN has 1 , 068 , 086 parameters vs. 1 , 063 , 178 for the LRU). We used the same hyperparameter optimization scheme.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks at the problem of online learning in RNNs from a perspective of exact gradient computation: it looks for network architectures for which exact gradients can be computed. It uses the recently proposed LRU (Orvieto et. al., 2023) to show that with linear and diagonal structure in recurrent dependencies, one can tractably compute the exact gradients online for a single layer network. For multi-layer networks, this simplified structure combined with truncation of gradient signal (only considering current time step) results in an approximate gradient.\n\nThe paper performs two sets of experiments. The first set examines the difference in the approximate gradient computed using the proposed method with the exact gradient. The second set of experiments focuses on learning tasks that require capturing long range dependencies. In both set of experiments the proposed method closes the gap to BPTT compared to the baselines used in the paper.\n\nSTRENGTHS:\n### Originality\n\nThe paper combines two known ideas, independent linear recurrent modules (LRU from Orvieto et. al., 2023) and gradient sparsification for online learning (Menick et. al., 2021), and presents them in new light.\n\n### Quality\n1. The paper is technically sound, has carefully designed experiments that support the claims well.\n2. claims supported by theory and/or experiments\n### Clarity\n1. The paper is well organized and clear for most part. There are some parts that can be improved (see weaknesses).\n### Significance\n1. The well designed experiments make the results presented in the paper useful for the community.\n\nWEAKNESSES:\n### Quality\n1. Along with other claims that are well supported, the authors claim that they have pushed the standard for what is possible through online recurrent learning. However, I believe that the current experiments, which are limited to simple long range dependency tasks, are not sufficient for such a grand claim. Specifically, since all the experiments in the paper use synthetic data, it will be nice, for example, to include an experiment on real world data like small scale language modeling and compare the results to an LSTM language model, and discuss the limitations of the work in terms of performance on real world data that has complex dependency structure that is not simple long-range structure. Such an experiment will show the limitations of online learning and help guide further research on the topic.\n2. Why does table 1 not contain sparse approximation method like SnAp-n applied to dense RNNs?\n\n### Clarity\n\nFollowing  are some mistakes/typos in the paper that can throw a reader off.\n1. There is a mix-up in the symbols in equation 1, and line 103. I think $x_t$ should be replaced with $h_t$ in Eq 1, and line 103 should state that $u_t$ is the input at time $t$.\n2. Line 109-111: I think you mean to say *absence of temporal non-linearity* in line 110.\n\nFollowing are some suggestions that could improve the paper:\n1. While the authors provide through discussion for single-layer networks (Sec 3.1), the more important case of multi-layer networks does not get the same amount of care. The entire discussion in section 3.2 happens without the use of detailed expressions. Figure 1 does provide some support to the discussion but I find it to be insufficient. At the least, the authors should accompany the pictorial description in Fig 1 with expression for $\\delta$ at each layer. This will help with the discussion of various points that come up in the experiment section 4.1 that centers around effect of dept. Specifically, an expression for $\\delta$ at a lower layer it will help the point made in line 201-205, that is later discussed in section 4.1 around line 245.\n\nQUESTIONS:\n### Typos and suggestions\n1. Line 130: Using them for complex differentiation allows using calculus rules that are similar to those for real valued functions.\n2. Line 167: \"Those\" as a vague reference here might cause confusion. It might be better to say $e^\\lambda$ and $e^B$ explicitly.\n\nLIMITATIONS:\nAlong with other claims that are well supported, the authors claim that they have pushed the standard for what is possible through online recurrent learning. However, I believe that the current experiments, which are limited to simple long range dependency tasks, are not sufficient for such a grand claim. Specifically, since all the experiments in the paper use synthetic data, it will be nice, for example, to include an experiment on real world data like small scale language modeling and compare the results to an LSTM language model, and discuss the limitations of the work in terms of performance on real world data that has complex dependency structure that is not simple long-range structure. Such an experiment will show the limitations of online learning and help guide further research on the topic.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors provide an online learning algorithm for linear recurrent units [23]. They take advantage of the fact that each unit of the LRU is an ‘independent recurrent module’ and thus RTRL for each LRU layer simplifies substantially in this case and becomes tractable. They test on some long-range dependency tasks.

STRENGTHS:
-	The authors have combined insights from recent developments like LRUs being as good as RNNs while being easier to train, and approximations of RTRL for online local learning as in biology.
-	They demonstrate good results on some long-range dependency tasks as compared to RNNs.

WEAKNESSES:
-	The learning algorithm proposed seems to me just e-prop [13] applied to the LRU. Indeed, e-prop also takes into account self-recurrence of each unit and 1-step lateral recurrence. With LRU, since there is no lateral recurrence, e-prop becomes exact within one LRU layer.
-	Their algorithm still relies on spatial backprop between layers, so this is still not local in space.
-	The authors claim that “numerous biological implementations and alternatives for spatial backpropagation have been proposed [e.g., 50–59], while essentially none exist yet for backpropagation-through-time [60].”. But various approximations for BPTT have also been proposed like [13].

QUESTIONS:
-	The authors should study the relation with e-prop and inform if there are any differences with it. To me, their algorithm is the same.
-	In Fig. 3, I expect that BP is BPTT, and Spat. Is just the spatial BPTT without temporal dependencies? If not, was BPTT not used?

Minor:
-	Mistakes in equation (1)! Cf. appendix A. I’ve classified this as minor, but it does not inspire confidence that the very first equation has many mistakes.
-	L304: inference -> inference

LIMITATIONS:
N/A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that applying an online learning algorithm to independent recurrent modules of linear recurrent units, drastically reduces the algorithm’s computational and memory requirements. They then show numerically that the algorithm’s gradient approximation for multi-layer networks is close to the “real gradient” (as provided by BPTT) and that the algorithm performs well across a range of tasks despite the approximation and using decoupled recurrent units.\n\nSTRENGTHS:\nThe authors take a promising path towards effective and efficient gradient-based online learning of recurrent neural networks: Instead of deriving a new approximation of BPTT or RTRL, they choose a network architecture that drastically reduces the computational and memory requirements when deriving online gradient updates for them. The paper provides a comprehensive overview and discusses relations and differences to previous work. The authors compare their proposed network architecture to other architectures and learning algorithms on the copy task. They further evaluate their architecture for three different learning algorithms on the sCIFAR and ListOps task. The paper is very well structured, written and accessible. The algorithm is derived in great detail, including a very helpful primer on complex differentiation, which is very helpful for understanding the research question, results and potential limitations. The figures are generally clear and accessible.\n\nWEAKNESSES:\n- While I want to strongly emphasise that I think it is awesome that the authors flag that the SnAp-1 algorithm is reducing to the proposed algorithm when being applied to the proposed network architecture; this opens questions about the contributions made by the paper. In how far is the proposed algorithm new? How is it different from SnAp-1? Maybe a list of contributions at the beginning of the paper could bring clarity? I also would like to flag here that I am not familiar with SnAp-1.\n- The authors write that the “most remarkable result” is that their online learning algorithm significantly outperforms learning of networks with densely connected recurrent neurons on the copy task. Given that the copy task requires to memorise a sequence of patterns with i.i.d. sampled entries, is it really surprising that having no interference between hidden units is helpful?\n- To show that the decoupled network architecture is performing reliably and well and is a promising alternative to coupled RNNs, in my opinion a comparisons on the more complex tasks, i.e., sCIFAR and ListOps, are missing (i.e. like the data in figure 3) .\n\nMinor:\n- Two typos in equation 1 (formula for yt)\n- No description for panels A-D in caption of figure 2\n- Maybe adding h-lines for 100% and 70% accuracies in figure 2?\n\nQUESTIONS:\n- Where is the high variance of the proposed algorithm in the ListOps task coming from?\n- In order to make an online task out of sCIFAR, the authors provide the label at every timestep. How is target encoded / scaled over time? What is the influence on performance of providing the target at each time step?\n- In lines 61-62 the authors write “Finally, in Section 4, we analyse our algorithm and relevant baselines [...] with sequence lengths up to over 1000 steps” – what experiment is this referring to and how are 1000 steps required?\n\nLIMITATIONS:\nA discussion of limitations of the work is missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a forward-only approach to gradient computation in linear recurrent layers. The approach yields exact gradients for a single layer and approximate gradients for multilayer networks. They show that their approach yields more accurate gradients and results in more successful learning than alternative approximations that enable forward-only gradient computation in recurrent layers, and they illustrate how their method is particularly adapted to the structure of linear recurrent layers. They show that their approach yields test performance similar to that of backpropagation through time for a multilayer LRU-based model on two tasks in the long-range arena (LRA) benchmark.\n\nSTRENGTHS:\nThe context, relevant background, and main contribution are presented clearly. The contribution itself is technically insightful and leverages a view specific to linear, complex-valued recurrent units. The topic of efficient learning in RNNs is well-motivated and contributes to a growing literature on linear recurrent parameterizations for sequence models. The experiments, particularly the illustrations on the copy task, are well-documented and show an insightful framing of both the performance advantages and the anticipated challenges of the proposed method.\n\nWEAKNESSES:\n- While the illustration of the method on the method on the copy task is informative, the broader empirical evaluation is limited. Apart from the copy task results, the entire basis for the claim that the algorithmic contribution enables learning of long-range dependencies is a small set of results on a subset of the LRA benchmark. The authors do not disclose why only a subset of LRA is used. The results are difficult to contextualize as it’s not clear what values indicate successful learning of the long-range dependencies in the data. Other comparative evaluations, e.g. in terms of computational or memory costs, are absent. This contribution would be impactful even without state-of-the-art performance along any of these dimensions, but a lack of context is harder to overcome.\n\n- The paper relies heavily on the LRU of Orvieto et al. (2023) and appears to contain some overlap in content with that work (e.g. Fig 1 left is nearly identical to Fig 1 left in Orvieto et al.). At some points, for example $\\S$4.2, the evaluation and analysis appears to be at least as much about the LRU itself as the algorithmic contribution of this submission. The authors could improve this work by more clearly delineating the present contribution from that of Orvieto et al. (2023).\n\n- The authors spend half of the discussion arguing for the potential importance of their work for neuroscience, which seems somewhat implausible and at odds with a relatively tight architectural and algorithmic focus up to that point (save for a brief aside on lines 170-171). The similarities in terms of locality, complex values, and modularity seem mostly superficial in light of the larger differences separating forward-only learning of LRUs from computational models of spiking neurons, to say nothing of actual, biological networks of neurons. If this analogy is important, it deserves more careful development in the main body of the paper.\n\nQUESTIONS:\n- Can the authors clarify the evidence for learning of long-range dependence and offer some additional context in line with the first point above? Do *any* of the results in Table 1 suggest that the model has learned long-range dependencies? Can the authors share their reasons for limiting evaluation to a subset of the LRA benchmark?\n\n- What is the relevance of the two paragraphs spanning lines 270-287 with respect to the contribution of the present paper? They seem to be largely focused on an (important) detail for initialization of linear RNN weights. Could this finding be interpreted as evidence that the comparisons are performed suboptimally, and other initialization schemes should have been explored?\n\n- Typically, “online” learning refers to contexts in which data continuously streams from a source and low-cost updates are made in real time, after which an observation is not revisited. Do the experiments here follow this setup, or are multiple forward passes over the data required? If the latter, what is the advantage in practice of this method over BPTT?\n\nLIMITATIONS:\nThere is no direct potential for negative social impact. Potential limitations for the method and results are either discussed or covered above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an online learning algorithm for recurrent neural networks, particularly targeting the learning of long-range dependencies. It builds upon the linear recurrent units and independent recurrent modules in multi-layer networks with complex-valued neural activities. The online update approach, originally proposed in the SnAp-1 algorithm for real values, is here extended to handle complex-valued neural activities the reduces the memory and computational requirements of training. As a result, this approach outperforms both spatial (online) backpropagation and prior approximate real-time recurrent learning approaches in copy task benchmark with sequence of 50 while having marginal improvement over spatial backpropagation for the long-range arena benchmarks.\n\nSTRENGTHS:\nThe proposed online learning algorithm effectively optimizes both single-layer and multi-layer recurrent neural network architectures. By transitioning from real-valued to complex-valued neural activities and employing a diagonal recurrent connectivity matrix, it maintains good performance on long-range temporal tasks and enhances online gradient estimation. It accomplishes exact online gradient computation within a single layer using only double the resources required for inference. In managing multi-layer linear recurrent units, proposed approach adeptly approximates backpropagated errors and augments hidden states, mitigating the memory-scaling issue inherent in real-time recurrent learning. Furthermore, it can be thought of a refinement to the SnAp-1 algorithm that can handle independent recurrent units online, offering robust theoretical guarantees despite the approximation used, and accurately computes gradients for all layers, thereby improving gradient alignment and boosting overall performance.\n\nWEAKNESSES:\n* The proposed algorithm, while innovative, has some limitations acknowledged by the authors. Its approximation of the error variable, δ, can degrade over time, particularly when the neural activity values converge around 1, leading to a disregard of future error information. This approximation error is compounded when backpropagated through multiple layers, resulting in only partial error signal backpropagation. Hence, the algorithm's effectiveness can be reduced, particularly when managing complex dynamics across many layers.\n\n* The novelty of the algorithm is also somewhat constrained, as it builds upon existing concepts of Linear Recurrent Units (LRUs) and online Recurrent Neural Network (RNN) training methodologies, adapting the mechanisms of the SnAp-1 algorithm to handle complex-valued entities in independent recurrent units.\n\n* The proposed algorithm's performance, while impressive on tasks like the copy task, still lags significantly behind full backpropagation-through-time (BPTT) on longer sequence tasks such as the long-range arena benchmarks with a sequence length of 1000.\n\n* Finally, the methodology employed for hyperparameter selection — using the hyperparameters from BPTT as a basis — isn't practically applicable in an online learning scenario where BPTT won't be available.\n\nQUESTIONS:\n* How would the hyperparameters be optimized for the online learning setting where BPTT results are not available?\n\n* There seems to be a significant drop in accuracy in LRA benchmarks compared to the BPTT. Does this mean that the proposed approach doesn’t scale well to larger sequences? If so, it would be good to discuss the typical sequence lengths that this approach would be ideal for.\n\nLIMITATIONS:\nlimitations of the works has been discussed briefly, but not the potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}