Title: Exploring the Promise and Limits of Real-Time Recurrent Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: real-time recurrent learning

online recurrent learning

recurrent neural networks

reinforcement learning

actor-critic

policy gradients
Abstract: Real-time recurrent learning (RTRL) for sequence-processing recurrent neural networks (RNNs) offers certain conceptual advantages over backpropagation through time (BPTT). RTRL requires neither caching past activations nor truncating context, and enables online learning. However, RTRL's time and space complexity makes it impractical. To overcome this problem, most recent work on RTRL focuses on approximation theories, while experiments are often limited to diagnostic settings. Here we explore the practical promise of RTRL in more realistic settings. We study actor-critic methods that combine RTRL and policy gradients, and test them in several subsets of DMLab-30, ProcGen, and Atari-2600 environments. On DMLab memory tasks, our system is competitive with or outperforms well-known IMPALA and R2D2 baselines trained on 10B frames, while using fewer than 1.2B environmental frames. To scale to such challenging tasks, we focus on certain well-known neural architectures with element-wise recurrence, allowing for tractable RTRL without approximation. We also discuss rarely addressed limitations of RTRL in real-world applications, such as its complexity in the multi-layer case.

Abstract

Real-time recurrent learning (RTRL) for sequence-processing recurrent neural networks (RNNs) offers certain conceptual advantages over backpropagation through time (BPTT). RTRL requires neither caching past activations nor truncating context, and enables online learning. However, RTRL's time and space complexity makes it impractical. To overcome this problem, most recent work on RTRL focuses on approximation theories, while experiments are often limited to diagnostic settings. Here we explore the practical promise of RTRL in more realistic settings. We study actor-critic methods that combine RTRL and policy gradients, and test them in several subsets of DMLab-30, ProcGen, and Atari-2600 environments. On DMLab memory tasks, our system is competitive with or outperforms well-known IMPALA and R2D2 baselines trained on 10 B frames, while using fewer than 1.2 B environmental frames. To scale to such challenging tasks, we focus on certain wellknown neural architectures with element-wise recurrence, allowing for tractable RTRL without approximation. We also discuss rarely addressed limitations of RTRL in real-world applications, such as its complexity in the multi-layer case. 1

1 Introduction

There are two classic learning algorithms to compute exact gradients for sequence-processing recur17 rent neural networks (RNNs): real-time recurrent learning (RTRL; [1, 2, 3, 4]) and backpropagation 18 through time (BPTT; [5, 6]) (reviewed in Sec. 2). In practice, BPTT is the only one commonly used 19 today, simply because BPTT is tractable while RTRL is not. In fact, the time and space complexities 20 of RTRL for a fully recurrent NN are quadratic and cubic in the number of hidden units, respectively, 21 which are prohibitive for any RNNs of practical sizes in real applications. Despite such an obvious 22 complexity bottleneck, RTRL has certain attractive conceptual advantages over BPTT. BPTT requires 23 to cache activations for each new element of the sequence processed by the model, for later gradient 24 computation. As the amount of these past activations to be stored grows linearly with the sequence 25 length, practitioners (constrained by the actual memory limit of their hardware) use the so-called trun26 cated BPTT (TBPTT; [7]) where they specify the maximum number of time steps for this storage, 27 giving up gradient components-and therefore credit assignments-that go beyond this time span. In 28 contrast, RTRL does not require storing past activations, and enables computation of untruncated 29 gradients for sequences of any arbitrary length. In addition, RTRL is an online learning algorithm 30 (more efficient than BPTT to process long sequences in the online scenario) that allows for updating 31 weights immediately after consuming every new input (assuming that the external error feedback to 32 the model output is also available for each input). These attractive advantages of RTRL still actively 33 motivate researchers to work towards practical RTRL (e.g., [8, 9, 10, 11, 12]). 34

The root of RTRL's high complexities is the computation and storage of the so-called sensitivity 35 matrix whose entries are derivatives of the hidden activations w.r.t. each trainable parameter of the 36 model involved in the recurrence (see Sec. 2). Most recent research on RTRL focuses on introducing 37 approximation methods into the computation and storage of this matrix. For example, Menick 38 et al. [11] introduce sparsity in both the weights of the RNN and updates of the temporal Jacobian 39 (which is an intermediate matrix needed to compute the sensitivity matrix). Another line of work 40 [8, 9, 10] proposes estimators based on low-rank decompositions of the sensitivity matrix that are less 41 expensive to compute and store than the original one. Silver et al. [12] explore random projections 42 of the sensitivity. The main research question in these lines of work is naturally focused around the 43 quality of the proposed approximation method. Consequently, the central goal of their experiments is 44 typically to test hyper-parameters and configurational choices that control the approximation quality 45 in diagnostic settings, rather than evaluating the full potential of RTRL in realistic tasks. In the end, 46 we still know very little about the true empirical promise of RTRL. Also, assuming that a solution is 47 found to the complexity bottleneck, what actual applications or algorithms would RTRL unlock? In 48 what scenarios would RTRL be able to replace BPTT in today's deep learning? 49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

Here we propose to study RTRL by looking ahead beyond research on approximations. We explore the full potential of RTRL in the settings where no approximation is needed, while at the same time, not restricting ourselves to toy tasks. For that, we focus on special RNN architectures with elementwise recurrence, that allow for tractable RTRL without any approximation. In fact, the quadratic/cubic complexities of the fully recurrent NNs can be simplified for certain neural architectures. Many wellknown RNN architectures, such as Quasi-RNNs [13] and Simple Recurrent Units [14], and even certain Linear Transformers [15, 16, 17], belong to this class of models (see Sec. 3.1). Note that the core idea underlying this observation is technically not new: Mozer [18, 19] already explore an RNN architecture with this property in the late 1980s to derive his focused backpropagation , and Javed et al. [20, 21] also exploit this in the architectural design of their RNNs (even though the problematic multi-layer case is ignored; we discuss it in Sec. 5). While such special RNNs may suffer from limited computational capabilities on certain tasks (i.e., one can come up with a synthetic/algorithmic task where such models fail; see Appendix B.1), they also often perform on par with fully recurrent NNs on many tasks (at least, this is the case for the tasks we explore in our experiments). For the purpose of this work, the RTRL-tractability property outweighs the potentially limited computational capabilities: these architectures allow us to focus on evaluating RTRL on challenging tasks with a scale that goes beyond the one typically used in prior RTRL work, and to draw conclusions without worrying about the quality of approximation. We study an actor-critic algorithm [22, 23, 24] that combines RTRL and recurrent policy gradients [25], allowing credit assignments throughout an entire episode in reinforcement learning (RL) with partially observable Markov decision processes (POMDPs; [26, 27]). We test the resulting algorithm, Real-Time Recurrent Actor-Critic method (R2AC), in several subsets of DMLab-30 [28], ProcGen [29], and Atari 2600 [30] environments, with a focus on memory tasks but also including reactive ones. In particular, on two memory environments of DMLab-30, our system is competitive with or outperforms the well-known IMPALA [31] and R2D2 [32] baselines, demonstrating certain practical benefits of RTRL at scale. Finally, working with concrete real-world tasks also sheds lights on further limitations of RTRL that are rarely (if not never) discussed in prior work. These observations are important for future research on practical RTRL. We highlight and discuss these general challenges of RTRL (Sec. 5).

78

79

80

81

82

83

2 Background

Here we first review real-time recurrent learning (RTRL; [1, 2, 3, 4]), which is a gradient-based learning algorithm for sequence-processing RNNs-an alternative to the now standard BPTT.

Preliminaries. Let t , T , N , and D be positive integers. We describe the corresponding learning algorithm for the following standard RNN architecture [33] that transforms an input x ( t ) ∈ R D to an output h ( t ) ∈ R N at every time step t as

where W ∈ R N × D and R ∈ R N × N are trainable parameters, s ( t ) ∈ R N , and σ denotes the 84 element-wise sigmoid function (we omit biases). For the derivation, it is convenient to describe each 85

component s k ( t ) ∈ R of vector s ( t ) for k ∈ { 1 , ..., N } , 86

In addition, we consider some loss function L total (1 , T ) = ∑ T t =1 L ( t ) ∈ R computed on an arbitrary 87 sequence of length T where L ( t ) ∈ R is the loss at each time step t , which is a function of h ( t ) 88 (we omit writing down explicit dependencies over the model parameters). Importantly, we assume 89 that L ( t ) can be computed solely from h ( t ) at step t (i.e., L ( t ) has no dependency on any other past 90 activations apart from h ( t -1) which is needed to compute h ( t ) ). 91

92

93

The role of a gradient-based learning algorithm is to efficiently compute the gradients of the loss w.r.t. the trainable parameters of the model, i.e., ∂ L total (1 , T ) ∂ W i,j ∈ R for all i ∈ { 1 , ..., N } and

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

j ∈ { 1 , ..., D } , and ∂ L total (1 , T ) ∂ R i,j ∈ R for all i, j ∈ { 1 , ..., N } . RTRL and BPTT differ in the way to compute these quantities. While we focus on RTRL here, for the sake of completeness, we also provide an analogous derivation for BPTT in Appendix A.3.

Real-Time Recurrent Learning (RTRL). RTRL can be derived by first decomposing the total loss L total (1 , T ) over time, and then summing all derivatives of each loss component L ( t ) w.r.t. intermediate variables s k ( t ) for all k ∈ { 1 , ..., N } :

In fact, unlike BPTT that can only compute the derivative of the total loss L total (1 , T ) efficiently, RTRL is an online algorithm that computes each term ∂ L ( t ) ∂ W i,j through the decomposition above.

The first factor ∂ L ( t ) ∂ s k ( t ) can be straightforwardly computed through standard backpropagation (as stated above, we assume there is no recurrent computation between s ( t ) and L ( t ) ). For the second factor ∂ s k ( t ) ∂ W i,j , which is an element of the so-called sensitivity matrix/tensor , we can derive a forward recursion formula , which can be obtained by directly differentiating Eq. 2:

where ✶ k = i denotes the indicator function: ✶ k = i = 1 if k = i , and 0 otherwise, and σ ′ denotes the derivative of the sigmoid, i.e, σ ′ ( s n ( t -1)) = σ ( s n ( t -1))(1 -σ ( s n ( t -1))) . The derivation is similar for ∂ L ( t ) ∂ R i,j where we obtain a recurrent formula to compute ∂ s k ( t ) ∂ R i,j . As this algorithm requires to store ∂ s k ( t ) ∂ W i,j and ∂ s k ( t ) ∂ R i,j , its space complexity is O (( D + N ) N 2 ) ∼ O ( N 3 ) . The time complexity to update the sensitivity matrix/tensor via Eq. 4 is O ( N 4 ) . To be fair with BPTT, it should be noted that O ( N 4 ) is the complexity for one update; this means that the time complexity to process a sequence of length T is O ( TN 4 ) .

Thanks to the forward recursion, the update frequency of RTRL is flexible: one can opt for the fully online learning , where we update the weights using ∂ L ( t ) ∂ W at every time step, or accumulate gradients for several time steps. It should be noted that frequent updates may result in staleness of the sensitivity matrix, as it accumulates updates computed using old weights (Eq. 4).

Note that algorithms similar to RTRL have been derived from several independent authors (see, e.g., 117 [3, 18], or [34, 35] for the continuous-time version). 118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

3 Method

Our main algorithm is an actor-critic method that combines RTRL with recurrent policy gradients, using a special RNN architecture that allows for tractable RTRL. Here we describe its main components: an element-wise LSTM with tractable RTRL (Sec. 3.1), and the actor-critic algorithm that builds upon IMPALA [31] (Sec. 3.2).

3.1 RTRL for LSTM with Element-wise Recurrence (eLSTM)

The core RNN architecture we use in this work is a variant of long short-term memory (LSTM; [36]) RNN with element-wise recurrence . Let ⊙ denote element-wise multiplication. At each time step t , it first transforms an input vector x ( t ) ∈ R D to a recurrent hidden state c ( t ) ∈ R N as follows:

where f ( t ) ∈ R N , z ( t ) ∈ R N are activations, F ∈ R N × D and Z ∈ R N × D are trainable weight matrices, and w f ∈ R N and w z ∈ R N are trainable weight vectors. These operations are followed by a gated feedforward NN to obtain an output h ( t ) ∈ R N as follows:

where O ∈ R N × D and W o ∈ R N × N are trainable weight matrices. This architecture can be seen as an extension of Quasi-RNN [13] with element-wise recurrence in the gates, or Simple Recurrent Units [14] without depth gating, and also relates to IndRNN [37]. While one could further discuss myriads of architectural details [38], most of them are irrelevant to our discussion on the complexity reduction in RTRL; the only essential property here is that 'recurrence' is element-wise. We use this simple architecture above, an LSTM with element-wise recurrence (or eLSTM), for all our experiments.

Furthermore, we restrict ourselves to the one-layer case (we discuss the multi-layer case later in Sec. 5), where we assume that there is no recurrence after this layer. Based on this assumption, gradients for the parameters O and W o in Eq. 7 can be computed by the standard backpropagation, as they are not involved in recurrence. Hence, the sensitivity matrices we need for RTRL (Sec. 2) are: ∂ c ( t ) ∂ F ,

∂ c ( t ) ∂ Z ∈ R N × N × N , and ∂ c ( t ) ∂ w f , ∂ c ( t ) ∂ w z ∈ R N × N . Through trivial derivations, we can show that each of these sensitivity matrices can be computed using a tractable forward recursion formula (we provide the full derivation in Appendix A.1). For example for ∂ c ( t ) ∂ F , we have, for i, j, k ∈ { 1 , ..., N } ,

̸

where we introduce an intermediate vector ˆ f ( t ) ∈ R N with components ˆ f i ( t ) ∈ R for convenience. 144 Consequently, the gradients for the weights can be computed as: 145

Finally , we can compactly summarise these equations using the standard matrix operations. By 146 introducing notations ˆ F ( t ) ∈ R N × N with ˆ F i,j ( t ) = ∂ c i ( t ) ∂ F i,j ∈ R , and e ( t ) ∈ R N with e i ( t ) = 147

∂ L ( t ) ∂ c i ( t ) ∈ R for i ∈ { 1 , ..., N } and j ∈ { 1 , ..., D } , Eqs. 8-10 above can be written as: 148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

where, for notational convenience, we introduce a function diag : R N → R N × N that constructs a diagonal matrix whose diagonal elements are those of the input vector; however, in practical implementations (e.g., in PyTorch), this can be directly handled as vector-matrix multiplications with broadcasting (this is an important note for complexity analysis). ⊗ denotes outer-product.

Analogously, we can derive compact update equations of sensitivity matrices and gradient computations for other parameters Z , w f and w z (as well as biases which are omitted here). The complete list of these equations is provided in Appendix A.1.

The RTRL algorithm above requires maintaining sensitivity matrices ˆ F ( t ) ∈ R N × N , and analogously defined ˆ Z ( t ) ∈ R N × N , ˆ w f ( t ) ∈ R N , and ˆ w z ( t ) ∈ R N (see Appendix A.1); thus, the space complexity is O ( N 2 ) . The per-step time complexity is O ( N 2 ) (see Eqs. 8-10). This is all tractable. Importantly, these equations 11-12 can be implemented as simple PyTorch code (just like the forward pass of the same model; Eqs. 5-7) without any non-standard logics. Note that many approximations of RTRL often involve computations that are not well supported yet in the standard deep learning library (e.g., efficiently handling custom sparsity), which is an extra barrier for scaling RTRL in practice.

Note that the derivation of RTRL for element-wise recurrent nets is not novel: similar methods can be found in Mozer [18, 19] from the late 1980s. This result itself is also not very surprising, since elementwise recurrence introduces obvious sparsity in the temporal Jacobian (which is part of the second term in Eq. 4). Nethertheless, we are not aware of any prior work pointing out that several modern RNN architectures such Quasi-RNN [13] or Simple Recurrent Units [14] yield tractable RTRL (in the one-layer case). Also, while this is not the focus of our experiments, we show an example of Linear Transformers/Fast Weight Programmers [15, 16, 17] that have tractable RTRL (details can be found in Appendix A.2), which is another conceptually interesting result. We also note that the famous LSTMalgorithm [36] (companion learning algorithm for the LSTM architecture) is a diagonal approximation of RTRL, so is the more recent SnAp-1 of Menick et al. [11]. Unlike in these works, the gradients computed by our RTRL algorithm above are exact for our eLSTM architecture. This allows us to draw conclusions from experimental results without worrying about the potential influence of approximation quality. We can evaluate the full potential of RTRL for this specific architecture.

Finally, this is also an interesting system from the biological standpoint. Each weight in the weight matrix/synaptic connections (e.g., F ∈ R N × N ) is augmented with the corresponding "memory" ( ˆ F ( t ) ∈ R N × N ) tied to its own learning process, which is updated in an online fashion, as the model observes more and more examples, through an Hebbian/outer product-based update rule (Eq. 12/Left).

3.2 Real-Time Recurrent Actor-Critic Policy Gradient Algorithm (R2AC)

The main algorithm we study in this work, Real-Time Recurrent Actor-Critic method (R2AC), combines RTRL with recurrent policy gradients. Our algorithm builds upon IMPALA [31]. Essentially, we replace the RNN archicture and its learning algorithm, LSTM/TBPTT in the standard recurrent IMPALA algorithm, by our eLSTM/RTRL (Sec. 3.1). While we refer to the original paper [31] for basic details of IMPALA, here we recapitulate some crucial aspects. Let M denote a positive integer. IMPALA is a distributed actor-critic algorithm where each actor interacts with the environment for a fixed number of steps M to obtain a state-action-reward trajectory segment of length M to be used by the learner to update the model parameters. M is an important hyper-parameter that is used to specify the number of steps M for M -step TD learning [39] of the critic, and the frequency of weight updates. Given the same number of environmental steps used for training, systems trained with a smaller M apply more weight updates than those trained with a higher M . For recurrent policies trained with TBPTT, M also represents the BPTT span (i.e., BPTT is carried out on the M -length trajectory segment; no gradient is propagated farther than M steps back in time; while the last state of the previous segment is used as the initial state of the new segment in the forward pass). In the case of RTRL, there is no gradient truncation, but since M controls the update frequency, the greater the M , the less frequently we update the parameters, and it potentially suffers less from sensitivity matrix staling. This setting allows for comparing TBPTT and RTRL in the setting where everything is equal (including the number of updates) except the actual gradients applied to the weights: truncated vs. untruncated ones.

Note that for R2AC with M = 1 , one could obtain a fully online recurrent actor-critic method. However, in practice, it is known that M > 1 is crucial (for TD learning of the critic) for optimal performance. In all our experiments, we have M > 1 . The main focus of this work is to evaluate learning with untruncated gradients, rather than the potential for online learning.

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

4 Experiments

4.1 Diagnostic Task

Since the main focus of this work is to evaluate RTRL-based algorithms beyond diagnostic tasks, we only conduct brief experiments on a classic diagnostic task used in recent RTRL research work focused on approximation methods [8, 9, 10, 11, 12]: the copy task. Since our RTRL algorithm (Sec. 3.1) requires no approximation, and the task is trivial, we achieve 100% accuracy provided that the RNN size is large enough and that training hyper-parameters are properly chosen. We confirm this for sequences with lengths of up to 1000 . Additional experimental details can be found in Appendix B.1.

4.2 Memory Tasks

Here we present the main experiments of this work: RL in POMDPs using realistic game environments requiring memory.

DMLab Memory Tasks. DMLab-30 [28] is a collection of 30 first-person 3D game environments, with a mix of both memory and reactive tasks. Here we focus on two well-known environments, rooms_select_nonmatching_object and rooms_watermaze , which are both categorised as 'memory' tasks according to Parisotto et al. [40]. The mean episode lengths of these tasks are about 100 and 1000 steps, respectively. As we apply an action repetition of 4, each 'step' corresponds to 4 environmental frames here. We refer to Appendix B.2 for further descriptions of these tasks, and experimental details. Our model architecture is based on that of IMPALA [31]. Both RTRL and TBPTT systems use our eLSTM (Sec. 3.1) as the recurrent layer with a hidden state size of 512. Everything is equal between these two systems except that the gradients are truncated in TBPTT but not in RTRL. To reduce the overall compute needed for the experiments, we first pre-train one TBPTT model for 50 M steps for rooms_select_nonmatching_object , and for 200 M steps for rooms_watermaze . Then, for all main training runs in this experiment, we initialise the parameters of the convolutional vision module from the same pre-trained model, and keep these parameters frozen (and thus, only train the recurrent layer and everything above it). For these main training runs, we train for 30 M and 100 M steps for rooms_select_nonmatching_object and rooms_watermaze , respectively; resulting in the total of 320 M and 1.2 B environmental frames. We compare RTRL and TBPTT for different values of M ∈ { 10 , 50 , 100 } (Sec. 3.2). We recall that M influences: the frequency of weight updates, M -step TD learning, as well as the backpropagation span for TBPTT.

Table 1 shows the corresponding scores, and the left part of Figure 1 shows the training curves. We observe that for select_nonmatching_object which has a short mean episode length of 100 steps, the performance of TBPTT and RTRL is similar even with M = 50 . The benefit of RTRL is only visible in the case with M = 10 . In contrast, for the more challenging rooms_watermaze task with a mean episode length of 1000 steps, RTRL outperforms TBPTT for all values of M ∈ { 10 , 50 , 100 } . Furthermore, with M = 50 or 100 , our RTRL system outperforms the IMPALA and R2D2 systems from prior work [32], while trained on fewer than 1.2 B frames. Note that R2D2 systems [32] are trained without action repetitions, and with a BPTT span of 80. This effectively demonstrates the practical benefit of RTRL in a realistic task requiring long-span credit assignments.

ProcGen. We test R2AC in another domain: ProcGen [29]. Most ProcGen environments are solvable using a feedforward policy even without frame-stacking [29]. There is a so-called memory-mode for certain games, making the task partially observable by making the world bigger, and restricting agents' observations to a limited area around them. However, in our preliminary experiments, we observe that even in these POMDP settings, both the feedforward and LSTM baselines perform similarly (see Appendix B.3). Nevertheless, we find one environment in the standard hard-mode , Chaser , which shows clear benefits of recurrent policies over those without memory. Chaser is similar to the classic game 'Pacman,' effectively requiring some counting capabilities to fully exploit power pellets valid for a limited time span. The mean episode length for this task is about 200 steps, where each step is an environmental frame as we apply no action repeat for ProcGen. Unlike in the DMLab experiments above, here we train all models from scratch for 200 M steps without pre-training the vision module (since training the vision parameters using RTRL is intractable, they are trained with truncated gradients, i.e., only the recurrent layer is trained using RTRL; we discuss this further in Sec. 5). We compare RTRL and TBPTT with M = 5 or 50 . The training curves are shown in the right part of Figure 1.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e7

10

20

30

40

50

60

Return

TBPTT

RTRL

(a) Non-matching,

M

= 10

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e8

10

20

30

40

50

60

Return

TBPTT

RTRL

M

= 10

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e8

0

1

2

3

4

5

6

7

8

Return

TBPTT

RTRL

Feedforward

(c) Chaser,

M

= 5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e7

10

20

30

40

50

60

Return

TBPTT

RTRL

(d) Non-matching,

M

= 100

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e8

35

40

45

50

55

60

Return

TBPTT

RTRL

M

= 100

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e8

1

2

3

4

5

6

7

8

9

Return

TBPTT

RTRL

Feedforward

(f) Chaser,

M

= 50

Similar to the rooms_select_nonmatching_object case above, with a sufficiently large M = 50 , 255 there is no difference between RTRL and TBPTT, while we observe benefits of RTRL when M = 5 . 256

257

258

259

260

261

262

263

4.3 General Evaluation

Here we evaluate R2AC more broadly, including environments which are mostly reactive.

Atari. Apart from some exceptions (such as Solaris [32]), many of the Atari game environments are considered to be fully observable when observations consist of a stack of 4 frames [41, 42]. However, it is also empirically known that, for certain games, recurrent policies yield higher performance than the feedforward ones having only access to 4 past frames (see, e.g., [43, 32, 44]). Here our general goal is to compare RTRL to TBPTT more broadly. We use five Atari environments: Breakout ,

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e8

40

60

80

100

120

140

160

180

Return

TBPTT

RTRL

Feedforward

(a)

Breakout

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e8

0

200

400

600

800

1000

1200

1400

Return

TBPTT

RTRL

Feedforward

(b)

Gravitar

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e8

500

1000

1500

2000

2500

3000

3500

Return

TBPTT

RTRL

Feedforward

(c)

MsPacman

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e8

0

2000

4000

6000

8000

10000

12000

Return

TBPTT

RTRL

Feedforward

(d)

Q*Bert

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Steps

1e8

2000

0

2000

4000

6000

8000

10000

12000

14000

Return

TBPTT

RTRL

Feedforward

(e) Seaquest

200

Gravitar , MsPacman , Q*bert , and Seaquest , following Kapturowski et al. [32]'s selection for ablations of their R2D2. Here we use M = 50 , and train for 200 M steps (with the action repeat of 4) from scratch. The learning curves are shown in Figure 2. With the exception of MsPacman (note that, unlike ProcGen/ Chaser above, 4-frame stacking is used) where we observe a slight performance degradation, RTRL performs equally well or better than TBPTT in all other environments.

DMLab Reactive Task. Finally, we also test our system on one environment of DMLab-30, room_keys_doors_puzzle , which is categorised as a reactive task according to Parisotto et al. [40]. We train with M = 100 for 100 M steps (with the action repeat of 4). The mean episode length is about 450 steps. Table 2/right shows the scores. Effectively, all feedforward, TBPTT, and RTRL systems perform nearly the same (at least within 100 M steps/400 M frames). We note that these scores are comparable to the one reported by the original IMPALA [31] which is 28 . 0 after training on 1 B frames, which is much worse than the score reported by Kapturowski et al. [32] for IMPALA trained using 10 B frames ( 54 . 6 ). We show this example to confirm that RTRL is effectively not helpful on a reactive task, unlike in the memory tasks above.

5 Limitations and Discussion

Here we discuss limitations of this work, which also sheds light on more general challenges of RTRL.

Multi-layer case of our RTRL. The most crucial limitation of our tractable-RTRL algorithm 280 for element-wise recurrent nets (Sec. 3.1) is its restriction to the one-layer case. By stacking two 281 such layers, the corresponding RTRL algorithm becomes intractable as we end up with the same 282 complexity bottleneck as in fully recurrent networks. This is simply because by composing two such 283 element-wise recurrent layers, we obtain a fully recurrent NN as a whole. This can be easily seen by 284 writing down the following equations. By introducing extra superscripts to denote the layer number, 285 in a stack of two element-wise LSTM layers of Eqs. 5-6 (we remove the output gate), we can express 286

the recurrent state c (2) ( t ) of the second layer at step t as a function of the recurrent state c (1) ( t -1) 287 of the first layer from the previous step as follows: 288

By looking at the first term of Eq. 13 and that of Eq. 16, one can see that there is full recurrence between c (2) ( t ) and c (1) ( t -1) via F (2) , which brings back the quadratic/cubic time and space complexity for the sensitivity of the recurrent state in the second layer w.r.t. parameters of the first layers. This limitation is not discussed in prior work [18, 19].

Complexity of multi-layer RTRL in general. Generally speaking, RTRL for the multi-layer case is rarely discussed (except Meert and Ludik [45]; 1997). This case is important in modern deep learning where stacking multiple layers is a standard. There are two important remarks to be made here.

First of all, even in an NN with a single RNN layer, if there is a layer with trainable parameters whose output is connected to the input of the RNN layer, a sensitivity matrix needs to be computed and stored for each of these parameters. A good illustration is the policy net used in all our RL experiments where our eLSTM layer takes the output of a deep (feedforward) convolutional net (the vision stem) as input. As training this vision stem using RTRL requires dealing with the corresponding sensitivity matrix, which is intractable, we train/pretrain the vision stem using TBPTT (Sec. 4.2;4.3). This is an important remark for RTRL research in general. For example, approximation methods proposed for the single-layer case may not scale to the multi-layer case; e.g., to exploit sparsity in the policy net above, it is not enough to assume weight sparsity in the RNN layer, but also in the vision stem.

Second, the multi-layer case [45] introduces more complexity growth to RTRL than to BPTT. Let L denote the number of layers. We seemlessly use BPTT with deep NNs, as its time and space complexity is linear in L . This is not the case for RTRL. With RTRL, for each recurrent layer, we need to store sensitivities of all parameters of all preceding layers. This implies that, for an L -layer RNN, parameters in the first layer require L sensitivity matrices, L -1 for the second layer, ..., etc., resulting in L +( L -1) + ( L -2) + ... +2+1 = L ( L +1) / 2 sensitivity matrices to be computed and stored. Given that multi-layer NNs are crucial today, this remains a big challenge for practical RTRL research.

Principled vs. practical solution. Another important aspect of RTRL research is that many realistic memory tasks have actual dependencies/credit assignment paths that are shorter than the maximum BPTT span we can afford in practice. In our experiments, with the exception of DMLab rooms_watermaze (Sec. 4.2), no task actually absolutely requires RTRL in practice; TBPTT with a large span suffices. Future improvements of the hardware may give a further advantage to TBPTT; the practical (simple) solution offered by TBPTT might be prioritised over the principled (complex) RTRL solution for dealing with long-span credit assignments. This is also somewhat reminiscent of the Transformer vs. RNN discussion regarding sequence processing with limited vs. unlimited context.

Sequence-level parallelism. While our study focuses on evaluation of untruncated gradients, another potential benefit of RTRL is online learning. For most standard self/supervised sequenceprocessing tasks such as language modelling, however, modern implementations are optimised to exploit access to the 'full' sequence, and to leverage parallel computation across the time axis (at least for training). While some hybrid RTRL-BPTT approaches [46] may still be able to exploit such a parallelism, fast online learning remains open engineering challenge even with tractable RTRL.

6 Conclusion

We demonstrate the empirical promise of RTRL in realistic settings. By focusing on RNNs with element-wise recurrence, we obtain tractable RTRL without approximation. We evaluate our reinforcement learning RTRL-based actor-critic in several popular game environments. In one of the challenging DMLab-30 memory environments, our system outperforms the well-known IMPALA and R2D2 baselines which use many more environmental steps. We also highlight general important limitations and further challenges of RTRL rarely discussed in prior work.

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

References

Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation , 1(2):270-280, 1989.

Ronald J Williams and David Zipser. Experimental analysis of the real-time recurrent learning algorithm. Connection science , 1(1):87-111, 1989.

Anthony J Robinson and Frank Fallside. The utility driven dynamic error propagation network , volume 1. University of Cambridge Department of Engineering Cambridge, 1987.

Anthony J Robinson. Dynamic error propagation networks. PhD thesis, University of Cambridge, 1989.

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature , 323(6088):533-536, 1986.

Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78(10):1550-1560, 1990.

Ronald J Williams and Jing Peng. An efficient gradient-based algorithm for online training of recurrent network trajectories. Neural computation , 2(4):490-501, 1990.

Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. In Int. Conf. on Learning Representations (ICLR) , Vancouver, Canada, April 2018.

Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random kronecker factors. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , pages 6594-6603, Montréal, Canada, December 2018.

Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal kronecker-sum approximation of real time recurrent learning. In Proc. Int. Conf. on Machine Learning (ICML) , volume 97, pages 604-613, Long Beach, CA, USA, June 2019.

Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, and Alex Graves. Practical real time recurrent learning with a sparse approximation. In Int. Conf. on Learning Representations (ICLR) , Virtual only, May 2021.

David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt. Learning by directional gradient descent. In Int. Conf. on Learning Representations (ICLR) , Virtual only, April 2022.

James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. In Int. Conf. on Learning Representations (ICLR) , Toulon, France, April 2017.

Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP) , pages 4470-4481, Brussels, Belgium, November 2018.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML) , Virtual only, July 2020.

Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, March 1991.

Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear Transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learning (ICML) , Virtual only, July 2021.

Michael C. Mozer. A focused backpropagation algorithm for temporal pattern recognition. 375 Complex Systems , 3(4):349-381, 1989. 376

Michael Mozer. Induction of multiscale temporal structure. In Proc. Advances in Neural 377 Information Processing Systems (NIPS) , pages 275-282, Denver, CO, USA, December 1991. 378

Khurram Javed, Martha White, and Richard S. Sutton. Scalable online recurrent learning using 379 columnar neural networks. Preprint arXiv:2103.05787 , 2021. 380

381

382

Khurram Javed, Haseeb Shah, Rich Sutton, and Martha White. Online real-time recurrent learning using sparse connections and selective learning. Preprint arXiv:2302.05326 , 2023.

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Proc. Advances in Neural Information Processing Systems (NIPS) , pages 1008-1014, Denver, CO, USA, November 1999.

Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proc. Advances in Neural Information Processing Systems (NIPS) , pages 1057-1063, Denver, CO, USA, November 1999.

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proc. Int. Conf. on Machine Learning (ICML) , pages 1928-1937, New York City, NY, USA, June 2016.

Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber. Recurrent policy gradients. Logic Journal of IGPL , 18(2):620-634, 2010.

Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence , 101(1-2):99-134, 1998.

Jürgen Schmidhuber. An on-line algorithm for dynamic reinforcement learning and planning in reactive environments. In Proc. Int. Joint Conf. on Neural Networks (IJCNN) , pages 253-258, San Diego, CA, USA, June 1990.

Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. Preprint arXiv:1612.03801 , 2016.

Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proc. Int. Conf. on Machine Learning (ICML) , pages 2048-2056, Virtual only, July 2020.

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In Proc. Int. Conf. on Machine Learning (ICML) , pages 1406-1415, Stockholm, Sweden, July 2018.

Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In Int. Conf. on Learning Representations (ICLR) , New Orleans, LA, USA, May 2019.

Jeffrey L Elman. Finding structure in time. Cognitive science , 14(2):179-211, 1990.

Barak A Pearlmutter. Learning state space trajectories in recurrent neural networks. Neural Computation , 1(2):263-269, 1989.

Michael Gherrity. A learning algorithm for analog, fully recurrent neural networks. In Proc. Int. Joint Conf. on Neural Networks (IJCNN) , volume 643, 1989.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper RNN. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pages 5457-5466, Salt Lake City, UT, USA, June 2018.

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. IEEE transactions on neural networks and learning systems , 28(10):2222-2232, 2016.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 1998.

Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing Transformers for reinforcement learning. In Proc. Int. Conf. on Machine Learning (ICML) , pages 7487-7498, Virtual only, July 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

Matthew J. Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI Fall Symposia , pages 29-37, Arlington, VA, USA, November 2015.

Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , pages 12329-12338, Vancouver, Canada, December 2019.

Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , Virtual only, December 2021.

Kürt Meert and Jacques Ludik. A multilayer real-time recurrent learning algorithm for improved convergence. In Proc. Int. Conf. on Artificial Neural Networks (ICANN) , pages 445-450, Lausanne, Switzerland, October 1997.

Jürgen Schmidhuber. A fixed size storage o ( n 3 ) time complexity learning algorithm for fully 450 recurrent continually running networks. Neural Computation , 4(2):243-248, 1992. 451

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate Realtime Recurrent Learning (RTRL) applied to recurrent RL, in contrast with traditional Backpropagation Through Time (BPTT). The authors' key insight is that they can significantly reduce time and space complexity by using single-layer elementwise recurrence. They propose an elementwise-LSTM (eLSTM), similar to IndRNN, for this purpose.\n\nThe authors execute a toy diagnostic task, then use IMPALA and R2D2 to train two variants of their eLSTM on pixels tasks -- one variant using BPTT and the other using RTRL.\n\nAs expected, the authors find that increasing the TBPTT length to large values provides the same performance as RTRL. Their is a notable performance gap when using short TBPTT, motivating the use of their approach. Although the tasks here do not necessarily require RTRL, one can imagine POMDPs that span hundreds of thousands of timesteps where this could be useful.\n\nSTRENGTHS:\n- The authors present an interesting and forward-looking idea. Although we are not aware of any common RL tasks today that require such long-term backpropgatation, it's clear that human-level decision making happens over timeframes much too long for BPTT to be tractable and for TBPTT to capture such dependencies. We suspect methods like RTRL will be necessary for truly intelligent agents.\n- The paper is generally well written, and it appears that the authors have dilligently reviewed prior literature.\n\nWEAKNESSES:\n- My biggest complaint is that there is no data showing empirical time or space efficiency of RTRL. My understanding is that the whole point of RTRL is decoupling training space efficiency from the sequence length. I do not expect RTRL to be faster for such short sequences, but I would like to compare practical wall-clock time or GPU memory usage.\n\n- I think it should be made explicit that in the traditional RL scenario (rollout workers separate from trainer, sync weights from trainer to workers every update), BPTT and RTRL are equivalent. It is really only the BPTT truncation that causes issues.\n\n- The paper could benefit from a mini-study further examining the effects of TBPTT vs BPTT. The watermaze/chaser experiments do this, but only for a very limited number of truncation lengths. It would be interesting to see results with truncation lengths at 10, 20, ... 200.\n\nQUESTIONS:\n- Line 24: to cache -> caching\n- Diagnostic task is run for a maximum length of 1000\n    - This is achievable using untruncated BPTT. Since BPTT scales with sequence length and RTRL does not, it would make more sense to do a very large sequence length.\n- It is not clear to me why varying $M$ matters for RTRL if training is not \"online\" in the sense that the rollout workers are not updating weights\n\nLIMITATIONS:\n- RTRL is relatively understudied and thus has numerous limitations in its current state. As it receives further attention, I suspect such limitations will be mitigated. It is nice to see the authors be honest about the limitations of RTRL throughout the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores an alternative to back-propagation through time (BPTT) for training recurrent neural networks, called real time recurrent learning (RTRL). The alternative algorithm does not depend on past activations, and also does not need truncation (which limits past context) like in BPTT to make it tractable. Although RTRL is very computationally complex for the general case, this paper exploits the fact that several recent recurrent architectures (Quasi-RNNs) allow for exact RTRL gradients to be computed efficiently. The method is evaluated on reinforcement learning tasks on several benchmarks and is compared with truncated BPTT  with LSTM while using Actor-Critic method as RL algorithms.\n\nSTRENGTHS:\n1.\tThe paper studies a very interesting direction of using RTRL instead of truncated BPTT. The difficulty of using BPTT for training recurrent architectures is well known and it might be a potential bottleneck in allowing recurrent networks in achieving their full performance in complex tasks in the real-world.  \n2.\tThe paper is written in a very clear way, clearly outlining the core contributions and the focus of this work.\n3.\tThe proposed method is very clean, elegant, well-explained, and simple. \n4.\tThe empirical evaluation is done very extensively on DMLab, ProcGen and Atari, and it's interesting to see that RTRL + A2C performs equivalent, or better than TBPTT, except for in a few scenarios. \n5.\tThe limitations section is well-written and shows a good understanding of the proposed technique.\n\nWEAKNESSES:\nWeaknesses / Questions:\n1. Ablate the dependence of M (episode rollout length) on R2AC. Since larger M leads to less frequent updates and shorter M leads to stale values in the sensitivity matrix, it would be interesting to see how the performance varies with this parameter. \n2. One modification to the empirical evaluation that I would suggest is to compare with LSTM (instead of eLSTM), and also to compare the best performance for both the training algorithms – TBPTT and R2AC for any value of M, tuned separately for the two algorithms.\n3. How does BPTT + LSTM with multiple hidden layers perform in the considered benchmarks?\n\nQUESTIONS:\n(see weaknesses section for questions)\n\nLIMITATIONS:\nThe limitations section is well-written and gives a clear explanation of the limits of the proposed line of work to train recurrent networks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper explores the use of RTRL in the case of scenario, where no approximation is needed.\n- The proposed \"method\" is not novel per se i.e., Mozer et. al put forward the derivation of RTRL for element-wise recurrence, but to the best of reviewer's ability no one has formally wrote about the tractability of the RTRL in case of some of the RNN architectures (Quasi-RNN or Simple Recurrent Units).\n- The paper does experiments for RL, where on different experiments the paper shows the method trained with RTRL can match or even sometimes outperform Truncated Backprop through time. \n\n==== \n\nRead the author's rebuttal.\n\nSTRENGTHS:\n- The paper is very well written.\n- The idea of using RTRL for RNNs in which element-wise recurrent is present is interesting.\n- The paper does experiments on many RL tasks, and compares the proposed method to TBPTT.\n\nWEAKNESSES:\n- Since the paper is about how well RTRL works with some RNNs, it will be useful to evaluate the proposed method on more tasks like language modelling or sequential image classification etc. \n- For all the experiments, it will also be useful to report a BPTT baseline i.e., where we backprop through the entire \"episode\". \n- It will also be useful to compare to various other approximations of RTRL (using the vanilla LSTM/GRU) or even consider the tasks consider in SnAp-1. \n- Not a weakness per se, but this is incorrect \"Silver et al. [12] explore random projections of the sensitivity\". DODGE work explores not just the random projections, but also learning the candidate direction (as a result of auxiliary tasks or via RL like synthetic gradients).\n\nQUESTIONS:\n- Evaluate the proposed method on more tasks like language modelling, sequential image classification as well as compare to SnAp-1.\n- Report the BPTT baseline, as well as some approximation of RTRL to just see how well those approximations work for the problems chosen in this work. \n- It will be helpful to make sure similar setup is followed throughout the experiments. For example: for some experiments, the paper uses the pre-trained visual encoder, or trains the visual encoder via the truncated BPTT gradient.\n\nLIMITATIONS:\nSee Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper modifies the LSTM architecture so that RTRL complexity, which is usually O(n^4), becomes O(n^2) where n is the number of recurrent units. This is achieved by constraining the recurrent weights to a diagonal structure. After performing some diagnostic tests, the paper presents experimental results using the proposed eLSTM in actor-critic RL settings, specifically DMLab, ProcGen, and Atari. The paper concludes with a discussion on the complexity of RTRL in the multi-layer case.\n\nSTRENGTHS:\nThe paper is well written and clearly structured. \n\nWhile the ideas presented in the paper are not novel per se, the application of these ideas to a modern RNN architecture in the context of RL is quite novel. \n\nThe work is quite relevant for the RNN/RL communities as unlocking RTRL for large-scale recurrent models is a long-standing problem and RL is an important field of application for RNNs. \n\nThe paper discusses some rarely addressed limitations about the complexity of RTRL in the multi-layer case as well as some other interesting points.\n\nWEAKNESSES:\nThe main weakness of the paper is the empirical evaluation of the method. This ranges from an incrompehensible selection of tasks over a severe lack of external baselines to insufficient evaluation methods (e.g., lack of significance testing). In the following, I will elaborate in greater detail on these and other shortcomings. \n\nOn Procgen and Atari, the paper lacks comparison to state-of-the-art methods or any other external baselines. In general, the authors seem to confuse the concepts of ablation studies and external baselines. For instance, TBPTT on the eLSTM architecture is an ablation study while TBPTT on the fully recurrent LSTM architecture is a baseline (which is missing in all experiments but should be included; see, e.g., [3] for design choices). The authors should compare to state-of-the-art baselines in all environments, not just DMLab-30. \n\nThe paper should experimentally explore the limitations arising from the diagonal constraint on the hidden-to-hidden interactions. This limitation competes with the bias in the gradient estimate used by the companion learning algorithm and SnAp-1 for LSTM. The authors should compare these RTRL variants in experiments to justify their proposed method as opposed to existing ones. \n\nThe core mechanism described in Eq. (5) that reduces the RTRL complexity has been proposed in [1] and should be cited properly. \n\nIn all RL experiments, the authors report mean and standard deviation over only 3 seeds (at least for DMLab-30 and Procgen; for Atari I didn't find any info) without any significance testing. In particular, the use of the standard deviation as a measure of uncertainty in combination with a low number of seeds has been critisized as bad practice and should be replaced by interval estimates such as IQM [2]. \n\nThe paper states that it focuses on \"RTRL-based algorithms beyond diagnostic tasks.\" However, it limits these experiments to the realm of RL, where the importance of memory is often unclear. The authors should investigate how the architectural changes affect performance in some standard supervised tasks. \n\n[1] Felix A. Gers, Jürgen Schmidhuber:\nRecurrent Nets that Time and Count. IJCNN (3) 2000: 189-194\n\n[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, Marc G. Bellemare:\nDeep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS 2021: 29304-29320\n\n[3] Tianwei Ni, Benjamin Eysenbach, Ruslan Salakhutdinov:\nRecurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. ICML 2022: 16691-16723\n\nQUESTIONS:\nThe proposed eLSTM architecture does not use any cell state activation function. This does not affect the complexity of RTRL so I wonder why this slightly off-standard decision was made. \n\nLine 21, 53, and 290: shouldn't the word \"quadratic\" be replaced by \"quartic\"? \n\nWhy did the authors dedicate resources to reactive tasks where no benefit are to be expected or an RNN might not be helpful to begin with? I had the impression that resources for experiments are limited. This exacerbates the question why reactive tasks were explored. \n\nThe paper states its focus is \"to evaluate learning with untruncated gradients rather than the potential for online learning.\" I wonder on what basis this decision was made as online updates could lead to high sample efficiency, one of the key problems in RL. I feel the paper missed a big chance by not researching this connection of online updates and sample efficiency. Instead, it focussed on reactive tasks. \n\nThe paper states that \"both the feedforward and LSTM baselines perform similarly\" in Procgen Memory mode. However, [5] shows that the feedforward baseline is outperformed by methods using memory in 4/6 environments. Moreover, the Procgen paper [4] (see appendix H) shows that LSTM or framestacking outperforms feedforward in 4/6 envirnments even in non-memory mode. \n\nIn summary, I really like the presented method (and the way it was presented) and believe that it bears quite some potential. However, the empirical evaluation is very weak and I do not believe it can be fixed in the short rebuttal period. Therefore, publication of the paper seems premature at this point. \n\n[4] Karl Cobbe, Christopher Hesse, Jacob Hilton, John Schulman:\nLeveraging Procedural Generation to Benchmark Reinforcement Learning. ICML 2020: 2048-2056\n\n[5] Fabian Paischer, Thomas Adler, Vihang P. Patil, Angela Bitto-Nemling, Markus Holzleitner, Sebastian Lehner, Hamid Eghbal-Zadeh, Sepp Hochreiter:\nHistory Compression via Language Models in Reinforcement Learning. ICML 2022: 17156-17185\n\nLIMITATIONS:\nThe authors give some interesting discussion on the limitation of their work, in particular the multi-layer RNN case.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}