SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The first equality uses the definition of  $e_{t + 1}^{\lambda}$ , the second the recurrent update of  $h_t$  as in Equation 8 and the fact that  $\gamma \odot Bx_{t + 1}$  does not depend on  $\lambda$ , and the third is the complex product rule. The
gradient of the loss with respect to  $\lambda_{i}$  is then equal to
$$
\begin{array}{l} \frac {\mathrm {d} L}{\mathrm {d} \lambda_ {i}} = \sum_ {t = 1} ^ {T} \sum_ {j} \frac {\mathrm {d} L _ {t}}{\mathrm {d} h _ {t , j}} \frac {\mathrm {d} h _ {t , j}}{\mathrm {d} \lambda_ {i}} (19) \\ = \sum_ {t = 1} ^ {T} \frac {\mathrm {d} L _ {t}}{\mathrm {d} h _ {t , i}} \frac {\mathrm {d} h _ {t , i}}{\mathrm {d} \lambda_ {i}} (20) \\ = \sum_ {t = 1} ^ {T} \delta_ {t, i} e _ {t, i} ^ {\lambda}. (21) \\ \end{array}
$$
In the first line, we applied forward-mode complex differentiation (combined with  $h_t$  being an holomorphic function of  $\lambda$ ) as in Equation 5. In the second, we used that  $h_i$  is independent of  $\lambda_j$  for  $i \neq j$ .
Update for  $\gamma$ . The derivation for the recurrent update of  $e^{\gamma}$  being very similar to the one for  $\lambda$ , we omit it. The fact that  $\gamma$  is a real variable does not change anything to this derivation. However, this makes the gradient computation slightly different. One can act as if  $\gamma$  was a complex variable  $\gamma_i^{\mathbb{C}}$  and get
$$
\frac {\mathrm {d} L}{\mathrm {d} \gamma_ {i} ^ {\mathbb {C}}} = \sum_ {t = 1} ^ {T} \delta_ {t, i} e _ {t, i} ^ {\gamma}. \tag {22}
$$
Now, looking at the real part of the previous equation, we have
$$
\frac {\mathrm {d} L}{\mathrm {d} \gamma_ {i}} = 2 \sum_ {t = 1} ^ {T} \operatorname {R e} \left[ \delta_ {t, i} e _ {t, i} ^ {\gamma} \right]. \tag {23}
$$
Note that the factor 2 will also indirectly appear in the update of  $\lambda$ , although it is not yet there in the gradient, as we have a real derivative here, when it is a complex one above.
Update for  $B$ . We have
$$
e _ {t + 1, j i} ^ {B} = \frac {\mathrm {d} h _ {t + 1 , j}}{\mathrm {d} B _ {j i}} = \frac {\mathrm {d}}{\mathrm {d} B _ {j i}} \left[ \lambda_ {j} h _ {t, j} + \gamma_ {j} B _ {j i} x _ {t + 1, i} \right] = \lambda_ {j} \frac {\mathrm {d} h _ {t , i}}{\mathrm {d} B _ {j i}} + \gamma_ {j} x _ {t + 1, i} \tag {24}
$$
$$
\frac {\mathrm {d} L}{\mathrm {d} B _ {j i}} = \sum_ {t = 1} ^ {T} \delta_ {t, j} e _ {t, j i} ^ {B}. \tag {25}
$$
Note that the update for  $e^B$  is more general than the one of Equation 6 in the main text in which  $\gamma = 1$ .
We base our implementation on the S5 [22] code base<sup>4</sup>. All networks are trained with the AdamW optimizer, with a linear learning rate warm up, followed by a one-cycle cosine learning rate decay. Following common practice, we do not apply weight decay for  $\lambda$ ,  $\gamma$ , and use a smaller learning rate for those parameters (global learning rate times a learning rate factor). By default, we initialize  $\lambda$  uniformly at random in the complex disk and  $\theta$  uniformly at random in  $[0, 2\pi]$ .
B.1 Copy task experimental details and hyperparameters
We report the hyperparameters we used and scanned over for the copy task in Tables 4 and 5. In Section 4.1, we take the default configuration reported in the LRU paper [23] for backpropagation-through-time and apply it to the different methods we consider. We use 25 epochs, with a linear warmup of 5 epochs. We tuned the learning rate for each method independently in the comparison of Figure 2.E and F and in the one of Table 1. For the 1-layer GRU architecture, following Menick et al. [15], we add an extra readout layer: a 1-hidden layer MLP with hidden dimension 1072 processes the hidden state to generate the output.
4https://github.com/lindermanlab/S5
Table 4: Hyperparameter configurations for Section 4.1. We use  $\left[\dots\right]$  to denote hyperparameters that were scanned over with grid search and  $\{\cdot \cdot \cdot \}$  to denote the variables of interest for the figure.
Hyperparameter	FIG.2.A/B	FIG.2.C/D	FIG.2.E/F
Learning rule	Ours	Ours	{Ours, Spat., Trunc., BP}
Pattern length	20	20	20
Padding	7	7	7
Training samples	20000	20000	20000
Number of layers	{1,2,3,4}	4	4
Recurrent state size N	64	64	64
Model size H	128	128	128
|λmin|	0	{0,0.5,0.75,0.9}	0
Epochs	25	25	25
Warmup	0	0	0
Batch-size	50	50	50
Base learning rate	10-3	2 × 10-3	2[0,1,2,3] 10-3
Learning rate factor	0.5	0.5	0.5
Dropout probability	0.1	0.1	0.1
Weight-decay	0	0	0
Table 5: Hyperparameter configurations for Section 4.2. We use  $\left[\dots\right]$  to denote hyperparameters that were scanned over.
Layer	LRU	Linear RNN	GRU	GRU
Number layers	4	4	1	4
Pattern length	20	20	20	20
Padding	7	7	7	7
Training samples	20000	20000	20000	20000
GLU	Yes	Yes	No	Yes
N	64	64	134	91
H	128	146	134	91
Extra readout	No	No	1072	No
Number parameters	268,430	267,956	269,488	270,011
Batch-size	20	20	20	20
Base learning rate	\( 2^{\left\lbrack 0,\cdots ,5\right\rbrack }{10}^{-3} \)	\( 2^{\left\lbrack 0,\cdots ,5\right\rbrack }{10}^{-3} \)	\( 2^{\left\lbrack 0,\cdots ,5\right\rbrack }{10}^{-3} \)	\( 2^{\left\lbrack 0,\cdots ,5\right\rbrack }{10}^{-3} \)
Learning rate factor	1	1	1	1
Dropout probability	0.1	0.1	0.1	0.1
Weight-decay	0	0	0	0
Table 6: Hyperparameter configurations for SCIFAR, IMDB and LISTOPS experiments. We use  $[\cdot \cdot \cdot ]$  to denote hyperparameters that were scanned over with grid search. By default, the  $\{\lambda ,\gamma \}$  parameters have no weight decay and have a slower learning rate (c.f. learning rate factor).
Hyperparameter	CIFAR	IMDB	LISTOPS
Number of layers	4	4	4
Recurrent state size N	128	128	128
Model size H	256	256	256
|λ|min	0.9	[0.0,0.9]	[0.0,0.9]
|λ|max	0.999	1	1
Batch-size	100	32	32
Base learning rate	[0.001,0.004]	[0.001,0.003]	[0.001,0.003]
Learning rate factor	0.5	0.5	0.5
Dropout probability	0.1	0	0.1
Weight-decay	[0.1,0.5]	0.05	0.05
Epochs	180	40	35
Warmup	18	4	0
B.2 Experimental details and hyperparameters for SCIFAR, IMDB and LISTOPS
For all experiments, we first ran a manual coarse-grained hyperparameter tuning to identify the most important parameters, and then ran the grid search described in 6. The final hyperparameters were each evaluated on 3 fresh seeds for the results reported in Table 2. The training time for our online learning rule on a single Nvidia RTX3090 GPU for SCIFAR, IMDB and LISTOPS was respectively 36, 10 and 40 hours.
For the comparison with linear RNNs, we kept the number  $N$  of hidden neurons fixed compared to the one used the LRU (c.f. Table 6), but changed the model size  $H$  to 294 in order to obtain the same number of parameters (our Linear RNN has 1,068,086 parameters vs. 1,063,178 for the LRU). We used the same hyperparameter optimization scheme.