SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[Ni et al., 2022] does not compare along the recurrent state size, but rather the hidden size, while utilizing various hidden sizes  $h$ . In other words, the LSTM recurrent size is twice that of the GRU. We let  $c = h / 32$  and  $m = h / c$ , so that  $mc = h$ . This produces an equivalent FFM configuration to the POPGym baseline when  $h = 128$ , with equivalent recurrent size in bytes to the LSTM (or equivalent in dimensions to the GRU). The paper truncates episodes into segments of length 32 or 64 depending on the task, so we let  $t_e = 128$  to ensure that information can persist between segments. Thanks to the determinism provided in the paper, readers should be able to reproduce our exact results using the random seeds 0, 1, 2, 3, 4. We utilize separate memory modules for the actor and critic, as done in the paper. We base our experiments off of the most recent commit at the time of writing, 4d9cbf1.
Model	MMER
MLP	0.067
PosMLP	0.064
FWP	0.112
FART	0.138
S4D	-0.180
TCN	0.233
FrSTACK	0.190
LMU	0.229
IndRNN	0.259
Elman	0.249
GRU	0.349
LSTM	0.255
DNC	0.065
FFM	0.400
Table 4:MMER score comparison from the POPGym paper
E Benchmark Lineplots
We provide cumulative max reward lineplots in Figure 9, Figure 7, and Figure 8 for all the experiments we ran.
Figure 7: SAC lineplots for POMDP-Baselines, where each trial is plotted separately.
Figure 8: TD3 lineplots for POMDP-Baselines, where each trial is plotted separately.
Figure 9: Lineplots for POPGym, where the shaded region represents the  $95\%$  bootstrapped confidence interval.
F POPGym Comparisons by Model
We provide Figure 10, Figure 11, Figure 12 showing the relative FFM return compared to the other 12 POPGym models, including temporal convolution, linear transformers, and more.
Figure 10: Relative POPGym returns compared to FFM.
Figure 11: Relative POPGym returns compared to FFM.
Figure 12: Relative POPGym returns compared to FFM.
G Efficiency Statistic Details
This section explains how we computed the bar plots showing efficiency statistics. We construct the POPGym models and perform one PPO training epoch. The train time metric corresponds to the time spent doing forward and backward passes over the data. One epoch corresponds to the epoch defined in POPGym: 30 minibatches of 65,336 transitions each, with an episode length of 1024, a hidden dimension of 128, and a recurrent dimension of 256. We do this 10 times and compute the mean and confidence interval. Torch GRU, LSTM, and Elman networks have specialized CUDA kernels, making them artificially faster than LMU and IndRNN which are written in Torch and require the use of for loops. We utilize the pure python implementation of these models, wrapping them in a for loop instead of utilizing their hand-designed CUDA kernels. We consider this a fair comparison since this makes the GRU, Elman, and LSTM networks still run slightly faster than IndRNN and LMU (Torch-native RNNs). FFM is also written in Torch and does not have access to specialized CUDA kernels. CUDA programmers more skilled than us could implement a custom FFM kernel that would see a speedup similar to the GRU/LSTM/Elman kernels.
To compute inference latency, we turn off Autograd and run inference for 1024 timesteps, computing the time for each forward pass. We do this 10 times and compute the mean and  $95\%$  confidence interval.
To compute memory usage, we utilize Torch's GPU memory tools and record the maximum memory usage at any point during the training statistic. Memory usage is constant between trials.
For reward, we take the mean reward over all environments, split by model and trial. We then report the mean and  $95\%$  confidence interval over trials and models.