SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show results for PPO, HELM, HELMv2, and SHELM on all Avalon tasks after  $10\mathrm{M}$  interaction steps in Table 2. Dreamerv2 attains the highest score on average after  $10\mathrm{M}$  timesteps, but performs on-par with SHELM. Further, there is no significant difference between the HELM variants though. Since SHELM is demonstrably capable of extracting semantics from Avalon environments (see Figure 12), we believe that the main reason for these results is that the Avalon tasks do not require the agent to use its memory.
To further investigate this finding we run the memory-less PPO baseline for  $50\mathrm{M}$  interaction steps and compare it to results reported in Albrecht et al. (2022) in Table 3. Surprisingly, our PPO baseline significantly outperforms the PPO baseline of Albrecht et al. (2022). Further, we find that our PPO baseline performs on-par with all other memory-based baselines, yielding further evidence that a memory mechanism is not imperative to solve the Avalon tasks.
We provide an additional ablation study in the choice of vision encoder for SHELM. Our qualitative results in Section 3.1 indicated that vision transformer based encoders are better in extracting semantics of synthetic environments. To corroborate this finding, we run an experiment on the MiniGrid-Memory environment where we exchange the ViT-B/16 with a ResNet-50. The results can be observed in Figure 15. We observe that the ViT-B/16 encoder is much better in discriminating between the different objects which results in a substantial improvement over the RN50 encoder.
Figure 11: Episodes sampled for a trained SHELM policy on the MiniGrid-MemoryS11-v0 environment. The object ball is consistently mapped to the token miner, while the object key maps to the token narrow.
F Hyperparameters and training details
Since the memory component of SHELM does not need not be trained, our memory requirements are comparably low. All our experiments were run on either a single GTX1080Ti or a single A100 GPU. The time requirements of our experiments vary for each environment. For MiniGrid and MiniWorld one run takes approximately two hours, while for Psychlab one run requires two days. These experiments were run on a single GTX1080Ti. For Avalon, we used a single A100 for training where one run to train for  $10\mathrm{M}$  interaction steps takes approximately 15 hours.
MiniGrid and MiniWorld We adapt the hyperparameter search from Paischer et al. (2022a). Particularly, we search for learning rate in  $\{5\mathrm{e} - 4,3\mathrm{e} - 4,1\mathrm{e} - 5,5\mathrm{e} - 5\}$ , entropy coefficient in  $\{0.05,0.01,0.005,0.001\}$ , rollout length in  $\{32,64,128\}$  for SHELM. To decrease wall-clock time of HELM variants, we vary the size of the memory register of TrXL such that it can fit the maximum episode length. We lower the number of interaction steps for the gridsearch if we observe convergence before the  $500\mathrm{k}$  interaction steps. If no convergence is observed within the  $500\mathrm{K}$  interaction steps, we tune for the entire duration. We apply the same scheme for tuning the LSTM baseline and tune the same hyperparameters as in Paischer et al. (2022a).
Avalon After visual inspection of token retrievals for observations we found that there is no substantial difference in retrieved tokens for observations in close proximity to each other. Therefore, we introduce an additional hyperparemeter, namely history-frameskip. Note that the history-frameskip is not functionally equivalent to the commonly used frameskip in, e.g., (Mnih et al., 2015). Rather, we actually discard frames within the skip. For example, for a history-frameskip of 10 the agent only
tree
$t = 0$
beach
t=64
dune
t=117
ditch
t=194
fruit
t=260
t=134
t=1242
cliff
t=368
cricket
t=219
sword
t=220
crab
t=232
tree
t=0
bee
$t = 23$
cherry
t=191
berry
t=70
palm
t=342
cliff
$t = 92$
lawn
t=503
frog
ocean
t=5017
turtle
$t = 32$
hole
t=62
stones
t=95
beach
t=258
dune
t=631
door
t=0
lock
t=16
Figure 12: Episodes sampled from a random policy on various Avalon tasks.
ring
t=24
fruit
t=66
observes the first and the eleventh frame in the history branch. The current branch observes every timestep as usual. We search over history-frameskip in  $\{3,5,10\}$  and adapt the memory of the agent to  $\{256,128,64\}$  timesteps respectively. Further we search over learning rate in  $\{2.5e - 4,1e - 4,7e - 5\}$ , and the number of retrieved tokens in  $\{1,2,4\}$ . If an observation is represented as more than one token, this effectively reduces the number of observations that fit into the memory register of TrXL, and thereby introduces a recency bias. Hyperparameters for the Dreamerv2 baseline are taken from (Albrecht et al., 2022). For the memory-less baseline we search over learning rates  $\{5e - 4,2.5e - 4,1e - 4,5e - 5\}$ . We used their respective codebase to run our experiments.
PsychLab Due to the computational complexity of the Psychlab environments we only run a gridsearch over the learning rate in  $\{5\mathrm{e} - 4,3\mathrm{e} - 4,1\mathrm{e} - 4,5\mathrm{e} - 5\}$ . Further we use 64 actors and set the rollout size to 256. For SHELM on continuous-recognition we only retrieve the closest token for an observation.
G Potential negative societal impact
Our method relies on the use of FMs, which are trained on non-curated datasets which were crawled from the web. Therefore, these models readily reflect the biases and prejudices found on the web and, consequently, so might the resulting RL agent trained by our method. Generally, deployment of RL
<sup>5</sup>https://github.com/Avalon-Benchmark/avalon
Figure 13: Sample observations for continuous recognition task of Psychlab. The agent must swipe in a certain direction depending on whether it has encountered an object before in the same episode.
Figure 14: Top-5 token retrievals for objects that are conflated by SHELM at a lower resolution. For higher resolution CLIP successfully maps the objects to different tokens.
Figure 15: Ablation study on the choice of vision encoder used for the retrieval from our semantic database. We report IQM and  $95\%$  CIs across 30 seeds on the MiniGrid-Memory task.
Table 2: Mean human normalized score for all Avalon tasks for PPO, HELMv2, SHELM, and Dreamerv2 after 10M interaction steps. We show mean and standard deviations.
Task	PPO	HELM	HELMv2	SHELM	Dreamerv2
eat	0.700 ± 0.070	0.671 ± 0.071	0.714 ± 0.063	0.693 ± 0.071	0.656 ± 0.077
move	0.312 ± 0.066	0.277 ± 0.068	0.294 ± 0.061	0.291 ± 0.063	0.362 ± 0.073
jump	0.232 ± 0.053	0.232 ± 0.058	0.219 ± 0.051	0.217 ± 0.050	0.233 ± 0.055
climb	0.120 ± 0.042	0.125 ± 0.039	0.211 ± 0.052	0.118 ± 0.038	0.140 ± 0.048
descend	0.203 ± 0.048	0.184 ± 0.044	0.108 ± 0.035	0.202 ± 0.048	0.283 ± 0.089
scramble	0.241 ± 0.056	0.213 ± 0.052	0.301 ± 0.056	0.271 ± 0.058	0.306 ± 0.089
stack	0.094 ± 0.035	0.058 ± 0.029	0.075 ± 0.031	0.115 ± 0.041	0.106 ± 0.041
bridge	0.020 ± 0.017	0.046 ± 0.026	0.040 ± 0.024	0.028 ± 0.018	0.046 ± 0.031
push	0.109 ± 0.039	0.069 ± 0.030	0.069 ± 0.029	0.110 ± 0.044	0.102 ± 0.044
throw	0.000 ± 0.000	0.017 ± 0.020	0.000 ± 0.000	0.000 ± 0.000	0.007 ± 0.011
hunt	0.059 ± 0.029	0.077 ± 0.033	0.091 ± 0.034	0.091 ± 0.039	0.067 ± 0.034
fight	0.179 ± 0.046	0.189 ± 0.048	0.207 ± 0.062	0.185 ± 0.051	0.299 ± 0.070
avoid	0.234 ± 0.048	0.192 ± 0.047	0.211 ± 0.045	0.227 ± 0.050	0.377 ± 0.062
explore	0.172 ± 0.045	0.194 ± 0.051	0.191 ± 0.050	0.204 ± 0.052	0.196 ± 0.051
open	0.086 ± 0.035	0.134 ± 0.042	0.040 ± 0.024	0.102 ± 0.038	0.055 ± 0.031
carry	0.080 ± 0.031	0.101 ± 0.041	0.073 ± 0.030	0.054 ± 0.027	0.087 ± 0.035
navigate	0.006 ± 0.007	0.000 ± 0.000	0.000 ± 0.003	0.006 ± 0.008	0.006 ± 0.008
find	0.005 ± 0.007	0.000 ± 0.000	0.000 ± 0.000	0.000 ± 0.000	0.003 ± 0.004
survive	0.049 ± 0.015	0.066 ± 0.005	0.041 ± 0.013	0.051 ± 0.016	0.033 ± 0.012
gather	0.007 ± 0.006	0.008 ± 0.019	0.009 ± 0.007	0.006 ± 0.003	0.006 ± 0.004
overall	0.146 ± 0.010	0.143 ± 0.010	0.145 ± 0.010	0.155 ± 0.011	0.168 ± 0.012
Table 3: Mean human normalized score over all Avalon tasks after 50M interaction steps. We show mean and standard deviations. Results for PPO, Dreamerv2, and Impala were taken from Albrecht et al. (2022). Asterisk indicates our own results.
Method	Mean Human Normalized Score
PPO	0.165 ± 0.014
Dreamerv2	0.199 ± 0.012
Impala	0.203 ± 0.013
PPO*	0.200 ± 0.012
agents in the real world requires a carefully designed interface to guarantee safe execution of selected actions. That said, we do not expect any negative societal impact from our memory mechanism. Our semantic memory enhances memory-based methods, thus, can yield more insights into why agents make certain decisions.
H Reproducibility Statement
We make all our code and random seeds used in our experiments, as well as obtained results publicly available at https://github.com/ml-jku/helm. The pretrained language encoder, as well as the pretrained CLIP encoder are publicly available on the huggingface hub (Wolf et al., 2020).