Semantic HELM: A Human-Readable Memory for Reinforcement Learning
Authors: Fabian Paischer
Thomas Adler
Markus Hofmarcher
Sepp Hochreiter
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Language Models
History Compression
Partial Observability
Foundation Models
Interpretability
Explainable AI
TL;DR: We introduce SHELM, a semantic and human-readable memory for Reinforcement learning and showcase its strengths on several memory-dependent environments.

Abstract: Reinforcement learning agents deployed in the real world often have to cope with partially observable environments. 
Therefore, most agents employ memory mechanisms to approximate the state of the environment. 
Recently, there have been impressive success stories in mastering partially observable environments, mostly in the realm of computer games like Dota 2, StarCraft II, or MineCraft. 
However, existing methods lack interpretability in the sense that it is not comprehensible for humans what the agent stores in its memory.
In this regard, we propose a novel memory mechanism that represents past events in human language.
Our method uses CLIP to associate visual inputs with language tokens. 
Then we feed these tokens to a pretrained language model that serves the agent as memory and provides it with a coherent and human-readable representation of the past.
We train our memory mechanism on a set of partially observable environments and find that it excels on tasks that require a memory component, while mostly attaining performance on-par with strong baselines on tasks that do not. 
On a challenging continuous recognition task, where memorizing the past is crucial, our memory mechanism converges two orders of magnitude faster than prior methods.
Since our memory mechanism is human-readable, we can peek at an agent's memory and check whether crucial pieces of information have been stored.
This significantly enhances troubleshooting and paves the way toward more interpretable agents.

Semantic HELM: A Human-Readable Memory for Reinforcement Learning
Fabian Paischer Thomas Adler ', Markus Hofmarcher's 2, Sepp Hochreiter ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, 2 JKU LIT SAL eSPML Lab, Institute for Machine Learning, Johannes Kepler University, Linz, Austria paischerOml j.u. at
Abstract
Reinforcement learning agents deployed in the real world often have to cope with partially observable environments. Therefore, most agents employ memory mechanisms to approximate the state of the environment. Recently, there have been impressive success stories mastering partially observable environments, mostly in the realm of computer games like Dota 2, StarCraft II, or MineCraft. However, existing methods lack interpretability in the sense that it is not comprehensible for humans what the agent stores in its memory. In this regard, we propose a novel memory mechanism that represents past events in human language. Our method uses CLIP to associate visual inputs with language tokens. Then we feed these tokens to a pretrained language model that serves the agent as memory and provides it with coherent and human-readable representation of the past. We train our memory mechanism on a set of partially observable environments and find that it excels tasks that require memory component, while mostly attaining performance on-par with strong baselines on tasks that do not. On a challenging continuous recognition task. where memorizing the past is crucial, our memory mechanism converges two orders of magnitude faster than prior methods. Since memory mechanism is numan-readable, we can peek check whether crucial pieces of information have been stored. This significantly enhances troubleshooting and paves the way toward more interpretable agents.
agent's memory and
Introduction
In reinforcement learning (RL) an agent interacts with an environment and learns from feedback provided in the form of a reward function. In many applications, especially in real-world scenarios, the true state of the environment is not directly accessible to the agent, but rather approximated via observations that reveal mere parts of it. In such environments, the capability to approximate the true state by virtue of an agent's perception is crucial (Åström, 1964; Kaelbling et al., 1998). To this end, many approaches track events that occurred in the past. The brute-force strategy is to simply store all past observations. However, this often infeasible and is much more efficient to store more abstract representations of the history. Thus, many RL algorithms use memory mechanisms such as LSTM (Hochreiter & Schmidhuber, 1997) or Transformer (Vaswani al., 2017) compress sequences of high-ding -ith-nmmnniionll observations. This has led to impressive successes mostly in the realm of mastering computer games on a human or even super-hure hhmann level. Some examples are Dota (Berner et al., 2019), StarCraft II (Vinyals et al., 2019), or MineCraft (Baker et al., 2022; Patil et al., 2022).
Most state-of oh-the-art methods dealing with partial observability in RL employ a memory mechanism that is intelligible for humans. In this regard, we draw inspiration from the semantic memory
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Memory
A gent
Environment
fiuit
sand
frog
Eron Eruit
Gree bee
door knob
beach
bill
loke
0g F
1=100
5-220
O:
Figure 1: We add semantic and human-readable memory to agent tackle partially observable RL tasks. We map visual observations On to the language domain via CLIP retrieval. The memory component, a pretrained language encoder, operates text only and compresses history of tokens into a vector hy The agent takes an action a, based on the current observation o, and the compressed history h4.
present humans (Yee et al., 2017) and propose represent past events in human language. Humans memorize abstract concepts rather than every detail of information they encountered in the past (Richards Frankland, 2017; Bowman & Zeithamova, 2018). Their ability abstract is heavily influenced by the exposure to language in early childhood (Waxman & Markow, 1995). Further, humans use language on a daily basis to abstract and pass on information. Therefore, language is natural choice representation for compounding information and has the key advantage of being human-readable. This enables analyzing critical pieces of information have entered the memory or not. Based on this data, it becomes clear which parts of the system require refinement. Moreover, natural language has been shown to be effective as compressed representation of past events RL (Paischer et al., 2022a).
Our proposed method, Semantic HELM (SHELM), leverages pre-trained foundation models to construct memory mechanism that does not require any training. We use CLIP (Radford et al., 2021) to associate visual inputs with language tokens. Thereby, the vocabulary of the CLIP language encoder serves as a semantic database of concepts from which we retrieve the closest tokens to given observation. These tokens are passed to a pretrained language model that serves as memory and provides the agent with coherent representation of the past.
We illustrate the benefits of a human-readable and semantic memory in partially observable RL problems, First, we conduct a qualitative analysis on whether CLIP vision encoder is capable of extracting semantics out of synthetic environments. Then, we test SHELM on a partially observable 2D MiniGrid (Chevalier-Boiswel 2018) environments. We find that even though these environments are only partially observable, they often do not necessitate memory mechanism as they are solvable by memory-less policy. The MiniGrid- Memoory task, however, clearly requires memory and SHELM reaches state-of-the-an performance. On more realistic 3D environments such as Avalon (Albrecht et al., 2022) and Psychlab turn,
et al., 2018), and 3D MiniWorld (Chevalier-Boll Boisvert,
(Leibo et al., 2018), SHELM successfully assesses the semantics of visual observations. requires approximately 100 times fewer interaction steps than prior methods on Psychlab's 's continuous recognition task that explicitly evaluates for memory capacity. Avalon, we find that the addition of our semantic memory performs on-par with the current state-of- the-art art for limited interaction budget, while adding interpretability to the memory.
2 Methods
Our goal is to express visual observations in language space such that the resulting representations become comprehensible for humans. To this end, we instantiate mapping from images to text in the form of pretrained components which entirely frozen during training. This way the available computational resources are invested into performance rather than interpretability. Before we describe SHELM, we briefly review the HELM framework, which serves as a starting point for our work.
2.1 Background
HELM (Paischer et al., 2022a) was proposed as a framework for RL in partially observable environ- ments. It utilizes pretrained language model (LM) memory mechanism that compresses past

---

top-k
a
0,3 crab accan
Actor Cribic Head
0.05 seatood
h
Similarity
CLIP
LM
A screen showing a {
CNN
CLIP VM
VIXI
VeLIP ience
while tree
episode
house cat
projec
crab
This
(a)
(b)
Figure 2: Architecture of SHELM. We compile semantic database S by encoding prompt-augmented tokens from the overlapping vocabularies of CLIP and the LM (a). Given an observation o, we retrieve the top-k embeddings present in S and select their corresponding text tokens (b). These tokens are passed to the LM which represents the memory module of SHELM (c). Ct-1 represents the memory cache of the LM which tracks past tokens.
observations. To map environment observations o, E IR" to the LM, it introduces the FrozenHop- field (FH) mechanism, which performs randomized attention E = (ei..... ...) E R of the LM, where & is the vocabulary size and m is the embedding dimension. Let P E mR be random matrix with entries sampled independently from (O.,n/m) The FH mechanism performs
pretrained token embeddings
It = E ( EP.))
(1)
where B is scaling factor that controls the dispersion of at within the convex hull of the token embeddings. This corresponds to spatial compression observations to mixture of tokens the LM embedding space. Since P is random, the association of observations e, is arbitrary, i.e., not meaningful. That is, the FH mechanism does not preserve semantics. For temporal compression, HELM leverages pretrained LM. At time HELM obtains compressed history representation by
with token embeddings
h LM(c-1, x.)
where e₁ represents the context cached in the memory register of the LM up to timestep t.
More recent work has shown that the FH mechanism is prone to representation collapse if observations visually similar to each other (Paischer et al., 2022b,c). They propose new method, namely HELMv2, which substitutes the random mapping with pretrained CLIPP encoder. Subsequently, they adopt a batch normalization layer (loffe & Szegedy, 2015) with fixed shifting and scaling parameters to transfer the image embedding to the language space. Consequently, HELMv2 computes the inputs the LM
D BNg= (CLIPvM(O.)).
where CLIP VM denotes the CLIP vision model and bg and ogg denote mean and standard deviation of the embedded vocabulary E. This effectively fits the statistics of the image embeddings to those of the LM embeddings. Since the embedding spaces of CLIP and the LM were trained independently they are not semantically aligned. Therefore, also HELMv2 fails to preserve semantics of observations and, consequently, the memory mechanism of HELMv2
human-readable.
2.2 Semantic HELM
Semantic HELM (SHELM) inherits the high-level architecture from HELM but introduces some changes to the memory module. Similarly to HELMv2, we also use CLIP to embed environment observations. However, we replace the batch normalization layer of HELMV2 with token-retrieval mechanism. We retrieve tokens that are similar to an observation in CLIP space and pass them the LM in the form of text they be regarded as textual descriptions of environment observations.

---

In a first step, we determine the overlap of the token vocabularies of CLIP and the LM. This is necessary to (i) to control for the number of tokens the LM receives per observation, and (ii) avoid loss of information due to different tokenizers used by CLIP and the LMM Thereby, we obtain a set of tokens V. Since CL.IP was pretrained on image-caption pairs, we augment each token u E with a set of pre-defined prompts P = {P1. P and can be designed to specifically target certain aspects of the different environments. We embed a token v in the CLIP output space by computing the average embedding of its prompt augmentations. That is, we define the function
P2,... }r
The contents of P are hyperparameters of our method
embed(v) = (PI
Σ PEP
(concat(p, v))-
We do this for every v E V. which results in set S of CLIP-embedded tokens
S
(embed(v))
(5)
We denote by arg max* extension of the arg max operator that returns the subset of the k largest elements in a set. For each observation we retrieve the most similar tokens in terms of cosine similarity by
S = arg max cossim(s, CL))) sES
(6)
where
y
cossim y) x y/l
Note that & = S is another hyperparameter of our method, namely the number of tokens that represent an observation. Effectively, & controls the degree of compression in the memory.
Finally, we embed single tokens a corresponding to the set of tokens in S in the LM embedding space and pass them to the LM. In this manner, provide the LM with textual representation of the current observation. While HELM and HELMv2 also leverage the latent structure of a pre-trained LM, they do not explicitly represent the observations in the form of text, thus, do not provide human-readable form of past observations. Another improvement over HELMv2 is that SHELM removes the restriction of HELMv2 that the embedding spaces of CLIP and LM must have the same dimensionality. In turn, any CLIP--lke encoder can be used as semantic extractor for SHELM. Figure shows an illustration of the methodology of SHELM.
3 Experiments
First, we investigate in Section 3. whether CLIP can extract semantics of artificial scenes. Next, we train SHELM on four different environments, namely MiniGrid (Chevalier-Boisver) et al., 2018), MiniWorld (Chevalier-Boise)
2018), Avalon (Albrecht et al., 2022), and Psychlab (Leibo et al., 2018). We compare SHELM to HELMv2, LSTM (a recurrent bascline based on the LSTM archi- tecture), and the popular 2reamerv2 (Hafner et al., 2021) and Dreamerv3 (Hafner et al., 2023). We show that semantic memory boosts performance in environments that are both heavily dependent on memory and photorealistic. Finally, in Section 3.6 we perform ablation studies on the benefit of semantics and the trade-off between interpretability and performance.
We train all HELM variants and LSTM with Proximal Policy Optimization Schulachlmann et al., 2017) on RGB observations. Following Paischer et al. (2022a), we instantiate the LM with pretrained TransformerXI. (TrXI., Dai et al., 2019) model. For training Dreamerv2 and Dreamerv3, we use the respective 2odebases and train on RGB observations. We report results via IQM (Agarwal et al., 202.) and 95% bootstrapped confidence intervals (CIs) unless mentioned otherwise. We follow (Colas et al., 2019) and perform a Welch's t-test with reduced significance level of 0.025 at the end of training to test for statistical significance. We elaborate on the architectural design and hyperparameter sweeps in Appendix F.
Following the original implementation at notebooks/Prompt. Bww.ramtt..gineeiing.... for ftreerrggngnrrtmageeeeeee....e ipynb
Dreameer22 Dreamerv3; https:/ /ttbtttt.../..e///g///etge...t wwwt//onwtaaa///ccctcc daanerv3

---

Desameee2
PPO
1.5
MinCnd
Vantryy
MinlWordd
0.3
8.6
Accumuaaed
04
02
6
100000
200000
200000
450000
500000
0.00
100000
200000
400000
B00000
Number of Imtrrciionn Steps
Number
Number Imaracionn Stepe
Figure 3: Left: on the MimmrriddMemoor task, Right: on eight Worl tasks. We report IQM and 95% CIs across 30 seeds for each method.
Accumulated reward for different methods on six MiniGrid environments, Middle:
3.1 Extracting semanties of virtual scenes
First, we analyse whether CLIP vision encoders are able to extract semantics from artificial scenes that are typically encountered in RL environments. We compare the two smallest ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2021) architectures, namely RN50 and ViT-B/16, since those add the least amount of computational overhead SHELM. In this regard, we look at observations sampled via a random policy, provide them to the CLIP vision encoder and retrieve the closest tokens in the CLIP embedding space as explained in Section 2.2. We find that the retrieval of tokens strongly varies between vision encoder architectures for MiniWorld and Avalon (see Figure 9 and Figure in the appendix). The differences are especially prominent for MiniWorld environments, where the ViT-B/16 encoder recognizes shapes and colors, whereas RN50 ranks entirely unrelated concepts highest. More photorealistic observations from the Avalon benchmark show similar, but less pronounced trend. We also observe strong bias toward abstract tokens such as screenshot, biome, or render. We alleviate the retrieval of these tokens by including them in our prompts for retrieval. Therefore, instead of using the prompt "An image of a <tok>", we consider prompts such as "A screenshot of a <tok>", or "A biome containing a <tok>", where <tok> stands for token in V. We only use prompts for Avalon and Psychlab, since the effect of this prompting scheme was negligible on the other environments. Table 1 in the appendix features the full list of prompts for the two environments. We also add an analysis BLIP-2 (Li et al., 2023) in Appendix B. We find that, while BLIP-2 correctly recognizes shapes, colors, and objects, it lacks accuracy in their compositionality. Based on this analysis ViT-B/16 encoder in combination with our designed prompts as semantics extractor for SHELM.
using off-the-shell captioning engines, such as
the
3.2 MiniGrid
We compare all methods on set of six partially observable grid-world environments as in as (Paischer et al., 2022a). Additionally, we train on the MiniGrid-Memory to as Memory. The Memory task requires the agent to match the object in the starting room to one of the two objects at the end of corridor. The agent's 's view is restricted so cannot observe both objects at the same time. Therefore, it needs to memorize the object in the starting room. If the agent chooses the wrong object at the end of the corridor, it receives no reward and the episode ends.
1-v0 environment, which we refer
Figure (left and middle, respectively) shows the results across the six MiniGrid environments and the Memory environment. On the MiniGrid environments, Dreamerv3 excels and converges much faster than any other method. However, final performance is approximately equal for Dreamerv3, HELMv2, and SHELM. Interestingly, although all MiniGrid environments are partially observable, they are solvable with a memory-less policy. In the Memory environment, SHELM performs on par with Dreamerv3 but exhibits faster convergence and more stable training. Figure 11 visualizes the tokens passed to the memory of SHELM after sampling episodes from trained policy. SHELM primarily maps the ball to the token miner, and the key to the token narrow. Although the retrieved tokens do not represent the semantically correct objects in the observations, they are still stored as two different language tokens in the memory. This enables SHELM to discriminate between them and enables faster learning mirrored in the learning curves. The LSTM bascline suffers from poor
5

---

Iree
beach
dume
ditch
fruit
clill
Method
Mean Human Normalized Score
HELM
0.143 0.010
HRLMVv2
0.145 + 0.010
L-368
PPO
0,146 + 0.010
tree
cherry
palm
lawn
frog
ocean
SHELM
0.155 + 0.011
Dreamerv2
191
L-342
L-5017
Figure 4: Left: Mean and standard deviation of human normalized score across all tasks the Avalon test worlds after 10 M timesteps. Right: Two sample episodes and their corresponding token mappings for episodes sampled from Avalon with random policy.
sample efficiency and does not learn to utilize its memory within the budget M interaction steps. LSTM requires approximately 5 M steps to solve the task.
3.3 MiniWorld
The MiniWorld benchmark suite provides set of minimalistic 3D environments. In line with Paischer et al. (2022b), we select eight different tasks from MiniWorld and train our method and all baselines on those. The tasks comprise collecting and placing objects, finding objects, or navigating through mazes. A detailed description of each task can be found Appendix A. Prior work on the MiniWorld benchmark mostly used single environments evaluate specialized approaches (Liu et al., 2021; Venuto et al., 2019; Zha et al., 2021; Khetarpal et al., 2020), or handcraft novel environments (Sahni et al., 2019; Hutsebaut ButssbbautBBuysee et al., 2020). To the best of our knowledge, HELMv2 is the only method that was evaluated on all eight tasks and has been state of the art so far (Paischer et al., 2022b).
Figure (right) shows the results for all methods. Interestingly, SHELM reaches performance on par with HELMv2, even though semantics can be extracted from the 3D observations to a certain degree (as explained in Section 3.1). Further, Dreamerv2 outperforms both and reaches state-of-the- performance. Surprisingly, Dreamerv3 attains significantly lower score and performs on par with HELM and the memory- PPO. This might be due to suboptimal hyperparameters, even though Hafner et al. (2023) claim that the choice of hyperparameterss transfers well across environments. LSTM again suffers from poor sample efficiency reaching the lowest scores out of all compared methods. Finally, we again observe that PPO can in principle solve all tasks, which yields further evidence that partial observability does not automatically imply necessity for memory. Our results suggest that memory mechanism result in enhanced sample efficiency (SHELM PPO). However, memory is not imperative to solve these tasks.
3.4 Avalon
Avalon an open-ended 3D survival environment consisting of 16 different tasks. The aim for the agent is survive long possible by defending against predators, hunting animals, and eating food order to restore energy. An episode ends if the agent has no energy left. This can happen if the agent receives environmental damage (e.g. from falling), being killed by a predator, does eat frequently. The agent receives a dense reward as the difference in energy between consecutive timesteps. Additionally, it receives delayed reward upon successful completion of task, e.g., eating food. The observation space are RGBD images as well as proprioceptive input that comprise, e.g., the agent's 's energy. We adopt the same architecture and training strategy as Albrecht al. (2022) for all HELM variants. Specifically, we add the history compression branch to the baseline PPO agent and train on all 16 tasks including their difficulty curriculum. The history compression branch is only based on RGB images and does not receive the proprioceptive state. We also train Dreamerv2, and a memory-less baseline (PPO), which is identical the PPO baseline in Albrecht et al. (2022). Due to computational constraints and since we observed superior performance of Dreamerv2 on 3D environments, we neglect Dreamerv3 and LSTM and train on a limited budget of 10 M interaction steps. We elaborate on training details and hyperparameter selection in Appendix F. The final performance of an agent is measured in terms of mean human normalized scores on curated set of 1000 test worlds.

---

Continuous Recognition
dog
kettle
Accumulated
20
SHELM
keys
flag
HELM22
10
Dreamen2
0.0
Number of Interaction Steps
Figure 5: Performance for all methods CR task from Psychlab. We report IQM and 95% CIs across 5 seeds (left). Observation and corresponding tokens for SHELM on CR environment (right).
The results are shown in Figure 4, left. For detailed results per task see Table 2 in the appendix. SHELM and Dreamerv2 yield the highest performance on average after 10 interaction steps. However, the difference to the memory-less PPO is not statistically significant. To further investigate this finding we train PPO for M0M interaction steps and compare Albrecht et al. (2022) in Appendix D. Indeed, we find that our memory- baseline attains scores on-par with memory-bar -based approaches trained for M interaction steps. This yields further evidence that Avalon does not necessitate the addition of memory mechanism. Finally, we can glance at SHELM's s memory to identify failure cases. We show observations of two episodes and their token correspondences for SHELM in Figure 4, right. The observations are mostly mapped to semantically meaningful tokens that represent parts
the baseline results reported in
image observations. However, we also observe some cases where CLIP retrieves tokens that are semantically related, but not correct, which we show in Figure 12 in the appendix.
3.5 PsychLab
The Psychlab environment suite (Leibo et al., 2018) consists of 8 complex tasks and designed to isolate various cognitive faculties of RL agents including vision and memory. The continuous recognition (CR) task of Psychlab was specifically designed target the memory capacity of an agent. In this task the agent is placed in front of a monitor which displays a stream of objects. The agent then needs to identify whether has already seen a particular object by swiping either left or right. If the agent correctly identifies an object it receives a reward of +1. A policy that always swipes one direction achieves reward of around 30, which we refer to the random baseline. Every reward higher than that indicates that an agent actually utilizes its memory. This task is ideal evaluate the memory capacity of an agent, since episodes last for about 2000 timesteps and usually require remembering up to 50 different objects.
We train all methods for 10 M interaction steps on the CR task (Figure 5). We neglect the memory- less baseline because this task is unsolvable without memory. SHELM indeed learns to effectively utilize its memory within 10M interaction steps and significantly outperforms all competitors. Other approaches require interaction steps in the range of billions until convergence (Parisotto et al., 2020; Fortunato al., 2019). These works report human normalized scores which are not publicly available. therefore we cannot compare SHELM to those. HELMv2 is the only competitor that attains performance better than random. but reaches significantly lower scores than SHELM. Surprisingly, both variants of Dreamer do not exceed performance of the random baseline. Finally, we inspect the memory of SHELM and show the token mappings that are passed on to the memory module for some sample observations in Figure 5. In most cases SHELM assigns semantically correct tokens to the displayed objects. However, we also show some cases where the token retrieval of SHELM conflates different objects in Figure 13. We find that this can mostly be attributed to the low resolution of observations. S can recover from these failure cases when using higher resolutions (see Figure 14).
7

---

3.6 Ablation studies
Are semantics important? We slightly alter the HELMv2 implementation from (Paischer et al., 2022b,c) by retrieving the closest tokens in the language space after their frozen batch normalization layer, and finally passing those to the LM. This setting is similar to SHELM in that the LM receives tokens in the form of text. However, semantics are not preserved since visual features are merely shifted the language space. We call this setting HEL.Mv2-Re and train on the Memory envi- ronment (see Figure 6). We find that if the mapping from observation to language is arbitrary, the performance decreases drastically.
Is it important to learn task-specific features? SHELM would be even more interpretable if not only the memory module but also the branch processing the current observation (CNN in Figure 2) could utilize language tokens. Therefore, we substitute the CNN with CLIP vision encoder (SHELM-CLIP) An important consequence of this methodological change is that even features from the current observation be interpreted by retrieval mechanism. However, Figure 6 suggests that it is vital to learn task-specific features.
Should the history branch operate on task- specific features? To answer this question we substitute the CLIP encoder in the history branch with the CNN encoder learned at the current timestep (SHELM-CNN). An immediate draw- back of that is the loss of readability, since we do not represent the observations in text form anymore. Further, we again observe a drastic drop in performance. The reason for this that the task-specific features must be learned before they actually provide any benefit to the history branch. Thus, we prefer to keep abstract seman- tic features in text form since that does not re- quire any training and has the additional benefit of being human-readable.
Memory
1.0
0.8
Accumulated Reward
0,6
0.4
SHELM
SHELM-CLIP
HELMv2-Ret
0.2
SHELM-LSTM
SHELM-CNI
0.0
Random
0.00
0.25
0.50
0.75
1.00
1.25
1.50
2,00
1e6
Is pretrained language encoder required? We replace the LM with an LSTM operating on the text tokens (SHELM-LSTM), This setting resulted in performance equivalent to randomly choosing an object at the end of the corridor. However, we believe that after longer training SHELM-LSTM can eventually learn to solve the task, since the LSTM baseline also solved the task after longer trainng. Thus, the simpler and more sample efficient method is to maintain the frozen pretrained encoder instead of learning a compressed history representation.
Number Interaction Steps
Figure 6: Ablation study on the effect of semantics, the influence of task-specific features in the history and the current timestep, and the importance of the pretrained LM. We report IQM and 95% CIs across 30 seeds on the MiniGrid-Memon task.
We show additional results for an ablation study where we exchange the ViT-B/16 vision encoder with RN50 in Appendix E. SHELM appears to be quite sensitive to the selection of vision encoder, which corroborates our qualitative findings in Section 3.
Related work
RL and partial observability RL with incomplete state information can necessitate a memory for storing the history of observations an agent encountered. However, storing the entire history is often infeasible. History Compression tries to answer the question of what information store in stream of observations (Schmidhuber, 1992; Schmidhuber et al., 1993; Zenke et al., 2017; Kirkpatrick et al., 2016; Schwarz et al., 201 8; Ruvolo Eaton, 2013). A plethora of prior works have used history compression to tackle credit assignment (Arjona-Medina et al., 2019; Patil er al., 2022: Widrich al., 2021; Holzleitner et al., 2021), and partial observability (Hausknecht Stone, 2015; Vinyals al., 2019; Berner et al., 2019; Pleines et al., 2022). The memory maintained by an agent can either
8

---

be external (Hill et al., 2021; Wayne et al., 2018), or directly integrated into the feature extraction pipeline via the network architecture. An orthogonal approach for history compression is training recurrent dynamics models (Ha & Schmidhuber, 2018 8; Pasukonis et al., 2022; Hafner et al., 2020, 2021). We believe language very well suited as medium for compression to summarize past events, as suggested by Paischer et al. (2022a).
Language in RL Language provides useful abstractions for RL. These abstractions can be leveraged to represent high-level skills (Sharma et al.. 2022; Jiang et al.. 2019; Jacob et al., 2021). LMs have been used to improve exploration in text-based environments (Yao al., 2020), or in visual environments via a language oracle (Mu et al., 2022). Pretrained vision-language models (VLMs) provide abstract embedding spaces that can be used for exploration in visual environments (TTam et al., 2022). We leverage VLMs to spatially compress visual observations to language tokens. Furthermore, pretrained LMs were leveraged for (i) initializing policies in text-based environments (Li et al., 2022), (ii) grounding to various environments (Andreas et al., 2018; Huang et al., 2023; Carta et al., 2023; Hill et al., 2021), I) (iii) sequence modeling in the offline RL setup (Reid et al., 2022), and (iv) generating high-level plans (Huang et al., 2022a.b; Wang et al., 2023; Singh et al., 2022; Ichter et al., 2022; Liang et al., 2022a; Dasgupta et al., 2023; Du et al., 2023; Shah al., 2022: Ahn et al., 2022; Zeng et al., 2022). Other works train agents to generate descriptions of virtual scenes (Yan al., 2022), or thought processes of humans (Hu & Clune, 2023). Additionally, language has been used for augmenting the reward function (Wang al., 2019; Bahdanau et al., 2019; Goyal et al., 2019; Carta et al., 2022; Kwon et al., 2023), or learning a dynamics model (Zhong et al., 2022, 2021, 2020; Wu et al., 2023). To manage language- based memory, Park et al. (2023) stores record of an agent's 's memory which comprises different levels of abstraction. Importantly, all agent-environment interactions in their work are scripted, while our agent enables memory based on language for visual inputs.
Language for interpretability In the realm of supervised learning, plethora of prior works had used language as human-interpreetbble medium to explain classification decisions in computer vision (Hendricks et al., 2016, 2018; Park al., 2018; Zellers et al., 2019; Hernandez et al., 2022), or in natural language processing (Andreas & Klein, 2017: Zaidan & Eisner, 2008; Camburu et al., 2018; Rajani et al., 2019; Narang al., 2020). Interpretability methods in RL are scarce and usually follow a post-hoe approach (Puiutta & Veith, 2020). Intrinsically interpretable models are designed to be inherently interpretable even during training time and preferable over post-hoc approaches (Rudin et al., 2021). They often restrict the complexity of the model class, which in turn results in reduced performance of the agent. Therefore, (Glanois et al., 2021) propose adopt modular approach to interpretabiliiyy To this end, our work focuses on the memory module. This enables us to provide some extent of intrinsic interpretability while exceeding performance of existing (non-interpretable) methods on tasks that necessitate memory.
Foundation models The advent of the Transformer architecture (Vaswani et al., 2017) gave rise to foundation models (FMs, Bommasani et al., 2021), such as GPT-3 (Brown et al., 2020). As shown by Petroni et al. (2019); Talmor et al. (2020); Kassner al. (2020); Mahowald al. (2023), pretrained LMS can learn abstract symbolic rules and show sparks of reasoning. We leverage their abstraction capabilities for history compression in RL. Further, vision FMs have been demonstrated to be well adaptable to foreign domains (Adler et al., 2020; Evci et al., 2022; Ostapenko et al., 2022; Parisi al., 2022). Our approach combines language-basee FMs with vision-langaus CLIP (Radford et al., 2021) or ALIGN (Jia et al., 2021). We use CLIP to obtain language tokens that semantically correspond to concepts present in synthetic environments.
models, such as
5 Limitations
Token-level abstraction One potential shortcoming of our method is that our retrieval is based on single tokens. However, we have shown that for environments where one or a few objects are important, this is sufficient. Further, our approach is very flexible and the semantic database can easily be augmented with more detailed descriptions of objects and their relationships. We aim to investigate more in this direction in future work.

---

Wall clock time A current limitation of SHELM is wall clock time. The rollout phase is particularly expensive since each timestep needs be propagated through the LM. Despite that, it is still more efficient than, e.g., Dreamerv2. This is due to the fact that the memory mechanism is kept frozen and need not be updated durint the training phase. potential solution for decreasing the complexity of the rollout phase would be distillation of the LM into smaller actor networks as in (Parisotto & Salakhutdinov, 2021).
Modality gap Our semantic mapping is limited by the inherent ability of CLIP vision encoders to extract semantically meaningful features in synthetic environments. However, CLIP suffers from the modality gap (Liang et al., 2022b), i.c., a mis-alignment between image and text embedding spaces. In the future, aim incorporating methods that mitigate the modality gap (Fürst et al., 2022; Ouali et al., 2023).
Distribution shift We use a pretrained CLIP model to retrieve tokens that are similar to visual observations. However, CLIP was pretrained on large-scale data crawled from the web. We have shown that it can still extract semantics of synthetic environments when those are sufficiently photorealistic, i.e., MiniWorld, Avalon, Psychlab. Further, prompting can enhance the quality of the retrieval. For environments such as MiniGrid, the retrieval yields tokens that do not correspond to aspects in the image. However, Gupta al. (2022) has shown that CLIP can handle task-specific texts for MiniGrid, thus, we would like to investigate the effect of augmenting our semantic database with such texts in the future.
6 Conclusion
In many real-world scenarios an agent requires a memory mechanism to deal with partial observability. Current memory mechanisms in RL act as black box where it is not comprehensible for humans what pieces of information were stored. To solve this problem, we proposed new method called Semanic HELM that represents past events in form of human-readable language by leveraging pretrained vision-language models. We showed compelling evidence that even for synthetic environments our memory mechanism can extract semantics from visual observations. Further, SHELM outperforms strong baselines on photorealistic memory-dependen environments through its human-readable memory module. In cases where SHELM fails to extract semanties from observations, we can investigate the cause by inspection of the memory module. Even in such cases, SHELM mostly performs on-par with other memory-based approaches.
We believe that we can further enhance our method by generating full captions from history obser- vations instead of only small number of words. This could enable (i) a long-term memory that textually summarizes all captions of corresponding observations, (ii) a potential for planning in form of text from visual observations similar to Patel et al. (2023), and (ii) modeling dynamics of an environment in language space.
Acknowledgements
The ELLIS Unit Linz, the LIT Ll the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects AI-MOTION (LIT-2011-6-YOU--22) DeepFlood (LITT- YOU-2-3 Medical Cognitive Computing Center (MC3), INCONTROL-RI (FFG-88 064),, PRIMAL (FFG-873979), S3AI (FFG-872172) DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), ALAGreenHCcatinGrrdstFeC 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stters (HORIZON- 16--022-CLLMMATE--000
We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), University SAL Labs initiative, FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation. We acknowledge FuroHPC Joint Undertaking for awarding us access to Karolina at IT4Innvvtiions (Czech Republic) and MeluXina at LuxProvide (Luxembourg).
10

---

References
Adler, T., Brandstetter, J., Widrich, M.. Mayr, A., Kreil, D. P., Kopp. M., Klambauer, G., and Hochreiter, S. Cross-Domain Few-Shot Few-ShottLerning Learning by Representation Fusion. CoRR, abs/2010.0644988 2020. arXiv: 2010.06498.
Agarwal, R.,. Schwarzer, M., Castro, P.S. Courville, A. C., and Bellemare, M. G. Deep Reinforcement Learning at the Edge of the Statistical Precipice. Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 19304-293200 2021.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J.. Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., Lee, K.-H., Levine, S., Lu. Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J.. Reyes, D., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., Yan, M., and Zeng, A. Do as can, not as say: Grounding language in robotic affordances, 2022.
Albrecht, J.. Fetterman, A. J., Fogelman, B., Kitanidis, E.,, Wróblewski, B., Seo, N., Rosenthal, M., Knutins, M., Polizzi, Z., Simon, J. B., and Qiu, K. Avalon: A benchmark for RL generalization using procedurally generated worlds. CoRR, abs/2210.13417, 2022. 13417.
doi: 10.48550/arXiv22210
Andreas, J. and Klein, D. Analogs of linguistic structure in deep representations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2893-2897, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 18653/v1/DI7-1311
Andreas, J., Klein, D.,, and Levine. S. Learning with Latent Language. In Walker, M. A., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pp. 2166-2179. Association for Computational Linguistics, 2018. doi: 10. 10.18853/v1/118-11
Arjona-Medina. J. A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., and Hochreiter, S. RUDDER: Return Decomposition for Delayed Rewards. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13544-13555 2019.
Bahdanau, D., Hill, F., Leike, J., Hughes, E., Hosseini, S. A., Kohli, P., and Grefenstette, E. Learning to Understand Goal Specifications by Modelling Reward. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Baker, B., Akkaya, I., Zhokhov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (VPT): learning to act by watching unlabeled online videos. CoRR, abs/2206. 11795, 2022. doi: (0.48550/arxi..220.... 11795.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C.,, Pachocki, J., Petrov, M., Pinto, H. P. d. O., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S. Dota 2 with Large Scale Deep Reinforcement Learning. CoRR, 2019.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., Arx, S. v., Bernstein, M. S., Bohg.
J.. Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E.. Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N. D., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, W., Krass, M. S., Krishna, R., Kuditipudi, R., and al, e. On the Opportunities and Risks of Foundation Models. CoRR, abs/2108 07255888 2021.
arXiv: 2 2008.0 008.07258.
11

---

Bowman, C. R. and Zeithamova, D. Abstract Memory Representations in the Ventromedial Prefrontal Cortex and Hippocampus Support Concept Generalization. Journal of Neuroscience, 38(10): 2605-2614, 2018. ISSN 0270-6474. doi: 10. 10.152//NFUROSC.288111177
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P.. Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C.,, Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Camburu, O., Rocktäschel, T., Lukasiewicz, T., and Blunsom, P. e-snli: Natural language inference with natural language explanations. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9560-9572, 2018.
Carta, T., Lamprier, S., Oudeyer, P., and Sigaud, O. EAGER: asking and answering questions for automatic reward shaping in language-guice guided RI. CoRR, abs/2206.096744 2022. doi: 10.48550/ arXiv.2206.09674.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P. Grounding large language models in interactive environments with online reinforcement learning. CoRR, abs/2302.00662. 2023. doi: 10.48550/nxiv.2332.000
Chevalier-ooisvertt M. MiniWorld: Minimalistic 3D Environment for RL & Robotics Research, 2018. Publication Title: GitHub repository.
Chevalier- Boiivver, M., Willems, L., and Pal, S. Minimalistic Gridworld Environment for OpenAI Gym, 2018 Publication Title: GitHub repository.
Colas, C., Sigaud, O., and Oudeyer, P. A hitchhiker's guide to statistical comparisons of reinforcement learning algorithms. In Reproducibility in Machine Learning, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net, 2019.
Dai. Z., Yang. Z., Yang, Y., Carbonell, J. G., Le, Q. V., and Salakhutdinov, R. Transformer-XL. Attentive Language Models beyond a Fixed-Length Context. In Korhonen, A., Traum, D. R., and Mårquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp- Association for Computational Linguistics, 2019. doi: 10. 10.18653/v1/p19-1288
Dasgupta, I., Kaeser-Chen, C., Marino, K., Ahuja, A., Babayan, S., Hill, F., and Fergus, R. Col- laborating with language models for embodied reasoning. CoRR, abs/2302.00763, 2023, doi: 10. 00448555WarXii.22220077666
Deng, J., Dong, W., Socher, R., R., K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp- 248-255, 2009. doi: 10.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Du, Y., Watkins, O., Wang. Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. CoRR, abs/2302.06692 2023. doi: 10.48550/acXii.2300..666
Evci, U., Dumoulin, V., Larochelle, H., and Mozer, M. C. Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.). International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 6009-6033 PMLR, 2022.
12

---

Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. MineDojo: Building Open-Ended Embodied Agents with Internet-Seat Knowledge. CoRR, Abs/2200.08885, 2022. doi: 10. 00448500/2Xx..2200088333 088533. arXiv: 2206.08853.
Fortunato, M., Tan, M., Faulkner, R., Hansen, S., Badia, A. P., Buttimore, G., Deck, C.,, Leibo, J. Z., and Blundell, C. Generalization of reinforcement learners with working and episodic memory. In Wallach, H. M., Larochelle, H.,, Beygelzimer. A., d' Alché-Buc, F., Fox, E. B.. and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12448-1222 12457, 2019.
Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang, F., Ramsauer, H., Kreil, D. P.. Kopp. M. K., Klambauer, G., Bitto-Nemling, A., and Hochreiter, S. CLOOB: Modern hopfield networks with infol .OOB outperform CLIP. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
Glanois, C., Weng, P., Zimmer, M., Li, D., Yang, T., Hao, J., and Liu, W. A survey on interpretable reinforcement learning. CoRR, abs/2112. 11.11122. 2021.
Goyal, P.. Niekum, S., and Mooney, R. Using Natural Language for Reward Shaping in Reinforee- ment Learning. In Kraus, S. (ed.), Proceedings of the Twenty-Eighe Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 2385-2391, ijeai.org. 2019. doi: 00.24933/jcci.22199333
Gupta, T., Karkus, P., Che, T., Xu, D., and Pavone, M. Foundation models for semantic novelty in reinforcement learning. CoRR, abs/2211.000 1.04778, 2022. doi: 00.48550/axxi.22211.4483
Ha, D. and Schmidhuber, J. Recurrent world models facilitate policy evolution. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), NeurIPS, pp. 2455-2467, 2018.
Hafner, D., Lillicrap, P., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. In ICLR. OpenReview.net, 2020.
Hafner, D., Lillicrap, T. P., Norouzi, M., and Ba, J. Mastering atari with discrete world models. In International Conference Learning Representations, 2021.
Hafner, D... Pasukonis. J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 2023.
Hausknecht, M. J. and Stone, P. Deep Recurrett Q-Learning for Partially Observable MDPs. In 2015 AAAI Fall Symposia, Arlington, Virginia, USA. November 12-14, 2015, pp. 29-37. AAAI Press, 2015.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1 10.110/22222226..000
Hendricks. L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., and Darrell, T. Generating visual explanations. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 3-19. Springer, 2016. doi:
Hendricks, L. A., Hu, R., Darrell, T., and Akata, Z. Grounding visual explanations. In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings, Part II, volume 11206 of Lecture Notes in Computer Science, pp. 269-286. Springer, 2018. 778-3-030-01211-88117
doi: 10.1007/
Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J. Natural language descriptions deep visual features. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
13

---

Hill, F., Tieleman, O., Glehn, T. v., Wong, N., Merzic, H., and Clark, S. Grounded Language Learning Fast and Slow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Hochreiter, S. and Schmidhuber. J. Long Short-Term Memory, Neural Comput., 98):11355-1780, 1997. doi: 10. 12.11662nccc.1997 199.
Holzleitner, M., Gruber, L., Arjona-Medina, J. A., Brandstetter, J., and Hochreiter, S. Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER, Trans. Large Scale Data Knowl. Centered Syst., 48: 10--130, 2021. doi: 10. 1111000978---6611 3-662-63519-3_5.
Hu, S. and Clune, J. Thought cloning: Learning to think while acting by imitating human thinking. CoRR, abs/2306.00323, 2023. doi: 10.48550/arxiv.225 Xttp://r.vv..3...000 230...0300...33.
Huang, W., Abbeel, P., Pathak. D., and Mordatch, I. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1148-9147. PMLR, 2022a.
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Jackson, T., Brown, N., Luu, L.,, Levine, S., Hausman, K., and Ichter, B. Inner monologue: Embodied reasoning through planning with language models. In Liu, K., Kulic, D., and Ichnowski, J. (eds.), Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pp. 1769-1782. PMLR, 2022b.
Huang, W., Xia, F., Shah, D., Driess, D., Zeng, A., Lu, Y., Florence. P., Mordatch, I., Levine, S., Hausman, K., and Ichter, B. Grounded decoding: Guiding text generation with grounded models for robot control. CoRR, abs/2303.00855. 2023. doi: 00.488550/rXX1..233.00000 00055..
coutsbbaut-uyyseee M., Mets, K., and Latré. S. Pre-trained word embeddings for coal-conditional transfer learning in reinforcement learning. CoRR, (bss2000.05196, 2020.
Ichter, B., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang. E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y. Parada, C., Rao, K., Sermanet, P., Toshev, A., Vanhoucke, V., Xia, F,, Xiao, T., Xu, P., Yan, M., Brown, N., Ahn, M., Cortes, O., Sievers, N., Tan, C., Xu, S., Reyes, D., Rettinghouse, J., Quiambao, J., Pastor, P., Luu, L., Lee, K., Kuang, Y., Jesmonth, S., Joshi, N. J., Jeffrey, K., Ruano, R. J., Hsu, J., Gopalakrishnan, K., David, B., Zeng, A., and Fu, C. K. Do I can, not as I say: Grounding language in robotic affordances. In Liu, K., Kulic, D., and Ichnowski, J. (eds.), Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pp. 287-318. PMLR, 2022.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456. JMLR.org. 2015.
Jacob, A. P., Lewis, M., and Andreas, J. Multitasking inhibits semantic drift. In Toutanova, K., Rumshisky, A., Zettlemoyer. L., Hakkani- Tür, D., Beltagy, I.. Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2021, Online, June 6-11, 2021, pp. 5351-5366. Association for Computational Linguisties, maiacc-main.421...
2021. doi: N0.18653/v1/200
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT. Parekh, 7., Pham, H., Le, Q. V., Sung, Y.HH., Li, 7.,, and Duerig, T. Scaling up visual and lasion-language representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
Jiang. Y., Gu, S. S., Murphy, K. P., and Finn, C. Language as an Abstraction for Hierarchical Deep Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
14

---

Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Planning and Acting in Partially Observable Stochastic Domains. Artificial Intellignnce, 101:99-134, 1998.
Kassner, N., Krojer, B., and Schütze, H. Are Pretrained Language Models Symbolic Reasoners over Knowledge? In Fernández, R. and Linzen, T. (eds.), Proceedings of the 24th Conference Computational Natural Language Learning, CoNLL 2020, Online, November 19-20, 2020, pp- 552-564. Association for Computational Linguistics, 2020. doi: Nt.18653/1//0220.comm
1.45.
Khetarpal, K., Klissarov, M., Chevalier- Chevalier-Boisvert, M., Bacon, P., and Precup. D. Options of interest: Temporal abstraction with interest functions. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp.
AAAI Press, 2020.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N. C., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Brabska-Barwinska. A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612,00779, 2016.
Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2023.
Leibo, J. Z., de Masson d' Autume, C., Zoran, D., Amos, D., Beattie, C., Anderson, K., Castañeda, A. G., Sanchez, M., Green, S., Gruslys, A., Legg, S., Hassabis, D., and Botvinick, M. M. Psychlab: A psychology laboratory for deep reinforcement learning agents. CoRR, abs/180 1801 1,01011666 16, 2018.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L., Chen, T., Huang, D.-A., Akyürek, E., Anandkumar, A., Andreas, J., Mordatch, Torralba, A., and Zhu, Pre-Trained Language Models for Interactive Decision-Making CoRR, abs/2202.01771, 2022. arXiv: 2202.01771.
Liang, J., Huang, W., Xia, F,, Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. CoRR, abs/2200.007753. 2022a. doi: 10.48550/arxiv.2200.7777
Liang W., Zhang, Y., Kwon, Y., Yeung. S., and Zou, Mind the gap: Understanding the modality gap in mudti-modal contrastive representation learning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, (eds.), Advances Neural Information Processing Systems, 2022b.
Liu, E. Z., Raghunathan, A., Liang, P., and Finn, C. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In Meila, M. and Zhang. T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 6925 6935. PMLR, 2021.
Mahowald, K., Ivanova, A., Blank, A., Kanwisher, N., Tenenbaum, J. B., and Fedorenko, Dissociating language and thought in large language models: a cognitive perspective, 2023.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nat., $18(7540):529-5555 2015. doi: 10. 1038/natrre 14236.
Mu, J., Zhong, V., Raileanu, R., Jiang, M., Goodman, N. D., Rocktäschel, T., and Grefenstette, E. Improving Intrinsic Exploration with Language Abstractions. CoRR, abs/2202.08999999 08938, 2022. arXiv: 2202.08938.
Narang, S., Raffel, C., Lee, K., Roberts, A., Fiedel, N., and Malkan, K. W15?! training text-to-text models to explain their predictions. CoRR, abs/2004. 14546, 2020.
Ostapenko, O., Lesort, T., Rodríguez, P., Arefin, M. R., Douillard, A., Rish, I., and Charlin, L. Foundational Models for Continual Learning: An Empirical Study of Latent Replay. arXiv preprint arXiv:2205.003: 2022.
15

---

Ouali, Y., Bulat, A., Martinez, B., and Tzimiropoulos, G. Black box few-shot adaptation for vision-languagg models, 2023.
Paischer, F., Adler, T., Patil, V. P., Bitto-Nemling, A., Holzleitner, M., Lehner, S., Eghbal-Zadeh
H., and Hochreiter, S. History compression via language models in reinforcement learning. In Chaudhuri, K., Jegelka, S., Song, L.., Szepesvári, C., Niu, G., and Sabato. S. (eds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 17156-17185, PMLR, 2022a.
Paischer, F,, Adler, T., Radler, A., Hofmarcher, M., and Hochreiter, S. Foundation models for history compression in reinforcement learning. In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022b.
Paischer, F., Adler, T., Radler, A., Hofmarcher, M., and Hochreiter, S. Toward semantic history compression for reinforcement learning. In Second Workshop on Language and Reinforcement Learning, 2022c.
Parisi, S., Rajeswaran, A.. Purushwalkam, S., and Gupta, A. The Unsurprising Effectiveness of Pre-Trained Vision Models for Control. In Chaudhuri, K., Jegelka, S., Song, L.,, Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1735--17771 PMLR, 2022.
Parisotto, E. and Salakhutdinov, R. R. Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Parisotto, E., Song, F., Rae, J., Pascanu, R., Gulcehre, C., Jayakumar, S., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., Botvinick, M., Heess, N., and Hadsell, R. Stabilizing Transformers for Reinforcement Learning. In International Conference on Machine Learning, pp. 7487-7498. PMLR, November 2020.
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., and Rohrbach, M. Multimodal explanations: Justifying decisions and pointing to the evidence. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 8779-8788. Computer Vision Foundation IEEE Computer Society, 2018 doi: 10.1109/CVPR.2218800099
Park, J. S., O' O'Brien, J. C.,, Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior, 2023.
Pasukonis, J., Lillicrap. T., and Hafner, D. Evaluating long-term memory in 3d mazes, 2022.
Patel, D., Eghbalzadeh, H., Kamra, N., luzzolino, M. L., Jain, U., and Desai, R. Pretrained language models as visual planners for human assistance, 2023.
Patil, V. P., Hofmarcher, M., Dinu, M.-.,, Dorfer, M., Blies, P. M., Brandstetter, J., Arjona-Medina,
J. A., and Hochreiter, S. Align-RUDDER: Learning From Few Demonsirations by Reward Redistribution. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, (eds.), International Conference on Machine Learning. ICML 2022. 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 17531-17572. PMLR, 2022.
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P. S. H., Bakhtin, A., Wu, Y., and Miller, A. H. Language Models as Knowledge Bases? In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 2463-2473, Association for Computational Linguistics, 2019. doi: 10.18653/11/D19111
Pleines, M., Pallasch, M., Zimmer, F., and Preuss, M. Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization. CoRR, abs/2205. 1.0005.11100 1104, 2022. doi: 10.48550/arXiv.2205. 11104. aiv: 2205.1 10..11100 1104.
16

---

Puiutta, E. and Veith, E. M. S. Explainable Reinforcement Learning: A Survey. In Holzinger, A.. Kieseberg, P., Tjoa, A. M., and Weippl, E. (eds.), Machine Learning and Knowledge Extraction, Lecture Notes in Computer Science, pp. 77-95, Cham, 2020. Springer International Publishing. ISBN 778-3-000--7332-- doi: 00007/978-3-030-7522-----
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark. J., Krueger, G., and Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision. In Meila, M. and Zhang, T. (eds.), Pruceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 3748-8766 PMLR, 2021.
Rajani, N. F., McCann, B., Xiong, C., and Socher, R. Explain yourself! leveraging language models for commonsense reasoning. In Korhonen, A., Traum, D. R., and Mårquez. L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4932-4942. Association for Computational Linguisties, 2019. doi: 10. 10.18633/v1/199 19telp/p111111 1987
Reid, M., Yamada, Y., and Gu, S. S. Can Wikipedia Help Offline Reinforcement Learning? CoRR, Abs/2201.12122, 2022.
arXiv: 2201.12122.
Richards, B. A. and Frankland, P. W. The Persistence and Transience of Memory. Neuron, 94(6): 1071-1084, 2017. ISSN 0896-6273, doi: 1ttp://oi..r/1111111/////// www.:/noooo.o..0......
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L.,, and Zhong, C. Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges, July 2021.
Ruvolo, P. and Eaton, E. ELLA: An Efficient Lifelong Learning Algorithm. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 507-515. JMLR.org, 2013.
Sahni, H., Buckley, T., Abbeel, P., and Kuzovkin, I. Addressing sample complexity in visual tasks using HER and hallucinatory gans. In Wallach, H. M., Larochelle, H., Beygelzimer, A., Alchéé Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5823-5833, 2019.
Schmidhuber,,1 J. Learning Complex, Extended Sequences Using the Principle of History Compression. Neural Comput., 4(2):234 242, 1992. doi: 10.1 10.1162/neco. 1ttp2//cco.999...44 1992.4.2.234.
Schmidhuber, J., Mozer, M. C., and Prelinger, D. Continuous history compression. In Hüning, H., Neuhauser, S., Raus, M., and Ritschel, W. (eds.), Proc. of Intl. Workshop on Neural Networks, RWTH Aachen, pp. 87-95. Augustinus, 1993.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms. CoRR, abs//107 0734777777777777 2017. arXiv: 1707.06347.
Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinske Barwinska, A., Tch, Y. W., Pascanu, R., and Hadsell, R. Progress & Compress: A scalable framework for continual learning. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholussmässan, Stockholm, Sweden. July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4535 4544. PMLR, 2018.
Shah, D., Osiński, B., brian ichter, and Levine, S. LM-nav: Robotic navigation with large pre-trained models of language, vision, and action. In 6th Annual Conference on Robot Learning, 2022.
Sharma, P., Torralba, A., and Andreas, J. Skill induction and planning with latent language. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational I. Linguistics (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 1713-1726. Association for Computational Linguistics, 2022. doi: 120.
10.18653/v
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Progprompt: Generating situated robot task plans using large language models. CoRR, abs/2209. 100009111300 11302, 2022, doi: 10.488500arxiv.2200.155
17

---

Talmor, A., Elazar, Y., Goldberg, Y., and Berant, J. oLMpies On what Language Model Pre-training Captures. Trans, Assoc. Comput. Linguistics, 8:743-758, 2020.
Tam, A. C.. Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, C. Y., Strouse, D. J., Wang, J. X., Banino, A., and Hill, F. Semantic Exploration from Language Abstractions and Pretrained Representations. CoRR, abs/2204.05080, 2022. doi: 10.48500arxx..2244450000 arXiv: 2204.05080.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, L.,, and Polosukhin, Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Venuto, D., Boussioux, L., Wang, J., Dali, R., Chakravorty, J., Bengio, Y., and Precup. D. Avoidance learning using observational reinforcement learning. CoRR, abs/1909. 1228, 2019.
Vinyals. O., Babuschkin, I., Czarnecki, W. M., Mathieu, M. Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang. A., Sifre, L., Cai, T., Agapiou, J. P.. Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gülçehre, C., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T. P., Kavukcuoglu, K. Hassabis, D., Apps, C., and Silver, D. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nat., 575(7782) 354,, 2019. doi: 10. 10.1038/541586-01999 172---
Wang. X., Huang, Q.. Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., and Zhang, L. Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6629-6638. Computer Vision Foundation IEEE, 2019. doi: 00.100//0P..221100006
Wang, Z., Cai, S., Liu, A., Ma, X., and Liang, Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, 2023.
Waxman, S. R. and Markow, D. Words as Invitations to Form Categories: Evidence from 12- to 13-Month-ON Infants. Cognitive Psychology, 29:257-302, 1995.
Wayne, G., Hung, C., Amos, D., Mirza, M., Ahuja, A., Bariskkk,a A., Rac, J. W. Mirowski, P., Leibo, J. 7.,, Santoro, A., Gemici, M., Reynolds, M., Harley, T., Abramson, J., Mohamed, S., Rezende, D. J., Saxton, D., Cain, A., Hillier, C., Silver, D., Kavukcuoglu, K., Botvinick, M. M., Hassabis, D., and Lillicrap, T. P. Unsupervised predictive memory in goal-directed agent. CoRR, abs/1803.1B 10760000...,,..001110011000000 2018
Widrich, M., Hofmarcher, M.. Patil, V. P.. Bitto-Nemling, A., and Hochreiter, S. Modern Hopfield Networks for Return Decomposition for Delayed Rewards. In Deep RL Workshop NeurIPS 2021, 2021.
Wolf, T., Debut, I.., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M.. Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020, Association for Computational Linguistics. doi: 10.18653/w1/2000 10..8665///222200mmmmmmme6mmmm emnlp-demos66..
Wu, Y., Fan, Y., Liang, P. P., Azaria, A., Li, Y., and Mitchell, T. M. Read and reap the rewards: Learning play atari with the help of instruction manuals, 2023.
Yan, C., Carnevale, F., Georgiev, P., Santoro, A., Guy, A., Muldal, A., Hung, C.-C., Abramson, J., Lillicrap, T., and Wayne, G. Intra-agent speech permits zero-shot task acquisition, 2022.
Yao, S., Rao, R., Hausknecht, M. J., and Narasimhan, K. Keep CALM and Explore: Language Models for Action Generation in Text-based Games. In Webber, B., Cohn, T., He, Y., and Liu, (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 8736-8754 Association for Computational
Linguisties, 2020. doi: Nt.18853/w1/200 wwt.::mmmmllmmammmmm
18

---

Yee, E., Jones, M., and McRae, K. Semantic Memory. 10 2017. doi: 10.1002/978 19170177 epen309.
Zaidan, O. and Eisner, J. Modeling annotators: generative approach to learning from annotator rationales. In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25-27 October 2008, Honolulu, Hawaii, USA, A meeting of SIGDAT, Special Interest Group of the ACL, pp. 31-40. ACL, 2008
Zellers, R., Bisk, Y., Farhadi, A., and Choi, Y. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Puttern Recognition,, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6720-6731. Computer Vision Foundation IEEE, 2019. doi: 10. P..1009CVVR.22119.MM
Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M. S., Sindhwani, V., Lee, J., Vanhoucke, V., and Florence, P. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language. CoRR, abs/2204,00598, 2022. doi: (0448550/axxi::224400888 arXiv: 2204.00598.
Zenke, F., Poole, B., and Ganguli, S. Continual Learning Through Synaptic Intelligence. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3987-3995. PMLR, 2017.
Zha, D., Ma, W., Yuan, L.,, Hu, X., and Liu, J. Rank the episodes: A simple approach for explo- ration in procedurally-generaaee environments. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Zhong, V., Rocktäschel, T., and Grefenstette, RTFM:
generalising to new environment dynamics via reading. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Zhong, V., Hanjie, A. W., Wang, S. I., Narasimhan, K., and Zettlemoyer, L.. SILG: the multi-domain symbolic interactive language grounding benchmark. In Ranzato, M., Beygelzimer, A., Dauphin, Advances Y. N., Liang, P., and Vaughan, J. W. Advancess in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 11555-21519) 2021.
Zhong, V., Mu, J., Zettlemoyer, I.., Grefenstette, E.,, and Rocktäschel, T. Improving policy learning via language dynamics distillation. CoRR, 0bs/2210000066 2022. doi: 11.488500/2Xiv.221000000
Åström, K. J. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10:174-205, 1964.
19

---

A Environments
We choose 8 diverse 3D environments of the MiniWorld benchmark suite:
CollectHealth:
The agent spawns in a room filled with acid and must collect medikits in order to survive as long possible.
The agent must reach red box that is located in one of four interconnected rooms.
FourRooms:
MazeS3Fast: procedurally generated maze in which the agent needs to find goal object.
PickupObjs: Several objects are placed in a large room and must be collected by the agent. Since the agent receives reward of I for each collected object, the reward is unbounded.
PutNext: Several boxes of various colors and sizes are placed in big room. The agent must put a red box next a yellow one.
Sign: The agent spawns in U-shaped maze containing various objects of different colors. One side of the maze contains sign which displays a color in written form. The aim is to collect all objects in the corresponding color.
TMaze:
The agent must navigate towards object that is randomly placed at either end of a T-junction.
YMaze: Same as TMaze, but with a Y-junction.
We neglect the OneRoom and the Hallway environments, since those are easily solved by all compared methods. Further, neglect the Sidewalk environment since it is essentially the same task as Hallway with different background. Since the rewards of PickupOhis and CollectHealth are unbounded, we normalize them to be in the range of [0, 1]. which is the reward received in all other environments. For a more detailed description of the MiniGrid environments we refer the reader to Paischer et al. (2022a).
B Token retrieval for synthetic environments
Since we perform retrieval a token level we investigate the effect of augmenting tokens with different prompts, and the effect of different vision encoders on retrieval performance. We analyse the former at the example of the Avalon environment. Figure 7 shows some examples. We observed that simply encoding single tokens in CLIP space results in a bias toward abstract tokens such as biome, screenshot. The same occurs for using the prompts originally used for zero-shot classification on the ImageNet dataser' (Deng et al., 2009). However, one can alleviate this bias by including these frequently retrieved tokens in the prompt itself (see Figure 7, bottom). However, we found this strategy to effective only for Avalon and Psychlab environments. The sets of prompts for both environments be found in Table For MiniGrid and MiniWorld we retrieve single tokens without any prompt.
Next, we investigate the influence of the choice of vision encoder architecture, e.g., RN50 ViT-B/16. We only consider those encoders since they induce the least complexity on our history compression pipeline. We show the closest tokens in CLIP space for observations of MiniWorld (see Figure 9) and Avalon environments (see Figure 8). For MiniWorld, CLIP can extract shapes and colors. However, the retrievals very noisy, which is mirrored in the attained score of SHELM. For Avalon, however, the token retrievals are more convincing. Generally, we find that retrievals using the RN50 encoder of CLIP tend to be more noisy than retrieval utilizing ViT-B/16 encoder.
Instead of using CLIP and our token retrieval we may be able to use pretrained image captioning systems, such as BLIP-2 (Li et al., 2023) directly. To investigate this, we use BLIP-2 to generate captions for MiniGrid-Memort MiniGrid- MiniGrid-Memory in the left column contain correctly detected colors (green and red), and shapes (square), but incorrectly combines them, c.g. there is neither a green square, nor a green arrow pointing to the left. A similar trend can be observed for Avalon in the middle comlumn, where BLIP-2 recognizes a frog, but incorrectly assumes that it stands on top of a rock, while the rock is
Avalon and Psychlab (Figure 10). The captions generated for the
'vvallbble at httpss//giiubbooom//eea//////LCt
20

---

biome
tree
No Prompts
screenshot
environment
pine
0.00
0.05
0.10
0.15
0.20
0.25
0.30
ImageNet Prompis
virtual
screenshot
tree
0.05
0.20
0.25
0.30
tree
Avalon Prompts
berry
dlive
Observation
Cosine similarity
Figure 7: Token rankings for ViT-B/16 encoder of CLIP on Avalon observations. Tokens are encoded with the CLIP language encoder without prompt (top), with ImageNet specific prompts (middle), or prompts designed for Avalon (bottom).
21

---

Observation
RN50
ViT-B/16
beech
tree
tree
bee
ddddle
stick
oak
beech
bush
olive
0.0
0.1
0.2
0.0
0.1
0.2
0.3
saddle
sand
rake
claw
rock
broom
hammer
cone
surf
hills
hill
hill
hills
climb
caim
mountain
ball
dew
shape
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Figure 8: Token rankings for RN50 and ViT-B/16 encoders of CLIP on Avalon observations.
clearly beside the frog in the foreground. The second observation for Avalon is very accurate though. For the two Psychlab observations in the right column, BLIP-2 correctly describes the objects on the screen, but also mentions redundant information, such "the objects are on screen". Overall, BLIP-2 correctly recognizes shapes, colors, and different objects, but often incorrect on their compositionality. Further, drawback of using off-the-shelf captioning engines is that we cannot control for the context length for each observation. This can quickly result in excessive context lengths, which in turn, results in drastic increase of computational complexity.
C Qualitative analyses
We show token mappings for the memory mechanism of SHELM to identify potential failures and successes. Figure shows few sample episodes from trained policy on the Memory environment. Clearly, CLIP is not capable of extracting semantics of the abstract 2D gridworld environments. Thereby, it maps the ball to the token miner and the key the token narrow. For minimalistic 3D environments, CLIP is capable of extracting colors and shapes of objects as can be seen in Figure 9. However, these results are still uninspiring since the token retrievals are very noisy.
We also visualize episodes collected by random policy from the Avalon environment in Figure 12. As opposed to the minimalistic MiniGrid and MiniWorld environments, CLIP successfully extracts semantically meaningful concepts which are subsequently passed to the memory. There are still some failure cases though. For example, in the second episode the agent is attacked by bee. The feelers of the bee are mistaken for a sword as the bee moves into close proximity of the agent. Further after the agent successfully defends itself against the bee, it mistakes the dead bec for crab. Alleviating
22

---

Observation
RN50
ViT-B/16
circuit
green
margin
cube
race
cubic
node
pin
box
0.00
0.10
0.20
0.0
0.1
margin
rot
circuit
devo
render
die
kart
render
race
0.00
0.05
0.10
0.15
0.20
0.0
0.1
0.2
render
cube
blome
render
nether
wall
lag
dungeon
room
cubic
0.00
0.05
0.10
0.15
0.0
0.1
Figure 9: Token rankings for RN50 and ViT-B/16 encoders of CLIP on MiniWorld observations.
cherry top
glass bottle screon
lock
pointing the let: low poly environment
Figure 10: Captions generated via BLIP-2 for observations of the EaniGrid-Memor task, Avalon and Psychlab.
23

---

Table 1: Prompts used for token retrieval for the Avalon and Psychlab environments.
Environment
Prompts
screenshot of a screenshot of a screenshot of many a biome containing a biome containing a biome containing many a biome full of
Avalon
render of render of a screenshot of a screenshot of a screen showing screen showing a
Psychlab
such failure cases would require grounding CLIP the respective environment as was done (Fan et al., 2022).
Further, visualize token mappings for objects the agent encounters in the continuous recognition task of Psychlab in Figure 13. The majority of token retrievals are semantically correct. However, sometimes the agent confuses different objects or conflates the same token for different objects. An example for that are two middle objects in row which are both mapped to the token icon, or the first two objects row that are mapped the token ruber. We suspect that this occurs due to downscaling to lower resolution which is conducted within the DMLab engine. Indeed, when taking a closer look at token retrievals Figure 14). Therefore, we consider two different aspects on how to alleviate this issue, (i) increasing the resolution of the observations and (ii) retrieving more than one token for an observation. The former results in increased computational complexity, which we aim to avoid, since the task is computationally very expensive already. The latter assumes that the retrieval differs least in their top-k most likely tokens and is viable solution.
higher resolution, they are mapped different tokens (see
D Additional results
We show results for PPO, HELM, HELMv2, and SHELM on all Avalon tasks after 10M interaction steps in Table 2. Dreamerv2 attains the highest score on average after 10M timesteps, but performs on-par with SHELM. Further, there is significant difference between the HELM variants though. Since SHELM is demonstrably capable of extracting semantics from Avalon environments (see Figure 12), believe that the main reason for these results is that the Avalon tasks do not require the agent to use its memory.
To further investigate this finding and compare to results reported in Albrecht et al. (2022) in Table 3. Surprisingly, our PPO baseline significantly outperforms the PPO baseline of Albrecht et al. (2022). Further, we find that our PPO baseline performs on-par with all other memory- based baselines, yielding further evidence that a memory mechanism not imperative to solve the Avalon tasks.
the memory-less PPO baseline for 50 interaction steps
E
Additional ablation studies
We provide an additional ablation study in the choice of vision encoder for SHELM. Our qualitative results in Section 3.1 indicated that vision transformer based encoders are better in extracting semantics of synthetic environments. MiniGrid-Memory environment where we exchange the ViT-B/16 with ResNet-50.. The results can be observed Figure 15. We observe that the ViT-B/16 encoder much better in discriminating between the different objects which results in substantial improvement over the RN50 encoder.
corroborate this finding, we run an experiment on
24

---

circuit
narrow
miner
miner
circuit
circuit
midi
screen
miner
t=
t=
t=2
t=3
t=4
t=5
t=
t=7
t
circuit
narrow
miner
miner
circuit
circuit
midi
screen
miner
t=
t=1
t=2
t=3
t=4
t=5
t=
t=7
t
midi
dungeon
miner
narrow
square
circuit
midi
screen
miner
t=
t=1
t=2
t=3
t=4
t=5
t=
t=7
t=
midi
square
miner
narrow
screenshot
screenshot
midi
screenshot
miner
t=
t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=
Figure 11: Episodes sampled for trained SHELM policy on the MiniGrid-MemorySS 1-v0 environ- ment. The object ball is consistently mapped to the token miner, while the object key maps to the token narrow.
F Hyperparameters and training details
Since the memory component of SHELM does not need not be trained, our memory requirements are comparably low. All our experiments were run on either a single GOXIO or a single A100 GPU. The time requirements of experiments vary for each environment. For MiniGrid and MiniWorld one run takes approximately two hours, while for Psychlab one run requires two days. These experiments where one run to train for 10M interaction steps takes approximately 15 hours.
run single GTX1080TT For Avalon, used single A100 for training
MiniGrid and MiniWorld We adapt the hyperparameter search from Paischer al. (2022a). Particularly, we search for learning rate in (5e-4, 3e-4, 1e-5, 5e-5), entropy coefficient in (0.05, 0.01, 0.005,000000 0.001), rollout length in (32, 64, 128) for SHELM. To decrease wall-clock time of HELM variants, we vary the size of the memory register of TrXL such that it can fit the maximum episode length.
lower the number of interaction steps for the gridsearch we observe convergence before the 500k interaction steps. If no convergence is observed within the 500K interaction steps, we tune for the entire duration. We apply the same scheme for tuning the LSTM baseline and tune the same hyperparameters in Paischer al. (2022a).
Avalon After visual inspection of token retrievals for observations we found that there is no substantial difference retrieved tokens for observations in close proximity to each other. Therefore, we introduce an additional hyperparemeter, namely history-frameskip. Note that the cistory-frameskee functionally equivalent the commonly used frameskip in, e.g., (Mnih et al., 2015). Rather, actually discard frames within the skip. For example, for distory-framessey of 10 the agent only
25

---

tree
beach
dune
ditch
fruit
cliff
t=0
t=64
t=117
t=194 cliff
t=260 ball
t= 368
tree
bee
berry
cricket
sword
crab
t=0
t=23
t=70
t=92
t=134
t=219
t=220
t=
tree
cherry
palm
lawn
frog
ocean
t=0
t=191
t=342
t=503
t= 1242
t=5017
turtle
hole
stones
beach
dune
t=32
t=62
t=
t=258
t= t=631
door
lock
ring
fruit
t=0
t=16
24
t=
Figure 12: Episodes sampled from random policy various Avalon tasks.
observes the first and the eleventh frame in the history branch. The current branch observes every timestep as usual. We search over to {256, 128, 64) timesteps respectively. Further we search over learning rate in {2.5e-4, 1e-4, 7e-5}, and the number of retrieved tokens {1, 2,. 4}. If an observation is represented as more than one token, this effectively reduces the number of observations that fit into the memory register of TrXL, and thereby introduces a recency bias. Hyperparameters for the Dreamerv2 baseline are taken from (Albrecht et al., 2022). For the memory-less baseline we search over learning rates (5e-4, 2.5e-4, le-4, 5e-5). We used their respective codebase to run our experiments.
in {3, 5, 10) and adapt the memory of the agent
PsychLab Due the computational complexity of the Psychlab environments only run gridsearch over the learning rate in (5e-4, 3e-4, le-4, 5e-5). Further we use 64 actors and set the rollout size to 256. For SHELM on continuous- mnoonnios we only retrieve the closest token for an observation.
G Potential negative societal impact
Our method relies the of FMs, which trained non-curated datasets which crawled from the web. Therefore, these models readily reflect the biases and prejudices found on the web and, consequently, might the resulting RI. agent trained by method. Generally, deployment of RI.
26

---

dog
bucket
equip
chocolate
dish
boiler
kettle
flag
block
pet
vest
rig
Icon
icon
deer
bottle
bag
broom
morpho
tuber
tuber
chair
hat
horse
spoon
jar
brain
grape
cone
keys
clam
Figure 13: Sample observations for continuous recognition task of Psychlab. The agent must swipe certain direction depending on whether it has encountered an object before in the same episode.
27

---

fruit
desk
PF
gorilla
bench
ape
stool
monkey
bureau
fry
homes
0.10
0.15
0.00 0.02
Cosine similarity
Cosine similarity
egg
lock
engineer
keys
hat
key
secure
Il
handle
0.00 0.02 0.05 0.07 0.10 0.125 Cosine similarity
0.00 0.02 0.05 0.07 0.10 0.125 Cosine similarity
Figure 14: Top-5 token retrievals for objects that conflated by SHELM a lower resolution. For higher resolution CLIP successfully maps the objects to different tokens.
Memory
1.0
0.8
Accumulated Reward
0.9
0.2
SHELM
SHELM-RN50
Random
0.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
1e6
Number of Interaction Steps
Figure 15: Ablation study on the choice of vision encoder used for the retrieval from our semantic database. We report IQM and 95% CIs across 30 seeds the MiniGriidMemmor task.
28

---

Table 2: Mean human normalized score for for all Avalon tasks for PPO, HELMv2, SHELM, and Dreamerv2 after 10M interaction steps. We show mean and standard deviations.
Task
PPO
HELM
HELMv2
SHELM
Dreamerv2
0.700 = 0.070
0.671 = 0.071
0.714
0.063
0.693 0.071
0.656 = 0,077
move
0.312 = 0.066
0.277 = 0.068
0.294
0
0.362 = 0.073
jump
0.232 = 0.053
0.232 = 0.058
0.051
0.217
0.233 = 0.055
climb
0.120 + 0.042
0.125 + 0.039
0.052
0
0.140 + 0.048
descend
0.203 + 0.048
0.184 + 0.044
0.108 + 0.035
0.283 + 0.089
scramble
0.241 + 0.056
0.213 + 0.052
0.301 + 0.056
0.271
0.058
0.306 + 0.089
stack
0.094 = 0.035
0.058 0.029
0.075 0.031
0
0.106 = 0.041
bridge
0.020 = 0.017
0.046 = 0.026
0.040 0.024
0.028 0.018
0.046 = 0.031
push
0.109 0.039
0.069 = 0.030
0.009
0.140
0.102 = 0.044
throw
0.000 = 0.000
0.017 = 0.020
0.000
0.000
0.007 = 0.011
hunt
0.059 0.029
0.077 0.033
0.091 0.034
0.091
0.067 0.034
fight
0.179 = 0.046
0.189 = 0.048
0.207
0.055
avoid
0.234 0.048
0.192 0.047
0.211
0.227
0.377 #
explore
0.172 + 0.045
0.194 + 0.051
0.191 + 0.050
0.204
0.196 + 0.051
open
0.086 0.035
0.134 + 0.042
0.040 + 0.024
0.102 0.038
0.055 0.031
carry
0.080 + 0.031
0.101 + 0.041
0.073 + 0.030
0.054 + 0.027
0.087 + 0.035
navigate
0.006 + 0.007
0.000 + 0.000
0.000 + 0.003
0.006 + 0.008
0.006 + 0.008
find
0.005 + 0.007
0.000 - 0.000
0.000 + 0.000
0.000 + 0.000
0.003 + 0.004
survive
0.049 0.015
0.066
0.005
0.041 0.013
0.051 + 0.016
0.033 0.012
gather
0.007 + 0.006
0.008 + 0.019
0.009 0.007
0.006 + 0.003
0.006 + 0.004
overall
0.146 + 0.010
0.143 + 0.010
0.145 0.010
0.155 + 0.011
0.168 + 0.012
Table Mean human normalized score over all Avalon tasks after 50M interaction steps. We show mean and standard deviations. Results for PPO, Dreamerv2, and Impala were taken from Albrecht al. (2022). Asterisk indicates our own results.
Method PPO Dreamerv2 Impala
Mean Human Normalized Score
0.165 + 0.014 0.199 = 0.012
0.203 = 0.013
PPO*
0.200 + 0.012
agents the world requires carefully designed interface to guarantee safe execution of selected actions. That said, we do not expect any negative societal impact from our memory mechanism. Our semantic memory enhances memory-based methods, thus, can yield more insights into why agents make certain decisions.
H Reproducibility Statement
We make all our code and random seeds used in our experiments, as well as obtained results publicly available at https: :/ttss///gite htt//gitttt thub. com/ml //mmmm11// /////mmmjjkk/// miljjuuhellm. The pretrained language encoder, well the pretrained CLIP encoder are publicly available on the huggingface hub (Wolf et al., 2020).
29


Reviews:
[{"review_id": "U9I6lNQcpp", "reviewer": "Reviewer_xnPR", "summary": "Paper presents a method to augment a feed-forward RL policy with an “interpretable” memory. In general, it is based off GTrXL and HELM, which uses transformer-XL (TrXL) to encode current visual observation (or further map it to some embedding through embedding look-up, CLIP encoder, etc) while paying attention to some past observations (through the memory mechanism in TrXL). Finally, both the output of TrXL (reterived memory) and current visual observation embedding are fed to actor and critic heads. The proposed method replace CLIP embedding with a bottleneck in explicit text, which is claimed to be more “interpretable”. The proposed method is evaluated on several domains again some baselines and ablations.", "questions": "See [Weaknesses]\n\n---\n\n### Post-rebuttal\n\nThank you for the detailed reply and the additional results. My concerns on baselines and limitations are mostly addressed. I will raise my score to 5. I will consider raising my score even further if the remaining questions can be addressed.\n\n---\nRaised my score to 6 after further discussions with the authors.\n\n\n", "limitations": "To ensure interpretability, the authors have to pre-define a set of sentences, which effectively limit the scope of visual observations that can be stored in the memory. Maybe using a generalist caption model like BLIP-2 could help mitigate this?", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+The paper is generally clear and well-written. Although there seems to be many legacies in this direction (HELM, HELMv2, etc), the authors clearly explains how their method is different from these prior arts, especially on what motivates the proposed changes, making it friendly to readers who might not quite familiar with memory in RL agents.\n\n+The study problem is relevant to the interest of NeurIPS. The use of language model, CLIP and other foundation models in RL should be of interest to many audiences of this conference, which could generate possible impact.\n\n+The proposed method generally makes sense. Instead of sending CLIP embedding directly to TrXL, introducing an explicit text bottleneck does bring more interpretability, and the results does not seem to indicate significant drawbacks from the previous model using implicit embeddings.\n\n+Results on lots of challenging domains (minigrid, MiniWorld, Avalon, PsychLab, etc) with ablations seems to justify the claims well.\n", "weaknesses": "Having said those above, I do have some concerns on baselines and results, listed below:\n\n-Although the authors mentioned four domains, no all baselines (PPO, PPO w/ LSTM, DreamerV2, HELM v1/v2, etc) are tested on all these environments. Ex., there is no results with DreamerV2 on Avalon. Can the authors comment on that? Also, I couldn’t find more justifications on the selection of these baselines – is the main goal of experiments to validate the effectiveness of memory mechanism in RL agents, or simply demonstrate that the proposed memory suites the tasks better? What is the state-of-the-art agent in the four environments? These are the points that needs to be further elaborate to help us understand the significance of the results.\n\nOne specific point: since DreamerV3 has been released, is there a certain reason to only include results from its precedent?\n\n-One main contribution made by this paper is to upgrade the CLIP + TrXL memory in HELMv2 to a more “interpretable” version. This raises two issues that do not seem to be addressed in the paper:\n\n1. Replacing the CLIP embedding in HELMv2 with some explicit text indeed introduces a semantic bottleneck in the memory pipeline, as the visual observations are now compressed into even lower bit discrete text instead of dense embeddings. In theory, this could bring some drawbacks on providing information of the past. Some of the results do indicate this, ex. Slower convergence rate on MiniGrid, worse results on MiniGrid-Memory but some do not, ex. PsychLab. Can the authors explain more on how could such differences emerge? In fact, description of each domain could have been more precise to allow more space for analysis. Specifically, analysis on domain-specific impact on convergence and end performances on different memory-based (w/ or w/o language bottleneck) or memory-free agents should be included.\n \n2. Interpretability seems to be a major matter of concern in this paper, however, I couldn’t find any results, metrics, or analysis in the main text dedicated to demonstrating the interpretability of the proposed method (with only an illustration in Figure 1, but I don’t think it count as a result). How interpretable can the proposed memory be? Does it indeed exhibit some interesting interpretable behavior? What is inside the memory when run on these four domains? Compared to baselines, does user trust this system more due to interpretability rather than better results? These are all questions that need to be answered to justify the interpretability of the proposed method.\n\n-Here are some ablations/baselines that could be missing:\n\n- GTrXL (transformer-XL instead of LSTM for RL)\n- HELM/SHELM w/ Different CLIP models (RN50, ViT, etc)\n- Fine-tune TrXL (it is frozen in SHELM no?)\n- Simply concat t-1 observation (CLIP embedding or text) with current visual observation\n\nMinor:\n-Which env is used for ablations?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and address the raised points as follows:\n\n**Baselines:**\n\nWe have **added the missing baselines**, specifically, Dreamerv2 on Avalon (these were originally shown only in Table 3 in the appendix), and PPO on MiniGrid and MiniWorld.\nDreamerv2 is SOTA on Avalon after 50M timesteps [1]. To the best of our knowledge HELMv2 is SOTA on the selected MiniGrid and MiniWorld environments [2].\nThe only existing results for Psychlab that we are aware of are GTrXL and the results reported in [3]. Unfortunately, they report human normalized scores and didn’t publish either human scores for normalization nor the codebase. Thus, we are unable to directly compare to these approaches. However, SHELM converges within 10M interaction steps whereas both prior methods require billions of interaction steps until convergence. From visual inspection the policy trained with SHELM behaves close to optimal. Results for fine-tuning TrXL and a GTrXL trained with PPO on MiniGrid were reported in [4], where HELM was more sample efficient. Also, no codebase for the original GTrXL implementation using V-MPO is publicly available.\n\nWe provide some **preliminary results for Dreamerv3** in the one-page pdf accompanying our general response. SHELM significantly outperforms Dreamerv3 on the continuous recognition task from Psychlab and is less sample efficient than SHELM on one of the MiniGrid environments. We will deliver results for Dreamerv3 on all remaining environments as well.\n\nWe also added **preliminary results on an ablation study** where SHELM uses a RN50 CLIP encoder instead of ViT-B16. The results corroborate our analysis in Section 3.1., i.e., that a ViT-based encoder is better suited for SHELM. We will add the final results as well as a discussion on these in our revised version.\n\n**Concatenating the t-1 observation** with the current observation essentially resembles framestacking which is guaranteed to fail on tasks such as MiniGrid-Memory and Psychlab, where memory dependencies span across longer time intervals.\n\n[1] Avalon: A benchmark for RL 379 generalization using procedurally generated worlds, Albrecht et al., NeurIPS 2022\n\n[2] Foundation models for history compression in reinforcement learning, Paischer et.al.,  NeurIPS FMDM Workshop, 2022.\n\n[3]  Generalization of reinforcement learners with working and episodic memory, Fortunato et al., NeurIPS 2019\n\n[4] History Compression via Language Models in Reinforcement Learning, Paischer et al., ICML 2022\n\n\n\n**Analysis of results:**\n\nThe reviewer states “worse results of SHELM on MiniGrid-Memory”, which we believe to be a mistake as SHELM significantly outperforms all other methods on MiniGrid-Memory.\nThe inconsistent results on MiniGrid and MiniWorld, however, are due to the fact that these environments are solvable without memory (results from PPO baseline we added on both environments). Memory can be helpful in these environments, e.g., increase sample efficiency, but is ultimately not required to solve the tasks. The only MiniGrid environment that truly requires memory is MiniGrid-Memory, where SHELM significantly outperforms the competitors. We will make this more clear in the revised version.\n\n**Results on interpretability:**\n\nThe retrieval examples that we showed resemble examples for what is stored in the memory. Figure 10 in the appendix shows examples for MiniGrid-Memory, and Figure 4 and 5 show examples for Avalon and Psychlab respectively. We will clarify this in our revised version. \n\nWe agree with the reviewer that some of our claims about interpretability were a bit unfortunate. Therefore, we have revised the claims w.r.t. interpretability of the decision-making process and user-trust. Specifically, we will rewrite the second paragraph in the introduction as well as tone down claims in the abstract to clarify that the enhanced interpretability aspect is enabled by representing past observations in the form of human-readable language. Further we will add intuitive examples on how and why this is useful, e.g., enhanced debugging of modular architectures in RL for developers.\n\n**Limitations:**\n\nA pre-defined set of sentences/prompts is not required, we could simply use the pre-defined vocabularies, i.e. single tokens and encode those with CLIP. However, in our analysis in Section 3.1 we found that prompts appear to make the token retrievals less noisy. We only use prompts for Avalon and Psychlab (shown in Table 1 in the appendix) though, since we did not observe any improvement for MiniGrid observations in our qualitative analysis. We will make this more explicit in the revised version. \n\nWe **added examples for BLIP-2 generated captions** for observations of MiniGrid, Avalon, and Psychlab. BLIP-2 works well for DMlab, but produces rather noisy captions for MiniGrid and Avalon. For now  we refrain from using captioning engines since they would drastically increase the required context length and substantially slow down SHELM. However, as mentioned in our conclusion we believe this is a fruitful avenue for future work.\n"}, {"review_id": "kCAWFaeIN0", "reviewer": "Reviewer_UimZ", "summary": "The paper looks at using a compressed representation of recent observations in the form of top-k CLIP-extracted text entities.  They look at training policies in a few different simulated environments with this representation.  They cast this as an extension of the prior \"HELM\" method, although their key idea does not need to necessarily be paired with HELM.", "questions": "Do each of the discussed weaknesses make sense?", "limitations": "The limitations section in my opinion is incomplete since the authors do not address the primary limitations I pointed out in the \"Weaknesses\" discussion.\nThere is no discussion of broader impacts.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The overall discussion, paper writing, introduction, and experimentation, all seem pretty \"solid\".  The authors seem to be devoted to a genuine scientific exploration and they build rapport with the reader such that they believe the experimentation and investigation is earnest. ", "weaknesses": "There are a couple main limitations in mind for me of the overall method:\n\n1) In the first place, the entire reason used to motivate the investigation was to add interpretability.  However, I'm not sure that a small list of short words, as extracted by CLIP, is a huge value-add for interpretability, since the downstream policy is still black-box.  It is possible that the downstream black-box part of the policy model learns correlations with the text features that are not as a human would interpret them to be.  This limitation is not really investigated empirically.  Instead, after the introduction, all of the results are about performance on the tasks and not interpretability of the performance on the tasks.\n\n2) Another huge limitation that is not discussed is that the proposed method would only be expected to work well in environments where there is a \"single or few things\" observable at a time in an image, and the mere existence of these objects is all the downstream policy needs to know.  The reason is that CLIP can be used okay as a zero-shot object detector, but it is not good at fine-grained understanding of environments.  For example, robot manipulation environments would not likely be well helped by the method.\n\nI consider the above 2 to be primary limitations of the method and my recommendation is primarily driven by those 2.  Here are a few other points to consider for the authors as they revise and for future work:\n\n- The term \"memory\" is repeatedly used here.  However, I'm not sure this is the best word choice for the author's method.  The authors are not creating any type of persistent memory mechanism.  Instead, the method only serves to compress past K observations.  The method works by operating on the past K observations, and compressing them all down to the top-k CLIP words.  Since nothing persistent is created, it's not necessarily a memory mechanism, I would argue.\n- \"Markovian baseline\" -- I know what Markov means, but I'm not sure exactly what they mean by Markovian baseline.  I think they mean that the baseline is memoryless and operates on only the current observation, which would assume that the environment is Markov, even though it's not.  But it also could mean that they give the agent full observability so that it is indeed Markov. It's not clear and should be clarified, moreover \"Markovian baseline\" is likely not the best choice to use as shorthand for it, even though it seems like the authors might have done this before.\n- I was about to suggest that an interesting baseline would be to, instead of using the top-k CLIP text entities, to instead just train a policy on the underlying CLIP vision embeddings themselves, but then I realized this seems to essentially be what HELMv2 was.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and address the concerns as follows:\n\n**First Weakness:**\n\nWe agree with the reviewer that the downstream policy itself remains a black box, but we would like to point out that we explicitly specify that our work only focuses on enhancing interpretability in the memory module (line 326-327).\nFurther, we believe our method adds value, for (i), developers of RL agents, and (ii), for potential applications in the real world. Developers can perform ad-hoc analysis on token mappings to determine whether semantics can be extracted, while post-hoc analysis allows troubleshooting to investigate if important bits of information were stored in the memory. We provide an example for the latter in appendix C.\nAgents interacting in the real world that are equipped with a semantic memory in the form of language can communicate their past findings to an end-user to allow, for example, tracking of lost objects.\n\nHaving said that, we agree with the reviewer that some of our claims about interpretability were a bit unfortunate. Therefore, we have revised the claims w.r.t. interpretability of the decision-making process and user-trust. Specifically, we will rewrite the second paragraph in the introduction as well as tone down claims in the abstract to clarify that the enhanced interpretability aspect is enabled by representing past observations in the form of human-readable language. Further we will add intuitive examples on how and why this is useful, e.g., enhanced debugging of modular architectures in RL for developers.\n\n**Second Weakness:**\n\nWe respectfully disagree with the reviewer that our semantic memory would not be beneficial for robot manipulation environments. Robots interacting in the real world will require a form of semantic scene understanding. In fact, other works have demonstrated the usefulness of CLIP for robot manipulation tasks, e.g. [1,2]. \nWe agree that a semantic memory alone will not be sufficient for solving fine-grained control tasks. However, our memory is modular and could easily be integrated into existing robotic manipulation pathways. We will elaborate on this in the revised version of our manuscript.\n\n[1] CLIPort: What and Where Pathways for Robotic Manipulation, Shridhar et al., CoRL 2021\n\n[2] CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation, Gadre et al., arXiv/2203.10421\n\n**Minor Weaknesses:**\n\n - The “semantic database” of token embeddings we retrieve from is persistent across all environments therefore we believe the terminology of “memory” is accurate. However we will differentiate between the “semantic memory” (i.e. database of tokens to retrieve from) and the “episodic memory”, which is represented by the LM.\n - We will change the terminology from “Markovian” to “Memory-less”, which is what our PPO method represents.\n"}, {"review_id": "jfqn0t2LY7", "reviewer": "Reviewer_a9Pk", "summary": "This paper introduces Semantic HELM, a memory mechanism for encoding an agent's past observations into a semantically meaningful space and recalling from the memory. It is an extension/modification of HELM, a previous work that uses a language model to encode past observations, but unlike prior methods, Semantic HELM's tokens are interpretable (i.e. semantically related to the observations) rather than random. The paper shows comparisons to prior methods on four environments, including ones requiring strong memory to complete successfully.", "questions": "See above for questions about vocab. ", "limitations": "See above - possibly some limitations about the vocab that I think the paper could address better. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I think the paper is extremely thorough in its experiments. It evaluates on four different environments and against reasonable baselines such as the previous HELM works and Dreamer. It even shows experiments where it is about par with other methods rather than cherrypicking environments where theirs does better. \n\nOn the Memory and PsychLab environments, which seem the most explicit about requiring memory, this method does perform extremely well compared to baselines. The method does have some natural advantage on PsychLab because it has the benefit of the knowledge from CLIP, but HELMv2 also has CLIP encodings, so this isn't an unfair advantage. \n\nThe idea is interesting and makes sense. I think the motivation for having a memory that is (somewhat) interpretable is pretty strong.\n\nThe ablation studies are really interesting and alway very useful and careful.", "weaknesses": "The interpretability claims should I think be somewhat caveated in the paper. As the authors acknowledge, SHELM can in fact end up learning associations that are not quite semantically correct (as stated in line 224). I get what the authors are saying here that overall it is much more aligned semantically than prior methods, but I think some of the statements about interpretability in the intro should be somewhat hedged.\n\n\nI did not quite understand from the paper how the vocab was determined and it seems potentially limiting. Reading section 2.2, it looks like it finds the token overlap between the LM and CLIP and appends a custom prompt. First, does this imply that you cannot encode anything longer than a single language token (not including the prompt?) That seems like a big limitation if true. Also, how are the prompts decided in each environment (is there a table somewhere)? This is some additional information that needs to be hand defined per envionment.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the positive feedback from the reviewer.\n\n**Weaknesses:**\n\n - We agree with the reviewer that some of our claims about interpretability were a bit unfortunate. Therefore, we have revised the claims w.r.t. interpretability of the decision-making process and user-trust. Specifically, we will rewrite the second paragraph in the introduction as well as tone down claims in the abstract to clarify that the enhanced interpretability aspect is enabled by representing past observations in the form of human-readable language. Further we will add intuitive examples on how and why this is useful, e.g., enhanced debugging of modular architectures in RL for developers.\n - For now, we only encode observations into a single token or a set of tokens. We agree, this is a limitation that we will make more explicit in our revised version. Table 1 in the appendix shows environment specific prompts and appendix B elaborates on the prompt selection strategy. We only used prompts for Psychlab and Avalon, since it did not seem to qualitatively improve retrieval. We also want to highlight that these prompts are not necessary, but useful since they make the retrieval process less noisy.\n\n"}, {"review_id": "vq1fIfoTNq", "reviewer": "Reviewer_FbDD", "summary": "This study presents Semantic History-Embedded Language Model (SHELM), a new interpretable memory mechanism for reinforcement learning (RL) agents in partially observable scenarios. Current memory methods are often uninterpretable for humans, which hampers their use in critical areas like autonomous driving or healthcare. SHELM addresses this by using human language to clarify the decision-making process. It employs CLIP to convert visual data into language tokens and a pre-trained language model to create an interpretable past record. Despite limitations in synthetic environments, SHELM performs well in photorealistic settings and shows robustness to environmental disturbances.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper's key strength is its development of the Semantic History-Embedded Language Model (SHELM), an innovative memory mechanism that brings improved interpretability to reinforcement learning. This is crucial for deploying RL agents in high-stake applications where human understanding of the decision-making process is vital. SHELM's ability to link visual inputs with language tokens makes the agent's memory easily understandable for humans. Moreover, the method demonstrates state-of-the-art performance in photorealistic environments and superior robustness to environmental noise compared to existing approaches. In addition, the narrative of the paper is very clean and clear, making it easy to follow.", "weaknesses": "- There are minor typos in the paper that should be corrected for revision. E.g. in line 56: \"a interpretable\" -> \"an interpretable\"\n- In the last paragraph of the Introduction, there is a claim saying: \"On Avalon we find that a semantic memory does not yield any benefits. In fact, we observe that agents with memory tend to learn faster, but reach lower final performance than a markovian baseline.\" I am wondering if there is any explanation for this. I also could not find where it has been discussed in the Experiments sections. Could you please provide a pointer for this?\n- How can we ensure that the actor-critic head does not ignore the features passed to it from the TrXL? This means that the agent may make decisions solely based on features passed to it from the CNN module. Even if this happens at very few decision steps, then this could lead to serious questions about the idea of the paper and the extracted insights.\n- This work endeavors to mitigate the black-box nature of deep RL agents through the utilization of a black-box tool (LM). I would use the metaphorical concept of \"fighting fire with fire\" to describe this paper. As discussed in the paper, the ultimate aspiration is to ensure interpretability throughout the decision-making process when applying such methodologies in practical scenarios. However, the lack of interpretability and transparency in the LM raises profound doubts regarding the fundamental premise of the paper.\n- From Figure 3, SHELM is only outperforming baselines on the MiniGrid Memory task. On other tasks, Either HELMv2 or Dreamerv2 is better. Is this a correct observation?\n- In line 145, there are references to Figures 8 and 9. But these figures are not in the main paper. I would suggest moving the discussions of those figures to the appendix where they are.\n- From the results of the paper, they are not as convincing as promised in the abstract. Could you please clarify this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and address the mentioned weaknesses as follows:\n\n - Thank you for pointing out the typos, we will correct them.\n - Thank you for pointing out the confusion about the Avalon results. Indeed this claim is wrong, since we observed performance on-par for Dreamerv2 and HELM with PPO. However, after 10M timesteps there is a significant improvement for HELMv2 and SHELM over PPO. Thus, the addition of memory can facilitate sample efficiency on Avalon.\n - If the task is unsolvable without a memory component, then the policy must rely on the history, which is the case for MiniGrid-Memory and Psychlab environments. While for MiniGrid-Memory only the final decision is dependent on the memory, the dependency on memory is very frequent, i.e. approx. every 5 steps for episodes of length > 300. We will point this out in our revised version.\n - We agree with the reviewer that the downstream policy itself remains a black-box, but we would like to point out that we explicitly specified that our work only focuses on enhancing interpretability in the memory module in line 326-327. However, we agree with the reviewer that some of our claims about interpretability were a bit unfortunate. Therefore, we have revised the claims w.r.t. interpretability of the decision-making process and user-trust. Specifically, we will rewrite the second paragraph in the introduction and tone down claims in the abstract to clarify that the enhanced interpretability aspect is enabled by representing past observations in the form of human-readable language. Further we will add intuitive examples on how and why this is useful, e.g., enhanced debugging of modular architectures in RL for developers.\n - Considering only Figure 3, then yes, SHELM only significantly outperforms the competitors in the MiniGrid-Memory environment. This is due to the fact that the other environments are solvable by a memory-less baseline (which we add in the revised version). Further, on Avalon SHELM reaches the highest performance after 10M interaction steps and on Psychlab, which presents the highest dependency on memory, SHELM significantly outperforms competitors. We will add a more extensive discussion about our experimental results and limitations in our revised version.\n - Thank you for pointing out the references to figures from the appendix. We will move the discussion about these results accordingly.\n - We have claimed in the abstract: “Our memory mechanism achieves state-of-the-art performance in photorealistic environments where memorizing the past is crucial to solve tasks.” Indeed, this is a bit misleading. While SHELM achieves SOTA performance on MiniGrid-Memory, we are unable to compare to prior SOTA methods on Psychalb since those report human normalized scores and didn’t publish either human scores for normalization nor the codebase. Also, we exclude Avalon from the claim since our experiments have shown that memory does not appear to be crucial to solve tasks there, as our experimental results show (memory-less based agent performs best). So we rephrased this claim to: “On a photorealistic continuous recognition task where memorizing the past is crucial, our memory mechanism converges two orders of magnitude faster than prior methods.”\n"}, {"review_id": "rL4T7y5JFe", "reviewer": "Reviewer_f2kZ", "summary": "This work introduces a a memory mechanism for RL agents that relies on foundation models but does not require training  for correct functioning. The main contribution of this memory architecture is that memory tokens are preserved in a \"natural language\" space, making easier for humans to interpret the decision process of the agent", "questions": "I didn't see mention of it in the paper but authors provide the code in the suplemental material, is there intention to publish it whit the paper?\n\n\n------\nPost rebuttal \n\nI believe authors have correctly addressed my concerns and I am updating my score", "limitations": "Authors address some limitation at the end of the work but I missed some further discussion and study on the interpretability of the method and it's limits, specially since authors state interpretability as one of the core goals of the work. Specifically, I miss authors including some in-depth analysis on how interpretable the decision making really is rather than relying so heavily in the assumption that it is interpretable because language is used in the memory stored.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work addresses long-term memory and interpretability, two challenging yet core research areas in the RL field. \n\nAuthors provide a very deep and broad empirical evaluation and analysis of their method and the baselines, highlighting clearly why each of them robust or weak on the different tests. The fact that the conclusions are drawn over 4 very different environments with 30 independent runs makes the reader confident on the results obtained. The ablation study included also reinforces the results.\n\nThe work does also a broad coverage of the existing literature and related works", "weaknesses": "The main weakness of this work and the reason why I cannot give a stronger mark with the current version is the clarity of the method and it's differences with HELMv2 which this work is heavily based on. Specifically I would extend section 2.2:\n\n* I would expand figure 2, where the method is explained, to highlight what are the key pieces that change from HELMv2. The paper does refer to HELMv2 and the main differences at a high level, but I think that including a depiction would greatly help the readers as I had to go several times through 2.1 and 2.2 to understand the differences\n* Lines 100-101, you state \"Instead of merely fitting the statistics of the respective embedding spaces we introduce\n101 a token-retrieval mechanism that preserves the semantics extracted by CLIP\" I think this requires detailed explaining\n* Lines 108-113 the steps are clearly explained but I miss the \"why\" are the authors introducing those changes, what each of them is trying to achieve\n* Line 119, you were explaining memory compression level then follow with  \"by performing this procedure at every time step, we build up a human-readable representation of the history of the current episode\" I would reiterate here what is making thins human readable (the representation, not the compression) and again contrast it with HELM.\n\nTo allocate space for the depictions and explanations I mention above I would merge parts of the experimental discussion or move some pieces to the appendix.\n\nI also think that authors give a lot of weight on interpretability of the method but then the work lacks doing a proper analysis of it as I note in the limitations box below\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and adopted the following changes:\n\nWe have updated Figure 2 of the paper to make the differences to HELMv2 more explicit (see Figure 1 in accompanying one-page pdf). Along with the updated figure we elaborate on the following points more in-depth:\n\n**Clarification on Line 100:**\n\nWe replace the batchnorm component of HELMv2 with a token-retrieval mechanism. The batchnorm component simply adapts the statistics of the CLIP embeddings ($\\mu_{\\text{CLIP}}$, $\\sigma_{\\text{CLIP}}$)  to those of the LM embedding space ($\\mu_\\boldsymbol{E}$, $\\sigma_\\boldsymbol{E}$). Specifically, the resulting CLIP embeddings are first centered by $\\mu_{\\text{CLIP}}$ and scaled by $\\sigma_{\\text{CLIP}}$, before being re-scaled by $\\sigma_\\boldsymbol{E}$ and shifted by $\\mu_\\boldsymbol{E}$. Thereby, the resulting embedding lives in the pretrained token embedding space of the TrXL. This process, however, does not transfer any semantics between the respective embedding spaces. We can sidestep this issue by first retrieving tokens similar to an observation in CLIP space and subsequently providing these tokens in text form to the LM.\n\n**Clarification on line 108-113:**\n\n - The overlap of the CLIP and TrXL vocabularies is determined to (i) avoid loss of information due to different tokenizers used by CLIP and TrXL, and (ii), to control for the number of tokens TrXL receives per observation.\n - CLIP was pre-trained on image-caption pairs and prompts have shown to significantly enhance retrieval performance. Following the original ImageNet prompting procedure of [1] (https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb) we augment each token in the vocabulary overlap with a set of prompts and represent each token as the mean over its prompt-augmented embeddings. \n - We will also clarify in Section 3.1 that we did not consistently observe improvements in token retrieval. For example in the MiniGrid and MiniWorld environments the prompts did not substantially change the set of retrieved tokens. On Avalon and DMLab, however, handcrafted prompts lead to a less noisy retrieval.\n\n**Clarification on line 119:**\n\nSince visual observations are represented in the form of language tokens in the memory, they can be interpreted by humans. While HELM and HELMv2 also leverage the latent structure of a pre-trained LM, they do not explicitly represent the observations in the form of text, thus, do not provide a human-readable form of past observations.\n\n**Question on Code:**\nYes, we have already added our code in the supplementary material during the submission and will make it publicly available.\n\nWe hope these changes clarify the points raised by the reviewer. We will update those in the revised version of our manuscript. Further, we will merge the second and third paragraph in the introduction and tone down claims w.r.t. interpretability such as “interpretable decision-making”, or “improved user-trust”. Also, we will put into perspective that the enhanced interpretability aspect is enabled by representing past observations in the form of language and added intuitive examples on how and why this is useful, e.g., enhanced debugging of modular architectures in RL for developers. This will provide some space to accommodate a more extensive elaboration on methodology. Finally, we will devote a new section to an extensive discussion about limitations of SHELM.\n"}, {"review_id": "y4fFyiTJC7", "reviewer": "Reviewer_RcPE", "summary": "This paper addresses the problem of partial observability by proposing a semantic-enhanced method. This approach converts environmental observations into human-readable language tokens and incorporates them into the hidden state embeddings. Relative to other methods, this technique exhibits resilience against representation collapse and is sufficiently explainable, making it viable for high-stakes domains.", "questions": "As stated in the weakness part. ", "limitations": "This paper contains the limitation part including the modality gap and captions diversity. \nI believe the author should dive more into the caption design of history observations for more complicated environments. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The presented method is concise and straightforward to implement. Moreover, its potential for broad applicability across diverse domains stems from the inherent generalizability of the foundational model (CLIP). \n\n(2) The paper provides a comprehensive review of foundational knowledge, enhancing its readability for varied audiences.\n\n(3) The extensive experiments conducted show the effectiveness of the Semantic HELM approach.", "weaknesses": "The proposed method, which embeds a subset of predefined tokens as semantic descriptions of each observation frame, appears to be a rudimentary solution to the partial observability problem. Put simply, it may only be suitable for simplistic environments that rely on recognizing specific objects. In comparison to a basic model that labels the current frame and concatenates the observation with its labels—thus serving as input to a memory-based policy—I don't see this method will stand out. \n\nIf the paper aims to show that a language-based memory adequately serves as policy memory, it would be more beneficial to delve into the specifics of description design rather than utilizing a sequence of independent objects. The paper should also engage with environments where past events significantly influence current decisions. I further question whether a singular language-modality memory is sufficient for more complex environments. For example, in three-dimensional video games like Minecraft, the absence of a visual-modality context may lead to inconsistent behavior sequences. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and address the following concerns:\n\n**Rudimentary approach:**\n\nWe agree with the reviewer that our approach is simple and builds on existing works. However, we disagree that SHELM is only suitable for simplistic environments. Our semantic memory performs particularly well for environments that contain real-world objects, as in Psychlab (Figure 5), where we can exploit the strength of the large-scale pre-training of CLIP. We strongly believe that agents deployed in the real world require a form of semantic memory to efficiently interact with their surroundings. Also, we have added results for Dreamerv3 on Psychlab. SHELM significantly outperforms Dreamerv3 there. Further, the description of a “basic approach that labels the current frame and concatenates the observation with its labels'' somewhat describes what we propose with SHELM.\nIf the reviewer is aware of existing works that we missed and are similar, we kindly ask the reviewer to make us aware of them.\n\n**Choice of environments:**\n\nOur MiniGrid-Memory and Psychlab environments are unsolvable without a memory component, thus past events significantly influence future decisions. While in the MiniGrid-Memory environment this is limited to the final decision in the POMDP, this effect is heavily exacerbated in the continuous recognition task of Psychlab. Each observed object heavily impacts future decisions.\n\n**Should memory be solely based on language?**\n\nThere is an increasing body of literature that a memory based on language can achieve impressive results on three-dimensional video games, e.g., VirtualHome [1], or Minecraft [2]. SHELM is different to these methods by providing a memory in the form of language for visual environments.\n\n[1] Pre-Trained Language Models for Interactive Decision-Making, Li et al., NeurIPS 2022\n\n[2] Voyager: An Open-Ended Embodied Agent with Large Language Models, Wang et al., arXiv/2305.16291, 2023\n\n**Description design:**\n\nWe agree with the reviewer that this is a particularly interesting aspect. We showed that a compression in the form of single tokens is already sufficient to solve complex tasks such as our continuous recognition task (Figure 5). Considering more extensive descriptions, however, poses an additional problem of required context length, especially if each observation is to be stored in the memory. This poses an interesting problem for future work.\n"}]

Author rebuttal:
We would like to thank the reviewers for the time they invested to give constructive feedback which helps us to greatly improve our manuscript.

We are glad the reviewers found our paper clear and well-written (xnPR, FbDD), relevant (xnPR), found our empirical analysis thorough (a9Pk, f2kZ) and our method straightforward to implement (RcPE).

To address the weaknesses pointed out to us we have made the following changes to our manuscript:

 - We have toned down claims w.r.t. interpretability of the decision-making process, user-trust, and high-stake environments. Further, we add a motivation to our approach and clarify that enhanced interpretability is caused by the fact that we represent past observation in the form of language. Also, we provide the reader with intuitive examples on why and how memory represented in written language may be useful.
 - We devote a section to limitations in the main paper and make a more in-depth analysis on results and potential shortcomings in terms of vocabulary, prompt design, and interpretability.
 - We extend our methodology section and figure 2 (see accompanying one-page pdf) to make the difference to HELMv2 more explicit and provide the reader with a deeper understanding of the methodology of SHELM. We have also updated ill-defined terminology such as “Markovian baseline”.
 - We provide additional results (see accompanying pdf) for
   - Memory-less PPO on MiniGrid and MiniWorld
   - Dreamerv2 on Avalon
   - Preliminary results on Dreamerv3 on MiniGrid and Psychlab
   - Preliminary ablation study for different vision encoders
   - BLIP-2 qualitative results for captioning of observations

Thereby, we have addressed the raised weaknesses by the reviewers and would gladly answer additional questions about our manuscript.


Meta-review: The paper proposes a new memory mechanism in RL that helps with partially observable environments. The memory mechanism is based on the words that a Visual-Language model has extracted from past observations (images).  As such the memory is an abstraction of past experiences, and can only capture the part of the state that can be captured in language.

The paper has received 2 x Accept, 1 x Weak Accept, 3 x Borderline Reject. 

On the positive side, most reviewers agree that the paper is well motivated (xnPR), the presented method is well explained (UimZ). They also find the approach novel (extension of another work HELMv2), and the novelty is sufficient. Further, they believe it has “potential for broad applicability” (RcPE). Also, most reviewers state that authors presents  “results on lots of challenging domains (minigrid, MiniWorld, Avalon, PsychLab, etc) “  and  “extensive and thorough experiments” (xnPR, RcPE, a9Pk, f2kZ). These experiments seem to be convincing in terms of performance benefits of the memory approach. 

There are two issues with the work that the reviewers point out. First, being of lesser concern, is that the suggested interpretability of the approach hasn’t been empirically validated and can be argued against (xnPR, UimZ, a9Pk, f2kZ). Nevertheless, in the rebuttal the authors propose to weaken this claim which puts the reviewers at ease.

The second concern, raised by UimZ, RcPE, is that the proposed memory consisting of words is too much of an abstraction — such representation isn’t great for manipulation tasks (UimZ) or complex environments (RcPE) where words aren’t sufficient to express what the agent sees. However, the experimental results demonstrate the merits of the approach, at least on the environments presented.

As such the AC, after consultation with SAC, accepts the paper to Neurips.
Decision: Accept (poster)