SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Authors: Cansu Sancaktar
Christian Gumbsch
Andrii Zadaianchuk
Pavel Kolev
Georg Martius
Venue: ICML 2025 poster (2025)
Keywords: intrinsic motivation
exploration
foundation models
model-based RL
Primary area: reinforcement_learning->deep_rl
TL;DR: We propose SENSEI to equip model-based RL agents with intrinsic motivation for semantically meaningful exploration using VLMs.

Abstract: Exploration is a cornerstone of reinforcement learning (RL). Intrinsic motivation attempts to decouple exploration from external, task-based rewards. However, established approaches to intrinsic motivation that follow general principles such as information gain, often only uncover low-level interactions. In contrast, children’s play suggests that they engage in meaningful high-level behavior by imitating or interacting with their caregivers. Recent work has focused on using foundation models to inject these semantic biases into exploration. However, these methods often rely on unrealistic assumptions, such as language-embedded environments or access to high-level actions. We propose SEmaNtically Sensible ExploratIon (SENSEI), a framework to equip model-based RL agents with an intrinsic motivation for semantically meaningful behavior. SENSEI distills a reward signal of interestingness from Vision Language Model (VLM) annotations, enabling an agent to predict these rewards through a world model. Using model-based RL, SENSEI trains an exploration policy that jointly maximizes semantic rewards and uncertainty. We show that in both robotic and video game-like simulations SENSEI discovers a variety of meaningful behaviors from image observations and low-level actions. SENSEI provides a general tool for learning from foundation model feedback, a crucial research direction, as VLMs become more powerful.

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Cansu Sancaktar 12 Christian Gumbsch * 113 Andrii Zadaianchuk Pavel Kolev!
Georg Martius 12
Abstract
1. Introduction
Exploration is comerstone of reinforcement learning (RL). Intrinsic motivation attempts to decouple exploration from external, task-based rewards. However, established approaches to in- trinsic motivation that follow general principles such as information gain, often only uncover low- level interactions. In contrast, children's play sug- gests that they engage in meaningful high-level behavior by imitating or interacting with their caregivers. Recent work has focused on using foundation models to inject these semantic biases into exploration. However, these methods often rely on unrealistic assumptions, such language- embedded enviromments or access to high-level actions. We propose SEmaNtically Sensible Ex- ploration (SENSEI), framework to equip model- based RL agents with an intrinsic motivation for semantically meaningful behavior. SENSEI dis- tills a reward signal of interestingness from Vision Language Model (VLM) annotations, enabling agent predict these rewards through world model. Using model-based RL, SENSEI trains an exploration policy that jointly maximizes seman- tic rewards and uncertainty. We show that in both robotic and video game-like simulations SENSEI discovers a variety of meaningful behaviors from image observations and low-level actions. SEN- SEI provides a general tool for learning from foun- dation model feedback, crucial research direc- tion, VLMs become more powerful.'
Achieving minttiiaallymotivvteeed learning in artificial agents has been long-standing dream, making it possible to decouple agents' learning from experimenter manually crafting and setting up tasks. Thus, the goal in intrinsically- motivated reinforcement learning (RL) is for agents to ex- plore their environment efficiently and autonomously. con- stituting a free play phase akin to children's curious play. Various intrinsic reward definitions have been proposed in the literature, such as aiming for state space coverage (Belle- mare et al., 2016; Tang et al., 2017; Burda et al., 2019), nov- elty retrospective surprise (Pathak et al., 2017; Schmidhu- ber, 1991), and information gain of a world model (Pathak et al., 2019; Sekar et al., 2020; Sancaktar et al., 2022). How- ever, when an agent starts learning from scratch, there is one fundamental problem: just because something is novel does not necessarily mean that it contains useful generalizable information for any sensible task (Dubey & Griffiths, 2017).
Imagine robot facing desk with several objects. The robot could explore by trying to move through the entire manipulable space or hitting the desk at various speeds. In contrast, human common sense would primarily focus on interacting with the objects or drawer the desk since po- tential task distributions likely revolve around those entities.
Agents exploring their environment with intrinsic motiva- tions suffer from a chicken-or-egg problem: how do you know something is interesting before you have tried it and experienced interesting consequences? This is a bottleneck for the types of behavior that an agent can unlock during free play. We argue that incorporating human priors into exploration could alleviate this roadblock. Similar points have been raised for children's 's play. During the first years of life, children are surrounded by their caregivers who ide- ally encourage and reinforce them while they explore their environment. Philosopher and psychologist Karl Groos has stipulated that there "a strong drive in children to observe the activities of their elders and incorporate those activities into their play" (Gray, 2017: Groos & Baldwin, 1901).
"Equal contribution 'Autonomous Learning, University of Tübingen 'Empirical Inference, Max Planck Institute for In- telligent Systems -oguniCCe Modeling, University of Tübingen "Cognitive and Clinical Neuroscience. TU Dresden *vISLab, University of Amsterdam. Correspondence to: Cansu Sancaktar Scansu.sancaken tue.mpg.de>, Christian Gumbsch <chris @ @uunbsch.mbb......
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A potential solution in the age of Large Language Mod- els (LLMs), is to utilize language as cultural-tansmitter to inject "human notions of interestingness" (Zhang et al.,
Project website
with
videos
and
code:

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
You: Which of the following imagrs is more interesting for cenvironmant discription>>?
VL.:: Trage is interesting,
distillation
You:
SENSEI
environment
agent
world model
Oi, T2
On Pr, Pr
VLM:
Image 2 is more interesting.
Figure 1. SENSEI overview: (a) During pre-training we prompt a VL.M to compare ohservations (e.g. images) from an environment with respect to their interestingness. We distill this ranking into reward function (SENSEI) for exploration. (b) An exploring agent not only receives observations (o:) and rewards (rr) from interactions with the environment, but also semantic exploration reward (r2mm) from SENSEL (c) The agent learns world model from its experience to judge the interestingness (f2"m) of states without querying SENSEI.
2023a) into RL agents' exploration. LLMs are trained on an immense amount of data produced mostly by humans. Thus, their responses likely mirror human preferences. However, the most prominent works in this domain assume (1) innguage-grounded environment (Zhang et al., 2023b; Du et al., 2023; Klissarov et al., 2023), (2) the availability of an offline dataset with exhaustive state-space coverage (Klis- sarov et al., 2023), or (3) access to high-level actions (Zhang al., 2023a; Du al., 2023). These assumptions de- tached from the reality of embodied agents, e.g. in robotics, which don't 't come with perfect state event captioners, pre- existing offline datasets nor with robust, high-level actions. Furthermore, none of these approaches learn an internal model of "interestinnneess" " Thus, they rely on the LLM, or a distilled module, to continuously guide their exploration.
through SENSEI enable fast learning of downstream tasks.
2. Method
We consider the setup of an agent interacting with a Partially Observable Markov Decision Process. At each time t, the agent performs an action a E A and receives an observation o, E O, composed of an image and potentially additional information. We assume that there exist one or more tasks in the environment with corresponding rewards r task R. However, during task-free exploration, the agent should select its behavior agnostic to task rewards.
We assume that SENSEI starts with a dataset Dinit C O, collected from self-supervised exploration with information gain as intrinsic reward (Sekar et al., 2020), thus not relying a pre-existing expert dataset. SENSEI has access to pretrained VLM and is provided with short description of the environment, either from a human expert generated by the VLM, based on some observations from Dinit, Prior to task-free exploration, SENSEI distills a semantic exploration reward function from VLM annotations (Sec. 2.1). During exploration, SENSEI learns a world model (Sec. 2.2) and optimizes an exploration policy through model-based RL (Sec. 2.3).
In this work, we propose SEmaNtically Sensible Explo- ration (SENSEI), framework for Vision Language Model (VLM) guided exploration for model- based RL agents, il- lustrated in Fig. 1. SENSEI starts with a short description of the environment and a dataset of observations (e.g. images) collected through self-supervised exploration. A VLM is prompted compare the observations pairwise with respect to their interestingness and the resulting ranking is distilled into reward function. When the agent explores its vironment, it receives semantically-groundee rewards from SENSEI. It learns to predict this exploration signal through its learned world model, corresponding to an internal model of "interestingness" The agent improves its exploration strategy by aiming for states for which it predicts high interestingness and then branching out to uncertain situations. Our main contributions are as follows:
exploration
2.1. Reward function distillation: MOTIFate SENSEI
Prior to task-free exploration, SENSEI needs to distill a semantically grounded intrinsic reward function R2 with learnable parameters v based on the preferences of a pre- trained VLM. While the overall framework of SENSEI is agnostic to the exact distillation method, we chose to use a vision-based extension of MOTIF (Klissarov et al., 2023; illustrated in Fig. 2a), which we refer to as VLM-MOTIF.
We propose SENSEI, a framework for foundation model- guided exploration with world models.
We show that SENSEI can explore rich, semantically meaningful behaviors with few prerequisites.
Original MOTIF (Klissarov et al., 2023) assumes an environ- ment where events captioned in natural language. Thus, they can use L.MSs to annotate captions of observations.
We demonstrate that the versatile world models learned
2

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
self-supervised exploration data
Gis
VLM
pairwise comparison
hp
h2
MOTIF
121
1/2
22
uninteresting
ranked data
interesting
Ry
intrinsic motivations
distilled reward function
(a) reward function distillation
(b) world model
Figure Intrinsic rewards SENSEI: (a) Prior to exploration, we prompt GPT-4 to compare images with respect the interestingness for a certain environment. From the resulting ranking we distill a reward function Rs using VLM-MOTIF. (b) Later, an agent learns an RSSM world model from exploration. From each state the agent predicts two intrinsic rewards, i.e., the distilled semantic rewards 2em and umcertainty-basel rewards dis
MOTIF consists of two phases. In the first phase of dataset annotation, the pretrained foundation model is used to com- pare pairs of observations, creating dataset of preferences. For this, prompt the VLM with an environment descrip- tion and pairs of observations from Dinit, asking the VLM which image it considers to be more interesting. The anno- tation function is given by the VLM O x O + V, where O is the space of observations, and y = (1, 0i is space of choices for the first, second or none of the observations. In the reward training phase, a reward function is derived from the VLM preferences using standard techniques from preference- RL (Wirth et al., 2017). A cross-entropy loss function is minimized on the dataset of preference pairs to learn a semantically grounded reward model R.p : O -> R. We use the final semantic reward function Ry when- ever the agent interacts with its environment: the agent not only receives observation o and reward r, after execut- ing an action de, but also receives exploration reward y <- Rs (o.) (see Fig. 1, center).
h4 each time i, the RSSM samples a new stochastic state 2; from posterior distribution Is computed from the cur- rent deterministic state h, and new observation o, (Eq. 1). The RSSM updates its deterministic memory ht+1 based on the action a, and previous latent states (Eq. 2). Then, the model predicts the next stochastic state be+1 (Eq. 3). Once the new observation 0++1 is received, the next posterior dø is computed and the process is repeated.
Besides encoding dynamics within its latent state, the RsSM is also trained to reconstruct external quantities y/ from its latent state via output heads 0g:
Output heads: Út ~ O h4s z1)
(4)
with Vr for, Ct Tt, r.mm The RSSM of DreamerV3 (Hafner et al., 2023) reconstructs observations or, episode continuations and rewards ти. For SENSEI, we addi- tionally predict the semantic exploration reward reem. The world model trained end-to-end to jointly optimize the evidence lower bound.
mamantically-groundde
2.2. World model: Let your SENSEI dream
Thus, world model learns predict semantic interest- ingness prem of states (see Fig. 1, right). We could base exploration exclusively on this signal. However, we (1) ex- pect to face many local optima when optimizing for this signal and (2) we do not want to only explore a fixed set of behaviors, but ensure that the agent goes for interesting and yet novel states. To overcome this limitation, Klissarov et al. (2023) post-process r rem and normalize it by episodic event message counts. As do not assume ground-trutt (2) countable event captions, we instead combine our seman-
We assume a model-based setting, i.c., the agent learns world model from its interactions. Following DreamerV3 (Hafner et al., 2023), we implement the world model as Recurrent State Space Model (RssM) (Hafner et al., 201 The RSSM with learnable parameters & is computed by
Posterior: Ze ~ q (Z+ het Ot)
(1)
Dynamics:
h141 = (a beg z,)
Prior:
E ps (2 h(41)
(3) tic reward signal with epistemic uncertainty. a quantity that was shown to be an effective objective for mosel-based exploration (Sekar et al., 2020; Pathak et al., 2017; San- caktar et al., 2022). Following Plan2Explore (Sekar et al.,
In short, the RSSM encodes all interactions through two la- tent states, stochastic state 2, and a deterministic memory
3

---

SENSEI:
Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
unimtrreiingg
unitterrcing
on semantic interestingness. The two trade-off factors 380 and geeppore, well the quantile k are hyperparameters. More details on this adaptation and hyperparameters can be found in Suppl. B. We learn an exploration policy based on rt using DreamerV3 (Hafner et al., 2023).
Intersiing
Interesting
(a) go: AREm Q+
(b) explore: Fi Q+
3. Related work
Figure 3. Switching exploration behavior in SENSEI: (a) When the agent is an uninteresting state it mainly strives to maximize "interestingness" (r1"m), e.g. by going to key. (b) When in an interesting state, the agent more strongly attempts to increase uncertainty (r2"h) by trying new actions, e.g. picking up the key.
Intrinsic rewards are applied either to facilitate exploration in tasks where direct rewards are sparse or in task-agnostic setting where they help collect diverse data. Many different reward signals have been proposed as exploration rewards (Baldassarre & Mirolli, 2013), such as prediction error (Schmidhuber, 1991; Pathak et al., 2017; Kim et al., 2020), Bayesian surprise (Storck et al., 1995; Blaes et al., 2019; Paolo et al., 2021), learning progress (Schmidhuber,
2020), we train an ensemble of N models with weights
{0) :...., O to predict the next stochastic latent states with
1991; Colas et al., 2019; Blaes et al., 2019), empower- ment (Klyubin et al., 2005; Mohamed & Jimenez Rezende, 2015), metrics for state-space coverage (Bellemare et al., 2016; Tang et al., 2017: Burda et al., 2019) and regularity (Sancaktar et al., 2024). While effective for low- dimensional observations, such objectives are challenging to apply for high-dimensional image observations. Here, alternatives are employing low-dimensional goal spaces (Colas al., 2019: OpenAI et al., 2021; Nair al., 2018; Pong et al., 2019; Zadaianchuk et al., 2021; Mendonca et al., 2021) learning latent world models (Hafner al., 2019a; 2023; Gumbsch et al., 2024) that can be employed for model-based exploration (Pathak et al., 2019; Sekar et al., 2020). In particular, Plan2Explore (Sekar et al., 2020) uses ensemble disagreement of latent space dynamics predictions as an intrinsic reward. While this a very general strategy for exploration, this could be limited in more challenging en- vironments where semantically meaningful or goal-directed behavior (Spelke, 1990) is needed for efficient exploration.
Ensemble predictor: 2
gon (2 Zc, Zc; az).
(5)
We quantify epistemic uncertainty as ensemble disagree- ment rfth by computing the variance over the ensemble predictions averaged over latent state dimensions J:
dis = J Σ Var(2).)) r
Thus, the model learns to predict two intrinsic rewards (fgem, relit) for a state-action-pair (Fig. 2b).
2.3. Exploration policy: Go and Explore with SENSEI
We could use a weighted sum of the two intrinsic reward signals, e.g. r + Brriss as the overall reward expl for op- timizing an exploration policy. However, idcally the weight- ing of the two signals should depend on the situation. In un- interesting states, want the agent to mostly pursue inter- estingness (via reem). However, once the agent has found an interesting state, we would like the agent to branch out and discover new behavior (via r This follows the principle of Go-Explore (Ecoffet et al., 2021), where the agent should first go to a subgoal and explore from there (illustrated in Fig. 3). We implement this using an adaptive threshold pa- rameter {3e0, 3explore) with gexplore > ,88, whose value depends on the following switching criteria:
Exploration with foundation models: Recent improve- ments of in-context learning of LLMs open additional ways to explore using human bias of interestingness during ex- ploration (Klissarov et al., 2023; Du et al., 2023; Zhang et al., 2023a) and skill learning (Colas et al., 2020; 2023; Zhang et al., 2023b). MOTIF (Klissarov al., 2023) lever- ages LLMs to derive intrinsic rewards by comparing pairs of event captions, demonstrating its efficacy in the com- plex game of NetHack (Küttler et al., 2020). Similarly, ELLM (Du et al., 2023) uses LLMs to guide RL agents towards goals that are meaningful, based on the agent's cur- rent state represented by text. Furthermore, OMNI (Zhang et al., 2023a) introduces novel method to prioritize tasks using LLMs Thereby, OMNI focuses on tasks that are not only learnable but also generally interesting. LAMP (Adeniji et al., 2023) proposes to use VLMS for reward mod- ulation by first generating a set of potential tasks with an LLM and then generating task-based rewards using VLMs.
gexplore, dis dis g
if fgem 2 Qu(me)));; (7) otherwise.
expl Tt
ggem
Here Qu denotes the 4th quantile of psem, which we esti- mate through moving average. Thus, until certain level of faem is reached, the exploration reward mainly aims to maximize interestingness. After exceeding this threshold, exploration more strongly favors invertainty-mxximiinn behaviors. As soon as the agent enters a less interesting state with peem < QK, SENSEI switches back to focusing
4

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
2
5
4
5
10
R4, SIESO0I
50
100
150
200
250
12
4
5
2
3
10
1F
Rs.
Sieval
20
40
60
80
100
120
140
1
2
3
4
4
Ra, Steven CENIERLL
Ry, Spawec
0.5
50
100
150
200
250
2
3
5
2
4
5
Ri. Seweet GENERAL
paeu (xorm.)
Rg.
0.5
50
100
150
200
250
1 2
4
0.5
1
3
5
pres (norm.)
0.4
0.3
Out
Critical hict
AAAAAAS hort by rolsent
0.2
CIP Pol
Ruy SEEEE
315
325
335
345
(a) screenshots
(b) semantic exploration reward
Figure Semantic exploration rewards for example trajectories: From top to bottom we show example trajectories for Minilack KeyRoom and KeyChest (see Fig. 10 for map view), Robodesk, and Pokémon Red. We showcase rewards from VLM-MOTIF distilled from GPT-4 annotations. The reward trajectories peak at the "intcresting" moments of exploration, such as opening a drawer in Robodesk, picking up the key in MiniHack, or landing a critical hit in Pokémon. For Robodesk we show reward trajectories for both SENSEI and version of SENSEI with a more general, zero pre- pre-knowledge prompting strategy (SENSEI GENERAL, see Sec. 4.2.2).
Most works that rely on VLMs as reward sources try to solve the reward specification problem in RL. In these works, a task is assumed to be described as a language caption (Cui al., 2022; Rocamonde et al., 2023; Baumli et al., 2023; Adeniji et al., 2023), goal image (Cui al., 2022), as video demonstration (Sontakke et al., 2023). In particular, RL-VLM-F (Wang et al., 2024) uses very similar setup to SENSEI. Pairs of images from initial rollouts are compared using a VLM to distill a reward function via MOTIF (Klis- sarov et al., 2023). However, we assume model-based setup and do not explicitly prompt the task and distill environment-speccffe but general exploration reward.
Reward-shaping through VLMs:
4. Results
Our experiments set out to empirically evaluate the follow- ing questions:
1. Does the distilled reward function Rs from VLM anno- tations encourage interesting behavior?
2. Can SENSEI discover semantically meaningful behavior during task-free exploration?
3. Is the world model learned via exploration suitable for later learning to efficiently solve downstream tasks?
4. Can SENSEI be combined with extrinsic rewards to solve tasks that require substantial exploration?
5

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
ngen opent dior
agre rest to eoit
castrocment pewsed
cane: willont
chost visl Beyy
enstrnment
plobol
piiked
pewoed
5001
400
66k-
43-
00k-
200
6k-
0
PTX PPX
PSX
POX VENE Monn
0 PSX
PSN
VIL3
VLM Moter
Pix
Moner
Monr
(a) interactions and rewards in KeyRoom-S15 $15
(b) interactions and rewards in KeyChest
Figure Interactions MiniHack: We plot the mean number interactions with task-relevar objects and environment reward (unknown to the agents) collected by SENSEI, Plan2Explore (P2X) and pure VI.M-MoTHH (SENSeI with no information gain, i.e. / = 0) for KeyRoom- S15 (a) and KeyChest (b). Error bars show the standard error (10 seeds).
We answer these questions by (1) illustrating that the seman- tic rewards obtained from VLM-MonH reflect interesting events in the environment, (2) quantitatively showing that SENSEI leads to more interaction-rich behavior during task- free exploration, (3) employing the learned world models to successfully train task-based policies and (4) combin- ing SENSEI'S exploration strategy with extrinsic rewards to tackle a challenging environment that cannot be efficiently explored using rewards alone. We use three fundamentally different types of environments:
In all environments, we collect the initial dataset Dinit with Plan2Explore (Sekar et al., 2020), the current state-of-the in exploration with pixel-based observations. We collect data from 500k steps in MiniHack and Pokémon Red and IM steps in Robodesk. For data annotation, we use GPT-4 (details in Suppl. D).
4.1. Reward function of SENSEI
In Fig. 4 we illustrate how the distilled VLM-MOTIF reward function Rg assigns semantic rewards r? for exemplary sequences. In MiniHack, r:tm clearly jumps for sigmificant events. Frames & 3 in KeyRo 11oom-115 5 and KeyChest. respectively, are right before the key is picked up. Later, r increases further once the agent is at the door or chest with a key (Frame 3 in KeyRoom S15 and Frames 4&5 in KeyChest). For Robodesk, interacting with objects, r.em increases, e.g., when open- ing the drawer or pushing the blocks. For Pokémon Red, r rises while winning battle, with surges for inflicting damage and drops for setbacks such as getting poisoned. More examples of Robodesk are shown Suppl. Fig. 17 and examples of Pokémon Red in Suppl. Fig. 26.
MiniHack (Samvelyan al., 2021) is sandbox design RL tasks based NetHack (Küttler al., 2020). In Mini- Hack, an agent needs to navigate dungeons by meaningfully interacting with its environment, e.g. open a door with a key. We tested two tasks: fetching key in a large room to unlock a smaller room with an exit (KeyRoom-Si 5) or fetching key to open a chest in a maze of rooms (KeyChest). MiniHack uses discrete actions. As observations we use pixel-based, egocentric views around the agent and binary flag indicating key pick-ups (details in Suppl. C.2).
that the robot is
Robodesk (Kannan et al., 2021) is a multi-task RL bench- mark in which a simulated robotic arm can interact with vari- ous objects on desk, including buttons, two types of blocks, ball, a sliding cabinet, drawer, and a bin. For different objects, there exist different tasks, e.g. open re Robodesk uses pixel-based observations and continuous actions. In order to deal with occlusions, we use images from two camera angles for VL.M annotations but only one camera angle as input to our agents (details in Suppl. C.I).
4.2. Task-free exploration
1.2.1. MINIHACK
We quantify the interactions uncovered by SENSEI during fisk-free exploration in two tasks of MiniHack. For task- relevant events, the mean number of interactions are plotted in Fig. 5. SENSEI focuses more semantically interesting interactions compared to Plan2Explore, e.g. picking up key, opening a locked door, or finding the chest with a key, As a result, SENSEI completes both tasks more frequently than Plan2Explore during task-free exploration, as evident by the higher number of collected rewards. We believe this indicates that SENSEI is well suited for initial task-free exploration in these environments, enabling the discovery of state-space regions crucial for solving downstream tasks.
Pokémon Red Game Boy role-playing game where players control trainer exploring a world of collectible crea- tures called Pokémon. Exploration is essential for progress, as players must navigate vast, interconnected world and master semantic battle system (c.g. Water beats Rock) to defeat Gym leaders and become Pokémon Champion. Our implementation (Whidden, 2023; Suarez, 2024) use the raw game screen as pixel-based observations and discrete actions corresponding Game Boy button presses (see Suppl. C.3).
6

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Upriglat block
Flat block
4.2.2. ROBODESK
12k
30k-
Next, we analyze exploration in the challenging visual control suite of Robodesk. Here compare 1M steps of exploration in SENSEI with Plan2Explore and Random Network Distillation (RND, Burda et al., 2019), strong model-free exploration approach that uses prediction errors of random image embeddings as intrinsic rewards to max- imize state space coverage. Fig. 6 plots the mean number of object interactions during exploration for the three meth- ods. On average, SENSEI interacts more with most available objects than the baselines. As result, in majority of tasks SENSEI receives more task rewards during exploration than Plan2Explore or RND (shown in Suppl. E.3). Qualita- tively, we observe that Plan2Explore mostly performs arm stretches', whereas RND mostly moves the arm around in the center of the screen, mostly hitting buttons, as they are also centered on the table, and occasionally hitting objects.
15k
6k-
RND
P2X SEINPI SENSEI GENERAL
RND
P2X
SENSRI SPNSES GENERAL
Bell
Buttons
8k
20k-
10k-
4K
RND
P2X SENSRI
RND
P2X SUNSKI SENSER
GENERAL
Drawer
Sliding cabinet
6k-
40k-
20%
Thus, our semantic exploration scheme leads more object interactions than uncertainty- based exploration, even in a low-level motor control robotic environment.
RND
P2X SENSEI
RND
P2X SENSEI SENSEI GENERAL
GENERAAL
Is an environment description by human expert neces- sary for SENSEI? In the previous SENSEI experiments, we provided a small environment description in the prompts for the VLM annotations. We investigate whether SENSEI relies on this external description in Robodesk, and compare against a version of SENSEI using a more general, zero- knowledge prompting strategy (SENSEI GENERAL). SEN- SEI GENERAL first prompts the VLM for environment description given image of the environment and uses the generated answer as context to annotate the dataset of preferences (details in Suppl. D.2). As shown in Fig. 6, SEN- SEI GENERAL interacts roughly as often with the relevant objects as SENSEI, outperforming both Plan2Explore and RND in terms of overall number of object interactions. Thus, injecting external environment knowledge the prompts is not necessary and this step can be fully automated. This further cements the generality of our approach.
Figure 6. Interactions in Robodesk: We plot the mean number of object interactions during 1M steps of exploration for SENSEI (en- vironment description provided by us). more general variant of SENSEI with "LM-ggnerated environment description (SENSEi GENERAL.) Plan2Fxplore (P2X), and Random Network Distilla- tion (RND). Error bars show the standard deviation (3 seeds).
Is information gain crucial for SENSEI? We show re- sults for exploration with pure semantic reward rf corre- sponding to SENSEI without an information gain reward rdis (8 = 0). In this VLM-MOTIF ablation, we emphasize the crucial role of the information gain objective. Optimizing only for the semantic reward rfem can cause the agent to get stuck in local optima and hinder further exploration. For example in KeyRoom, the agent with "LM-MOTIF often picks up the key. However, it fails to explore the room well enough after key pick-ups to find and open the door and reach the exit, as reflected in the interaction metrics in Fig. 5. We observe a similar scenario for KeyChest: although the pure VLM-MOTIF agent reaches the chest often after hav- ing picked up the key, it collects substantially less rewards than SENSEL. For the episode to end, the agent needs to use the key to open the chest. The VLM-MOTIF agent, however, simply hovers around the chest. As being at the chest with key is an "interesting" state and opening chest immedi- ately terminates the episode, there is no real incentive for the agent to explore chest openings. This ablation shows the importance of combining novelty and usefulness in order to continually push the frontier of experience.
Ablations We perform ablations to see (1) how noisy an- notations from VLMS affect SENSEI compared to oracle annotator, (2) how much the behavior richness of the initial dataset affects SENSHI'S performance (Suppl. Fig. 14), and (3) ablate our Go-explore switching strategy. We observe that as VLMs get better, there is indeed more to gain from SENSEL, and richer exploration data helps SENSEI bootstrap faster. See Suppl. E.2 for more information. We further showcase the robustness of our Go-Fxplore switching strat- egy in terms of hyperparameter sensitivity compared to variant of SENSEI where the semantic and disagreement rewards get fixed weights (Suppl. E.6).
3 Interestingly, this can still lead to solving tasks. For example, stretching the against the sliding cabinet can close it, and stretching the arm toward a block can push it off the table.
7

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
SENSEI
PlanoExplore
D3eamerV3
PPO
SENSEI GENERAL
Plan2Esplore
Dreamer V3
8
segments
score
6
0..5
0.5
4-
2
0
0.5
0.5
100
200
300
400
1e6 steps
1e6 steps
4
episodes
(a) KeyRoom-SL S15
(b) KeyChest
(b) map segments explored
Figure? 7. Downstream task performance in MiniHack: Mcan
3
3
episode scores for (a) KeyRoom-S1 and (b) ReyChest, using world models learned during task-free exploration with SENSEI and PlanoExplore (P2X). We also compare learning from scratch with DreamerV3 and PPO. Shaded areas show standard error (10 seeds); curves are smoothed with window size of 3.
Pokemmnn level
2
50
100
0
100
200
300
400
4.3. Fast downstream task learning
map
% seeds
episndes
(c) highest level of Pokémon in party
We hypothesize that world models learned from richer ex- ploration would enable model-based RL agents to quickly learn to solve new downstream tasks. We investigate this in MiniHack by running DreamerV3 (Hafner et al., 2023) using the previously explored world models to learn a novel task-based policy. To this end, we initialize DreamerV3 with the pre-trained world models from the initial 500K steps of exploration (see Sec. 4.2). We compare world models from task-free exploration with either SENSEI or Plan2Explore. Additionally, we compare running DreamerV3 and training Proximal Policy Optimization (PPO, Schulman et al. 2017), a state-of-the-art model-free baseline, from scratch.
(a) map progress
Figure 8. Task-based exploration in Pokémon Red comparing SENSEI GENERAL to Plan2Explore and DreamerV3 for 750k steps. We partition the overall game map into unique map segments for different routes, towns, or buildings (details and full map in Suppl. C.3). We sequentially numbered segments that need to be traversed from game start (0) to the first Gym (9) and plot the percentage of random seeds that reach each segment (a). Temporal exploration trends are visualized by plotting the mean number of unique map segments visited (b) and the highest level of the agent's Pokémon (c) over episodes, smoothed with moving average (window size 5). Shaded areas indicate standard error (5 seeds).
Figure 7 shows the performance of task-based policies over training. previously explored world model from SENSEI allows the agent learn to solve the task faster than all other baselines. Compared to Plan2Explore, SENSEI allo- cates more resources to explore the relevant dynamics in the environment, e.g. opening the chest more, resulting in well-suited world model for policy optimization. Unlike the clear improvements of SENSEI, task-free exploration with Plan2Explore does not outperform learning a task policy from scratch with DreamerV3 consistently across environ- ments. In KeyRoom, the model-free baseline PPO takes more than 20M steps to consistently solve the task (full PPO curves in Supp. Fig. 13). Thus, in this task SENSEI outper- forms PPO by roughly two orders of magnitude. This shows the improved sample efficiency of our approach: combining foundation model-go -uided exploration and model-based RL. In KeyChest, the model-free bascline PPO shows the first successes in the tasks early during training, but on average takes longer to learn to reliably solve the task.
4.4. Task-based exploration
Some environments are complex that meaningful task progress is only possible through effective exploration. We investigate such task-based exploration in Pokémon Red. We compare 750k steps of exploration with SEN- SEI GENERAL (using rz + r eeppl) to Plan2Explore (using rt + Brilht and DreamerV3 (using only rt). Thus, all agents receive extrinsic task rewards r; during exploration, but only Plan2Explore and SENSEI also utilize intrinsic signals.
Progress in the game requires navigating a vast world to reach Pokémon Gyms, as well as assembling and training a strong team to defeat Gym Leaders. To evaluate spatial exploration, we partition the game world into distinct map segments, corresponding to towns, routes, forests, or build- ings (details in Suppl. C.3). Over the course of exploration, SENSEI consistently discovers new segments, outperform- ing baselines in terms of total map coverage (Fig. 8b). To assess whether this exploration is goal-directed, Figure 8a labels the specific segments required to progress from the game start (segment 0) to the first Gym (segment 9), and plots the segments reached per method (bigh-resolution map
In a supplementary experiment (Suppl. E.7), we analyze fast downstream task learning also on representative Robodesk tasks, and demonstrate more sample-efficient policy learning compared to exploration with Plan2Explore.
8

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
in Fig. 12). Only SENSEI reaches the first Gym, demonstrat- ing superior exploration aligned with the game's objectives.
Wang et al., 2024), SENSEI learns an internal model of inter- estingness. This is sensible design choice when working with world models (as detailed in Suppl. A.3), enabling SEN- SEI to predict semantic rewards also while imagining states during policy training. We demonstrate that this can lead to significantly faster learning, since both VLM guidance as well as model-based RL improve sample efficiency.
To assess battle-related progress, we track the levels of the agent's Pokémon (Fig. 8c), which serve as proxy for battle experience and overall strength. Dreamer fails to sufficiently explore the battle system and does not manage to train its Pokémon. Its highest-level Pokémon remains at the same level as at the start of the episode. In con- trast, SENSEI begins leveling up its Pokémon early during exploration, and from episode 100 onward, it consistently achieves higher levels than Plan2Explore. From episode 390 onward, SENSEI, on average, obtains twice as many level-ups per episode as Plan2Explore, indicating greater battle success and a higher potential for future encounters.
Limitations SENSEI benefits from fully-observabye obser- vations, e.g. images that capture all relevant aspects of the environment. The VLM annotations degrade when dealing with occlusions. In Robodesk we mitigate this using multi- plc camera angles. In future work this could be remedied further by annotating videos to better convey temporal or partially-observable information.
Together, these results highlight SENSEI'S ability to per- form meaningful, goal-directed exploration in rich, open- ended environments. We provide more detailed analysis in Suppl. E.8.
Future work One promising direction is to systemati- cally investigate iterative refinement of the semantic reward function. In Suppl. E.9, we show that incorporating SENSEI- collected data into a second round of annotations reduces out-of-din distribution errors for states not present in the initial annotation set. We believe this iterative process can un- lock increasingly complex behaviors with each generation. Another avenue is to explore SENSEI in photorealistic or real-world environments. Photorealism of observations are likely to help VLM annotations because large portion of VLMs' training data comes from real world photos or videos. Thus, SENSEI is likely to scale well to these settings.
Second generation of annotations With more explo- ration, and due to the vastness of the world in Pokémon Red, SENSEI increasingly enters regions outside the distri- bution of its annotation data (Dinit). This leads to degraded semantic rewards. In Suppl. E.9, we show how this lim- itation can be addressed by refining the reward function through a second round of VLM annotations on SENSEI- collected data. When continuing the SENSEI run used for annotation for 200 additional episodes, now using the up- dated reward function, the agent is able to defeat the first Gym and obtain the Boulder Badge. This marks critical milestone, highlighting the strong potential of iterative se- mantic reward refinement to unlock meaningful progress in complex environments.
Acknowledgements
The authors thank Sebastian Blaes and Onno Eberhard for helpful discussions. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS- IS) for supporting Cansu Sancaktar and Christian Gumb- sch. Georg Martius is a member of the Machine Learn- ing Cluster of Excellence, EXC number 2064/1 Project number 390727645. We acknowledge the financial sup- port from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01 8039BB. This work was supported by the Volkswagen Stiftung (No 98 571).
5. Discussion
We have introduced SENSEL. a framework for guiding the exploration of model- based agents through foundation mod- els. SENSEI bootstraps a model of interestingness from previously generated play data. On this dataset, SENSEI prompts a VLM to compare images with respect to their interestingness and distills a semantic reward function. SEN- SEI learns an exploration policy via bodel- RL using two sources of intrinsic rewards: (1) trying to reach states with high semantic interestingness and (2) branching out from these states to maximize epistemic uncertainty. We show that in the video game environments of MiniHack and Pokémon Red and a robotic simulation, this strategy leads to more meaningful interactions, e.g. opening chest with key or manipulating objects desk.
Impact Statement
This work introduces a framework for semantically mean- ingful exploration in reinforcement learning (RL), guided by intrinsic rewards distilled from lision-language mod- els (VLMs). The approach enables agents to efficiently discover useful, high-level behaviors without relying on task-based rewards. RL often suffers from computational inefficiencies due to extensive trial-and-eme processes, but effective exploration strategies can alleviate this by guiding agents towards more purposeful behaviors. In set-
Internal model of interestingness Unlike prior work of foundation guidee exploration (Klissarov et al., 2023;

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
tings, exploration poses additional challenges due safery concerns, aimless interactions can lead damage unsafe situations. By emphasizing semantically meaning- ful exploration, our approach offers step toward more energy-efficient and potentially safer exploration. We have identified no significant ethical concerns beyond standard considerations for responsibly deploying autonomous learn- ing agents.
Colas, C., Teodorescu, L., Oudeyer, P.-.., Yuan, X., and Côté, M.-A. Augmenting autotelic agents with large language models. In Conference on Lifelong Learning Agents, pp. 205-226. PMLR, 2023.
Cui, Y., Nickum, S., Gupta, A., Kumar, V., and Rajeswaran, A. Can foundation models perform zero-shot task specifi- cation for robot manipulation? In Learning for Dynamics and Control Conference, pp. 893-905. PMLR, 2022.
References
Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pp. 8657- 8677. PMLR, 2023.
Adeniji, A., Xie, A., Sferrazza, C., Seo, Y., James, S., and Abbeel, P. Language reward modulation for pretraining reinforcement learning. arXiv preprint arXiv:2308. 12270, 2023.
Baldassarre, G. and Mirolli, M. Intrinsically Motivated Learning in Natural and Artificial Systems. Springer Publishing Company, Incorporated, 2013. ISBN 364232374X.
Dubey, R. and Griffiths, L.. A rational analysis of curiosity. In Proceedings of the 39th Annual Conference of the Cognitive Science Society, pp. 307-312, 2017.
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune. First return, then explore. Nature, 590(7847); 580-586, 2021.
Baumli, K., Baveja, S., BBehbahani, F., Chan, H., Comanici, G., Flennerhag, S., Gazeau, M., Holsheimer, K., Horgan, D., Laskin, M., et al. Vision-language models source of rewards. arXiv preprint arXiv:2312.09187, 2023.
Gray, What exactly play, and why is it such powerful vehicle for learning? Topics in Language Disorders, 37 (3):217-228,, 2017.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based ex- ploration and intrinsic motivation. Advances in Neural Information Processing Systems (NeurIPS), 2016.
Groos, K. and Baldwin, E. The Play of Man. Appleton, 1901.
Blaes, S., Vlastelica, M., Zhu, J.-J., and Martius, G. Control What You Can: Intrinsically motivated task- planning agent. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Gumbsch, C., Sajid, N., Martius, G., and Butz, M. V. Learn- ing hierarchical world models with adaptive temporal ab- stractions from discrete latent dynamics. In The Twelfth International Conference on Learning Representations, 2024.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Explo- ration by random network distillation. International Conference on Learning Representations (ICLR), 2019.
Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv: S.fY:11803.11222 10122, 2018.
Chevelier-Boisvertt, M., Dai, B., Towers, M., Perez- Vicente, R., Willems, I.., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances in Neural Information Processing Sys- tems, 36, 2024.
Hafner, D. Benchmarking the spectrum of agent capabilities. arXiv preprint (rXiv:2109.06780 2021.
Hafner, D.
ing research. Embodied: Fast reinforcement learn- embodi ed, 2024.
com/daning
Colas, C., Fournier, P., Chetouani, M.. Sigaud, O., and Oudeyer, P.-.. CURIOUS: Intrinsically motivated mod- ular multi-goal reinforcement learning. In International Conference on Machine Learning (ICMI.), 2019.
Hafner, D.,, Lillicrap, T., Ba. J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019a.
Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin- Frier, C., Dominey, P., and Oudeyer, P.-.. Language as a cognitive tool to imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems, 33:3761-3777 -3774, 2020.
Hafner, D.,. Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International conference on machine learning, 2019b.
10

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Hafner, D., Lee, K.-H., Fischer, I., and Abbeel, P. Deep hierarchical planning from pixels. In Advances in Neural Information Processing Systems, volume 35, 2022.
Paolo, G., Coninx, A., Doncieux, S., and Laflaquière, A. Sparse reward exploration via novelty search and emit- ters. In Proceedings of the Genetic and Evolutionary Computation Conference, Pp. 154-162, 2021.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint 0rXiv:2301.04144:1 2023.
Pathak, D., Agrawal, P., Efros, A., and Darrell, T. Curiosity-driver exploration by self-supervised predic- tion. In International Conference on Machine Learning (ICML), 2017.
Hansen, N., Su, H., and Wang, X. TD-MPC2: Scalable, ro- bust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2024.
Pathak, D., Gandhi, D., and Gupta, A. Self-supervised ex- ploration via disagreement. In International Conference Machine Learning (ICML), 2019.
Kannan, H., Hafner, D., Finn, C., and Erhan, D. Robodesk: multi-task reinforcement learning benchmark, https: / //qithub. com/ http://ooooeeereeeeeeeae www.eeeeerrrrbrreeeeeeeeee 2021.
Pleines, M., Addis, D., Rubinstein, D., Zimmer, F., Preuss, M., and Whidden, P. Playing pokémon red via deep reinforcement learning, arXiv preprint arXiv:2502.19922 2025.
Kim, K., Sano, M., De Freitas, J., Haber, N., and Yamins, D. Active world model learning with progress curiosity. In International Conference Machine Learning (ICML), 2020.
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. Skew-fit: State-covering self-supervised re- inforcement learning. arXiv preprint arXiv: 1003::990003360 036698. 2019.
Klissarov, M., D'Oro, P., Sodhani, S., Raileanu, R., Bacon, P.-L., Vincent, P., Zhang, A., and Henaff, M. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint ar arXiv:2310.00166, 9 2023.
Rocamonde, J., Montesinos, V., Nava, E., Perez, E., and Lindner, D. Vision-language models are zero-shot re- ward models for reinforcement learning, arXiv preprint arXiv:2310. 20l::2300.2200 12921, 2023.
Klyubin, A., Polani, D., and Nehaniv, C. Empowerment: a universal agent-centric measure of control. IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135 Vol.1, 2005.
Samvelyan, M., Kirk, R., Kurin, V., Parker-Holder, J., Jiang, M., Hambro, E., Petroni, F., Kuttler, H., Grefenstette, E., and Rocktäschel, T. Minihack the planet: A sandbox for open-ended reinforcement learning research. In Neural In- formation Processing Systems Datasets and Benchmarks Track, 2021.
Küttler, H., Nardelli, N., Miller, A., Raileanu, R., Selvatici, M., Grefenstette, E., and Rocktäschel, T. The nethack learning environment. In Advances in Neural Information Processing Systems, volume 33. pp. 7671-7684 2020.
Sancaktar, C., Blaes, S., and Martius, G. Curious explo- ration via structured world models yields zero-shot object manipulation. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.
Mendonca, R., Rybkin, O., Daniilidis, K., Hafner, D., and Pathak, D. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 0124379-24399 2021
Sancaktar, C., Piater, J., and Martius, G. Regularity as intrin- sic reward for free play. Advances in Neural Information Processing Systems, 36, 2024.
Mohamed, S. and Jimenez Rezende, D. Variational in- formation maximisation for intrinsically motivated rein- forcement learning. In Advances Neural Information Processing Systems (NeurIPS), 2015.
Schmidhuber, J. A possibility for implementing curiosity and boredom in model-building neural controllers. In Pro- ceedings of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, 1991.
Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.0637777777 2017.
OpenAI, Plappert, M., Sampedro, R., Xu, T., Akkaya, I., Kosaraju, V., Welinder, P., D'Sa R., Petron, A., de Oliveira Pinto, H. P., Paino, A., Noh, H., Weng, L., Yuan, Q., Chu, C., and Zaremba, W. Asymmetric self- play for automatic goal discovery in robotic manipulation. arXiv:2 048822, 2021.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via self-super world models. In International Conference on Machine Learning (ICML., 2020.

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Sontakke, S. A., Zhang, J., Arnold, S. M., Pertsch, K., Bryik; E., Sadigh, D., Finn, C., and Itti, L. Roboclip: demonstration is enough to learn robot policies. arXiv preprint arXiv:2310.07899, 2023.
Spelke, E. S. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
Storck, J., Hochreiter, S., and Schmidhuber, J. Reinforce- ment driven information acquisition non-deterministic environments. In Proceedings of the International Con- ference on Artificial Neural Networks, pp. 159-164, Paris, 1995. EC2 & Cie.
Suarez, Pokegym:
Pokemon red gymnasium environment for reinforcement learning. https: /pypi...... org/ project / /ttp///p/egooo/////oo//oe/.ataa/ 0 1/, 2024. v0.1.1.
Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F,, and Abbeel, exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Wang, Y., Sun, Z., Zhang, J., Xian, Z., Biyik, E., Held, D., and Erickson, Z. Rl-vIm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint ar ar21:22402.0368,, 2024.
Whidden, P. Ai plays pokemon. https://githin Pon//Whiddy poee////poeennneeieeeeemeetttss
2023. YouTube video: https://youtu.bel DCYLT37IMBY.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. A survey of preference- based reinforcement learning meth- ods. Journal of Machine Learning Research, 2017.
Wu, P., Escontrela, A., Hafner, D., Abbeel, P., and Goldberg, K. Daydreamer: World models for physical robot learn- ing. In Conference on Robot Learning, pp. 2226-2240. PMLR, 2023.
Zadaianchuk, A., Seitzer, M., and Martius, G. Self- supervised visual reinforcement learning with object- centric representations. International Conference on Learning Representations, 2021.
Zhang, J., Lehman, J., Stanley, K., and Clune, J. Omni: Open-endedness via models of human notions of interest- ingness. arXiv preprint arXiv:2306.0177 017111 2023a.
Zhang, J., Zhang, J., Pertsch, K., Liu, Z., Ren, X., Chang, M., Sun, S.-H., and Lim, J. J. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. 7th Annual Conference on Robot Learning, 2023b.
12

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Supplementary Material for: SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
A. SENSEI: Implementation Details
We provide code with VLM-MOTIF checkpoints at https::/ssthuub..om/mrtiiiii llteettiussssssseeeeeeeeeeeeeeee sensei.
A.1. World model
RSSM We base RSSM implementation on V3 (Hafner et al., 2023). For MiniHack we use the small model size setting with roughly 18M parameters (h, dimensions: 512, CNN multiplier: 32, dense hidden units: 512, MLP layers:
2). For the more complicated Robodesk environment, we use the medium model size with around 37M parameters (h: dimensions: 1024, CNN multiplier: 48, dense hidden units: 640, MLP layers: 3). By default, when the input observation o, is only an image, it is en- and decoded through CNNs. For MiniHack, we have an additional inventory flag that is processed by separate MLP, as is customary for the Dreamer line of work when dealing with multimodal inputs (Wu et al., 2023).
The MLP decoder outputs Bernoulli distribution from which sample the decoded inventory flag.
Reward predictors To handle rewards of widely varying magnitudes, DreamerV3 uses twohot codes predicted in symlog space when predicting rewards (Hafner et al., 2023). We use the same setup for all reward prediction heads, i.e., for extrinsic rewards rf for task i or the semantic exploration reward r During task-free exploration, the gradients from reward predictions are stopped to not further affect world model training. We do this to keep the world model task-agnostic and to avoid biasing task-fres exploration. Similarly, to avoid overfitting to the exploration regime, we also stop the gradients from the semantic reward prediction heads.
Plan2Explore Both our Plan2Explore baseline well as our ensemble predictors (Eq. 5) based on the re- implementation top of DreamerV3. The most notable difference is that in original Plan2Explore the ensemble is trained to predict image encodings (Sekar et al., 2020), whereas the new version is trained to predict stochastic states z Recent re-implementations (Hafner, 2021; Hafner et al., 2022; Gumbsch et al., 2024) also used Plan2Explore with ensemble disagreement over zt as a bascline and verified a strong exploration performance. For our experiments on task- based exploration (Sec. 4.4), we use a weighted sum of extrinsic and intrinsic rewards, i.e. re + angle. We determined the best value a 0.5 through a hyperparameter search, for SENSHI'S hyperparameters.
Quantile estimation We update our estimate of the quantile (fsem) whenever train the exploration policy. For this, we compute the k-th quantile of prem in each training batch (16 x 16). We keep an exponential moving average over these estimates with smoothing factor of a = 0.99.
Reward weighting
In practice, we compute exploration rewards (Eq. 7) overall five reward factors:
explore prom + gexplore pdi if prem 2 (isee)))
r expl = Xi. +
(8)
00 + g
otherwise.
ie. x E 10 } to scale extrinsic rewards Tz, a to scale semantic rewards prem and / to scale uncertainty-based rewards r We set x= = 0 for task-free exploration and for task-based exploration (Sec. 4.4). When training the value function with DreamerV3, the scale of the reward sources are normalized. To compute this normalization for the exploration policy we use a explore and gexplore of the high percentile region of interestingness (2 Qk).
A.2. Semantie Reward Distillation: VLM-MOTIF
For the semantic reward function Rg : 0 + I. we use a 2Dcoonvolutoonal neural network to encode the images. We use 3 convolutional layers, where we progressively increase the number of channels to num num_channel s ma_ 64. The
13

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
output then gets downsampled via pooling before going into two-layer MLP with hidden dimensions 256 & 512 and outputting the scalar reward value. Additionally, in MiniHack include inventory information via separate multi-layer perceptron (MLP) head, consisting of 2 layers with 512 hidden units. The extracted features are concatenated with the image features and get further processed by the output MLP. The training hyperparameters for all Rig can be found in Suppl. B.
A.3. Design Choice: Semantic Reward Predictions
World models typically encode and predict dynamics fully in self-learned latent state (Ha & Schmidhuber, 2018; Hafner et al., 2023; Hansen et al., 2024). Thus, for world model to predict r? at any point in time t, we need mapping from latent states to semantic rewards. We chose to directly predict frem using a reward prediction head of the RSSM. Another option would to decode the latent state to images and use those as inputs for MOTIF. However, we believe this has several disadvantages: (1) Decoding latent states to images a computationally costly step that would significantly decrease computational efficiency. (2) We would use an indirect target (the image) instead of the direct target (r7mm)) for training the semantic reward predictions. There would exist gradient signal correct somewhat reasonable image predictions that lead to inconsistent reward predictions at given state. (3) The image predictions of the RSSM can contain artifacts, blurriness or hallucinations. Since MOTIF is only trained on real images from the simulation, we will likely encounter out-of-distribution errors.
B. Hyperparameters
We provide the hyperparameters used for the world model, exploration policy, VLM-MOTIP annotations & reward model training well the environment-speccie settings.
Name
Value KeyRoom
Robodesk
KeyChest
Pokémon Red
World Model
RSSM size
M
S
L
Ensemble size N
8
8
8
8
Train ratio
512
512
512
512
Exploration policy
Quantile
0.75 - - 0.80
0.90
0.90
0.6
0-0-0-0
explore
0.1 - -.0 0.01
0.3
0.25
0.025
gexplore
1-1-1-1
0.5
a.0
1-1-1-1
0.5
0-0-0-0
0.1
0.05
0.1
Annotations for MOTIF
VLM
GPT-4 turbo (right) & GPT-4 omni (left)
GPT-4 omni
GPT-4 omni
GPT-4 omni
Temperature
0.2
0.2
0.2
0.2
Dataset size
200K
100K
100K
100K
Image res.
224 x 224
80 x 80
80 80
1202 x 1080
MOTIF Training
Batch size
32 64 - 32 32
32
32
32
Learning rate
10 5 5- 3x -x 10
10-4
10
10-5
Weight decay
50-5 0
10-5
10-4
Environment
Action repeat
Episode length
250
600
800
1k/2k/4k
Steps of exploration
1M
500K
500k
750k
14

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
(a) Default observations
(b) Our observations
(c) Left camera
Figure 9. Robodesk environment: We modify the default tow-down camera view (a) side view with less occlusion (b). For annotation
with GPT-4 we also provide a left camera observation (c).
For the exploration policy in Robodesk we use different values for the four different variants tested. The values listed here stand for, from left to right: GPT-4 with Plan2Explore (P2X) data using two camera angles for VLM annotations, GPT-4 with P2X data using only the right camera angle, Oracle with P2X data, and Oracle with CEE-US data (corresponding to a more interaction-rich exploration dataset Dimit). The VLM-MOTIF hyperparameters are also listed in the same order.
64 64 pixel images for all environments. However, for the GPT annotations use higher resolution images, as shown in the table. Inside the environment step function, the rendering performed at these higher resolutions, and this image is input to the semantic reward function Rg. The image is then scaled down to 64 x 64 as part of the observation that the RSSM is trained on.
Image resolution For the world model
Baselines We run D3eamerV3 with the same world model setup as SENSEI and Plan2Explore. We use an open source PPO (Schulman et al., 2017) implementation Hafner (2024)* optimized to work well across multiple environments with a fixed set of hyperparameters (details Hafner al., 2023, supplementary material). We build our RND (Burda et al., 2019) implementation on top of PPO. For the predictor and target network by dense layers. We only use the intrinsic reward train PPO agent. Intrinsic rewards normalized as outlined in Burda et al. (2019). While Burda et al. (2019) also normalize input observations through a running statistics, we found that using LayerNorm at the input layer leads to slightly more interactions in Robodesk.
ResNet with 3 convolutional layers followed
C. Environment Details
C.1. Robodesk
Robodesk (Kannan et al., 2021) is multi-task RL benchmark in which a robot can interact with various objects on a desk. We an episode length of 250 time steps.
Observations Robodesk uses only an image observation, depicting the current scene, which scale down (64 x 64 pixels). However, we found that the default top-down view often had occlusions and was hard to interpret from a single image (Fig. 9a). Thus, we used a different camera angle showing the robot from one side (Fig, 9c). With this view objects and the drawer were rarely occluded; however, lights that turn on from button presses were not as visible anymore.
Actions The continuous 5-dimensional actions control the movement of the end effector. We use an action repeat of to speed up the simulation. Thus, IM steps of exploration correspond to 2M actions in the environment.
Interaction metrics We track how often the robot interacted with different objects to quantify the behavior during exploration. Specifically, we track the velocity of joints and object positions. For buttons, sliding cabinet, or drawer, we
fhttps: http:::://///htu/t/////e thub. com/ d jar embodi ed, version v1.2
15

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
(a) KeyRoom-S0 S15
(b) KeyChest
(c) egocentric view
Figure 10. MiniHlckk We consider two tasks KeyRoom-S15 (a) and KeyChest (b). The agent receives egocentric view of the
environment as its observation (c).
check if the joint position changes than fixed value (0.02). For all other objects, check if any of their z-y-2 velocities exceed threshold (0.02).
Tasks We the sparse reward versions of all the tasks available in the environment, For some tasks, add easier
versions. All tasks describe interactions with one multiple objects:
Buttons: Pushing the red (push_red)) blue (push_blue), or green (push_green) button.
Sliding cabinet: Opening the sliding cabinet fully (open_s1 ide).
Drawer: Opening the drawer fully (open_drawer), opening the drawer half-way (open aopen_drawer_mediine or opening it slightly (open (open_drawer) @alaww__lghtte We introduced the latter tasks.
Upright Block: Lifting the upright block
pushing off the table shelf).
(upright_b)
or putting into the shelf (upright_blook
Flat Block: Lifting the flat block (lift_fla) block), pushing it off the table oflat_blook_offttbbbll into the bin (flat_block block in bii), or into the shelf (flat block in snelf.....
Both blocks: Stacking both blocks (stack).
Ball: Lifting the ball (lift_ball), dropping it into the bin (ball_ [i_bbin) or putting it into the shelf (ball_in_shelff)
C.2. MiniHack
Observations In MiniHack multiple observation and action spaces are possible. We use egocentric, pixel-based observa- tions centered on the agent (1:2 grids, example in Fig. 10c). In addition to that, we provide the agent's inventory. By default, in MiniHack the inventory is given as an array strings (UTF8 encoded), and different player characters have different starting equipment based on the character classes NetHack. We simplify this by providing only a binary flag that indicates the agent has picked up new item. This sufficient for the problems consider, which maximally one new item can be collected and starting equipment cannot be used.
Environments Here we detail enviromments we tackle:
In the benchmark KeyRoom-S13 problem (Fig. 10a), the agent needs fetch a key in a large room (15 15 grids) to enter smaller room and find staircase exit the dungeon. We the default action space but enable autopickup and therefore remove the PICKUP action. We use episode length of 600 time steps, which is 1.5 times longer than the default episode length.
KeyChest is novel environment designed by us, based on DeyCorridorS44 from MiniGrid (Chevalier-Boiss Boisvert et al., 2024) (see Fig. 10b). The agent starts in a corridor randomly connected to different rooms. A key is hidden in one room and chest in another room. The goal is open the chest with the key in the inventory. Object positions are
16

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
PIDGEY .2
AAAAAAAAAA
16
NF:
22/ 22
FIGHT PAR
ITEM RUN
(a) map navigation
(b) battle screen
(c) low-resolution observation
Figure 11. Pokémon Red poses strong exploration challenge it requires an agent to leam (1) to navigate complex overworld map
(a) and (2) to battle and catch Pokémon (b). We only use the down-scaled game screen image as observations (c).
randomized. The action space for this task contains 5 discrete actions for moving the agent in 4 cardinal directions (UP, RIGHT, DOWN, LEFT) and an OPEN-action to open a chest when standing next to it with a key in the inventory. Episodes terminate when the chest opened. We enable auto-pickup. so no additional action is needed to pick up the key when stepping on it. We use episode length of 800 time steps.
Rewards All environments use sparse reward of r, 1, which the agent only receives upon accomplishing the task. A small punishment (r, -0.01) is given, when the agent performs an action that does not alter the screen.
Image remapping Empirically, we found that GPT-4 may encounter problems if we provide the image observations as is. For example, when using the default character in the KeyRoom- S1 5 environment (Rogue), GPT-4 sometimes throws content violation errors. We suspect that this due to the character wearing a helmet with horns, which could be mistaken for demonic or satanic imagery. Thus, we pre-po pre-proeeseed the images before returning them from the environment. We render all characters the Tourists, friendly looking character with Hawaiian shirt and straw hat. Furthermore, GPT-4 sometimes mistakes entrance staircases for exit staircases. Since the entrance staircases serve no particular purpose and not different from the regular floor, we remap all entrance staircases to floors.
C.3. Pokémon Red
We evaluate SENSEI's ability for semantic exploration in the classic Game Boy game, Pokémon Red. Pokémon Red presents extremely challenging exploration problem due to: (1) its vast and interconnected world, composed of towns, routes, forests, and other areas that must be navigated, and (2) its complex battle system, which requires semantic knowledge to understand type interactions (e.g., Water attacks are strong against Fire-type Pokémon). Thus, an agent needs strong exploration capabilities to progress in the game's primary objective, i.c., becoming the Pokémon Champion by defcating Gym Leaders and collecting their badges. We base our implementation on PokeGym v0.1.1 (Suarez, 2024), which is based on Whidden (2023), with minor modifications as detailed below.
Observations Unlike previous RL agents applied to Pokémon (Whidden, 2023; Pleines al., 2025), we provide only the raw game screen image input, without access to the internal game state additional memory. When input into the world model, we downscale the game screen (to 64x64 pixels) in order to save compute (see Fig. 11c.. For VLM annotations, we use the original size (1202x108) pixels), such that all text is clearly readable (see Fig. 11b).
The agent controls the game using a 6-dimensional action space corresponding to Game Boy button presses (Left, Right, Up. Down, A. B). Since the game only advances upon button presses (except during attack animations) we only apply an action every 1.5 seconds real time game play (frame skip of 96) to manage episode length.
Actions
17

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Rewards The agent receives weighted sum of rewards rt Σ Ø'rt for different in-game events i. Rewarded events include leveling up, catching Pokémon, encountering strong opponents, healing Pokémon, visiting new map tiles, earning badges, and penalty for blacking out after losing battle. We leave the default values ø' from Suarez (2024) except we increase ditiles = 0.1 for reaching new map tiles (previously set to 0.01), we found this improves exploration.
Episode Length To scaffold exploration, we gradually increase the maximum episode length over environment steps: 1k length for 0-25k steps, 2k length for 25-50k, 4k length for 50-75k, and 8k length when continuing exploration beyong 75k steps.
Starting point As in Whidden (2023), we skip the initial "tutorial" phase of the game, and start only when the player can freely move and catch Pokémon. In-game, this corresponds the point after delivering Oak's Parcel and receiving the Pokédex and Poké Balls from Professor Oak. We use the same checkpoint as Whidden (2023), starting with level Squirtle named AAAAAAAAAA
Map segments The accessible game world in Pokémon Red prior to defeating the first Gym is already expansive, comprising variety of interconnected areas. Many of these are distinct maps that load separately when the player enters or exits a building (see Fig. 12). To evaluate spatial exploration, we segment this world into discrete map segments, as illustrated in Fig. 8a. Each route and town is treated as a separate segment, as are buildings or enclosed areas like forests that load dedicated sub-map. This amounts 25 map segments that can be accessed before beating the first Pokémon Gym. Reaching the first Pokémon Gym requires navigating through such segments, which we enumerate in order of appearance (ef. Fig. 8a):
Number
Name
Type building town route
Oak's Lab
Pallet Town
Route
Viridian City
town
4
Route 2
route
5
Viridian Forest South Gate
building
6
Viridian Forest
forest
Viridian Forest North Gate
building
Pewter City
town
Pewter Gym
building
This segmentation allows us to quantify exploration progress by measuring the highest-indexed map segment reached during an episode.
18

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Figure 12. Full map of Pokémon Red accessible prior defeating the first Gym. If entering or exiting building brings the agent to a submap, this is indicated by line. The map is modified from https: ool.o.ooooooooooooo.oooo www...veeuu......o com pom////// / pokemon- red- weed-bue--ee blue-map html
19

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
D. VLM prompting
We prompt the VLM with somewhat general descriptions of the environments that we consider. Here we provide the full prompts for all environments.
D.I. Robodesk
In Robodesk, for each query, we provide two observation images (resolution 224 224) with the following prompt:
Here are two images in simulated envi ronment with robot in front of desk. Your task is to pick between these images based on how interesting they are. Which image is more interesting in terms of the showcased behavior? For context following points would constitute interestingness:
(1) The robot.
currently holding an object its gripper. (2) The robot is pushing
interacting object around or pushing button or opening the drawer
with entities the desk. (3) Objects
the desk are in an interesting configuration: e.g. a stack. Being far away from the desk with the robot arm retracted or just stretching your arm without interactions, is a sign the image is not interesting. Answer in maximum one word: 0 for image 1, for image 2, 2 for both images and if you have no clue.
Due to occlusions, we annotate the same pair from the initial dataset Dinit with the same prompt using images from two camera angles: right (Fig. 9c) and left (Fig. 9c). pair is deemed valid only the GPT-4 response is the same across both camera angles, otherwise the pair and the annotation are removed from the dataset. The VLM-MOTIF training as well as the world model training are still executed using only the right camera image, such that during free play with SENSEI we only rely on the right camera images as input.
for the right camera annotations and GPT-4 omni for the left camera annotations. To cap the dataset size of pairs at 200K and still make sure that we got contrastive pairs of interesting vs. uninteresting behaviors, we pre-filtered the pairs using our Oracle of interestingness for the Plan2Explore dataset. We believe that in future work this step can be omitted by simply generating more random pairs and expanding the dataset size. After the two-angle image amnotation, we use the subset of the dataset where both left and right camera annotations agree. This corresponds 139409 pairs from the original 200K dataset, 69.7% of
In Robodesk,
GPT-4 Turbo 1ett-4-turbo-2024-000
2ott-4o-2024-055
D.2. Robodesk: Zero Pre-Knowledge General Prompt
To demonstrate the generality of SENSEL, we also test a version with general prompt (SENSEI GENERAL), using a multi-turn strategy for getting the annotations for pair of images. This version of the prompt does not contain any external environment specific knowledge, First, we show picture from the robotic environment (shown in Fig. 9c) and ask the VLM (gpt-40-2024-05-4 describe what it sees in the scene using the following prompt:
will show you an image from simulated environment. Describe the environment to me and what you deem the agent and then explain what types of behavior observations would be useful and interesting for the agent explore in this setting?
We obtain the following response from GPT:
20

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
The image depicts cts simulated envi ronment featuring robotic arm interacting with a table that has various objects it. The objects include buttons of different colors, a red ball, a green cube, and a blue rectangular block. The setting appears to be designed for testing robotic manipulation and interaction with objects.
Agent: The robotic arm is the agent in this environment. It is likely programmed to interact with the objects on the table, possibly for tasks involving sorting, stacking, Useful and Interesting Behaviors/Observation
pressing buttons.
1. Object Manipulation: Grasping and moving objects like the ball, cube, and block. Testing different grips and force applications to handle objects of various shapes and weights. 2. Button Interaction: - Pressing buttons to observe effects, such as changes in the environment or triggering specific actions.
Tearning which buttons correspond to which actions or outcomes. 3. Spatial
Awareness: Navigating the arm around the table without knocking over objects. Understanding the spatial relationship between objects for tasks like stacking. 4. Task Learning:
Completing specific tasks like sorting objects by color or shape. Learning sequences of actions to achieve complex goals. 5. Feedback and Adaptation: Using sensory feedback to adjust actions in real-time.
Adapting to changes in the such as objects being moved or removed. 6. Exploration and Experimentation: Experimenting with different strategies for interacting with objects. Exploring the effects of different actions the environment. These behaviors would help the agent develop skills in manipulation, decision-making, and adaptive learning, which are crucial for more complex real-world applications.
Next, we show the pair of images that we would like GPT-4 annotate with the following prompt:
Now will show you two images. Based your answer, choose which of these images is more interesting in terms of the showcased behavior? Answer in maximum one word: 0 for image 1, 1 for image 2, 2 for both images and 3 if you have no clue.
We obtain the final annotation for the pair as the answer to this last prompt.
In this experiment, the environment description is generated using the first prompt only once. For subsequent annotations, the initial question-answer pair is included in GPT-4's context window as history. In the second turn, the new image pair is presented using the final prompt. This approach more cost-efficient, as in-context tokens are less expensive than generated ones, and time-efficient, as generating the full environment description sequentially requires more processing time. With this multi-turn strategy, annotating each image pair costs approximately $0.004, compared to $0.002 per annotation with the single-turn prompt.
D.3. MiniHack
For MiniHack provide two observation images (resolution 80 80) and the inventory. For the inventory, we only consider items that were picked up and not items in the agent's starting equipment'. We provide the inventory text descriptions. The different options are shown in purple.
"The starting equipment from NetHack game and irrelevant and inaccesible our tasks.
21

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Your task is help play the video game MiniHack MiniHack is roguelike game where agent needs to navigate through rooms and escape a dungeon. For succeeding in the game, finding items, collecting items and exploring new rooms is crucial. Images are egocentric around the agent, who is standing on dotted blue flcor. Your task is to pick between two game states, composed of images and an inventory descriptions, based on how interesting and useful they are.
there any difference between the first and secend game state in terms of how interesting it is? The images depict the current view. {The first agent has a key named The Master Key of Thievery in their inventory., The second agent has a key named The Master Key of Thievery in their inventory., Both agents have a key named The Master Key of Thievery in their inventory., Both agents have no items in their inventory.
the first state is more interesting, if the second state is more interesting, if both states interesting and 3 if nothing is interesting or you very unsure.
Think it through and then answer in maximum one word:
For MiniHack we use GPT-4 omni (qpt-40-2024-05-1)
D.4. Pokémon Red
For Pokémon Red we set the goal to reach the first Pokémon Gym (the first boss battle of the game): We assume that was extensively trained on game play data and various walkthroughs of Pokémon Red
GPT-4 (gpt-4o- 4000t-40-002444- and contains sufficient knowledge of the game. Thus, we again use multi-turn strategy for image annotations, with SENSEI GENERAL (see Suppl. D.2), where first ask the VLM for game play description, which then as context for further annotations. We first asked the VLM::
Your task is help me play the Game Boy game Pokémon Red. just obtained my starter Pokémon, Squirtle. My goal is to find and defeat the Gym Leader, Brock. What do need to do, and which areas do I need to traverse to achieve this goal? Keep it very short.
We generated five different responses from GPT-4. We sample from them uniformly as context for image-based comparisons. We provide two observation images with the following prompt:
Here are two screenshots from the game. Which image depicts game state that is closer to my goal? Answer in maximum one word: 0 if the first state is better, 1 if the second state is better.
D.5. Oracle for Interestingness
In Robodesk, we also use an Oracle of interestingness annotate the pairs as ablation (see Suppl. E.2). Our goal here is to showcase upper-bound of performance SENSEI without the noisiness of VLMs. For the Oracle, deem a state interesting if: (1) any of the entities are in motion (here only for the ball make an exception that the ball should be in motion with the end effector close to it as the ball in the environment is unimpeded by friction), (2) if the drawer is opened, (3) if the drawer/sliding cabinet not yet in motion, but the end effector is very close to their handles, (4) if the upright and flat blocks are not yet in motion but the end effector very close to them (almost touching), (5) if the stacking task is solved. With these statements, we essentially cover the range of tasks defined the Robodesk environment, as they are shown in Fig. 16.
22

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
score
0.5
score
0.5
PPO
0
PPO
10
20
30
40
0
10
15
1e6 steps
1e6 steps
(a) extrinsic phase in KeyRoom-So S15
(b) extrinsic phase in KeyChest
Figure PPO performance in MiniHack: We plot the mean episode score obtained by PPO during evaluation the MiniHack lasks KeyRoom-S 105m-115 (a) and KeyChest (b). Shaded areas depict the standard error (10 seeds). We apply smoothing over the score trajectorics with window size 30.
E. Extended Results
E.1. MiniHack: Extended Results
Figure 13 shows the full trajectory of evalutation scores for Proximal Policy Optimization (PPO, Schulman et al. 2017) in MiniHack when trained until convergence. While PPO manages to learn to solve all tasks, it can be much less sample efficient than the model-based agents we evaluated (see Fig. 7), especially in KeyRoom-S15, Here SENSEI outperforms PPO terms of sample efficiency in one to two orders of magnitude.
E.2. Robodesk: SENSEI Ablations
In Robodesk, we compare different versions of SENSEI in order to analyze the effect of the VLM and the initial exploration data on SENSEI performance (Fig. 14). First, we showcase SENSEI results when annotating the initial exploration dataset from Plan2Explore with only the right camera images. In this case, we use the whole 200K pairs in the dataset, without any pruning. another ablation, we replace the VLM (GPT-4) with hand-crafted Oracle (see Suppl. D.5 for how the oracle computed) for annotating the pairs. After the oracle annotations, we distill these preferences into VLM-MOTIF for SENSEI, following the same procedure before. Furthermore, collected either by CEE-US (Sancaktar al., 2022) by Plan2Explore for the oracle SENSEI versions. CEE-US uses vector- rector-based position of entities for information cicliionngii----e gain-based exploration, in comparison to Plan2Explore, which works on the pixel-level. Due to the privileged inputs, Dosi CEE-US US contains more complex interactions. We compare 1M steps of exploration with the four versions of SENSEI and Plan2Explore.
compare two initial datasets Dinit of self-supervised exploration
On average, all versions of SENSEI interact more with the objects than Plan2Explore and our semantic exploration reward lead to object interactions than pure epistemic uncertainty-bant exploration. SENSEI with Oracle for both the Plan2Explore and especially the CEE-US initial datasets show the most object interactions. We believe this further
Interaction counts
100k
Plan2Explore (P2X)
SENSEI, GPT-4 & P2X
SENSEI, GPT-4 & P2X right cam
SENSEI, Oracle
50k
I
SENSEI, Oracle & CEE-US
Sliding cabinet
Ball
Upright block
Flat block
Buttons
Drawer
Figure Interactions Robodesk: We plot the mean over the number of interactions with objects in environment during exploration for different versions of SENSEI (Oracle vs. VLM, CEE-US (Sancaktar et al., 2022) vs. Plan2Explore to create the data to label Dimit)
Plan2Explore. We also ablate SENSEI using only right camera angle VLM annotations on the Plan2Explore dataset. Error bars show the standard deviation (3 seeds).
23

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
showcases that the VLM provides much noisier signal of interestingness, making it harder optimize for.
The initial exploration dataset Dimit influences with which objects SENSEI interacts. Qualitatively, we observe Plan2Explore performing mostly arm stretches. Interestingly, this can still lead to solving tasks during exploration. For example, stretching the arm against the sliding cabinet can close it, and stretching the arm toward the upright block can push it off the table. As a result, SENSEI with Plan2Explore Oracle focuses mainly on the sliding cabinet and the upright block, reinforcing the existing trends in the initial dataset from which VLM-MOTIF is distilled.
For CEE-US data, Oracle SENSEI interacts more with the other objects, such as the ball and the flat block, as well as the drawer. The difference between the Oracle annotator SENSEI versions with CEE-US Plan2Explore data showcases that there is still lot to be gained from richer initial dataset for SENSEI, which could be obtained via multiple rounds of SENSEI exploration.
If a VLM annotates images instead of the Oracle, SENSEI shows similar behavioral trends, but overall less object interactions, such that neither of the GPT-4 annotations on the Plan2Explore data completely match the performance the oracle annotator.
Finally, when we compare the performance for SENSEI using GPT-4 annotations with two-angle camera images vs. only the right camera angle image, since the drawer is more clearly visible in the left camera view. However, as the ball and blocks are mainly initialized on the right side of the table, the pure right camera angle SENSEI generates more interactions with these objects during exploration. Another factor here is that for the right camera angle we retain all 200K pairs for WLM-MOTE distillation, whereas we only keep ca. 70% the pairs in the case of SENSEI using both cameras for annotation.
that the two-angle version performs better in terms of drawer interactions. This is expected
E.3. Robodesk: Rewards
In addition to interaction metrics, we count the number of times task rewards are collected during exploration. We observe that for the majority of tasks SENSEI solves more tasks in the environment during play than Plan2Explore. Note that for the open_sseeee slide task you need to open the slide fully one direction, which achieved in abundance in Plan2Explore runs by simply stretching the arm. The full interaction metrics of exploring how the slide moves left-right is not necessarily reflected in the task rewards, can be seen in comparison to Fig. 6. Similar arguments also apply for opening the drawer fully vs. opening and closing the drawer more dynamically. Additionally as the bin not really visible in our camera angle, solving in_bin tasks are more due to the objects that go off the table landing by chance the bin for all methods, such that higher statistics for of off_abbl rewards also lead to higher i bn_bii rewards.
open_drawer_ligee
4,000
2,000
P2X
SENSEI
RND
Figure 15. Collected open drawer. _iigtthttttttt exploration (3 seeds).
rewards for during
24

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
stack
upright block
off. -oockooff_rre
table
ball.off.table
10,000
1,500-
100
1,000
50
5,000
500
0
0
flat. block in shelf
flat. block off table
ball in shelf
800
1,500
300
600
1,000
200
400
100
500
200
0
0-
upright block in shelf
lift. fltt.flat.llock
lift_ball
1,500
400
200
1,000
200
100
500
0
lift lift..p block block
flat. block in bin
ball in bin
80
20
40
60
40
10
20
20
0
push_blue
push.red
push-green
1,000
8,000
800
1,000
6,000
600
4,000
400
500
2,000
200
0
olin
open.drawer
open.drawer. light
300
40,000
1 10
200
20,000
50,000
100
0
0
Plan2Explore (P2X)
SENSEI, GPT-4 & P2X VL.MMMttif RND
Figure 16. Robodesk environment rewards: We plot the mean number of sparse rewards (successful task completions) discovered during 1M steps of task-free exploration for all tasks for Plan2Explore, SENSEL, pure VIMM-MTTIF, and the RND baseline.
25

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Rg. SENSEI GENERAL
R.g SENSEI
2
34
5
4
5
0.5
-0.5
0
100
150
200
250
2
3
4
5
2
4
200 (HOIILL
0.5
50
100
150
200
250
5
4
seca (uorm..)
0.5
50
100
150
200
250
(a) screenshots
(b) semantic exploration reward
Figure 17. Semantic exploration rewards for example trajectories with VLM-MOTIF using general vs. specialized prompts: For semantic rewards distilled from GPT-4 annotations using prompt
three example Robodesk episodes, showcase specialized to the environment vs. general prompt using multi-turn annotations. The reward trajectories for both the general and specialized prompts peak at the "interesting" moments of exploration, such as opening drawer pushing the blocks. With zero external knowledge injection, the general prompt version of VLM-MOTIF is highly correlated with its specialized prompt counterpart.
E.4. Robodesk: VLM-MOTIF with General Prompt
In this section, we investigate the distilled reward function when using a general prompting strategy (SENSEI GENERAL, Suppl. D.2). As shown in Fig. 17, the semantic reward r for the general prompt seems to show high positive correlation or qualitatively matches with the VLM-MOTIF distilled using the specialized prompt in Robodesk (see Suppl. D.I). Thus, we manage to distill a reward function that peaks at interesting moments of exploration without injecting any environment specific knowledge into the prompt.
E,5. Robodesk:
Baselines
We present two other baselines in Robodesk: RND trained with PPO and pure VLM-MOTIF, and analyze the interaction metrics in Fig. 18. On average, SENSEI interacts more with most available objects than the baselines. RND mostly moves the arm around the center of the screen, occasionally hitting objects or mostly buttons. is important to note that the robot arm in Robodesk is mostly initialized close to the buttons. Pure VLM-MOTIP is an ablation of SENSEI without any information gain objective. Here,
the importance of the information gain reward to ensure diverse exploration. Unlike SENSEI, we see that VLM-MOTIF interacts with specific entities: mostly the buttons, the drawer and the flat block. The lack of interaction with the cabinet, the upright block and the ball are expected as these entities are spatially further away from the robot initialization pose. Once high semantic rewards are found in the vicinity by interacting with the drawer and buttons, there is no incentive for pure VLM-MOTIF to explore further. On the other hand SENSEI aims to discover interesting and yet novel behaviors, ensuring better coverage across the different useful behaviors in the environment.
26

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Upright block
Flat block
Hull
300
12k-
20%
15%
6k-
P2X SENSRI VLM RND
P2X
VLM RND MOTIP
P2X RND
MOTTP
MOTIP
Buttons
Drawer
Sliding cabinet
40k
18k-
12K
20k
0-
PIX SENSEI VLM AND
P2X SENSEI
VLM RND
P2X SENSEI VLM BND
Figure 18. Interactions in Robodesk: We plot the mean over the number of interactions with any object during 1M steps of exploration for SENSEL, Plan2Explore (P2X), pure VLM-Mont MOTIF and Random Network Distillation (RND) trained with PPO policy, as model-free exploration baseline. Error bars show the standard deviation (3 scods).
E.6. Robodesk: SENSEI without Dynamic Scaling and Analyzing Hyperparameter Sensitivity
In this section, ablate the dynamic scaling of the semantic reward ri and the information gain reward r terms in SENSEL. In SENSEL, adjust the weight of these two terms based whether seem has reached the high percentile region of interestingness (r.em 2 Qa), as per equation Eq. 8. In this ablation, we instead use linear combination with fixed weights a and B, such that the exploration reward is given by:
expl ar + Br
(9)
We present the results in Fig. 19 for 6 different sets of fixed weights. First of all, we observe that none of the fixed scale settings outperform SENSEI nor do they consistently perform as well as SENSEL. Second of all, we see that the exploration behavior is very sensitive to the choice of the weights a and 3. For larger a values, the behavior collapses to mostly interacting with the drawer, buttons and the flat block, with larger fluctuations. This mode is very similar to the case of pure VI.M-MOTIF presented in Fig. 18.
Next, we test the hyperparameter sensitivity of SENSEI with dynamic scaling of the reward weights. We see in Fig. 20, that across all 4 hyperparameter configurations, SENSEI is better at least par with Plan2Explore, and we don't 't observe any behavior collapse as in the fixed scale setting. We argue that although the dynamic scaling introduces additional hyperparameters, the overall behavior is much more robust and less dependent on hyperparameter tuning.
Table Hyperparameter configurations for SENSEI presented in the main experiments and the 3 other configurations that are shown in Fig. 20.
SENSEI
SENSEI HP1
SEnSEi HP2
SENSEI HP3
Quantile
0.75
0.80
0.85
0.75
explore
0.1
0.01
0.1
0.05
gexplore
1
1
a
1
1
1
580
0
0
0
0
27

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Upright block
Flat block
30k
12K-
15k
P2X
0.0
F2X
25
01.5
0.5
1.0
25
0.5
Buttons
6/c
20k
3kc
10k-
P2X
SKANSSI
PaN
SUNNSLL
(3
1.0
1.01
1.0
11.
1.5
1.0
0.1
26
0.5
Drawer
Sliding cabines
Ok
600c
10k-
200
20K
P2X
1.0
1.0
1.0
0.1
25
0.5
10
0.1
25
0.5
Figure 19. Comparing interactions in Robodesk between SENSEI and fixed scaling of rewards: We plot the mean over the number of interactions with any object during IM steps exploration SENSEI Plan2Explore (P2X) and ablation SENSEL, where we do not dynamically adjust the weight of the reward terms based on the current semantic reward. For this ablation, reward is computed as rt exppl or
pem + Br dis with fixed weights and Error bars show standard deviation (3 seeds).
E.7. Robodesk: Downstream task learning
We investigate whether the world model learned by SENSEI is versa- tile enough to efficiently support downstream task learning in the multi- task environment of Robodesk. We evaluate this on three representa- tive tasks: opening the drawer (Drawer-Mee Perawer-Mediime pushing the upright block off the table (Upright-Block Boittt-Book-eoe welock-off---e Off-Table), and lifting the ball (Li Bt-Bl.....
Dreamer
Score
0.5
As in our MiniHack experiments (see Sec. 4.3), we initialize Dream- erV3 agent with pre- pre-trained world model obtained from IM steps of SENsEI-driven exploration. We compare this setup to exploration with Plan2Explore, running each agent for 1.2M steps or until SENSEI con- verges on the task.
2
1c6 stops
Figure 22, Dreamer in Robodesk for upright
block_ table (5 seeds, = SEM).
28

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Upright blck
Flat block
12k
Saxaei
I11"2
IIP3
HP1
II'
HP3
Ball
Buttons
201-
10k-
P2X
SENSEI
Sprveri
Server
P2X
SEaSEI
Serverr
SENSEI
Dzswer
Sliding cabinet
40k
3k-
0-
P2X
SEXSEI
SENSEI
Sexses
SEaSEI
P2X
SteesnI
SICUSII
Figure 20. Comparing interactions in Robodesk for SENSEI with different hyperparameters: We plot the mean over the number of interactions with any object during steps of exploration SENSEI (winner hyperparameter configuration) Plan2Explore (P2X) and SENSEI with different hyperparameters specified in Table 1. Error hars show the standard deviation seeds).
Fig. 21 shows the performance of task-specific policies training. The world model explored by SENSEI enables significantly faster learning compared to Plan2Explore. SEnSEI reliably learns to open the drawer and push the block off the table, though it does not fully converge 100% success the ball-lifting task. Nevertheless, across all tasks, SENSEI achieves higher success rates than Plan2Explore.
Finally, we also train Dreamer V3 agent for one of the tasks (Upright-BLock-off----- Dreamer does not fully learn solve the task within 2M environment steps. This shows that exploration crucial to reliably leamn to solve these sparse reward tasks.
from scratch (Fig. 22).
29

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
SENSEI
Plan2Eipporee
1
Scor
0.5
Soore
0.5
Soore
0.5
0-
0-
1.2
leb stops
166 steps
le6 stope
(a) Drawer-Medium
(b) Uprigpt-Blloooo Block-Off-Tabie
(c) Lift-Ball
Figure 21. Downstream task performance in Robodesk: We plot the mean of the episode score obtained during evaluation for three exemplary Robodesk tasks (a) Drawer-Medh Mediiu,, (b) Upright-Bloce -ooottt--ooo-------------------- -fbbeeeee and (c) Lift-Ball with world models learned from SENSEI vs. Plan2Explore (P2X) exploration. Shaded areas depict the standard error (10 seeds) and apply smoothing over the score trajectories with window size 3.
E.8. Pokémon Red: Extended Results
We evaluate the exploration of battle-related mechanics in Pokémon Red, by tracking the Pokémon party size and their levels and plot their distribution in Fig. 23. Only Plan2Explore and SENSEI explored the battle mechanics of catching Pokémon and leveling them. Dreamer fails to consistently engage in battles or catch Pokémon. Throughout exploration SENSEI assembles the strongest teams in terms of highest individual level and summed total levels of Pokémon in the party (Fig. 23b & 23c).
We visualize how exploration progressess over episodes in Fig. 24. Despite being only trained on task rewards, Dreamer quickly outperformed by SENSEI and Plan2Explore in terms of rewards achieved (Fig. 24a). This highlights the complexity of the Pokémon environment and how casily agents get stuck in local optima without structured exploration. Overall, SENSEI and Plan2Explore reach comparable reward levels.
How do SENSEI and Plan2Explore allocate their exploration? Plan2Explore appears to prioritize catching Pokémon: after around 50 episodes, it consistently maintains a larger party than SENSEI (Fig. 24b). This also reflected in the number of Pokémon caught by both methods (Fig. 25). Plan2Explore manages to collect wider variety of Pokémon across seeds. SENSEL on the other hand, allocates more resources to map-based progress towards the Gym (Fig. 24c) and leveling its Pokémon team through battles, consistently reaching higher Pokémon levels from episode 100 onward (Fig. 8c).
We hypothesize that SENSEI 's exploration is shaped by GPT-4's preferences and our prompt emphasizing the goal of defeating the first Gym Leader. Most carly-game wild Pokémon are weak against Brock's ' Rock/Ground type team, and only the Water-type starter Squirtle has a type advantage. As a result, the most promising strategy, reflected in SENSEI behavior, is to repeatedly battle and level Squirtle in preparation for the Gym.
SENSEI GENERAL
Plan2Explore
Dreamer
100
100
%6 opisodes
opisodes
opisodss
10
60
0.1
20
8
9
10
11
12
13
14
15
6.0 10 13 14
18
22 25 25
30 36
(a) party size
(b) highest levels of Pokémon in party
(c) summed levels of Pokémon in party
Figure Pokémon interaction statistics: We report the distribution maximum party size (a), the highest individual Pokémon level (b), and the total sum of levels across all Pokémon in the party (c) for 750k steps of SENSEL, Plan2Explore, and DreamerV3, Error bars indicate standard error across 5 seeds.
30

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
SENSEI GENERAL
Plan2Explore
DreamerV3
100
wetur per step
man segment
6-
3000
party
02
4-
100
350 400
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
episodes
episades
episodes
(a) return per step
(b) maximum party size
(c) maximum map segment
Figure 24. Exploration progress in Pokémon Red: We plot the mean return per step (a) and maximum party size reached (b) over episodes. To visualize map progress, we plot the maximum map segment reached between episode starting point (0) and the Gym (9) (c). We compare SENSEI GENERAL to Plan2Explore (r, + ou?" and Dreamer (rx). Shaded areas depict the standard error (5 seeds) and we apply smoothing over the score trajectories with window size 5.
E.9. Pokémon Red: Second Generation VLM-MOTIF
A limitation of our current SENSEI implementation is that if an agent encounters entirely novel states, the semantic reward may contain mostly noise, it is outside of the training distribution for VLM-MOTIF. Here, exploration relies on epistemic uncertainty rewards (r2fis, similar to Plan2Explore. This can be the case for large, open-ended environment such as in Pokémon Red. The initial SENSEI dataset contains 100K pairs from Plan2Explore run (500K steps exploration), reaching at most Viridian Forest (map segment 6, Fig. 8a). SENSEI reaches the frontier of Viridian Forest more often, occasionally breaking into Pewter City, where the first Pokémon Gym is located. However, the fist-ggenrrtionn VLM-MOTIF has never seen this area and thus relies on information gain explore beyond Viridian Forest.
We visualize these failure cases in Fig. 26. For instance, the first-generation reward function incorrectly assigns higher interestingness to task-irrelevant museum (top row, frames 3 & 5) than to the Pewter Gym (top row, frames 2 & 4), which is the actual task goal. Upon entering the Gym, the semantic reward drops sharply (middle row, frames 3-5), further highlighting this OOD failure.
To address this, we propose a simple refinement procedure: re-annotating data collected from a SENSEI run. We sample 50K additional observation pairs explored by SENSEI and annotate them using the same VLM prompting strategy. We then distill ccoondggnneration semantic reward function Rgg trained on both the original and newly collected annotations, now including previously unseen areas such as Pewter City and the Gym. As shown qualitatively in Fig. 26 (Generation 2), the refined reward function corrects earlier misjudgments: it spikes correctly upon seeing the Gym and peaks when the agent enters and faces the Gym Leader (middle row, frame 5). This demonstrates the potential of iterative semantic refinement. We see this as a promising extension to SENSEI, enabling agents to continuously improve their internal reward models by incorporating data from newly explored regions.
We evaluate the effect of the refined semantic reward function on exploration by continuing the original SENSEI runs for 200 additional episodes. Specifically, resume exploration from the same point used to collect the second-generation annotations, comparing runs that use either the refined reward function (Generation 2) or the original (Generation 1). To
100
Plan2Explore SENSEI
% of soods
80
60
Dreamer
40
20
0
Squirtle
Caterpie
Metapod
Weedle
Kakuna
Pidgey
Rattata
Pikachu
Nidorand
Figure 25. Pokémon caught: For SENSEI GENERAL, Plan2Explore and DreamerV3 we plot the ratio of seeds that managod obtain the listed Pokémon at least once during 750k steps of exploration. Other Pokémon were never caught.
31

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Generation 1 Ry
Generation 2 Ri
3
4
5
(norm.)
4
0.8
0.6
0.4
750
775 t
800
1
2
5
5
0.5
0999990009
AAA
1,650
1,670
1,690
2
3
4
5
0.5
prm (norm.)
HEXOLEE
MEGEOLE
3
NERDLE
4
5
0.4
0.3
0.2
315
325
t 335
345
(a) screenshots
(b) semantic exploration reward
Figure 26. First and second generation semantic rewards Pokémon Red: visualize semantic rewards from reward function Hyg learned either purely from Plan2Explore data (Generation 1) or refined based on SENSEI data. Generation 2 semantic rewards rf correctly peak when seeing Pewter Gym (top row, & 4) or entering the Gym (middle row, 3 -5). These images are out-of-distribution for Generation 1 Rg, which is incorrectly yielding low sematic rewards r In battle-related game play both reward functions are highly correlated.
assess whether simply running baselines longer would yield similar benefits, also continue For fairness and to avoid biasing our baselines, we select the only Plan2Explore seed that managed to reach Pewter City.
of Plan2Explore.
Exploration progress across these runs is visualized in Fig. 27, where dashed gray line marks the onset of the second- generation annotations and the divergence point between Generation and Generation 2. SENSEI Generation 2 broadly continues the exploration trends of Generation 1, outperforming Plan2Explore on metries except Pokémon party size (Fig. 27e). However there are a few subtle differences between SENSEI Generation
2: Both SENSEI Generation 1 & seem mostly avoid catching new Pokémon (Fig. 27c) and instead focus solely on leveling up their Squirtl. Thereby, SENSEI Generation 2 reaches slightly higher levels than Generation 1 (Fig. 27b). We hypothesize that SENSEI'S focus on leveling Squirtle comes from our prompt defining its objective "to find and defeat the Gym Leader, Brock". The best option to beat Brock is high-level Squirtle whose Water attacks are super effective against Brock's 's team. While SENSEI Generation 2 visits fewer overall map segments than Generation (Fig. 27d), it concentrates more on segments and 9 (Fig. 27c), corresponding to Pewter City and the Pewter Gym. This focused exploration, combined with a stronger Squirtle, allows SENSEI Generation to achieve critical milestone: defeating Brock, the Pewter City Gym Leader, and obtaining the Boulder Badge. This event occurs twice over the course of continued exploration (Fig. 27f), with the first badge earned after 570 episodes, or roughly 1.8 million environment steps. None of our baselines
able to achieve this milestone before.
This demonstrates the potential of iterative semantic refinement. We see this as promising extension to SENSEI, enabling agents to continuously improve their internal reward models by incorporating data from previously unexplored regions.
32

---

SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
SENSEI Generation
SENSEI Generation 2
Plan2tixplore
15
return jas stepp
600
lighes level
mtsp seguneat
6
100
10
4
800
0-
100
2000
300
400
500
600
100
200
300
400
500
600
100
200
300
500
600
episodes
episodes
episodes
(a) return per step
(b) highest Pokémon level
(c) maximum map segment
badges obtained
NW
10-
3--
patty
0.5
#
100
200
300
400
500
600
100
300
-100
500
200
300
400
500
600
eplsodes
epásodes
eplsodes
(d) map segments explored
(e) maximum party size
(I) Gym badges obtained
Figure 27. Exploration with first and second generation semantic rewards in Pokémon Red: We track exploration progress when continuing one seed of SENSEI with Ry annotated on its previous exploration data (Generation 2) and compare against continuing with the previous R2 (Generation 1) longer runs of Plan2Explore. We plot return per step (a), highest Pokémon level achieved (b). maximum map segment reached (c), overall number of map segments explored (d), maximum party size reached (e) or Gym badges obtained (f) over episodes. The gray dashed line marks the annotation Generation 2 Rg. (a)-(e) we apply smoothing over the trajectories with window size 10.
F. Computation
SENSEI has 3 phases: (1) annotation of data pairs (offline), (2) reward model, i.e. VLM-MOTIF, training (offline), (3) online RL training with environment interactions (DreamerV3). All experiments were performed on an internal compute cluster.
Dataset Annotation The annotation of data pairs is done using the OpenAl API, such that single CPU is sufficient. For instance for Robodesk with dataset size of 200K pairs, parallelized this over 200 CPUs, where annotated K pairs each, which took approximately 40 minutes. Note that annotations SENSEI itself. Each annotation using the single-turn strategy cost $0.002 with 2itt-4-2024-0555 3 and $0.004 with 2ot-44-urbo-2000 2024-04-----00-00 -99. The multi-turn prompting for the zero-knowledge Robodesk annotations also cost $0.004 per pair with gpt-4o-2024 1000002220--115
fully offline and do not affect the runtime of
Reward Model Training epochs. Using e.g. Tesla 1100-SXM2-32GB, this took 20min. We ran grid search over different hyperparameters for VLM-MOTII training (batch size, learning rate, weight decay, network size), testing for total of 18 different combinations, and we chose the reward model with the best validation loss to use in SENSEI runs.
After annotating the dataset, we train the YLM-MOTIF network using a single GPU for 50
Online Model-based RL Training SENSEI is built on top of DreamerV3, just like our main baseline Plan2Explore. On a NVIDIA A100-SXM4-80GG, SENSEI runs at ca. 7.5Hz, Plan2Explore runs at ca. 10Hz and pure VLM-MOTIF runs at ca. 8. 8.7Hz.
33


Reviews:
[{"review_id": "3K0ayWSfL7", "reviewer": "Reviewer_7mHN", "summary": "The paper proposes incorporating human priors into RL exploration to encourage policies to internalize a model of *interestingness*. This is done by first annotating pairs of frames for interestingness using VLMs (which, owing to training on internet-scale human data, has incorporated these priors). This is then distilled into a reward model, which in turn is distilled into the world model of a model-based Reinforcement Learning policy. The authors additionally use an epistemic uncertainty-based reward, incentivizing the agent to try new behaviors in interesting states.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and highlighting that our research direction is important and acknowledging that our experimental results are interesting.\n\n### New Environment: Pokémon Red\n\nAs per your suggestion, we now apply SENSEI to another environment: the classic Game Boy game Pokémon Red. This is a challenging environment due to its (1) vast, interconnected world and (2) complex battle system requiring semantic knowledge (e.g. Water beats Rock). Strong exploration is essential for progressing toward the main goal: becoming the Pokémon Champion by defeating Gym Leaders. We use the PokeGym v0.1.1 setup [1], based on [2], with:\n\n- Observations: Only the raw 64×64 game screen, unlike previous RL applications which additionally used internal game states or memory [2,3].\n- Actions: A 6-button Game Boy action space (Left, Right, Up, Down, A, B).\n- Rewards: Event-based rewards following [1].\n- Episode Length: Episode length is increased with environment steps: 1k until 250k steps, 2k during 250–500k, 4k during 500–750k, etc. We take actions every 1.5seconds of gameplay.\n- Game Start: We skip the tutorial, until the agent can move freely and catch Pokémon, starting after receiving the Pokédex. We use the same checkpoint as [2], beginning with a level 6 Squirtle.\n\nWe compare SENSEI-General to Plan2Explore. Both receive extrinsic task-based rewards alongside self-generated exploration rewards. SENSEI's initial dataset is collected using 500k steps of Plan2Explore (100K pairs). We use a general prompt: \n\n'Your task is to help me play the Game Boy game Pokémon Red. I just obtained my starter Pokémon, Squirtle. My goal is to find and defeat the Gym Leader, Brock. What do I need to do, and which areas do I need to traverse to achieve this goal? Keep it very short.'\n\nWe generate five different responses from GPT-4 and sample from them uniformly as context for image-based comparisons.\n\nThe results are shown on our rebuttal webpage: https://sites.google.com/view/sensei-icml-rebuttal. We plot the histogram of the maximum map reached by the agent at each episode throughout the training run for SENSEI and Plan2Explore (5 seeds). Our results show that SENSEI progresses further into the game, and most notably manages to reach the first Gym in Pewter City, unlike Plan2Explore. Both SENSEI and Plan2Explore explore catching and training Pokémon, with SENSEI typically assembling a party of slightly higher levelled Pokémon than Plan2Explore. In order to succeed in the game you need to have a diverse team with high levels. This means achieving a high party level (higher counts on the right side of the histogram).As Pokémon unlocks new maps over time, we also explore using generations of SENSEI: refining the semantic reward using data from an earlier SENSEI run. See our response to Reviewer 9xtC for details.\nThank you for your great suggestion of adding another environment. Let us know if you have further questions about our setup or results.\n\n### Approximating human interestingness \n\nWe assume “proxy-of-a-proxy” refers to our two-step distillation process of 1) first training motif from VLM annotations and 2) learning to predict semantic rewards in the world model. We understand the reviewer’s concern, but want to emphasize why both design choices are essential. We will explain this here in detail and add more emphasis on these aspects in our updated paper.\n\n1. Distilling a semantic reward function $R_\\psi$ from VLM annotations is necessary as it allows us to not deal with a large and slow VLM inside the fast RL loop. Additionally, directly querying VLMs can lead to noisy outputs, which can further derail the agent’s learning.\n\n2. Learning a mapping from world model states to semantic rewards is a necessary design choice when working with latent state world models as we point out in the discussion and detail in Suppl. A. 3. \n\nWorld models such as the RSSM typically encode and predict dynamics fully in a self-learned latent state. Thus, for a world model to predict $r^{sem}_t$ at any point in time, we need a mapping from latent states to semantic rewards. \n\nThis begs the question whether our mapping is the best way to do it. Another option would be to decode the latent state to images and use those as inputs for Motif. However, we believe this has several disadvantages: (1) Inefficiency—image decoding from latent states is expensive and slows down training. (2) Artifacts—decoded images can be blurry or unrealistic, causing a distribution shift and making Motif, trained on real environment images, encounter out-of-distribution errors.\nWe hope this motivates our two-step distillation process. We thank the reviewer for allowing us to clarify this and update the paper to better emphasize this in the method section and discussion.\n\n[1]  Suarez (2024): https://pypi.org/project/pokegym/0.1.1/\n\n[2]  Whidden (2023), https://github.com/PWhiddy/PokemonRedExperiments,  \n\n[3]    Pleines et al. (2025): Pokemon Red via Reinforcement Learning", "claims_and_evidence": "While the experimental results are interesting, in my opinion, results on just two environments (Minihack, Robodesk) fall short of concretely establishing the claims.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I did not find any major issues with the experimental designs.", "relation_to_broader_scientific_literature": "Intrinsic Motivation and Reward Shaping quite important in the field of Reinforcement Learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "While the paper proposes a novel take on intrinsic motivation in terms of human priors, the major weakness is the limited experimental analysis. I would like to see this framework extended to at least one other environment.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "* It appears that the two-stage process of transferring the VLM's interestingness analysis into the agent's world model -- via a distilled reward model -- may lead to significant information loss. At this stage, the approach seems like a proxy of a proxy for human interestingness, making it quite indirect and prone to potential failure points. Did the authors explore simpler alternatives?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWZ3XyXfhy", "reviewer": "Reviewer_9xtC", "summary": "The paper proposes SENSEI, a framework designed to enhance exploration in model-based RL by integrating semantic guidance from VLMs. SENSEI distills a reward signal of interestingness from VLM-generated annotations of observations, guiding agents toward semantically meaningful interactions. This intrinsic reward system enables the learning of versatile world models, with internal models of interestingness, improving exploration efficiency. Empirical results in robotic and video game simulations demonstrate that SENSEI successfully encourages agents to discover meaningful behaviors from low-level actions and raw image observations, significantly enhancing downstream task learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful feedback and recognition of SENSEI’s novelty, thorough evaluations, and strong empirical evidence. Below, we address your key questions and minor comments.\n\n### SENSEI in newly unlocked areas\n\n> What happens if exploring a behavior in one area unlocks uncertain behaviors in a previously explored area?\n\nAs the reviewer correctly noted, we differentiate two cases: 1) the newly unlocked behavior exists in the initial self-exploration dataset, 2) the behavior is entirely novel.\n\nCase 1 is naturally accommodated by SENSEI, as these semantic relationships are reflected in the learned VLM-Motif. For example, in MiniHack-KeyChest, an agent might initially discover a locked chest before finding the key. Upon acquiring the key, the previously explored chest gains new semantic significance, yielding higher semantic rewards. SENSEI adapts seamlessly, as semantic meaning evolves with discoveries. In contrast, Plan2Explore and other uncertainty-driven methods lack an inherent mechanism for handling such scenarios.\n\nCase 2 connects to our new experiments and the reviewer's question 3:\n\n### Second Generation of SENSEI Annotations\n\n> Since the distilled reward function is fixed during world model training, what if the agent discovers behaviors or states absent from the initial self-supervised data?\n\nGreat point. If an agent encounters entirely novel states, the semantic reward may contain mostly noise, as it’s out of distribution for the trained VLM-Motif. Here, exploration relies on epistemic uncertainty rewards ($r^{dis}$), similar to Plan2Explore. In such cases, we can refine our notion of interestingness with new data. We adopt a generational perspective: run SENSEI for a first generation, then perform a second round of VLM annotations to distill new semantic rewards based on newly explored regions.\n\nWe tested this in Pokémon Red, comparing SENSEI-General to Plan2Explore (see Reviewer 7mHN response for environment details and results). The initial SENSEI dataset contains 100K pairs from a Plan2Explore run (500K steps), reaching at most Viridian Forest. SENSEI reaches the frontier of Viridian Forest more often, occasionally breaking into Pewter City, where the first gym is located. However, the first-generation VLM-Motif has never seen this gym and thus relies on information gain to explore beyond Viridian Forest. After 750K steps, we sample 50K new pairs from a SENSEI run and re-annotate them, forming a refined second-generation semantic reward function ($R_\\psi$) now informed by previously unseen maps.\n\nQualitative results (Figure E2) on our rebuttal webpage [https://sites.google.com/view/sensei-icml-rebuttal](https://sites.google.com/view/sensei-icml-rebuttal) show that second-generation annotations correctly emphasize Pokémon Gyms—unlike the initial generation. This is essential for the agent to further explore the gym and beat the gym leader.\n\nWe hope this motivates extending SENSEI to complex domains where new behaviors are unlocked through generations.\n\n### Semantic Rewards and Evolving Experience\n\n> What happens if what’s considered interesting changes over time?\n\nWhile our semantic reward $r^{sem}$ is independent of agent experience, we assume a VLM inherently ranks more complex tasks (e.g., stacking) higher than simpler ones (e.g., pushing). However, SENSEI also incorporates epistemic uncertainty ($r^{dis}$) into its exploration reward ($r^{expl}$, Eq. 7), so the agent engages in interesting and yet novel behaviors. As exploration progresses, $r^{dis}$ naturally decreases as the world model improves (Eq. 6).\n\nConsider a case where the VLM assigns equal interestingness to pushing and stacking. Initially, both are uncertain, yielding similar $r^{expl}$. As the agent masters pushing, $r^{dis}$ diminishes, lowering $r^{expl}$ for that skill. The agent then prioritizes stacking—aligning exploration with increasingly complex tasks.\n\n### Minor Comments\n\n**Human evaluations in Robodesk**  \n\n> In the Robodesk evaluation, it would be useful to include a human evaluation of behaviors to support the claim that “our semantic exploration reward leads to more meaningful behavior than purely epistemic uncertainty-based exploration.”\n\nTo avoid overstating, we revise the sentence to: “On average, our semantic reward leads to more object interactions than purely epistemic uncertainty-based exploration.”\n\n> Why is SENSEI-General not shown in MiniHack?\n\nThe MiniHack prompt is already highly general. We describe it as a “rogue-like game” with a generic description and note the egocentric view, without explicitly mentioning doors, chests. The key is only mentioned if it is part of the agent's inventory (Supp C3.3). This differs from Robodesk, where we specify relevant objects. Given this broad framing, we don’t expect SENSEI-General to yield significantly different results.\n\nWe appreciate the reviewer’s insightful questions and welcome any further discussion.", "claims_and_evidence": "Yes, the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method, e.g., learning an internal model of interestingness via SENSEI, is novel and the evaluation criteria make sense for the problem.\n\nIn the Robodesk evaluation, it would be useful to include a human evaluation of the produced behaviors to support the claim that “our semantic exploration reward seems to lead to more meaningful behavior than purely epistemic uncertainty-based exploration.”", "theoretical_claims": "Yes, they are correct.\n\nThe writing in Section 2.3 can make the distinction between r_t_sem and r_t_sem_hat clearer.", "experimental_designs_or_analyses": "Yes, the experiments are sound and valid. Experiments are conducted over two domains and thorough analyses are done on both.\n\nI appreciate the SENSEI General experiments on Robodesk, showing that SENSEI is not dependent on the injection of external environment knowledge. However, why is SENSEI General not shown in the MiniHack environment? It will be useful to show it as an ablation in both domains.", "relation_to_broader_scientific_literature": "SENSEI is most related to intrinsically motivated RL. Unlike traditional intrinsic exploration methods, which emphasize novelty or state-space coverage, this work integrates semantic guidance from VLMs to provide more human-aligned, meaningful intrinsic rewards, following recent trends of leveraging foundation models for guiding exploration (e.g., Motif, OMNI). Additionally, by incorporating model-based RL frameworks and world models, the approach advances the capability of RL agents to predict semantic “interestingness,” thus bridging human-defined semantic priors with effective exploration in complex environments.", "essential_references_not_discussed": "I don’t have any additional ones to suggest.", "other_strengths_and_weaknesses": "Strengths\n- The proposed method and idea is novel and interesting. Thorough experiments also show how having an internal model of interestingness in the world model can contribute to downstream tasks.\n\nWeaknesses\n- See other sections.", "other_comments_or_suggestions": "See other sections.", "questions_for_authors": "1. What happens if exploring a behavior in one area unlocks uncertain behaviors in a previously explored area? For example, suppose the agent has already explored a box in a room but then presses a button at the other end of the room, opening the box. Now, previously explored spaces contain newly unlocked, interesting behaviors. How might SENSEI or future extensions handle this scenario?\n2. The current version of SENSEI makes use of a fixed distilled reward function. However, what happens if what’s considered interesting changes over time? For example, in robot manipulation, initially, interacting with objects might be interesting, but once the robot has mastered basic manipulation, the focus of interestingness might shift towards more challenging tasks, such as assembling or stacking objects. How might SENSEI perform in scenarios where the definition of interesting behaviors evolves?\n3. Relatedly, since the distilled reward function is not updated during world model training, what if the agent discovers behaviors or states not present in the initial self-supervised exploration data? For instance, consider an agent accidentally falling into a ditch. Since the initial exploration data did not include this environment, the distilled reward function lacks information regarding the interestingness of exploring this new scenario. How might SENSEI cope with this situation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "smYQ9ACwPW", "reviewer": "Reviewer_wRRZ", "summary": "The paper introduces a novel framework for intrinsic motivation in reinforcement learning (RL) agents, enabling them to explore environments meaningfully without relying on task-specific rewards. The authors propose SEmaNtically Sensible ExploratIon (SENSEI), which leverages Vision Language Models (VLMs) to guide exploration by distilling a reward signal reflecting observations' semantic interestingness. The method demonstrates the ability to discover meaningful behaviors in complex environments and promises to accelerate downstream task learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and greatly appreciate that they found our paper well-organized, our experiments thorough, and our evidence clear and convincing. We aim to address the remaining concerns below.\n> The paper lacks comparisons to recent exploration methods beyond P2X like Curiosity-Driven Exploration via Disagreement or Skew-Fit\n\nFirst, we note that we compare not only against P2X, but also RND and VLM-Motif—a reimplementation of Motif adapted to our world model and VLM setting. Thus, we compare against 3 strong exploration baselines.\n\nSecond, Plan2Explore represents the state-of-the-art in curiosity-driven exploration via disagreement for pixel-based inputs. We are not aware of other disagreement-based methods suitable for comparison.\n\nSkew-Fit, on the other hand, falls into the goal-based exploration paradigm, where goals are sampled to maximize state coverage. Our RND baseline also targets state-space coverage. We believe goal-based methods are orthogonal to our setting. In addition, we note that Hu el al. 2023 [1] shows (Fig. 4 of main paper) that an improved model-based adaptation of Skew-Fit is performing worse or similar to Plan2Explore on a range of environments, making Plan2Explore the stronger baseline to compare to. \n \n> Testing SENSEI on a broader range of tasks (e.g., navigation, social interaction) would better demonstrate its versatility.\n\nWe have now added a new environment—Pokémon Red—which involves navigation over a large map and interactions with Pokémon and gym leaders. See our response to Reviewer 7mHN for details and our rebuttal webpage (https://sites.google.com/view/sensei-icml-rebuttal) for results.\n\n> This paper lacks a detailed methodological comparison with Plan2Explore.\n\nOur method builds on Plan2Explore by adding semantic rewards and dynamic reward scaling. We apologize if this was unclear. We have clarified this connection in the method section and discussion.\nWe also empirically analyze the contributions of both additions via comparisons to VLM-Motif and through ablations on dynamic reward scaling (Suppl. D.6).\n\n> In Figure 8, why do you choose observations from view (b) apart from the reasons you mentioned in the paper?\n\nIn Robodesk’s default view, the robot arm often occludes objects, presenting challenges for learning. Early tests also showed better VLM annotations from the right camera view, which we therefore used.\n\n> However, the definition of interestingness is a very subjective concept. Is the exploration of all tasks considered interesting?\n\nWe agree that interestingness is subjective. However, we argue that across cultures, commonalities exist in what humans find interesting—reflected in the large-scale training data of VLMs. The tasks in our environments were all designed by humans, implying inherent human interest.\n\nFor annotation with SENSEI-General in Robodesk, we first ask the VLM for a description of the environment and what could be interesting. We observe strong alignment between the VLM’s responses and the environment’s task distribution (Supp. C.3.2).\n\n> In Figure 7, SENSEI has already trained for 500K steps before fine-tuning on downstream tasks. Would it be fair to let DreamerV3 train for an additional 500K steps?\n\nExcellent point—our method does spend more time in the environment. However, note that SENSEI does not optimize for any particular task during exploration. Even with additional training, Dreamer, in our experiments, does not match SENSEI’s performance.\n\nFor example, in MiniHack-KeyRoomS15, SENSEI solves the task reliably after 300K steps of task-based training (on top of 500K steps exploration), totaling 800K steps. Dreamer, even after 1M steps, does not reliably solve the task across seeds.\n\nTo emphasize this further, we now compare SENSEI in a Robodesk task (Upright-Block-Off-Table) after 1M exploration steps to Dreamer trained from scratch. SENSEI solves the task across all seeds in 600K steps; Dreamer only succeeds in one seed after 2M steps. Even counting SENSEI’s 1M exploration, it outperforms Dreamer with 1.6M total steps (see our rebuttal webpage, Figure E4).\n\n> Task scenarios like Noisy TV may lead to excessive exploration of novel areas, resulting in exploration failure. Have you considered these?\n\nIndeed, this is a known issue in intrinsically motivated RL. However, due to the ensemble disagreement formulation—used in Plan2Explore and our method—this is not a concern. In stochastic environments, the ensemble predictions converge to the process mean with enough samples, resulting in 0 disagreement and no further reward. We refer the reviewer to [2] for more details.\n\n[1] Hu, E. et al., Planning goals for exploration, ICLR 2023.\n\n[2] Pathak, D. et al., Self-supervised exploration via disagreement, ICML 2019.", "claims_and_evidence": "The claims in the SENSEI paper are supported mainly by clear and convincing evidence, though some aspects could benefit from further validation or broader testing.\n\nThe paper suggests SENSEI could scale to real-world applications, but experiments are limited to simulated environments (MiniHack, Robodesk). Photorealism and occlusion handling are acknowledged limitations.\n\nThe paper lacks comparisons to recent exploration methods beyond P2X (e.g., Curiosity-Driven Exploration via Disagreement, Skew-Fit). This limits the strength of claims about SENSEI’s superiority.", "methods_and_evaluation_criteria": "The SENSEI paper's proposed methods and evaluation criteria are well-aligned with the problem of semantically guided exploration in reinforcement learning (RL). There are some potential improvements for this paper: (1) Including more recent exploration methods (e.g., Curiosity-Driven Exploration via Disagreement, Skew-Fit) would strengthen the evaluation by providing a broader context for SENSEI’s performance. (2) Evaluating SENSEI on a wider range of tasks (e.g., navigation, manipulation, and social interaction) could further demonstrate its versatility.", "theoretical_claims": "There is no proof in the paper.", "experimental_designs_or_analyses": "Yes. The experimental designs and analyses in the SENSEI paper are mostly sound, with clear motivations and appropriate metrics. However, the robustness of VLM annotations, the quality of the world model, and the diversity of tasks could be more rigorously tested. Addressing these issues would strengthen the paper’s claims and provide a more comprehensive evaluation of the proposed methods.", "relation_to_broader_scientific_literature": "The SENSEI paper's key contributions are closely related to the broader scientific literature on reinforcement learning (RL), intrinsic motivation, and the use of foundation models for exploration. \n\nIntrinsic Motivation: SENSEI builds on intrinsic motivation in RL, where agents explore environments without external rewards. Prior methods (e.g., curiosity-driven exploration, prediction error, Bayesian surprise) focus on low-level interactions, while SENSEI leverages VLMs to guide exploration toward semantically meaningful behaviors (similar to children’s play).\n\nModel-Based RL: SENSEI uses a Recurrent State Space Model (RSSM) to predict semantic rewards, aligning with prior work on world models (Ha & Schmidhuber, 2018; Hafner et al., 2023). The key innovation is predicting semantic rewards directly from latent states, enabling the agent to evaluate hypothetical states.", "essential_references_not_discussed": "In most real-world scenarios, there are potential complex disturbances that can affect the model's learning and judgment. Task scenarios like Noisy TV [1,2,3] may lead to excessive exploration of novel areas, resulting in exploration failure.\n\n**References**\n\n[1] urgen Schmidhuber, J. (1991). Adaptive confidence and adaptive curiosity. Technical Report FKI {149 {91 (revised), Technische Universit at M unchen, Institut f ur Informatik.\n\n[2] Mavor-Parker, A., Young, K., Barry, C., & Griffin, L. (2022, June). How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference on Machine Learning (pp. 15220-15240). PMLR.\n\n[3] Huang, K., Wan, S., Shao, M., Sun, H. H., Gan, L., Feng, S., & Zhan, D. C. (2025). Leveraging Separated World Model for Exploration in Visually Distracted Environments. Advances in Neural Information Processing Systems, 37, 82350-82374.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-organized, clearly explaining the method, experiments, and results.\n2. The experiments are thorough, with ablation studies (e.g., Figure 17 for fixed reward weighting, Figure 18 for hyperparameter sensitivity) and comparisons to strong baselines (e.g., Plan2Explore, RND). The results are presented with standard errors and multiple seeds, ensuring reproducibility.\n\n**Weaknesses**\n\n1. The paper lacks comparisons to state-of-the-art exploration methods like Curiosity-Driven Exploration via Disagreement or Skew-Fit. Including these would provide a more comprehensive evaluation of SENSEI’s performance.\n2. The evaluation focuses on two MiniHack tasks and a subset of Robodesk tasks. Testing SENSEI on a broader range of tasks (e.g., navigation, social interaction) would better demonstrate its versatility.\n3. This paper lacks a detailed methodological comparison with Plan2Explore. However, if viewed from another perspective, it resembles an exploration of the intrinsic rewards of disagreement based on the world model's ensemble dynamics in Plan2Explore, adding a VLMs reward related to the task's interestingness, and then considering how to balance these two rewards. Of course, engineering improvements are also important, especially if the results are outstanding.", "other_comments_or_suggestions": "See questions.", "questions_for_authors": "1. In Figure 8, why do you choose observations from view (b) apart from the reasons you mentioned in the paper? Have you conducted any ablation studies on this design?\n2. From the provided prompts, the rewards given by the large model are based on interestingness. However, the definition of interestingness is a very subjective concept. Is the exploration of all tasks considered interesting?\n3. In Figure 7, SENSEI has already trained for 500K steps before fine-tuning on downstream tasks. Would it be fair to let DreamerV3 train for an additional 500K steps before conducting a formal comparison of downstream tasks?\n4. In most real-world scenarios, there are potential complex disturbances that can affect the model's learning and judgment. Task scenarios like Noisy TV [1,2,3] may lead to excessive exploration of novel areas, resulting in exploration failure. Have you considered these noisy exploration scenarios?\n\n**References**\n\n[1] urgen Schmidhuber, J. (1991). Adaptive confidence and adaptive curiosity. Technical Report FKI {149 {91 (revised), Technische Universit at M unchen, Institut f ur Informatik.\n\n[2] Mavor-Parker, A., Young, K., Barry, C., & Griffin, L. (2022, June). How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference on Machine Learning (pp. 15220-15240). PMLR.\n\n[3] Huang, K., Wan, S., Shao, M., Sun, H. H., Gan, L., Feng, S., & Zhan, D. C. (2025). Leveraging Separated World Model for Exploration in Visually Distracted Environments. Advances in Neural Information Processing Systems, 37, 82350-82374.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oUPCQ1rBTh", "reviewer": "Reviewer_bmzW", "summary": "This paper introduces SENSEI (SEmaNtically Sensible ExploratIon), a LM-based framework for guiding the exploration phase of RL agents towards \"interesting\" states. It trains a reward model based on VLM-generated rankings of interestingness on prior exploration data. The final exploration reward is the sum of the aforementioned interestingness score and an uncertainty score computed by ensemble disagreement (taken from Plan2Explore [1]). It dynamically reweights these two scores to first go towards an interesting state, then explore from there. Experiments are performed in two domains: MiniHack (navigating dungeons and finding keys) and Robodesk (simulated robot arm interacts with objects on a desk). Results show that this method, compared to baselines:\n1. succeeds in directing towards the agent towards semantically meaningful states during exploration\n2. solves tasks faster", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive review. We appreciate that you found our paper comprehensive, the introduced concepts interesting, and our experimental design sound.\n\n### VLM-based exploration baselines\n\nExisting VLM-based exploration methods (Omni, Motif, ELLM) rely on assumptions that limit their applicability to our setting, such as requiring expert demonstrations (Motif), language-grounded environments (Omni, Motif, ELLM), or high-level actions (Omni, ELLM). In contrast, SENSEI operates in a more general RL setup without these constraints.\n\nMotif also uses an initial offline expert dataset. Omni requires access to the full task list and corresponding rewards. Motif and ELLM both rely on event-counting to bias novelty. In contrast, we show how novelty and semantic interestingness can be seamlessly combined with minimal assumptions, enabling deployment in low-level control environments while preserving scalability.\n\nAdditionally, none of these methods use model-based RL, which is central to SENSEI’s sample-efficient exploration. For fairness, we adapt Motif into our VLM-Motif baseline. Our results show SENSEI outperforms VLM-Motif, highlighting the benefits of dynamic reweighting and the importance of incorporating information gain within a model-based approach. We will add this discussion to the final version.\n\n### Offline VLM usage\n\nFor details on why offline VLM usage is necessary/preferred and on our distillation procedure, please see our answer to Reviewer 7mHN (Section: Approximating human interestingness).\n\n### Diversity of Envs and corresponding interestingness\n\n> There is also a question of how large and diverse these environments are, whereby the goal state may be the only \"interesting\" state there is.\n\nWe now include a diverse environment: Pokémon Red. It requires navigating a large map and mastering battle mechanics to catch/fight wild Pokémon and defeat Trainers. Here, interestingness is multi-faceted (map progression, Pokémon collection, etc.), and SENSEI shows consistent gains. Please see our response to Reviewer 7mHN and our rebuttal webpage: [https://sites.google.com/view/sensei-icml-rebuttal](https://sites.google.com/view/sensei-icml-rebuttal).\n\n### Pre-Collected Dataset\n\nSENSEI uses pre-collected data, unlike Plan2Explore, but our dataset size is chosen to ensure coverage of interesting states—not to be minimal. Crucially, SENSEI remains effective with significantly less data. In a new ablation, we show that in MiniHack-KeyRoom, using just ¼ of the dataset (25k samples) yields similar interaction statistics (Webpage Figure E5).\n\nIn real-world settings, various offline robotics datasets (e.g., OpenX-Embodiment, TriFinger RL) contain expert and non-expert control data, making SENSEI directly applicable. Alternatively, SENSEI could begin with exploration purely via $r^{dis}$, then distill the reward from collected data. We appreciate this suggestion and will include the discussion in the final version. It aligns well with our generational SENSEI perspective, further supported by the Pokémon experiment (see response to Reviewer 9xtC).\n\n### Alignment Between Goals and \"Interestingness\"\n\nIf a task’s goals or subgoals don’t align with \"interestingness\", SENSEI relies on epistemic uncertainty ($r^{dis}$) to discover them. In practice, this has not been limiting.\n\nFor instance, in MiniHack-KeyRoom, the goal is to reach a staircase (Fig. 4, top, img 5), which GPT-4 does not consider especially interesting. Standing near the exit yields lower semantic rewards (point 5) than standing near the key and door (points 3&4). Still, SENSEI reliably solves the task and collects more rewards than Plan2Explore by reaching the staircase.\n\n### Interestingness vs Information content\n\nInterestingness captures semantic relationships between entities, which may not be reflected in simple information content heuristics.\n\nFor example, in MiniHack-KeyChest, compare: (1) “Agent is next to a locked chest with a key in the inventory” and (2) “Agent is next to a locked chest with no key”. Both have similar information content, but (1) is more interesting due to the key-chest relation.\n\n> Can interestingness be adjusted based on existing information (e.g., a new object is more interesting than a familiar one)?\n\nThis is precisely what SENSEI achieves by combining interestingness with disagreement (information gain). We aim to explore regions that are both interesting and novel.\n\n> What kinds of states are \"interesting\"? How is this distinct from human-like subgoals in earlier work?\n\nWe assume VLMs incorporate priors aligned with human preferences. Thus, human-like subgoals would likely be favored by VLMs over random states. However, unlike prior goal-based work, we don’t assume access to a subgoal set. Instead, we use Motif to compute a continuous reward, approximating interestingness.\n\nWe hope this response addresses the reviewer’s concerns and we would be happy to clarify further if needed.", "claims_and_evidence": "Examining the evidence presented for the main claims made in the submission (1 & 2 in summary):\n1. SENSEI guides LMs towards \"interesting\" states: The evidence shown for this claim includes a small amount of qualitative samples (Figure 4), and a quantitive analysis showing evidence that agents generally interact more with objects under SENSEI than P2X. The precie definition of \"interestingness\" is worth some discussion in the paper: For example, why are object interactions considered \"interesting\"? Breakdown of other types of interactions deemed interesting by the reward model could be useful: e.g. seeing information-dense states. Are different types of interactions on the same object considered more or less interesting (e.g. pressing a button vs. moving it)? Furthermore, can interestingness be adjustable based on existing information (e.g. a new object is more interesting than an object seen before)? This claim can benefit from being slightly more precise -- or including a more rigorous analyses of what types of states have high \"interesting\" reward.\n2. SENSEI solves task faster: This seems fairly well-substantiated from the end-to-end experiments in Figure 4. SENSEI outperforms Plan2Explore, DreamerV3, and PPO, all of which do not use a interestingness-based reward. It could be clarified that P2E is basically exactly the same as SENSEI but without the interesting-ness reward, representing a minimal ablation of the contributed portion.", "methods_and_evaluation_criteria": "SENSEI relies on data that is pre-collected from an exploration policy in order to distill an \"interesting-ness\" reward function from. Given that the policy relies on pre-collected data, it seems like an unfair comparison to compare against baselines that are purely from-scratch and not bootstrapped with any initial exploration data: perhaps a fairer comparison (and a more realistic setting) would be to look at the transferability of the distilled interesting-ness model across environments (even though this itself would mean SENSEI has access to some extra domain-specific information that other methods don't), or start P2X with the world model it's built up with the existing exploration data, and have it continue exploring from there. Why not just use the VLM online to give interestingness rewards?\n\nThe choice of evaluation datasets makes sense to me. The paper shows improvements on two very different domains: one with navigating a dungeon and one with interacting with objects on a desk. Perhaps adding a real-world domain would also be useful here (but I don't think it's essential for this work).", "theoretical_claims": "No theoretical claims in paper", "experimental_designs_or_analyses": "The experimental designs generally seem sound. Experiments were conducted on two different (simulated) environments. Comparisons against Plan2Explore (the method upon which this paper is based -- which is missing the interestingness score) and Random Network Distillation (a model-free method trained with PPO), demonstrate 1) evidence of increased semantically-meaningful object interactions, 2) empirical improvements on downstream tasks, when the interestingness score is incorporated into the model.\n\nOne question is how much these environments may already be in GPT-4o, which may already a sense of what goals exist in these environments, and thus output \"interestingness\" scores based on proximity to the goal. There is also a question of how large and diverse these environments are, whereby the goal state may be the only \"interesting\" state there is.\n\nSome additional analyses to consider:\n1. Is downstream performance improvement predicated on the goal state (or a subgoal state) being one of (or proximate to) the identified interesting states? What happens in settings where goals may not necessarily be aligned with \"interestingness\"?\n2. To what extent is GPT4-o necessary for ranking \"interestingness\" and not just heuristics like total information content of the state?\n3. To what extent is GPT4-o's rating of interestingness generalizable to diverse (potentially real-world) environments and goals?", "relation_to_broader_scientific_literature": "Using LLMs to provide semantic signal for exploration has been studied extensively over the past few years, including by certain methods cited by the authors themselves (e.g. ELLM, OMNI, LAMP). It's unclear to me why this method would be preferred over others. For example, ELLM does not require exploration data to be gathered ahead of time, so might even be preferable in certain cases. (On the other hand, if the distilled model is general across environments, I can see a cost-based argument for not needing to re-query GPT-4o each time a new environment is encountered -- though I am not certain that is the case for this method.) A comparison to these methods should be included as baseline(s), or at the very least a discussion on when/why would SENSEI be preferred over these techniques.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Overall, the paper is quite comprehensive with its experiments/analyses and has done a good job making a case for the necessity of an interestingness score. The two phases of exploration seems like a very interesting concept.\nMy main concerns have been listed above: specifically, I think the two critical concerns I have with this paper are:\n\n1. Novelty with respect to prior literature: there has been lots of prior work on LLM/VLM-guided exploration, and it is unclear why SENSEI is advantageous over these prior methods. It would be great if the authors to include these baselines, or at least a discussion of when/why SENSEI is better than them.\n\n2. The method assumes access to a *pre-collected* dataset of exploration data on the specific environment that the agent is trying to explore. It is unclear to me when this setting where pop up in the real world, and why this data isn't just used to initialize the agent, who can then choose to continue exploring from there (if it so desires). Moreover, none of the baselines assume access to this pre-collected dataset, so the evaluation isn't quite comparable.", "other_comments_or_suggestions": "None", "questions_for_authors": "If you have any important questions for the authors, please carefully formulate them here. Please reserve your questions for cases where the response would likely change your evaluation of the paper, clarify a point in the paper that you found confusing, or address a critical limitation you identified. Please number your questions so authors can easily refer to them in the response, and explain how possible responses would change your evaluation of the paper.\n\n1. it is unclear to me why the distillation step is even necessary: why not just use the VLM directly online?\n2. it is unclear whether the VLM is even necessary, or what exactly it is doing -- it would be insightful to compare to heuristic baselines like information content of the image, number of objects in the state, etc.\n3. What kinds of states are \"interesting\" and how is this distinct from (or perhaps the same as) generating human-like subgoals from earlier papers?\n\nPlease also see questions from above. More details about baselines Plan2Explore and RND would be useful in the main paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies using VLMs to derive a signal of "interestingness" for accelerating RL. The reviewers and I generally agree that the experiments are satisfactory evaluations of the method, and the method is a somewhat-useful contribution to the growing research area of using foundation models as a source of learning or exploration signals. The author responses addressed some of the remaining methodological questions. However, there are still lingering questions around the motivation for e.g. reliance on pre-collected data to initialize, whether the extent of the novelty is sufficient to merit publication, etc. The paper will certainly be of some interest, but potentially within a narrow community.
Decision: Accept (poster)