SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models

Authors: Cansu Sancaktar, Christian Gumbsch, Andrii Zadaianchuk, Pavel Kolev, Georg Martius
Primary area: reinforcement learning
Keywords: intrinsic motivation, exploration, foundation models, model-based RL

Abstract:
Exploring useful behavior is a keystone of reinforcement learning (RL). Intrinsic motivation attempts to decouple exploration from external, task-based rewards. However, existing approaches to intrinsic motivation that follow general principles such as information gain, mostly uncover low-level interactions. In contrast, children’s play suggests that they engage in meaningful high-level behavior by imitating or interacting with their caregivers. Recent work has focused on using foundation models to inject these semantic biases into exploration. However, these methods often rely on unrealistic assumptions, such as environments already embedded in language or access to high-level actions. To bridge this gap, we propose SEmaNtically Sensible ExploratIon (SENSEI), a framework to equip model- based RL agents with intrinsic motivation for semantically meaningful behavior. To do so, we distill an intrinsic reward signal of interestingness from Vision Language Model (VLM) annotations. The agent learns to predict and maximize these intrinsic rewards using a world model learned directly from intrinsic rewards, image observations, and low-level actions. We show that in both robotic and video game-like simulations SENSEI manages to discover a variety of meaningful behaviors. We believe SENSEI provides a general tool for integrating feedback from foundation models into autonomous agents, a crucial research direction, as openly available VLMs become more powerful.

Decision: reject
Reviewer scores: 3, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729169253098

thread_type:
official_review

<thread_object>
creator: reviewer_dzbp

<content>
summary:
The authors introduce SEmaNtically Sensible Exploration (SENSEI), a technique inspired by the previous work MOTIF, which uses Large-Language Models (LLMs) to express preferences between different observations in order to generate a reward model for a (model-free) reinforcement learning agent. The authors propose adapting this technique for Vision-Language Models (VLMs) and integrating it with world model learning. After an initial data collection phase, an "interestingness" reward model is distilled from the VLM’s preferences over image observations. This reward model is combined with the Recurrent State-Space Model (RSSM) of DreamerV3 and the ensemble technique used in Plan2Explore to develop an exploration policy that is influenced by the learned semantic reward. The results demonstrate that this combination of VLM-MOTIF and Plan2Explore leads to more effective exploration than relying on either technique alone. Additionally, downstream task experiments show that SENSEI outperforms baseline methods in terms of task performance.

strengths:
- The extension of MOTIF to VLMs feels natural and improves the applicability of such techniques beyond environments with text representations.
- The paper does not solely rely on the semantic reward model but recognizes that epistemic uncertainty is important to consider. The authors analyse this and propose a well-motivated adaptive solution that combines the two sources of information to achieve more effective exploration. 
- The paper provides a solid empirical validation; the experiments are overall well-explained and detailed and show that SENSEI outperforms baseline approaches in terms of exploration quality and downstream task performance

weaknesses:
- The proposed method seems to rely on domain-specific prompt engineering. For instance, as shown in Appendix C3, in the Robodesk environment, the VLM is provided with detailed, manually crafted prompts specifying what constitutes "interesting" behavior. This significantly limits the generality and practicality of the proposed method.
-  The method heavily relies on the dataset collected through a self-supervised exploration technique, which is then used for annotation and reward model training. A key concern here is that the distilled reward model is inherently biased towards the specific data collected through this relatively simple exploration technique. This may not pose issues in small, simple environments but is likely to be fragile in more complex, large-scale environments. The reward model is static, meaning it cannot adapt to observations outside the distribution of the pre-collected dataset, limiting its ability to generalize to novel situations.
- In the Appendix, it is mentioned that the authors used 139409 pairs of observations for Robodesk. If I understand correctly, this means the VLM was prompted 139409 times to obtain the annotations for a single environment. If so, the practicality and scalability of the approach becomes slightly concerning. The cost of the experiments is not mentioned in the paper, but it is likely that querying VLMs this often could become computationally expensive, particularly in larger-scale environments.

questions:
**Suggestions**
- Regarding the reliance on detailed prompt engineering, one potential resolution could be to lean more heavily on the prior knowledge and inference capabilities of the VLM. Instead of providing explicit descriptions of the environment, the VLM could be tasked with inferring the context from the visual observations. This would allow the model to deduce what constitutes interesting behavior without needing environment-specific prompts. While this might sacrifice some performance, it could provide a more general solution applicable across a variety of visual environments without manual prompt engineering for each domain.
- A suggestion regarding the concern about over-reliance on the initial pre-collected dataset in larger, more complex environments would be to consider introducing additional phases of data collection and retraining. After the initial exploration and training with the distilled reward model, the agent (using DreamerV3 + SENSEI) could collect new data during subsequent exploration, which could then be annotated and used to retrain a new reward model. Although this approach would still be biased by the initial dataset, I believe that this direction of research could progressively reduce the bias and help the model better adapt to novel observations.

**Minor comments / Questions**
- Line 146, small error 'In reward training phase'
- I would suggest omitting the parts from your section titles in Section 2, such as "UNLEASH YOUR SENSEI", as it does not fit the professional tone of the rest of the paper.
- At a high level, in your proposed exploration scheme, the agent is incentivized to first find interesting states and then switch to an uncertainty-maximizing approach. However, what is not completely clear to me is how or when it then switches back to $\beta_{Go}$ to continue finding new interesting states?
- In Figure 5, why is there no comparison with VLM-MOTIF, whereas in Figure 4, there is?
- In Figure 4, the SENSEI agent performs poorly in the "at chest with key" scenario compared to VLM-MOTIF. You mention that this is because "being at the chest with a key is an 'interesting' state, so there is no real incentive for the agent to explore what would happen if the chest was opened." However, intuitively, one might expect that an open chest would be considered even more interesting. Is there any further intuition as to why the VLM preferences do not guide the agent in exploring this further? Have you seen in annotations that if presented as a pair, the key+chest was rated more interesting than an open chest?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729488205773

thread_type:
official_review

<thread_object>
creator: reviewer_5hj6

<content>
summary:
The authors propose SENSEI, a framework to equip model-based RL agents with intrinsic motivation for semantically meaningful behavior.
They distill an intrinsic reward signal of interestingness from VLM annotations, following MOTIF. The agent learns to predict and maximize these intrinsic rewards using a world model, using RL algorithms similar to Dreamerv3. They conduct experiments on both robotic and
video game-like simulations and achieve good results.

strengths:
1. Good motivation and experiments. The authors explain their methods very clearly, and their experiments are diverse and convincing. Incorporating VLM into reinforcement learning is a hot topic, but this kind of integration sounds novel.

2. Various related work. The coverage of the huge amount of related work is about as thorough as possible.

3. Sufficient ablation study. The authors conduct several ablations to demonstrate the effectiveness of each component.

weaknesses:
1. It seems that most components come from existing work. 

2. MOTIF uses much weaker VLMs, so it's possible that the performance gain mostly come from the improvement of VLMs. The authors could perform experiments to demonstrate the extent to which performance declines when using a less powerful model.

questions:
1. The authors may briefly explain how long it will take to train SENSEI (dataset annotation, reward training and RL training) and the baselines using the same computation resources.

2. The authors emphasize a realistic setting. It would be much harder to create a 200K dataset of preferences if we want to deal with real world embodied agents. Have the authors tried to use a smaller dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730316799599

thread_type:
official_review

<thread_object>
creator: reviewer_71ub

<content>
summary:
This paper proposes a way to combine VLM feedback and exploration bonuses. The main idea of this paper is to combine Plan2Explore, a latent ensemble disagreement-based exploration method, with a MOTIF-based intrinsic reward that tells the agent how interesting a state is. Specifically, SENSEI first trains a semantic reward model based on a dataset using MOTIF (w/ preference-based reward learning), and then runs Plan2Explore augmented with the learned semantic reward function. The authors also propose an additional mechanism to adaptively control the coefficients for the semantic and ensemble-based intrinsic rewards. They show that their method (SENSEI) outperforms Plan2Explore and VLM-MOTIF on MiniHack and Robodesk tasks.

strengths:
* The proposed method is straightforward and reasonable.
* The authors empirically show that SENSEI improves exploration on both MiniHack and Robodesk, and the paper has some analysis results as well.
* The paper is generally well-written. In particular, I enjoyed reading Sections 1 and 2.

weaknesses:
* The method is largely a straightforward combination of two existing techniques: Plan2Explore and MOTIF. While I think this *alone* doesn't necessarily constitute a ground for rejection, I do think the results are not terribly surprising nor extremely insightful in the current form. It may have been more informative to practitioners if the authors had focused much more on ablation studies or analyses (beyond those in Appendix D). For example: Is MOTIF the only way to define a semantic reward model, and if so, how/why is it better than other alternatives? Is having two phases necessary (can we not have a separate pre-training stage)? Is the adaptive coefficient adjusting strategy (Eq. (7)) necessary? How sensitive is the performance to this strategy?
* The experimental results are somewhat limited. The authors only compare SENSEI to its ablations (P2X, VLM-MOTIF), and it is unclear how SENSEI performs compared to other types of exploration methods like LEXA/PEG, or VLM-based exploration methods like ELLM/OMNI/LAMP. I don't expect comparisons with all of these baselines, but I think it is important to have at least some comparisons with different types of previous methods in each category.
* The hyperparameter table in Appendix B implies that SENSEI is potentially sensitive to various hyperparameters. For example, it seems SENSEI requires careful, individual tuning of the quantile hyperparameter for each individual task in Robodesk (it uses 0.75, 0.75, 0.80, and 0.85 for each task). How were these values chosen, and did the authors perform the same level of hyperparameter tuning for the baselines as well? Did the authors decouple the runs for hyperparameter tuning and for the results (the results will otherwise be biased)? How sensitive is SENSEI to these individually tuned hyperparameters (quantile, batch size, learning rate, weight decay, etc.)?
* The authors use only 3 seeds for the experiments, which further makes it difficult to evaluate the empirical contributions of the method.

questions:
Please answer the questions in the weaknesses section above. The paper is mostly clear, and I don't have any other specific clarification questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730372224362

thread_type:
official_review

<thread_object>
creator: reviewer_qrft

<content>
summary:
This paper presents SENSEI, a model-based RL framework that guides agents' exploration by eliciting a human notion of "interestingness" from a vision-language model (VLM). The VLM annotates preferences between pairs of observations, gathered through self-supervised exploration. This annotated preference dataset is then used to train a reward model, SENSEI, which is subsequently distilled into the agent’s world model. The proposed approach demonstrates superior performance over baselines in experimental settings, including robotic and video game-like simulations.

While the paper introduces promising ideas, I believe it needs further development before it is ready for publication.

strengths:
- Exploring how human priors can be integrated into RL agents using VLMs is a valuable research direction.
- The combination of intrinsic rewards derived from both VLMs and epistemic uncertainty effectively enhances performance.
- The graphical illustrations are clear and helpful, aiding readers in understanding the methodology and results.

weaknesses:
- The primary concern is the incorporation of a human notion of "interestingness" in the decision-making process. I believe environments where humans can clearly express preferences between observations based on interestingness are limited. For instance, in settings like the game of Go or simple mazes with few interactive objects, it can be challenging to determine what constitutes an "interesting" observation. The authors attempt to address this in the prompt design described in Appendix C.3, where they define "interestingness" for specific environments. However, defining “interestingness” specifically for each environment may limit the method’s ability to incorporate true human priors. In my opinion, the actual prompt the authors use is simply a more specified version of that used in [1], which prompts for the option “most likely to make progress towards the goal” or “most likely to show improvement with respect to the goal.” As a result, the distinction between this work and previous methods may be less substantial. To enhance originality and better substantiate the claims, I recommend refining the prompt design to more clearly elicit human priors from VLMs and including a thorough ablation study on different prompt choices.
- Additionally, I found the formulation unnecessarily complex. The VLM preferences are distilled twice, first into the reward model (SENSEI) and then into the world model. A more straightforward approach might be to incorporate the reward model directly into the world model to simplify the structure and better convey the core idea.

[1] Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. The Twelfth International Conference on Learning Representations, 2024.

questions:
- There is only one baseline in the main experiment. This paper could benefit from including more baselines to enable a more comprehensive comparative analysis.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732484129512

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers for their constructive feedback and appreciate that they found our research direction “well-motivated” and “valuable”, the method to be “straightforward and reasonable”, the experiments “diverse and convincing” with “ solid empirical validation”, and the paper “generally well-written”. 

We notice that common suggestions among reviewers were more experimental evaluations, which we hope to address in this rebuttal. As a short summary, we provide 1) an ablation for self-supervised annotations using non-environment specific prompts, 2) a new exploration baseline (RND [ref 1]),  3) ablations on hyperparameter sensitivity and our novel coefficient adaptation strategy, 4) we provide more details on implementation, for example on the computational resources used, and 5) we ran more random seeds. We provide more details below and in the individual responses.

# Summary of changes

## General prompt with zero-knowledge for image annotation (qRFT & dZbp)

We investigated whether we can employ SENSEI using a more general prompting strategy without prior knowledge about the environment. In this general prompt setting (detailed in Suppl. C.3.2) we first prompt the VLM for an environment description given a screenshot and use this context to annotate image pairs with respect to their interestingness. In Suppl. D.4 we qualitatively analyze the resulting reward function for Robodesk and illustrate how the semantic rewards positively correlate or even match the ones obtained from our original, more specified, reward function.  

## New baselines in Robodesk (qRFT & 71Ub & dZbp)

We’ve added Random Network Distillation (RND) [ref 1] with a PPO policy as a new model-free exploration baseline in Robodesk. SENSEI outperforms RND in almost all interaction metrics (Fig. 5) and discovered rewards (Fig 13), except for button presses. We also added VLM-Motif as a baseline in Robodesk, which SENSEI also outperforms in most interactions metrics (Fig. 15) and rewards (Fig. 13), showcasing the importance of the information gain objective of SENSEI.  

## New ablations on hyperparameter sensitivity and dynamic coefficient adaption (71Ub)

We extensively analyze the effects of the dynamic coefficient adaptation strategy (Eq. 7) and the hyperparameter sensitivity of SENSEI in Robodesk in Suppl. D.6. In sum, the coefficient adaptation strategy makes our method much more robust to the exact choice of hyperparameters. Without coefficient adaptation, our method can still achieve a high performance for a subset of object interactions, but with coefficient adaptation, it achieves a high rate of interactions across the board.


## Computational resources (5Hj6 & dZbp)

We now detail our computational resources in a new supplementary section (Supp D.7).


## More seeds (71Ub)

We now run all Minihack experiments with 2 more seeds, so a total of 5 seeds. We will continue increasing the number of seeds for the camera-ready version, for both Minihack and Robodesk.

## PPO
 
By revisiting PPO, we also found an inconsistency in our implementation affecting PPO’s performance in MiniHack-KeyChest. We apologize for this. After re-running the experiments, PPO’s sample efficiency is increased in KeyChest. However, SENSEI still learns to reliably solve the task faster than PPO and, thus, the main message of these experiments remains unchanged.

We hope the detailed responses to the individual reviewers additionally address remaining questions. We thank all reviewers again for their time and efforts spent reviewing our paper.

## References

[ref 1] Burda, Yuri, et al. "Exploration by random network distillation.", ICLR 2019.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a model-based RL framework that guides agents' exploration through human-defined "interestingness" from a vision-language model.  This work compares with existing baselines and shows superior performance. The core issue for this approach is the scalability of it. It can be challenging to find “interesting” parts purely from images.  Moreover, the overall method is complicated, which undermines the impact. Thus, I recommend the authors to polish the manuscript and submit to another conference.