Vision-Language Models Provide Promptable Representations for Reinforcement Learning
Authors: William Chen
Oier Mees
Aviral Kumar
Sergey Levine
Venue: Accepted by TMLR ()

Abstract: Humans can quickly learn new behaviors by leveraging background world knowledge. In contrast, agents trained with reinforcement learning (RL) typically learn behaviors from scratch. We thus propose a novel approach that uses the vast amounts of general and indexable world knowledge encoded in vision-language models (VLMs) pre-trained on Internet-scale data for embodied RL. We initialize policies with VLMs by using them as promptable representations: embeddings that encode semantic features of visual observations based on the VLM's internal knowledge and reasoning capabilities, as elicited through prompts that provide task context and auxiliary information. We evaluate our approach on visually-complex, long horizon RL tasks in Minecraft and robot navigation in Habitat. We find that our policies trained on embeddings from off-the-shelf, general-purpose VLMs outperform equivalent policies trained on generic, non-promptable image embeddings. We also find our approach outperforms instruction-following methods and performs comparably to domain-specific embeddings. Finally, we show that our approach can use chain-of-thought prompting to produce representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times.

Published in Transactions on Machine Learning Research (03/2025)
Vision-Language Models Provide Promptable Representa- tions for Reinforcement Learning
William Chen U.C. Berkeley
Oier Mees U.C. Berkeley
Aviral Kumar Carnegie Mellon University
Sergey Levine
U.C. Berkeley
Reviewed on OpenReview: https: // openreview. net/ forum? id- vQDKYYuqWA
//
Abstract
Humans quickly learn new behaviors by leveraging background world knowledge. In contrast, agents trained with reinforcement learning (RL) lypically learn behaviors from scratch. We thus propose novel approach that uses the vast amounts of general and index- able world knowledge encoded in vision-language models (VLMs) pre-trained on Internet- scale data for embodied R.. We initialize policies with VLMs by using them as promptable representations: embeddings that encode semantic features of visual observations basod on the VLM's internal knowledge and reasoning capabilities, as elicited through prompts that provide task context and auxiliary information. We evaluate our approach on visually- complex, long horizon RL tasks in Minecraft and robot navigation in Habitat. We find that our policies trained on embeddings from off-tho-shelf, general-purpose VLMs outper- form equivalent policies trained on generic, non-promptable image embeddings. We also find our approach outperforms instruction foolloition-ollowing methods and performs comparably to sppain-specific embeddings. Finally, we show that our approach can use chain-of- chain-of-thought prompting produce representations of common-scnss semantic reasoning, improving pol- icy performance in novel scenes by 1.5 times.
1 Introduction
Embodied decision-making often requires representations informed by world knowledge for perceptual grounding, planning, and control. Humman rapidly learn to perform sensorimotor tasks by drawing on prior knowledge, which might be high-level and abstract ("If I'm cooking something that needs milk, the milk is probably in the refrigerator") or grounded and low-level (e.g., what refrigerators and milk look like). These capabilities would be highly beneficial for reinforcement learning (RL) too: we aim for our agents interpret (asks in terms of concepts that can be reasoned about with relevant prior knowledge and grounded with previously-learned representations, thus enabling more efficient learning. However, doing requires condensed source of vast amounts of general-purpose world knowledge, captured in form that allows us to specifically index into and access tosk-relevant information. Therefore, we need representations that are contextual, such that agents can use a concise task context to draw out relevant. background knowledge, abstractions, and grounded features that aid it in acquiring a new behavior.
An approach to facilitate this involves integrating RL agents with the prior knowledge and reasoning abilities of pre-trained foundation models. Transeorrer-based language models (LMs) and vision-language models

---

Published in Transactions on Machine Learning Research (03/2025)
Minecraft Taskc Combat Spider
Habitat Find Follet
Obserstiora
Oeseraatins
VIMI
Learned
Task Env
VEN
Learsed
Task Env
Actora
"Sades selhoooss Ins oack iters audde 11's smager
Vouall bakel bayt whynott"
them araider."
ALPS PaaCoo Trales
Figure 1: Example instantiations PR2L tasks in Minecraft and Habitat. query a VLM with task- relevant prompt about observations to produce promptable representations, which we train a policy on via RL. Rather than directly asking for actions or specifying the task, the prompt enables indexing into the VLM's prior world knowlodge to access relevatl information. This prompt also allows us to inject auxiliary information and elicit chain-of-thongg reasoning.
(VLMs) are trained on Internet-scale data to enable generalization in downstream tasks requiring facts or common sense. Moreover, in-context learning (Brown et al., 2020), chain-of-thought reasoning (CoT) (Wei) et al. 2023), and instruction fine-tuning (Ouyang et al., 2022] have provided better ways to index into (V)LMs' knowledge and steer their capabilities based on user nceds. These successes have seen some transfer embodied control, with (V)LMs being used to reason about goals to produce executable plans (Ahn et al. 2022 or encoders of useful information (like instructions (Liu et al.
2023) feedback (Sharma et al. 2023 that the control policy utilizes. Both these paradigms have major limitations: actions generated by LMs are often not appropriately grounded, unless the tasks and scenes are amenable to being expressed or captioned in language. Even then, (V)LMs are often only suited to producing subtask plans, not low-level control signals. On the other hand, using (V)LMs to simply encode inputs under-utilizes their knowledge and reasoning abilities, instead focusing on producing embeddings that reflect the compositionality of language (e.g., so an -ostuuttoonfolllwinng policy may generalize). This motivates the development of an algorithm for learning to produce low-level actions that are grounded and leverage (V)LMs' knowledge and reasoning.
To this end, we introduce Promptable Representations for Reinforcement Learning (PR2L): flexible framework for steering VLMs into producing semantic features, which (i) integrate observations with prior task knowledge and (ii) are grounded into actions via RL (see Figure Specifically, we ask a VLM questions about observations that are related to the given control task, priming to attend to task-relevant features in the image based on both its internal world knowledge, reasoning capabilities, and any supplemental information injected via prompting. The VLM then encodes this information in decoded text, which is discarded, and associated embeddings, which serve as inputs to a learned policy. In contrast to the standard approach of using pre-trained image encoders to convert visual inputs into generic features for downstream learning, our method yields task-specific features capturing information particularly conducive to learning considered task. Thus, the VLM does not just produce an un-grounded encoding of instructions, but embeddings containing semantic information relevant to the task, that is both grounded and informed by the VLM's 8 prior knowledge.
To the best of our knowledge, we introduce the first approach for initializing RL policies with generative VLM representations. We demonstrate our approach on tasks in Minecraft (Fan et al.) 2022 and Habitat (Savva et al. 2019), as they present semantically-ridd problems representative of many practical, realistic, and challenging applications of RL. We find that PR2L outperforms equivalent policics trained on vision-only embeddings or with instruction-conduuoonnng popular ways of using pre-trained image models and VLMs respectively for control. We also show that promptable representations extracted from general-purpose VLMs are competitive with domain-specific representations. Our results highlight how visually-complex control tasks can benefit from accessing the knowledge captured within VLMs via prompting in both online and offline RL settings.
2

---

Published in Transactions on Machine Learning Research (03/2025)
2 Related Works
Vision- models. In this work, we utilize generalive VLMs (such as Li et al. 2022 2023a et al. (2023) Karamcheti et al.] (2024)): models that generate language in response to an image and text prompt passed as input. This is in contrast to other designs of combining vision and language that either generate images or segmentation (Rombach et al., 2022 Kirillov et al., 2023) and contrastive representations (Radford et al. 2021). Formallyy the VLM enables sampling from p(æ1: (,),, where æ1:K represents the K tokens of the output, I is the imput image(s), is the prompt, and p the distribution over natural language responses produced by the VLM on those inputs. Typically, the VLM is pre-trained on tasks that require building association between vision and language such captioning. All these tasks require learning to attend to certain semantic features of input images depending on the given prompt. For auto-regressive generative VLMs, this distribution is factorized as I[, c, Typical architectures parameterize these distributions using weights that define a representation oe c, Ø1a-1), which depends on the image I, the prompt. e, and the previously emitted tokens, and a decoder p(r o, f1s-1)), which defines a distribution over the next token.
Dai
Embodied (V)LM reasoning. Many recent works have leveraged (V) as priors over effective plans for given goal. These works use the network's ' language modeling and rego-seeeessvve generation capabilities to extract such priors as textual subtask sequences (Ahn et al.
2022:
Huang ct al.,
2022b Sharma et al.
2022) or code (Liang el al. 2023 Singh et al. 2022: Zeng et al.]
2022 Vemprala et al. 2023 thereby using
the (V)LM to decompose
tasks into executable parts. These systems often need grounding
mochanisms to ensure plan fcasibility (c.g., affordance estimators (Ahn et al. 2022), scene captioners (Zeng) et al. 2022), or trajectory labelers (Palo et al., (023)). They also often assume access to low-level policies that can execute these subtasks, such as robot pick-and-place skills (Ahn et al., 2022 Liang et al.] 2023)
which is often strong assumption. These methods generally do not address how such policies can be acquired, nor how these low-level skills can themselves benefit from the prior knowledge in (V)LMs. Even works in this area that use RL still use LV)MMs as state-dependent priors over reasonable high-level goals to learn (Du et al. 2023). This is a key difference from our work: instead of considering priors on plans/goal, we rely on VLM's implicit knowledge of the world to extract representations which encode task-relevant information. We train policy to convert these features into low-level actions via standard RL, meaning the VLM does not need to know how to take actions for task.
Embodied (V)LM pre-training Other works use (V)LMs to embed useful information such as instruc- tions [Lu et al. 2023 Myers et al. 2023 Lynch & Sermanet 2021 Mees et al. 2023 O.M.T. et al. 2023). feedback (Sharma et al. 2023 Bucker et al., 2022). reward specifications FFan et al, 2022), and data for world modeling (Lin et al. 2023b Narasimhan et al.. 2018). These works use (V)LMs as encoders of the compositional semantic structure of input text and images, which aids in generalization: an instruction- conditioned model may never have learned to grasp apples (but can grasp other objects), but by interacting with them in other ways and receiving associated language descriptions, the model might still be able to grasp them zero-shot. In contrast, our method produces embeddings that are informed by world knowledge and reasoning, both from prompting and pre-training. Rather than just specifying that the task is to acquire an apple, we ask a VLM to parse observations into task-relevant features, such as whether there is an apple in the image or if the observed location likely contains apples - information that is useful even in single-task RL. Thus, we use VLMs to help RT, solve new tasks, not just to follow instructions.
These two categories are not mutually exclusive: Brohan al. (2023a) use VLMs to understand instructions, but also reasoning (e.g., reasoning that the "correct bowl" for a strawberry is one that contains fruits); Palo et al. (023) use LM to reason about goal subtasks and a VLM to know when trajectory matches subtask, automating the demonstration
sallcttion/abellinn of Ahn et al. (2022). while Adeniji et al. (2023) use a similar approach to pretrain a language-conditionel RL policy that is transferable to learning other tasks; and Shridhar et al. (2021) use CLIP to merge vision and text instructions directly into a form that a Transporter (Zeng et al. 2020) policy can operationalize. Nevertheless, these works primarily focus on instruction-followeee for robot manipulation. Our approach instead prompts VLM to supplement RL with representations of world knowledge, not. instructions. In addition, except. for Adeniji et al. (2023). these
3

---

Published in Transactions on Machine Learning Research (03/2025)
Vision-Language Model
Policy
Action
Frozen LLM Transformer Layers
Policy Network
Non-visual Observations
Summary Embed
Image Encoder
Tokenizer
Detokenizer
Learned Transformer
Encoder
Decoder
Dacoded Text *Vas, spider."
"Spiders Minecraftt black. (heve e spideer in inagw?"
CLS
Schematic of how we extract task-relevant features from the VLM and use them in a policy trained with RL. These representations can incorporate task context from the prompt, while generic image embeddings cannot. As generative VLM's embeddings can be variable length, the policy has a Transformer layer that takes in these embeddings and a "CLS" token, thereby condensing all inputs into a single summary vector.
Figure 2:
works focus on behavior cloning (BC), assuming access to demonstrations for policy learning, whereas our framework can be used for both online RL and offline RL/BC.
3 PR2L: Promptable Representations for Reinforcement Learning
We adopt the standard framework of partially-observed Markov decision process deep RL, wherein the objective is to find a policy mapping states actions that maximizes the expected returns. Our goal is to supplement RL with task-relevant information extracted from VLMs containing general-purpost knowledge, One way to index into this information is by prompting the model to get it to produce semantic information relevant to given control task. Therefore, our approach, PR2L, queries a VLM with a task-relevant prompt for each visual observation received by the agent, and receives both the decoded text and, critically, the intermediate representations, which we refer to as promptable representations. Even though the decoded text might often not be correct or directly usable for choosing the action, our key insight that these VLM embeddings still provide useful semantic features for training control policies via RL. This recipe enables to incorporate semantic information without the need of re-training or fine-tuning VLM to directly output actions, proposed by BBoohnn et al. (2023)). Note that our method is not instruction- instruction-ffolowingg method, and it does not require task instruction to perform well. Instead, our approach still learns control via RL, while benefiting from the incorporation of background contert. In this section, we will describe various components of our approach, accompanied by practical design choices and considerations.
3.1 Promptable Representations
In principle, one directly query VLM to produce actions for task given a visual observation. While this may work when high-level goals or subtasks are sulficient, VLMSs are empirically poor at yielding the low-level actions used commonly in RL (Huang et al. 2022a). As VLMs are trained to follow instructions and answer questions about images, it is more appropriate to use these models to extract and reason about semantic features about observations that are conducive to being linked to actions. We thus elicit features that are useful for the downstream task by querying these VLMS with task-relevant prompts that provide contextual task information, thereby causing the VLM to attend to and interpret appropriate parts of observed images. Extracting these features naïvely by only using the VLM's decoded tezt has its own challenges: such models often suffer from hallucinations (Ji et al.) 2023) and an inability to report what they "know" in language, even when their embeddings contain such information (Kadavath et al. 2022 Hu & Levy 2023). However, even when the lext is bad, the underlying represenlations still contain va valuable granular world information that is potentially lost in the projection to language (Li et al.) 2021 Wiedemann et al. 2019 Huang et al., 2023 Li et al. (023b). Thus, we do not directly condition upon the generated text and instead provide our policy the underlying embeddings produced by the VLM in response prompts asking about relevant semantic features in observations.
4

---

Published in Transactions on Machine Learning Research (03/2025)
Which parts of the network can be used as promptable representations? The VLMs we consider are all based on the Transformer architecture (Vaswani et. al. 2017), which treats the prompt, input image(s), and decoded text as token sequences. This architecture provides a source of learned representations by computing embeddings for cach token at every layer based on the previous layer's token embeddings. In terms of the generalive VLM formalism introduced prior, a TransSormer-based VLM's representations dh, C, F1.-1) consist of N embeddings per token (the outputs of the N self-attention layers) in the input image T, prompt e, and decoded text F1--11 The decoder p(x) extracts the final layer's embedding of the most recent token 2t-1, projecting it to distribution over the token vocabulary and allowing for it to be sampled. When given visual observation and task prompt, the tokens representing the prompt, image, and answer consequently encode task-relevant semantic information. Thus, for each observation, we use the VLM to sample response to the task prompt Iı:K ~ p(x We then use some or all of these token embeddings Φx (,, c, as our promptable representations and feed them, along with any non-visual observation information, as a state representation into our neural policy trained with RL.
In summary, our approach involves creating a task-relevant prompt. that provides context and auxiliary information. This prompt, alongside the current visual observation from the environment, is fed to into the VLM to generate tokens. While these tokens are used for decoding, they are ultimately discarded. Instead, we utilize the representations produced by the VLM (associated with the image, prompt, and decoded text) as input for our policy, which is trained via an off-the-shelf online RL algorithm to produce appropriate actions. A schematic of our approach is depicted in Figure 2 and a code snippet example is presented in Appendix !
3.2 Design Choices for PR2L
To instantiate this idea, we need to make some concrete design choices in practice. First, the representations of the VLM's decoded text depend on the chosen decoding scheme: greedy decoding is fast and determin- istic, but may yield low-probability decoded tokens; beam search improves on this by considering multiple "branches" of decoded text, at the cost of requiring more compute time (for potentially small improvements); lastly, sampling-based decoding can quickly yield estimates of the maximum likelihood answer, but at the cost of introducing stochasticity, which may increase variance. Given the inherent high-variance of our tasks (due to sparse rewards and partial observability) and the expense of VLM decoding, we opt for greedy decoding or fixed-seed sampling.
Sccond, one must choose which VLM layers' embeddings to utilize in the policy. While theoretically, all layers of the VLM could be used, pre-trained Transformer models tend to encode valuable high-level semannic information in their later layers (Tenney et al.) 2019 Jawahar et al.) 2019)). Thus, we opt to only feed the final few layers' representations into our policy. As these representation sequences are of variable length, incorporate encoder-decoder Transformer layer in the policy. At each time step in a trajectory, this layer receives variable-length VLM representations, which are attended to and converted into fixed-length summarization by the embeddings of a learned "CLS" token (Devlin et al. 2019) in the decoder (green in Figure 2 This design choice is akin to attention pooling, variants of which are an established way for extracting and aggregating representation ions from Transformers (Zhai et al. 2022: Karamcheti al. 2023 Lee ct al., 2019), but other approaches, like global average pooling, may also be viablc. We also note that this policy can receive the observed image directly, so as to not lose any visual information from being processed by the VLM. However, we do not do this in our experiments in order to more clearly isolate and demonstrate the usefulness of the VLM's representations in particular.
Finally, while it is possible to fine-tune the VLM for RL end-to-end with the policy (Brohan et al. 2023a this incurs substantial compute, memory, and time overhead, particularly with larger VLMs. heles,, we find that our approach performs better than not using the language and prompting components of the VLM. This holds true even when the VLM frozen, and only the policy is trained via RI, or when the decoded text occasionally fails to answer the task-specific prompt correctly.
5

---

Published in Transactions on Machine Learning Research (03/2025)
3.3 Task- Relevant Prompt Design
How do design good prompts to elicit useful representations from VLMs? As we aim to extract good state representations from the VLM for a downstream policy, we do not use instructions or task descriptions, but task-relevant prompts: questions that make the VLM attend to and encode semantic features in the image that are useful for the RL policy learning to solve the task (Borja-Diat et al., 2022). For instance, if the task is to find a toilet within a house, appropriate prompts include "What room is this?" and "Would a toilct be found here?" Intuitively, the answers to these questions help determinc good actions (e.g.. look around the room or explore elsewhere), making the corresponding representations good for representing the state for a policy. Answering the questions will require the VLM to attend to task-relevant features in the scene, relying on the model's internal conception of what things look like and common-sense semantic relations. One can also prompt the VLM to use chain of thought (Wei et al. 2023) to explain its generated text, often requiring it to reason about task-relevant features in the image, resulting in further enrichment of the state representations. Finally, prompts can provide helpful auxiliary information: egg., one can describe what certain entities of interest look like, aiding the VLM in detecting them even if they were not commonly found in the model's pre-training data.
Note that prompts based on instructions or task descriptions do not enjoy the above properties: while the goal of those prior methods is to be able to directly query the VLM for the optimal action, the goal of task-relevant prompts is to produce a useful state representation, such that running RL with them can learn a more performant policy with fewer samples. While the former is not possible without task-specific training data for the VLM in the control task, the latter proves beneficial with off-the-shelf VLMs.
Evaluating and designing prompts for RL. Since the specific representations elicited from the VLM are determined by the prompt, want to design prompts that produce promptable representations that maximize performance the downstream task. The brute-force approach would involve running RL with each candidate prompt to measure its efficacy, but this would be computationally very expensive. Tn lieu of this, we evaluate candidate prompts on a small dataset of observations labeled with semantic features of interest for the considered task. Example features include whether task-relevant entities are in the image, the relative position of said entities, or even actions (if expert retonstrations are available). We test. prompts by querying the VLM and checking how well the resulting decoded text for each image matches ground truth labels. As this is only practical for small, discrete spaces that are easily expressed in words, we see how well small model can fit the VLM's embeddings to the labels (akin to probing in self-supervised learning (Shi et al. 2016 Belinkov & Glass 2019)). While this does not directly optimize for task performance, it does act as a proxy that ensures a prompt's 's resulting representations encode certain semantic features which are helpful for the task.
4 Experimental Setups
Our experiments analyze whether promptable representations from VLMs provide benefits downstream control, thus providing effective vehicle for transferring Internet-scale knowledge to RL. We aim to show that PR2L is a good source of state representations, even with our current VLMs that are bad at reasoning about actions- - as such models become more performant, we expect such representations to be even better. We thus design exporiments to answer the following: (1) Can promptable representations obtained via task- specific prompls enable more performant and sample-efficiet learning than those of non-promptable image encoders pre-trained for vision or control? (2) How does PR2L compare to approaches that directly "ask" the VLM to generate good actions for a task specified in the prompt?
(3) How does PR2L fare against other popular learning approaches or purely visual features in our domains of interest?
4.1 Domain 1: Minecraft
We first conduct experiments in Minecraft, which provides control tasks that require associating visual obscrvations with rich semantic information to succood. Morcover, since these observations are distinct from the images in the the pre-training dataset of the VLM, succeeding on these tasks relies crucially on the efficacy of the task-specific prompt in meaningfully affecting the learned representation, enabling to
6

---

Published in Transactions on Machine Learning Research (03/2025)
stress-test our method. E.g., while spiders in Minecraft somewhat resemble real-life spiders, they exhibit stylistic exaggerations such as bright red eyes and large black body. If the task-specific prompt is indeed effective in informing the VLM of these facts, it would produce a representation that is more conducive to policy learning and this would be reflected in task performance. For this domain, we use the half-precision Vicuna-7B version of the InsI instruction-tan generative VLM (Dai et al. 2023 2023 to produce promptable representations.
Chiang et al.
Minecraft tasks. We consider all programmatic Minecraft tasks evaluated by Fan et al. 2022 combat spider, milk cow, shear sheep, combat zombie, combat enderman, and combat pigmar D The remaining tasks considered by Fan et al. (2022) are creative tasks, which do not have programmatic reward functions or success detectors, so we cannot directly train RL agents on them. We follow the Mije definitions of observation action spaces and reward function structures for these tasks: at each time step, the policy observes an egocentric RGB image, its pose, and its previously action; the policy can choose a discrete action to turn the agent by changing the agent's pitch and/or yaw in discrete increments, move, attack, or use a held item. These tasks are long horizon, with maximum episode length of 500- 1000 and taking roughly 200 steps for learned policy to complete them. See Figure 3 for example observations and Appendix B.1 for more details.
Comparisons. We compare PR21 to five performant classes of approaches for RL in Minecraft: (a) Meth- ods using nou-promp prompromtta representations of visual observations. This does not use prompting altogether, instead using lask-agnostic embeddings from the VLM's image encoder (specifically, the ViT-g/14 from In- structBLIP blue in Figure 2). While these representations are still pre-trained, PR2L utilizes prompting to produce task-specific representations. For fair comparison, we use the ezact same policy architecture and hyperparameters for this baseline in PR2L, ensuring that performance differences come from prompting for better representations from the VLM. (b) Methods that directly "asks" the VLM to output actions to execute on the agent. This adapts the approach of Brohan et al. (2023a) to our setting and directly outputs the action from the VLM. While Brohan et al.] (2023a) also finc-tune the VLM backbone, we are unable to do so using our compule resources. To compensale, we do not just execule the action from the VLM, but train an RL policy to map this decoded action to a better one. Note that if the VLM already decodes good action texts, simply copying over this action via RL should be casy. (c) Methods for efficient RL from pixels via model-based approaches. We choose Dreamer v3, since it has proven to be successful at learn- ing Minecraft tasks from scratch (Hafner et al. 2023).
(d) Methods leveraging pretrained representations specifically useful for embodied control, though which are non con-prompttble and non- Mon-Miierraft specific. We choose VC-1 and R3M (Majumdar et al. 2023 Nair et al. 2022). (c) Methods using models pre-trained on large-scale Minecraft data. These serve as "oracle" " comparisons, as these representations are explicitly fine-tuned on Minccraft YouTube videos, whereas our pre-trained VLM is both frozen and not trained on any Minecraft video data. We choose MineCLIP, VPT, and STEVE-1 1 as our sources of Minecraft-specific representations (ann et al. 2022 Baker et al. 2022 Lifshitz et al. 2023).
We use PPO (Schulman et al. 2017) as our base RL algorithm for all non-Dreamer Minecraft policies. We also note that we do not compare against non-RL methods, such as Voyager (which uses LLMs to write high-level code skills, abstracting away low-level control to hand-written APIs that use oracle information). See Appendix B.2 for training details and E.1 for further discussion of such non-learned systems.
4.2 Domain 2: Habitat
A major advantage of VLMs pre-trained on Internet-scale data is their reasoning and generalization capa- bilities. To evaluate this, we run offline BC and RL experiments in the Habitat household simulator. In contrast to Minecraft, tasks in this domain require connecting naturalistic images with real-world common sense about the structure and contents of typical home environments. Our experiments evaluate (1) whether PR2L confers the generalization properties of VLMs to our policies, (2) whether PR2L-based policies can leverage the scmantic reasoning capabilitics of the underlying VLM (c.g., via thain-of-thought Wei et al.
Fan al. 2022 also consider hont cow/sheep. However, we omit them as we were unable to replicate their results those lasks; all approaches failed to learn them.
7

---

Published in Transactions on Machine Learning Research (03/2025)
PR21. Prompt
RT-2-style Baseline Prompt
Change Auxiliary Text Ablation Prompt
Spiders Minecralt black
want. to fighi spider. can stlack. move, or turn. What should I do? waut to milk a cow, I can use my bucket, move, turn. What should I do? want to shear a sheep. I can use my sheurs. What should do?
Combat Spider
Is there a spider in this image?
Is there spider in this image?
Cows in Minecraft are black and white
Milk Cow
Is there now in this image?
Is there cow this image?
Sheep in Minccraft are usullly white.
Shear Sheep
Is there sheep in this image?
move
Is there shoep this image?
I want to fight a [larget entäty.. I can attack, move, or turn. What should do!
Other Combal Tasis
Is there [target entity] this image!
Table 1:
Prompts used Minecraft for querying the VLM with PR2T, comparison (b), and the change auxillary text ablation. For the last column, we remove the auxiliary text for combat spider, and add it in for the other two.
2023 and (3) whether PR2T, can learn entirely from stale, offline data sources. We use Llama2-7B Prismatic VLM for the Habitat experiments Karamcheti et al. (2024).
Habitat tasks. We consider the ObjectNay task suite in 3D scanned household scenes from the HM3D dataset (Savva et al. 2019; Yadav et al.) 2023a; Ramakrishnan et al.] 2021). ). These tasks involve a simulated robot traversing home environment
find an instance of specified object (toilet, bed, sofa, television, plant, chair) in the shortest path possible. The full benchmark consists of 80 houschold scenes intended to train the agent and 20 for validation. We change the observation space to consist of just. RGB vision, previous action, posc, and target object class, omitting depth images to ensure that observed performance differences come from the quality of promptable representations vs. umpromptable ones. Like with MineDojo, these tasks are long horizon, taking 80 steps for a privileged shortest path follower to succeed and 150+ for humans. See Figure 3 for example observations and Appendix C for more details.
Comparisons. To see if PR2L can leverage VLM reasoning capabilities, we train two PR2L policies, one with and one without thain-of-thought prompting (see Section 4.3 We also train a policy Prismatic VLM image encoder embeddings (equivalent to Minecraft approach (a), but with Dino+SSgILIP Caron et al. [2021 ps collected with Habitat-Was Ramrakhya et al.
Zhai et al. (2023) on a human demonstration datasct collected from the ObjectNav training scenes
(022) and used by past works on large-scale BC on pre- trained visual representations Ramrakhya et al.
2023 Yadav et al. (2023b); Majumdar et al. (2023 As it previously achieved stato-of-tho-art performance among those works, we also compare against two policies using VC-1 as an encoder MMjuundar et al., 2023), either using just its summarizing CLS token or using learned Transformer layer to condense its patch embeddings. We adopt the same LSTM-based recurrent architecture used by that work, but replace the image embeddings with a learned Transformer layer that condenses our input token embeddings (from the VLM, VLM image encoder, or VC-1) into single summary embedding, as done with Minecraft.
Due to computational constraints, we train all policies on just under a tenth of the full dataset of 77k trajectories/ 12M steps. In contrast, other works using this dataset train on the entire dataset. Nevertheless, we evaluate on the unseen validation scenes, thereby testing how well PR2L generalizes.
4.3 Designing Task-Specific Prompts for Minecraft and Habitat
We now discuss how to design prompts for PR2L. As noted in Section 3.3 these are not instructions or task descriptions, but prompts that force the VLM to encode semantic information useful for the task in its representation. The simplest relevant feature for our Minecraft tasks is the presence the target entity in an observation. Thus, we choose "Is there a [target entity] in this image?" as the base of our chosen prompt. We also pick two alternate prompts per task that prepend different amounts of auxiliary information about the target entity. E.g., for combut spider, one candidate is "Spiders in Minecraft are black." To choose between these candidates, we mcasure how well the VLM is able to decode a correct answer to the prompt question of whether or not the larget entity is present in the image on a small annolated dataset. Full details of this prompt evaluation scheme for the first three Minecraft tasks are presented in Appendix [A] and Table 5 We find that auxiliary text only helps with detecting spiders while systematically and significantly degrading the detection of sheep and cows. Our ablations show that this detection success rate metric correlates with performance of the RL policy. Additionally, the prompts used for comparison (b) follow the prompt structure
8

---

Published in Transactions on Machine Learning Research (03/2025)
Bueeliues Dreamer
Oraclee
Task
PR2L (Ours)
VLM Image Encoder
Ry-l-slyle
VC-1
R3M
MineCLIP
VPT
STEVE-
Conobet Späder
97.6
14.9
31.2
71.5
9.7
5.4 +
72:2
72.8 +: 8.7
176.9 = 19.8
197. = 19.8
88.8
14.0
Milk Cou
223.4
35.4
95.2 1 18.7
125.6 28.9
21.0
2:0.2:12
96.8 1 16.3
100.0 4: 11.1
191.4 1: 83.3
85.5 4: 14.5
75.2
15.4
Shoar Shorp
37.0
4.4
23.0
26.2 322
20.9 1:2
26.5
17.6
23.1 3.7
24.1
18.2
2.2
Conbatt Zazobie
24.6
1.6
14.8 2.0
18.2 2.1
1.8 + 0.2
5.6
+ 1.0
5.8 1.4
56.6 + 8.8
91.2
S.8
23.6
3.4
Enderman
52.2
72.1
14-4
19.2
59.3
Combal Pigman
46.4
3.3
36.5 3.7
35.1
2 2.5
5.8
4:
1.5
33.7 1: 4.9
31.4
1: 4.2
182.0
7.9
169.0 2: 7.8
98.3
8.4
Table 2: Performance of PR2L, baseline, and oracle approaches in Minecraft tasks. Values reported are IQM successes and standard errors. PR2T. universally outperforms all baselines. As they are trained on Minecraft-specift data, the oracles outperform PR2L in half the comparisons (italicized).
PR21 (Ours)
VC-1
CLS
VC-1 + Patch Embeds
Task (# Episodes)
VLM Image Encoder
With CoT
Without CoT
40 Epochs
120 Epochs
40 Epochs
120 Epochs
A Average (2000)
41.9%
27.8%
11.6%
6.8%
8.9%
13.6%
15.8%
Toilet (398)
37.2%
22.9%
8.8%
2.8%
2.0%
7.0%
9.3%
Bed (433)
45.0%
28.9%
12.9%
6.7%
9.9%
14.8%
19.2%
Sofa (376)
48.1% 1
34.3%
11.7%
9.8%
14.4%
17.0%
19.4%
Chair (438)
51.2%
40.9%
17.5%
11.7%
15.0%
22.4%
23.8%
Television (281)
26.7%
10.3%
5.0%
2.8%
3.2%
4.6%
4.6%
Plant (84)
23.8%
8.3%
9.1%
1.2%
1.2%
9.5%
9.5%
Table 3: Performance of PR2L and baselines Habitat ObjectNav tasks. Following prior works, values reported validation scenes. PR2L (with without CoT) does better than all other approaches. PR2L with CoT does the best, universally achieving more than double the performance of all non-PR2L approaches and 7% higher average performance than PR2L without CoT. Note that PR2T. and image encoder palicies trained for 40 epochs, but VC-I palicies' performance saturated 120, we report. their performance both times.
average success rates
prescribed by Brohan et al. (2023a), which motivated this comparison. In these prompts, also provide list of actions that the VLM can choose from to the policy. All chosen prompls are presented in Table
For Habitat, we choose the prompt "Would [target object] be found here? Why or why not?" As opposed to the Minccraft prompts, this does not just identify the presence of a target object in the image, but draws on general knowledge from the VLM to determine if the observed location would contain the target object, even if said object is not in view. The second part of the prompt then leads the VLM provide chain of thought (CoT) (Wei et al., 2023) rationale for its final answer. This CoT draws out task-relevant VLM world knowledge by explicitly reasoning about visual semantic concepts, that useful to learning policy (see Table 4). We note such raasningssraaionnlles are akin to the "embodied" features that aid in picking good robot actions (as shown by Zawalski et al. 2024)), as opposed to "traditional" CoTs that breaks down a problem into simpler reasoning steps. To investigate if PR2L enables embodied agents to benefit from these VLM common-senss reasoning capabilities (even they do not directly reason about actions), we train PR2L policics both with and without the sccond part of the prompt.
5 Results
Minecraft results. We report the interquartile mean (IQM) and standard error number of successes over 16 seeds for all Minecraft tasks in Table 2 PR2L uniformly outperforms the non-oracle approaches (a) using non-promptable image embeddings, (b) directly asking the VLM for actions, (c) learning from scratch Drcamer, and (d) using conppromptable control-spocific embeddings.
PR2L outperforms (a) the VLM image encoder baseline, even though both approaches receive the same visual features, with PR2L simply transforming those features via prompting LLM (with no additional information from the environment), thus supporting that prompting does shape representations in a beneficial way for learning control lasks. We provide an analysis of why PR2L states are better than (b) R2T-2-style ones in Appendix[H.] We observe that PR2L embeddings are bimodally distributed, with transitions leading to high reward clustered at one mode. This structure likely enables more efficient learning, thereby showing how control tasks can benefit from extracting prior knowledge encoded in VLMs by prompting them with task context, even when the VLM does not know how to act. For (c) the model-based comparisons, we find
9

---

Published in Transactions on Machine Learning Research (03/2025)
that Dreamer is not as conducive at learning our Minecraft tasks. We hypothesize this is because our tasks are comparatively shorter than the ones considered by Hafner et al. (2023), so learning model is less beneficial (while PR2L provides immediately-useful representations). Additionally, we note that all our approaches involve interacting with partially-observable, non-stationary entitics, which the Dreamer model may have a hard time learning. See Appendix E.2 for further discussion. Finally, (e) the oracles outperform PR2T, in combat padderman/pigaan all but STEVE-1 do better in combat ppider/zombirn and none do better shear theep/mmily cow. We hypothesize this because endermen and pigmen are Ninecraft-specific entities, giving rise to comparatively poor representations in the VLM (which is trained exclusively on natural images). In contrast, Minecraft combies/spidens are heavily stylized, but still somewhat resemble other depictions of such creatures, while Minecraft cows and sheep are the closest to their naturalistic counterparts, making PR2L more effective. Even though our VLM is not trained on Minecraft data, its representations yield better policies in half the oracle comparisons.
We provide ablations in Table g and Appendix F We find that (1) PR2L performs worse when is un- prompled or does not decode text. This suggests that the representations that are good to learn from are elicited specifically by answering prompt the VLM representations from generating unprompted text or from simply using the VLM as an image and prompt encoder (without generation) are not as effective (though we note they nonetheless outperform the image encoder baseline). (2) Our prompt evaluation scheme successfully identified cases where auxiliary text improves/ /degrades performance, indicating that "ability to adequately answer a prompt" is a promising proxy for quality of representations for RL. (3) A policy with oracle entity detection does worse than PR2L, suggesting our prompt is not just eliciting that feature from the VL...
Habitat results. Following prior works, we report success rates the ObjectNav validation episodes in Table 3 PR2L with CoT outperforms all other policies on all tasks, including an almost 4x performance increase over the VLM image encoder baselines- again, suggesting that using promptable representations for control improves over the base purely-visual embeddings. While PR2L without CoT still does better than all baselines, we find CoT prompting improves policy performance (by 1.5x, from 27.8% success rale to 41.9%), likely because provides the policy with useful generalizable features: e.g., even if the agent comes across unfamiliar room while scarching for a toilet, it still knows to look elsewhere if the VLM reasons that, due to the presence of a bed, the room is likely a bedroom (which is unlikely to contain toilets). Thus, even if the VLM cannot reason about actions, our resulls indicale that PR2T, provides promising way of using ils ability to reason about image semantics and common sense for control. See Table 4 for CoT examples.
While we do not beat VC-1's reported SOTA BC performance (60.3% success rate when VC-1 is frozen Majumdar et al. (2023)) ), we note that said performance is achieved with (1) over ten times more training data and gradient steps and (2) image augmentations to prevent overfitting. Our VC-1 policies were trained the same amount of data our PR2L agent and for 1-3x as many gradient steps, but perform far worse, suggesting that PR2L is significantly more sample- and compute-effiiiet than VC-1 policies. Additionally, PR2L does not use any explicit countermeasures to overfitting, yet still generalizes well to unseen ObjectNav scenes (aided by the VLM's representations of reasoning).
Finally, we analyze policies trained with offline RL in a simplified Habitat setting in Appendices D where we find that VLM representations align well with the returns of an optimal policy.
6 Conclusion
We propose Promptable Representations for Reinforcement Learning, a method for extracting semantic features from images by prompting VLMs with task context to leverage their extensive general-purpose prior knowledge. We demonstrate PR2L in Minecraft and Habitat, domains that benefit from interpreting obscrvations in terms of semantic concepts that can be related to task context. This framework for using VLMs for control opens new directions. For example, other types of foundation models pre-trained with more sophisticated methods could also be used for PR2L: e.g., ones trained on physical interactions might yield features which encode physics or action knowledge, rather than just common-sense visual semantics. Developing and using such models with PR2L offers an exciting way to transfer diverse prior knowledge broad range of control applications.
10

---

Published in Transactions on Machine Learning Research (03/2025)
Target
Example VLM Responses to "Would [target] be found here? Why or why not?"
"No, toilet is not found in this room as it is a bedroom. Toilets are typically found in bathrooms
Toilel
The image shows large bed with a wooden headboard, nightstands, and lamps
Yes, a toilet would be found in this room because it is a bathrom.
The presence of toilet in the imange is expected, as bathros are designed to accommodate toilets for personal bygiene
"No, bed would not be found in this room because it is a dising zoom.
Bed
The presence of a dining table chaits suggests that this room is designed eattng meals and socializing, rather than sloeping"
"Yes, a had would he found brewnse room deseribed as bodroom.
bed television, lamp, and a dresser-
which are commin Burnishings in bedroom
'No, sola is not found here because a kitchen area with appliinces and cabiuets.
Sofa
Solas are typically found in living rooms or other common areas
Yes, there is black leather sofa the living room. It has a red pillow on It. It larpe sectioonal couch
Table 4: Example VLM responses the Habitat prompt for various images. Beyond just detecting the target, prompting
the VLM for CoT elicits relevant common sense, which semantically relates other useful visual features. By using the underlying VLM embeddings as state representation, the policy thus integrates the VLM's knowledge and reasoning into
docision-making,
A limilation of PR2L is that prompts are currently hand-crafted based on the user's conception of useful task features. While coming up with good prompts for our tasks was not hard, the process of evaluating and improving them could be automated - some possibilities that we leave to future works include adopting prompt-optimization techniques used for LLM reasoning, such as using differentiable soft-prompts or meta- level RT over prompts (Wen et al. 2023: Deng et al.] 2022).
We also find that the quality of representations largely depends on the VLM -LM-c.g, c.g., InstructBLIP could not reason well about Habitat scenes, but the more recent Prismatic VLMs are more capable in that regard, enabling our CoT experiments. Thus, as VL.M capabilities are expected to increase, we expect the quality of their representations for PR2L to also improve, but also require more sophisticated methods for evaluating which VLM's reprosentations are most suited to policy learning. While methods in self-supervised repre- sentation learning do this by evaluating representation information content or effective rank (Thilak et al., 2023 Garrido et al. 2023), other options would be to consider features that are specifically good for RL, e.g.; the degree to which representations of different states are "collapsed," " that has been shown to hinder TD-learning (Kumar et al. 2021).
Lastly, the size and speed of VLMs can limit their applicability. Our policics typically achieve 3-5 Hz inference speeds, comparable to those of robot policies built on large models Brohan et al. (2023b a); O.M.T.
et al. 2023). Likewise, our VLM sizes are comparable to models used for policies in prior works (Brohan 2023a Szot et al. 2024p. Additionally, due using off off-the-shelf VLMs, approaches that speed up standard VLM inference also apply here (c.g., Zawalski et al. (2024) shows that VLM policies can be sped up by compiling the VLM).. While inference speeds may hinder online policy learning, we find that offline approaches (which can parallelize training and data generation) used for Habitat help remedy this.
et al.
References
Ademi Adeniji, Amber Xie, Carmelo Sferrazza, Younggyo Seo, Stephen James, and Pieter Abbeel. Language reward modulation for pretraining reinforcement learning, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea. Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmilry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do can and not as say: Grounding language in robotic affordances. 2022.
11

---

Published in Transactions on Machine Learning Research (03/2025)
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022.
Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey, 2019.
Jessica Borja-Diaz, Oier Mees, Gabriel Kalweit, Lukas Hermann, Joschka Boedecker, and Wolfram Burgard. Affordance learning from play for sample-efficien policy learning, In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, USA, 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang- Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-languag--action models transfer web knowledge to robotic control, 2023a.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Docksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vnong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Adilya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Aduances in Neural Information Processing Systems, volume 33, pp. 1877-1901 Cur- ran Associates, Inc., 2020. URL https: http::///ccceeeg...eeeee neurips. wwwpsps.ccppcppppprrrrrrr.. http:accpppplpepee/eaaaaaaaaeeee /ilees/papeer/222//iiie 0457 htt.r444418bba114222444 1att:eacc1264aaaaaaaaaaaaaa Paper. pdf
Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, and Rogerio Bonatti. Latte: Language trajectory transformer, 2022.
Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, Emerging properties in self-supervised vision transformers, 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL httpp:: http::://mmss...oo hrtpm/ss..rg/bog//// 2023-03-30-vicuna//
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression, 2017.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Lá, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
Mingkai Deng, Jianyu Wang, theng-Ping, Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep indirectional transformers for language understanding, 2019.
Ziluo Ding, Hao Luo, Ke Li, Junpeng Yue, Tiejun Huang, and Zongqing Lu. vision-language model for minecraft, 2023.
Clip4mc: An rl-friendly
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models, 2023.
Kiana Ehsani, Tanmay Gupta, Rose Hendrix, Jordi Salvador, Luca Weihs, Kuo-Hao Zeng, Kunal Pratap Singh, Yejin Kim, Winson Han, Alvaro Herrasti, Ranjay Krishna, Dustin Schwenk, Eli VanderBilt, and Aniruddha Kembhavi. Imitating shortest paths in simulation enables effective navigation and manipulation in the real world, 2023.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De- An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Neural Information Processing Systems, 2022, 2022.
Quentin Garrido, Randall Balestricro, Laurent Najman, and Yann Lecun. Rankme: Assessing the down-
stream performance of pretrained self-supervised representations by their rank, 2023. URL https:
//axxiv.og// wttgg//ab////...oo 02885
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2023.
Jennifer Hu and Roger Levy. Prompt-based methods may underestimate large language models' linguistic generalizations, 2023.
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), London, UK, 2023.
Wenlong Huang, Picter Abbeel, Decpak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022b.
Ganesh Jawahar, Benott Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019. Association for Computational Linguistics.
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, mar 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac HadffelddDoods, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Gan guli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravce, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Junyon Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, Weijun Hong, Zhongyue Huang, Haicheng Chen, Guangjun Zeng, Yue Lin, Vincent Micheli, Eloi Alonso, François Fleuret, Alexander Nikulin, Yury Belousov, Oleg Svid- chenko, and Alcksei Shpilman. Minerl diamond 2021 competition: Overviow, results, and lessons learned, 2022.
Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for roboties, 2023. URL https://arxii. httpp////aaaiv.or//aa/////// wwwp:or/////b/22222/2 12766
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditionee language models, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehoad, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning, 2020.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit mnderppaaraattteiizttonn inhibits data-efficieul deep reinforcement learning, 2021. URL httppe///arrvv.oor///////000...0
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-base permutation-invarian neural networks, 2019. URL
https: https://arxiv.. attpaaawi....aaaaaaaa/a
00825
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping Aanguag--mmag pre-training with frozen image encoders and large language models, 2023a.
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg, Emergent world representations: Exploring a sequence model trained on a synthetic task, 2023b.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023.
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith. Steve-1: A generative model for text-to-behavior in minecraft, 2023.
Haowei Lin, Zihao Wang, Jianzhu Ma, and Yitao Liang. Meu: A task-centric framework for open-ended agent evaluation in minecraft, 2023a.
Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language. 2023b.
Hao Liu, Lisa Lee, Kimin Lee, and Pieter Abbeel. Instructioo-ffllowinn agents with multimodal transformer, 2023.
Haokuan Luo, Albert Yuc, Zhang- Wei Hong, and Pulkit Agrawal. Stubborn: A strong bascline for indoor object navigation, 2022.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data, 2021.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence?, 2023.
Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
Vivek Myers, Andre Hc, Kuan Fang, Homer Walke, Philippe Hanscn-Estruch, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. Goal representations for instruction fol- lowing: A semi-supervised language interface to control, 2023.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation, 2022.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning, 2018.
Kolby Nottingham, Prithviraj Ammanabrohu, Alanc Suhr, Yejin Choi, Hannanch Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling, 2023.
O.M.T., Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. /attps://octo moteooomooels...... mooeets--mmmleessghhhhhue github. 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkinn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller. Towards unified agent with foundation models, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanorvisto, Maximilian Ernestus, and Noah Dormann. Stable-baseliness: Reliable reinforcement learning impleementations. Journal of Machine Learning Re- search, 22(268):1 8, 2021. URL http://jmlr httg//apers///22222133...
Santhosh K. Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X. Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai, 2021.
Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale, 2022.
Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Abhishek Das. Pirlnav: Pretraining with imitation and rl finetuning for objectnav, 2023.
Robin Rombach, Andreas Blattmann, Dominik Torenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
15

---

Published in Transactions on Machine Learning Research (03/2025)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms, 2017.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language, 2022.
Pralyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Tor- ralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. In Robotics: Science and Systems, 2022, 2023.
Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Melhods in Naturul Language Processing, pp. 1526 1534, November 2016.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Na- talic Mackraz, Devon Hjelm, and Alexander Toshev. Large language models as generalizable policies for embodied tasks, 2024.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline, 2019.
Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint embedding ssl archi- tectures, 2023. URL https:://axxv..rrrr www.grg//////r//er.ooooeeeeeee/e
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilitics. Technical report, Microsoft, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodicd agent with large language models, 2023a.
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang, Describe, explain, plan and select: Interactive planning with large language models enables open-workee apen-world multi-task agents, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Gold- stein. Hard prompts made easy: Gradient- based discrete optimization for prompt tuning and discovery. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 51008 51025. Curran Associates, Inc., 2023. URL https://proccedng...nrree nwwriss..errrr.ccccoo... http://w.........ooooooo/oooa//a pftp////////ppe////////eee/ 400-4801864/00022227777222211111
Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. Does bert make any sense? inter- pretable word sense disambiguation with contextualized embeddings, 2019.
Karmesh Yadav, Jacob Krantz, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Jimmy Yang, Austin Wang, John Turner, Aaron Gokaslan, Vincent-Pierre Berges, Roozbeh Mootaghi, Oleksandr Maksymets, Angel X Chang, Manolis Savva, Alexander Clegg, Devendra. Singh Chaplot, and Dhruv Batra. Habitat challenge 2023. https://aihabitan httpa//iatattt.reg/caaa///////// 2023a.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Olek- sandr Maksymets, and Dhruv Batra. Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav, 2023b.
Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4me: Skill reinforcement learning and planning for open-world minecraft tasks, 2023.
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv preprint orXiv:2407.08693, 2024.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Arm- strong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johmny Lee. Transporter nelworks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratie models: Composing zero-shot multimodal reasoning with language, 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023.
Bohan Zhou, Ke Li, Jiechuan Jiang, and Zongqing Lu. Learning from visual observation via offline pretrained stale-lo-go transformer, 2023.
Minzhao Zhu, Yifeng Li, and Tao Kong. Integrating map-based method with end-to-end learning, 2022.
URL
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for worn-world environments via large language models with text-based knowledge and memory, 2023.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Target Entity
Prompt
True Positive Rate
True Negative Rate
"Is there spider in this image?"
22.27%
100.00%
Spider
"Spiders in Minecraft are black.
73.42%
94.54%
Is there a spider in this image?"
"Spiders in Minecraft are black
and have red eyes and long, thin
50.50%
99.85%
legs. Is there spider in this image?"
"Is there a now this image?'
71.00%
45,41%
Cow
"Cows in Minecraft are black and white. Is there a cow in this image?"
98.22%
2.00%
"Cows in Minecraft are black and white and have four legs.
96.67%
7.35%
Ts there in this image?"
there a sheep this image?'
88.00%
59.83%
Sheep
"Sheep in Minecraft are white. Is there a sheep in this image?"
100.00%
0.00%
"Sheep in Minecraft have four legs.
white and
100.00%
0.00%
Is there sheep in this image?"
Table 5: InstructBLIP's performance at decoding text indicating that it detected the presence of a target entity when given different prompts. We use this as proxy metric for prompl engineering for RL, allowing us to determine which prompt to use for PR2L.
A Prompt Evaluation for RL in Minecraft
We discuss how to evaluate prompts to use with PR2L, by showcasing an example for a Minecraft task. We start by noting that the presence and relative location of the entity of interest. for each task (i.e., spiders, sheep, or cows) are good features for the policy to have. To evaluate if a prompt elicits these features from the VLM, we collect a small dataset of videos in which each Minecraft entity of interest is on the left, right, middle, or not on screen for the entirety of the clip. Each video is collected by a human player screen recording visual observations from Minecraft of the entity from different angles for around 30 seconds at 30 frames per second (with the exception of the video where the entity is not prosent, which is minute long).
We propose prompts that target each of the two features we labeled. First, we evaluate prompts that ask "Is there a(n) [entity] in this image?" As the answers to these questions are just yes/no, we see how well the VLM can directly generate the correct answer for cach frame in the collected vidcos. The VLM should answer "yes" for frames the three videos where the larget entity is on the left, right, or middle of the screen and "no" for the final video. Second, we evaluate if our prompts can extract the entity's relative position (left, right, middle) in the videos where it is present. We note that the prompts we tried could not extract this feature in the decoded text (e.g., asking "Is the [entity] on the left, right, or middle of the screen?" will always cause the VL.M to decode the same text). Thus, we try to see if this feature can be extracted from the decoded texts' representations. We measure this by fitting a three-category linear classifier of the entity's position given the token- wise mean of the decoded tokens' final embeddings. This is an unsophisticated and unexpressive classifier, i.e., we do not have to worry about the model potentially memorizing the data, which mcans that good classification performance corresponds to an casy extractability of said feature.
We evaluate three types of prompts per task entity for the first feature: one simply asking if the entity is present in the image (e.g., "Is there spider in this image?") and two others adding varying amounts of auxiliary information about visual characteristies of the entity (c.g., "Spiders in Minccraft are black. Is there a spider in this image?" and "Spiders in Minecraft are black and have red eyes and long, thin legs. Is there a spider in this image?"). We present evaluations of all such prompts in Table 5 We find that the VLM benefits greatly from auxiliary information for the spider case only, likely because spiders in Minecraft are the most dissimilar the ones present in natural images of real spiders, whereas cows and sheep are still comparatively similar, especially in terms of scale and color. However, adding too much auxiliary information degrades
18

---

Published in Transactions on Machine Learning Research (03/2025)
Tasks in MineDojo
Combat Spider
Milk Cow
Shear Sheep
Tasks in Habitat
ObjectNav: Find [toilet sofa bed].
Figure 3: Example tasks, observations, and task-relevant prompts from MineDojo and Habitat.
performance, perhaps because the input prompt becomes too long, and therefore is out-of-distribution for the types of prompts that. the VLM was pre-trained on. This same argument may explain why auxiliary information degrades performance for the other two target entities as well, causing them to almost always answer that said entities are present, even when they are not. Once more, considering that these targets exhibit higher degree of visual resemblance to to their real counterparts compared to Minecraft spiders, it is reasonable to infer that the VLM would not benefit from auxiliary information. Furthermore, taking into account that the auxiliary information we gave is more common-sense than the information given for the spider, it could imply that the prompls are also more likely to be ont-of-distribution (given that "sheep are white" is so obvious that people would not bother expressing it in language), causing the systematic performance degradation.
For the probing evaluation, find that all three prompts reach similar final linear classifiabilittes for each of their target entities, shown in Figure 4 While this does not aid in choosing one prompt over another, it does confirm that the VLM's decoded embeddings for cach prompt still contains this valuable and gramular position information about the target entity, even though the inpul prompl did not ask for il.
B MineDojo Details
B.1 Environment Details
Spaces. The observation space for the Minecraft tasks consists of the following:
RGB:
Egocentric RGB images from the agent. (160, 256, 3)-size tensor of integers E (0, 1, 25.)).
2. Position: Cartesian coordinates of agent in world frame. 3-clement vector of floats.
Pitch, Yaw: Orientation of agent in world frame in degrees. Note that we limit the pitch to 15" above the horizon to 75 below for combat spider, which makes learning casier (as the agent otherwise often spends a significant amount of time looking straight up or down). Two 1-clement vectors of floats.
19

---

Published in Transactions on Machine Learning Research (03/2025)
4. Previous Action: The previous action taken by the agent. Set to operation at the start of each episode. One-hot vector of size |A| 53 for combat spider and 89 otherwise (see below).
This differs from the simplified observation space used in Fan et al. (0022) in that we do not use any nearby voxel label information and impose pitch limits for combat spider. This observation space is the same for all Minccraft experiments.
The action space is discrete, consisting of 53 or 89 different actions:
Lincar Glassifter Accuracy of Relative Position of Spidor
1.0
1. Turn:
Change the yaw and pitch of the agent. The yaw and pitch can be changed up to +900 in multiples of 15 As they can both be changed at the same time, there are 9 x 9 81 total different turning ac- tions. The turning action where the yaw and pitch changes are both 0 is the no op- eration action. Note that, since impose pitch limits for the spider task, we also limit the change in pitch to +300, meaning there are only 45 turning actions in that case.
Cisoricston accursstonncuusss Accursey
0.8
0.6
0.4
0.2
Solders Ninecraft black there spis this Image?
Spaders Minocraft are black and have red eyes and long, thin logs.
spider image?
0.0
Uneer Classtter Acturacy Relative Fostiion
1.0
cissiiicston accurscy
0.8
2. Move: Move forward, backward, left, right, jump up, or jump forward for actions to- tal.
0.4
Is there cow this Image?
Minecrat: black white. there this there this image?
3. Attack: Swing the held item at whatever targeted at the center of the agent's view.
four legs.
Uneer Classiteer Accuracy of Relwtive Position of Sheep
4. Use Item: Use the held item on what- ever is targeted at the center of the agent's 's view. This is used to milk cows or shear sheep (with an empty bucket or shears re- spectively). If holding sword and shield,
Ciazmiicatonnccc Acturscy
0.6
0.4
this action will block attacks with the latter.
Is there a shoep this Image?
Sheep in Ninecraft are white, is there a sheep this image?
Sheep Kinscraft white and have four legs. is there a sheep in this image?
This non- con-combat spider action space is the same as the simplified one in Fan et al. (2022). All experi- ments for a given task share the same action space.
0.0
100
200
300
400
500
Epochs
World specifications. MineDojo implements fast reset functionality that use. Instead of gen- erating entirely new world for each episode, fast reset simply respawns the player and all specified en- tities in the same world instance, but with fully re- stored items, health points, and other relevant task quantities. This lowers the time overhead of re- sets significantly, but also mcans that some changes to the world (like block destruction) are persistent. However, breaking blocks generally takes multi- ple time steps of taking the same action (and does not directly lead to any reward), the agent empirically does not break many blocks aside from tall grass (which is destroyed with a single strike from any held item). We keep all reset parameters (like the agent respawn radius, how far away entities can spawn from the agent, etc) at their default values provided by MineDojo.
We train a linear classifier to predict the relative position of the target entity feft//right/miidll based on the av- erage VLM embeddings decoded in response to each associated candidate prompt. We find that all three candidate prompts per task elicit embeddings that are similarly highly conducive to this classification scheme.
Figure 4:
We stage all tasks in the same area of the same sateraammtticlllyygeeeeaa world: namely, a sunflower plains biome in the world with seed 123. This is the default location for the implementation of the spider
20

---

Published in Transactions on Machine Learning Research (03/2025)
Task Combot Zombie
Hyperparameter
Combot Spider
Milk Cow
Shear Sheep
Combot Enderman
Combot Pigman
Total Train Steps
150000
100000
Rollout Steps
2048
Action Entropy Coefficient
5e-3
Value Function Coefficient
0.5
Max LR
5o-5
1e4
1o44
5e-5
l044
5e-5
Min LR
5e-6
le-4
le-d
5e-6
le-4
5e-6
Batch Size
64
Update Epochs
10
0.99
GAE A
0.95
Clip Range
0.2
Max Gradient Norm
0.5
Normalize Advantage
True
Table 6: PPO hyperparameters for Minecraft tasks, shared by the baselines, our method, and ablations.
combat task MineDojo. We choose this specific world/location as it represents a prototypical Minecraft scene with relatively casily-traversable terrain (thus making learning faster and casier).
Additional task details and reward functions. We provide additional notes about our Minecraft tasks.
Combat spider: Upon detecting the agent, the spider approaches and attacks; if the agent's health is depleted, then the episode terminates in failure. The agent receives +1 reward for striking any entity and +10 for defeating the spider. We also include several distractor animals (a cow, pig, chicken, and sheep) that passively wander the task space; the agent can reward game by striking these animals, making credit assignment of success rewards and the overall task harder.
Milk cow: The agent also holds wheat in its off hand, which causes the cow to approach the agent when detected and sufficiently nearby. For each episode, track the minimum visually-observed distance between the agent and the cow at each time step. The agent receives +0.1 Admial reward for decreasing this minimum distance (where Admin < 0 the change in this minimum distance at a given time step) and +10 for successfully milking the cow.
Shear sheep: As with milk cow, the agent holds wheat in off hand to cause the sheep to approach it. The reward function also has the same structure as that task, albeit with different coefficients: +|Admin for decreasing the minimum distance to the sheep and +10 for shearing it.
Combal zombie: Same combal spider, but the enemy is a zombie. We increase the episode length to 1000, as the zombie has more health points than the spider.
Combat enderman: Same as combat spider, but the enemy is an Enderman. As with combat zombic, we increase the episode length to 1000. Note that Endermen are non-hostile (until directly looked at for sufficiently long or attacked) and have significantly more health points than other enemies. We thus enchant the agent's sword to deal more damage and decrease the initial spawn distance of the enderman from the agent.
Combat pigman: Same as combat spider, but the enemy is a hostile zombie pigman. As with combat zombie, we increase the episode length to 1000.
B.2 Policy and Training Details
For our actual RL algorithm, we use the Stablo-Baselines? (version 2.0.0) implementation of clipping- based PPO (Raffin et. al.) 2021), with hyperparameters presented Table G Many of these parameters are the same as the ones presented by Fan et al. (2022). For the spider trials, we use cosine learning rate schedule:
+ cas
eurrent train step
total train steps
train step) = Min LR + -Max LR Min LR)
(1)
2
21

---

Published in Transactions on Machine Learning Research (03/2025)
Policy Transformer Hyperparameters
Transformer Token Size
512
128
Transformer Foodforward Dim
512 128
Transformer Number Heads
2
Transformer Number Decoder Layers 1
Transformer Number Encoder Layers
Transformer Output Dim
Transformer Dropout
0.1
Transformer Nonlinearity
ReLU
Policy MLP Hyperparameters
Number Hidden Layers
Hidden Layer Size
128
Activation Function
tanh
VLM Generation Hyperparameters
Max Tokens Generated
Min Tokens Generated
Decoding Scheme
Greedy
Table 7: All policy hyperparameters for all Minecraft tasks. Smaller token sizes and feedforward dimensions are used for combat (zombie/ weteeeeeeennninmmnneeeeeee
We also present the policy and VLM hyperparameters in Table The hyperparaamtters and architecture of the MLP part of the policy are primarily defined by the default values and structure defined by the Stable- Baselinsss ActorGriticPolicy class. Note that the generation ablation, VLM image encoder bascline, and MineCLIP trials do not generate text with the VLM, and so all do not use the associated process's 's hyperparaneterrs The MineCLIP trials also do not use Transformer layer in the policy, due to not receiving token sequence embeddings. It instead just uses a MLP, but with two hidden layers (to supplement the lowerod policy expressivity due the lack of Transformer layer).
Additionally, InstructBI.IP's token embeddings larger than ViT-g/14 11's (used in the VLM image en- coder baseline), and may carry more information. However, the VLM does not receive any privileged information over the image encoder from the lask environment any additional information in the VLM's representations is therefore purely from the model's prompted internal knowledge. Still, to ensure consistent policy expressivity, we include a learned linear layer projecting all representations for this baseline and our approach to the same size (512 dimensions) so that the rest of the policy is the same for both.
Minecraft training runs were run on 16 A5000 GPUs (to accommodate the 16 seeds).
C Habitat ObjectNav Details
C.1 Environment Details
The spaces and agent/task specifications are largely the same the defaults provided by Habitat, as specified in the HM3D ObjectNav configuration file (Savva et al., 2019).
Spaces. The observation space for Habitat consists of the following:
1. RGB:
Egocentric RGB images from the agent. (480, 640, 3)-size tensor of integers E 10,1.... 2555. By default, agents also receive depth images, but we remove them to ensure that state representations are grounded primarily in visual observations.
2. Position:
Horizontal Cartesian coordinates of agent. 2-element vector of floats.
3. Compass: Yaw of the agent. Single floats.
22

---

Published in Transactions on Machine Learning Research (03/2025)
4. Previous Action: The previous action taken by the agent. Set to operation at the start of each episode. One-hot vector of size |A]] 4.
Object Goal: Which object the agent is aiming to find. One-hot vector of size 3.
The action space is the standard Habitat-Lab action space, though we remove the pitch-changing actions, leaving only four:
Turn: Turn left or right, changing the yaw by 30%.
2. Move Forward: Move forward fixed amount or until the agent collides with something.
3. Stop: Ends the episode, indicating that the agent believes it has found the goal object.
All observations, actions, and associated dynamics are deterministic.
World specifications. In ObjectNav, an agent is spawned in household enviromment. and must find and navigate to instance of specified target object in efficient path possible. Doing effectively requires a common-sense understanding of where household objects are often found and the structure of standard homes.
Habitat provides a standardized train-validation split, consisting of 80 houschold scenes for training (from which one can run online RL or collect data for offline RL or BC) and 20 novel scenes for validation, thereby testing policies' generalization capabilities. These scenes come from the Sabitat-Matterpar 3D v1 dataset Ramakrishnan et al. 2021).
C.2 Policy and Training Details
In line with previous work (Ramrakhya et al. 2023 Yaday et al. 2023b; Majumdar et al. 2023). we train our policies with behavior cloning (BC) on the Habitat-Wad human demonstration dataset of 77k trajectories (12M steps) (Ramrakhya et al. 2022). We adopt many of the same design choices provided by said prior works, but with a few critical differences:
Due to compute limitations, we were unable to train on the full dataset (as those original works used 512 parallel environments to roll out demo trajectories and collect data). Instead, we used a subset of the dataset, built by dividing the dataset. by both target object and scene, then sampling every tenth demo. This would ensure that training data still contained examples from every training scene + target object combination that existed. In total, our subsampled datasct contains approximately 1.1M steps over 7550 trajectories.
2.
We adopt the same optimizer, scheduler, and associated hyperparameters Majumdar et al. (023 but find learning rate of 1e 4 to be more effective than their 1e - 3.
Rather than sampling partial trajectory rollouts from 512 parallel environments as done by Ma- jumdar et al. (2023), our batches contain full trajectories, though with the same total number of transitions per batch in that work. This means that our batches potentially contain less diverse data (due to observations from fewer different total scenes being present), but allow us to com-- pute up-to-date full trajectory hidden states for the RNN portion of our policy. We use gradient accumulation to achieve this, once again due to compute limitations.
4. While Majumdar et al. (2023) trains for 24k gradient steps (observing approximately 400M transi- tions.), we find using only approximately a tenth of that (40 epochs through our smaller dataset, so around 40M transitions) to reach peak performance for our policy. The scheduler still assumes the full training run will last for 400M transitions, so our LR decays at the same rate as with VC-1. Furthermore, for fairness, we leave our VC-1 bascline policies (trained on our subsampled datasots) training beyond 40 epochs, and report their validation performance at both 40 and 120 epochs (when its performance saturates).
23

---

Published in Transactions on Machine Learning Research (03/2025)
5. For policies that receive visual observations sequence of tokens (PR2L, VC-1 with patch embed- dings), we apply 2D average pooling with kernel sizes of 4 x 4 to reduce down to 16 tokens. Then, we pass those tokens through learned Transformer layer, instead of the learned compression layer used by Majumdar et al. (2023). We do this to ensure that policy performance differences are due lo represenlation quality, not architecture.
6. We employ inflection upweighting during training, as done by Ramrakhya et al. (2023 Yaday et al. (2023b); Majumdar el al.] (2023). However, we also categorically upweight the cross entropy loss of stopping and turning by 1.5 (due to them being uncommon but important), as we observe this increases learning speed for all policies.
We do not employ any image augmentation or loss regularization prevent overfitting. However, we find our policy exhibits strong generalization performance in unseen validation scenes nonetheless.
For PR2L-specific design choices:
1. Our chosen VLM is the Prismatic VLM Karamcheti et al. 2024 with Digo-S as vision backbone and Llara2-7B-pure as the language backbone. We use the 224px version, which maps images to 256 visual tokens (which, as described above, get compressed into 16 via pooling).
2. To reduce the size of VLM representations for PR2L, we embed one observation (sampled uniformly at random) from each trajectory in our subsampled dataset with our VLM, then compute all resulting tokens' principle component vectors. We then use said vectors to lower all tokens' dimensionality down from 4096 to 1024 (i.e., corresponding approximately to their first 1024 principle components).
3. Like with the Minecraft experiments, we take the VLM's last two layers' embeddings and treat them as our promptable representations. However, unlike with Minecraft, we stack each VLM token's two embeddings (forming new embeddings of size 2048), rather than concatenate all of them.
For generating text. in response to our task-relevant prompt, we use sample-based decoding with fixed random seed prior to the decoding with temperature 0.4 and 32 48 new tokens generated.
The learned Transformer layer of our policy the same as the one used the Minecraft experiments, but with token embedding sizes of 1024.
All Habitat training done A100 GPU server. Generation of data and evaluations were done on 16 A5000 GPUs for parallelization.
D Simplified Habitat Offline RL Experiments
While our primary Habitat experiments use behavior cloning to stay consistent with past works, we also run offline RL experiments on a simplified version ObjectNav to better explore how VLM representations aid action learning. We discuss the details of said setting now.
D.1 Environment Details
We pick 32 reconstructed 3D home environments with at least one instance of each the three target objects (toilct, bed, and sofa) and an annotator quality score of lcast 4 out of 5. We choose to remove plants and televisions from the goal object set due to finding numerous unlabeled instances of them. Additionally, we remove chairs, as they are significantly more common than other goal objects and thus usually can be found in much shorter episodes. This simplified problem formulation enables us to remove many of the "tricks" that aid ObjectNav, such as using omnidirectional views or policies with history; our agent makes action decisions purely based on its current visual observation and pose, allowing us to do "vanilla" " RL to better isolate the effect of PR2L.
To generate data, we use Habitat's built-in greedy shortest geodesic path follower. Imitating such demon- strations allows policies to learn unintuitively emergent and performant navigation behaviors (Ehsani et al.
24

---

Published in Transactions on Machine Learning Research (03/2025)
2023) at scale. For each defined starting location in our considered households, we autonomously collect data by using the path follower to navigate to each reachable instance of the corresponding goal object. This yields high quality, near-optimal data. We then supplement our dataset by generating lower-quality data. Specifically, for each computed near-optimal path from a starting location to a goal object instance, we choose to inject action noise parlway through the trajectory (uniformly at random from 0 - 90% of the way through). At that point, all subsequent actions have a 0- 50% probability (again chosen uniformly at random) of being a random action other than the one specified by the path follower. To ensure that paths are sufficiently long, we choose to make the probability of choosing the stop action 10% and the other two movement actions 45% In total, we collect 107518 observations over 2364 trajectories.
Reward functions. The ObjectNay challenge evaluates agents based on the average "success weighted by path length' (SPL) metric (Yadav et al.) 2023ah: if an agent succeeds at taking the stop action while close to an instance of the goal object, it gets SPL(p, 1) = max(1.p] points, where is the actual shortest path from the starting point to an instance of the goal object and is the length of the path that the agent actually took during that particular episode, If the agent stops while not close to the target object, the SPL is 0. Thus, taking the most efficient path to the nearest goal object and stopping yields a maximum SPI of 1.
We use this to design our reward function. Specifically; when the agent stops, it receives a reward of +10SPL(p,I), Additionally, we add shaping reward of the change in geodesic distance to the nearest goal object instance each time the agent moves (where lowering that distance yields a positive reward).
D.2 Policy and Training Details
For our offline RL experiments in Habitat, we use Conservative Q-Learning (CQL) on top of the Stable- Baslines3 Contrib codebase's implementation of Quantile Regression DQN (QR-DQN) Kumar et al. (2020); Dabney et al. 2017]. We choose to multiply the QR-DQN component of the CQL loss by 0.2. Using the notation proposed by Kumar et al. (2020), this is equivalent to a = 5, which said work also uses. Other hyperparameters are 7 -= 1, ? 0.99, fixed learning rate of 1e 4, 100 epochs, and 50 quantiles (no exploration hyperparameters are specified, since we do not generate any new online data).
The policy architecture used for Habitat experiments are the same those used for PPO, though the final network outputs quantile Q-values for each action (rather than just distribution over actions). The action with the highest mean quantile value is chosen at evaluation time.
During training, we shuffle the data and load full offline trajectorics until the buffer has at least 32 x 1024 32768 transitions or all trajectories have been loaded once that epoch. We then uniformly sample and train on batches of size 512 transitions from the buffer until each transition has been trained on once in expectation (c.g.. number of transitions the buffer batches). Each batch is used for 8 gradient steps before the next is 512
sampled. We choose this data loading scheme to fit the training infrastructure provided by Stable-Baselines3 while not using up too much memory at once.
D.3 Experiments and Results
Our primary comparison is once again between our promptable representations and general-purpose non- promptable ones. We thus repeat the baseline described previously for Minecraft in Section 4. training a single agent for all three ObjectNav tasks using both PR2L and the VLM image encoder representations. We empirically note that. longer visual embedding sequences tend to perform better in Habitat. To rol for this, we opt to use InstructBLIIP's Q-Former unprompted embeddings instcad of the VIT emboddings directly (which are much longer than PR2L's embedding sequences). As InstructBLIP uses the former representations to extract visual features to be projected into language embedding space, this serves to close the gap in embedding sequence length between our two conditions while still providing us with general visual features that the VLM processes via prompting. In this case, we use the same InstructBLIP model as the Minecraft experiments and choose "What room is this?" as our task-relevant prompt.
We report evaluation success rates and average returns for the simplified Habitat ObjectNav sctting in Figure PR2L achieves nearly double the average success rate of the baseline (60.4% vs. 35.2%), supporting the hypothesis that PR2L works especially well when exploration is not needed. Lastly, in Appendix H.2 we
25

---

Published in Transactions on Machine Learning Research (03/2025)
Success Rates Per Category
Average Returns Per Category
1.0
10
0.8
Average Returns
Success Rate
0.6
0.4
0.2
0.0
Average
Toilet
Bed
Sofa
Average
Toilet
Bed
Sofa
Target Object
Target Object
PR2L (ours)
VLM Image Encoder Baseline
Figure 5: Offiin RL performance of PR2L and baselines in Habitat ObjectNav. Plots show final evaluation success rates and average returns per target object and overall. PR2L outperforms the bascline in all cases.
find that PR2L causes the VLM to produce highly structured representations that correlate with an expert policy's value function: high-value states are typically labeled by the VLM as being from room where one would expect to find the target object.
E
Extended Discussion of Tasks and Results
E.1 Notes on Task-specific Systems
We designed experiments to specifically investigate the use of VLM embeddings as task-specific promptable representations for downstream sensorimotor policy learning. As such, compare with other works that propose evaluate either lcarning from scratch from pre-trained representations, but not to systems in Minecraft and Habital that require domaan--peciiic engineered systems beyond just policy learning (such Luo et al. (2022); Zhu et al. (2022)) or which target learning or producing higher-level plans or abstractions (such as Wang ct al. (202bb
Such comparisons are not made as these works either aim to investigate other problems in control or are aim- ing develop highly specialized and task-specific systems (whereas we present general approach for policy learning). For instance, Voyager shows how LLM can reason about and compose high-level hand-crafted control primitives (Wang et al. 2023a). Voyager's ability to complete harder tasks comes from its access to powerful hand-crafted high-level primitives that extensively leverage oracle information, which are com- posed into skills by GPT-4 (which does not handle any low-level control). Said hand-coded control primitives used in Voyager are very advanced and do much of the heavy-lifting. In particular, Voyager gives GPT-4 access to dedicated killMob (allMob(<entity name>) control primitive function. This function calls a separate bot.pvp attack att. (<entity>) (hand-written) function, which calls hard-coded oracle pathfinder, aiming controller, and altack function to repeatedly approach and attack the specified entity until is defeated. Thus, for Voyager, the skill for hunting sheep simply fills in the powerful killMob () primitive function with "sheep" as the target, abstracting away all low-level control via the oracle hand-written controllers.
Vitally, unlike PR2L, Voyager does not investigate how to use (V)LMs to learn these primitives. thus cannot be applied to settings that lack such primitives (e.g., because oracle path planners are not available, like in Habitat). This makes PR2L complementary: we directly learn policy link obscrvations low- level actions (turning, moving, attacking, etc) via RL with no oracle information, while Voyager aims compose pre-existing primitives into skills via LLMss
26

---

Published in Transactions on Machine Learning Research (03/2025)
Ablations Change Aux. Text
Task
PR2L (Ours)
VLM Image Encoder
No Prompt
No Generation
Oracle Detector
Combatt Spider
97.6 +: 14.9
51.2 9.3
T2.6 + 14.2
66.6 + 11.8
80.1 + 12.6
58.0 + 13.4
Milk Cow
223.4 + 35.4
95.2 + 18.7
116.6 + 25.9
160.2 + 23.6
80.5 + 17.8
178.4 = 42.5
Shoar Sheep
37.0 - 4.4
23.0 + 3.6
23.8 + 3.2
26.1 + 4.5
27.8 = 4.6
27.4 = 9.3
Table 8: Minecraft ablations, VLM image encoder baseline, and our full approach. All achieve worse performance than PR2L. Values final IQM success counts and intervals are the standard error.
E.2 Notes on Dreamer v3
We note that PR2I. just proposes to VLMs a source of task-specifie representations for RL tasks; it does not prescribe which learning algorithm to use. Therefore, in principle, one could use Dreamer in conjunction with PR2L and gain benefits from both the VLM representation and the choice of a strong model-based RL algorithm. However, while we leave this to future works, our Minecraft comparison (c) measures how well the approach does on our Minccraft tasks (as the original paper focuses more on the component subtasks involved in the find diamond lask, all of which do not involve interacting with moving entities).
We find that Dreamer v3 is unable to learn our six tasks given the same number of environment interactions that PR2L+PPO was trained on. We hypothesize that this is due to its visual reconstruction--eese world model not being suited for tasks requiring interaction with partially-observable non-stationary autonomous entities (which all our tasks involve). We note that the last two rows of the figure visualizing model recon- structions in the original Dreamer v3 paper shows that its world model fails to reconstruct an observed pig (Hafner et al.) 2023), supporting our hypothesis. This highlights the need for robust representations that are conducive to world model learning, with PR2L's capabilities to elicit task-relevant visual semantic features via prompting being one possibility for doing so.
F
Ablations
We run four ablations on combat spider, milk cow, and shear sheep to isolate and understand the importance of various components of PR2L. First, we run PR2L with no prompt to see if prompting with task context actually tailors the VLM's generated reprosentations favorably towards the target task, improving over unprompted VLM. Note that this is not the same just using the image encoder (comparison (a)), as this ablation still decodes through the VLM, just with an empty prompt. Second, we run PR2L with our chosen prompt, but no generation of text - i.e., the policy only receives the embeddings associated with the image and prompt (the left and middle red groupings of tokens in Figure 2 but not the right-most group). This tests the hypothesis that representations of generated lext might. make certain task-relevant features more salient: e.g., the embeddings for "Is there a cow in this image?", might not encode the presence of a cow clearly if the VLM generates "Yes" in response, impacting downstream performance. Third, to check if our prompt evaluation strategy provides good proxy for downstream task performance while tuning prompts for P2RL, we run PR2T. with alternative prompts that were not predicted to be the best, as per our criterion in Appendix A We thus remove the auxiliary text from the prompt for combat spider and add it for milk cow and shear sheep. Lastly, to see if PR2L embeddings are just better due to them encoding entity detection, we train a VLM image encoder policy with an additional ground truth oracle target entity detector as a feature.
Results from these additional experiments presented in Table g In general, all ablations perform worse than PR2L. For milk cow, we note the most performant ablation is generation, perhaps because the generated text is often wrong; among the chosen prompts, it yiclds the lowest true positive and negative rales for classifying the presence of its corresponding larget entity (see Table in Appendix A), though adding auxiliary text makes it even worse, perhaps explaining why milk cow experienced the largest performance decrease from adding back in. Based on these overall trends, we conclude that (i) the promptable and generative aspects of VLM representations are important for extracting good features for control tasks and (ii) our simple evaluation scheme is effective proxy measure of how good prompt is for PR2L.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Behavior Cloning Success Rate
Combat
Spider
Milk
Cow
Shear Sheep
PR2L (ours)
VLM Image Encoder Baseline
0.0
0.2
0.4
0.6
0.8
1.0
Success Rate
Figure 6: Success rates for BC on either PR2L or VLM image encoder baseline representations for all original tasks. PR2T. excels at combat spider, even after the policy is trained for single epoch.
G Minecraft Behavior Cloning Experiments
We collected expert policy data by training policy on MineCLIP embeddings complction on all of our original lasks and saving all transitions to create offline dataset. We then embedded all transitions with either PR2L the VLM image encoder. Finally, train policies with behavior cloning (BC) successful trajectories under specified length (300 for combat spider, 250 for milk cow, and 500 for shear sheep) from either set of embeddings for all three tasks, then evaluate their task success rates.
Results are presented in Figure 6 We first note that, since the expert data was collected from a policy trained on MineCLIP embeddings, the shear sheep policy not very effective (as we found in Table 2). Both resulting shear sheep BC policies are likewise not very performant. We find that combat spider in particular shows very large gap in performance: the PR2L agent achieves approximately twice the success rate of the VLM image encoder agent after training for just a single epoch. The comparatively small amount of training and data necessary to achieve near-expert performance for this lask supports our hypothesis that promptable representations from general-purpose VLMs do not help with exploration (they work better in offline cases, where exploration is not a problem), but instcad are particularly conducive to being linked to appropriate actions even though the VLM is not producing actions itself. Further investigation of this hypothesis is presented in Appendix H
H Representation Analysis
Why do our prompts yield higher performance than one asking for actions or eestruction-olllwiinn Intu- itively, despite appropriate responses to our task-relevant prompts not directly encoding actions, there should be a strong correlation: e.g,, when fighting spider, if the spider is in view and the VLM detects this, then good policy should know to attack to get rewards. We therefore wish to investigate if our representations are conducive to easily deciding when certain rewarding actions would be appropriate for given task if it is, then such a policy may be more casily learned by RL, which would explain PR2L's 's improved performance over the baselines.
H.1 Minecraft Analysis
To investigate this, we use the emboddings of our offline data from the BC experiments (collected by training MineCLIP encoder policy to high performance on all of our original three lasks, as discussed in Appendix G We specifically look at the embeddings produced by a VLM when given our standard task-relevant prompts and when given the instructions used for our RT-2-style bascline. We then perform principal component analysis (PCA) on the tokenwise average of all embeddings for each observation, thereby projecting the embeddings to 2D space with maximum variance.
28

---

Published in Transactions on Machine Learning Research (03/2025)
Spider Reps
Reps
Sheep
15
Abback
12
12
-10
Mosement
-10
20
-20
-25
-20
20
Combat Spider Instrudion Raps PCA
Instuction Raps PCA
Sheep Instruction heps
12-
15
-10
-1
-3
-13
15 13
16
15
10
10
15 15 10 15
PC1
Figure 7: PCA of PR2L representations of observations from twenty episode rollouts of expert policies in all three Minecraft tasks. Larger points correspond to transitions where the expert received > 0.1 reward. We vary the prompt be either our task-relevant prompt or the RT-2-style baseline instruction prompt. Our prompt's representations are bi-modal, with the clusters on the left corresponding to the VLM outputting "yes" (the entity is in view). We find that most functional actions (orange points) that yielded rewards are located in said clusters. Note that, since these expert policies are trained on top of MineCLIP embeddings, the shear sheep policy is not very performant, as seen in Table 2
We visualize these low-dimensional space in Figure 7 for the final 20 successful obsorvations from each task, with the point colors orange and blue respectively indicating whether the observation results a functional action (attack or use item) or movement (translation or rotation) by the expert policy. Additionally, we enlarge points corresponding to when the agent received rewards in order to recognize which actions aided in or achieved the task objective.
We find that our considered prompts resulted in bimodal distribution over representations, wherein the left-side cluster corresponds to the VLM outputting "yes (the entity is in view)" and the right-side one corresponds to "no." Additionally, observations resulting in functional actions that received rewards (large orange points in Figure 7 tend to be on the left-side ("yes") cluster for representations elicited by our prompt, but are more widely distributed in the instruction prompt case, in agrcement with intuition. This is especially clear in the milk cow plot, wherein nearly all rewarding functional actions (using the bucket on the cow to successfully collect milk) are in the lower left corner.
This analysis supports that the representations yielded by InstructBLIP in responsc to our chosen style of prompls are more structured than representations from instructions. Such structure is useful in identifying and learning rewarding actions, even when said actions were taken from an expert policy trained on unrelated embeddings. This suggests that such representations may similarly be more conducive being mapped good actions via RL, which we observe empirically (as our prompt's representations yield more performant policies than the instructions for the RT-2-style baseline).
29

---

Published in Transactions on Machine Learning Research (03/2025)
Bid Wike MAZ. hapa PCA
Fin snd PAZ. PCA
Find Sera 1934
30-
20
Figure PCA of PR2L (above) and image encoder (below) representations of observations from thirty episode rollouts of expert policies in all Habitat tasks. The points' colors correspond to their value under Habitat's built-in oracle shorlest path follower (a near-optimal policy). More yellow is better. Boxes correspond to points the VLM has labeled a given household room, in response to the task prompt of "What room is this?" This analysis aligns with intuition: for find toilet, high value observations lend to be labeled as bathrooms (orange box), fond bed's tend to be labeled as bedrooms (blue), and fond sofa's s are labeled as living rooms (red).
H.2 Habitat Analysis
Likewise, we conduct a similar analysis on the Habitat data from our simplified setting. Specifically, we wish to see if PR2T. produces representations that are conducive to extracting the value function of good policy. Since the chosen Habitat ObjectNav prompt is "What room is this?" we expect the state representations to be clustered based room categories. Intuitively, states corresponding to the room one is likely to find the target object should have the highest values.
As shown in F we thus used PCA to project expert trajectories' PR2L and general image encoder state representations (gencrated with Habitat's geodesic shortest path follower) to two dimensions, then colored each one based on their value under said near-optimal policy. We also plotted the mean and standard deviation of all points labelod as each room, visualizing them as axis-aligned bounding boxes. Note that each upper subplot in Figure g has cluster of points far from all boxes. These correspond to the VLM generating nothing garbage data with room label.
This visualization qualitatively agrees with intuition. High value states tend to be grouped with the room the corresponding larget object is often found in: find loile! corresponds to bathrooms, find bed to bedrooms, and find sofe to living rooms. Comparatively, the general image encoder features do not have such semantically meaningful groupings; all observations are clustered together and, within that single grouping, high-value observations are more spread out. This all supports the idea that prompting allows representations take structures that correlate well to value functions of good policies.
30

---

Published in Transactions on Machine Learning Research (03/2025)
Code Snippets
We provide some code snippets showcasing instantiations of PR2L.
class Policy (torch Module):
def __init__ (self,
actions, subedddim=40900
"""Policy that accepts pronptable reps imput **
super O.__init___
Project down VLM embed dimensions
self embed_fc torch. Linear (tf etf_embed_din, 1024) Predict actions
action torch. an. Linear (1024, num_actions]
Transformer layer to condense promptable reps to token self transforner torch. un. Transforner 1024,
1, num jun_enccoer__aet nun_encoder_Layere-., num Lun_decoder_layera. dim Ain_feeaforward-102. batch_
self torch (mbedding 1024) cls tokens
def forward(self. x):
aeq, bs, traj_len, nun_tokens,
seq. shape
[batchrtraj num tokens, token size] seq seq. reehapee (bs traj_len, nun_tokens, -1) [batchetraj_len, num tokens]
mask mask. reshape (bs traj_len, aun_tokens)
# Project down # [batch+traj _len, num tokens, tf dim]
self. embed_fc(seq)
Get CLS embedding cls self cla(torct zeros traj_len, 1].
device=seq device, devicce dtype-int))
summary embedding [batch*traj_
din]
els_embed self transformer
seq, Encoder input cls, Decoder input
Apply mask
sre_ key wwwpppddddss--aaaana
memory_kep key____e_____e wer.yy.eeypaaddgnaaae.. mask-nask,
# [batch, traj_len, d m_ddde]]
cls embed cla_embed, reshape (bs, traj_len, -1)
# Predict actions
# [batch, traj_len, actions] return self
(cle_embed)
Listing 1: Example policy for PR2L.
def process_ote (model, processor, image, prompt, device, last_n=2):
inputs processor (images=inage, text=pronpt, return_tensors="pp" to (device)
Generate text in response to prompt and extract enbeddings outputs model generatee
**inputs, hit.u__hidden_stttee-tte
return_ saarrnddeetiiinggneeteee-- Any generation parameters (nin/nax tokens, temp, etc)
31

---

RGB obs from the env and process it with the VLM Published in Transactions on Machine Learning Research (03/2025)
he outputa "'iddeensstate""
image prompt embeds
Any additional processing should happen here (eg pooling of visual tokens)
[last_n, nun ing prompt tokens, tf_embed_dian saaage_and_proatt._
torch. cat [0],,, dim-0) [=last_n:]
Get decoded token embeds
# (last_n, nun decoded tokens, tf_embed_din] dec_emba = []
for dec_hs in he [1:1:
# [last_n, 1, tf [f__embed_diml
dec_hs = torch cat (dec_hs, din=0) [-last_n:] dec_embs append (dec_be)
[last_n, num decoded tokens, tf tf_embed_ din]
_ec_embs- torch.ca (dec_embs, dim=1)
[last_n, total tokens]
seq_ emba torch. corch.cat ([image and aaddppromt_eebae dec_embs], dim-1) if__mmedd_eee dim seq_enbs... shape [-1]
[ba=1, seq_len=1, last_ntnum total tokens, tf ef_embed_dinl
seq_emba seq_embs. req_embs.reehape (1, 1, -1, (f_embed_dix)
nask torch.. zeros ends [:-1], type=int)
return seq_embs, mask
Listing Example code for extracting promptable representations from a VLM.
Create VLM and processor (InstructBLIP, for example)
model mntrrcliiffrrrodddooolloleaaaaan wttprrionnrooooprrteeeee pro.apprrraaneed Saaeerooree/msrruutbbi---e
processor InstructBlipProceesor from trom_pretrained ("Salesfore/intteetbbii -iccuna -7b*)
Set device, also change dtype desired
device "cuda:0"
model todel.to (devicee)
Create env env
Create policy. This can be trained via RL or as needed.
policy = Policy (env aun_actions). (device)
Define resk-velevant prozpt
pronpt = "Would a toilet be found here? Why or why not?"
To predict action,
obs env reset()
seq, mask process_ (model, processor, obs, prompt, device)
Then, pass through the policy get action logits and step act set_logite policy forward ((seq, mask)) reshape (env. aum_actions) argmax (act_ (act_logitt) env. step (action)
Listing 3: Example usage of the above function and policy.
J Extended Literature Review
Learning in Minecraft. We now consider some current approaches for creating autonomous learning systems for tasks in Minecraft. Such works highlight some of the difficulties prevalent in tasks in said environment. For instance, since Minecraft tasks take place in dynamic open world, it be difficult
32

---

Published in Transactions on Machine Learning Research (03/2025)
for agent to determine what goal it is attempting to reach and how close it is to reaching that goal. Cai et al. (2023) tackles these issues by introducing and integrating training scheme for self-supervised goal-conditioned representations and a horizon predictor. [Zhou et al. (2023) learns a model from visual observations to discriminate between expert state sequences and non-expert ones, which provides a source of intrinsic rewards for downstream RL tasks (as it pushes the policy to learn match the expert state distribution, which tend to be "good" states for accomplishing tasks Minccraft).
Foundation Models and Minecraft. Likewise, there has been much interest in applying foundation models especially (V)LMs to Minecraft tasks. Baker et al. (2022) pretrains large scale videos, which enabled the first agent that could learn to acquire diamond tools (thereby completing a longstanding challenge in the MineRL competition Kanervisto et al. (2022)) ). LMs have subsequently also been used to produce graphs of proposed skills to learn or technology tree advancements to make in the form of structured language Nottingham al.
2023 Zhu al. 2023 Yuan al. 2023 Wang et al. 2023b). Other works propose use the LLM generate actions or code submodules given textual descriptions of observations or agent 2023a Finally, VLMs have been used largely for canguage-condiitioned reward shaping [Fan et al. 2022 Ding et al. 2023). In contrast, use VLMs as source of representations for learning of atomic tasks (as defined by Lin et al. (2023a)) that have pro-defined reward functions; the latter works can thus be used in conjunction with our proposed approach for tasks where these vision- language reward functions are appropriate.
states (Wang al.
33


Reviews:
[{"review_id": "hpUl6k8W8I", "reviewer": "Reviewer_fqhF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- World modeling is an important research topic. The paper shows one approach that can elicit good representations for a downstream policy learning task\n\n- The paper proposes to design prompt that provide additional task-related context rather than training an instruction following model\n\n- The evaluations is fairly solid. Empirical evaluation uses two different datasets and two VLMs and several baselines. These approaches should be interested to readers of TMLR\n\n# Weaknesses\n\n- The paper appears to suggest that prompts should be designed using task-related context. However, there is not suggestion on how to come up with a generic way to design prompts for representation extraction. \n\n- Continuing above point, it is not clear to this reviewer that the method in the paper can be used with newer VLMs that are better than the VLMs used in the paper. What is lacking is a \"sensitivity analysis\" of the prompts to VLMs. The paper uses one specific VLMs per dataset. This is understandable due to concerns on compute but it does leave the reviewer wondering what is the takeaway for a reader of the paper\n\n- No analysis on what constitutes a good prompt that leads to a good representation for policy learning\n\nThe concern that needs to be addressed by the paper is that there is no clear strategy or recipe that can work across tasks and perhaps VLMs. The authors may want to address this either via further experiments (see requested changes) or a discussion in the paper on why a generic strategy may not be possible but how the users of the proposed approach may want to design prompts. This would greatly enhance the usefulness of the high level message in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to extract semantically meaningful representations by prompting pretrained vision language models (VLMs) with task-related (task-speciic) prompts. The idea in the paper builds on previous observations that suggest that representations carry semantically rich information about the world. These extracted representations are used to train policies in a RL setting. Evaluations on 2 datasets suggest that the method, PR2L, works better than several baselines including image embeddings alone, model-based RL methods, methods trained on datasets considered in the paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- There is a paragraph in Section 3 **Evaluating and designing prompts for RL** Are there any experimental results that can support the assertions made in this paragraph about the sufficiency of the surrogate task for evaluating representations?\n\n- Do the same prompting strategy work for different VLMs for a given task? \n\n- Is there any analysis on the extracted embeddings that can be done that sheds light on what is necessary to construct good prompts? SSL literature has the following papers that suggest how one might predict downstream performance on a task or class of tasks. See the following papers (and references therein for more)\n[1] RankMe https://arxiv.org/abs/2210.02885\n[2] LiDAR https://arxiv.org/abs/2312.04000", "broader_impact_concerns": "When it comes to designing promptable representations the users of the proposed method may want to be mindful of the same concerns other users of VLMs may have. No specific statement in the paper is necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6gJJ5IpF3h", "reviewer": "Reviewer_wAy6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The experiment proved the effectiveness of the proposed methods\n* The overall pipeline is relatively easy to understand\n* This work combines VLM for embodied AI, which is interesting and important\n\nWeaknesses:\n* For different tasks, do we train different policies? And possible, could you give an example of action space?\n* For Table 5, it seems that the model is sensitive to the prompt. Is there any explanation or any methods to make the model robust for the prompt?\n* If the author provides code, it will be easy for other researchers to follow", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes an approach that leverages the extensive general and indexable world knowledge embedded in vision-language models (VLMs) pre-trained on internet-scale data for embodied reinforcement learning (RL). The experiments demonstrate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check weakness", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mvy2Opi3RG", "reviewer": "Reviewer_niqv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper is in general clear and well-written\n- P2RL is well explained, and the experimental design is sound\n- The results are notable.  PL2R seems indeed to outperform significantly previous methods\n- I believe the method will become better as more advanced VLMs are developed, which is an important strength of the method.\n\n### Weaknesses\n- As the authors already mentioned, the biggest limitation is the handcrafting required for PR2L to work well. This may be a limitation in environments where users may find it hard to define a 'useful' prompt, and thus in more practical applications. As a proxy for how much hand-crafting makes potentially a difference, it would be good to report the performance of PR2L when using prompts which are not necessarily very good, and analyze the drop in performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new framework, **PR2L**, which allows to leverage the knowledge already present in current VLMs to improve classical RL policies, which usually learn their behaviour entirely from interactions with the environment. \n\nPR2L consider the task relevant features (defined the entire embedding of the last layers of the model)  produced by a VLM upon elicitation with a task-specific prompt; these features are fed as input to the RL policies. \n\nThe authors evaluated their method, against various baselines (vision-only embeddings, instruction-tuning, etc.), across 2 environments: Minecraft and Habitat. The results showcase that the approach beside on the proposed promptable representations outperforms intsruction-following methods and are competitive w.r.t. domain-specific embeddings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I request that the authors report the SOTA results from previous methods (when possible), even if the compute resources may not be equivalent. For example, consider VC-1 reported in Table 3. While it is true that PR2L outperforms this method when trained on the same amount of data, I would not consider a given the fact that PR2L will have better performance when increasing the data. It would be useful to thus have in context the absolute results, as to have a fairer comparison with previous methods.\n\n- Not being 100% familiar with all the previous work, I would find it potentially useful to have a table that summarizes the main differences across the various methods considered and PR2L. Such table would for example include a small prompt example if relevant, if hand-crafting is required, etc. That would be useful to have an immediate grasp of the strengths and weaknesses of the various methods, providing a better context when going over the paper.\n\n\nJust some small typos I found.\n\nSection 1: 'to the best our knowledge' -> 'to the best of our knowledge'\n\nSection 3.2, last paragraph: ',this' -> ', this'", "broader_impact_concerns": "No ethical concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nqB4WJW2DZ", "reviewer": "Reviewer_tAan", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The manuscript is clearly written, concepts are explained concisely and effectively;\n - Idea is novel and interesting;\n - Experiments provide evidence in support of the claims made in the manuscript.\n\nWeaknesses:\n - Prompt construction is ad-hoc per task, and relies on human knowledge;\n - The proposed method requires inference with large VLMs at inference time, which can be expensive and time-consuming;\n \n I will expand on a few questions below, and I am looking forward to discussing with the authors!", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission proposes to prompt general-purpose vision-language models (VLMs) with task-specific questions to obtain representations to train reinforcement learning (RL) policies.\n\nExperiments are conducted on two different dataset to compare with existing methods and study the effect of different prompting strategies.\n\nResults show that the proposed method PR2L, outperforms existing methods by achieving better performance on the same amount of data.\n\nLimitations and future research directions are discussed in the conclusion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Sec. 2, Embodied VLM reasoning.\n\n> Even works in this area that use RL ...\n\nI am not sure I understand what \"reasonable high-level goals to learn\" means here. Could the authors clarify and expand on the message they are trying to convey here?\n\n2. Sec. 3.1, first paragraph.\n\n> Thus, we disregard the generated text and instead provide our policy the embeddings ...\n\nCould the authors expand on what they mean by disregarding? My understanding is that the \"generated text\" is used in the form of embeddings, without decoding them to human-language. If the policy had the same vocabulary and token embeddings as the VLM, then this would be equivalent to decoding the generated text and embedding it again to the same tokens. In fact, there is an ablation study where the decoded *tokens* are not included as input of the policy, and performance degrades. Am I missing something?\n\n3. Sec. 3.1, Which parts of the network paragraph.\n\n> We then use some or all of these token embeddings ...\n\nCould the authors clearly state which tokens they use, and how they choose them? What do the authors mean with \"along with any non-visual observation information\", could they provide examples?\n\n4. Sec. 3.2, second paragraph.\n\n> Thus, we opt to only feed the final few layers' representation into our policy ...\n\n\"The final few layers'\" is imprecise and should be clarified. Which layers are chosen, and how? How are tokens coming from different layers combined as input to the RL policy? Is the past of the trajectory also included as an input to the RL policy, if so, how?\n\n5. Sec. 3.3, paragraph at the bottom of the page. \n\n> The goal of task-relevant ... is to accelerate learning.\n\nI think readers might be interested in seeing a comparison of the runtimes of the different methods. Prompting requires decoding, which can be time-consuming. Could the authors include training times to clarify in which terms training is accelerated, or how it would scale with, for example, to the full Habitat experiment dataset?\n\nRelated to this point, do the authors have any intuition on how decoding could be replaced with a less time-consuming task? For example, could some form of in-context learning help with the quality of the representations without needing to decode? It might also be interesting to readers to see the main takeaways of the ablation experiment with and without decoding in the main text of the manuscript. Right now, it is only mentioned in passing, but it is important.\n\nOverall, Sec. 3.3 reads very high-level and it provides general guidelines. Readers may expect this section to be more technical, and to provide insights on how the task of designing prompts may be automated/generalized to different tasks. It may be helpful to clarify that prompt design, within the scope of the submission, relies completely on human knowledge. Some ways to go beyond this setting could be mentioned as future research directions.\n\nThe evaluation of the prompts in terms of detection performance made me wonder whether the VLM could be replaced with a classifier trained on the detection task. It could be an interesting baseline to compare with, in order to understand whether the more general-purpose knowledge in VLMs provides an advantage compared to a simpler model with limited reasoning capabilities.\n\nIt could be interesting to see how a couple different VLMs compare on the same task.\n\n---\n\n**Minor comments**\n\n1. Sec. 2, Vision-language models paragraph. \"like\" is colloquial, could the authors rephrase with more formal language?\n2. Sec. 2, Embodied VLM reasoning paragraph. Repetition in \"the model's language modeling\".\n3. Sec. 2, Embodied VLM pre-training. \"Figuring out\" is colloquial.\n4. Sec. 4.1. Please make sure citations are within parentheses unless they are a part of the text.\n5. Table 2: Could the authors replace the absolute numbers with success rates as in Table 3?\n6. Tables: Could the authors boldface the winner method for each task?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R9i40TiOTz", "reviewer": "Reviewer_YwUW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. the proposed method is straightforward to understand, easy and cheap to deploy\n\n2. presentation is easy to follow\n\nWeaknesses:\n\n1. the authors do not discuss latency comparison of all the methods: it is not surprising that using more computation by adding vlm description of task features, but how much more cost is needed to utilize these representations?\n\n2. in section 4.3 and other relevant parts of the paper, the authors describe their method as using chain of thought, but my personal opinion is the prompting method is more like providing a rationale rather than chain of thought: chain of thought should be more about decomposing tasks into elementary units and computing the results as exemplified in [1].\n\n3. I don't completely follow all of the rationales of architectural design: e.g. the authors mention that the trained transformer layer is present because \"these representation sequences are of variable length\". It does not seem to be very intuitive as to why the authors have to train another module to solve this problem instead of using other alternative methods, e.g. padding/truncating/pooling.\n\n[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes that VLMs can be prompted to provide useful information to train efficient RL agents.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is generally quite interesting! The authors are encouraged to address the concerns mentioned in the weaknesses", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The strengths of this work lie in its clear writing, high generalizability of the method, and excellent experimental performance. The reviewers unanimously recognized the contribution of this work and considered the paper to be generally correct. The reviewers' concerns mainly stem from the limited discussion of related work and more comparisons with baselines. Although the authors provided detailed responses to the reviewers' comments during the rebuttal period, I still tend to have the authors incorporate these changes into the new version of the paper and submit it.

Some of the suggested edits for the final version:
- Fully discuss the comparison with existing work mentioned by the reviewers, focusing on analyzing the differences in method design, motivation, and experimental results.
- Address the lack of ablation studies by using different LLMs on the same task or using the same LLM on different tasks.
- Fix minor typos pointed out by the reviewers.
Decision: Accept (poster)