Guide Your Agent with Adaptive Multimodal Rewards
Authors: Changyeon Kim
Younggyo Seo
Hao Liu
Lisa Lee
Jinwoo Shin
Honglak Lee
Kimin Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Multimodal Representation
Imitation Learning
TL;DR: We introduce a novel imitation learning framework that uses the visual-text alignment score as a reward signal for better generalization.

Abstract: Developing an agent capable of adapting to unseen environments remains a difficult challenge in imitation learning. This work presents Adaptive Return-conditioned Policy (ARP), an efficient framework designed to enhance the agent's generalization ability using natural language task descriptions and pre-trained multimodal encoders. Our key idea is to calculate a similarity between visual observations and natural language instructions in the pre-trained multimodal embedding space (such as CLIP) and use it as a reward signal. We then train a return-conditioned policy using expert demonstrations labeled with multimodal rewards. Because the multimodal rewards provide adaptive signals at each timestep, our ARP effectively mitigates the goal misgeneralization. This results in superior generalization performances even when faced with unseen text instructions, compared to existing text-conditioned policies. To improve the quality of rewards, we also introduce a fine-tuning method for pre-trained multimodal encoders, further enhancing the performance. Video demonstrations and source code are available on the project website: \url{https://sites.google.com/view/2023arp}.

Guide Your Agent with Adaptive Multimodal Rewards
Changyeon Kim Jinwoo Shin'
Younggyo Seo?
Hao Liu?
Lisa Lee*
Honglak Lee
Kimin Lee'
'KAIST DDyson Robot Learning Lab
UCC Berkeley
*Google DeepMind
University of Michigan
LLG Al Research
Abstract
Developing an agent capable of adapting to unseen environments remains difficult challenge imitation learning. This work presents Adaptive Return-onnditioee Policy (ARP), an efficient framework designed to enhance the agent's generaliza- tion ability using natural language task descriptions and pre-trained multimodal encoders. Our key idea is to calculate similarity between visual observations and natural language instructions in the pre-trained multimodal emhedding space (such as CLIP) and use it as reward signal. We then train return-conditioned policy using expert demonstrations labeled with multimodal rewards. Because the multimodal rewards provide adaptive signals at each timestep, our ARP effectively mitigates the goal misgeneralization. This results in superior generalization per- formances even when faced with unseen text instructions, compared to existing text-conditioned policies. To improve the quality of rewards, we also introduce fine-tuning method for pre- pre-trained multimodal encoders, further enhancing the performance. Video demonstrations and source code are available on the project website: https: sttps://siitee google wwt./viee/e22aaaa.....ee
1 Introduction
Imitation learning (IL) has achieved promising results in learning behaviors directly from expert demonstrations, reducing the necessity for costly and potentially dangerous interactions with envi- ronments [32, 59]. These approaches have recently been applied to learn control policies directly from pixel observations [7, 36, 57]. However, IL policies frequently struggle generalize to new environments, often resulting in a lack of meaningful behavior [14, 58, 68, 79] due to overfitting to various aspects of training data. Several approaches have been proposed to train IL agents capable of adapting to unseen environments and tasks. These approaches include conditioning on a single expert demonstration [18, 20], utilizing video of human demonstration [6, 76], and incorporating the goal image [16, 23]. However, such prior methods assume that information about target behaviors in test environments available to the agent, which is impractical many wal-world problems.
One alternative approach for improving generalization performance is to guide the agent with natural language: training agents conditioned language instructions [7, 50, 72]. Recent studies have indeed demonstrated that text-conditioned policies incorporated with large pre-trained multimodal models [22, 56] exhibit strong generalization abilities [46, 66). However, simply relying on text representations may fail to provide helpful information to agents in challenging scenarios. For example, consider a text-conditioned policy (see Figure la) trained to collect coin, which is positioned at the end of the map, following the text instruction "collect a coin". When we deploy the learned agent to test environments where the coin's location is randomized, often fails collect the coin. This is because, when relying solely on expert demonstrations, the agent might mistakenly think that the goal is to navigate to the end of the level (see supporting results in Section 4.... This example shows the simple text-conditioned policy fails to fully exploit the provided text instruction and suffers from goal misgeneralization (i.e., pursuing undesired goals, even when trained with correct specification) [15, 64].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Text-conditioned Policy Action
Train env: coin ts the for right
Action
Xi
enocdee
encoder
ercoddrr
Treetes
Ossevsblrr D1
Obseraatio
Toe Instrntttiionn
Adaptive Return-conditioned Policy
Action #1
coin middle
Transformar recureet neural ratweet
s4
Rotur
encoder
enoode
enooder
(a) Comparison of ARP with baseline
(b) Multimodal reward curve from CoinRun
Figure I: (a) ARP utilizes the similarity between visual observations and text instructions in the pre-trained multimodal representation space reward and then trains the return-conditioned policy using demonstrations with multimodal reward labels. (b) Curves of multimodal reward for fine-tuned CLIP [56] the trajectory from CoinRun environment. Multimodal reward consistently increases as the agent approaches the goal, and this trend remains consistent regardless of the training and test environment, suggesting the potential to guide agents toward target objects in test environments.
In this paper, we introduce Adaptive Return-conditioned Policy (ARP), novel IL method designed to enhance generalization capabilities. Our main idea is to measure the similarity between visual observations and natural language task descriptions in the pre-trained multimodal embedding space (such as CLIP [56]) and use it as a reward signal. Subsequently, we train a ctuun-conditioned policy using demonstrations annotated with these multimodal reward labels. Unlike prior IL work that relies on static text representations [46, 51, 67], our trained policies make decisions based on multimodal reward signals computed each timestep (see the bottom figure of Figure 1a).
We find that our multimodal reward can provide a consistent signal to the agent in both training and test environments (see Figure lb). This consistency helps prevent agents from pursuing unintended goals (i.e., mitigating goal misgeneralization) and thus improves generalization performance when compared to rext-conditioned policies. Furthermore, we introduce pre-trained multimodal encoders using in-domain data (i.c., expert demonstrations) to enhance the quality of the reward signal. We demonstrate that when using rewards derived from fine-tuned encoders, the agent cxhibits superior generalization performance compared to the agent with frozen encoders in test environments. Notably, we also observe that ARP effectively guides agents in test environments with unseen text instructions associated with new objects of unseen colors and shapes (see supporting results in Section 4.3).
tinettunigg scheme that adapts
In summary, our key contributions are as follows:
We propose Adaptive Return-conditioned Policy (ARP), novel IL framework that trains return- couum-conditioned policy using adaptive multimodal rewards from are-trained encoders.
We introduce a fine-tuning scheme that adapts pre-trained CLIP models using in-domain expert demonstrations to improve the quality of multimodal rewards.
We show that our framework effectively mitigates goal misgeneralization, resulting in better generalization when compared to text-conditioned baselines. We further show that ARP can execute unseen text instructions associated with new objects of unseen colors and shapes.
We demonstrate that our method exhibits comparable generalization performance baselines that consume goal images from test environments, even though our method solely relies on natural language instruction.
Source code and expert demonstrations used for our experiments are available https: / //github. com/ csmile- wwwmimee-e0///0e 1006/0000R ARP, .gi

---

2 Related Work
Generalization in imitation learning Addressing the challenge of generalization in imitation learning is crucial for deploying trained agents in real-world scenarios. Previous approaches have shown improvements in generalization to test environments by conditioning agents
robot demonstration [18, 20], a video of a human performing the desired task [76, 6], or a goal image [16, 44, 23]. However, these approaches have disadvantage: they can be impractical to adopt in real- world scenarios where the information about target behaviors in test environments is not guaranteed. In this work, we propose an efficient yet effective method for achieving generalization even in the absence of specific information about test environments. We accomplish this by leveraging multimodal reward computed with current visual observations and task instructions in the pre-trained multimodal embedding space.
Pre-trained representation for reinforcement learning and imitation learning Recently, there has been growing interest in leveraging pre-trained representations for robot learning algorithms that benefit from large-scale data [54, 73, 61, 52]. In particular, language-condittoned agents have seen significant advancements by leveraging pre-trained vision-languaae models [46, 66, 78, 38]. drawing inspiration from the effectiveness of multimodal representation learning techniques like CLIP [56]. For example, InstructRL [46] utilizes pre-trained multimodal encoder [22] to encode the alignment between multiple camera observations and text instructions and trains transformer- based behavior cloning policy using encoded representations. visual observations and text instructions in the pre-trained multimodal embedding space in the form of a reward to guide agents in the test environment adaptively.
contrast, our work utilizes the similarity between
We provide more discussions on related work in more detail in Appendix B.
Method
In this section, introduce Adaptive for enhancing generalization ability using multimodal rewards from pre-trate crairained encoders. We first describe the problem setup in Section 3.1. Section 3.2 introduces how define the multimodal reward in the pre-trained CLIP embedding spaces and use it for training return-ondditoned policies. Additionally, we propose new fine-tuning scheme that adapts pre-trained multimodal encoders with in-domain data to enhance the quality of rewards in Section 3.3.
Policy (ARP), a novel IL framework
3.1 Preliminaries
We consider the visual imitation learning (IL) framework, where an agent learns to solve target task from expert demonstrations containing visual observations. We assume access to a dataset D {T,} i= consisting of N expert trajectories observation, a means the action, and T denotes the maximum timestep. These expert demonstrations utilized to train the policy via behavior cloning. As single visual observation is sufficient for fully describing the underlying state of the task, we approximate the current state by stacking consecutive past observations following common practice [53, 74].
(Ou, ag .... 0r, or) where represents the visual
We also assume that a text instruction x E X that describes how to achieve the goal for solving tasks is given in addition to expert demonstrations. The standard approach to utilize this text instruction is to train a texi-conditioned policy 7 [ar]ose, x). It has been observed that utilizing pre-trained multimodal encoders (like CLIP [56] and M3AE [22]) is very effective in modeling this eot-connitioned policy [46, 49, 66, 67]. However, as shown in the upper figure of Figure 1a, these approaches provide the same text representations regardless of changes in visual observations. Consequently, they would not provide the agent with adaptive signals when encountering previously unseen environments. To address this limitation, to compute similarity with the current visual observation within the pre-trained multimodal embedding space. We then employ this similarity as a reward signal. This approach allows the reward value to be adjusted as the visual observation changes, providing the agent with an adaptive signal (see Figure lb).
propose an alternative framework that leverages
3.2 Adaptive Return-Conditioned Policy
Multimodal reward
To provide more detailed task information to the agent that adapts over timesteps, we propose to use the visual-text alignment score from pre-trained multimodal encoders.
3

---

Specifically, we compute the alignment score between visual observation at current timestep t and text instruction x the pre-trainn pre-trained multimodal embedding space as follows:
(o, x) = (ot),
(I)
Here, 6 represents a similarity metric in the representation space of pre-trained encoders: a visual encoder l parameterized by á and a text encoder f parameterized by vi. While our method is compatible with any multimodal encoders and metric, we adopt the cosine similarity between CLIP [56] text and visual embeddings in this work. We label each expert state-action trajectory as r- = (Ro, Ou, ag.... Rr, or, a7) where R₂ = Ei=e (o;, x) denotes the multimodal return for the rest of the trajectory at timestep t'. The set of return-labeled demonstrations is denoted as D (T
Return-conditiones policy Using return-labeled demonstrations D", we train return-conditioned policy To vololoos,, Ra) parameterized by using the dataset D and minimize the following objective:
(0) = E,"
Σ EST
UTolarlogu Re), aj)
Here, / represents the loss function, which is either the cross entropy loss when the action space is defined in discrete space or the mean squared error when defined in continuous space.
The main advantage of our method lies in its adaptability in the deployment by adjusting to multimodal rewards computed in test environments (see Figure la). At the test time, trained policy predicts the action a, based on the target multimodal return R, and the observation og. Since the target return R2 is recursively updated based on the multimodal reward rx, it can provide timestep- wist signal to the agent, enabling it to adapt its behavior accordingly. We find that this signal effectively guides the agent to prevent pursuing undesired goals (see Section 4.1 and Section 4.2), and it also enhances generalization performance in environments with unseen text instructions associated with objects having previously unseen configurations (as discussed in Section 4.3).
In our experiments, we implement two different types of ARP using Decision Transformer (DT) [8], referred to ARP-DT, and using Recurrent State Space Model (RSSM) [25], referred to ARP- RSSM. Further details of the proposed architectures are provided in Appendix A.
3.3 Finiin Pre-trained Multimodal Encoders
Despite the effectiveness of our method with pre-trained CLIP multimodal representations, there may be a domain gap between the images used for pre-training and the visual observations available from the environment. This domain gap can sometimes lead to the generation of unreliable, misleading reward signals. To address this issue, we propose fine-tuning schemes for pre-trained multimodal encoders ( , s using in-domain dataset (expert demonstrations) D in order to improve the quality of multimodal rewards. Specifically, we propose fine-tuning objectives based on the following two desiderata: reward should (i) remain consistent within similar timesteps and (ii) be robust visual distractions.
Temporal smoothness To encourage the consistency of the multimodal reward over timesteps. we adopt the objective of value implicit pre-traue pre-training (VIP) [52] that aims to learn smooth reward functions from action-free videos. The main idea of VIP to (i) capture long-range dependency by attracting the representations of the first and goal frames and (ii) inject local smoothness by encouraging the distance between intermediate frames to represent progress toward the goal. We extend this idea to our multimodal setup by replacing the goal frame with the text instruction describing the task objective and using our multimodal reward R as below:
Lv v/) = (1 7) Eor~o, .. ( (01, x]]
long range dependenccy luss
+ log Eio, (cxp(ro o (0 x) +1 T (o.1
local smoothness
'We assume the discount factor as 1 in our experiments, and our method can also be applied in the setup with the discount factor less than 1.

---

Training
Test
Training
Test
Training
Test
(a) CoinRun
(b) Maze
(c) Maze
Figure 2: Environments from OpenAl Procgen benchmarks [10] used in our experiments. We train our agents using expert demonstrations collected in environments with multiple visual variations. We then perform evaluations on environments from unseen levels with target objects in unseen locations. See Section 4.1 for more details.
Here O denotes set of initial visual observations D. One can see that the local smoothness loss is the one-step temporal difference loss, which recursively trains the r (or, to regress -1+ --r,e To.v (Ot+1, x). This then induces the reward to represent the remaining steps to achieve the text-specified goal x [31], making rewards from consecutive observations smooth.
Robustness visual distractions To further encourage our multimodal reward to be robust to visual distractions that should not affect the agent (e.g., changing textures backgrounds), introduce the inverse dynamics model (IDM) objective [55, 33, 41]:
(oDM(O, )) = (-)))==))).)))))((((((( (o:), f (o1+1), ) ( (x)) a;)],
(4)
where g(-) denotes the prediction layer which outputs de, predicted estimate of a,, and represents the loss function which is either the cross entropy loss when the action space defined in discrete space or the mean squared error when it's defined in continuous space. By learning to predict actions taken by the agent using the observations from consecutive timesteps, tuned encoders learn to ignore aspects within the observations that should not affect the agent.
Fine-tuning objective We combine both VIP loss and IDM loss the training objective fine- pre-trained multimodal encoders in our model:
(r v/) = (vi v) + B (OM v),
where 3 is a scale hyperparameter. We find that both objectives synergistically contribute to improving the performance (see Table 7 for supporting experiments).
Experiments
We design our experiments to investigate the following questions:
1. Can our method prevent agents from pursuing undesired goals test environments? (see Section 4.1 and Section 4.2)
2. Can ARP follow unseen text instructions? (see Section 4.3)
3. Is ARP comparable to goal image-conditioned policy? (see Section 4.4)
4. Can ARP induce well-aligned representation in test environments? (see Section 4.5)
5. What is the effect of each component in our framework? (see Section 4.6)
4.1 Procgen Experiments
Environments We evaluate our method on three different environments proposed in Langosco et al. [15], which are variants derived from OpenAl Procgen benchmarks [10]. We assess the generalization ability of trained agents when faced with test environments that cannot be solved without following true task success conditions.
CoinRun: The training dataset consists of expert demonstrations where the agent collects coin that is consistently positioned on the far right of the map, and the text instruction is "The goal
5

---

Training Performance
Test Performance
100
80
Normalized Score (%)
80
40
20
20
CoinRun
CoinRun
InstructRL
ARP-DT (Durs)
ARP-DT+ [Ours)
Figure 3: appert-normalized scores training/test environments. The result shows the mean and standard variation averaged over three runs. ARP-DT denotes the model that uses pre-trained CLIP representations, and ARP-DT+ denotes the model that uses fine-tuned CLIP representations (see Section 3.3) for computing the multimodal reward.
is to collect the coin.". Note the agent may mistakenly interpret that the goal is to proceed to the end the level, as this also leads to reaching the coin when relying solely on the expert demonstrations. We evaluate the agent in environments where the coin's location is randomized (see Figure 2a) to verify that the trained agent truly follows the intended objective.
Maze The training dataset consists of expert demonstrations where the agent reaches yellow cheese that is always located at the top right corner, and the text instruction is "Navigate a maze to collect the yellow cheese.". The agent may misinterpret that the goal is to proceed the far right comner, as it also results in reaching the yellow cheese when relying only on expert demonstrations. To verify that the trained agent follows the intended objective, we assess the trained agents the test environment where the cheese is placed at random position (see Figure 2b).
Maze II: The training dataset consists of expert demonstrations where the agent approaches yellow diagonal line located a random position, and the text instruction is "Navigate a maze to collect the line.". The agent might misinterpret the goal as reaching object with a yellow color because also leads to collecting the object with a line shape when relying only on expert demonstrations, For evaluation, we consider modified environment with two objects: yellow gem and a red diagonal line. The goal of the agent is to reach the diagonal line, regardless of its color, to verify that the agent truly follows the intended objective (sce Figure 2c).
Implementation For all experiments, we utilize the open-sourced pre-trained CLIP model? with VrT-B/16 architecture to generate multimodal rewards. Our return-conditioned policy is implemented based on the official implementation of InstruciRL [46], and implementation details are the same unless otherwise specified. To collect expert demonstrations used for training data, first train PPG [11] agents on 500 training levels that exhibit ample visual variations for 200M timesteps per task. We then gather 500 rollouts for CoinRun and 1000 rollouts for Maze in training environments. All models are trained for 50 epochs on two GPUs with a batch size 64 and a context length of 4. Our code and datasets are available at https: //gi thub. www.cooccccccooo con/csmile--e 1006/ 1006/RR...... ARP. git. Further training details, including hyperparameter settings, can found in Appendix C.
Evaluation We evaluate the zero-shot performance of trained agents in test environments from different levels (i.e., different map layouts and backgrounds) where the target object is either placed in unseen locations with unseen shapes. To quantify the performance of trained agents, report the expert-normalized scores on both training and test environments. To report training performance, measure the average success rate of trained agents over 100 rollouts training environments and divide it by the average success rate from the expert PPG agent used to collect demonstrations. For test performance, we train separate expert PPG agent in test environments and compute normalinorm scores in the same manner.
Baseline and our method As baseline, we consider InstructRL [46], one of the state-of-the-ar text-conditioned policies. InstructRL utilizes a transformerrbased policy and pre-trained M3AE [22]
htttps///gthabb..oooooos/////1

---

Training
SHANT
AGD RCSN (Oun)
AN HCSMI 20uns
Succiss Rate (16)
Test
60
Training
(a) Pick Up Cup
(b) Training/test performance
Figure 4: (a) Image observation of training and test environments for Pick Up Cup task in RLBench benchmarks [34]. (b) Success rates both training and test environments. The result represents the mean and standard deviation over four different seeds. ARP-RSSM denotes the model that uses frozen CLIP representations for computing the multimodal reward, and ARP-RSSM+ denotes the model that incorporates fine-tuning scheme in Section 3.3.
representations for encoding visual observations and text instructions. For our methods, we use return-conditioned policy based Decision Transformer (DT) [8, 43] architecture, denoted as ARP-DT (see Appendix A for details). We consider two variations: the model that uses frozen CLIP representations (denoted ARP-DT) and the model that uses fine-tuned CLIP representations (denoted as ARP-DT+) for computing the multimodal reward. We use the same M3AE model to encode visual observations and the
transformer architecture for policy training. The main difference is that our model uses sequence with multimodal return, while the baseline uses static text representations with the concatenation of visual representations.
Comparison with language-conditioned agents Figure shows that method significantly outperforms InstructRI. in all three tasks. In particular, ARP-DT outperforms InstrucIRL. test environments while achieving similar training performance. This result implies that method effectively guides the agent away from pursuing unintended goals through the adaptive multimodal reward signal, thereby mitigating goal misgeneralization. Moreover, observe that ARP-DT+, which uses the multimodal reward from the fine-tuned CLIP model, achieves superior performance to ARP-DT. Considering that the only difference between ARP-DT and ARP-DT+ is using different multimodal rewards, this result shows that improving the quality of reward can lead to better generalization performance.
4.2 RLBench Experiments
Environment We also demonstrate the effectiveness of our framework on RLBench [34], which serves as a standard benchmark for visual-based robotic manipulations. Specifically, we focus on Pick Up Cup task, where the robot arm is instructed grasp and lift the cup. We train agents using 100 expert demonstrations collected from environments where the position of the target cup changes above the cyan-colored line each episode (see the upper figure of Figure 4a). Then, we evaluate the agents in test environment, where the target cup is positioned below the cyan- cyan-colored line (see the lower figure Figure 4a). The natural language instruction x used is " "grasp the red cup and lift off the surface with the robotic arm." For evaluation, we measure the average success rate over 500 episodes where the object position varies in each episode.
Setup As baseline, we consider MV-MWM [62], which initially trains multi-view autoencoder by reconstructing patches from randomly masked viewpoints and subsequently learns a world model based on the autoencoder representations. We use the same procedure for training the multi-view autoencoders for our method and baseline. The main difference is that while MV-MWM does not use any text instruction as an input, our method trains policy conditioned on the multimodal return as well. In our experiments, we closely follow the experimental setup and implementation of the imitation learning experiments MV-MWM. Specifically, we adopt a single-view control setup where the representation learning is conducted using images from both the front and wrist cameras, but world model learning is performed solely using the front camera. For our methods, we train the return-connn conditiond policy based on the recurrent state-space model (RSSM) [25]. denoted
7

---

as ARP-RSSM (see Appendix A for more details). We consider two variants of this model: the model utilizing frozen CLIP representations (referred as ARP-RSSM) and the model that employs fine-tuned CLIP representations (refemed to as ARP-RSSM+). To compute multimodal rewards using both frozen and fine-tuned CLIP, we employ the same setup as Procgen experiments. Additional details are in Appendix D.
Results Figure showcases the enhanced generalization performance of ARP-RSSM+ agents in test environments, increasing from 20.37% to 50.93%. This result implies that our method facilitates the agent in reaching target cups in unseen locations by employing adaptive rewards. Conversely, ARP-RSSM, which uses frozen CLIP representations, demonstrates similar performance to MV- MWM in both training and test environments, unlike the result in Section 4.1. We expect this is because achieving target goals for robotic manipulation tasks in RLBench requires more fine-grained controls than game- like environments.
4.3 Generalization to Unseen Instructions
(a) CoinRun bluegem
(b) Maze III
Figure 5: Test environments used for experiments in Section 4.3.
We also evaluate our method in test environments where the agent is now required to reach a different object with an unseen shape, color, and location by following unseen language instructions associ- ated with this new object. First, train agents in environments with the objective of collecting yellow coin, which is always positioned in the far right corner, and learned agents are tested on unseen environments where the target object changes to blue gem, and the target object's location randomized. This new environment is referred to as CoinRun-bluegen (see Figure 5a), and we provide unseen instruction, "The goal is collect the blue gem." to the agents. Table 1 shows that our method significantly outperforms the texl-conditioned policy (InstruciRL) even in CoiunRun- bluegem. This result indicates that our multimodal reward can provide adaptive signals for reaching target objects even when the color and shape change.
Table I: Expert-normalized scores on CoinRum-
bluegem test environments (see Figure 5a).
Model
Test Performance
InstruciRL
63.99% 3.7%
ARP-DT (Ours)
77.05% + 2.99
ARP-DT+ (Ours)
79.06% + 6.08
In addition, we verify the effectiveness of our multi- modal reward in distinguishing lomikag-looking dis- tractors and guiding the agent the correct goal. To this end, we train agents using demonstrations from Maze II environments, where the objective is to col- lect the yellow line. Trained agents are tested in an augmented version of Maze II test environments: we place a yellow gem, a red diagonal line, and a red straight line in the random position of the map (denoted Maze III in Figure 5b), and instruct the trained agent to reach the red diagonal line (x = -aavigate a maze to collect the red diagonal line."). Table 2 shows that our method outperforms the baseline in Maze III, indicating that our multimodal reward can provide adaptive signals for achieving goals by distinguishing distractors.
Table 2: Expert-normalized scores on Maze III test environments (see Figure 5b).
Model
Test Performance
InstruceRL
21.21% % + 1.52%
ARP-ITT (Ours)
33.33% + 4.01%
ARP-DT+ (Ours)
38.38% 3.15%
4.4 Comparison with Coal-Conditioned Agents
We compare our method with goal-conditioned methods, assuming the availability of goal images in both training and test environments. First it is essential to note that suggested baselines rely on additional information from the test environment because they assume the presence of a goal image during the test time. In contrast, our method relies solely natural language instruction and
8

---

Training Performance
Test Performance
100
Norrna ized Score (9%)
60
20-
CoinRun
CoinRun
Maze
(Ours)
Figure 6: Expert-normalizer scores on training/test environments. The result shows the mean and standard variation averaged over three runs. ARP-DT shows comparable or even better generalization ability compared to goal-conditioned baselines.
does not necessitate any extra information about the test environment, As baselines, we consider goal-conditioned version InstructRL (denoted as GC-InstructRL) which uses visual observations concatenated with goal image at each timestep. We also consider variant of our algorithm that uses the distance of CLIP visual representation to the goal image for reward (denoted as GC-DT).
Figure 6 illustrates the training and test performance of goal-conditioned baselines and ARP-DT. First, we observe that GC-DT outperforms GC-InstrucRRL. in all test environments. Note that utilizing goal image is the only distinction between GC-DT and GC-IIstrucrRL. This result suggests that our return-conditooned policy helps enhance generalization performance. Additionally, we find that ARP-DT demonstrates comparable results GC-DT and even surpasses GC-InstrucIRL in all three tasks. Importantly, should emphasized that while goal-conditioned baselines rely on the goal image of the test environment (which can be challenging provide in real-worle scenarios), ARP-DT solely relies on natural language instruction for the task. These findings highlight the potential of our method to be applicable in real-world scenarios where the agent cannot acquire information from the test environment.
4.5 Embedding Analysis
To support the effectiveness of our framework in generalization, analyze whether proposed method can induce meaningful abstractions in test environments. Our experimental design aims to address the key requirements for improved generalization in test environments: (i) the agent should consistently assign similar representations to similar behaviors even when the map configuration changed, and (ii) the agent should effectively differentiate between goal-rcaching behaviors and misleading behaviors. To this end, we measure the cycle-consistency of hidden representation from trained agents following [3, 42]. For two trajectories ~ and 72 with the same length N. we first choose i < N and find its nearest neighbor j = arg minygN [lh a22,) h(o², 222))12. where h(-)) causal transformer ARP-DT (refer to Appendix A for details).
denotes the output manner, we find the nearest neighbor of j. which is denoted as A = arg mingşs [| al2R) h((22, az ll. We define i cycle-consistent if i k| 1, can return its original point. The presence of cycle-consistency entails precise alignment of two trajectories within the hidden space.
similar
In our experiments, we first collect the set of success/failure trajectories from N different levels in CoinRun test environment, which is denoted as 7 or 7 where hidden representations from trained agents at each timestep across all trajectories. We then measure cycle-consistency across these representations using three different pairs of trajectories (see Figure 7):
N. Next, we extract
1. (T sucee (f): We compute the cycle-consistency between success trajectories from different levels. This indicates whether the trained agents behave similar manner success cases, regardless of different visual contexts.
2. (rili T (t): Similarly, compute the cycle-consistency between failure trajectories from different levels.
3. (Tilice Thal (4): We measure the cycle-consistency between success trajectory and failure trajectory from the same level. This evaluates whether the agent can act differently success and failure cases.

---

(Trdecs Tan
Level
Table 3: We investigate the cycle consistency of trained agents' hidden representations different sets of trajec- tories in CoinRun environments. The results are presented as the mean and standard deviation averaged over three different seeds, Scores within one standard deviation from the highest average score are marked in bold. (1) and (L) denotes that higher/lower values are better, respectively.
(Plier Tine
(ra Tall
Level is
(Ss0oct Tall
Figure 7: Visual observations of trajec- tories in CoinRun environments. We construct different pairs to evaluate whether trained agents have well-aligned hidden representations.
Instrekk
ARP-ITT
ARP-DT+
(Tager (*)
22.66%
1. LOS
24.75%
1915
29.07%
1.W5
(Ta vad) tt)
14.12%
24.95%
24.29%
1.00
(Fibo Tra) (4)
35.09%
2009
30.18%
1:19
5.65%
Note that (t) / (4) implies that higher/lower value is better, respectively. For implementation, we first select ten different levels in CoinRun test environment with different coin locations. We then collect success and failure trajectories from each level and use the last ten timesteps of each trajectory for measuring cycle-consistency. Table 3 shows the percentage of timesteps that are cycle-consistent with other trajectories from different pairs. Similar to results described in Section 4.1, our proposed methods significantly improve cycle-consistency compared to InstrucIRL in all cases. Moreover, ARP-DT+, which utilizes the multimodal reward from the fine-tuned CLIP model, outperforms ARP-DT with the frozen CLIP.
4.6 Ablation Studies
Effect of pre-trained multimodal representations To verify the effectiveness of pre-trained multimodal representations, we compare ARP-DT+ with agents using multimodal rewards obtained from a smaller- scale multimodal transformer, which was trained from scratch using VIP and IDM objectives, denoted ARP-DT+ (scratch). Table 4 shows a significant de- crease in performance for ARP-DT+ (scratch) when compared to ARP-DT+ across all environments, par- ticularly in the training performance within Maze environments. These findings highlight the crucial role of pre-training in improving the efficacy of our multimodal rewards.
Table 4: Ablation study of using pre-trained CLIP representations.
Bay
Modal
Thain (%)
Tist (9)
CoinRas
ARP-II++
ARP-DT- (scratch)
90.28% 1os5
77.08%
72.36% 4.4
62. 62.48%
Maze
ARP DT+ (scratch)
ARP-DT-
75.47%
18.87%
36. 138
32.5216 2210
Mare II
ARP (scratch)
ARP-IVT-
22.51% a x
1898
40.05% -7.00
37.62%
Effect of fine-tuni fine-tuning objectives In Table 5, we ex- amine the effect of fine-tuning objectives by reporting the performance of ARP-DT fine-tuned with or with- out the VIP loss Lvi (Equation 3) and the IDM loss LIMM (Equation 4). We find that the performance of ARP-DT improves with either LVIP or LIDM, which shows the effectiveness of the proposed losses that en- courage temporal smoothness and robustness visual distractions. We also note that the performance with both objectives the best, which implies that both losses synergistically contribute improving the quality of the rewards.
Table 5: Ablation study of fine-tuning objec- tives: LVV and LIDM in CoinRun.
Lvw
LIDM
Train (%)
Test (%)
x
x
89.58%
2008%
63.32
2010%
x
/
89.24%
6.11%
67.34 % 2.66%
<
x
90.28%
2.17%
70.35
% 1,15%
J
90.28%
= 1.59%
72.36
% +.48%
5 Conclusion
In this paper, we present Adaptive Return condiiicond Policy, simple but effective IL framework for improving generalization capabilities. Our approach trains return-conditioned policy using the adaptive signal computed with pre-trained multimodal representations. Extensive experimental results demonstrate that our method can mitigate goal misgeneralization and execute unseen text instructions associated with new objects compared to text-conditioned baselines. We hope our framework could facilitate future research to further explore the potential of using multimodal rewards to guide II. agents in real-world applications.
10

---

Acknowledgments and Disclosure of Funding
We want thank Sihyun Yu, Junsu Kim, and anonymous reviewers for providing helpful feedback and suggestions for improving our paper. This research is supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) NN..2022-0-0095.. Self-directed AI Agents with Problem-solving Capability; No.201944 00075, Artificial Intelligence Graduate School Program (KAIST); No. 10022022--00014. Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics). This material based upon work supported by the Google Cloud Research Credits program with the award (RDA2-TICV.2D/T-HAHIE
References
[1] Abdal, Rameen, Zhu, Peihao, Femiani, John, Mitra, Niloy, and Wonka, Peter. Clip2stylegan: Unsupervised extraction of stylegan edit directions. In ACM SIGGRAPH 2022 conference proceedings, 2022.
[2] Ahn, Michael, Brohan, Anthony, Brown, Noah, Chebotar, Yevgen, Cortes, Omar, David, Byron, Finn, Chelsea, Gopalakrishnan, Keerthana, Hausman, Karol, Herzog, Alex, al. Do can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[3] Aytar, Yusuf, Pfaff, Tobias, Budden, David, Paine, Thomas, Wang. Ziyu, and De Freitas, Nando. Playing hard exploration games by watching youtube. Advances in Neural Information Processing Systems, 31, 2018.
[4] Bacchus, Fahiem, Boutilier, Craig, and Grove, Adam. Rewarding behaviors. In Proceedings of the National Conference on Artificial Intelligence, 1996.
[5] Bacchus, Fahiem, Boutilier, Craig, and Grove, Adam. Structured solution methods for non- markovian decision processes. In Association for the Advancement of Artificial Intelligence, 1997.
[6] Bonardi, Alessandro, James, Stephen, and Davison, Andrew J. Learning one-shot imitation from humans without humans. IEEE Robotics and Automation Letters, 5(2), 2020.
[7] Brohan, Anthony, Brown, Noah, Carbajal, Justice, Chebotar, Yevgen, Dabis, Joseph, Finn, Chelsea, Gopalakrishnan, Keerthana, Hausman, Karol, Herzog, Alex, Hsu, Jasmine, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:22
2022. [8] Chen, Lili, Lu, Kevin, Rajeswaran, Aravind, Lee, Kimin, Grover, Aditya, Laskin, Misha, Abbeel, Pieter, Srinivas, Aravind, and Mordatch, Igor. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021.
[9] Cho, Jaemin, Yoon, Seunghyun, Kale, Ajinkya, Dernoncourt, Franck, Bui, Trung, and Bansal, Mohit. Fine-grainee image captioning with CLIP reward. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2022.
[10] Cobbe, Karl, Hesse, Chris, Hilton, Jacob, and Schulman, John. Leveraging procedural genera- tion to benchmark reinforcement learning. International conference on machine learning, 2020.
[11] Cobbe, Karl W, Hilton, Jacob, Klimov, Oleg, and Schulman, John. Phasic policy gradient. In International Conference on Machine Learning, 2021.
[12) Crowson, Katherine, Biderman, Stella, Kornis, Daniel, Stander, Dashiell, Hallahan, Eric, Castricato, Louis, and Raff, Edward. Vqgan-clip: Open domain image generation and editing with natural language guidance. In Computer Vision-ECCS 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, 2022.
[13] Cui, Yuchen, Niekum, Scott, Gupta, Abhinav, Kumar, Vikash, and Rajeswaran, Aravind. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, 2022.

---

[14] de Haan, Pim, Jayaraman, Dinesh, and Levine, Sergey. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, 2019.
[15] Di Langosco, Lauro Langosco, Koch, Jack, Sharkey, Lee D, Pfau, Jacob, and Krueger, David. Goal misgeneralization in deep reinforcement learning, In International Conference on Machine Learning, 2022.
[16] Ding, Yiming, Florensa, Carlos, Abbeel, Pieter, and Phiclipp, Mariano. Goal-conditioned imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
[17] Driess, Danny, Xia, Fei, Sajjadi, Mehdi SM, Lynch, Corey, Chowdhery, Aakanksha, Ichter, Brian, Wahid, Ayzaan, Tompson, Jonathan, Vuong, Quan, Yu, Tianhe, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03377. 2023.
[18) Duan, Yan, Andrychowicz. Marcin, Stadie, Bradly, Jonathan Ho, OpenAI, Schneider, Jonas, Sutskever, Ilya, Abbeel, Pieter, and Zaremba, Wojeiech. One-shot imitation learning. In Advances Neural Information Processing Systems, 2017.
[19] Fan, Linxi, Wang. Guanzhi, Jiang, Yunfan, Mandlekar, Ajay, Yang, Yuncong. Zhu, Haoyi, Tang. Andrew, Huang, De-An, Zhu, Yuke, and Anandkumar, Anima. Minedojo: Building open- open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206. 200.00853,, 2022.
[20] Finn, Chelsea, Yu, Tianhe, Zhang, Tianhao, Abbeel, Pieter, and Levine, Sergey. One-shot visual imitation learning via meta-learning. In Conference on Robot Learning, 2017.
[21] Gao, Peng. Geng, Shijie, Zhang, Renrui, Ma, Teli, Fang, Rongyao, Zhang, Yongfeng, Li, Hongsheng, and Qiao, Yu. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv: 10.00544 2021.
[22] Geng, Xinyang, Liu, Hao, Lee, Lisa, Schuurams, Dale, Levine, Sergey, and Abheel, Pieter. Multi- modal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. 14204, 2022.
[23] Ghosh, Dibya, Gupta, Abhishek, Reddy, Ashwin, Fu, Justin, Devin, Coline Manon, Eysenbach, Benjamin, and Levine, Sergey. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021.
[24] Goyal, Prasoon, Nickum, Scott, and Mooney, Raymond Using natural language for reward shaping in reinforcement learning. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 2385-2391, 2019.
[25] Hafner, Danijar, Lillicrap. Timothy, Fischer, Ian, Villegas, Ruben, Ha, David, Lee, Honglak, and Davidson, James. Learning latent dynamics for planning from pixels. In International conference on machine learning, 2019.
[26] Hessel, Jack, Holtzman, Ari, Forbes, Maxwell, Bras, Ronan Le, and Choi, Yejin. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
[27] Hill, Felix, Mokra, Sona, Wong, Nathaniel, and Harley, Tim. Human inslroctionnfollooiing with deep reinforcement learning via transfer-leamning from text. arXiv preprint arXiv:2005.09388 2020.
(28) Hu, Hengyuan and Sadigh, Dorsa. Language instructed reinforcement learning for human-ai coordination. In International Conference on Machine Learning, 2023.
[29] Huang, Wenlong, Abbeel, Pieter, Pathak, Deepak, and Mordatch, Igor. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
[30] Huang, Wenlong, Xia, Fei, Xiao, Ted, Chan, Harris, Liang, Jacky, Florence, Pete, Zeng, Andy. Tompson, Jonathan, Mordatch, Igor, Chebotar, Yevgen, Sermanet, Pierre, Jackson, Tomas, Brown, Noah, Luu, Linda, Levine, Sergey, Hausman, Karol, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. 6th Annual Conference on Robot Learning. 2022.
12

---

[31]
Huang, Zhiao, Liu, Fangchen, and Su, Hao. Mapping state space using landmarks for universal goal reaching. In Advances in Neural Information Processing Systems, 2019.
[32]
Hussein, Ahmed, Gaber, Mohamed Medhat, Elyan, Eyad, and Jayne, Chrisina. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[33] Islam, Riashat, Tomar, Manan, Lamb, Alex, Efroni, Yonathan, Zang, Hongyu, Didolkar, Aniket, Misra, Dipendra, Li, Xin, van Seijen, Harm, Combes, Remi Tachet des, et al. Agent- controller representations: Principled offline rl with rich exogenous information. arXiv preprint W.Xv02211.000144, 2022.
[34] James, Stephen, Ma, Zicong, Arrojo, David Rovick, and Davison, Andrew J. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, (2), 2020.
[35] James, Stephen, Wada, Kentaro, Laidlow, Tristan, and Davison, Andrew J. Coarse-to-fine g-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[36] Jang, Eric. Irpan, Alex, Khansari, Mohi, Kappler. Daniel, Ebert, Frederik, Lynch, Corey, Levine, Sergey, and Finn, Chelsea. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, 2022.
[37] Jeong, Jongheon, Zou, Yang, Kim, Taewan, Zhang, Dongqing, Ravichandran, Avinash, and Dabeer, Onkar. WinCLIP: Zero-/few-shot anomaly classification and segmentation. arXiv preprint arXiv:2303. 14814, 2023.
[38] Jiang. Yunfan, Gupta, Agrim, Zhang, Zichen, Wang, Guanzhi, Dou, Yonggiang, Chen, Yanjun, Fei-Fei, Li, Anandkumar, Anima, Zhu, Yuke, and Fan, Linxi. Vima: General robot manipulation with multimodal prompts. In International Conference Machine Learning, 2023.
[39] Kim, Changyeon, Park, Jongjin, Shin, Jinwoo, Lee, Honglak, Abbeel, Pieter, and Lee, Kimin. Preference transformer: Modeling human preferences using transformers for RL. In Interna- tional Conference on Learning Representations, 2023.
[40] Kwon, Minae, Xie, Sang Michael, Bullard, Kalesha, and Sadigh, Dorsa. Reward design with language models. In International Conference on Learning Representations, 2023,
[41] Lamb, Alex, Islam, Riashat, Efroni, Yonathan, Didolkar, Aniket Rajiv, Misra, Dipendra, Foster, Dylan J, Molu, Lekan P, Chari, Rajan, Krishnamurthy, Akshay, and Langford, John. Guaranteed discovery of control-endogenous latent states with multi-step inverse models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[42] Lee, Kimin, Lee, Kibok, Shin, Jinwoo, and Lee, Honglak, Network randomization: simple technique for generalization in deep reinforcement learning. In International Conference on Learning Representations, 2020.
[43] Lee, Kuang-Huei, Nachum, Ofir, Yang, Mengjiao Sherry, Lee, Lisa, Freeman, Daniel, Guadar- rama, Sergio, Fischer. Ian, Xu, Winnie, Jang, Eric, Michalewski, Henryk, et al. Multi-game decision transformers. In Advances in Neural Information Processing Systems, 2022.
[44] Lee, Youngwoon, Szot, Andrew, Sun, Shao-Hua, and Lim, Joseph J. Generalizable imitation learning from observation via inferring goal proximity. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. Wortman (eds.), Advances in Neural Information Processing Systems, 2021.
[45] Liang, Jacky, Huang, Wenlong, Xia, Fei, Xu, Peng, Hausman, Karol, brian ichter, Florence, Pete. and Zeng, Andy. Code policies: Language model programs for embodied control. In Workshop on Language and Robotics at CoRL 2022, 2022.
[46] Liu, Hao, Lee, Lisa, Lee, Kimin, and Abbeel, Pieter. coatuuctionfollovviin agents with jointly pre-trained vision- vision-language models. arXiv preprint arXiv:2210.133311 2022.
[47] Loshchilov, Ilya and Hutter, Frank. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
13

---

[48] Lynch, Corey and Sermanet, Pierre. arXiv:2005.07648, 40, 2020.
Grounding language in play.
arXiv preprint
[49] Lynch, Corey and Sermanet, Pierre. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005. FrX4:2200.006488, 2020.
[50] Lynch, Corey, Wahid, Ayzaan, Tompson, Jonathan, Ding, Tianli, Betker, James, Baruch, Robert, Armstrong, Travis, and Florence, Pete. Interactive language: Talking to robots in real time. arXiv preprint N0.0:22200006407,,22 2022.
[51] Ma, Yecheng Jason, Liang, William, Som, Vaidehi, Kumar, Vikash, Zhang, Amy, Bastani, Osbert, and Jayaraman, Dinesh. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, 2023.
[52] Ma, Yecheng Jason, Sodhani, Shagun, Jayaraman, Dinesh, Bastani, Osbert, Kumar, Vikash, and Zhang, Amy. VIP: Towards universal visual reward and representation via value-implicit pre-training, In International Conference on Learning Representations, 2023.
[53] Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, loannis, Wierstra, Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5600 2013.
[54] Nair, Suraj. Rajeswaran, Aravind, Kumar, Vikash, Finn, Chelsea, and Gupta, Abhinav. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.1260 12601,, 2022.
[55] Pathak, Deepak, Agrawal, Pulkit, Efros, Alexei A, and Darrell, Trevor. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
[56] Radford, Alec, Kim, Jong Wook, Hallacy, Chris, Ramesh, Aditya, Goh, Gabriel, Agarwal, Sandhini, Sastry, Girish, Askell, Amanda, Mishkin, Pamela, Clark, Jack, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[57] Reed, Scott, Zolna, Konrad, Parisotto, Emilio, Colmenarejo, Sergio Gomez, Novikov, Alexander, Barth-Maron, Gabriel, Gimenez, Mai, Sulsky, Yury, Kay, Jackie, Springenberg, Jost Tobias, et al. A generalist agent. arXiv preprint 2205::22000067555 2022.
[58] Ross, Stéphane and Bagnell, Drew. Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statistics, 2010.
[59] Schaal, Stefan. Learning from demonstration. Advances in Neural Information Processing Systems, 9, 1996.
[60] Seo, Younggyo, Hafner, Danijar, Liu, Hao, Liu, Fangchen, James, Stephen, Lee, Kimin, and Abbeel, Pieter. Masked world models for visual control. In Conference on Robot Learning, 2022.
[61] Seo, Younggyo, Lee, Kimin, James, Stephen L, and Abbeel, Pieter. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning. 2022.
[62] Seo, Younggyo, Kim, Junsu, James, Stephen, Lee, Kimin, Shin, Jinwoo, and Abbeel, Pieter. Multi-view masked world models for visual robotic manipulation. arXiv preprint 200.:23002004008, 2023.
[63] Shah, Dhruv, Osiński, Blažej. Levine, Sergey, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, 2023.
[64] Shah, Rohin, Varma, Vikrant, Kumar, Ramana, Phuong, Mary, Krakovna, Victoria, Uesato, Jonathan, and Kenton, Zac. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprint arXiv:2210.01790. 2022.
14

---

[65] Sharma, Pratyusha, Sundaralingam, Balakumar, Blukis, Valts, Paxton, Chris, Hermans, Tucker, Torralba, Antonio, Andreas, Jacob, and Fox, Dieter. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05 2022.
[66] Shridhar, Mohit, Manuelli, Lucas, and Fox, Dieter. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning. 2022.
[67] Shridhar, Mohit, Manuelli, Lucas, and Fox, Dieter. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference Robot Learning. 2023.
[68] Song, Xingyou, Jiang, Yiding, Tu, Stephen, Du, Yilun, and Neyshabur, Behnam. Observational overfitting in reinforcement learning. In International Conference on Learning Representations, 2020.
[69] Stepputtis, Simon, Camphell, Joseph, Phielipp, Mariano, Lee, Stefan, Baral, Chitta, and Ben Amor, Heni. Language-conditione imitation learning for robot manipulation tasks. In Advances Neural Information Processing Systems, 2020.
[70] Vinker, Yael, Pajouheshgar, Ehsan, Bo, Jessica Y, Bachmann, Roman Christian, Bermano, Amit Haim, Cohen-Or, Daniel, Zamir, Amir, and Shamir, Ariel. Clipasso: Semantically-awar object sketching. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022.
[71] Walmer, Matthew, Suri, Saksham, Gupta, Kamal, and Shrivastava, Abhinav. Teaching matters: Investigating the role of supervision in vision transformers. arXiv preprint arXiv:2212.03862, 2022.
[72] Winograd, Terry. Understanding natural language. Cognitive psychology, 3(1), 1972.
[73] Xiao, Tete, Radosavovic, Ilija, Darrell, Trevor, and Malik, Jitendra. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.066 220.00673, 2022.
[74] Yarats, Denis, Fergus, Rob, Lazaric, Alessandro, and Pinto, Lerrel. Mastering visual continuous control: Improved fata-augmented reinforcement learning. arXiv preprint arXi::2107.0%64 2021.
[75] Ying, Lance, Zhi-Xuan, Tan, Mansinghka, Vikash, and Tenenbaum, Joshua B. Inferring the goals of communicating agents from actions and instructions. arXiv preprint arXiv:2306. 16207, 2023.
[76] Yu, Tianhe, Finn, Chelsea, Xie, Annie, Dasari, Sudeep, Zhang, Tianhao, Abbeel, Pieter, and Levine, Sergey. Ohot imitation from observing humans via fomain-adaptio meta-learning. arXiv preprini arXiv: 1802.0157, 2018.
[77] Yu, Wenhao, Gileadi, Nimrod, Fu, Chuyuan, Kirmani, Sean, Lee, Kuang-Huei, Arenas, Montse Gonzalez, Chiang, Hao-Tien Lewis, Erez, Tom, Hasenclever, Leonard, Humplik, Jan, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.00 08647, 2023.
[78] Zeng, Andy, Wong, Adrian, Welker, Stefan, Choromanski, Krzysztof, Tombari, Federico, Purohit, Aveek, Ryoo, Michael, Sindhwani, Vikas, Lee, Johnny, Vanhoucke, Vincent, al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
[79] Zhang, Chiyuan, Vinyals, Oriol, Munos, Remi, and Bengio, Samy. A study on overfitting in deep reinforcement learning, arXiv preprint arXiv:1804.0689) 2018.
[80] Zhang, Renrui, Guo, Ziyu, Zhang, Wei, Li, Kunchang, Miao, Xupeng, Cui, Bin, Qiao, Yu, Gao, Peng, and Li, Hongsheng. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
15

---

Supplementary Material
Guide Your Agent with Adaptive Multimodal Rewards
A ARP Architecture Details
Transformer-based policy (ARP-DT) To train agents following adaptive multimodal reward signals in Procgen experiments, we introduce a retum-conditioned policy Te based on Decision Transformer architecture [8, 43]. Specifically, train decoder-only transformer to autoregressively model the following sequence:
(01, R1, a1, 02, R2, 02 .... or, Rr, QT)
Given the expert trajectory т, we first compute the target returns {[E H of expert demonstrations by computing the multimodal reward in Equation I, and we tokenize the sequence using embedding layer for each modality. Our model и parameterized by 8 comprises the following components:
Action decoder:
To (og. act. Rgt)
Return decoder: To Ru ost, act, Ret)
All tokens are fed into the causal transformer, and it produces output embeddings. The action decoder receives the embedding and predicts action å, and the return decoder takes the embedding for predicting multimodal return RG to encourage the agent to be aware of the multimodal return. Following Lee et al. [43], we train the model to predict not only the next action but also the next multimodal return by minimizing the objective below:
Σ (To(0elosi ace, Rg)),a)) + X MSE(TT((Rllosu Ret), R²) (6) 1
-0)
= ET>D
where CE is the cross entropy loss, MSE the mean squared error, and À is hyperparameter that adjusts the scale return prediction. We call ARP methods based on this architecture ARP-DT.
RSSM-based policy (ARP-RSSM) To demonstrate the versatility of our framework across different model architectures, we introduce eeturn-conditione policy based on world models [60, 62] for RLBench experiments. Specifically, implement the world model variant of the recurrent state-space model (RSSM; (25]) and condition it on multimodal return. The world model comprises the following components:
Encoder:
fo(s)18--1. Rs)
Ôr (ār St)
Decoder:
Rs po (Be (st)
Dynamics model:
8, po (st. sc-1, Oc-1)
Policy:
â ~ po p (ée(se)
The encoder extracts state 8, from previous state 8t-1. previous action dr-1, current observation os and the target multimodal return R, Σr= R(o,, x) with a recurrent architecture. The decoder reconstructs o, to provide learning signal for model states and predicts R, to encourage the agent to be aware of the multimodal return. The policy predicts action d, using state 8t. All model parameters 0 are optimized by minimizing the objective below:
((0) = EY~D
Σ TST
In po(z (st) - In (o, (s:) - In po (a)
(8)
KL (g) (St a1-1,
11,1 2t, Re) (o))ppp((((ss111.001
We refer to ARP methods based on this architecture ARP-RSSM.
16

---

B Extended Related Work
Training instruction-folloving agents Humans excel at understanding and utilizing language instructions to adapt to unfamiliar situations. Consequently, there has been significant interest in training policies that incorporate natural language in IL [48, 69, 36, 7] by learning policy conditioned on both current observation and the text instruction of the task. In parallel, recent studies have leveraged large language models (LLMs) for generating inner plans over pre-defined skills from natural language instructions for solving various robotic manipulation tasks [2, 45, 30, 27, 17]. Our method can be thought of as one of language-conditione conditioned imitation learning, which leverages natural language instructions reward signal by utilizzig the similarity between visual observations and natural language instructions in the pre-trained multimodal embedding space.
Task specification with text instructions Leveraging text instructions for task guidance is a common practice among humans. Building upon this concept, prior approaches have harnessed text instructions to direct agents in various ways, including the shaping of reward functions [24, 75, 77, 40, 51], addressing misbehavior correction [65], and exploring human-Al coordination [75, 28). The closest work to ours Ma al. [51], which extends the objective function of VIP [52] to include text instructions as goals for training visual- lanuuage aligned representations. The primary distinction that it uses static text representation concatenated with visual representations for policy learning. akin to other baselines [46, 67J. In contrast, our approach employs multimodal reward defined by measuring the similarity between image observations and text instructions in the pre-trained multimodal embedding space.
Utilizing CLIP [56] for supervision signals Recent work utilize CLIP scores [26] or CLIP- based perceptual loss [70] for improving image-text alignment in various domains including image generation [1. 12]. image captioning [26, 9]. and anomaly detection [37]. Similar to our approach,
some work [13, 19] have also leveraged CLIP scores as supervision signals to address reward-scarce tasks with reinforcement learning. Fan et al. [19] propose a video-language model that is pre-trained using large-scale, real-world videos paired with their transcripts, and it utilizes the similarity between video-text representations as the reward for reinforcement learning. In our study, we focus on the udaptability of the multimodal reward, which empowers the agent achieve desired goals in previously unseen test environments. Furthermore, we employ pre-trained multimodal representations without the need for resource- inttuccee pre-training. and we introduce fine-tuning scheme for better reward quality that can be easily implemented with a small set of in-domain demonstrations.
C Procgen Experiment Details
This section describes the details for implementing Adaptive Return eeturn-codiiiooeed Policy and provides our source code in the supplementary material.
Environment details We utilize publicly available implementation' to replicate the environments introduced by Di Langosco et al. [15]. We modify the simulator of the environments to render higher-resolution images to leverage pre-trained multimodal representations for both method and baselines. In this particular setup, the observations obtained from the environment at each timestep t comprise an RGB image with dimensions of 256 x 256 x 3 and natural language instruction delineating the desired goal. Throughout our experiments, we adhere to the hard environment difficully as described in [10]. Maximum episode length for all tasks is 500. To gather expert demonstrations used for training data, we train PPG [11] agents on 500 training levels for 200M timesteps per task using hyperparameters provided in Cobbe et al. [11]. For evaluation purposes, we assess the test performance on 1,000 different levels, encompassing previously unseen themes and goals that differ from those employed in training.
Both Instrac(RL. (Liu et al., 2022) and ARP employ ViT-B/16 the transformer-policy and pre-trained multimodal transformer encoder (M3AE: [22]) in all experi- ments, unless stated otherwise. For fine-tuning pre-trained multimodal encoders, we adopt the Adapter [21, 80] to effectively fine-tune the pre-trained CLIP embeddings without overfitting.
Architecture details
17

---

In detail, we attach extra linear layers to both the visual and text encoders and use the weighted sum of the oulput from the linear layers and the original pre-trained feature for computing visual/text repre- sentations. Throughout the training, we only apply gradients to the weight of these adapter layers and freeze both CLIPP' visual and textual encoders. Furthermore, we utilize multi- features obtained by concatenating intermediate layer representations with the final output representation as the input for the adapter layers, drawing inspiration from Liu et al. [46] and Walmer et al. [71]. Finally, the multimodal reward is computed using the cosine similarity between the multi-scale features from the image and text encoders. See Appendix E for qualitative results of our multimodal rewards. Inspired by Gao al. [21], attach an additional 2-layer MLP to the end of a pre-trained multimodal transformer encoder and use the weighted sum of the output from MLP and the original pre-trained feature for obtaining image/text representations. In the training phase, we apply gradients only the weight of these MI.P layers. Through empirical evaluation, we observe that this architecture yields superior performance in both our method and the baseline.
We resize images to 224 x 224 x 3 before computing multimodal rewards, and 256 x 256 3 RGB observations for training the cruut-coondiiooed policy. To stabilize training, we normalize multimodal returns following the method proposed by Chen et al. (2021), dividing them by 1000 in all experiments. We the AdamW optimizer (Loshchilov et al., 2018) with a learning rate of 5 x 10-1 and weight decay x 10-5 A cosine decay schedule utilized adjust the training learning rate. In CoinRun experiments, data augmentation techniques such as color jitter and random rotation are applied to the RGB images o, while maintaining alignment in the context. However, no augmentation is used to RGB images in Maze I/II experiments. For scaling the return prediction loss in training the return couun-coodiiioned policy, we set À = 0.01 in CoinRun experiments and À 0.001 in Maze I/II experiments. During the fine-tuning of the pre-trained multimodal encoder, a 2-layer MLP is attached to the end of both CLIP image and text encoders. An extra 2-layer MLP is added as an action prediction layer for the IDM objective. The model is trained for 20 epochs, and the one with the lowest validation loss is used for generating multimodal rewards. To scale the IDM loss in fine-tuning CLIP, we employ B = 1.5 in CoinRun experiments and /3 2.0 in Maze 1/ll experiments. In evaluation, we choose the target multimodal return as 90% quantile of the multimodal return from the dataset D* in all experiments.
Training details
Computation We use 24 CPU cores (Intel Xeon CPU @ 2.2GHz) and 2 GPUs (NVIDIA A100 40GB GPU) for training return-conditioned policy. The training of ARP for 50 epochs takes approxi- mately 4 hours for CoinRun experiments with the largest dataset size. For fine-tuning CLIP, we use 24 CPU cores (Intel Xeon CPU @ 2.2GHz) and GPU (NVIDIA A100 40GB GPU), which takes approximately 1.5 hours for Coinrun experiments.
Hypeparameters
We report the hyperparameters used in our experiments in Table 6.
Table 6: Hyperparameters of ARP-DT. Unless specified, we use the same hyperparameters used in InstructRL [46].
Hyperparameter
Value
Policy batch size
64
Policy epochs
50
Policy context length
Policy learning rate
0.000S
Policy optimizer
AdamW [47]
Policy optimizer momentum
81 0.9, 82 0.999
Policy weight decay
0.00005
Policy learning rate docay
Linear warmup and cosine decay (sco code for dctails)
Policy context length
Policy transformer size
2 layers. 4 heads, 768 units
Fine-tuned CLIP batch size
64
Fine-tuned CLIP epochs
20
Fine-tuned CLIP leamiing rate
0.0001
Fine-tuned CLIP weight decay
0.001
Fine-tunee CLIP adapter layer size
layers, 1024 units
Fine-tuned CLIP optimizer
AdamW [47] /g = 0.9, 8g = 0.999
Fine-tuned CLIP optimizer momentum
18

---

D RLBench Experiment Details
Environment details Visual observations obtained from RLBench environment each timestep t comprise an RGB image with dimensions of 256 x 256 3 and natural language instruction delineating the desired goal. Pick Up Cup task, we adjusted the environment to generate the target cup with different regions in training and evaluation. Specifically, we generate the target cup from 80% of the area of the table in training environments. In evaluation, we generate the cup from remaining 20% of the area of the table. We choose an action mode of RLBench that specifies the delta of joint positions for all experiments.
Training details For training data, we collect 100 expert demonstrations using scripted policy provided by RLBench simulator. For each demonstration, we extract the keypoints [35] and train the agent output the relative change in (x, y. 2) position between keypoints. We train the agents to output the prediction of (x.y.)) position changes.
set the maximum episode length to 500. We closely follow the implementation details and use the same hyperparameters deseribed in the imitation learning experiments in MV-MWM [62]. For all experiments, we utilize the open-sourced pre-trained CLIP model with ViT-B/16 architecture from huggingface transformers library and fine-tune CLIP based on that model. We train our method for 100K iterations and MV-MWM for 200K iterations until convergence. We apply data augmentation in training, including color jittering over RGB images. In evaluation, we choose the target multimodal return as 50% quantile of the multimodal return from training demonstrations in all experiments.
E Qualitative Results of Multimodal Rewards
In Figure 8, 9, 10, we present the curves of multimodal rewards for frozen/fine-tur CLIP in the trajectories from training/held-out evaluation environments. We find that the multimodal reward exhibits an overall increasing trend as the agent approaches the goal in both frozen and fine-tuned CLIP, irrespective of the training and held-out evaluation environments. Furthermore, we observe that fine-tuned CLIP not only induces reward that is temporally smoother in the intermediate stages compared to frozen CLIP (see Figure 1) but also demonstrates a steeper upward reward curve (see Figure 9, 10). These results support the claim that the quality of multimodal rewards from the fine-tuned CLIP outperforms those from the frozen CLIP (Section 4.1). Video examples of the trajectories are provided in the supplementary material.
19

---

1.0
Multimodal Reward
0.8
0.6
0.4
0.2
Frozen CLIP
Fine-tuned CLIP
0.0
0
20
40
60
80
Timestep
(a) Multimodal reward curve in the training environment.
1.0
Multimodal Reward
0.8
4
0.6
2
0.4
0.2
Frozen CLIP
Fine-tuned CLIP
0.0
20
40
60
100
Timestep
(b) Multimodal reward curve in the held-out evaluation environment.
Figure 8: Qualitative results of multimodal rewards in CoinRun environments.
20

---

1.0
Multimodal Reward
0.8
0.6
0.4
0.2
Frozen CLIP
Fine-tuned CLIP
0.0
0
20
40
60
80
Timestep
2
(a) Multimodal reward curve in the training environment.
1.0
Multimodal Reward
0.8
0.6
2
0.4
0.2
Frozen CLIP
Fine-tuned CLIP
0.0
10
20
30
40
50
70
Timestep
(b) Multimodal reward curve in the held-out evaluation environment.
Figure 9; Qualitative results of multimodal rewards in Maze I environments,
21

---

1.0
Multimodal Reward
0.8
0.6
2
0.4
0.2
Frozen CLIP
Fine-tuned CLIP
0.0
20
40
60
80
Timestep
2
(a) Multimodal reward curve in the training environment.
1.0
Multimodal Reward
0.8
1
2
3
0.6
0.4
0.2
Frozen CLIP
Fine-tuned CLIP
0.0
10
30
40
50
Timestep
(b) Multimodal reward curve in the held-out evaluation environment.
Figure 10: Qualitative results of multimodal rewards in Maze II environments.
22

---

Additional Experiments
Effect of return prediction We investigate the effect of including the return prediction loss M, R7)) in Equation 6, which encourages policy to be more aware conditioned re-
Table 7: Ablation study of the return prediction loss MSE(R, R₁) in CoinRun environments.
MSL Re. RG)
L
Train (9)
Test (56
turns. In Table 7, we observe that the performance of ARP-DT becomes much more sensitive to the quality of multimodal rewards when trained with the return prediction loss. For instance, without
90.28%
+221%
56.32%
1.55%
/
92.01 %
1,98
56.28%
7.01%
89.58%
2.08%
63.32%
2.015
90.28%
1.9996
72.36%
1.48%
the return prediction loss, the evaluation perfor- mance becomes almost the same with or without the fine-tuning scheme, which suggests that the model is insensitive the quality of rewards. On the other hand, with the prediction loss, the performance increases as the quality of reward improves. This implies that the model becomes aware of the returns and can thus follow the adaptive signal from the multimodal reward.
Effect scaling return prediction loss We in- vestigate how the coefficient X, which determines the weight of the return prediction loss in training retur-conditioned policy, affects the performance of ARP-DT. To this end, we test various values of in CoinRun environments. Table 8 shows the perfor- mance of ARP in training/hell--ou evaluation envi- ronments with different A. We find that performance is not significantly different according to the value of À in the held-out evaluation environments. These results indicate that ARP is robust to the choice of hyperparameter A.
Table 8: Ablation studies of the hyperparame- ter À in CoinRun environments.
LeT
Train (%)
Test (%)
0.001
X
89.93% + 1.94
62.65%
+ 10.12%
85.42% + 1.80%
71.69%
5.71%
0.01
X
89.58 %.%
2.0%%
63.32%
2.18%
90.28%
1.59%
72.36%
3.48%
0.1
87.15%
+: 2.62%
62.65%
+: 10.31%
85.76%
3.18%
73.37%
3.48%
1.0
X
87.15%
4.70%
61.64%
6.38%
81.25%
+ 1.04%
73.37%
2.66%
Training
100
Normalized Score (%)
20
CoinRun
Maze
ColnRRn
Maze
Maze II
Random
Inst
Figure 11: Expert-normalized scores on training/evaluationn environments of ARP trained using multimodal rewards generated with (i) instructive text (i.e., Inst) and (ii) random (i.e., Random). The result shows the mean and standard deviation averaged over three runs.
Ablation studies on text instructions To investigate whether ARP-DT makes decisions based on the adaptive signal from the multimodal reward, we evaluate the quality of rewards generated with (i) instructive text (i.e,, Inst) and (ii) random text (i.e., Random) in three different environments. Specifically, we use a natural language instruction for each environment, as described in Section 4 for Inst, and "NeurIPS 2023 will be held again at the New Orleans Ernest N. Morial Convention Center" for Random. We find that using random text instructions significantly declines performance both training and evaluation environments. These results highlight the importance of using the instructive text and demonstrate that ARP-DT indeed depends on the adaptive signal from the multimodal reward for solving tasks at deployment time.
23

---

G Limitation and future work
One limitation of our work that we currently rely on single image-text pair compute the multi- modal reward at every timestep t. Although our approach has shown effectiveness both quantitatively and qualitatively, there are tasks where rewards depend on the history of past observations (i.e., non-Markovian) [4, 5, 39]. To address this limitation, it would be valuable to explore the extension of our method to incorporate video-text pairs for calculating multimodal rewards. Another aspect to consider is that the tasks we have examined so far are relatively simple, as they involve only single condition for success. tackle more complex problems, we are interested investigating approaches that leverage large language models [29, 30, 17] with method.
H Potential negative impacts
We do not anticipate significant negative societal impacts in that our method is now limited to applying in simulated environments. However, if method is applied in real-world scenarios, privacy concerns may arise, considering that behavior cloning agents used in such applications, like autonomous driving [63] real-time control [7, 17], require large amounts of data, which often contain controversial information. Additionally, behavior cloning policy presents challenge as imitates specified demonstrations, potentially including undesirable actions. If some evil actions included in expert demonstrations (e.g., behaviors that may be violent to the pedestrians are contained in the training data for mobile manipulation tasks), the policy could have significant negative impacts on users. To address this concern future directions should focus on developing agents with safe adaptation in addition performance enhancement efforts.
24


Reviews:
[{"review_id": "KpZ70mwdCa", "reviewer": "Reviewer_qrbz", "summary": "This work tackles the goal misgeneralization problem in goal-conditioned RL agent setups. They propose to leverage pre-trained multimodal models, i.e., CLIP, to serve as a reward function that simultaneously estimate both text and image modalities for deciding an engineered reward for each time step. The authors train a transformer-based model to take a sequence of encoded state representation, estimated reward from CLIP, and historical actions, to predict the next reward and state representations for learning a return-conditioned policy. Additionally, to encourage smoother and more robust reward functions, they adopt value implicit pre-training and inverse dynamic model to incentivize better adapted rewards. The experimental results on OpenAI Procgen benchmarks demonstrate better generalization abilities over baselines without the proposed reward engineering scheme.", "questions": "- Why don’t you choose tasks and/or domains that are closer to CLIP’s training distribution (i.e., more real-world imageries), such as some egocentric embodied tasks or navigational agent?\n- Do you think your method, or perhaps, CLIP-based rewards, are trustworthy enough to generalize well and replace the actual environmental rewards? I.e., can we assume we can simply adopt the engineered rewards and disregard the environmental success signals to train an agent and it still performs (maybe almost) on par with that using the environment reward. (I guess one will need to further train a termination signal from CLIP, too.)", "limitations": "- The authors did not explicitly point out the limitations of this work. I suggest the authors discuss more on this matter in both their proposed ideas and the execution of the experiments.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Utilizing CLIP as an adaptive reward function is sound and easy to implement.\n- The proposed idea is neat and should be transferable or up-scalable to more complex tasks and environments.\n- The adoptions of both VIP and IDM are to the point for the proposed return-conditioned policy training.\n- The reward evaluation protocols are well-motivated and justified.", "weaknesses": "- While the experimental setups are solid, the adopted environments (testbed tasks) are far from real-world imageries, and the adaptation may not utilize very well (and much) what CLIP has learned during the pre-training. More challenging environments are needed to justify whether the multimodal rewards that coming out from a pre-trained models are indeed useful. I.e., a valid baseline is to just train a small sizable multimodal transformer (with both VIP and IDM applied) and see if the pre-training alignment is indeed that important/useful.\n- To really (and effectively) showcase the generalization ability, I suggest experiments that are conducted on generalizing to unseen (but perhaps visually similar) tasks as a testbed for the multimodal rewards. Using the testbeds this work adopt, for example, Maze I transferring to Maze II would be an interesting and insightful experimental setup.\n- Literature reviews: using CLIP scores or CLIP-based perceptual loss [1] has been quite popular these days. The author should provide more in-depth discussion on related works along using pre-trained multimodal models as a strong supervision signal.\n\n[1] Vinker, Yael, et al. \"Clipasso: Semantically-aware object sketching.\" ACM Transactions on Graphics (TOG) 41.4 (2022): 1-11.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer qrbz,\n\nWe sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n---\n\n**[W1] Effect of pre-trained models.**\n\nTo verify the effectiveness of pre-trained multimodal representations, we compare our method with agents using multimodal rewards obtained from a smaller-scale multimodal transformer, which was trained from scratch using VIP and IDM objectives, denoted as MRDT+ (scratch).  Figure 3 in global response shows a significant decrease in performance for MRDT+ (scratch) when compared to MRDT+ across all environments, particularly in the training performance within Maze environments. These findings highlight the crucial role of pre-training in improving the efficacy of our multimodal rewards.\n\n---\n\n**[W2] Transferring tasks for proving generalization ability.**\n\nFollowing your suggestion, we have evaluated agents trained using demonstrations from Maze I environments, where the goal is to reach the yellow cheese, in unseen Maze II environments, where the target object changes to a yellow line. Note that the only difference is that the agent now needs to reach a different object and it receives different natural language instruction. As depicted in the table presented below, we observe that our method outperforms the baseline in this unseen transfer scenario. These results indicate that our approach effectively guides agents towards desired goals in unseen environments, even when there is a change in the accompanying text command.\n\n| Normalized Score | Maze II (Test)  |\n|------------------|-----------------|\n| InstructRL       | 43.36% +- 1.36% |\n| MRDT (Ours)      | 47.88% +- 6.11% |\n| MRDT+ (Ours)     | 49.23% +- 2.82% |\n\n---\n\n**[W3] Literature review for recent work utilizing CLIP scores / CLIP-based perceptual loss.**\n\nFollowing your suggestion, we will revise our manuscript with a more comprehensive literature survey of utilizing CLIP scores.\n\nRecent work utilize CLIP scores [1] or CLIP-based perceptual loss [2] for improving image-text alignment in various domains including image generation [3,4], image captioning [1, 5], and anomaly detection [6]. Similar to our approach, some work [7,8] have also leveraged CLIP scores as supervision signals to address reward-scarce tasks with reinforcement learning. Fan et al. [8] proposes a video-language model that is pre-trained using large-scale, real-world videos paired with their transcripts, and it utilizes the similarity between video-text representations as the reward for reinforcement learning.\n\nIn our study, we focus on the adaptability of the multimodal reward, which empowers the agent to achieve desired goals in previously unseen test environments. Furthermore, we employ pre-trained multimodal representations without the need for resource-intensive pre-training, and we introduce a fine-tuning scheme for better reward quality that can be easily implemented with a small set of in-domain demonstrations.\n\n---\n\n**[Q1] Selecting tasks close to CLIP’s training distribution.**\n\nFollowing your suggestion, we evaluated our method in RLBench, which provides more realistic visual observations compared to Procgen. We observe that our method (MR-RSSM+, 50.93%) significantly outperforms the baseline (MV-MWM, 20.37%) in the unseen test environment. Please refer to the [GR1] in the global response. \n\n---\n\n**[Q2] Comparison between CLIP-based rewards and actual task rewards.**\n\nIn our study, we have demonstrated the enhanced generalization performance of our proposed method with only natural language instructions in various environments. However, our approach may exhibit limitations when confronted with non-Markovian states. Since the computation of current multimodal rewards relies on the representations of image-text pairs, it might not effectively capture complex spatio-temporal relationships. Extending our method to incorporate video-text pairs for computing multimodal reward would be an interesting future direction for solving more complex environments characterized by extended horizons and multiple objectives\n.\n\n---\n\n**[L1] Clarification on the limitations of our work.**\n\nWe discuss some limitations of our work in Appendix F. Apologies for the confusion. We will move it to the main text in the final version.\n\nA primary limitation of our method is its current reliance on a single image-text pair to compute the multimodal reward. This design could potentially be a limitation in dealing with complex tasks that demand an understanding of spatio-temporal relationships across past observations. An intriguing avenue for future research would involve extending our methodology to incorporate video-text pairs by utilizing pre-trained video-text multimodal representations. \n\nFrom an experimental perspective, our approach does exhibit some weaknesses. For instance, we introduce certain hyperparameters for fine-tuning pre-trained multimodal representations (e.g., $\\gamma$ and $\\beta$ as discussed in Section 3.3 of the submitted draft) that depend on the specific task. Additionally, the quality of the multimodal reward may prove sensitive to the choice of prompts.\n\n\n---\n\n**References**\\\n[1] CLIPScore: A Reference-free Evaluation Metric for Image Captioning, EMNLP 2021.\\\n[2] CLIPasso: Semantically-Aware Object Sketching, ACM Transactions on Graphics (TOG), 41(4):1-1, 2022.\\\n[3] StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators, ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.\\\n[4] Vqgan-clip: Open domain image generation and editing with natural language guidance, ECCV 2022.\\\n[5] Fine-grained image captioning with CLIP reward, NAACL 2022.\\\n[6] WinCLIP: Zero-/few-shot anomaly classification and segmentation, CVPR 2023.\\\n[7] Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?, L4DC 2022.\\\n[8] Minedojo: Building open-ended embodied agents with internet-scale knowledge, NeurIPS 2022.\n"}, {"review_id": "YX5PCFpdRK", "reviewer": "Reviewer_9gB5", "summary": "This paper presents a framework called Multimodal Reward Decision Transformer (MRDT) that uses the visual-text alignment score from pre-trained vision-language models (after careful fine-tuning using the in-domain data) as the reward signals in visual-based reinforcement learning. Specifically, the authors propose to train a return-conditioned policy based on Decision Transformer (DT) with the improved reward signals learned by the tuned VLMs. The method is shown to generalize better than without the multimodal reward strategy in environments with unseen goals partly due to the knowledge captured in pre-trained models.", "questions": "Need to also discuss some other existing relevant literature on this subject, e.g.,\n\n[1] Reward Design with Language Models\n\n[2] LIV: Language-Image Representations and Rewards for Robotic Control", "limitations": "The major limitation is the lack of adequate evaluations. The approach is an otherwise promising one.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method leverage pre-trained VLMs in a simple yet effective way for reward learning.\n+ This is an important direction to study foundational models for decision-making. Compared to directly using pre-trained visual embedding models, this strategy of reward learning is more promising so far, IMO.", "weaknesses": "- Limited experiments. The authors only perform their studies on three tasks in ProcGen, far from most published work on this subject.\n- Due to the limited experiments, it might require higher technical novelties. However, the novelty is also limited since the proposed method is built on top of several existing methods (VIP, IDM, CLIP-Adapter, etc.)\n\nI am open to raising my score given more experimental evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9gB5,\n\nWe sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n---\n\n**[W1] Limited experiments.**\n\nIn response to your concern, we have investigated the efficacy of our method within RLBench, which serves as a standard benchmark for visual-based robotic manipulations. We observe that our method can guide the agent to achieve unseen goals also in robotic manipulation tasks requiring fine-grained controls by incorporating our fine-tuning schemes. e.g., MR-RSSM+ achieves a performance of 50.93%, whereas MV-MWM achieves only 20.37%. Please refer to the [GR1] in the global response.\n\n---\n\n**[W2] Discussion over existing relevant literature.**\n\nWe would like to highlight that LIV [1] is a concurrent work to ours in that it was made public after the submission deadline of NeurIPS 2023. LIV shares similarities with our approach by extending the objective function of VIP [2] to include natural language instructions as goals for training visual-language aligned representations. The primary distinction lies in LIV's use of static text representation, concatenated with visual representations, for policy learning, akin to other baselines (e.g., InstructRL [3]). In contrast, our approach introduces a novel imitation learning framework, employing image-text alignment in the pre-trained multimodal embedding space as a reward to train a return-conditioned policy.\n\n\"Reward Design\" [4] is also relevant to our research as it specifies a proxy reward function using pre-trained representations. Notably, it differs from our work in that it utilizes Large Language Model (LLM) to define the proxy reward function through prompting instructions. In contrast, our method capitalizes on the similarity between text instructions and current visual observations within the pre-trained multimodal embedding space for the reward signal. Furthermore, while \"Reward Design\" converts LLM responses into binary rewards, our method employs adaptive reward signals that dynamically adjust at each timestep to guide agents in previously unseen environments.\n\nWe will enhance the depth of our discussions in the final draft by incorporating relevant papers that explore the utilization of language instructions for shaping reward functions [5, 8, 9], addressing misbehavior correction [7], and investigating human-AI coordination [6, 10].\n\n---\n\n**References**\\\n[1] LIV: Language-Image Representations and Rewards for Robotic Control, ICML 2023.\\\n[2] VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training, ICLR 2023.\\\n[3] Instruction-Following Agents with Multimodal Transformer, ArXiv 2023.\\\n[4] Reward Design with Large Language Models, ICLR 2023.\\\n[5] Language to Rewards for Robotic Skill Synthesis, ArXiv 2023.\\\n[6] Inferring the Goals of Communicating Agents from Actions and Instructions, Theory-of-Mind Workshop @ ICML 2023.\\\n[7] Correcting robot plans with natural language feedback, ArXiv 2022.\\\n[8] Using natural language for reward shaping in reinforcement learning, IJCAI 2019.\\\n[9] Inferring rewards from language in context, ACL 2022.\\\n[10] Language instructed reinforcement learning for human-ai coordination, ICML 2023.\n"}, {"review_id": "qJBvYvtM9W", "reviewer": "Reviewer_nhjK", "summary": "The authors propose a formulation of the reward-conditioned decision transformer but do so using a learned reward function estimated by a pretrained CLIP style model which maps expert demonstrations and textual descriptions of these demonstrations into the same domain. They evaluate their method on the coinrun and Maze navigation domains. \n", "questions": "Please help me understand how to distinguish goal misgeneralization is different from overfitting to training tasks. Further, it's not clear to me how adding language prevents overfitting. Consider an extreme case, where the dataset of demonstrations only contains trajectories moving towards the coin placed in the exact same location. How does adding language prevent overfitting here? The language label here will still be the same across trajectories. It's unclear to me why language would prevent overfitting.  ", "limitations": "The paper evaluation is a bit tenuous and I would encourage expanding the scope of the experiments. Currently, a lot of the experiments are geared towards evaluating the reward. Since a pretrained model, i.e., CLIP is being used to generate the rewards, it would be useful for readers to know how much of a domain alignment is needed for this work and to what extent the method breaks when trained on vastly OOD environments. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method is principled and the intuitive: The authors describe how to finetune existing language and vision models and use alignment between the observation and the text a reward for a decision transformer.\n2. The method figures are clear and easy to understand (except Fig1 see weaknesses).\n", "weaknesses": "1. Example poorly motivated: How is goal misgeneralization any different from policies overfitting to training environments? Figure 1 needs improvement since the whole paper is motivated by it and currently it is impossible to tell what is happening in the 2 panes.\n2. Comparison with baselines: The idea of using CLIP for reward generation is not entirely new and Zest https://arxiv.org/pdf/2204.11134.pdf is a very close related work published about a year ago. Comparing to this paper is a natural baseline.\n3. Generalization across language commands : Does the method enable generalization using language commands? The only experiments are demonstrated are domain generalization (unclear how OOD the test tasks are in comparison to the train).\n4. Unclear how the method prevents goal misgeneralization: While the introduction motivates the work from this problem, the experimental section does not lend any credence to the fact that this method would prevent such a problem. Experiment\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nhjK,\n\nWe sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n---\n\n**[W1] Clarification of Figure 1 in the submitted draft.**\n\nFigure 1 in the submitted draft presents an illustrative example of goal misgeneralization. In this scenario, an agent trained to collect a coin consistently positioned in the far right corner (left) might learn a behavior that directs it toward the far right corner of the level, even if the actual task reward comes from collecting the coin. As a result, the agent achieves a low task return when the location of the coin is randomized by heading to the far right of the level (right). \n\n---\n\n**[W2] Comparison with other baseline (ZeST).**\n\nFollowing your suggestion, we have devised a variant of our method in which the reward is computed based on the goal image, opposed to utilizing natural language instructions, denoted as GC-DT. It is considered as an improved version of ZeST + DT [1] in that it utilizes M3AE representations and additionally optimizes return-prediction objective. Despite not requiring any information from the test environment, MRDT shows comparable generalization performance to goal-conditioned baselines. For more details, please refer to [GR2] in the global response.\n\n---\n\n**[W3-1] Generalization across language commands.**\n\nWe would like to remark that we investigated how language command affects the generalization performance in Section 4.3 of the submitted draft. Specifically, our observations indicate that the utilization of instructive text results in significantly improved generalization performance compared to employing random text. \n\nNonetheless, following your suggestion, we have evaluated agents trained using demonstrations from Maze I environments, where the goal is to reach the yellow cheese, in unseen Maze II environments, where the target object changes to a yellow line. Note that the only distinction is that the agent now needs to reach a different object and receive different natural language instructions. As depicted in the table presented below, we observe that our method outperforms the baseline, demonstrating that our method achieves enhanced generalization through the understanding of language commands.\n\n| Normalized Score | Maze II (Test)  |\n|------------------|-----------------|\n| InstructRL       | 43.36% +- 1.36% |\n| MRDT (Ours)      | 47.88% +- 6.11% |\n| MRDT+ (Ours)     | 49.23% +- 2.82% |\n\n---\n\n**[W3-2] How do out-of-distribution (OOD) test environments compare to the training environments?**\n\nRegarding the test environment, our evaluation encompasses not only unseen configuration of the map but also previously unseen goals (i.e., different goal position or goal object) that differ from those used in training.\n\n---\n\n**[W4, Q2] How does our method prevent goal misgeneralization?**\n\nAs you pointed out, prior approaches [2,3] have a drawback that the agent receives the same signal at every timestep from text instruction, manifested as single hidden embedding from pre-trained representation space. In contrast, our method computes the multimodal reward between current image observation and text instruction at every timestep. This reward dynamically adjusts based on visual observations, enabling it to guide the agent to achieve the goal even in unseen test environments (see Figure 9, 10, 11 in the submitted draft). Subsequently, we train a return-conditioned policy using demonstrations with reward labels, which makes decisions based on the adaptive multimodal reward signal. As a result, our approach facilitates an enhanced ability to generalize across different goals and effectively prevents goal misgeneralization within test environments. Please refer to Figure 4 in the attached pdf for visual descriptions.\n\n---\n\n**[Q1] Distinguishing goal misgeneralization with overfitting to training tasks.**\n\nGoal misgeneralization [4,5] is a special case of overfitting to training tasks. It arises when an agent is overfitted to a misleading goal that isn't connected to the actual source of the true task reward, resulting in the agent achieving a low task return in the test environment.\n\n---\n\n**[L1-1] Degree of domain alignment required for pre-trained models**\n\nTasks demanding delicate control, such as manipulation, would require a more precise multimodal reward achieved through fine-tuning of pre-trained multimodal representations. For instance, as shown in Figure 1 in the attached pdf, we observe that MR-RSSM, employing frozen CLIP representations, demonstrates similar performance to baseline in RLBench. On the other hand, MR-RSSM+, which integrates our fine-tuning scheme, achieved significantly enhanced performance in test environments. This outcome indicates that our fine-tuning scheme becomes more crucial in complex tasks like robotic manipulations.\n\n---\n\n**[L1-2] Condition for breaking our method.**\n\nGiven that our multimodal reward heavily relies on CLIP image-text pair representations, it might encounter challenges in handling language instructions that involve complex spatio-temporal relationships and relative positions of multiple objects., e.g., Pick up the leftmost of the three cups in RLBench environments. For solving this problem, extending our approach to incorporate video-text pairs for multimodal reward computation would be an interesting future direction.\n\n---\n\n**References**\\\n[1] Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?, L4DC 2022.\\\n[2] Instruction-Following Agents with Multimodal Transformer, ArXiv 2023.\\\n[3] Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation, CoRL 2022.\n[4] Goal Misgeneralization in Deep Reinforcement Learning, ICML 2022.\\\n[5] Goal Misgeneralization: Why Correct Speciﬁcations Aren’t Enough For Correct Goals, ArXiv 2023.\\\n"}, {"review_id": "TGTVv1jWR6", "reviewer": "Reviewer_HCX5", "summary": "The paper proposes an interesting and novel idea of using large pretrained mulitmodal models to compute image-text alignment \nscore to use as a reward to train return-conditioned policies using a Decision Transformer (DT). They propose a novel \nimitation learning framework called Multimodal Reward Decision Transformer (MRDT) which trains a return-conditioned policy\nusing adaptive reward signals from image-text multimodal rewards.\n\nThe authors also propose a fine-tuning scheme that uses VIP and IDM objective to fine-tune using CLIP-Adapter using data from in-doman demonstrations to improve performance.\n\nThe paper also shows MRDT successfully improves generalization to unseen levels and unseen goals on 3 environments from OpenAI procgen benchmark. \n\nAuthors also present an analysis on how MRDT helps mitigate goal misgeneralization and present a set of metrics to evaluate the quality of multimodal rewards.", "questions": "1. The proposed method is interesting and a novel but intuitive idea to leverage pre-trained multi-modal models as rewards for learning a policy. \nThe idea is novel and seems promising, the main concern I have is on the breadth of the experiment section. It'd be nice if authors can show results on slightly\n more complex environments to support the effectiveness of MRDT. I'd be open to updating my rating if authors can demonstrate the effectiveness of MRDT on different benchmarks/tasks.\n2. As outlined in the weakness section, it would be great if authors can discuss more about why the reward spikes initially for finetuned CLIP model as shown in the analysis in appendix B.\n3. I'd appreciate if the authors can also discuss the goal misrepresentation problem I describe in point 3 of the weakness section.", "limitations": "1. The experiments section seems lacking at the moment and could benefit from evaluation on different benchmarks\n2. Multi-modal rewards post finetuning exhibit unexpected behavior which could prove to be a big problem when finetuning policies \nusing these rewards. These rewards can sometimes misrepresent the value of the state agent is in.\n3. It is not clear how well the approach would work in more complex tasks with partial observability and non-markovian states. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Paper is well-written and easy to follow\n2. The proposed approach is interesting and is shown to be effective on the OpenAI Procgen benchmark when generalizing to \nunseen levels and unseen goals. It outperforms prior work InstructRL on all 3 environments by a decent margin.\nThe results also show that finetuning using proposed fine-tuning scheme further improves the results.\n3. The analysis on quality of multimodal rewards is quite informative. It gives a clear insight on how the multimodal rewards differ \nfor different cases . The analysis shows finetuned multimodal rewards better capture distinction, distance (S) and distance(F). \n4. The analysis in figure 8 shows the rewards generated with instructions aligned with the goal state gets higher reward compared \nto a random instructions. This demonstrations a pretrained multimodal model like CLIP can give good reward signal to the policy.", "weaknesses": "1. The paper proposes an interesting idea but the experiment section is lacking breadth. I'd appreciate if authors consider evaluating their \nmethod on multiple benchmarks which have multi-modal input. Similar to InstructRL paper authors can consider a subset of tasks from RLBench \nfor evaluation of the proposed method. The environments in ProcGen are quite simple and the generalization is only being tested for\n unseen instantiations of a single task (for 3 environments). Does MRDT work well if we want a single policy that can achive different goals? \nOne example is Object Navigation problem (but in simpler grid world environments), where the agent has to navigate to single instance of one of the n target object categories i.e. Find a chair/Find a sofa.\n It'd be nice if authors can add more experiments from different benchmarks to the paper.\n\n2. It seems like for all 3 environments in ProcGen benchmark a better reward can be cosine between image embeddings of current state vs the expected goal state. \nHave authors tried using this simple baseline? Does this lead to better or worse results than MRDT? The concern I have is that these environments are quite simple \nand might not require multi-modal rewards. \n\n3. Does using multimodal rewards from CLIP like model leads to goal misrepresentation problem? For example, let's consider a augmented version of Maze II environment. \nIf in addition to the yellow gem and diagonal line there was a straight red/yellow line and the task was \"\"Navigate a maze to collect the diagonal line\"\" or \"\"Navigate a maze to collect straight line\"\",\n how would all the baselines perform? Are these multi-modal rewards capable of clearly distinguishing these similar looking goals? or this leads to agents learning a average policy where it\n sometimes confuses the straight line with diagonal line because of misrepresented reward signal?\n\n4. The qualitative results in appendix B are quite interesting and I appreciate authors added these results in the supplementary. In Figure 9, for the coinrun task the \nFine-tuned CLIP rewards are almost as high as the reward that agent achieves at the goal state. This seems concerning and hints towards multi-modal rewards not being able to distinguish \ngoal state from a random subset of states. This could lead to policy learning arbitary behaviors or not learning the task at all if trained/finetuned using RL instead of imitation learning. \nDo authors have any insights on why the finetuned reward model show unexpected behavior? And how can this be mitigated so that we can successfully use these rewards for online training/finetuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer HCX5,\n\nWe sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n---\n\n**[W1, L1] Limited experiments.**\n\nIn response to your concern, we have investigated the efficacy of our method within RLBench in global response. As shown in Figure 1b in the attached pdf, we observe that our method can guide the agent to achieve unseen goals also in robotic manipulation tasks requiring fine-grained controls by incorporating our fine-tuning schemes. e.g., MR-RSSM+ achieves a performance of 50.93%, whereas MV-MWM achieves only 20.37%. Please refer to [GR1] in the global response for more details.\n\nIn terms of object navigation problems you suggested, it would be an interesting future direction to investigate the effectiveness of our framework in multi-task setups.\n\n---\n\n**[W2] Comparison with goal-conditioned baselines.**\n\nThank you for your intriguing question. First of all, it is important to note that the suggested baseline relies on additional information from the test environment because it assumes availability of a goal image (or state) during the test time. On the other hand, our method solely relies on natural language instruction and does not require any extra access to the test environment. Despite not requiring any additional information, we observe that our method (MRDT) achieves comparable generalization performance to goal-conditioned baselines (GC-DT). Please refer to [GR2] in the global response.\n\n---\n\n**[W3, Q3, L2] Goal misrepresentation problem that can be caused by using multimodal rewards.**\n\nThank you for pointing this out. To address your point, we have evaluated agents trained using demonstrations from Maze II environments, where the objective is to collect the yellow line (see Figure 5a in the attached pdf), in an augmented version of Maze II test environments. Specifically, we place a yellow gem, a red diagonal line, and a red straight line in the random position of the map, and instruct the trained agent to reach the red diagonal line. We denote this environment as Maze III. The goal of this evaluation is to verify that our multimodal reward clearly distinguishes similar looking distractors and guides the agent to the correct goal. Please refer to Figure 5c in the attached pdf for image observation of Maze III.\n\nAs depicted in the table presented below, we observe that our method outperforms the baseline also in Maze III. This result demonstrates that our multimodal reward can provide adaptive signals for achieving the goal by distinguishing distractors using natural language instructions.\n\n| Normalized Score | Maze II (Test)      | Maze III (Test)       |\n|------------------|-----------------|-----------------|\n| InstructRL       | 21.90% +- 3.30% | 21.21% +- 1.52% |\n| MRDT (Ours)      | 26.67% +- 5.41% | 33.33% +- 4.01% |\n| MRDT+ (Ours)     | 40.95% +- 2.97% | 38.38% +- 3.15% |\n\n---\n\n**[W4, Q2, L2] Unexpected behavior of the fine-tuned reward model.**\n\nThe unexpected behavior observed in multimodal rewards in fine-tuned CLIP in Figure 9 of the submitted draft could be attributed to the existence of round obstacles which have a similar shape to coin. (Please refer to the first frame of Figure 9b in the submitted draft.) However, as shown in Figure 9, 10, 11 of the submitted draft, our multimodal reward consistently exhibits an identifiable increasing pattern as the agent progresses toward the goal. Moreover, the issue of learning irrelevant behaviors might be mitigated through the utilization of our return-conditioned policy, as it is built upon a sequential architecture (DT [1], RSSM [2]). This architecture allows for the interpretation of the same reward value within the context of the agent's past trajectory. This contextual understanding can potentially alleviate the unexpected behavior and contribute to more effective utilization of the rewards for online training and finetuning purposes.\n\n---\n\n**[L3] How well the approach would work in more complex tasks (partial observability, non-Markovian states)?**\n\nAs shown in Figure 1 in the attached pdf, we observed that our method can guide the agent to achieve unseen goals in RLBench environment by incorporating our fine-tuning schemes. Considering that robotic manipulation tasks require more fine-grained control for gripping the object compared to Procgen environments, it remarks that our framework can be applicable in more complex tasks. However, our method does possess a limitation in addressing non-Markovian states, as the multimodal reward computation relies on image-text pair representations, potentially missing spatio-temporal relationships. An intriguing avenue for exploration is the extension of our approach to incorporate video-text pairs for multimodal reward computation. This limitation is discussed further in Appendix F of our draft.\n\n---\n\n**References**\\\n[1] Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021.\\\n[2] Learning Latent Dynamics for Planning from Pixels, ICML 2019."}]

Author rebuttal:
Dear reviewers,

We sincerely appreciate your time and effort in reviewing our manuscript. As the reviewers highlighted, our work proposes a simple but effective framework (qrbz, HCX5, 9gB5) for imitation learning with a novel adaptive reward that leverages pre-trained multimodal representations (9gB5, HCX5), and demonstrates strong empirical performance (HCX5). Our paper also provides informative analyses (qrbz, HCX5) and clear presentations (HCX5, nhjK).

We appreciate the reviewers’ insightful comments on our paper. In the attached pdf, we have included the following additional results to address the reviewer’s comments:
- RLBench experiment (Figure 1)
- Comparison with goal-conditioned baselines (Figure 2)
- Ablation study of MRDT+: comparison with smaller multimodal transformer trained from scratch (Figure 3)
- Concept figure depicting the difference between MRDT and baseline (Figure 4)
- Image observation of Procgen Maze environments used in our additional experiments (Figure 5)

---

The main concerns raised by multiple reviewers were (1) limited experiments and (2) lack of comparison to prior work that utilizes goal images. For (1), we prepare additional results on RLBench [1]. For (2), we prepared additional experiments comparing our method with goal-conditioned baselines. We describe the details of each experiment below.

**[GR1] RLBench experiment**

To further verify the effectiveness of our method, we evaluate on RLBench [1], which serves as a standard benchmark for visual-based robotic manipulations. Specifically, we test on the Pick Up Cup task, where the robot arm is instructed to grasp and lift the cup. We train agents in environments where the position of the cup to lift changes in each episode, and then evaluate the agents in a test environment, where the target cup is placed in an unseen region (refer to Figure 1a in the attached pdf). 

As a baseline, we consider MV-MWM [2]. We closely follow the experimental setup and implementation of the imitation learning experiments in MV-MWM. For training agents, we implement the return-conditioned policy using a variant of the recurrent state-space model (RSSM ; [4]), where the recurrent encoder is additionally conditioned on multimodal return. We refer to our model that utilizes frozen CLIP representations for computing the multimodal reward as MR-RSSM, and our model that incorporates the proposed fine-tuning scheme as MR-RSSM+.

Figure 1b in global response showcases the enhanced generalization performance of MR-RSSM+ agents in test environments, increasing from 20.37\% to 50.93\%. This result implies that our method facilitates the agent in reaching unseen goals by employing adaptive rewards on complex robotics tasks.

**[GR2] Comparison with goal-conditioned baselines**

We compare our method with goal-conditioned methods assuming the availability of goal images in both training and test environments. First of all, it is important to note that the suggested baseline relies on additional information from the test environment because it assumes the presence of a goal image (or state) during the test time. On the other hand, our method solely relies on natural language instruction and does not necessitate any extra access to the test environment.

For goal-conditioned baseline, we consider a variant of our algorithm that uses the distance of CLIP visual representation to the goal image as a reward, denoted as GC-DT. The training and test performance of the goal-conditioned baselines and MRDT are visually presented in Figure 2 in the attached pdf.

We observe that MRDT demonstrates comparable results to GC-DT in all three tasks. Importantly, it should be emphasized that while goal-conditioned baselines rely on the goal image of the test environment (which can be challenging to provide in real-world scenarios), MRDT solely relies on a natural language instruction for the task. These results show the potential of our method to be applicable in real-world scenarios where the agent cannot acquire information from the test environment.

---

We sincerely believe that our enhanced manuscript will deliver the benefits of our proposed method to the audience of NeurIPS much better.

Thank you very much!\
Authors.

**References**\
[1] RLBench: The Robot Learning Benchmark & Learning Environment, IEEE Robotics and Automation Letters, 5(2), 2020.\
[2] Multi-View Masked World Models for Visual Robotic Manipulation, ICML 2023.\
[3] Mastering Atari with Discrete World Models, ICLR 2021.\
[4] Learning Latent Dynamics for Planning from Pixels, ICML 2019.

Meta-review: This paper presents a framework, Multimodal Reward Decision Transformer (MRDT), that leverages pre-trained vision-language models like CLIP for reward generation in goal-conditioned reinforcement learning (RL). It aims to address the issue of goal misgeneralization on the OpenAI Procgen benchmark. The reviewers had mixed opinions about the work, but after discussion, most concerns are addressed and reviewers lean towards the acceptance of this work. 
Strengths highlighted include the novelty of using CLIP as an adaptive reward function, a potentially scalable approach, and methodological soundness. However, reviewers also emphasize various limitations, particularly around the breadth of experimental validation. There is a consistent call for tests on more complex or real-world tasks to assess the true capabilities and generalizability of MRDT. Despite these criticisms, the paper's strengths in pioneering the use of pre-trained multimodal models for RL and good experimental protocols merit its Accept status, although further validation is advised for future work.
Decision: Accept (poster)