SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Task specification with text instructions Leveraging text instructions for task guidance is a common practice among humans. Building upon this concept, prior approaches have harnessed text instructions to direct agents in various ways, including the shaping of reward functions [24, 75, 77, 40, 51], addressing misbehavior correction [65], and exploring human-AI coordination [75, 28]. The closest work to ours is Ma et al. [51], which extends the objective function of VIP [52] to include text instructions as goals for training visual-language aligned representations. The primary distinction is that it uses static text representation concatenated with visual representations for policy learning, akin to other baselines [46, 67]. In contrast, our approach employs multimodal reward defined by measuring the similarity between image observations and text instructions in the pre-trained multimodal embedding space.
Utilizing CLIP [56] for supervision signals Recent work utilize CLIP scores [26] or CLIP-based perceptual loss [70] for improving image-text alignment in various domains including image generation [1, 12], image captioning [26, 9], and anomaly detection [37]. Similar to our approach, some work [13, 19] have also leveraged CLIP scores as supervision signals to address reward-scarce tasks with reinforcement learning. Fan et al. [19] propose a video-language model that is pre-trained using large-scale, real-world videos paired with their transcripts, and it utilizes the similarity between video-text representations as the reward for reinforcement learning. In our study, we focus on the adaptability of the multimodal reward, which empowers the agent to achieve desired goals in previously unseen test environments. Furthermore, we employ pre-trained multimodal representations without the need for resource-intensive pre-training, and we introduce a fine-tuning scheme for better reward quality that can be easily implemented with a small set of in-domain demonstrations.
C Procgen Experiment Details
This section describes the details for implementing Adaptive Return-conditioned Policy and provides our source code in the supplementary material.
Environment details We utilize a publicly available implementation<sup>3</sup> to replicate the environments introduced by Di Langosco et al. [15]. We modify the simulator of the environments to render higher-resolution images to leverage pre-trained multimodal representations for both our method and baselines. In this particular setup, the observations obtained from the environment at each timestep  $t$  comprise an RGB image with dimensions of  $256 \times 256 \times 3$  and a natural language instruction delineating the desired goal. Throughout our experiments, we adhere to the hard environment difficulty as described in [10]. Maximum episode length for all tasks is 500. To gather expert demonstrations used for training data, we train PPG [11] agents on 500 training levels for 200M timesteps per task using hyperparameters provided in Cobbe et al. [11]. For evaluation purposes, we assess the test performance on 1,000 different levels, encompassing previously unseen themes and goals that differ from those employed in training.
Architecture details Both InstructRL (Liu et al., 2022) and ARP employ ViT-B/16 as the transformer-policy and pre-trained multimodal transformer encoder (M3AE; [22]) in all experiments, unless stated otherwise. For fine-tuning pre-trained multimodal encoders, we adopt the CLIP-Adapter [21, 80] to effectively fine-tune the pre-trained CLIP embeddings without overfitting.
<sup>3</sup>https://github.com/JacobPfau/procgenAISC
In detail, we attach extra linear layers to both the visual and text encoders and use the weighted sum of the output from the linear layers and the original pre-trained feature for computing visual/text representations. Throughout the training, we only apply gradients to the weight of these adapter layers and freeze both CLIP's visual and textual encoders. Furthermore, we utilize multi-scale features obtained by concatenating intermediate layer representations with the final output representation as the input for the adapter layers, drawing inspiration from Liu et al. [46] and Walmer et al. [71]. Finally, the multimodal reward is computed using the cosine similarity between the multi-scale features from the image and text encoders. See Appendix E for qualitative results of our multimodal rewards. Inspired by Gao et al. [21], we attach an additional 2-layer MLP to the end of a pre-trained multimodal transformer encoder and use the weighted sum of the output from MLP and the original pre-trained feature for obtaining image/text representations. In the training phase, we apply gradients only to the weight of these MLP layers. Through empirical evaluation, we observe that this architecture yields superior performance in both our method and the baseline.
Training details We resize images to  $224 \times 224 \times 3$  before computing multimodal rewards, and we use  $256 \times 256 \times 3$  RGB observations for training the return-conditioned policy. To stabilize training, we normalize multimodal returns following the method proposed by Chen et al. (2021), dividing them by 1000 in all experiments. We use the AdamW optimizer (Loshchilov et al., 2018) with a learning rate of  $5 \times 10^{-4}$  and weight decay  $5 \times 10^{-5}$ . A cosine decay schedule is utilized to adjust the training learning rate. In CoinRun experiments, data augmentation techniques such as color jitter and random rotation are applied to the RGB images  $o_t$  while maintaining alignment in the context. However, no augmentation is used to RGB images in Maze I/II experiments. For scaling the return prediction loss in training the return-conditioned policy, we set  $\lambda = 0.01$  in CoinRun experiments and  $\lambda = 0.001$  in Maze I/II experiments. During the fine-tuning of the pre-trained multimodal encoder, a 2-layer MLP is attached to the end of both CLIP image and text encoders. An extra 2-layer MLP is added as an action prediction layer for the IDM objective. The model is trained for 20 epochs, and the one with the lowest validation loss is used for generating multimodal rewards. To scale the IDM loss in fine-tuning CLIP, we employ  $\beta = 1.5$  in CoinRun experiments and  $\beta = 2.0$  in Maze I/II experiments. In evaluation, we choose the target multimodal return as  $90\%$  quantile of the multimodal return from the dataset  $\mathcal{D}^*$  in all experiments.
Computation We use 24 CPU cores (Intel Xeon CPU @ 2.2GHz) and 2 GPUs (NVIDIA A100 40GB GPU) for training return-conditioned policy. The training of ARP for 50 epochs takes approximately 4 hours for CoinRun experiments with the largest dataset size. For fine-tuning CLIP, we use 24 CPU cores (Intel Xeon CPU @ 2.2GHz) and 1 GPU (NVIDIA A100 40GB GPU), which takes approximately 1.5 hours for Coinrun experiments.
Hypeparameters We report the hyperparameters used in our experiments in Table 6.
Table 6: Hyperparameters of ARP-DT. Unless specified, we use the same hyperparameters used in InstructRL [46].
Hyperparameter	Value
Policy batch size	64
Policy epochs	50
Policy context length	4
Policy learning rate	0.0005
Policy optimizer	AdamW [47]
Policy optimizer momentum	β1=0.9, β2=0.999
Policy weight decay	0.00005
Policy learning rate decay	Linear warmup and cosine decay (see code for details)
Policy context length	4
Policy transformer size	2 layers, 4 heads, 768 units
Fine-tuned CLIP batch size	64
Fine-tuned CLIP epochs	20
Fine-tuned CLIP learning rate	0.0001
Fine-tuned CLIP weight decay	0.001
Fine-tuned CLIP adapter layer size	2 layers, 1024 units
Fine-tuned CLIP optimizer	AdamW [47]
Fine-tuned CLIP optimizer momentum	β1=0.9, β2=0.999
D RLBench Experiment Details
Environment details Visual observations obtained from RLBench environment at each timestep  $t$  comprise an RGB image with dimensions of  $256 \times 256 \times 3$  and a natural language instruction delineating the desired goal. In Pick Up Cup task, we adjusted the environment to generate the target cup with different regions in training and evaluation. Specifically, we generate the target cup from  $80\%$  of the area of the table in training environments. In evaluation, we generate the cup from the remaining  $20\%$  of the area of the table. We choose an action mode of RLBench that specifies the delta of joint positions for all experiments.
Training details For training data, we collect 100 expert demonstrations using scripted policy provided by RLBench simulator. For each demonstration, we extract the keypoints [35] and train the agent to output the relative change in  $(\mathrm{x},\mathrm{y},\mathrm{z})$  position between keypoints. We train the agents to output the prediction of  $(\mathrm{x},\mathrm{y},\mathrm{z})$  position changes. We set the maximum episode length to 500. We closely follow the implementation details and use the same hyperparameters described in the imitation learning experiments in MV-MWM [62]. For all experiments, we utilize the open-sourced pre-trained CLIP model with ViT-B/16 architecture from huggingface transformers library<sup>4</sup> and we fine-tune CLIP based on that model. We train our method for 100K iterations and MV-MWM for 200K iterations until convergence. We apply data augmentation in training, including color jittering over RGB images. In evaluation, we choose the target multimodal return as  $50\%$  quantile of the multimodal return from training demonstrations in all experiments.
E Qualitative Results of Multimodal Rewards
In Figure 8, 9, 10, we present the curves of multimodal rewards for frozen/fine-tuned CLIP in the trajectories from training/held-out evaluation environments. We find that the multimodal reward exhibits an overall increasing trend as the agent approaches the goal in both frozen and fine-tuned CLIP, irrespective of the training and held-out evaluation environments. Furthermore, we observe that fine-tuned CLIP not only induces a reward that is temporally smoother in the intermediate stages compared to frozen CLIP (see Figure 1) but also demonstrates a steeper upward reward curve (see Figure 9, 10). These results support the claim that the quality of multimodal rewards from the fine-tuned CLIP outperforms those from the frozen CLIP (Section 4.1). Video examples of the trajectories are provided in the supplementary material.
<sup>4</sup>https://huggingface.co/openai/clip-vit-base-patch16
1
(a) Multimodal reward curve in the training environment.
1
(b) Multimodal reward curve in the held-out evaluation environment.
Figure 8: Qualitative results of multimodal rewards in CoinRun environments.
(a) Multimodal reward curve in the training environment.
(b) Multimodal reward curve in the held-out evaluation environment.
Figure 9: Qualitative results of multimodal rewards in Maze I environments.
(b) Multimodal reward curve in the held-out evaluation environment.
Figure 10: Qualitative results of multimodal rewards in Maze II environments.
Effect of return prediction We investigate the effect of including the return prediction loss  $\mathrm{MSE}(\hat{R}_t,\hat{R}_t^*)$  in Equation 6, which encourages the policy to be more aware of conditioned returns. In Table 7, we observe that the performance of ARP-DT becomes much more sensitive to the quality of multimodal rewards when trained with the return prediction loss. For instance, without the return prediction loss, the evaluation perfor
Table 7: Ablation study of the return prediction loss  $\mathrm{MSE}(\hat{R}_t,\hat{R}_t^*)$  in CoinRun environments.
MSE(ˆt,ˆt*)	LFT	Train (%)	Test (%)
X	X	90.28% ± 4.21%	56.32% ± 3.55%
✓	92.01% ± 3.18%	56.28% ± 2.01%
✓	X	89.58% ± 2.08%	63.32% ± 2.01%
✓	90.28% ± 1.59%	72.36% ± 3.48%
mance becomes almost the same with or without the fine-tuning scheme, which suggests that the model is insensitive to the quality of rewards. On the other hand, with the prediction loss, the performance increases as the quality of reward improves. This implies that the model becomes aware of the returns and can thus follow the adaptive signal from the multimodal reward.
Effect of scaling return prediction loss We investigate how the coefficient  $\lambda$ , which determines the weight of the return prediction loss in training return-conditioned policy, affects the performance of ARP-DT. To this end, we test various values of  $\lambda$  in CoinRun environments. Table 8 shows the performance of ARP in training/held-out evaluation environments with different  $\lambda$ . We find that performance is not significantly different according to the value of  $\lambda$  in the held-out evaluation environments. These results indicate that ARP is robust to the choice of hyperparameter  $\lambda$ .
Table 8: Ablation studies of the hyperparameter  $\lambda$  in CoinRun environments.
λ	\( \mathcal{L}_{\text{FT}} \)	Train (%)	Test (%)
0.001	X	89.93%	± 3.94%	62.65%	± 10.12%
✓	85.42%	± 1.80%	71.69%	± 5.71%
0.01	X	89.58%	± 2.08%	63.32%	± 2.01%
✓	90.28%	± 1.59%	72.36%	± 3.48%
0.1	X	87.15%	± 2.62%	62.65%	± 10.31%
✓	85.76%	± 3.18%	73.37%	± 3.48%
1.0	X	87.15%	± 4.70%	61.64%	± 6.38%
✓	81.25%	± 1.04%	73.37%	± 2.66%
Figure 11: Expert-normalized scores on training/evaluation environments of ARP trained using multimodal rewards generated with (i) instructive text (i.e., Inst) and (ii) random text (i.e., Random). The result shows the mean and standard deviation averaged over three runs.
Ablation studies on text instructions To investigate whether ARP-DT makes decisions based on the adaptive signal from the multimodal reward, we evaluate the quality of rewards generated with (i) instructive text (i.e., Inst) and (ii) random text (i.e., Random) in three different environments. Specifically, we use a natural language instruction for each environment, as described in Section 4 for Inst, and "NeurIPS 2023 will be held again at the New Orleans Ernest N. Morial Convention Center" for Random. We find that using random text instructions significantly declines performance in both training and evaluation environments. These results highlight the importance of using the instructive text and demonstrate that ARP-DT indeed depends on the adaptive signal from the multimodal reward for solving tasks at deployment time.
G Limitation and future work
One limitation of our work is that we currently rely on a single image-text pair to compute the multimodal reward at every timestep  $t$ . Although our approach has shown effectiveness both quantitatively and qualitatively, there are tasks where rewards depend on the history of past observations (i.e., non-Markovian) [4, 5, 39]. To address this limitation, it would be valuable to explore the extension of our method to incorporate video-text pairs for calculating multimodal rewards. Another aspect to consider is that the tasks we have examined so far are relatively simple, as they involve only a single condition for success. To tackle more complex problems, we are interested in investigating approaches that leverage large language models [29, 30, 2, 17] with our method.
H Potential negative impacts
We do not anticipate significant negative societal impacts in that our method is now limited to applying in simulated environments. However, if our method is applied in real-world scenarios, privacy concerns may arise, considering that behavior cloning agents used in such applications, like autonomous driving [63] or real-time control [7, 17], require large amounts of data, which often contain controversial information. Additionally, a behavior cloning policy presents a challenge as it imitates specified demonstrations, potentially including undesirable actions. If some evil actions are included in expert demonstrations (e.g., behaviors that may be violent to the pedestrians are contained in the training data for mobile manipulation tasks), the policy could have significant negative impacts on users. To address this concern, future directions should focus on developing agents with safe adaptation in addition to performance enhancement efforts.