SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While we previously presented joint learning of a policy and the attention module  $\mathcal{G}$ , here we also present how to update  $\mathcal{G}$  for a pretrained policy  $\pi$  to make the policy adaptable to domain changes. In this case, we add a policy prompt  $p_{\mathrm{pol}}^v$  to concentrate on task-relevant features from observations for
the pretrained policy  $\pi$  so that prompted embedding  $\tilde{z}_0$  with the task-relevant features is incorporated into the guided-attention-based ensemble, i.e.,  $\pi (\mathcal{G}(\tilde{z}_0,\mathbf{z}))$ , where  $\tilde{z}_0 = T_\phi (o,p_{\mathrm{pol}}^v)$ .
Table 2: Prompt Ensemble with a Pretrained Policy. The pretrained policies of each task are learned on 4 source domains. The Source column presents the performance for those source domains. In all evaluations, we use 40 unseen target domains. The Target column presents the performance for the unseen target domains not used for policy training.
(a) Zero-shot Performance in AI2THOR with Visual Navigation and Room Rearrangement Tasks
Method	ObjectNav. (Aln.)	PointNav. (Not Aln.)	ImageNav. (Not Aln.)	RoomR. (Not Aln.)
Source	Target	Source	Target	Souce	Target	Scoure	Target
Pretrained	87.5±17.2	65.8±19.1	95.3±4.6	80.9±1.6	77.2±3.3	56.2±2.2	87.3±3.1	75.2±13.2
CONPE	88.4±1.7	72.8±3.1	98.9±1.0	84.4±1.0	79.2±1.4	61.6±1.1	93.3±1.2	82.2±14.4
(b) Zero-shot Performance in egocentric-Metaworld with 4 Different Robot Manipulation Tasks
Method	Reach (Aln.)	Reach-Wall (Not Aln.)	Button-Press (Not Aln.)	Door-Open (Not Aln.)
Source	Target	Source	Target	Source	Target	Source	Target
Pretrained	100.0±0.0	65.7±6.4	100.0±0.0	58.0±5.8	100.0±0.0	16.8±2.3	100.0±0.0	35.6±6.2
CONPE	100.0±0.0	74.7±5.0	100.0±0.0	75.7±9.0	100.0±0.0	73.7±8.3	100.0±0.0	93.2±1.1
Table 2 reports zero-shot performance for the scenarios when a pretrained policy is given. We evaluate two different cases: aligned (Aln.) when prompt-based contrastive learning is conducted on data from the same task of a pretrained policy; otherwise, not aligned (Not Aln.). In AI2THOR, we use data from the object goal navigation task for prompt-based contrastive learning, while each pretrained policy is learned individually through one of tasks including object goal navigation, point goal navigation, image goal navigation, and room rearrangement. Similarly, in egocentric-Metaworld, we use data from the reach task for prompt-based contrastive learning, while each pretrained policy is learned individually through one of tasks including reach, reach-wall, button-press, and door-open. In Table 2(a), CONPE enhances zero-shot performance of the pretrained policies by  $3.5\sim 7.0\%$  for unseen target domains in AI2THOR. This prompt ensemble adaptation requires only 400K samples, equivalent to  $10\%$  of the total samples used for policy learning. In Table 2(b), CONPE significantly boosts zero-shot performance of the pretrained policies by  $9.0\sim 57.6\%$  in egocentric-Metaworld.
Here we conduct ablation studies with AI2THOR. All the performances are reported in success rates.
Table 3: Prompt Ensemble Scalability
n	Source	Seen Target	Unseen Target
2	98.7±0.4	40.5±2.2	43.0±2.9
5	96.1±0.6	59.2±9.6	45.0±10.1
10	96.3±1.0	83.3±0.3	79.7±6.4
16	91.8±2.0	83.8±1.3	77.1±6.2
18	98.5±1.8	83.3±2.3	79.0±4.5
Table 4: Prompt Ensemble Methods
Ensemble Method	Source	Seen Target	Unseen Target
COM-UNI-AVG	52.9±12.2	51.4±7.6	43.1±12.7
COM-WEI-AVG	63.6±11.2	42.3±5.2	50.3±6.1
ENS-UNI-AVG	88.6±1.4	79.8±3.4	65.5±5.0
ENS-WEI-AVG	94.1±6.3	75.5±3.5	61.2±12.7
CONPE	96.3±1.0	83.3±0.3	79.7±6.4
Prompt Ensemble Scalability. Table 3 evaluates CONPE with respect to the number of prompts  $(n)$ . CONPE effectively enhances zero-shot performance for both seen and unseen target domains through prompt ensemble that captures various domain factors. Compared to the case of  $n = 2$ , for  $n = 10$ , there was a significant improvement in zero-shot performance for both seen and unseen target domains, with increases of  $42.8\%$  and  $36.7\%$ , respectively. For  $n \geq 10$ , we observe stable performance that specifies that CONPE can scale for combining multiple prompts to some extent.
Prompt Ensemble Methods. Tabel 4 compares the performance of various prompt integration methods [25, 26, 27] including our guided attention-based prompt ensemble. We denote prompt-level integration as COM, and prompted embeddings-level integration as ENS. UNI-AVG and WEI-AVG refer to uniform average and weighted average mechanisms, respectively. CONPE achieves superior success rates over the most competitive ensemble method ENS-UNI-AVG, showing  $3.5\%$  and  $14.2\%$  performance gain for seen and unseen target domains.
Table 5: Prompt Ensemble Adaptation
Optimization	Source	Target
Pretrained	95.3±4.6	80.9±1.6
w/o pvpol	59.5±2.9	54.2±0.6
w pvpol	98.9±1.0	84.4±1.0
E2E	96.4±0.5	83.3±3.3
Table 6: Semantic Regularized Data Augmentation
δ	w/o Semantic	w Semantic
Source	Target	Source	Target
0.1	97.4±3.8	83.6±8.9	100.0±0.0	84.1±10.2
0.2	94.7±0.0	77.6±12.8	94.8±7.4	79.7±9.1
0.3	84.2±3.7	75.3±8.8	96.1±1.9	83.1±10.0
0.4	80.3±16.2	74.0±14.9	86.9±3.8	81.3±12.3
Prompt Ensemble Adaptation Method. Table 5 shows the effect of our ensemble adaptation method for the situation when a pretrained policy is given. As explained in Section 4.2, in this situation, CONPE can update the attention module with an additional prompt  $p_{\mathrm{pol}}^v$ . Note that  $p_{\mathrm{pol}}^v$  corresponds to this case, while w/o  $p_{\mathrm{pol}}^v$  corresponds to the other case of using the attention module without  $p_{\mathrm{pol}}^v$ . In addition, E2E denotes the fine-tuning of both the policy and the attention module along with  $p_{\mathrm{pol}}^v$ . The results demonstrate that our method enhances the zero-shot performance of the pretrained policy, showing that  $p_{\mathrm{pol}}^v$  facilitates the extraction of task-specific features.
Semantic Regularized Data Augmentation. So far, we have only utilized vision data, but here, we discuss one extension of CONPE using semantic information. Specifically, we use a few samples of object-level text descriptions to regularize the data augmentation process in policy learning. This aims to mitigate overfitting issues [28, 29]. The detailed explanations can be found in Appendix. As shown in Table 6, CONPE with semantic data (w Semantic) consistently yields better performance than CONPE without semantic data (w/o Semantic) for all noise scale settings  $(\delta)$ . Note that the noise scale manages the variance of augmented prompted embeddings. This experiment indicates that CONPE can be improved by incorporating semantic information.
Adaptation in Embodied AI. In the literature of robotics, numerous studies focused on developing generalized visual encoders for robotic agents across various domains [30, 31], exploiting pretrained visual encoders [32, 33], and establishing robust control policies with domain randomized techniques [34, 35]. Furthermore, in the field of learning embodied agents, a few works addressed adaptation issues of agents to unseen scenarios in complex environments, using data augmentation techniques [36, 37, 38, 39, 40] or adopting self-supervised learning schemes [41, 42, 43]. Recently, several works showed the feasibility and benefits of adopting large-scale pretrained vision-language models for embodied agents [7, 44, 45, 46]. Our work is in the same vein of these prior works of embodied agents, but unlike them, we explore visual prompt learning and ensembles, aiming to enhance both zero-shot performance and sample-efficiency.
Decoupled RL Structure. The decoupled structure, where a state representation model is separated from RL, has been investigated in vision-based RL [47, 2, 16]. Recently, contrastive representation learning on expert trajectories gains much interest, as it allows expert behavior patterns to be incorporated into the state encoder even when a policy is not jointly learned [1, 20]. They established generalized state representations, yet in that direction, sample-efficiency issues in both representation learning and policy learning remain unexplored.
Prompt-based Learning. Prompt-based learning or prompt tuning is a parameter-efficient optimization method for large pretrained models. Prompt tuning was used for computer vision tasks, optimizing a few learnable vectors in the text encoder [18], and it was also adopted for vision transformer models to handle a wide range of downstream tasks [17]. Recently, visual prompting [48] was introduced, and both visual and text prompt tuning were explored together in the multi-modal embedding space [49, 50]. We also use visual prompt tuning, but we concentrate on the ensemble of multiple prompts to tackle complex embodied RL tasks. We take advantage of the fact that the generalized representation capability of different prompts can vary depending on a given task and domain, and thus we strategically utilize them to enable zero-shot adaptation of RL policies.
**Limitation.** Our CONPE framework exploits visual inputs and their relevant domain factors for policy adaptation. For environments where domain changes extend beyond those domain factors, the adaptability of the framework might be constrained. In our future work, we will adapt the framework with semantic knowledge based on pretrained language models to improve the policy generalization capability for embodied agents in dynamic complex environments and to cover various scenarios associated with multi-modal agent interfaces.
Conclusion. In this work, we presented the CONPE framework, a novel approach that allows embodied RL agents to adapt in a zero-shot manner across diverse visual domains, exploring the ensemble structure that incorporates multiple contrastive visual prompts. The ensemble facilitates domain-invariant and task-specific state representations, thus enabling the agents to generalize to visual variations influenced by specific domain factors. Through various experiments, we demonstrated that the framework can enhance policy adaptation across various domains for vision-based object navigation, rearrangement, manipulation tasks as well as autonomous driving tasks.
We would like to thank anonymous reviewers for their valuable comments and suggestions. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-01045, 2022-0-00043, 2020-0-01821, 2019-0-00421) and by the National Research Foundation of Korea (NRF) grant funded by the MSIT (No. NRF-2020M3C1C2A01080819, RS-2023-00213118).
[1] Adam Stooke et al. "Decoupling representation learning from reinforcement learning". In: Proceedings of the 38th International Conference on Machine Learning. 2021, pp. 9870-9879.
[2] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. "CURL: Contrastive unsupervised representations for reinforcement learning". In: Proceedings of the 37th International Conference on Machine Learning. 2020, pp. 5639-5650.
[3] Max Schwarzer et al. "Data-efficient reinforcement learning with self-predictive representations". In: arXiv preprint arXiv:2007.05929 (2020).
[4] Shangda Li et al. "Unsupervised domain adaptation for visual navigation". In: arXiv preprint arXiv:2010.14543 (2020).
[5] Ziad Al-Halah, Santhosh K. Ramakrishnan, and Kristen Grauman. “Zero experience required: Plug & play modular transfer learning for semantic visual navigation”. In: Proceedings of the 9th International Conference on Vision and Pattern Recognition. 2021, pp. 17010–17020.
[6] Qianfan Zhao et al. "Zero-shot object goal visual navigation". In: arXiv preprint arXiv:2206.07423 (2022).
[7] Apoorv Khandelwal et al. "Simple but Effective: CLIP embeddings for embodied AI". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 14809-14818.
[8] Arjun Majumdar et al. "Zson: Zero-shot object-goal navigation using multimodal goal embeddings". In: arXiv preprint arXiv:2206.12403 (2022).
[9] Eric Kolve et al. "Ai2-thor: An interactive 3d environment for visual ai". In: arXiv preprint arXiv:1712.05474 (2017).
[10] Alexey Dosovitskiy et al. "CARLA: An open urban driving simulator". In: Proceedings of the 1st Conference on Robot Learning. 2017, pp. 1-16.
[11] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[12] Qihang zhang, Zheghao Peng, and Bolei Zhou. "Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining". In: Proceedings of the 17th European Conference on Computer Vision. 2022, pp. 111-128.
[13] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding". In: arXiv preprint arXiv:1807.03748 (2018).
[38] Xiaofeng Gao et al. "Dialfred: Dialogue-enabled agents for embodied instruction following". In: arXiv preprint arXiv:2202.13330 (2022).
[39] Chong Liu et al. “Vision-language navigation with random environmental mixup”. In: Proceedings of the International Conference on Computer Vision. 2021, pp. 1624–1634.
[40] Jialu Li, Hao Tan, and Mohit Bansal. "EnvEdit: Environment Editing for Vision-and-Language Navigation". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 15386-15396.
[41] Xin Wang et al. "Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 6629-6638.
[42] Roberto Bigazzi et al. "Explore and explain: self-supervised navigation and recounting". In: Proceeding of the 25th International Conference on Pattern Recognition. 2020, pp. 1152-1159.
[43] Eun Sun Lee et al. "MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agents". In: Proceeding of the 17th European Conference on Computer Vision. 2022, pp. 338-354.
[44] Vishnu Sashank Dorbala et al. "CLIP-Nav: Using CLIP for zero-shot vision-and-language navigation". In: arXiv preprint arXiv:2211.16649 (2022).
[45] Samir Yitzhak Gadre et al. "CLIP on Wheels: zero-shot object navigation as object localization and exploration". In: arXiv preprint arXiv:2203.10421 (2022).
[46] Dhruv Shah et al. "Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action". In: Proceedings of the 6th Conference on Robot Learning. 2022, pp. 492–504.
[47] Irina Higgins et al. "DARLA: Improving zero-shot transfer in reinforcement learning". In: Proceedings of the 34th International Conference on Machine Learning. 2018, pp. 1480-1490.
[48] Hyojin Bahng et al. "Exploring visual prompts for adapting large-scale models". In: arXiv preprint arXiv:2203.17274 (2022).
[49] Muhammad Uzair Khattak et al. "MaPLe: Multi-modal prompt learning". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[50] Yuhang Zang et al. "Unified vision and language prompt learning." In: arXiv preprint arXiv:2210.07225 (2022).