Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents
Authors: Wonje Choi
Woo Kyung Kim
SeungHyun Kim
Honguk Woo
Venue: NeurIPS 2023 poster (2023)
Keywords: Prompt Learining
Domain Adaptation
Embodied AI
TL;DR: Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents

Abstract: For embodied reinforcement learning (RL) agents interacting with the environment, it is desirable to have rapid policy adaptation to unseen visual observations, but achieving zero-shot adaptation capability is considered as a challenging problem in the RL context. To address the problem, we present a novel contrastive prompt ensemble (ConPE) framework which utilizes a pretrained vision-language model and a set of visual prompts, thus enables efficient policy learning and adaptation upon a wide range of environmental and physical changes encountered by embodied agents. Specifically, we devise a guided-attention-based ensemble approach with multiple visual prompts on the vision-language model to construct robust state representations. Each prompt is contrastively learned in terms of an individual domain factors that significantly affects the agent's egocentric perception and observation. For a given task, the attention-based ensemble and policy are jointly learned so that the resulting state representations not only generalize to various domains but are also optimized for learning the task. Through experiments, we show that ConPE outperforms other state-of-the-art algorithms for several embodied agent tasks including navigation in AI2THOR, manipulation in Metaworld, and autonomous driving in CARLA, while also improving the sample efficiency of policy learning and adaptation.

Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents
Wonje Choi, Woo Kyung Kim, SeungHyun Kim, Honguk Woo"
Department of Computer Science and Engineering
Sungkyunkwan University
{wjchoi1995, kwk2696, kimsh571, hwoo)@skku. edu
Abstract
For embodied reinforcement learning (RL) agents interacting with the environment, it is desirable to have rapid policy adaptation unseen visual observations, but achieving zero-shot adaptation capability is considered as challenging problem in the RL context. To address the problem, present novel contrastive prompt ensemble (CONPE) framework which utilizes pretrained vision-language model and of visual prompts, thus enabling efficient policy learning and adaptation upon a wide range of environmental and physical changes encountered by embod- ied agents. Specifically, we devise a guided-attentee attention-baser based ensemble approach with multiple visual prompts on the vision- langualan model to construct robust state representations. Each prompt is contrastively learned in terms of an individual domain factor that significantly affects the agent's egocentric perception and obser- vation. For given task, the attention-ban based ensemble and policy are jointly learned so that the resulting state representations not only generalize to various domains but are also optimized for learning the task. Through experiments, we show that CONPE outperforms other state-of-the-art algorithms for several embodied agent tasks including navigation AI2THOR, manipulation egocentric- and autonomous driving in CARLA, while also improving the sample efficiency of policy learning and adaptation.
Introduction
In the literature of vision-based reinforcement learning (RL), with the advance of unsupervised techniques and large-scale pretrained models for computer vision, the decoupled structure, in which visual encoders are separately trained and used later for policy learning, has gained popularity [T] 2.3. This decoupling demonstrates high efficiency in low data regimes with sparse reward signals, compared end-to-end RI. In this regard, several works on adopting the decoupled structure to
embodied agents interacting with the environment were introduced [4] [5] and specifically, pretrained
vision models (e.g., ResNet (6) or vision-language models (e.g., CLIP in (7,8) were exploited for visual state representation encoders. Yet, it is non-trivial achieve zero-shot adaptation to visual domain changes the environment with high diversity and non-stationarity, which are inherent for embodied agents. It was rarely investigated how to optimize those popular large-scale pretrained models to ensure the shoo capability embodied agents.
Embodied agents have several environmental and physical properties, such as egocentric camera position, stride length, and illumination, which are domain factors making significant changes in agents' perception and observation. In the target (deployment) environment with uncalibrated settings on those domain factors, RI. policies relying on pretrained visual encoders remain vulnerable to domain changes.
Honguk Woo is the corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure provides an example of egocen- tric visual domain changes experienced by embodied agents due to different camera po- sitions. When policies learned in the source environment are applied to the target en- vironment, zero-shot performance can significantly degraded, unless the visual en- coder could adapt not only to environmental Figure Visual Domain Changes of Embodied Agents differences but also to the physical diversity of agents. In this paper, we investigate RL policy adaptation techniques for embodied agents to enable zero-shot adaptation to domain changes, by leveraging prompt-based learning for pretrained models in the decoupled RI structure. To this end, we present CONPE, a novel contrastive prompt ensemble framework that uses the CLIP vision- lision-language model the visual encoder, and facilitates dynamic adjustments of visual state representations against domain changes through an ensemble of contrastively learned visual prompts. In CONPE, the ensemble employs attention-basee state composition on multiple visual embeddings from the same input observation, where each embedding corresponds to state representation individually prompted for a specific domain factor. Specifically, the cosine similarity between an input observation and its respective prompted embeddings is used to calculate attention weights effectively.
Source Environment
Target Environments
Through experiments, we demonstrate the benefits of our approach. First, RL policies learned via CONPE achieve competitive zero-shot performance upon wide variety of egocentric visual domain variations for several embodied agent tasks, such as navigation tasks in AI2THOR (9. vision-based robot manipulation tasks in egocentric- egocentric-Metaworre Metaworld, and autonomous driving tasks in CARLA (10). For instance, the policy via CONPE outperforms EmbCLIP (7) in zero-shot performance by 20.7% for unseen target domains in the AI2THOR object navigation. Second, our approach achieves high sample-efficiency in the decoupled RL structure, For instance, CONPE requires less than 50.0% and 16.7% of the samples compared ATC T
and 60% and 50% of the samples compared to EmbCLIP to achieve comparable performance in seen and unseen target domains in the AI2THOR object navigation.
In the context of RL, our work is the first to explore policy adaptation using visual prompts for embodied agents, achieving superior zero-shot performance and high sample-efficiency. The main contributions of our work are as follows.
We present novel CONPE framework with an ensemble of visual contrastive prompts, which enables zero-shot adaptation for vision-based embodied RL agents.
We devise visual prompt-based contrastive learning and guided- sendd-atteniionbbseee prompt ensemble algorithms to represent task-specific information the CLIP embedding space.
We experimentally show that policies via CONPE achieve comparable or superior zero- shot performance, compared to other state-of-the-ar baselines, for several tasks. We also demonstrate high sample-efficiency in policy learning and adaptation.
We create the datasets with various visual domains in AI2THOR, egocentric- egocentric-Metaworld and CARLA, and make them publicly accessible for further research on RL policy adaptation.
2 Problem Formulation
In RL formulation, a learning environment is defined as Markov decision process (MDP) of (S, P, RR) with state space s E S, action space a E A, transition probability P S x + S and reward function R: S A + R. The objective of RL is to find an optimal policy x" S - A maximizing the sum of discounted rewards. For embodied agents, states might not be fully observable, and the environment is represented by partially observable MDP (POMDP) of tuple AS, R, 2, with an observation space o E Ω and a conditional observation probability x A Q.
Given visual domains in the dynamic environment, we consider policy adaptation to find the optimal policy that remains invariant across the domains or is transferable to some target domain, where each domain is represented by a POMDP and domain changes are formulated by different O. We denote domains as D (2, O). Aiming to enable zero-shot adaptation to various domains, we formulate

---

(1) Prompt- Contrastive Leaming
(ii) Guided-A tention-based Prompt Ensemble (iii) Zero-shot Policy Deployment
Rnightnes
P
CAM
Pf Sinde Sindel Leughtng
Loumiing
Pronyy Ersembles
plysioa
Clai % (ouet?
peopetty
Coztrust Punction
Space
Domsit genernlantion
donais
(Gardeed ittrttinn warghts
Visul Franet Pool,
Policy y
Policy
action
Tanget BEK.
Figure 2: CONPE Framework. The CLIP visual encoder is enhanced offline via (i) prompt-based contrastive learning that generates the visual prompt pool, and a policy is learned online by (ii) guided-attention-bale prompt ensemble that uses the prompt pool. In (iii) zero-shot deployment, the policy is immediately evaluated upon domain changes.
the policy adaptation problem as finding the optimal policy x' such that
argmax
E Drp(D)
Σ y (R(ss,((o)))
(I)
where p(D) is given domain distribution and y is discount factor of the environment.
For embodied agents, the same state can be differently observed depending on the configuration of properties such as egocentric camera position, stride length, illumination, and object style. We refer to such a property causing domain changes in the environment as domain factor. Practical scenarios often involve the interplay of multiple domain factors in the environment.
3
Our Approach
3.1 Framework Structure
To enable zero-shot policy adaptation unseen domains, develop the CONPE framework consist- ing of (i) prompt-based contrastive learning with the CLIP visual encoder, (ii) guided-attention-be prompt ensemble, and (iii) zero-shot policy deployment, illustrated in Figure The capability of the CLIP visual encoder is enhanced using multiple visual prompts that are contrastively learned on expert demonstrations for several domain factors. This establishes the visual prompt pool in (i). Then, the prompts are used to train the guided-attention baseed ensemble with the environment in (ii). To enhance learning efficiency and interpretability of attention weights, we use the cosine similarity of embeddings. The attention module and policy are jointly learned for specific task so that resulting state representations tend to generalize across various domains and optimized for task learning, In deployment, non-stationary environment where its visual domain varies according to the environment conditions and agent physical properties is considered, and the zero-shot performance evaluated in (iii).
3.2 Prompt-based Contrastive Learning
To construct iamain-invariant representations with respect to a specific domain factor for egocentric perception data, we adopt several contrastive tasks for visual prompt learning, which can be learned on a few expert demonstrations, For this, we use visual prompt
p² = Je", e" ef T Rd
c.
(2)
where ef is a continuous learnable vector with image patch embedding dimension d (e.g.. 768 for CLIP visual encoder) and is the length of visual prompt. Let pretrained model To parameterized by & maps observations
to the embedding space 2. With a contrast function I2] to discriminate whether observation pair is positive not,
P. Ω Ω + {0, [

---

Pi
PS
p
P5
0
Pi
Visual encoder Ta (o.pl)
Emheddings
30
11
4
...
Zn
11
2
3
24
35
2
83
94
9s
8n
mlp
mlp
mlp
mlp
mlp
mlp
U1
U2
U3
44
Ug
4
softmax
2
On
(
(a22)
(013)
(014)
(())
(0))))
Attention Module G
Task saak-specific state representation 2
Policy (Z)
Figure 3: based Prompt Ensemble. The cosine similarity- -uideed attention module yields task-specific state representations from multiple prompted embeddings and learned with policy network æ.
consider an m-sized batch of observation pairs Bp = {(oi, Jimm containing one positive pair
foo, o'2) P(ox, o4) }} for some m. Then, enhance the capability of Ta by learning visual prompt p" through contrastive learning, where the contrastive loss function (13] defined as
C log log ( S(To(ok, p"), To(o), ""Tooee))))
1
cxp ( (x, v}
|>>||||| ) (3)
Eier S(To(o,,p"), To To(o S(z, v)
As in [14, for latent vectors 2, y E Z, their similarity in the embedding space 2 is calculated by S(x, y), where À is hyperparameter. By conducting the prompt-based contrastive learning on different domain factors, we obtain a visual prompt pool
p .p..P%....PPP)
Through this process, each visual prompt in p" encapsulates domain-invariant knowledge pertinent its respective domain factor.
3.3 Guidedd-Ateention-based Prompt Ensemble
To effectively integrate individual prompted embeddings from multiple visual prompts into a task- specific state representation, devise guidedd-ttennionnbassee prompt ensemble structure, shown in Figure 3 where the attention weights on the embeddings are dynamically computed via the attention module g for each observation.
Given observation o and the learned visual prompt pool p", an image embedding 2g = To(o) and prompted embeddings attention module G, where attention weights WP for each prompted embedding 2, are optimized. Since directly computing the attention weights using 2g and is prone to have uninterpretable local optima, we introduce a guidance score 9; based on the cosine similarity between the input image and (Ro.&,)
[=] (o, pi,,.., ..., zn] are calculated. Then, =o and z are fed to the
visual prompted image embeddings in Z, i.e., = || Given that larger 9: signifies stronger conformity of an observation to the domain factor relevant to the prompted embedding 2,. use gi to steer the attention weights, aiming to not only improve learning efficiency but also provide interpretability. With guidance Jt, compute the attention weights Wi by
exp(u:/7) w5 Σx exp(ux/7)*
(En, ks)
u;
91
(5)
Vd

---

Algorithm Procedure of CONPE Framework
Dataset D {(1,, (1),,,)))..:. replay buffer Zp & 0, pretrained vision-language model Visual prompt pool p = "Pi:",,p'l attention module G, policy
1: / Prompt-based Contrastive Learning //
2: for ..... ni do
3: while not converge do
4: Sample batch Bp. = ((0j,0))))s ~
5: Update prompt p t P. VCCON (V, BR, using
6: end while
7: end for
8: 1* Prompt Ensemble- basee Policy Learning */
9: for each environment step do
10: Sample action
x (g (o) z)) using (
11: ZD + ZD U ( a, r)r
Jointly optimize policy and module G (f Zp
13: end for
where K; is the projection of zx, d is dimension of and T is a softmax temperature. Then, state embedding Z is obtained by
Z = G(=0, 2)
Wizi.
(6)
Algorithm [ shows the procedures in CONPE, where the first half corresponds to prompt-based contrastive learning (in Section 3.2) and the other half corresponds to joint learning of policy π T(Z) and the attention module G. As G optimized by given RL task objective in the source domains (in line 12), the resulting Z tends to be task-specific, while Z is also nomain-invariant by the ensemble of contrastively learned visual prompts based on G with respect the combinations of multiple domain factors. The entire algorithm can be found in Appendix.
Evaluation
10 environments, specifically configured for embodied agent tasks with dynamic domain changes. These environments allow us factors to explore various domain factors such as camera settings, stride length, rotation degree, gravity, illuminations, wind speeds, and others. prompt-based contrastive learning (in Section 3.2). we use small dataset of expert demonstrations for each domain factor (i.c., 10 episodes per domain factor). For prompt ensemble-bre based policy learning (in Section 3.3). we use few source domains randomly generated through combinatorial variations of the seen domain factors (i.c., 4 source domains). In zero-shot evaluations, we use target domains that can be categorized as either seen or unseen. The seen target domains those encountered during the phase of prompt-based contrastive learning, while these domains are not present during the phase of prompt ensemble-base based policy learning. On the other hand, the unseen target domains refer to those that are entirely new, implying that they not encountered during either learning phases.
Experiments. We use AI2THOR (9), Metaworld [15), and CARLA
Baselines. We implement several baselines for comparison. LUSR 16 is reconstruction- based domain adaptation method RL, which uses the variational autoencoder structure for robust rep- resentations. CURL (2) and ACT [ employ contrastive learning in RL frameworks for high sample-efficiency and generalization to visual domains. ACO [12] utilizes augmentation-die behavior-driven contrastive tasks in the context of RL. EmbCLIP is a state-of-the ohe-----rtttttt embodied Al model, which exploits the pretrained CLIP visual encoder for visual state representations.
and
Implementation. We implement CONPE using the CLIP model with ViT-B/32, similar to VPT [17 and CoOp (18). prompt-based contrastive learning, we adopt various contrastive learning schemes
including augmentation-drivel where the prompt length sets to 8. policy learning, we exploit online learning (i.e., PPO (23))
19 and behavior-driven [2 20 21 22 contrastive learning,
for AI2THOR and imitation learning (i.e., DAGGER (4) for egocentric Metaworld and CARLA.
eggcentric-Mettaorlld
5

---

Table 1: Zero-shot Performance. The policies of each method (CONPE and the baselines) are learned on source domains. The Source column presents the performance for those source domains. In all evaluations, 30 seen target domains and 10 unseen target domains. The Seen Targer column presents the performance for the seen target domains, and the Unseen Target column presents the performance for the unseen target domains. The unseen target domains are not used for representation learning.
(a) Zero-shot Performance in AI2THOR with Object Point Goal Navigation Tasks
ObjectNav.
PointNaw.
Method
Source
Seen Target
Unseen Target
Source
Seen Target
Unseen Target
LUSR
53.3.11.1
21.3.1199
15.1118
85.614.6
71.8.33.8
62.415.8
CURL
51.3.11.0
8.010.1
6.911.
70.8.7.4
55.212.7
54.8.33.0
ATC
82.2+9.7
72.3#3.3
51.318.6
95.0=3.3
89.1+1.9
81.9#3.6
ACO
55.0+23..8
39.6#21.5
35.8=5.8
91.1=6.3
73.4+2.0
67,5+2.8
EmbCLIP
89.3+3.0
77.6#1.3
59.0=16.4
95.3+4.6
84.5+19
77.4+14
CONPE
96.3+1.0
83.3+0..3
79.7:6.4
97.8+1.0
89.7+1.6
84.3+2.0
(b) Zero-shot Performance in agocentric-Metawone with Reach and Reach-wall Tasks
Reach
Reach-Wall
Method
Source
Seen Target
Unseen Target
Source
Seen Target
Unseen Target
LUSR
100.0:10.0
16.0:115.1
14.7::2.3
50.0=110.0
33.3:16.1
30.7:16.4
CURL
100.0=0.0
53.315.0
46.7
43.3=15.3
2.0:0.00
0.7+1.2
ATC
100.0=0.0
71.318.1
72.042.0
66.725.8
5.3:112
4.0:10.0
ACO
100.0+0.0
52.0+22.0
44.0+3355
63.3+15.3
8.74-2.3
4.7+1.2
EmbCLIP
100.010.0
64.716..1
66.714.22
100.0:1..0
58.011..2
49.315.0
CONPE
100.010.0
88.713.1
86.7-3.1
100.0:10.0
75.313.11
67.311.3
(c) Zero-shot Performance in CARLA with Different Maps
Map
Map
Method
Source
Seen Target
Unseen Target
Source
Seen Target
Unseen Target
LUSR
2141.9
635.11606.22 2
1073.91112.6
2279.6
1173.7
2159.41146.5
CURL
945.4
864.21138..J 0
1256.0111.6
1050.1
1089.91824.0
2190.3110.2
ATC
2280.5
1684.4+338..
1073.7+618.8
2272.2
2253.9+218.7
2200.1=307.8
ACO
2265.8
1545.611596.1
1330.01144..5
2270.6
2360.9188.0
2415.51553.0
EmbCLIP
2235.7
1732.21588..6
1415,11669.99
2262.7
2139.11655.9
240133122.3
CONPE
2237.5
1738.0+163.3
1933.4+299
2277.2
2422.5=79.6
2512.9::157
4.1 Zero-shot Performance
Table T shows zero performance CONPE and the baselines across source, seen and unseen target domains. We evaluate with different seeds and report the average performance (i.e., task success rate in AI2THOR and egocentric -ererrorddd,. the sum of rewards in CARLA). As shown in Table [aa] CONPE outperforms the baselines in the AI2THOR tasks. It particularly surpasses the most competitive baseline, EmbCLIP, by achieving 5.2 5.7% higher success rate for seen target domains, and 6.9 20.7 7% for unseen target domains. For egocentric-Metaworrla
shown in Table [(b) CONPE demonstrates superior performance with a significant success rate for both seen and unseen target domains, which is 17.3 ~ 24.0% and 18.0 20.0% higher than EmbCLIP, respectively. For autonomous driving in CARLA, we take into account external environment factors, such weather conditions and times of day, as domain factors that can influence the driving task. In Table I(c) CONPE consistently maintains competitive zero-shot performance across all conditions, outperforming the baselines.
In these experiments, LUSR shows relatively low success rates, as the reconstruction-bant represen- tation model can abate some task-specific information from observations, which is critical to conduct vision-based complex RI. tasks. EmbCLIP shows the most comparative performance among the baselines, but its zero-shot performance for target domains is not comparable to CONPE. In contrast,

---

CONPE effectively estimates the domain shifts pertaining to each domain factor through the use of guided attention weights, leading robust performance both seen and unseen target domains.
100
100
R0
50
S000000 KOrK
80
8
0.0M 0.5M 1.M
1.5M
2.0M
2.5
3.004
3.5M
4.53
9.3M 0.535 1.M
1.534
2:M
2.53
3.0
3.5M
4.M
Timedope
Timesige
(a) Success Rate in Seen Target Domain
(b) Success Rate in Unseon Target Domain
Figure 4: Sample-efficiency
Prompt Ensemble- Policy Learning for Object Navigation in AITTHOR. The x-axis represents the number of samples (timesteps) used for policy learning, while the y-axis represents the task success rate for zero-shot evaluation.
Sample Efficiency. Figure presents performance with respect to samples (timesteps) that are used by CONPE and baselines for policy learning. Compared the most competitive baseline EmbCLIP, CONPE requires less than 60.0% timesteps (online samples) for seen target domains and 50.0% for unseen larget domains have comparable success rates.
Prompt
dipoxaLL
2)
I) Intra Prompted Embedding
2) Inter Prompted Embodding
(a) Prompted Embeddings
(b) Attention Weights
Figure 5: Prompt Ensemble Interpretability. In (a), the embeddings in the big circle are intra prompted embeddings obtained by varying domains within a domain factor, and the embeddings in the rectangle inter prompted embeddings obtained by changing the visual prompts with aligned observation. The closely located intra prompted embeddings indicate the domain-invarian knowledge, while inter prompted embeddings clustered by different visual prompts indicate the alignment between the visual prompts and the domain factors. In (b), each cell represents attention weight w, applied for prompted embedding 24.
Prompt Ensemble Interpretability. Figure 5(a) visualizes the prompted embeddings using the prompt pool obtained through CONPE. For intra prompted embeddings, we use observation pairs, where each pair is generated by varying domains within domain factor. We observe that the embeddings are paired to form the domain cimaiininnvariat invariant knowledge because the visual prompt is learned through prompt-basee based contrastive learning. The inter prompted embeddings specify that each prompt distinctly clusters prompted embeddings that correspond to the domains associated to its domain factor. Figure 5(b) shows examples of the attention weight matrix of CONPE for four different domains. The x-axis denotes the visual prompts and the y-axis denotes timesteps. This shows the consistency of the attention weights on the prompts across the timesteps in the same domain.
4.2 Prompt Ensemble with Pretrained Policy
While we previously presented joint learning of policy and the attention module G, here we also present how to update G for a pretrained policy и to make the policy adaptable domain changes. In this case, we add policy prompt p to concentrate task-relevant features from observations for

---

the pretrained policy т so that prompted embedding 20 with the task-relevant features is incorporated into the guided-atten satieed--atention--ae ensemble, i.e. (9 z)), where En To Ppa).
Table 2:
Prompt Ensemble with a Pretrained Policy. The pretrained policies of each task are learned on 4 source domains. The Source column presents the performance for those source domains. In all evaluations, we use 40 unseen target domains. The Target column presents the performance for the unseen target domains not used for policy training.
(a) Zero-shot Performance in AI2THOR with Visual Navigation and Room Rearrangement Tasks
ObjectNav. (Aln.)
PointNav. (Not Aln.)
ImageNav. (Not Aln.)
RoomR. (Not Aln.)
Method
Source
Target
Source
Target
Souce
Target
Scoure
Target
Pretrained
17.5
65.8+19.1
95.3+ 98.9+1.0
80.9+166
77.2+3.3 79.2+1.4
56.2+2.2 61.6+1.1
87.3+3.1 93.
75.2=13.2
CONPE
88.4+1.7
7..
84.4+1.0
82.2-14.4
(b) Zero- shot Performance in egocentric Metaworld with 4 Different Robot Manipulation Tasks
Reach (Aln.)
Reach- Wall (Not Aln.)
Button-Press (Not Aln.)
Door-Open (Not Aln.)
Method
Source
Target
Source
Target
Source
Target
Source
Target
Pretrained
100.0:0.0
65. 06.7466.4
100.010.0
58.0
100.010.0
16.842.3
100.010.0
35.6=6.2
CONPE
100.0=0.00
74.7=5.0
100.0=0.0
75.749.0
100.0=0.0
73.748.3
100.0+0.0
93.2+t1.1
Table 2 reports zero-shot performance for the scenarios when pretrained policy is given. We evaluate two different cases: aligned (Aln.) when prompt-based contrastive learning is conducted on data from the same task of pretrained policy; otherwise, not aligned (Not Aln.). In AI2THOR, we use data from the object goal navigation task for prompt- baset contrastive learning, while each pretrained policy is learned individually through one of tasks including object goal navigation, point goal navigation, image goal navigation, and room rearrangement. Similarly, in Meteetrrri--ttaorrdd we use data from the reach task for prompt-hased contrastive learning, while each pretrained policy learned individually through one of tasks including reach, sachhwall,, button-press, and door-open. In Table 2(a) CONPE enhances zero-shot performance of the pretrained policies by 7.0% for unseen target domains in AI2THOR. This prompt ensemble adaptation requires only 400K samples, equivalent to 10% of the total samples used for policy learning. In Table 2(b] CONPE significantly in egocentric-Metavordd.
boosts zero-shot performance of the pretrained policies by
4.3
Ablation Study
Here we conduct ablation studies with AI2THOR. All the performances are reported in success rates.
Table 3: Prompt Ensemble Scalability
Table 4: Prompt Ensemble Methods
Source
Seen Target
Unseen Target
Ensemble Method
Source
Seen Target
Unseen Target
98.7++0.4
40.5++2.2
43.0+22.9
COM-UNI-AVG
12.9+11.22
51.4+7.6
12.77
5
96.1+
59.2+99.6
45.0+ 10.1
COM WEI-AVG
63.6+11.2
4.2
50.3+6.1
10
96.3+11.0
83.3+0.3
79.7+6..4
ENS-UNI-AVG
88.6#1.4
79.8+3.4
65.5#5.00
16
91.8++2.0
83.8+1.3
77.1+66.2
ENS-WEI-AVG
94.1.66.3
75.5
61.2.112.7
18
98.5411.8
83.312.
79.0:14.5
CONPE
96.3.11.
83 83.3.10.3
79.716.4 44
Prompt Ensemble Scalability. Table CONPE with respect to the number of prompts (n). CONPE effectively enhances zero-shot performance for both seen and unseen target domains through prompt ensemble that captures various domain factors. Compared to the case of n = 2, for
10, there was significant improvement in zero-shot performance for both seen and unseen target domains, with increases of 42.8% and 36.7%, respectively. For n 2 10, we observe stable performance that specifies that CONPE can scale for combining multiple prompts to some extent.
Prompt Ensemble Methods. Tabel compares the performance of various prompt integration methods [25 26 27 including our guided attention-basee prompt ensemble. We denote prompt-level integration as COM, and prompted embeddings-level integration as ENS. UNI-AVG and WEI-AVG refer to uniform average and weighted average mechanisms, respectively. CONPE achieves superior success rates over the most competitive ensemble method ENS-UNI-AVG, showing 3.5% and 14.2% performance gain for seen and unseen target domains.
8

---

Table 5: Prompt Ensemble Adaptation
Table 6: Semantic Regularized Data Augmentation
Optimization
Source
Target
6
w/o Semantic
Semantic
Pretrained
95.3+4.6
80.9+1.6
Source
Target
Source
Target
w/o Poot
59.
54.2:10.6
97.4 97.4++3.8
83.6+
100.0+0.0
84.1 10.2
0.2
94.7=0.0
77.6=12.8
94.8+7.4
79.749.1
w Poot
98.9+1100
84.4+1.0
84.213.7
75.318.8
96.111.9
83.1110.0
E2E
96.4+0.5
83.3
0.4
80.3416.2
74.0=1499
86.913.8
81.3112.3
Prompt Ensemble Adaptation Method. Table shows the effect of our ensemble adaptation method for the situation when pretrained policy given. As explained in Section 4.2 in this situation, CONPE can update the attention module with an additional prompt Prot- Note that Piot corresponds to this case, while w/o Pool corresponds to the other case of using the attention module without plot- In addition, E2E denotes the fine-tuning of both the policy and the attention module along with Prot- The results demonstrate that our method enhances the zero-shot performance of the pretrained policy, showing that Piu facilitates the extraction of task-specific features.
Semantic Regularized Data Augmentation. So far, we have only utilized vision data, but here, we discuss one extension of CONPE using semantic information. Specifically, we use few samples of object-level text descriptions to regularize the data augmentation process policy learning. This aims mitigate overfitting issues [8 29 The detailed explanations can be found in Appendix. As shown in Table CONPE with semantic data (w Semantic) consistently yields better performance than CONPE without semantic data (w/o Semantic) for all noise scale settings (6). Note that the noise scale manages the variance of augmented prompted embeddings. This experiment indicates that CONPE can be improved by incorporating semantic information.
Related Work
Adaptation in Embodied AI. In the literature of robotics, numerous studies focused on developing generalized visual encoders for robotic agents across various domains 3T). exploiting pretrained
visual encoders [32 333, and establishing robust control policies with domain randomized tech-
niques (34] 35| Furthermore, in the field of learning embodied agents, few works addressed
adaptation issues of agents to unseen scenarios in complex environments, using data augmentation
techniques [36] 37 38 39 (40 or adopting self-supervised learning schemes [41] 42, 43 Recently,
several works showed the fcasibility and benefits of adopting large-scale pretrained rision-language
models for embodied agents 77 44 45 46). Our work is in the same vein of these prior works of embodied agents, but unlike them, explore visual prompt learning and ensembles, aiming to enhance both zero-shot performance and sample-efficienyy.
Decoupled RL Structure. The decoupled structure, where state representation model is separated from RL, has been investigated in dision-based RL [47] 2 [6]. Recently, contrastive representation it allows expert behavior patterns to be
learning on expert trajectories gains much interest, incorporated into the state encoder even when a policy is not jointly learned [1 20). They established
generalized state representations, yet in that direction, sample-effiiinncy issues in both representation learning and policy learning remain unexplored.
Prompt-based Learning. Prospt-based learning or prompt tuning is a parameter-efficient opti- mization method for large pretrained models. Prompt tuning was used for computer vision tasks, optimizing few learnable vectors in the text encoder [18] and it was also adopted for vision transformer models to handle a wide range of downstream tasks [77] Recently, visual prompting (48) was introduced, and both visual and text prompt tuning were explored together in the multi-modal embedding space [49 50). We also use visual prompt tuning, but we concentrate on the ensemble of multiple prompts to tackle complex embodied RL tasks. We take advantage of the fact that the generalized representation capability of different prompts can vary depending on a given task and domain, and thus we strategically utilize them to enable zero-shot adaptation of RL policies.

---

6 Conclusion
Limitation. Our CONPE framework exploits visual inputs and their relevant domain factors for policy adaptation. For environments where domain changes extend beyond those domain factors, the adaptability of the framework might be constrained. our future work, we will adapt the framework with semantic knowledge based on pretrained language models to improve the policy generalization capability for embodied agents in dynamic complex environments and cover various scenarios associated with molli- agent interfaces.
Conclusion, In this work, we presented the CONPE framework, novel approach that allows embodied RI agents to adapt in zero-shot manner across diverse visual domains, exploring the ensemble structure that incorporates multiple contrastive visual prompts. The ensemble facilitates invarinnnvvaiant and task-specific state representations, thus enabling the agents to generalize to vi- sual variations influenced by specific domain factors. Through various experiments, we demonstrated that the framework can enhance policy adaptation across various domains for vision-based object navigation, rearrangement, manipulation tasks as well as autonomous driving tasks.
7 Acknowledgement
We would like to thank anonymous reviewers for their valuable comments and suggestions. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-01044 1022-0-00043 2020-0- 01821, 2019-0-00421) and by the National Research Foundation of Korea (NRF) grant funded by the MSIT (No. PRE-2020MSCIC2A000000 19, RS-2023-00213118)
References
[1] Adam Stooke et al. "Decoupling representation learning from reinforcement learning". In: Proceedings of the 38th International Conference on Machine Learning. 2021, pp. 9870-9879.
[2] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. "CURL: Contrastive unsupervised repre- sentations for reinforcement learning". In: Proceedings of the 37th International Conference on Machine Learning. 2020, pp. 5639-5650.
[3] Max Schwarzer al. "Data-efficient reinforcement learning with self-predictive representa- tions". In: arXiv preprint arXiv:2007.05929 (2020).
[4] Shangda Li et al. "Unsupervised domain adaptation for visual navigation". In: arXiv preprint arXiv:2010. 14543 (2020).
[5] Ziad Al-Halah, Santhosh K. Ramakrishnan, and Kristen Grauman. "Zero experience required:
Plug & play modular transfer learning for semantic visual navigation". In: Proceedings the 9th International Conference on Vision and Pattern Recognition. 2021, pp. 17010-17020.
[6] Qianfan Zhao et al. "Zero-shot object goal visual navigation". In: arXiv preprint arXiv:2206.077 074433 (2022).
[7] Apoorv Khandelwal et al. "Simple but Effective: CLIP embeddings for embodied AI". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2022, pp. 14809-14818. 18.
[8] Arjun Majumdar et al. "Zson: Zero-shot object-goal navigation using multimodal goal embed- dings". In: arXiv preprint arXiv:2206. / /2403 (2022).
[9] Eric Kolve et al. "Ai2-thor: An interactive 3d environment for visual ai".. In: arXiv preprint arXiv:1712.0522 04447444.444444 (2017).
[10] Alexey Dosovitskiy et al. "CARLA: An open urban driving simulator". In: Proceedings of the
1st Conference on Robot Learning. 2017, pp. 1-16.
[11] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press,
2018.
[12] Qihang zhang, Zheghao Peng, and Bolei Zhou. "Learning drive by watching youtube videos: Action-conditioned contrastive policy pretraining". In: Proceedings of the 17th European Conference on Computer Vision. 2022, pp. 111-128.
[13] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding". In: arXiv preprint arxiv: 1807.03383 (2018).
10

---

[14] Ting Chen et al. "A simple framework for contrastive learning of visual representations". In: Proceedings of the 37th International Conference on Machine Learning. 2020, pp. 1597-1607. [15] Tianhe Yu et al. Meta-Worlde A benchmark and evaluation for multi-task and meta reinforce- ment learning". In: Proceedings of the 3rd Conference on Robot Learning. 2019, pp. 1094 1100.
[16] Jinwei Xing et al. "Domain adaptation in reinforcement learning via latent unified state representation". In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021,
pp. 10452-10459.
[17] Menglin Jía et al. "Visual prompt tuning". In: Proceedings of the 17th European Conference on Computer Vision. 2022, pp. 709-727.
[18] Kaiyang Zhou et al. "Learning to prompt for vision-language models". In: International Journal of Computer Vision (2022).
[19] Bang You al. "Integrating contrastive learning with dynamic models for reinforcement learning from images". . In: Neurocomputing (2022).
[20] Minbeom Kim et al. "Action-driven contrastive representation for reinforcement learning". In:
PLOS ONE (2022).
[21] Rishabh Agarwal et al. "Contrastive behavioral similarity embeddings for generalization in reinforcement learning". In: arXiv preprint arXiv:2101. 201.22000..5665 (2021).
[22] Young Jae Lee et al. "STACoRe: Spatio-temporal and action-based contrastive representations for reinforcement learning in Atari". In: Neural Networks (2023).
[23] John Schulman et al. "Proximal policy optimization algorithms". In: arXiv preprint
arXiv:/707.06347 (2017).
[24] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. "A reduction of imitation learning
and structured prediction to no-regret online learning" In: Proceedings of the 14th Interna- tional Conference on Artificial Intelligence and Statisties. 2011, pp. 627-635.
[25] Pengfei Liu et al. "Pre-train, prompt. and predict: A systematic survey of prompting methods in natural language processing". In: ACM Computing Surveys (2023).
[26]
Minh-Thang
Thang Luong, Hieu Pham, and Christopher D Manning. "Effective approaches to attention-baat neural machine translation". In: arXiv preprint arXiv: 04025 (2015).
[27] Ashish Vaswani et al. "Attention all you need". In: Advances in neural information process- ing systems (2017).
[28] Denis Yarats, Ilya Kostrikov, and Rob Fergus. "Image Augmentation Is All You Need: Regu- larizing deep reinforcement learning from pixels". In: Proceedings of the 9th International Conference on Learning Representations. 2021.
[29] Samarth Sinha, Ajay Mandlekar, and Animesh Garg. "S4RL: Surprisingly Simple Self- Supervision for Offline Reinforcement Learning in Robotics". In: Proceedings of the 5th Conference on Robotics Learning. 2021, pp. 907-917.
[30] Arjun Majumdar et al. "Where are we in the search for an Artificial Visual Cortex for Embodicd Intelligence?" ?" In: arXiv preprint 2rX (2023).
[31] Suraj Nair al. "R3m: A universal visual representation for robot manipulation". In: arXiv preprint arXiv:2203: 12601 (2022).
[32] Austin Stone et al. "Open-world object manipulation using pre-trained vision-language mod- els". In: arXiv preprint arXiv:2303. 00905 (2023).
[33] Jiange Yang et al. "Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick- Robots". In: arXiv preprini arXiv:2306.0577 2506661666 (2023).
[34] Dhruv Shah al. "Gnm: A general navigation model to drive any robot". In: Proceedings of the IEEE International Conference on Robotics and Automation. IEEE. 2023, pp. 7226-72677333
[35] Noriaki Hirose et al. "ExAug: Robot-conditioned navigation policies via geometric experi- ence augmentation" " In: Proceedings of the IEEE International Conference on Robotics and Automation. IEEE. 2023, pp. 4077-4084.
[36] Daniel Fried al. "Speaker-follower models for vision-and-languuage navigation". In: Pro- ceedings of the 31th Conference on Neural Information Processing Systems. 2018, pp. 3318- 3329.
[37] Felix Yu et al. "Take the scenic route: Improving generalization in vision-and-language navigation". In: arXiv preprint arXiv:2003. 14266 (2020).

---

[38]
Xiaofeng Gao et al. "Dialfred: Dialogue-ente enabled agents for embodied instruction following". In: arXiv preprint arXiv:2202.13330 (2022).
[39]
Chong Liu et al. "Vision-language navigation with random environmental mixup". In: Pro- ceedings of the International Conference on Computer Vision. 2021, pp. 1624-1634,
[40] Jialu Li, Hao Tan, and Mohit Bansal. "EnvEdit: Environment Editing for Assion-add-Languaa
Navigation". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. pp. 15386-15396.
[41] Xin Wang et al. "Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation". : In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 0629-6638.
[42] Roberto Bigazzi et al. "Explore and explain: self-supervised navigation and recounting". In: Proceeding of the 25th International Conference on Pattern Recognition. 2020, pp. 1152-1159.
[43] Eun Sun Lee al. "MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agents". In: Proceeding of the 17th European Conference Computer Vision. 2022,
pp. 338-354.
[44] Vishnu Sashank Dorbala et al. "CLIP-Nav: Using CLIP for zero-shot vision-and-- and-language navigation". In: arXiv preprint arXiv:2211.1666 (2022).
[45] Samir Yitzhak Gadre al. "CLIP on Wheels: zero-shot object navigation as object localization and exploration". In: arXiv preprint vrXiv:2203.1042 (2022).
[46] Dhruv Shah et al. "Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action". In: Proceedings of the 6th Conference on Robot Learning. 2022, pp. 492- 504.
[47] Irina Higgins al. "DARLA: Improving zero-shot transfer in reinforcement learning". In: Proceedings of the 34th International Conference on Machine Learning. 2018, pp. 1480-1490.
[48] Hyojin Bahng et al. "Exploring visual prompts for adapting large-scale models". In: arXiv preprint arXiv:2203. 17274 (2022).
[49] Muhammad Uzair Khattak et al. "MaPLe: Multi-modal prompt learning.. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2023.
[50] Yuhang Zang et al. "Unified vision and language prompt learning." In: arXiv preprint
12X2:22200.77225 (2022).
12


Reviews:
[{"review_id": "5ENXFVlbG2", "reviewer": "Reviewer_QsRA", "summary": "This paper presents a novel framework for “efficient policy adaptation”, i.e. adapting learned policies to changes in the domain (e.g., lighting, camera height etc.), using a very small amount of in-domain data (few-shot and zero-shot). The proposed approach comprises learning “visual prompts” for modeling each different kind of variation in the domain, and using an attention head over these prompts to learn policies that are invariant to these factors. Empirical analysis in three different settings (AI2THOR, Metaworld and CARLA) show that the proposed method outperforms other representation learning alternatives.", "questions": "Please see the concerns and questions raised in the detailed weaknesses section above.", "limitations": "**I did not find limitations** or societal impact sections (the latter is likely not required) in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper studies a problem that is central to the sim2real generalization of robotic policies trained in simulation: how well do these policies generalize to varying conditions in the domain that are out of the agents control. The proposed approach, while having a lot of moving parts, seems to be extremely data-efficient (~10 expert episodes and small amount of diversity used), which can make a strong case for real-world applications. I also found the ablation analysis to be thorough and convincing regarding the design decisions made.", "weaknesses": "- [Method Presentation, Reproducibility] My biggest concern with the paper in it’s current form is the poor presentation of method details and reproducibility. I appreciate that the authors have provided code in the supplemental material, which will really help reproduce the results and build on them, but as a research article of novel, scholarly work, the paper is extremely hard to parse and missing on several key details. Updating these and clarifying the integral components of the paper better would be imperative to the impact of the paper. Some questions:\n\t- In Section 3.2, it is not clear what the learned parameters are: only the prompts $p^v$, or both prompts and $\\mathcal{T}_\\phi$. The objective only says $p^v$, but the text says “we fine-tune $\\mathcal{T}_\\phi$ by learning a visual prompt…” This is _really_ important to understanding why the method works: if T is indeed being finetuned, how is this standardized across the tasks/domains? Why would $p^v$ learn anything useful or interpretable at all?\n\t- I don’t think it is ever mentioned what specific pre-trained representation $\\mathcal{T}$ actually is. At some point, the authors say it is CLIP but it is not clear how CLIP is used. CLIP consists of two encoders (text and visual): which of these is $\\mathcal{T}$? It also seems to be overloaded as something with one or two arguments, and while this is somewhat understandable in context, this should be clarified for correctness.\n\t- “each visual prompt in $p^v$ encapsulates a specific domain-invariant knowledge” is a somewhat strong statement. Is there any reason this is interpretably something domain-invariant? Empirical/illustrative insights would really be helpful here, or even a qualitative discussion.\n\t- [L141-143] This statement is very convoluted and seems unclear. Can the authors very concretely define what they mean by “task” and “domain”? Maybe this is well-understood by a sub-community, but certainly seems vague to me in the context of a representation learning paper. \"As G is optimized by a given RL task objective in the source domains (in line 12), the resulting Z tends to be task-specific, while Z is also domain-invariant by the ensemble of contrastively learned visual prompts based on G with respect to various domain factors.”\n\n- [Prior work] In the introduction, the authors make some rather bold claims that I think should be qualified. In [L27-30], the authors say “It was rarely investigated how to optimize…, to ensure the zero-shot capability of embodied agents”. Two thoughts here:\n\t- It should be clarified if this statement is regarding zero-shot generalization of visual encoders or entire policies. This has indeed been studied by many works: both for encoders [5] and policies, most recently in the navigation/embodied AI settings by [1, 2]. It would be nice to acknowledge some of this prior work and position the paper more carefully in their context (not saying there are no differences or it reduces the contributions of this paper).\n\t- And for what it’s worth, the current paper does not really address this. The closest reported results analyze how policies trained with a small set of domain properties generalize to slightly different properties.\n\n- This trend continues with statements like “egocentric visual domain changes experienced by embodied agents due to different camera positions”. This has indeed been studied for visual encoders [5, 6] trained on diverse data such as Ego4D, as well as for policies [1, 2, 3] (both these papers specifically studied the case of different camera positions and FoV, and even different embodiments etc.).\n\n- [Task & eval Description] Some of the lack of key information seeps into the evaluations as well. Separating it for better readability. I was unable to interpret the results without getting more clarity on what the experiment setup is and what is being reported. Please see some specific concerns below:\n\t- What do tasks/domains etc. mean specifically for each of the environments considered in the paper? Some details seem to be in the appendix, but even a brief discussion for completeness in the main paper with pointers to appendix should suffice.\n\t- I don’t think the specific task in CARLA is defined anywhere, even in the appendix.\n\t- What does “zero shot” mean in Section 4.1? Is it trained in the same environment for the same task but with different lighting conditions/camera parameters etc.? Or is everything different? What is the exact experiment? The text as well as the captions are uninformative about this (e.g., “The Seen Target column presents the performance for the seen target domains, and the Unseen Target column presents the performance for the unseen target domains.”) I suspect that the words “zero-shot performance” are too vague and broad and mis-communicate some of the findings. I would like to see some clarity on this before I can interpret the results.\n\t- The caption in Fig 5 is pretty hard to parse, could the authors clarify what is being visualized in the left image and why it indicates \"domain-invariant knowledge” ?\n\t- What is the “pretrained policy” in Section 4.2 actually pretrained on? It seems like individual policies are used for each of the different tasks in each environment, so what exactly is being “pretrained”? It’s really not clear what the distinction between pretrain/prior results are, or what tasks/domains mean and what the exact experiment is.\n\t- In the experiments, I see that the policy training used ~10 episodes per task. How much data was used for training the visual prompts, though? Seems like that would be quite high (which is fine since this is likely done procedurally in sim).\n\n- A broader question regarding the setup of the experiments to me is how much of an improvement a sophisticated scheme like the one proposed over something more naive, like (i) learning a parametric policy that simply conditions on the different factors of variation [3], or (ii) training a joint policy on data from multiple domains across all these parameters, akin to domain randomization, [1, 2] so the factors of variation can be learned to be ignored. I imagine either of these “simpler” methods would require more data, but given this is a study in sim where data is free, does the proposed method help in certain ways? An analysis of this could be really important.\n \n[1] Putta et al., “Emergence of Implicit System Identification via Embodiment Randomization”, 2023\n\n[2] Shah et al., “GNM: A General Navigation Model to Drive Any Robot”, 2022\n\n[3] Hirose et al., “ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentation”, 2022\n\n[5] Majumdar et al., “Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?”, 2023\n\n[6] Nair et al., “R3M: A Universal Visual Representation for Robot Manipulation”, 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness 1-1.**\nIn Sec. 3.2, when using the phrase \"fine-tune $\\mathcal{T}_{\\phi}$\", it denotes the tuning process of visual prompts, a process we term visual prompt-tuning. This does not encompass parameter updating $\\phi$ in CLIP visual encoder $\\mathcal{T}$. Via the visual prompt-tuning, the encoder's parameters $\\phi$ remain frozen, precluding  modification.\nTo eliminate any misunderstanding, we will substitute fine-tuning with prompt-tuning.\n### **Weakness 1-2.**\nWith CLIP visual encoder, we adopt the prompt-based contrastive learning to extract features pertinent a specific factor. We construct an ensemble of the visual prompts to manage domain shifts brought about by a combination of these factors. By learning each visual prompt with a few demos (while CLIP encoder is frozen), we can encapsulate knowledge pertinent to a specific factor within a prompt.\nWe also use the CLIP text encoder for Semantic Regularized Data Augmentation in Table 6.\n### **Weakness 1-3.**\nTo mitigate discrepancies between domains, we incorporate contrastive learning into our adaptation approach. It is consistent with [7,8], for which contrastive learning serves for aligning the representations from domains, enabling cross-domain transfer of knowledge.\nWe empirically prove that our framework generalizes for diverse domain scenarios in Sec. 4.1 and Appendix F. Those include 1) Alignment visualization for prompted embeddings of different domains, 2) Robustness of the ensemble representation across different domains w.r.t a specific domain factor, and 3) Interpretability of attention weights prompt ensemble.\n### **Weakness 1-4.**\nOur work aligns with visual RL that \"task'\" refers to a specific objective of an agent (e.g., Object Goal Navigation), and determined by rewards.\nA \"domain\" pertains to the visual observation space and observation probability (e.g., domains with a high camera position, the agent receives high position camera views.); each is characterized by a combination of factors, as in Lines 86-94.\nWe include the definition table in PDF as Global Author Rebuttal.\n### **Weakness 2-1.**\nWe appreciate the reviewer's valuable feedback on related works in robotics.\nWe will revise the Introduction and Related Work to articulate the position of our works with references [1, 2, 3, 5, 6].\nOur work distinguishes itself from the prior works in that the visual encoder's prompts are only updated for policy adaptation, and it provides a novel parameter-efficient, sample-efficient RL adaptation structure, while leveraging pre-trained visual models.\n### **Weakness 2-2.**\nGeneralizing for each individual domain factor can provide a basic understanding, but we presume it doesn't necessarily prepare an agent for intricate multi-factorial situations; e.g., mastering driving on wet roads focuses on a single factor (road wetness), and the challenges associated with it is not the same as navigating scenarios characterized by combinations of multiple factors: weather, road conditions, and car types.\n### **Weakness 4-3.**\n\"zero-shot\" pertains to the evaluation scenarios where a policy is evaluated in target domains, not encountered during the training.\nAs in Figure 2, our training consists of two phases: (i) visual prompt-tuning (prompt-based contrastive learning) and (ii) policy's joint learning with the prompt ensemble (guided-attention-based prompt ensemble).\nIn evaluation, the target domains are either \"unseen target domains\" or \"seen target domain\": the former denotes entirely new domains, which were not encountered during both (i) and (ii) training. In contrast, the latter denotes new domains, which are encountered during (i) but not during (ii).\n### **Weakness 4-4.**\nIn Figure 5(a)-1), we visualize prompted embeddings of two distinct domains, both defined by different configurations of the same factor, to evaluate the alignment; prompted embeddings from these two domains align closely.\nFigure 5(a)-2) illustrates that each prompt distinctly clusters prompted embeddings corresponding to domains associated to its domain factor.\nBoth visualizations provide insight into how these prompts capture domain-invariant knowledge with respect to a specific domain factor.\nFigure 5(b) illustrates the visualizations of attention weights during zero-shot adaptation. Each prompt receives varying attention weights across diverse domains, indicating ConPE's ability to dynamically adjust its focus using the perceived domain that is configured by multiple domain factors.\n### **Weakness 4-5.**\nIn Sec. 4.2, \"pre-trained\" denotes a policy fully trained in the source domains for a specific RL task.\n(i.e., associated with a reward function).\nFor a pre-trained policy, the model architecture for training it employs the CLIP visual encoder without prompts. This is different from the other cases of zero-shot performance, in Sec. 4.1, where a policy is jointly learned with the prompt ensemble through an RL objective.\nConPE supports two methods: joint learning in Sec. 4.1 and tuning of the prompt ensemble with a pre-trained policy in Sec. 4.2.\n### **Weakness 4-6.**\nThe datasets comprise 28,464 samples for AI2THOR, 4,700 samples for Metaworld, and 73,900 samples for CARLA, respectively (details in Table 11-13 of Appendix.). \n### **Weakness 5.**\nRegarding (i): as shown in [3], the policy conditioned on known specific parameters is quite intuitively structured in the perspective of multi-task learning, where a distinctly parameterized condition can be used as input to infer task-specific contexts of multi-tasks.\nUnlike this parametric policy, we do not assume any known configurations of domain factors for target domains. This evaluation assumption is consistent with zero-shot policy adaptation in target domains.\nRegarding (ii): we will include test on domain randomized encoders in revised manuscript.\n\n[7] Kang et al. \"Contrastive adaptation network for unsupervised domain adaptation.\", 2019\n\n[8] Thota et al. \"Contrastive domain adaptation.\", 2021"}, {"review_id": "lziqhK9knw", "reviewer": "Reviewer_TuT5", "summary": "This paper presents a method for improving zero-shot domain adaptation for Embodied AI agents by learning a task-specific state representation.\n\nSpecifically, $n$ visual prompts are learned by fine-tuning the agent's visual encoder (i.e. CLIP) contrastively on observations. Each visual prompt is specific to a \"domain factor\" (such as camera position or illumination). Observations are positively paired if their state is the same (but the observations themselves may differ depending on domain factors).\n\nThen, an ensembling technique is used to compute a task-specific state representation. This is an average of the prompt encodings, weighted based on their cosine similarities to the observation encoding. This is used to optimize the policy using RL.", "questions": "- It is said that $T_\\phi$ is \"fine-tuned\" but I believe this must be frozen and only the visual prompts are fine-tuned. Is that correct? Should be clarified if so.\n\n- Could the authors please clarify whether they use RoboTHOR or iTHOR for the AI2-THOR benchmarks? E.g. the ObjectNav task is available for both environments.", "limitations": "As far as I understand, this method may not be generalizable between simulators/environments, because the domain factors may not be compatible between these. Future work will be necessary to help improve such generalization -- fine-tuning the visual encoder offline is likely a valid approach, but factor-specific visual prompting or ensembling is not.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method presented in this paper is relatively simple and appears to provide significant gains over SOTA for both source and (seen/unseen) target domains on several benchmarks (AI2-THOR tasks, egocentric-Metaworld, CARLA).\n\n- To the best of my knowledge, visual prompting has not previously been explored in Embodied AI. This paper also emphasizes learning visual variations that are specific to domain factors (and shows that this helps generalize to unseen domains).\n\n- This paper includes several ablations, e.g. for prompt ensemble scalability and ensembling methods.\n\n- Zero-shot domain adaptation is a necessary and under-explored capability in Embodied AI, as it has many downstream applications (e.g. Sim2Real for robotics).", "weaknesses": "- It's a bit hard to interpret the sample efficiency experiment in Fig. 4. For the seen target domain, it's true that the performance is higher for ConPE than EmbCLIP at 0.5M steps, before either method has converged (approx. at 1M steps). But it's a bit sparse, as success is only measured every 0.5M steps (and the performance for ConPE is always higher than EmbCLIP anyway). Something somewhat similar could be said for the unseen target domain. Also, what benchmark does this correspond to? If this is averaged over several benchmarks, these should probably be split into different plots.\n\n- I believe the details related to the domain factors could be clarified. This seems to be an important component of the methodology and several things are unclear: what is the full list of domain factors (and do these vary per environment?), how were these domain factors chosen, what is the variance per domain factor within one environment (are these discrete or continuous values? are they uniformly sampled per observation?), and how were these subsampled for the experiment in Table 3. Most importantly: how does visual prompting perform without any domain factors? I.e. what are the baseline metrics if you fine-tune a single visual prompt for each environment?\n  - These points are particularly important for the scientific contribution of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author response for Weakness 1.**\n|Method|0.5 M|0.6 M|0.7 M|0.8 M|0.9 M|1.0 M|\n|-|-|-|-|-|-|-|\n|EmbCLIP    |$54.38$|$60.35$|$60.00$|$66.00$|$66.38$|$70.72$|\n|ATC        |$56.61$|$54.30$|$54.12$|$52.54$|$61.61$|$60.45$|\n|ConPE      |$64.52$|$68.16$|$69.91$|$71.58$|$71.50$|$75.53$|\n\nWe appreciate this reviewer's valuable feedback on the sample efficiency in Figure 4 of our manuscript.\n\nFigure 4 shows the sample efficiency for the Object Navigation task in AI2THOR.\nTo clarify the sample efficiency, we measure the policy's performance at each 0.1 million timesteps during 1 million timesteps, as illustrated in the tables above.\n\nFor the seen target domains, we observe that ConPE requires, on average, 30.0\\% fewer samples to achieve approximately 70.0\\% success rate, compared to the most comparative baseline. These results indicate that the framework is the most sample efficient for zero-shot adaptation to seen target domains.\n\nWe will update Figure 4 in our revised manuscript, showing the detailed learning progress with more plots during the initial stages before 1.5 million timesteps for both seen and unseen target domains.\n\n### **Author response for Weakness 2-1**: [list of domain factors, how chosen, what is the variance (discrete or continuous values, sampling)]\nThe examples of domain factors for each environment (i.e., AI2THOR, Metaworld, CARLA) are provided in Appendix D. We will include the domain factor list (i.e., Camera field of views, stride length, rotation degrees, and illuminations in AI2THOR; Camera positions, gravity, wind speeds, and illuminations in Metaworld; Camera positions, camera field of views, weather conditions, times of day, and different ranges of action magnitude in CARLA.) in our revised manuscript.\n\nWe have chosen these domain factors based on a couple of considerations: how significantly they affect domain shifts in relation to the agent's performance, and how effectively we can create different domain data for each domain factor through diverse configurations of the simulators we use. We also consider previous studies in embodied RL [1, 2, 3] in which domain factors like camera position, camera FoV, and stride length were used.\n\nEach domain factor can be represented as either discrete or continuous values, depending on its inherent nature. For example, we treat weather conditions as a discrete factor, which can be classified as either clear, rainy, cloudy, or other; Conversely, we represent brightness as a continuous factor, with a range extending from $0.0$ to $1.0$.\n\nRegarding sampling methods, these domain factors are individually drawn from a uniform distribution to produce combinatorial domain variations within the simulation environment.\n\nWe will detail these domain factors including the sampling method in Appendix of our revised manuscript.\n\n### **Author response for Weakness 2-2**: [experiment in Table 3]\nIn Table 3 of our manuscript, $n$ represents how many prompts we use for training domain factors within the AI2THOR object navigation environment. In this experiment, we use total 18 different domain factors in AI2THOR to show the scalability of our framework in terms of multiple visual prompts.\n\nFor example, when $n=5$, the domain factors used include illuminations, FoV, stride length, rotate degree, and look up/down degree, thus requiring the respective 5 individual prompts in our ensemble structure.\n\n### **Author response for Weakness 2-3**: [without domain factors, with a single visual prompt]\nFor a specific domain in the environment, it is feasible to fine-tune the corresponding visual prompt for policy adaptation, aiming to improve the performance of the source domain. However, it is expected that the tuned visual prompt easily overfits to the source domain, that is trained.\n\nAs our objective is to ensure robust zero-shot performance for unseen target domains other than the source domain, this situation is not desirable. The zero-shot performance can significantly degrade when overfitted. Thus, we explore an ensemble structure of multiple visual prompts, each encapsulating knowledge pertinent to its respective domain factor.\n\nIn fact, if we showed the case of $n=1$ in Table 3, it would correspond to the single prompt case. Given the low performance result by the other case of $n=2$ in the table, we expect a relatively low performance by the case of $n=1$ for seen and unseen target domains. We will confirm this in our revised manuscript.  \n\nWe will also expand this comparison including the single prompt optimized to the source domain as well as the domain-randomized single prompt in our revised manuscript.\n\n### **Author response for Question 1.**\nAs noted by the reviewer, the visual encoder $\\mathcal{T}_{\\phi}$ remains frozen and only the visual prompts are updated, same as other visual prompt-tuning approaches [4, 5, 6].\n\nWe will use the term \"visual prompt-tuning\" than \"fine-tuning\" in our revised manuscript. This term more accurately conveys our intent, tuning only the visual prompts, as opposed to the encoder parameters.\n\n### **Author response for Question 2.**\nWe use iTHOR for the AI2-THOR benchmarks, because iTHOR involves more diverse object types (e.g., fridge, Window, Stove burner) and tasks (e.g., Object Goal Navigation, Point Goal Navigation, Room Rearrangement, Image Goal Navigation), compared to RoboTHOR.\n\n[1] Chattopadhyay, Prithvijit, et al. \"Robustnav: Towards benchmarking robustness in embodied navigation.\", 2021.\n\n[2] Shah, Dhruv, et al. \"Gnm: A general navigation model to drive any robot.\", 2023.\n\n[3] Julian, Ryan, et al. \"Efficient adaptation for end-to-end vision-based robotic manipulation.\", 2020\n\n[4] Jia, Menglin, et al. \"Visual prompt tuning.\" , 2022\n    \n[5] Sohn, Kihyuk, et al. \"Visual prompt tuning for generative transfer learning.\", 2023.\n    \n[6] Peng, Xiangyu, et al. \"Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning.\", 2022"}, {"review_id": "bpAhaz6LVu", "reviewer": "Reviewer_aYZP", "summary": "The authors propose a method to address zero shot adaptation to novel visual domains. They do so by:\ni) Learning a set of visual prompts via contrastive learning that encode specific domain invariant knowledge.\nii) Using a prompt ensemble based on embedding similarity scores (attention)", "questions": "\n* Why was CLIP the choice of Visual encoder?\n* Could you elaborate on the expert dataset used to do the prompt based contrastive learning?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors have incorporated the idea of image prompting to improve the model's capacity to adapt to novel environments in an effective manner.\n* The experiments have been conducted on re implemented baselines from other works with all the hyperparameters specified in the appendix, which is deeply appreciated.\n* The diagrams are a big help and very well done!", "weaknesses": "* I wish there were a deeper analysis on understanding/ interpreting the prompts better. Maybe something like visual analyses of images that tended to use similar prompt clusters thus seeing what invariant information the prompt encoded. \n* I think the reader would benefit from a deeper understanding of the actual differences in the seen vs unseen domains, to get a sense of how similar or different they are. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author response for Weakness 1.**\nIn Figure 5(a)-1) of our manuscript, we visualize intra prompted embeddings from two distinct domains, both defined by different configurations of the same domain factor, to evaluate their alignment. The figure specifies that the prompted embeddings from these two domains align closely. For example, consider a scenario where the domain factor is \"weather\". The prompted embeddings might represent two different weather conditions, \"clear\" and \"rainy\", each presenting notably different visual representations. Even though these are two distinct weather scenarios, the embeddings align closely. This suggests that the prompt effectively captures the essence of weather as a domain factor. We include additonal examples in PDF as Global Author Rebuttal.\n\nFigure 5(a)-2) illustrates that each prompt distinctly clusters prompted embeddings corresponding to domains associated to its domain factor. If we consider multiple domain factors like \"camera position\", \"weather\", and \"time of day\", each prompt would cluster embeddings specific to its domain factor. For instance, one prompt might cluster all \"rainy\" observations, placing them nearer to each other compared to observations from different \"camera position\" or \"time of day\".\n\nWe will include additional examples for intra and inter prompted embedding representations in Appendix of our revised manuscript.\nThese visualizations provide insight into how these prompts capture domain-invariant knowledge with respect to a specific domain factor. They also demonstrate the efficacy of a similarity-based guidance during the ensemble process.\n\nFigure 5(b) illustrates the visualizations of attention weights during zero-shot adaptation.\nEach prompt receives varying attention weights across diverse domains, indicating ConPE's ability to dynamically adjust its focus using the perceived domain that is configured by multiple domain factors.\nFor example, when the agent navigates in a \"rainy\" and \"night\" domain environment, the attention weights are likely higher for the prompt specifically trained for \"weather\" and \"times of day\" factors, as opposed to when navigating in a \"high camera position\" setting, where the prompt relative to \"camera position\" might be more emphasized.\n\nThis demonstrates consistent assignments of attention weights to the prompted embeddings throughout the timesteps while adapting to the current domain.\n\n### **Author response for Weakness 2.**\nIn Tables 11-13 in Appendix, we present the configurations of our datasets including environment image samples and specific configurations used to set the values of domain factors.\n\nIn the CARLA environment, we incorporate \"weather conditions\" and \"times of day\" as domain factors. The distinction between seen and unseen domains, combined by these two domain factors, aligns with the seen and unseen domain configurations presented in reference works [1, 2]. For example, \"noon\" and \"sunny day\" could be a seen domain, and \"night\" and \"rainy weather\" could be a unseen domain. As such, the differences between seen and unseen domains depend on configurations of domain factors in each simulator.\nIt is also worth noting that our domain specification based on combinatorial configurations of multiple domain factors is scalable to include various domains, so other factors such as \"camera positions\", \"camera field of view\" and \"different ranges of action magnitude\" are seamlessly integrated, provided that they are supported and configurable in CARLA. We include domain examples in the PDF.\n\nWe will include a detailed table to describe the differences of the seen and unseen target domains in Appendix of revised manuscript.\n\n### **Author response for Question 1.**\nWe use the CLIP model as our visual encoder since our primary intent is to leverage the capabilities of pre-trained visual encoders, which have seen significant advancements in the fields of computer vision, aiming to make the decoupled RL model structure more suited for zero-shot policy adaptation across domains.\nOur focus is on parameter-efficient approaches using visual prompt-tuning and ensembling for policy adaptation.\n\nSpecifically, CLIP's Transformer architecture allows us to employ visual prompting (in a way that the prompt is incorporated into tokenized input embeddings) without much modification to the architecture.\n\nIt is also worthwhile to note that our framework can accommodate other pre-trained encoders than CLIP, especially those based on the Transformer structure.\n\nIn Table below, we conduct the experiment with another Transformer-based model (ViT [3]) learned on the ImageNet dataset. As shown, while ViT shows competitive performance for the source domains, its performance degrades significantly in zero-shot adaptation.\n\nWe conjecture that this degradation of ViT is attributed to the relatively small size of training datasets (1,281,167 samples) used for visual encoder training. In contrast, CLIP is trained on the dataset of 400 million samples, thereby demonstrating robust generality.\n\n|Method|Source|Seen Target|Unseen Target|\n|-|-|-|-|\n|ViT[3] |$96.41 \\%$|$27.34 \\%$|$24.44 \\%$|\n|ConPE  |$96.11 \\%$|$87.50 \\%$|$73.21 \\%$|\n\n### **Author response for Question 2.**\nThe statistics of our expert datasets used for prompt learning are detailed in Tables 11-13 in Appendix.\nEach dataset contains expert demonstrations from multiple domains, where each domain is determined by varying a specific domain factor.\n\nWe use 28,464 samples for AI2THOR, 4,700 samples for Metaworld, and 73,900 samples for CARLA, and regarding to the experts, we devise rule-based policies tailored to each environment.\n\n[1] Xing et al. \"Domain adaptation in reinforcement learning via latent unified state representation.\", 2021.\n\n[2] Li et al. \"Domain adaptive state representation alignment for reinforcement learning.\", 2022\n\n[3] Dosovitskiy et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\", 2020"}, {"review_id": "jn9jc7eOb8", "reviewer": "Reviewer_okL6", "summary": "The authors propose an approach for rapid policy adaptation to unseen visual observations for embodied agents. Their approach called CONPE (contrastive prompt ensemble) leverages contrastive learning to learn an ensemble of prompts for policy visual encoders that enables the policy to ultimately adjust to the domain shift. The results are demonstrated on multiple simulated environments, along with various ablations. ", "questions": "- It would be helpful to define source, seen and unseen target domains more concretely for Tab.1. I am a little confused between source and seen. Likewise, it would be useful to define what the authors mean by zero-shot for Tab.1 — aren’t a prompt pool and policy are jointly learnt for CONPE in Tab.1? \n- L255: Unclear what ENS approach is. Please provide a 1-line description with a reference, if it is from an existing paper.\n- Tab.5:  Shouldn’t E2E be the upper bound? Why does it have lower performance than CONPE?\n", "limitations": "(see also the weakness section)\n- This might be out of scope for current work but the results are only shown on simulated environments. Would be great to see some discussion on how CONPE might work in real world.\n- A limitation section is also missing.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work addresses a well-motivated problem, which the authors explain well.\n- The authors’ build up on advances in contrastive learning to develop a simple yet novel approach for policy adaption in embodied settings.\n- The authors demonstrate their approach in multiple environments/tasks: AI2THOR, CARLA, and Metaworld.\n- The authors also present extensive experiments, ablations, and analysis to provide a holistic understanding of their approach. ", "weaknesses": "- The terminology used in the paper is confusing at places. The authors never define what they mean by a “prompt” or “zero-shot performance”. This is especially confusing in Sec. 3 where they explain the framework. In the overview in Sec.3.1 they talk about zero-shot policy deployment (L99). However, Sec.3.2 only talks about jointly learning the policy and the guided attention for the prompt ensemble (also main results tab.1 — see my comment in questions section on this). Overall, since the authors also use specialized data for contrastive learning of domain-invariant prompts to later enable policy adaptation (through joint learning of policy), I am wondering whether their approach can really be called zero-shot. Despite this, I do think their approach is useful for policy adaptation amongst domain shifts.  Hence, I recommend an acceptance. However, I encourage the authors to clarify their terminology and claims especially around “zero-shot”.\n- The paper is also missing a limitation section. It is unclear where CONPE struggles. I encourage the authors to add one. What happens when a domain factor is varied beyond a degree present in the prompt learning data? e.g., a camera with much smaller view or more pan-tilt than present in the prompt training data. Lastly, I would also be interested in understanding how CONPE might work in real-world for instance, for robots. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author response for Weakness 1. and Question 1. [prompt and zero-shot]**\nIn Line 122 in Section 3.2, the term \"prompt\" is defined as a learnable vector appended to the input, with the intent of extracting domain-invariant knowledge from pre-trained visual encoders. This definition of prompt is commonly used in the context of pre-trained language and vision models, particularly in [1, 2, 3] where visual prompt-tuning is used similarly to our work.\n\nIn our work, \"zero-shot evaluation\" pertains to the evaluation scenarios where a policy is evaluated in target domains, which were not encountered during its training. Accordingly, zero-shot evaluation occurs immediately, without any additional training.\n\nAs clearly illustrated in Figure 2 of our manuscript, our training process consists of two training phases: (i) visual prompt-tuning (prompt-based contrastive learning) and (ii) policy's joint learning with the prompt ensemble (guided-attention-based prompt ensemble).\n\nIn policy train, \"source domain\" is comprised of the domain factors that have seen in prompt learning.\nIn our evaluation, the target domains are categorized as two different sets: \"unseen target domains\" and \"seen target domain\": the former denotes entirely new domains, which were not encountered during both (i) and (ii) training. In contrast, the latter denotes new domains, which are encountered during (i) but not during (ii).\nSuch categorization of target domains aligns with the prior works in the application of visual domain RL adaptation such as [4, 5]. Note that the zero-shot performance for these target domains is represented in different columns in Table 1(a)-(c) of our manuscript.\n\nAs formally defined in Section 2 of our manuscript, each domain is represented by a POMDP, and domain shifts are formulated by varied conditional observation probabilities. We also formulate each domain as an MDP (POMDP), defined by tuples of state, action, reward, and others, as same as [4, 5]. Domain examples in our datasets can be found in Tables 11-13 of Appendix.\n\nFor clarity, we will update this explanation on zero-shot evaluation and target domain categorizations in our revised manuscript.\n\n### **Author response for Weakness 2. and Limitations [missing limitation section]**\n\nWe will highlight the limitations of our approach in the Conclusion Section, as recommended by the reviewer. Indeed, an inherent limitation of our ConPE framework is its architectural dependence on visual inputs. While this offers significant advantages in specific contexts, it might introduce challenges when the visual input is obscured and significantly varied as the reviewer mentioned.\n\nAs briefly described as our future direction in the Conclusion Section of our manuscript, we consider semantic representations to tackle the limitations of visual input dependence. As a next step, we are keen on exploring the capabilities of multi-modal pre-trained models to further enhance embodied policy adaptation.\n\nAs for adapting to field of view settings, we conduct several experiments in the AI2THOR environment, where we manipulate an agent's field of view to create either abnormally small (narrow observations) or excessively large (wide observations) views. Our ConPE framework shows slightly better performance than other baselines in both extreme cases, but its robustness appears to be variable.\n\nSimilarly, for pan-tilt adjustments, we test with the Metaworld and CARLA environments. We observe performance inconsistency in Metaworld, with limited views surrounding a robotic arm. On the other hand, the CARLA environment demonstrates better resilience in ConPE's performance, likely due to the comprehensive camera settings capturing a wider range of relevant visual information.\n\nWe also notice this inconsistency in performance could be attributed to the differences in image rendering quality and the volume of visual information available among the simulators. Further in-depth investigation is needed, as the reviewer queries, to achieve robustness across diverse simulators and environments.\n\n### **Author response for Question 2. [ENS explanation]**\nENS (ENSenble) refers to an ensemble method conducted on prompt embeddings, akin to the approach in [6], wherein prompts are integrated at the embedding-level.\nWe further categorize this approach to ENS-UNI-AVG and ENS-WEI-AVG. In the former, the prompt embeddings are integrated using uniform average, while the latter, weighted averaging with attention is employed to integrate prompt embeddings.\n\n### **Author response for Question 3. [performance score]**\nWe notice such performance degradation in the end-to-end (E2E) setting, which is attributed to the necessity of fitting a larger number of parameters to the source domain. Updating a large model can lead to a decrease in overall generalization capability.\n\nThis observation is consistent with findings reported in the RL literature [7], where the generalization capability tends to degrade after fine-tuning the entire model with a large number of parameters.\n\n[1] Jia, Menglin, et al. \"Visual prompt tuning.\", 2022.\n\n[2] Zhou, Kaiyang, et al. \"Learning to prompt for vision-language models.\", 2022.\n\n[3] Zhou, Kaiyang, et al. \"Conditional prompt learning for vision-language models.\", 2022.\n\n[4] Xing, Jinwei, et al. \"Domain adaptation in reinforcement learning via latent unified state representation.\", 2021.\n\n[5] Li, Dongfen, et al. \"Domain adaptive state representation alignment for reinforcement learning.\", 2022\n\n[6] Liu, Pengfei, et al. \"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.\", 2023\n\n[7] Hakhamaneshi, Kourosh, et al. \"Hierarchical few-shot imitation with skill transition models.\", 2021"}, {"review_id": "Genudn2KgP", "reviewer": "Reviewer_gM92", "summary": "This paper proposes a novel contrastive prompt ensemble (ConPE) framework for embodied reinforcement learning (RL) agents. ConPE utilizes a pretrained vision-language model and a set of visual prompts to construct robust state representations. Each prompt is contrastively learned in terms of an individual domain factor that significantly affects the agent's egocentric perception and observation. The attention-based ensemble and policy are jointly learned so that the resulting state representations not only generalize to various domains but are also optimized for learning the task. Through experiments, ConPE outperforms other state-of-the-art algorithms for several embodied agent tasks, including navigation in AI2THOR, manipulation in Metaworld, and autonomous driving in CARLA.", "questions": "Clarify and address the concern mentioned in the above section.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a novel idea that uses visual prompt-based contrastive learning and guided-attention-based prompt\n72 ensemble algorithms for better representation learning that leads to improved vision-based RL policy performance. The idea is neat and the method is technically sound.\n\nThe experimental results show that the proposed method can outperform prior vision-based RL methods in navigation and manipulation tasks. The authors also provide extensive ablation studies for better understanding of the algorithm.", "weaknesses": "The evaluated tasks in the paper are relatively toy-like. For example, the authors only evaluated the method in reaching tasks for manipulation, which is overly simple. It would be more convincing if the authors could include more challenging and realistic tasks such as more dexterous manipulation tasks like pick and place and kitchen rearrangement and navigation tasks such as Habitat object navigation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author response for Weakness 1. [more challenging and realistic tasks]**\n\nThe simulators that we used (e.g., AI2THOR, CARLA) are widely used in the field of Embodied AI [1, 2, 3, 4, 5].\nThe observation state from these environments contains a wide variety of objects, and their image quality is regarded as being photo-realistic [1, 2, 5].\nWhile we hold the belief that the complexity of the environments we utilized stands strong in comparison to other existing RL environments (e.g., mujoco [6], vizdoom [7] and Megaverse [8]), we agree that there is room to extend our research into more challenging and realistic environments in future work.\n\nAdditionally, we believe we have tackled a challenging issue.\nThe complexities of practical scenarios often involve the interplay of multiple domain factors, each contributing to the environment overall context in which a policy operates. \n\nGeneralizing with respect to each individual domain factor can provide a basic understanding, but we presume that that doesn't necessarily prepare an agent for intricate multi-factorial situations. \n\nFor example, consider the car driving example. Mastering driving on wet roads focus on a single domain factor (road wetness), and the challenges associated with it is not the same as some navigating scenarios characterized by combinations of multiple factors, such as varying weather, diverse road conditions, and different car types, in addition to such wet surface. Our work aims to provide a novel parameter-efficient, sample-efficient RL adaptation structure, while leveraging pre-trained visual models, to tackle such complex realistic situations affected by multiple domain factors.  \n\n[1] Duan, Jiafei, et al. \"A survey of embodied ai: From simulators to research tasks.\" IEEE Transactions on Emerging Topics in Computational Intelligence 6.2 (2022): 230-244.\n\n[2] Kolve, Eric, et al. \"Ai2-thor: An interactive 3d environment for visual ai.\" arXiv preprint arXiv:1712.05474 (2017).\n\n[3] Batra, Dhruv, et al. \"Rearrangement: A challenge for embodied ai.\" arXiv preprint arXiv:2011.01975 (2020).\n\n[4] Weihs, Luca, et al. \"Allenact: A framework for embodied ai research.\" arXiv preprint arXiv:2008.12760 (2020).\n\n[5] Dosovitskiy, Alexey, et al. \"CARLA: An open urban driving simulator.\" Conference on robot learning. PMLR, 2017.\n\n[6] Todorov, Emanuel, Tom Erez, and Yuval Tassa. \"Mujoco: A physics engine for model-based control.\" 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012.\n\n[7] Kempka, Michał, et al. \"Vizdoom: A doom-based ai research platform for visual reinforcement learning.\" 2016 IEEE conference on computational intelligence and games (CIG). IEEE, 2016.\n\n[8] Petrenko, Aleksei, et al. \"Megaverse: Simulating embodied agents at one million experiences per second.\" International Conference on Machine Learning. PMLR, 2021."}, {"review_id": "my2qO3PDtK", "reviewer": "Reviewer_FnQE", "summary": "This paper proposes a novel framework with an ensemble of visual contrastive prompts for efficient adaptation of vision-based robot learning tasks, named CONPE. The visual prompts are firstly learned from a contrastive learning procedure with different domain factors, which are expected to be domain orthogonal. Then the prompts are applied for downstream vision robot learning tasks along with a guided-attention-based prompt ensemble module, which adaptively learns an ensemble of vision features and prompts, to obtain task specific and domain invariant visual features, for efficient robot learning tasks. The experimental results demonstrate the effectiveness of the proposed framework.", "questions": "1.\tWhat’s Stage 1’s training data and strategy? How about tune the visual prompts for downstream data in Stage 2?\n2.\tSeveral numbers reported in Section 4 worths discussing, e.g., the performance of ATC and EmbCLIP in Tab. 1 (a), why would the methods’ performance on unseen domain be better than seen domain?\n", "limitations": "Environment generalization and adaptation is necessary for robot learning, it is encouraged to explore the potential of visual prompt for generalization of robot arms and other visual factors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper proposes an intuitive yet efficient method to address a meaningful issue of environment generalization in robotics learning. The approach combines the natural of contrastive learning and visual prompt and surpasses existing baselines according to the experimental results.", "weaknesses": "1. As shown in Fig. 2 (iii), it indicates that the framework enables robot learning model to generalize to unseen physical property. But according to the statement, the physical property is more likely to refer to angle of view (also can be seen in Fig. 1. However, for most manipulation tasks, the change of the appearance of robot arm/hand brings more disruption, which worths discussion but is missing in this paper.\n2. Visual representations in virtual environments are simpler than in real world, which has more complex object layout, texture, etc. Thus, many methods in work in Venv but fail to be deployed. Supportive real-world experimental result is crucial.\n3.\tSome related works on data-driven environment generalization for robot learning should be discussed, e.g., \"Open-World Object Manipulation using Pre-trained Vision-Language Models\", and \"Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author response for Weakness 1. [physical property]**\nWe appreciate the reviewer for this observation. The physical properties that we focus on include camera fields of view, camera positions and action magnitudes which are frequently considered embodiment configurations in the embodied AI literature [1, 2].\nThe reviewer's point regarding the impact of the change in the appearance of a robot arm/hand is quite valid. Manipulation tasks can be sensitive to the appearance and configuration of the robot's end effector.\n\nHowever, given the constraints of the simulators that we have utilized, our concentration on variations of camera fields of view, camera positions, and action magnitudes seemed the most pragmatic and relevant for us. Furthermore, in our current research, our primary emphasis is not on individual domain factors, but rather on our unified framework architecture with visual prompt ensemble, which can address a broad spectrum of adaptation issues.\n\nIn future research, we intend to investigate the effects of changes in the robot's appearance on its learning capability and performance.\n\n### **Author response for Weakness 2. [real-world experimental]**\nWe agree that the distinction between virtual environments and real-world scenarios is a valid concern.\n\nThe simulator that we used, such as AI2THOR and CARLA is widely recognized for high fidelity and photo-realistic qualities [3, 4, 5].  \nWhile the simulators we employed offer detailed simulations, it is essential to approach the potential of our ConPE framework in real-world applications with caution. We understand that there are inherent differences between these virtual environments and actual real-world conditions. We plan to further explore research utilizing real-world data and applications.\n\n### **Author response for Weakness 3. [related works]**\nThese works [6, 7] mentioned by the reviewer resonate with our goal of harnessing pre-trained models to enhance RL policy adaptation in various areas including robotic manipulation.\n\nWe will include these references in the Related Work Section in our revised manuscript. Intriguingly, these works primarily focus on exploring language-grounded policies, providing us with some rational to consider the incorporation of both visual and language encoders.\n\n### **Author response for Question 1. [training data and strategy]**\nFor prompt learning, we use the dataset gathered from various domains, where each domain is determined by varying a specific domain factor while keeping the others consistent.\nWe devise rule-based policies for each environment and collect 28,464 samples for AI2THOR, 4,700 samples for Metaworld, and 73,900 samples for CARLA.\n\nIn Stage 1 (prompt-based contrastive learning, as depicted in the left-side of Figure 2), given the dataset, we individually train the prompts through several contrastive learning algorithms.\nWe describe the details in Appendix A, including behavior-driven contrastive learning, augmentation-driven contrastive learning, and timestep-driven contrastive learning.\n\nIn Stage 2 (guided-attention-based prompt ensemble, as depicted in the middle of Figure 2), we train the ensemble module and policy on the source domains, while keeping the visual prompts frozen (which are learned in Stage 1).\n\n### **Author response for Question 2. [performance scores]**\nIn certain cases, the zero-shot performance in unseen target domains can exceed that of the source or seen target domains. While this might appear less consistent, similar observations have been reported in prior works [8, 9] in domain adaptation research.\nThis phenomenon can be largely attributed to the disparities between the source and target domains, particularly when the source domains are relatively more challenging than the target domains. We frequently experience this pattern as domains differ by the complex interplay of multiple factors.\n\n### **Author response for Limitations [generalization of robot arms and other visual factors]**\nWe will include the limitation of our work, partially related to domain factors considered in our current experiments, in the Conclusion Section of our revised manuscript.\n\nWe appreciate the reviewer's insightful comment on the limitations and agree that further work is needed to explore the potential of our framework toward more complex, realistic situations that are characterized by a multitude of different domain factors and associated with long-horizon control tasks in robotics environments.\n\n[1] Chattopadhyay, Prithvijit, et al. \"Robustnav: Towards benchmarking robustness in embodied navigation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[2] Shah, Dhruv, et al. \"Gnm: A general navigation model to drive any robot.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.\n\n[3] Duan, Jiafei, et al. \"A survey of embodied ai: From simulators to research tasks.\" IEEE Transactions on Emerging Topics in Computational Intelligence 6.2 (2022): 230-244.\n\n[4] Kolve, Eric, et al. \"Ai2-thor: An interactive 3d environment for visual ai.\" arXiv preprint arXiv:1712.05474 (2017).\n\n[5] Dosovitskiy, Alexey, et al. \"CARLA: An open urban driving simulator.\" Conference on robot learning. PMLR, 2017.\n\n[6] Stone, Austin, et al. \"Open-world object manipulation using pre-trained vision-language models.\" arXiv preprint arXiv:2303.00905 (2023).\n\n[7] Yang, Jiange, et al. \"Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots.\" arXiv preprint arXiv:2306.05716 (2023).\n\n[8] Xing, Jinwei, et al. \"Domain adaptation in reinforcement learning via latent unified state representation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\n[9] Li, Dongfen, et al. \"Domain adaptive state representation alignment for reinforcement learning.\" Information Sciences 609 (2022): 1353-1368."}]

Author rebuttal:
We attached global author rebuttal. Please refer to it.

Meta-review: This paper focussed on zero-shot domain adaptation to unseen visual observations for embodied agents. The proposed approach CONPE (contrastive prompt ensemble) fine-tunes the observation/visual encoder contrastively to tackle domain gaps. An ensemble over this  helps with representations for specific tasks, demonstrated across AI2THOR, Metaworld, and CARLA simulators, widely used for training embodied perception-control agents.
 
This submission received six reviews that unanimously support acceptance. Key strengths that most reviewers agree on are (1) simple and effective approach tackling generalization via zero-shot adaptation, (2) extensive empirical evidence across several simulators, tasks, and baselines, (3) novelty around marrying visual prompting with embodied actor policies, (4) clear presentation via helpful visuals and analysis. Two reviewers updated reviews (including one rating update) after the author response period, which were also incorporated in the decision-making. 

The AC concurs with the unanimous decision of the reviewers and recommends acceptance. The authors are encouraged to utilize their constructive and detailed discussion with reviewers towards further raising the quality and impact of their work.
Decision: Accept (poster)