Title: Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents
Venue: NeurIPS 2023
Accepted: yes
Keywords: Prompt Learining

Domain Adaptation

Embodied AI
Abstract: For embodied reinforcement learning (RL) agents interacting with the environment, it is desirable to have rapid policy adaptation to unseen visual observations, but achieving zero-shot adaptation capability is considered as a challenging problem in the RL context. To address the problem, we present a novel contrastive prompt ensemble (ConPE) framework which utilizes a pretrained vision-language model and a set of visual prompts, thus enables efficient policy learning and adaptation upon a wide range of environmental and physical changes encountered by embodied agents. Specifically, we devise a guided-attention-based ensemble approach with multiple visual prompts on the vision-language model to construct robust state representations. Each prompt is contrastively learned in terms of an individual domain factors that significantly affects the agent's egocentric perception and observation. For a given task, the attention-based ensemble and policy are jointly learned so that the resulting state representations not only generalize to various domains but are also optimized for learning the task. Through experiments, we show that ConPE outperforms other state-of-the-art algorithms for several embodied agent tasks including navigation in AI2THOR, manipulation in Metaworld, and autonomous driving in CARLA, while also improving the sample efficiency of policy learning and adaptation.

Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents

Abstract

For embodied reinforcement learning (RL) agents interacting with the environment, it is desirable to have rapid policy adaptation to unseen visual observations, but achieving zero-shot adaptation capability is considered as a challenging problem in the RL context. To address the problem, we present a novel contrastive prompt ensemble (CONPE) framework which utilizes a pretrained vision-language model and a set of visual prompts, thus enabling efficient policy learning and adaptation upon a wide range of environmental and physical changes encountered by embodied agents. Specifically, we devise a guided-attention-based ensemble approach with multiple visual prompts on the vision-language model to construct robust state representations. Each prompt is contrastively learned in terms of an individual domain factor that significantly affects the agent's egocentric perception and observation. For a given task, the attention-based ensemble and policy are jointly learned so that the resulting state representations not only generalize to various domains but are also optimized for learning the task. Through experiments, we show that CONPE outperforms other state-of-the-art algorithms for several embodied agent tasks including navigation in AI2THOR, manipulation in egocentric-Metaworld, and autonomous driving in CARLA, while also improving the sample efficiency of policy learning and adaptation.

1 Introduction

In the literature of vision-based reinforcement learning (RL), with the advance of unsupervised techniques and large-scale pretrained models for computer vision, the decoupled structure, in which visual encoders are separately trained and used later for policy learning, has gained popularity [1, 2, 3]. This decoupling demonstrates high efficiency in low data regimes with sparse reward signals, compared to end-to-end RL. In this regard, several works on adopting the decoupled structure to embodied agents interacting with the environment were introduced [4, 5], and specifically, pretrained vision models (e.g., ResNet in [6]) or vision-language models (e.g., CLIP in [7, 8]) were exploited for visual state representation encoders. Yet, it is non-trivial to achieve zero-shot adaptation to visual domain changes in the environment with high diversity and non-stationarity, which are inherent for embodied agents. It was rarely investigated how to optimize those popular large-scale pretrained models to ensure the zero-shot capability of embodied agents.

Embodied agents have several environmental and physical properties, such as egocentric camera position, stride length, and illumination, which are domain factors making significant changes in agents' perception and observation. In the target (deployment) environment with uncalibrated settings on those domain factors, RL policies relying on pretrained visual encoders remain vulnerable to domain changes.

Figure 1 provides an example of egocentric visual domain changes experienced by embodied agents due to different camera positions. When policies learned in the source environment are applied to the target environment, zero-shot performance can be significantly degraded, unless the visual encoder could adapt not only to environmental differences but also to the physical diversity

Source Environment

Target Environments

of agents. In this paper, we investigate RL policy adaptation techniques for embodied agents to enable zero-shot adaptation to domain changes, by leveraging prompt-based learning for pretrained models in the decoupled RL structure. To this end, we present CONPE, a novel contrastive prompt ensemble framework that uses the CLIP vision-language model as the visual encoder, and facilitates dynamic adjustments of visual state representations against domain changes through an ensemble of contrastively learned visual prompts. In CONPE, the ensemble employs attention-based state composition on multiple visual embeddings from the same input observation, where each embedding corresponds to a state representation individually prompted for a specific domain factor. Specifically, the cosine similarity between an input observation and its respective prompted embeddings is used to calculate attention weights effectively.

Through experiments, we demonstrate the benefits of our approach. First, RL policies learned via CONPE achieve competitive zero-shot performance upon a wide variety of egocentric visual domain variations for several embodied agent tasks, such as navigation tasks in AI2THOR [9], vision-based robot manipulation tasks in egocentric-Metaworld, and autonomous driving tasks in CARLA [10]. For instance, the policy via CONPE outperforms EmbCLIP [7] in zero-shot performance by 20.7% for unseen target domains in the AI2THOR object navigation. Second, our approach achieves high sample-efficiency in the decoupled RL structure. For instance, CONPE requires less than 50.0% and 16.7% of the samples compared to ATC [1] and 60% and 50% of the samples compared to EmbCLIP to achieve comparable performance in seen and unseen target domains in the AI2THOR object navigation.

In the context of RL, our work is the first to explore policy adaptation using visual prompts for embodied agents, achieving superior zero-shot performance and high sample-efficiency. The main contributions of our work are as follows.

We present a novel CONPE framework with an ensemble of visual contrastive prompts, which enables zero-shot adaptation for vision-based embodied RL agents.

We devise visual prompt-based contrastive learning and guided-attention-based prompt ensemble algorithms to represent task-specific information in the CLIP embedding space.

We experimentally show that policies via CONPE achieve comparable or superior zeroshot performance, compared to other state-of-the-art baselines, for several tasks. We also demonstrate high sample-efficiency in policy learning and adaptation.

We create the datasets with various visual domains in AI2THOR, egocentric-Metaworld and CARLA, and make them publicly accessible for further research on RL policy adaptation.

2 Problem Formulation

In RL formulation, a learning environment is defined as a Markov decision process (MDP) of ( S, A, P , R ) with state space s ∈ S , action space a ∈ A , transition probability P : S × A → S and reward function R : S × A → R . The objective of RL is to find an optimal policy π ∗ : S → A maximizing the sum of discounted rewards. For embodied agents, states might not be fully observable, and the environment is represented by a partially observable MDP (POMDP) of a tuple ( S, A, P , R, Ω , O ) with an observation space o ∈ Ω and a conditional observation probability [11] O : S × A → Ω .

Given visual domains in the dynamic environment, we consider policy adaptation to find the optimal policy that remains invariant across the domains or is transferable to some target domain, where each domain is represented by a POMDP and domain changes are formulated by different O . We denote domains as D = (Ω , O ) . Aiming to enable zero-shot adaptation to various domains, we formulate

(i) Prompt-based Contrastive Learning

Visual Prompt Pool,

𝐩 𝑣

⋮

Brightness

𝑝1

𝑣

CAM

𝑝2

𝑣

Stride Length

𝑝3

𝑣

CLIP

𝒯 𝜙

(𝑜, 𝑝

𝑖

𝑣

)

𝓏𝑘

𝓏𝑘

′

𝓏𝑖≠𝑘

′

𝑝 𝑖

𝑣

domain factor

Contrast Function

𝓏𝑘

′

Pos.

Neg.

𝓏𝑘

𝓏𝑖≠𝑘

′

⋮

+

(ii) Guided-Attention-based Prompt Ensemble

Source env.

⋮

Prompt Ensemble

𝒢

𝒵

𝓏0

𝓏1

𝓏2

𝓏3

𝓏4

𝓏5

Guided attention weights

Policy

𝜋

(iii) Zero-shot Policy Deployment

Target env.

Unseen

visual

variation

Unseen

physical

property

Domain generalization

Policy

𝜋

Contrastive Learning

action

the policy adaptation problem as finding the optimal policy π ∗ such that

where p ( D ) is a given domain distribution and γ is a discount factor of the environment.

For embodied agents, the same state can be differently observed depending on the configuration of properties such as egocentric camera position, stride length, illumination, and object style. We refer to such a property causing domain changes in the environment as a domain factor . Practical scenarios often involve the interplay of multiple domain factors in the environment.

3 Our Approach

3.1 Framework Structure

To enable zero-shot policy adaptation to unseen domains, we develop the CONPE framework consisting of (i) prompt-based contrastive learning with the CLIP visual encoder, (ii) guided-attention-based prompt ensemble, and (iii) zero-shot policy deployment, as illustrated in Figure 2. The capability of the CLIP visual encoder is enhanced using multiple visual prompts that are contrastively learned on expert demonstrations for several domain factors. This establishes the visual prompt pool in (i). Then, the prompts are used to train the guided-attention-based ensemble with the environment in (ii). To enhance learning efficiency and interpretability of attention weights, we use the cosine similarity of embeddings. The attention module and policy are jointly learned for a specific task so that resulting state representations tend to generalize across various domains and be optimized for task learning. In deployment, a non-stationary environment where its visual domain varies according to the environment conditions and agent physical properties is considered, and the zero-shot performance is evaluated in (iii).

3.2 Prompt-based Contrastive Learning

To construct domain-invariant representations with respect to a specific domain factor for egocentric perception data, we adopt several contrastive tasks for visual prompt learning, which can be learned on a few expert demonstrations. For this, we use a visual prompt

where e v i is a continuous learnable vector with the image patch embedding dimension d (e.g., 768 for CLIP visual encoder) and u is the length of a visual prompt. Let a pretrained model T ϕ parameterized by ϕ maps observations o ∈ Ω to the embedding space Z . With a contrast function P : Ω × Ω → { 0 , 1 } [1, 2, 12] to discriminate whether an observation pair is positive or not,

𝓏0

⋯

Embeddings

𝒛

Attention Module

𝓖 𝓏0, 𝒛

Visual encoder

𝒯 𝜙

(𝑜, 𝑝

𝑖

𝑣

)

⋯

〮

⋯

𝑂

𝑝1

𝑣

𝑂

𝑝2

𝑣

𝑂

𝑝𝑛

𝑣

𝑂

𝑔1

𝑔2

𝑔𝑛

⋯

Task-specific state representation

Z

Policy

𝜋 (𝑍)

⋯

𝑝3

𝑣

𝑂

𝜔𝑛

𝑢𝑛

〮

𝑚𝑙𝑝

𝑔3

𝑢5

𝑚𝑙𝑝

〮

⋯

𝜔1

𝑚𝑙𝑝

〮

𝑢1

𝑚𝑙𝑝

𝜔2

〮

𝑢2

𝓏1

𝓏𝑛

𝓏2

⋯

𝓏3

𝓏4

𝓏5

𝓏1

𝓏𝑛

𝓏2

⋯

𝓏3

𝓏4

𝓏5

𝓏1

𝓏𝑛

𝓏2

⋯

𝓏3

𝓏4

𝓏5

𝑝4

𝑣

𝑂

𝑝5

𝑣

𝑂

𝑔5

𝑔4

𝜔3

𝑢3

𝑚𝑙𝑝

〮

𝜔4

𝑢4

𝑚𝑙𝑝

〮

softmax

consider an m -sized batch of observation pairs B P = { ( o i , o ′ i ) } i ≤ m containing one positive pair { ( o k , o ′ k ) | P ( o k , o ′ k ) = 1 } for some k ≤ m . Then, we enhance the capability of T ϕ by learning a visual prompt p v through contrastive learning, where the contrastive loss function [13] is defined as

̸

As in [14], for latent vectors x, y ∈ Z , their similarity in the embedding space Z is calculated by S ( x, y ) , where λ is a hyperparameter. By conducting the prompt-based contrastive learning on n different domain factors, we obtain a visual prompt pool

Through this process, each visual prompt in p v encapsulates domain-invariant knowledge pertinent to its respective domain factor.

3.3 Guided-Attention-based Prompt Ensemble

To effectively integrate individual prompted embeddings from multiple visual prompts into a taskspecific state representation, we devise a guided-attention-based prompt ensemble structure, as shown in Figure 3 where the attention weights on the embeddings are dynamically computed via the attention module G for each observation.

Given observation o and the learned visual prompt pool p v , an image embedding z 0 = T ϕ ( o ) and prompted embeddings z = [ z 1 = T ϕ ( o, p v 1 ) , ..., z n ] are calculated. Then, z o and z are fed to the attention module G , where attention weights ω i for each prompted embedding z i are optimized. Since directly computing the attention weights using z 0 and z is prone to have an uninterpretable local optima, we introduce a guidance score g i based on the cosine similarity between the input image and visual prompted image embeddings in Z , i.e., g i = ⟨ z 0 ,z i ⟩ ∥ z 0 ∥∥ z i ∥ . Given that larger g i signifies a stronger conformity of an observation to the domain factor relevant to the prompted embedding z i , we use g i to steer the attention weights, aiming to not only improve learning efficiency but also provide interpretability. With guidance g i , we compute the attention weights ω i by

where k i is the projection of z i , d is dimension of z , and τ is a softmax temperature. Then, state embedding Z is obtained by

Algorithm 1 shows the procedures in CONPE, where the first half corresponds to prompt-based contrastive learning (in Section 3.2) and the other half corresponds to joint learning of a policy π ( Z ) and the attention module G . As G is optimized by a given RL task objective in the source domains (in line 12), the resulting Z tends to be task-specific, while Z is also domain-invariant by the ensemble of contrastively learned visual prompts based on G with respect to the combinations of multiple domain factors. The entire algorithm can be found in Appendix.

4 Evaluation

Experiments. We use AI2THOR [9], Metaworld [15], and CARLA [10] environments, specifically configured for embodied agent tasks with dynamic domain changes. These environments allow us to explore various domain factors such as camera settings, stride length, rotation degree, gravity, illuminations, wind speeds, and others. For prompt-based contrastive learning (in Section 3.2), we use a small dataset of expert demonstrations for each domain factor (i.e., 10 episodes per domain factor). For prompt ensemble-based policy learning (in Section 3.3), we use a few source domains randomly generated through combinatorial variations of the seen domain factors (i.e., 4 source domains). In our zero-shot evaluations, we use target domains that can be categorized as either seen or unseen. The seen target domains are those encountered during the phase of prompt-based contrastive learning, while these domains are not present during the phase of prompt ensemble-based policy learning. On the other hand, the unseen target domains refer to those that are entirely new, implying that they are not encountered during either learning phases.

Baselines. We implement several baselines for comparison. LUSR [16] is a reconstruction-based domain adaptation method in RL, which uses the variational autoencoder structure for robust representations. CURL [2] and ACT [1] employ contrastive learning in RL frameworks for high sample-efficiency and generalization to visual domains. ACO [12] utilizes augmentation-driven and behavior-driven contrastive tasks in the context of RL. EmbCLIP [7] is a state-of-the-art embodied AI model, which exploits the pretrained CLIP visual encoder for visual state representations.

Implementation. We implement CONPE using the CLIP model with ViT-B/32, similar to VPT [17] and CoOp [18]. In prompt-based contrastive learning, we adopt various contrastive learning schemes including augmentation-driven [2, 1, 19] and behavior-driven [12, 20, 21, 22] contrastive learning, where the prompt length sets to be 8. In policy learning, we exploit online learning (i.e., PPO [23]) for AI2THOR and imitation learning (i.e., DAGGER [24]) for egocentric-Metaworld and CARLA.

Table 1: Zero-shot Performance. The policies of each method (CONPE and the baselines) are learned on 4 source domains. The Source column presents the performance for those source domains. In all evaluations, we use 30 seen target domains and 10 unseen target domains. The Seen Target column presents the performance for the seen target domains, and the Unseen Target column presents the performance for the unseen target domains. The unseen target domains are not used for representation learning.

4.1 Zero-shot Performance

Table 1 shows zero-shot performance of CONPE and the baselines across source, seen and unseen target domains. We evaluate with 3 different seeds and report the average performance (i.e., task success rate in AI2THOR and egocentric-Metaworld, the sum of rewards in CARLA). As shown in Table 1(a), CONPE outperforms the baselines in the AI2THOR tasks. It particularly surpasses the most competitive baseline, EmbCLIP, by achieving 5 . 2 ∼ 5 . 7% higher success rate for seen target domains, and 6 . 9 ∼ 20 . 7% for unseen target domains. For egocentric-Metaworld, as shown in Table 1(b), CONPE demonstrates superior performance with a significant success rate for both seen and unseen target domains, which is 17 . 3 ∼ 24 . 0% and 18 . 0 ∼ 20 . 0% higher than EmbCLIP, respectively. For autonomous driving in CARLA, we take into account external environment factors, such as weather conditions and times of day, as domain factors that can influence the driving task. In Table 1(c), CONPE consistently maintains competitive zero-shot performance across all conditions, outperforming the baselines.

In these experiments, LUSR shows relatively low success rates, as the reconstruction-based representation model can abate some task-specific information from observations, which is critical to conduct vision-based complex RL tasks. EmbCLIP shows the most comparative performance among the baselines, but its zero-shot performance for target domains is not comparable to CONPE. In contrast,

CONPE effectively estimates the domain shifts pertaining to each domain factor through the use of guided attention weights, leading to robust performance in both seen and unseen target domains.

0.0M

0.5M

1.0M

1.5M

2.0M

2.5M

3.0M

3.5M

4.0M

Timesteps

0

20

40

60

80

100

Success Rate (%)

EmbCLIP

ATC

(a) Success Rate in Seen Target Domain

0.0M

0.5M

1.0M

1.5M

2.0M

2.5M

3.0M

3.5M

4.0M

Timesteps

0

20

40

60

80

100

Success Rate (%)

(b) Success Rate in Unseen Target Domain

Sample Efficiency. Figure 4 presents performance with respect to samples (timesteps) that are used by CONPE and baselines for policy learning. Compared to the most competitive baseline EmbCLIP, CONPE requires less than 60 . 0% timesteps (online samples) for seen target domains and 50 . 0% for unseen target domains to have comparable success rates.

1) Intra Prompted Embedding

2) Inter Prompted Embedding

1)

2)

(a) Prompted Embeddings

Prompt

Timestep

(b) Attention Weights

梦

1.0

0.8

0.6

0.4

0.2

0.0

Prompt Ensemble Interpretability. Figure 5(a) visualizes the prompted embeddings using the prompt pool obtained through CONPE. For intra prompted embeddings, we use observation pairs, where each pair is generated by varying domains within a domain factor. We observe that the embeddings are paired to form the domain-invariant knowledge because the visual prompt is learned through prompt-based contrastive learning. The inter prompted embeddings specify that each prompt distinctly clusters prompted embeddings that correspond to the domains associated to its domain factor. Figure 5(b) shows examples of the attention weight matrix of CONPE for four different domains. The x-axis denotes the visual prompts and the y-axis denotes timesteps. This shows the consistency of the attention weights on the prompts across the timesteps in the same domain.

4.2 Prompt Ensemble with a Pretrained Policy

While we previously presented joint learning of a policy and the attention module G , here we also present how to update G for a pretrained policy π to make the policy adaptable to domain changes. In this case, we add a policy prompt p v pol to concentrate on task-relevant features from observations for

the pretrained policy π so that prompted embedding ˜ z 0 with the task-relevant features is incorporated into the guided-attention-based ensemble, i.e., π ( G (˜ z 0 , z )) , where ˜ z 0 = T ϕ ( o, p v pol ) .

Table 2 reports zero-shot performance for the scenarios when a pretrained policy is given. We evaluate two different cases: aligned (Aln.) when prompt-based contrastive learning is conducted on data from the same task of a pretrained policy; otherwise, not aligned (Not Aln.). In AI2THOR, we use data from the object goal navigation task for prompt-based contrastive learning, while each pretrained policy is learned individually through one of tasks including object goal navigation, point goal navigation, image goal navigation, and room rearrangement. Similarly, in egocentric-Metaworld, we use data from the reach task for prompt-based contrastive learning, while each pretrained policy is learned individually through one of tasks including reach, reach-wall, button-press, and door-open. In Table 2(a), CONPE enhances zero-shot performance of the pretrained policies by 3 . 5 ∼ 7 . 0% for unseen target domains in AI2THOR. This prompt ensemble adaptation requires only 400K samples, equivalent to 10% of the total samples used for policy learning. In Table 2(b), CONPE significantly boosts zero-shot performance of the pretrained policies by 9 . 0 ∼ 57 . 6% in egocentric-Metaworld.

4.3 Ablation Study

Here we conduct ablation studies with AI2THOR. All the performances are reported in success rates.

Prompt Ensemble Scalability. Table 3 evaluates CONPE with respect to the number of prompts ( n ). CONPE effectively enhances zero-shot performance for both seen and unseen target domains through prompt ensemble that captures various domain factors. Compared to the case of n = 2 , for n = 10 , there was a significant improvement in zero-shot performance for both seen and unseen target domains, with increases of 42 . 8% and 36 . 7% , respectively. For n ≥ 10 , we observe stable performance that specifies that CONPE can scale for combining multiple prompts to some extent.

Prompt Ensemble Methods. Tabel 4 compares the performance of various prompt integration methods [25, 26, 27] including our guided attention-based prompt ensemble. We denote prompt-level integration as COM, and prompted embeddings-level integration as ENS. UNI-AVG and WEI-AVG refer to uniform average and weighted average mechanisms, respectively. CONPE achieves superior success rates over the most competitive ensemble method ENS-UNI-AVG, showing 3 . 5% and 14 . 2% performance gain for seen and unseen target domains.

Optimization

Source

Pretrained w/o

p

v

v

pol

w

p

pol

E2E

95

.

3

±

4

.

59

.

98

5

.

96

±

2

±

9

.

4

.

1

±

0

6

9

.

.

0

5

Target

80

.

9

±

1

.

54

.

84

2

±

.

83

0

±

4

.

3

±

6

.

6

1

3

.

0

.

3

Prompt Ensemble Adaptation Method. Table 5 shows the effect of our ensemble adaptation method for the situation when a pretrained policy is given. As explained in Section 4.2, in this situation, CONPE can update the attention module with an additional prompt p v pol . Note that p v pol corresponds to this case, while w/o p v pol corresponds to the other case of using the attention module without p v pol . In addition, E2E denotes the fine-tuning of both the policy and the attention module along with p v pol . The results demonstrate that our method enhances the zero-shot performance of the pretrained policy, showing that p v pol facilitates the extraction of task-specific features.

Semantic Regularized Data Augmentation. So far, we have only utilized vision data, but here, we discuss one extension of CONPE using semantic information. Specifically, we use a few samples of object-level text descriptions to regularize the data augmentation process in policy learning. This aims to mitigate overfitting issues [28, 29]. The detailed explanations can be found in Appendix. As shown in Table 6, CONPE with semantic data (w Semantic) consistently yields better performance than CONPE without semantic data (w/o Semantic) for all noise scale settings ( δ ). Note that the noise scale manages the variance of augmented prompted embeddings. This experiment indicates that CONPE can be improved by incorporating semantic information.

5 Related Work

Adaptation in Embodied AI. In the literature of robotics, numerous studies focused on developing generalized visual encoders for robotic agents across various domains [30, 31], exploiting pretrained visual encoders [32, 33], and establishing robust control policies with domain randomized techniques [34, 35]. Furthermore, in the field of learning embodied agents, a few works addressed adaptation issues of agents to unseen scenarios in complex environments, using data augmentation techniques [36, 37, 38, 39, 40] or adopting self-supervised learning schemes [41, 42, 43]. Recently, several works showed the feasibility and benefits of adopting large-scale pretrained vision-language models for embodied agents [7, 44, 45, 46]. Our work is in the same vein of these prior works of embodied agents, but unlike them, we explore visual prompt learning and ensembles, aiming to enhance both zero-shot performance and sample-efficiency.

Decoupled RL Structure. The decoupled structure, where a state representation model is separated from RL, has been investigated in vision-based RL [47, 2, 16]. Recently, contrastive representation learning on expert trajectories gains much interest, as it allows expert behavior patterns to be incorporated into the state encoder even when a policy is not jointly learned [1, 20]. They established generalized state representations, yet in that direction, sample-efficiency issues in both representation learning and policy learning remain unexplored.

Prompt-based Learning. Prompt-based learning or prompt tuning is a parameter-efficient optimization method for large pretrained models. Prompt tuning was used for computer vision tasks, optimizing a few learnable vectors in the text encoder [18], and it was also adopted for vision transformer models to handle a wide range of downstream tasks [17]. Recently, visual prompting [48] was introduced, and both visual and text prompt tuning were explored together in the multi-modal embedding space [49, 50]. We also use visual prompt tuning, but we concentrate on the ensemble of multiple prompts to tackle complex embodied RL tasks. We take advantage of the fact that the generalized representation capability of different prompts can vary depending on a given task and domain, and thus we strategically utilize them to enable zero-shot adaptation of RL policies.

6 Conclusion

Limitation. Our CONPE framework exploits visual inputs and their relevant domain factors for policy adaptation. For environments where domain changes extend beyond those domain factors, the adaptability of the framework might be constrained. In our future work, we will adapt the framework with semantic knowledge based on pretrained language models to improve the policy generalization capability for embodied agents in dynamic complex environments and to cover various scenarios associated with multi-modal agent interfaces.

Conclusion. In this work, we presented the CONPE framework, a novel approach that allows embodied RL agents to adapt in a zero-shot manner across diverse visual domains, exploring the ensemble structure that incorporates multiple contrastive visual prompts. The ensemble facilitates domain-invariant and task-specific state representations, thus enabling the agents to generalize to visual variations influenced by specific domain factors. Through various experiments, we demonstrated that the framework can enhance policy adaptation across various domains for vision-based object navigation, rearrangement, manipulation tasks as well as autonomous driving tasks.

References

Adam Stooke et al. 'Decoupling representation learning from reinforcement learning'. In: Proceedings of the 38th International Conference on Machine Learning . 2021, pp. 9870-9879.

Aravind Srinivas, Michael Laskin, and Pieter Abbeel. 'CURL: Contrastive unsupervised representations for reinforcement learning'. In: Proceedings of the 37th International Conference on Machine Learning . 2020, pp. 5639-5650.

Max Schwarzer et al. 'Data-efficient reinforcement learning with self-predictive representations'. In: arXiv preprint arXiv:2007.05929 (2020).

Shangda Li et al. 'Unsupervised domain adaptation for visual navigation'. In: arXiv preprint arXiv:2010.14543 (2020).

Ziad Al-Halah, Santhosh K. Ramakrishnan, and Kristen Grauman. 'Zero experience required: Plug & play modular transfer learning for semantic visual navigation'. In: Proceedings of the 9th International Conference on Vision and Pattern Recognition . 2021, pp. 17010-17020.

Qianfan Zhao et al. 'Zero-shot object goal visual navigation'. In: arXiv preprint arXiv:2206.07423 (2022).

Apoorv Khandelwal et al. 'Simple but Effective: CLIP embeddings for embodied AI'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 14809-14818.

Arjun Majumdar et al. 'Zson: Zero-shot object-goal navigation using multimodal goal embeddings'. In: arXiv preprint arXiv:2206.12403 (2022).

Eric Kolve et al. 'Ai2-thor: An interactive 3d environment for visual ai'. In: arXiv preprint arXiv:1712.05474 (2017).

Alexey Dosovitskiy et al. 'CARLA: An open urban driving simulator'. In: Proceedings of the 1st Conference on Robot Learning . 2017, pp. 1-16.

Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

Qihang zhang, Zheghao Peng, and Bolei Zhou. 'Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining'. In: Proceedings of the 17th European Conference on Computer Vision . 2022, pp. 111-128.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 'Representation learning with contrastive predictive coding'. In: arXiv preprint arXiv:1807.03748 (2018).

Ting Chen et al. 'A simple framework for contrastive learning of visual representations'. In: Proceedings of the 37th International Conference on Machine Learning . 2020, pp. 1597-1607.

Tianhe Yu et al. 'Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning'. In: Proceedings of the 3rd Conference on Robot Learning . 2019, pp. 10941100.

Jinwei Xing et al. 'Domain adaptation in reinforcement learning via latent unified state representation'. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence . 2021, pp. 10452-10459.

Menglin Jia et al. 'Visual prompt tuning'. In: Proceedings of the 17th European Conference on Computer Vision . 2022, pp. 709-727.

Kaiyang Zhou et al. 'Learning to prompt for vision-language models'. In: International Journal of Computer Vision (2022).

Bang You et al. 'Integrating contrastive learning with dynamic models for reinforcement learning from images'. In: Neurocomputing (2022).

Minbeom Kim et al. 'Action-driven contrastive representation for reinforcement learning'. In: PLOS ONE (2022).

Rishabh Agarwal et al. 'Contrastive behavioral similarity embeddings for generalization in reinforcement learning'. In: arXiv preprint arXiv:2101.05265 (2021).

Young Jae Lee et al. 'STACoRe: Spatio-temporal and action-based contrastive representations for reinforcement learning in Atari'. In: Neural Networks (2023).

John Schulman et al. 'Proximal policy optimization algorithms'. In: arXiv preprint arXiv:1707.06347 (2017).

Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. 'A reduction of imitation learning and structured prediction to no-regret online learning'. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics . 2011, pp. 627-635.

Pengfei Liu et al. 'Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing'. In: ACM Computing Surveys (2023).

Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 'Effective approaches to attention-based neural machine translation'. In: arXiv preprint arXiv:1508.04025 (2015).

Ashish Vaswani et al. 'Attention is all you need'. In: Advances in neural information processing systems (2017).

Denis Yarats, Ilya Kostrikov, and Rob Fergus. 'Image Augmentation Is All You Need: Regularizing deep reinforcement learning from pixels'. In: Proceedings of the 9th International Conference on Learning Representations . 2021.

Samarth Sinha, Ajay Mandlekar, and Animesh Garg. 'S4RL: Surprisingly Simple SelfSupervision for Offline Reinforcement Learning in Robotics'. In: Proceedings of the 5th Conference on Robotics Learning . 2021, pp. 907-917.

Arjun Majumdar et al. 'Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?' In: arXiv preprint arXiv:2303.18240 (2023).

Suraj Nair et al. 'R3m: A universal visual representation for robot manipulation'. In: arXiv preprint arXiv:2203.12601 (2022).

Austin Stone et al. 'Open-world object manipulation using pre-trained vision-language models'. In: arXiv preprint arXiv:2303.00905 (2023).

Jiange Yang et al. 'Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots'. In: arXiv preprint arXiv:2306.05716 (2023).

Dhruv Shah et al. 'Gnm: A general navigation model to drive any robot'. In: Proceedings of the IEEE International Conference on Robotics and Automation . IEEE. 2023, pp. 7226-7233.

Noriaki Hirose et al. 'ExAug: Robot-conditioned navigation policies via geometric experience augmentation'. In: Proceedings of the IEEE International Conference on Robotics and Automation . IEEE. 2023, pp. 4077-4084.

Daniel Fried et al. 'Speaker-follower models for vision-and-language navigation'. In: Proceedings of the 31th Conference on Neural Information Processing Systems . 2018, pp. 33183329.

Felix Yu et al. 'Take the scenic route: Improving generalization in vision-and-language navigation'. In: arXiv preprint arXiv:2003.14269 (2020).

Xiaofeng Gao et al. 'Dialfred: Dialogue-enabled agents for embodied instruction following'. In: arXiv preprint arXiv:2202.13330 (2022).

Chong Liu et al. 'Vision-language navigation with random environmental mixup'. In: Proceedings of the International Conference on Computer Vision . 2021, pp. 1624-1634.

Jialu Li, Hao Tan, and Mohit Bansal. 'EnvEdit: Environment Editing for Vision-and-Language Navigation'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 15386-15396.

Xin Wang et al. 'Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2019, pp. 6629-6638.

Roberto Bigazzi et al. 'Explore and explain: self-supervised navigation and recounting'. In: Proceeding of the 25th International Conference on Pattern Recognition . 2020, pp. 1152-1159.

Eun Sun Lee et al. 'MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agents'. In: Proceeding of the 17th European Conference on Computer Vision . 2022, pp. 338-354.

Vishnu Sashank Dorbala et al. 'CLIP-Nav: Using CLIP for zero-shot vision-and-language navigation'. In: arXiv preprint arXiv:2211.16649 (2022).

Samir Yitzhak Gadre et al. 'CLIP on Wheels: zero-shot object navigation as object localization and exploration'. In: arXiv preprint arXiv:2203.10421 (2022).

Dhruv Shah et al. 'Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action'. In: Proceedings of the 6th Conference on Robot Learning . 2022, pp. 492504.

Irina Higgins et al. 'DARLA: Improving zero-shot transfer in reinforcement learning'. In: Proceedings of the 34th International Conference on Machine Learning . 2018, pp. 1480-1490.

Hyojin Bahng et al. 'Exploring visual prompts for adapting large-scale models'. In: arXiv preprint arXiv:2203.17274 (2022).

Muhammad Uzair Khattak et al. 'MaPLe: Multi-modal prompt learning'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2023.

Yuhang Zang et al. 'Unified vision and language prompt learning.' In: arXiv preprint arXiv:2210.07225 (2022).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework with an ensemble of visual contrastive prompts for efficient adaptation of vision-based robot learning tasks, named CONPE. The visual prompts are firstly learned from a contrastive learning procedure with different domain factors, which are expected to be domain orthogonal. Then the prompts are applied for downstream vision robot learning tasks along with a guided-attention-based prompt ensemble module, which adaptively learns an ensemble of vision features and prompts, to obtain task specific and domain invariant visual features, for efficient robot learning tasks. The experimental results demonstrate the effectiveness of the proposed framework.\n\nSTRENGTHS:\nThis paper proposes an intuitive yet efficient method to address a meaningful issue of environment generalization in robotics learning. The approach combines the natural of contrastive learning and visual prompt and surpasses existing baselines according to the experimental results.\n\nWEAKNESSES:\n1. As shown in Fig. 2 (iii), it indicates that the framework enables robot learning model to generalize to unseen physical property. But according to the statement, the physical property is more likely to refer to angle of view (also can be seen in Fig. 1. However, for most manipulation tasks, the change of the appearance of robot arm/hand brings more disruption, which worths discussion but is missing in this paper.\n2. Visual representations in virtual environments are simpler than in real world, which has more complex object layout, texture, etc. Thus, many methods in work in Venv but fail to be deployed. Supportive real-world experimental result is crucial.\n3.\tSome related works on data-driven environment generalization for robot learning should be discussed, e.g., \"Open-World Object Manipulation using Pre-trained Vision-Language Models\", and \"Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots\".\n\nQUESTIONS:\n1.\tWhat’s Stage 1’s training data and strategy? How about tune the visual prompts for downstream data in Stage 2?\n2.\tSeveral numbers reported in Section 4 worths discussing, e.g., the performance of ATC and EmbCLIP in Tab. 1 (a), why would the methods’ performance on unseen domain be better than seen domain?\n\nLIMITATIONS:\nEnvironment generalization and adaptation is necessary for robot learning, it is encouraged to explore the potential of visual prompt for generalization of robot arms and other visual factors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel contrastive prompt ensemble (ConPE) framework for embodied reinforcement learning (RL) agents. ConPE utilizes a pretrained vision-language model and a set of visual prompts to construct robust state representations. Each prompt is contrastively learned in terms of an individual domain factor that significantly affects the agent's egocentric perception and observation. The attention-based ensemble and policy are jointly learned so that the resulting state representations not only generalize to various domains but are also optimized for learning the task. Through experiments, ConPE outperforms other state-of-the-art algorithms for several embodied agent tasks, including navigation in AI2THOR, manipulation in Metaworld, and autonomous driving in CARLA.\n\nSTRENGTHS:\nThis paper presents a novel idea that uses visual prompt-based contrastive learning and guided-attention-based prompt\n72 ensemble algorithms for better representation learning that leads to improved vision-based RL policy performance. The idea is neat and the method is technically sound.\n\nThe experimental results show that the proposed method can outperform prior vision-based RL methods in navigation and manipulation tasks. The authors also provide extensive ablation studies for better understanding of the algorithm.\n\nWEAKNESSES:\nThe evaluated tasks in the paper are relatively toy-like. For example, the authors only evaluated the method in reaching tasks for manipulation, which is overly simple. It would be more convincing if the authors could include more challenging and realistic tasks such as more dexterous manipulation tasks like pick and place and kitchen rearrangement and navigation tasks such as Habitat object navigation.\n\nQUESTIONS:\nClarify and address the concern mentioned in the above section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach for rapid policy adaptation to unseen visual observations for embodied agents. Their approach called CONPE (contrastive prompt ensemble) leverages contrastive learning to learn an ensemble of prompts for policy visual encoders that enables the policy to ultimately adjust to the domain shift. The results are demonstrated on multiple simulated environments, along with various ablations.\n\nSTRENGTHS:\n- The work addresses a well-motivated problem, which the authors explain well.\n- The authors’ build up on advances in contrastive learning to develop a simple yet novel approach for policy adaption in embodied settings.\n- The authors demonstrate their approach in multiple environments/tasks: AI2THOR, CARLA, and Metaworld.\n- The authors also present extensive experiments, ablations, and analysis to provide a holistic understanding of their approach.\n\nWEAKNESSES:\n- The terminology used in the paper is confusing at places. The authors never define what they mean by a “prompt” or “zero-shot performance”. This is especially confusing in Sec. 3 where they explain the framework. In the overview in Sec.3.1 they talk about zero-shot policy deployment (L99). However, Sec.3.2 only talks about jointly learning the policy and the guided attention for the prompt ensemble (also main results tab.1 — see my comment in questions section on this). Overall, since the authors also use specialized data for contrastive learning of domain-invariant prompts to later enable policy adaptation (through joint learning of policy), I am wondering whether their approach can really be called zero-shot. Despite this, I do think their approach is useful for policy adaptation amongst domain shifts.  Hence, I recommend an acceptance. However, I encourage the authors to clarify their terminology and claims especially around “zero-shot”.\n- The paper is also missing a limitation section. It is unclear where CONPE struggles. I encourage the authors to add one. What happens when a domain factor is varied beyond a degree present in the prompt learning data? e.g., a camera with much smaller view or more pan-tilt than present in the prompt training data. Lastly, I would also be interested in understanding how CONPE might work in real-world for instance, for robots.\n\nQUESTIONS:\n- It would be helpful to define source, seen and unseen target domains more concretely for Tab.1. I am a little confused between source and seen. Likewise, it would be useful to define what the authors mean by zero-shot for Tab.1 — aren’t a prompt pool and policy are jointly learnt for CONPE in Tab.1? \n- L255: Unclear what ENS approach is. Please provide a 1-line description with a reference, if it is from an existing paper.\n- Tab.5:  Shouldn’t E2E be the upper bound? Why does it have lower performance than CONPE?\n\nLIMITATIONS:\n(see also the weakness section)\n- This might be out of scope for current work but the results are only shown on simulated environments. Would be great to see some discussion on how CONPE might work in real world.\n- A limitation section is also missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

comments: SUMMARY:
The authors propose a method to address zero shot adaptation to novel visual domains. They do so by:
i) Learning a set of visual prompts via contrastive learning that encode specific domain invariant knowledge.
ii) Using a prompt ensemble based on embedding similarity scores (attention)

STRENGTHS:
* The authors have incorporated the idea of image prompting to improve the model's capacity to adapt to novel environments in an effective manner.
* The experiments have been conducted on re implemented baselines from other works with all the hyperparameters specified in the appendix, which is deeply appreciated.
* The diagrams are a big help and very well done!

WEAKNESSES:
* I wish there were a deeper analysis on understanding/ interpreting the prompts better. Maybe something like visual analyses of images that tended to use similar prompt clusters thus seeing what invariant information the prompt encoded. 
* I think the reader would benefit from a deeper understanding of the actual differences in the seen vs unseen domains, to get a sense of how similar or different they are.

QUESTIONS:
* Why was CLIP the choice of Visual encoder?
* Could you elaborate on the expert dataset used to do the prompt based contrastive learning?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for improving zero-shot domain adaptation for Embodied AI agents by learning a task-specific state representation.\n\nSpecifically, $n$ visual prompts are learned by fine-tuning the agent's visual encoder (i.e. CLIP) contrastively on observations. Each visual prompt is specific to a \"domain factor\" (such as camera position or illumination). Observations are positively paired if their state is the same (but the observations themselves may differ depending on domain factors).\n\nThen, an ensembling technique is used to compute a task-specific state representation. This is an average of the prompt encodings, weighted based on their cosine similarities to the observation encoding. This is used to optimize the policy using RL.\n\nSTRENGTHS:\n- The method presented in this paper is relatively simple and appears to provide significant gains over SOTA for both source and (seen/unseen) target domains on several benchmarks (AI2-THOR tasks, egocentric-Metaworld, CARLA).\n\n- To the best of my knowledge, visual prompting has not previously been explored in Embodied AI. This paper also emphasizes learning visual variations that are specific to domain factors (and shows that this helps generalize to unseen domains).\n\n- This paper includes several ablations, e.g. for prompt ensemble scalability and ensembling methods.\n\n- Zero-shot domain adaptation is a necessary and under-explored capability in Embodied AI, as it has many downstream applications (e.g. Sim2Real for robotics).\n\nWEAKNESSES:\n- It's a bit hard to interpret the sample efficiency experiment in Fig. 4. For the seen target domain, it's true that the performance is higher for ConPE than EmbCLIP at 0.5M steps, before either method has converged (approx. at 1M steps). But it's a bit sparse, as success is only measured every 0.5M steps (and the performance for ConPE is always higher than EmbCLIP anyway). Something somewhat similar could be said for the unseen target domain. Also, what benchmark does this correspond to? If this is averaged over several benchmarks, these should probably be split into different plots.\n\n- I believe the details related to the domain factors could be clarified. This seems to be an important component of the methodology and several things are unclear: what is the full list of domain factors (and do these vary per environment?), how were these domain factors chosen, what is the variance per domain factor within one environment (are these discrete or continuous values? are they uniformly sampled per observation?), and how were these subsampled for the experiment in Table 3. Most importantly: how does visual prompting perform without any domain factors? I.e. what are the baseline metrics if you fine-tune a single visual prompt for each environment?\n  - These points are particularly important for the scientific contribution of this work.\n\nQUESTIONS:\n- It is said that $T_\\phi$ is \"fine-tuned\" but I believe this must be frozen and only the visual prompts are fine-tuned. Is that correct? Should be clarified if so.\n\n- Could the authors please clarify whether they use RoboTHOR or iTHOR for the AI2-THOR benchmarks? E.g. the ObjectNav task is available for both environments.\n\nLIMITATIONS:\nAs far as I understand, this method may not be generalizable between simulators/environments, because the domain factors may not be compatible between these. Future work will be necessary to help improve such generalization -- fine-tuning the visual encoder offline is likely a valid approach, but factor-specific visual prompting or ensembling is not.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework for “efficient policy adaptation”, i.e. adapting learned policies to changes in the domain (e.g., lighting, camera height etc.), using a very small amount of in-domain data (few-shot and zero-shot). The proposed approach comprises learning “visual prompts” for modeling each different kind of variation in the domain, and using an attention head over these prompts to learn policies that are invariant to these factors. Empirical analysis in three different settings (AI2THOR, Metaworld and CARLA) show that the proposed method outperforms other representation learning alternatives.\n\nSTRENGTHS:\nThe paper studies a problem that is central to the sim2real generalization of robotic policies trained in simulation: how well do these policies generalize to varying conditions in the domain that are out of the agents control. The proposed approach, while having a lot of moving parts, seems to be extremely data-efficient (~10 expert episodes and small amount of diversity used), which can make a strong case for real-world applications. I also found the ablation analysis to be thorough and convincing regarding the design decisions made.\n\nWEAKNESSES:\n- [Method Presentation, Reproducibility] My biggest concern with the paper in it’s current form is the poor presentation of method details and reproducibility. I appreciate that the authors have provided code in the supplemental material, which will really help reproduce the results and build on them, but as a research article of novel, scholarly work, the paper is extremely hard to parse and missing on several key details. Updating these and clarifying the integral components of the paper better would be imperative to the impact of the paper. Some questions:\n\t- In Section 3.2, it is not clear what the learned parameters are: only the prompts $p^v$, or both prompts and $\\mathcal{T}_\\phi$. The objective only says $p^v$, but the text says “we fine-tune $\\mathcal{T}_\\phi$ by learning a visual prompt…” This is _really_ important to understanding why the method works: if T is indeed being finetuned, how is this standardized across the tasks/domains? Why would $p^v$ learn anything useful or interpretable at all?\n\t- I don’t think it is ever mentioned what specific pre-trained representation $\\mathcal{T}$ actually is. At some point, the authors say it is CLIP but it is not clear how CLIP is used. CLIP consists of two encoders (text and visual): which of these is $\\mathcal{T}$? It also seems to be overloaded as something with one or two arguments, and while this is somewhat understandable in context, this should be clarified for correctness.\n\t- “each visual prompt in $p^v$ encapsulates a specific domain-invariant knowledge” is a somewhat strong statement. Is there any reason this is interpretably something domain-invariant? Empirical/illustrative insights would really be helpful here, or even a qualitative discussion.\n\t- [L141-143] This statement is very convoluted and seems unclear. Can the authors very concretely define what they mean by “task” and “domain”? Maybe this is well-understood by a sub-community, but certainly seems vague to me in the context of a representation learning paper. \"As G is optimized by a given RL task objective in the source domains (in line 12), the resulting Z tends to be task-specific, while Z is also domain-invariant by the ensemble of contrastively learned visual prompts based on G with respect to various domain factors.”\n\n- [Prior work] In the introduction, the authors make some rather bold claims that I think should be qualified. In [L27-30], the authors say “It was rarely investigated how to optimize…, to ensure the zero-shot capability of embodied agents”. Two thoughts here:\n\t- It should be clarified if this statement is regarding zero-shot generalization of visual encoders or entire policies. This has indeed been studied by many works: both for encoders [5] and policies, most recently in the navigation/embodied AI settings by [1, 2]. It would be nice to acknowledge some of this prior work and position the paper more carefully in their context (not saying there are no differences or it reduces the contributions of this paper).\n\t- And for what it’s worth, the current paper does not really address this. The closest reported results analyze how policies trained with a small set of domain properties generalize to slightly different properties.\n\n- This trend continues with statements like “egocentric visual domain changes experienced by embodied agents due to different camera positions”. This has indeed been studied for visual encoders [5, 6] trained on diverse data such as Ego4D, as well as for policies [1, 2, 3] (both these papers specifically studied the case of different camera positions and FoV, and even different embodiments etc.).\n\n- [Task & eval Description] Some of the lack of key information seeps into the evaluations as well. Separating it for better readability. I was unable to interpret the results without getting more clarity on what the experiment setup is and what is being reported. Please see some specific concerns below:\n\t- What do tasks/domains etc. mean specifically for each of the environments considered in the paper? Some details seem to be in the appendix, but even a brief discussion for completeness in the main paper with pointers to appendix should suffice.\n\t- I don’t think the specific task in CARLA is defined anywhere, even in the appendix.\n\t- What does “zero shot” mean in Section 4.1? Is it trained in the same environment for the same task but with different lighting conditions/camera parameters etc.? Or is everything different? What is the exact experiment? The text as well as the captions are uninformative about this (e.g., “The Seen Target column presents the performance for the seen target domains, and the Unseen Target column presents the performance for the unseen target domains.”) I suspect that the words “zero-shot performance” are too vague and broad and mis-communicate some of the findings. I would like to see some clarity on this before I can interpret the results.\n\t- The caption in Fig 5 is pretty hard to parse, could the authors clarify what is being visualized in the left image and why it indicates \"domain-invariant knowledge” ?\n\t- What is the “pretrained policy” in Section 4.2 actually pretrained on? It seems like individual policies are used for each of the different tasks in each environment, so what exactly is being “pretrained”? It’s really not clear what the distinction between pretrain/prior results are, or what tasks/domains mean and what the exact experiment is.\n\t- In the experiments, I see that the policy training used ~10 episodes per task. How much data was used for training the visual prompts, though? Seems like that would be quite high (which is fine since this is likely done procedurally in sim).\n\n- A broader question regarding the setup of the experiments to me is how much of an improvement a sophisticated scheme like the one proposed over something more naive, like (i) learning a parametric policy that simply conditions on the different factors of variation [3], or (ii) training a joint policy on data from multiple domains across all these parameters, akin to domain randomization, [1, 2] so the factors of variation can be learned to be ignored. I imagine either of these “simpler” methods would require more data, but given this is a study in sim where data is free, does the proposed method help in certain ways? An analysis of this could be really important.\n \n[1] Putta et al., “Emergence of Implicit System Identification via Embodiment Randomization”, 2023\n\n[2] Shah et al., “GNM: A General Navigation Model to Drive Any Robot”, 2022\n\n[3] Hirose et al., “ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentation”, 2022\n\n[5] Majumdar et al., “Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?”, 2023\n\n[6] Nair et al., “R3M: A Universal Visual Representation for Robot Manipulation”, 2022\n\nQUESTIONS:\nPlease see the concerns and questions raised in the detailed weaknesses section above.\n\nLIMITATIONS:\n**I did not find limitations** or societal impact sections (the latter is likely not required) in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}