PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation
Authors: Jialu Li
Mohit Bansal
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision-and-Language Navigation
diffusion models
image inpainting for panorama generation

Abstract: Vision-and-Language Navigation requires the agent to follow language instructions to navigate through 3D environments. One main challenge in Vision-and-Language Navigation is the limited availability of photorealistic training environments, which makes it hard to generalize to new and unseen environments. To address this problem, we propose PanoGen, a generation method that can potentially create an infinite number of diverse panoramic environments conditioned on text. Specifically, we collect room descriptions by captioning the room images in existing Matterport3D environments, and leverage a state-of-the-art text-to-image diffusion model to generate the new panoramic environments. We use recursive outpainting over the generated images to create consistent 360-degree panorama views. Our new panoramic environments share similar semantic information with the original environments by conditioning on text descriptions, which ensures the co-occurrence of objects in the panorama follows human intuition, and creates enough diversity in room appearance and layout with image outpainting. Lastly, we explore two ways of utilizing PanoGen in VLN pre-training and fine-tuning. We generate instructions for paths in our PanoGen environments with a speaker built on a pre-trained vision-and-language model for VLN pre-training, and augment the visual observation with our panoramic environments during agents' fine-tuning to avoid overfitting to seen environments. Empirically, learning with our PanoGen environments achieves the new state-of-the-art on the Room-to-Room, Room-for-Room, and CVDN datasets. Besides, we find that pre-training with our PanoGen speaker data is especially effective for CVDN, which has under-specified instructions and needs commonsense knowledge to reach the target. Lastly, we show that the agent can benefit from training with more generated panoramic environments, suggesting promising results for scaling up the PanoGen environments to enhance agents' generalization to unseen environments.

PANOGEN: Text-Conditions Panoramic Environment
Generation for Vision-and-Language Navigation
Jialu Li
Mohit Bansal
UNC Chapel Hill
(jialuli, mbansal)@cs unc. edu
https://pano gen. wet.ggittub.... io
Abstract
Vision-and-Laanguae Languag Navigation requires the agent to follow language instructions navigate through 3D environments. One main challenge Visionaand-Laguaag Navigation is the limited availability of photorealistic training environments, which makes hard generalize to new and unseen environments. To address this problem, we propose PANOGEN, a generation method that can potentially cre- ate an infinite number of diverse panoramic environments conditioned on text. Specifically, we collect room descriptions by captioning the room images in ex- isting Matterport33 environments, and leverage a state-of-the-ard text-to-image diffusion model to generate the new panoramic environments. We use recursive outpainting over the generated images to create consistent 360-degree panorama views. Our new panoramic environments share similar semantic information with the original environments by conditioning on text descriptions, which ensures the co-occurrence of objects in the panorama follows human intuition, and creates enough diversity in room appearance and layout with image outpainting. Lastly, we explore two ways of utilizing PANOGEN in VLN pre- vreitraining and fine-tuning. We generate instructions for paths in our PANOGEN environments with a speaker built pre-trained vision-and lad---aa--n model for VLN pre-training, and augment the visual observation with our panoramic environments during agents' fine-tuning to avoid overfitting seen environments. Empirically, learning with our PANOGEN environments achieves the new state-of-the-art on the Noom-to-Room Room-for- Room, and CVDN datasets, Besides, we find that pre-training with our PANOGEN speaker data is especially effective for CVDN, which has under-specified instruc- tions and needs commonsense knowledge reach the target. Lastly, we show that the agent can benefit from training with more generated panoramic environments, suggesting promising results for scaling up the PANOGEN environments enhance agents' generalization to unseen environments.
Introduction
Vision-and-Language Language Navigation (VLN) requires an agent to make sequential decisions based on both language instructions and visual environments. In indoor instruction-guile guided navigation, many datasets have been proposed. These datasets aim to enhance agents' ability to understand detailed navigation instructions [2], dialogue style instructions [52], instructions in different languages [26], and high-level ubject-finding instructions [43]. Though many efforts have been proposed to help the agent learn diverse instruction inputs, all these datasets are built on the same 3D room environments from Matterport3D, which only contains 60 different room environments for agents' training. This is because diverse photorealistic 3D room environments with a large number of sampled soman-height viewpoints are very hard to collect. This limited availability of training environments poses challenges for the agent to learn the navigation policy well, and generalize to unseen new room environments.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Room Panorama Gaptioning
Toxtro waaaooo-iaaeeeeee Generation
Textio-Pannoran Generation
Parotama:
Irput
Captioms
avd Sie aneeed piok
ebeerer Practingg lin
Stable Diflusion
Discretize Panorama
Stable Diffusion
One Ganenated
Avieei vinane piot
Captione
LERI
O elevtios
Genamtted
Outpainting
12 heading!
Visws
elevation
Outgsint View
Image Capdion
Image Stitching
Image Stttching
N Captiona
vindow rain pool becison with lag fan.
X
Incoherent
Panorama
Cohement
Figure I: Overview of our PANOGEN. We first generate captions for all the room panoramas in the Matterport3D dataset. Each panorama is discretized into 36 views, we show 15 views here for a better view of each discretized image. Then, we generate panoramic environments with recursive outpainting over a single image generated from the text caption.
Several works in VLN have been proposed to address this challenge. [51] first proposes to perform dropout on environments during training to avoid the agent overfitting and [31] further propose to edit the existing environments by mixing up environments and changing the style and appearance of the environments. However, these approaches are still limited by the 3D environments in Matterport3I., and do not create new environments with different objects and layouts, which is important for agents' generalization unseen environments. [11] takes one step further and introduces more unannotated 3D room environments from Habitat Matterport3D dataset (HM3D) [46], with machine-generatee instructions augment the training environments and enhance agents' generalization performance. While 900 environments are introduced, their method still relies on existing manually-captured 3D environments in HM3D, which cannot be further scaled up due to the very expensive environment collection process. Hence, in this paper, we aim to explore the possibility of generating unlimited panoramic environments without any human annotation, and investigate how to effectively utilize the generated panoramic environments for improving navigation agents' ability to generalize to unseen environments.
environments. [34]
To this end, we propose PANOGEN, a generation method that can potentially create infinite diverse panoramic environments conditioned on text. As shown in Figure we first collect descriptions of room environments by using state-offthe-art vision-language model BLIP-2 [32] to annotate the view images in the Matterport3I dataset. Then, use text-to-image diffusion models generate diverse room images based on the text captions. As the agent navigates in egocentric panorama observation, learning from disjoint single-view images (middle column in Figure 1) will confuse the agent, and it cannot learn the spatial relationship between objects due to inconsistent views. Hence, keep the observations coherent the same panorama, we additionally propose recursive image outpainting approach, which generates missing observations beyond the original image boundaries (right column in Figure 1). Specifically, we choose one generated image the panorama as the starting point, and gradually rotate the camera angle right, left, up, and down, and then outpaint the unseen observation based on text descriptions. Lastly, we explore and compare two training methods to effectively utilize the generated panoramic environments. In the first method, we train a speaker to automatically generate instructions for the generated panoramic environments, based on a pre-trained model mPLUG [27]. which has the sfate-of-the-art performance for zero-shot vidco captioning. We pre-train the VLN agent with both the original training data and the generated instruction eestruction-traecctee trajectory pairs from panoramic environments. In the second method, instead of learning from speaker data, we directly fine-tune the VLN agents on both the original environments and our panoramic environments by randomly replacing some of the original observations with our panoramic environments to avoid overfitting to training environments.
We conduct experiments on Room-to-Room (R2R) [2], Ccom-for-Rooe (R4R) [21], and CVDN [52] datasets. We measure agents' ability following fine-grained instructions various lengths (R2R, and R4R), and under-specified dialogue instructions (CVDN). Empirical results demonstrate that training with our PANOGEN environments could improve the SotA agents by 2.7% in success rate and 1.9% in SPL on the R2R test leaderboard, and improve the goal progress by 1.59 meters the CVDN

---

test leaderboard, achieving relative gain of 28.5% compared with previous SotA agents. The large improvement on the CVDN dataset suggests that our PANOGEN introduces diverse commonsense knowledge of the room environments, and helps navigation when given under- snderr-peeiifed dialogue instructions. Moreover, we analyze the impact of the number of our PANOGEN environments used for VINN training, and demonstrate the potential of scaling up our PANOGEN environments for further enhancing agents' generalization ability. Lastly, we use both qualitative and quantitative analysis to demonstrate good alignment between our generated instructions and PANOGEN environments.
2 Related Work
Vision-and- Lang-agg Navigation Vision-ande Arroon-nnd-aanguee Language Navigation is the task that requires an agent to navigate through the visual environment based on language instructions. Many datasets [2, 21, 52, 8, 39, 49, 41, 26, 43] have been proposed for this task. In this paper, we focus on indoor navigation, specifically Room-to-Rot Ro-Room dataset (R2R). Room-four-Rond
dataset (R4R), and Cooperative Vision- and-Dialog Navigation dataset (CVDN). To solve this challenging task, early works build their model based on a sequence-to-seqeene LSTM model [51, 35, 29, 30, 55, 33]. [36] first utilizes pre-trained vision-and-language transformers for learning an agent to pick the correct path. [20] enhances the transformer-based agent with recurrent state to model navigation history, and [10] proposes a hierarchical architecture to encode both spatial and temporal information in navigation history. Besides, some works explore utilizing graph information to build better global visual representation [12, 53, 63, 15, 9]. some works propose better proxy tasks for learning better temporal knowledge [44] and decision making ability [10, 28] during pre-training, and other works explore enriching the navigation agent with the ability to generate text [56] and image [28]. In this paper, we build our approach over the state-of-the-arr agent DUET [12].
Navigation One main challenge in Vision-and- Language Navigation is to learn from limited available training environments and generalize to unseen new environments. Though many datasets have been proposed for indoor instruction-guider navigation with diverse instruction inputs [2, 26, 21, 52, 43), their navigation environments are all from Matterport3D [7], which only contains 61 environments for training, and 29 environments for unseen validation and testing. Previous works aim to address this challenge from multiple aspects. One line of work tries to mitigate the environment bias during training by dropping out environment features [51] and learning environment-agnosta visual representation [30, 57]. Another line of research aims to generate or introduce more environments for VLN training. Specifically, [31] and [34] edit the existing environments by mixing up environments or changing room style and object appearances. However, these approaches are still limited by the existing Matterport3D environments. [11] introduces 900 environments from HM3D [46] with generated instructions to address the environment scarcity and learn a more generalizable agent. However, it's expensive to collect 3D environments, and thus their approach is hard to be further scaled up. Different from them, our proposed PANOGEN generates new panoramic environments without the need of any human annotation, and could potentially generate unlimited panoramic environments. Lastly, [24] generates future views based on history observations in a given trajectory so as to build a world model for more informed planning, and [25] synthesizes views at different locations in the original environments as data augmentation for VLN training. Our work focuses on introducing diverse set of new environments (in appearance), while maintaining consistency in the panorama, using text-conditioned recursive image outpainting.
Environment Scarcity in Iision-and-Lange
Text-to-Image Generation Text-to-Image generation has been one of the main research areas in both NL. and CV. With the advances of GANs [16], previous works aim to generate high resolution images with stacked generators [60, 61], and improve the perceptual quality of an image by adding regularization losses [6]. However, GANs are hard to optimize [17, 37] and face the problem of mode collapse [38]. Recently. [47] proposes utilize the latent diffusion models to generate images with high resolution. By first encoding the image into a low dimension latent space that is perceptually equivalent to the pixel space, the latent diffusion model is computationally more efficient while maintaining good generation quality. Diffusion model shows better performance on several image generation tasks like image inpainting [62], image outpainting [59], and image synthesis [42]. [50] first extends the diffusion models to 3D indoor scene synthesis by conditioning it on geometry and editing prompts. Different from them, we propose to generate coherent panorama environments with diverse objects based on detailed text descriptions.
3

---

N-1 Remaining Views
Generated View
Rotate Camera Angle
Right
Up
Down
Image Captioning
Outpaint Recursively
Generated Captions
Image Outpainting
bedrocm with bed and dresser.
Image
Siftching
Figure 2: Given image generated based the room description, rotate the camera angle and outpaint the unseen observation recursively to generate a consistent 360-degree panorama view.
3 PANOGEN: Generating Panoramic Environments
In this section, we describe our PANOGEN method, which generates panoramic environments from text descriptions for Vision-and-Languaae Navigation. Our PANOGEN addresses two main challenges for panoramic environment generation. First, the panorama observation of an indoor room environment will contain multiple objects with a complex layout, which makes it hard to use short description to cover all the details in the panorama. Thus, first discretize the panorama into 36 views and generate each view separately. However, the separately generated single images will be inconsistent with each other. Hence, to generate consistent panoramic environments, we propose a recursive image outpainting approach (Sec. 3.2). Second, the generated panoramic environment should align with human commonsense knowledge. For example, when generating panorama for bedroom, the panorama can consist of objects like "bed", "dresser", but not "refrigerator". To address this problem, instead of utilizing large amount of available room image captions from the web for different portions the panorama, which might have unreasonable co-occurrence of objects, we directly generate text descriptions for the panoramas in the Matterport3D environments (Sec. 3.1).
3.1 Room Description Collection
We first collect room descriptions from the Matterport3D environments (Figure Right). To maxi- mally generate panorama environments with diverse objects and reasonable layouts, we discretize the panorama into 36 views and caption the discretized views separately. The generated captions contain more detailed description of the objects in the smaller region of the panorama. We then utilize the pre-trained dision-and-languaa BLIP-2 trained by bootstrapping off-the-shelf pre-trained image encoders and large language models. Benefiting from the general cross-modal alignment between text and image learned during pre-training, and the zero-shot generation ability of large language models, BLIP-2 can generate creative and informative captions for images in the Matterport3D dataset.
model from BLIP-2 [32] to caption the discretized room images.
3.2 Generating Panorama with Text-Conditioned Image Outpainting
While many works [47, 22, 40, 48, 3] have focused on building better models for text-to-image generation, how to transfer the advances in text-to-image generation to (ext-to-panorama generation still less explored. [4] combines multiple diffusion generation processes with shared constraints to generate a panorama based on the text description. However, their panorama generation process single text description of the full panorama. One text description is
only be conditioned not enough for complex indoor navigation environments with diverse objects. Thus, we propose to generate the panorama view by outpainting recursively based multiple text descriptions.
As discussed Sec. 3.1, each discretized image the Matterport3D environment paired with caption generated by BLIP-2. With the detailed descriptions, use state-of oh--t--rrttt text-to-image

---

diffusion model Stable Diffusion [47] to generate the images. However, the panorama view is not continuous we directly 'stitch' the generated images. To address this problem (Figure 2), we first generate one single view with zero camera elevation in the panorama based on its caption. The view with zero camera elevation serves as a better starting point, as it usually contains more important information in the panorama, whereas the positive elevation view usually consists of objects on the ceiling, and the negative clevation view usually faces the ground. Next, we rotate the generated image right, up and down by p, %, Pv. %, p&% respectively, and outpaint the unseen observation based on the caption of the nearby view. By conditioning the image generation on both the text caption and nearby view, the generated images are consistent in style and can be stitched into coherent panorama. For example, as shown in Figure 2, the caption "a bedroom with a bed and a dresser" mentions "bed", and the bed is already partially generated in the image. When outpainting the unseen observation, our generation model will complete the bed and generate coherent surrounding views, instead of generating new bed with different appearance. We repeat this process until we generate all 36 views in the panorama. We generate one panorama for each panorama in the R2R training environments. In total, our PANoGEN have 7,644 panoramas, stitched from 275,184 images.
Utilizing Panoramic Environments for VLN Training
In this section, we first introduce the problem setup of VLN in Sec. 4.1, and the general training procedures for VLN in Sec. 4.2. Then, we introduce the two ways that we utilize the generated panoramic environments. Specifically, we first explore generating new instruction trajeoootttjjjtrrryy pairs in the panoramic environments and utilize these data in VLN pre- wre-training (Sec. 4.3). Then, we directly utilize the generated panoramic environments to replace the original observation during VLN fine-tuning to avoid agents' overfitting to training environments (Sec. 4.4).
4.1 Problem Setup
In Vision-and and- Language Navigation, the agent needs to navigate through the environment based on natural language instructions. Formally, each time step I, the agent takes in the full language instruction 7, a panoramic observation P; of the current location, and navigation history observations [P.) The agent needs to pick the next step from a set of navigable locations { We follow the setup in [12], where navigable locations include both adjacent locations which can be reached in one step, and global viewpoints which are observed but not visited in navigation history. Navigation finishes when the agent predicts 'STOP' action or reaches the maximum navigation steps.
4.2 VLN Training Procedures
Training a Vision- and-Language Navigation agent contains two stages: pre- veeitraining and fine-tuning.
Pre-training, In the pre-training stage, the agent is pre-trained with three proxy tasks: Masked Lan- guage Modeling (MLM), Instruction and Trajectory Matching (ITM), and Single Action Prediction (SAP). Specifically, in Masked Language Modeling, the agents need predict the randomly masked words given both the unmasked instructions and the full trajectory observations. In Instruction and Trajectory Matching, the agent needs to learn pick the correct instruction and trajectory pair from one positive pair and four negative pairs. Two of the negatives come from other trajectories in the same batch, and another two of the negatives shuffle the original trajectories so that the agent could learn the order of the observations in the trajectory. In Next Action Prediction, the agent mimics the navigation task by doing single action prediction based on full instruction and navigation history.
Fine-tuning, In the fine-tuning stage, we follow [12] and train the agent with the supervision from the pseudo interactive demonstration. Specifically, each time step, the ground truth navigation trajectory is sampled based on the current policy learned by the agent. This sampling process enables the agent explore the environment and generalize better.
4.3 Generating Instructions for Paths in Panoramic Environments
Given the high quality and coherent panoramic environments from our PANOGEN, we investigate how to utilize these environments for mitigating the data scarcity problem VLN agents' training and enhancing VLN agents' generalization ability.
5

---

As our PANOGEN environments are generated conditioned on text captions, it shares similar semantics with the original environments, but the room layout and appearance will be different from the original environments. Thus, the instruction for traversing the original environment will not be aligned well with the new panoramic environment. To address this problem, we train a speaker to generate instructions for the new panoramic environments.
Previous works [51, 14, 31, 18] train a small LSTM-based speaker from scratch on VLN datasets to generate instructions for unannotated paths in the Matterport3D environments. However, as these speakers are not pre-trained on larger image-text datasets, they lack general visual grounding knowledge and therefore it's hard to generate instructions with diverse entity mentions that do not appear in the small training data. Marky [54] improves the speaker by utilizing multilingual T5 (mT5) [58], which is a text-to-text encoder-decoder transformer. mT5 enables the speaker to generate instructions with a large multi-lingual vocabulary, and has prior text domain knowledge. However, mT5 only has text domain knowledge, and will need to learn the visual grounding knowledge from scratch on the limited training data. Hence, to introduce general cross-modal alignment information to the agent, we propose to build our speaker based mPLUG [27], vision-language model that can do both multi-modal reasoning and generation.
mPLUG is transformer pretrained on image and text pairs. To adapt it to navigation trajectory, which is sequence of panorama images, we first simplify the panorama representation to the single view representation which the agent is facing. The single views are first encoded with CLIP-ViT/B- 16. The encoded image patches are then flattened and concatenated as the input for instruction generation. We fine-tune mPLUG on the Room-to-Room (R2R) training data, and use it to automatically generate instructions for all the paths in the R2R training data with our new panoramic observation. In total, our speaker data has 4,675 instruction-traaectryy pairs.
Following [12], we use R2R dataset [2], and Prevalent dataset [18] for agents' pre-training Moreover, we further pre-train the agent with our speaker data, which introduces our panoramic environments to the agent and improves the agents' generalization ability.
4.4 Learning from Panoramic Environments during Fine-tuning
We further enhance the VLN fine-tuning by incorporating our panoramic environments. Specifically, we randomly replace m% of the observations in the trajectory with our panoramic environments during fine-tuning. This observation replacement helps the agent avoid overfitting to the limited training environments. As our panoramic environments are generated conditioned on text descriptions of the room, thus the semantic underlying the panoramic observation is similar to the original environments. This ensures that when the replacement ratio m% not large, after replacing the original environments with our panoramic observations, the instruction and the observations still have reasonable alignment (discussed in Sec. 6.4).
Experimental Setup
5.1 Dataset and Evaluation Metrics
We evaluate our agent on three datasets: Room-to-Room dataset (R2R) [2], Cooperative Vision-and- Dialog Navigation dataset (CVDN) [52], and Room-for-Re Room dataset (R4R) [21]. The training set contains 61 different room environments, while the unseen validation set and test set contains 11, and 18 room environments that unseen during training. Details of the dataset can be found in Appendix.
We evaluate our model on the following metrics: (1) Success Rate (SR), which measures whether the agent stops within meters to the target. (2) Success Rate Weighted by Path Length (SPL) [1], which penalizes long paths that explore the environment to reach the target instead of following the instructions. (3) Goal Progress (GP), which calculates the distance that the agent moves toward the
target location. (4) Navigation Error (NE), which is the distance between the stop location and the
target. (5) Trajectory Length (TL), which counts the total navigation length of the agent. We consider
SPL as the main metric for R2R and R4R dataset, and GP as the main metric for CVDN datasct.

---

Table 1: Test leaderboard performance for Room-to-Rooe dataset and CVDN dataset. indicates approaches that augment the training environments. For "EnvEdit", we report the non-ensemble performance for fair comparison to all other methods. Best results in bold, and second best results are underlined.
Models
Room-to-Room dataset
CVDN
Validation Unseen
Test
Val
Test
TI.
NE
SR
SPL.
TI.
NE +
SR
SPL.
GP
GP f
RecBERT [20]
12.01
3.93
63.0
57.0
12.35
4.09
63.0
57.0
NDH-Full [23]
5,51
5,27
HAMT [10]
11.46
2.29
66.0
61.0
12.27
3.93
65.0
60.0
5.13
5.58
EnvEdit *
12.13
3.22
67.9
62.9
SE3DS [25]
3.29
69.0
62.0
3.67
66.0
60.0
DUET [12]
13.94
3.31
72.0
60.0
14.73
3.65
69.0
59.0
DUET-CLIP
12.92
3.19
72.8
63.4
5.50
PANOGEN
13.40
3.03
74.2
64.3
14.38
3.31
71.7
61.9
5.93
7.17
Betterpon 20
Quise Diftcaton Puntbon Dischelord Vienea
Figure 3: Qualitative analysis of the panoramic environments generated with our PANOGEN. "Mat- terport3D" is the original environment for VLN tasks. "Stable Diffusion for Discretized Views" is the concatenation of separately generated discretized views given text captions.
5.2 Implementation Details
In panoramic environment generation, we caption all the view images the training environments in R2R dataset with FLII--2-FlanT5-xxx We utilize stable-diffusion--2211 base model generate the single view based on caption only, and use stable-diffusion-vl wtw.diffusiion.l.--aaaiiiiiii model to outpaint the unseen observation for the rotated views. In speaker data generation, we build our speaker model based on mPLUG-base. For navigation training, we adopt the agent architecture from DUET [12]. More Implementation Details be found in Appendix.
Results and Analysis
In this section, we first present state-of- ofate-of-the-arr performance on test leaderboard on Room-to-Rowe toom dataset and CVDN dataset in Sec. 6. Then, we show some qualitative examples our generated panoramic in Sec. 6.2. Besides, we further use ablation studies to demonstrate that utilizing our speaker data in the panoramic environments during pre-training, with new panorama during fine-tuning effective for enhancing VLN agents' performance in unseen environments in Sec. 6.3 and Sec. 6.4. Moreover, we investigate how the number the panorama environments influence the performance in Sec. 6.5, and compare PanoGen environments with the existing environment augmentation approach in Sec. 6.6. Lastly, we include two quantitative evaluations of the quality of generated speaker data in Sec. 6.7.
random replacing the observation
6.1 Test Set Results
We show our method's performance on both the Room-to-Roome (R2R) and the Cooperative Vision- and-Dialog Navigation (CVDN) dataset. We adopt DUET [12] architecture for our navigation agent.

---

Table 2: Ablation results for utilizing our speaker data during pre- pre-training validation unseen set
for R2R, CVDN, and R4R datasets.
Models
Room-to-Roo to-m
CVDN
Room-for-Room
TL
NE 4
SR
SPL t
TL
GP
TL
NE 4
SR 1
SPL 1
DUET [12]
13.94
3.31
72
60
DUET-CLIP
12.92
3.19
72.84
63.37
24.09
5.50
21.04
6.06
46.61
41.94
PANOGEN+Env-oniy
14.21
2.99
73.35
62.12
PANDGEN-FnnDDrop
13.57
3.05
73.69
63.44
25.17
5.81
22.88
6.17
46.06
40.33
PANoGEN+mPLUG
14.58
2.85
74.20
62.81
24.66
5.93
18.32
6.12
45.78
42.52
Different from DUET which uses ViT-B/16 [13] pre-trained on ImageNet to extract features, we use CLIP-VIT-B/16 [45] it shows better performance on R2R dataset ("DUET" vs. "DUET-CLIP" in Table 1). As shown in Table 1, fine-tuning with panoramic environments from our PANOGEN improves previous SotA agent DUET [12] by 2.7% in success rate, and 2.9% SPL on Room-to- Room test leaderboard. This demonstrates the effectiveness of utilizzig our generated panoramic environments improve agents' generalization unseen environments. Moreover, pre-training with our speaker data improves the goal progress by 1.59 meters on CVDN test leaderboard, relative gain of 28.5% compared with previous SotA agent HAMT [10]. This large improvement demonstrates that learning from our PANOGEN environments are especially helpful for following under-specified instructions in unseen environments which need commonsense knowledge of the visual observation to reach the target. On both the Room-to- to-Room dataset and CVDN dataset, learning with our PANOGEN environments achieves the new state-of-the-ar performance.
6.2 Qualitative Analysis of Panoramic Environment
We show some panoramic environment generated with our PANOGEN in Figure 3. We could see that directly generating discretized views based on caption will be disjoint and inconsistent (Row "Stable Diffusion for Discretized Views"). In comparison, our recursive outpainting approach could generate continuous views that can be stitched together to form a high-quality panorama (Row "PANOGEN").
Besides the high quality and coherency, our generated panorama environments is able preserve the wide range of objects appeared in the original environments, while generating them with new appearance and different room layout. For example, in the left generated panorama, it contains a corridor view, and shows multiple rooms that are connected to the corridor (e.g., bedroom, and bathroom). This layout also follows human's commonsense knowledge, where the bedroom and bathroom can be connected with corridor. We include more qualitative examples of our panoramic environment and speaker data in Appendix.
6.3 Effectiveness of Speaker Data in Pre-training
In this section, we show the effectiivness of utilizing the speaker data generated for PANOGEN environments for VLN pre-training. As shown Table 2, pre-training the VLN agent with our speaker data ("PANOGEN+mPLUU") improves the baseline ("DUET-CLIP") by 1.36% in success rate, and 0.2 meters in navigation eтror on R2R dataset. Besides, we observe 0.43 meters absolute improvement in goal progress (a relative gain of 7.8%) improvements in CVDN demonstrates that the agent learns useful commonsense knowledge from the diverse visual environments in our PANOGEN, and thus generalize well the unseen environments when navigating based on under-specified instructions CVDN dataset. Pre-training with our speaker data also shows slight improvement in R4R, improving the SPL by 0.58%.
CVDN validation unseen set. The large
Moreover, we compare our speaker ("PANOGEN+mPLUG")) with the widely used speaker from We find that utilizing either speaker data improves the
EnvDrop [51] ("PANOGEN+Env-Doog) baseline performance on R2R and CVDN dataset, demonstrating the effectiveness of our PANOGEN environments for improving agents' generalization to unseen environments. Besides, pre-training with speaker data generated with mPLUG shows better performance in success rate on R2R and R4R dataset and higher goal progress on CVDN dataset, demonstrating the effectiveness of our speaker.
Lastly, we show the effectiveness of generating new instructions for PanoGen environments during pre-training. Compared with only pre-training on original instructions and PanoGen environments
8

---

Table 3: Ablation results for replacing the original environment with panoramic observation during fine-tuning on validation unseen set for R2R, CVDN and R4R datasets.
Models
Room-to-Rooe
CVDN
Room-for-Room
TI.
NE 4
SR
SPI.
TT.
GP
TI.
NE
SR
SPL f
DUET [12]
13.94
3.31
72
60
DUET-CLIP
12.92
3.19
72.84
63.37
24.09
5.50
21.04
6.06
46.61
41.94
PANoGEN-Replace
13.76
2.99
74.41
63.88
23.29
5.63
18.62
6.02
47.78
44.25
(*PANoGEN--an-oooll)) find that generating new instructions with mPLUG speaker improves the performance by 0.85% in success rate, and 0.69% SPL on R2R unseen set.
6.4 Effectiveness of Panorama Replacement in Fine-tuning
We demonstrate that using the environments from PANOGEN observation aug- mentation during fine-tuning can improve agents' generalization to unseen environments. As shown in Table 3, randomly replacing the observation in the trajectory with our panoramic environments ""PANOGEN+Replace"
Table 4: Comparison of replacing different ratio of the viewpoints the trajectory with the panoramic environments generated with our PANOGEN on Room-to-Roo validation unseen set.
improves the navigation performance on all the three datasets. Specifically, our approach improves the SR by 1.57% on R2R dataset, 0.13 meters CVDN dataset, and 2.31 % in SPL on R4R dataset. The consistent improve- ments in all the three datasets demonstrate the usefulness of our PANOGEN environments.
in goal progress
No.
Ratio
Validation Unseen
TL
NE J
SR t
SPL f
0.0
12.92
3.19
72.84
63.37
2
0.1
13.16
3.16
72.84
63.24
We further show that it's important to balance the ratio of replaced observations in the full navi-
3
0.3
13.76
2.99
74.41
63.88
4
0.5
13.03
3.19
72.84
63.84
gation trajectory. As shown in Table replacing 30% of the viewpoints in the navigation trajec- tory achieves the best performance. The trajec- tory will not be aligned with the instruction well if we replace a large ratio of viewpoints with our PANoGEn environments, and thus the improvements in R2R validation unseen set smaller.
5
0.7
12.62
3.18
72.33
63.93
6.5 Impact of the Number of Panoramic Environments
In this section, we investigate the impact of the number of panoramic environments used during VLN fine-tuning. Specifically, we randomly se- lect 10 scans and 30 scans out of the 61 scans in the R2R training environments. During VLN fine-tuning, if the navigation trajectory belongs to these scans, we replace the original observa- tion with generated panoramic environments with a fixed probability. As shown in Table 5, we observe that training with more panoramic environments consistently improves the perfor- mance (No. 1-4) 4). Furthermore, for every panorama in the original R2R training environ- ments, we generate one more panoramic view with our PANOGEN. In this case, the number of panoramic environments we generate is twice the number of the original R2R training environments. As shown in Table we observe that the gain in SPL is still not satured yet (No. 5 vs No. 4), and gradually increases when we add more our PANOGEN environments for VIN fine-tuning. This suggests that it's promising to generate more panoramic environments with our approach further enhance agents' generalizability.
Table 5: Comparison of replacing the original vironments with different number scans of our panoramic environments.
No.
Scans
Validation Unseen
TL
NE |
SR
SPL t
1
0
12.92
3.19
72.84
63.37
2
10
13.94
3.00
72.80
62.48
3
30
13.86
3.05
73.69
62,88
4
61
13.76
2.99
74.41
63.88
5
122
13.40
3.03
74.20
64.27

---

6.6 Comparison with Other Environment Augmentation Approaches
In this section, we demonstrate that training with our PanoGen environments achieves bet- ter generalization performance compared to other environment augmentation approaches.
Specifically, we compare our approach with two existing approaches that augment the environ- ments to avoid overfitting: EnvEdit [31] and EnvDrop [51]. For adapting EnvEdit to DUET, we follow the batch mixing approach in EnvEdit and randomly replace half of the data in a batch with the edited environments which change the appearance of the objects during VLN finetun- ing. For adapting DnvDrop to DUET, we replace the regular dropout layer in DUET with the pro- posed environment-level dropout layer during fine-tuning. As the results shown in Table 6, training with our PanoGen environments achieves better performance than previous approaches.
Table 6: Comparison of training with different environment augmentation approach on Room-to- Room validation unseen set.
Model
Validation Unseen
TL
NE L
SR f
SPL
EnvEdit EnvDrop PanoGen
13.61
3.03
72.80
63.17
13,28
3.12
72.58
62.40
13.40
3.03
74.20
64.27
6.7 Quantitative Evaluation of Generated Speaker Data
In this section, we evaluate the quality of our generated instructions by measuring its alignment with the environment, and measuring its BERTScore compared with the original instructions. First, we measure the similarity between the generated instructions and the PanoGen environments. We hypothesize that higher similarity between instruction and trajectory pairs in the embedding space can indicate better alignment between the instruction and the trajectory. Specifically, we represent the trajectory representation by averaging the image embeddings of the viewpoints in the trajectory. The image embedding of each viewpoint is encoded with with CLIP text encoder. We calculate the cosine similarity between the instruction representation and the trajectory representation. As shown in Table 7, we find that the similarity between our PanoGen environments and instructions generated with mPLUG is higher than instructions generated with EnvDrop (No.2 vs No. 3). Besides, we calculate the similarity between randomly replacing 30% of the viewpoints with PanoGen environments and the original instructions (to mimic the observation replacement fine-tuning). We average the score over 5 runs to mitigate randomness. We find that randomly replacing the observation doesn't lead decrease in similarity (No. No. 1).
We also encode the instruction
Second, we calculate the BERTScore of instructions generated by our mPLUG based speaker and the En- vDrop speaker. Specifically, use both speakers gen- erate the instructions on the R2R validation unseen set. We use Bart-base to calcu- late the BERTScore. Our speaker achieves BERTScore of 71.8, while the EnvDrop speaker achieves a BERTScore of 70.5.
Table 7: Comparison of similarity between generated instructions and environments.
No.
Instruction
Environment
Similarity
Original
Original
0.2845
2
EnwDrop
PanoGen
0.2669
mPLUG
PanoGen
0.2714
Original
30% PanoGen, 70% Original 0.2893 (1: 0.0001)
7 Conclusion
In this paper, we propose PANOGEN, generation approach which can potentially create infinite diverse panoramic environments conditioned on text. Specifically, we propose a recursive image outpainting that reconstructs the missing observations in the panorama gradually based on text caption to generate coherent panorama with diverse objects and room layouts. We then investigate two training methods to effectively utilize the generated panoramic environments during VLN pre-training and fine-tuning. Learning from our PANOGEN achieves the new SotA on both CVDN dataset and R2R dataset, demonstrating effectiveness for enhancing agents' generalization to unseen environments. Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.
10

---

8 Acknowledgement
This work was supported by ARO W911NF2110220, ONR N00014-23-1-2333 DARPA KAIROS FA8750-19-2-1000 1004, and DARPA MCS N66001-19-2-403, The views contained in this article are of the authors and not of the funding agency.
References
[1] P. Anderson, A. Chang, D. Chaplot, Dosovitskiy, S. Gupta, Koltun, J. Kosecka, Malik, R. Mot- taghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint 18Xiv 2018.
[2] P. Anderson, Q. Wu, D. Teney, Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and-language navigation: Interpreting essually-grounde navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6674-3683, 2018.
[3] Y. Balaji. S. Nah, X. Huang. A. Vahdat, J. Song. K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint 2rX11:2211.1332.. 2022.
[4] O. Bar-Tal, L. Yariv, Y. Lipman, and T. Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08//3 2, 2023.
[5] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392- 18402, 2023.
[6] M. Cha. Y. Gwon, and H. Kung. Adversarial nets with perceptual losses for text-to-image synthesis. In 2017 IEEE 27th international workshop on machine learning for signal processing (MLSP), pages 1-6. TEEE, 2017.
[7] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. 7eng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments, arXiv preprint arXiv:1709.0611 2017.
[8] H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538-12547, 2019.
[9] Chen, C. Gao, E. Meng, Q Zhang, and S. Liu. Reinforced structured statc-evolution for vision-languaa navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
pages
[10] S. Chen, P.-.. Guhur, C. Schmid, and I. Laptev. History aware multimodal transformer for vision-and- language navigation. Advances in neural information processing systems, 34:5834-5847 2021.
[11] S. Chen, P.-L.. Guhur, M. Tapaswi, C. Schmid, and Laptev. Learning from unlabeled 3d environments for Dision-and-languagg navigation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXIX, pages 638 655. Springer, 2022.
[12] S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev. Think global, act local: Dual-scale graph transformer for vision-and and language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16537-16547, 2022.
[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Hcigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2rXhv:2010.11999 2020.
[14] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Kirk-aiika,,r,ck, K. Saenko, D. Klein, and T. Darrell, follower models for vision-and-languaee navigation. Advances in Neural Information Processing Systems, 31, 2018.
[15] C. Gao, X. Peng, M. Yan, H. Wang, I.. Yang, H. Ren, H. Li, and S. Liu. Adaptive zone-aware hierarchical planner for vision-language navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14911-14 1-110,, 2023.
[16] 1. Goodfellow, J. Pouget-Abadics M. Mirza, B. Xu. D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications the ACM, 63(11):139 144, 2020.
11

---

[17] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. Advancex in neural information processing systems, 30, 2017.
[18] W. Hao, Li, X. Li, L.. Carin, and J. Gao. Towards learning generic agent for vision-and-languuae language navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137-13146, 2020.
[19] Y. Hong, C. Rodriguez-Opazo. Q. Wu, and Gould. Sub-instruction aware vision-and-language navigation. arXiv preprint arXiv:2004.022 027007 2020.
[20] Y. Hong. Q. Wu, Y. Qi. C. Rodriguez-Opazo, and S. Gould. A recurrent vision-and-language bert for navigation. arXiv preprint urXiv:2011.13222, 2020.
[21] V. Jain, G. Magalhaes, A. Ku, A. Vaswani, E. Ie, and J. Baldridge. Stay on the path: Instruction fidelity in vision-and-languaae navigation. arXiv preprint /905. 2019.
[22] M. Kang. J.-Y. Zhu, Zhang, Park, Shechtman, S. Paris, and Park. Scaling up gans for text-to- image synthesis, arXiv preprint urXiv:2303.0551, 2023.
[23] H. Kim, J. Li, and M. Bansal. Ndh-full: Learning and evaluating navigational agents full-length dialogue. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processngg 2021.
[24] J. Y. Koh, H. Lee, Yang. J. Baldridge, and P. Anderson. Pathdreamer: A world model for indoor navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14738- 14748, 2021.
[25] J. Y. Koh, H. Agrawal, D. Batra, R. Tucker, A. Waters, H. Lee, Y. Yang, J. Baldridge, and P. Anderson. Simple and effective synthesis of indoor 3d scenes. arXiv preprint wrXiv:2204.0296 2022.
[26] A. Ku, P. Anderson, R. Patel, F. Ie, and J. Baldridge. Room-across-room; Multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:20100009544 2020.
(27] C. Li, H. Xu, Tian, W. Wang, M. Yan, B. Bi, J. Ye, H. Chen, G. Xu, Z. Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint 12205 2022.
[28] J. Li and M. Bansal. Improving vision-and- languuge navigation by generating future-view image semanties. arXiv preprint arXiv:2304.04900 200.400997,,222.
[29] J. Li, H. Tan, and M. Bansal. Improving cross-modal alignment in vision language navigation via syntactic information. arXiv preprint arXiv:2/04.09590 2021.
[30] J. Li, H. Tan, and M. Bansal. Clear: Improving vision-language navigation with cross- lingulin environment- agnostic representations. arXiv preprint arXiv:2207.00 20..02207.0288, 2022.
[31] J. Li, H. Tan, and M. Bansal. Envedit: Environment editing for vision-and and looon-and-language -anguaa navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15407- 15417, 2022.
[32] Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint wtt.::::3311..999. 2023.
[33] X. Li, C. Li, Q. Xia, Y. Bisk, A. Celikyilmaz, Gao, N. Smith, and Y. Choi. Robust navigation with language pretraining and stochastic sampling. arXiv preprint 19X9v:1999.02244, 2019.
[34] C. Liu. F. Zhu, X. Chang, X. Liang. Z. Ge, and Y.DD. Shen. Vision-language navigation with random environmental mixup. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1644-1654, 2021.
[35] C.-Y. Ma. J. Lu, Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong. Self- monitoring navigation agent via auxiliary progress estimation. arXiv preprint arXiv: 03035, 2019.
[36] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra. Improving vision-and- language navigation with image-text pairs from the web. In Computer Rision-ECVV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part VI 16, pages 259-2725 259-274. Springer, 2020.
[37] L. Mescheder. On the convergence properties of gan training, arXiv preprint arXiv:1801.0444 1:16. 2018.
12

---

[38] L. Metz. B. Poole, D. Pfau, and J. Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint 1.X1v.161..0213,, 2016.
[39] K. Nguyen and H. Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraginn imitation learning. arXiv preprint (rXiv:/999.011 2019.
[40] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew. I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741, 2021.
[41] A. Padmakumar, Thomason, A. Shrivastava, Lange, A. Narayan- Chean, Gella, R. Piramuthu, G. Tur. and D. Hakkami- Tur. Teach: Task-drived embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2017-2025, 2022.
[42] T. Park, M.-Y. Liu, T.-.. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially-adaptive normal- ization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2337-2346,, 2019.
[43] Y. Qi, Q. Wu, P. Anderson, X. Wang, W. Y. Wang, C. Shen, and A. v. d. Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9982-9991, 2020.
[44] Y. Qiao. Y. Qi. Y. Hong, Z. Yu, P. Wang, and Q Wu. Hop: history-and ard-rrder aware pre-training for laiio-aad-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15418-15427, 2022.
[45] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748 8763. PMLR, 2021.
[46] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. Habitat-matterpon 3d dataset (HM3dd) 1000 large-scale 3d environments for embodied AL. Thirty-fiith Conference on Neurul Information Processing Systems Dutasets and Benchmarks Track (Round 2), 2021. URL https:// openreview
[47] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10699 June 2022.
[48] C. Saharia, W. Chan, S. Saxena, L. Li. J. Whang. E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
[49] M. Shridhar, J. Thomason. D. Gordon, Y. Bisk, Han. R. Mottaghi, Zeullmoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Pruceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[50] I.. Song, L. Cao, H. Xu, K. Kang, F. Tang, J. Yuan, and Y. Zhao. Roomdreamer: Text-driven 3d indoor scene synthesis with coherent geometry and texture. arXiv preprint arXiv:2305.1133/ 2023.
[51] H. Tan, L. Yu, and M. Bansal. Learning to navigate unseen environments: Back translation with environ-
mental dropout. arXiv preprint 1904:::/044.4410 2019.
[52] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Dision-and-diale navigation. In Conference Robot Learning, pages 394 406. PMLR, 2020.
[53] H. Wang, W. Wang, W. Liang, C. Xiong, and J. Shen. Structured scene memory for vision-language navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 8455-8464, 2021.
[54] S. Wang, C. Montgomery, J. Orbay, V. Birodkar, A. Faust, I. Gur, N. Jaques, A. Waters, J. Baldridge, and P. Anderson. Less is more: Generating grounded navigation instructions from landmarks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15428-15438 2022.
[55] X. Wang. Q Huang. A. Celikyilmaz, J. Gao, D. Shen, Y.-F Wang. W. Y. Wang. and L. Zhang. Reinforced cross modal matching and self supervied imitation learning for dision-language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6629-6638, 2019.
13

---

[56] X. Wang, W. Wang. J. Shao, and Y. Yang. Lana: A language-capable navigator for instruction following and generation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 19048-19058, 2023.
[57] X. E. Wang. V. Jain, E. le. W. Y. Wang, Z. Kozareva, and S. Ravi Enviromment--ggnotte multitask learning natural language grounded navigation. In Computer Fision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XXIV 16, pages 413 Springer, 2020.
[58] L. Xue, N. Constant, A. Roberts, Kale, Al-Rfou, A. Siddhant, Barua, and C. Raffel. mt5; massively multilingual pre pre-trinee text-to-text transformer. arXiv preprint arXiv:2010.11933 2020.
[59] Z. Yang, Dong, P. Liu, Yang, and Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10561-10570, 2019.
[60] Zhang. T. Xu,
Zhang, Wang, X. Huang. and D. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. Proceedings of the IEEE international conference computer vision, pages 5907 -9155 2017.
[61] H. Zhang, T. Xu, H. Li, S. Zhang. X. Wang. X. Huang, and D. N. Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence, (188):1447 1962, 2018.
[62] L. Zhang, Q. Chen, B. Hu, and S. Jiang. Text-guided neural image inpainting. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1302-1310, 2020.
[63] Y. Zhao, J. Chen, C. Gao, W. Wang, L. Yang. H. Ren, H. Xia, and S. Liu. Target-driven structured transformer planner for vision-language navigation. In Proceedings the 30th ACM International Conference on Multimedia, pages 4203, 2022.
14

---

Appendix
In this supplementary, we provide the following:
Detailed description the datasets we use in Sec. A, and more implementation details in Sec. B.
Ablation performance of our PANOGEN REVERIE dataset in Sec. C.
Performance of improving panoramic view consistency across different navigation steps in Sec. D.
More examples of the panoramic environments generated by our PANOGEN in Sec. E, and alignment with speaker data in Sec. F.
Limitations and broader impacts in Sec. G, and licenses in Sec. H.
A
Datasets
We evaluate our agent on three datasets: Room-to- Ro-Room dataset (R2R) [2]. Cooperative Vision-and- -Dialog Navigation dataset (CVDN) [52], and Room-for- Room dataset (R4R) [21].
R2R. Room-to-Room Ro-m datasct contains detailed instructions to guide the agents navigate toward the target location step by step. The ground truth paths are the shortest path between the start location and the end location. The training contains different room environments, while the unseon validation and contain 11, and 18 room environments that are unseen during training.
R4R. Room-for-ER for--oooo-oo dataset created by concatenating adjacent paths in the Room-to-Room dataset. this case, the ground truth path is not the shortest path. This encourages the agent to follow the instructions to reach the target instead of exploring the environment bias and reach the target by directly navigating the shortest path.
CVDN. Cooperative fision-and Dialog Navigation dataset contains interactive dialogue instructions. The dialogue usually contains ander-specifies instructions, and the agent needs navigate based on both dialogue histories and the commonsense knowledge of the room. The room environments in the training set, unseen validation set, and test set follow the split in Room-to-Room dataset.
B Implementation Details
panoramic environment generation, we caption the view images the training environments in R2R dataset with BL.I--2-FlanT T5.xxI. We utilize stable-diffusion-v.. base model generate single view based on caption only, and use stable-diffusion-y ittedddffuuooonol.l--orraaaiiiii model to outpaint the unseen observation for the rotated views. It takes 2 days 6 A 100ss to generate all the environments.
In speaker data generation, build our speaker model hased mPLUU-hase, which has 350M parameters and utilizes VIT/B-16 16 the visual backbone. We train the speaker for epochs two days.
A6000 GPU with batch size
navigation training, we adopt the agent architecture from DUET [12]. follow the training hyperparameters in DUET. Different from DUET, we utilize one A6000 GPU. We pre-train the agent with batch size 64 for 150k iterations, and then fine-tune the agent with batch size for 40k iterations. Both the pre-training and fine-tuning take approximately one day finish. We report reproduced baseline performance with CLIP-ViT/B. features for fair comparison. The best model selected based on performance on validation unseen set.
to extract the visual features. We train the model on
C Ablation Performance on REVERIE Dataset
We demonstrate the effectiveness of approach REVERIE dataset in Table We observe that pre- training with our speaker data improves the baseline by 2.64% in success rate, while fine-tuning with ob- servation replacement from PanoGen environments achieves 4.60% absolute improvement in success rate. This demonstrates that our approach generalizes well to navigation tasks that have inder-specified instruc- tions.
Table 8: Ablation performance on REVERIE un- seen set.
Model
Validation Unseen
SR 1
SPI.1
RGS
RGSPI f
DUETCLIP
46.58
34.14
22.89 32.80
31.70
PanoGen-mPPLCG
Panooen++Rollaee
51.18
34,99
33.26
22.99
D Improving Consistency Across Steps in Panorama Environment
15

---

Metterport 3D
Paruden
Stabin Diftusionn Bitcretaed VitcretaeddVeee Visaa.
Figure 4: Qualitative analysis of the panoramic environments generated with our PANOGEN. "Mat- terport3D" is the original environment for VLN tasks. "Stable Diffusion for Discretized Views" is the concatenation separately generated discretized views given text captions.
our PANOGEN environments, we didn't i explicitly constrain the consistency between views at different navigation steps. In this section, show initial exploration of improving the view consistency across steps. Specifically, we fine-tune instructpix2pix [S] based on aub-instruction annotations [19] to gener- ate the next views given previous step in the trajec- tory. After generating consistent discrete views with instructpix2pix at different viewpoints, we then paint the view with our approach to create panoramic environments with better consistency across naviga- tion steps. As shown in Table 9, we observe that improving consistency across navigation steps slightly helps with the navigation performance the R2R unseen set, improving the SPL by 0.21%.
Table 9: Performance of improving consistency across navigation steps in our panoramic environ-
ments on Room-to-Room validation unseen set.
Model
Validation Unseen
TL
NE
SR
SPL f
DUET-CLIP
12.92
3.19
72.8
63.4
PanoGen
13.76
2.99
74.41
63.88
PanoGen+Con
13.91
3,00
74.46
64.09
E
Qualitative Example for PanoGen Environment
We show more panoramic environments generated with our PANOGEN in Figure 4. We observe that directly concatenating discretized views generated separately will generate inconsistent panoramas (Row "Stable Diffu- sion for Discretized Views"). In comparison, our PANOGEN can generate continuous views with reasonable layout and object co-occurrence (Row "PANoGEN"). Moreover, our approach can generate panorama for both indoor and outdoor environments. Though generating the outdoor environments might not benefit agents' indoor navigation ability directly, our approach demonstrates its potential to be applied to panorama generation with different content (e.g., landscape).
F Qualitative Example for Alignment between Speaker Data and PanoGen Environment
We include one panorama-instruction example Figure to demonstrate the alignment between the instruction and the environment. We show that our panorama environments are more diverse in appearance, and the instruction data generated by our mPLUG based speaker contains more details. Besides, the general semantic information across different steps is still reasonable in consistency.
G Limitations and Broader Impacts
Vision-and-Language LanguLge Navigation tasks can be used in many real-world applications, for example, a home service robot can bring things to the owner based on natural language instructions. In this paper, our proposed method generates panoramic environments for VLN training, and significantly improves navigation agents' generalization ability to unseen environments given limited human-annotated training data. Our approach reduces the efforts of re-training the agents in every new environment when adapting to real-world scenarios.
We also note that there are some limitations of our work. First, this work directly utilizes stable diffusion models trained for inpainting on "laion-aesthetic aesthetics v2 5+", Though the zero-shot generation performance is good, further improvement might be observed if further trained room images. Second, we investigate specific
16

---

Original Environment
PanoGen Environment
Human: Walk past the TV and continue toward the bathroom. Stop before walking through the bathroom door.
EnvDrop:
Turn around and walk to the right side of the room. wait there.
walk around bathroom. stop in front of the sink.
Ours:
and
Figure 5: Qualitative analysis of panorams-trreccooryy-nsntttooo pairs.
task Vision-and Lad-Language Navigation in this paper. but the proposed method can be potentially used in other embodied tasks like concept learning and grounding in panoramic environments. We will explore other useful and interesting tasks in the future.
H
Licenses
We provide the licenses of the existing assets we in this paper in Table 10.
Table 10: A list of the licenses of the existing assets used in this paper.
Asset
License
Pytorch
BSD-style
Huggingface Transformers
Apache License 2.0 BSD 3-Clause "New" or "Revised" License
Torchvision
Room-to-Room
MIT
Room-for-Be Shom-for-Room
Apache License 2.0 MIT
Cooperative
Dision-and-Dialog Navigation
BLIP-2
BSD 3-Clause "New" or "Revised" License Apache License 2.0
mPLUG
DUET
N/A
Stable Diffusion
CreativeML Open RAIL-M
17


Reviews:
[{"review_id": "uxbJmyylQO", "reviewer": "Reviewer_VRWy", "summary": "This paper proposes a creative system-level solution for VL-navigation agent training, by incorporating the recent T2I model to help increase the data diversity while follow the context and human intuition, which facilitate effective pre-training for performing domain-specific tasks. The proposed pipeline with image capturing and recursive in-painting can potentially create an infinite number of diverse panoramic environments conditioned on text. The author has further proposed two ways to demonstrate the usage of this generated panoramic data, by replacing existing images or generating new trajectory-instruction pairs. They have well demonstrated its effectiveness on existing VL-Nav benchmarks, with better generalization to unseen environments.", "questions": "When generating realistic trajectory from the panorama images, will any depth information (like from pre-trained depth) be used as well?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed method of leveraging existing T2I models for VL-Nav tasks is creative and provides a cost-effective way to generate rich data for training embodied agents.\n\nIn addition to the effective method of generating panoramas for indoor environments, the author also provides a complete pipeline for using the data and evaluates their idea on well-known public benchmarks. This required considerable effort, but will inspire follow-up work in the community to leverage pre-trained generative models to create diverse training data for downstream tasks.\n\nThe proposed framework has achieved exciting results in improving existing methods in instructed navigation tasks, such as improving goal progress by 1.59 meters on the CVDN test set.\n\nThe ablation experiments with different ways and ratios of using the generated panorama data are also commendable, as they help to achieve a better system design.", "weaknesses": "- It seems that the comparison to previous practices is missing. There have been multiple different practices used to avoid overfitting and increase data diversity, such as manually doing domain randomization, doing re-lighting to existing environments, or doing joint training over multiple different datasets for the same task. If possible, it would be helpful to know how this proposed framework compares to existing practices.\n\n- Another concern is the quality of the synthetic data. While nearby view in-painting might make sense, the final generated panorama can sometimes be counterintuitive and lack loop closure characteristics. The generated trajectory-instruction pair can also sometimes make no sense, and in the replaced view case, the image may differ from the original one a lot regarding the layout. It would be helpful to know if there is a way to measure the quality of the generated data besides the final evaluation in the target tasks, since some generated data might be harmful to the tasks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer VRWy**\n\n> Q1: Comparison with previous practices which avoid overfitting and increase data diversity. \n\nFirst, our approach is compatible with the existing instruction augmentation approach PREVALENT[1]. As we described in L229-L231, we follow our baseline approach DUET and pre-train the VLN agent on both R2R data and PREVALENT data, which contains synthetic instructions for unannotated paths in the seen environments. \n\nBesides, we further compare our approach with two existing approaches that augment the environments to avoid overfitting: EnvEdit[2] and EnvDrop[3]. For adapting EnvEdit to DUET, we follow the batch mixing approach in EnvEdit and randomly replace half of the data in a batch with the edited environments which change the appearance of the objects during VLN finetuning. For adapting EnvDrop to DUET, we replace the regular dropout layer in DUET with the proposed environment-level dropout layer during fine-tuning.  As the results shown below, training with our PanoGen environments achieves better performance than previous approaches. \n\n| Model   | TL    | NE   | SR    | SPL   |\n|---------|-------|------|-------|-------|\n| EnvEdit | 13.61 | 3.03 | 72.80 | 63.17 |\n| EnvDrop | 13.28 | 3.12 | 72.58 | 62.40 |\n| PanoGen | 13.40 | **3.03** | **74.2**  | **64.3**  |\n\n\n> Q2: Measurement of quality of generated data. \nWe measure the quality of our generated data from two aspects. \n* **Similarity between instructions and environments.** First, we measure the similarity between the generated instructions and the PanoGen environments. We hypothesize that higher similarity between instruction and trajectory pairs in embedding space can indicate better alignment between instruction and trajectory. Specifically, we represent the trajectory representation by averaging the image embeddings of the viewpoints in the trajectory. The image embedding of each viewpoint is encoded with CLIP-ViT/16. We also encode the instruction with CLIP text encoder. We calculate the cosine similarity between the instruction representation and the trajectory representation. As shown in the Table below, we find that the similarity between our PanoGen environments and instructions generated with mPLUG is higher than instructions generated with baseline EnvDrop (No.2 vs No. 3). Besides, we calculate the similarity between randomly replacing 30% of the viewpoints with PanoGen environments and the original instructions (to mimic the observation replacement fine-tuning). We average the score over 5 runs to mitigate randomness. We find that randomly replacing the observation doesn’t lead to decrease in similarity (No. 4 vs No. 1). \n\n| No. | Instruction | Environment               | Similarity           |\n|-----|-------------|---------------------------|----------------------|\n| 1   | Original    | Original                  | 0.2845               |\n| 2   | EnvDrop     | PanoGen                   | 0.2669               |\n| 3   | mPLUG       | PanoGen                   | 0.2714               |\n| 4   | Original    | 30% PanoGen, 70% Original | 0.2893 ($\\pm$0.0001) |\n\n* **Automatic BERTScore evaluation over instructions.** Second, we calculate the BERTScore of instructions generated by our mPLUG based speaker and the EnvDrop speaker. Specifically, we use both speakers to generate the instructions on the validation unseen set of R2R data. We use Bart-base as the base model to calculate the BERTScore. Our speaker achieves a BERTScore of 71.8, while the EnvDrop speaker achieves a BERTScore of 70.5. \n\n**Qualitative example.** Beside the above automatic evaluation results, we also include one panorama-instruction example in the general response pdf for qualitative analysis. Our PanoGen environments contain similar semantics as the original environments, while being much more diverse. The instruction generated by our mPLUG based speaker also contains more detailed instructions. \n\n**Loop closure characteristics.** Lastly, we are working on improving the loop closure characteristics of our panoramas. Specifically, we generate the last view by conditioning on both its nearby views and inpaint the middle observation. Due to time limitations, we cannot share the results on the VLN task yet, but we will include the results in the final version (and will also try to report it before the rebuttal discussion periods end), as an investigation for how much loop closure characteristics impact the navigation performance. \n\n> Q3: Utilization of depth information. \n\nDepth information is not used in our trajectory generation. We believe utilizing depth information for generating environments that are more consistent in 3D space can be interesting future work. \n\n\n[1] Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training. In CVPR 2020. \n\n[2] EnvEdit: Environment Editing for Vision-and-Language Navigation. In CVPR 2022. \n\n[3] Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout. In NAACL 2019. \n"}, {"review_id": "mn9hOOP8GI", "reviewer": "Reviewer_tReg", "summary": "In this paper, the authors propose to leverage the generative model to create panoramic images for agent training. A recursive inpainting method is adopted to generate 360-degree panorama views, which aims to ensure the co-occurrence of objects and enough diversity. Experiments are conducted on R2R, R4R, and CVDN datasets, confirming the effectiveness of the proposed method. ", "questions": "- Why were the performance results of DUET-CLIP on the test set not provided?", "limitations": "NA", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed pipeline is reasonable.\n\n- The performance improvement is promising when using PANOGEN. \n\n- The paper is well-written and easy to follow.", "weaknesses": "- My main concern is the lack of novelty. In my view, the authors simply leverage and modify the existing text2img model to conduct data augmentation for VLN agents, and there is no specific design for the VLN method. Thus I believe this work does not provide enough technical contribution to the VLN community.\n\n- As shown in Table 2 and 3, compared to DUET-CLIP on R2R, the proposed method does not show performance improvement on the SPL. \n\n- More qualitative results of generated panoramic environments need to be provided for better examining the spatiotemporal consistency of panoramas.\n\n- Lacking experiments on datasets with high-level instruction such as REVERIE.\n\n- Missing some references such as [1-4].\n\n[1] HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation. In TPAMI.\n\n[2] Adaptive Zone-Aware Hierarchical Planner for Vision-Language Navigation. In CVPR 2023.\n\n[3] LANA: A Language-Capable Navigator for Instruction Following and Generation. In CVPR 2023.\n\n[4] Reinforced Structured State-Evolution for Vision-Language Navigation. In CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer tReg**\n\n> Q1: Novelty. \n\nTo adapt the text2image model for VLN, our main technical contributions include: \n* We propose a novel inpainting way to generate consistent panorama views for VLN instead of single image views (Sec. 3.2). \n* We propose a multi-modal transformer-based model to generate instructions for our PanoGen environments (Sec. 4.3), which shows superior performance than the baseline speaker when evaluated on downstream tasks, and higher similarity with the PanoGen environments. \n* We propose two ways of effectively incorporating PanoGen environments in VLN training -- (1) utilizing the generated instruction-trajectory pairs in pre-training (Sec. 4.3) (2) replacing observations with augment environments during fine-tuning (Sec. 4.4).  \n\nWe believe all these aspects are novel and have specific designs suited for VLN tasks, and hence achieve SotA on multiple VLN tasks (i.e., R2R, CVDN, REVERIE). Besides, our approach can be potentially generalized to other text-guided embodied tasks in panoramic environments, and impact a larger research field besides VLN. \n\n> Q2: Performance compared with DUET-CLIP on R2R. \n\nThough having slightly lower SPL on R2R, it achieves significant improvement on CVDN (0.43m absolute improvement in GP, which is a relative 7% improvement in GP). Besides, it also achieves better performance on REVERIE as we showed in Q4. \n\n> Q3: Qualitative results of PanoGen environments. \n\nWe provide two more qualitative examples of our PanoGen environments in Appendix Figure 1. Besides, we provide one more panorama trajectory-instruction pair example in the general response pdf for qualitative analysis. Our PanoGen environments contain similar semantics as the original environments, while being much more diverse. The instruction generated by our mPLUG based speaker also contains more detailed instructions. \n\n> Q4: REVERIE performance. \n\nWe show our approach’s performance on REVERIE in the following Table. Both pre-training with our speaker data and fine-tuning with observation replacement achieve a large performance improvement (4.6% absolute improvement in SR) than the CLIP baseline on the REVERIE dataset. \n\n| Model           | SR    | SPL   | RGS   | RGSPL |\n|-----------------|-------|-------|-------|-------|\n| DUET            | 46.98 | 33.73 | 32.15 | **23.03** |\n| DUET-CLIP       | 46.58 | 34.14 | 31.70 | 22.89 |\n| PanoGen+mPLUG   | 49.22 | 33.44 | 32.80 | 22.45 |\n| PanoGen+Replace | **51.18** | **34.99** | **33.26** | 22.99 |\n\n> Q5: Missing references. \n\nThanks so much for pointing out these references. We will definitely add and discuss them in the final version. \n\n> Q6: DUET-CLIP test leaderboard performance. \n\nThe test leaderboard only has limited submission times, and the community usually uses it to submit only the final models instead of intermediate baselines (to avoid excessive utilization of the test set and maintain its blind nature). The validation unseen set also contains unseen environments during training to test the generalization ability of our proposed agents, which our agent shows better performance than the baseline approach. \n"}, {"review_id": "GZJmM1Faep", "reviewer": "Reviewer_Wxjf", "summary": "This paper presents a new data augmentation method for VLN tasks. The proposed method first generates captions for each view and then recursively generates new images to ensure the consistency among multi-views. The authors demonstrate two ways to utilize the newly generated panorama on three benchmarks: R2R, CVDN, R4R. Better results are achieved compared to the previous SoTA.", "questions": "See details in weaknesses.", "limitations": "See details in weaknesses.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow, with clearly stated objectives and technical descriptions.\n2. Strong performances are achieved on three different VLN tasks.\n3. Comprehensive and insightful ablation study is provided.\n4. Quantitative and qualitative results show the importance of multi-view consistency to the VLN tasks.", "weaknesses": "1. Will the generated panorama in two consecutive steps be largely different? If yes, will it be better if reduce the difference across steps?\n2. What are the impacts on high-level VLN tasks, such as REVERIE?\n3. It could be better to make comparisons to previous data augmentation methods, such as EnvDrop and PREVALENT.\n\n====== after rebuttal ============\nThanks for the authors' responses. All my concerns have been well addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer Wxjf**\n\n> Q1: Consistency between panorama of two steps. \n\nWe show one qualitative analysis in the general response pdf. We could observe that though we didn’t explicitly improve consistency across steps, the general semantic information is still reasonable across steps. For the given example, we could always observe that the navigation happens in a bedroom with a sofa and an outside view of the sea. \n\n> Q2:  REVERIE performance. \n\nWe show our approach’s performance on REVERIE in the following Table. Both pre-training with our speaker data and fine-tuning with observation replacement achieve a large performance improvement (4.6% absolute improvement in SR) than the CLIP baseline on the REVERIE dataset. \n\n| Model           | SR    | SPL   | RGS   | RGSPL |\n|-----------------|-------|-------|-------|-------|\n| DUET            | 46.98 | 33.73 | 32.15 | **23.03** |\n| DUET-CLIP       | 46.58 | 34.14 | 31.70 | 22.89 |\n| PanoGen+mPLUG   | 49.22 | 33.44 | 32.80 | 22.45 |\n| PanoGen+Replace | **51.18** | **34.99** | **33.26** | 22.99 |\n\n> Q3: Comparison with EnvDrop and Prevalent. \n\n* **Comparison with PREVALENT**. Our approach is compatible with the existing instruction augmentation approach PREVALENT[1]. As we described in L229-L231, we follow our baseline approach DUET and pre-train the VLN agent on both R2R data and PREVALENT data, which contains synthetic instructions for unannotated paths in the seen environments. \n* **Comparison with EnvDrop**. We adapt EnvDrop to DUET by replacing the regular dropout layer in the image encoder with environment-level dropout. Besides, we also compare with another environment-level data augmentation approach EnvEdit [1], where we follow the batch mixing approach in EnvEdit and randomly replace half of the data in a batch with the edited environments which change the appearance of the objects during VLN finetuning. The results shown in the Table below demonstrate the effectiveness of training with our PanoGen environments. \n\n| Model   | TL    | NE   | SR    | SPL   |\n|---------|-------|------|-------|-------|\n| EnvEdit | 13.61 | 3.03 | 72.80 | 63.17 |\n| EnvDrop | 13.28 | 3.12 | 72.58 | 62.40 |\n| PanoGen | 13.40 | **3.03** | **74.2**  | **64.3**  |\n\n[1] EnvEdit: Environment Editing for Vision-and-Language Navigation. In CVPR 2022. \n"}, {"review_id": "Z1S1ogKeIt", "reviewer": "Reviewer_wyXn", "summary": "This paper proposed a data augmentation method named PANOGEN, which generates panoramic environments. The proposed method employs a recursive image inpainting technique to generate coherent panoramic environments and incorporates these augmented environments in both the pre-training and fine-tuning stages. Experimental evaluations demonstrate the effectiveness of the proposed method in enhancing performance in VLN tasks.", "questions": "It would be better to indicate in Table that DUET-CLIP is the baseline to make it easier to make comparisons.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. PANOGEN generates new environments without human annotation, which is novel and solves the problem of environment scarcity. \n2. The paper is well-written and easy to follow.\n3. The experiment is sufficient and ablation studies are strict.\n", "weaknesses": "1. It seems that it lacks a discussion about the selection of the image caption model since the caption is a critical section. And I am wondering how it would be to directly generate the panorama view from the instruction.\n2. More discussion on the effectiveness of panorama replacement in fine-tuning is needed. I'm still confused about why using the generated environment is better than the original one because of the larger data.\n3. It would be better if the visualization results were made into a panorama-instruction format so that readers can compare easily, and there are no failure cases. \n4. I could not find any examples of generated instructions or a measure of the quality of the generation, thus it is hard to evaluate the performance of the Speaker.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer wyXn**\n\n> Q1: Discussion of selection of image captioning model, and clarification for generating panorama from text. \n\n**Image captioning model choice.** We choose BLIP-2 for image captioning since it has sota/good zero-shot performance on multiple image captioning benchmarks (Table1 in [BLIP-2 paper](https://arxiv.org/pdf/2301.12597.pdf)). Besides, we compare the captions generated by OFA[1] and BLIP-2, and manually go through 50 caption outputs on the R2R seen environments to decide to use BLIP-2. \n\n**Panorama generation from text clarification.** To generate panoramas from text captions, we propose a novel image inpainting approach to gradually generate the panorama from single images (L148-L171). Specifically, each panorama is discretized into 36 views, and we generate captions for each view. Then, we choose one view in the middle elevation as the starting point, and generate the image given the caption. We gradually rotate the image and inpaint the unseen observation recursively to generate the full panorama from multiple text captions. \n\n> Q2: Effectiveness of panorama replacement in fine-tuning.\n\nDue to the limited available training environments for VLN, the previous paper shows that the agent tends to overfit the low-level appearance features of the environment [2]. Thus, training by randomly replacing the observation with PanoGen environments brings more diverse environments during training, and can avoid overfitting and improve generalization (L32-L51). \n\n> Q3: Qualitative example of panorama-instruction pairs. \n\nWe include one qualitative example of panorama-instruction pair in the general response pdf. Our PanoGen environments contain similar semantics as the original environments, while being much more diverse. \n\n> Q4: Evaluation of speaker. \n\n**Automatic BERTScore evaluation over instructions.** We calculate the BERTScore of instructions generated by our mPLUG based speaker and the EnvDrop speaker. Specifically, we use both speakers to generate the instructions on the validation unseen set of R2R data. We use Bart-base as the base model to calculate the bert score. Our speaker achieves a BERTScore of 71.8, while EnvDrop speaker achieves a BERTScore of 70.5. \n\n**Qualitative example.** Besides the evaluation with BERTScore, we also include the instructions generated with EnvDrop speaker and our speaker in the general response pdf. We find that our speaker tends to generate instructions with more details, while the EnvDrop instructions only mention “walk to the right side of the room”. \n\n> Q5: Emphasize DUET-CLIP baseline. \n\nThanks for pointing out. We will add “DUET-CLIP is considered as our baseline approach.” in the Table captions. \n\n[1] OFA: Unifying Architectures, Tasks, and Modalities through a Simple Sequence-toSequence Learning Framework. In ICML 2022. \n\n[2] Diagnosing the Environment Bias in Vision-and-Language Navigation. In IJCAI 2020. \n"}, {"review_id": "lstVrPw1nZ", "reviewer": "Reviewer_PpHD", "summary": "This paper proposes a nove approach for creating an infinite number of diverse panoramic environments conditioned on text for visual-language-navigation (VLN). Specifically, the authors use stable diffusion with captions of images from a existing dataset to generate in-door panoramic views. Recursive inpainting is used to ensure the consistency across views. The authors also present pretraining and finetuning strategies that are designed to maximize the benefits of the synthesized data, and they demonstrate its effectiveness across various benchmarks.", "questions": "1. In this paper, panoramic generation is conditioned on captions from existing data. However, it is also possible to create panoramic views in an unconditioned manner. Then mPLUG can also create instructions over this data. Will such data help VLN training? \n\n2. Is it necessary to have novel layouts with instructions, as I mentioned in Weakness (2)? Can we just use ControlNet/GLIGEN?", "limitations": "The authors have discussed about the limitations and potential impact in the supp.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper demonstates that strong generative models (like Stable Diffusion) are potential to augment traning data for robotics learning (VLN in this paper). Similar ideas could be adopted in other areas like autonomous driving.\n\n2. The experiments over different benchmarks show that incorporating virtual panoramic scenes generated by Stable Diffusion is useful.", "weaknesses": "1. There are no examples for mPLUG captioning. It would be great to contain several samples of {previous_img, previous_caption, generated_img, generated_caption}. With such kind of samples can reader better understand how mPLUG creates suitable captions for generated images with different layouts and appearances.\n\n2. This method creates novel data with different appearance, scene layouts, and instructions compared to original data. And experiments show that incorporating these three factors together is helpful. However, It is not very clear which specific part or all parts have played a helpful role. If novel layouts and instructions do not help, I believe incorporating ControlNet/GLIGEN can provide novel appearances with the same arrangements, no need for finetuning mPLUG and captioning, making the overall method simpler and easier to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer PpHD**\n\n> Q1: Qualitative examples of generated data. \n\nWe include one panorama-trajectory-instruction pair in the general response pdf. As shown in the Figure, our PanoGen environment is more diverse in appearance, but still maintains good alignment in semantics with the original environments. Besides, we find that the instructions generated with our mPLUG based speaker are better in detail compared with the baseline EnvDrop speaker, which clearly indicates that the agent should stop in front of the sink inside the bathroom. \n\n> Q2: Ablations on instructions and layouts. \n\n* To demonstrate the effectiveness of generating new instructions for PanoGen environments, we experiment with pre-training with PanoGen environments and the original instructions. As shown in the Table below, without the instructions generated for the PanoGen environments, we observe 0.8% drop in SR, and 2.2% drop in SPL. \n\n* To demonstrate the effectiveness of layouts, we compare our approach with EnvEdit[1], which only edits the appearance of the original environments by synthesizing based on semantic segmentation (similar idea as using GLIGEN). As shown in the Table below, training with PanoGen environments further improves training with EnvEdit environments by 1.4\\% in SR and 1.1\\% in SPL. \n\n| Model          | SR   | SPL  |\n|----------------|------|------|\n| PanoGen        | 74.2 | 64.3 |\n| - instructions | 73.4 | 62.1 |\n| - layouts      | 72.8 | 63.2 |\n\n> Q3: Unconditional panorama generation. \n\nDue to time limitations, we cannot train a model to generate a large number of new panoramas unconditionally. Instead, we experiment with randomly switching existing panoramas in PanoGen environments to mimic the unconditional panorama generation process. Then, we use our mPLUG based speaker to generate instructions for these environments. As shown in the Table below, training with this data will decrease the performance by 1.2\\% in SR and 1.4\\% in SPL. We attribute the performance decrease to the large inconsistency between consecutive viewpoints when using unconditional generation. Conditioning the generation of text captions can implicitly maintain some consistency between consecutive viewpoints. \n\n| Model            | SR   | SPL  |\n|------------------|------|------|\n| PanoGen          | 74.2 | 64.3 |\n| Random switching | 73.0 | 62.9 |\n\n\n[1] EnvEdit: Environment Editing for Vision-and-Language Navigation. In CVPR 2022. \n"}]

Author rebuttal:
**General Response**

We thank all the reviewers for their thoughtful feedback. We are glad that they find our work novel and creative (Reviewer vRWy, wyXn), and provides a cost-effective way to tackle the data scarcity problem for VLN and potentially more general robotics learning (Reviewer vRWy, wyXn, PpHD). We thank them for recognizing the promising improvement brought by our approach (Reviewer tReg, Wxjf, VrWy), and acknowledging that our ablation studies as comprehensive and insightful (Reviewer Wxjf, wyXn). We thank them for thinking our paper as well-written and easy to follow (Reviewer tReg, Wxjf, wyXn). We address their questions below and will include them in the final version. 

Meta-review: The paper proposes a new data generation method for VLN. The initial concerns for the paper mainly include missing comparison with previous data augmentation methods, lacking evaluations on additional datasets, no qualitative examples and marginal performance gain. The rebuttal addressed most of these concerns. All reviewers except reviewer tReg voted to accept the paper. The reviewer tReg did not respond to the authors’ rebuttal, but the AC considered that the clarification on the novelty issue and additional results in the rebuttal are convincing to alleviate the reviewer’s concerns. Therefore, the AC recommends accepting the paper.
Decision: Accept (poster)