SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate our model on the following metrics: (1) Success Rate (SR), which measures whether the agent stops within 3 meters to the target. (2) Success Rate Weighted by Path Length (SPL) [1], which penalizes long paths that explore the environment to reach the target instead of following the instructions. (3) Goal Progress (GP), which calculates the distance that the agent moves toward the target location. (4) Navigation Error (NE), which is the distance between the stop location and the target. (5) Trajectory Length (TL), which counts the total navigation length of the agent. We consider SPL as the main metric for R2R and R4R dataset, and GP as the main metric for CVDN dataset.
Table 1: Test leaderboard performance for Room-to-Room dataset and CVDN dataset.  $\spadesuit$  indicates approaches that augment the training environments. For "EnvEdit", we report the non-ensemble performance for a fair comparison to all other methods. Best results are in bold, and second best results are underlined.
Models	Room-to-Room dataset	CVDN
Validation Unseen	Test	Val	Test
TL	NE ↓	SR ↑	SPL ↑	TL	NE ↓	SR ↑	SPL ↑	GP ↑	GP ↑
RecBERT [20]	12.01	3.93	63.0	57.0	12.35	4.09	63.0	57.0	-	-
NDH-Full [23]	-	-	-	-	-	-	-	-	5.51	5.27
HAMT [10]	11.46	2.29	66.0	61.0	12.27	3.93	65.0	60.0	5.13	5.58
EnvEdit♠	12.13	3.22	67.9	62.9	-	-	-	-	-	-
SE3DS♠[25]	-	3.29	69.0	62.0	-	3.67	66.0	60.0
DUET [12]	13.94	3.31	72.0	60.0	14.73	3.65	69.0	59.0	-	-
DUET-CLIP	12.92	3.19	72.8	63.4	-	-	-	-	5.50	-
PANOGEN	13.40	3.03	74.2	64.3	14.38	3.31	71.7	61.9	5.93	7.17
Figure 3: Qualitative analysis of the panoramic environments generated with our PANOGEN. "Matterport3D" is the original environment for VLN tasks. "Stable Diffusion for Discretized Views" is the concatenation of separately generated discretized views given text captions.
In panoramic environment generation, we caption all the view images in the training environments in R2R dataset with BLIP-2-FlanT5-xxL. We utilize stable-diffusion-v2.1 base model to generate the single view based on caption only, and use stable-diffusion-v1.5-inpainting model to outpaint the unseen observation for the rotated views. In speaker data generation, we build our speaker model based on mPLUG-base. For navigation training, we adopt the agent architecture from DUET [12]. More Implementation Details can be found in Appendix.
In this section, we first present state-of-the-art performance on test leaderboard on Room-to-Room dataset and CVDN dataset in Sec. 6.1. Then, we show some qualitative examples of our generated panoramic in Sec. 6.2. Besides, we further use ablation studies to demonstrate that utilizing our speaker data in the panoramic environments during pre-training, and random replacing the observation with new panorama during fine-tuning are effective for enhancing VLN agents' performance in unseen environments in Sec. 6.3 and Sec. 6.4. Moreover, we investigate how the number of the panorama environments influence the performance in Sec. 6.5, and compare our PanoGen environments with the existing environment augmentation approach in Sec. 6.6. Lastly, we include two quantitative evaluations of the quality of our generated speaker data in Sec. 6.7.
6.1 Test Set Results
We show our method's performance on both the Room-to-Room (R2R) and the Cooperative Vision-and-Dialog Navigation (CVDN) dataset. We adopt DUET [12] architecture for our navigation agent.
Table 2: Ablation results for utilizing our speaker data during pre-training on validation unseen set for R2R, CVDN, and R4R datasets.
Models	Room-to-Room	CVDN	Room-for-Room
TL	NE ↓	SR ↑	SPL ↑	TL	GP ↑	TL	NE ↓	SR ↑	SPL ↑
DUET [12]	13.94	3.31	72	60	-	-	-	-	-	-
DUET-CLIP	12.92	3.19	72.84	63.37	24.09	5.50	21.04	6.06	46.61	41.94
PANOGEN+Env-only	14.21	2.99	73.35	62.12	-	-	-	-	-	-
PANOGEN+EnvDrop	13.57	3.05	73.69	63.44	25.17	5.81	22.88	6.17	46.06	40.33
PANOGEN+mPLUG	14.58	2.85	74.20	62.81	24.66	5.93	18.32	6.12	45.78	42.52
Different from DUET which uses ViT-B/16 [13] pre-trained on ImageNet to extract features, we use CLIP-ViT-B/16 [45] as it shows better performance on R2R dataset ("DUET" vs. "DUET-CLIP" in Table 1). As shown in Table 1, fine-tuning with panoramic environments from our PANOGEN improves previous SotA agent DUET [12] by  $2.7\%$  in success rate, and  $2.9\%$  in SPL on Room-to-Room test leaderboard. This demonstrates the effectiveness of utilizing our generated panoramic environments to improve agents' generalization to unseen environments. Moreover, pre-training with our speaker data improves the goal progress by 1.59 meters on CVDN test leaderboard, a relative gain of  $28.5\%$  compared with previous SotA agent HAMT [10]. This large improvement demonstrates that learning from our PANOGEN environments are especially helpful for following under-specified instructions in unseen environments which need commonsense knowledge of the visual observation to reach the target. On both the Room-to-Room dataset and CVDN dataset, learning with our PANOGEN environments achieves the new state-of-the-art performance.
6.2 Qualitative Analysis of Panoramic Environment
We show some panoramic environment generated with our PANOGEN in Figure 3. We could see that directly generating discretized views based on caption will be disjoint and inconsistent (Row "Stable Diffusion for Discretized Views"). In comparison, our recursive outpainting approach could generate continuous views that can be stitched together to form a high-quality panorama (Row "PANOGEN").
Besides the high quality and coherency, our generated panorama environments is able to preserve the wide range of objects appeared in the original environments, while generating them with new appearance and different room layout. For example, in the left generated panorama, it contains a corridor view, and shows multiple rooms that are connected to the corridor (e.g., bedroom, and bathroom). This layout also follows human's commonsense knowledge, where the bedroom and bathroom can be connected with a corridor. We include more qualitative examples of our panoramic environment and speaker data in Appendix.
6.3 Effectiveness of Speaker Data in Pre-training
In this section, we show the effectiveness of utilizing the speaker data generated for our PANOGEN environments for VLN pre-training. As shown in Table 2, pre-training the VLN agent with our speaker data ("PANOGEN+mPLUG") improves the baseline ("DUET-CLIP") by  $1.36\%$  in success rate, and 0.2 meters in navigation error on R2R dataset. Besides, we observe 0.43 meters absolute improvement in goal progress (a relative gain of  $7.8\%$ ) on CVDN validation unseen set. The large improvements in CVDN demonstrate that the agent learns useful commonsense knowledge from the diverse visual environments in our PANOGEN, and thus generalize well to the unseen environments when navigating based on under-specified instructions in CVDN dataset. Pre-training with our speaker data also shows slight improvement in R4R, improving the SPL by  $0.58\%$ .
Moreover, we compare our speaker ("PANOGEN+mPLUG") with the widely used speaker from EnvDrop [51] ("PANOGEN+EnvDrop"). We find that utilizing either speaker data improves the baseline performance on R2R and CVDN dataset, demonstrating the effectiveness of our PANOGEN environments for improving agents' generalization to unseen environments. Besides, pre-training with speaker data generated with mPLUG shows better performance in success rate on R2R and R4R dataset and higher goal progress on CVDN dataset, demonstrating the effectiveness of our speaker.
Lastly, we show the effectiveness of generating new instructions for PanoGen environments during pre-training. Compared with only pre-training on original instructions and PanoGen environments
Table 3: Ablation results for replacing the original environment with our panoramic observation during fine-tuning on validation unseen set for R2R, CVDN and R4R datasets.
Models	Room-to-Room	CVDN	Room-for-Room
TL	NE ↓	SR ↑	SPL ↑	TL	GP ↑	TL	NE ↓	SR ↑	SPL ↑
DUET [12]	13.94	3.31	72	60	-	-	-	-	-	-
DUET-CLIP	12.92	3.19	72.84	63.37	24.09	5.50	21.04	6.06	46.61	41.94
PANOGEN+Replace	13.76	2.99	74.41	63.88	23.29	5.63	18.62	6.02	47.78	44.25
("PANOGEN+Env-only"), we find that generating new instructions with our mPLUG speaker improves the performance by  $0.85\%$  in success rate, and  $0.69\%$  in SPL on R2R unseen set.
6.4 Effectiveness of Panorama Replacement in Fine-tuning
We demonstrate that using the environments from our PANOGEN as observation augmentation during fine-tuning can improve agents' generalization to unseen environments.
As shown in Table 3, randomly replacing the observation in the trajectory with our panoramic environments ("PANOGEN+Replace") improves the navigation performance on all the three datasets. Specifically, our approach improves the SR by  $1.57\%$  on R2R dataset, 0.13 meters in goal progress on CVDN dataset, and  $2.31\%$  in SPL on R4R dataset. The consistent improvements in all the three datasets demonstrate the usefulness of our PANOGEN environments.
We further show that it's important to balance the ratio of replaced observations in the full navigation trajectory. As shown in Table 4, replacing  $30\%$  of the viewpoints in the navigation trajectory achieves the best performance. The trajec
Table 4: Comparison of replacing different ratio of the viewpoints in the trajectory with the panoramic environments generated with our PANOGEN on Room-to-Room validation unseen set.
No.	Ratio	Validation Unseen
TL	NE ↓	SR ↑	SPL ↑
1	0.0	12.92	3.19	72.84	63.37
2	0.1	13.16	3.16	72.84	63.24
3	0.3	13.76	2.99	74.41	63.88
4	0.5	13.03	3.19	72.84	63.84
5	0.7	12.62	3.18	72.33	63.93
tory will not be aligned with the instruction well if we replace a large ratio of viewpoints with our PANOGEN environments, and thus the improvements in R2R validation unseen set is smaller.
6.5 Impact of the Number of Panoramic Environments
In this section, we investigate the impact of the number of panoramic environments used during VLN fine-tuning. Specifically, we randomly select 10 scans and 30 scans out of the 61 scans in the R2R training environments. During VLN fine-tuning, if the navigation trajectory belongs to these scans, we replace the original observation with our generated panoramic environments with a fixed probability. As shown in Table 5, we observe that training with more panoramic environments consistently improves the performance (No. 1 - 4). Furthermore, for every panorama in the original R2R training environments, we generate one more panoramic view
Table 5: Comparison of replacing the original environments with different number of scans of our panoramic environments.
No.	# Scans	Validation Unseen
TL	NE ↓	SR ↑	SPL ↑
1	0	12.92	3.19	72.84	63.37
2	10	13.94	3.00	72.80	62.48
3	30	13.86	3.05	73.69	62.88
4	61	13.76	2.99	74.41	63.88
5	122	13.40	3.03	74.20	64.27
with our PANOGEN. In this case, the number of panoramic environments we generate is twice the number of the original R2R training environments. As shown in Table 5, we observe that the gain in SPL is still not saturated yet (No. 5 vs No. 4), and gradually increases when we add more our PANOGEN environments for VLN fine-tuning. This suggests that it's promising to generate more panoramic environments with our approach to further enhance agents' generalizability.
6.6 Comparison with Other Environment Augmentation Approaches
In this section, we demonstrate that training a ter generalization performance compared to
Specifically, we compare our approach with two existing approaches that augment the environments to avoid overfitting: EnvEdit [31] and EnvDrop [51]. For adapting EnvEdit to DUET, we follow the batch mixing approach in EnvEdit and randomly replace half of the data in a batch with the edited environments which change the appearance of the objects during VLN finetuning. For adapting EnvDrop to DUET, we replace the regular dropout layer in DUET with the proposed environment-level dropout layer during fine-tuning. As the results shown in Table 6,
with our PanoGen environments achieves betoother environment augmentation approaches.
Table 6: Comparison of training with different environment augmentation approach on Room-to-Room validation unseen set.
Model	Validation Unseen
TL	NE ↓	SR ↑	SPL ↑
EnvEdit	13.61	3.03	72.80	63.17
EnvDrop	13.28	3.12	72.58	62.40
PanoGen	13.40	3.03	74.20	64.27
training with our PanoGen environments achieves better performance than previous approaches.
6.7 Quantitative Evaluation of Generated Speaker Data
In this section, we evaluate the quality of our generated instructions by measuring its alignment with the environment, and measuring its BERTScore compared with the original instructions. First, we measure the similarity between the generated instructions and the PanoGen environments. We hypothesize that higher similarity between instruction and trajectory pairs in the embedding space can indicate better alignment between the instruction and the trajectory. Specifically, we represent the trajectory representation by averaging the image embeddings of the viewpoints in the trajectory. The image embedding of each viewpoint is encoded with CLIP-ViT/16. We also encode the instruction with CLIP text encoder. We calculate the cosine similarity between the instruction representation and the trajectory representation. As shown in Table 7, we find that the similarity between our PanoGen environments and instructions generated with mPLUG is higher than instructions generated with EnvDrop (No.2 vs No. 3). Besides, we calculate the similarity between randomly replacing  $30\%$  of the viewpoints with PanoGen environments and the original instructions (to mimic the observation replacement fine-tuning). We average the score over 5 runs to mitigate randomness. We find that randomly replacing the observation doesn't lead to decrease in similarity (No. 4 vs No. 1).
Second, we calculate the BERTScore of instructions generated by our mPLUG based speaker and the EnvDrop speaker. Specifically, we use both speakers to generate the instructions on the R2R validation unseen set. We use Bart-base to calculate the BERTScore. Our
Table 7: Comparison of similarity between generated instructions and environments.
No.	Instruction	Environment	Similarity
1	Original	Original	0.2845
2	EnvDrop	PanoGen	0.2669
3	mPLUG	PanoGen	0.2714
4	Original	30% PanoGen, 70% Original	0.2893 (± 0.0001)
speaker achieves a BERTScore of 71.8, while the EnvDrop speaker achieves a BERTScore of 70.5.
In this paper, we propose PANOGEN, a generation approach which can potentially create infinite diverse panoramic environments conditioned on text. Specifically, we propose a recursive image outpainting that reconstructs the missing observations in the panorama gradually based on text caption to generate coherent panorama with diverse objects and room layouts. We then investigate two training methods to effectively utilize the generated panoramic environments during VLN pre-training and fine-tuning. Learning from our PANOGEN achieves the new SotA on both CVDN dataset and R2R dataset, demonstrating its effectiveness for enhancing agents' generalization to unseen environments. Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.
This work was supported by ARO W911NF2110220, ONR N00014-23-1-2356, DARPA KAIROS FA8750-19-2-1004, and DARPA MCS N66001-19-2-403. The views contained in this article are of the authors and not of the funding agency.
[56] X. Wang, W. Wang, J. Shao, and Y. Yang. Lana: A language-capable navigator for instruction following and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19048-19058, 2023.
[57] X. E. Wang, V. Jain, E. Ie, W. Y. Wang, Z. Kozareva, and S. Ravi. Environment-agnostic multitask learning for natural language grounded navigation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIV 16, pages 413–430. Springer, 2020.
[58] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.
[59] Z. Yang, J. Dong, P. Liu, Y. Yang, and S. Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10561-10570, 2019.
[60] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907-5915, 2017.
[61] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence, 41(8):1947-1962, 2018.
[62] L. Zhang, Q. Chen, B. Hu, and S. Jiang. Text-guided neural image inpainting. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1302-1310, 2020.
[63] Y. Zhao, J. Chen, C. Gao, W. Wang, L. Yang, H. Ren, H. Xia, and S. Liu. Target-driven structured transformer planner for vision-language navigation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4194-4203, 2022.
In this supplementary, we provide the following:
- Detailed description of the datasets we use in Sec. A, and more implementation details in Sec. B.
- Ablation performance of our PANOGEN on REVERIE dataset in Sec. C.
- Performance of improving panoramic view consistency across different navigation steps in Sec. D.
- More examples of the panoramic environments generated by our PANOGEN in Sec. E, and alignment with speaker data in Sec. F.
- Limitations and broader impacts in Sec. G, and licenses in Sec. H.
We evaluate our agent on three datasets: Room-to-Room dataset (R2R) [2], Cooperative Vision-and-Dialog Navigation dataset (CVDN) [52], and Room-for-Room dataset (R4R) [21].
R2R. Room-to-Room dataset contains detailed instructions to guide the agents navigate toward the target location step by step. The ground truth paths are the shortest path between the start location and the end location. The training set contains 61 different room environments, while the unseen validation set and test set contain 11, and 18 room environments that are unseen during training.
R4R. Room-for-Room dataset is created by concatenating the adjacent paths in the Room-to-Room dataset. In this case, the ground truth path is not the shortest path. This encourages the agent to follow the instructions to reach the target instead of exploring the environment bias and reach the target by directly navigating the shortest path.
CVDN. Cooperative Vision-and-Dialog Navigation dataset contains interactive dialogue instructions. The dialogue usually contains under-specified instructions, and the agent needs to navigate based on both the dialogue histories and the commonsense knowledge of the room. The room environments in the training set, unseen validation set, and test set follow the split in Room-to-Room dataset.
In panoramic environment generation, we caption all the view images in the training environments in R2R dataset with BLIP-2-FlanT5-xxL. We utilize stable-diffusion-v2.1 base model to generate the single view based on caption only, and use stable-diffusion-v1.5-outpainting model to outpaint the unseen observation for the rotated views. It takes 2 days on 6 A100s to generate all the environments.
In speaker data generation, we build our speaker model based on mPLUG-base, which has 350M parameters and utilizes ViT/B-16 as the visual backbone. We train the speaker for 4 epochs on one A6000 GPU with batch size 16 for two days.
For navigation training, we adopt the agent architecture from DUET [12]. We follow the training hyperparameters in DUET. Different from DUET, we utilize CLIP-ViT/B-16 to extract the visual features. We train the model on one A6000 GPU. We pre-train the agent with batch size 64 for 150k iterations, and then fine-tune the agent with batch size 8 for 40k iterations. Both the pre-training and fine-tuning take approximately one day to finish. We report reproduced baseline performance with CLIP-ViT/B-16 features for a fair comparison. The best model is selected based on performance on validation unseen set.
C Ablation Performance on REVERIE Dataset
We demonstrate the effectiveness of our approach on REVERIE dataset in Table 8. We observe that pretraining with our speaker data improves the baseline by  $2.64\%$  in success rate, while fine-tuning with observation replacement from PanoGen environments achieves  $4.60\%$  absolute improvement in success rate. This demonstrates that our approach generalizes well to navigation tasks that have under-specified instructions.
Table 8: Ablation performance on REVERIE unseen set.
Model	Validation Unseen
SR ↑	SPL ↑	RGS ↑	RGSPL ↑
DUET	46.98	33.73	32.15	23.03
DUET-CLIP	46.58	34.14	31.70	22.89
PanoGen+mPLUG	49.22	33.44	32.80	22.45
PanoGen+Replace	51.18	34.99	33.26	22.99
D Improving Consistency Across Steps in Panorama Environment
PanoGen
Dedicalized for Discretized Dif
Figure 4: Qualitative analysis of the panoramic environments generated with our PANOGEN. "Matterport3D" is the original environment for VLN tasks. "Stable Diffusion for Discretized Views" is the concatenation of separately generated discretized views given text captions.
In our PANOGEN environments, we didn't explicitly constrain the consistency between views at different navigation steps. In this section, we show our initial exploration of improving the view consistency across steps. Specifically, we fine-tune instructpix2pix [5] based on sub-instruction annotations [19] to generate the next views given previous step in the trajectory. After generating consistent discrete views with instructpix2pix at different viewpoints, we then outpaint the view with our approach to create panoramic environments with better consistency across navigation steps. As shown in Table 9, we observe that
Table 9: Performance of improving consistency across navigation steps in our panoramic environments on Room-to-Room validation unseen set.
Model	Validation Unseen
TL	NE ↓	SR ↑	SPL ↑
DUET-CLIP	12.92	3.19	72.8	63.4
PanoGen	13.76	2.99	74.41	63.88
PanoGen+Con	13.91	3.00	74.46	64.09
improving consistency across navigation steps slightly helps with the navigation performance on the R2R unseen set, improving the SPL by  $0.21\%$ .
E Qualitative Example for PanoGen Environment
We show more panoramic environments generated with our PANOGEN in Figure 4. We observe that directly concatenating discretized views generated separately will generate inconsistent panoramas (Row "Stable Diffusion for Discretized Views"). In comparison, our PANOGEN can generate continuous views with reasonable layout and object co-occurrence (Row "PANOGEN"). Moreover, our approach can generate panorama for both indoor and outdoor environments. Though generating the outdoor environments might not benefit agents' indoor navigation ability directly, our approach demonstrates its potential to be applied to panorama generation with different content (e.g., landscape).
F Qualitative Example for Alignment between Speaker Data and PanoGen Environment
We include one panorama-instruction example in Figure 5 to demonstrate the alignment between the instruction and the environment. We show that our panorama environments are more diverse in appearance, and the instruction data generated by our mPLUG based speaker contains more details. Besides, the general semantic information across different steps is still reasonable in consistency.
Vision-and-Language Navigation tasks can be used in many real-world applications, for example, a home service robot can bring things to the owner based on natural language instructions. In this paper, our proposed method generates panoramic environments for VLN training, and significantly improves navigation agents' generalization ability to unseen environments given limited human-annotated training data. Our approach reduces the efforts of re-training the agents in every new environment when adapting to real-world scenarios.