SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PANOGEN: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation
Jialu Li Mohit Bansal
UNC Chapel Hill
{jialuli,mbansal}@cs.unc.edu
https://pano-gen.github.io
Vision-and-Language Navigation requires the agent to follow language instructions to navigate through 3D environments. One main challenge in Vision-and-Language Navigation is the limited availability of photorealistic training environments, which makes it hard to generalize to new and unseen environments. To address this problem, we propose PANOGEN, a generation method that can potentially create an infinite number of diverse panoramic environments conditioned on text. Specifically, we collect room descriptions by captioning the room images in existing Matterport3D environments, and leverage a state-of-the-art text-to-image diffusion model to generate the new panoramic environments. We use recursive outpainting over the generated images to create consistent 360-degree panorama views. Our new panoramic environments share similar semantic information with the original environments by conditioning on text descriptions, which ensures the co-occurrence of objects in the panorama follows human intuition, and creates enough diversity in room appearance and layout with image outpainting. Lastly, we explore two ways of utilizing PANOGEN in VLN pre-training and fine-tuning. We generate instructions for paths in our PANOGEN environments with a speaker built on a pre-trained vision-and-language model for VLN pre-training, and augment the visual observation with our panoramic environments during agents' fine-tuning to avoid overfitting to seen environments. Empirically, learning with our PANOGEN environments achieves the new state-of-the-art on the Room-to-Room, Room-for-Room, and CVDN datasets. Besides, we find that pre-training with our PANOGEN speaker data is especially effective for CVDN, which has under-specified instructions and needs commonsense knowledge to reach the target. Lastly, we show that the agent can benefit from training with more generated panoramic environments, suggesting promising results for scaling up the PANOGEN environments to enhance agents' generalization to unseen environments.
Vision-and-Language Navigation (VLN) requires an agent to make sequential decisions based on both language instructions and visual environments. In indoor instruction-guided navigation, many datasets have been proposed. These datasets aim to enhance agents' ability to understand detailed navigation instructions [2], dialogue style instructions [52], instructions in different languages [26], and high-level object-finding instructions [43]. Though many efforts have been proposed to help the agent learn diverse instruction inputs, all these datasets are built on the same 3D room environments from Matterport3D, which only contains 60 different room environments for agents' training. This is because diverse photorealistic 3D room environments with a large number of sampled human-height viewpoints are very hard to collect. This limited availability of training environments poses challenges for the agent to learn the navigation policy well, and generalize to unseen new room environments.
Figure 1: Overview of our PANOGEN. We first generate captions for all the room panoramas in the Matterport3D dataset. Each panorama is discretized into 36 views, we show 15 views here for a better view of each discretized image. Then, we generate panoramic environments with recursive outpainting over a single image generated from the text caption.
Several works in VLN have been proposed to address this challenge. [51] first proposes to perform dropout on environments during training to avoid the agent overfitting to seen environments. [34] and [31] further propose to edit the existing environments by mixing up environments and changing the style and appearance of the environments. However, these approaches are still limited by the 3D environments in Matterport3D, and do not create new environments with different objects and layouts, which is important for agents' generalization to unseen environments. [11] takes one step further and introduces more unannotated 3D room environments from Habitat Matterport3D dataset (HM3D) [46], with machine-generated instructions to augment the training environments and enhance agents' generalization performance. While 900 environments are introduced, their method still relies on existing manually-captured 3D environments in HM3D, which cannot be further scaled up due to the very expensive environment collection process. Hence, in this paper, we aim to explore the possibility of generating unlimited panoramic environments without any human annotation, and investigate how to effectively utilize the generated panoramic environments for improving navigation agents' ability to generalize to unseen environments.
To this end, we propose PANOGEN, a generation method that can potentially create infinite diverse panoramic environments conditioned on text. As shown in Figure 1, we first collect descriptions of room environments by using a state-of-the-art vision-language model BLIP-2 [32] to annotate the view images in the Matterport3D dataset. Then, we use text-to-image diffusion models to generate diverse room images based on the text captions. As the agent navigates in egocentric panorama observation, learning from disjoint single-view images (middle column in Figure 1) will confuse the agent, and it cannot learn the spatial relationship between objects due to inconsistent views. Hence, to keep the observations coherent in the same panorama, we additionally propose a recursive image outpainting approach, which generates missing observations beyond the original image boundaries (right column in Figure 1). Specifically, we choose one generated image in the panorama as the starting point, and gradually rotate the camera angle right, left, up, and down, and then outpaint the unseen observation based on text descriptions. Lastly, we explore and compare two training methods to effectively utilize the generated panoramic environments. In the first method, we train a speaker to automatically generate instructions for the generated panoramic environments, based on a pre-trained vision-and-language model mPLUG [27], which has the state-of-the-art performance for zero-shot video captioning. We pre-train the VLN agent with both the original training data and the generated instruction-trajectory pairs from our panoramic environments. In the second method, instead of learning from speaker data, we directly fine-tune the VLN agents on both the original environments and our panoramic environments by randomly replacing some of the original observations with our panoramic environments to avoid overfitting to training environments.
We conduct experiments on Room-to-Room (R2R) [2], Room-for-Room (R4R) [21], and CVDN [52] datasets. We measure agents' ability in following fine-grained instructions of various lengths (R2R, and R4R), and under-specified dialogue instructions (CVDN). Empirical results demonstrate that training with our PANOGEN environments could improve the SotA agents by  $2.7\%$  in success rate and  $1.9\%$  in SPL on the R2R test leaderboard, and improve the goal progress by 1.59 meters on the CVDN
test leaderboard, achieving a relative gain of  $28.5\%$  compared with previous SotA agents. The large improvement on the CVDN dataset suggests that our PANOGEN introduces diverse commonsense knowledge of the room environments, and helps navigation when given under-specified dialogue instructions. Moreover, we analyze the impact of the number of our PANOGEN environments used for VLN training, and demonstrate the potential of scaling up our PANOGEN environments for further enhancing agents' generalization ability. Lastly, we use both qualitative and quantitative analysis to demonstrate good alignment between our generated instructions and PANOGEN environments.
Vision-and-Language Navigation Vision-and-Language Navigation is the task that requires an agent to navigate through the visual environment based on language instructions. Many datasets [2, 21, 52, 8, 39, 49, 41, 26, 43] have been proposed for this task. In this paper, we focus on indoor navigation, specifically Room-to-Room dataset (R2R), Room-four-Room dataset (R4R), and Cooperative Vision-and-Dialog Navigation dataset (CVDN). To solve this challenging task, early works build their model based on a sequence-to-sequence LSTM model [51, 35, 29, 30, 55, 33]. [36] first utilizes pre-trained vision-and-language transformers for learning an agent to pick the correct path. [20] enhances the transformer-based agent with a recurrent state to model navigation history, and [10] proposes a hierarchical architecture to encode both spatial and temporal information in navigation history. Besides, some works explore utilizing graph information to build better global visual representation [12, 53, 63, 15, 9], some works propose better proxy tasks for learning better temporal knowledge [44] and decision making ability [10, 28] during pre-training, and other works explore enriching the navigation agent with the ability to generate text [56] and image [28]. In this paper, we build our approach over the state-of-the-art agent DUET [12].
Environment Scarcity in Vision-and-Language Navigation One main challenge in Vision-and-Language Navigation is to learn from limited available training environments and generalize to unseen new environments. Though many datasets have been proposed for indoor instruction-guided navigation with diverse instruction inputs [2, 26, 21, 52, 43], their navigation environments are all from Matterport3D [7], which only contains 61 environments for training, and 29 environments for unseen validation and testing. Previous works aim to address this challenge from multiple aspects. One line of work tries to mitigate the environment bias during training by dropping out environment features [51] and learning environment-agnostic visual representation [30, 57]. Another line of research aims to generate or introduce more environments for VLN training. Specifically, [31] and [34] edit the existing environments by mixing up environments or changing room style and object appearances. However, these approaches are still limited by the existing Matterport3D environments. [11] introduces 900 environments from HM3D [46] with generated instructions to address the environment scarcity and learn a more generalizable agent. However, it's expensive to collect 3D environments, and thus their approach is hard to be further scaled up. Different from them, our proposed PANOGEN generates new panoramic environments without the need of any human annotation, and could potentially generate unlimited panoramic environments. Lastly, [24] generates future views based on history observations in a given trajectory so as to build a world model for more informed planning, and [25] synthesizes views at different locations in the original environments as data augmentation for VLN training. Our work focuses on introducing a diverse set of new environments (in appearance), while maintaining consistency in the panorama, using text-conditioned recursive image outpainting.
Text-to-Image Generation Text-to-Image generation has been one of the main research areas in both NLP and CV. With the advances of GANs [16], previous works aim to generate high resolution images with stacked generators [60, 61], and improve the perceptual quality of an image by adding regularization losses [6]. However, GANs are hard to optimize [17, 37] and face the problem of mode collapse [38]. Recently, [47] proposes to utilize the latent diffusion models to generate images with high resolution. By first encoding the image into a low dimension latent space that is perceptually equivalent to the pixel space, the latent diffusion model is computationally more efficient while maintaining good generation quality. Diffusion model shows better performance on several image generation tasks like image inpainting [62], image outpainting [59], and image synthesis [42]. [50] first extends the diffusion models to 3D indoor scene synthesis by conditioning it on geometry and editing prompts. Different from them, we propose to generate coherent panorama environments with diverse objects based on detailed text descriptions.
Figure 2: Given an image generated based on the room description, we rotate the camera angle and outpaint the unseen observation recursively to generate a consistent 360-degree panorama view.
3 PANOGEN: Generating Panoramic Environments
In this section, we describe our PANOGEN method, which generates panoramic environments from text descriptions for Vision-and-Language Navigation. Our PANOGEN addresses two main challenges for panoramic environment generation. First, the panorama observation of an indoor room environment will contain multiple objects with a complex layout, which makes it hard to use a short description to cover all the details in the panorama. Thus, we first discretize the panorama into 36 views and generate each view separately. However, the separately generated single images will be inconsistent with each other. Hence, to generate consistent panoramic environments, we propose a recursive image outpainting approach (Sec. 3.2). Second, the generated panoramic environment should align with human commonsense knowledge. For example, when generating a panorama for a bedroom, the panorama can consist of objects like "bed", "dresser", but not "refrigerator". To address this problem, instead of utilizing a large amount of available room image captions from the web for different portions in the panorama, which might have unreasonable co-occurrence of objects, we directly generate text descriptions for the panoramas in the Matterport3D environments (Sec. 3.1).
3.1 Room Description Collection
We first collect room descriptions from the Matterport3D environments (Figure 2 Right). To maximally generate panorama environments with diverse objects and reasonable layouts, we discretize the panorama into 36 views and caption the discretized views separately. The generated captions contain a more detailed description of the objects in the smaller region of the panorama. We then utilize the pre-trained vision-and-language model from BLIP-2 [32] to caption the discretized room images. BLIP-2 is trained by bootstrapping off-the-shelf pre-trained image encoders and large language models. Benefiting from the general cross-modal alignment between text and image learned during pre-training, and the zero-shot generation ability of large language models, BLIP-2 can generate creative and informative captions for images in the Matterport3D dataset.
3.2 Generating Panorama with Text-Conditioned Image Outpainting
While many works [47, 22, 40, 48, 3] have focused on building better models for text-to-image generation, how to transfer the advances in text-to-image generation to text-to-panorama generation is still less explored. [4] combines multiple diffusion generation processes with shared constraints to generate a panorama based on the text description. However, their panorama generation process can only be conditioned on one single text description of the full panorama. One text description is not enough for complex indoor navigation environments with diverse objects. Thus, we propose to generate the panorama view by outpainting recursively based on multiple text descriptions.
As discussed in Sec. 3.1, each discretized image in the Matterport3D environment is paired with a caption generated by BLIP-2. With the detailed descriptions, we use a state-of-the-art text-to-image
diffusion model Stable Diffusion [47] to generate the images. However, the panorama view is not continuous if we directly 'stitch' the generated images. To address this problem (Figure 2), we first generate one single view with zero camera elevation in the panorama based on its caption. The view with zero camera elevation serves as a better starting point, as it usually contains more important information in the panorama, whereas the positive elevation view usually consists of objects on the ceiling, and the negative elevation view usually faces the ground. Next, we rotate the generated image right, up and down by  $p_r\%$ ,  $p_u\%$ ,  $p_d\%$  respectively, and outpaint the unseen observation based on the caption of the nearby view. By conditioning the image generation on both the text caption and nearby view, the generated images are consistent in style and can be stitched into a coherent panorama. For example, as shown in Figure 2, the caption "a bedroom with a bed and a dresser" mentions "bed", and the bed is already partially generated in the image. When outpainting the unseen observation, our generation model will complete the bed and generate coherent surrounding views, instead of generating a new bed with different appearance. We repeat this process until we generate all 36 views in the panorama. We generate one panorama for each panorama in the R2R training environments. In total, our PANOGEN have 7,644 panoramas, stitched from 275,184 images.
4 Utilizing Panoramic Environments for VLN Training
In this section, we first introduce the problem setup of VLN in Sec. 4.1, and the general training procedures for VLN in Sec. 4.2. Then, we introduce the two ways that we utilize the generated panoramic environments. Specifically, we first explore generating new instruction-trajectory pairs in the panoramic environments and utilize these data in VLN pre-training (Sec. 4.3). Then, we directly utilize the generated panoramic environments to replace the original observation during VLN fine-tuning to avoid agents' overfitting to training environments (Sec. 4.4).
In Vision-and-Language Navigation, the agent needs to navigate through the environment based on natural language instructions. Formally, at each time step  $t$ , the agent takes in the full language instruction  $I$ , a panoramic observation  $P_{t}$  of the current location, and navigation history observations  $\{P_{i}\}_{i = 1}^{t - 1}$ . The agent needs to pick the next step from a set of navigable locations  $\{g_{t,k}\}_{k = 1}^{K}$ . We follow the setup in [12], where navigable locations include both adjacent locations which can be reached in one step, and global viewpoints which are observed but not visited in navigation history. Navigation finishes when the agent predicts 'STOP' action or reaches the maximum navigation steps.
4.2 VLN Training Procedures
Training a Vision-and-Language Navigation agent contains two stages: pre-training and fine-tuning.
Pre-training. In the pre-training stage, the agent is pre-trained with three proxy tasks: Masked Language Modeling (MLM), Instruction and Trajectory Matching (ITM), and Single Action Prediction (SAP). Specifically, in Masked Language Modeling, the agents need to predict the randomly masked words given both the unmasked instructions and the full trajectory observations. In Instruction and Trajectory Matching, the agent needs to learn to pick the correct instruction and trajectory pair from one positive pair and four negative pairs. Two of the negatives come from other trajectories in the same batch, and another two of the negatives shuffle the original trajectories so that the agent could learn the order of the observations in the trajectory. In Next Action Prediction, the agent mimics the navigation task by doing single action prediction based on full instruction and navigation history.
Fine-tuning. In the fine-tuning stage, we follow [12] and train the agent with the supervision from the pseudo interactive demonstration. Specifically, at each time step, the ground truth navigation trajectory is sampled based on the current policy learned by the agent. This sampling process enables the agent to explore the environment and generalize better.
4.3 Generating Instructions for Paths in Panoramic Environments
Given the high quality and coherent panoramic environments from our PANOGEN, we investigate how to utilize these environments for mitigating the data scarcity problem in VLN agents' training and enhancing VLN agents' generalization ability.
As our PANOGEN environments are generated conditioned on text captions, it shares similar semantics with the original environments, but the room layout and appearance will be different from the original environments. Thus, the instruction for traversing the original environment will not be aligned well with the new panoramic environment. To address this problem, we train a speaker to generate instructions for the new panoramic environments.
Previous works [51, 14, 31, 18] train a small LSTM-based speaker from scratch on VLN datasets to generate instructions for unannotated paths in the Matterport3D environments. However, as these speakers are not pre-trained on larger image-text datasets, they lack general visual grounding knowledge and therefore it's hard to generate instructions with diverse entity mentions that do not appear in the small training data. Marky [54] improves the speaker by utilizing multilingual T5 (mT5) [58], which is a text-to-text encoder-decoder transformer. mT5 enables the speaker to generate instructions with a large multi-lingual vocabulary, and has prior text domain knowledge. However, mT5 only has text domain knowledge, and will need to learn the visual grounding knowledge from scratch on the limited training data. Hence, to introduce general cross-modal alignment information to the agent, we propose to build our speaker based on mPLUG [27], a vision-language model that can do both multi-modal reasoning and generation.
mPLUG is a transformer pretrained on image and text pairs. To adapt it to navigation trajectory, which is a sequence of panorama images, we first simplify the panorama representation to the single view representation which the agent is facing. The single views are first encoded with CLIP-ViT/B-16. The encoded image patches are then flattened and concatenated as the input for instruction generation. We fine-tune mPLUG on the Room-to-Room (R2R) training data, and use it to automatically generate instructions for all the paths in the R2R training data with our new panoramic observation. In total, our speaker data has 4,675 instruction-trajectory pairs.
Following [12], we use R2R dataset [2], and Prevalent dataset [18] for agents' pre-training. Moreover, we further pre-train the agent with our speaker data, which introduces our panoramic environments to the agent and improves the agents' generalization ability.
4.4 Learning from Panoramic Environments during Fine-tuning
We further enhance the VLN fine-tuning by incorporating our panoramic environments. Specifically, we randomly replace  $m\%$  of the observations in the trajectory with our panoramic environments during fine-tuning. This observation replacement helps the agent avoid overfitting to the limited training environments. As our panoramic environments are generated conditioned on text descriptions of the room, thus the semantic underlying the panoramic observation is similar to the original environments. This ensures that when the replacement ratio  $m\%$  is not large, after replacing the original environments with our panoramic observations, the instruction and the observations still have reasonable alignment (discussed in Sec. 6.4).
5.1 Dataset and Evaluation Metrics
We evaluate our agent on three datasets: Room-to-Room dataset (R2R) [2], Cooperative Vision-and-Dialog Navigation dataset (CVDN) [52], and Room-for-Room dataset (R4R) [21]. The training set contains 61 different room environments, while the unseen validation set and test set contains 11, and 18 room environments that are unseen during training. Details of the dataset can be found in Appendix.