Title: Frequency-Enhanced Data Augmentation for Vision-and-Language Navigation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Vision-and-Language Navigation; High-Frequency; Data Augmentation
Abstract: Vision-and-Language Navigation (VLN) is a challenging task that requires an agent to navigate through complex environments based on natural language instructions. In contrast to conventional approaches, which primarily focus on the spatial domain exploration, we propose a paradigm shift toward the Fourier domain. This alternative perspective aims to enhance visual-textual matching, ultimately improving the agent's ability to understand and execute navigation tasks based on the given instructions. In this study, we first explore the significance of high-frequency information in VLN and provide evidence that it is instrumental in bolstering visual-textual matching processes. Building upon this insight, we further propose a sophisticated and versatile Frequency-enhanced Data Augmentation (FDA) technique to improve the VLN model's capability of capturing critical high-frequency information. Specifically, this approach requires the agent to navigate in environments where only a subset of high-frequency visual information corresponds with the provided textual instructions, ultimately fostering the agent's ability to selectively discern and capture pertinent high-frequency features according to the given instructions. Promising results on R2R, RxR, CVDN and REVERIE demonstrate that our FDA can be readily integrated with existing VLN approaches, improving performance without adding extra parameters, and keeping models simple and efficient. The code is available at https://github.com/hekj/FDA.

Frequency-enhanced Data Augmentation for Vision-and-Language Navigation

Abstract

Vision-and-Language Navigation (VLN) is a challenging task that requires an agent to navigate through complex environments based on natural language instructions. In contrast to conventional approaches, which primarily focus on the spatial domain exploration, we propose a paradigm shift toward the Fourier domain. This alternative perspective aims to enhance visual-textual matching, ultimately improving the agent's ability to understand and execute navigation tasks based on the given instructions. In this study, we first explore the significance of high-frequency information in VLN and provide evidence that it is instrumental in bolstering visual-textual matching processes. Building upon this insight, we further propose a sophisticated and versatile Frequency-enhanced Data Augmentation (FDA) technique to improve the VLN model's capability of capturing critical high-frequency information. Specifically, this approach requires the agent to navigate in environments where only a subset of high-frequency visual information corresponds with the provided textual instructions, ultimately fostering the agent's ability to selectively discern and capture pertinent high-frequency features according to the given instructions. Promising results on R2R, RxR, CVDN and REVERIE demonstrate that our FDA can be readily integrated with existing VLN approaches, improving performance without adding extra parameters, and keeping models simple and efficient. The code is available at https://github.com/hekj/FDA.

1 Introduction

Vision-and-Language Navigation (VLN) requires an embodied agent to navigate in complex environments following human instructions. This research area has garnered substantial attention due to its potential applicability in human-robot interaction contexts, such as service and rescue robotics. Despite the significant advancements [5, 12, 23, 25, 55, 34, 9, 42, 10, 37, 64, 50, 20, 68, 60] achieved in devising deep learning models for VLN, the task remains arduous, primarily due to the complexities involved in attaining precise visual-textual matching.

Recent studies in the field of VLN have focused on enhancing embodied agents' ability to navigate complex environments using natural language instructions. These studies aim to develop more robust and efficient deep learning models that can tackle the challenges of visual-textual matching, thereby

其

improving the agent's capacity to understand and execute navigation tasks based on given instructions. Some of the recent studies have investigated attention mechanisms [58, 25, 9, 64], detection models [44, 1, 10], and fine-grained trajectory-instruction pairs [66, 24, 21] to enhance cross-modal matching from the spatial domain perspective.

Validation Unseen

90

06

Normal

Normal

LF-Perturbed

LF-Perturbed

80

78.55

HF-Perturbed

80

HF-Perturbed

76.3

75.02

70

70

71.78

70.2

64.15

63.86

65.64

61.12

61.26

%60

960

58.15

53.68

51.03

51.04

50

49.07

50

43.88

45.13

44.4

40

40

30

30

TD-STP

DUET

TD-STP

DUET

Beyond the spatial domain perspective, in this paper, we are interested in investigating the effectiveness of other domain perspectives for VLN. We present a paradigm shift toward the Fourier domain to enhance visual textual matching, a research area that has received limited prior investigation. Specifically, high-frequency and low-frequency information pertains to distinct components of an image when analyzed within the Fourier domain. High-frequency information encompasses rapid changes, fine details, edges, and texture patterns, as illustrated in Figure 1 (part of the blue background). Conversely, low-frequency information signifies the overall structure and global features of an image, including large shapes and smooth color gradients, thereby capturing the general appearance and layouts, as shown in Figure 1 (part of the orange background).

We simply investigate the sensitivity of benchmark methods to low and high-frequency information by perturbing the low-frequency or high-frequency components in images. Three powerful baseline models, i.e. , HAMT [9], DUET [10], and TD-STP [64], are used to analyze the significance of low/high-frequency information on both R2R validation seen and unseen splits, wherein the navigation views are disrupted in the Fourier domain. As Shown in Figure 2, the three models maintain a relatively high Success Rate (SR) under low-frequency perturbations. However, the SR on validation seen and unseen splits decrease markedly under the high-frequency perturbation navigation process, exhibiting lower SR metrics compared to low-frequency perturbations. For example, on the validation seen dataset, the SR results for HAMT [9], DUET [10], and TD-STP [64] decrease by 31.44, 27.52, and 27.23, respectively, when affected by high-frequency perturbations. These findings reveal that VLN approaches exhibit a marked sensitivity to high-frequency information.

In light of these observations, we hypothesize that high-frequency information from images may be of paramount importance for cross-modal navigation tasks. Through extensive experiments, we ascertain that high-frequency information indeed serves a pivotal role in enriching visual representations, thereby contributing to a marked enhancement in the navigation performance of models when

faced with novel environments. To efficiently leverage the benefits of high-frequency information, we further propose a Frequency-enhanced Data Augmentation (FDA) tailored for VLN, a simplistic yet efficacious approach to augment the model's ability to capture essential high-frequency information. Specifically, the FDA method utilizes the Discrete Fourier Transform on navigation views, extracting high and low-frequency components from RGB channels. It replaces part of the high-frequency components with those from an interference image to introduce high-frequency perturbations. Augmented data is obtained by applying inverse Fourier Transform to the combination of the perturbed high-frequency and original low-frequency components. By training the agent to match original instructions with both the original and augmented navigation views concurrently, the FDA method encourages the agent to hone its ability to capture the relevant high-frequency information that best aligns with the given instruction. This, in turn, enhances the agent's overall performance in VLN tasks, making it more adept at identifying and leveraging critical high-frequency features.

Leveraging FDA during training, existing SoTA methods can achieve significant performance improvements without the need for additional parameters. This demonstrates the effectiveness of FDA in enhancing visual-textual matching capabilities in Vision-and-Language Navigation (VLN) tasks while maintaining model simplicity and efficiency. Our contributions are summarized as follows: 1) we conduct a first-of-its-kind in-depth analysis of frequency domain information in VLN task, highlighting the importance of high-frequency information in improving navigation performance. This novel perspective offers fresh research opportunities for the community to explore and enhance VLN models. 2) We further introduce a simple, effective data augmentation method, Frequencyenhanced Data Augmentation (FDA), which enhances a model's ability to discern and capture essential high-frequency information without added complexity, offering practical solutions for the research community. 3) Our method has achieved superior results on various cross-modal navigation tasks, i.e. , R2R, RxR, CVDN and REVERIE, and shown great adaptability across different models.

2 Frequency Perspective for Vision-and-Language Navigation

In this section, we first introduce the VLN problem formulation. Then we explore the potential value of high-frequency and low-frequency information for VLN. Finally, we present the Frequencyenhanced Data Augmentation (FDA) for enhancing model's ability to capture essential high-frequency information.

2.1 Problem Formulation

According to the VLN setup, an agent navigates in an in-door environment E = { p 1 , p 2 , ..., p | E | } with numbers of preset points p i in it, following a human instruction T = { w 1 , w 2 , ..., w | T | } . Assuming at step t, the agent standing on the point p i t can perceive the surrounding panoramic vision O t = ( o k t ) 36 k =1 around it which is composed by 36 discrete observations o k t . Each observation o k t = ( I k t , θ k t , ϕ k t ) is the combination of the k th view I k t with its relative heading θ k t and elevation ϕ k t to the oriented view. The neighboring navigable points N ( p i t ) distribute among some of these views. The agent selects the next point from these neighboring points N ( p i t ) based on the correlation between instruction T and the observations o k t where N ( p i t ) locate in. Then, the agent will be teleported to that selected point. This navigation continues until the agent predicts a stop action or exceeds a preset step threshold. Navigation is considered a success when the agent stops within 3 meters of the target destination.

2.2 High Frequency or Low Frequency: Which Benefits VLN Performance?

Considering the observations of Figure 2, we posit that high-frequency information in images may be of paramount importance for cross-modal navigation tasks. In order to validate this hypothesis, we conduct a straightforward experiment that entails the fusion of original image features with their corresponding high-frequency or low-frequency components. These amalgamated features subsequently serve as input for the navigation network during training and testing, as shown in Figure 3. The results on TD-STP [64] are present in Table 1. We can observe the integration of high-frequency features leads to an improvement in navigation accuracy, such as SR increasing from 69.65 to 70.37 for unseen environments. Conversely, incorporating low-frequency features appears to have an adverse effect, resulting in a decrease in accuracy with SR dropping from 69.65 to 68.75. The above observation highlights the critical role of high-frequency information in cross-modal navigation. This is because 1) high-frequency information encompasses fine details such as edges, corners, and texture patterns. These details are essential for accurately identifying and differentiating objects, scenes, and locations, which can lead to more effective visual-textual matching and better

FFT+GHPF

𝐻

!

𝐻

"

𝐻

#

iFFT

Visual

Encoder

Navigator

INSTRUCTION

…go upstairs, …,

stop there.

Visual

Encoder

Textual

Encoder

FFT+GLPF

iFFT

Visual

Encoder

Navigator

INSTRUCTION

…go upstairs, …,

stop there.

Visual

Encoder

Textual

Encoder

𝐿

!

𝐿

"

𝐿

#

口

(a) Navigator with high-frequency information as extra input

(b) Navigator with low-frequency information as extra input

navigation performance. 2) Models trained with high-frequency information tend to be more robust to environmental variations and exhibit greater generalization ability to unseen environments, as the model learns to focus on a more diverse set of features, rather than just memorizing specific low-frequency, global patterns present in the training data.

2.3 Frequency-enhanced Data Augmentation

To systematically capitalize on the advantages of high-frequency information, we subsequently introduce a Frequency-enhanced Data Augmentation (FDA) method, specifically designed for VLN tasks. This approach is both straightforward and effective in bolstering the model's capacity to apprehend and incorporate crucial high-frequency information. As illustrated in Figure 4, the reference image I is a navigation view corresponding to the navigation instruction T (' Go forward, pass by the refrigerator, turn left behind the dining table, go straight through the doorway... "). The interference image ˆ I is another navigation view sampled from the Matterport3d (Mp3d) dataset [6] randomly. To prevent information leakage, all interference images are sampled from training/validation seen splits, and no image from the validation unseen and test splits is used. First, we transform these two images to the frequency domain space via FFT, resulting in two frequency spectrums F { rgb } I and F { rgb } ˆ I :

where F { rgb } denotes the Fourier Transform on RGB color channels. Afterward, we apply High-Pass and Low-Pass Gaussian Filters on the two frequency spectrums to obtain the reference high-frequency H { rgb } , reference low-frequency L { rgb } and interference high-frequency ˆ H { rgb } .

where G h and G l denote the Gaussian High-Pass Filter (GHPF) and Gaussian Low-Pass Filter (GLPF), and ⊙ is element-wise multiplication. Then, we mix the high-frequency components of these two

FFT

!

𝐻

!

𝐻

!

𝐻

"

𝐻

#

𝐿

!

𝐿

"

𝐿

#

FFT

Sample

𝐻

!

𝐻

"

𝐿

!

𝐿

"

𝐿

#

Mix

iFFT

Mix

INSTRUCTION

Go forward, pass by the

refrigerator, turn left behind

the dining table, go straight

through the doorway…

Augmented Image

ALIGN

GHPF

GHPF

&

GLPF

!

𝐻

#

!

𝐻

"

!

𝐻

#

!

𝐻

#

images. Specifically, for each RGB channel of the reference image, there is a certain probability that its high-frequency component is replaced by the interference high-frequency from the same channel:

where H { rgb } mix is the mixed high frequency. We combine it with the reference low frequency L { rgb } and then apply iFFT to obtain the frequency-enhanced augmented image I mix :

Finally, the original image I and augmented image I mix share the same textual instruction label T and are used alternately to train the agent during the training phase:

where L ( θ ) denotes the navigation loss considering both the original image I and frequency-enhanced augmented image I mix , θ denotes the parameters of the navigator.

3 Experiments

3.1 Datasets

The visual environments are based on the photo-realistic dataset Matterport3d (Mp3d) [6]. There are a total of 90 houses, with 61, 11, and 18 houses allocated for training/validation seen, validation unseen, and test splits, respectively. Four datasets containing the instruction-trajectory pairs have been adopted: R2R [5], RxR [29], CVDN [52] and REVERIE [45]. R2R includes 7,189 trajectories and each trajectory has 3 corresponding English instructions. RxR has 16,522 trajectories and 126,069 multilingual (English, Hindi and Telugu) instructions in total. Compared to R2R, its trajectories are longer and instructions are more complex. CVDN is based on human-to-human dialogs. The multi-turn dialogs instruct the agent to navigate to the destinations. There are 2,050 dialogs and over 7,000 trajectories in this dataset. REVERIE contains a total of 21,702 high-level instructions, including descriptions of navigation destinations and the objects to be found.

3.2 Evaluation Metrics

According to the conventions of each task, we adopt a total of ten evaluation metrics, namely Success Rate (SR), Success Rate weighted by Path Length (SPL), Trajectory Length (TL), Navigation Error (NE), normalized Dynamic Time Warping (nDTW), Success weighted by normalized Dynamic Time Warping (SDTW), Goal Progress (GP), Navigation Oracle Success Rate (OSR), Remote Grounding Success Rate (RGS) and Remote Grounding Success Rate weighted by Navigation Path Length (RGSPL). SR measures the proportion of successful navigation. SPL is a trade-off metric between SR and TL. TL is the average trajectory length. NE measures the average distance between the final position and the destination. nDTW measures the fidelity between the predicted trajectory and the groundtruth trajectory. SDTW combines the evaluation of SR with nDTW. GP measures the progress toward the goal. OSR measures the proportion of successful arrivals at or passage through the destinations. RGS represents the proportion of successful groundings to the correct object. RGSPL is a trade-off metric between RGS and TL. R2R uses the SR, SPL, TL and NE metrics. RxR adopts the SR, SPL, nDTW and SDTW metrics. CVDN uses the GP metric. REVERIE adopts the TL, OSR, SR, SPL, RGS and RGSPL metrics.

3.3 Ablation Study

FDA Assists in the R2R Task. We enhance both Transformer-based and LSTM-based strong baselines, i.e. , HAMT [9], DUET [10], TD-STP [64], Follower [15] and EnvDrop [49], with our FDA method on R2R task, as shown in Table 2. The three enhanced Transformer-based models have achieved clear advantages over the baselines, with SR improved by 1.88, 1.66, 2.13 and SPL improved by 2.01, 0.88, 1.02, respectively on the validation unseen split. Meanwhile, our FDA method can also boost the performances of the two LSTM-based models, with SR improved by 7.3 and 3.4, respectively. These results demonstrate the adaptability of our approach across various models, which can significantly improve their performances and alleviate overfitting.

FDAFacilitates Performances on Other Cross-Modal Navigation Tasks. Furthermore, we evaluate our FDA method on RxR, CVDN and REVERIE tasks. Table 3 shows the results on validation unseen splits of RxR and CVDN tasks. The HAMT equipped with FDA has obtained notable superior performances compared to the baseline. On RxR task, the SR and SDTW have been improved by 1.7 and 2.0, respectively. On CVDN task, our method could boost the GP performance by 0.36. Table 4 shows our method's performance on REVERIE task. The DUET enhanced by FDA significantly outperforms the baseline, with SR and RGS improved by 2.92 and 3.49. These observations indicate

Table 4: Ablabtion on the REVERIE task.

that our approach can be easily and effectively integrated into other cross-modal navigation tasks while ensuring a significant performance boost.

Performances on Navigation Scenes with High-Frequency Interference. As analyzed before, existing VLN methods are severely limited in high-frequency perturbed navigation scenes. Table 5 shows more details. We observe that the baselines can perform well under the normal navigation scenes. Setting 1: Then, we introduce high-frequency interference, whose corresponding spatial image is the navigation views randomly sampled from MP3d [6], into seen/unseen navigation views as described in the method section to get the high-frequency perturbed unseen navigation scenes. Navigating under such scenes, all models have experienced a significant decrease in SR metric. Taking TD-STP as an example, its SR decreases by as much as 27.23 and 24.52 on validation seen and validation unseen, respectively. The fact that the models are easily and severely affected by high-frequency interference suggests their lack of ability to effectively identify and capture the necessary high-frequency information based on the instructions. However, with the assistance of our FDA, all models show notable improvement in performance. HAMT, DUET, and TD-STP see SR metric increases of 23.5, 18.41, 19.96 on validation seen split, and 13.92, 14.69, 16.98 on validation unseen split. Setting 2: To distinguish the source of interference from the training phase, we have also sampled high-frequency interference from the 1000 classes of ImageNet [48]. Due to the out-ofdomain interference, we find that the SR of baseline models is significantly more affected under this setting. Taking TD-STP as an example, its SR on validation seen and validation unseen decreases by 35.75 and 27.55, respectively. In such challenging scenes, our FDA-enhanced models whose training data only contains high-frequency and low-frequency information from Mp3d, perform remarkably well, improving the SR of HAMT, DUET, and TD-STP by 17.83, 19.2, and 22.53 on validation seen, and by 16.69, 19.08, and 19.03 on validation unseen, respectively. In most cases, the navigation performances of the models have gained even greater improvements compared to the first setting. These two settings illustrate FDA-enhanced models excelling in both the seen/unseen perturbed images (val seen/unseen) and the seen/unseen perturbation images (MP3d/ImageNet). This serves as compelling evidence of our method's remarkable capability to recognize and capture essential high-frequency information to boost navigation performance.

Comparison with Existing Data Augmentation Methods. Table 6 shows the comparison with the visual data augmentation method ENVEDIT 2 [33], and textual data augmentation methods Speaker [15] and PREVALENT [20]. Compared to ENVEDIT which aims to increase the diversity of the environment by using GANs to alter environment style, our method, which also serves as a visual data augmentation method, demonstrates a clear superiority, with SR of HAMT/TD-STP increased by 0.21/1.96 on validation unseen, respectively. In comparison with the textual augmentation methods, the results indicate that our FDA method demonstrates consistent improvements on both baseline models Follower and TD-STP -3 . Additionally, it effectively complements the two textual augmentation methods, further enhancing cross-modal navigation performance. Our FDA has demonstrated superior performance without relying on external models such as visual/textual generative models. As a result, our method could greatly reduce the complexity of data augmentation work in VLN field.

3.4 Comparison with the State-of-the-art Methods

We compare our method with SoTA methods on R2R, RxR, CVDN and REVERIE tasks.

On R2R task, Table 7 shows our method achieves the best performance on both validation unseen and test splits. Compared to the DUET model which has been fed with the extra object features not included in our method, the SR/SPL of our method still outperforms it by 2/4 points on validation unseen. Compared to TD-STP, Our method shows SR/SPL improvements of 2/1 and 3/2 points on validation unseen and test splits. On RxR task, Table 8 shows the comparisons between our method and previous methods. The results show that our method outperforms the previous SoTA HAMT with SPL/nDTW improved by 1.8/2.1 and 0.4/1.5 on validation unseen and test splits, respectively. On CVDN task, our method also shows superior performance over previous methods in Table 9. Compared to HAMT, the previous SoTA, our method gains a GP improvement of 0.05/0.36/0.25 on validation seen, validation unseen and test splits, respectively. On REVERIE task, as shown in Table 10, our method outperforms the previous SoTA DUET with the increase of 2.92/2.98/3.49/3.30 and 1.21/0.49/2.03/1.47 in SR/SPL/RGS/RGSPL on validation unseen and test splits, respectively.

The remarkable performance analyzed above has demonstrated the excellence of our method in enhancing cross-modal navigation, as well as the outstanding adaptability across different tasks.

4 Related Work

Vision-and-Language Navigation. Anderson et al. [5] are the first to introduce the Vision-andLanguage Navigation, where an agent is asked to follow an English instruction in R2R to navigate.

After that, model architecture [58, 4, 24, 12, 23, 25, 55, 1, 34, 9, 42, 10, 37, 7, 59, 2, 3, 22], strategy design [28, 40, 39, 17, 8, 18], training paradigm [66, 62, 54, 38], loss design [65, 36, 32], reward shaping [27, 56, 21], pretraining [35, 20, 41, 46, 49, 61, 47] and probing experiments [26, 63, 67] have been widely explored. Then, a more challenging task RxR [29] containing multilingual instructions and longer trajectories is proposed. A series of works [34, 32, 21, 9] have explored the above and make significant progress. CVDN [52] is a task asking an agent to navigate following multi-turn human-to-human dialog. In addition to requiring the agent to have cross-modal understanding, it also demands a certain level of reasoning ability. Works [50, 20, 69, 68, 60, 9] focusing on that have been investigated to advance the development of this field. While these works have conducted extensive exploration, they have almost neglected the frequency domain that our work has demonstrated to be crucial for VLN navigation.

Data Augmentation in Vision-and-Language Navigation. In terms of text augmentation, Fried et al. [15], Wang et al. [57] and Dou et al. [14] use instruction generators to generate pseudo-instructions according to sampled trajectories. In terms of vision augmentation, Fu et al. [16] learn a sampler to sample the challenging paths to enhance the agent's ability for handling them. Tan et al. [51] use environment dropout on seen environments to mimic unseen environments and generate the pseudo-instructions for it with instruction generator. Parvaneh et al. [43] learn a model to generate counterfactuals on seen environments to simulate unseen environments. Li et al. [33] adopt GAN to transfer the environment style for increasing the diversity of the environment. However, all of these works focus on the augmentation in the spatial domain and have ignored the high-frequency information which is crucial for cross-modal matching. Moreover, they have to rely on extra generator models, which require intricate design and will increase the model burden.

Data Augmentation in Robotics. Domain randomization is a form of Data Augmentation (DA) that has shown effectiveness in transferring reinforcement learning policies from simulation to the real world [53]. In contrast, Cobbe et al. [11] propose a modification to Cutout [13], where multiple rectangular regions of varying sizes are masked with a random color for each observation. Lee et al. [31] use a randomized (convolutional) neural network that adds random perturbations to the input observations, which helps the trained agents learn more robust features that are invariant across diverse and randomized environments. RAD [30] investigates the general DAs for robotics on both pixelbased and state-based inputs. In this paper, we propose a simple and effective frequency-enhanced DA for the cross-modal task VLN, a fundamental task in robotics.

5 Conclusion

Shifting from the perspective of spatial domain previous works mainly focus on, we provide a firstof-its-kind in-depth analysis of the frequency domain information in VLN tasks. Our findings have demonstrated the importance of high-frequency information in enhancing navigation performance. In light of the findings, we further propose the Frequency-enhanced Data Augmentation (FDA) method, a simple and effective data augmentation technique to boost a model's capability to discern and capture essential high-frequency information without the need for additional auxiliary generative models. Our method on a range of VLN tasks, including R2R, RxR, CVDN and REVERIE, achieves notable performances and shows great adaptability across different VLN models. We hope our work can offer a practical solution and bring new insights to the VLN research community.

Limitations and Future Work. Our work focuses on enhancing the model's general ability to recognize and capture essential high-frequency information. However, we have not yet explored the fine-grained correlation between frequency and specific scenes or categories. This area of investigation remains an avenue for future exploration.

References

D. An, Y. Qi, Y. Huang, Q. Wu, L. Wang, and T. Tan. Neighbor-view enhanced model for vision and language navigation. In ACM International Conference on Multimedia , 2021.

D. An, Y. Qi, Y. Li, Y. Huang, L. Wang, T. Tan, and J. Shao. Bevbert: Multimodal map pre-training for language-guided navigation. In International Conference on Computer Vision , pages 2737-2748, 2023.

D. An, H. Wang, W. Wang, Z. Wang, Y. Huang, K. He, and L. Wang. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. arXiv preprint arXiv:2304.03047 , 2023.

P. Anderson, A. Shrivastava, D. Parikh, D. Batra, and S. Lee. Chasing ghosts: Instruction following as bayesian state tracking. Advances in Neural Information Processing Systems , 2019.

P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In IEEE Conference on Computer Vision and Pattern Recognition , 2018.

A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 , 2017.

J. Chen, C. Gao, E. Meng, Q. Zhang, and S. Liu. Reinforced structured state-evolution for vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

P. Chen, D. Ji, K. Lin, R. Zeng, T. Li, M. Tan, and C. Gan. Weakly-supervised multi-granularity map learning for vision-and-language navigation. Advances in Neural Information Processing Systems , 35:38149-38161, 2022.

S. Chen, P.-L. Guhur, C. Schmid, and I. Laptev. History aware multimodal transformer for vision-andlanguage navigation. Advances in Neural Information Processing Systems , 34, 2021.

S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning . PMLR, 2019.

Z. Deng, K. Narasimhan, and O. Russakovsky. Evolving graphical planner: Contextual global planning for vision-and-language navigation. Advances in Neural Information Processing Systems , 33, 2020.

T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Z.-Y. Dou and N. Peng. FOAM: A follower-aware speaker model for vision-and-language navigation. In North American Chapter of the Association for Computational Linguistics , Seattle, United States, July 2022. Association for Computational Linguistics.

D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell. Speaker-follower models for vision-and-language navigation. Advances in Neural Information Processing Systems , 2018.

T.-J. Fu, X. E. Wang, M. F. Peterson, S. T. Grafton, M. P. Eckstein, and W. Y. Wang. Counterfactual vision-and-language navigation via adversarial path sampler. In European Conference on Computer Vision . Springer, 2020.

C. Gao, X. Peng, M. Yan, H. Wang, L. Yang, H. Ren, H. Li, and S. Liu. Adaptive zone-aware hierarchical planner for vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 14911-14920, 2023.

G. Georgakis, K. Schmeckpeper, K. Wanchoo, S. Dan, E. Miltsakaki, D. Roth, and K. Daniilidis. Crossmodal map learning for vision and language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 15460-15470, 2022.

P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid. Airbert: In-domain pretraining for visionand-language navigation. In International Conference on Computer Vision , 2021.

W. Hao, C. Li, X. Li, L. Carin, and J. Gao. Towards learning a generic agent for vision-and-language navigation via pre-training. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

K. He, Y. Huang, Q. Wu, J. Yang, D. An, S. Sima, and L. Wang. Landmark-rxr: Solving vision-andlanguage navigation with fine-grained alignment supervision. Advances in Neural Information Processing Systems , 34, 2021.

K. He, Y. Jing, Y. Huang, Z. Lu, D. An, and L. Wang. Memory-adaptive vision-and-language navigation. Available at SSRN 4539118 .

Y. Hong, C. Rodriguez, Y. Qi, Q. Wu, and S. Gould. Language and visual entity relationship graph for agent navigation. Advances in Neural Information Processing Systems , 33, 2020.

Y. Hong, C. Rodriguez-Opazo, Q. Wu, and S. Gould. Sub-instruction aware vision-and-language navigation. Empirical Methods in Natural Language Processing , 2020.

Y. Hong, Q. Wu, Y. Qi, C. Rodriguez-Opazo, and S. Gould. A recurrent vision-and-language bert for navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

R. Hu, D. Fried, A. Rohrbach, D. Klein, T. Darrell, and K. Saenko. Are you looking? grounding to multiple modalities in vision-and-language navigation. In Annual Meeting of the Association for Computational Linguistics , 2019.

V. Jain, G. Magalhaes, A. Ku, A. Vaswani, E. Ie, and J. Baldridge. Stay on the path: Instruction fidelity in vision-and-language navigation. Association for Computational Linguistics , 2019.

L. Ke, X. Li, Y. Bisk, A. Holtzman, Z. Gan, J. Liu, J. Gao, Y. Choi, and S. Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. Empirical Methods in Natural Language Processing , 2020.

M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems , 33, 2020.

K. Lee, K. Lee, J. Shin, and H. Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference on Learning Representations , 2020.

J. Li, H. Tan, and M. Bansal. CLEAR: Improving vision-language navigation with cross-lingual, environment-agnostic representations. In North American Chapter of the Association for Computational Linguistics , 2022.

J. Li, H. Tan, and M. Bansal. Envedit: Environment editing for vision-and-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

J. Li, H. H. Tan, and M. Bansal. Improving cross-modal alignment in vision language navigation via syntactic information. In North American Chapter of the Association for Computational Linguistics , 2021.

X. Li, C. Li, Q. Xia, Y. Bisk, A. Celikyilmaz, J. Gao, N. Smith, and Y. Choi. Robust navigation with language pretraining and stochastic sampling. In Empirical Methods in Natural Language Processing , 2019.

X. Liang, F. Zhu, Y. Zhu, B. Lin, B. Wang, and X. Liang. Contrastive instruction-trajectory learning for vision-language navigation. In AAAI Conference on Artificial Intelligence , volume 36, 2022.

B. Lin, Y. Zhu, Z. Chen, X. Liang, J. Liu, and X. Liang. Adapt: Vision-language navigation with modality-aligned action prompts. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

B. Lin, Y. Zhu, Y. Long, X. Liang, Q. Ye, and L. Lin. Adversarial reinforced instruction attacker for robust vision-language navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10), 2021.

C.-Y. Ma, J. Lu, Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong. Self-monitoring navigation agent via auxiliary progress estimation. International Conference on Learning Representations , 2019.

C.-Y. Ma, Z. Wu, G. AlRegib, C. Xiong, and Z. Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra. Improving vision-and-language navigation with image-text pairs from the web. In European Conference on Computer Vision , 2020.

A. Moudgil, A. Majumdar, H. Agrawal, S. Lee, and D. Batra. Soat: A scene-and object-aware transformer for vision-and-language navigation. Advances in Neural Information Processing Systems , 34, 2021.

A. Parvaneh, E. Abbasnejad, D. Teney, J. Q. Shi, and A. Van den Hengel. Counterfactual vision-andlanguage navigation: Unravelling the unseen. Advances in Neural Information Processing Systems , 33, 2020.

Y. Qi, Z. Pan, S. Zhang, A. van den Hengel, and Q. Wu. Object-and-action aware model for visual language navigation. In European Conference on Computer Vision , 2020.

Y. Qi, Q. Wu, P. Anderson, X. Wang, W. Y. Wang, C. Shen, and A. van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Y. Qiao, Y. Qi, Y. Hong, Z. Yu, P. Wang, and Q. Wu. Hop: History-and-order aware pre-training for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Y. Qiao, Y. Qi, Y. Hong, Z. Yu, P. Wang, and Q. Wu. Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115(3), 2015.

S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, and K. Keutzer. How much can CLIP benefit vision-and-language tasks? In International Conference on Learning Representations , 2022.

A. Shrivastava, K. Gopalakrishnan, Y. Liu, R. Piramuthu, G. Tur, D. Parikh, and D. Hakkani-Tur. VISITRON: Visual semantics-aligned interactively trained object-navigator. In Association for Computational Linguistics , Dublin, Ireland, May 2022. Association for Computational Linguistics.

H. Tan, L. Yu, and M. Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In North American Chapter of the Association for Computational Linguistics , 2019.

J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Vision-and-dialog navigation. In Conference on Robot Learning , 2020.

J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In International Conference on Intelligent Robots and Systems . IEEE, 2017.

H. Wang, W. Liang, J. Shen, L. Van Gool, and W. Wang. Counterfactual cycle-consistent learning for instruction following and generation in vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

H. Wang, W. Wang, W. Liang, C. Xiong, and J. Shen. Structured scene memory for vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

H. Wang, Q. Wu, and C. Shen. Soft expert reward learning for vision-and-language navigation. In European Conference on Computer Vision . Springer, 2020.

S. Wang, C. Montgomery, J. Orbay, V. Birodkar, A. Faust, I. Gur, N. Jaques, A. Waters, J. Baldridge, and P. Anderson. Less is more: Generating grounded navigation instructions from landmarks. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.-F. Wang, W. Y. Wang, and L. Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

X. Wang, W. Wang, J. Shao, and Y. Yang. Lana: A language-capable navigator for instruction following and generation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 19048-19058, 2023.

X. E. Wang, V. Jain, E. Ie, W. Y. Wang, Z. Kozareva, and S. Ravi. Environment-agnostic multitask learning for natural language grounded navigation. In European Conference on Computer Vision . Springer, 2020.

S. Wu, X. Fu, F. Wu, and Z.-J. Zha. Cross-modal semantic alignment pre-training for vision-and-language navigation. In ACM International Conference on Multimedia , 2022.

J. Zhang, J. Fan, J. Peng, et al. Curriculum learning for vision-and-language navigation. Advances in Neural Information Processing Systems , 34, 2021.

Y. Zhang, H. Tan, and M. Bansal. Diagnosing the environment bias in vision-and-language navigation. In International Joint Conferences on Artificial Intelligence , 2021.

Y. Zhao, J. Chen, C. Gao, W. Wang, L. Yang, H. Ren, H. Xia, and S. Liu. Target-driven structured transformer planner for vision-language navigation. In ACM International Conference on Multimedia , 2022.

F. Zhu, Y. Zhu, X. Chang, and X. Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

W. Zhu, H. Hu, J. Chen, Z. Deng, V. Jain, E. Ie, and F. Sha. Babywalk: Going farther in vision-and-language navigation by taking baby steps. Association for Computational Linguistics , 2020.

W. Zhu, Y. Qi, P. Narayana, K. Sone, S. Basu, X. Wang, Q. Wu, M. Eckstein, and W. Y. Wang. Diagnosing vision-and-language navigation: What really matters. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Seattle, United States, July 2022. Association for Computational Linguistics.

Y. Zhu, Y. Weng, F. Zhu, X. Liang, Q. Ye, Y. Lu, and J. Jiao. Self-motivated communication agent for real-world vision-dialog navigation. In International Conference on Computer Vision , October 2021.

Y. Zhu, F. Zhu, Z. Zhan, B. Lin, J. Jiao, X. Chang, and X. Liang. Vision-dialog navigation by exploring cross-modal memory. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a Frequency enhanced Data Augmentation (FDA) technique to improve generalization/robustness to unseen environments in the vision-language navigation task. This work provides an initial analysis that shows high frequency information impacts performance. Results are provided on R2R, RxR, and CVDN datasets and are better than prior work when using the same models.\n\nSTRENGTHS:\n- The paper is clearly written and easy to follow.\n- Experimentally the authors demonstrate that the proposed FDA data augmentation technique improves performance on most of the models and validation splits.\n\nWEAKNESSES:\n- The premise of the paper is slightly confusing. The high and low frequency perturbed images essentially are different domains, so of course performance drops. Additionally, counter to what the writing of the paper details, low frequency information also helps on SR for the val seen split in Table 1, and the unseen results vary but < 1 point on SR.\n- FDA can hurt performance or doesn’t improve as much on seen compared to unseen results (e.g., R2R DUET, val seen).\n- Table 4. It seems like an obvious result that a method trained with FDA is going to do better on high frequency perturbed images, this table may not be necessary or at least does not need to be included in the main text. Are there downstream scenarios where this kind of perturbation would even realistically happen?\n- While many types of data augmentation are referenced in the related work section, the authors only experimentally compare to one (ENVEDIT) data augmentation work, and do not try baselines with other data augmentations that may affect similar information to the high frequency perturbation. Because of this, it feels the experiments are incomplete, and generally underwhelming due to the small SR performance changes.\n\nQUESTIONS:\n**Questions:**\n- Figure 1 and the original analysis that founds this paper begs the question: how much of the performance effect shown by high frequency perturbation is caused by loss of color in the resulting image? What if you made the image black and white, or the low frequency image black and white? The analysis feels lacking to truly demonstrate that it is frequency information that matters.\n- Did you try other methods of fusing the high frequency information into the training images? Or other training regimes instead of alternating the images per even and odd training steps per Eq. (6)?\n- Are all of the results for a single experiment run?\n\n**Comments and Suggestions:**\n- L38-L40, I suggest notating where in the Figure to look. I.e. \"Figure 1 (middle)\" or \"Figure 1 (right)\" instead of repeating just \"Figure 1\".\n- L59-64 is a very long single sentence, it should be broken up to several shorter sentences to make it easier to follow.\n- Figure 4: there is room within the figure to have an abbreviation legend since there are a ton of acronyms in the figure that are not defined in the figure caption - i.e., what is GHPF? FFT? etc. They are defined in earlier figures but it would be better to have them defined within the scope of this figure, too.\n- It would be better if figures appeared on pages they were referenced.\n- L111-118, I am not really sure what the difference is between the points being made in 2) and 3).\n- The notation in Eq. (2): It switched between using the $F$ notation for frequency and the $\\mathcal{F}$ notation for FFT. Isn’t the high and low frequency $H$ on the original image $I$, while $\\hat{H}$ should be on the interference image $\\hat{I}$? Right now $H$ and $\\hat{H}$ have the same definition in the equation.\n- L156 \"Lengt\" --> \"length\", L153 \"7k\" --> \"7,000\", no comma in \"2050\" --> \"2,050\"\n- L158 \"Process\" --> \"Progress\"?\n- L178 says GP performance increases by 3.6, but the table shows a **0.36** improvement.\n- L230 add a space after \"et al.\"; generally throughout paper this is an issue.\n\nLIMITATIONS:\nThe initial analysis prompting this work is not very convincing of the importance of high frequency information in images, and the paper does not compare to multiple data augmentation techniques. It could have additional baselines for simple image augmentations or other augmentation methods from prior work. This is particularly important because the performance improvement is not very large across the different models and datasets. Given the small performance differences, if the reported results are from a single experiment it is not convincing that the FDA approach is consistently effective.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a Frequency-enhanced Data Augmentation (FDA) for VLN models to enhance their ability to capture high-frequency information, thus improving their performance. First, at each step, they take the view as Reference Image and randomly sample an image as Interference Image. Then they transform these two images to the frequency domain space via FFT and mix the high-frequency components of two images. Finally, they obtain the augmented image via iFFT and alternately use original views’ images and augmented images to train the agent. the Experiments on different benchmarks reveal the proposed augmentation method is helpful to VLN models. However, I have some concerns about this paper. My detailed comments are as follows.\n\nSTRENGTHS:\n1. The authors take an analysis of frequency information domain by fuse images’ high-frequency or low-frequency components into original images, and find the importance of high-frequency for the agent’s better performance in VLN tasks.\n\n2. The authors propose a simple data augmentation method, Frequency-enhanced Data Augmentation (FDA), which needs no external models and does not add complexity. The proposed method enhances the models’ ability to capture necessary high-frequency information and has been demonstrated to improve the models’ performance on multiple datasets.\n\nWEAKNESSES:\n1. I wonder why the authors choose to incorporate the high-frequency components from randomly sampled interference images into the original image, rather than using a fixed interference image, or utilizing the high-frequency components of the image itself, or randomizing certain high-frequency components of the original image. It would be better if the author could discuss these approaches or include them in the ablation experiments.\n\n2. While the importance of high frequency compared to low frequency has been demonstrated by other experiments, it would be beneficial to conduct an ablation experiment utilizing low frequency for data augmentation under the same settings.\n\n3. Some novel SoTA models are not shown in the tables in this paper, such as Airbert [1] on R2R benchmark. It would be better for the authors to add these state-of-the-art models into the comparative experiments which would demonstrate the strengths and limitations of the proposed method more comprehensively. Also, it is better to add the brief introduction about the VLN works on continued environment, such as [2,3] for the sake of completeness\n\n4. There appear several low-level calculation/statistical errors that should have been avoided in the paper. For instance, on line 216, it should be \"4 and 3 points,\" and on line 222, it should be \"3.3 and 3.3 (2 and 1.7).\" Additionally, on line 178, it should be \"boost the GP performance by 0.36.\" The author should carefully review the data in the tables and the calculations presented throughout the text.\n\n\n**Minor issues\n\n1. In Equation 2, since Equation 1 already uses $F_{\\hat{I}}^{\\{rgb\\}}$ to represent the operation result of $\\mathcal{F}^{\\{rgb\\}}(\\hat{I})$, it would be better to use $F_{\\hat{I}}^{\\{rgb\\}}$ instead of $\\mathcal{F}^{\\{rgb\\}}(\\hat{I})$ in Equation 2.\n\n2. On page 5, line 156, “Path Lengt” should be “Path Length”.\n\n3. On page 6, line 178, “boost the GP performance by 3.6” should be “boost the GP performance by 0.36”.\n\n4. On page 8, line 216, “by 4 and 2 points” should be “by 4 and 3 points”.\n\n5. On page 8, line 222, “by 3.4 and 3.3” should be “by 3.3 and 3.3”.\n\n[1] Airbert: In-domain Pretraining for Vision-and-Language Navigation, ICCV 2021\n\n[2] Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation, NeurIPS 2022.\n\n[3] Cross-modal Map Learning for Vision and Language Navigation, CVPR 2022.\n\nQUESTIONS:\nThe manuscript is well organized and the proposed consistent improvement upon several strong baselines on 3 datasets. It would be better for the authors to evaluate the proposed FDA technique on other multi-model tasks, not limited to the VLN task.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors investigate the VLN task from a Fourier domain perspective, which aims to enhance the vision-language matching ability of agents. A Frequency-enhanced Data Augmentation (FDA) technique is proposed to improve the capability of capturing high-frequency information, which is seen as an important factor for training VLN agents. The experimental results show that baseline models can be further improved by adding the FDA technique.\n\nSTRENGTHS:\n- This work investigates the VLN task from an alternative perspective, i.e., the Fourier domain.\n\n- A reasonable FDA method is proposed to enhance the baseline model to capture high-frequency information without the need for additional parameters. \n\n- With the FDA technique, the performances of three transformer-based methods are further improved, confirming its effectiveness.\n\nWEAKNESSES:\n- The adopted HAMT, DUET, and TD-STP are all transformer-based agents. However, the LSTM-based agents with FDA also need to be verified. \n\n- In Table 2, DUET+FDA exhibits a significant performance drop on the val-seen set compared to the original DUET, e.g., SPL from 73.71\\% to 69.66\\%.\n\n- Lacking experimental comparison on the high-level instruction REVERIE dataset.\n\n- As shown in Table 2 of the supplementary material, different random seeds have a significant impact on the model performance. For example, seed-3 and seed-4 differ by 1\\% in SPL.\n\n- The writing needs to be further polished, e.g., the highest accuracy numbers in all tables should be bolded, and the title of section 3.4 should be 'Comparison with the State-of-the-art Methods'. \n\n- Missing numerous references such as [1-4].\n\n[1] Adaptive Zone-Aware Hierarchical Planner for Vision-Language Navigation. In CVPR 2023.\n\n[2] HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation. In TPAMI.\n\n[3] Reinforced Structured State-Evolution for Vision-Language Navigation. In CVPR 2022.\n\n[4] LANA: A Language-Capable Navigator for Instruction Following and Generation. In CVPR 2023.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper aims at investigating the Fourier domain of vision-and-language navigation (VLN). They propose Frequency-enhanced Data Augmentation (FDA) to improve the visual-text matching processing based on high-frequency visual information. Their method can achieve promising results efficiently on various VLN datasets, including R2R, RxR, and CVDN, with the same parameter size.

STRENGTHS:
+ This paper is well-written and easy to follow.
+ The aspect of the Fourier domain and lower-/higer- frequency visual representation is new in VLN.
+ Though data augmentation is not novel in VLN, the frequency-based sampling seems effective, which helps achieve new/competitive SOTA on various datasets.
+ The sensitive analysis in Figure 2 and Table 1 is a good intro convincing.

WEAKNESSES:
+ Though the performance looks effective, what is the motivation to rely on Fourier representation instead of the widely-used learned visual features (maybe I missed it, but I cannot find a clear answer in the draft)?
+ Does visual information get lost during the Fourier transform? How to overcome this issue during the navigation process?
+ A detailed analysis of data augmentation should be considered.  For example, how much data can lead to how much improvement? If we keep training with more data, does the navigator works better and better?
+ There should include some qualitative examples to demonstrate how Fourier features can actually lead to the final goal. The comparison with baselines is required, and both successful/failed cases should be involved to guide future research.

QUESTIONS:
Please see Weakness

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the sensitivity of existing state of the art VLN models to high frequency perturbations in the image features, and identifies that this could be a way to improve the models performance by making them more robust. It also provides evidence to show that adding high frequency information explicitly to the models may improve their performance. Taken together, this implies that if the model can be made more sensitive to changes in high frequency components while being able to discriminate which ones are relevant to the instruction, it could improve the generalization power of the models. The paper then proposes a data augmentation approach which involves replacing high frequency components of images with those of randomly selected other images, and training the model on this data. The resulting models degrade less in performance when evaluated on perturbed images and improve performance on standard benchmarks compared to the baseline models. \n\n-------\nEdit: Updated score after clarifications from the rebuttal!\n\nSTRENGTHS:\n## Originality\nAs far as I know, this is a new exploration in terms of data augmentation by perturbing visual features in the frequency domain\n\n## Quality, Clarity, Significance\nThe paper provides an interesting exploration and provides evidence that supports their hypothesis. It is well written and easy to follow. The paper provides experiments based on three state of the art models, and provides a lightweight method to improve models' performance.\n\nWEAKNESSES:\n## Clarity\n* Lacking in implementation details regarding choice of model components such as image encoder, text encoder, etc. \n\n## Discrepancies in numbers reported\n* The numbers reported in the table for RxR seem to not be consistent from those in literature (for example in terms of NDTW, HAMT on RxR Val seen achieves 65.3 and 63.1 on val unseen according to their paper [1] as opposed to the 63.1 and 59.94 reported here. \n* In Table 5 (Comparison to EnvEdit), the numbers reported for HAMT + EnvEdit  for SR and SPL are 67.1 and 62.84 as opposed to 68.9 and 64.4 in the EnvEdit paper [2] \n* Line 177: Text states that the improvement in GP is by 3.6 points whereas the table mentions 0.36 points.\n \nCould the authors please explain these discrepancies?\n\n## Minor issues / typos\n* Line 156: Path Lengt -> Path Length \n* Line 158: Goal Process -> Goal Progress\n* Table 4 caption. Please state more clearly that this evaluation set is non-standard R2R, and is created as part of this paper\n\n\n[1] History Aware Multimodal Transformer for Vision-and-Language Navigation. Shizhe Chen et al.\n[2] ENVEDIT: Environment Editing for Vision-and-Language Navigation. Jialu Li et al.\n\nQUESTIONS:\n* What happens if you just do an interpolation between one of the channels from a random image, instead of doing it in the fourier domain? This could be additional proof that augmentation by manipulating in the fourier domain is most effective.\n* According to Eq 3, the resulting image is likely to have perturbed features for more most of the channels - how did you decide the ratio for keeping original vs replacing with the distractor image features. Is this applied for all the views and all the images in the trajectory?\n* What happens if you train only with the augmented data? what was the reason you chose the set up in (6)\n* Since the edge and corner features are what this paper finds to be specifically useful, and given that the image feature extractors based on CNNs also extract such features in early layers, have the authors considered or tried reusing intermediate features from the image extractor to emphasize these features?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}