SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Performances on Navigation Scenes with High-Frequency Interference. As analyzed before, existing VLN methods are severely limited in high-frequency perturbed navigation scenes. Table 5 shows more details. We observe that the baselines can perform well under the normal navigation scenes. Setting 1: Then, we introduce high-frequency interference, whose corresponding spatial image is the navigation views randomly sampled from MP3d [6], into seen/unseen navigation views as described in the method section to get the high-frequency perturbed unseen navigation scenes. Navigating under such scenes, all models have experienced a significant decrease in SR metric. Taking TD-STP as an example, its SR decreases by as much as 27.23 and 24.52 on validation seen and validation unseen, respectively. The fact that the models are easily and severely affected by high-frequency interference suggests their lack of ability to effectively identify and capture the necessary high-frequency information based on the instructions. However, with the assistance of our FDA, all models show notable improvement in performance. HAMT, DUET, and TD-STP see SR metric increases of 23.5, 18.41, 19.96 on validation seen split, and 13.92, 14.69, 16.98 on validation unseen split. Setting 2: To distinguish the source of interference from the training phase, we have also sampled high-frequency interference from the 1000 classes of ImageNet [48]. Due to the out-of-domain interference, we find that the SR of baseline models is significantly more affected under this setting. Taking TD-STP as an example, its SR on validation seen and validation unseen decreases by 35.75 and 27.55, respectively. In such challenging scenes, our FDA-enhanced models whose training data only contains high-frequency and low-frequency information from Mp3d, perform remarkably well, improving the SR of HAMT, DUET, and TD-STP by 17.83, 19.2, and 22.53 on validation seen, and by 16.69, 19.08, and 19.03 on validation unseen, respectively. In most cases, the navigation performances of the models have gained even greater improvements compared to the first setting. These two settings illustrate FDA-enhanced models excelling in both the seen/unseen perturbed images (val seen/unseen) and the seen/unseen perturbation images (MP3d/ImageNet). This serves as compelling evidence of our method's remarkable capability to recognize and capture essential high-frequency information to boost navigation performance.
Table 6: Comparison with the visual and textual data augmentation methods on R2R task.
Validation Unseen
TL	NE↓	SR↑	SPL↑
Visual Data Augmentation	HAMT	11.50	3.57	65.64	60.15
HAMT+ENVEDIT	11.15	3.48	67.31	62.84
HAMT+FDA	11.86	3.44	67.52	62.16
TD-STP	13.53	3.28	69.65	62.97
TD-STP+ENVEDIT	14.70	3.20	69.82	62.55
TD-STP+FDA	13.68	3.02	71.78	64.00
Textual Data Augmentation	Follower	-	-	38.3	-
Follower+FDA	-	-	45.6	-
Follower+Speaker	-	-	40.4	-
Follower+Speaker+FDA	-	-	46.3	-
TD-STP-	13.78	3.62	65.73	58.72
TD-STP+FDA	14.77	3.44	66.96	59.72
TD-STP+PREVALENT	13.53	3.28	69.65	62.97
TD-STP+PREVALENT+FDA	13.68	3.02	71.78	64.00
Comparison with Existing Data Augmentation Methods. Table 6 shows the comparison with the visual data augmentation method ENVEDIT $^2$  [33], and textual data augmentation methods Speaker [15] and PREVALENT [20]. Compared to ENVEDIT which aims to increase the diversity of the environment by using GANs to alter environment style, our method, which also serves as a visual data augmentation method, demonstrates a clear superiority, with SR of HAMT/TD-STP increased by 0.21/1.96 on validation unseen, respectively. In comparison with the textual augmentation methods, the results indicate that our FDA method demonstrates consistent improvements on both baseline models Follower and TD-STP $^{-3}$ . Additionally, it effectively complements the two textual augmentation methods, further enhancing cross-modal navigation performance. Our FDA has demonstrated superior performance without relying on external models such as visual/textual generative models. As a result, our method could greatly reduce the complexity of data augmentation work in VLN field.
3.4 Comparison with the State-of-the-art Methods
We compare our method with SoTA methods on R2R, RxR, CVDN and REVERIE tasks.
On R2R task, Table 7 shows our method achieves the best performance on both validation unseen and test splits. Compared to the DUET model which has been fed with the extra object features not included in our method, the SR/SPL of our method still outperforms it by 2/4 points on validation unseen. Compared to TD-STP, Our method shows SR/SPL improvements of 2/1 and 3/2 points on validation unseen and test splits. On RxR task, Table 8 shows the comparisons between our method and previous methods. The results show that our method outperforms the previous SoTA HAMT with SPL/nDTW improved by 1.8/2.1 and 0.4/1.5 on validation unseen and test splits, respectively. On CVDN task, our method also shows superior performance over previous methods in Table 9. Compared to HAMT, the previous SoTA, our method gains a GP improvement of 0.05/0.36/0.25 on validation seen, validation unseen and test splits, respectively. On REVERIE task, as shown in Table 10, our method outperforms the previous SoTA DUET with the increase of 2.92/2.98/3.49/3.30 and 1.21/0.49/2.03/1.47 in SR/SPL/RGS/RGSPL on validation unseen and test splits, respectively.
The remarkable performance analyzed above has demonstrated the excellence of our method in enhancing cross-modal navigation, as well as the outstanding adaptability across different tasks.
Vision-and-Language Navigation. Anderson et al. [5] are the first to introduce the Vision-and-Language Navigation, where an agent is asked to follow an English instruction in R2R to navigate.
2Unlike ENVEDIT, our method operates without external models. To ensure fairness, we compare our method with ENVEDIT's Editing Style, which relies on the fewest external models.
${}^{3}$  TD-STP  ${}^{ - }$  denotes the TD-STP model after excluding data augmentation PREVALENT.
Table 7: Comparison with SoTA methods on R2R task.
Validation Unseen	Test Unseen
TL	NE↓	SR↑	SPL↑	TL	NE↓	SR↑	SPL↑
Seq2Seq [5]	8.39	7.81	22	-	8.13	7.85	20	18
Speaker-Follower [15]	6.62	-	35	-	14.82	6.62	35	28
SMNA [39]	-	5.52	45	32	18.04	5.67	48	35
RCM [58]	11.46	6.09	43	-	11.97	6.12	43	38
EGP [12]	-	4.83	56	44	-	5.34	53	42
EnvDrop [51]	10.70	5.22	52	48	11.66	5.23	51	47
PREVALENT [20]	10.19	4.71	58	53	10.51	5.30	54	51
RelGraph [23]	9.99	4.73	57	53	10.29	4.75	55	52
RecBert [25]	12.01	3.93	63	57	12.35	4.09	63	57
Airbert [19]	11.78	4.01	62	56	12.41	4.13	62	57
DUET [10]	13.94	3.31	72	60	14.73	3.65	69	59
DUET* [10]	12.95	3.38	70	60	-	-	-	-
HAMT [9]	11.46	2.29	66	61	12.27	3.93	65	60
HAMT* [9]	11.50	3.57	66	60	-	-	-	-
TD-STP [64]	-	3.22	70	63	-	3.73	67	61
TD-STP* [64]	13.53	3.28	70	63	14.71	3.79	66	60
Ours (TD-STP + FDA)	13.68	3.02	72	64	14.76	3.41	69	62
Table 8: Comparison with SoTA methods on RxR task.
Validation Unseen	Test Unseen
nDTW↑	SDTW↑	SR↑	SPL↑	nDTW↑	SDTW↑	SR↑	SPL↑
Multilingual Baseline [29]	38.9	18.2	22.8	-	36.8	16.9	21.0	18.6
Monolingual Baseline [29]	44.5	23.1	28.5	-	41.1	20.6	25.4	22.6
SAA [34]	52.7	33.0	40.0	36.0	46.8	29.1	35.4	31.6
EnvDrop + CLIP-ViL [49]	55.7	-	42.6	-	51.1	32.4	38.3	35.2
HAMT [9]	63.1	48.3	56.5	56.0	59.9	45.2	53.1	46.6
HAMT* [9]	64.3	49.6	58.1	54.0	60.5	47.0	55.5	48.4
Ours (HAMT+FDA)	66.4	51.6	59.8	55.8	62.0	47.3	55.5	48.8
After that, model architecture [58, 4, 24, 12, 23, 25, 55, 1, 34, 9, 42, 10, 37, 7, 59, 2, 3, 22], strategy design [28, 40, 39, 17, 8, 18], training paradigm [66, 62, 54, 38], loss design [65, 36, 32], reward shaping [27, 56, 21], pretraining [35, 20, 41, 46, 49, 61, 47] and probing experiments [26, 63, 67] have been widely explored. Then, a more challenging task RxR [29] containing multilingual instructions and longer trajectories is proposed. A series of works [34, 32, 21, 9] have explored the above and make significant progress. CVDN [52] is a task asking an agent to navigate following multi-turn human-to-human dialog. In addition to requiring the agent to have cross-modal understanding, it also demands a certain level of reasoning ability. Works [50, 20, 69, 68, 60, 9] focusing on that have been investigated to advance the development of this field. While these works have conducted extensive exploration, they have almost neglected the frequency domain that our work has demonstrated to be crucial for VLN navigation.
Table 9: Comparison with SoTA methods on CVDN task.
Val Seen	Val Unseen	Test Unseen
PREVALENT [20]	-	3.15	2.44
CMN [69]	7.05	2.97	2.95
VISITRON [50]	5.11	3.25	3.11
SCoA [68]	7.11	2.85	3.31
MT-RCM+EnvAg [60]	5.07	4.65	3.91
HAMT [9]	6.91	5.13	5.58
HAMT* [9]	7.43	5.03	5.43
Ours (HAMT+FDA)	7.48	5.39	5.68
Table 10: Comparison with SoTA methods on REVERIE task.
Validation Unseen	Test Unseen
TL	OSR↑	SR↑	SPL↑	RGS↑	RGSPL↑	TL	OSR↑	SR↑	SPL↑	RGS↑	RGSPL↑
RecBERT [25]	16.78	35.02	30.67	24.90	18.77	15.27	15.86	32.91	29.61	23.99	16.50	13.51
Airbert [19]	18.71	34.51	27.89	21.88	18.23	14.18	17.91	34.20	30.28	23.61	16.83	13.28
HAMT [9]	14.08	36.84	32.95	30.20	18.92	17.28	13.62	33.41	30.40	26.67	14.88	13.08
DUET [10]	22.11	51.07	46.98	33.73	32.15	23.03	21.30	56.91	52.51	36.06	31.88	22.06
DUET* [10]	18.76	48.59	44.65	32.92	28.57	21.01	16.31	52.77	48.41	35.96	28.31	20.61
DUET+FDA	19.04	51.41	47.57	35.90	32.06	24.31	17.30	53.54	49.62	36.45	30.34	22.08
Data Augmentation in Vision-and-Language Navigation. In terms of text augmentation, Fried et al. [15], Wang et al. [57] and Dou et al. [14] use instruction generators to generate pseudo-instructions according to sampled trajectories. In terms of vision augmentation, Fu et al. [16] learn a sampler to sample the challenging paths to enhance the agent's ability for handling them. Tan et al. [51] use environment dropout on seen environments to mimic unseen environments and generate the pseudo-instructions for it with instruction generator. Parvaneh et al. [43] learn a model to generate counterfactuals on seen environments to simulate unseen environments. Li et al. [33] adopt GAN to transfer the environment style for increasing the diversity of the environment. However, all of these works focus on the augmentation in the spatial domain and have ignored the high-frequency information which is crucial for cross-modal matching. Moreover, they have to rely on extra generator models, which require intricate design and will increase the model burden.
Data Augmentation in Robotics. Domain randomization is a form of Data Augmentation (DA) that has shown effectiveness in transferring reinforcement learning policies from simulation to the real world [53]. In contrast, Cobbe et al. [11] propose a modification to Cutout [13], where multiple rectangular regions of varying sizes are masked with a random color for each observation. Lee et al. [31] use a randomized (convolutional) neural network that adds random perturbations to the input observations, which helps the trained agents learn more robust features that are invariant across diverse and randomized environments. RAD [30] investigates the general DAs for robotics on both pixel-based and state-based inputs. In this paper, we propose a simple and effective frequency-enhanced DA for the cross-modal task VLN, a fundamental task in robotics.
Shifting from the perspective of spatial domain previous works mainly focus on, we provide a first-of-its-kind in-depth analysis of the frequency domain information in VLN tasks. Our findings have demonstrated the importance of high-frequency information in enhancing navigation performance. In light of the findings, we further propose the Frequency-enhanced Data Augmentation (FDA) method, a simple and effective data augmentation technique to boost a model's capability to discern and capture essential high-frequency information without the need for additional auxiliary generative models. Our method on a range of VLN tasks, including R2R, RxR, CVDN and REVERIE, achieves notable performances and shows great adaptability across different VLN models. We hope our work can offer a practical solution and bring new insights to the VLN research community.
Limitations and Future Work. Our work focuses on enhancing the model's general ability to recognize and capture essential high-frequency information. However, we have not yet explored the fine-grained correlation between frequency and specific scenes or categories. This area of investigation remains an avenue for future exploration.
This project is supported by National Key R&D Program of China (2022ZD0117900), National Natural Science Foundation of China (62236010 and 62276261), Key Research Program of Frontier Sciences CAS Grant No. ZDBS-LYJSC032, the National Research Foundation, Singapore, under its Medium Sized Center for Advanced Robotics Technology Innovation, and the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0006).
[61] S. Wu, X. Fu, F. Wu, and Z.-J. Zha. Cross-modal semantic alignment pre-training for vision-and-language navigation. In ACM International Conference on Multimedia, 2022.
[62] J. Zhang, J. Fan, J. Peng, et al. Curriculum learning for vision-and-language navigation. Advances in Neural Information Processing Systems, 34, 2021.
[63] Y. Zhang, H. Tan, and M. Bansal. Diagnosing the environment bias in vision-and-language navigation. In International Joint Conferences on Artificial Intelligence, 2021.
[64] Y. Zhao, J. Chen, C. Gao, W. Wang, L. Yang, H. Ren, H. Xia, and S. Liu. Target-driven structured transformer planner for vision-language navigation. In ACM International Conference on Multimedia, 2022.
[65] F. Zhu, Y. Zhu, X. Chang, and X. Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[66] W. Zhu, H. Hu, J. Chen, Z. Deng, V. Jain, E. Ie, and F. Sha. Babywalk: Going farther in vision-and-language navigation by taking baby steps. Association for Computational Linguistics, 2020.
[67] W. Zhu, Y. Qi, P. Narayana, K. Sone, S. Basu, X. Wang, Q. Wu, M. Eckstein, and W. Y. Wang. Diagnosing vision-and-language navigation: What really matters. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, July 2022. Association for Computational Linguistics.
[68] Y. Zhu, Y. Weng, F. Zhu, X. Liang, Q. Ye, Y. Lu, and J. Jiao. Self-motivated communication agent for real-world vision-dialog navigation. In International Conference on Computer Vision, October 2021.
[69] Y. Zhu, F. Zhu, Z. Zhan, B. Lin, J. Jiao, X. Chang, and X. Liang. Vision-dialog navigation by exploring cross-modal memory. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.