Frequency-Enhanced Data Augmentation for Vision-and-Language Navigation
Authors: Keji He
Chenyang Si
Zhihe Lu
Yan Huang
Liang Wang
Xinchao Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision-and-Language Navigation; High-Frequency; Data Augmentation

Abstract: Vision-and-Language Navigation (VLN) is a challenging task that requires an agent to navigate through complex environments based on natural language instructions. In contrast to conventional approaches, which primarily focus on the spatial domain exploration, we propose a paradigm shift toward the Fourier domain. This alternative perspective aims to enhance visual-textual matching, ultimately improving the agent's ability to understand and execute navigation tasks based on the given instructions. In this study, we first explore the significance of high-frequency information in VLN and provide evidence that it is instrumental in bolstering visual-textual matching processes. Building upon this insight, we further propose a sophisticated and versatile Frequency-enhanced Data Augmentation (FDA) technique to improve the VLN model's capability of capturing critical high-frequency information. Specifically, this approach requires the agent to navigate in environments where only a subset of high-frequency visual information corresponds with the provided textual instructions, ultimately fostering the agent's ability to selectively discern and capture pertinent high-frequency features according to the given instructions. Promising results on R2R, RxR, CVDN and REVERIE demonstrate that our FDA can be readily integrated with existing VLN approaches, improving performance without adding extra parameters, and keeping models simple and efficient. The code is available at https://github.com/hekj/FDA.

Frequency-enhanced Data Augmentation for Vision-and-Language Navigation
Keji He1.23
Chenyang S
Zhihe Lu Yan Huang!.2 Liang Wang" 1.2 Xinchao Wang" =3
Center for Research Intelligent Perception and Computing
National Key Laboratory Multi-modal Artificial Intelligence Systems Institute of Automation, Chinese Academy of Scienees
School of Artificial Intelligence, University Chinese Academy Sciences National University of Singapore
Nanyyng Technological University
mejj.heecripac.oo aa.ac.aaac....... chenyang siintt.......... edu.sg
[zhihelu, xinchao) enu..edu.s....
Cyhuang, wangliang''nlpr. ii..aa.a.
Abstract
Vision-and- Lan--uuag--L Navigation (VLN) is challenging task that requires an agent navigate through complex environments based natural language instructions. In contrast to conventional approaches, which primarily focus on the spatial domain exploration, we propose paradigm shift toward the Fourier domain. This alterna- tive perspective aims to enhance visual-textual matching, ultimately improving the agent's ability understand and execute navigation tasks based the given instruc- tions. In this study, we first explore the significance high-frequency information in VLN and provide evidence that it is instrumental in bolstering visual-textual matching processes. Building upon this insight, we further propose a sophisticated Data Augmentation (FDA) technique im-
and versatile Frequency-enhant prove the VLN model's capability of capturing critical high-frequency information. Specifically, this approach requires the agent to navigate in environments where only subset of textual instructions, ultimately fostering the agent's 'ss ability selectively discern and capture pertinent high-frequency features according to the given instructions. Promising results on R2R, RxR, CVDN and REVERTE demonstrate that our FDA can be readily integrated with existing VIN approaches, improving performance without adding extra parameters, and keeping models simple and efficient. The code is available at
visual information corresponds with the provided
Introduction
Assionaand-Languag Navigation (VLN) requires an embodied agent to navigate in complex envi- ronments following human instructions. This research area has garnered substantial attention due its potential applicability in human-robot interaction contexts, such as service and rescue robotics.
Despite the sigmificant udvancements [2] 23 25] 34 42 10 37 64 [50] 20 68 60) achieved in devising deep learning models for VLN, the task remains arduous, primarily due to the complexities involved attaining precise visual-textual matching.
Recent studies in the field of VLN have focused on enhancing embodied agents' ability to navigate complex environments using natural language instructions. These studies aim develop robust and efficient deep learning models that can tackle the challenges of visual-textual matching, thereby
"Corresponiing author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure I: Examples of the high-frequencr and low- wwwfrrquenccy information. Blue background part is the high-frequency spectrum and high-frequency information in spatial domain after inverse Fourier Transform. Orange background part is about these two contents of the low-frequency information.
improving the agent's capacity to understand and execute navigation tasks based on given instructions.
Some of the recent studies have investigated attention mechanisms
[8
25
9 (4 detection models
44 1 [10, and fine-grained trajectory-ine rajectory-instruction instructionnnn pairs 666 24
enhance cross-moaal matching
from the spatial domain perspective.
Validation Seen
Validation Unseen
Normal
No mal Ut-Purturbed re-Porturbdd
Pedturtee Porturbed
80
55
80-
76.3
75.02
64.15
63.86
64
00
00
50.15
53.00
50
43.6 86
.4.6
40
30
30
DUET HANT
Figure 2: Sensitivity analysis of benchmark methods to high and low-frequency information, i.e., HAMT (9) DUET [10), and TD-STP [64]. Normal denotes the normal navigation scenes. HF- Perturbed and LF-Perturbed denote the navigation scenes whose high-frequency and low-frequency have been perturbed, respectively.
Beyond the spatial domain perspective, in this paper, we are interested in investigating the effective- ness of other domain perspectives for VLN. We present paradigm shift toward the Fourier domain to enhance visual textual matching, research area that has received limited prior investigation. Specifi- cally, high-frequency and low-frequency information pertains to distinct components of an image when analyzed within the Fourier domain. High-frequency information encompasses rapid changes, fine details, edges, and texture patterns, as illustrated in Figure | (part of the blue background). Conversely, low- www-rreuency information signifies the overall structure and global features of an image, including large shapes and smooth color gradients, thereby capturing the general appearance and layouts, as shown in Figure (part of the orange background).
We simply investigate the sensitivity of benchmark methods to low and high-frequency information by perturbing the freeuencyy or frequeryy components in images. Three powerful baseline models, i.e., HAMT (9. DUET [10, and TD-STP [64), are used to analyze the significance of low/high-frequency information on both R2R validation seen and unseen splits, wherein the navigation views are disrupted in the Fourier domain. As Shown in Figure 2 the three models maintain a relatively high Success Rate (SR) under low-frequency perturbations. However, the SR on validation and unseen splits decrease markedly under the high- nigh-frequency perturbation navigation process, exhibiting lower SR metries compared to low- www-freueency perturbations. For example, on the validation seen dataset, the SR results for HAMT 19, DUET [10], and TD-STP [64] decrease by 31.44, 27.52, and 27.23, respectively, when affected by high-freeg frequency perturbations. These findings reveal that VLN approaches exhibit marked sensitivity to uighfrreuency information.
In light of these observations, we hypothesize that high-frequency information from images may be of paramount importance for cross-modal navigation tasks. Through extensive experiments, we ascertain that high-frequency information indeed serves a pivotal role enriching visual representa- tions, thereby contributing marked enhancement in the navigation performance of models when

---

faced with novel environments. To efficiently leverage the benefits of high-freee fegg-ffeuuency information, we further propose a Frequency-enhane
Data Augmentation (FDA) tailored for VLN, simplistic yet efficacious approach to augment the model's s ability to capture essential high-freqnency information. Specificallyy, the FDA method utilizes the Discrete Fourier Transform on navigation views, extracting high and low-frequency components from RGB channels. It replaces part of the high-frequency components with those from an interference image to introduce high- freeefryy perturbations. Aug- mented data is obtained by applying inverse Fourier Transform to the combination of the perturbed gigh-freuuency and original low-frequency components. By training the agent to match original instructions with both the original and augmented navigation views concurrently, the FDA method encourages the agent to hone its ability to capture the relevant high-frequency information that best aligns with the given instruction. This, in turn, enhances the agent's overall performance in VLN tasks, making it more adept at identifying and leveraging critical high-frequency features.
Leveraging FDA during training, existing SoTA methods can achieve significant performance im- provements without the need for additional parameters. This demonstrates the effectiveness of FDA in enhancing visual-textual matching capabilities in Vision-and aiion-and-annuage Navigation (VLN) tasks while maintaining model simplicity and efficiency. Our contributions are summarized as follows: 1) we conduct first-of-its-indd in-depth analysis of frequency domain information in VLN task, highlighting the importance of feghffreyeeccy information in improving navigation performance. This novel perspective offers fresh research opportunities for the community to explore and enhance VLN models. 2) We further introduce a simple, effective data augmentation method, Frequency- enhanced Data Augmentation (FDA), which enhances model's ability to discern and capture essential high-frequency information without added complexity, offering practical solutions for the research community. 3) Our method has achieved superior results various cross-modal navigation tasks, ie., R2R, RxR, CVDN and REVERIE, and shown great adaptability across different models.
2 Frequency Perspective for Vision-and-Languaae Navigation
In this section, we first introduce the VLN problem formulation. Then we explore the potential value of high-frequency and low-frequency information for VLN. Finally, we present the Frequency- enhanced Data Augmentation (FDA) for enhancing model's ability to capture essential high-frequency information.
2.1 Problem Formulation
According the VLN setup, an agent navigates in an in-door environment E = (p- 12: yl [l with numbers of preset points p in it, following a human instruction T = {wi, W2, w Assuming at stept, I, the agent standing on the point På can perceive the surrounding panoramic vision O, = (of ) around it which is composed by 36 discrete observations of. Each observation of = (I o of is the combination of the kiph. view T with its relative heading 0f and elevation of to the oriented view. The neighboring navigable points N (pi) distribute among some of these views. The agent selects the next point from these neighboring points N (pi) based on the correlation between instruction T and the observations of where N (p:) locate in. Then, the agent will be teleported to that selected point. This navigation continues until the agent predicts a stop action exceeds preset step threshold. Navigation is considered a success when the agent stops within meters of the target destination.
k=
2.2 High Frequency or Low Frequency: Which Benefits VLN Performance?
Considering the observations of Figure we posit that high-frequency information in images may be of paramount importance for cross-modal navigation tasks. In order to validate this hypothesis, we conduct straightforward experiment that entails the fusion of original image features with their corresponding high-frequency or low-frequency components. These amalgamated features subsequently serve as input for the navigation network during training and testing, as shown in Figure 3 The results on TD-STP [64] are present in Table I We can observe the integration of high- frequecyyy features leads to an improvement in navigation accuracy, such as SR increasing from 69.65 to 70.37 for unseen environments. Conversely, incorporating low- lowffrequency features appears to have an adverse effect, resulting in a decrease in accuracy with SR dropping from 69.65 68.75. The above observation highlights the critical role of tigh-frequency information in cross-modal navigation. This is because 1) high-frequency information encompasses fine details such as edges, corners, and texture patterns. These details are essential for accurately identifying and differentiating objects, scenes, and locations, which can lead to more effective visual-textus matching and better
3

---

IFFT
IFFT
OFT-GIPE
Enceder
Encoder
Emooder
Visual
Visual
Navigitor
Navigator
INSTRUCTION upstairs,
Emooder
INSTRUCTION
Testual
upstairs,
stop
stop
(a) Navigator with high-frequene frequency information as extra input
(b) Navigator with low cow-frequency information as extra impot
Figure 3: Integrate high- frequency and low- low-frequency information into the navigator (TD-STP). FFT and iFFT denote the Fourier Transform and inverse Fourier Transform. GHPF and GLPF denote the Gaussian High-Pass Filter and Gaussian Low-Pass Filter, respectively. The green background part is the pipeline of the baseline model (TD-STP).
Table 1:
Results R2R task: integrating high high-frequency (HF) and low- freeueegynnyy (LF) information.
Validation Seen
Validation Unseen
TL
NE1
SRt
SPL1
TL.
NEІ.
SR^
SPL1
TD-STP
12.66
2.53
76.30
71.70
13.53
3.28
69.65
62.97
TD-STP+LF
12.83
2.42
76.49
71.36
14.29
3.30
68.75
61.96
TD-STP+HF
12.29
2.30
78.75
73.78
13.68
3.14
70.37
63.62
navigation performance. 2) Models trained with high-frequency information tend to be more robust environmental variations and exhibit greater generalization ability unseen environments, as the model learns to focus on a more diverse set of features, rather than just memorizing specific low-frequency, global patterns present in the training data.
2.3 Frequency-enbancee Data Augmentation
To systematically capitalize on the advantages of high-frequency information, we subsequently intro- duce a Trequency-enhanced Data Augmentation (FDA) method, specifically designed for VLN tasks. This approach both straightforward and effective bolstering the model's capacity to apprehend and incorporate crucial ligh-frequenc, information. As illustrated in Figure 4 the reference image is a navigation view corresponding to the navigation instruction T ("Go forward, pass by the refrigerator, turn left behind the dining table, go straight through the doorway...") The interference image î is another navigation view sampled from the Matterport3d (Mp3d) dataset (6) randomly. To prevent information leakage, all interference images are sampled from and image from the validation unseen and splits is used. First, we transform these images to the frequency domain space via FFT, resulting two frequency spectrums F (((gb)) and | prgb))
seen splits,
F(rab) Flrsb)(I), F(rab) Flrsb) (f)
where Firgb) denotes the Fourier Transform on RGB color channels. Afterward, we apply High-Pass and Low-Pass Gaussian Filters on the two frequency spectrums obtain the reference high-frequency H (rgb), reference low-frequency L frgbt and interference high-frequency f/ (rgb),
Hrgb) Gn @ (robb)
[(rsb) Gi @ (((a)) fr(reb) Gn. o (rgb)
where Gn and Gj denote the Gaussian High-Pass Filter (GHPF) and Gaussian Low-Pass Filter (GLPF), and O is clement-win wise multiplication. Then, mix the high-frequee freeuuereeeeyyy components of these two

---

Interference Image
INSTRUCTION
Go forwurd, pass by the refrigerator, left behind the dining table, go straight through the dourway...
FFT
H
Sample
H
GHPF
f f
Mix
ALIGN
H
H
Reference Image
FFT
H
H
Mix
H
AM
iFFT
Augmented Image
GHPFF
&
L7
Lr
GLPF
1.P
L
L
Figure 4: Our approach mixes the interference high-frequency and the reference high-frequency together. Then the augmented image obtained by applying inverse Fourier Transform on combination of the mixed high-frequency and reference low-frequency. Finally, the augmented image containing only a portion of the reference high-frequency aligned to the instruction to encourage agent discern and capture the reference high-frequence information. FFT and iFFT represent the Fourier Transform and inverse Fourier Transform. GHPF and GLPF denote the Gaussian High-Pau Pass Filter and Gaussian Low-Pas. Filter, respectively.
images. Specifically, for each RGB channel of the reference image, there is a certain probability that its high-frequency component replaced by the interference high-frequency from the same channel:
H fr.
probability of 1/3
H mix = Mix(H A =
others : CE (r,g.b)
(3)
H trgb] Mir (H((H)))))), fr(rab))
(4)
where H [rggg mix is the mixed high frequency. We combine it with the reference low frequency L (rob) and then apply iFFT to obtain the frequency-enhancee augmented image Imi::
Imi (FF((F)))))) mix
JF
(rgh) (rgh) miz
(rgh)).
(5)
Finally, the original image and augmented image micc share the same textual instruction label T and are used alternately to train the agent during the training phase:
Navigator Loss(1, T, T,0), Navigator Loss( mixx; T, 0), even-numbered step
odd-numbered step
((
(6)
where (()) denotes the navigation loss considering both the original image / and frequency-enhance augmented image Imiux denotes the parameters of the navigator.
Experiments
3.1 Datasets
The visual environments are based on the photo-realistic dataset Matterport3d (Mp3d) (6]. There are a total of 90 houses, with 61, 11, and 18 houses allocated for
seen, validation unseen, and test splits, respectively. Four datasets containing the instruction-traacctory pairs have been adopted: R2R [5), RxR [29]. CVDN [52] and REVERIE [45] R2R includes 7, 7,189 trajectories and each trajectory has corresponding English instructions. RxR has 16,522 trajectories and 126,069 multilingual (English, Hindi and Telugu) instructions in total. Compared to R2R, its trajectories are longer and instructions are more complex. CVDN based on human-to-human dialogs. The multi-turn dialogs instruct the agent to navigate to the destinations. There are 2,050 dialogs and over 7,000 trajectories in this dataset. REVERIE contains total of 70, high-level instructions, including descriptions of navigation destinations and the objects to be found.

---

3.2
Evaluation Metries
According to the conventions of each task, we adopt a total of ten evaluation metrics, namely Success Rate (SR), Success Rate weighted by Path Length (SPL), Trajectory Length (TL). Navigation Error (NE), normalized Dynamic Time Warping (nDTW), Success weighted by normalized Dynamic Time Warping (SDTW), Goal Progress (GP), Navigation Oracle Success Rate (OSR), Remote Grounding Success Rate (RGS) and Remote Grounding Success Rate weighted by Navigation Path Length (RGSPL). SR measures the proportion of successful navigation. SPL is trade-off metric between SR and TL. TL is the average trajectory length. NE measures the average distance between the final position and the destination. nDTW measures the fidelity between the predicted trajectory and the groundtruth trajectory. SDTW combines the evaluation SR with nDTW. GP measures the progress toward the goal. OSR measures the proportion of successful arrivals at or passage through the destinations. RGS represents the proportion of successful groundings to the correct object. RGSPL is trade-off metric between RGS and TL. R2R uses the SR. SPL, TL and NE metrics. RxR adopts the SR. SPL, nDTW and SDTW metrics. CVDN uses the GP metric. REVERIE adopts the TL, OSR, SR, SPI., RGS and RGSPL metrics.
3.3 Ablation Study
Table 2: Models enhanced by the FDA method on R2R task.
Validation Unseen
TL
NEG
SR*
SPL
HAMT
11.50
3.57
65.64
60.15
HAMT+FDA
11.86
3.44
67.52
62.16
Transformer-based Models
DUET
12.95
3.38
70.20
60.28
DUET+FDA
13.61
3.16
71.86
61.16
TD-STP
13.53
3.28
69.65
62.97
TD-STP+FDA
13.68
3.02
71.78
64.00
Follower
38.3
FoD
45.6
LSTM-based Models
EnyDrop
10.08
5.15
51.1
47.8
Drop+FFAD
13.43
5.00
54.5
49.1
FDA Assists in the R2R Task. We enhance both Transformer-basee and LSTM- based strong baselines, i.e., HAMT D. DUET [IO], TD-STP (64, Follower [15] and Env [49]. with our FDA method on R2R task. as shown in Table 2 The three enhanced Transformer based models have achieved clear advantages over the baselines, with SR improved by 1 1.88, 1.66, 2.13 and SPL. improved by 2.01, 0.88, 1.02, respectively on the validation unseen split. Meanwhile, our FDA method can also boost the performances of the two LSTM-based models, with SR improved by 7.3 and 3.4, respectively. These results demonstrate the adaptability of our approach across various models, which can significantly improve their performances and alleviate overfitting.
Table 3: Ablations on the RxR and CVDN tasks. RxR
CVDN
nDTWt
SDTW1
SR
SPL1
GP1
HAMT 64.3
49.6
58.1
54.0
5.03
HAMT+FDA
66.4
51.6
59.8
55.8
5.39
FDA Facilitates Performances on Other Cross-Modal Navigation Tasks. Furthermore, we evaluate our FDA method on RxR, CVDN and REVERIE tasks. Table (ooleBBshook shows the results on validation unseen splits of RxR and CVDN tasks. The HAMT equipped with FDA has obtained notable superior performances compared to the baseline. On RxR task, the SR and SDTW have been improved by 1.7 and 2.0, respectively. On CVDN task, our method could boost the GP performance by 0.36. Table shows our method's performance on REVERIE task. The DUET enhanced by FDA significantly outperforms the baseline, with SR and RGS improved by 2.92 and 3.49. These observations indicate

---

Table 4: Ablabtion on the REVERIE task.
Validation Unseen
TL
OSR
SR1
SPL1
RGSt
RGSPL1
DUET 18.76
48.59
44.65
32.92
28.57
21.01
DUET+FDA
19.04
51.41
47.57
35.90
32.06
24.31
Table 5: Performances in navigation scenes with feee-freyeency interference on R2R task. Red background part represents the baselines in normal navigation scenes. White and gray background parts represent the baselines and our augmented models scenes with high-freuuncc interference. Note that this evaluation setting is non-standard R2R, and is specially designed to validate our method.
Validation Seen
Validation Unseen
TL
NEJ
SRT
SPL.1
TI.
NEJ.
SR*
SPL1
HAMT
11.06
2.49
75.02
71.78
11.50
3.57
65.64
60.15
[MP3d] HAMT
15.47
3.61
43.88
38.94
15.76
5.55
44.40
36.93
[MP3d] HAMT+FDA
11.99
3.40
67.38
62.93
13.67
4.40
58.32
51.12
[ImageNet] HAMT
17.19
41.23
33.37
18.60
6.10
38.87
31.04
[ImageNet] HAMT+FDA
13.26
4.24
59.06
52.90
13.43
4.67
55.56
48,99
DUET
11.73
2.38
78.55
73.71
12.95
3.38
70,20
60.28
[MP3d] DUET
16.96
5.31
51.03
41.30
15.75
5.19
51.04
39.20
[MP3d] DUET+FDA
14.06
3.17
69.44
60.78
14.69
3.74
65.73
53.16
ImageNet] DUET
17.39
6.11
43.29
33.41
16.79
5.78
44.61
32.72
[ImageNet] DUET+FDA
17.01
3.90
62.49
59.79
16.89
3,94
63.69
49.78
TD-STP
12.66
2.53
76.30
71.70
13.53
3.28
69.65
62.97
[MP3d] TD-STP
21.34
5.23
49.07
40.62
21.25
5.73
45.13
37.41
[MP3d] TD- TD-STP+FDA
13.50
3.04
69.93
63.44
16.65
4.02
62.11
52.77
NmtgeNet] TD-STP
21.81
5.91
40.55
32.95
21.47
5.74
42.10
34.17
[ImageNet] TD-STP+FDA
16.20
3.69
63.08
54.74
16.80
4.02
61.13
51.37
that our approach can be easily and effectively integrated into other cross-modal navigation tasks while ensuring a significant performance boost.
Performances on Navigation Scenes with High-Frequency Interference. As analyzed before, existing VLN methods are severely limited in fighffreueecy perturbed navigation scenes. Table showws more details. We observe that the baselines can perform well under the normal navigation scenes. Setting Then, we introduce high-frequency interference, whose corresponding spatial image is the navigation views randomly sampled from MP3d [6), into seen/unseen navigation views as described in the method section to get the high-frequency perturbed unseen navigation scenes. Navigating under such scenes, all models have experienced a significant decrease in SR metric. Taking TD-STP as an example, its SR decreases by as much as 27.23 and 24.52 on validation seen easily and severely affected by
and validation unseen, respectively. The fact that the models high-frequency interference suggests their lack of ability to effectively identify and capture the necessary high-frequency information based on the instructions. However, with the assistance of our FDA, all models show notable improvement in performance. HAMT, DUET, and TD-STP see SR metric increases of 23.5, 18.41, 19.96 on validation seen split, and 13.92, 14.69, 16.98 on validation unseen split. Setting 2: To distinguish the source of interference from the training phase, we have also sampled high-freyuency interference from the 1000 classes of ImageNet 48 Due the out-of- domain interference, we find that the SR of baseline models is significantly more affected under this setting. Taking TD-STP as an example, its SR on validation seen and validation unseen decreases by 35.75 and 27.55, respectively. In such challenging scenes, our FDA enA-cnhanced models whose training data only contains figgufryy and low-frequency information from Mp3d, perform remarkably well, improving the SR of HAMT, DUET, and TD-STP by 17.83, 19.2, and 22.53 on validation seen, and by 16.69, 19.08. and 19.03 on validation unseen, respectively. In most cases, the navigation performances of the models have gained even greater improvements compared to the first setting, These two settings illustrate FDA-enhanced models excelling both the seen/unseen perturbed images (val seen/unseen) and the seen/unseen perturbation images (MP33/ImaaeeNtt. This serves as compelling evidence of our method's remarkable capability to recognize and capture essential high- figgufrequency information to boost navigation performance.
7

---

Table 6: Comparison with the visual and textual data augmentation methods on R2R task.
Validation Unseen
TL
NEL
SR
SPL
HAMT HAMT+ENVEDIT
11.50
3.57
65.64
60.15
11.15
3.48
67.31
62.84
HAMT+FDA
11.86
3.44
67.52
62.16
Visual Data Augmentation
TD-STP
13.53
3.28
69.65
62.97
ED-STD+ENVEDDIT
14.70
3.20
69.82
62.55
TD-STP+FDA
13.68
3.02
71.78
64.00
Follower
38.3
Follower+FDA
45.6
Follower+ Speaker
40.4
SpaaerrAS
46.3
Textual Data Augmentation
TD-STP-
13.78
3.62
65.73
58.72
DD-STP=+FDD
14.77
3.44
66.96
59.72
STP=+PREVVLENT
13.53
3.28
69.65
62.97
TD-STP APEEAAEEREA
13.68
3.02
71.78
64.00
Comparison with Existing Data Augmentation Methods. Table shows the comparison with the visual data augmentation method ENVEDIT 133, and textual data augmentation methods Speaker [3] and PREVALENT [20]. Compared to ENVEDIT which aims to increase the diversity of the environment by using GANs to alter environment style, method, which also serves as visual data augmentation method, demonstrates clear superiority, with SR of HAMT/TD-ST increased by 0.21/1.96 on validation unseen, respectively. In comparison with the textual augmentation methods, the results indicate that our FDA method demonstrates consistent improvements on both baseline models Follower and TD-STP Additionally, it effectively complements the two textual augmenta- tion methods, further enhancing cross-modal navigation performance. Our FDA has demonstrated superior performance without relying on external models such as visual/textual generative models. As result, our method could greatly reduce the complexity of data augmentation work in VLN field.
3.4 Comparison with the State-of-the-art Methods
We compare method with SoTA methods on R2R, RxR, CVDN and REVERIE tasks.
On R2R task, 7 shows our method achieves the best performance on both validation unseen and test splits. Compared to the DUET model which has been fed with the extra object features not included in our method, the SR/SPL of our method still outperforms by 2/4 points on validation unseen. Compared to TD-STP, Our method shows SR/SPL improvements of 2/1 and 3/2 points validation unseen and test splits. On RxR task, Table S shows the comparisons between our method and previous methods. The results show that our method outperforms the previous SoTA HAMT with SPL/nDTW improved by 1.8/2.1 and 0.4/1.5 on validation unseen and test splits, respectively, On CVDN task, method also shows superior performance Compared to HAMT, the previous SoTA, our method gains a GP improvement of 0.05/0.36/0.25 on validation seen, validation unseen and test splits, respectively. On REVERIE task, as shown in Table
previous methods in Table 9
10 our method outperforms the previous SoTA DUET with the increase 2. 002/2.98/3.49/3 and validation unseen and test splits, respectively.
1.21/0.49/2.03/1 .47 in SR/SPLIRGS//RGSPL
The remarkable performance analyzed above has demonstrated the excellence of our method in enhancing cross-modal navigation, well as the outstanding adaptability across different tasks.
4 Related Work
Vision-and-Language Navigation. Anderson et al. 5 are the first to introduce the Vision-and- Language Navigation, where agent is asked to follow an English instruction in R2R to navigate
Unlike ENVEDIT, our method operates without externl models. To ensure faireess, we compare our method with ENVEDIT's Editing Style, which relies on the fewest external models.
"TD-STP- denotes the TD-STP model after excluding data augmentation PREVALENT.

---

Table 7: Comparison with SoTA methods on R2R task.
Validation Unseen
Test Unseen
TL
NEP
SR1
SPL1
TL
NEL
SRT
SPL1
Seq2Seq
8.39
7.81
22
8.13
20
Speaker Follower 15
6.62
35
14.82
6.62
35
28
SMNA (39
5.52
45
32
18.04
5.67
48
35
RCM [58
11.46
6.09
43
11.97
6.12
43
38
EGP [12
4.83
56
44
5.34
53
42
EnvDrop 51
10.70
5.22
52
48
11.66
5.23
51
47
PREVALENT
20
10.19
4.71
58
53
10.51
5.30
ReIGraph
22
9.99
4.73
57
53
10.29
4.75
55
52
RecBert
25
12.01
3.93
63
57
12.35
4.09
63
57
Airbert
19
11.78
4.01
62
56
12.41
4.13
62
57
DUET 10
13.94
3.31
72
60
14.73
3.65
69
59
DUET
12.95
70
60
HAMT
11.46
66
61
12.27
3.93
65
HAMT
99
11.50
3.57
66
60
TD-STP
664
3.22
70
63
3.73
67
61
TD-STP"
64
13.53
3.28
70
63
14.71
3.79
66
60
Ours (TD-STP FDA)
13.68
3.02
72
64
14.76
3.41
69
Table 8: Comparison with SoTA methods on RxR task.
Validation Unseen
Test Unseen
nDTWt
SDTW
SRT
SPLt
nDTW1
SDTW1
SRT
SPL1
Multilingual Baseline 29
38.9
18.2
22.8
36,8
16.9
21.0
18.6
Monolingual Baseline
44.5
23.1
28.5
41.1
22.6
SAA 24
52.7
33.0
40.0
36.0
46.8
29.1
35.4
31.6
Envprop
CLIP-VIL
49
55.7
42.6
51.1
35.2
HAMT
63.1
48.3
56.5
56.0
46.6
HAMT
64.3
49.6
58.1
54.0
60.5
47.0
55.5
48.4
Ours (HAMT+FDA)
66.4
51.6
59.8
55.8
62.0
47.3
55.5
48.8
After that, model architecture design [28 40 39
58
24
25
55
34
10
59
3
22
strategy
17
18 training paradigm gm (66,
62
54
38
loss design 665
26 reward
36 32
shaping 27 56 pretraining 35
20
4l 46
49
61
47 and probing experiments
have containing multilingual instructions
been widely explored. Then, more challenging task RxR 229
and longer trajectories is proposed. A series of works 14 32 21 have explored the above and make significant progress. CVDN [52) is task asking agent to navigate following multi-turn noman-to-humal dialog. In addition requiring the agent have cross modal understanding. also focusing on that have been development of this field. While these works have conducted extensive
demands certain level of reasoning ability. Works (50) 20 69 68
60 9
investigated to advance exploration, they have almost neglected the frequency domain that our work has demonstrated to be crucial for VLN navigation.
Table Comparison with SoTA methods on CVDN task.
Val Seen
Val Unseen
Test Unseen
PREVALENT
20
3.15
2.44
CMN 69
7.05
2.97
2,95
VISITRON 50
5.11
3.25
3.11
SCoA
68
7.11
2.85
3.31
MT-RCM+ (60
5.07
4.65
3.91
HAMT
6.91
5.13
5.58
HAMT
7.43
5.03
5.43
Ours (HAMT+FDA)
7.48
5.39
5.68

---

Table 10: Comparison with SoTA methods on REVERIE task.
Validation Unseen
Test Unseen
TL
OSR
SRT
SPL
RGS1
RGSPL
TL
OSR
SR1
SPL1
RGS1
RGSPL1
RecBERT
25
16.78
35.02
30.67
24.90
18.77
15.27
15.86
32.91
29.61
23.99
16.50
13.51
Airbert
19
18.71
34.51
27.89
21.88
18.23
14.18
17.91
34.20
30.28
23.61
16.83
13.28
HAMT
14.08
36.84
32.95
30.20
18.92
17.28
13.62
33.41
30.40
26.67
14.88
13.08
DUET [0
22.11
51.07
46.98
33.73
32.15
23.03
21.30
56,91
52.51
36,06
31.88
22.06
DUET* 100
18.76
48.59
44.65
32.92
28.57
21.01
16.31
52.77
48.41
35.96
28.31
20.61
DUET-FDA
19.04
51.41
47.57
35,90
32.06
24.31
17.30
53.54
49.62
36.45
30.34
22.08
Data Augmentation in Vision-and-LLanguage Navigation. In terms of text augmentation, Fried et al. [5], Wang et al. [57] and Dou et al. [14] instruction generators generate pseudo-instruction terms of vision augmentation, Fu et al. [16] learn a sampler to sample the challenging paths to enhance the agent's ability for handling them. Tan al. [51] use environment dropout on seen environments to mimic unseen environments and generate the pseudo-instructions for it with instruction generator. Parvaneh al. 43 learn model generate counterfactuals on seen environments to simulate unseen environments. Li et al. [33 adopt GAN to transfer the environment style for increasing the diversity of the environment. However, all of these works focus on the augmentation in the spatial domain and have ignored the high-frrou highffrequency information which crucial for mooal matching. Moreover, they have to rely on extra generator models, which require intricate design and will increase the model burden.
according to sampled trajectories.
Data Augmentation in Robotics. Domain randomization is form of Data Augmentation (DA) that has shown effectiveness transferring reinforcement learning policies from simulation to the real world [53]. In contrast, Cobbe el al. [II] propose modification Cutout [13], where multiple rectangular regions of varying sizes are masked with random color for each observation. Lee et al. 33 use randomized (convolutional) neural network that adds random perturbations the input observations, which helps the trained agents learn more robust features that are invariant across diverse and randomized environments. RAD 30] investigates the general DAs for robotics on both pixel- based and state-based inputs. In this paper, we propose simple and effective frequency-enhane DA for the cross-modal task VLN, fundamental task in robotics.
Conclusion
Shifting from the perspective of spatial domain previous works mainly focus on, we provide first- of-its-kind in-depth analysis of the frequency domain information in VLN tasks. Our findings have demonstrated the importance of high-frequency information in enhancing navigation performance. In light of the findings, we further propose the Frequency-enhance Data Augmentation (FDA) method, a simple and effective data augmentation technique to boost a model's capability to discern and capture essential high-freque gighffrequency information without the need for additional auxiliary generative models. Our method on a range of VLN tasks, including R2R, RxR, CVDN and REVERIE, achieves notable performances and shows great adaptability across different VLN models. We hope our work can offer practical solution and bring new insights to the VLN research community.
Limitations and Future Work. Our work focuses on enhancing the model's general ability to recognize and capture essential high-frequency information. However, we have not yet explored the fine-grained correlation between frequency and specific scenes or categories. This area of investigation remains an avenue for future exploration.
Acknowledgements
This project supported by National Key R&D Program of China (20227D01 17900), National Natural Science Foundation of China (62236010 and 62276261), Key Research Program of Frontier Sciences CAS Grant No. ZDBS-LYJSC032, the National Research Foundation, Singapore, under its Medium Sized Center for Advanced Robotics Technology Innovation, and the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: TE0-T222201220000000
10

---

References
[1] D. An, Y. Qi. Y. Huang, Q. Wu, L. Wang, and T. Tan. Neighbor-view enhanced model for vision and language navigation. In ACM International Conference on Multimedia, 2021.
[2] D. An, Y. Qi. Y. Li, Y. Huang, I. Wang, T. Tan, and J. Shao. Bevbert: Multimodal map pre-training for language-guided navigation. In International Conference Computer Vision. pages 2737-2748, 2023.
[3] D. An, H. Wang, W. Wang, Z. Wang, Y. Huang, K. Hc, and L. Wang. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. arXiv preprint arX1v:2304.03044, 2023.
[4] P. Anderson, A. Shrivastava, D. Parikh, D. Batra, and S. Lee. Chasing ghosts: Instruction following as bayesian state tracking. Advances in Neural Information Processing Systems, 2019.
[5] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and-language navigation: Interpreting visually-groundel navigation instructions in real environments. In IEFE Conference on Computer Vision and Pattern Recognition, 2018.
[6] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng. and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments, arXiv preprint arXiv:1709.06/35 2017.
[7] J. Chen, C. Gao, E. Meng, Q. Zhang, and S. Liu. Reinforeed structured evolutevo for vision- lasoon-anguuaee navigation. In JEEE Conference Computer Vision and Pattern Recognition, 2022.
[8] P. Chen, D. Ji, K. Lin, R. Zeng, T. Li, M. Tan, and C. Gan. Weakly-supervised multi guati-granularity map learning for vision-and-language navigation. Advances in Neurul Information Processing Systems. 35:38149-38161 2022.
[9] S. Chen, P.-I.. Guhur, C. Schmid, and I. Laptev. History aware multimodal transformer for vision-and- language navigation. Advances in Neural Information Processing Systems, 34, 2021.
[10] S. Chen, P.-L.. Guhur, M. Tapaswi, C. Schmid, and I, Laptev. Think global, act local: Dual-scale graph transformer for bision-and-lanee and-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[11] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying gencralization in reinforcement learning. In International Conference Machine Learning. PMLR, 2019.
[12] Z. Deng. K. Narasimhan, and O. Russakovsky. Evolving graphical planner: Contextual global planning for aatoo--nndllaguuee navigation. Advances in Neural Information Processing Systems, 33, 2020.
[13] T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708 1008. 100.00452, 2017.
[14] Z.-Y. Dou and Peng. FOAM: A follower-aware speaker model for vision-and- laiio--nd-langguaee navigation. In North American Chapter of the Association for Computational Linguistics, Seattle, United States, July 2022. Association for Computational Linguistics.
[15] D. Fried, R. Hu, V. Cirik, A. Rohrbach, Andreas, L.-P. Morency, T. Berg-Kirkpar Kirkpatrik,, K. Saenko, D. Klein, and T. Darrell, Speaker-followat models for vision-and-langune navigation. Advances in Neural Information Processing Systems, 2018.
[16] T.JJ Fu, X. E. Wang, M. F. Peterson, S. T. Grafton, M. P. Eckstein, and W. Y. Wang. Counterfactual vision-and and languae navigation via adversarial path sampler. In European Conference on Computer Vision. Springer, 2020.
[17] C. Gao, X. Peng. M. Yan, H. Wang, L. Yang, H. Ren, H. Li, and S. Liu. Adaptive zone-aware hierarchical planner for vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 14911-14920 2023.
[18] G. Georgakis, K. Schmeckpeper, K. Wanchoo, S. Dan, E. Miltsakaki, D. Roth, and K. Daniilidis. Cross- modal map learning for vision and language navigation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 15460- 100000154770 15470, 2022.
[19] P.-L. Guhur, M. Tapaswi. S. Chen. I. Laptev, and C. Schmid. Airbert: In-domain pretraining for vision- and-language navigation. In International Conference on Computer Vision, 2021.
[20] W. Hao, C. Li, X. Li, L. Carin, and J. Gao. Towards learning a generic agent for vision and language navigation via pre-training, In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
11

---

[21] K. He, Y. Huang. Q. Wu, J. Yang, D. An, S. Sima, and L. Wang. Landmark-rxr: Solving vision-and- language navigation with fine-grained alignment supervision. Advances in Neural Information Processing Systems, 34, 2021.
[22] K. He. Y. Jing, Y. Huang. Z. Lu, D. An, and L. Wang. Memory-adaptic vision-and-lann -anguage navigation. Available at SSRN 4539118.
[23] Y. Hong, C. Rodriguez, Y. Qi. Q. Wu, and S. Gould. Language and visual entity relationship graph for agent navigation. Advances in Neural Information Processing Syxtems, 33, 2020.
[24] Y. Hong, C. Modriguez-Opazzz Q. Wu, and S. Gould. Sub-instruction aware vision-and- and--anguagee navigation. Empirical Methods in Natural Language Prucessing, 2020.
[25] Y. Hong. Q. Wu, Y. Qi, C. Rodriguez-Opaza, and S. Gould. A recurrent vision-and-language bert for navigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[26] R. Hu, D. Fried, A. Rohrbach, D. Klein, T. Darrell, and K. Saenko. Are you looking? grounding to multiple modalities in vision-and- and-language navigation. In Annual Meeting of the Association for Computational Linguistics, 2019.
[27] V. Jain, G. Magalhaes, A. Ku, A. Vaswani, E. Ie, and J. Baldridge. Stay on the path: Instruction fidelity in vision-and-land looo-ane-language navigation. Association for Computational Linguistics, 2019.
[28] L. Ke, X. Li, Y. Bisk, A. Holtzman, Z. Gan, J. Liu, Gao, Y. Choi, and S. Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In JEEE Conference on Computer Vision and Pattern Recognition, 2019.
[29] A. Ku, P. Anderson, R. Patel, E. le, and Baldridge. Room-across-room Multilingual vision-and- -annlauugl navigation with dense spatiotemporal grounding. Empirical Methods in Natural Language Processing. 2020.
[30] M. Laskin, K. Lee, Stooke, I.. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data, Advances in Neural Information Processing Systems, 33, 2020.
[31] K. Lee, K. Lee, J. Shin, and H. Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference on Learning Representations, 2020.
[32] J. Li, H. Tan, and M. Bansal. CLEAR:
Improving vision-language navigation with cross- loossllnggua,, environment-agmme agmostic representations. In North American Chapter of the Association for Computational Linguistics, 2022.
[33] J. Li, H. Tan, and M. Bansal. Envedit: Environment editing for vision-and-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[34] J. Li, H. H. Tan, and M. Bansal. Improving cross- cross-modal modal alignment in vision language navigation via syntactic information. In North American Chapter of the Association for Computaatonal Linguistics, 2021.
[35] X. Li, C. Li, Q. Xia, Y. Bisk, A. Celikyilmaz, Gao, N. Smith, and Y. Choi. Robust navigation with language pretraining and stochastic sampling. In Empirical Methods in Natural Language Processing, 2019.
[36] X. Liang, F. Zhu, Y. Zhu, Lin, Wang, and X. Liang. Contrastive instruction-triecctory learning for vision-language navigation. In AAAI Conference on Artificial Intelligence, volume 36, 2022.
[37] B. Lin, Y. Zhu, Z. Chen, X. Liang, J. Liu, and X. Liang. Adapt: Vision-languaae navigation with modality--llgneen action prompts. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[38] B. Lin, Y. Zhu, Y. Long, X. Liang. Q Ye, and L. Lin. Adversarial reinforced instruction attacker for robust vision-language navigation. IEFE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 2021.
[39] C.-Y. Ma. J. Lu, Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong. Self monitoring navigation agent via auxiliary progress estimation. International Conference on Learning Representations, 2019.
[40] C.-Y. Ma, Z. Wu, G. AlRegib, C. Xiong, and Z. Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In IEEE Conference Computer Vision and Pattern Recognition, 2019.
[41] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra. Improving vision-and- lannguugee navigation with image-text pairs from the web. In European Conference on Computer Vision, 2020.
12

---

[42] A. Moougii, A. Majumdar, H. Agrawal, S. Lee. and D. Batra. Soat: A scene-and object-aware transformer for vision-and-languee and-language navigation. Advances in Neural Information Processing Systems, 34, 2021.
[43] A. Parvanch, E. Abbasnejad. D. Teney, J. Q. Shi, and A. Van den Hengel. Counterfactual vision-and- language navigation: Unravelling the unseen, Advances in Neural Information Processing Systems, 33, 2020.
[44] Y. Qi, Z. Pan, S. Zhang, A. van den Hengel, and Q. Wu. Object-and-actionn aware model for visual language navigation. In European Conference on Computer Vision, 2020.
[45] Y. Qi, Q. Wu, P. Anderson, X. Wang. W. Y. Wang, C. Shen, and A. van den Hengel. Reverie: Remote emboddied visual referring expression in real indoor environments. In JEEE Conference on Computer Vision and Pattern Recognition, 2020.
[46] Y. Qiao, Y. Qi, Y. Hong, Z. Yu, P. Wang, and Q. Wu. Hop: iistory-and-order aware pre-training for vision- and aaee--nd--annnn language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[47] Y. Qiao, Y. Qi, Y. Hong. Z. Yu, Wang, and Q. Wu. Hop+: History-enhaneed and order-aware pre-training for vision-and- -nd--angaa---eeeeeeeee-e---- navigation. IEEE Transactions on Puttern Analysis and Machine Intelligence. 2023.
[48] O. Russakovsky, J. Deng. H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bersstcin, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 2015.
[49] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, and K. Keutzer. How much can CLIP benefit Aisionandd-languag tasks? In International Conference on Learning Representations, 2022.
[50] A. Shrivastava, K. Gopalakrishnan, Y. Liu, R. Piramuthu, G. Tur, D. Parikh, and D. Hakkani-Tuur. VIS- ITRON: Visual semantics-aligned interactively trained ohject-navigator. In Association for Computational Linguistics, Dublin, Ireland, May 2022. Association for Computational Linguisties.
[51] H. Tan, L. Yu, and M. Bansal. Learning to navigate unseen environments: Back translation with en- vironmental dropout. In North American Chapter of the Association for Computational Linguistics, 2019.
[52] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Vision-and-dialog navigation. In Conference on Robot Learning 2020.
[53] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In International Conference on Intelligent Robots and Systems. IEFF, 2017.
[54] H. Wang. W. Liang, Shen, L. Van Gool, and W. Wang. Counterfactual cycle-consistent learning for instruction following and generation in vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[55] H. Wang, W. Wang, Liang, C. Xiong, and Shen, Structured scene memory for lasion-language navigation. In IEFE Conference on Computer Vision and Pattern Recognition, 2021.
[56] H. Wang, Q. Wu, and C. Shen. Soft expert reward learning for vision-and-language navigation. In European Conference on Computer Vision. Springer, 2020.
[57] S. Wang, C. Montgomery, J. Orbay, V. Birodkar, A. Faust, I. Gur, N. Jaques, A. Waters, J. Baldridge, and P. Anderson. Less is more: Generating grounded navigation instructions from landmarks. In IEFE Conference on Computer Vision and Pattern Recognition, 2022.
[58] X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.-F. Wang, W. Y. Wang, and L. Zhang. Reinforced cross modal matching and supervised imitation learning for vision-lant lanounl navigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[59] X. Wang, W. Wang, J. Shao, and Y. Yang. Lana: A language-capable navigator for instruction following and generation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 19048-19058, 2023.
[60] X. E. Wang. V. Jain, E. le. W. Y. Wang, Z. Kozareva, and S. Ravi, sesiioommet--ggnosoi multitask learning for natural language grounded navigation. In European Conference on Computer Vision. Springer, 2020.
13

---

[61] S. Wu, X. Fu, F. Wu, and Z.-J. Zha. Cross-modal semantic alignment pre-training for vision-and-languaal navigation. In ACM International Conference on Multimedia, 2022.
[62] J. 7hang, J. Fan, J. Peng, et al. Curriculum learning for vision-and-languuge navigation. Advances in Neural Information Processing Systems, 34, 2021.
[63] Y. Zhang, H. Tan, and M. Bansal. Diagnosing the environment bias in vision-and-language navigation. In International Joint Conferences Artificial Intelligence, 2021.
[64] Y. Zhao, J. Chen, C. Gao, W. Wang, I.. Yang. H. Ren, H. Xia, and S. Liu. Target-driven structured transformer planner for vision-language navigation. In ACM International Conference on Multimedia, 2022.
[65] F. Zhu, Y. Zhu, X. Chang, and X. Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[66] W. Zhu, H. Hu, Chen, Z. Deng, Jain, E. le, and F. Sha. Babywalk: Going farther ssioo-and-language navigation by taking baby steps. Association for Computational Linguistics, 2020.
[67] W. Zhu, Y. Qi, P. Narayana, Sone, S. Basu, X. Wang. Wu, M. Fekstein, and W. Y. Wang. Diagnosing vision-and- -anaa-aa- navigation: What really matters. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, July 2022. Association for Computational Linguistics.
[68] Y. Zhu, Y. Weng. F. Zhu, X. Liang, Q. Ye, Y. Lu, and J. Jiao. Self-motivated communication agent for real-world vision-die vision-dialog navigation. In International Conference on Computer Vision, October 2021.
[69] Y. Zhu, F. Zhu, Z. Zhan, B. Lin, J. Jiao, X. Chang, and X. Liang. Vision-dialog navigation by exploring cross-modal memory. In IEEE Conference Computer Vision and Puttern Recognition, 2020.
14


Reviews:
[{"review_id": "Z405T8aG3m", "reviewer": "Reviewer_wftp", "summary": "This paper investigates the sensitivity of existing state of the art VLN models to high frequency perturbations in the image features, and identifies that this could be a way to improve the models performance by making them more robust. It also provides evidence to show that adding high frequency information explicitly to the models may improve their performance. Taken together, this implies that if the model can be made more sensitive to changes in high frequency components while being able to discriminate which ones are relevant to the instruction, it could improve the generalization power of the models. The paper then proposes a data augmentation approach which involves replacing high frequency components of images with those of randomly selected other images, and training the model on this data. The resulting models degrade less in performance when evaluated on perturbed images and improve performance on standard benchmarks compared to the baseline models. \n\n-------\nEdit: Updated score after clarifications from the rebuttal!", "questions": "* What happens if you just do an interpolation between one of the channels from a random image, instead of doing it in the fourier domain? This could be additional proof that augmentation by manipulating in the fourier domain is most effective.\n* According to Eq 3, the resulting image is likely to have perturbed features for more most of the channels - how did you decide the ratio for keeping original vs replacing with the distractor image features. Is this applied for all the views and all the images in the trajectory?\n* What happens if you train only with the augmented data? what was the reason you chose the set up in (6)\n* Since the edge and corner features are what this paper finds to be specifically useful, and given that the image feature extractors based on CNNs also extract such features in early layers, have the authors considered or tried reusing intermediate features from the image extractor to emphasize these features?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "## Originality\nAs far as I know, this is a new exploration in terms of data augmentation by perturbing visual features in the frequency domain\n\n## Quality, Clarity, Significance\nThe paper provides an interesting exploration and provides evidence that supports their hypothesis. It is well written and easy to follow. The paper provides experiments based on three state of the art models, and provides a lightweight method to improve models' performance.", "weaknesses": "## Clarity\n* Lacking in implementation details regarding choice of model components such as image encoder, text encoder, etc. \n\n## Discrepancies in numbers reported\n* The numbers reported in the table for RxR seem to not be consistent from those in literature (for example in terms of NDTW, HAMT on RxR Val seen achieves 65.3 and 63.1 on val unseen according to their paper [1] as opposed to the 63.1 and 59.94 reported here. \n* In Table 5 (Comparison to EnvEdit), the numbers reported for HAMT + EnvEdit  for SR and SPL are 67.1 and 62.84 as opposed to 68.9 and 64.4 in the EnvEdit paper [2] \n* Line 177: Text states that the improvement in GP is by 3.6 points whereas the table mentions 0.36 points.\n \nCould the authors please explain these discrepancies?\n\n## Minor issues / typos\n* Line 156: Path Lengt -> Path Length \n* Line 158: Goal Process -> Goal Progress\n* Table 4 caption. Please state more clearly that this evaluation set is non-standard R2R, and is created as part of this paper\n\n\n[1] History Aware Multimodal Transformer for Vision-and-Language Navigation. Shizhe Chen et al.\n[2] ENVEDIT: Environment Editing for Vision-and-Language Navigation. Jialu Li et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Lacking in implementation details regarding choice of model components such as image encoder, text encoder, etc.**\n\nA1: Thank you for your concern. Our model components closely match baseline models (TD-STP, DUET, HAMT). For R2R, TD-STP, DUET, and HAMT use ViT-B/16 for image encoder and BERT for text encoder. For CVDN, HAMT uses ViT-B/16 for image encoder and BERT for text encoder. For RxR, HAMT uses CLIP for image encoder and XLM-R for text encoder. As our focus is image-level data augmentation, we’ve detailed image encoders in Supplementary Section D. We’ll add text encoder information for better reader understanding.\n\n**Q2: The numbers reported in the table for RxR seem to not be consistent from those in literature (for example in terms of NDTW, HAMT on RxR Val seen achieves 65.3 and 63.1 on val unseen according to their paper [1] as opposed to the 63.1 and 59.94 reported here.**\n\nA2: Thanks for pointing this out. We anticipate that the reviewer erroneously refers to the incorrect numbers. In essence, our research aligns with the results in paper [1]. The numbers 65.3 and 63.1 in Table 13 of [1] are for val seen and val unseen, respectively. In our paper, the values 63.1 and 59.94 in Table 7 correspond to val unseen and test unseen, respectively. For [1]’s test unseen (nDTW=59.94), we suggest looking at its Table 12.\n\n[1] S. Chen, et al. History aware multimodal transformer for vision-andlanguage navigation. NeurIPS. 2021.\n\n**Q3: In Table 5 (Comparison to EnvEdit), the numbers reported for HAMT + EnvEdit for SR and SPL are 67.1 and 62.84 as opposed to 68.9 and 64.4 in the EnvEdit paper [2].**\n\nA3: Thanks for your question. We apologize for the unclear expression. ENVEDIT [2] implements a total of three types of data augmentation: Editing Style, Editing Appearance, and Editing Objects. The latter two require the semantic segmentation model to provide object and semantic information in addition to the generative model. The performance of 68.9 and 64.4 reported in [2] actually results from the ensemble of the three models trained on the three types of augmented ENVEDIT data, respectively. Unlike ENVEDIT, our method operates without external models. To ensure fairness, we compare our approach with Editing Style, which uses the fewest external models. [2] reports SR and SPL as 67.3 and 62.6, respectively. Our reimplementation confirms these as 67.31 and 62.84, seen in our Table 5. Our current statement (L205: “a strong data augmentation method ENVEDIT in VLN, which aims to increase the diversity of the environment using GANs to alter the environment style.\") is not entirely clear, and we will enhance its clarity in our revisions.\n\n[2] J. Li, et al. Envedit: Environment editing for vision-and-language navigation. CVPR. 2022.\n\n**Q4: Line 177: Text states that the improvement in GP is by 3.6 points whereas the table mentions 0.36 points.**\n\nA4: We sincerely apologize for the oversight and greatly appreciate your attention to detail. The accurate value is indeed 0.36. We will rectify this error in the revised version.\n\n**Q5: Minor issues / typos.**\n\nA5: Thank you for your correction. We will further correct typos, and provide clearer clarifications in the revised version as suggested.\n\n**Q6: What happens if you just do an interpolation between one of the channels from a random image, instead of doing it in the fourier domain? This could be additional proof that augmentation by manipulating in the fourier domain is most effective.**\n\nA6: Thanks for your suggestion. The interpolation method achieves SR/SPL of 70.16/62.79 on R2R val unseen, notably lower than FDA’s 71.78/64.00. This indicates Fourier domain manipulation is superior since it can fully leverage high frequency for augmentation, which the spatial domain cannot achieve.\n\n**Q7: According to Eq 3, the resulting image is likely to have perturbed features for more most of the channels - how did you decide the ratio for keeping original vs replacing with the distractor image features. Is this applied for all the views and all the images in the trajectory?**\n\nA7: Thanks for your question. We empirically selected the current ratio, which has shown quite remarkable results. This ratio is consistently used for all views and images in the trajectory.\n\n**Q8: What happens if you train only with the augmented data?**\n\nA8: Thank you for bringing up this concern. The model trained only on augmented data achieves SR 67.01 and SPL 57.86 on R2R val unseen, performing worse than the model trained with both original and augmented data (SR 71.78, SPL 64.00). Exclusive reliance on augmented data may lack proper visual cues for the model to emphasize crucial high-frequency components, thus affecting cross-modal alignment.\n\n**Q9: What was the reason you chose the set up in (6)?**\n\nA9: Empirically, using the approach in setting (6) yields better performance (SR 71.78) than random selection of original and augmented data (SR 71.65). Thus, we opted for setting (6).\n\n**Q10: Since the edge and corner features are what this paper finds to be specifically useful, and given that the image feature extractors based on CNNs also extract such features in early layers, have the authors considered or tried reusing intermediate features from the image extractor to emphasize these features?**\n\nA10: Thanks for your interesting question. We conduct a preliminary experiment following a structure similar to Figure 3, using the layer1 features of CLIP-ResNet50. It showed an improvement over the baseline (SR 69.65), reaching SR 70.11. This underscores high-frequency information’s significance in cross-modal navigation. In contrast, our method achieves far better performance (SR 71.78) without additional parameters or model reliance."}, {"review_id": "RGuNHIa791", "reviewer": "Reviewer_gjxf", "summary": "This paper aims at investigating the Fourier domain of vision-and-language navigation (VLN). They propose Frequency-enhanced Data Augmentation (FDA) to improve the visual-text matching processing based on high-frequency visual information. Their method can achieve promising results efficiently on various VLN datasets, including R2R, RxR, and CVDN, with the same parameter size.", "questions": "Please see Weakness", "limitations": "N/A\n", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "+ This paper is well-written and easy to follow.\n+ The aspect of the Fourier domain and lower-/higer- frequency visual representation is new in VLN.\n+ Though data augmentation is not novel in VLN, the frequency-based sampling seems effective, which helps achieve new/competitive SOTA on various datasets.\n+ The sensitive analysis in Figure 2 and Table 1 is a good intro convincing.", "weaknesses": "+ Though the performance looks effective, what is the motivation to rely on Fourier representation instead of the widely-used learned visual features (maybe I missed it, but I cannot find a clear answer in the draft)?\n+ Does visual information get lost during the Fourier transform? How to overcome this issue during the navigation process?\n+ A detailed analysis of data augmentation should be considered.  For example, how much data can lead to how much improvement? If we keep training with more data, does the navigator works better and better?\n+ There should include some qualitative examples to demonstrate how Fourier features can actually lead to the final goal. The comparison with baselines is required, and both successful/failed cases should be involved to guide future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Though the performance looks effective, what is the motivation to rely on Fourier representation instead of the widely-used learned visual features (maybe I missed it, but I cannot find a clear answer in the draft)?**\n\nA1: Thank you for your question. Our method is essentially a combination of Fourier transform and learned visual features. In data augmentation phase, we rely on Fourier transform to obtain the high and low-frequency information of the reference image and the interference image. By using the Mix operation (Formula 3) and Fourier inverse transforms, we generate the augmented image. However, during the training and testing phases, we do not use Fourier transforms. Instead, we utilize learned visual features. In the training phase, we feed the reference image and augmented image into the image encoder to extract learned visual features. Subsequently, we input these features into the baseline model to learn downstream visual features and perform navigation reasoning based on them. During the test phase, we directly extract visual features from the reference image using the image encoder and feed them into the FDA-enhanced model to accomplish navigation reasoning. Because of this characteristic of our method, our lightweight approach can easily enhance existing navigation models based on learned visual features, such as TD-STP, DUET and HAMT. We apologize for any unclear expressions and will ensure clarity in the revised version.\n\n**Q2: Does visual information get lost during the Fourier transform? How to overcome this issue during the navigation process?**\n\nA2: Thanks for pointing out your concern. We implement the Fourier forward and inverse transforms using numpy.fft.fftn and numpy.fft.ifftn, which almost incur no information loss. Meanwhile, our method enhances the ability to capture high-frequency information of the model itself. As a result, during the test phase, the model directly processes RGB images, same to its baseline model, without relying on Fourier transform.\n\n**Q3: A detailed analysis of data augmentation should be considered. For example, how much data can lead to how much improvement? If we keep training with more data, does the navigator works better and better?**\n\nA3: Thanks for your constructive suggestion. In order to identify some underlying patterns, we conduct a preliminary exploratory experiment using current augmented data volume’s 10%, 100%, and 200%. Compared to the baseline model, which achieves an SR of 69.65, the model trained with 10% augmented data exhibits an SR of 69.77. Similarly, the model trained with 100% augmented data achieves an SR of 71.78, and the model trained with 200% augmented data achieves an SR of 71.86. As the volume of augmented data increases, the model’s performance gradually improves; however, this improvement trend begins to plateau. We will add this discussion to the revision.\n\n**Q4: There should include some qualitative examples to demonstrate how Fourier features can actually lead to the final goal. The comparison with baselines is required, and both successful/failed cases should be involved to guide future research.**\n\nA4: Thanks for your valuable suggestions. Due to the fact that our method does not require the Fourier features during navigation, but only relies on learned visual features, it is not possible to perform a direct qualitative analysis of Fourier features. However, we have still attempted to analyze the results from other perspectives. Firstly, we conduct navigation experiments under high-frequency interference settings (Section 3.3) and provide corresponding visualizations (Appendix - Figure 1, Figure 2). The results confirm that our model can effectively capture the necessary high-frequency information from images. Secondly, we provide visualizations of visual and textual matching (Appendix - Figure 3, Figure 4), which demonstrates that models with stronger high-frequency capturing abilities can utilize the captured high-frequency information to improve cross-modal attention weight assignment and cross-modal matching. Following your suggestions, we will add failed cases in the revised version and make every effort to provide in-depth analysis to assist other researchers in the community with their future research endeavors."}, {"review_id": "QLqDGivFpa", "reviewer": "Reviewer_r2mS", "summary": "In this paper, the authors investigate the VLN task from a Fourier domain perspective, which aims to enhance the vision-language matching ability of agents. A Frequency-enhanced Data Augmentation (FDA) technique is proposed to improve the capability of capturing high-frequency information, which is seen as an important factor for training VLN agents. The experimental results show that baseline models can be further improved by adding the FDA technique.", "questions": "See Weaknesses.", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work investigates the VLN task from an alternative perspective, i.e., the Fourier domain.\n\n- A reasonable FDA method is proposed to enhance the baseline model to capture high-frequency information without the need for additional parameters. \n\n- With the FDA technique, the performances of three transformer-based methods are further improved, confirming its effectiveness.", "weaknesses": "- The adopted HAMT, DUET, and TD-STP are all transformer-based agents. However, the LSTM-based agents with FDA also need to be verified. \n\n- In Table 2, DUET+FDA exhibits a significant performance drop on the val-seen set compared to the original DUET, e.g., SPL from 73.71\\% to 69.66\\%.\n\n- Lacking experimental comparison on the high-level instruction REVERIE dataset.\n\n- As shown in Table 2 of the supplementary material, different random seeds have a significant impact on the model performance. For example, seed-3 and seed-4 differ by 1\\% in SPL.\n\n- The writing needs to be further polished, e.g., the highest accuracy numbers in all tables should be bolded, and the title of section 3.4 should be 'Comparison with the State-of-the-art Methods'. \n\n- Missing numerous references such as [1-4].\n\n[1] Adaptive Zone-Aware Hierarchical Planner for Vision-Language Navigation. In CVPR 2023.\n\n[2] HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation. In TPAMI.\n\n[3] Reinforced Structured State-Evolution for Vision-Language Navigation. In CVPR 2022.\n\n[4] LANA: A Language-Capable Navigator for Instruction Following and Generation. In CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The adopted HAMT, DUET, and TD-STP are all transformer-based agents. However, the LSTM-based agents with FDA also need to be verified.**\n\nA1: Thanks for your valuable suggestion. We conduct experiments on two LSTM-based models: Agent [1] and Follower [2]. The first baseline model, Agent, achieves an SR of 51.1 on R2R val unseen. After applying our FDA enhancement, the SR of the model increases to 54.5, a notable 3.4 improvement. The second baseline model, Follower, achieves an SR of 38.3 on R2R val unseen. After FDA enhancement, the SR of the model increases to 45.6, representing an impressive 7.3 improvement. These results demonstrate the effectiveness of our method not only for Transformer-based models but also for LSTM-based models. We will include these suggested experiments in the revised version.\n\n[1] S. Shen, et al. How much can CLIP benefit vision-and-language tasks? ICLR. 2022.\n\n[2] D. Fried, et al. Speaker-follower models for vision-and-language navigation. NeurIPS. 2018.\n\n**Q2: In Table 2, DUET+FDA exhibits a significant performance drop on the val-seen set compared to the original DUET, e.g., SPL from 73.71 to 69.66.**\n\nA2: Thank you for your comment. The phenomenon mentioned is a result of severe overfitting of DUET to the val seen split. The issue of overfitting in seen environments can impact the assessment of the model’s true navigational capability. Thus, seen environments have less significance as a reference in the VLN task. Instead, VLN unseen environments are the main way to evaluate navigation ability. This is because navigating unseen environments best reflects the model’s generalization ability. On the VLN competition leaderboards (both R2R and RxR), agents’ navigation capabilities are only evaluated and ranked in unseen environments.\n\nHence, we base our best model selection solely on VLN’s performance in the val unseen split. In practice, during the training process, DUET+FDA exhibited an SR of 78.55 on val seen and 71.48 on val unseen. Nevertheless, due to its underperformance in the unseen environments compared to the model reported (val seen SR 76.00, val unseen SR 71.86) in our paper, its navigational ability is considered inferior. On validation unseen split, our method, applied to HAMT, DUET, and TD-STP, substantially improves the SR by 1.88, 1.66, and 2.13, respectively. Notably, our method significantly improves the performance of TD-STP to achieve a new SoTA performance.\n\nMoreover, our method can effectively mitigate overfitting in models. It reduces the gap in SR between the seen and unseen splits compared to the baseline models, reducing it from 9.38, 8.35, and 6.65 to 8.29, 4.14, and 5.89, respectively.\n\n**Q3: Lacking experimental comparison on the high-level instruction REVERIE dataset.**\n\nA3: Thanks for your valuable suggestion and we will add it to our revision. For your reference, we have conducted experiments on the REVERIE dataset using the SoTA model DUET. Based on our method, the model’s performance on SR, SPL, RGS, and RGSPL metrics shows significant improvement compared to the baseline model, with an increase of 2.69/3.54/3.18/3.45, from 44.65/32.92/28.57/21.01 to 47.34/36.46/31.75/24.46. The comprehensive improvement of the agent’s navigation capabilities in the R2R, RxR, CVDN, and REVERIE tasks thoroughly validates the effectiveness of our method.\n\n**Q4: As shown in Table 2 of the supplementary material, different random seeds have a significant impact on the model performance. For example, seed-3 and seed-4 differ by 1 in SPL.**\n\nA4: Thank you for your comment. The fluctuations observed in the navigation metrics within Table 2 of the Appendix are considered normal. The average values for SR and SPL are 71.78 and 63.64, respectively. The maximum deviations of SR and SPL from their respective means, across the four experimental groups, are only 0.76, which is a reasonable value. We notice that HAMT [1] and EnvAg [2] also reported multiple experimental results, and our deviation values are quite similar to them. SR deviation reaches 1.4 and 2.14 for HAMT and EnvAg, while SPL deviation hits 1.1 and 2.12 for the same.\n\n[1] S. Chen, et al. History aware multimodal transformer for vision-andlanguage navigation. NeurIPS. 2021.\n\n[2] X. E. Wang, et al. Environment-agnostic multitask learning for natural language grounded navigation. ECCV. 2020.\n\n**Q5: The writing needs to be further polished, e.g., the highest accuracy numbers in all tables should be bolded, and the title of section 3.4 should be ’Comparison with the State-of-the-art Methods’.**\n\nA5: Thank you for your valuable suggestions. We are committed to earnestly incorporating your suggestions to enhance the clarity and precision of our expression.\n\n**Q6: Missing numerous references such as [1-4].**\n\nA6: Thank you for the reminder. We sincerely apologize for overlooking these four outstanding and interesting works. Among them, two CVPR 2023 papers were not published at the time of our submission. We will include proper citations for all of them in the revised version.\n\n[1] Gao, Chen, et al. Adaptive Zone-Aware Hierarchical Planner for Vision-Language Navigation. CVPR. 2023.\n\n[2] Qiao, Yanyuan, et al. HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation. TPAMI. 2023.\n\n[3] Chen, Jinyu, et al. Reinforced structured state-evolution for vision-language navigation. CVPR. 2022.\n\n[4] Wang, Xiaohan, et al. LANA: A Language-Capable Navigator for Instruction Following and Generation. CVPR. 2023."}, {"review_id": "QfzK1HG574", "reviewer": "Reviewer_LWMZ", "summary": "In this paper, the authors propose a Frequency-enhanced Data Augmentation (FDA) for VLN models to enhance their ability to capture high-frequency information, thus improving their performance. First, at each step, they take the view as Reference Image and randomly sample an image as Interference Image. Then they transform these two images to the frequency domain space via FFT and mix the high-frequency components of two images. Finally, they obtain the augmented image via iFFT and alternately use original views’ images and augmented images to train the agent. the Experiments on different benchmarks reveal the proposed augmentation method is helpful to VLN models. However, I have some concerns about this paper. My detailed comments are as follows.", "questions": "The manuscript is well organized and the proposed consistent improvement upon several strong baselines on 3 datasets. It would be better for the authors to evaluate the proposed FDA technique on other multi-model tasks, not limited to the VLN task.", "limitations": "The authors have adequately addressed the limitations", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The authors take an analysis of frequency information domain by fuse images’ high-frequency or low-frequency components into original images, and find the importance of high-frequency for the agent’s better performance in VLN tasks.\n\n2. The authors propose a simple data augmentation method, Frequency-enhanced Data Augmentation (FDA), which needs no external models and does not add complexity. The proposed method enhances the models’ ability to capture necessary high-frequency information and has been demonstrated to improve the models’ performance on multiple datasets.", "weaknesses": "1. I wonder why the authors choose to incorporate the high-frequency components from randomly sampled interference images into the original image, rather than using a fixed interference image, or utilizing the high-frequency components of the image itself, or randomizing certain high-frequency components of the original image. It would be better if the author could discuss these approaches or include them in the ablation experiments.\n\n2. While the importance of high frequency compared to low frequency has been demonstrated by other experiments, it would be beneficial to conduct an ablation experiment utilizing low frequency for data augmentation under the same settings.\n\n3. Some novel SoTA models are not shown in the tables in this paper, such as Airbert [1] on R2R benchmark. It would be better for the authors to add these state-of-the-art models into the comparative experiments which would demonstrate the strengths and limitations of the proposed method more comprehensively. Also, it is better to add the brief introduction about the VLN works on continued environment, such as [2,3] for the sake of completeness\n\n4. There appear several low-level calculation/statistical errors that should have been avoided in the paper. For instance, on line 216, it should be \"4 and 3 points,\" and on line 222, it should be \"3.3 and 3.3 (2 and 1.7).\" Additionally, on line 178, it should be \"boost the GP performance by 0.36.\" The author should carefully review the data in the tables and the calculations presented throughout the text.\n\n\n**Minor issues\n\n1. In Equation 2, since Equation 1 already uses $F_{\\hat{I}}^{\\{rgb\\}}$ to represent the operation result of $\\mathcal{F}^{\\{rgb\\}}(\\hat{I})$, it would be better to use $F_{\\hat{I}}^{\\{rgb\\}}$ instead of $\\mathcal{F}^{\\{rgb\\}}(\\hat{I})$ in Equation 2.\n\n2. On page 5, line 156, “Path Lengt” should be “Path Length”.\n\n3. On page 6, line 178, “boost the GP performance by 3.6” should be “boost the GP performance by 0.36”.\n\n4. On page 8, line 216, “by 4 and 2 points” should be “by 4 and 3 points”.\n\n5. On page 8, line 222, “by 3.4 and 3.3” should be “by 3.3 and 3.3”.\n\n[1] Airbert: In-domain Pretraining for Vision-and-Language Navigation, ICCV 2021\n\n[2] Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation, NeurIPS 2022.\n\n[3] Cross-modal Map Learning for Vision and Language Navigation, CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: I wonder why the authors choose to incorporate the high-frequency components from randomly sampled interference images into the original image, rather than using a fixed interference image, or utilizing the high-frequency components of the image itself, or randomizing certain high-frequency components of the original image. It would be better if the author could discuss these approaches or include them in the ablation experiments.**\n\nA1: Thank you for your intriguing suggestion. The reason we randomly incorporate the high-frequency components of the interference image into the original image is to provide a more diverse set of high-frequency negative samples. In response to your suggestion, we conduct experiments under three settings: (1) Setting 1 - only using a fixed interference image’s high-frequency components as interference; (2) Setting 2 - using the high-frequency components of one channel in the image as interference for other channels; (3) Setting 3 -randomizing certain high-frequency components of the original image. In Setting 1, the model achieves SR score of 70.92. In Setting 2, the score is 69.26, and in Setting 3, the score is 69.73. Compared to the baseline model whose SR is 69.65, we observe a slight decline in performance in Setting 2, a slight improvement in Setting 3, and the most significant improvement in Setting 1. This suggests that incorporating high-frequency negative samples from other images effectively enhances the model’s ability to capture the required high-frequency information and improve cross-modal alignment. Comparing our method with Setting 1, we achieve superior performance with SR score of 71.78. The diverse sampling of high-frequency negative samples further improves the model’s ability to capture the necessary high-frequency information and enhance cross-modal alignment. We will add this discussion to the revision as suggested.\n\n**Q2: While the importance of high frequency compared to low frequency has been demonstrated by other experiments, it would be beneficial to conduct an ablation experiment utilizing low frequency for data augmentation under the same settings.**\n\nA2: Thanks for your valuable suggestion. In this experiment, we achieve an SR of 69.52. The result exhibits a minor decrement when compared to the baseline’s SR of 69.65, and notably trails behind the remarkable SR of 71.78 attained through high-frequency-based data augmentation. These findings will be incorporated into the revised version as recommended.\n\n**Q3: Some novel SoTA models are not shown in the tables in this paper, such as Airbert [1] on R2R benchmark. It would be better for the authors to add these state-of-the-art models into the comparative experiments which would demonstrate the strengths and limitations of the proposed method more comprehensively. Also, it is better to add the brief introduction about the VLN works on continued environment, such as [2,3] for the sake of completeness。**\n\nA3: Thanks for your valuable suggestion. We apologize for the oversight. Airbert [1] is an outstanding work, and we will include a comparison with it in the revised version. Additionally, as per your suggestion, we will introduce an overview of research on VLN-CE, such as [2,3].\n\n[1] Guhur, Pierre-Louis, et al. Airbert: In-domain pretraining for vision-and-language navigation. ICCV. 2021.\n\n[2] Chen, Peihao, et al. Weakly-supervised multi-granularity map learning for vision-and-language navigation. NeurIPS. 2022.\n\n[3] Georgakis, Georgios, et al. Cross-modal map learning for vision and language navigation. CVPR. 2022.\n\n**Q4: There appear several low-level calculation/statistical errors that should have been avoided in the paper. For instance, on line 216, it should be \"4 and 3 points,\" and on line 222, it should be \"3.3 and 3.3 (2 and 1.7).\" Additionally, on line 178, it should be \"boost the GP performance by 0.36.\" The author should carefully review the data in the tables and the calculations presented throughout the text.**\n\nA4: Thank you very much for your thorough review. We will definitely correct these errors in the revised version.\n\n**Q5: Minor issues.**\n\nA5: Thank you very much for your attention to detail. We will correct these errors in the revision.\n\n**Q6: The manuscript is well organized and the proposed consistent improvement upon several strong baselines on 3 datasets. It would be better for the authors to evaluate the proposed FDA technique on other multi-model tasks, not limited to the VLN task.**\n\nA6: Thank you for your sincere and enlightening feedback. In response to your suggestions, we conduct additional experiment on the REVERIE task using the SoTA model DUET. Based on our method, the model’s performance on SR, SPL, RGS, and RGSPL metrics shows significant improvement compared to the baseline model, with gains of 2.69/3.54/3.18/3.45, from 44.65/32.92/28.57/21.01 to 47.34/36.46/31.75/24.46. The comprehensive improvement of the agent’s navigation capabilities in the R2R, RxR, CVDN, and REVERIE tasks thoroughly validates the effectiveness of our method. In the next phase of our research, we will further extend the findings of this study to a broader range of cross-modal tasks."}, {"review_id": "aBYlJiUOYA", "reviewer": "Reviewer_bEiK", "summary": "Authors propose a Frequency enhanced Data Augmentation (FDA) technique to improve generalization/robustness to unseen environments in the vision-language navigation task. This work provides an initial analysis that shows high frequency information impacts performance. Results are provided on R2R, RxR, and CVDN datasets and are better than prior work when using the same models. ", "questions": "**Questions:**\n- Figure 1 and the original analysis that founds this paper begs the question: how much of the performance effect shown by high frequency perturbation is caused by loss of color in the resulting image? What if you made the image black and white, or the low frequency image black and white? The analysis feels lacking to truly demonstrate that it is frequency information that matters.\n- Did you try other methods of fusing the high frequency information into the training images? Or other training regimes instead of alternating the images per even and odd training steps per Eq. (6)?\n- Are all of the results for a single experiment run?\n\n**Comments and Suggestions:**\n- L38-L40, I suggest notating where in the Figure to look. I.e. \"Figure 1 (middle)\" or \"Figure 1 (right)\" instead of repeating just \"Figure 1\".\n- L59-64 is a very long single sentence, it should be broken up to several shorter sentences to make it easier to follow.\n- Figure 4: there is room within the figure to have an abbreviation legend since there are a ton of acronyms in the figure that are not defined in the figure caption - i.e., what is GHPF? FFT? etc. They are defined in earlier figures but it would be better to have them defined within the scope of this figure, too.\n- It would be better if figures appeared on pages they were referenced.\n- L111-118, I am not really sure what the difference is between the points being made in 2) and 3).\n- The notation in Eq. (2): It switched between using the $F$ notation for frequency and the $\\mathcal{F}$ notation for FFT. Isn’t the high and low frequency $H$ on the original image $I$, while $\\hat{H}$ should be on the interference image $\\hat{I}$? Right now $H$ and $\\hat{H}$ have the same definition in the equation.\n- L156 \"Lengt\" --> \"length\", L153 \"7k\" --> \"7,000\", no comma in \"2050\" --> \"2,050\"\n- L158 \"Process\" --> \"Progress\"?\n- L178 says GP performance increases by 3.6, but the table shows a **0.36** improvement.\n- L230 add a space after \"et al.\"; generally throughout paper this is an issue.", "limitations": "The initial analysis prompting this work is not very convincing of the importance of high frequency information in images, and the paper does not compare to multiple data augmentation techniques. It could have additional baselines for simple image augmentations or other augmentation methods from prior work. This is particularly important because the performance improvement is not very large across the different models and datasets. Given the small performance differences, if the reported results are from a single experiment it is not convincing that the FDA approach is consistently effective. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- Experimentally the authors demonstrate that the proposed FDA data augmentation technique improves performance on most of the models and validation splits.", "weaknesses": "- The premise of the paper is slightly confusing. The high and low frequency perturbed images essentially are different domains, so of course performance drops. Additionally, counter to what the writing of the paper details, low frequency information also helps on SR for the val seen split in Table 1, and the unseen results vary but < 1 point on SR.\n- FDA can hurt performance or doesn’t improve as much on seen compared to unseen results (e.g., R2R DUET, val seen).\n- Table 4. It seems like an obvious result that a method trained with FDA is going to do better on high frequency perturbed images, this table may not be necessary or at least does not need to be included in the main text. Are there downstream scenarios where this kind of perturbation would even realistically happen?\n- While many types of data augmentation are referenced in the related work section, the authors only experimentally compare to one (ENVEDIT) data augmentation work, and do not try baselines with other data augmentations that may affect similar information to the high frequency perturbation. Because of this, it feels the experiments are incomplete, and generally underwhelming due to the small SR performance changes.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Answer to Weakness-1:** Thanks for your comments. Figure 2 reveals important insights into the behavior of baseline models: HAMT, DUET, and TD-STP. These models show relatively high SR under low-frequency perturbations but experience significant SR drops under high-frequency perturbations. This indicates VLN models’ deficiency in effectively capturing the required high-frequency information.\n\nIn Table 1, the small 0.19 SR improvement on val seen might stem from overfitting, given the significant 0.9 SR drop on val unseen. In VLN, navigation ability is primarily evaluated using unseen environments for better generalization. On val unseen, TD-STP+high frequency shows a 0.72 SR increase over the baseline and a 1.62 improvement compared to TD-STP+low frequency. This indicates high-frequency plays a more crucial role than low-frequency in cross-modal navigation.\n\nConsidering the above-mentioned reasons, we design our approach to enhance the agent’s ability to capture crucial high-frequency information necessary for cross-modal navigation.\n\nMoreover, an improvement in SR and SPL on a strong (SoTA) baseline (SR 69.65, SPL 62.97), with gains of 0.72 and 0.65, reaching 70.37 and 63.62, constitutes a noteworthy advancement. Furthermore, TD-STP+HF outperforms TD-STP+LF by 1.62/1.66 in SR/SPL on val unseen, showcasing substantial enhancement.\n\n**Answer to Weakness-2:**  Thanks for your concern. The mentioned phenomenon results from severe overfitting of DUET to val seen. This overfitting issue in seen environments distorts the model’s true navigational capability assessment. Thus, seen environments have less significance as a reference in VLN. Instead, unseen environments are the main way to evaluate navigation ability, as they best reflect generalization capacity. On the VLN official competition leaderboards, agents are only evaluated and ranked in unseen environments.\n\nHence, our optimal model choice is solely based on val unseen performance. In practice, during training, DUET+FDA exhibited 78.55/71.48 SR on val seen/unseen. Due to its underperformance on the val unseen compared to the paper’s reported model (76.00/71.86 on validation seen/unseen), its navigational ability is deemed inferior. For val unseen, our method, applied to HAMT, DUET, and TD-STP, notably enhances SR by 1.88/1.66/2.13, respectively.\n\n**Answer to Weakness-3:**  Table 4 is to illustrate FDA-enhanced models excelling in both the unseen perturbed images (val unseen) and the unseen perturbation images (ImageNet). While such scenarios may not exist in reality, they serve as compelling evidence of our method’s remarkable capability to capture essential high-frequency information for cross-modal navigation. This consideration led us to include it in the main manuscript.\n\n**Answer to Weakness-4-Part1:** Since no existing VLN data augmentation method aligns with our approach, which uniquely requires no additional parameters, data, or external models, comparisons with such methods are not available. Instead, we conduct a comparison with ENVEDIT, the only method sharing a similar image-level data augmentation strategy with ours.\n\nFor reference, we’ve added comparisons between the FDA method with two other methods: Speaker [11*] and PREVALENT [13*], as shown in Table 1 of the attached PDF, using Baseline-1 [11*] and Baseline-2 [53*] models (* indicates the reference number in the main paper.). The result shows that our method achieves an additional gain of 5.2 compared to the Speaker based on extra model. Furthermore, our approach can effectively complement the two data augmentation methods.\n\n**Answer to Weakness-4-Part2-[small SR performance changes]:**  Our method applied to HAMT, DUET, and TDSTP significantly boosts val unseen performance of these SoTA methods, with SR increases of 1.88/1.66/2.13 respectively. Our improvements compared to the previous SoTA methods (DUET, TD-STP) rival or even surpass that of previous studies [29*, 37*, 50*, 1, 2, 3] (mainly below 2 points). Additionally, we experiment on two LSTM-based models: Baseline-1 [11*] and Baseline-3 [39*], as depicted in Table 2 of the attached PDF. These two models represent impressive 7.3/3.4 improvements.\n\n[1] KERM: ... CVPR. 2023.\n\n[2] GeoVLN: ... CVPR. 2023.\n\n[3] Local Slot Attention for VLN. CVPR. 2023.\n\n**Answer to Question-1:** We use the method in Figure 4 to generate high-frequency interference (HFI) and low-frequency interference (LFI) images for Figure 2. See the “Augmented Image\" in Figure 4 for the HFI image. Since the image with HFI retains low-frequency components where the color information mainly resides (as illustrated in Figure 1), the color remains largely intact in the HFI image. However, in Figure 2, the model performs notably worse with HFI scenarios compared to LFI scenarios. This underscores that high-frequency, not color, is a key factor influencing cross-modal navigation.\n\n**Answer to Question-2:** We attempted to introduce high-frequency interference using weighted fusion on original images. The SR on val unseen was 70.54, notably below our method’s 71.78. Thus, we chose the current approach. Additionally, we experimented with random training on original and augmented images during navigation. The result is slightly inferior to the current method, with a 71.65 SR.\n\n**Answer to Question-3:** In Appendix Section B, we experiment with four seeds. Results consistently show notable navigation performance enhancement. SR improves from 1.74 to 2.89, with a mean increase of 2.13.\n\n**Answer to Comments and Suggestions:** Regarding 2) and 3) (L111-118): 2) highlights high-frequency information’s role in enhancing robustness for performance under environmental variations. 3) shows high-frequency information’s role in reducing overfitting for performance in new environments. We sincerely appreciate your thoughtful feedback regarding the issues with formulas, spelling, formatting, and expression. We will earnestly make the necessary revisions as suggested."}]

Author rebuttal:
Dear Chairs and Reviewers,

We would like to express our heartfelt gratitude to you for managing this paper and dedicating your valuable time to providing insightful comments. We also extend our sincere appreciation to the reviewers for recognizing the significance of our work. We have taken meticulous care in addressing all the concerns raised by the reviewers. For detailed information regarding the specific concerns, we kindly direct you to the Rebuttal Section.

The attached PDF file includes Tables 1-3 showing our experimental results:

1. Table 1 shows the comparison between our FDA method with two classic data augmentation methods: Speaker and PREVALENT.

    The result shows that our method, **relying on no additional parameters, data and external model**, achieves an **additional gain of 5.2 on SR** compared to the Speaker based on extra model. Furthermore, our approach can effectively **complement the two data augmentation methods**.

2. Table 2 represents the result of different LSTM-based models enhanced by the FDA method on R2R. 


    The two LSTM-based models represent **impressive 7.3/3.4 improvements on SR**, demonstrating **the effectiveness of our method not only for Transformer-based models but also for LSTM-based models**.

3. Table 3 shows the results of our FDA method on the REVERIE task with the SoTA baseline DUET.
 
    Based on our method, the model’s performance on SR, SPL, RGS, and RGSPL metrics shows significant improvement compared to the baseline model, with **significant gains of 2.69/3.54/3.18/3.45**, from 44.65/32.92/28.57/21.01 to 47.34/36.46/31.75/24.46. The comprehensive improvement of the agent’s navigation capabilities in **the R2R, RxR, CVDN, and REVERIE tasks thoroughly validates the effectiveness of our method**.

If you have any further questions, we are more than willing to engage in further discussions.

Best wishes,

Paper12863 Authors

Meta-review: This paper proposes Frequency-enhanced Data Augmentation (FDA) technique aimed to improve generalization, robustness, and performance on in VLN. Results on R2R, RxR, and CVDN empirically show that capturing critical high-frequency information w/ FDA helps.

The committee found the paper to be (1) well-written, (2) backed with strong suite of experiments across multiple base models and multiple VLN datasets, (3)  is a lightweight edit to improve performance across multiple base models, (3) investigates a novel aspect of this vision-language-embodiment task probing importance of high-frequency visual information and correspondence in text. 

The discussion phase helped resolve concerns of the reviewers and revealed helpful updates to the draft.

The AC concurs with the unanimous decision of the reviewers and recommends acceptance. Also, the AC requests the authors to follow through on the promised revisions in the rebuttals (clarity, failure cases, new experiments, etc.).
Decision: Accept (poster)