Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed history-aware multi-modal transformers for vision and language navigation tasks. Different from prior work (RecBert, PREVALENT), the proposed method takes long-horizon history with vision transformers and spatial relationship prediction proxy tasks. The proposed method achieves the state of the art performance on VLN R2R tasks. 
main_review: [Strength]

- Very good performance on VLN R2R datasets. 

- The idea of encoding the whole history as input is intuitive, and the paper is well written and the authors perform extensive ablation study to verify the effectiveness of each module. 

- The author also try with different visual backbone and imitation learning and A3C for the tasks. 

[Weakness]

- In the contribution section, the author claims their model is the first fully transformer-based model for VLN tasks. This is not super clear to me, it will be better to illustrate more precisely. 

- Since the model encodes the whole history image as input, I wonder what is the computation time during inference time. 
limitations_and_societal_impact: Yes, the author addresses the limitation and potential negative social impact of the work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a history-aware multimodal transformer (HAMT) style model for Vision-and-Language Navigation task which explicitly takes the history of observations as the input for the transformer model. HAMT efficiently encodes all past panoramic observations using a hierarchical vision transformer. It consists of unimodal transformers for text, history (of panoramic visual observations), and current visual observation and then fuses these representations together using a cross-modal transformer. They also propose additional auxiliary proxy tasks for end-to-end training of HAMT. These are Single-step Action Prediction & Regression (SAP/SAR) and Spatial Relationship Prediction (SPREL). Through extensive ablations, they show that how each of their contributions makes sense and gets state-of-the-art on the single-run R2R task, R4R, and R2R-Back & R2R-Last (two augmented tasks they described in the paper).
main_review: ### Originality and Significance:
-	This paper proposes a novel modelling approach where they explicitly model the history of panoramic visual observations in addition to instruction and current visual observation. Their “history-aware” transformer style model first encodes individual images with Vision Transformer and then combine it with unimodal transformer output of instruction and current visual observation using a cross-modal transformer.
-	They also propose 2 new auxiliary tasks for pretraining their model: Single-step Action Prediction/Regression and Spatial Relationship Prediction.
-	They show results on 2 VLN tasks: R2R and R4R. 
    -	Using ablations, they show that their history encoding for VLN improves Success Rate by 4.2% and 2.1% SPL over temporal-only baseline.
    -	They show that using all their proxy tasks and visual representation finetuning provides gains of 12.2% SPL (48.7 -> 60.9) and 11.3% SR (54.4 -> 65.7).
    -	They achieve state-of-the-art on R2R and beats recurrent-VLN-BERT by 3% on SPL and 2% on SR.
    -	They get state-of-the-art on R4R and two additional VLN setup tasks (introduced in the paper), R2R-Back where the agent needs to remember the trajectory and come back to starting position, R2R-Last where only the last instruction is provided.

### Quality and Clarity:
The paper is well-written, is clear about their claims and explanations and is easy to follow.

### Typos:
L98: and-to-end -> end-to-end

limitations_and_societal_impact: None
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a History Aware Multimodal Transformer to incorporate a long-horizon history into multimodal decision making. The authors perform experiments on three VLN datasets (R2R, R4R, and R2R-last), and demonstrate that HAMT is particularly effective for navigation tasks with longer trajectories.

main_review: This paper introduces a History Aware Multimodal Transformer to incorporate a long-horizon history into multimodal decision making. The authors perform experiments on three VLN datasets (R2R, R4R, and R2R-last), and demonstrate that HAMT is particularly effective for navigation tasks with longer trajectories.

Strengths:
- A new multimodal transformer based method for VLN is introduced particularly for solving the longer paths.
- The proposed HAMT outperforms the state of the art and achieves excellent results on three VLN datasets. 
- Training HAMT with auxiliary proxy tasks is interesting.

Weaknesses:
- More clarification and ablation study is needed about the training strategy. Have the authors tried to train HAMT in a single stage end-to-end rather than the current two-stage strategy?
- Experimental details were not made clear and many important hyper-parameters are missing, making this paper hard to reproduce. Are all the Panoramic Transformer layers randomly initialized, or are they initialized from certain checkpoints? How do you initialize the discount factor? Besides, the authors introduce a new VLN framework, thus it is important to report some model details and training details, for example, the numbers of model parameters, memory cost and inference time (the model seems to be rather complex), and so on.
- Line 160, how do you get the special token? From my guess you are treating it as a totally free parameter during training. If so, how do you initialize it?
- The presentation of the paper can be further improved. For example, the authors could give a brief formulation of Temporal Transformer, Panoramic Transformer, and Multimodal Transformer first, and then give a unified and specific description.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a transformer-based model for VLN, which makes use of history trajectory information. Panoramic visual observations of all passed viewpoints are embedded by two transformer modules, one handling panorama at each step and the other one handling panorama representations of all steps. The well-known pre-training technology is employed. Experiments on R4R, R2R, and R2R variants are conducted.
main_review: The main contribution of this work are twofold: (1) introducing navigated trajectory into transformers in the form of hierarchical fashion; (2) employing the pre-training technology. Experiments are conducted on two commonly used datasets R2R and R4R. The authors also created two additional datasets, R2R-Back and R2R-Last, to further demonstrate the effectiveness of the proposed history-aware model.

The writing is not very clear as some crucial details are missing. For example, (1) Do ViT and Panoramic Transformer are shared between time steps as shown in Figure 2a? (2) Is the panorama at t=0 fed into the model for steps after t=0? If so, is the repeated calculation necessary? (3) Regarding observation encoding, 36 view embeddings are added with navigable embeddings as shown in Figure 1. However, one view may have zero or multiple navigable viewpoints. How are such cases handled by the proposed model? (4) How do you guarantee and verify the information is not lost regarding the claim in line 156 "In this way, xxx without information loss".

Regarding the performance, the main contribution hierarchical history module only brings 1%~2% SPL improvement on unseen split (which is the main reference) as shown in Table 2 (row Recurrent vs row Hierarchical), but the pre-triaing technology brings about 10% improvement. Performance is further improved if ViT is used. However, the pre-training technology is not the contribution of this work as they are taken from existing work. Thus, the claimed contribution is not matched with the experimental results. Furthermore, some ablation studies are missing, such as for the ViT setting in Table 3a, how about remove both PT and e2e? What are the contributions of SAP and SAR  respectively to the final performance?  In addition, it is weird not using the final model (ViT+PT+e2e) to do the ablation study in Table 3b.

For comparisons on R2R-Back and R2R-Last, it is not very fair for existing methods as they are not designed for such tasks. In addition, do you retrain them on R2R-Back and R2R-Last datasets?

Overall, I think this work is not ready for publication based on the novelty, writing, technique contribution, and experiment issues listed above and below.

Other issues:
1. Why zero out v_*^o or a_*^o (line 198) for spatial relationship prediction? How does it affect the performance if not zero out them?
2. Is there any explanation for the performance drop when optimizing visual representation in ViT setting of Table 3a?
3. The explanation for marginal SPL improvement in line 328 is not reasonable, because for the R2R task the SPL metric is a good measurement for path fidelity as the ground-truth path is exactly the shortest path. 
4. Since pre-training technology is utilised, the proposed model should be tested on more downstream tasks, such as Touchdown, REVERIE, CVDN, etc.
limitations_and_societal_impact: Thanks for the response from the authors. Most of my issues have been addressed, such as more results on downstream tasks and clarifications about model details. Overall, I think this is a good work but not good enough for NeurIPS. This is because most of the performance comes from the application of existing pre-training instead of the proposed history-aware scheme (10% vs 2% improvement), and there is no novelty in neither pretraining model design nor loss functions. 

I have read initial comments from other reviewers, which tend to be positive. However, if this paper does not get lucky due to my review, I suggest rewriting the paper in line with the contribution importance of each proposed module, and paying extra attention to the newly designed xxx-back VLN task which usually can be a separate article.
review_title: null

Meta-review: This paper presents an effective history-aware multi-modal transformer for VLN tasks, which captures well the long-horizon history and spatial relationships.  This method efficiently encodes all past panoramic observations using a hierarchical vision transformer, with promising performances on a variety of datasets.  It achieves SOTA on R2R.  The main concerns raised by reviewers are the unclear writing and the relatively smaller gap from using a new history-aware scheme compared to pretraining models. The authors later provide extensive experiments to demonstrate the generalization capability and large improvements by their modules in more downstream tasks and more diverse datasets.  The experiments indeed validating well the contribution of the proposed transformers. Considering the positive comments by three reviewers and the responses solve the most issues by reviewer 7aBY, the AC thus recommends accepting this paper.
Decision: Accept (Poster)