Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a new fine-grained VLN dataset called Landmark-RxR which builds on top of RxR. Using the fine-grained annotations, they propose two kinds of focal-oriented rewards for improving local cross-modal alignment. They also propose a re-initialization mechanism to evaluate the navigation such that it is insensitive to difficult points.
main_review: ### Originality and Significance:
-	Previous papers in VLN space have tried to split the instructions to sub-instructions using heuristics, but it is the first paper to create sub-goal level human annotations for sub-instructions for the instructions in RxR. This dataset will provide more fine-grained supervision for cross-modal alignment training.
-	Through their experiments, they support the claims that they make. 
    -	Fine-grained data contribute to instruction domain generalization: They show a 7.1% increase in performance when using R2R data and sub-instructions from Landmark-RxR as opposed to just using R2R instructions or R2R+RxR instructions.
    -	Comparison with SoTA: Their model with soft focal-oriented rewards outperforms the RCM model by 4.9 in sDTW and 8.1% on SR.


### Quality and Clarity:

The paper is well-written and easy to follow. 

### Typos:
-	Table 4 Column heading should be “Validation Unseen”. It is currently “Validation Seen”.

limitations_and_societal_impact: None
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes augmenting the en-RxR dataset by splitting the trajectories into finer grained sub-trajectories (by human annotators). It also introduces soft and hard focal reward formulations for their policy gradient agent which seem to do well empirically. 
main_review: Observing that cross-modal alignment is an import aspect leading to higher performance in VLN metrics, this paper makes meaningful contributions both in terms of adding more fine-grained data to RxR dataset and exploiting this additional signal to improve performance on the original task. I have a few suggestions below:

Suggestions

- Can you split Table 3 and report numbers for en-RxR and Landmark-RxR separately?

- Presentation
  - Remove all occurrences of "obviously"
  - There are several typos throughout the manuscript, please correct those

- Related work section missing relevant citations:

  - Transferable representation learning in vision and-language navigation. Huang et al
  - Multi-modal discriminative model for vision-and-language navigation. Huang et al
  - Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments. Chen et al
  - Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View. Mehta et al
limitations_and_societal_impact: No negative societal impact of this work
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes to improve vision and language navigation by augmenting the RxR dataset with corresponded subinstructions and subtrajectories.  Using the notion of critical points (points that are more important/helpful in the navigation process), two kinds of focal-oriented rewards (soft and hard) are proposed to encourage the predicted trajectory to be close to the demonstration trajectories at critical points.  The authors conduct experiments on R2R and RxR and show that using the collected subinstructions and subtrajectories and proposed losses, the trained agent is able to outperform the baseline RCM model.  For the RxR dataset, the training data consisted both of original RxR data as well as an augmented dataset consisting of concatenating the collected subinstruction and subtrajectories together.

Main contributions
- Landmark-RxR: Additional annotations on top of the RxR dataset providing sub-instructions paired with sub-trajectories
- Proposed to use modified rewards for improved cross-modal alignment
- Ablation experiments of proposed method on the R2R and RxR dataset
main_review: Originality: Limited.  Much of what is done in this work has been proposed in prior work (having paired sub-instruction and sub-trajectory data, concatenating simpler instructions/trajectories together, the focus-oriented reward is also similar to rewards from prior work).  The main novelty lies in the application sub-instructions to the RxR dataset, using the sub-instructions to generate more novel combinations, and having the reward to restricted to a set of critical points. 

Quality: There are missing discussion and comparisons wrt prior work.  There is no discussion on the limitations and weaknesses of the work.

Clarity: The writing is hard to understand at places.  The paper is also missing some important details (see "Other comments and questions" for details)

Significance: The Landmark-RxR dataset will be useful for the community.  The significance of the proposed method is unclear.

Strengths:
- The collected Landmark-RxR dataset would be valuable
- Experiments show that using the subinstructions and the proposed focal losses improves over the baseline RCM model

Weaknesses: 
- Missing detailed discussion and comparison against related work (Jain et al, 2019, Ilharco et al, 2019 both used fidelity-oriented rewards with RCM).  There should be more detailed description of how this work relates and clear experiments comparing against the prior work. 
- Weak experimental setup.
  - Missing comparisons against relevant prior work.  This includes the close works of RCM with fidelity oriented reward [Jain et al, 2019, Ilharco et al, 2019], as well as other prior work using sub-instructions (such as BabyWalk [Zhu et al, 2020], [Hong et al 2020]).
  - Proposed method appears to considerably underperform prior work with RCM (Wang et al, 2019, Jain et al, 2019, Ilharco et al, 2019)
  - For Table 4 (RxR against SOTA), results are reported only for on set (not all sets of validation seen, unseen, testset).  It's also not clear which set Table 4 numbers corresponds to (Table 4 indicates validation seen, but the test (lines 295) imply validation unseen - actual numbers appear to match val-unseen).  
- There is no discussion of training variance
- Contribution of the method is unclear
- The writing is also poor, making it difficult to understand at places.  There are also terminology/wording that are not properly introduced and thus unclear. 

Other comments and questions:

- What is the relationship between critical points and landmarks?

- When is the re-initialization mechanism used?  Is it only for evaluation?  Or is it used during training? inference?

- Is there a reason why the loss number metric is not normalized against the number of landmarks?  The lack of normalization makes it challenging to intuitively determine how good the agent is without some understanding of the total number of landmarks.

- It's unclear how the sub-trajectory accuracy is computed.  Is this an average of the nDTW of all sub-trajectories?  If it is a nDTW, then the name "sub-trajectory accuracy" does not seem accurate (wouldn't "sub-trajectory nDTW" be better?)

- How does the proposed soft and hard focal-oriented reward relate to the fidelity-based reward from Jain et al, 2019?

- Line 201: What specifically is the distance function used? The L2 Euclidean distance?
- Line 204: How are the critical points sampled? Is a random set of the landmarks uniformly sampled?

- In the experiments, the R_nDTW and R_SR rewards are not fully explained.  How do these rewards correspond to what was introduced in prior work?  The combination of R_nDTW and R_SR seem to work well (comparable with the proposed method).

- The subinstructions/subtrajectories are concatenated together for data augmentation.  Was there experiments comparing it against using data augmentation with the speaker-follower model?

- Line 254: The claim that performance of model#1 and model#4 are equivalent is inaccurate.  While they have similar SR for val-unseen, the performance of the two methods are clearly different for other metrics

- Overall the writing quality was poor, making it confusing and hard to understand.  See below for details:
  - From the introduction, it was hard to understand what exactly the dataset consisted of and what specifically was meant by various terms such as "fine-grained perspective" and "landmarks".  Overall, I found the term "landmark" to be very confusing as I would expect a "landmark" (for a indoor scene) to be say a "bed" or a "painting on the wall" and correspond to either visual landmarks or landmarks identified by text.  But what it seems to refer to is a location on the trajectory (defined in the paper as "marked end points of sub-trajectories").  The term "critical point" was also not precisely defined.   Would all landmarks qualify to be a "critical point"?

  - Line 57-58: "english Guide part" - the details of the RxR dataset has not been introduced so it is hard to understand what "english Guide" refers to. It's also unclear what "fine-grained" and "currently the largest scale" means.

  - The set of annotation principles should be described in the main paper, not the supplemental.  Overall, the description in the main paper was not very informative.  The information in supplemental is clearer and more informative.  In addition to the annotation principles, the supplemental also explains how it differs from prior fine-grained dataset in a clear way, which was lacking from the main paper.  Table 1 from the supplement should be in the main paper (Table 1 in the main paper is better suited to be in the supplement)

  - Line 245: It would make more sense if the "Hyper parameters" were described after the "Training policy" and before the "Evaluation metrics"

  - Some other example of phrases with poor wording that are difficult to understand:
    - Line 38: "no sub-instruction can we get from RxR" 
    - Line 85: "to navigate in recurrent mechanism like LSTM efficiently"
    - Line 95: "spatially-temporally aligned priori"
    - Line 96: "a plain result"
    - Line 183: "in reverse order originally"
    - Line 230: "The agent reasons in one stage and end-to-end manner"

  - Some examples of wording / typos
    - Line 3, 24, and others: "prediction trajectory" => "predicted trajectory"
    - Line 98: "Besides," => "In addition,"
    - Line 105: "supports" => "allows"
    - Line 106-107: "During the annotation progress" => "During annotation"
    - Line 111: "firstly" => ""
    - Figure 2 caption: "mean" => "indicate"
    - Line 157: "succeeds to reach within 3 meters" => "reaches within 3 meters"
    - Line 142: "firstly" => ""
    - Line 219: "the data contain two parts" => "the training data consists of two parts"
    - Table 2 caption: "have little help" => "provides little improvement"

   - References are not alphabetically sorted, references not also not capitalized correctly

Post-Rebuttal:
- I increased my rating from 4 to 5.  The clarification by the authors has mitigated some of my concerns.  However, I still believe that the exposition lacking in clarity and detail, with terminology that is not clearly defined.   Due to the contribution of the dataset and the authors clarifications, I increase my score to 5.   I believe the current submission is not clear enough and a revision is needed.  
limitations_and_societal_impact: There was no discussion of the limitations of the work or the any negative social impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles the problem of Vision and Language Navigation (VLN). The main contribution of the paper is a dataset called Landmark RxR. The instructions in the RxR dataset have been broken down into smaller segments by human annotators, where each segment corresponds to navigation to a landmark. A landmark is a point at which it is difficult to make a decision (due to ambiguity, clutter, etc). The paper proposes a "re-initialization" strategy to better evaluate VLN agents. Two types of reward functions based on the deviation from the landmarks are defined. The paper reports results on R2R, en-RxR and Landmark RxR datasets.
main_review: **Strengths**

- The fine-grained annotation of the RxR dataset can be useful in certain navigation scenarios.

- The paper shows state-of-the-art results on the en-RxR dataset.

- One of the proposed reward functions (Soft focal-oriented reward) provides significant improvements compared to standard goal-oriented or fidelity-oriented reward functions.

**Weaknesses**

One major issue is that it is not clear if the improvements are provided by the landmark annotations or just by breaking the long trajectories. Two baseline experiments should be added to the experiment section: (1) divide the instructions into equal size segments (2) divide the instructions into segments of random size (ending at a period). Without these two experiments, it is hard to judge if the provided dataset is a useful contribution or not.

Also, there are several critical points that are not clear:

- Is the re-initialization mechanism used only for computing SA and SSA metrics? or is it used for computing the other metrics as well?

- The performance drops when complete instructions are used in Table 2 (model #5 vs model #6), but using complete instructions provides improvement in Table 3 (line 266). What is the explanation for that?

- Why does the method use only two points among the landmarks (line 250)?

- Regarding Table 2, which dataset has been used for rows that do not have a tick for R2R?

I will adjust my rating based on the rebuttal. There are also numerous typos.  

**Post-rebuttal comments**

The rebuttal addresses my concerns (the baseline experiments and the response to the questions) so I retain my initial rating. 
limitations_and_societal_impact: No, the paper does not address the limitations or the potential negative societal impact.
review_title: null

Meta-review: This work introduces a new resource for the English fold of RxR and shows how these annotations lead to significant gains in the original related domains/tasks.  The authors have provided substantial clarifications and new results during the discussion which are critical to understanding their work.  These need to be included in the final revision.  Note, in particular, many of the concerns raised by dQiv and the results for gPfu.  Some discussion of whether the resource can provide utility to the full RxR task (not just English) would also be appreciated.
Decision: Accept (Poster)