Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes VXN, a large-scale 3D indoor dataset for multimodal, multitask navigation in continuous and audiovisual complex environments. It can conduct four embodied navigation tasks, including image-goal navigation, audio-goal navigation, object-goal navigation, and vision-language navigation. Moreover, the authors devise a framework named VIENNA to simultaneously learn four navigation tasks within a single unified parsing-and-query scheme. Experiments show that the VIENNA outperforms the baselines among four navigation tasks.
strengths_and_weaknesses: # Strengths
1. The authors are the first to innovatively combine four navigation tasks and propose a dataset that can simultaneously adopt them.
2. The authors propose an agent that can solve the four navigation tasks using one single model without switching among different models and the model size is considerable.
3. The proposed VIENNA agent leverages multisensory input for navigation. Input in different modalities provides supplementary information. This setting is more practical in the real world, where robots use multi-modal information for finishing their tasks.
4. The proposed methods outperform the existing SOTA in every single task by leveraging multisensory input and multi-task learning. Also, the authors have conducted a thoughtful ablation study to evaluate the effectiveness of each component and claim.

# Weaknesses
1. My main concerns are what kinds of knowledge can be reused and how to reuse this knowledge? In Section 4.3, the authors claim that the knowledge re-usage is conducted by \tau. However, in image-goal navigation, \tau_A does not contain any information about what kind of images should be found. So how can \tau_A help to retrieve target image-related audio for navigation? More explanations are needed.
2. Some details are missing. Please explain the meaning and benefits of the vectors \mu_{1:t} in section 4.1. 
3. VIENNA adopts the same reward function for the four navigation tasks. However, the reward function design will be different for different navigation tasks. For example, in object-goal navigation, a positive reward should be given when an object with goal-related semantics is found. It would be an interesting future research direction for considering different kinds of rewards for different tasks.

questions: My main concerns are what kinds of knowledge can be reused and how to reuse this knowledge? Please explain more about this question.
limitations: Yes, the authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a new visual navigation benchmark that combines containing information from different modalities. Then the authors propose a modular model for the proposed benchmark including Episodic Encoder to encode the history, Target Parser to interpret the target, and a planner to predict the navigation action. 
strengths_and_weaknesses: Strengths:

1. The proposed task itself is a timely movement in navigation area. With the advancement in individual navigation task, it is natural to consider tasks with input of more modalities. 

2. Considering the complexity of the task, the proposed modular model is a great baseline. The modular structure is designed via analyzing the task component. The architecture itself also shows some intuition on the problem. For example, the author shows a good demo about how to handle input from different modalities.

3. Both the paper and the appendix are detailed and well-written.

Weakness: 

1. The main contribution is a proposed dataset, but it does not provide the full dataset or a sample in the supplemental material.

2. The authors claim the task includes four navigation tasks. Actually, it is just a new task including the input types from the four tasks.

3. One serious issue in navigation task is that sometimes utilizing more modality leads to a poor performance for some model. Therefore I think it is necessary to ablate different modality input and test the performance. For example, check the paper mentioned in Section 4.1.2 of [1]. There are limited ablation study in section 5.2. 

4. The paper should include more SOTA models on the four navigation tasks as baselines. The proposed model with more inputs should at least outperform these models with less input information.   


Suggested citation:
[1] A survey on visual navigation for artificial agents with deep reinforcement learning
[2] Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
[3] Core Challenges of Social Robot Navigation: A Survey

questions: See Weakness


limitations: Yes, they discussed that in the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a unified model for 4 popular navigation tasks (image-goal navigation, object-goal navigation, audio-goal navigation, vision-language navigation). A Transformer based architecture is utilized to combine these 4 tasks. The target signal (image, audio, etc.) is basically used as the query. The paper also proposes a dataset that supports all 4 tasks. The experiment section provides results for single-task and multi-task settings and provides comparisons with task-specific baselines. 
strengths_and_weaknesses: **Strengths:**
- The paper takes a step towards unified models for Embodied AI. This is an important direction since agents should be able to perform various tasks simultaneously.

- The proposed dataset, which supports multiple types of navigation, will be useful for the community.

**Weaknesses:**
- The weakest part of the paper is its experiment section. The conclusion that are drawn from the experiments are not accurate.

&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; (1) The difference between most of the single-task and multi-task results is within the range of training noise (8.3 vs 8.8 for object goal navigation or 10.6 vs 11.3 for audio goal navigation). The results of these types of training algorithms vary greatly depending on the initialization, sampling and various other factors. It is great that the multi-task model works as well as the single-task models. However, claims such as "VIENNA also beats other task-specific competitors" are not correct. 

&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; (2) Statements such as L345: "VIENNA_ST suffers from relatively large performance" are not accurate. If the training noise is considered the drop is the same for single-task and multi-task settings. 

&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; (3) Numbers in Tables 4b and 4d are within the range of noise. No conclusion can be drawn from these tables. Statements such as "such strategy greatly boosts the performance" are not accurate. 

&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; I will consider raising the score if these statements are corrected in the revision.

- It is not clear how models are chosen for evaluation. Is it the model that achieves the best performance on the validation set, trained for a specific number of epoch, or other criteria? 



questions: - L226: what is "epoch embedding" ?


- How is MDPPO different from DDPPO? Does it just balance the task samples? 


- What are Navigation Error (NE) and Oracle success Rate (OR) ?


- The sequence length for the Transformer is 100. That means there is a huge memory consumption, and the sequence might not fit into a GPU. Is there any specific trick used ?
limitations: Yes, the limitations are discussed adequately.
review_title: null

Meta-review: metareview: This paper introduces a novel indoor navigation dataset that is both continuous and audio+visual.  Within this setting, they include popular tasks and their audio-generalizations (e.g. image-goal nav --> audio-goal nav).  Particularly of note is the leveraging of unification of these tasks during training for a better overall agent.  This is a necessary and important step for the community.

There are several minor concerns regarding exposition and claims which were addressed in responses/updates which will strengthen the final paper. This includes task/model variances and clarifying why the reported variances are smaller than typically seen in related EAI tasks.
confidence: Certain
Decision: Accept