Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a Neural SLAM-based approach for tackling embodied multimodal tasks in ALFRED benchmark. The approach, called Affordance-aware Multimodal Neural SLAM (AMSLAM), utilizes several modalities for exploration, predicts an affordance-aware semantic map, and plans over it at the same time. The approach achieves 40% improvement over prior published work.
main_review: Strengths:
- This submission is well motivated. Training robotic assistants to follow human instructions to complete an interactive task is an important and challenging problem. 

- The proposed method achieves good performance, improving 40% over prior published approaches to achieve 23.48% success rate. 

Weaknesses:
- The writing clarity and readability can be improved significantly. The method seems extremely complicated and is very difficult to understand. There are way too many forward and backward references in Section 4. Even after reading over the methods section twice and going over the details in the Appendix, I am unable to understand many details in the method. For example:
    - In sec 4, what is meaning of module operating at the "subgoal level"? In contrast to what in the E.T. paper? Why is the explored area considered an extra modality?
    - In sec 4.1, how does the agent predict the task described in the language instructions? How does the agent know the subgoals in each task? How does the agent switch between exploration and execution phases? What is the difference between planned and prediction actions?  
    - In Figure 1, it is unclear where and how the semantic map is used. It is unclear where the explored area is coming from (my understanding is that the environment does not provide the explored area). The caption refers to "affordance-aware semantic representation" which is not shown in the figure. What is the meaning of the "actual" task? 
    - In Sec 4.2, what are the delta rule-based functions?
    - In Sec 4.3, how does the agent know the which low-level instruction corresponds to the current navigation and subsequent object interaction subgoals?

- The method seems to use of lot of arbitrary choices, rules and hacks. For example, "we define a single-step explored region as a binary map where a 5×3 rectangle grid (as a hard-coded field of view)", "We manually inject 4 RotateRight after every 2 MoveAhead predicted by our module to acquire 360◦ views of the scenes", "we further inject two LookUp or two LookDown actions alternately after every exploration action", "The agent then goes through 6 horizons {60◦, 45◦, ..., 0◦, −15◦} and obtains the mask prediction (with confidence > 0.8)". All these choices seem to be specific to the Alfred benchmark. I can not imagine a robot operating in the real-world in this manner. My worry is that the authors have exploited unrealistic approximations in simulation environments (such as discrete action and state space, no noise in motion and pose sensors and use of high-level interaction actions), and over-optimized the design choices specific to this benchmark, which are likely to not result in better performance in more realistic settings.

- The technical contributions of this paper are unclear:
   - The authors claim in the introduction "we propose the first multimodal exploration module that takes language instruction as guidance and keeps track of visited regions." I believe the Active Neural SLAM model, which AMSLAM is based on, also keeps track of visited regions to learn exploration. Based on authors' definition, even Active Neural SLAM is multimodal as it used pose, explored area and visual observations as input. Is the addition of language instructions as input the only change to the model?
    - The authors claim they introduce "Affordance-aware semantic representation that estimates object positions, heights, and relative spatial relationships". I believe the semantic representations in Chaplot et al. 2020b and Blukis et al. 2021 also estimate object positions, heights and relative spatial relationships. It is unclear what makes the proposed representation "affordance-aware". 
summary_of_the_review: Overall, the motivation is good and the method achieves a good performance on a public ALFRED benchmark. However, the authors seem to have made many design choices specific to this benchmark which are likely to not result in better results on realistic tasks. The technical contributions of this paper are unclear and the writing clarity needs to be improved significantly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework that improves the state-of-the-art performance of the ALFRED benchmark (accomplishing navigation and interaction tasks given language instructions in AI2THOR environments) by 40% relatively. Two claimed main technical contributions are: 1) leveraging multimodal inputs, specifically the newly incorporated language instructions and visited area maps, during exploration, 2) proposing an affordance-aware semantic representation that marks the object locations, heights, and possible agent interaction spots. Results that are 40% relatively better than the state-of-the-art are observed and several ablation studies over key network modules are also presented.
main_review: strengths:
- results are much better than state-of-the-art and the improved margins are significant.
- the proposed technical module of learning an affordance-aware semantic representation is valid, reasonable, and somewhat novel.
- the system designs are really solid in mixing up many well-designed heuristics and learned modules into a big system that produces very good results.

weaknesses:
- my biggest concern is that the paper presentation is not selling the framework well as an easily-readable research paper, for example, a) there are no qualitative figures showing the performance of the work, only tables with numbers are presented. It is quite important to show qualitative results and your qualitative comparisons to baselines, additionally with some analysis figures clearly proving the effectiveness of each proposed module, for readers to be better convinced where your superior performance comes from. b) the writing for Sec. 4 can be improved. The current writing is quite messy mixing up the two contributions together. Also, many important details (e.g. the architecture to learn the affordance maps and the aggregation steps) are put in supplementary. c) for the experiments, the authors spent many more words on Sec 5.1 and 5.2 than Sec 5.3 (the main results). The main results subsection is very short and does not contain analysis over the tables or show qualitative figures to analyze the results.
- another of my major concerns is whether or not using the language instructions during the exploration stage is a reasonable and realistic setting. My opinion is that we should not assume we know the task (goal) and the instructions for subgoals beforehand. and during the exploration. Otherwise, we need different exploration steps given different tasks. Or, at least, you need to report your exploration steps as part of the task execution steps, since you need different exploration steps given any new test task. Please clarify if I'm wrong with this. But, the confusion regarding this point prevents me from recognizing the claimed technical contribution on leveraging multimodal inputs during exploration.
- though there are some novel designs for the affordance learning part, there are previous works (Qi et al, 2019, and Nagarajan & Grauman, 2020) that have proposed the essential ideas, and I don't see too much difference other than adding some heuristics-derived waypoint definitions. Please clarify if I missed or misunderstood the main differences from previous works.
- though I recognize that the presented system achieves amazing results and significantly refreshes the state-of-the-art, it's unclear how much performance gains come from combining many state-of-the-art architectures, say transformers or bert, and how much comes from the claimed two novel technical design points. Can you compare to baseline methods with the same architectures or report the parameter amounts for different models, to give us a sense of this?
- one minor issue is to use \citep instead of \cite in latex for a more valid citation format of papers
summary_of_the_review: Overall, I have a mixed feeling. On one hand, the performance is really good and significant. On the other hand, the two claimed technical contributions are not stated clearly about their significance of differences than the previous methods or the validity of the design. Also, this paper is more like a system report other than a well-written research paper with clearly presented and analyzed novel techniques. Adding qualitative figures and analysis will definitely help on this front.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for solving the ALFRED task (following language instructions to perform a set of household tasks). The model has two main components: (1) Affordance-aware Semantic Representation (2) Multi-modal exploration. The former component estimates the location of the objects in the scene. The latter provides an exploration strategy by using instructions, images, previous actions and previously explored areas as input. 
main_review: **Strengths**

- The method achieves the state-of-the-art performance on the ALFRED benchmark, which is quite challenging.

- The set of ablation experiments are informative.


**Weaknesses**

- The paper is heavily engineered for the ALFRED benchmark. I am not sure if any of these modules can be used for generic embodied tasks.

- The paper makes a big deal about the "affordance-aware" representation in the introduction and other places in the text, but it is actually a simple heuristic that the agent backs up a few steps when it is next to an object. According to the text "the agent will back up 3 steps if the target object is a fridge, 2 steps if it is a safe, a cabinet or a drawer, and 1 step for everything else". These types of heuristics are not generalizable or scalable. The affordance-aware representation should be learned instead.

- There is a strong assumption that "we know the changes (∆x, ∆y, ∆r) of the agent after each action". These types of assumptions are valid only in simulation. Several previous works have made similar assumptions. However, those approaches are suitable only for simulation. In reality, the estimates for agent pose are quite noisy. I would like to see how robust the proposed approach is against noise in the agent pose.

- It is not clear if the exploration steps and the steps used for capturing panoramic views are included in the evaluations.
summary_of_the_review: The paper proposes an approach which is heavily engineered for the ALFRED benchmark and I do not see a major novelty that generalizes to other embodied tasks. The method is mainly a mixture of heuristics and a set of pre-trained components that are glued together for the ALFRED tasks. However, it achieves good performance on the ALFRED benchmark, hence, my borderline rating.
review_title: null

Meta-review: This paper presents a SLAM based approach for the ALFRED benchmark. The presented method, Affordance aware Multimodal Neural SLAM has two key advantages over past works: It uses a multimodal exploration strategy and it predicts an affordance aware semantic map. It also obtains a very large performance improvement over the ALFRED benchmark. The reviewers for this paper were quite impressed by the large improvements obtained by this technique. However, there were two major concerns across the reviews: (1) Are the design choices made in this paper heavily engineered towards ALFRED ? (2) Does the work make too many assumptions about the setting (unrealistic assumptions that may not really hold in more realistic environments or the real world) ? The authors have provided a detailed response and answered many questions posed to them, but the reviewers continue to have concerns about the generalizability of the proposed method. Another point of concern pointed out by a reviewer is whether it is reasonable in a realistic setting to perform exploration with a knowledge of the downstream task. This point has not really been answered satisfactorily by the authors. My takeaway is that the method presented by the authors clearly works on ALFRED. But it contains several design choices that are largely ALFRED specific and in some cases unrealistic. This provides fewer benefits to readers looking for more general insights that can be valuable across a suite of tasks. As a result of this, and in spite of the large gains, I recommend rejecting this paper.
Decision: Reject