Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new kind of audio-visual navigation task which requires agent to generalize to novel audio sources in 3D indoor environments from the Matterport dataset. The proposed approach leverages multiple sources of information of which objects the source is likely coming from, where they might be found, and where the source might be present in the house to tackle the task. In experiments, they show that the proposed approach significantly outperforms existing methods on unseen houses, unheard sounds. 

strength_and_weaknesses: Strengths: 
- The task definition is interesting. Generalizing to unheard sounds is an interesting split to look at, and I am glad that this is being looked at. 

- The proposed approach is somewhat intuitive and seems to work well, specially on unheard sounds. 

Weaknesses: 
- It's not clear if the pre-training happened on all sources of sounds (seen and unseen). The formulation seems to suggest that the audio classification model was trained on all sources (seen and unseen). I think this is a bit problematic, since it makes the assumption that at test time, no novel sound source will exist. 

- If the pre-training involved both seen and unseen sources, then comparison with SAVI is unfair because SAVI presumably didn't have access to these sounds. 

- It'd be nice to show results for classification models apart from navigation results. These will help the readers understand how classification accuracy correlates with navigation accuracy. 

- I also think it'd be nice to replace certain parts of the model with "oracle" modules. For instance, if the audio classification model, or the location prediction model was perfect, how will the performance get affected. Doing these sort of analyses helps build a much stronger intuition for the task.
clarity,_quality,_novelty_and_reproducibility: I think the paper can be clearer. There are certain mistakes -- for example, in figure 2 (b), they have GCN, but figure 2(a) have GENS. The approach was also quite hard to follow. More importantly, what do the authors mean by unheard sounds is not clear until the very end. 
summary_of_the_review: The task is interesting, and the approach is intuitive. But, I think the paper is also missing certain analyses (classification model performance, ablations by replacing with oracle modules) that will further strengthen the paper. 

Update after rebuttal: I thank the authors for providing responses to all my concerns and the experimental results for Oracle modules. I am tending towards acceptance for the paper. I am hesitating to give a strong accept is because I would have liked to see experiments where *not all sounds* are heard during the pre-training stage. Since they are building a new benchmark, it'd be nice to push the limits a little bit more and cover the case where a truly novel sound source is present during test stage. For that reason, I am only updating the score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the Semantic Audio-Visual Navigation (SAVi) and proposes the use of Knowledge-driven scene priors for encoding the object-region relations, spatial knowledge, and background knowledge to address the embodied navigation problem. The method named K-SAVEN incorporates graph encoder networks to model audio and visual inputs, as well as applies scene memory transformer to capture long-term visual-audio dependencies. Numbers of pre-training tasks are defined, and a multimodal dataset is curated to support the learning. Moreover, a new SAVi task is defined to evaluate the agent performance on novel objects and sounds. 
strength_and_weaknesses: Strengths:

1.	The paper proposes novel and effective methods and defined a valuable task to address SAVi, which will be highly beneficial to future research in relevant directions. The use of knowledge graph to connect sounds, objects and regions could be an important step towards generalizable audio-visual navigation in novel environments. Results show significant improvement over previous approaches. Moreover, the new task offers a setting for evaluating the agent’s generalization ability on novel objects (and sounds).

2.	It is great to see the comprehensive details presented in the method section and the Appendix, e.g., the knowledge graph construction and representation, including the resultant graphs in Table 3 and Table 4.

3.	Overall, the paper is clearly presented.

Weaknesses:

1.	It is wonderful to see the new experiments added to this paper, but they can go deeper. Since knowledge graph is the key novelty, I expect more experiments on using the knowledge and the graph construction (e.g., the source and type of knowledge, the connection between objects and regions, etc.). Although this work is built based on SAVi and several methods are inherited, it will be valuable to see the influence of the pre-training and classification tasks in the proposed architecture. Moreover, it would greatly strengthen the paper if the method is evaluated on more dataset such as Replica and compare with the baselines.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, figures and tables are clearly presented. The proposed methods such as knowledge graph, local prediction and direct-to-reverberant ratio estimation are novel in addressing the SAVi problem. The new task supports a more comprehensive and meaningful evaluation of the agent’s performance, which should be beneficial to future research. The authors have stated to release the code and data upon acceptance, which I believe the reproducibility of this work is promising.
summary_of_the_review: This is the second time that I review this paper. Comparing to its previous ICLR and CVPR versions, it is great to see manuscript has been significantly improved and most of my previous concerns are nicely addressed, especially the new experiments in Table 1 (top), Table 2, and Table 7 are added to justify the method and designs, as well as the insightful analysis section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to propose two architectural contributions and a new task for semantic audio-visual embodied navigation. 

The paper first employs a knowledge graph for encoding object-object, object-region, and region-region relations. Then, the paper introduces multiple auxiliary models to facilitate audio-visual embodied navigation. 
Furthermore, the paper proposes a new task of semantic audio-visual embodied navigation. In the new task, the agent is designed to steer toward an object without unheard sound.

The paper also demonstrates performance improvements in all permutation of settings, ie., seen/unseen houses with heard/unheard sounds.

strength_and_weaknesses: Pros: The paper demonstrates the overall architecture clearly in Figure 2.

Cons: 
- It is hard for readers to catch up on the key points of the paper. Since almost every subsection only contains one long paragraph, readers might fail to understand the meaning.
- The paper claims to introduce pre-defined semantic information to encode object-object relationships. However, in ICLR 2019, [1] proposed a scene prior graph learned from Visual Genome dataset. It would be helpful if the author could provide a comparison between the proposed semantic information with the semantic knowledge learned in ScenePriors [1].
- The second contribution of the paper is to establish object-object, object-region, and region-region relationships via a knowledge graph. However, in ICLR 2021, [2] proposed a transformer-based architecture to encode similar relationships in visual navigation. The author first misses this reference and second lacks the difference with VTNet [2] in building the aforementioned relationships. 
- The paper does not provide convincing analysis for some ablation experiments. For example, in both SH/HS and UH/HS, introducing $GEN^b$ alongside $GEN^v$ will lead to a performance drop compared with only adopting $GEN^v$. Could the author provide some extra explanation for these results?
- Furthermore, it is hard for me to understand why the performance of GEN^s is lower than that of $GEN^b$ in both SH/US and UH/US settings. Could the author please provide more experiments for readers to understand these experiment results?  



[1] Yang, W., Wang, X., Farhadi, A., Gupta, A., & Mottaghi, R. (2018). Visual semantic navigation using scene priors. arXiv preprint arXiv:1810.06543.

[2] Du, H., Yu, X., & Zheng, L. (2020, September). VTNet: Visual Transformer Network for Object Goal Navigation. In International Conference on Learning Representations.


clarity,_quality,_novelty_and_reproducibility: Clarity & Quality:
- The paper is a little bit hard to understand and needs more effort in organizing the structure. 
- More analysis for ablation studies are necessary.


Novelty:
- The paper does not provide clear and detailed explanations for the novelty of some contributions.
- The new audio-visual navigation task is novel, while the new task does not seem to match with other contributions. 

summary_of_the_review: Overall, the paper raises a novel audio-visual navigation task with unheard sounds. However, the other three contributions need more explanations to justify their novelty.

I lean toward rejecting the current version. However, if the author persuades me that all these three contributions are meaningful, I would like to modify my recommendation to be above the acceptance threshold.

Besides the review for this paper, I believe that this kind of open-set setting is interesting and will definitely benefit our community. However, the goal of raising a new task is not because it is a new task. **In my opinion, the goal should be to provide more insights or chances for the community members.** To achieve that, I recommend the author to think questions about some specific problems in open-set tasks and designing methods following corresponding motivations. At final, I hope to see the edited paper in the future.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new task in audio-visual navigation, where the goal is to navigate to unheard sounds. The developed model is based on a knowledge graph that encodes the relationship between objects and regions. 
The main strengths of the paper are introducing a novel task and the extensive set of experiments. There are two major weaknesses: (1) The sounds are not unheard. Those objects are used in the pre-training stage. (2) The methods introduced share a lot of similarities with prior work.

justification_for_why_not_higher_score: The issues raised during the review and the virtual meeting are major issues and outweigh the positive points of the paper.
Decision: Reject