Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new method for the self-supervised localization of sounding objects in image. The paper focus on two important limitations of current methodologies 1) they do not handle well sample with no visible or audible sound sources and 2) they are prone to overfitting due to using early stopping criteria on test sets. To address these issues, the main methodological contributions of the paper are - - the introduction of slow-moving momentum audio and visual encoders
- the use of extreme dropout on visual encoders
- the introduction of a loss function that jointly enforces sounding region selection and cross-modal instance discrimination, using different projections for each tasks.

Experiments reveal that the propose method establishes a new state-of-the-art on standard datasets as well as on newly designed datasets that include negative samples. Careful ablation studies on the components of the proposed approach and on the metric parameters are also conducted.
strengths_and_weaknesses: ## Strengths
- The paper is clear and pleasing to read. The formalization of the method and chosen notations are sound and easy to follow.
- Beyond establishing a new state of the art, it is also a welcome contribution to research reproducibility in this field.
- Experiments are convincing and are rigorously designed to avoid biases, in particular in the choice of metric threshold.

## Weaknesses
- **There is unfortunately a potential fatal mistake in the paper: the precision results of Table 3 seem to have been copy-pasted from Table 2. It is not clear whether the discussions and conclusions drawn by the authors throughout the paper will still hold after fixing this. This must be discussed during the rebuttal phase. Hence, I give the paper a grade of 5, which would become an 8 if this issue is satisfyingly fixed.**
- There are several points in the paper that deserve to be clarified, see next section.
questions: - In equation (1) and equation (4), the loss function depends on the index i. It should be denoted \mathcal{L}_i to improve clarity. Is it the case that over a mini-batch containing several audio-visual pairs (a_i,v_i), i=1...P, the minimized loss is actually the sum of \mathcal{L}_i for i=1...P ? Moreover, are the sum over j in the loss actually taken over $j\ne i$ ? Please make this precise.

- In eq. (4): typo: I guess the denominator of the second term should contain P_ji instead of P_ij, otherwise it doesn't make sense (the two terms are identical).

- L182: while this is probably obvious to the authors, it is not entirely clear to me what the categories "small, medium, large and huge" refer to. The total image size? The sounding object size? How is this defined?

- L187: the bracketed term "(usually set at γ)" is confusing and should probably be removed.

- Section 4.1: Maybe I missed it but I do not see how the (x,y) locations are selected inside each image to compute local visual features. At random? On a regular grid? And from how many pixel are the visual features are computed?

limitations: The authors have correctly addressed the limitations of their work, mostly the poor performances on small objects.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes new metrics and training recipes in order to avoid problematic behavior of sound localization methods which tend to overfit to predicting correctly masks for sounding objects which appear on-screen but completely neglect input images with not on-screen sounding objects. The authors also propose a momentum audio-visual approach which shows state-of-the-art results on widely used benchmark datasets.

strengths_and_weaknesses: ***Pros:***
- The approach is simple and well described so researchers can reproduce the results.
- The paper discusses a really important problem of rejecting regions with not on-screen sounding objects which is beneficial for the community. The paper also extends benchmark datasets with well detailed evaluation dataset extensions to include the capability of models to reject off-screen sounds.
- The results on widely used benchmark datasets are robust and the paper is clearly written.

***Cons:***
- The paper addresses the problem of the models in the literature to overfitting, whereas the main problem is that one could also use hard-negatives and negative samples during training directly (e.g. using a loss function that tries to make the predicted audio-visual mask zero for frames which are not aligned with the input audio). A similar problem (videos with no on-screen sounds) has also been addressed in open-domain audio-visual sound source separation [A] using a multi-instance loss and measuring the off-screen power suppression for sounds which do not appear on-screen. It is not so straightforward why the authors did not try to optimize a straightforward loss on the mask of hard negative examples.
- Picking the confidence threshold also depends on the dataset and thus, although there is no strict dependence on supervised data to train a model with the proposed methodology, this approach also needs supervised data in practice to fine-tune the confidence threshold in order to shift the operating point of the model towards a desired one.
- The ReNet weights which are used come from supervised training on image classification tasks (Line 222: “we initialize the visual encoder with ImageNet [39] pre-trained weights”), please correct me if I am wrong. In this case, the authors either need to remove the terminology about “unsupervised” and replace it with partially/semi/less - supervised or they can show results where their method does not need supervised pre-training to work. 

[A] Tzinis E, Wisdom S, Jansen A, Hershey S, Remez T, Ellis D, Hershey JR. Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds. In International Conference on Learning Representations 2021.

I am more than happy to increase my score if the authors address my concerns. To be honest, the only reason that in the current state I vote in favor of NOT accepting this great paper is the misuse of the term “unsupervised” when in all the experiments the backbone image encoder was pre-trained on the whole ImageNet (and this is the reason that I used this rating on the soundness axis).
questions: - How can the proposed method be extended towards removing the dependencies on object detection models and object saliency maps?  
- The authors can remove the gray background in Figures 2 and 3.
limitations: In my opinion, the authors aptly address several limitations of their work in the main text (see Section 4.5).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper describes two contributions to audio-visual source localization: 1) an evaluation protocol designed to explicitly include negative example cases (where no sound source is visible in the image), and 2) model components designed to address shortcomings in prior work.
The authors compare the proposed model extensions to prior work, using both the proposed evaluation (and extended datasets) and previous evaluation schemes.
Ablation studies quantify the relative contributions of each modeling component across the new metrics.
strengths_and_weaknesses: Strengths:

The explicit inclusion of negative examples for this task makes a great deal of intuitive sense, and addresses a real shortcoming of previous work in this area.

The evaluation appears to be carefully done, and the proposed modeling components do seem provide a substantive improvement for localization.



Weaknesses:

The proposed model components (section 3.2) could be explained better.
In most cases, I do not see a direct connection between a specific problem with prior models and how these ideas should be expected to help.

questions: I found the description of momentum confusing and lacking context.  There are numerous "dimensions" in which momentum could reasonably be applied here—time, visual extent, training iteration—and a reasonable case could be made for any of them.  (E.g., one should expect smooth variation in the visual extent if sound sources are compact visual objects, and sound activation should be piecewise constant in time for most types.)  I *think* you mean momentum in the training iteration sense, but I'm not 100% certain and could not find any definitive statement to this effect in the text.  Could you please clarify, and explicit about what you're doing?

Similarly, I don't think I fully understood the motivation for SLID.  It seems like the "xid" branch is incentivized to match every visual region to audio from some example in the batch (by virtue of maximizing a softmax, eq 3), but why is this reasonable?  Shouldn't we expect most images to have regions that correspond to no audio at all?

What are the definitions for "small, medium, large, huge" in line 182?

Section 4.1 omits the audio sampling rate, without which the numbers describing the audio encoder are meaningless.  Please clarify.

What is the criterion for determining "best" (bold) in tables 2 and 3?  Of course the "ours" method is bold, but in both tables, there are prior examples (EZ-VSL+OGL) that achieve higher/comparable scores (eg 37.86 vs 37.79).  The authors do state that the experiment is too expensive to support error bars, but some statement of a meaningful difference should be provided.

Several answers to the author checklist are incorrect or left as "N/A".  Specifically:
- 1.c, broken reference (no such text exists)
- 3.d, section 4.1 does not include the specified details
- 4.a, does not appear to cite all tools used (i.e. software), only prior papers describing models
- 4.c, 4.e: answer should be "no", not "N/A" (you are using existing assets, and did not address these points)

limitations: Some technical limitations are discussed, but no societal implications are mentioned.  The problem being studied here has obvious applications in surveillance that could be mentioned.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new Momentum-based approach for Visual Sound Localization, termed as MoSVL. It focuses on two key audio-visual sound source localization problems: 1) previous studies are not truly un-supervised and have overfitting problems easily when scaled up to large datasets. 2) the authors claim that all baseline methods fail to consider the silence in the sound localization problem, hence a new mechanism with evaluation protocols is proposed. 
strengths_and_weaknesses: Strengths:

- [minor +] The audio-visual learning is on the surge and is of great importance. Hence the focused problem has good practical impacts.

- [minor +] The paper is generally easy to follow.

Weaknesses:

- [major -] My biggest concern is that the paper **directly ignores some very very important and relevant works**, DSOL [1] and IEr [2]. They are neither compared nor even discussed/cited in the paper. In particular, DSOL [1] targets to solve the discriminative sounding object localization in multi-sound scenarios **as well as coping with the silent objects**. They suppress those silent but visible objects by multiplying the audio category probability distributions to the visual object distributions, hence those silent objects will not take effect. Besides, DSOL **also propose a manner to effectively evaluate the sound localization performance**. They create a 2x2 grid, with two sounding objects and two silent objects to evaluate cIoU and NSA (Non-Sound Areas). If the model mistakenly localize the sounds to the silent regions, both the cIoU and NSA will perform poorly. In IEr [2], the authors propose to not only deal with the in-the-scene silent objects, but also suppress the off-screen background noise by cross-modal matching. All these two methods are proved to successfully solve the silence problem in the sound localization. However, the authors claim that this is an unsolved problem and it constitutes a very important observation and contribution of this paper.

- [major -] Lack of Novelty. To solve the overfitting problem, the authors propose to add more dropout to the visual features and perform the EMA update on the encoders. However, these are all very basic tricks and couldn't independently serve as a novelty or contribution to the community. Besides, the authors fail to very carefully analyze the effect of dropout or ways to prevent overfitting. Why the dropout works? Why other tricks that help to prevent overfitting won't work, like data augmentation, network parameter normalization, etc.

- [major -] Lack of Demo Video. The lack of demo video weakens the qualitative results of this paper. I could not straight judge the performance when silent objects exist. 

- [minor -] No code. As promised by the authors in the checklist, they will include code or data in the supplementary material. Although the authors say that the dataset is too large to submit. The code is not included, which makes doubt the answer [YES] in the checklist and the re-producibility of this paper.

[1] Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching, Hu et al., NeurIPS 2020.

[2] Visual Sound Localization in-the-Wild by Cross-Modal Interference Erasing, Liu et al., AAAI 2022.

Overall, the ignorance of very relevant works makes the claim, contribution and novelty of the paper poor. The focused problem has already been solved, and the corresponding evaluation protocols have already been proposed, while they are not even discussed in the paper. This makes the main body of the paper meaningless. The left part is the method to solve the overfitting problem, which are quite trivial tricks to me. The authors fail to make their best efforts to delve into the phenomenon and analyze why such methods could alleviate the overfitting problem. 
questions: My concerns are elaborate in the above weaknesses field. The lack of comparison and discussions with important related works makes the majority of the paper meaningless. 
limitations: They are discussed in the supplementary material, but it shows up as Section ?? in the checklist. I recommend the authors to pay more attention to the checklist, since this is a very important and useful part in NeurIPS submission.
review_title: null

Meta-review: metareview: The authors seem to have addressed most if not all of the reviewers recommendations, leading to a much improved paper compared to the initial manuscript. The updated scores from the reviewers reflect the major improvements and therefore I recommend this paper be accepted in its updated form.
confidence: Certain
Decision: Accept