Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In order to localize the object emitting a specified sound, i.e., sounding object localization, this paper proposes a Two-stream Universal Referring localization Network (TURN) concerning the localization stream and alignment stream. The TURN model correlates the referring image localization via the localization stream and text-audio semantic space via the alignment stream. Such a mechanism transfers the audio query through the learned codebooks, retrieved from the text-audio semantic space, to indirectly trigger the referring object detector. In addition, the authors provide an adaptive sampling strategy for helping training. The experiments show the proposed model on par with the existing methods.
strengths_and_weaknesses: [Strengths] 
+ The idea of utilizing the zero-resource perspective for tackling sounding object localization is interesting.
+ The manuscript is well organized and has adequate references.

[Weaknesses] 
- The technical novelty is limited since the TURN model is mainly composed of DETR [10], VQ-VAE [55], and VQ-WAV2VEC [7]. The TURN model shows an application employing these methods [7,10,55] with an additional adaptive sampling strategy for helping the model training.
- The proposed method makes the task of sounding object localization from the audio-image matching problem to the audio-text-image matching one. The proposed method seemingly complicated the original task of acquiring the box-level prediction. The proposed method also implies that the extra data are needed yet ignoring the available training data from the field of sounding object localization. However, the performance gain is not significant.
- While evaluating the model, does the TURN model evaluated with the box-level prediction? It seems that the box-level prediction is not the same as the heat-map prediction, as shown in [A].
- Figure 2 needs more descriptions and notations to gain a better understanding.
- What is the so-called `patch’ in line 168 used in a DETR backbone [10]?
- The loss terms mentioned in section 3.5 should be consistent with the notations in figure 2.
- The reference is not sufficient. For example, some related methods are shown as follows.
[A] Arda Senocak, Hyeonggon Ryu, Junsik Kim, In So Kweon: Less Can Be More: Sound Source Localization With a Classification Model. WACV 2022: 577-586.

questions: The major concern is the motivation to treat the sounding object localization task as a combination of image grounding and audio retrieval tasks. Such treatment seems complicated to the target task and the resulting model. In particular, previous methods [48, A] showed that the heatmaps retrieved from the sounding object localization could generate the corresponding boxes.
limitations: No limitations and potential negative societal impact included.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: * The paper proposes a method for training a sounding object localization model when there is no paired data available. It achieves this by transferring knowledge from paired data from overlapping modalities i.e. image-text and text-audio datasets.
* The idea is that by aligning the text representations from the different paired datasets into the same space, we can learn to match the accompanying audio representation with the appropriate visual embedding.
* The text and audio representations are projected into a common space using codebooks that are learned through the self-supervised training method of the vq-wav2vec [7] idea. The image representation is combined with the text representation from the codebooks and passed through a patch encoder to perform bounding box regression.
* The experimental results on the VGSS dataset shows that the proposed method can perform better than other methods that use supervised data for training.

strengths_and_weaknesses: ### Strengths
* The proposed method builds on ideas developed for individual modalities and brings them together to learn multimodal models in low resource settings.
* The paper addresses the problem of sample weighting since there can be domain gaps in the source and intermediate domains. It uses an objective function that weighs the text embeddings such that they are hard to distinguish by a domain discriminator.
* The results on the VGSS dataset are significantly better than the methods that have been trained in a supervised manner.
* The paper also addresses the question of how to pick the source and intermediate domains via an ablation study.

### Weaknesses
* The IOU result on MUSIC-Solo is oddly very low compared to the other methods. The paper doesn’t provide any explanation or investigation for this result.
* The paper doesn’t include an experiment that shows how to combine the annotated data along with the un-annotated data. It would be interesting to see how much additional benefit can be obtained from using un-annotated data.

questions: * Can you provide results from an experiment that combined annotated and un-annotated data?
limitations: The paper can include a discussion the biases of different data sources that can propagate to the results, especially when performing zero-shot recognition.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies sounding object localization from a zero-resource perspective. To solve this, the paper proposes a Two-stream Universal Referring localization Network(TURN), which includes a localization stream and an alignment stream. In order to solve the domain gap, an adaptive sampling strategy is used in this paper. There are many terms of loss function in this paper, the authors have carried out sufficient ablation experiments to prove the effectiveness of each part. The experiments in this paper prove the effectiveness of the model.
strengths_and_weaknesses: ### Strengths
+ This paper proposes a Two-stream Universal Referring localization Network(TURN) to combine two domains(Image Grounding and Audio Retrieval) to solve zero-resource sounding object localization.

+ An interesting point in this paper is that an additional classifier is trained to measure the domain gap of the sample, based on which the data is sampled for training. 

+ This paper also conducted sufficient ablation experiments on each loss in the model, proving the effectiveness of each part. In addition, the authors analyze the failure cases in the appendix to illustrate the limitations of the method.

### Weakness

- In Line 37-Line 39, the paper mentioned that most of the current models generate a heatmap rather than a bounding box, which bring much inconvenience. However, the inconvenience is not described in detail in paper.

- Paper [1,2] also proposes a method to give the bounding box, which is not compared with the method in this paper.

- The paper does not explain how the texts of two domains are aligned during training. Although an adaptive sampling strategy is proposed to solve domain gap, in the training process, the texts from two domains may be irrelevant in the same batch.(For example, in the same training batch, the image grounding text description may be "a car was going on the road" but the text description in audio retrieval may be "a dog is barking on the road")

- The method in this paper can only give a single bounding box. For the mixed sound scenes, the method can not deal with it correctly.


[1]  D. Hu, et al., “Class-aware sounding objects localization via audiovisual correspondence,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[2] T. Afouras, et al. “Self-supervised object detection from audio-visual correspondence,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: 1. In line 167，is each element in set $V$ $\hat{v}_i$ or $v_i$? From the following, should it be $v_i$?
2. Why does equation(5) use a cross-modality way in the intermediate domain? In the loss function, the contrastive learning loss has been used to align the audio modal and the text modal. What is the effect if the cross-modality way is not used here?
3. The result of DML and DSOL is heatmap. Is it fair to compare the IOU calculated by heatmap and the bounding box of ground truth with the one calculated by the bounding box and the bounding box of ground truth?

The authors are expected to solve the problems in the weakness and questions, then I can accordingly modify my ratings.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to locate sounding objects in an audiovisual scene without resorting to specialized training data for sound localization. It designs a new framework Two-stream Universal Referring localization Network to achieve multi-modal knowledge transfer among visual, text and audio modalities. Specifically, the localization stream locates objects from related texts using DETR structure, the alignment stream establishes a universal feature space shared by audio and text. In this way, it manages to localize the objects related to the specific sounds without extra annotations. The experiments show the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1. This paper is well motivated, and the idea of transferring knowledge from three modalities is interesting. It solves the sounding object localization task from a new perspective, i.e., directly regress bounding box instead of generate heatmaps, and does not require extra human annotations.
2. The paper is well written and easy to follow, and most technical details are well illustrated. Though some are borrowd from previous works, but generally it is a suitable application in this task and effective in multi-modal konwledge transfer.
3. The adaptive sampling strategy makes sense and the learning procedure of the sampling weight is interesting. It fully utilizes domain information and selects reasonable training samples for text audio knowledge transfer.
4. The results on single-source sounding object localization are promising. And the analysis on the failure cases is valuable. It demonstrates that this new formulation of sound localization instead of heatmaps is worth exploring and also figures out how to improve it.

Weaknesses:
1. The biggest concern is that this paper only performs experiments on single-source data, lacks the results on multi-source scenarios. On signle-source data, the sound localization is comparatively easy and the performance is mostly decided by the localization stream since the alignment is trivial. But on multi-source cases, it can validate whether the alignment stream establishes satisfactory semantic text audio alignment. And from my perspective, the image grounding task can help discriminate object semantics, and the text audio alignment also gather similar semantics, it should be able to deal with multi-source scenes with minor modifications. So I strongly suggest authors to try to expand the framework to more complex scenes for a test.
2. The visualizations in Sec.4.4 show that multi-box cases are diffcult to process. Deeper analysis on what causes this problem is desired. For example, when providing the text embedding as query how will the localization result be. Better to figure out whether it is a problem from localization stream or alignment stream.
questions: My questions are stated in the weakness part, mostly on the experiments.
limitations: Yes, the authors solve the limitations of lacking benchmarks. And there is no obvious negative social impact.
review_title: null

Meta-review: metareview: This paper presents an effective transfer-based two-stream architecture for zero-resource sounding object localization, which is an interesting paper that will benefit research in zero-resource tasks in multimodal settings. The experiment results are pretty impressive, both quantitatively and qualitatively. The rebuttal successfully addressed some of the major concerns and, in the end, there is a general consensus about accepting the paper.
confidence: Certain
Decision: Accept