Towards Effective Multi-Modal Interchanges in Zero-Resource Sounding Object Localization
Authors: Yang Zhao
Chen Zhang
Haifeng Huang
Haoyuan Li
Zhou Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: sounding object localization
knowledge transfer
multi-modal
zero-resource learning

Abstract: Aiming to locate the object that emits a specified sound in complex scenes, the task of sounding object localization bridges two perception-oriented modalities of vision and acoustics, and brings enormous research value to the comprehensive perceptual understanding of machine intelligence. Although there are massive training data collected in this field, few of them contain accurate bounding box annotations, hindering the learning process and further application of proposed models. In order to address this problem, we try to explore an effective multi-modal knowledge transfer strategy to obtain precise knowledge from other similar tasks and transfer it through well-aligned multi-modal data to deal with this task in a zero-resource manner. Concretely, we design and propose a novel \textit{Two-stream Universal Referring localization Network} (TURN), which is composed of a localization stream and an alignment stream to carry out different functions. The former is utilized to extract the knowledge related to referring object localization from the image grounding task, while the latter is devised to learn a universal semantic space shared between texts and audios. Moreover, we further develop an adaptive sampling strategy to automatically identify the overlap between different data domains, thus boosting the performance and stability of our model. The extensive experiments on various publicly-available benchmarks demonstrate that TURN can achieve competitive performance compared with the state-of-the-art approaches without using any data in this field, which verifies the feasibility of our proposed mechanisms and strategies.  

Towards Effective Multi-Modal Interchanges in Zero-Resource Sounding Object Localization
Yang Zhao", Chen Zhang", Haifeng Huang", Haoyuan Li, and Zhou Zhao
Zhejiang University
Glibaba-Zhejianl University Joint Research Institute of Frontier Technologies {awalk, zc99, huanghaifeng, lihaoyuan, zhaozhou}@zzj edu cn
Abstract
Aiming to locate the object that emits a specified sound in complex scenes, the task of sounding object localization bridges two penception-oriented modalities of vision and acoustics, and brings enormous research value to the comprehensive per- ceptual understanding of machine intelligence. Although there massive training data collected in this field, few of them contain accurate bounding box annotations, hindering the learning process and further application of proposed models. In order to address this problem, we try to explore an effective multi-modal knowledge transfer strategy obtain precise knowledge from other similar tasks and transfer it through well- well-aligned multi-modal data to deal with this task in tero-resoure, man- ner. Concretely, we design and propose a novel Two-stream Universal Referring localization Network (TURN), which is composed of a localization stream and an alignment stream to carry out different functions. The former utilized to extract the knowledge related to referring object localization from the image grounding task, while the latter is devised learn universal semantic space shared between texts and audios. Moreover, we further develop an adaptive sampling strategy automatically identify the overlap between different data domains, thus boosting the performance and stability of our model. The extensive experiments on various publicly-availe benchmarks demonstrate that TURN can achieve competitive performance compared with the state-of-the-a approaches without using any data in this field, which verifies the feasibility of our proposed mechanisms and strategies. The code is available at htttps//ggttbbbcoo///ww///Z////
Introduction
When interacting with the external environment, humans can gather and integrate much informa- tion through different types of perception (such as vision, hearing, smell, and taste), thus making corresponding judgments and decisions based on this. To effectively mimic this ability to bridge knowledge from various modalities, researchers have been trying to explore the potential of machine intelligence from different aspects. Among them, crucial part of audio-visual learning, the task of sounding object localization has gained an increasing amount of attention.
Given a specific sound, the goal of sounding object localization to ascertain the object that emits it in a complex scene. To tackle this problem, lot of efforts are made to model the co-occurrence patterns in the visual and acoustic data and accordingly identify the possible target region. And for the research and development in this field, there have been numerous aligned data pairs collected into several publicly-availan benchmarks. Nevertheless, few of them contain available box annotations, which forces the models to acquire knowledge weakly supervised manner and increases the
Equal Contribution. Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

difficulty in the training process. Apart from this, the annotations in the test split are usually given as bounding boxes, which keeps consistent with other spatial localization tasks. But restricted by the architecture and learning paradigm, most of the proposed models can only generate results in the format of heatmaps but not bounding boxes, bringing much inconvenience to the further utilization of located regions.
In spite of this, when we look into another subproblem of referring object localization, i.c., image grounding, we may find that the data in this field are usually equipped with accurate bounding box annotations, which brings higher information density and better model performances. So is there any suitable solution to extract and migrate the knowledge from this field to our target scenario? The answer is Yes. As with the pivot-based alignment strategies adopted in the zero-resource machine translation [6]] 40 23 52]. we can turn to the datasets of audio retrieval and use series of aligned text-audio pairs as intermediaries to establish an efficient knowledge transfer pipeline to tackle the target problem in zero-resource manner. Although it sounds straightforward and feasible theoretically, such a learning scheme is still faced with two kinds of practical challenges: 1) Domain gap, which consists of the essential discrepancies between all these three data domains. As depicted in the green part of Figure I the queries in image grounding put more emphasis on the appearance, quantity and spatial relationship of the objects, while those in audio retrieval tend to describe action, state and temporal change about the targets. Meanwhile, similar gaps also exist in the visual inputs (illustrated in the red part) and the acoustic data in the other two modality pairs. 2) Modality gap. which stems from the intrinsic properties within different modalities and the complicated relationships between them. The noise and redundant information in these three modalities will hamper the transfer process from different aspects, and the non-monotonie alignment between modalities also makes hard to learn modsl representation in fine granularity.
Gues clear glass containing green plant on the left with two similar sitting to right
Image Grounding
Gmgg1
Image Gap
$19
Text Gop
2
Guo2 : Wind and rain sounds,
then man begins to talk
Gang
Audio Retrievol
Audio Gap
Sounding Object Locolization
Figure 1: The major challenges in our problem setting.
To this end, we propose a novel Two-stream Universal Referring localization Network (TURN) to handle the problem of sounding object localization in zero-resource way and reduce the potential performance degradation from the challenges described above. Specifically, we utilize a localization stream model the relationship between visual elements and referring queries so to unearth valuable location knowledge from image grounding data. Meanwhile, we develop an alignment stream construct universal semantic space for the linguistic and acoustic data, thus enabling efficient knowledge transfer to our target task. Moreover, considering that different data items will affect the transfcrability of knowledge strategy to automatically identify the overlap between various data domains and accordingly modulate the sampling weights of data, which enhances the stability of the learning process and improves the all-around performance of our model.
varying extent, we further devise an adaptive sampling
In conclusion, our contributions this paper can be summarized in the following aspects:
We explore the task of sounding object localization from zero-resource perspective for the first time and discuss the feasibility of transfer-based solutions in this setting.
We devise novel adaptive sampling mechanism in the area of multi-modal knowledge transfer, which can automatically discover the overlap between different data domains and adjust the sampling weights of data, thus enhancing the comprehensive performance of the learning model.
We propose an effective two-stream architecture for the target of zero-resource sounding object localization. Without using any training data in this field, TURN achieves competitive
2

---

performance on publicly-available benchmarks, including MUSIC and VGGSS, which demonstrates the feasibility of our proposed solution.
2 Related Work
Audio-Tex: Representation Learning Cross-modal representation learning aims to capture the relationship between different modalities and project embeddings to the same latent space. Though
many works have studied image-text [22 [15 44 and audio-visual representation learning [5] 38 [3],
audio-text representation learning is still under-studied. Some works [38 8 35] directly computed
the similarity between embeddings but cannot well encode the semantic and acoustic information
into the embeddings at the same time. Other works such as Chen et al. [2] attempted to learn interactive moddll representation with both acoustic and textual information followed the spirit of contrastive eagguage-imagee pre-tra wre-tranning, but might be deeply affected by the insufficiency of training data. Considering that VQ-VAE (56 6 68) has shown its outstanding performance in discrete representation learning, we try to learn the audio-text representation via cross-modal reconstruction and codebook sharing based on the structure of VQ-VAE.
Transfer Learning Typical transfer learning is usually applied to solve the problem of data
distribution inconsistency between source and target domains. The mainstream approaches this area usually tackle this problem from one of these three perspectives: data inputs [66 3
66
intermediate representations g 20
58 67 and learning paradigms 36
30
32
However, these works are based on the assumption that the source and target domains are conceptually close and modally consistent. In order to handle the situation where data from the source domain and target domain are not closely related, Tan et al. (54) formulated the paradigm of Transitive Transfer Learning (TTL.) and exploited an auxiliary annotated intermediate domain to transfer knowledge. And considering the cost of constructing the intermediate domain, Tan et al. 55 further proposed a novel learning schema Distant Domain Transfer Learning (DDTL), in which the intermediate domain will be automatically selected and constructed from multiple datasets without labels, thus achieving the knowledge transfer between completely unrelated domains. Recently, some researchers have also discussed the usage of transfer learning strategies in the multi-modal scenarios, which covers the field of image caption [39], audio caption [7T], machine translation [40] and so forth. In this paper, we further explore the application of transfer learning paradigms in the task of zero-resource sounding object localization as a complement to this field.
Referring Object Localization Given complicated scene with multiple objects and specific referential query, the task of referring object localization is to ascertain the location of the object that best matches the given reference. Depending on the modality of queries, this field can be subdivided into Image Grounding and Sounding Object Localization. In the scenario of image grounding, the queries are given in natural language descriptions, and fully-annotated data are denoted to facilitate this area. Consequently, a lot of supervised approaches and architectures are proposed to tackle this problem, which can be further categorized into proposal-based 24 28
59
60
62, 69 72 and
The former estimate the confidences of all the pre-defined in ranking-basedd way, while the latter directly regress the coordinates of bounding boxes based on the fully aggregated and fused representations. When it comes to acoustic queries, clear box annotations are usually unavailable in the training partition of previously proposed datasets 14 48 70]. Therefore, researchers in this field turn to explore the 25 based on contrastive learning paradigms and produce pixel-level heatmap results in a weakly-supervise
proposal-free ones 17 33 47 64
63
proposals and choose the best
solutions 226 42 45 5T
way. Although sounding object localization shares lots of commonalities in essence with image grounding, there is still lack of discussions and analyses of their relationships and characteristics. In this paper, we make pioneering attempt to link these two universal referring object localization.
subproblems and explore
3 Method
3.1 Problem Formulation
As depicted in Figure 2 our goal is to extract adequate knowledge from the data of image grounding and modulate it with the assistance of well-alig well-aligned audio-text corpora in the field of audio retrieval,
3

---

Locoliaation Stream
Lreg
A
V
CNN
DETR Encoder
Potch Encoder
Cior
Cadebooks
CNN
Textl:A yallow hawl with broccalt
Text Encoder
Text Decouler
DETR
Audio Encoder
Domain Clossifir
T
Text2: woman tolks pesrs.
Text Encoder
Text Decoder
Codebooks
Patch Encoder
CNN
Audio Encoder
Audio Decoder
Arpocom
Allgnment
Stream
0
8t"h
Leade
a
Training Phase
Testing Phose
Mesiied Rustion
UJ Gradiens
W, W3 Somple Welgit
Figure 2: Overall architecture diagram of our model TURN. Best viewed in color mode and with zoom-in. The green line localization stream and other lines are combined to form alignment stream.
thus achieving acceptable generalization performance the task of sounding object localization. For the sake clarity, we regard the data image grounding, audio retrieval and sounding object localization are from the source. intermediate and target domains with the notations of D* D and D', respectively. Under this definition, this problem setting can be also considered as a knowledge transfer and generalization from D* to D via D To keep consistent with the other prior works and the characteristics of different domains, only a limited amount of data from source and intermediate domains are accessible to the model during training. and the annotated testing data of target domain will be finally used to measure the overall generalization ability in the evaluation.
3.2 Model Architecture and Data Specification
Overall Network Structure The overall workflow of our proposed model is depicted in Figure Specifically, our model conducts the calculations in two parallel pathways referred to as localization and alignment streams in the following descriptions. The localization stream focuses modeling dependencies within visual elements and capturing relationships between inputs and queries, thus accordingly producing the target bounding boxes. In this part, a series of pre-trained backbones are first utilized to embed the input images into a group of object-aware features. And these features will be further refined and aggregated with the guidance of query conditions and then used predict the coordinates of target regions. Meanwhile, the alignment stream is developed to align the data distribution of different domains and modalities into shared space and generate debiased query conditions to serve as references for the localization stream. In order to construct semantically complete latent space of query embeddings, we exploit a group of shared vector quantizers and devise coossreconstruuction mechanism to alleviate the domain and modality gaps existing in the input audio and text data. Moreover, we additionally employ an auxiliary classifier to adjust the sampling weight during the training phase to help our model adaptively absorb knowledge from different domains.
Inputs and Annotations Before the detailed descriptions of our proposed architecture, first formulate the concrete symbolic definitions of the inputs, annotations and representations. To begin with, the original data from source, intermediate and largel domains are denoted as D
{(V", T b²)= E (V*, (T,T,B)), D {(T "T" E (ym { and D' = [(V', A A',b') e omit the superscript in the following explanations when there is no need to distinguish between different domains. In these notations, V E [0, |3x x T e N and A Rd. represent the pixel matrix, word index sequence, and log-mel spectrogram of the image, text and audio inputs, respectively. As for the annotation format, b = [a, y, w. h) E [0, 1]* represents the target bounding box of queried object.
(V", Tt, ]. For simplification,
3.3 Localization Stream
As the basic component of referring object localization, the localization stream is applied to obtain location-related knowledge from the source domain and produce target bounding box predictions.

---

Given an image containing multiple objects V E [ H
H&W and semantically condensed query condition q E R.,, we first project the visual input into a series of object-aware embeddings V [Vi...., Vn,J via a pre-trained DETR backbone 110] and a group of multi-layer perceptrons, where V E R and ny is the input sequence length of the DETR encoder.
After the initial extraction procedure, we combine the guidance information from query conditions into the visual representations via a simple flement-wise multiplication. And then, we employ another transformer encoder component to model the dependencies and relations among these query-aware visual features and generate a bunch of fully fused representations, which can be given by
My (O, ......, In,)), where V, V; q q q/12-
(I)
In this formula, the notation ((...) stands for the transformer encoder module and we conduct a 12-norm to prevent numerical instability caused by the dot product calculation.
Finally, we calculate the aggregated representation by averaging all the patch features and utilize a multi-layer perceptron to generate the prediction of bounding boxes, given by
b a WW (Re + cj)) + c²),
where Wi Rddd W R and cj c E R are all learnable parameters, a+)) is the sigmoid function and [0, 1]4 is the predicted bounding box.
3.4 Alignment Stream
In the overall knowledge transfer procedure, the alignment stream plays a crucial role in extracting the domain- and modality -ivvarittt representations from the original inputs and providing query condition features as the reference of the localization stream.
In the training phase, we first project the text data into a series of continuous word embeddings, denoted as T = {ii .......;, iं.,} and Tum (ir.... ..... (y, respectively. And the log-mel spectrogram of audio data will be compressed via a stack of convolutional layers, given as Å TM {af",... & J. Afterward, utilize two transformer encoders to generate context-aware linguistic and acoustic features and aggregate them into a single semantically representation t, a E R as formulated in Equation 3 It's worth noting that the word embedding table and text encoder are shared between source and intermediate domains so as to filter out the domain -specffc bias within the original data.
E 1 Le (E tL.)). a= a= (o,. &L.)). La
Next, we expect to learn a universal semantic representation space where the data with the same semantics but from different domains or modalities will be mapped to adjacent positions so that the localization of target objects can be independent of the query source. Regarding the remarkable performance of variational autoencoder in fitting distributions of variables, we employ the straight-
through VQ-VAF (56) module achieve this goal. Following the strategy adopted in 7 we split the aggregated features into total of chunks and utilize group of codehooks (.)()), (.),.... V (-))
to project them into multiple shared semantic spaces and reorganize them into series of debiased
query representations ł, å E Rd, which will serve as the query condition term in Equation | and can be given by
E [T, (T.); (G)E))
(ā))) (a,(),))
(4)
where [:] is the concatenation operator and E, a, E Igd/a represents the i-th part of and a (i.e., ? 6= -- :U::...;u. Eg] and a = ;a,;
Afterward, further devise a welf-cross-reeconstriid query representations and enhance the mutual information between aligned texts and audios.
mechanism to improve the robustness of
(5) where T maak and Amask are the modified inputs which a certain proportion of the original data into the <mask> embedding and ((X, Y) is a mask mask-prredcct transformer decoder to reconstruct the masked elements of with the reference of X. As formulated Equation 5 the texts from the source domain are reconstructed with the guidance of itself and the reconstruction in the intermediate domain will be performed a cross- modsliyyy way. Thereafter, the aggregated features of one modality will be able to get modulated in a data-driven manner serve reasonable substitutes of the other one.
T = (i T ); Tu 22 (ām. Tm ( A Ω2 (i A m mask))
5

---

3.5 Training and Inference
As mentioned above, the entire architecture can be trained in an end-to-end mechanism. Given the annotation b from the source domain, we follow the learning strategy adopted in [17] and apply combination of regression- based and loU-based loss functions to optimize the localization stream, formulated as
Lioe = A₁ Creg (6, b) A2Ciou (b, b)
where the freg (-..) :) is the smooth 1 loss to optimize the regressions of each coordinate separately, and (.))(".)) is the GloU loss to conduct an overall constraint on predicted boxes.
As for the alignment stream, the optimization target is composed of the following three aspects, which will be presented in the logical order of calculations. Initially, in order maximize the possibility of mapping the aligned linguistic and acoustic data to the same discrete code, we apply contrastive in every mini-batch to constrain the distribution of
learning paradigm with the InfoNCE loss 50 57 aggregation vectors, given by
log exp(D(t,.. cxp(D((",aa"/Ta a/T)/TT))
Ceontra
Σ 1m 2j exp(D(t,", (00)),aa)).)) a//)/To)
(7)
where is the size of each mini-batch and Ta the temperature hyper-parameter. y( measures the distance between two vectors and y. and we use the the normalized inner-product to calculate the similaiim distance in this part.
And next, all the codebooks will conduct calculations in parallel and get optimized in a K K-means based manner individually, which can be abstracted as
Loode
||sg[[] - (f) + B||r - sg. (f)]lI,
where f denotes the input of codebooks and sg(:) represents the stop gradient operation.
Moreover, two reconstruction losses will be utilized indirectly modulate condition representations. given by
Drecon = Lee (T, Droom Li, (Ā, A) where Loe(-.)) and Ci, (:.)) are the cross-entropy and 4 loss functions, respectively. In conclusion, the optimization of alignment stream is summarized as
Cain A3Coode + Ag Crecon) + Asfcontra:
(10)
Finally, the overall loss function combined as ftoe + Luin to learn our proposed architecture, where Ai..... As the balancing factors to control the magnitude of corresponding terms. And in the process of inference, we can simply take the projected embeddings of audio data as the condition representations to the localization stream and generate the calculation result.
3.6 Adaptive Sampling Strategy
Although we have already obtained set of domain- and modality-invanl invariant representations and performed sufficient aligmments, the huge inherent gaps between the source and intermediate domains still heavily hamper the transferability and generalization ability. As illustrated previously, there tend to be some domain-specifee data samples in the corpora. For example, "The half of the sandwich on the right" might be a good item for image grounding, but there merely exists any acoustic information in this scenario. And it will also be too abstract imagine the concrete object corresponding to "A loud
sound takes place." " Generally, sample like "A car was going on the road." may serve as excellent point to bridge knowledge. To this end, we try to estimate the domuin-exclusivity of data items and take this as reference for data sampling. Specifically, given all the queries from T and T first employ extra classifier (b-) to discriminate between them and use the domain labels to optimize this module, formulated as
Lext
Σ (og(a(((t))) tET
Σ - (1(1-)((((")))
(11)
After this training procedure, the classifier will respond with higher confidence to the data from the source domain and lower confidence to those from the intermediate domain. On this basis, we can

---

safely assume that t" with lower confidence (tt") and t with higher ((t")) ought to be shared between these two domains and located within the intersection part.
Consequently, the data sampling weights will be assigned adaptively via the following mechanism.
exp( exp(-0((t) (+))//²)
exp(o(t,")//=
- Σ, expl (xp((=((()))) (t,)//+*)
(12)
Σ,
and we have
max (e(()) min ((()))
(13)
In(k)
where wf is the probability for the i-th item to be selected into
in the training phase. In this formula, the term r" applied to smooth the distribution of sampling weights and the expected ratio of the maximum and minimum in all weights, constraining the range of possible results and reducing the impacts of outliers to some extent.
Experiments and Results
4.1 Datasets and Metries
Image Grounding Datasets We choose RefCOCO 65 /ReCCOOO+ (65//ee (37 as the image grounding datasets and conduct in-depth studies on the RefCOCOg dataset. The statistic information of these three datasets can be found on the webpage of Tensorflow. We follow the data partitions from Yu et al. 665 and only use the training set the overall learning process.
Audio Retrieval Datasets As for the audio-text datasets, we use Clotho [9] and AudioCaps [29] Clotho is sourced from the online platform Freesound [21] and contains 4.981 audio samples and 24,905 captions (5 captions for each audio clip). AudioCaps contains video clips and corresponding natural language captions, and we only use the audio clips extracted from the videos, resulting in totally 50,960 audio-text pairs for training.
Sounding Object Localization Datasets For the dev set, we use the annotated subset of Flickr- SoundNet [48), which is built on Flickr-SunndNet dataset 4 and contains 2,786 annotated image- audio pairs. For the test set, we choose VGGSS [4] and MUSIC 170] VGGSS is derived from VGGSound [3] and contains 5,158 audio-image pairs. MUSIC, as a musical instrument video dataset, consists of 685 untrimmed videos of musical solos and duets spanning 11 instrument categories. We follow the data process pipeline of Hu et al. [27], and finally we use 489 solo videos for test.
we adopt Consensus Intersection over Union (cloU) and Area Under Curve (AUC) to evaluate the performance of our model. It's noteworthy that cloU and vanilla IoU will be equivalent under the circumstance of n = 1 as shown in Equation 14
Metries Following 43
48 14
m 1). cloU((m,) m) = IoU m),
min
(14)
where (my by and m are the binary masks converted from ground-truth and predicted boxes.
4.2 Comparisons with State-of-the-art Methods
Though TURN is designed for the zero-resource scenario, it still reaches a counterpart to other full-resource state-of-the-art sounding object localization methods. Here we compare our TURN with previous leading methods on VGGSS and MUSIC-Solo as Table and Table Table reeppectivel And is noteworthy that retain all the samples with multiple box annotations of VGGSS in the entire evaluation process. We can find that TURN outperforms all of the previous methods in VGGSS in both cloU and AUC metries. In MUSIC-Solo, the results among all the methods with a reasonable IoU score. It is also worth noting that the cloU and AUC of both datasets suffer sharp drop when random vector takes the place of the actual query in the localization stream (denoted as TURN (w/o Query)). This observation shows that is unreliable to retrieve the salient region as the predictions by only the localization stream and further indicates
similar TURN obtains the best AUC
7

---

the validity of our proposed knowledge transfer strategy. Apart from this, we can also note that the loU@0.5 metric on the MUSIC-Solo dataset relatively low when compared to other methods. We infer that this phenomenon may stem from the inherent annotation gap between image grounding and sounding object localization, and the low proportion of music instruments the training datasets.
Table 1: Performances on VGGSS dataset.
Table 2: Performances on MUSIC-Solo dataset.
Method
cloU @.05
AUC
Method
loU@0.5
AUC
Attention 48
17.1
28.7
object-that-sound
2
26.1
35.8
AVobject
29.7
35.7
DMC 25
29.1
38.0
HardWay 14
31.9
37.0
Attention 48
37.2
38.7
LessMore
32.2
36.6
Sound-of-pixx
40.5
43.3
SSPL 51
33.9
38.0
DSOL 26
51.4
43.6
TURN (w/o Query)
12.2
26.9
TURN (w/o Query)
16.9
34.1
TURN (w/ Query)
34.6
39.1
TURN (w/ Query)
33.7
45.2
4.3 Ablation Studies
In this part, we conduct ablation studies to verify the effectiveness of each design detail in TURN according to the results on VGGSS and MUSIC-Solo.
Choice of Source and Intermediate Datasets TURN needs to generalize knowledge from the source domain the target domain with the assistance of the intermediate domain. Therefore, make a comparison between different selections of source and intermediate datasets, which is illustrated in Table Consistent with our expectations, the best choice for the source domain is RefCOCOg. Compared with RefCOCO and RefCOCO+, the grounded textual descriptions in RefCOCOg contain more details about the object itself, which vital when aligned to the texts from the intermediate domain. As for intermediate domain datasets, it is straightforward to find that AudioCaps is more suitable for the MUSIC-Solo dataset, and Clotho helps VGGSS more. The reason can be attributed to the fact that AudioCaps contains more instrument-relatee entries and indicates the names of different instruments in the descriptions, while the audio quality in Clotho is much higher, and there are multiple descriptions assigned to the same audio, which reduces noise disturbances in general.
Table 3: The results of different source and intermediate domains on VGGSS and MUSIC-Solo.
VGGSS
MUSIC
Source
Intermediate
cloU@0.5
AUC
loU@0.5
AUC
RefCOCO
AudioCaps
23.13
33.89
15.87
36.06
RefCOCO+
AudioCaps
27.26
35.27
29.12
43.95
RefCOCOg
AudioCaps
33.58
38.57
33.68
45.17
RefCOCO
Clotho
16.40
31.50
21.43
41.68
RefCOCO+
Clotho
22.51
33.19
36.47
42.89
RefCOCOg
Clotho 34.57
39.07
28.21
42,42
Effect of Different Training Settings In order to verify the effectiveness of regularization terms in our model, we conduct individual ablation study on each term. As illustrated in Table the contrastive loss takes the most significant effect, implying plays a leading role in the alignment of data from different modalities. And the codebook loss optimizes the learning process of shared semantic space, determining the effect of knowledge transfer. Although the reconstruction loss has no direct influence representations, it can still improve the performance by enhancing mutual information between different modalities. In addition, we also evaluate the adaptive sampling strategy, and the performance margin indicates the benefit it brings to the learning process.
Choice of parame-ps in Codebooks As described in Section 3.4 we use a group of codebooks to project aggregated features in the alignment stream. Here we investigate how the group number of codebooks and total codebook size influence the localization performance. From Figure we can observe that the total codebook size cannot be either too large (depicted by the green line, the embeddings from different modalities might be scattered into different partitions of the codebook space) too small (shown by the blue line, the codebook capacity is not enough). And given fixed

---

Table 4: The performance of different experimental settings on VGGSS dataset.
Contrastive
Codebook
Reconstruction
Adaptive Sampling
Clotho
AudioCaps
Loss
LOss
Loss
cloU
AUC
cloU
AUC
34.57
39.07
33.58
38.57
28.56
36.75
29.24
36.55
30.09
36.55
28.42
36.09
V
V
V
27.53
36.37
27.70
36.13
26.44
34.62
21.21
34.23
total size, the codebook group number should also be tuned in a proper range. According to the experiment results, when the codebook number is set to be and the total codebook size is set to be 512, TURN gets the highest cloU and AUC.
35.0
25
32.5
20
30.0
15-
27.5
2
The group number codebooks
Figure 3: Effect of the hyper- parametrrsss of codebooks on VGGSS. The curves of cloU and AUC are drawn in the solid lines and the dashed lines, respectively.
Choice Reconstruction Settings As formulated in Eq, 5 the reconstruction text and audio data will be conducted in Self-Cross manner, which is expected improve the mutual information between text and audio pairs and eliminate modality and domain gaps between different data sources. To evaluate the effect of our reconstruction strategy, further examine other combinations in the format of "A-B", where denotes the reconstruction type adopted in the source domain and B denotes the adopted in the intermediate domain. The performance comparison can be found in Figure 4 It's worth noting that "No" means "no reconstruction will be applied.", ", and "Self/ Cross" means cross-modality manner". From Figure 4 find that the reconstruction in each domain is essential and decisive to the final performance, and the Self-Cross one behaves superior the other combinations.
that "the reconstruction will be conducted in uni- /
Self-No
clou
Self-Self
Self Cross
28
32
34
38
40
Figure 4: The performance different reconstruction settings on VGGSS dataset.
4.4 Qualitative Analyses
In order to comprehensively evaluate the performance of our TURN, we will analyze some success cases both the VGGSS and MUSIC-Solo datasets, which are shown in Figure 5 (The failure ones will be discussed in the supplementary material From these cases, we can easily find that our
And more details about experiments also be found in the supplementary material.

---

model obtains the basic ability to discriminate the corresponding target according to the specified sound from complex scene. Limited by the structural design, when faced with the multi-bos scenarios of the VGGSS dataset, our model can only try to randomly choose one (c) or take their union (d) as the prediction among multiple equivalent candidates. Although this will result in a lower metric score, these cases can still qualitatively indicate the satisfactory performance of our proposed architecture.
EVANS
(a) Cello Performance
(b) Stack Train
(c) Thrush Birds
(d) Drum Set Performance
Figure 5: Success cases with one or more ground-truth annotations in MUSIC-Solo (a) and VGGSS (b, and d). The predictions and ground-truths marked in green and red boxes, respectively.
5 Conclusion
In this paper, develop TURN, an effective two-stream architecture for zero-resource sounding object localization, which consists of localization stream and an alignment stream. We explore the task of sounding object localization from a zero-resource perspective for the first time and discuss the feasibility of transfer-basee solutions in this setting. Without using any training data from field sounding object localization, our proposed TURN reaches counterpart to other full- resource SOTA methods on publicly-availabl benchmarks (including MUSIC and VGGSS), which demonstrates the feasibility of our TURN. More detailed analyses further verify the effectiveness of the overall mechanism and architecture, and the qualitative analyses also provide more intuitive and comprehensive perception the results.
Acknowledgments and Disclosure of Funding
This work is supported by Zhejiang Natural Science Foundation LR 19199220006 and National Natural Science Foundation of China under Grant No. 6o.62272397 and No.61 836002. And this research also supported by Yiwise and Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies.
References
[1] Triantafyllos Afouras, Andrew Owens, Joon Chung, and Andrew Zisserman. Self-supervieed learning of audio-visual objects from video. ArXiv, abs/2008.0422 2020.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pages 435-451, 2018.
[3] Yusuf Aytar and Andrew Zisserman. Tabula rasa: Model transfer for object category detection. 201 International Conference on Computer Vision, pages 2252-2259, 2011.
[4] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 29, 2016.
[5] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. See, hear, and read: Deep aligned representations. arXiv preprint
2017.
Alexei Bacvski, Steffen Schneider, and Michael Auli. wq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2019.
[7] Alexei Baevski, Steffen Schneider, and Michael Auli, vq-wav2vec: Self-supervised learning of discrete speech representations. ArXiv, abs/1910.054522 2020.
[8] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recognition. Advances in Neural Information Processing Systems, 34, 2021.
10

---

[9] Shai Ben-David, John Blitzer. Koby Crammer. and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
[10] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ArXiv, 12bs/2005.1287... 2020.
[11] Fabio Maria Carlucci, P. Russo, Tatiana Tommasi, and Barbara Caputo. Hallucinating agnostic images to generalize across domains. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3227-3234, 2019.
[12] Chen Chen, Nana Hou, Yuchen Hu, Heqing Zou, Xiaofeng Qi, and Eng Siong Chng. Interactive audio-text representation for automated audio captioning with contrastive learning. arXiv preprint 22Xx::2203.15566 2022.
[13] Honglie Chen, Weidi Xie. Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[14] Honglic Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. 2021 IEEE/CVF Conference on Computer Vision and Pattenn Recognition (CVPR), pages 16862-16871, 2021.
[15] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer, 2020.
[16] Wenyuan Dai, Qiang Yang. Gui-Rong Xue, and Yong Yu. Boosting for transfer learning. In ICML '07. 2007.
[17] Jiajun Deng, Zhengyuan Yang. Tianlang Chen, Wen gang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1759, 2021.
[18] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NeurlPS, 2019.
[19] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning datasct. ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 736-740, 2020.
[20] Sarah Erfani, Mahsa Baktashmotlagh, Masud Moshtaghi, Xuan Nguyen, Christopher Leckie, James Bailey, and Rao Kotagiri. Robust domain generalisation by enforcing distribution invariance. In Proceedings the Twenty-Fiit FFitt International Joint Conference on Artificial Intelligence (IJCAI-16), pages 1455-1461. AAAI Press, 2016.
[21] Frederic Font, Gerard Roma, and Xavier Serra. Freesound technical demo, Proceedings of the 21st ACM international conference on Multimedia, pages 41 1-412, 2013.
[22] Xiangteng He and Yuxin Peng. Fine-grained visual-textual representation learning. IEEE Transactions on Circuits and Systems for Video Technology, 30(2):520-531, 2019.
[23] Julian Hitschler. Shigehiko Schamoni, and Stefan Riezler. Multimodal pivots for image caption translation. ArXiv, abs/1601.03916 2016.
[24] Richang Hong, Daqing Liu, Xiaoyu Mo. Xiangnan He, and Hanwang Zhang. Learning to composc and reason with language tree structures for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:684 696, 2022.
[25] Di Hu, Feiping Nic, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. 2019 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9240-9249, 2019.
[26] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discrim- inative sounding objects localization via self-supervised audiovisual matching. ArXiv, abs/2010.05466. 2020.
[27] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Dis- criminative sounding objects localization via sulf-supe audiovisual matching. Advances in Neural Information Processing Systems, 33:10077 10087, 2020.

---

[28] Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. Modeling relationships in referential expressions with compositional modular networks. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 4418 1188-447,220 2017.
[29] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 119-132, 2019.
[30] Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learning for domain generalization. ArXiv, abs/1 100003663, 2018.
[31] Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng, Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 624 639, 2018.
[32] Yiying Li, Yongxin Yang. Wei Zhou, and Timothy M. Hospedales. Feature-critic networks for heteroge- neous domain generalization. In ICML, 2019.
[33] Yue Liao, Si Liu, Guanbin Li, Fei Wang. Yanjie Chen, Chen Qian, and Bo Li. A real-time cross moos-ito correlation filtering method for referring expression comprehension. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10877-10886, 2020.
[34] Daqing Liu, Hanwang Zhang, Zhengjun Zha, and Feng Wu. Learning to assemble neural module tree networks for visual grounding. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 4672-4681, 2019.
[35] Siyu Lou, Xuenan Xu, Mengyue Wu, and Kai Yu. Audio-text retrieval in context. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4793-4797. IEEE, 2022.
[36] Massimiliano Mancini, Samuel Rota Buld, Barbara Caputo, and Elisa Ricci. Best sources forward: Domain generalization through source-specific nets. 2018 25th IEFE International Conference on Image Processing (ICIP), pages 1353-1357, 2018.
[37] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana-Maria Camburu, Alan Loddon Yuille, and Kevin P. Murphy. Generation and comprehension of unambiguous object descriptions. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11 11-20, 2016.
[38] Arsha Nagrani, Samuel Albanie, and Audrew Zisserman. Learnable pins: Cross-modal embeddings for person identity. In Proceedings of the European Conference on Computer Vision (ECCV), pages 71-88, 2018.
[39] Arsha Nagrani, Paul Hongsuck Sco, Bryan Scybold, Anja Hauth, Santiago Manén, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. ArXiv, abs/2204.00679 2022.
[40] Hideki Nakayama and Noriki Nishida. Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot. Machine Translation, 31:49-64, 2017.
[41] Sinno Jialin Pan, Ivor Wai-Hung Tsang, James Tin-Yau Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22: 22:199-210, 2011.
[42] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu. Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. ArXiv, abs/2007. 063555, 2020.
[43] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu. Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In European Conference on Computer Vision, pages 292-308. Springer, 2020.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, PMLR, 2021.
[45] Janani Ramas and Sukhendu Das. See the sound, hear the pixels. 2020 IEEE Winter Conference on Applicutions of Computer Vision (WACV), pages 2959-2968. 2020.
[46] Jongbin Ryu, Gitack Kwon. Ming-Hsuan Yang. and Jongwoo Lim. Generalized convolutional forest networks for domain generalization and visual recognition. In ICLR, 2020.
12

---

[47] Arka Sadhu, Kan Chen, and Ramakant Nevatia. Zero-shot grounding of objects from natural language queries. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4693-4702, 2019.
[48] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In-So Kweon. Learning localize sound source in visual scenes. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4358-4366, 2018.
[49] Arda Scnocak, Hyeong Sun Ryu, Junsik Kim, and In-So Kweon. Less can be more: Sound source localization with classification model. 2022 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 577-586, 2022.
[50] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
[51] Zengjie Song, Yuxi Wang, Junsong Fan, Tieniu Tan, and Zhaoxiang Zhang. Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes. ArXiv, 2ttp/22033134122. 2022.
[52] Lucia Specia, Stella Frank, K. Sima' an, and Desmond Elliott. A shared task multimodal machine translation and crosslingual image description. In WMT, 2016.
[53] Didac Surís. Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i Nieto. Cross-modee modal embeddings for video and audio retrieval. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[54] Ben Tan, Yangqiu Song, Erheng Zhong, and Qiang Yang. Transitive transfer learning. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015.
[55] Ben Tan, Yu Zhang, Sinno Jialin Pan, and Qiang Yang. Distant domain transfer learning. In AAAI, 2017.
[56] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[57] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, ArXiv, abs/ /e///887.03777 007.00378.222 2018
[58] Jindong Wang, Yigiang Chen, Wenjie Feng, Han Yu, Meiyu Huang, and Qiang Yang. Transfer learning with dynamic distribution adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11 (1):1-25, 2020.
[59] Liwei Wang. Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:3944407, 2019.
[60] Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel. Neighhourhood watch: Referring expression comprehension via language-guided graph attention networks. 2019 IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1960-1968. 2019.
[61] Hua Wu and Haifeng Wang. Pivot language approach for phrase-batd based statistical machine translation. Machine Translation, 21:165- 2007.
[62] Sibei Yang. Guanbin Li. and Yizhou Yu. Dynamic graph attention for referring expression comprehension.
2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4643 4652. 2019.
[63] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4682-4692, 2019.
[64] Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. ArXiv, abs/2008.01055 2020.
[65] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg, Modeling context in referring expressions. ArXiv, 011116600002222 2016.
[66] Xiangyu Yue, Yang Zhang. Sicheng Zhao, Alberto L. Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to- generalization without access- ing target domain data. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2100 2110 10, 2019.
13

---

[67] Chen Zhang, Yi Ren, Xu Tan. Jinglin Liu, Kejun Zhang, Tao Qin, Sheng Zhao, and Tie-Yan Liu. De- noispeech: Denoising text speech with frame-level noise modeling. In ICASSP 2021-2021 IEEE International Conference on Acousticx, Speech Signal Processing (ICASSP), pages 7063-7067. IEEE, 2021.
[68] Chen Zhang. Xu Tan, Yi Ren, Qin, Kejun Zhang, and Tie- Yan Liu. Uwspeech: Speech to speech translation for unwritten languages. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14319-14327, 2021.
[69] Hanwang Zhang, Yulci Niu, and Shih-Fu Chang. Grounding referring expressions in images by variational context. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4158 4166, 2018.
[70] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh H. McDermott, and Antonio Torralba. The sound of pixcls. ArXiv, abs/1 804.03160, 2018.
[71] Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, and Yejin Choi. Connecting the dots between audio and text without parallel data through visual knowledge transfer. ArXiv, abs/21 12.08995, 2021.
[72] Bohan Zhuang. Qi Wu, Chunhua Shen, Ian D. Reid, and Anton den Hengel. Parallel attention: A unified framework for visual object discovery through dialogs and queries, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4252-4261, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
3. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
14


Reviews:
[{"review_id": "odkq83oS-bG", "reviewer": "Reviewer_oJxK", "summary": "In order to localize the object emitting a specified sound, i.e., sounding object localization, this paper proposes a Two-stream Universal Referring localization Network (TURN) concerning the localization stream and alignment stream. The TURN model correlates the referring image localization via the localization stream and text-audio semantic space via the alignment stream. Such a mechanism transfers the audio query through the learned codebooks, retrieved from the text-audio semantic space, to indirectly trigger the referring object detector. In addition, the authors provide an adaptive sampling strategy for helping training. The experiments show the proposed model on par with the existing methods.", "questions": "The major concern is the motivation to treat the sounding object localization task as a combination of image grounding and audio retrieval tasks. Such treatment seems complicated to the target task and the resulting model. In particular, previous methods [48, A] showed that the heatmaps retrieved from the sounding object localization could generate the corresponding boxes.", "limitations": "No limitations and potential negative societal impact included.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths] \n+ The idea of utilizing the zero-resource perspective for tackling sounding object localization is interesting.\n+ The manuscript is well organized and has adequate references.\n\n[Weaknesses] \n- The technical novelty is limited since the TURN model is mainly composed of DETR [10], VQ-VAE [55], and VQ-WAV2VEC [7]. The TURN model shows an application employing these methods [7,10,55] with an additional adaptive sampling strategy for helping the model training.\n- The proposed method makes the task of sounding object localization from the audio-image matching problem to the audio-text-image matching one. The proposed method seemingly complicated the original task of acquiring the box-level prediction. The proposed method also implies that the extra data are needed yet ignoring the available training data from the field of sounding object localization. However, the performance gain is not significant.\n- While evaluating the model, does the TURN model evaluated with the box-level prediction? It seems that the box-level prediction is not the same as the heat-map prediction, as shown in [A].\n- Figure 2 needs more descriptions and notations to gain a better understanding.\n- What is the so-called `patch’ in line 168 used in a DETR backbone [10]?\n- The loss terms mentioned in section 3.5 should be consistent with the notations in figure 2.\n- The reference is not sufficient. For example, some related methods are shown as follows.\n[A] Arda Senocak, Hyeonggon Ryu, Junsik Kim, In So Kweon: Less Can Be More: Sound Source Localization With a Classification Model. WACV 2022: 577-586.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o88oHIoP_W", "reviewer": "Reviewer_YYdX", "summary": "This paper aims to locate sounding objects in an audiovisual scene without resorting to specialized training data for sound localization. It designs a new framework Two-stream Universal Referring localization Network to achieve multi-modal knowledge transfer among visual, text and audio modalities. Specifically, the localization stream locates objects from related texts using DETR structure, the alignment stream establishes a universal feature space shared by audio and text. In this way, it manages to localize the objects related to the specific sounds without extra annotations. The experiments show the effectiveness of the proposed method.", "questions": "My questions are stated in the weakness part, mostly on the experiments.", "limitations": "Yes, the authors solve the limitations of lacking benchmarks. And there is no obvious negative social impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is well motivated, and the idea of transferring knowledge from three modalities is interesting. It solves the sounding object localization task from a new perspective, i.e., directly regress bounding box instead of generate heatmaps, and does not require extra human annotations.\n2. The paper is well written and easy to follow, and most technical details are well illustrated. Though some are borrowd from previous works, but generally it is a suitable application in this task and effective in multi-modal konwledge transfer.\n3. The adaptive sampling strategy makes sense and the learning procedure of the sampling weight is interesting. It fully utilizes domain information and selects reasonable training samples for text audio knowledge transfer.\n4. The results on single-source sounding object localization are promising. And the analysis on the failure cases is valuable. It demonstrates that this new formulation of sound localization instead of heatmaps is worth exploring and also figures out how to improve it.\n\nWeaknesses:\n1. The biggest concern is that this paper only performs experiments on single-source data, lacks the results on multi-source scenarios. On signle-source data, the sound localization is comparatively easy and the performance is mostly decided by the localization stream since the alignment is trivial. But on multi-source cases, it can validate whether the alignment stream establishes satisfactory semantic text audio alignment. And from my perspective, the image grounding task can help discriminate object semantics, and the text audio alignment also gather similar semantics, it should be able to deal with multi-source scenes with minor modifications. So I strongly suggest authors to try to expand the framework to more complex scenes for a test.\n2. The visualizations in Sec.4.4 show that multi-box cases are diffcult to process. Deeper analysis on what causes this problem is desired. For example, when providing the text embedding as query how will the localization result be. Better to figure out whether it is a problem from localization stream or alignment stream.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WJB21zT7Zq", "reviewer": "Reviewer_w7AN", "summary": "This paper studies sounding object localization from a zero-resource perspective. To solve this, the paper proposes a Two-stream Universal Referring localization Network(TURN), which includes a localization stream and an alignment stream. In order to solve the domain gap, an adaptive sampling strategy is used in this paper. There are many terms of loss function in this paper, the authors have carried out sufficient ablation experiments to prove the effectiveness of each part. The experiments in this paper prove the effectiveness of the model.", "questions": "1. In line 167，is each element in set $V$ $\\hat{v}_i$ or $v_i$? From the following, should it be $v_i$?\n2. Why does equation(5) use a cross-modality way in the intermediate domain? In the loss function, the contrastive learning loss has been used to align the audio modal and the text modal. What is the effect if the cross-modality way is not used here?\n3. The result of DML and DSOL is heatmap. Is it fair to compare the IOU calculated by heatmap and the bounding box of ground truth with the one calculated by the bounding box and the bounding box of ground truth?\n\nThe authors are expected to solve the problems in the weakness and questions, then I can accordingly modify my ratings.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n+ This paper proposes a Two-stream Universal Referring localization Network(TURN) to combine two domains(Image Grounding and Audio Retrieval) to solve zero-resource sounding object localization.\n\n+ An interesting point in this paper is that an additional classifier is trained to measure the domain gap of the sample, based on which the data is sampled for training. \n\n+ This paper also conducted sufficient ablation experiments on each loss in the model, proving the effectiveness of each part. In addition, the authors analyze the failure cases in the appendix to illustrate the limitations of the method.\n\n### Weakness\n\n- In Line 37-Line 39, the paper mentioned that most of the current models generate a heatmap rather than a bounding box, which bring much inconvenience. However, the inconvenience is not described in detail in paper.\n\n- Paper [1,2] also proposes a method to give the bounding box, which is not compared with the method in this paper.\n\n- The paper does not explain how the texts of two domains are aligned during training. Although an adaptive sampling strategy is proposed to solve domain gap, in the training process, the texts from two domains may be irrelevant in the same batch.(For example, in the same training batch, the image grounding text description may be \"a car was going on the road\" but the text description in audio retrieval may be \"a dog is barking on the road\")\n\n- The method in this paper can only give a single bounding box. For the mixed sound scenes, the method can not deal with it correctly.\n\n\n[1]  D. Hu, et al., “Class-aware sounding objects localization via audiovisual correspondence,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.\n[2] T. Afouras, et al. “Self-supervised object detection from audio-visual correspondence,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MzjZ-qEDzup", "reviewer": "Reviewer_mRAH", "summary": "* The paper proposes a method for training a sounding object localization model when there is no paired data available. It achieves this by transferring knowledge from paired data from overlapping modalities i.e. image-text and text-audio datasets.\n* The idea is that by aligning the text representations from the different paired datasets into the same space, we can learn to match the accompanying audio representation with the appropriate visual embedding.\n* The text and audio representations are projected into a common space using codebooks that are learned through the self-supervised training method of the vq-wav2vec [7] idea. The image representation is combined with the text representation from the codebooks and passed through a patch encoder to perform bounding box regression.\n* The experimental results on the VGSS dataset shows that the proposed method can perform better than other methods that use supervised data for training.\n", "questions": "* Can you provide results from an experiment that combined annotated and un-annotated data?", "limitations": "The paper can include a discussion the biases of different data sources that can propagate to the results, especially when performing zero-shot recognition.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The proposed method builds on ideas developed for individual modalities and brings them together to learn multimodal models in low resource settings.\n* The paper addresses the problem of sample weighting since there can be domain gaps in the source and intermediate domains. It uses an objective function that weighs the text embeddings such that they are hard to distinguish by a domain discriminator.\n* The results on the VGSS dataset are significantly better than the methods that have been trained in a supervised manner.\n* The paper also addresses the question of how to pick the source and intermediate domains via an ablation study.\n\n### Weaknesses\n* The IOU result on MUSIC-Solo is oddly very low compared to the other methods. The paper doesn’t provide any explanation or investigation for this result.\n* The paper doesn’t include an experiment that shows how to combine the annotated data along with the un-annotated data. It would be interesting to see how much additional benefit can be obtained from using un-annotated data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept