SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details. For audio clips, we pass  $96 \times 64$  log-mel spectrograms to VGGish, and the output is a 512D vector, while the raw waveform of the original 3s audio clip is sent to SoundNet. For frames, we used an input image of size  $256 \times 256 \times 3$ , with  $224 \times 224 \times 512$  as output. We choose RandAug [51] as strong augmentation, while random cropping, resizing, and random horizontal flip as weak augmentation. We set  $\delta$  as 0.6 and  $\tau$  as 0.7. More experiments of hyperparameters are shown in Appendix C.5.
Table 2: Comparison results on VGG-ss. Models are trained on VGG-Sound 10k and 144k.
Methods	VGG-Sound 10k	VGG-Sound 144k
CIoU	AUC	CIoU	AUC
Attention10k [14, 15]	16.00	28.30	18.50	30.20
LVS [28]	27.70	34.90	34.40	38.20
EZVSL [9]	32.30	33.68	34.38	37.70
SLAVC† [10]	37.80	39.48	39.20	39.46
SSPL [29]	31.40	36.90	33.90	38.00
SSL-TIE† [30]	36.80	37.21	38.60	39.60
Attention10k-SSL† [14, 15]	38.60	38.26	39.20	38.52
Ours (|Dl|=256)	41.20 (39.40)	40.68 (38.70)	43.60 (39.40)	41.88 (38.70)
Ours (|Dl|=2k)	43.20 (42.60)	40.82 (40.75)	45.60 (42.60)	43.24 (40.75)
Ours (|Dl|=4k)	46.80 (43.80)	43.18 (41.63)	48.80 (43.80)	45.76 (41.63)
5.2 Comparison with the State-of-the-art Methods
Comprehensive experiments show that DMT achieves the state-of-the-art performance among all existing methods on both datasets, and showcases several advantages.
Effective Utilization of Finite Annotations and Remarkable Performance. We tested DMT's localization performance with varying amounts of labeled data and found that it consistently outperforms state-of-the-art methods when with 256, 2k, and 4k labeled data. Notably, even with just 256 labeled data, DMT achieved an accuracy of  $87.2\%$  to  $87.6\%$ , showing a significant improvement in CIoU by around 10 absolute points compared to preceding models. Additionally, our model shows a  $3\%$  absolute improvement in CIoU compared to a supervised-only model. Furthermore, DMT maintains superior performance in complex and open environments, as demonstrated in Table 2 and Table 3c, indicating strong generalization capabilities. These results highlight DMT's ability to improve localization performance by utilizing more unlabeled data.
Table 3: Performance comparisons in existing issues (small objects localization and false positives) and complex scenarios (open set). The results of small objects and open set are tested on the VGG-ss dataset, while false positives are reported on the Flickr dataset.
Methods	Small Testset	Medium Testset
MSE↓	IoU↑	MSE↓	IoU↑
LVS [28]	0.515	0.021	0.441	0.265
EZVSL [9]	0.566	0.023	0.467	0.268
SLAVC [10]	0.705	0.021	0.568	0.220
Ours	0.160	0.025	0.174	0.335
(a) Small objects.
Methods	AP↑	max-F1↑	Acc↑
LVS [28]	9.80	17.90	19.60
DMC [2]	25.56	41.80	52.80
EZVSL [9]	46.30	54.60	66.40
SLAVC [10]	51.63	59.10	83.60
Ours	53.56	62.80	91.60
(b) False positives.
Methods	CIoU↑	AUC↑
LVS [28]	26.30	34.70
EZVSL [9]	39.57	39.60
SLAVC [10]	38.92	41.17
Ours	43.12	42.81
(c) Open set.
Significant Advancement in Small Subset Localization. We categorize objects based on their bounding box pixel area into small  $(1\sim 32^{2})$ , medium  $(32^{2}\sim 96^{2})$ , large  $(96^{2}\sim 144^{2})$  and huge  $(144^{2}\sim 224^{2})$ . We tested different methods on small and medium objects in the VGG-Sound dataset, focusing on the challenges of detecting small objects and reducing false positives mentioned earlier. The results in Table 3a show that DMT significantly improves performance, especially in terms of MSE metric. Despite some errors in the IoU metric, DMT still outperforms previous methods. The results in Figure 1 show that DMT accurately locates sounding objects with clear boundaries and precisely convergence to object contours, unlike previous methods that often have excessive or insufficient foreground regions, especially for small objects. These results demonstrate the effective and precise localization of small objects achieved by DMT. More experiments of different object sizes are in Appendix C.7.
Capability of Learning Rich Semantic Information. We present visualized predictions for testsets of varying sizes in Figure 1 (Left). It is evident that our approach achieves remarkable precision in localizing sounding sources. It accurately locates the position of sounding objects and precisely converges to their boundaries, while prior methods usually have excessive or insufficient foreground regions, particularly in the case of small objects.
Table 5: Main ablation study results. Models are trained on Flickr 144k and tested on Flickr-SoundNet testset, where 'S' and 'V' denote SoundNet and VGGish respectively.
Modules	Performance
# Teachers	Backbone	Filter	IPL	EMA	CIoU↑	AUC↑	MSE↓
One teacher	(a). S	X	X	X	80.20	53.57	0.458
(b). V	X	X	X	81.80	55.92	0.379
Dual teachers	(c). S+V	X	X	X	82.20	55.16	0.382
(d). S+V	X	X	✓	82.80	59.38	0.375
(e). S+V	X	✓	X	83.60	62.83	0.359
(f). S+V	✓	X	X	84.80	65.58	0.259
(g). S+V	✓	X	✓	86.20	66.56	0.260
(h). S+V	X	✓	✓	86.60	66.35	0.274
(i). S+V	✓	✓	X	88.60	66.68	0.260
(j). S+V	✓	✓	✓	90.40	69.36	0.237
It is worth noting that our method can even find out sounding objects overlooked in the manual annotations. For instance, in Figure 1 (Left, 4-th row), the heatmap reveals the presence of a piano, which is omitted in the manual annotation process. Furthermore, we assessed the model's capability to identify false positives, signifying instances where sounding objects are occasionally not visually observable within the image (off-screen), as shown in 3b. This reflects the ability of Dual Mean-Teacher to extract audio semantic information and effectively localize multiple sounding objects within a scene, a feat that eludes other methods. We attribute this capability to the semantic alignment of audio-visual features achieved through the pre-trained VGGish and SoundNet backbone.
Capacity for Cross-Domain Generalization and Multi-Source Localization. We tested DMT's generalization across different domains and its ability to localize multiple objects. Models trained using VGG-ss 144k were directly evaluated on MUSIC-solo [37], MUSIC-duet [37], and MUSIC-synthetic datasets [31, 52]. Figure 3 demonstrates DMT's strong generalization performance in the music domain, outperforming other method. As shown in Figure 3, the previous method struggles to accurately localize multi
ple sounding objects, either missing them or including all sounding objects within a large foreground area. In contrast, DMT localizes each instrument accurately and separately. However, without category information for fine-grained training, it leads to sub-optimal performance in differentiating between multiple active and silent instruments. There is still significant room for improvement with multiple sounding objects and we plan to address this issue in future work.
Figure 3: Performance on music-domain.
5.3 Extensions of Dual Mean-Teacher
We replicated several existing methods and integrated them into our framework. Notably, the integration of the Dual Mean-Teacher showcases its ability to significantly enhance the performance of other existing methods. In Table 4, one can observe a noteworthy improvement in the CIoU of EZVSL from  $62.65\%$  to  $87.20\%$ , and SLAVC rising from  $66.80\%$  to  $88.80\%$ , which further reinforces the efficacy of our framework and highlight its flexible extensibility.
Table 4: Extension results of DMT with various audio backbones, with 'R', 'V' and 'S' denoting ResNet, VGGish and SoundNet.
Methods	Backbones	CIoU↑	AUC↑	MSE↓
EZVSL w/o DMT	R	62.65	54.89	0.428
EZVSL w/ DMT	R+V	85.30	65.80	0.312
EZVSL w/ DMT	R+S	85.95	66.12	0.298
EZVSL w/ DMT	V+S	87.20	67.74	0.256
SLAVC w/o DMT	R	66.80	56.30	0.386
SLAVC w/ DMT	R+V	86.10	66.24	0.288
SLAVC w/ DMT	R+S	86.30	66.58	0.283
SLAVC w/ DMT	V+S	88.80	68.69	0.247
What is the individual contribution of each module to the performance gains? In this section, we progressively analyze the performance gain from each module in detail. We choose a self-supervised approach as our baseline. Table 5 presents the results on the Flickr 144k training set with  $10\%$  annotated samples.
(i). Initially, we apply it under the semi-supervised pseudo-labeling mechanism, using only one backbone, as shown in Table 5(a) and Table 5(b). The localization performance improves with annotated data supervision, but the gain is limited due to poor-quality pseudo-labels.
(ii) Next, we extend it to a two-backbone architecture and sequentially introduce the filter, IPL, and EMA modules. The results demonstrate that all three modules contribute to performance improvement  $(3\%, 1.8\%, 1\%)$ . Notably, Filter shows the most significant impact on the model's improvement, for it effectively rejects noisy samples, ensuring the stability of the model.
(iii) Finally, we can observe that optimal performance is achieved through the joint integration of the three modules by effectively suppressing confirmation bias.
How does annotation help localization? We aim to demonstrate that even with extremely limited labeled data, significant performance can still be achieved. To this end, we investigated the performance of our model from  $0.5\%$  to  $10\%$ , and report the results in Table 6. Our model consistently outperforms state-of-the-art approaches across all ratios. Furthermore, with a constant amount of unlabeled data, as the proportion of labeled data increases, our model's performance continues to improve, highlighting the significant impact of labeled data.
We investigated the impact of varying amounts of unlabeled data while keeping labeled data constant. Experimental results in Table 6 show that increasing the amount of unlabeled data improves localization performance, which seems contradictory to the previous conclusion about the proportion of labeled data, but actually, it demonstrates that labeled data can effectively leverage the unlabeled data. Based on our analysis, labeled data not only provides annotation information but also effectively enhances the power of unlabeled data, resulting in significant performance improvements.
Table 6: Performance on various labeled ratios % and multiple  $\times$  on Flickr 144k.
Labeled ratio %	CIoU	AUC
0.5% (200/40k)	84.80	63.58
1% (400/40k)	86.20	65.16
2% (800/40k)	87.20	65.94
5% (2k/40k)	87.60	67.44
10% (4k/40k)	88.40	68.12
Multiple ×	CIoU	AUC
2.5× (4k/10k)	88.00	67.80
5× (4k/20k)	88.20	67.91
10× (4k/40k)	88.40	68.12
20× (4k/80k)	89.20	68.44
40× (4k/200k)	91.20	71.36
Why can DMT outperform the existing semi-supervised AVSL method? Both naive SSL and DMT have utilized labeled and unlabeled data. However, a key distinction is that naive SSL employs unlabeled data only for contrastive loss, whereas DMT leverages pseudo-labels to incorporate unlabeled data into both contrastive loss and supervised loss, which amplifies the utilization of unlabeled data, thus enhancing generalization capability.
Data Utilization. We supplement the comparison experiments with fixed labeled data and an increase in unlabeled data from 10k to 200k, as shown in left part of 7. As the amount of unlabeled data increases, naive SSL exhibits only marginal improvement, whereas DMT shows more performance gains, indicating DMT can better use unlabeled data.
Generalization Ability. The right part of 7 highlights the limitations of naive SSL in the open set and in-the-wild datasets, suggesting that adding a supervised loss alone may lead to overfitting and weaken generalization. In contrast, DMT effectively leverages pseudo-label for improved generalization capability.
Table 7: Comparison of two SS-AVSL methods. * denotes the results from the original paper. 'sim-avsl' denotes the simple self-supervised AVSL model we use. We report the CIOU below.
2.5k/10k	2.5k/144k	2.5k/200k	open set	cross-datasets
attention10k + naive SSL	84.00*/83.68	84.40*/84.08	84.24	19.60	62.20
attention10k + DMT (ours)	88.00	89.52	90.40	42.64	87.26
sim-avsl + naive SSL	83.84	84.24	84.40	20.80	60.60
sim-avsl + DMT (ours)	88.24	89.76	91.12	43.10	89.80
Is it necessary to warm up dual teachers? We believe that the initialization of teachers and students is crucial, for the quality of pseudo-labels has a significant impact on model performance. To validate the effectiveness of this idea, we experimented to study the warm-up stage's impact on the model. We find that without the Warm-Up Stage, the model's improvement is very slow, and the performance eventually deteriorates. This indicates that without a good initialization, the model can
(a) Filter.
(b) IPL.
Figure 5: The effect of each component (Noise Filtering, IPL and EMA) in DMT to suppress confirmation bias, together with the number of filtered samples for pseudo labeling depicted in (d).
(c) Different modules.
(d) Filtered number.
accumulate errors, leading to confirmation bias issues. Therefore, we can conclude that Warm-Up is essential as it effectively suppresses confirmation bias in the early stages of training.
How to effectively mitigate confirmation bias in AVSL? In Section 4.3, we present the origins and mitigation strategies for confirmation bias in the localization task. In this section, we will demonstrate this. Figure 5a depicts the quality of pseudo labels before and after applying the filter module. It is evident that the quality of pseudo labels can be significantly improved after filtering. This observation highlights the effectiveness of the filter in eliminating noisy samples, which tend to be false-positive instances. Figure 5b depicts the comparison between using the direct outputs of each teacher as pseudo labels and utilizing their intersection, known as IPL. Where the purple dashed line represents the initialization value. The results
Figure 4: Effect of Warm-Up Stage.
clearly indicate that employing the direct outputs alone leads to the accumulation of bias, causing a deterioration in the quality of pseudo labels throughout the training process. Conversely, IPL consistently ensures the preservation of high-quality pseudo labels, thus mitigating the impact of bias.
Furthermore, Figure 5c visually presents the trend of model performance, revealing that the absence of any of these three modules results in a decline in model performance. However, we can see that EMA only affects the final performance of the model, and without the Filter module, the model's performance will be significantly affected by noise. Without the IPL module, the model will experience a continuous decline in performance due to erroneous estimation of pseudo-labels. Therefore, we find that the Noise Filtering module and IPL modules play a significant role in addressing the confirmation bias problem. Moreover, Figure 5d reflects that under the joint action of the three modules, DMT generates more accurate pseudo-labels and its performance continues to improve steadily.
In this paper, we advance the naive SS-AVSL work and propose a novel Semi-Supervised Audio-Visual Source Localization (SS-AVSL) framework, namely Dual Mean-Teacher (DMT), considering the importance of both limited annotated and abundant unlabeled data. From a unified perspective, existing self-supervised (weakly-supervised) AVSL methods could be referred to as a single student structure, while DMT employs dual teacher-student pairs to filter out noisy samples via the agreement of two teachers and generate high-quality pseudo labels to avoid confirmation bias. DMT has greatly enhanced AVSL performance and addressed intractable issues like false positives and inaccurate localization of tiny objects. Moreover, DMT is a learning paradigm and could be seamlessly incorporated into existing AVSL methods and consistently boost their performance.
We hope this work will bring more attention to SS-AVSL, provoke a reconsideration of pseudolabeling, bias avoidance, and better utilization of the underlying unlabeled data, and thus stimulate more semi-supervised learning research in this dense prediction task.
We would like to thank the National Natural Science Foundation of China under Grant 61773374 and the Major Basic Research Projects of Natural Science Foundation of Shandong Province under Grant ZR2019ZD07. We also appreciate Shuailei Ma, Kecheng Zheng, and Ziyi Wang for their valuable and insightful discussions.
[1] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[2] Hao Zhu, Man-Di Luo, Rui Wang, Ai-Hua Zheng, and Ran He. Deep audio-visual learning: A survey. International Journal of Automation and Computing, 18:351–376, 2021.
[3] Janani Ramaswamy. What makes the sound?: A dual-modality interacting network for audiovisual event localization. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4372-4376, 2020.
[4] Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. arXiv preprint arXiv:2008.09622, 2020.
[5] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15516–15525, June 2021.
[6] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds. arXiv preprint arXiv:2011.01143, 2020.
[7] Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R Hershey. Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation. In European Conference on Computer Vision, pages 368–385. Springer, 2022.
[8] Hanyu Xuan, Lei Luo, Zhenyu Zhang, Jian Yang, and Yan Yan. Discriminative cross-modality attention network for temporal inconsistent audio-visual event localization. IEEE Transactions on Image Processing, 30:7878-7888, 2021.
[9] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. arXiv preprint arXiv:2203.09324, 2022.
[10] Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. In Advances in Neural Information Processing Systems, 2022.
[11] Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[12] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3614-3633, 2021.
[13] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, pages 1-20, 2022.
[14] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358-4366, 2018.
[15] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound sources in visual scenes: Analysis and applications. TPAMI, 43(5):1605-1619, 2019.
[47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[49] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 iee international conference on acoustics, speech and signal processing (icassp), pages 131-135. IEEE, 2017.
[50] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in Neural Information Processing Systems, 2016.
[51] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[52] Di Hu, Yake Wei, Rui Qian, Weiyao Lin, Ruihua Song, and Ji-Rong Wen. Class-aware sounding objects localization via audiovisual correspondence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9844-9859, 2021.
Brief Introduction. The appendix is structured into four main sections: Algorithm, Experimental Settings, Supplementary Experiments, and Further Analysis. The main contents are as follows:
A. Algorithm: Pseudo-codes and algorithm details.
- B. Experimental Settings: More detailed description of the datasets, backbones, metrics formula, implementation details and baselines.
- C. Supplementary Experiments: Results of cross-dataset evaluation, comparison of different predicted map, exploration of Warm-Up speed and its influence on the final results, false-positive rejection capability of Noise Filtering, investigation of hyperparameters for Filter, IPL, and EMA, effect of data augmentation and quality analysis (visualization).
D. Further Analysis: Theoretical elaboration on the challenges faced by existing contrastive learning methods, and explanation of why contrastive learning alone cannot achieve precise localization.
A Algorithm
To make it more clear, Dual Mean-Teacher is specifically depicted in Algorithm 1.
Algorithm 1 Dual Mean-Teacher algorithm.
1: Input:  $\mathcal{D}_u = \{(a_i,v_i)\}$ ,  $\mathcal{D}_l = \{(v_i,a_i),\mathcal{G}_i\}$  {labeled data and unlabeled data.}  
2: while not reach the maximum iteration do  
3: for  $(a_i,v_i)$  in  $\mathcal{D}_u$  do  
4: while not reach the convergence of Warm-Up do  
5:  $\mathcal{L}_{\mathrm{Warm - Up}} = \mathbb{E}_{(a_i,v_i)\sim \mathcal{D}_l}H(\mathcal{G}_i,\mathcal{P}_i^t)$  {Supervised learning on labeled data.}  
6: end while  
7: Get the pseudo-labels  $\mathcal{M}_i^{t,A},\mathcal{M}_i^{t,B}$  from dual teachers  
8: if IoU( $\mathcal{M}_i^{t,A},\mathcal{M}_i^{t,B}$ )  $\geq \tau$  then  
9:  $\mathcal{IPL}(a_i,v_i) = \mathcal{M}_i^{t,A}\cdot \mathcal{M}_i^{t,B}$  {Compute Intersection of Pseudo-Labels (IPL).}  
10:  $\hat{\mathcal{G}}_i = \mathcal{IPL}(a_i,v_i)$  {Update the pseudo-label  $\hat{\mathcal{G}}_i$  of unlabeled data.}  
11: Add  $(a_i,v_i)$  to new dataset  $\mathcal{D}_u'$   
12: end if  
13: end for  
14:  $\mathcal{D}_{mix} = \mathcal{D}_l\cup \mathcal{D}_u'$  {Mix the filtered unlabeled data and labeled data.}  
15:  $\mathcal{L}_{\mathrm{full}} = (\mathcal{L}_{\mathrm{sup}}^A +\mathcal{L}_{\mathrm{sup}}^B) + \lambda_u(\mathcal{L}_{\mathrm{unsup}}^A +\mathcal{L}_{\mathrm{unsup}}^B)$ . {Students learning.}  
16:  $\theta_m^t\gets \beta \theta_{m - 1}^t +(1 - \beta)\theta_m^s$  {Students update teachers via EMA.}  
17: end while  
18: Return: Dual teachers and students model parameters.
NOTING TIPS:
Train. Warm-Up Stage is essentially a supervised learning. The performance gains of subsequent Unbiased-Learning Stage over Warm-Up Stage is actually the performance gains of our semi-supervised framework over vanilla supervised training on the same labelled dataset  $\mathcal{D}_l$ , which proves the validity of the proposed Dual Mean-Teacher, as shown in the main results in Table 1 and Table 2.
Inference. For the localization result of  $i_{th}$  audio-visual pair, we merge the outputs of the dual teachers to create a predicted map as below. Comparison of different predicted maps are described in C.2.
$$
\mathcal {P} _ {i} = \frac {1}{2} \left(\mathcal {P} _ {i} ^ {t, A} + \mathcal {P} _ {i} ^ {t, B}\right). \tag {15}
$$