SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation. The input image is resized to a resolution of  $224 \times 224$ . The input audio takes the log spectrograms extracted from 3s of audio at a sample rate of  $22050\mathrm{Hz}$ . Following previous works [9, 10], we apply STFT to generate an input tensor of size  $257 \times 300$  (257 frequency bands over 300 timesteps) using 50ms windows with a hop size of  $25\mathrm{ms}$ . We follow the prior audio-visual segmentation work [1] and use the ResNet50 [76] as the audio and visual encoder. The visual model is initialized using weights pre-trained on ImageNet [77]. The model is trained with the Adam optimizer with default hyper-parameters  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , and a learning rate of 1e-4. The model is trained for 20 epochs with a batch size of 64.
4.2 Comparison to Prior Work
In this work, we propose a novel and effective framework for weakly-supervised audio-visual segmentation. To validate the effectiveness of the proposed WS-AVS, we comprehensively compare it to previous audio-visual segmentation, weakly-supervised semantic segmentation, and visual sound source localization baselines: 1) AVS [1](ws): the weakly-supervised version of the first audio-visual segmentation work, where we removed the pixel-level annotations for training; 2) CAM [18]: the first baseline using class-activation maps as pseudo segmentation masks for weakly-supervised object localization; 3) EZ-VSL [9]: the state-of-the-art visual sound source localization approach with coarse source maps as output; 4)  $\mathrm{C}^2\mathrm{AM}$  [30]: the very recent work for weakly-supervised semantic segmentation with only the image as input.
The quantitative comparison results are reported in Table 1. As can be seen, we achieve the best performance in terms of all metrics compared to previous weakly-supervised baselines. In particular,
Figure 2: Qualitative comparisons with weakly-supervised audio-visual segmentation (AVS [1]), weakly-supervised semantic segmentation (CAM [18],  $\mathrm{C}^2\mathrm{AM}$  [30]), and visual sound source localization (EZ-VSL [9]) baselines on single-source segmentation. The proposed WS-AVS produces much more accurate and high-quality segmentation maps for sounding objects.
the proposed WS-AVS significantly outperforms AVS [1](ws), the weakly-supervised version of the state-of-the-art audio-visual segmentation approach, by 21.50 mIoU and 26.77 F-score. Moreover, we achieve superior performance gains of 3.26 mIoU and 15.21 F-score compared to  $C^2$ AM [30], which implies the importance of the proposed multi-scale multiple-instance contrastive learning for addressing the modality uncertainty in audio-visual fusion. Meanwhile, our WS-AVS outperforms EZ-VSL [9], the current state-of-the-art visual sound source localization baseline, by a large margin, where we achieve the performance gains of 4.73 mIoU and 16.06 F-score. These significant improvements demonstrate the superiority of our method in single-source audio-visual segmentation.
In addition, significant gains in multi-source sound localization can be observed in Table 1. Compared to AVS [1](ws), the weakly-supervised version of the state-of-the-art audio-visual segmentation approach, we achieve the results gains of  $22.09\mathrm{mIoU}$  and 31.15 F1 score. Furthermore, the proposed approach still outperforms  $\mathrm{C}^2\mathrm{AM}$  [30] by  $5.52\mathrm{mIoU}$  and 17.29 F1 score. We also achieve highly better results than EZ-VSL [9], the current state-of-the-art visual sound source localization baseline. These results validate the effectiveness of our approach in learning multi-scale multiple-instance semantics from mixtures and images for multi-source localization.
In order to qualitatively evaluate the localization maps, we compare the proposed WS-AVS with weakly-supervised AVS [1], CAM [18],  $\mathrm{C^2AM}$  [30], and EZ-VSL [9] on single-source segmentation in Figure 2. From comparisons, three main observations can be derived: 1) Without pixel-level annotations, AVS [1], the first audio-visual segmentation fails to predict the masks for sounding objects; 2) the quality of segmentation masks generated by our method is much better than the strong visual sound source baseline, EZ-VSL [9]; 3) the proposed WS-AVS achieves competitive even better results on predicted masks against the weakly-supervised semantic segmentation baseline [30] by using contrast class-agnostic maps for prediction. These visualizations further showcase the superiority of our simple WS-AVS with multi-scale multiple-instance contrastive learning to guide segmentation for predicting accurate source masks.
In this section, we performed ablation studies to demonstrate the benefit of introducing the Audio-Visual Fusion with multi-scale multiple-instance contrastive learning and Pseudo Mask Refinement module by contrastive class-agnostic map. We also conducted extensive experiments to explore the effect of fusion stages and batch size on weakly-supervised audio-visual segmentation. Furthermore,
Table 2: Ablation studies on Audio-Visual Fusion (AVF) and Pseudo Mask Refinement (PMR).
AVF	PMR	mIoU (↑)	F-score (↑)
X	X	12.63	24.99
✓	X	31.85 (+19.22)	39.78 (+14.79)
X	✓	32.46 (+19.83)	43.92 (+18.93)
✓	✓	34.13 (+21.50)	51.76 (+26.77)
Figure 3: Effect of fusion stages and batch size on mIoU and F-score.
we visualized high-quality pseudo masks from Pseudo Mask Refinement in weakly-supervised training.
Audio-Visual Fusion & Pseudo Mask Refinement. To validate the effectiveness of the introduced audio-visual fusion with multi-scale multiple-instance contrastive learning (AVF) and Pseudo Mask Refinement by contrastive class-agnostic map (PMR), we ablate the necessity of each module and report the quantitative results in Table 2. We can observe that adding AVF to the vanilla baseline highly increases the results of single-source audio-visual segmentation by  $19.22\mathrm{mIoU}$  and 14.79 F-score, which demonstrates the benefit of multi-scale multiple-instance contrastive learning  $(\mathbf{M}^2\mathbf{ICL})$  in extracting aligned cross-modal feature in fusion for source segmentation. Meanwhile, introducing only PMR in the baseline also increases the segmentation performance in terms of all metrics. More importantly, incorporating AVF with  $\mathbf{M}^2\mathbf{ICL}$  and PMR together into the baseline significantly raises the performance by  $21.50\mathrm{mIoU}$  and 26.77 F-score. These improving results validate the importance of AVF with  $\mathbf{M}^2\mathbf{ICL}$  and PMR with contrastive class-agnostic map in addressing both modality and spatial challenges for generating accurate audio-visua masks.
Effect of Fusion Stages and Batch Size. The number of fusion stages and batch size used in the proposed  $\mathbf{M}^2\mathrm{ICL}$  affect the extracted cross-modal representations for audio-visual segmentation. To explore such effects more comprehensively, we varied the number of fusion stages from  $\{1,2,3,4\}$  and ablated the batch size from  $\{8,16,32,64,128\}$ . The comparison results of segmentation performance are shown in Figure 3. When the number of fusion stages is 4 and using batch size of 64 in  $\mathbf{M}^2\mathrm{ICL}$ , we achieve the best segmentation performance in terms of all metrics. With the increase of fusion stages from 1 to 4, the proposed WS-AVS consistently raises results, which shows the importance of multi-scale visual features in audio-visual fusion for learning discriminative cross-modal representations. Regarding the batch size, the performance of the proposed WS-AVS climbs with the increase of the batch size from 8 to 64. However, increasing the batch size from 64 to 128 will not continually improve the result since there might be some false negatives in the mini-batch for this training set with a relatively smaller size.
Generated Pseudo Mask. Generating reliable pseudo masks with contrast class-agnostic maps is critical for us to train the weakly-supervised framework. To better evaluate the quality of generated pseudo masks, we visualize the pseudo mask and class-agnostic map in Figure 4. As can be observed in the last column, the class-agnostic map can successfully localize the sounding object in the image. Furthermore, the pseudo masks in the third column predicted by Eq. 6 have high quality and they are even fine-grained compared to the manually annotated ground-truth masks. These meaningful visualization results further showcase the success of the Pseudo Mask Refinement in extracting reliable pseudo masks as guidance for single-source audio-visual segmentation.
Figure 4: Qualitative comparisons of the ground-truth mask, pseudo mask, and class-agnostic map. Note that the generated pseudo mask is used for weakly-supervised training.
4.4 Limitation
Although the proposed WS-AVS achieves superior results on single-source audio-visual segmentation, the performance gains of our approach on the mIoU metric are not significant. One possible reason is that our model easily overfits across the training phase, and the solution is to incorporate dropout and momentum encoders together for weakly-supervised audio-visual segmentation. Meanwhile, we notice that our model performs worse on mixed sound sources, such as a scenario with both female singing and playing tabla. The future work could be to add separation objectives or assign semantic labels to both audio and visual segments in audible videos.
In this work, we successfully investigate the weakly-supervised audio-visual segmentation to eliminate pixel-level annotations as supervision. Then we present WS-AVS, a novel Weakly-Supervised Audio-Visual Segmentation framework with multi-scale multiple-instance contrastive learning for capturing multi-scale audio-visual alignment. Furthermore, we conduct extensive experiments on AVSBench to demonstrate the superiority of our WS-AVS against previous weakly-supervised audio-visual segmentation, weakly-supervised semantic segmentation, and visual sound source localization approaches.
Broader Impact. The proposed approach successfully predicts segmentation masks of sounding sources from manually-collected datasets on the web, which might cause the model to learn internal biases in the data. For instance, the model could fail to discover rare but crucial sound sources. Therefore, these issues should be addressed for the deployment of real applications.
[1] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In European Conference on Computer Vision, 2022. 1, 2, 3, 4, 5, 6, 7
[2] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4358-4366, 2018. 1, 2
[3] Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh H. McDermott, and Antonio Torralba. Self-supervised audio-visual co-segmentation. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2357–2361, 2019. 1, 2
[4] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9248-9257, 2019. 1, 2
[71] Prem Seetharaman, Gordon Wichern, Shrikant Venkataramani, and Jonathan Le Roux. Class-conditional embeddings for music source separation. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 301–305, 2019. 3
[72] Fatemeh Pishdadian, Gordon Wichern, and Jonathan Le Roux. Finding strength in weakness: Learning to separate sounds with weak supervision. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2386-2399, 2019. 3
[73] Efthymios Tzinis, Scott Wisdom, John R. Hershey, Aren Jansen, and Daniel P. W. Ellis. Improving universal sound separation using sound classification. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 96-100, 2020. 3
[74] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollar. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6399-6408, 2019. 4, 5
[75] Jiang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Jiashi Feng, and Jianmin Jiang. A simple pooling-based design for real-time salient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3917-3926, 2019. 5
[76] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. 6
[77] Jia Deng, Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. 6