SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxin Guo $^{1,2,3}$ , Shijie Ma $^{1,2}$ , Hu Su $^{1,2}$ , Zhiqing Wang $^{1,2}$ , Yuhao Zhao $^{1,2}$ , Wei Zou $^{1,2*}$ , Siyang Sun $^{3}$ , Yun Zheng $^{3}$
$^{1}$ School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China  
 $^{2}$ State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation of Chinese Academy of Sciences, Beijing, China
$^{3}$ DAMO Academy, Alibaba Group {guoyuxin2021, wei.zou}@ia.ac.cn
Audio-Visual Source Localization (AVSL) aims to locate sounding objects within video frames given the paired audio clips. Existing methods predominantly rely on self-supervised contrastive learning of audio-visual correspondence. Without any bounding-box annotations, they struggle to achieve precise localization, especially for small objects, and suffer from blurry boundaries and false positives. Moreover, the naive semi-supervised method is poor in fully leveraging the information of abundant unlabeled data. In this paper, we propose a novel semi-supervised learning framework for AVSL, namely Dual Mean-Teacher (DMT), comprising two teacher-student structures to circumvent the confirmation bias issue. Specifically, two teachers, pre-trained on limited labeled data, are employed to filter out noisy samples via the consensus between their predictions, and then generate high-quality pseudo-labels by intersecting their confidence maps. The sufficient utilization of both labeled and unlabeled data and the proposed unbiased framework enable DMT to outperform current state-of-the-art methods by a large margin, with CIoU of  $90.4\%$  and  $48.8\%$  on Flickr-SoundNet and VGG-Sound Source, obtaining  $8.9\%$ ,  $9.6\%$  and  $4.6\%$ ,  $6.4\%$  improvements over self- and semi-supervised methods respectively, given only  $< 3\%$  positional-annotations. We also extend our framework to some existing AVSL methods and consistently boost their performance. Our code is available at https://github.com/gyx-gloria/DMT.
Visual and auditory perception is crucial for observing the world. When we hear a sound, our brain will extract semantic information and locate the sounding source. In this work, we focus on Audio-Visual Source Localization (AVSL) [1, 2], with the purpose of accurately locating sounding objects in frames based on their paired audio clips. Beyond this scope, AVSL also plays a crucial role in many downstream tasks including environmental perception [3], navigation [4, 5], sound separation [6, 7] and event localization [8]. Therefore, accurate localization is of utmost importance.
In the literature of AVSL [9, 10, 11], the conventional paradigm is to employ self-supervised contrastive learning based on audio-visual correspondence. However, most of them suffer from some serious challenges. From the performance perspective, there are issues such as blurry boundaries, inability to converge to specific objects, and the predicted sounding regions that are too large to accurately locate objects, especially small objects. In terms of the learning stage, a single model alone is unable to recognize and filter out false positives, i.e., noisy samples with no visible sounding sources, which could affect the entire learning process of the model.
Figure 1: Comparison of existing Audio-Visual Source Localization (AVSL) methods and the proposed Dual Mean-Teacher (DMT). Left: DMT has greatly addressed severe issues including inaccurate small object localization, blurry boundaries, and instability. Right: DMT outperforms previous by a large margin on Flickr and VGG-ss datasets.
In essence, AVSL is a dense prediction task, which can not be directly accomplished from a shared global image representation [12], requiring models to capture fine local features in order to accurately predict object locations, i.e., achieving precise pixel-level localization is not feasible without positional annotations. Unluckily, the number of samples with location labels is extremely limited. As a result, we resort to Semi-Supervised Learning [13] (SSL) to fully leverage the labeled data.
Considering that self-supervised AVSL is not fully learnable, Attention10k [14, 15] extended the self-supervised model to an SSL model by directly appending a supervised loss on labeled data, which is the first semi-supervised attempt in the field. Nevertheless, simply leveraging labeled data might lead to overfitting and neglect to fully harness the underlying unlabeled data. Given these issues, we resort to pseudo-labeling [16]. However, directly introducing pseudo-labeling could lead to confirmation bias [17] which cannot be adequately rectified by a single model.
To tackle these challenges, we break away from traditional self-supervised learning and propose a more sophisticated Semi-Supervised Audio-Visual Source Localization (SS-AVSL) framework, called Dual Mean-Teacher (DMT), which adopts a double teacher-student structure in a two-stage training manner. We consider previous AVSL methods as a single student unit. To fully leverage positional annotations and training data, we extend it to a classic semi-supervised framework MeanTeacher [18]. To address the issue of confirmation bias, we expand it into a dual independent teacher-student structure with designed modules of Noise Filtering, Intersections of Pseudo-Labels (IPL), as shown in Figure 2. Specifically, teachers are pre-trained on a limited amount of labeled data in Warm-Up stage, establishing a solid foundation, in the subsequent Unbiased-Learning Stage, dual teachers filter out noisy samples and rectify pseudo-labels. In more detail, the Noise Filtering module effectively rejects noise samples by leveraging consensus, i.e., agreement, between dual teachers, ensuring high-quality training data, then IPL module generates precise pseudo-labels by intersecting the predictions from both teachers. DMT eliminates the influence of confirmation bias by rejecting noisy samples and improving the quality of pseudo-labels, which effectively tackles the issues of false positives and greatly improves localization performance.
In summary, our method contributes to the following three aspects. Firstly, we introduce a novel unbiased framework based on a pseudo-labeling mechanism for semi-supervised AVSL, which could maximize the utilization of both labeled and unlabeled data, effectively address the challenge of limited annotated data, and mitigate the issue of confirmation bias. Moreover, compared to existing approaches, DMT achieves much remarkable localization performance, with better small object localization and stronger generalization capability, which significantly elevate the performance of current methods. Finally, DMT can be summarized as a semi-supervised learning paradigm and could be combined with existing (weakly-supervised) AVSL methods to consistently boost their performance.
Semi-Supervised Learning. Semi-Supervised Learning (SSL) [13, 19] leverages a small amount of labeled data to unlock the potential of unlabeled data for better model learning. One line of work relies on consistency regularization [18, 20, 21] to encourage models to behave similarly under different perturbations. An orthogonal idea is to generate high-quality pseudo-labels [16, 22] on
unlabeled data to retrain models for better performance. The quality of pseudo-labels is crucial. Current methods [23, 24, 25] combine the above two paradigms to achieve remarkable results.
Audio-Visual Source Localization. The key to Audio-Visual Sound Localization (AVSL) endeavors to establish the correspondence between visual objects and their corresponding sounds by contrastive learning [26, 27]. Most existing methods predominantly utilize self-supervised or weakly-supervised approaches (for all of them employ pre-trained backbone). Some classical works, such as Attention10k [14, 15], DMC [2], LVS [28], EZVSL [9], SSPL [29], SSL-TIE [30] achieve improving performance over time. Other methods like DSOL [31], CoarsetoFine [32], mix and localize [33], and AVGN [11] pay attention to multi-source localization. In addition, some studies also address the issue of false positives and false negatives in self-supervised learning. For example, SLAVC [10] focuses on false positives and effectively overcomes the overfitting issue. IER [34] proposes a label-free method that targets the suppression of non-sounding objects, while Robust [35] considers both false positive and false negative issues. AVID [36] detects false negatives by defining sets of positive and negative samples via cross-modal agreement. Source separation [37, 38] and generative models [39] also achieve good results. However, most AVSL methods exhibit subpar performance in the absence of annotation.
Semi-Supervised Learning in Localization. Semi-Supervised Object Detection (SSOD) is one of the few applications of SSL in the localization field. The majority of SSOD methods, such as [40, 41, 42], utilize pseudo-labeling to enhance the localization performance. Moreover, some works like [43, 44] focus on the confirmation bias in SSOD. Similar to object detection, AVSL is a pixelwise dense prediction task, heavily reliant on high-quality pseudo-labels. Attention10k [14, 15] is the first SS-AVSL work. It extends a self-supervised model to a semi-supervised framework by simply adding a supervised loss, aiming at fixing the false conclusions generated by weakly-supervised methods. However, this naive method may lead to overfitting and neglects the full utilization of unlabeled data. In contrast, we introduce a novel SS-AVSL framework based on pseudo-label mechanism, which can address confirmation bias and maximize the utilization of both labeled and unlabeled data, to achieve stronger localization performance.
Problem Definition. Audio-Visual Source Localization (AVSL) aims to accurately locate the sound source within a given visual scene. We denote audio-visual pairs as  $(a_i, v_i)$ , where  $a_i$  and  $v_i$  represent the audio and visual modality, respectively. The objective is to generate a pixel-wise confidence map  $\mathcal{P}$  indicating the location of the sound source.
Contrastive Learning in AVSL. Self-supervised AVSL methods commonly leverage audio-visual correspondence to maximize the similarity between frames and their corresponding audio clips (positive pairs) while minimizing the similarity among unpaired ones (negative pairs):
$$
\mathcal {L} _ {\text {u n s u p}} = - \mathbb {E} _ {\left(a _ {i}, v _ {i}\right) \sim \mathcal {D} _ {u}} \left[ \log \frac {\exp \left(s \left(g \left(a _ {i}\right) , f \left(v _ {i}\right)\right) / \tau_ {t}\right)}{\sum_ {j = 1} ^ {n} \exp \left(s \left(g \left(a _ {i}\right) , f \left(v _ {j}\right)\right) / \tau_ {t}\right)} + \log \frac {\exp \left(s \left(f \left(v _ {i}\right) , g \left(a _ {i}\right)\right) / \tau_ {t}\right)}{\sum_ {j = 1} ^ {n} \exp \left(s \left(f \left(v _ {i}\right) , g \left(a _ {j}\right)\right) / \tau_ {t}\right)} \right].
$$
where  $\mathcal{D}_u$  are unlabeled datasets,  $g(\cdot)$  and  $f(\cdot)$  are audio and visual feature extractors.  $\tau_t$  is the temperature coefficient.  $s(\cdot)$  is consistency matching criterion. The predicted map  $\mathcal{P}_i$  is typically calculated with cosine similarity  $\sin (\cdot)$  to represent the confidence of the presence of sounding objects:
$$
\mathcal {P} _ {i} = \operatorname {s i m} \left(g \left(a _ {i}\right), f \left(v _ {i}\right)\right) = \frac {\langle g \left(a _ {i}\right) , f \left(v _ {i}\right) \rangle}{\| g \left(a _ {i}\right) \| \cdot \| f \left(v _ {i}\right) \|}. \tag {2}
$$
Learning with (Pseudo) Labels in AVSL. When labeled data are available, one could apply supervised loss directly to learn to localize:
$$
\mathcal {L} _ {\sup } = \mathbb {E} _ {i \sim \mathcal {D}} H \left(\mathcal {G} _ {i}, \mathcal {P} _ {i}\right). \tag {3}
$$
where  $\mathcal{G}_i$  could be the ground truth or generated pseudo-labels, both  $\mathcal{G}_i$  and  $\mathcal{P}_i$  are in the form of binary confidence map.  $H(\cdot ,\cdot)$  is cross-entropy function across the two-dimensional spatial axes.
Figure 2: Overview of the proposed Dual Mean-Teacher framework. Left: Overall learning process of dual teacher-student pairs, two students are guided by both ground-truth labeled data and filtered unlabeled data with the Intersection of Pseudo-Labels (IPL). Upper-right: Details of Noise Filtering and IPL. Dual teachers reject noise samples based on their consensus and generate pseudo-labels on filtered data. Lower-right: Details of AVSL pipeline. Students are learned through contrastive learning and predict confidence maps for supervised learning with (pseudo) labels.
4 Dual Mean-Teacher
Overview. In this section, we mainly describe Dual Mean-Teacher (DMT) in the order of the learning process. Specifically, in the Warm-Up Stage (Section 4.1), two teachers are pre-trained on bounding-box annotated data to obtain a stable initialization. In the subsequent Unbiased-Learning Stage (Section 4.2), the Noise Filtering module and Intersection of Pseudo-Label (IPL) module collectively filter out noisy samples and generate high-quality pseudo-labels by two teachers to guide students' training, teachers are in turn updated by exponential moving average (EMA) of students.
From a unified perspective, existing AVSL methods can be viewed as a single student, which is later expanded into the semi-supervised classical framework Mean-Teacher [18], in order to fully utilize limited labeled data. To effectively address the confirmation bias issue, we further extend it to a double teacher-student framework, as shown in Figure 2. DMT adopts a teacher-student architecture, where each teacher and student contains two independent AVSL pipelines for audio-visual contrastive learning and generating localization results. Teachers provide students with stable unlabeled samples after Noise Filtering and their generated IPL. Students pass back the parameters to the teachers.
General Notations. Cross-entropy function  $H(\cdot, \cdot)$  and two feature extractors  $f(\cdot)$  and  $g(\cdot)$  are already discussed in Section 3. By default, subscript  $i$  indicates the  $i$ -th sample, while superscript  $A$ ,  $B$  denote two teacher-student pairs with  $t$  and  $s$  indicating teacher and student, respectively.  $\mathcal{D}_l$  and  $\mathcal{D}_u$  are labeled and unlabeled datasets.  $\mathcal{G}_i$  and  $\mathcal{P}_i$  are ground-truth and predicted confidence maps, both with size of  $H \times W$ . We apply strong  $\mathcal{A}(\cdot)$  or weak  $\alpha(\cdot)$  augmentation on visual inputs.
4.1 Warm-Up Stage
The quality of pseudo-labels is crucial to SSL, especially for localization tasks. Before generating pseudo-labels, we first pre-train dual teachers with bounding-box annotated data to achieve preliminary localization performance. In order to avoid overfitting, we apply strong augmentation and get augmented labeled dataset  $\mathcal{D}_l = \{(a_i,\mathcal{A}(v_i)),\mathcal{G}_i\}$ . After extracting visual features  $f^{t}(\mathcal{A}(v_{i}))$  and auditory features  $g^{t}(a_{i})$ , the predicted map  $\mathcal{P}_i^t$  can be obtained by Eq. (2). Then we utilize bounding-box annotations  $\mathcal{G}_i$  as supervision:
$$
\mathcal {L} _ {\text {W a r m - U p}} = \mathbb {E} _ {\left(a _ {i}, v _ {i}\right) \sim \mathcal {D} _ {l}} H \left(\mathcal {G} _ {i}, \mathcal {P} _ {i} ^ {t}\right). \tag {4}
$$
4.2 Unbiased-Learning Stage
Noise Filtering. To mitigate confirmation bias, it is crucial to filter out noisy samples that are more likely to be false positives. As depicted in Figure 2, two predicted maps of the same sample are generated by dual teachers. It is clear that samples with higher reliability can be identified when the two predicted maps exhibit higher similarity, i.e., there is more agreement and consensus between dual teachers, then the sample is reserved for pseudo-labelling. Conversely, when there is a significant discrepancy between the two maps, the sample will be considered as a false positive, such as frames without distinguishable sound objects or sounds that cannot be accurately represented by a bounding box (e.g., wind sounds), such samples are rejected and discarded.
Intersection of Pseudo-Labels (IPL). By intersecting the foreground regions of two predicted maps on the filtered samples, one can generate positional pseudo-labels, named Intersection of Pseudo-Labels (IPL), to guide students' learning. With the pre-defined foreground threshold  $\delta$ , two predicted maps  $\mathcal{P}_i^{t,A}, \mathcal{P}_i^{t,B}$  could be transferred to binary maps  $\mathcal{M}_i^{t,A}$  and  $\mathcal{M}_i^{t,B}$ . Weak augmentation  $\alpha(\cdot)$  is employed for teachers to generate high-quality pseudo-labels:
$$
\mathcal {P} _ {i} ^ {t} = \operatorname {s i m} \left(g ^ {t} \left(a _ {i}\right), f ^ {t} \left(\alpha \left(v _ {i}\right)\right)\right), \tag {5}
$$
$$
\mathcal {M} _ {i} ^ {t} = \mathbb {1} \left(\mathcal {P} _ {i} ^ {t} \geq \delta\right). \tag {6}
$$
We adopt the Intersection over Union (IoU) metric to quantify the similarity between the two maps  $\mathcal{M}_i^{t,A}$ ,  $\mathcal{M}_i^{t,B}$ . If the IoU score exceeds the threshold  $\tau$ , the sample will be accepted, and the intersection of those two maps will be generated as its pseudo-label (IPL). Otherwise, it will be filtered out as a noise sample.
$$
\mathcal {D} _ {u} ^ {\prime} = \left\{\left(a _ {i}, v _ {i}\right) \mid \operatorname {I o U} \left(\mathcal {M} _ {i} ^ {t, A}, \mathcal {M} _ {i} ^ {t, B}\right) \geq \tau , \forall \left(a _ {i}, v _ {i}\right) \in \mathcal {D} _ {u} \right\}. \tag {7}
$$
$$
\mathcal {I P L} \left(a _ {i}, v _ {i}\right) = \mathcal {M} _ {i} ^ {t, A} \cdot \mathcal {M} _ {i} ^ {t, B}. \tag {8}
$$
The newly selected unlabeled dataset is applied to the student model along with the corresponding high-quality IPL.
Students Learning without bias. To suppress confirmation bias more effectively, we mix labeled and new unlabeled datasets. Both ground-truth annotations and high-quality IPL are employed to train the student models:
$$
\mathcal {D} _ {m i x} = \mathcal {D} _ {l} \cup \mathcal {D} _ {u} ^ {\prime} = \left\{\left(a _ {i}, v _ {i}\right), \widehat {\mathcal {G}} _ {i} \right\}, \text {w h e r e} \widehat {\mathcal {G}} _ {i} = \left\{ \begin{array}{c c} \mathcal {G} _ {i} & \text {i f} \left(a _ {i}, v _ {i}\right) \in \mathcal {D} _ {l} \\ \mathcal {I P L} \left(a _ {i}, v _ {i}\right) & \text {i f} \left(a _ {i}, v _ {i}\right) \in \mathcal {D} _ {u} ^ {\prime} \end{array} \right. \tag {9}
$$
In addition, we incorporate consistency regularization [45] in the semi-supervised learning process. Specifically, for a given sample, we obtain IPL from the teachers on weakly augmented images while strong augmentations are applied for samples of students. By enforcing consistency between IPL and students' predictions, DMT could be more stable with better generalization ability.
$$
\mathcal {P} _ {i} ^ {s} = \operatorname {s i m} \left(g ^ {s} \left(a _ {i}\right), f ^ {s} \left(\mathcal {A} \left(v _ {i}\right)\right)\right), \tag {10}
$$
$$
\mathcal {L} _ {\sup } = \mathbb {E} _ {i \sim \mathcal {D} _ {m i x}} H \left(\widehat {\mathcal {G}} _ {i}, \mathcal {P} _ {i} ^ {s}\right). \tag {11}
$$
Similar to the AVSL method mentioned in Section 3, students are also trained by audio-visual correspondence of contrastive learning loss. Here, we introduce an attention module to add attention to the sounding region in the frame:
$$
f _ {a t t} (v _ {i}) = \frac {\exp (\mathcal {P} _ {i} (x , y))}{\sum_ {x , y} \exp (\mathcal {P} _ {i} (x , y))} \cdot f (v _ {i}). \tag {12}
$$
Then, the full semi-supervised loss could be derived with  $\mathcal{L}_{\mathrm{sup}}$  (see Eq. (11)) on  $\mathcal{D}_{mix}$  and  $\mathcal{L}_{\mathrm{unsup}}$  (see Eq. (1)) on  $\mathcal{D}_u$ :
$$
\mathcal {L} _ {\text {f u l l}} = \left(\mathcal {L} _ {\sup } ^ {A} + \mathcal {L} _ {\sup } ^ {B}\right) + \lambda_ {u} \left(\mathcal {L} _ {\operatorname {u n s u p}} ^ {A} + \mathcal {L} _ {\operatorname {u n s u p}} ^ {B}\right). \tag {13}
$$
Update of Students and Teachers. Students are updated via gradient descent of  $\mathcal{L}_{\mathrm{full}}$  , while dual teachers are updated through the exponential moving average (EMA) of corresponding students:
$$
\theta_ {m} ^ {s} \leftarrow \theta_ {m - 1} ^ {s} - \gamma \frac {\partial \mathcal {L} _ {\text {f u l l}}}{\partial \theta_ {m - 1} ^ {s}}, \quad \theta_ {m} ^ {t} \leftarrow \beta \theta_ {m - 1} ^ {t} + (1 - \beta) \theta_ {m} ^ {s}. \tag {14}
$$
The slowly progressing teachers can be regarded as the ensemble of students in recent training iterations, which enables stable progress in training.
4.3 Unbiased Superiority of Dual Mean-Teacher
For dense prediction tasks such as AVSL, employing pseudo-labels for model training can easily accumulate errors and lead to sub-optimal results. In our DMT framework, The unbiased characteristics could be attributed to the following three factors: (i). Noise Filtering ensures that only stable samples are utilized to train. (ii). IPL generates high-quality pseudo-labels. (iii). Pre-train dual teachers on bounding-box annotated data with strong augmentation in Warm-Up Stage. The above conclusion will be validated in subsequent ablation studies in Section 5.4.
Table 1: Comparison results on Flickr-SoundNet. Models are trained on Flickr 10k and 144k.  $\dagger$  indicates our reproduced results, others are borrowed from original papers. Attention10k-SSL is of 2k labeled data supervision. We report the proposed DMT results from both stages as stage-2 (stage-1).  $|\mathcal{D}_l|$  denotes the number of labeled data.
Methods	Flickr 10k	Flickr 144k
CIoU	AUC	CIoU	AUC
Attention10k [14, 15]	43.60	44.90	66.00	55.80
CoarsetoFine [32]	52.20	49.60	-	-
DMC [2]	-	-	67.10	56.80
LVS [28]	58.20	52.50	69.90	57.30
EZVSL [9]	62.65	54.89	72.69	58.70
SLAVC†[10]	66.80	56.30	73.84	58.98
SSPL [29]	74.30	58.70	75.90	61.00
SSL-TIE†[30]	75.50	58.80	81.50	61.10
Attention10k-SSL [14, 15]	82.40	61.40	83.80	61.72
Ours (|Dl|=256)	87.20 (84.40)	65.77 (59.60)	87.60 (84.40)	66.28 (59.60)
Ours (|Dl|=2k)	87.80 (85.60)	66.20 (63.18)	88.20 (85.60)	66.63 (63.18)
Ours (|Dl|=4k)	88.80 (86.20)	67.81 (65.56)	90.40 (86.20)	69.36 (65.56)
With limited annotated data, DMT could significantly raise the performance of AVSL and address the severe issues, e.g., false positives and poor localization ability on small objects. Then, we direct our focus towards answering the following questions with ablation experiments 5.4:
- What is the individual contribution of each module to the performance gains?
- How does annotation enhance localization performance?
- Why can DMT outperform the existing semi-supervised AVSL method?
- Is it necessary to warm up dual teachers?
- How to effectively mitigate confirmation bias in AVSL?
Datasets. We conduct experiments on two large-scale audio-visual datasets: Flickr-SoundNet [14, 15] and VGG Sound Source [46], where there are 5,000 and 5,158 bounding-box annotated samples, respectively. For labeled data, we randomly select 4,250 for training, 500 for validating, and keep the same test sets with 250 samples as previous works [9, 10, 28, 30]. Moreover, we select a subset of 10k and 144k unlabeled samples to train as before. Details can be found in Appendix B.1.
Audio and Visual Backbones. For visual backbones, we followed prior work and used ResNet-18 [47] pre-trained on ImageNet [48]. For audio backbones, we select the pre-trained VGGish [49] and SoundNet [50] with semantic audio information. Further details can be found in Appendix B.2.
Metrics. We report the Consensus Intersection over Union (CIoU) and Area Under Curve (AUC), following previous settings [14, 15]. CIoU represents the localization accuracy, samples with IoU above the threshold  $\delta = 0.5$  are considered to be accurately located. Considering small objects, we introduce Mean Square Error (MSE), which measures the average pixel-wise difference between two maps without binarization, making it more suitable for evaluating dense prediction tasks on small objects. More details are shown in Appendix B.3.