SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yingying Fan, Yu Wu, Bo Du, Yutian Lin*
School of Computer Science, Hubei Luojia Laboratory, Wuhan University
{fanyingying_CS, wuyucs, dubo, yutian.lin}@whu.edu.cn
We focus on the weakly-supervised audio-visual video parsing task (AVVP), which aims to identify and locate all the events in audio and visual modalities. Previous works only concentrate on video-level overall label denoising across modalities, but overlook the segment-level label noise, where adjacent video segments (i.e., 1-second video clips) may contain different events. However, recognizing events in the segment is challenging because its label could be any combination of events that occur in the video. To address this issue, we consider tackling AVVP from the language perspective, since language could freely describe how various events appear in each segment beyond fixed labels. Specifically, we design language prompts to describe all cases of event appearance for each video. Then, the similarity between language prompts and segments is calculated, where the event of the most similar prompt is regarded as the segment-level label. In addition, to deal with the unlabeled segments, we propose to perform dynamic re-weighting on the unreliable segments to adjust their labels. Experiments show that our simple yet effective approach outperforms state-of-the-art methods by a large margin. Code and data are available at https://github.com/fyyCS/LSLD.
Classic audio-visual tasks such as audio-visual event localization and audio-visual representation learning have been well explored in previous works [1, 2, 3] and have achieved promising performance. However, most existing research studies on these tasks assume that audio and visual are correlated and temporally aligned, which is not always the case. For example, a person can hear a dog barking but not see it in the video. To this end, we focus on the more general audio-visual video parsing (AVVP) task, which aims to recognize all events in each modality and localize their temporal boundaries.
Since the full annotation process is time-consuming and expensive, Tian et al. [4] address AVVP in a weakly-supervised manner, which only requires video-level labels, without specifying the modality and temporal boundaries. They develop a hybrid attention network to leverage unimodal and cross-modal information and propose a simple Multimodal Multiple Instance Learning (MMIL) framework to aggregate segment-level predictions into video-level ones. However, weakly-supervised labels cannot determine which modality the event comes from. Wu et al. [5] then introduce a video-level label refinement method and adopt contrastive learning for audio and visual temporal alignment. Furthermore, Chen et al. [6] propose another label denoising method based on alleviating modality-specific noise through cross-modal loss patterns for both modalities.
However, previous works [5, 6] only considered the noisy label at the overall video instance level, while ignoring the segment-level label noise. Segment-level label noise indicates the fact that not every segment of a video contains all the events of the video. For example, a person may be speaking
in the video for a few seconds, and then he disappears in the following seconds. Therefore, it is inappropriate to consider speech as a supervised signal for all segments, where speech can be regarded as a noisy label for some segments of this video. It could harm the network in event classification on segments when training with such segment-level noisy labels.
The challenge of learning reliable segment-level labels comes from that the label of each segment is not predefined, but could be any combination of events that occur in the video, e.g., a segment may contain both speech and music events or none of them. To address this flexible label assignment challenge, we propose to leverage a third modality, the language, which enables better scene understanding beyond the labels. Language can describe what is happening in the scene, and who is involved, and identify segments not covered by the existing labels. Therefore, we propose to address AVVP from the language perspective, where natural language descriptions serve as a bridge connecting video and flexible weak labels.
In this paper, we propose a Language guided Segment-level Label Denoising (LSLD) strategy. We leverage the generalization capability of language to construct prompts that indicate all cases of event appearance for a segment. To connect vision and language, we introduce a vision-language pre-trained model, CLIP [7], into our work. Specifically, CLIP is used to extract the visual features for all segments, and the similarity between the segment and all prompts is calculated for label denoising. In addition, to further mitigate the effect of noisy segment labels, we perform a dynamic re-weighting strategy on the segments that may be mislabeled. Specifically, we calculate the similarity between the language description of each event and all the segments across the dataset. We do this for both the segments with or without the event label (i.e., potentially positive and negative clips) separately to compare their similarity distribution. If a segment's similarity falls within the overlap of the positive and negative groups, we consider it an unreliable segment and adjust its label dynamically according to its similarity to the event.
For the audio track, we introduce a large-scale audio-language pre-trained model LAION-CLAP [8] to denoise the audio label. The main contributions of our work are summarized as follows:
- We introduce the language modality to the AVVP task and construct language prompts that indicate all cases of event appearance for a segment, where the prompt with the highest similarity is regarded as the segment label to avoid segment-level label noise.
- We propose a simple yet effective Language-guided Segment-level Label Denoising (LSLD) mechanism based on CLAP and CLIP to align the modality of audio/visual and language for segment-level label denoising. In addition, we mine the visual segments with unreliable labels and learn from them with dynamic weight.
- Experiments show our method achieves comparable performance with the state-of-the-art methods. Especially, we improve visual metric significantly from  $66.4\%$  to  $71.3\%$  on the segment level over the state-of-the-art.
Audio-Visual Learning. Audio-visual learning includes various interesting tasks such as audio-visual representation learning [9, 10, 11, 12, 13, 14, 3, 15], audio-visual event localization [1, 16, 17, 18, 19, 20, 21], audio-visual sound separation [22, 23, 24, 25, 26, 27], audio-visual sound source localization [28, 29, 30, 31], and audio-visual action recognition [32, 33, 34, 35]. Most previous work on these tasks assumes that audio and visual are temporally aligned and always correlated. However, in many videos, audio and visual do not share the same information at a certain time, for instance, something that can be seen is not audible and the sound source is not visible. Therefore, we focus on the audio-visual video parsing task and aim to separately identify the events that happen in each modality with only weakly-supervised video-level labels for training.
Audio-Visual Video Parsing. Audio-visual video parsing (AVVP) task [4] intends to recognize events in each modality and localize their temporal boundaries. Since labeling each modality and segment of a video is time-consuming and labor-intensive, recent work has been conducted in a weakly-supervised manner. Tian et al. [4] utilize a hybrid attention network and multiple instance learning mechanism to aggregate segment features into video-level ones. Wu et al. [5] adopt the exchange of unrelated video and audio tracks to denoise weakly-supervised labels at the video level. More recently, to explicitly group semantic-aware multi-modal contexts, Mo et al. [36] propose
Figure 1: Algorithm Overview. (a) Language prompts construction provides prompts for a video to indicate all cases of event appearance based on the weak label of the video. (b) Language guided segment-Level noisy-label removing calculates the similarity between prompts and segments, where the prompt with the highest similarity is taken as the segment label. (c) Dynamic re-weighting for unreliable segments gathers the segments with and without the event using the denoised labels and calculates their similarity to the event name respectively, then re-weight the segments' label on the overlap region according to their similarity. (d) Training with Denoised Label on Segment-Level. Finally, with denoised segment-level labels, we train the network following [4]. CLIP and CLAP are frozen during the process. In the figure, “ $\rightarrow$ 0” denotes label removal,  $\otimes$  is the cross product, and  $\widehat{\mathbf{M}}$  denotes the maximum function.
a Multi-modal Grouping Network to learn more discriminative audio and visual representations. Chen et al. [6] propose a dynamic training mechanism, which lowers the modality-specific noise at the video level. Different from the above approaches, we propose to investigate segment-level label noise. Specifically, we introduce the language modality into the AVVP task and remove noisy labels on the segment level by constructing prompts and calculating the text-image similarity.
Language-Supervised Vision Representation Learning. Language-supervised vision representation learning aims at learning visual representations from language data. Early works for visual representation learning include predicting the bag-of-words representation [37] or training n-gram models [38] on the YFCC100M dataset [39]. While some work like ICMLM [40] and VirTex [41] produce effective visual representations of COCO Captions [42]. Soon after, CLIP [7] quickly draws attention due to its simplicity, scale, and generalization capabilities. CLIP is pre-trained on 400 million image-text pairs using the contrastive learning approach and can be transferred to multi-modal downstream tasks like image caption [43], and video caption [44] due to its zero-shot transfer ability. Thus we employ CLIP as our image encoder to bridge the gap between the image and language descriptions for segment-level label noising.
In this paper, we propose a novel Language guided Segment-level Label Denoising (LSLD) method for AVVP, which aims to remove the segment-level noise with the help of natural language. The overview of the proposed method is shown in Fig. 1, where language prompts are generated to calculate the similarity with the segments for denoising. Then dynamic re-weighting is adopted to adjust unreliable segment labels. With the newly assigned segment-level label, the network is trained end-to-end to better classify events on segments.
Following, we introduce our method in detail. We begin with the preliminary and our baseline method (Sec. 3.1). Then we illustrate the language prompt construction (Sec. 3.2), segment-level label denoising (Sec. 3.3), and the dynamic re-weighting strategy (Sec. 3.4).
Problem Setup and Notations. Given a  $T$ -segments audio-visual video sequence  $S = \{V_{t}, A_{t}\}_{t=1}^{T}$ ,  $A$  and  $V$  are the audio and visual track, respectively, and each segment is of one second long. Our goal is to predict the event label of each segment in audio and visual, which may contain multiple or no events at all. During evaluation, for the  $t$ -th video segment  $(V_{t}, A_{t})$ , the target is denoted as  $(\mathbf{y}_{t}^{a}, \mathbf{y}_{t}^{v}, \mathbf{y}_{t}^{av}) \in \mathbb{R}^{1 \times C}$ , where  $\mathbf{y}_{t}^{a}, \mathbf{y}_{t}^{v}$ , and  $\mathbf{y}_{t}^{av}$  are labels for audio, visual, and audio-visual events, respectively, and  $C$  is the number of event classes. During training, we can only obtain the weak label of the video as a whole, which is not specific to modality and segment.
**Baseline Framework.** To obtain cross-modal and uni-modal information, we adopt HAN [4] as our baseline. HAN [4] extracts visual feature  $\mathbf{F}^v = \{\mathbf{f}_t^v\}_{t=1}^T \in \mathbb{R}^{T \times d}$  and audio feature  $\mathbf{F}^a = \{\mathbf{f}_t^a\}_{t=1}^T \in \mathbb{R}^{T \times d}$  by Resnet [45] and VGGish [46], where  $d$  is the feature dimension. Then it adopts self-attention and cross-attention layers to aggregate cross-modal and uni-modal information at each segment. After gathering the audio and visual features, the segment-level event prediction for audio and visual  $(\mathbf{p}_t^a, \mathbf{p}_t^v \in \mathbb{R}^{1 \times C})$  can be obtained through an FC layer and a sigmoid function. Then the video-level prediction is aggregated by attentive MMIL pooling, which is formulated as:
$$
\mathbf {p} ^ {a} = \sum_ {t = 1} ^ {T} \mathbf {w} _ {t} ^ {a} \mathbf {p} _ {t} ^ {a}, \quad \mathbf {p} ^ {v} = \sum_ {t = 1} ^ {T} \mathbf {w} _ {t} ^ {v} \mathbf {p} _ {t} ^ {v}, \quad \mathbf {p} ^ {a v} = \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \mathbf {W} _ {t} [ m ] \odot \mathbf {P} _ {t} [ m ], \tag {1}
$$
where  $\mathbf{W}_t = \{\mathbf{w}_t^a, \mathbf{w}_t^v\}$  denote the temporal attention weights,  $\mathbf{P}_t = \{\mathbf{p}_t^a, \mathbf{p}_t^v\}$  includes audio and visual predictions,  $\mathbf{M}$  is the number of modalities. Finally, binary cross-entropy (BCE) loss is adopted to optimize the model in a weakly-supervised way:
$$
\mathcal {L} _ {\text {b a s e}} = \operatorname {B C E} \left(\mathbf {y} ^ {a v}, \mathrm {p} ^ {a v}\right) + \operatorname {B C E} \left(\overline {{\mathbf {y}}} ^ {a}, \mathbf {p} ^ {a}\right) + \operatorname {B C E} \left(\overline {{\mathbf {y}}} ^ {v}, \mathbf {p} ^ {v}\right), \tag {2}
$$
where  $\overline{\mathbf{y}}^a$ ,  $\overline{\mathbf{y}}^v$  are video-level audio and visual labels by performing label smooth on  $\mathbf{y}^{av}$  to mitigate the impact of noisy labels. As the baseline method only utilizes video-level labels during training, which is affected by the segment-level noise, we propose a Language guided Segment-level Label Denoising (LSLD) method to address segment-level noise.
3.2 Language Prompt Construction
As discussed above, during training, we can only access video-level labels, but for each segment, the label could be any combination of events, which is flexible. To address the flexible label assignment challenge, we introduce the language modality into our task, which describes how the events occur. For each video, to consider all cases of event appearance, we create prompts for each case according to its video-level labels. As shown in Fig. 1 (a), taking a video labeled with two classes named A and B as an example, we construct 4 prompts describing A appears, B appears, both classes appear and none of them appear, respectively. Finally, the language prompts for the video could be:
[Class  $A$ , Class  $B$ , Class  $A$  and  $B$ , other],
where other means no pre-defined event appears.
3.3 Language Guided Segment-Level Label Denoising
With language prompts generated in Sec. 3.2, we intend to remove segment-level noisy labels by language guidance. Specifically, we calculate the similarity between each segment and prompts, where the prompt with the highest similarity is more likely to describe the segment accurately. As illustrated in Fig. 1 (b), take the visual modality for example, we feed the constructed language prompts into CLIP's text encoder to obtain the text features, and each frame of the video is fed into CLIP's image encoder to extract the image feature. We denote the text feature as  $\mathbf{f}_t \in \mathbb{R}^{N \times d}$  and the visual feature as  $\mathbf{f}_v \in \mathbb{R}^{T \times d}$ , where  $N$  is the number of prompts,  $T$  is the number of segments, and  $d$  is feature dimension. Then we calculate the text-image similarity matrix  $\mathbf{s} \in \mathbb{R}^{N \times T}$  on the normalized text and image feature:
$$
\mathbf {s} = \operatorname {S o f t m a x} \left(\frac {\mathbf {f} _ {t}}{\| \mathbf {f} _ {t} \| _ {2}} \otimes \left(\frac {\mathbf {f} _ {v}}{\| \mathbf {f} _ {v} \| _ {2}}\right) ^ {\top}\right), \tag {3}
$$
where  $\otimes$  denotes the matrix multiplication. For the audio modality, we feed the audio into CLAP's audio encoder and the language prompts into CLAP's text encoder.
With the similarity matrix, the language prompt with the highest similarity for each segment is obtained, which infers the segments' corresponding event label. For instance, as shown in Fig.1 (b), the video is labeled with class A and class B, while the most similar prompt of its second segment is Class A, which means that this segment most likely contains only with class A. In this case, we treat class B as the noisy class of this segment for label denoising. Likewise, if the most similar prompt is other, both class A and class B will be treated as noisy labels.
With segment-level label denoising, each audio/visual segment is assigned its own event label  $\widetilde{\mathbf{y}}_t^a /\widetilde{\mathbf{y}}_t^v$  instead of being regarded as having the same label as the whole video. In this way, the labels depict the audio/visual segments more accurately which boosts network learning.
3.4 Dynamic Re-weighting for Unreliable Segments
In Sec. 3.3, each segment is re-labeled to avoid segment-level noise. However, such noise is inevitable, because some segments may be mislabeled. To further alleviate the impact of mislabeled segments, in this section, we propose to mine the unreliable segments and learn from them with dynamic weight.
To mine the unreliable segments, for each event class, we aggregate the segments with and without the event into two groups (i.e., potentially positive and negative segments) by the denoised segment-level labels. Then, we calculate the similarity between the visual segments and the language prompts of the class for the two groups separately to compare their similarity distribution. The whole process is shown in Fig. 1 (c), where the similarity distributions of with/ without Car are presented. We observe that though some visual segments in orange are labeled as without a Car, their similarity is still high, while some segments in blue that are labeled as with a Car are not that similar to the class description. Therefore, we consider the overlap region of the two similarity groups as mislabeled segments.
Then, we propose a dynamic re-weighting strategy to learn unreliable segments softly. Specifically, for an event and a corresponding unreliable segment, the weighted similarity between the segment and event description is adopted as the soft label. Then, for the t-th segment of a video, its label for event class  $c$  can be calculated as:
$$
\hat {\mathbf {y}} _ {t} ^ {v} [ c ] = \left\{ \begin{array}{c c} M i n (1, \alpha \times \mathbf {s} _ {\mathbf {c}}), & \widetilde {\mathbf {y}} _ {t} ^ {v} [ c ] = 1 a n d \mathbf {s} _ {\mathbf {c}} \leq M a x _ {\mathrm {w / o}}, \\ M i n (1, \beta \times \mathbf {s} _ {\mathbf {c}}), & \widetilde {\mathbf {y}} _ {t} ^ {v} [ c ] = 0 a n d \mathbf {s} _ {\mathbf {c}} \geq M i n _ {\mathrm {w}}, \\ \widetilde {\mathbf {y}} _ {t} ^ {v} [ c ], & o t h e r w i s e, \end{array} \right. \tag {4}
$$
where  $Min()$  is a function to return the item with the lowest value,  $\alpha$  and  $\beta$  are parameters for the segment with/ without the event  $c$ .  $\mathbf{s_c}$  is the similarity between segments and class description using Eq. 3.  $Min_{w}$  is the lowest similarity of the segments with the event,  $Max_{w/o}$  is the highest similarity of segments w/o the event. If the similarity of a segment w/o the event is higher than  $Min_{w}$ , then its similarity is set to  $Min(1, \beta \times \mathbf{s_c})$ . If the similarity of a segment with the event is lower than  $Max_{w/o}$ , then its similarity is set to  $Min(1, \alpha \times \mathbf{s_c})$ . Note that, during the application, we set  $\alpha > \beta$  to ensure the segments with  $\widetilde{\mathbf{y}}_t^v[c] = 1$  still have more possibility of the event appearance. We only adopt dynamic re-weighting on the label of unreliable visual segments.
After dynamic re-weighting, we use the segment-level label to optimize the model on the audio and visual modality. The pipeline is shown in Fig. 1 (d). Specifically, we replace the video-level label  $\overline{\mathbf{y}}^v$  with our segment-level label  $\hat{\mathbf{y}}_t^v$ , and  $\overline{\mathbf{y}}^a$  with the segment-level label  $\widetilde{\mathbf{y}}_t^a$ , in Eq. 2. Notably, we also use CLAP/CLIP to extract the audio/visual features during training since CLAP/CLIP trains from large-scale datasets and learns richer audio/visual information.
Dataset. In the AVVP task, we only evaluate our method on the Look, Listen and Parse (LLP) Dataset [4] following previous AVVP work. The dataset contains 11,849 video clips 10 seconds long from 25 different event categories, including human activities, animals, musical instruments, etc. For the training process, we use 10,000 video clips with only video-level event labels. The remaining
Table 1: Comparisons with the state-of-the-art methods on LLP dataset. "CLAP+CLIP" means we use the audio and visual features extracted by CLAP [8] and CLIP [7] respectively, while * represents our baseline framework. The best results are marked in bold.
Method	Segment-Level	Event-level
A	V	A-V	Type	Event	A	V	A-V	Type	Event
AVE [1]	47.2	37.1	35.4	39.9	41.6	40.4	34.7	31.6	35.5	36.5
AVSDN [16]	47.8	52.0	37.1	45.7	50.8	34.1	46.3	26.5	35.6	37.7
HAN [4]*	60.1	52.9	48.9	54.0	55.4	51.3	48.9	43.0	47.7	48.0
MA [5]	60.3	60.0	55.1	58.9	57.9	53.6	56.4	49.0	53.0	50.6
CVCMS [47]	60.8	63.5	57.0	60.5	59.5	53.8	58.9	49.5	54.0	52.1
DHHN [48]	61.4	63.4	56.8	60.5	59.5	54.6	60.8	51.1	55.5	53.3
MM-Pyramid [49]	61.1	60.3	55.8	59.7	59.1	53.8	56.7	49.4	54.1	51.2
MGN [36]	60.8	55.4	50.4	55.5	57.2	51.1	52.4	44.4	49.3	49.1
JoMoLD [6]	61.3	63.8	57.2	60.8	59.9	53.9	59.9	49.6	54.5	52.5
CMPAE [50]	64.2	66.4	59.2	63.3	62.8	56.6	63.7	51.8	57.4	55.7
LSLD (ours)	62.7	67.1	59.4	63.1	62.2	55.7	64.3	52.6	57.6	55.2
LSLD+CLAP+CLIP	68.7	71.3	63.4	67.8	68.2	61.5	67.4	55.9	61.6	60.6
Table 2: Ablation study of each component of LSLD. Here, "DeN" denotes language-guided segment-level label denoising. "ReW" denotes the dynamic weighting strategy on unreliable segments. All experiments are based on CLAP and CLIP features.
DeN	ReW	Segment-Level	Event-level
A	V	A-V	Type	Event	A	V	A-V	Type	Event
X	X	64.1	54.9	49.0	56.0	60.0	53.3	51.0	41.8	48.7	51.6
✓	X	67.3	70.1	61.2	66.2	67.2	60.8	66.3	54.3	60.5	60.2
✓	✓	68.7	71.3	63.4	67.8	68.2	61.5	67.4	55.9	61.6	60.6
1849 validation and test videos possess modality and segment-specific labels (i.e., start and end time of each event on audio and visual track). We conduct experiments following the official data splits from the LLP dataset.
Evaluation Metrics. We evaluate the parsing performance of audio, visual, and audio-visual events under both segment-level and event-level metrics by adopting F-scores. The segment-level metrics can evaluate segment-wise event prediction performance. For the event-level metrics, we extract events by concatenating consecutive positive segments in the same event and compute event-level F-score with mIoU=0.5 as the threshold. In addition, we evaluate the parsing performance of audio-visual events using Type@AV and Event@AV metrics. Type@AV is calculated by averaging audio, visual, and audio-visual event evaluation results while Event@AV considers all audio and visual event categories for each sample instead of directly averaging results from different events.