SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yung-Hsuan Lai, $^{1}$  Yen-Chun Chen, $^{2}$  Yu-Chiang Frank Wang $^{1,3}$
$^{1}$ National Taiwan University  $^{2}$ Microsoft  $^{3}$ NVIDIA
r10942097@ntu.edu.tw, chen.yenchun.tw@gmail.com, frankwang@nvidia.com
Audio-visual learning has been a major pillar of multi-modal machine learning, where the community mostly focused on its modality-aligned setting, i.e., the audio and visual modality are both assumed to signal the prediction target. With the Look, Listen, and Parse dataset (LLP), we investigate the under-explored unaligned setting, where the goal is to recognize audio and visual events in a video with only weak labels observed. Such weak video-level labels only tell what events happen without knowing the modality they are perceived (audio, visual, or both). To enhance learning in this challenging setting, we incorporate large-scale contrastively pre-trained models as the modality teachers. A simple, effective, and generic method, termed Visual-Audio Label Elaboration (VALOR), is innovated to harvest modality labels for the training events. Empirical studies show that the harvested labels significantly improve an attentional baseline by 8.0 in average F-score (Type@AV). Surprisingly, we found that modality-independent teachers outperform their modality-fused counterparts since they are noise-proof from the other potentially unaligned modality. Moreover, our best model achieves the new state-of-the-art on all metrics of LLP by a substantial margin (+5.4 F-score for Type@AV). VALOR is further generalized to Audio-Visual Event Localization and achieves the new state-of-the-art as well.<sup>1</sup>
Multi-modal learning has become a pivotal topic in modern machine learning research. Audio-visual learning is undoubtedly one of the primary focuses, as human frequently uses both hearing and vision to perceive the surrounding environment. Countless researchers have devoted to its modality-aligned setting with a strong assumption that the audio and visual modality both contain learnable clues to the desired prediction target. Numerous audio-visual tasks and algorithms have then been proposed, such as audio-visual speech recognition [1, 66, 68], audio-visual action recognition [22, 53, 81], sound generation from visual data [18, 69, 84], audio-visual question answering [40, 89], and many more. However, almost all real-world events can be audible while invisible, and vice versa, depending on how they are perceived. For example, a mother doing dishes in the kitchen might hear a baby crying from the living room, but be unable to directly see what is happening to the baby.
Having observed this potential modality mismatch in generic videos, Tian et al. [73] proposed the Audio-Visual Video Parsing (AVVP) task, which aims to recognize events in videos independently of the audio and visual modalities and also temporally localize these events. AVVP presents an unaligned setting of audio-visual learning since all 25 event types considered can be audio-only, visual-only, or audio-visual. Unfortunately, due to the laborious labeling process, Tian et al. [73] created this dataset (Look, Listen, and Parse; LLP) in a weakly-supervised setting.2 More specifically,
$^{1}$ Code is available at: https://github.com/Franklin905/VALOR.
$^{2}$ AVVP, LLP are used interchangeably in the literature. We use AVVP for the task, and LLP for the dataset.
Figure 1: Modality-unaligned samples from LLP. Note that recent AVVP approaches like HAN [73] are vulnerable to unaligned data modality and produce incorrect predictions.
only video-level event annotations are available at training. In other words, the modality (audio, visual, or both) and the timestamp of which an event occurs are not given to the learning model.
The AVVP task poses significant challenges from three different perspectives. First, an event is typically modality independent, i.e., knowing an event occurs in one modality says nothing about the other modality. As illustrated in Fig. 1, a sleeping dog is seen but may not be heard; conversely, a violin being played could sometimes go out of the camera view. Second, existing works heavily rely on the Multi-modal Multiple Instance Learning (MMIL) loss [73] to soft-select the modality (and timestamp), given only weak modality-less labels. This would be challenging for models to learn the correct event modality without observing a large amount of data. The uni-modal guided loss via label smoothing is also used to introduce uncertainty to the weak labels and thus regularize modality recognition. However, we hypothesize this improvement could be sub-optimal because no explicit modality information is introduced. Finally, AVVP requires models to predict events for all 1-second segments in a given video. Learning from weak video-level labels without timestamps makes it challenging for models to predict on a per-segment basis.
To address the above challenges in AVVP, we propose to incorporate large-scale pre-trained open-vocabulary models, namely CLIP [56] and CLAP [79], to enhance learning with weak labels. Pre-trained on pixels and waveforms (and contrastively pre-trained with natural language), these models are inherently isolated from potential spurious noise from the other modality. Another benefit is the applicability of their prediction in an open-vocabulary fashion. Therefore, to benefit from CLIP and CLAP, we aim to harvest explicit modality learning signals from them. Moreover, we aim to inference these models per video segment, yielding fine-grained temporal annotations.
While it might be tempting to naively treat these pre-trained models as teachers and then applies knowledge distillation (KD) [28], this could be sub-optimal as some events are difficult to distinguish from a single modality, even for humans. For example, cars, motorcycles, and lawn mowers all produce similar sounds. To better utilize CLIP and CLAP, we introduce Visual-Audio Label Elaboration (VALOR), to harvest modality and timestamp labels in LLP. We prompt CLIP/CLAP with natural language description of all visual/audio event types for each video segment-by-segment and then extract labels when a threshold is met. Additionally, implausible events are filtered out using the original weak labels accompanied with the video to mitigate the above indistinguishable problem. VALOR constructs fine-grained temporal labels in both modalities so that models have access to explicit training signals.
In addition to achieving the promising performance of AVVP, we observe that modality-independent teachers, CLIP and CLAP, generate more reliable labels than a modality-fused one, a cross-modal transformer. We also showcase the generalization capability of VALOR via the Audio-Visual Event Localization (AVE) task, in which our method also achieves the new state-of-the-art. Our contributions are summarized as follows:
- A simple and effective AVVP framework, VALOR, is proposed to harvest modality and temporal labels directly from video-level annotations, with an absolute improvement of  $+8.0$  F-score.
- We are the first to point out that modality independence could be crucial for audio-visual learning in the unaligned and weakly-supervised setup.
- Our VALOR achieves new state-of-the-art results with significant improvements on AVVP (+5.4 F-score), with generalization to AVE (+4.4 accuracy) jointly verified.
Audio-Visual Video Parsing (AVVP) The AVVP [73] task is to recognize events of interest in a video in both visual and audio modalities and to temporally identify the associated frames. For the benchmark dataset of Look, Listen, and Parse (LLP), a  $T$ -second video is split into  $T$  non-overlapping segments. Each video segment is paired with a set of multi-class event labels  $(\pmb{y}_t^v, \pmb{y}_t^a) \in \{0, 1\}^C$  ( $\pmb{y}_t^v$ : visual events,  $\pmb{y}_t^a$ : audio events,  $C$ : number of event types). However, in the training split, the dense 'segment-level' labels  $(\pmb{y}_t^v, \pmb{y}_t^a)$  are not available. Instead, only the global modality-less 'video-level' labels  $\pmb{y} := \max_t \{\pmb{y}_t^v \wedge \pmb{y}_t^a\}_{t=1}^T$  are provided ( $\wedge$ : element-wise 'logical and'). In other words, AVVP models need to be learned in a weakly-supervised setting.
Baseline Model We now briefly review the model of Hybrid Attention Network (HAN) [73], which is a common baseline for AVVP. In HAN, ResNet-152 [26] and  $\mathrm{R}(2 + 1)\mathrm{D}$  [75] are employed to extract 2D and 3D visual features. Subsequently, they are concatenated and projected into segment-level features  $\pmb{F}^{v} = \{\pmb{f}_{t}^{v}\}_{t = 1}^{T}\in \mathbb{R}^{T\times d}$  ( $d$ : hidden dimension). Segment-level audio features  $\pmb{F}^{a} = \{\pmb{f}_{t}^{a}\}_{t = 1}^{T}\in \mathbb{R}^{T\times d}$  are extracted using VGGish [27] and projected to the same dimension. HAN takes these features and aggregates the intra-modal and cross-modal information through self-attention and cross-attention:
$$
\tilde {\boldsymbol {f}} _ {t} ^ {a} = \boldsymbol {f} _ {t} ^ {a} + \operatorname {A t t} \left(\boldsymbol {f} _ {t} ^ {a}, \boldsymbol {F} ^ {a}, \boldsymbol {F} ^ {a}\right) + \operatorname {A t t} \left(\boldsymbol {f} _ {t} ^ {a}, \boldsymbol {F} ^ {v}, \boldsymbol {F} ^ {v}\right) \tag {1}
$$
$$
\tilde {\boldsymbol {f}} _ {t} ^ {v} = \boldsymbol {f} _ {t} ^ {v} + \operatorname {A t t} \left(\boldsymbol {f} _ {t} ^ {v}, \boldsymbol {F} ^ {v}, \boldsymbol {F} ^ {v}\right) + \operatorname {A t t} \left(\boldsymbol {f} _ {t} ^ {v}, \boldsymbol {F} ^ {a}, \boldsymbol {F} ^ {a}\right), \tag {2}
$$
where  $\mathrm{Att}(\pmb {q},\pmb {K},\pmb {V})$  denotes multi-head attention [76]. Following Transformer's practice, the outputs are further fed through LayerNorms [6] and a 2-layer FFN to yield  $\hat{\pmb{f}}_t^a,\hat{\pmb{f}}_t^v$  . With another linear layer, the hidden features are transformed into categorical logits  $z_{t}^{v},z_{t}^{a}$  for visual and audio events, respectively. Finally, the segment-level audio and visual event categorical probabilities,  $\pmb{p}_t^a$  and  $\pmb{p}_t^v (\in [0,1]^C)$ , are obtained by applying Sigmoid activation.
As a key module in Tian et al. [73], Multi-modal Multiple Instance Learning pooling (MMIL) is applied to address the above weakly-supervised learning task, which predicts the audio and visual event probabilities  $(\pmb{p}^m, m \in \{a, v\}$ , audio and visual modalities) as:
$$
\boldsymbol {A} ^ {m} = \left\{\boldsymbol {\alpha} _ {t} ^ {m} \right\} _ {t} = \operatorname {s o f t m a x} _ {t} \left(\hat {\boldsymbol {F}} ^ {m} \boldsymbol {W} ^ {m}\right), \quad \boldsymbol {p} ^ {m} = \sum_ {t} \boldsymbol {\alpha} _ {t} ^ {m} \odot \boldsymbol {p} _ {t} ^ {m}, \tag {3}
$$
where trainable parameters  $\pmb{W}^{m}\in \mathbb{R}^{d\times C}$  are implemented as linear layers (  $\odot$  : element-wise product). For video-level event probability  $\pmb{p}$ :
$$
\boldsymbol {\mathsf {B}} = \left\{\left\{\beta_ {t} ^ {m} \right\} _ {t} \right\} _ {m} = \operatorname {s o f t m a x} _ {m} (\hat {\boldsymbol {X}} \boldsymbol {W}), \quad \boldsymbol {p} = \sum_ {m} \sum_ {t} \beta_ {t} ^ {m} \odot \boldsymbol {\alpha} _ {t} ^ {m} \odot \boldsymbol {p} _ {t} ^ {m}, \tag {4}
$$
where  $\hat{\mathbf{X}} = \{\hat{\mathbf{F}}^m\}_{m} \in \mathbb{R}^{2 \times T \times d}$  and  $\mathbf{W}$  as a trainable linear layer. Moreover, modality training targets are obtained via label smoothing (LS) [71]:  $\tilde{\mathbf{y}}^m = \mathrm{LS}(\mathbf{y})$ . Finally, the model is trained with binary cross entropy (BCE) as the loss function:
$$
\mathcal {L} _ {\text {b a s e}} = \mathcal {L} _ {\text {v i d e o}} + \mathcal {L} _ {\text {g u i d e d}} ^ {a} + \mathcal {L} _ {\text {g u i d e d}} ^ {v}, \quad \mathcal {L} _ {\text {v i d e o}} = \operatorname {B C E} (\boldsymbol {p}, \boldsymbol {y}), \quad \mathcal {L} _ {\text {g u i d e d}} ^ {m} = \operatorname {B C E} \left(\boldsymbol {p} ^ {m}, \tilde {\boldsymbol {y}} ^ {m}\right). \tag {5}
$$
In summary, by the attention mechanisms introduced in HAN, MMIL pooling assigns event labels for each modality across time segments with only video-level event labels observed during training.
With only video-level event labels observed during training, we address three major challenges of AVVP: 1) modality independence of events' occurrence, 2) reliance on MMIL pooling for event label assignment under insufficient data, and 3) demand for dense temporal predictions. To address these challenges, we propose to leverage large-scale pre-trained contrastive models, CLIP and CLAP, to extract modality-aware, temporally dense training signals to guide model learning.
3.1 Zero-Shot Transfer of Contrastive Pre-trained Models
Radford et al. [56] proposed Contrastive Language-Image Pre-training (CLIP) to utilize web-scale image-text pairs to train a strong image encoder. As a result, CLIP overthrows the limitation of predicting predefined categories. Due to its large training data size (400M), CLIP has demonstrated remarkable zero-shot performance on a wide range of visual recognition tasks. All the above motivates us to incorporate CLIP to improve visual event recognition in AVVP.
In our work, CLIP's visual understanding of AVVP is extracted as the following. We extract  $T$  evenly spaced video frames and pass them into CLIP's image encoder to obtain the visual features  $\{\pmb{f}_t^{\mathrm{CLIP}}\}_{t=1}^T \in \mathbb{R}^{T \times d_2}$  ( $d_2$ : the dimension of CLIP's feature). For simplicity and readability, we will
Figure 2: VALOR framework. With modality-independent label elaboration via CLIP and CLAP, the harvested temporally dense labels serve as additional modality- and time-aware cues.
omit the time subscript  $t$  for the remainder of this paper when there is no ambiguity. Next, we convert the AVVP event categories to concepts that CLIP understands. A caption for each event is constructed by adding a "A photo of" prefix to the event's natural language form. These captions are processed by the CLIP's text encoder, resulting in event features  $\pmb{G}^{\mathrm{CLIP}} = \{\pmb{g}_c^{\mathrm{CLIP}}\}_{c=1}^C \in \mathbb{R}^{C \times d_2}$ , where  $c$  indexes the events, and  $\pmb{g}_c$  represents the text feature of the  $c$ -th event. Frame-level event logits  $\pmb{z}^{\mathrm{CLIP}} \in \mathbb{R}^C$  can be obtained by calculating the inner products:
$$
z ^ {\mathrm {C L I P}} = f ^ {\mathrm {C L I P}} G ^ {\mathrm {C L I P} ^ {\top}}. \tag {6}
$$
In light of the notable success of CLIP [56], several studies have sprouted to research on learning representative audio embeddings and text embeddings through Contrastive Language-Audio Pretraining (CLAP) [13, 15, 25, 49, 79]. In the same way as images and text are encoded in CLIP, web-scale audios and text are pre-trained with a contrastive objective in CLAP. Symmetrically, we obtain CLAP's understanding of AVVP audios as the following. From the audio channel, the raw waveform is extracted and split into  $T$  segments with the same lengths and then fed into CLAP, yielding segment-level audio features  $\{f_t^{\mathrm{CLAP}}\}_{t} \in \mathbb{R}^{T \times d_3}$  ( $d_3$ : the dimension of CLAP's feature). On the other hand, an audio event caption is constructed by adding the prefix "This is a sound of" to each AVVP event's name. Processed by the CLAP text encoder, we obtain  $G^{\mathrm{CLAP}} = \{g_c^{\mathrm{CLAP}}\}_{c} \in \mathbb{R}^{C \times d_3}$ . Segment-level audio event logits  $z^{\mathrm{CLAP}} \in \mathbb{R}^C$  are obtained by the inner products:
$$
\boldsymbol {z} ^ {\text {C L A P}} = \boldsymbol {f} ^ {\text {C L A P}} \boldsymbol {G} ^ {\text {C L A P} ^ {\top}}. \tag {7}
$$
We note that Eqn. (6) and (7) can be viewed as CLIP's and CLAP's understanding of the associate video frame in the event space of AVVP.
3.2 Harvesting Training Signals
Given the logits  $z^{\mathrm{CLIP}}$  and  $z^{\mathrm{CLAP}}$ , we aim to convert them to useful training signals for the AVVP task. An intuitive idea is to teach our model via knowledge distillation (KD) [28]. To deploy KD in training, segment-level normalized probabilities are first computed:  $q^{P} = \operatorname{softmax}_{c}(z^{P})$ ,  $q^{m} = \operatorname{softmax}_{c}(z^{m})$ , where  $(m,P) \in \{(v,\mathrm{CLIP}), (a,\mathrm{CLAP})\}$  denotes data modality (audio/visual) and pre-trained model (CLIP/CLAP) pair. Next, KL-divergence for all segments is calculated:  $\mathcal{L}_{\mathrm{KD}}^{m} = \sum_{t}\mathrm{KL}(q_{t}^{P},q_{t}^{m})$ . Finally, KD training is done by optimizing the loss function:
$$
\mathcal {L} _ {\mathrm {K D}} = \mathcal {L} _ {\text {v i d e o}} + \mathcal {L} _ {\mathrm {K D}} ^ {a} + \mathcal {L} _ {\mathrm {K D}} ^ {v}. \tag {8}
$$
However, as we find out empirically (shown in Table 4), this is not the optimal usage of CLIP and CLAP. We hypothesize that some events are hard to distinguish from a single modality, e.g. cars, motorcycles, and lawn mowers produce the sound of an engine. Therefore, we design VALOR, utilizing video-level labels to filter out the impossible events, hence mitigating the confusion.
Visual-Audio Label Elaboration (VALOR) To better exploit CLIP and CLAP, we design a simple yet effective method, VALOR, to harvest dense labels in both modalities. In particular, we first define class-dependent thresholds  $\pmb{\theta}^{P}\in \mathbb{R}^{C}$  for each modality to obtain segment-level labels from logits. Next, the impossible events are excluded using the given video-level labels, done via logical AND. Formally, this process can be written as:  $\hat{\pmb{y}}_t^m = \{\pmb {z}_t^P >\pmb {\theta}^P\} \wedge \pmb {y},$  with the overall loss function:
$$
\mathcal {L} _ {\text {V A L O R}} = \mathcal {L} _ {\text {v i d e o}} + \mathcal {L} _ {\text {V A L O R}} ^ {a} + \mathcal {L} _ {\text {V A L O R}} ^ {v}, \quad \mathcal {L} _ {\text {V A L O R}} ^ {m} = \sum_ {t} \mathrm {B C E} \left(\boldsymbol {p} _ {t} ^ {m}, \hat {\boldsymbol {y}} _ {t} ^ {m}\right). \tag {9}
$$
To summarize, we design a simple yet effective method, VALOR, to utilize large-scale pre-trained contrastive models, CLIP and CLAP, to generate segment-level labels in both modalities. Due to the nature of immunity to spurious noise from the other modality, the contrastive pre-training methods, and the large pre-training dataset size, CLIP and CLAP are able to provide reliable labels in visual and audio modality, respectively. In addition, they are able to provide temporally dense labels to explicitly guide the model in learning events in each segment.
4.1 Audio-Visual Video Parsing with Look, Listen, and Parse
For AVVP, research flourishes along two orthogonal directions: enhancing the model architecture and label refinement. Architectural improvements include cross-modal co-occurrence module [45], class-aware uni-modal features and cross-modal grouping [51], and Multi-Modal Pyramid attention [87]. On the other hand, label refinement shares a similar spirit with ours. MA [77] corrupted the data by swapping the audio channel of two videos with disjoint video-level event sets. The model's likelihood of the corrupted data was then used to determine the modality label. More recently, JoMoLD [11] utilized a two-stage approach. First, an AVVP model was trained as usual. Next, another model was trained while denoising the weak labels with prior belief from the first model. Both MA and JoMoLD produced global modality labels without timestamps. Concurrent to ours, VPLAN [96] and LSLD [16] generate dense temporal visual annotations with CLIP; however, the audio labels remain absent. Our VALOR represents a unified framework to elaborate the weak labels, along modality and temporal dimension, via zero-shot transfer of pre-trained models. We further emphasize the importance of modality independence when synthesizing modality supervision.
4.2 More Audio-Visual Learning
Audio-Visual Event Localization (AVE) Tian et al. [72] proposed AVE to recognize the audiovisual event in a video while localizing its temporal boundaries. Numerous studies have been conducted, including Lin et al. [46] with seq2seq models, Lin and Wang [44] using intra&inter frame Transformers, Wu et al. [78] via dual attention matching, audio-spatial channel-attention by Xu et al. [82], positive sample propagation from Zhou et al. [95], and Xia and Zhao [80] employing background suppression. We generalize VALOR to AVE's weakly supervised setting.
Audio-Visual Assistance While significant advancements have been made in speech recognition, speech enhancement, and action recognition, noise or bias residing in the uni-modal data is still problematic. An effective solution could involve integrating data from an additional modality. This research direction encompasses various areas including speech recognition [1, 30, 66, 68], speaker recognition [12, 14, 55, 61, 63, 67], action recognition [22, 35, 36, 53, 81], speech enhancement or separation [2, 3, 34, 39, 50, 60], and object sound separation [7, 20, 21, 59, 74, 83, 91, 92].
Audio-Visual Correspondence and Understanding Humans possess an impressive capacity to deduce occurrences in one sensory modality using information solely from another. This fascinating human ability to perceive across modalities has inspired researchers to delve into sound generation from visual data [18, 19, 37, 45, 54, 69, 84, 93], video generation from audio [38, 41, 43, 90, 94], and audio-visual retrieval [42, 70]. In the pursuit of understanding how humans process audio-visual events, numerous studies have been undertaken on audio-visual understanding tasks such as sound localization in videos [5, 31, 32, 52, 64], audio-visual navigation [8-10, 17, 48, 86, 88], and audio-visual question answering [4, 24, 29, 40, 62, 65, 89].
Dataset and Metrics The LLP dataset is composed of 11849 10-second Youtube video clips covering 25 event categories, such as human activities, musical instruments, vehicles, and animals. The dataset is divided into training, validation, and testing splits, containing 10,000, 649, and 1200 clips, respectively. The official evaluation uses F-score to evaluate audio (A), visual (V), and
Table 1: AVVP benchmark. Note that pseudo label denoising is not applied for VPLAN†. VALOR+ is trained on a thinner yet deeper HAN of similar size. VALOR++ further uses CLIP and CLAP as feature extractors and significantly boosts all metrics. The best numbers are in bold and the second best numbers are underlined.
Methods	Segment-level	Event-level
A	V	AV	Type	Event	A	V	AV	Type	Event
AVE [72]	47.2	37.1	35.4	39.9	41.6	40.4	34.7	31.6	35.5	36.5
AVSDN [46]	47.8	52.0	37.1	45.7	50.8	34.1	46.3	26.5	35.6	37.7
HAN [73]	60.1	52.9	48.9	54.0	55.4	51.3	48.9	43.0	47.7	48.0
MM-Pyr [87]	60.9	54.4	50.0	55.1	57.6	52.7	51.8	44.4	49.9	50.5
MGN [51]	60.8	55.4	50.4	55.5	57.2	51.1	52.4	44.4	49.3	49.1
CVCMS [47]	59.2	59.9	53.4	57.5	58.1	51.3	55.5	46.2	51.0	49.7
DHHN [33]	61.3	58.3	52.9	57.5	58.1	54.0	55.1	47.3	51.5	51.5
MA [77]	60.3	60.0	55.1	58.9	57.9	53.6	56.4	49.0	53.0	50.6
JoMoLD [11]	61.3	63.8	57.2	60.8	59.9	53.9	59.9	49.6	54.5	52.5
\( VPLAN^† \)[96]	60.5	64.8	58.3	61.2	59.4	51.4	61.5	51.2	54.7	50.8
VALOR	61.8	65.9	58.4	62.0	61.5	55.4	62.6	52.2	56.7	54.2
VALOR+	62.8	66.7	60.0	63.2	62.3	57.1	63.9	54.4	58.5	55.9
VALOR++	68.1	68.4	61.9	66.2	66.8	61.2	64.7	55.5	60.4	59.0
audio-visual (AV) events separately. Type@AV (Type) is the averaged F-scores of A, V, and AV. Event@AV (Event) measures the ability to detect events in both modalities by combining audio and visual event detection results. Different from segment-level metrics, the event-level metrics treat consecutive positive segments as a whole, and mIoU of 0.5 is applied to calculate F-scores.
Implementation Details Unless otherwise specified, VALOR uses HAN under a fair setting w.r.t. previous works with same data pre-processing. For the visual feature extraction, video frames are sampled at 8 frames per second. Additionally, we conduct experiments using CLIP and CLAP as feature extractors. The pre-trained ViT-L CLIP and HTSAT-RoBERTa-fusion CLAP are used to generate labels and extract features. Note that for all experiments with CLAP, we use the implementation from Wu et al. [79] pre-trained on LAION-Audio-630K. We do not use the version pre-trained on AudioSet (a larger pre-training corpus) since it overlaps with the AVVP validation and testing videos.