SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7: Two Different HAN Model Architectures. The "standard" model architecture is used in VALOR. The "variant" model architecture is used in VALOR+ and VALOR++.
HAN model	standard	variant
Model Arch. Hyper-parameters
hidden dim	512	256
hidden layers	1	4
trainable params	5.1M	5.05M
Training Hyper-parameters
peak learning rate	1e-4	3e-4
min learning rate	1e-6	3e-6
Table 8: Ablation study of whether using video-level labels as filters. Left: the ablation study of using video-level labels in audio label elaboration. Right: the ablation study of using video-level labels in visual label elaboration. To save the time required for tuning event thresholds, we have transformed class-dependent event thresholds into unified event thresholds, which means that the thresholds for each event are the same.
Video labels as filters	Event Thresholds	Segment-level
A	V	AV	Type	Event
×	unified	47.9	64.5	49.2	53.9	53.8
✓	unified	63.4	65.8	60.2	63.1	62.2
✓	class-dependent	62.7	66.3	61.0	63.4	61.8
Video labels as filters	Event Thresholds	Segment-level
A	V	AV	Type	Event
×	unified	62.8	53.8	50.9	55.9	58.6
✓	unified	62.3	65.9	60.6	62.9	60.9
✓	class-dependent	62.7	66.3	61.0	63.4	61.8
to unified does not significantly degrade the model's performance, whether in the audio or visual modality. However, if video label filtering is not applied, the resulting audio and visual pseudo labels become highly inaccurate, leading to a model with an audio F-score of only 47.9 and a visual F-score of only 53.8.
D VALOR with Pseudo Label Denoising
In this section, we explore the application of Pseudo Label Denoising (PLD), as proposed in VPLAN [96], to refine the segment-level labels generated by our method. The hyper-parameters for the PLD, specifically  $K = 4$  and  $\alpha = 6$  for the visual modality, and  $K = 10$  and  $\alpha = 10$  for the audio modality, are chosen based on the visual and audio F-scores on the validation split. From Table 9, we can see that PLD is less effective in refining our pseudo labels compared to VPLAN's pseudo labels (+1.5 v.s. +2.22 in segment-level metrics and +2.28 v.s. +3.41 in event-level metrics). However, it's worth noting the visual segment-level labels derived from our method before PLD are nearly as accurate as those from VPLAN after PLD (72.34 v.s. 72.51). Although we do implement PLD in the audio modality, no noticeable improvement is recorded for any audio pseudo labels. Referring to Table 10, the model trained with our denoised segment-level labels improves marginally. Nevertheless, we outperform VPLAN on Type@AV and Event@AV F-scores in segment-level and event-level metrics.
E Qualitative Comparison with Previous AVVP Works
Aside from quantitative comparison with previous AVVP works, we perform a qualitative evaluation as well. In Figure 8, we qualitatively compare with the baseline method HAN [73] and the state-of-the-art method JoMoLD [11]. In the top video example, JoMoLD erroneously predicts the "Speech" audio event, while all other methods accurately identify the audio events. In the bottom example, HAN produces identical temporal annotations for the "Speech" event in both modalities, despite the event only occurring audibly. Additionally, our method provides annotations that more closely align with the ground truth than either HAN or JoMoLD does when the events occur intermittently, which is challenging for models to generate accurate predictions.
F More Audio-Visual Event Localization Details
**Baseline Method** We adopt the baseline model HAN to aggregate uni-modal and cross-modal temporal information as we have done in the AVVP task. For brevity, we introduce our baseline method from the procedure after feature aggregation. The segment-level audio features and visual features,  $\hat{\pmb{f}}_t^a$  and  $\hat{\pmb{f}}_t^v (\in \mathbb{R}^d)$ , output from HAN are processed through a 2-layer feed-forward network (FFN) to yield the uni-modal segment-level predictions (logits),  $\pmb{z}_t^a$  and  $\pmb{z}_t^v (\in \mathbb{R}^{(C + 1)})$ , respectively:
$$
\boldsymbol {z} _ {t} ^ {m} = \operatorname {F F N} \left(\hat {\boldsymbol {f}} _ {t} ^ {m}\right), m \in \{a, v \}, \tag {10}
$$
Table 9: PLD refinement. We evaluate the fidelity (F-score) of the segment-level pseudo labels before and after pseudo label denoising (PLD). PLD is less effective in refining our pseudo labels compared to VPLAN's pseudo labels. However, the visual segment-level labels generated from our method before PLD are nearly as accurate as those generated from VPLAN after PLD (72.34 v.s. 72.51). Results are reported on the validation split.
Methods	PLD	Audio	Visual
Seg	Event	Seg	Event
VALOR	X	80.78	71.69	72.34	66.36
VALOR	✓	80.78	71.69	73.84 (+1.5)	68.64 (+2.28)
VPLAN [96]	X	-	-	70.29	64.68
VPLAN [96]	✓	-	-	72.51 (+2.22)	68.09 (+3.41)
Table 10: Results of Training with Denoised Labels. We outperform VPLAN on Type@AV and Event@AV F-scores in segment-level and event-level metrics with and without PLD. Results are reported on the testing split.
Methods	PLD	Segment-level	Event-level
Type	Event	Type	Event
VALOR	X	62.0	61.5	56.7	54.2
VALOR	✓	62.2	61.9	56.6	53.7
VPLAN [96]	X	61.2	59.4	54.7	50.8
VPLAN [96]	✓	62.0	60.1	55.6	51.3
Figure 8: Qualitative Comparison with Previous AVVP Works. In general, the predictions generated by our method VALOR are more accurate than those produced by the other methods. "GT" denotes the ground truth annotations. We compare with HAN [73] and JoMoLD [11].
where  $C + 1$  denotes the number of event classes and the "background" event. Since segment-level labels are not available in the weakly-supervised setting, we simply infer video-level logits  $z \in \mathbb{R}^{C + 1}$  by averaging all logits over time dimension  $t$  and modality dimension  $m$ . Finally, the binary cross-entropy loss is applied to train
the model:
$$
\mathcal {L} _ {\text {v i d e o}} ^ {\text {a v e}} = \operatorname {B C E} (\operatorname {S i g m o i d} (z), y), \quad z = \frac {1}{2 T} \sum_ {t} \sum_ {m} z _ {t} ^ {m} \tag {11}
$$
Harvesting Training Signals The main idea of our method is to leverage large-scale open-vocabulary pre-trained models to provide modality-specific segment-level pseudo labels. We elaborate on how these pseudo labels are generated. Initially, segment-level audio logits and visual logits,  $\pmb{z}_t^{\mathrm{CLAP}}$  and  $\pmb{z}_t^{\mathrm{CLIP}}$  ( $\in \mathbb{R}^C$ ), are generated in a manner identical to the AVVP task. Then, we use two sets of class-dependent thresholds,  $\phi^{\mathrm{CLAP}}$  and  $\phi^{\mathrm{CLIP}}$  ( $\in \mathbb{R}^C$ ), to construct the uni-modal segment-level labels  $\hat{\pmb{y}}_t^a$  and  $\hat{\pmb{y}}_t^v$  ( $\in \mathbb{R}^C$ ), respectively:
$$
\hat {\boldsymbol {y}} _ {t} ^ {m} = \boldsymbol {y} \wedge \left\{\boldsymbol {z} _ {t} ^ {P} > \phi^ {P} \right\}, (m, P) \in \{(v, \text {C L I P}), (a, \text {C L A P}) \} \tag {12}
$$
In addition, we append an additional event "background" to the end of the segment-level labels  $\hat{y}_t^m$  to expand the dimension to  $\mathbb{R}^{C + 1}$ . If  $\hat{y}_t^m$  consists solely of zeros, we assign the last dimension ("background") a value of one; otherwise, we assign it a value of zero. In other words, if an event could possibly occur in a video and the pre-trained model has a certain confidence that the event is present in a specific video segment, that segment will be labeled as containing the event; otherwise, the segment will be labeled as "background". Having prepared the segment-level pseudo labels  $\hat{y}_t^a$  and  $\hat{y}_t^v$ , we compute binary cross-entropy loss in individual modality and combine them to optimize the whole model instead of using the video-level loss  $\mathcal{L}_{\mathrm{video}}^{\mathrm{ave}}$ :
$$
\mathcal {L} _ {\text {A L O R}} ^ {\text {a v e}} = \operatorname {B C E} \left(\operatorname {S i g m o i d} \left(\boldsymbol {z} _ {t} ^ {a}\right), \hat {\boldsymbol {y}} _ {t} ^ {a}\right) + \operatorname {B C E} \left(\operatorname {S i g m o i d} \left(\boldsymbol {z} _ {t} ^ {v}\right), \hat {\boldsymbol {y}} _ {t} ^ {v}\right) \tag {13}
$$
Dataset & Evaluation Metrics The Audio-Visual Event (AVE) Dataset [72] is composed of 4143 10-second video clips from AudioSet [23] that cover 28 real-world event categories, such as human activities, musical instruments, vehicles, and animals. Each clip contains an event and is uniformly split into ten segments. Each segment is annotated with an event category if the event can be detected through both visual and auditory cues; otherwise, the segment is labeled as background. The AVE task is divided into a supervised setting and a weakly-supervised setting. In the former, we can obtain ground truth labels for each segment during training; in the latter, similar to the AVVP task setting, we can only obtain video-level labels. As with the AVVP task, we address the AVE task under the weakly-supervised setting. We follow [72] to split the AVE dataset into training, validation, and testing split and report the results on the testing split. Following the previous work [72], we use the accuracy of segment-level event category predictions as the evaluation metric.
Implementation Details The pre-trained ViT-L CLIP and  $\mathrm{R}(2 + 1)\mathrm{D}$  are used to extract 2D and 3D visual features, respectively, which are then concatenated to represent low-level visual features. The pre-trained HTSAT-RoBERTa-fusion CLAP is used to extract audio features. We adopt the standard HAN model (1-layer and 512-dim) in this task and train the model with AdamW optimizer, configured with  $\beta_{1} = 0.5$ ,  $\beta_{1} = 0.999$  and weight decay set to  $1e - 3$ . A learning rate scheduling of linear warm-up for 10 epochs to the peak learning rate of  $3e - 4$  and cosine annealing decay to the minimum learning rate of  $3e - 6$  is adopted. The batch size and the number of total training epochs are 16 and 120, respectively. We clip the gradient norm at 1.0 during training.