SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\sum_ {i} \log p \left(y ^ {(i)} \mid \boldsymbol {a} ^ {(i)}\right), \tag {6}
$$
where  $y^{(i)}$  is the gold label of  $a^{(i)}$ . For supervised multi-label classification, the classification head estimates the likelihood that an audio clip has some textual label. We thus minimize the per-label binary cross-entropy loss:
$$
\sum_ {i} \sum_ {l} \log p (l = 1 | \boldsymbol {a} ^ {(i)}), \tag {7}
$$
where  $l$  enumerates all possible audio labels.
ESC50 and US8K classification. We initialize the audio encoder from random initialization, CLIP, and  $\mathsf{V}\mathsf{I}\mathsf{P}\sim \mathsf{A}\mathsf{T}$  , respectively. Among them,  $\mathsf{V}\mathsf{I}\mathsf{P}\sim \mathsf{A}\mathsf{T}$  performs best. It surpasses random initialization and CLIP on both datasets (see Table 7). Notably, it outperforms the strong baseline AST-P on ESC50  $(+0.1\%)$  , though AST-P has used gold audio labels for supervised pre-training.
AS classification. We consider balanced and unbalanced training for AS classification and train an individual model on the balanced set and the unbalanced set, respectively. Since the audio encoder has been pre-trained on the unbalanced AudioSet training set, it can be directly used without further fine-tuning. Nevertheless, we fine-tune the last  $k$  layers of the Transformer architecture of  $\mathsf{v}_{\mathsf{IP}}\sim \mathsf{A}_{\mathsf{NT}}$  and investigate whether task-specific fine-tuning helps (see Figure 8). When  $k = 0$  the model is
<sup>9</sup>We find that  $\mathrm{VIP} \sim \mathrm{ANT}$  initialization leads to fast convergence, so it can bring better classification results than other initialization methods with the same number of training epochs.
STAT.	AudioSet	ESC50	US8K	AudioCaps	Clotho
# Train	2041789 (unbalanced)	2000 (5-fold)	8732 (10-fold)	44118 (×1 caption)	3839 (dev-train)
# Dev	22160 (balanced)	1045 (dev-val)
# Val	441 (×5 caption)	1045 (dev-test)
# Test	20371 (balanced)	860 (×5 caption)	1043 (withheld)
# Class	527	50	10	5 captions / audio
Duration	10s	5s	0-4s	10s	15-30s
Task	Multi-label CLF	Multi-class CLF	Multi-class CLF	Captioning	Captioning
Source	YouTube	Freesound	Freesound	YouTube (AudioSet)	Freesound
Table 6: Statistics of the data used in this paper. CLF is the abbreviation of "classification". In AudioSet (Gemmeke et al., 2017) audio clips come from distinct videos. Balanced split means that there are at least 59 samples for each of 527 sound classes. We managed to download 18036 out of 22160 videos in the balanced training split, 16416 out of 20371 videos in the test / evaluation split, and 1715367 out of 2041789 videos in the unbalanced split.
AS Classification
Dataset	AST	AST*	AST†	VIP~ANT
Unbalanced	43.4	44.7
Balanced	34.7	35.8	31.4	37.9
US8K and ESC50 Classification
Dataset	AST-S	AST-P	CLIP	VIP~ANT
US8K	82.5±6.0	86.0±2.8
ESC50	88.7±0.7	95.6±0.4	89.7±1.5	95.7±1.4
Table 7: Multi-label classification mAPs  $(\%)$  on AS and Supervised audio classification accuracies  $(\%)$  on ESC50 and US8K. AST, AST-S, and AST-P indicates the results reported by Gong et al. (2021). We follow their suggestions and test the their best model  $\left(\mathrm{AST}^{\star}\right)$  on our test set. Note that the best model has been trained on the combination of balanced and unbalanced AS training sets.  $\dagger$  indicates that we follow the settings of AST and train it on our data. CLIP and vIP~ANT indicate that the audio encoder is initialized from CLIP and from vIP~ANT, respectively.
basically a linear probe. It inspects if contrastive pre-training learns separable audio representations. As we increase  $k$ , i.e., fine-tuning more layers, the model exhibits a tendency of over-fitting the training set. We use  $k = 4$  as a trade-off between under-fitting and over-fitting. Our model achieves the best mAP of  $37.9\%$  for balanced training, which surpasses AST by  $6.5\%$  (see Table 7). While for unbalanced training, we find it crucial to fine-tune the whole model. Again, our model outperforms AST (+1.4% mAP).
D Position embedding interpolation
Clotho (Drossos et al., 2020) audio has a duration of 15-30 seconds, which is longer than 10-second audio clips used in pre-training. To apply our pre
trained audio encoder to Clotho audio-caption retrieval, we up-sample the pre-trained positional embeddings to account for the longer audio token sequences. Table 8 shows retrieval performance of 10-second audio and 18-second audio. In general, longer audio gives rise to better audio-caption retrieval performance.
E VAT versus AT fine-tuning
Given caption-augmented AudioCaps audio (Kim et al., 2019), we can improve the pre-trained audio encoder via contrastive vision-audio-text (VAT) fine-tuning and contrastive audio-text (AT) fine-tuning. Figure 9 shows a comparison between the two fine-tuning techniques on zero-shot ESC50 classification and AudioCaps audio retrieval. In general, AT fine-tuning results in better results on the two tasks.
F Analyzing tri-modal representations
To better understand the geometry of tri-modal embeddings of our pivoting, unsupervised curation, and supervised curation, we study how AT fine-tuning influences the tri-modal representation space. Specifically, we analyze  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}}$  (pivoting),  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}} + \mathsf{AT}$  (w/ RC) (unsupervised curation), and  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}} + \mathsf{AT}$  (w/ GC) (supervised curation) using pivotability.
Pivotability measures how likely images can pivot audio and text. We quantify it for each aligned VAT triplet via a two-step retrieval probe. Starting at a given audio clip, we retrieve  $k$  nearest image neighbors; for each image neighbor, we retrieve
Model	10-second Clotho (eval)	18-second Clotho (eval)
Text→Audio	Audio→Text	Text→Audio	Audio→Text
R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
Zero-resource	VA-Rand	1.4	7.4	3.2	13.1	1.3	7.5	3.2	13.5
vIP~ANT	1.9	10.1	6.1	23.7	1.9	9.5	7.0	25.6
+AT w/ AC	5.9	26.3	8.2	30.3	6.7	29.1	7.1	30.7
+AT w/ FC	5.7	26.6	6.6	28.0	6.5	27.7	7.8	29.7
+AT w/ VC	5.2	25.2	7.0	25.9	5.5	25.6	7.6	28.2
+AT w/ RC	3.5	16.3	5.7	23.6	3.5	16.9	5.5	24.9
Zero-shot	+AT w/ GL	6.0	27.1	6.1	25.4	6.7	29.0	6.8	27.0
+AT w/ GC	10.2	39.0	10.3	37.2	11.1	40.5	11.8	41.0
Table 8: Interpolating positional embeddings to account for Clotho audios which are longer than 10 seconds.
(a) Zero-shot classification (CLF) accuracy versus level of supervision.
(b) R@1 of audio retrieval with text as the query versus level of supervision.
the top-5 nearest captions. Since each audio clip has 5 gold captions, we compute pivotability as the ratio of the number of retrieved gold captions to 5. A gold caption may be retrieved more than one time, but we always count it as 1, so pivotability is always between 0 and 1.
We conduct this experiment on AudioCaps test set. For each  $k$ , i.e., how many images will be retrieved for a given audio clip, we average pivotability scores over all test triplets (see Figure 10).
Which pairs are pivotable? To study what kinds of audio are more likely to be pivoted with text by images, we set  $k = 5$ , i.e., 5 images will be retrieved for each given audio clip. We consider an AT pair as pivotable if at least 3 out of 5 gold captions of the audio clip are retrieved, i.e., pivotability is equal to or larger than 0.6. Figure 11 illustrates the categories of the audio clips in pivotable AT pairs. Unsurprisingly, audio about speech and vehicle is more pivotable because the two categories are
Figure 9: Comparing VAT and AT fine-tuning on zero-shot ESC50 classification and AudioCaps audio retrieval.
Figure 10: Tri-modal pivotability. +AT (w/ GC) and +AT (w/ RC) indicate that  $\mathsf{v}_{\mathsf{IP}}\sim \mathsf{A}_{\mathsf{NT}}$  is further finetuned on GC and RC, respectively.
among the top three frequent categories in AS. $^{10}$  Given that AT fine-tuning improves Audio  $\rightarrow$  Image retrieval, we wonder if it could also help find novel categories of audio that can be pivoted with
10 Music is the second most frequent category in AS. It is not shown in the figure because AudioCaps excludes all music audio.
Figure 11: Categories of the audio that can be pivoted with text by images. Larger text indicates that the related audio is more likely to be pivoted with text.
text. We find that this is indeed the case (see Table 9). For example,  $\mathsf{vIP}\sim \mathsf{ANT} + \mathsf{AT}$  (w/ GC) finds more fine-grained speech categories because most AT pairs in AudioCaps are about speech. In contrast,  $\mathsf{vIP}\sim \mathsf{ANT} + \mathsf{AT}$  (w/ RC) finds two additional novel insect categories, presumably because RC suffers from less data bias than GC.
+AT w/ GC	‘female speech, woman speaking’, ‘narration, monologue’, ‘vibration’
+AT w/ RC	‘bee, wasp, etc.’, ‘female speech, woman speaking’, ‘insect’, ‘narra-tion, monologue’, ‘vibration’
Table 9: Comparing against  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}}$ , the two fine-tuned versions of  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}}$  find novel audio categories in pivotable AT pairs.
Asymmetric audio-text retrieval performance. For Text  $\rightarrow$  Audio retrieval, our unsupervised pivoting model is not as good as on Audio  $\rightarrow$  Text. This could be because audio is intrinsically more difficult to retrieve with specificity than text in our corpus, e.g., because sound events co-occur (a baby may cry in street with sirens in the background or in a room with dogs barking), there may be a broader range of captions that accurately describe them. However, it could also be the case that AT alignment is bounded by VT alignment because VA pre-training biases audio representations towards image representations. We check this hypothesis by conducting image-text retrieval on AudioCaps. AudioCaps provides aligned image-audio-text triplets, so we simply replace audio with the corresponding image. We find that the Text  $\rightarrow$  Image retrieval performance of CLIP is much better than the Text  $\rightarrow$  Audio retrieval performance of VIP\~ANT (see the OracleAV-CLIP row of Table 3). It is also close
Figure 12: Per-class accuracy on ESC50.
to the Image  $\rightarrow$  Text retrieval performance of CLIP. In contrast,  $\mathsf{V}\mathsf{I}\mathsf{P}\sim \mathsf{A}\mathsf{T}$  exhibits a large gap between the Text  $\rightarrow$  Audio retrieval performance and the Audio  $\rightarrow$  Text retrieval performance.
Per-class accuracy on ESC50 is illustrated in Figure 12.