SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	AudioCaps	Clotho
Text→Audio	Audio→Text	Text→Audio	Audio→Text
R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
Supervised SoTA	18.0	62.0	21.0	62.7	4.0	25.4	4.8	25.8
Zero-resource	VA-Rand	1.3	7.3	5.6	24.5	1.3	7.5	3.2	13.5
VIP~ANT	0.8	7.9	10.1	38.1	1.9	9.5	7.0	25.6
+AT w/ AC	9.9	45.6	15.2	52.9	6.7	29.1	7.1	30.7
+AT w/ FC	8.9	41.5	14.7	50.0	6.5	27.7	7.8	29.7
+AT w/ VC	6.9	35.7	13.5	49.4	5.5	25.6	7.6	28.2
+AT w/ RC	3.8	19.9	10.7	38.1	3.5	16.9	5.5	24.9
Zero-shot	+AT w/ GL	12.4	52.9	13.0	51.2	6.7	29.0	6.8	27.0
+AT w/ GC	27.7	78.0	34.3	79.7	11.1	40.5	11.8	41.0
OracleAV-CLIP	4.8	27.8	6.6	31.2
Table 3: Audio caption retrieval performance (\%) on AudioCaps test set and Clotho evaluation set. "Supervised SoTA" is from Oncescu et al. (2021b). OracleAV-CLIP: we replace audio with the corresponding image and evaluate image-text retrieval performance of CLIP (Radford et al., 2021). VA-Rand is the same as vIP~ANT but he audio encoder is initialized randomly, instead of from CLIP visual weights. We further fine-tune vIP~ANT on AT data curated in different fashions, e.g., AC, FC, VC, and RC are mined explicitly from the zero-resource pivoting model (see Table 2 for details).
Model	ESC50	US8K	AS
Supervised	95.7±1.4	86.0±2.8	37.9
Zero-resource	VA-Rand	37.6(33.0)	41.9(38.1)	1.7(2.0)
vIP~ANT	57.1(49.9)	44.7(37.8)	2.6(2.8)
+AT w/ AC	62.8(55.7)	54.0(47.0)	11.6(12.3)
+AT w/ FC	62.5(58.0)	52.7(50.0)	11.2(12.2)
+AT w/ VC	61.9(58.0)	52.7(50.3)	8.9(10.7)
+AT w/ RC	51.6(36.1)	42.3(28.5)	4.1(4.6)
Zero-shot	Wav2CLIP	41.4	40.4
+AT w/ GL	67.2(64.5)	62.6(61.0)	15.4(18.9)
+AT w/ GC	69.5(64.2)	71.9(67.1)	13.3(13.6)
AudioCLIP	69.4	65.3
Table 4: Zero-shot audio classification accuracies  $(\%)$  on ESC50 and US8K and mAPs  $(\%)$  on AudioSet (AS). "Supervised" = upper bound performance of vIP~ANT when fine-tuned with supervised audio labels. In the zero-shot / zero-resource settings, we use a prompt 'the sound of' by default (results in parenthesis are without the prompt). "+"AT" = fine-tuned vIP~ANT on AT pairs with different curations. AudioCLIP is pre-trained using the 2 million textual labels of AudioSet; +AT w/ GL and +AT w/ GC are trained with only 44K labels / captions. Wav2CLIP is most directly comparable to our zero-resource pivoting model vIP~ANT with unsupervised curation.
different sound source from AudioCaps and is used for out-of-domain evaluation.
We study the out-of-domain generalizability of our models by applying them to Clotho directly,
without further fine-tuning on it.7
Zero-shot audio classification. We consider the following three widely used datasets for audio classification.
(1) ESC50 (Piczak, 2015) contains 2000 audio clips from 50 classes. Each audio clip has a duration of 5 seconds and a single textual label. We follow the standard  $k$ -fold data splits.
(2) US8K (Salamon et al., 2014) contains 8732 audio clips from 10 classes. Each audio clip has a duration less than 4 seconds and a single textual label. We follow the standard  $k$ -fold data splits.
(3) AudioSet (Gemmeke et al., 2017) is a benchmark dataset for multi-label classification. AudioSet provides balanced and unbalanced training sets. The balanced set consists of 22 thousand audio clips and the unbalanced set contains around 2 million audio clips. It also provides 20 thousand balanced audio clips for evaluation (more data statistics can be found in Table 6 in Appendix A).
For each audio clip  $\pmb{a}$ , we first compute the cosine similarity between it and every possible textual label in the tri-modal representation space. Then we predict the label  $t$  with the highest similarity:
$$
\underset {i} {\arg \max } \cos (\boldsymbol {t} ^ {(i)}, \boldsymbol {a}). \tag {5}
$$
Clotho audio clips (15-30s) are longer than our pretraining audio clips (10s). See Appendix D for adaptation details.
Our prediction results for AT retrieval are given in Table 3 and for zero-shot classification in Table 4 (Appendix F contains qualitative results of the trimodal representations).
Initializing with visual CLIP weights helps. Comparing VA-Rand to  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}}$  we see accuracy increases in all classification and retrieval setups. For example, on AudioCaps,  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}}$  outperforms VA-Rand by  $4.5\%$  R@1 and  $13.6\%$  R@10. This confirms that the findings of Gong et al. (2021) carry-over to unsupervised audio pre-training.
Pivoting works well for Audio  $\rightarrow$  Text. VIP\~ANT exhibits surprisingly strong performance on AT retrieval tasks and zero-shot classification. For example, it outperforms the supervised baseline (Oncescu et al., 2021b) by  $2.2\%$  R@1 for text retrieval, without being trained or fine-tuned on Clotho, and without ever having seen an aligned AT pair.
Prompting (usually) helps. Inspired by the zero-shot image classification setups of CLIP (Radford et al., 2021), we prefix textual labels with a prompt in zero-shot audio classification. We empirically find that the prompt 'the sound of' works well. Using it greatly improves zero-shot multi-class classification accuracy (see Table 4). Take  $\mathsf{v_{IP}}\sim \mathsf{A_{NT}}$ , the prompt gives rise to an improvement of  $7.2\%$  on ESC50 and  $6.9\%$  on US8K, but hurts multi-label classification performance on AS.
Random curation helps. Even when the audio-text pairs used to train that objective are sampled entirely at random (+AT w/ RC),  $\mathsf{V}\mathsf{I}\mathsf{P}\sim \mathsf{A}\mathsf{T}$  improves, e.g., R@1 for Text  $\rightarrow$  Audio retrieval increases from  $0.8\%$  to  $3.8\%$ . We conjecture that RC at least makes audio representations aware of and lean towards the text cluster of the joint VT representation space. While this result also holds for AS classification  $(+1.5\% \mathrm{mAP})$ , performance decreases for ESC50  $(-5.5\%$  accuracy) and US8K  $(-2.4\%$  accuracy).
Figure 6: Per-class accuracy on US8K.
Unsupervised curation is universally helpful.  $\mathsf{VIP}\sim \mathsf{A}\mathsf{T}$  fine-tuned with unsupervised audio captions  $(+\mathrm{AT~w / AC})$  outperforms both pivoting  $(\mathsf{VIP}\sim \mathsf{A}\mathsf{T})$  and random curation  $(+\mathrm{AT~w / RC})$  in all cases. Thus, explicitly mining unsupervised AT pairs can be a helpful zero-resource approach. Performance with automatically generated captions (FC) is similar to captions written by humans (AC).
Supervision is still the most helpful. Finetuning  $\mathrm{VIP} \sim \mathrm{A}_{\mathrm{NT}}$  on GC pairs leads to the highest accuracies on ESC50 and US8K. However, we do not see similar improvements on AS, presumably because multi-label classification is more challenging and requires more direct language supervision, such as audio labels. This is further evident when we fine-tune  $\mathrm{VIP} \sim \mathrm{A}_{\mathrm{NT}}$  on GL and obtain the highest accuracy (18.9% mAP) on AS (see Table 4).
For retrieval, GL uses only audio labels as the text, which provide less dense language supervision than GC and is thus slightly worse than GC, but still, it gives better AT alignment than all automatic methods. As captions become semantically further from the audio-caption domain, e.g.,  $\mathrm{GC} < \mathrm{AC} < \mathrm{FC} < \mathrm{VC}$ , the AT alignment becomes weaker, and thus leading to worse retrieval performance. The fine-tuned audio encoder generalizes to the out-of-domain Clotho successfully, displaying a trend similar to AudioCaps.
Supervision improves per-class accuracy in general. We further plot zero-shot classification accuracy for each audio class (see Figure 6 for US8K and Figure 12 in Appendix G for ESC50). Clearly, language supervision improves per-class accuracy in general. The highest improvement is observed on 'siren' because 'siren' rarely appears in image descriptions while GC contains a lot of textual descriptions of 'vehicle' audio.
Concretely, VA pre-training pushes audio embeddings towards the image cluster (V) of the VT space of the pretrained CLIP, but it does not guarantee that audio embeddings will be as close to the text cluster (T) of the VT space as to V. Random curation provides an estimate of the text-cluster's distributional properties, i.e., the audio embeddings are moved on top of the distribution of the text cluster of the VT space explicitly; surprisingly, this crude "semantic-free" alignment method improves the quality of audio-text alignment.
(a) R@1 of AT retrieval on AudioCaps test set.
(b) Zero-shot classification (CLF) on ESC50 and US8K.
Figure 7: Audio retrieval and zero-shot classification performance versus level of language supervision.
4.2 Level of language supervision
We have observed that AT fine-tuning on AT pairs mined without any additional supervision (e.g., AC, FC, and VC) can improve the AT alignment, but supervised alignments are still the most effective. But: how much supervised data is really needed? To understand the relationship between supervision and performance, we vary the number of gold AT pairs (i.e., training samples of AudioCaps) used for AT fine-tuning. On the audio-text retrieval task (see Figure 7a), unsurprisingly, fine-tuning on more aligned AT pairs results in higher audiotext retrieval / zero-shot classification performance. Surprisingly, using only 442 (around  $1\%$ ) AT pairs of AudioCaps gives rise to as strong AT alignment as VT alignment (cf. OracleAV-CLIP in Table 3).
As we increase the number of supervised AT pairs used during fine-tuning, we observe a roughly linear relationship between zero-shot performance and the log of the number of supervised pairs (this observation is similar to Kaplan et al. (2020)'s observations regarding Transformers). While it is not clear how reliable extrapolations from this roughly linear trend are, we roughly estimate the amount of annotated AT pairs required for the zero-shot performance to equal human parity for ESC50 of  $81\%$  (Piczak, 2015): our estimate is that  $2^{21} \approx 2\mathrm{M}$  supervised audio caption pairs would be needed. We are hopeful both (1) that larger curated audiotext datasets will become available; and (2) that future work can improve the data efficiency of the pre-training process.
We have presented  $\mathsf{VIP}\sim \mathsf{ANT}$  for unsupervised audio-text alignment induction. Based on the pivoting idea, our model learns image-text alignment
and image-audio alignment explicitly and separately via bi-modal contrastive pre-training. The image modality is shared between the two and thus pivots audio and text in the tri-modal embedding space implicitly, without using any paired audio-text data. We empirically find that our model achieves strong performance on zero-shot audiotext tasks. We further strengthen the audio-text alignment by using varying kinds of audio-text supervision. Experimental results show that even un-aligned audio-caption pairs can help.
We would like to thank the AI2 Mosaic team for discussions, the AI2 Beaker team for computing support, and the anonymous reviewers for their suggestions. Yanpeng would like to thank Ivan Titov for his comments on the draft. The work was partially supported by the European Research Council (ERC Starting Grant BroadSem 678254), the Dutch National Science Foundation (NWO VIDI 639.022.518), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), DARPA SemaFor program, and Google Cloud Compute.
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. In Thirty-Fifth Conference on Neural Information Processing Systems.
Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. 2020. Self-supervised multimodal versatile networks. In Advances in Neural
Information Processing Systems, volume 33, pages 25-37. Curran Associates, Inc.
Venkatesh Boddapati, Andrej Petef, Jim Rasmusson, and Lars Lundberg. 2017. Classifying environmental sounds using image recognition networks. Procedia Computer Science, 112:2048-2056. Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 21st International Conference, KES-20176-8 September 2017, Marseille, France.
Peter F. Brown, Jennifer C. Lai, and Robert L. Mercer. 1991. Aligning sentences in parallel corpora. In 29th Annual Meeting of the Association for Computational Linguistics, pages 169-176, Berkeley, California, USA. Association for Computational Linguistics.
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. 2017. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 421-425.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Cloth: an audio captioning dataset. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736-740.
Frederic Font, Gerard Roma, and Xavier Serra. 2013. Freesound technical demo. In Proceedings of the 21st ACM International Conference on Multimedia, MM '13, page 411-412, New York, NY, USA. Association for Computing Machinery.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780.
Morton Ann Gernsbacher. 2015. Video captions benefit everyone. Policy Insights from the Behavioral and Brain Sciences, 2(1):195-202. PMID: 28066803.
Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571-575.
Andrey Guzhov, Federico Raue, Jorn Hees, and Andreas Dengel. 2021a. Audioclip: Extending CLIP to image, text and audio. CoRR, abs/2106.13043.
Andrey Guzhov, Federico Raue, Jorn Hees, and Andreas Dengel. 2021b. Esresnet: Environmental sound classification based on visual domain models. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4933-4940.
Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal pivots for image caption translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2399-2409, Berlin, Germany. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08361.
Bongjun Kim and Bryan Pardo. 2019. Sound event detection using point-labeled data. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 1-5.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. AudioCaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 119-132, Minneapolis, Minnesota. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Qiuqiang Kong, Yong Xu, Wenwu Wang, and Mark D. Plumbley. 2018. Audio set classification with attention model: A probabilistic perspective. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 316-320.
Bruno Korbar, Du Tran, and Lorenzo Torresani. 2018. Cooperative learning of audio and video models from self-supervised synchronization. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Anurag Kumar, Maksim Khadkevich, and Christian Fügen. 2018. Knowledge transfer from weakly labeled audio using convolutional neural network for sound events and scenes. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 326-330.
Anurag Kumar and Bhiksha Raj. 2017. Audio event and scene recognition: A unified approach using strongly and weakly labeled data. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 3475-3482.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7327-7337.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision - ECCV 2020, pages 121-137, Cham. Springer International Publishing.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and Larry Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV. European Conference on Computer Vision.
Richard F. Lyon. 2010. Machine hearing: An emerging field [exploratory tsp]. IEEE Signal Processing Magazine, 27(5):131-139.
Brian McFee, Justin Salamon, and Juan Pablo Bello. 2018. Adaptive pooling operators for weakly labeled sound event detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(11):2180-2193.
Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021. Attention bottlenecks for multimodal fusion. In Advances in Neural Information Processing Systems.
Hideki Nakayama and Noriki Nishida. 2017. Zeroresource machine translation by multimodal encoder-decoder network with multimedia pivot. Machine Translation, 31(1/2):49-64.
Andreea-Maria Oncescu, João F. Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. 2021a. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2265-2269.
Andreea-Maria Oncescu, A. Sophia Koepke, João F. Henriques, Zeynep Akata, and Samuel Albanie. 2021b. Audio Retrieval with Natural Language Queries. In Proc. Interspeech 2021, pages 2411-2415.
Karol J. Piczak. 2015. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pages 1015-1018. ACM Press.
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech
Recognition and Understanding. IEEE Signal Processing Society. IEEE Catalog No.: CFP11SRW-USB.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015. A dataset for movie description. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3202-3212.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM International Conference on Multimedia, MM '14, page 1041-1044, New York, NY, USA. Association for Computing Machinery.
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Lucia Specia, Stella Frank, Khalil Sima'an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 543-553, Berlin, Germany. Association for Computational Linguistics.
Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 484-491, Rochester, New York. Association for Computational Linguistics.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Luyu Wang, Pauline Luc, Adrià Recasens, Jean-Baptiste Alayrac, and Aïron van den Oord. 2021. Multimodal self-supervised learning of general audio representations. CoRR, abs/2104.12807.
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. 2021. Wav2clip: Learning robust audio representations from CLIP. CoRR, abs/2110.11499.
Hua Wu and Haifeng Wang. 2007. Pivot language approach for phrase-based statistical machine translation. Machine Translation, 21(3):165-181.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msrvtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288-5296.
Yang You, Igor Gitman, and Boris Ginsburg. 2017. Scaling SGD batch size to 32k forImagenet training. CoRR, abs/1708.03888.
This supplementary material includes (1) data statistics (\$A), (2) hyperparameters of optimizers (\$B), (3) supervised audio classification (\$C), (4) interpolating pre-trained position embeddings for Clotho audio-caption retrieval (\$D), (5) comparison between VAT fine-tuning and AT fine-tuning (\$E), (6) a qualitative study of the geometry of the tri-modal embedding space (\$F), and (7) additional findings from the audio-text retrieval task (\$G).
A Data statistics
Table 6 presents data statistics of all the datasets used in the paper.
B Optimizer hyperparameters
Table 5 presents optimizer hyperparameters used in our learning tasks.
Hyperparam.	VA	AT	ESC50	US8K
Optimizer	LARS (You et al., 2017)
Batch size	432	64	50
Weight decay	1e-6
LR of weight	2e-1	1e0
LR of bias	4.8e-3	2.4e-2
Warmup epoch	10
Training epoch	25	50
Hyperparam.	AS balanced	AS unbalanced
Optimizer	Adam (Kingma and Ba, 2015)
Batch size	12	128
Weight decay	1e-7
Learning rate	5e-5
Warmup step	1000
Training epoch	25	5
LR scheduler	MultiStepLR (γ = 0.5)
Table 5: Hyperparameters of the optimizers used for VA pre-training, AL fine-tuning, ESC50 classification, US8K classification, balanced AS classification, and unbalanced AS classification. The learning rate (LR) in balanced AS classification is scheduled by epoch: 5, 9, 10, 11, 12 epochs. In unbalanced AS classification it is scheduled by optimization step: 7.5, 15, 20, 25, 35, 40, 45, 50 thousand steps.
C Supervised audio classification
To perform supervised audio classification, we add a classification head (a linear layer) on top of the pre-trained audio encoder. For multi-class classification, the classification head projects the vector representation of an audio clip onto the class space.
Figure 8: Fine-tuning last  $k = 0,2,4,6,8$  layers of the pre-trained audio encoder for AS classification. mAP is measured on the AS balanced training set per fine-tuning epoch.
We fine-tune the model by minimizing the cross-entropy loss: