SlowGuess/NAACL_2022_OCR huggingface dataset:
Yanpeng Zhao\* Jack Hessel Youngjae Yu Ximing LuRowan Zellers Yejin Choi
$\spadesuit$ Institute for Language, Cognition and Computation, University of Edinburgh  
 $\spadesuit$ Paul G. Allen School of Computer Science & Engineering, University of Washington  
 $\diamond$ Allen Institute for Artificial Intelligence
Machines that can represent and describe environmental soundscapes have practical potential, e.g., for audio tagging and captioning. Prevailing learning paradigms of audio-text connections have been relying on parallel audiotext data, which is, however, scarcely available on the web. We propose VIP~ANT that induces Audio-Text alignment without using any parallel audio-text data. Our key idea is to share the image modality between bi-modal imagetext representations and bi-modal image-audio representations; the image modality functions as a pivot and connects audio and text in a trimodal embedding space implicitly.
In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by  $2.2\%$  R@1. We further investigate cases of minimal audio-text supervision, finding that, e.g., just a few hundred supervised audio-text pairs increase the zero-shot audio classification accuracy by  $8\%$  on US8K. However, to match human parity on some zero-shot tasks, our empirical scaling experiments suggest that we would need about  $2^{21} \approx 2\mathrm{M}$  supervised audio-caption pairs. Our work opens up new avenues for learning audio-text connections with little to no parallel audio-text data.
Environmental sound provides rich perspectives on the physical world. For example, if we hear: joyful laughing, a playful scream, and a splash; we not only can visualize literal objects / actions that might have given rise to the audio scene, but also, we can reason about plausible higher-level facets, e.g., a child speeding down a water slide at a water park, splashing through the water (see Figure 1).
Figure 1:  $\mathsf{VIP}\sim \mathsf{ANT}$  pivots audio and text via visual imagination.
Machines capable of parsing, representing, and describing such environmental sound hold practical promise. For example, according to the National Association of the Deaf's captioning guide, accessible audio caption generation systems should go beyond speech recognition (i.e., identifying speakers and transcribing the literal content of their speech) and provide the textual description of all the sound effects, e.g., "a large group of people talking excitedly at a party", in order to provide the full information contained in that audio.
The dominant paradigm for studying machine hearing (Lyon, 2010) has been through human-annotated audio-text data, where text is either free-form audio descriptions (e.g., "the sound of heavy rain") or tagsets (Salamon et al., 2014; Gemmeke et al., 2017; Kim et al., 2019; Drossos et al., 2020). But existing supervised audio-text resources are limited. While some audio-text co-occurrences can be sourced from audio-tag co-occurrences (Font et al., 2013) or from video captioning data (Rohrbach et al., 2015; Xu et al., 2016; Oncescu et al., 2021a), they are either not sufficiently related to environmental sound or limited in their scale and coverage.
$^{1}$ nad.org's captioning guide; Gernsbacher (2015) discusses the benefits of video captions beyond d/Deaf users.
*Work was partially done during an internship at AI2.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4492 - 4507 July 10-15, 2022 ©2022 Association for Computational Linguistics
(video, audio)
WIKIPEDIA The Free Encyclopedia
(image, text)
...it has some fantastic slides with exciting names like The Nucleus, that is an indoor water roller-coaster...
.audio, text)
Figure 2: Video-audio and image-text co-occurrences are abundantly available on the web to support the learning of video-audio alignment and image-text alignment (e.g., via large-scale video-audio and image-text pre-training), but audio-text co-occurrences are not.
People shout in the splashing water.
In this paper, we study large-scale audio-text alignment without paired audio-text (AT) data. Inspired by pivot-based models for unsupervised machine translation (Wu and Wang, 2007; Utiyama and Isahara, 2007), we propose  $\mathsf{VIP}\sim \mathsf{ANT}$ , short for Visually Pivoted Audio and(N) Text.  $\mathsf{VIP}\sim \mathsf{ANT}$  uses images as a pivot modality to connect audio and text. It parallels our motivating example: hearing a sound, humans can visually imagine the associated situation and literally describe it. Pivoting is practically viable because there are abundantly available image-text (VT) and video-audio (VA) co-occurrences on the web, from which bimodal correspondence models can be trained (see Figure 2). By linking audio and text implicitly via the combination of the VT and VA models, we enable zero-resource connection between audio and text, i.e.,  $\mathsf{VIP}\sim \mathsf{ANT}$  can reason about audio-text connections despite never having observed these modalities co-occur explicitly.
We evaluate on zero-shot audio-text retrieval and zero-shot audio classification. On the Clotho caption retrieval task (Drossos et al., 2020), without any parallel AT data,  $\nu_{\mathrm{IP}} \sim \mathsf{A}_{\mathrm{NT}}$  surpasses the supervised state of the art by  $2.2\%$  R@1; on zero-shot audio classification tasks, it establishes new state of the arts, achieving  $57.1\%$  accuracy on ESC50 (Piczak, 2015) and  $44.7\%$  accuracy on US8K (Salamon et al., 2014). We also show that the zero-resource pivoting AT model  $\nu_{\mathrm{IP}} \sim \mathsf{A}_{\mathrm{NT}}$  can
be improved by:
(1) Unsupervised curation: whereby noisy AT pairs are explicitly mined from the pivoting model and serve as additional training data (e.g., +5.7% on ESC50 and +9.3% on US8K); and
(2) Few-shot curation: whereby a small number of human-annotated audio caption pairs are made available at training time (e.g., a few hundred pairs increases the zero-shot audio classification accuracy by  $8\%$  on US8K).
However, for ESC-50, according to the empirical scaling relationship we find, it would require around  $2^{21} \approx 2\mathrm{M}$  aligned audio-text pairs for the zero-shot model to match human parity on ESC50 under our setup, which is an order-of-magnitude more than the largest currently-available audio-text corpus of Kim et al. (2019).
Supervised audio representation learning. While automatic speech recognition has been a core focus of the audio processing community, environment sound classification has emerged as a new challenge and is drawing more attention (Salamon et al., 2014; Piczak, 2015; Gemmeke et al., 2017). Some prior work in learning sound event representations are supervised by category labels (Dai et al., 2017; Boddapati et al., 2017; Kumar et al., 2018; Guzhov et al., 2021b; Gong et al., 2021). Others use weaker forms of supervision for tagging (Kumar and Raj, 2017; Kong et al., 2018) and localization (McFee et al., 2018; Kim and Pardo, 2019).
Learning audio representations from visual imagination. There are two main paradigms for using visual information to derive audio representations. In the two-stage setup, an image encoder is first pre-trained; these weights are used as the initialization of the supervised audio model (Guzhov et al., 2021b; Gong et al., 2021). The other adopts contrastive learning: it exploits the image-audio alignment inherent in videos and learns audio and image / video representations jointly (Korbar et al., 2018; Wang et al., 2021; Nagrani et al., 2021). We use insights from both directions by (1) using CLIP's image encoder, which has been pre-trained on image-text pairs (Radford et al., 2021), to initialize an audio encoder and (2) using contrastive pretraining on image-audio pairs. Throughout training, we do not require any labeled images or audio.
Model	AE Initialization	Objective	AT Supervision	VT Alignment	Zero-shot AT Retrieval
MMV (Alayrac et al., 2020)	Random	Lbi-bi	None	Trainable	X
VATT (Akbari et al., 2021)	Random	Lbi-bi	None	Trainable	X
AudioCLIP (Guzhov et al., 2021a)	ImageNet	Ltri	2M Audio Tags	Trainable	X
Wav2CLIP (Wu et al., 2021)	Random	Lbi-bi	None	Frozen	X
VIP~ANT (ours)	Image CLIP	Lbi-bi	None	Frozen	✓
VIP~ANT +AT (ours)	Image CLIP	Lbi-bi	Caption Curation	Frozen	✓
Table 1: Survey of recent prior work studying for tri-modal (images, audio, and text) representation learning. AE is short for Audio Encoder. Some work experiments with more than one objective, we report the best or the one it advocates. Importantly, we report zero-shot audio-text retrieval between audio and full-sentence text descriptions, along with scaling laws associated with that setup.
Tri-modal learning of audio-text alignment. Our work extends recent work that generalizes the bi-modal contrastive learning to a tri-modal setting (Alayrac et al., 2020; Akbari et al., 2021). While they also connect audio and text implicitly by using images as a pivot, the quality of this audio-text alignment has rarely been studied. To our knowledge, we present the first comprehensive evaluation of the inferred audio-text alignment via zero-shot retrieval / classification.
The work closest to ours are Audio-CLIP (Guzhov et al., 2021a) and Wav2CLIP (Wu et al., 2021). AudioCLIP's pre-training setup is similar to ours, but requires human-annotated textual labels of audio, while ours does not. Wav2CLIP is concurrent with our work; while similar-in-spirit, our model not only performs significantly better, but also, we more closely explore methods for improving audio-text alignment, e.g., unsupervised curation.
Pivot-based alignment models. The pivoting idea for alignment learning can date back to Brown et al. (1991). Language pivots (Wu and Wang, 2007; Utiyama and Isahara, 2007) and image pivots (Specia et al., 2016; Hitschler et al., 2016; Nakayama and Nishida, 2017) have been explored in zero-resource machine translation. Pivot-based models have also been shown to be helpful in learning image-text alignment (Li et al., 2020).
We first formalize tri-modal learning by assuming available co-occurrence data for every pair of modalities (§ 3.1). Then we present bi-bi-modal pre-training as an alternative when there is no paired audio-text data, and implement vIP~ANT via bi-bi-modal pre-training (§ 3.2). Finally, we describe model variants for cases of varying AT
Figure 3: Learning paradigm of  $\mathsf{VIP}\sim \mathsf{ANT}$
supervision (§ 3.3).
3.1 Tri-modal representation learning
Tri-modal representation learning between images, audio, and text aims to derive representations from co-occurrence patterns among the three modalities (Alayrac et al., 2020; Akbari et al., 2021). We consider a simple tri-modal representation space, which relies on encoding functions  $g_V: V \to \mathbf{V}$ ,  $g_A: A \to \mathbf{A}$ , and  $g_T: T \to \mathbf{T}$  to map images  $v$ , audio  $a$ , and text  $t$  ( $v \in V, a \in A$ , and  $t \in T$ ), respectively, to a shared vector space:  $\boldsymbol{v}, \boldsymbol{a}, \boldsymbol{t} \in \mathbb{R}^d$  ( $\boldsymbol{v} \in \mathbf{V}, \boldsymbol{a} \in \mathbf{A}$ , and  $\boldsymbol{t} \in \mathbf{T}$ ). Instead of pre-specifying the precise semantics of this continuous space, vector similarities across modalities are optimized to reconstruct co-occurrence patterns in training corpora, i.e., two vectors should have a higher dot product if they are more likely to co-occur. We use contrastive learning with the InfoNCE loss (Sohn, 2016; van den Oord et al., 2018):
$$
\mathcal {L} _ {c s t} (A, B) =
$$
$$
\sum_ {i} \frac {\exp s (\boldsymbol {a} ^ {(i)} , \boldsymbol {b} ^ {(i)})}{\sum_ {\boldsymbol {a}} \exp s (\boldsymbol {a} , \boldsymbol {b} ^ {(i)})} + \frac {\exp s (\boldsymbol {a} ^ {(i)} , \boldsymbol {b} ^ {(i)})}{\sum_ {\boldsymbol {b}} \exp s (\boldsymbol {a} ^ {(i)} , \boldsymbol {b})}, \tag {1}
$$
where  $A, B$  are two sets of data points from two different modal domains, respectively;  $\pmb{a}^{(i)}, \pmb{b}^{(i)}$
are vector representations of the co-occurring pair  $(a^{(i)},b^{(i)})$  which are encoded by  $g_{A}(a^{(i)})$  and  $g_{B}(b^{(i)})$ , respectively;  $s(\pmb {a},\pmb {b})$  computes the similarity between  $\pmb{a}$  and  $\pmb{b}$ , which we take to be scaled cosine similarity.
If we had access to co-occurrence data between all pairs of modalities, we could optimize the trimodal loss:
$$
\mathcal {L} _ {t r i} (V, A, T) =
$$
$$
\mathcal {L} _ {c s t} (V, A) + \mathcal {L} _ {c s t} (A, T) + \mathcal {L} _ {c s t} (V, T). \tag {2}
$$
3.2 Visually pivoted audio and text
Differently from image-text and image-audio pairs, which are abundantly available on the web, audiotext data is scarce. Instead of Equation 2, in  $\mathsf{VIP}\sim \mathsf{ANT}$ , we consider a "bi-bi-modal" loss, which doesn't require AT data.
$$
\mathcal {L} _ {b i - b i} (V, A, T) = \mathcal {L} _ {c s t} (V, A) + \mathcal {L} _ {c s t} (V, T). \tag {3}
$$
The image encoder is shared between the VA alignment model (i.e.,  $\mathcal{L}_{cst}(V,A)$ ) and the VT alignment model (i.e.,  $\mathcal{L}_{cst}(V,T)$ ) and thus provides a zero-resource connection between audio and text in the tri-modal embedding space implicitly.
3.2.1 Model architecture
Image and text encoders. Instead of learning  $g_V$  and  $g_T$  from scratch, we build on a pre-trained CLIP model, which has been pre-trained on WebImageText (WIT), a dataset of 400 million image-text pairs gathered from the internet (Radford et al., 2021). CLIP has been shown highly performant on VT tasks, e.g., zero-shot image classification. We use the ViT-B/32 model in this work, which consists of a 12-layer vision Transformer (ViT) and a 12-layer language Transformer (Vaswani et al., 2017; Dosovitskiy et al., 2021). Given CLIP's strong VT alignment, we use its image encoder as  $g_V$  and text encoder as  $g_T$ . During learning,  $g_V$  and  $g_T$  are kept frozen and thus the joint VT representation space is untouched (see Figure 3). We minimize only the first loss term of Equation 3:
$$
\min  _ {\Theta_ {A}} \mathcal {L} _ {c s t} (V, A), \tag {4}
$$
where  $\Theta_A$  are the trainable parameters of the audio encoder  $g_{A}$
Figure 4: Left: three-channel image versus one-channel Spectrogram features of audio. We use ViT (Dosovitskiy et al., 2021) to encode images and audio. ViT uses a convolution layer to encode non-overlapped image patches into a sequence of image tokens, but for audio we modify the convolution stride to allow for overlaps between neighbor patches.
Right: adapting the convolution layer of ViT for audio encoding. For simplicity's sake, we omit the output channels of kernel weights and positional embeddings.
Audio encoder. Our audio encoder has the same vision Transformer architecture as CLIP's image encoder (ViT-B/32). In § 4, we show that initializing the audio encoder with CLIP's visual weights significantly improves convergence speed and accuracy. The architectural modifications which enable the use of visual CLIP's architecture for audio are (see Figure 4 for an illustration):<sup>2</sup>
(1) We customize the convolution stride to allow for overlaps between neighbor patches of Spectrogram features of audio.
(2) In the input embedding layer, we average the kernel weights of the convolution layer along the input channel to account for 1-channel Mel-filter bank features of audio (cf. RGB channels of images).
(3) We up-sample the 2-dimensional positional embeddings of image tokens to account for longer audio token sequences.
3.2.2 Bi-bi-modal pre-training details
Video-audio co-occurrences. To optimize Equation 4, we gather VA co-occurrences from AudioSet (AS; Gemmeke et al. (2017)),<sup>3</sup> which contains temporally aligned audio and video frames from 10-second clips gathered from around 2 million YouTube videos. To construct aligned image-audio pairs from AS, we adopt a sparse sampling approach (Lei et al., 2021): we first, extract four
$^{2}$ https://github.com/zhaoyanpeng/vipant
<sup>3</sup>https://github.com/zhaoyanpeng/audioset-dl
Figure 5: Image  $\rightarrow$  Audio retrieval performance per image-audio pre-training epoch, evaluated on the AS balanced training set. "CLIP" and "Rand" indicates that the audio encoder is initialized from CLIP's image encoder and has random initialization, respectively.
equal-spaced video frames from each clip. Then, during training, we randomly sample a frame from the four, and treat it as co-occurring with the corresponding audio clip. At test time, we always use the second video frame as the middle frame to construct image-audio pairs. We use the unbalanced training set, which consists of around 2 million video clips, to pre-train the audio encoder. Since AudioSet does not provide an official validation set, we validate the audio encoder and tune model hyperparameters on the balanced training set.
Audio preprocessing. We use Kaldi (Povey et al., 2011) to create Mel-filter bank features (FBANK) from the raw audio signals. Specifically, we use the Hanning window, 128 triangular Melfrequency bins, and 10 millisecond frameshift. We always use the first audio channel when an audio clip has more than one channel. We apply two normalizations: (1) before applying Kaldi, we subtract the mean from the raw audio signals; and (2) we compute the mean and standard deviation of FBANK on the unbalanced AS training set, and then normalize the FBANK of each audio clip. For data augmentation, inspired by Gong et al. (2021), we use frequency masking and time masking: we randomly mask out one-fifth FBANK along the time dimension and one-forth FBANK along the frequency dimension during training.
Training dynamics. The architecture of our audio encoder follows the vision Transformer of CLIP (ViT-B/32, see Radford et al. (2021) for more details). For the trade-off of efficiency and efficacy, we set the convolution stride to  $16 \times 24$ . This re
sults in around 300 audio tokens for a kernel size of  $32 \times 32$  and an input size of  $1000 \times 128$  (all in the form of time  $\times$  frequency). We optimize the model with LARS (You et al., 2017), where the initial learning rates for model weights and model biases are set to 2e-1 and 4.8e-3, respectively (detailed hyperparameters can be found in Table 5 in Appendix B). We pre-train our model on 4 NVIDIA Quadro RTX 8000 GPUs and for 25 epochs. We empirically set the batch size to 432 to fit the GPU memory. The full pre-training can be done within 24 hours.
Evaluation. We measure the VA pre-training performance by retrieval precision and recall:
$$
\begin{array}{l} p = \frac {\# (\text {r e l e v a n t i t e m s a m o n g t h e r e t r i e v e d})}{\# (\text {r e t r i e v e d i t e m s})}, \\ r = \frac {\# (\text {r e l e v a n t i t e m s a m o n g t h e r e t r i e v e d})}{\# (\text {r e l e v a n t i t e m s})}. \\ \end{array}
$$
Audio is relevant if it has the same set $^4$  of labels as the image query, and vice versa. We average precisions and recalls over all samples in the balanced AS training set. Figure 5 illustrates the top-1 retrieval performance with images as the query (similar trends are observed when using audio as the query). Compared with random initialization, initializing the audio encoder from CLIP's image encoder leads to faster convergence and better VA alignment. As we will see, this performance on VA retrieval transfers to downstream AT tasks.
3.3 Unsupervised and few-shot curation
To improve the AT alignment beyond pivoting, we consider curating audio-text pairs, and then performing an additional fine-tuning step by training the audio encoder with the AT loss, i.e.,  $\mathcal{L}_{cst}(A,T)$ . During AT fine-tuning, we keep the text encoder  $g_{T}$  frozen and only fine-tune the audio encoder.
Unsupervised curation. We consider explicitly mining AT pairs from  $\mathsf{vIP}\sim \mathsf{ANT}$ . Because this zeroresource method uses no human supervision, we refer to it as "unsupervised curation." Concretely, for each video segment in AudioSet, we extract a video frame, and input that frame to the original CLIP image encoder. Then, we encode a large
4Recall that each audio clip in AudioSet is annotated with multiple labels.
<sup>5</sup>Since our goal is to improve AT alignment, we primarily focus on AT fine-tuning; nonetheless, we compare AT fine-tuning to full VAT fine-tuning as in Equation 2 in Appendix E.
Supervised (Zero-resource)	AC
example	Audio-focused Captions originate from the training captions of AudioCaps and Clotho. We perform caption retrieval by using CLIP and the prompt "the sound of". (1080078 aligned pairs)
A balloon is rubbed quickly and slowly to make squeaking sounds.
FC
example	Free Captions are generated by priming GPT-J with MSCOCO captions. We perform caption retrieval by using CLIP and the prompt "a photo of". (1224621 aligned pairs)
The blue colored person is jumping on the white and yellow beach ball.
VC
example	Vision-focused Captions originate from MSCOCO. We perform caption retrieval by using CLIP and the prompt "a photo of". (1172276 aligned pairs)
A sky view looking at a large parachute in the sky.
RC
example	Random Captions indicates that we break the gold AL alignment in AudioCaps by randomly sampling a caption for each audio clip. They are used as a lower bound on the quality of AL alignment. (44118 aligned pairs)
A whoosh sound is heard loudly as a car revs its engines.
GL
example	Gold textual Labels are used to construct AL pairs. (120816 aligned pairs)
Gurgling
GC
example	Gold Captions from AudioCaps provide an upper bound on the quality of AL alignment. (44118 aligned pairs)
Children screaming in the background as the sound of water flowing by.
Table 2: Different ways of curating AT pairs. Gurgling is described as "the bubbling sound of water flowing through a narrow constriction, such as from a bottle with a narrow neck". The example comes from this YouTube video: 1O7-QuhweZE.
set of candidate captions, and perform Image  $\rightarrow$  Text retrieval over them by using the CLIP text encoder. The top candidate captions according to cosine similarity are then paired with the audio that corresponds to the original video clip.
We consider multiple caption sources to search over. As noted by Kim et al. (2019), captions for images and captions for environmental audio are significantly different in focus. We consider two vision-focused caption sets: (1) MSCOCO (Lin et al., 2014) captions (VC); and (2) because MSCOCO captions are limited to 80 object categories, we generate free-captions from GPT-J (Wang and Komatsuzaki, 2021) conditioned on MSCOCO captions as a prompt (FC). We additionally consider audio-focused captions from the training set of AudioCaps (Kim et al., 2019) and Clotho (Drossos et al., 2020) (AC). As a baseline, we also consider a random caption alignment, which assigns a random caption from AC to each clip (instead of pivoting on images). The upper half of Table 2 summarizes different ways of curating AT pairs without additional supervision.
Few-shot curation. We also explore the effect of incorporating limited amounts of AT supervision, specifically, via captions from AudioCaps (GC) and textual labels of AudioCaps (GL) (see the bottom half of Table 2).
4 Audio-text experiments
We use two types of tasks to evaluate the quality of the audio-text alignments learned by our model: AT retrieval and zero-shot audio classification.
AT retrieval. We conduct audio-text retrieval on AudioCaps and Clotho for in-domain evaluation and out-of-domain evaluation, respectively:
(1) AudioCaps (Kim et al., 2019) builds on AudioSet (Gemmeke et al., 2017) and provides captions for a subset of audio clips in AudioSet (sourced from YouTube). As we have pre-trained the audio encoder on AudioSet, we consider audiotext retrieval on AudioCaps as in-domain evaluation.
(2) Clotho (Drossos et al., 2020) consists of audio clips which have a duration of 15-30 seconds and come from Freesound (Font et al., 2013). It has a
<sup>6</sup>We do not use the alignment of these captions — just the captions themselves.