SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Soham Deshmukh<sup>1</sup> Benjamin Elizalde<sup>1</sup> Rita Singh<sup>2</sup> Huaming Wang<sup>1</sup>
<sup>1</sup>Microsoft  <sup>2</sup>Carnegie Mellon University
{sdeshmukh, benjaminm, huawang}@microsoft.com, rsingh@cs.cmu.edu
In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering state-of-the-art performance. However, current models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question Answering. We introduce Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and text, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model. The unified architecture of Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions. When evaluated on 21 downstream tasks, our approach yields state-of-the-art performance in several of them. Our results show that connecting language models with audio models is a major step towards general-purpose audio understanding<sup>1</sup>.
Figure 1: Examples of audio and text prompt inputs and their corresponding textual responses. Images are for illustration purposes only. Our proposed model Pengi enables close-ended tasks, such as classification or retrieval and open-ended tasks, such as captioning or question & answering.
Machine Listening breaks down audio understanding into separate and independent audio tasks. For example, Sound Event and Scene Classification, Audio Retrieval, and Audio Captioning. Because these audio tasks are intrinsically related, we can leverage from Transfer Learning (TL). TL focuses on applying knowledge gained while solving one task to solve a related task. The learning method involves pre-training a model with a large compilation of datasets from different tasks followed by fine-tuning on a target dataset. These models have shown the potential to learn general-purpose audio
$^{1}$ Code is available here: https://github.com/microsoft/Pengi
representations [53] that can successfully be used in a variety of downstream tasks. To leverage from larger amounts of audio that is unlabeled, the community has employed Self-Supervised and Unsupervised Learning [50, 51, 44, 19]. These methods do not require labels [53, 7] and have achieved state-of-the-art performance. However, both methods require an additional fine-tuning step before they can be applied to any downstream task.
To address this drawback, another Transfer Learning (TL) method called Zero-Shot Learning provides direct inference capabilities and removes the need of fine-tuning. These models use contrastive objectives to learn the similarity between natural language descriptions and audio content to provide a score that identifies the most probable class label for a given testing audio. Examples are CLAP [15], Mulan [26], and LAION-CLAP [58]. Despite not seeing the training data of a target task, Zero-Shot models achieve surprising performance in close-ended tasks, such as classification and retrieval. However, these models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question Answering (AQA).
Current audio models that can perform open-ended tasks do not support or have not been evaluated on closed-ended tasks [37, 31]. It is yet to be explored how to leverage TL to enable both types of tasks in the audio domain. We drew inspiration from recent advances in Natural Language Processing (NLP) and Visual Language Models (VLM). In NLP, Raffel et. al. [49] explored a unified framework called T5 where all text-based tasks are framed as text input to text output problems. T5 was trained with a single objective function and supported a diverse set of tasks, like translation, question & answering, and classification. FLAN [8] showed that language models trained on a collection of text tasks phrased as instructions, enabled models to respond better to similar instructions at inference time. This TL technique showed performance improvement across a range of models, prompting setups, and evaluation tasks. On the other hand, VLM incorporates visual information by combining a language model and an image encoder to transfer knowledge across modalities. Tasks are framed as text and image input to text output problems. Captioning training consists of optimizing a text generation objective, and can transfer moderately well to visual question & answering in the zero-shot settings. Examples include, Frozen [52], Flamingo [2], and other models [54, 52, 2, 42, 39]. But their performance on close-ended tasks still lags behind contrastive models [47, 59]. In the audio domain, there are no models that resemble any of these capabilities, let alone that support both close-ended and open-ended audio tasks simultaneously.
In this paper, we introduce Pengi, a novel Audio Language Model (ALM) that takes as input, an audio recording and a text prompt, and generates free-form text as output. To the best of our knowledge, the following contributions are achieved for the first time in the literature:
- A novel Audio Language Model capable of supporting multiple close-ended and open-ended audio tasks without any additional fine-tuning or task-specific extensions of the architecture. Pengi draws inspiration from VLM but tackles intrinsic challenges in the audio domain.
- We propose a new learning framework where we frame all audio tasks as audio and text input to text output tasks. Our framework uses a single training procedure and a captioning objective function. For training, we designed new audio task templates inspired by Instruction Tuning.
- We extensively evaluated Pengi on 21 downstream tasks across various audio domains yielding state-of-the-art performance in several of them. Thus, establishing a baseline for general-purpose ALM.
Audio Language Models. In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques [50, 51, 43, 22, 25, 24, 3, 15, 26, 23, 57, 58, 41, 12, 14, 16]. These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering SoTA performance. However, current models can tackle either close-ended tasks or open-ended tasks. ALM pose a new learning paradigm for audio processing that can support all tasks. The language modeling approaches to audio find utility in generating audio given an input description [4, 1]. But it is yet to be explored how to train them for general-purpose audio understanding and what their performance would be.
Language Models. Transfer Learning has been extensively utilized in Natural Language Processing with the recent shift to Zero-Shot and Few-Shot Learning [29, 48, 5, 55]. The work by Raffel et. al. [49] explored a unified framework for text tasks by converting all text-based tasks into the text-to-text
format. The experimental results showed the methods can achieve SoTA results when combined and scaled. FLAN [55] released in 2022 uses instruction fine-tuning to fine-tune an existing language model on a large set of varied instructions. Pengi adapts a similar idea for the audio domain, where each audio-tasks is considered a text generation task conditional on the input text and input audio. This allows audio tasks to be represented in (audio-text)-text format and enables learning a single unified model for all the tasks. For training, we created (audio-text)-text templates for audio tasks and trained Pengi with them.
Visual Language Models. Inspired by the success of Transfer Learning and Few-Shot Learning in NLP, a host of VLM were proposed for vision tasks. VLM intend to extend the pre-trained language model and adapt them to incorporate visual information. VisualBERT [35] and SimVLM [54] explored different ways to convert images into tokens and jointly train the model on interleaved images and text. Inspired by prefix-tuning [36] and prompt-tuning [34], Frozen [52] and Clipcap [42], use a frozen language model and align the image embeddings for the language model. To better fuse image information, Flamingo [2] uses a gated-cross-attention dense layer in the language model. The interleaved image-text training also enables Flamingo to do few-shot learning. Drawing parallels with VLM, Pengi can be considered an ALM based on audio conditional prefix tuning where the prompt is produced by an audio encoder.
In this section, we describe Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text generation tasks. It takes as input, an audio recording and a text prompt, and generates free-form text as output. The unified architecture in Figure 2 enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions of the architecture.
3.1 Unified Architecture
Figure 2: Pengi has a unified architecture that takes as input, an audio recording and a text prompt, and generates free-form text as output. At training, the architecture learns an audio encoder  $a_{\phi}$  and a mapping network  $m_{1}$  to represent an input audio as a sequence of continuous embeddings. A frozen text encoder  $g_{\psi}$  and a learnable mapping  $m_{2}$  do the same for the corresponding text input. Both sequences are concatenated as a prefix to leverage from a pre-trained frozen autoregressive language model  $f_{\theta}$  to perform multiple tasks. At inference, the language model generates tokens autoregressively conditioned on the audio and text input.
Audio Encoder. The audio encoder  $a_{\phi}$  transforms the raw audio input into an audio embedding. We used the audio transformer backbone from CLAP [15] as our audio encoder due to its success in diverse audio and multimodal tasks. Models in Computer Vision [42, 2, 39] use a frozen image encoder like CLIP, but CLAP is trained on a magnitude smaller collection of audio-text pairs. Therefore, we unfroze its weights for our training procedure.
Text Encoder. The text encoder  $g_{\psi}$  transforms the input text prompt into a text embedding. The prompt can be any form of natural language, such as a task-specific prompt or a question. The text encoder is frozen so its weights are not updated during training. The text encoder can be any off-the-shelf text encoder and allows our architecture to learn and perform well in close-ended tasks.
Mapping Networks and Prefix. To construct the prefix to be fed to the causal language model, we used two mapping networks ( $m_{1}$  and  $m_{2}$ ). The mapping networks [42] convert an embedding into a sequence of  $k$  embeddings. The audio embedding is transformed by  $m_{1}$  and the text embedding by  $m_{2}$ , both are trainable. Both sequences are concatenated to form the fixed-length prefix.
Causal Language Model. To generate the text output we used a pre-trained autoregressive causal language model which is kept frozen during training and inference [52]. Even though the language model is frozen, the audio prefix receives gradients enabling the parameters of mapping network  $(m_{1})$  and audio encoder  $a_{\phi}$  to be optimized with gradient descent and backpropagation. At inference, the language model generates tokens autoregressively conditioned on the audio and text prefix.
3.2 Training and Inference
We propose a new learning framework where we frame all audio tasks as audio and text input to text output tasks. Our framework uses a single training procedure and objective function. Let the training data in audio-text-to-text format be referred to as  $\{x^i,t^i,c^i\}$  where  $x^{i},t^{i}$  and  $c^i$  are the  $i^{th}$  audio file,  $i^{th}$  input text, and  $i^{th}$  output text or caption respectively.
To create a prefix, the audio encoder  $a_{\phi}$  and mapping network  $m_{1}$  projects the audio  $x^{i}$  into a sequence of  $k$  embeddings. Similarly, the text encoder  $g_{\psi}$  and mapping network  $m_{2}$  projects the input text  $t^i$  into a sequence of  $k$  embeddings. Both sequences are concatenated to form prefix  $p^i$  for the pre-trained frozen language model  $f_{\theta}$ .
$$
p ^ {i} = p _ {1} ^ {i}, \dots , p _ {2 k} ^ {i} = \operatorname {c o n c a t} \left\{m _ {1} \left(a _ {\phi} \left(x ^ {i}\right)\right), m _ {2} \left(g _ {\psi} \left(t ^ {i}\right)\right) \right\} \tag {1}
$$
The language model  $f_{\theta}$  is fed with the prefix-caption concatenation of all  $\{z_i\}_{i=1}^N$ , where  $z_i$  is:
$$
z ^ {i} = p _ {1} ^ {i}, \dots , p _ {2 k} ^ {i}, c _ {1} ^ {i}, \dots , c _ {l} ^ {i} \tag {2}
$$
The model is trained as a standard captioning system, where it learns to predict a caption (text tokens)  $c^i$  conditioned on the prefix in an autoregressive fashion. We used Cross-Entropy as the loss function:
$$
\mathcal {L} = - \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {l} \log p _ {\gamma} \left(c _ {j} ^ {i} \mid p _ {1} ^ {i}, \dots , p _ {2 k} ^ {i}, c _ {1} ^ {i}, \dots , c _ {j - 1} ^ {i}\right) \tag {3}
$$
where  $\gamma$  denotes model's trainable parameters which include audio encoder parameters  $\phi$  and parameters from both mapping networks. The text encoder and the causal language model are frozen.
At inference time, the prefix is constructed using the test audio and a text prompt. The causal language model  $f_{\theta}$  generates the next token sequentially conditioned on the prefix. The language model assigns probabilities to all vocabulary tokens at each prediction, which are used to determine the next token depending on the choice of decoding. In our experiments, we used beam search decoding with a beam size of 5 for inference and downstream tasks.
4.1 Training Datasets and Templates
Our Audio Language Model Pengi is trained on a collection of audio-text tasks phrased as instruction templates. The templates are inspired by instruction tuning and enable models to respond better to similar instructions at inference time. This TL technique is novel for audio and yielded performance improvement across a range of input prompting examples and downstream tasks.
The training datasets are modified to adapt to our proposed framework (audio-text)-to-text format by constructing 8 audio-task templates. Before our study, there was no evidence that the templates could lead to good performance across open- and close-ended tasks. Each template consists of audio input, input text prompt, and text output. Examples are "this is the sound of", "this emotion is" or "question: {question}". All the templates are in Table 1, out of which one template is the Auxiliary task "generate metadata". With it, we add audio-text pairs that are not task-specific. Drawing parallels, this training data setup is inspired by instruction tuning format of FLAN [55, 8]. Defining new templates or variations of the ones proposed here is a promising direction to explore.
Task	Input prompt	Output format
Audio Captioning	generate audio caption	{caption}
Audio QA	question: {question}	{answer}
Sound Event Classification	this is a sound of	{event a}, {event b},..
Acoustic Scene Classification	this acoustic scene is	{scene}
Task	Input prompt	Output format
Speech Emotion Recognition	this emotion is	{emotion}
Speech Sentiment Recognition	this sentiment is	{sentiment}
Music Analysis	music analysis	this is a sound of music in language {language} and genre {genre}..
Music Note Analysis	this music note is	produced by {instrument}, pitch {pitch},..
Auxiliary	generate metadata	{metadata}
Table 1: The training datasets are modified to adapt to our proposed framework (audio-text)-to-text format by constructing 8 audio-task templates. Each template consists of audio input, input text prompt, and text output. The  $\{\}$  symbol indicates variable content. The Auxiliary task template allowed us to add audio-text pairs that are not task-specific.
The training data is collected from multiple audio datasets coming from different sources. In all, we collected 3.4 million audio-text pairs and mapped them to the 8 templates. The number of training pairs makes this model one of the largest if not the largest non-speech audio model in literature. We use only the training set of each dataset. The datasets and their mapping to a task are the following. Sound Event Classification: AudioSet [21], FSD50K[20]; Acoustic Scene Classification: CochlScene [27]; Speech Emotion and Sentiment Recognition: MSP Podcast [38], CMU MOSI [60], CMU MOSEI [61], MELD [46]; Music Analysis: NSynth [17], FMA [9]; Audio Captioning: AudioCaps [30], ClothoV2 [13]; Audio Question and Answering: ClothoAQA [37]; Auxiliary: WavText5K [11], SoundDescs [33], MACS [40], WavCaps [41], FreeSound [18] and FindSound $^2$ .
The unified architecture of Pengi enables open-ended tasks and close-ended tasks.
Open-ended tasks. This task type requires free-form text generation and there is flexibility in the correctness of the output. Examples are Audio Captioning and AQ&A. Pengi will take as input the testing audio and the desired prompt to generate the text output. It does not require any additional fine-tuning or task-specific components.
Close-ended tasks. This task type is restricted to predefined values that can be classes or numbers. Examples are classification and retrieval. Pengi will take as input the testing audio and the desired prompt. Ideally, the free-form text output from Pengi should contain the exact predefined value. For example, a predefined class is "dog" but Pengi may output "dog barking" or "canine". Although these answers are reasonable, they are incorrect under most metrics. To evaluate the correctness, we proposed two methods: Log-likelihood and Text matching (Fig. 3). Unless explicitly mentioned, all experiments in our paper use the Text-matching method for evaluation.
Log-likelihood: We take the concatenated prefix from a testing audio, the prompt, and append one of the predefined values (e.g class name, number) to create a candidate output. We would have  $N$  candidate outputs corresponding to  $N$  predefined values. For example in classification, if we have 100 testing audios and 5 classes, we would have 5 output candidates per audio. The outputs and the predefined values are used to compute Log-likelihood scores and determine the model's prediction. This method is expensive for the extensive evaluation in our study.
Figure 3: Text-matching method used during inference for close-ended tasks. TE indicates Text Embedding.
Text-matching: In this setup, the free-form output is matched to the predefined values using text embeddings (Fig.3). For example, in a classification setting, we compute sentence-level text embeddings for Pengi's output and for all the class labels in a given dataset. Then, we calculate cosine similarity to determine the model's prediction. We used Pengi's text encoder to compute the embeddings, but any off-the-shelf text encoder could be used.
$^{2}$ https://www.findsounds.com
Downstream tasks. We used 21 downstream tasks (Table 2) to benchmark the open-ended and close-ended capabilities of Pengi. The open-ended tasks consist of Audio Captioning and AQA. The close-ended tasks consist of classification, regression, and retrieval. Datasets like Clotho have more than one type of annotations, so they are used for multiple tasks like Audio Captioning and Text-to-Audio Retrieval.
Domain	Dataset	Files	Dur. (secs)	Output Type	Metric	Setup
Audio Captioning	Clotho	7k	15 - 30	Cap.	SPIDEr	train/val/test
AudioCaps	39k	10	Cap.	SPIDEr	train/val/test
Audio Question Answering	ClothoAQA	2k	15 - 30	Q&A	ACC	train/val/test
Sound Event Classification	ESC50	2k	5	MC (50)	ACC	5 folds
FSD50K	51k	0.3 - 30	ML (200)	mAP	train/val/test
UrbanSound8K	8k	≤ 4	MC (10)	ACC	10 folds
DCASE2017 Task4	52k	10	MC (17)	ACC	train/val/test
Music Analysis	GT. Music Speech	120	30	B (2)	ACC	10 folds
GT. Music Genre	1k	30	MC (10)	ACC	10 folds
Instrument Classification	Beijing Opera	236	4.77	MC (4)	ACC	5 folds
NS. Instruments	305k	4	MC (11)	ACC	train/val/test
Music Note Analysis	NS. Pitch	305k	4	Reg.	ACC	train/val/test
NS. Velocity	305k	4	MC (11)	ACC	train/val/test
NS. Sonic	305k	4	ML (10)	ACC	train/val/test
Acoustic Scene Classification	TUT 2017	6.3k	10	MC (15)	ACC	train/val/test
Emotion Recognition	CREMA-D	7k	5	MC (6)	ACC	5 folds
RAVDESS	2.5k	≤ 5	MC (8)	ACC	5 folds
Vocal Sound Classification	Vocal Sound	21k	5	MC (6)	ACC	train/val/test
Surveillance	Surveil. Applications	585	≤ 33	MC (6)	ACC	train/val/test
Text-to-Audio Retrieval	Clotho	7k	15 - 30	Ret.	R@1	train/val/test
AudioCaps	39k	10	Ret.	R@1	train/val/test
Table 2: We extensively evaluated Pengi across 21 downstream tasks from various domains. The first two domains are open-ended tasks and the rest are close-ended tasks. For the "Output Type" column, Cap. refers to captioning, MC to multiclass, B indicates binary, Reg. indicates regression, and Ret. retrieval.
Encoders and mappers. We used the audio transformer HTSAT[6] as our audio encoder and CLIP's [47] text encoder. The audio is sampled at  $44.1\mathrm{kHz}$  and is converted to a log Mel spectrograms with 64 Mel bins, a hop size of  $320~\mathrm{ms}$ , and a window size of  $1024~\mathrm{ms}$  in the range of  $50 - 8000\mathrm{Hz}$ . We randomly truncated all audio files to 7 seconds in length for HTSAT. The max length of the text encoder is set to 40 for computational efficiency. We performed another step of CLAP (Contrastive Language-Audio Pretraining) training using the above two encoders [15]. This enables experiments where the audio encoder can be kept frozen to see the utility of CLAP's [15] audio embeddings similar to VLM [42, 52, 2]. The mapping networks  $m_{1}$  and  $m_{2}$  each use an 8-layer transformer with a prefix length of 40. The total prefix length after concatenating the audio and text is 80. The hyper-parameters of the encoders and the CLAP training are mostly left as in the original papers, the details are in Appendix D.
Causal Language Model. We used the GPT2 line of models, specifically GPT2-base (124M). The model is kept frozen through all the experiments.
Pre-training. We used Adam Optimiser [32] for 60 epochs and with a batch size of 384 on 20 V100 GPUs. We used a linear schedule with 2000 warmup steps and a base learning rate of 1e-4.
5.1 Benchmarking Pengi
We assessed Pengi on 21 downstream tasks covering various domains. Pengi is the first audio model that can perform both, open-ended and close-ended tasks. A fair comparison against another model that can perform both is not possible. We chose CLAP [15] as the baseline because it is the only Zero-Shot model with a comprehensive evaluation (16 downstream tasks). The next best evaluation was only on 8 tasks. Thus, providing no evidence of performance across domains like speech and music, which tend to be the most difficult. Moreover, we compared against SoTA results even if it came from different models and learning methods. We compared against SoTa Zero-Shot models in Table 8, a subset of Table 3, for Sound Event Classification. Even against SoTA from supervised
Audio Captioning ↑	AQA ↑	Sound Event Classification ↑
Model	AudioCaps	Clotho	ClothoAQA	ESC50	FSD50K	US8K	DCASE17 Task 4
CLAP	X	X	X	0.826	0.3024	0.7324	0.3
Pengi	0.4667	0.2709	0.6453	0.9195	0.4676	0.7185	0.338
Acoustic Scene Classification↑	Music ↑	Instrument Classification ↑	Music Note Analysis↑
Model	TUT2017	Music Speech	Music Genres	Beijing Opera	Instrument family	NS. Pitch	NS. Velocity	NS. Qualities
CLAP	0.2963	1.0	0.252	0.2963	0.2949	-	-	-
Pengi	0.3525	0.9688	0.3525	0.6229	0.5007	0.8676	0.3728	0.386