SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Emotion Recognition↑	Vocal Sound Classification↑	Action Recog.↑	Survei Istance.↑
Model	CRE MA-D	RAV DESS	Vocal Sound	ESC50 Actions	SESA
CLAP	0.1784	0.1599	0.4945	0.497	0.7487
Pengi	0.1846	0.2032	0.6035	0.5277	0.5402
Table 3: We used CLAP [15] as a baseline comparison because of its strong performance on a wide range of downstream tasks. The  $\cdot$  symbol indicates numbers were not available, whole  $\mathcal{X}$  indicates that the model cannot support the task. Higher is better for all numbers. The evaluation metric is mAP for FSD50k, AudioSet, ESC50-Actions, and NSynth sonic; F1 score for DCASE17; and SPIDEr for AudioCaps and Clotho captioning. All other downstream tasks use Accuracy.
learning models in Tables 5 and 7 for AQ&A and Audio Captioning respectively. Table 9, against SSL, supervised and trained on speech audio models.
Open-ended tasks. Pengi sets new state-of-the-art performance for open-ended tasks. We used Audio Captioning and AQA for open-ended tasks. The CLAP model can only support close-ended tasks and cannot perform open-ended tasks without additional modules and fine-tuning. Therefore, we compared against supervised trained models in Section 5.2.
Close-ended tasks. Pengi performs better than CLAP on most audio classification tasks, and can also outperform the literature. Although CLAP and Pengi employed different learning methods and used a different amount of training data, it is to be noted that Pengi can compete with strong contrastive methods like CLAP and other methods in the literature.
5.2 Audio Captioning and AQA
Audio Captioning. Pengi's performance outperformed supervised models in the two captioning tasks AudioCaps and Clotho, as shown in Table 7. The captioning competition IEEE DCASE 2022  $^3$  ranks models based on the metric SPIDER, a combination of CIDEr and SPICE. Specifically, for AudioCaps Pengi outperformed the literature by a relative  $6.6\%$  and for Clotho by a relative  $26\%$ . All models used both, AudioCaps and Clotho datasets in training. One of the best captioning models is from Kim et al. [31]. The authors followed a similar training procedure to ours with audio encoders and a language model. Unlike Pengi, which uses a single audio encoder, they employed two mapping networks to capture both global and temporal features from the audio. Despite having two audio representations, the model underperformed our approach.
Similar to Multi-Task Learning [62, 10], we hypothesize that learning a shared audio encoder and mapping networks helps Pengi to solve individual tasks better. We addressed this hypothesis by conducting an ablation study in Table 4. In experiment  $A$ , we trained and evaluated Pengi only on audio captioning data with text prompts of "generate audio caption". Then, we contrasted audio captioning performance against experiment  $B$ , where we trained on data across different tasks, in other words, our proposed setup in this paper. From Table 4, we see consistent improvement in both AudioCaps and Clotho downstream tasks. Specifically, experiment  $B$  outperforms experiment  $A$  by a relative  $2.5\%$  and  $2.3\%$  on AudioCaps and Clotho respectively. This indicates that Pengi's shared architecture does help in improving performance on individual tasks.
Exp.	Eval. dataset	Audio Captioning ↑	Model	Audio Q&A ↑	Model	Retr.	Text-to-Audio Retrieval ↑
\( BLUE_1 \)	\( SPIDEr \)	R@1	R@5	R@10
A	AudioCaps	0.6439	0.4551	\( M_1 \)	0.575	Chen et al.	Clotho	1.5	4.4	7.5
B	AudioCaps	0.6912	0.4667	\( M_2 \)	0.627	Gont. et al.	Clotho	2.1	7.0	12.0
A	Clotho	0.5619	0.2648	\( M_3 \)	0.635	Mei et al.	Clotho	4.0	14.1	21.6
B	Clotho	0.5702	0.2709	Pengi	0.645	Kim et al.	Clotho	7.6	19.6	28.8
Soham et al.	Clotho	16.7	41.0	54.1
Pengi	Clotho	9.4	26.1	36.7
Table 4: Effect of shared audio encoder training
Table 5: AQ&A results
Table 6: T2A Retrieval results
<sup>3</sup>https://dcasecommunity/challenge2022/task-automatic-audio-c captioning
Model	Eval. dataset	\( {\mathrm{{BLUE}}}_{1} \)	\( {\mathrm{{BLUE}}}_{2} \)	\( {\mathrm{{BLUE}}}_{3} \)	\( {\mathrm{{BLUE}}}_{4} \)	METEOR	\( {\mathrm{{ROUGE}}}_{L} \)	CIDEr	SPICE	SPIDEr
Chen et al.	AudioCaps	0.489	0.292	0.178	0.106	0.152	0.346	0.265	0.093	0.179
Gontier et al.	AudioCaps	0.635	0.461	0.322	0.219	0.208	0.450	0.612	0.153	0.383
Mei et al.	AudioCaps	0.682	0.507	0.369	0.266	0.238	0.488	0.701	0.166	0.434
Kim et al.	AudioCaps	0.708	0.547	0.402	0.283	0.238	0.499	0.710	0.167	0.438
Pengi	AudioCaps	0.691	0.419	0.371	0.253	0.232	0.482	0.752	0.182	0.467
Chen et al.	Clotho	0.516	0.325	0.215	0.141	0.153	0.350	0.314	0.102	0.208
Gontier et al.	Clotho	0.461	0.282	0.182	0.117	0.136	0.318	0.251	0.083	0.167
Mei et al.	Clotho	0.516	0.318	0.204	0.127	0.157	0.351	0.313	0.105	0.209
Kim et al.	Clotho	0.539	0.346	0.227	0.142	0.159	0.366	0.319	0.111	0.215
Pengi	Clotho	0.57	0.369	0.242	0.15	0.172	0.375	0.416	0.126	0.271
AQA. Pengi outperformed the existing literature [37]. Authors in [37] collected the only dataset available (ClothoAQA). They converted the AQA task into a classification task, instead of a generation task. Authors trained and fine-tuned a model in a supervised setup. In contrast, we used the free-form text from Pengi, where the answer is correct only when it directly matches the human response. Note that Pengi includes the training set of ClothoAQA among its training sets, but there is no further fine-tuning on this task. The results are shown in Table 5. The first column indicates three different baseline models from [37]. Pengi achieved  $64.5\%$  and outperformed the existing supervised benchmark by a relative  $1.5\%$ .
5.3 Zero-Shot Sound Event Classification
We compared Pengi's classification performance against Zero-Shot contrastive models in the literature. The existing literature restricts the training and evaluation tasks to a few sound event datasets. Hence, we matched our comparisons to sound event datasets. The downstream datasets of ESC50, US8k, DCASE17 Task4 contain audio files and labels not seen by Pengi during training. We considered these three datasets to constitute a zero-shot setup for Pengi. For FSD50k, the audio files in the training split have been used for training Pengi. Hence, we do not consider this a pure zero-shot setup but nonetheless, report numbers for insights.
On Zero-Shot ESC50 performance, Pengi beats AudioCLIP [23], CLAP [15], and LAION CLAP [58] by  $32\%$ ,  $11\%$ , and  $1\%$  respectively (See Table 8). Interestingly, human performance on ESC50 is  $81\%$  accuracy and Pengi's performance is  $92\%$ . Mei et. al. [41] added ChatGPT augmented audio-text pairs to CLAP training [58] and showed an improvement in performance from  $91\%$  to  $94\%$  on ESC50. On US8k, Pengi performed better than Wav2CLIP and AudioCLIP but lower than CLAP and LAION CLAP. Overall, even though Pengi is a text generation model, its Zero-Shot performance on close-ended Sound Event Classification is competitive.
Table 7: Pengi outperforms the best Audio Captioning performance from supervised models. All models used both, AudioCaps and Clotho datasets in training. SPIDER is the metric used to rank models in IEEE DCASE Challenge. Higher is better for all metrics.
Zero-Shot Sound Event Classification ↑
Model	ESC50	FSD50K	US8K	DCASE17 Task 4
Wav2CLIP	0.414	0.030	0.404	-
AudioCLIP	0.694	-	0.653	-
CLAP	0.826	0.302	0.732	0.3
LAION	0.91	-	0.77	-
Pengi	0.92	0.468	0.719	0.338
Table 8: The literature on Zero-Shot audio models only reports performance on Sound Event Classification datasets. Pengi's classification performance is competitive. The  $-$  indicates numbers are not available. The evaluation metric for DCASE17 is the F1 score while FSD50K employs mAP, ESC50 and US8K use Accuracy.
5.4 Text-to-Audio Retrieval
For Text-to-Audio Retrieval in a contrastive learning setup, the user query is converted into a text embedding which is then used to retrieve the top  $k$  audios by their audio embeddings [11, 58]. Pengi is a generative model and does not allow a contrastive setup. Although Pengi has an audio encoder and a text encoder that could replicate the contrastive setup, we wanted to evaluate our model from the generative perspective. First, Pengi is used to index a database by generating audio captions for all the audio recordings. Second, the user text query is matched directly to the dataset captions. The associated audio files of the top  $k$  dataset captions are considered to be the top  $k$  retrieved audio. Note that the cosine similarity computation is between two text embeddings and not audio and text embeddings. Thus, the quality of generated captions for indexing the dataset is important for retrieval performance.
In Table 6, we compared Pengi's Text-to-Audio retrieval performance against the literature. The models used for comparison are audio captioning models using the above-described procedure of indexing and query matching, and not the contrastive-like setup. Pengi outperforms the literature on R@1. However, contrastive models [15],[58], [11] are substantially better than generative models for the task of directly matching text to audio for retrieval. An example of contrastive model performance is shown in Table 6 as a gray row.
5.5 Next text-token prediction for learning audio representations
Pengi uses next-text token prediction to learn audio representations, hence a natural question is: "Can next text-token prediction objective help in learning general purpose audio representations?" To answer this question, we performed linear probe [47] and shallow learning [53] experiments. After Pengi's pre-training, we took the audio encoder  $a_{\phi}$  in Fig 2 and trained one, two, or three fully-connected linear layer(s) with cross-entropy on top. Note that, we kept Pengi's audio encoder frozen and it did not include the mapping network  $m_{1}$ . We selected representative datasets from the domain of Sound Events, Music, and Speech Emotion for the linear probe experiment. Pengi's linear probe (one layer) and shallow learning (two or three layers) numbers are compared against the best single model submissions from the HEAR challenge [53] in Table 9. The results from HEAR challenge reported the maximum of both settings ( $L_{1}$  or  $L_{2}$ ,  $L_{3}$ ). Apart from Wav2vec2 which is trained on speech data, all other models were trained on non-speech audio. Pengi's linear probe  $L_{1}$  and  $L_{3}$  performance is consistently better than CLAP [15]. In the Sound Events and Music domain, Pengi outperformed other models. In the Speech Emotion domain, Pengi performed better than non-speech models but lower than models trained on speech (Wav2vec2). The experiment indicates that the next token prediction does help in learning audio representations useful for various domains.
Model	Sound Events ↑	Music ↑	Speech Emotion ↑
ESC50	FSD50k	GTZAN Genres	Opera	RAVDESS	CREMA-D
YAMNet	0.8375	-	0.847	0.9405	0.479	0.4533
Open L3	0.7505	0.4470	0.879	0.9746	0.604	0.5497
Wav2CLIP	0.7589	0.3617	0.748	0.9363	0.684	0.5116
PaNN	0.9085	-	0.860	0.9112	0.429	0.5550
Wav2Vec2	0.5610	0.3417	0.780	0.9067	-	0.6562
CLAP (L1)	0.8995	0.5024	0.73	0.6399	0.4044	0.2315
CLAP (L3)	0.9310	0.5690	0.8330	0.8263	0.4512	0.2830
Pengi (ZS)	0.9195	0.4676	0.3525	0.6229	0.2032	0.1846
Pengi (L1)	0.8915	0.5608	0.8000	0.9193	0.4774	0.5057
Pengi (L3)	0.9485	0.6235	0.9010	0.9883	0.6108	0.5916
Table 9: Shallow learning experiment where the audio encoder is frozen in all the experiments. ZS is zero-shot and  $L_{i}$  indicates  $i$  linear layers used. Unless specified, each model reports the best of  $L_{1}, L_{2},$  and  $L_{3}$ .
Trade-off between close-ended and open-ended tasks performance. The classification and text generation performance of Pengi is competitive against contrastive models. However, text-based retrieval performance lags behind that of contrastive models [11, 58]. Although these models excel at retrieval, they are limited to close-ended tasks. Thus, there is a trade-off between both types of learning methods proposed so far in the literature.
Limitations inherent to Language Models. Pengi benefits from the encyclopedic knowledge of pre-trained Language Models (LM). However, as pretrained LM is a component of Pengi, they also inherit their limitations. For example, LM are known to hallucinate [28] and specific to Pengi, can produce responses not grounded or conditioned on audio. Similarly, Pengi falls back to LM behavior if no audio is provided or if the audio knowledge is limited. Therefore, the risks of LM, namely propagating stereotypes, and biases and potentially producing offensive language are still applicable to Pengi. The recent works [48, 56] in the NLP field try to address these issues. However, specifically studying risks and limitations can uncover new insights that can accelerate the development of ALMs.
We proposed Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and a text prompt, and generates free-form text as output. Pengi is capable of handling both, close-ended and open-ended audio tasks. We benchmarked Pengi on 21 downstream tasks and show it yields SoTA performance in several of them. Our findings break ground in prompting language models with audio for general-purpose audio understanding.
[58] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. arXiv preprint arXiv:2211.06687, 2022.
[59] L. Yuan, D. Chen, Y.-L. Chen, N. Codella, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
[60] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259, 2016.
[61] A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, 2018.
[62] Y. Zhang and Q. Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586-5609, 2022. doi: 10.1109/TKDE.2021.3070203.
The appendix is organized as follows: In the first few sections (A - C), we describe a set of additional experiments. In Section D, we discuss the contrastive model and its training details. In Section E, we evaluate Pengi's performance with the audio encoder kept frozen. In Section F, we analyze the effect of text encoder on zero-shot performance. In Section G, we analyze and highlight types of Pengi errors. Lastly, in section H, we compare contrastive and generative pretraining.
Figure 4: More examples of audio and text prompt input and their corresponding textual responses. Images are for illustration purposes only.
Figure 5: The user can also add an additional second text input and guide the output of Pengi. For example, the user can add "in the background" after the audio and text prefix and Pengi produces the output "a person is speaking". Compared to Fig 2, the output of Pengi changes to what the user has prompted in the second text input which is about background sounds.
A Additional text input
Pengi takes as input, an audio recording and text, and generates free-form text as output. During inference, an audio encoder  $a_{\phi}$  and a mapping network  $m_{1}$  represent each audio recording as a sequence of continuous embeddings. Similarly, a text encoder  $g_{\phi}$  and a mapping network  $m_{2}$  does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model  $f_{\theta}$ . The language model generates tokens starting from the prefix.
The text input acts as task induction and helps guide the language model to produce the desired output. Let's take an example of human speech recording. A text input of "generate audio caption" will generate a caption like "a person speaking with a car moving in the background", while a text input of "this sentiment is" will produce a response like "negative". However, there are instances where we want to guide the language model further to answer or complete a specific query we had. We can do this by additional text input. This is depicted in Fig 5. The second text input gets tokenized by the frozen language model's tokenizer and converted into continuous embedding by the frozen language model's embedding function. Therefore, the new prefix consists of a sequence of embeddings associated with audio, first text input, and second text input which originates from the audio encoder, text encoder, and frozen language model's embedding function respectively.
Some examples and the effects of the second text input are shown in Fig 6. Empirically, we have seen the additional second input produces meaningful output only when used with text input of "generate metadata". The examples shown in Fig 6 are cherry-picked. The additional text input often causes Pengi to lose track of the audio data and hallucinate its own text or fall back to frozen language model behavior. It is not clear how to ground the output in audio information when additional text input is provided. Further investigation in this direction will enable new scenarios including in-context learning.
Figure 6: Examples of audio-text input with additional text input and their corresponding textual responses. Images are for illustration purposes only. The  $-$  symbol indicates additional text input was not used.
B Inferring audio prefix
The audio encoder and text encoder followed by mapping networks, jointly forms the prefix which prompts the frozen language model. To understand more about Pengi's natural language response, we try to interpret prefixes as a sequence of tokens or words. Each prefix embedding is mapped to the highest similarity token from the GPT2 vocabulary [42]. The similarity method used is cosine similarity. This is possible as the prefix and GPT2 embeddings occupy the same latent space. We
use this method on a few examples from the ESC50 dataset [45]. The examples of Pengi's generated output and the inferred audio prefix are shown in Table 10. The interpretations are hard to follow but do contain salient words that are related to audio content. For example, each inferred audio prefix contains words associated with content of audio like babies, thunder, chicken, etc which also appear in corresponding Pengi's natural language output.
One reason interpreted prefix does not have a clear structure is that the mapping network has to do two things at once - comprehend both the audio and text input and guide the fixed language model. Mokady et. al.[42] observed that the interpreted prefix is more comprehensible when GPT2 is also fine-tuned. A similar method can be followed to infer the text input prefix, but we didn't find any interpretable insights there.
Text output	Inferred audio prefix
a baby is crying loudly and loudly	and, the my's the first the and and fixme the the supern the. coma
the BST in in improvis the babies in in the noises from noises
in the ( the the and innovative for
and- the bigHUD the the the and as[])
the theth P the. weather the close landscape. thunder in- the
Audiostorms interview click in the the and i unsettling,
a thunder claps and then a thunderstorm hits
a rooster is crowing loudly	and at the newone the new the and to OUR the theron the. chickens
theities the in imperson the chickens to in the Audio sitcom.
chickens in the ( the the, Mumbai the
a bird is singing in the background	and, the great bird the first the and and OUR the the number La
the in bird the one great and photography and bird that. in
Audio owl interview singing being: the and I innovative,
C Effect of text prompts
The choice of input text prompt changes Pengi's downstream task performance. We analyze the performance of seven of the input text prompts defined in Section 4.1 for downstream tasks. For some tasks, only specific prompts are applicable, for example, 'question: {}' prompt for AQA and 'this emotion is' for emotion recognition. Pengi's performance on each downstream task corresponding to the different input text prompts is shown in Table 11). In summary, we see that the prompt 'generate metadata' works well on average for close-ended downstream tasks.
Table 10: Examples of Pengi output and their corresponding inferred audio prefix. The input text prompt is "generate audio caption" for all examples. We bold the salient words relating to the input audio and text output.
Text prompts ↑
Downstream Dataset	question: {}	generate audio caption	generate metadata	this is a sound of	this acoustic scene is	this music note is	this emotion is
Clotho Cap.	-	0.2709	-	-	-	-	-
AudioCaps Cap.	-	0.4667	-	-	-	-	-
ClothoAQA	0.6453	-	-	-	-	-	-
ESC50	-	0.8870	0.9195	0.6910	-	-	-
FSD50k	-	0.4676	0.4504	0.4572	-	-	-
US8k	-	0.7185	0.6585	0.5731	-	-	-
DCASE17	-	0.3150	0.3143	0.3506	-	-	-
AudioSet	-	0.1216	0.1230	0.1635	-	-	-
TUT 2017	-	0.2562	0.3525	0.2216	0.1716	-	-
GTZAN Genres	-	0.3230	0.3420	0.3180	-	-	-
GTZAN MS	-	0.9440	0.9606	0.9922	-	-	-
Opera	-	0.2373	0.6229	0.4449	-	-	-
NSynth Instrument	-	-	-	-	-	0.5007	-
NSynth Pitch	-	-	-	-	-	0.8676	-
NSynth Velocity	-	-	-	-	-	0.3728	-
NSynth Qualities	-	-	-	-	-	0.3860	-
RAVDESS	-	-	-	-	-	-	0.1846
CREMAD	-	-	-	-	-	-	0.2032
Vocal Sounds	-	0.5778	0.6035	0.5688	-	-	-
SESA	-	0.5162	0.5402	0.5350	-	-	-
ESC50 Actions	-	0.5277	0.5111	0.4846	-	-	-
Clotho Ret. (T2A)	-	0.0938	-	-	-	-	-
AudioCaps Ret. (T2A)	-	0.1771	0.1407	-	-	-	-
Clotho Ret. (A2T)	-	0.1148	-	-	-	-	-
AudioCaps Ret. (A2T)	-	0.1819	0.1771	-	-	-	-
Table 11: We use different text prompts and observe the performance on downstream tasks. ' -' indicates the prompt is not used. The metrics used for each downstream tasks are same as Table 3.
D Constrastive Learning model details
We follow and train a CLAP [15] model for the choice of contrastive model used in our experiments. We use transformer-based audio and text encoder. The audio encoder is HTSAT [6] and the text encoder is from CLIP [47]. Both the encoders are followed by a linear transformation called the projection layer. We finetune both the encoder and their projection layers. After contrastive training, the audio encoder and text encoder are used in Pengi.
Consider a batch size of  $N$ . Let the audio and text embedding be represented by  $E_{t} \in \mathcal{R}^{N \times d}$  and  $E_{a} \in \mathcal{R}^{N \times d}$ . Then the resulting similarity matrix  $C$  is:
$$
C = \tau \left(E _ {t} \cdot E _ {a} ^ {T}\right) \tag {4}
$$
We use the loss function  $(\mathcal{L})$  of symmetric cross-entropy: projections
$$
\mathcal {L} = 0. 5 \left(\ell_ {\text {t e x t}} (C) + \ell_ {\text {a u d i o}} (C)\right) \tag {5}
$$
where  $\ell_{k} = \frac{1}{N}\sum_{i = 0}^{N}\log diag(\text{softmax}(C))$  along text and audio axis respectively.
implementation details. The audio is sampled at  $44.1\mathrm{kHz}$  and is converted to a log Mel spectrogram with 64 Mel bins, a hop size of 320 secs, and a window size of 1024 secs in the range of  $50 - 8000\mathrm{Hz}$ . We randomly truncate all audio files to 7 seconds in length for HTSAT. All models are trained with Adam Optimiser [32] for 45 epochs with a batch size of 1536 on 20 V100 GPUs. We use a linear schedule with 2000 warmup steps and a base learning rate of 1e-4.
Results. To verify the training, we check our CLAP's performance on the ESC50 dataset. The results are shown in Table 15.
Model	ESC50
Wav2CLIP	0.414
AudioCLIP	0.694
CLAP	0.826
LAION	0.91
CLAP (ours)	0.89
E Frozen audio encoder
The audio encoder  $a_{\phi}$  transforms the raw audio input into an audio embedding. We used the audio transformer backbone from CLAP trained in Section D as our audio encoder in our experiments. In Computer Vision, Visual Language Models [42, 2, 39] use an image encoder from CLIP [47] which is frozen throughout experiments. However, there is a magnitude order difference in data collection of image-text vs audio-text pairs. Therefore, for Pengi we train the audio encoder as well. Nonetheless, we report numbers on Pengi's performance if the audio encoder is kept frozen. The results are shown in Table 13. Frozen Pengi underperforms Pengi across all downstream tasks.