Pengi: An Audio Language Model for Audio Tasks
Authors: Soham Deshmukh
Benjamin Elizalde
Rita Singh
Huaming Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: audio language model
audio representation learning
audio and speech processing
multi-task and transfer learning
TL;DR: Tackling multiple audio tasks with a single Audio Language Model called Pengi

Abstract: In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering state-of-the-art performance. However, current models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question Answering. We introduce Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and text, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model. The unified architecture of Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions. When evaluated on 21 downstream tasks, our approach yields state-of-the-art performance in several of them. Our results show that connecting language models with audio models is a major step towards general-purpose audio understanding.

Pengi: An Audio Language Model for Audio Tasks
Soham Deshmukh' Benjamin Elizalde'
Rita Singh²
Huaming Wang
Miicoosoft Carnegie Mellon University
(sdeshmukh, benjaminm, huawang)mmicroes com, rsinghues. cmu edu
Abstract
In the domain of audio processing. Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches have led to the development of versatile models capable of tackling wide array of tasks, while delivering state-of-the-art performance. However, current models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question Answering. We introduce Pengi, novel Audio Language Model that leverages Transfer Learning by framing all audio tasks text-generation tasks. It takes as input, audio recording, and text, and generates free-form text as output. The input audio represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt pre-trained frozen language model. The unified architecture Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning task-specific extensions. When evaluated on 21 downstream tasks, our approach yields state-of-the-ar performance in several them. Our results show that connecting language models with audio models is major step towards general cenerra--urppse purpose audio understanding
Audio input
Text input
Text output
this is sound
train, railway and locomotive
4:4+
this emotion is
happy
the of the crash onto the shore then recede
444++
generate audio caption
question: what type of animal is making light sound background?
bird
Figure 1: Examples of audio and text prompt inputs and their corresponding textual responses. Images for illustration purposes only. Our proposed model Pengi enables close-ended tasks, such as classification or retrieval and open-ended tasks, such as captioning or question & answering.
Introduction
Machine Listening breaks down audio understanding into separate and independent audio tasks. For example, Sound Event and Scene Classification, Audio Retrieval, and Audio Captioning. Because these audio tasks are intrinsically related, we can leverage from Transfer Learning (TL). TL focuses on applying knowledge gained while solving one task solve a related task. The learning method involves pre-training a model with a large compilation of datasets from different tasks followed by fine-tuning target dataset. These models have shown the potential to learn general-purpose audio
'Code available here:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

representations [53] that can successfully be used in variety of downstream tasks. To leverage from larger amounts of audio that is unlabeled, the community has employed Self-Suuer and
Unsupervised Learning 50 51 44
19]. These methods do not require labels [53 7 and have
achieved state-of-the-an performance. However, both methods require an additional fine-tuning step before they can be applied to any downstream task.
To address this drawback, another Transfer Learning (TL) method called Zero-Shot Learning provides direct inference capabilities and removes the need of fine-tuning, These models use contrastive objectives to learn the similarity between natural language descriptions and audio content to provide a score that identifies the most probable class label for given testing audio. Examples are CLAP [15], Mulan [26). and LAION-CLAL [58]. Despite not seeing the training data of a target task, Zero-Shi Shot models achieve surprising performance in close-ended tasks, such as classification and retrieval. However, these models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question Answering (AQA).
Current audio models that can perform open-ended tasks do not support or have not been evaluated on closed-ended tasks (37 31.. It is yet to be explored how to leverage TL to enable both types of tasks in the audio domain. We drew inspiration from recent advances in Natural Language Processing (NLP) and Visual Language Models (VLM). In NLP, Raffel et. al. 49 explored a unified framework called T5 where all text-based tasks are framed as text input to text output problems. T5 was trained with a single objective function and supported diverse set of tasks, like translation, question & answering, and classification. FLAN g showed that language models trained on collection of text tasks phrased as instructions, enabled models to respond better to similar instructions at inference time. This TL. technique showed performance improvement across a range of models, prompting setups, and evaluation tasks. On the other hand, VLM incorporates visual information by combining a language model and an image encoder to transfer knowledge across modalities. Tasks are framed as text and image input to text output problems. Captioning training consists of optimizing a text generation objective, and can transfer moderately well to visual question & answering in the zero-shot settings. Examples include, Frozen [52), Flamingo 22. and other models [54, 52 2 42 [3]] But their performance on close-ended tasks still lags behind contrastive models [47 59]. In the audio domain,
there are no models that resemble any of these capabilities, let alone that support both close-ended and open-ended audio tasks simultancously.
In this paper, we introduce Pengi, a novel Audio Language Model (ALM) that takes as input, an audio recording and a text prompt, and generates free-form text as output. To the best of our knowledge, the following contributions are achieved for the first time in the literature:
A novel Audio Language Model capable of supporting multiple close-ended and open-ended audio tasks without any additional fine-tuning or -peccicc extensions of the architecture. Pengi draws inspiration from VLM but tackles intrinsic challenges in the audio domain.
We propose a new learning framework where we frame all audio tasks as audio and text input to text output tasks. Our framework uses a single training procedure and a caption- ing objective function. For training, we designed new audio task templates inspired by Instruction Tuning.
We extensively evaluated Pengi on 21 downstream tasks across various audio domains yielding state-of-the-art performance in several of them. Thus, establishing a bascline for general-purpose ALM.
2 Related Work
Audio Language Models. In the domain of audio processing, Transfer I Learning has facilitated the
rise of Self- Supervised Learning and Zero-Shot Learning techniques (50, [1 43 22 25 24 3 [5]
26 23
14 16). These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering SoTA performance. However, current models can tackle either close-ended tasks open-ended tasks. ALM pose new learning paradigm for audio processing that can support all tasks. The language modeling approaches to audio find utility in generating audio given an input description 44 []. But it is yet to be explored how to train them for general-purposs audio understanding and what their performance would be.
57 58
41
12
Language Models. Transfer Learning has been extensively utilized in Natural Language Processing with the recent shift to Zero-Shot and Few-Shot Learning (29 48 5 (5) The work by Raffel et. al. 49 explored a unified framework for text tasks by converting all baxed tasks into the text-to-text
2

---

Causal Language Model / format. The experimental results showed the methods can achieve SoTA results when combined and scaled. FLAN [55] released in 2022 uses instruction fine-tuning to fine-tune an existing language model on a large set of varied instructions. Pengi adapts similar idea for the audio domain, where each audio-tasks is considered a text generation task conditional on the input text and input audio. This allows audio tasks to be represented in (audio-text)-tex) format and enables learning a single unified model for all the tasks. For training, we created (audio-tex()-tex templates for audio tasks and trained Pengi with them.
Visual Language Models. Inspired by the success of Transfer Learning and Few-Shot Learning in NLP, host of VLM were proposed for vision tasks. VLM intend to extend the pre-trained language model and adapt them to incrporate visual information. VisualBERT [35] and SimVLM
54 explored different ways to convert images into tokens and jointly train the model on interleaved
images and text. Inspired by -refi--tuning (36) and prompt-tuning 34 Frozen [52] and Clipeap 42].
use a frozen language model and align the image embeddings for the language model. To better fuse image information, Flamingo uses a sated-croos-attentiin dense layer in the language model. The interleaved image-text training also enables Flamingo to do few-shot learning. Drawing parallels with VLM, Pengi can considered an ALM based on audio conditional prefix tuning where the prompt is produced by an audio encoder.
3 Approach
In this section, we describe Pengi, novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text generation tasks. takes as input, an audio recording and a text prompt, and generates free-form text as output. The unified architecture in Figure 2 enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions of the architecture.
3.1 Unified Architecture
the waves of the
ocean
the waves of the
ocean
Audio Encoder
Text & Encoder
generate audio caption
Figure 2: O Pengi has unified architecture that takes input, audio recording and text prompt, and generates free-form text as output. training, the architecture learns audio encoder ag and mapping network mh to represent an input audio as sequence of continuous embeddings. frozen text encoder gy and a learnable mapping m2 do the same for the corresponding text input. Both sequences are concatenated as a prefix to leverage from pre-trained frozen autoregressive language model fa to perform multiple tasks. At inference, the language model generates tokens autoregrossively conditioned on the audio and text input.
Audio Encoder. The audio encoder ag transforms the raw audio input into an audio embedding. We used the audio transformer backbone from CLAP [15 as our audio encoder due its success in diverse audio and multimodal tasks. Models in Computer Vision 42 2 39 use frozen image encoder like CLIP, but CLAP is trained on a magnitude smaller collection of audio-text pairs. Therefore, unfroze its weights for our training procedure.

---

Text Encoder. The text encoder Pe transforms the input text prompt into a text embedding. The prompt can be any form of natural language, such as task-specific prompt or a question. The text encoder is frozen so its weights are not updated during training. The text encoder can be any off-the-shelf text encoder and allows our architecture to learn and perform well in close-ended tasks.
Mapping Networks and Prefix. To construct the prefix to be fed to the causal language model, we used two mapping networks (m and m2). The mapping networks 42 convert an embedding into sequence of A embeddings. The audio embedding is transformed by my and the text embedding by m2, both are trainable. Both sequences are concatenated to form the fixed-length prefix.
Causal Language Model. To generate the text output we used a pre- are-trained autoregressive causal language model which is kept frozen during training and inference [52]. Even though the language model is frozen, the audio prefix receives gradients enabling the parameters of mapping network (m1) and audio encoder 0g to be optimized with gradient descent and backpropagation. At inference, the language model generates tokens autoregressively conditioned on the audio and text prefix.
3.2 Training and Inference
We propose a new learning framework where we frame all audio tasks as audio and text input to text output tasks. Our framework uses single training procedure and objective function. Let the training data in audio-text-to-text format be referred to as (x", (,,,'',)) c,)) where x², l' and et are the jth audio file, 7th input text, and 7th output text or caption respectively.
To create a prefix, the audio encoder ag and mapping network mi projects the audio of into a sequence of & embeddings. Similarly, the text encoder go and mapping network m2 projects the input text e into a sequence of & embeddings. Both sequences are concatenated to form prefix pt for the pre-trained frozen language model fa.
p² pi..... Pbe = p
(1)
The language model fo fed with the
concatenation of all { where 2, is:
pl. ..... PIR ci...., of
(2)
The model is trained as a standard captioning system, where it learns to predict a caption (text tokens) et conditioned on the prefix in an autoregressive fashion. We used Eotss-y as the loss function:
N log Py PZx, (.......-)))
C
where "y denotes model's trainable parameters which include audio encoder parameters 6 and parameters from both mapping networks. The text encoder and the causal language model are frozen.
At inference time, the prefix constructed using the test audio and a text prompt. The causal language model fo generates the next token sequentially conditioned on the prefix. The language model assigns probabilities to all vocabulary tokens at each prediction, which are used determine the next token depending the choice of decoding. In our experiments, we used beam search decoding with beam size of for inference and downstream tasks.
4 Experiments
4.1 Training Datasets and Templates
Our Audio Language Model Pengi is trained on a collection of audio-text tasks phrased as instruction templates. The templates are inspired by instruction tuning and enable models to respond better to similar instructions at inference time. This TL technique is novel for audio and yielded performance improvement across range of input prompting examples and downstream tasks.
The training datasets are modified to adapt to our proposed framework (audio-text)-to- constructing 8 audio-task templates. Before our study, there was no evidence that the templates could lead to good performance across open- and close-ended tasks. Each template consists of audio input, input text prompt, and text output. Examples are "this is the sound of", "this emotion is" or "question: (question)" J". All the templates are T out of which one template the Auxiliary task "generate metadata". With it, we add audio-text pairs that are not task-specific. Drawing parallels, this training data setup is inspired by instruction tuning format of FLAN (55 g. Defining new templates or variations of the ones proposed here is promising direction to explore.
format by

---

Tisk
Input prompt
Output formst
Audio
Tisk
Inpul prompt
Oulput fornat
Speech Emnotion
Recognitive
this emotion is
[emotioon]
Captioning
generate sadio caption
[esption]
Speech Sentiment
this sentiment is music analysis
[sentiment]
Audio
question: (question) this sound of this acoustic scene is [ccene]
(answer)
Recognition
Sound Event
Music Analysis
in langugee (langunge) and geare (genre)
music
Classificution
[event a),
fevent b)...
Acoustic Scene
Classification
Note Analysis
Music
this music nete is
produced by [insitument]] pitch [picch)
Auxillary
generate metadata
[metadiata]
Table I: The training datasets are modified to adapt to our proposed framework (audio-text)-tor constructing 8 audio-task templates. Each template consists of audio input, input text prompt, and text output. The [] symbol indicates variable content. The Auxiliary task template allowed us to add audio-text pairs that not task-specific.
format by
The training data is collected from multiple audio datasets coming from different sources. In all, we collected 3.4 million audio-text pairs and mapped them to the 8 templates. The number of training pairs makes this model one of the largest if not the largest non-speech audio model in literature. We use only the training set of each dataset. The datasets and their mapping to task are the following. Sound Event Classification: AudioSet (21), ESD50K[20) Acoustic Scene Classification: CochlScene 271: Speech Emotion and Sentiment Recognition: MSP Podcast [38], CMU MOSI [60]. CMU MOSEI [6]] MELD | Music Analysis: NSynth (17), FMA 99: Audio Captioning: AudioCaps
B0,, V2 [3; Audio Question and Answering: ClothoAQA [77: Auxiliary: WayTexx5K (
SoundDescs
33 MACS 40)
WavCaps
41 FreeSound 18 and FindSound
4.2
Downstream Tasks
The unified architecture of Pengi enables open-ended tasks and close-ended tasks.
Open-ended tasks. This task type requires free-form text generation and there is flexibility in the correctness of the output. Examples are Audio Captioning and AQ&A. Pengi will take as input the testing audio and the desired prompt generate the text output. It does not require any additional fine-tuning or task-specific components.
Close-ended tasks. This task type is restricted to predefined values that can be classes or numbers. Examples are classification and retrieval. Pengi will take as input the testing audio and the desired prompt. Ideally, the free-form text output from Pengi should contain the exact predefined value. For example, a predefined class is "dog" but Pengi may output "dog barking" or "canine". Although these answers are reasonable, they are incorrect under most metrics. To evaluate the correctness, proposed two methods: Log-liken Lig-liielihood and Text matching (Fig. (3). Unless explicitly mentioned, all experiments in our paper use the Text-matching method for evaluation.
Log-likelihod:: We take the concatenated prefix from testing audio, the prompt, and append one of the predefined values (c.g class name, number) to create candidate output. We would have N candidate outputs corresponding to V predefined values. For example in classification, we have 100 testing audios and 5 classes, we would have 5 output candidates per audio. The outputs and the predefined values are used to compute lokeliielioood scores and determine the model's prediction. This method is expen- sive for the extensive evaluation in our study.
Rhythmic crashing of the beach
TE
Cosine smilaniyy
i Pengi
TE
Clasues
Dog ba Maniinn Gurchal See
generate audio caption
Figure 3: Text-matching method used during inference for close-ended tasks. TE indicates Text Embedding.
Text matthiathhin:: In this setup, the free-form output is matched to the predefined values using text em- beddings (Fig For example, in classification setting, we compute sentence-level text embeddings for Pengi's 's output and for all the class labels in a given dataset. Then, we calculate cosine similarity to determine the model's 's prediction. We used Pengi's 's text encoder to compute the embeddings, but any off-the-sher text encoder could be used.
htttps://wee http://wwwwwiidsuuud...oo
5

---

Downstream tasks. We used 21 downstream tasks (Table 2 to benchmark the open- epen-ended and close-ended capabilities of Pengi. The open-ended tasks consist of Audio Captioning and AQA. The llose-ended tasks consist of classification, regression, and retrieval. Datasets like Clotho have more than one type of annotations, so they are used for multiple tasks like Audio Captioning and Text-to-Audio Retrieval.
Domain
Dataset
Files
Dur. (sccs)
Output Type
Metric
Setup
Audio Captioning Audio Question
Clotho
7k
15
Cap.
SPIDEr
train/val/est
AudioCaps
39k
Cap.
SPIDEr
train/valiest
Answering
ClotheAQA
2k
15- 30
Q&AA
ACC
train/valiest
ESC50
2k
(30)
ACC
5 Tolds
Sound Event
FSD50K
51k
0.3 30
(L.(200)
mAP
train/valltst
Classification
UrhunSoundKK
MC
ACC
10 folds
DCASE2207 Task4
52k
10
(17)
ACC
train/val/vstt
Music Analysis
GT. Music Speech
120
30
(2)
ACC
10 Tolds
GT. Music Genre
1k
30
MC (10)
ACC
10 folds
Instrument Classification
Beijing Opera
236
4.77
MC (4)
ACC
5 tolds
Instruments
305K
(11)
ACC
train/val/iest
NS. Pitch
305k
Reg.
ACC
train/valiest
Music Note Analysis
NS. Velocity
305k
MC (1))
ACC
train/valMest
NS. Sonie
305k
ML (10)
ACC
train/val/tss
Acoustic Scene
Clascification
TUT 2017
6.3k
MC (15)
ACC
train/val/test
Emnttion Recognition
CREMA-D RAVDESS
MC (6)
ACC
Folds
2.5k
S
MC (8)
ACC
5 folds
Vocal Sound
Vocal Sound
21k
5
MC (6)
ACC
train/val/tsst
Classification
Surveil.
Surveillance
Applications
585
S 33
MC
ACC
train/val/est
lext-toAAddio
Clotho
7k
15-30 30
Ret.
R
train/valhest
Retrieval
AudioCaps
39k
10
Ret.
G
train/val/ess
Table 2: Wc extensively evaluated Pengi across 21 downstream tasks from various domains. The first two domains open-ended tasks and the rest close-ended tasks. For the "Output Type" column, Cap. refers to captioning, MC multiclass, indicates binary, Reg. indicates regression, and Ret. retrieval.
4.3 Implementation details
Encoders and mappers. We used the audio transformer HTSAT[ as our audio encoder and CLIP's [47] text encoder. The audio is sampled at 44.1 kHz and is converted to log Mel spectrograms with 64 Mel bins, a hop size of 320 ms, and a window size of 1024 ms in the range of 50-8000 Hz. We randomly truncated all audio files to 7 seconds in length for HTSAT. The max length of the text encoder is set to 40 for computational efficiency. We performed another step of CLAP (Contrastive Language-Audio Pretraining) training using the above two encoders (15). This enables experiments where the audio encoder can be kept frozen to see the utility of CLAP's [15) audio embeddings
similar to VLM 2 52
The mapping networks mi and m2 each use an 8-layer transformer
with a prefix length of 40. The total prefix length after concatenating the audio and text is 80. The prper-parameters of the encoders and the CLAP training are mostly left as in the original papers, the details are in Appendix D
Causal Language Model. We used the GPT2 line of models, specifically GPT2-base (124M). The model is kept frozen through all the experiments.
Pre-training. We used Adam Optimiser (32] for 60 epochs and with batch size of 384 on 20 V100 GPUs We used a linear schedule with 2000 warmup steps and a base learning rate of le-4.
5 Results
5.1
Benchmarking Pengi
We assessed Pengi on 21 downstream tasks covering various domains. Pengi is the first audio model that can perform both, open-ended and close-ended tasks. A fair comparison against another model that can perform both is not possible. We chose CLAP (15) as the baseline because it is the only Zero-Shot model with comprehensive evaluation (16 downstream tasks). The next best evaluation was only on 8 tasks. Thus, providing no evidence of performance across domains like speech and music, which tend to be the most difficult. Moreover, we compared against SoTA results even if it came from different models and learning methods. We compared against SoT Zero-Shot models in Table g subset of Table 3 for Sound Event Classification. Even against SoTA from supervised

---

Audio Captioiing T
AQA f
Sound Event Classilcation 1
ICCASE77
Model
AudioCaap
Clotho
CloibaAQA
ESC50
FSDSOK
USSK
Task 4
CLAP
x
x
X
0.826
0.3024
0.7324
0.3
Pengi
0.4667
0.2709
0.6453
0.9195
0.4676
0.7185
Acoustic Scene
Classification
Music f
Instrument Classification f
Music Note Analysist
Model
TUT2017
Speech
Music
Genres
Music
Beijing
Opera
Instrument
family
Pitch
NS.
Velocity
Qualitics
CLAP
0.2963
1.0 0,9688
0.252
2963
0.2549
Pongi
0.3525
0,3525
0.6229
0.5007
0,8676
0.3728
0.386
Emotion Recognition
Clavsification
Vocal Sound
Recog."
Action
llance
Survei
RAV
Vocal
ESC50
Model
MA-D
DESS
Sound
Actions
SESA
CLAAP
0.8784
0.1599
0.4945
0.497
0.7487
Pengi
0.1846
0.2032
0.6035
0.5277
0,5402
Table 3: We used CLAP [15 as bascline comparison because of its strong performance on a wide range of
downstream tasks. The '-" symbol indicates numbers were not available, whole 'x' indicates that the model cannot support the task. Higher better for all numbers. The evaluation metric mAP for FSD50k, AudioSet, ESC50-Actions, and NSynth sonic; score for DCASE17; and SPIDEr for AudioCaps and Clotho captioning. All other downstream tasks use Accuracy.
learning models in Tables and 7 AQ&A and Audio Captioning respectively. Table against SSL, supervised and trained on speech audio models.
Open-ended tasks. Pengi sets new state-of-the-att performance for open-ended tasks. We used Audio Captioning and AQA for open-ended tasks. The CLAP model can only support close-ended tasks and cannot perform open-ended tasks without additional modules and fine-tuning. Therefore, we compared against supervised trained models in Section (5.2
Close-ended tasks. Pengi performs better than CLAP on most audio classification tasks, and can also outperform the literature. Although CLAP and Pengi employed different learning methods and used different amount of training data, it is to be noted that Pengi can compete with strong contrastive methods like CLAP and other methods in the literature.
5.2 Audio Captioning and AQA
Audio Captioning. Pengi's performance outperformed supervised models in the two captioning tasks AudioCaps and Clotho, as shown in Table 7 The captioning competition IEEE DCASE 2022 ranks models based on the metric SPIDEr, combination of CIDEr and SPICE. Specifically, for AudioCaps Pengi outperformed the literature by a relative 6.6% and for Clotho by a relative 26%. All models used both, AudioCaps and Clotho datasets in training. One of the best captioning models is from Kim et al. [3I]. The authors followed a similar training procedure to ours with audio encoders and language model. Unlike Pengi, which uses single audio encoder, they employed two mapping networks to capture both global and temporal features from the audio. Despite having two audio representations, the model underperformed our approach.
Similar to Multi-Tase Learning [62] [0], we hypothesize that learning a shared audio encoder and mapping networks helps Pengi to solve individual tasks better. We addressed this hypothesis by conducting an ablation study in Table 4 In experiment A, we trained and evaluated Pengi only on audio-captioning data with text prompts of "generate audio caption". Then, we contrasted audio captioning performance against experiment B, where we trained on data across different tasks, in see consistent improvement in both AudioCaps and Clotho downstream tasks. Specifically, experiment B outperforms experiment A by a relative 2.5% and 2.3% AudioCaps and Clotho respectively. This indicates that Pengi's i's shared architecture does help in improving performance on individual tasks.
other words, our proposed setup in this paper. From Table [
Audio Captoning 1
Audio Retrieval f
Exp.
Eval. dataset
BLUE,
SPIDEr
Audio Q&A
Model
Retr.
R@1
R@8
R@P10
Model
Acc
al.
Clotho
4.4
7.5
AudioCaps
0.6439
0.451
M,
0.575
Gont, et al.
Clotho
2.1
7.0
12.0
AudioCaps
0.6912
0.4667
M2
0.627
Mei et
Clatho
14.1
21.6
Clotho
0.5199
0.2648
M3
0,635
Kim et
Clotho
7.6
19.6
28.8
Clotho
0.5702
0.2709
Pengi
0.645
Soham et al.
Clotho
16.7
41.0
54.1
Pengi
Clotho
26.1
36.7
Table 4: Effect of shared audio encoder training
Table S: AQ&A results
Table 6: T2A Retrieval results
https://dcase
7

---

Eval.
Model
BLUB,
BLUF2
BLUFF
BLU.
METFOR
ROUGE
dataset
Chen al.
AudioCaps
0.346
0.179
Gontier et al.
AudioCaps
0.635
0.461
0.322
0.219
0.208
0.450
0.612
0.153
0.383
Mei et al.
AudioCaps
0.682
0.507
0.369
0.266
0.238
0.488
0.701
0.166
0.434
Kim al.
AudioCaps
0.708
0.547
0.402
0.283
0.238
0.499
0.710
0.167
0.438
Pengi
AudioCaps
0.691
0.419
0.371
0.253
0.232
0.482
0.752
0,182
0.467
Chen et al.
Clotho
0.516
0.325
0.215
0.141
0.153
0,350
0.314
0.102
0.208
Gontier et al.
Clotho
0.461
0.282
0.182
0.117
0.136
0.318
0.251
0.083
0.167
Mei al.
Clotho
0.516
0.318
0.204
0.127
0.157
0.351
0.313
0.105
0.209
Clotho
0.366
0.215
Pongi
Clotho
0.57
0,369
0.242
0.15
0.172
0,375
0,416
0.126
0.271
Table Pengi oulperforms the best Audio Captioning performance from supervised models. All models used both, AudioCaps and Clotho datasets in training. SPIDEr is the metric used rank models IEEE DCASE Challenge. Higher is better for all metrics.
AQA. Pengi outperformed the existing literature 37 Authors in 37 collected the only dataset available (ClothoAQA). They converted the AQA task into a classification task, instead of generation task. Authors trained and fine-tuned model in supervised setup. In contrast, we used the free-form text from Pengi, where the answer is correct only when it directly matches the human response. Note that Pengi includes the training set of ClothoAQA among its training sets, but there is no further fine-tuning on this task. The results are shown in Table[3] The first column indicates three different baseline models from (37]. Pengi achieved 64.5% and outperformed the existing supervised benchmark by a relative 1.5%.
5.3
Zero-Shot Sound Event Classification
We compared Pengi's 's classification performance against Zero-Shot contrastive models in the literature. The existing literature restricts the training and evaluation tasks to few sound event datasets. Hence, matched our comparisons to sound event datasets. The downstream datasets of ESC50, US8k, DCASE17 Task4 contain audio files and labels not seen by Pengi during training. We considered these three datasets to constitute zero-shot setup for Pengi. For FSD50k, the audio files in the training split have been used for training Pengi. Hence, we do not consider this a pure zero-shot setup but nonetheless, report numbers for insights.
On Zero-Shot FSC500 performance, Pengi beats AudioCLIP 23 CLAP [5]. and LAION CLAP [58) by 32%, 11 11%, and 1% respectively (See Table 8). Interestingly, human performance ESC50 is % accuracy and Pengi's performance 92%. Mei et. al. [41] added ChatGPT augmented audio-text pairs to CLAP training
(58) and showed improvement in performance from 91% to 94% on ESC50. On US8k, Pengi performed better than Wav2CLIP and AudioCLIP but lower than CLAP and LAION CLAP. Overall, even though Pengi is text generation model, its Zero-Sha Shot performance on close-ended Sound Event Classification is competitive.
Zero-Shot Sound Event Classificatio f
DCASE17
Model
ESC50
FSD50K
US8K
Task
Wav2CLIP
0.404
AudioCI.IP
0.694
0.653
CLAP
0.826
0,302
0.732
LAION
0.91
0.77
Pengi
0.92
0.468
0.719
0,338
Table 8: The literature Zero-Shot audio models only reports performance on Sound Event Classification datascts. Pengi's classification performance is competitive. The '-' indicates numbers are not available. The evaluation metric for DCASE17 is the Fl score while FSD50K employs mAP, ESC50 and US8K use Accuracy.
5.4 Texto Retrieval
For Aexio Retrieval in a contrastive learning setup, the user query is converted into a text embedding which is then used to retrieve the top & audios by their audio embeddings [11 [1][8]. Pengi a generative model and does not allow a contrastive setup. Although Pengi has an audio encoder and text encoder that could replicate the contrastive setup, we wanted to evaluate our model from the generative perspective. First, Pengi used index a database by generating audio captions for all the audio recordings. Second, the user text query is matched directly to the dataset captions. The associated audio files of the top dataset captions are considered to be the top retrieved audio. Note that the cosine similarity computation is between two text embeddings and not audio and text embeddings. Thus, the quality of generated captions for indexing the dataset important for retrieval performance.
8

---

In Table 6 we compared Pengi's text-to-Audio retrieval performance against the literature. The models used for comparison are audio captioning models using the above-described procedure of indexing and query matching, and not the contrastive- like setup. Pengi outperforms the literature on R@ However, contrastive models [15. (58), [1] are substantially better than generative models for the task of directly matching text to audio for retrieval. An example of contrastive model performance is shown in Table 6 as a gray row.
5.5
Next text-token prediction for learning audio representations
Pengi uses next-text token prediction to learn audio representations, hence natural question is: "Can next text-token prediction objective help in learning general purpose
purposs audio representations?" To answer this question, we performed linear probe [47] and shallow learning (53) experiments. After Pengi's pre-training, we took the audio encoder ag in Fig 2 and trained one, two, or three fully-connected linear layer(s) with cross-entropy on top. Note that, we kept Pengi's audio encoder frozen and it did not include the mapping network m.. We selected representative datasets from the domain of Sound Events, Music, and Speech Emotion for the linear probe experiment. Pengi's linear probe (one layer) and shallow learning (two or three layers) numbers are compared against the best single model submissions from the HEAR challenge (53) in Table The results from HEAR challenge reported the maximum of both settings (7,1 or T2, Taa). Apart from Wav2vec2 which is trained on speech data, all other models were trained on non-speech audio. Pengi's 's linear probe I, and La performance is consistently better than CLAP [T5]. In the Sound Events and Music domain, Pengi outperformed other models. In the Speech Emotion domain, Pengi performed better than non-speech models but lower than models trained on speech (Wav2vec2). The experiment indicates that the next token prediction does help in learning audio representations useful for various domains.
Sound Events
Music
Speech Emotion
Model
ESC50
FSDS0k
GTZAN Genres
Opera
RAVDESS
CREMA-D
YAMNet
0,8375
0.847
0,9405
0.479
0,4533
Open 1.3
0.7505
0.4470
0.879
0.9746
0.604
0,5497
Wav2CLIP
0,7589
0,3617
0.748
0,9363
0.684
0,5116
PaNN
0.9085
0,860
0.9112
0,5550
War2Nec2
0.5610
0.3417
0.780
0.9067
0.6562
CLAP (L1)
0,8995
0.5024
0.73
0.6399
0,4044
0,2315
CLAP (L.)
0.9310
0.5690
0.8330
0.8263
0.4512
0,2830
Pongi (ZS)
0.9195
0,4676
0.3525
0.6229
0.2032
0.1846
Pengi (L,)
0.8915
0,5608
0.8000
Pengi (La)
0.9485
0.6235
0.9010
0,9883
0.6108
0,5916
Table 9: Shallow learning experiment where the audio encoder is frozen in all the experiments. ZS is zero-shot and L, indicates linear layers used. Unless specified, cach model reports the best of L1, L2, and La.
6 Limitations
Trade-off between close-ended and open-ended tasks performance. The classification and text generation performance of Pengi is competitive against contrastive models. However, retrieval performance lags behind that of contrastive models (8). Although these models excel at retrieval, they are limited to close-ended tasks. Thus, there is trade-off between both types of learning methods proposed far in the literature.
Limitations inherent to Language Models. Pengi benefits from the encyclopedic knowledge of pre-trained Language Models (LM). However, pretrained LM is component of Pengi. they also inherit their limitations. For example, LM are known to hallucinate [28) and specific to Pengi. can produce responses not grounded or conditioned on audio. Similarly, Pengi falls back to LM behavior no audio is provided or if the audio knowledge is limited. Therefore, the risks of LMM, namely propagating stereotypes, and biases and potentially producing offensive language are still applicable to Pengi. The recent works [48 [56] in the NLP field try to address these issues. However, specifically studying risks and limitations can uncover new insights that can accelerate the development of ALMs.
7 Conclusions
We proposed Pengi, novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text- cext-generation tasks. It takes as input, an audio recording, and text prompt, and generates free-form text as output. Pengi capable of handling both, close-ended and open-ended audio tasks. We benchmarked Pengi on 21 downstream tasks and show it yields SoTA performance in several of them. Our findings break ground in prompting language models with audio for general- purpose audio understanding.

---

References
[1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen,
A. Roberts, M. Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint 2023.
arXiv:2301.1133
[2] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716
2022.
[3] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vee 2.0: A framework for sule-super learning of speech representations. Advances neural information processing systems, 2020.
[4] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, O. Teboul, D. Grang- ier, M. Tagliasacchi, and N. Zeghidour. Audiolm: a language modeling approach to audio generation. arXiv preprint arXiv:2209.03/ 143, 2022.
[5] T. Broww, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov. Hts-at: A hicrarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[7] S. Chen, Y. Wu, C. Wang. S. Liu, D. Tompkins, Z. Chen, and F. Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212. 220000058, 2022.
[8] H. W. Chung, L.. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al. Scaling instruction-fineture language models. arXiv preprint arXiv:2210.1 11416, 2022.
[9] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson. FMA: A dataset for music analysis.
In 18th International Society for Music Information Retrieval Conference (ISMIR), 2017.
[10] Deshmukh, B. Raj, and R. Singh. Improving weakly supervised sound event detection with self-supervised auxiliary tasks. pages 596-600, 08 2021. doi: 090214477/nterspeecch20001111111
[11] S. Deshmukh, B. Elizalde, and H. Wang. Audio Retrieval with WavText5Ke and CLAP Training. In Proc. INTERSPEECH 2023, pages 2948-2952 2023. doi: /t14/////nnnrreecch22223-- 136.
[12] H. Dhamyal, B. Elizalde, S. Deshmukh, H. Wang. B. Raj. and R. Singh. Describing emotions with acoustic property prompts for speech emotion recognition. arXiv preprint arXiv:221 1.07737 2022.
[13] K. Drossos, S. Lipping, and T. Virtanen. Clotho: an audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. doi: 10. ww..10//CASSS94077...... 000.:777..2200002522
[14] B. Elizalde, S. Zarar, and Raj. Cross modal audio search and retrieval with joint embeddings based on text and audio. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
[15] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang. Clap: Learning audio concepts from natural language supervision. arXiv preprint MiXiv:20060046660 2022.
[16] B. M. Elizalde. Never-ending learning of sounds. Carnegie Mellon University, 2020.
[17] J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learning, pages 1068-1077. PMLR, 2017.
[18) E. Fonseca, J. Pons Puig, X. Favory, F. Font Corbera, D. Bogdanov, A. Ferraro, S. Oramas, A. Porter, and X. Serra. Freesound datasets: a platform for the creation of open audio datasets. In Hu X, Cunningham SJ, Turnbull D, Duan Z, editors. Proceedings of the 18th ISMIR Conference; 2017 oct 23-27; Suzhou, China. :Canad/Cnndd):: International Society for Music Information Retrieval. International Society for Music Information Retrieval (ISMIR), 2017.
[19] E. Fonseca, D. Ortego, K. McGuinness, N. E. O'Connor, and X. Serra. Unsupervised contrastive learning of sound event representations. In ICASSP 2021 - 202 IEEE International Conference on Acoustics, Speech and Signal Processing (OC.ingggAAAAA (ICASSP), 2021, doi: 10. 19.11999/ASS99997222201.. 9415009.
10

---

[20] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra. Fsd50k: An open dataset of nlmann-lbblle sound events, JEEE/ACM Transactions on Audio. Speech, and Language Processing, 2022. doi: 10.1109/TASLLP20022 PTT./ASS....021.113320000
[21] F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and lluman-labelee dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780, 2017. doi: 10. 10L1009CAASP2221..7792222
[22] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. Glass. Ssast: Self-supervised audio spectrogram transformer. 36: 10699-00777, Jun. 2022. doi: 10.
URL https
//ojs. woojsaaaai.org//dee...... /ttAAA//aaatiiceei/////......../
[23] A. Guzhov, Raue, Hees, and A. Dengel. Audioclip: Extending clip to image, text and audio. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
doi: 10.1 D0.100//CCSSS444922..00 202.997772222
[24] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3466 2021.
[25] P. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35: 28708-28720, 2022.
[26] Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y. L. and D. P. W. Bllis. Mulan: joint embedding of music audio and natural language. In International Society for Music Information Retrieval Conference, 2022.
[27] I.-Y. Jeong and J. Park. Cochlscene: Acquisition of acoustic scene data using crowdsourcing. In 2022 Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 17-21. IEEE, 2022.
[28] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, J. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023.
[29] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[30] C. D. Kim, B. Kim, H. Lee, and G. Kim. AudioCaps: Generating Captions for Audios in The Wild. In NAACL-HLT, 2019.
[31] M. Kim, Sung-Bin, and T.-H. Oh. Prefix tuning for automated audio captioning. arXiv preprint arX1492303.17449 2023.
[32] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
http://////ax org/ abs/ /tss/412... 6980
[33] A. S. Koepke, A.- Oncescu, J. Henriques, 7. Akata, and S. Albanie. Audio retrieval with natural language queries: benchmark study. IEEE Transactions on Multimedia, 2022. doi:
10. 1..10007TMM22222334999777
[34] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[35] I.. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: Asimple and performant baseline for vision and language. arXiv preprint
2019.
[36] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, 2021.
[37] Lipping. P. Sudarsanam, K. Drossos, and T. Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140-1144. IEEE, 2022.
[38] R. Lotfian and C. Busso. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 10(4):471-483, 2017.
11

---

[39] O. Mañas, P. Rodriguez Lopez, S. Ahmadi, A. Nematzadeh, Y. Goyal, and A. Agrawal. MAPL: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2023.
[40] I. Martín-Moraie and A. Mesaros. What is the ground truth? reliability of multi-annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO). 2021.
[41] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. arXiv preprint ar arXiv:2303. 17395, 2023.
[42] R. Mokady, A. Hertz, and A. H. Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111 1..0934,, 2021.
[43] D. Nizumii, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino. Byol for audio: Self- supe-vupe learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks, IJCNN 2021, 2021.
[44] D. Nizumii, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino. Byol for audio: Self-supervised learning for general-purpont audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021. doi: 10.1 10.1109/NNN52388..222 (DNI05222872221.... 10001.9534444
[45] K. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pages 1015-1018. ACM Press, 2015. ISBN
978-1-4503-3459-4 doi: 10. 111114//773333.286633933
[46] Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pages 527-536, 2019.
[47] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 2021.
[48] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 12. /446, 2021.
[49] Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21 (1)):4485-555) 2020.
[50] A. Saeed, D. Grangier, and N. Zeghidour. Contrastive learning of general- purpose audio representations. In ICASSP 202/ 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[51] J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. de Chaumont Quitry, M. Tagliasacchi, I. Shavitt, D. Emanuel, and Y. Haviv. Towards learning a universal non-semantic representation of speech. Proc. Interspeech 2020, pages 140-144, 2020.
[52] M. Tsimpoukelli, J. L.. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 200-212, 2021.
[53] J. Turian, J. Shier, et al. HEAR: Holistic Evaluation of Audio Representations. In NeurlPS 202/ Competitions and Demonstrations Track, 2022.
[54] 7.. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual lan- guage model pretraining with weak supervision. In International Conference on Learning Representations.
[55] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.. Finetuned language models zero-shot learners. In International Conference on Learning Representations, 2021.
[56] L. Weidinger, Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04355 2021.
[57] H.-H. Wu, P. Seetharaman, K. Kumar, et al. Wav2elip: Learning robust audio representations from clip. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. doi: 10LL099ICAASSP449222200999711119
12

---

[58] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Kirg-Kirkpatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and pe-word-t-ccattinn augmentation. arXiv preprini arXiv:22
2022.
[59] L. Yuan, D. Chen, Y.-L. Chen, N. Codella, et al. Florence: A new foundation model for computer vision. arXiv preprint 1rXiv:2111.1145 2021.
[60] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency. Mosi: multimodal corpus of sentiment
intensity and subjectivity analysis in online opinion videos. arXiv preprint 1606.066 2016.
[61] A. B. Zadeh, P. Liang, S. Poria, E. Cambria, and I..P.. Morency. Multimodal language analysis in the wild: Cmu- timu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2236-2246, 2018.
[62] Y. Zhang and Q Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586-5600 5609, 2022. doi: 10.1109/TKDDE.022.33700000
13

---

The appendix is organized as follows: In the first few sections AAl we describe set of additional experiments. In Section [ we discuss the contrastive model and training details. In Section E we evaluate Pengi's performance with the audio encoder kept frozen. In Section Fi we analyze the effect of text encoder on zero-shot performance. Section G. we analyze and highlight types of Pengi errors. Lastly, in section H we compare contrastive and generative pretraining.
Audio input
Text input
Text oulput
this sound
train, railway and locomolive
+++
this emotion is
happy
the of the crash onto shore recede
#+++
generate audio caption
question: what type of animal making the light sound the background?
it is bird
this sound of music language
music analysis
english and genre hip-hop. The associated tags are glitch, idm, dustep
the acoustic scene is
park
question: is the engine healthy?
sentiment
positive
Figure 4: More examples of audio and prompt input and their corresponding textual responses. Images are for illustration purposes only.
person
speaking
f9
Language
the
bockground
mm
m
A/
generate
Figure 5: The user can also add an additional second text input and guide the output of Pengi. For example, the user can add "in the hackground" after the audio and text prefix and Pengi produces the output "a person is speaking". Compared to Fig the output of Pengi changes to what the user has prompted in the second text input which is about background sounds.
14

---

A Additional text input
Pengi takes as input, an audio recording and text, and generates free-form text as output. During inference, an audio encoder ag and mapping network my represent each audio recording as sequence of continuous embeddings. Similarly, text encoder d6 and mapping network m2 does the same for the corresponding text input. Both sequences are combined pre-trained frozen language model fo. The language model generates tokens starting from the prefix.
prefix to prompt
The text input acts task induction and helps guide the language model to produce the desired output. Let's take an example of human speech recording. A text input of "generate audio caption" will generate a caption like "a person speaking with car moving in the background", while a text input of "this sentiment is" will produce a response like "negative". However, there are instances where we want to guide the language model further answer or complete a specific query we had. We can do this by additional text input. This is depicted in Fig The second text input gets tokenized by the frozen language model's tokenizer and converted into continuous embedding by the frozen language model's 'ss embedding function. Therefore, the new prefix consists of sequence of embeddings associated with audio, first text input, and second text input which originates from the audio encoder, text encoder, and frozen language model's embedding function respectively.
Some examples and the effects the second text input are shown in Fig 6 Empirically, we have seen the additional second input produces meaningful output only when used with text input of "generate metadata". The examples shown in Fig c are cherry-picked. The additional text input often causes Pengi to lose track of the audio data and hallucinate its own text or fall back to frozen language model behavior. not clear how ground the output in audio information when additional text input is provided. Further investigation in this direction will enable new scenarios including in-context learning.
Audio input
Additional text input
Text output
bird is chirping
the bird is called
robin
Text input
name the bird.
this is bird called robin
generate metadata
the background is
quiet
mention forest.
a blackbird is singing in the forest
Audio input
Additional text input
Text output
choir singing
at the beginning
vocalist is singing
Text input
in the end
people applaud
generate metadata
the background
sound of drum beat
Figure 6: Examples of audio-text input with additional text input and their corresponding textual responses. Images are for illustration purposes only. The '-" symbol indicates additional text input was not used.
B Inferring audio prefix
The audio encoder and text encoder followed by mapping networks, jointly forms the prefix which prompts the frozen language model. To understand more about Pengi's natural language response, we try to interpret prefixes as a sequence of tokens or words. Each prefix embedding is mapped to the highest similarity token from the GPT2 vocabulary [42] The similarity method used is cosine similarity. This is possible the prefix and GPT2 embeddings occupy the same latent space. We
15

---

[4]] The examples of Pengi's generated The interpretations are hard follow but do contain salient words that are related to audio content. For example, each inferred audio prefix contains words associated with content of audio like babies, thunder, chicken, etc which also appear in corresponding Pengi's 's natural language output.
use this method on a few examples from the ESC50 dataset output and the inferred audio prefix are shown in Table [T]
One reason interpreted prefix does not have a clear structure is that the mapping network has to do two things at once comprehend both the audio and text input and guide the fixed language model. Mokady et. al. 44 observed that the interpreted prefix is more comprehensible when GPT2 is also fine-tuned. A similar method can be followed to infer the text input prefix, but we didn't find any interpretable insights there.
Text oulput
Interred audio prefix
and, the my's 's the first the and and fixme the the supern the, coma the BST in in improvis the babies in in the noises from noises in the ( the the and innovative for
a baby is crying loudly and loudly
and- the bigHUD the the the and as"] the theth P the. weather the close andscape. thunder in- the Audiostorms interview click in the the and unsettling, and at the newone the new the and to OUR the theron the. chickens theitics the in imperson the chickens to in the Audio sitcom.
thunder claps and then thunderstorm hits
rooster is crowing loudly
chickens in the ( the the, Mumbai the
and. the great bird the first the and and OUR the the number La
bird is singing in the background
the in bird the one great and photography and bird that.
Audio owl interview singing being: the and innovative.
Table 10:
Examples of Pengi output and their corresponding inferred audio prefix. The input text prompt is "generate audio caption" for all examples. We bold the salient words relating to the input audio and text output.
C Effect of text prompts
The choice of input text prompt changes Pengi's downstream task performance. We analyze the performance of seven of the input text prompts defined in S. for downstream tasks. For some tasks, only specific prompts are applicable, for example, 'question: //' prompt for AQA and 'this emotion is' for emotion recognition. Pengi's performance on each downstream task corresponding to the different input text prompts is shown in Table In summary, we see that the prompt 'generate metadata" works well on average for close-ended downstream tasks.
Text prompts
Downstream
question:
generate audio
generate metadata
this is sound
this acoustic scene
this music
this emotion
Dataset
caption
note
Clotho Cap.
0,2709
AudioCaps Cap.
0.4667
ClothnAQA
0.6453
ESC50
0,8870
0.9195
0.6910
FSD50K
0.4676
0.4504
USB8
0,7185
0.6585
0.5731
DCASE17
0.3150
0.3143
0.3506
AudioSet
0,1216
0.1230
0.1635
TU1'2017
0.2562
0,3525
0,2216
0.1716
GTZAN Genres
0,3230
0.3420
0.3180
GTZAN MS
0,9440
0.9606
0.9922
Opera
0.2373
0.6229
0.4449
NSynth Instrument
0.5007
NSynth Pitch
0.8676
NSynth Velocity
0.3728
NSynth Qualities
0.3860
RAVDESS
0.1846
CREMAD
0.2032
Vocal Sounds
0.5778
0.5688
SESA
0,5162
0.5402
0.5350
ESC50 Actions
0.5277
0.5111
0.4846
Clotho Ret. (T2A)
0.0938
AudioCaps Ret. (T2A)
0.1771
0.1407
Clotho Ret. (A2T)
0.1448
AudioCaps Rei. (A2T)
0.1819
0.1771
Table 11: We use different text prompts and observe the performance on downstream tasks. 1-2 indicates the prompt is not used. The metrics used for each downstream tasks are same as Table 3
16

---

D Constrastive Learning model details
We follow and train a CLAP [5] model for the choice of contrastive model used our experiments. We use cansformer-based audio and text encoder. The audio encoder is HTSAT [6] and the text encoder is from CLIP [47]. Both the encoders are followed by a linear transformation called the projection layer. We finetune both the encoder and their projection layers. After contrastive training,
the audio encoder and text encoder are used in Pengi.
Consider a batch size of N. Let the audio and text embedding be represented by Eg E R Nxd and Eg R Nxd, Then the resulting similarity matrix is:
C = 7 T(E Feg )
(4)
We use the loss function (L) of symmetric cross-entropy: projections
L = 0.5(ftex:(C) + (C))
(5)
where 4x = Ei= log diag(softmazz
along text and audio axis respectively.
implementation details. The audio is sampled at 44.1 kHz and is converted to a log Mel spectrogram with 64 Mel bins, hop size of 320 secs, and a window size of 1024 secs in the range of 50-8000 Hz. We randomly truncate all audio files to 7 seconds in length for HTSAT, All models are trained with Adam Optimiser [32] for 45 epochs with batch size of 1536 on 20 V100 GPUs. We use linear schedule with 2000 warmup steps and a base learning rate of le-4.
Results. To verify the training, we check our CLAP's performance on the ESC50 dataset. The results
are shown in Table 15
Model
ESC50
Way2CLIP
0.414
AudioCLIP
0.694
CLAP
0.826
LAION
0.91
CLAP (ours)
0.89
Table 12: CLAP zero-shot performance on ESC50
E Frozen audio encoder
The audio encoder as transforms the audio input into audio embedding. We used the audio transformer backbone from CLAP trained Section D as our audio encoder in our experiments. In Computer Vision, Visual Language Models [4] 2 39 use an image encoder from CLIP [47] which is frozen throughout experiments. However, there is a magnitude order difference in data collection of image-text audio-text pairs. Therefore, for Pengi we train the audio encoder as well. Nonetheless, we report numbers on Pengi's performance if the audio encoder is kept frozen. The results are shown Frozen Pengi underperforms Pengi across all downstream tasks.
in Table 13
Audio Captioning 1
Audio Q&A 1
Sound Event Classification 1
DCASE17
Model
AudioCaps
Clotho
ClothaQA
ESC50
FSDS0K
US8K
Task 4
Frowen Pengi
0.4535
0.2577
0.6395
0.8950
0.4117
0.6319
0.3225
Pengi
0.4667
0.2709
0.6453
0.9195
0.4676
0.7185
0.338
Acoustic Scene
Music
Instrument Classification
Note Analysist
Classification
Music Speech
Music
Beijing
Instrument
NS.
Model
TUT2017
Genres
Opera
family
Pith
Velocity
Qualities
Frozen Pengi
0.3449
0.9219
0.2550
0.814
0.2949
0.7131
0.3330
0.3830
Pengi
0.3525
0.9688
0.3825
0.6229
0.50M)
0.8676
0.3728
0.3860
Vocal Sound
Action
Survei
Emotion Recognition
Classificationt
Recog.
llance.
CRE
RAV
Vocal
ESC50
Model
SESA
MA-D
DESS
Sound
Actions
Frozen Pengi
0.1816
0.1312
0.3371
0,5196
0.5316
Pongi
0.1846
0.2032
0.6035
0.5277
0,5402
Table 13:
The model 'Frozen Pengi' indicates Pengi with audio encoder frozen. The symbol indicates numbers were not available while 'X' indicates that the model cannot support the task. Higher is better for all numbers. The evaluation metric mAP for FSD50k, AudioSet, and NSynth sonic: Fl score for DCASE17; and SPIDEr for AudioCaps and Clotho captioning. All other downstream tasks use Accuracy.
17

---

Audio Captioning 7
Audio QEA T
Sound Event Classiication T
DCASE17
Model
AudioCaps
Clotho
CloibaQQ
ESC50
FSDS0K
US8K
Task 4
Exp
0.4857
0.2545
0.6316
0.9215
0.4478
6882
0.3314
Pengi
0.9195
0.4676
Acoustic Scene
Music f
Instrument Classification
Music Note Analysist
Classificationt
Model
TUT2017
Speech
Music
Genres
Music
Beijing
Opera
Instrument
family
Pitch
Velocity
Qualitics
Exp B
0.3241 0.3525
0.9609
0.317
66684
0.5
0.8591 0.8676
0.3708 0.3728
0.377
Pongi
0.9688
0.3525
0.6229
0.5007
0.386
Recognition
Clavsification
Vocal Sound
Recog.1
Action
Survei llance f
Vocal
ESC50
Model
SESA
MA-D
DESS
Sound
Actions
Exp
0.1728
0.1769
0.5798
0,5282
0.4923
Pengi
0.1846
0,2032
0.6035
0.5277
0.5402
Table 14: Exp B is Pengi with mapper ma but without the text encoder. The evaluation metric is mAP for FSD50k, AudioSet, ESCCOn-cctonn, and NSynth sonic; FI score for DCASE17; and SPIDEr for AudioCaps and Clotho captioning. All other downstream tasks use Accuracy.
F Effect of text encoder
Pengi's 's architecture in Figure consists of text encoder gy that transforms the input text into text embeddings. Then a mapping network m2 converts these embeddings into a sequence of k embeddings. natural question that arises here is "Why an explicit mapping needed for inpur tex??". We conducted two experiments to evaluate the effect of omitting my and/or the text encoder. We denote Exp as Pengi without the text encoder and m2 (input text directly to LM), and Exp B as Pengi without the text encoder but with m2 (input text to m2). In Exp A, we found that removing resulted in loss of coherence between the input text prompt and the output text. For example, an input prompt about identifying and thus random performance. In Exp B, we removed the text encoder but retained mg- The Exp architecture is depicted in Fig Z and its results are shown in Table [4] By removing the text encoder, model performs slightly lower than the proposed architecture with both components.
emotion class "the emotion is resulted in random text output
the
wavee
ocean
Causal Language Model %
Prefix
the
waves
the
ocean
7P22 i
%6 Audio Encoder
Embedder
audio caption
Figure 7: Pengi architecture without the text encoder Iv. The text prompt is tokenized and embedded by text embedder, followed by the mapping network mg. The results of this architecture are shown in Table 13
18

---

Audio Captioning"
Audio Q&A
Sound Event Classifiation T
DCASE17
Model
AudioCaps
Clotho
CluibeAQA
ESC50
FSDS0K
US8K
Task
CLAP
X
x
X
0.8916
0,3398
0.7661
0.3387
Pengi
0.4667
0.2709
0.6453
0.7185
0.3380
Accustic Scene
Music 1
Instrument Classification
Music Note Analysist
Classification
Model
TUT2017
Specch
Geures
Music
Beijing
Opera
Instrument
family
Pitch
Velocity
NS.
Qualitics
CLAP*
0:3037
1.0 0,9688
0.479
0.4025
0.415
0.1337 0.8676
0.2185 0,3728
0.2545
Pongi
0,3525
0.3525
0.6229
0.5007
0.386
Emotion Recognition
Vocal Sound
Action
Survei
Clacsification
Recog.1
llance.
CRE
Vocal
ESC50
Model
MA D
DESS
Sound
Actions
SESA
CI.AP
0.1512
0. 0692
0.5522
0.508
0,7094
Pengi
0.1846
0.2032
0.6035
0,5277
0,5402
Table 15: We train a new CLAP* model on the same 3.4M pairs training data used Pengi. The 'x" indicates that
the model cannot support the task. Higher is better for all numbers. The evaluation metric is mAP for FSD50k, AudioSet, ESC50- Actions, and NSynth sonic; F1 score for DCASE17; and SPIDEr for AudioCaps and Clotho captioning. All other downstream tasks use Accuracy.
G Different type of Pengi errors
There are three types of errors that lead to drop in Pengi's performance. We categorize them into audio concept errors, hierarchy errors, and text-matching errors.
Audio concept errors. These types of errors are when the model gets the base audio concepts wrong. For example, while generating an audio caption, the model predicts it as "a sound of dog barking in a neighboring field" instead of "a sound of door knocks with cars moving nearby". This indicates the model fails to detect the sound event of door knock and confuses it with dog barking. These are Pengi model errors stemming from the audio encoder.
Heirarchy errors. The hierarchy error comes from a mismatch between Pengi's model prediction and the target domain classification. For example, in classifying sound events, Pengi predicts the sound as "domestic sounds", however for ESC50, the target classification requires more fine-grained classification within domestic sounds like Vaceum cleaner, Toilet flush, brushing teeth, etc. If text matching is used for classification, then the model will not be able to categorize "domestic sounds" into any of the fine- ane-grained classes. To solve this error and get more fine- mme-grained response, we can use improved text prompts or switch to the log-likeldhood method.
Text-matchin errors. The text-matching errors the errors that result from the text embeddings or the eext-matching method used. This means depending on the text embedding and similarity method used, the performance of Pengi on close-ended tasks will change.
H Constrastive Learning and Generative Pretraining
We compare our model Pengi with CLAP [15] rfat--ffthhearr art Zero-Shot model that has been evaluated on 16 downstream tasks. However, CLAP is trained on a smaller amount of audio-text data. This leads to ask: "Is the improved performance due the larger training data or the generative pretraining?" We already know that generative pretraining allows us to perform open-ended tasks like Audio Captioning, AQA, which are not possible with contrastive models. But this does not tell us if: generative pretraining is beneficial for dose-eened tasks like classification?. To answer this question, we train CLAP model with the same data 4.1b that we use to train Pengi. We call this model CLAP*
Results.
The results shown in Table 15 We see generative pertaining (Pengi) outperforming contrastive learning (CLAP") on average. Moreover, with generative pretraining, the model can perform open-ended tasks like Audio Captioning and Audio Question Answering.
An interesting observation Pengi outperforms human performance (81%) on ESC50. Humans have limitations inherent to how much information participant can handle at once. In the case of ESC50, humans listen to the audio once, and have to remember the audio content, task description, and choose among 50 different classes. Moreover, listeners have different degrees of familiarity with prototypical content from different sound classes, whereas Pengi has been exposed to similar content during training. In sense, Pengi is an expert listener, whereas the humans in the listening experiment were not.
19


Reviews:
[{"review_id": "8eO7w5zHqp", "reviewer": "Reviewer_iz2N", "summary": "The authors present an approach to combine multiple non-ASR audio tasks into a single model. Motivated by audio language models and VLM, the authors propose using a frozen LLM to create an audio LM that can be used for open-ended (purely generative) and close-ended (classification) tasks. The main idea is to use an audio encoder and a text encoder to create fixed length prompts that can be used with a frozen LM after training. The authors train various tasks by using task specific text prompts (“generate audio caption”, “this is the sound of”, etc.). In this sense, the model is similar to a multitask learning model (more on this below). Results show that the model is competitive compared to other approaches in the literature that are task-specific, and an alternative multi-task model, CLAP.", "questions": "1) The authors tokenize and map text prompts. Why is an explicit mapping needed? Can’t they re-use the same tokenizer as the LLM?\n2) Why does the audio and text prompt prefix have to be of fixed length? Does the length of the input audio affect quality?\n3) Line 180 and Fig. 3: Text matching is not appropriately described and is a little hard to follow. Consider providing a more detailed description. Are the embeddings created for each token or for the entire sentence? If for the sentence, how are they created (summarized)?\n4) The authors claim sota in a few different close-ended tasks. Are the gains coming from the new formulation (audio LM) or the extra datasets that the authors are now using? For example, what if the authors train a single model with multiple classification heads for the N-tasks (close-ended, at least)? Does it work worse than the current model that uses the frozen LLM and the proposed architecture? \n5) Line 181: When computing log likelihoods, do the authors normalize based on the number of tokens in the desired values (dog barking vs. sea, e.g.)? Does the number of tokens affect the overall score?\n6) Sec. 5.5: Does the presented result show that perhaps a good portion of the performance comes from training data. For instance, wav2vec2 works better on emotion recognition, likely because the training data is relevant. Are the training set the same for the remaining non-speech models in Tab. 9? \n\nMinor\n1) The authors use close-ended and open-ended tasks a lot, but only define it Sec. 4.2. Consider moving this to an earlier section to avoid confusion.\n2) The authors say the text encoder is frozen (Line 111). Where does this encoder come from?\n3) Why are the results in Tab. 4 for the best setting, worse than those in Tab. 7?\n5) Please provide more descriptive captions for Tab. 4 – 6. \n6) Line 271: Why does Pengi outperform human performance by such a large margin? It is interesting, so perhaps an explanation is warranted.\n7) It would be useful to include the best results from Mei et al to Tab. 8. Also, it’d be useful to include some representative contrastive methods to Tab. 6 for audio retrieval.\n8) Why does Pengi work better on some zero-shot tasks and not the others, compared to techniques like CLAP? Is it related to the training data?\n\nTypos:\n1) Line 51: For example, -> Examples include\n2) Line 70: Missing citation (?)\n3) Figure 2 caption: a text a prompt -> a text prompt\n4) Line 151: question: question -> question: {question}\n5) Line 205: 320 secs / 1024 secs: Perhaps the authors mean milliseconds?\n6) Line 270: Missing Table# (Tab. 8?).", "limitations": "1) Does not support ASR, which is, arguably, one of the most important audio task.\n2) Unclear if the model can learn new tasks. The existing tasks use a fixed prefix prompt, which is tokenized and mapped. This makes it cumbersome to add new tasks to the model. This is a significant drawback compared to current LLMs that can learn new tasks by prompting.\n3) How much does the LLM help? What is the quality if the LM component is trained from scratch with the current dataset?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A single model that can handle multiple tasks, both open-ended and close-ended.\n- One of the first approaches that incorporate VLM / LLMs to create a general purpose audio LM tasks. \n- Results show gains using LLM for building the model, compared to a baseline that doesn't.", "weaknesses": "- While the model can cover a number of tasks with competitive results, the model still performs worse on some of the tasks considered compared to task-specific models. \n- Furthermore, it is unclear if the method generalizes to new tasks, which is a main strength of non-audio LLMs / VLMs. The authors show zero-shot capabilities for new labels within an existing task. Although impressive, not entirely novel given prior works have approached the problem along similar veins (like CLAP, e.g.) but in limited settings. The results in Tab. 9 also show poor performance in zero-shot settings for a new task.\n- The results in Tab. 8 also show that the model is not as best as some of the other models on zero-shot classification task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contribution and providing detailed comments. We hope that our response resolves your concerns and welcome any follow-up questions.\n***\n**Q1. The authors tokenize and map ... re-use the same tokenizer as the LLM**\\\nWe conduct an experiment and discuss the findings and results in global response question 1.\n\n**Q2: Why does the audio and text prompt ... of the input audio affect quality?**\\\nWe choose the audio and text prompt prefix to be of length 40. The length choice is a hyperparameter and does not have to be fixed. On the second question, yes, the length of the input audio affects downstream task performance. The model is trained with 7-second audio clips, so it performs best for audios of similar duration. Shorter audio degrades performance, especially for long sound events like bell gong.\n\n**Q3: Line 180 and Fig. 3: Text matching is not appropriately described ... they created (summarized)?**\\\nWe apologize and will revise the description. For the text matching, the text encoder generates a sentence-level embedding for Pengi's textual output. The sentence-level embedding corresponds to the embedding of the [CLS] token from the text encoder.\n\n**Q4: The authors claim sota in a few different ... the proposed architecture?**\\\nWe appreciate the reviewer’s question. More objectives and parameters may improve training, but only if the model can converge with multiple losses.  Also, adding heads is complex and hard to scale. For example, we have 22 tasks, so we need 22 heads, with different losses and outputs (labels, regression, descriptions, etc). Pengi can scale to N tasks, with a single loss and training procedure, and the same output format (free-form text). To check the benefit of our formulation and remove the effect of additional data, we train a new CLAP (CLAP*) on the same 3.4M pairs as Pengi. We attach this as Table 2 in PDF in the global response. Overall, Pengi's strong performance still holds.\n\n**Q5: Line 181: When computing log likelihoods, do the authors normalize ...  the overall score?**\\\nYes, we normalized based on the number of tokens for the log-likelihood method. We found that the number of tokens does affect the overall score, a larger number of tokens may decrease performance.\n\n**Q6: Sec. 5.5: Does the presented result show that ... in Tab. 9?**\\\nThe quality of the embeddings depends in good part on the training data. For example, adding more speech or music-related data usually correlates with improvement on relevant task. However, other components in Pengi are equally important, such as the audio encoder, text encoder, and LLM. They provide information, such as acoustic and text semantics and context. For instance, if our model never saw the word \"dog\" in the training data, but it saw \"bark\" and \"animal\", it can still associate and generate \"dog barking\" as a description.\n\n**Minor question 1, 3, 4 and Typos**\\\nWe thank the reviewer and will fix the issues and typos the reviewer pointed out. \n\n**MQ2: The authors say the text encoder is frozen ... encoder come from?**\\\nThis can be any off-the-shelf text encoder. We tested with text encoders from CLAP, BERT, T5, and CLIP and found little to no difference on downstream task performance.\n\n**MQ5: Line 271: Why does Pengi outperform human performance ... warranted.**\\\nHumans have limitations inherent to how much information a participant can handle at once. In the case of ESC50, humans listen to the audio once, and have to remember the audio content, task description, and choose among 50 different classes. Moreover, listeners have different degrees of familiarity with prototypical content from different sound classes, whereas Pengi has been exposed to similar content during training. In a sense, Pengi is an expert listener, whereas the humans in the listening experiment were not.\n\n**MQ6: It would be useful to include the best results from Mei ... audio retrieval.**\\\nThe evaluation method for retrieval is different for contrastive and generative models, which makes a fair comparison difficult. We comment on this as well as the performance difference in Section 6 paragraph 1. As a follow-up, we will update our manuscript to explicitly mention these numbers in Table 6.\n\n**MQ7: Why does Pengi work better on some zero-shot tasks ... to the training data?**\\\nThis is due to multiple factors like training data, contrastive learning formulation, and having a common audio-text multimodal space.\n\n**L1: Does not support ASR, which is, arguably, one of the most important audio task.**\\\nPengi focuses on non-speech audio but we include some speech-related downstream tasks like speech emotion recognition to evaluate generalization. We didn't include speech audio and their transcripts, therefore Pengi does not support ASR. To the best of our knowledge, bridging speech, non-speech audio, and music in a single model is still an open problem.\n\n**L2: Unclear if the model can learn new tasks ... can learn new tasks by prompting.**\\\nWe have explored this direction in Appendix Section A, where we show that the user can use a fixed text prompt like “generate metadata” and then provide more information or ask follow-up questions. This enables the user to steer the conversation with additional prompts (Fig. 6), such as “the background is”, “mention forest.”, etc. However, we acknowledge that Pengi still falls short of VLMs like Flamingo in this aspect, and we plan to investigate this further in the future.\n\n**L3: How much does the LLM help? What is ... scratch with the current dataset?**\\\nWe appreciate the reviewer’s comment. Our default LM is GPT-2 (124M), but we also tested GPT2-XL (1.5B). A larger LM improved performance on the open-ended task of Audio Q&A, had mixed results on Audio Captioning, and no significant change on close-ended tasks. We attach the AQA results below.\n| LLM | Parameters | Audio Q&A|\n------|------------|----------|\n| GPT2-base | 128M |  0.645   |\n| GPT2-XL   | 1.5B |  0.701   |"}, {"review_id": "QPVN2Ck3Ep", "reviewer": "Reviewer_o28f", "summary": "This paper proposes a new audio-language learning model by treating existing audio tasks as text-generation tasks. The model architecture allows both open-ended and close-ended tasks. By evaluating on 22 downstream tasks, this paper shows competitive performance on many of them.", "questions": "My main concern about this work is its performance against SOTA. I'd appreciate authors' clarification on that.\n\n==== post rebuttal ===\nI'd like to thank the authors for their responses. However, the performance improvement against LAION is not really convincing and comparisons are lacking, and thus I'm keeping the original score.", "limitations": "It has been discussed in Section 6.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of treating various forms of audio tasks as text generation task is reasonable, which allows scaling the size of the data to large models.\n\nThe evaluation is pretty extensive. Authors compared the proposed model on multiple benchmarks against multiple models.\n\nThe paper is well-written and easy to follow.", "weaknesses": "It seems that the strongest baseline is LAION CLAP [53], while authors did not compare with it in Table 3 or 9. In Table 8, Pengi did not outperform LAION, which seems to indicate that this model does not outperform the SOTA approach.\n\nMy other concern is that the model's performance is largely based on the LLM and thus limited to their weaknesses. This has been discussed in the limitation section as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contribution and providing constructive feedback! We address every question and hope that our response resolves your concerns. Any follow-up questions are welcome. \n***\nWe present a novel and unified model that can handle both open-ended and close-ended audio tasks without relying on external modules or fine-tuning. Our key contributions are: (1) Pengi, the first Audio Language Model (ALM) in the literature, (2) audio task templates inspired by Instruction Tuning, and (3) a method to perform close-ended tasks with ALM. We evaluate Pengi on 22 downstream tasks and show that it achieves state-of-the-art results on most of them. Therefore, our main goal is not to beat SoTA all around with a single model, but to demonstrate the versatility and strength in performance of our method.\n\n**Questions 1. It seems that the strongest baseline is LAION CLAP [53], while authors did not compare with it in Table 3 or 9. In Table 8, Pengi did not outperform LAION, which seems to indicate that this model does not outperform the SOTA approach.**\\\nWe chose CLAP because it is the only Zero-Shot model with a comprehensive evaluation (16 downstream tasks). It is not conclusive if LAION will outperform Pengi in every task. First, the LAION model was only tested on 4 datasets related to sound event classification. Thus, providing no evidence of performance on multilabel classification (FSD50K) and across other domains like speech and music, which tend to be the most difficult. Second, in Table, 8 Pengi underperformed LAION on US8K but outperformed LAION on ESC50. LAION model cannot perform Audio Captioning or Audio Q&A without additional modules and finetuning.\n\n**Questions 2. My other concern is that the model's performance is largely based on the LLM and thus limited to their weaknesses. This has been discussed in the limitation section as well.**\\\nWe agree with the reviewer. All the limitations of LLMs inherent to LLMs apply to Pengi as well. We explore this in Section 6.  \n\n**Questions 3. My main concern about this work is its performance against SOTA. I'd appreciate authors' clarification on that.**\\\nIn addition to our comment in Response 1, we compared against SoTa performance throughout our paper. We compared against SoTa Zero-Shot models in Table 8, a subset of Table 3, for Sound Event Classification. Against SoTa from supervised learning models in Table 5 and 7 for Audio Q&A and Audio Captioning respectively. Against SSL, supervised, and trained on speech audio models in Table 9. \n\nTo enhance our baseline from Table 3, we trained a CLAP model (CLAP*) on the same amount of pairs of data as Pengi (3.4M) to put aside variations due to data. Pengi's strong performance still holds. We attach this as Table 2 in PDF in the global response."}, {"review_id": "cKDuWwiwWf", "reviewer": "Reviewer_vWxh", "summary": "This work is inspired by the visual language models (VLM) in the literature and presents an audio language model (ALM) for various audio tasks, including both open-ended and close-ended tasks. It also presents a comprehensive evaluation of the proposed ALM on a range of both open-ended and close-ended tasks, and showed very promising results. \n\n", "questions": "- I feel it is a little bit counter-intuitive that in Figure 2, a `m2` mapping network is used after the text encoder, as text encoder  in Figure 2 is already encoded the text prompt into the same space as the text in the response part. What's the purpose of  `m2` here and do you have experimental results to show using `m2` is helpful ? \n\n- It is also unclear to me how the network handle the variable length in text prompt. It looks like to me that both audio prompts and  text prompts in this works has a fixed length, i.e., 40 tokens for each part. However, it is unclear from the paper, how the authors would handle variable lengths of text prompt. It is also unclear what the effect of this fixed length (40 in the experiments) on the audio task performance. \n\n- The authors have proposed 2 methods for evaluating the proposed model's performance on close-ended tasks: log-likelihood and text matching. However, all the following experiments using text matching and there is no comparison of log-likelihood based method vs text matching methods. Are they give relatively close results? \n", "limitations": "The authors have addressed the limitations of the proposed methods adequately. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper presents an innovative audio language model which leverage a pretrained audio encoder and a pretrained language model. Though the proposed network architecture is largely borrowed from vision language models like [47], it is still relatively new to the audio domain. \n\nThis paper is well-written and presents comprehensive evaluations on 22 downstream tasks. ", "weaknesses": "I feel this work is somewhat limited in that it only uses a relatively small language model (GPT-2 line, 124M params) and there is no study of how the language model can affect the performance on the downstream tasks. It is possible that with a stronger language model, the performance of the proposed model on open-ended tasks can be further improved, while the performance on close-ended tasks may more depending on the audio encoder quality. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing our contribution and novelty! We hope that our response resolves your concerns. Any follow-up questions are welcome.\n***\n**Questions 1. I feel this work is somewhat limited in that it only uses a relatively small language model (GPT-2 line, 124M params) and there is no study of how the language model can affect the performance on the downstream tasks. It is possible that with a stronger language model, the performance of the proposed model on open-ended tasks can be further improved, while the performance on close-ended tasks may more depending on the audio encoder quality.**\\\nWe appreciate the reviewer’s comment. Our default language model is GPT-2 base (124M), but we also tested GPT2-XL (1.5B). A larger LM improved performance on the open-ended task of Audio Q&A but have mixed results on Audio Captioning --one dataset improved the other one worsen. For close-ended tasks, there was no significant change. We attach the AQA results below. We agree the audio encoder does impact task performance, and presents an exciting direction for future work.\n| LLM | Parameters | Audio Q&A|\n------|------------|----------|\n| GPT2-base | 128M |  0.645   |\n| GPT2-XL   | 1.5B |  0.701   |\n\n**Questions 2. I feel it is a little bit counter-intuitive that in Figure 2, a m2 mapping network is used after the text encoder, as text encoder in Figure 2 is already encoded the text prompt into the same space as the text in the response part. What's the purpose of m2 here and do you have experimental results to show using m2 is helpful ?**\\\nWe appreciate the reviewer's question. We conduct an experiment and discuss the findings and results in global response question 1.\n\n**Questions 3. It is also unclear to me how the network handle the variable length in text prompt. It looks like to me that both audio prompts and text prompts in this works has a fixed length, i.e., 40 tokens for each part. However, it is unclear from the paper, how the authors would handle variable lengths of text prompt. It is also unclear what the effect of this fixed length (40 in the experiments) on the audio task performance.**\\\nThe text encoder can handle input text of variable length. It produces sentence-level embedding for the input text. The mapping network $m_2$ then converts the sentence-level embedding into a fixed-length sequence of embeddings (prefix). The prefix length is a hyperparameter that we set to 40. Likewise, the model can deal with audio input of variable duration, and it maps the audio representation to a prefix of length 40. We empirically found that a prefix size of 40 achieved the best performance compared to a prefix size of 20 or 80.\n\n**Questions 4. The authors have proposed 2 methods for evaluating the proposed model's performance on close-ended tasks: log-likelihood and text matching. However, all the following experiments using text matching and there is no comparison of log-likelihood based method vs text matching methods. Are they give relatively close results?**\\\nWe chose the text-matching method for experiments because it is computationally less expensive. The log-likelihood method requires more computation and has lower performance on most datasets than the text-matching methods. However, the log-likelihood method has an advantage on out-of-domain or rare words that the text encoder cannot recognize. For instance, when identifying specific bird species by their song, the log-likelihood method outperforms the text-matching methods. We concur that a large-scale study is necessary to determine which evaluation method is superior."}, {"review_id": "qgFMvCtlZA", "reviewer": "Reviewer_sUJx", "summary": "This paper explores Large Language Models (LLMs) in the context of audio processing. The core idea is to do audio-injected instruction tuning for pre-trained LLM. The method is simple: collecting a lot of audio-text paired data and use it to fine-tune a pre-trained CLAP audio encoder together with a frozen LLM. Audio is injected by concatenating the CLAP feature sequence before text. Taking advantage on the nature of LLMs training strategy, the pair relation can be provided in any form (e.g., captioning, label, metadata) and simply put into a pre-defined template to train with next token prediction. Evaluation is done on a wide collection of audio-related tasks using the output sentence of the LLM given the input audio and question.", "questions": "- The use of text encoder: line 112 said it's fundamental but there seems to be no ablation on this. Having LLMs taking text as input seems to be more intuitive. Any idea why?\n- For line 270: \"... by 32%\", isn't this 22-ish%?\n", "limitations": "- Generalizability (data): Table 8 shows that in the cases (namely, ESC50, US8K, and DCASE17) where Pengi have not seen the dataset, results are not as competitive. It would be good if more details on evaluation pipeline can be provided and interesting to see some error analysis.\n- Generalizability (format): All the tasks this paper considered are covered by the templates hence the input is not in free form. One of the key strength of instruction-tuned LLM is the ability to take any input and answer accordingly. are there examples where the question (text input) is never seen during training?", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- This is the first (together with few concurrent papers) work to explore LLM for audio processing.\n- Performance on audio captioning is solid.\n- The method is simple in a good way, and this work can be served as a baseline for audio processing with LLMs if the evaluation can be more completed (see weaknesses).", "weaknesses": "The biggest concern I have for this paper is that **the evaluation seems to be limited/biased**.\n\n- Table 3: In Pengi's framework, CLAP serves as a feature extractor that is pre-trained and unfrozen. It is obvious that Pengi can (and should) be making improvement over CLAP considering the scale of the system, the total amount of data used, and the computational power required.  Comparing Pengi to state-of-the-art methods on each bench mark would better justify its value.  (Imho, this is also why LLM is a great success - they generalized to and performed well on most of the tasks so good to the point where people can live with the amount of data/computation it costs.) Even if the numbers are not overwhelming, it will still add value to this work as a first step of exploring LLM's application in audio.\n- Table6/Section5.4: It is unclear how the authors obtained the retrieval performance for existing contrastive methods using their text-to-text retrieval pipeline. How do they index the dataset with text given the audio/text encoder from contrastive learning? Moreover, the text-to-text setup could be biased and unfair for other methods, since Pengi builds on top of pre-trained LLM and focuses on text during training. While a 100% fair comparison might not be possible, it is clearly unfair to use significantly more resource AND compare in a way favoring the large model. Table 5/7 are good examples where at least the evaluation protocol is fair and I don't see a reason why Table 6 should be treated differently. Again, even if the numbers are not good enough, it would still help the community by establishing a standard for general purpose audio model.\n\nOverall, this work is interesting in terms of exploring general audio processing model, but the evaluation should be improved. I would happily raise my score if the above-mentioned concerns can be addressed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing our contribution and novelty! We take every comment seriously and address every concern point by point. We hope that our response resolves your concerns. Any follow-up questions are welcome.\n***\n**Questions 1. Table 3: In Pengi's framework, CLAP serves as a feature ... it will still add value to this work as a first step of exploring LLM's application in audio.**\\\nAs the reviewer suggested, our goal is to add value exploring LLM's for audio rather than establishing SoTa in every task. For Table 3, we chose CLAP because it is the only Zero-Shot model with a comprehensive evaluation (16 downstream tasks). The next best evaluation was only on 8. Thus, providing no evidence of performance across other domains like speech and music, which tend to be the most difficult. For the rest of the Tables, we compared against SoTA results even if it came from different models and learning methods. We compared against SoTa Zero-Shot models in Table 8, a subset of Table 3, for Sound Event Classification. Even against SoTa from supervised learning models in Table 5 and 7 for Audio Q&A and Audio Captioning respectively. Table 9, against SSL, supervised, and trained on speech audio. To enhance our baseline from Table 3, we trained a CLAP model on the same amount of pairs data as Pengi (3.4M) to put aside variations due to data. Pengi's strong performance still holds. We attach this as Table 2 in PDF in the global response.\n\n**Questions 2. Table6/Section5.4: It is unclear how the authors obtained the retrieval performance ... it would still help the community by establishing a standard for general purpose audio model.**\\\nWe acknowledge the reviewer’s point. Models trained with contrastive learning learn the similarity between audio and text and can do audio-to-text and text-to-audio retrieval in one step. Text generation models like standard audio captioning models and Pengi, don't learn the multimodal similarity and thus require additional steps to compare text and audio. Therefore, in Table 6, we compared Pengi using a setup proposed by Kim et al. [1] and the best-performing models using the same setup. Due to the inherent difference between contrastive and generative models, there is no fair comparison. However, for a purely numerical comparison, contrastive learning methods surpass all text-generation methods (Pengi or others) for the retrieval task. We explain this in Section 6 paragraph 1. We will include these numbers in Table 6 to provide a reference to the readers.\n\n[1] Kim, Minkyu, Kim Sung-Bin, and Tae-Hyun Oh. \"Prefix tuning for automated audio captioning.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\n**Questions 3. The use of text encoder: line 112 said it's fundamental but there seems to be no ablation on this. Having LLMs taking text as input seems to be more intuitive. Any idea why?**\\\nWe appreciate the reviewer's question. We conduct an experiment and discuss the findings and results in global response question 1.\n\n**Questions 4. For line 270: \"... by 32%\", isn't this 22-ish%?**\\\nWe are sorry for any misunderstanding. We report relative percentage improvements and not absolute percentage improvements. Since AudioCLIP and Pengi score 0.694 and 0.92 respectively on ESC50, we calculate the relative improvement as (0.92-0.694)/0.694 = 32.5%.\n\n**Questions 5. Generalizability (data): Table 8 shows that in the cases (namely, ESC50, US8K, and DCASE17) where Pengi have not seen the dataset, results are not as competitive. It would be good if more details on evaluation pipeline can be provided and interesting to see some error analysis.**\\\nWe evaluated the generalizability (data) of Pengi by testing it on 22 downstream tasks (Table 3) and comparing its performance with different SoTA models in the literature (Table 4-9). Pengi’s zero-shot performance achieves SOTA on several but not all downstream tasks. Table 8, a subset of Table 3, evaluates zero-shot sound event classification on 4 tasks and includes the best models in the literature. None of the models, including Pengi, have seen the dataset during training. We believe Pengi's results are competitive at 92% vs SoTA of 91% for ESC50 and 72% vs SoTA of 77%. To better understand the errors made by Pengi, we conduct an error analysis. We identify three types of errors that cause a decline in Pengi’s performance. We classify them into audio concept errors, hierarchy errors, and text-matching errors and describe them in Appendix Section F.\n\n**Questions 6. Generalizability (format): All the tasks this paper considered are covered by the templates hence the input is not in free form. One of the key strength of instruction-tuned LLM is the ability to take any input and answer accordingly. are there examples where the question (text input) is never seen during training?**\\\n We apologize for any confusion and we would clarify in the manuscript that:\n - Pengi can handle different ways of phrasing the same prompt, thanks to its text encoder. For instance, it can recognize that “this is a sound of” and “detect sound events” are asking for the same kind of output (e.g.“dog barking\") even if the template only has “this is a sound of”. If the prompt is completely different from the ones it was trained on, Pengi defaults to generating metadata as a general response.\n- We agree with the reviewer that it is an interesting challenge to make LLMs respond to any input appropriately. We have explored this direction in Appendix Section A, where we show that the user can use a default text prompt like “generate metadata” and then provide more information or ask follow-up questions. This enables the user to steer the conversation with additional unseen prompts (Fig. 6), such as “the background is”, “mention forest.”, etc. We acknowledge that this is not Pengi's strength. The literature suggests that scaling up training data could improve this issue."}, {"review_id": "9lFGGr37iS", "reviewer": "Reviewer_4yQP", "summary": "Pengi is a novel audio language model that leverages transfer learning to frame all audio tasks as text-generation tasks. It takes an audio recording and text as input, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder, and the text input is represented similarly by a text encoder. Both sequences are then combined as a prefix to prompt a pre-trained frozen language model. When evaluated on 22 downstream tasks, Pengi achieved state-of-the-art performance in several of them. ", "questions": "1. Please add more baseline - a cascade model for each task.\n2. Does the model has capabilities of ASR?\n3. Can you add ablation analysis of each component? ", "limitations": ".", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The proposed model is a novel audio language model that can be used for multiple audio tasks, including close-ended and open-ended tasks. It does not require any additional fine-tuning or task-specific extensions. \n\n2. The paper introduce a new learning framework that frames all audio tasks as audio and text input to text output tasks. This framework uses a single training procedure and a captioning objective function. For training, Pengi uses new audio task templates inspired by Instruction Tuning.\n\n3. Pengi has been extensively evaluated on 21 (or 22?) downstream tasks across various audio domains. It achieved state-of-the-art performance in several of these tasks, establishing a baseline for general-purpose ALM.", "weaknesses": ".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing our contribution! We address every question and hope that our response resolves your concerns. Any follow-up questions are welcome. \n***\n**Questions 1. Please add more baseline - a cascade model for each task.**\\\nFor Table 3, we chose CLAP because it is the only Zero-Shot model with a comprehensive evaluation (16 downstream tasks). The next best evaluation was only on 8. Thus, providing no evidence of performance across other domains like speech and music, which tend to be the most difficult. For the rest of the Tables, we compared against SoTA results even if it came from different models and learning methods. We compared against SoTa Zero-Shot models in Table 8, a subset of Table 3, for Sound Event Classification. Even against SoTa from supervised learning models in Table 5 and 7 for Audio Q&A and Audio Captioning respectively. Table 9, against SSL, supervised and trained on speech audio. Training with ensemble models (cascade) will provide insights into how methods compliment, but it was our scope.\n\n**Questions 2. Does the model has capabilities of ASR?**\\\nPengi was not trained on any speech audio-transcript data, so it does not support ASR. We believe that the key to integrating audio and speech tasks is to develop a universal audio encoder architecture, which is an exciting and important direction for the future, but beyond the scope of this work.\n\n**Questions 3. Can you add ablation analysis of each component?**\\\nWe performed different ablation studies and a new experiment for text encoder: \n1) We study the choice of text encoder and its mapper- m2 for Pengi. We denote exp A as Pengi without both the text encoder and m2 and exp B as Pengi without the text encoder. In exp A, we found that removing both m2​ and the text encoder resulted in a loss of coherence between the input text prompt and the output text. For example, an input prompt about identifying an emotion class \"the emotion is \" resulted in random text output and thus random performance. In exp B, we found comparable performance to Pengi. We attach this as Table 1 in PDF in the global response. \n2) We analyzed the choice of text prompt in performance in Appendix Section C, and found that the prompt \"generate metadata\" is a good default choice. \n3) We analyzed the prefix output from the audio encoder-mapper in Appendix Section B and found that it contains relevant keywords present in the output text, but can be noisy. \n4) We analyzed freezing and unfreezing the audio encoder in Appendix Section E, and found that unfreezing the encoder yielded better performance."}]

Author rebuttal:
We sincerely thank all the reviewers for recognizing our contribution and providing constructive feedback, especially for acknowledging that **this paper presents a novel audio language model** (Reviewer vWxh, 4yQP), **Performance on audio captioning is solid.** (Reviewer sUJx), **One of the first approaches that incorporate LLMs to create a general purpose audio LM** (Reviewer iz2N, sUJx) and **the evaluation is pretty extensive. Authors compared the proposed model on multiple benchmarks against multiple models** (Reviewer o28f, Reviewer vWxh, Reviewer 4yQP).
***
We would like to re-emphasize the novelty and technical contributions of this work. We present a novel and unified model that can handle both open-ended and close-ended audio tasks without relying on external modules or fine-tuning. Our key contributions are: (1) Pengi, the first Audio Language Model (ALM) in the literature, (2) audio task templates inspired by Instruction Tuning, and (3) an extensive evaluation on 22 downstream tasks showing that our unified model can achieve competitive and even sota performance in several tasks.

We summarize the two main questions brought up by the reviewers and address them here: \
**Question 1: Why is an explicit mapping needed for input text? What's the purpose of m2 here and do you have experimental results to show using m2 is helpful ?**\
We need a mapping network $m_2$ to bring the output of the text encoder into the space of the LM. The text input is a sentence that is passed to the text encoder. The encoder outputs a sentence-level embedding that is passed to $m_2$, which outputs a sequence of embeddings that the LM can "understand". We conducted two experiments to evaluate the effect of omitting $m_2$ and/or the text encoder. We denote Exp A as Pengi without the text encoder and $m_2$ (input text directly to LM), and Exp B as Pengi without the text encoder​ but with $m_2$ (input text to $m_2$).​ In Exp A, we found that removing $m_2$ resulted in a loss of coherence between the input text prompt and the output text. For example, an input prompt about identifying an emotion class "the emotion is " resulted in random text output and thus random performance. In Exp B, we removed text encoder but retain $m_2$. We obtained slightly lower results than our proposed architecture with both components. We attached this as Table 1 in the PDF below.

**Question 2: Concern about using CLAP as baseline. Whether the performance gains are due to additional data or Pengi's ALM formulation?**\
We chose CLAP as the baseline because it is the only Zero-Shot model with a comprehensive evaluation (16 downstream tasks). The next best evaluation was only on 8 tasks. Thus, providing no evidence of performance across other domains like speech and music, which tend to be the most difficult. For the rest of the Tables, we compared against SoTA results even if it came from different models and learning methods. We compared against SoTa Zero-Shot models in Table 8, a subset of Table 3, for Sound Event Classification. Even against SoTa from supervised learning models in Tables 5 and 7 for Audio Q&A and Audio Captioning respectively. Table 9, against SSL, supervised and trained on speech audio models. 

To enhance our baseline from Table 3 and answer whether the performance gains are due to additional data or Pengi's ALM formulation, we performed a new experiment. We trained a CLAP model (CLAP*) on the same amount of 3.4M audio-text pairs as Pengi. We observed Pengi's strong performance still holds. We attached this as Table 2 in the PDF.

Meta-review: The paper presents Pengi, an "audio language model" (ALM) which is a text language model that is conditioned on a both audio and text prefixes. The name is misleading, as the model doesn't generate audio and the language modeling over audio tokens is not central to the generation, it is a prefixing/conditioning mechanism. Pengi is compared to CLAP in various audio tasks, where is mostly beats it. There is no support for ASR, which admittedly is a special(ized) task. The contributions of the paper are relatively limited, but several reviewers see merit in unifying lots of audio tasks in a "prompting+LLM generation" way. The method is simple and the paper is easy to follow. The rebuttal convinced some reviewers of the experimental validation but not o28f, who had valid concerns. Overall, the paper is borderline (in term of scope and results), but being sound and SOTA on enough benchmarks with a single model, it deserves inclusion in NeurIPS.
Decision: Accept (poster)