SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 12: CLAP zero-shot performance on ESC50
Audio Captioning ↑	Audio Q&A ↑	Sound Event Classification ↑
Model	AudioCaps	Clotho	ClothoAQA	ESC50	FSD50K	US8K	DCASE17 Task 4
Frozen Pengi	0.4535	0.2577	0.6395	0.8950	0.4117	0.6319	0.3225
Pengi	0.4667	0.2709	0.6453	0.9195	0.4676	0.7185	0.338
Acoustic Scene Classification↑	Music ↑	Instrument Classification ↑	Music Note Analysis↑
Model	TUT2017	Music Speech	Music Genres	Beijing Opera	Instrument family	NS. Pitch	NS. Velocity	NS. Qualities
Frozen Pengi	0.3449	0.9219	0.2550	0.4814	0.2949	0.7131	0.3330	0.3830
Pengi	0.3525	0.9688	0.3525	0.6229	0.5007	0.8676	0.3728	0.3860
Emotion Recognition↑	Vocal Sound Classification↑	Action Recog.↑	Survei llance.↑
Model	CRE MA-D	RAV DESS	Vocal Sound	ESC50 Actions	SESA
Frozen Pengi	0.1816	0.1312	0.5371	0.5196	0.5316
Pengi	0.1846	0.2032	0.6035	0.5277	0.5402
Table 13: The model 'Frozen Pengi' indicates Pengi with audio encoder frozen. The '-' symbol indicates numbers were not available while  $X$  indicates that the model cannot support the task. Higher is better for all numbers. The evaluation metric is mAP for FSD50k, AudioSet, and NSynth sonic; F1 score for DCASE17; and SPIDEr for AudioCaps and Clotho captioning. All other downstream tasks use Accuracy.
Audio Captioning ↑	Audio Q&A ↑	Sound Event Classification ↑
Model	AudioCaps	Clotho	ClothoAQA	ESC50	FSD50K	US8K	DCASE17 Task 4
Exp B	0.4857	0.2545	0.6316	0.9215	0.4478	6882	0.3314
Pengi	0.4667	0.2709	0.6453	0.9195	0.4676	0.7185	0.3380
Acoustic Scene Classification↑	Music ↑	Instrument Classification ↑	Music Note Analysis↑
Model	TUT2017	Music Speech	Music Genres	Beijing Opera	Instrument family	NS. Pitch	NS. Velocity	NS. Qualities
Exp B	0.3241	0.9609	0.317	0.6864	0.5	0.8591	0.3708	0.377
Pengi	0.3525	0.9688	0.3525	0.6229	0.5007	0.8676	0.3728	0.386
Emotion Recognition↑	Vocal Sound Classification↑	Action Recog.↑	Survei Istance.↑
Model	CRE MA-D	RAV DESS	Vocal Sound	ESC50 Actions	SESA
Exp B	0.1728	0.1769	0.5798	0.5282	0.4923
Pengi	0.1846	0.2032	0.6035	0.5277	0.5402
Table 14: Exp B is Pengi with mapper  $m_2$  but without the text encoder. The evaluation metric is mAP for FSD50k, AudioSet, ESC50-Actions, and NSynth sonic; F1 score for DCASE17; and SPIDEr for AudioCaps and Clotho captioning. All other downstream tasks use Accuracy.
F Effect of text encoder
Pengi's architecture in Figure 2 consists of a text encoder  $g_{\psi}$  that transforms the input text into text embeddings. Then a mapping network  $m_2$  converts these embeddings into a sequence of k embeddings. A natural question that arises here is "Why is an explicit mapping needed for input text?" We conducted two experiments to evaluate the effect of omitting  $m_2$  and/or the text encoder. We denote Exp A as Pengi without the text encoder and  $m_2$  (input text directly to LM), and Exp B as Pengi without the text encoder but with  $m_2$  (input text to  $m_2$ ). In Exp A, we found that removing resulted in a loss of coherence between the input text prompt and the output text. For example, an input prompt about identifying an emotion class "the emotion is" resulted in random text output and thus random performance. In Exp B, we removed the text encoder but retained  $m_2$ . The Exp B architecture is depicted in Fig 7 and its results are shown in Table 14. By removing the text encoder, the model performs slightly lower than the proposed architecture with both components.
Figure 7: Pengi architecture without the text encoder  $g_{\psi}$ . The text prompt is tokenized and embedded by text embedder, followed by the mapping network  $m_2$ . The results of this architecture are shown in Table 13
Audio Captioning ↑	Audio Q&A ↑	Sound Event Classification ↑
Model	AudioCaps	Clotho	ClothoAQA	ESC50	FSD50K	US8K	DCASE17 Task 4
CLAP*	X	X	X	0.8916	0.3398	0.7661	0.3387
Pengi	0.4667	0.2709	0.6453	0.9195	0.4676	0.7185	0.3380
Acoustic Scene Classification↑	Music ↑	Instrument Classification ↑	Music Note Analysis↑
Model	TUT2017	Music Speech	Music Genres	Beijing Opera	Instrument family	NS. Pitch	NS. Velocity	NS. Qualities
CLAP*	0.3037	1.0	0.479	0.4025	0.415	0.1337	0.2185	0.2545
Pengi	0.3525	0.9688	0.3525	0.6229	0.5007	0.8676	0.3728	0.386
Emotion Recognition↑	Vocal Sound Classification↑	Action Recog.↑	Survei llance.↑
Model	CRE MA-D	RAV DESS	Vocal Sound	ESC50 Actions	SESA
CLAP*	0.1512	0.1692	0.5522	0.508	0.7094
Pengi	0.1846	0.2032	0.6035	0.5277	0.5402
Table 15: We train a new CLAP* model on the same 3.4M pairs training data used Pengi. The  $\mathbf{x}$  indicates that the model cannot support the task. Higher is better for all numbers. The evaluation metric is mAP for FSD50k, AudioSet, ESC50-Actions, and NSynth sonic; F1 score for DCASE17; and SPIDEr for AudioCaps and Cloth captioning. All other downstream tasks use Accuracy.
G Different type of Pengi errors
There are three types of errors that lead to a drop in Pengi's performance. We categorize them into audio concept errors, hierarchy errors, and text-matching errors.
Audio concept errors. These types of errors are when the model gets the base audio concepts wrong. For example, while generating an audio caption, the model predicts it as "a sound of a dog barking in a neighboring field" instead of "a sound of door knocks with cars moving nearby". This indicates the model fails to detect the sound event of a door knock and confuses it with dog barking. These are Pengi model errors stemming from the audio encoder.
Heirarchy errors. The hierarchy error comes from a mismatch between Pengi's model prediction and the target domain classification. For example, in classifying sound events, Pengi predicts the sound as "domestic sounds", however for ESC50, the target classification requires a more fine-grained classification within domestic sounds like Vaccum cleaner, Toilet flush, brushing teeth, etc. If text matching is used for classification, then the model will not be able to categorize "domestic sounds" into any of the fine-grained classes. To solve this error and get a more fine-grained response, we can use improved text prompts or switch to the log-likelihood method.
Text-matching errors. The text-matching errors are the errors that result from the text embeddings or the text-matching method used. This means depending on the text embedding and similarity method used, the performance of Pengi on close-ended tasks will change.
H Constrastive Learning and Generative Pretraining
We compare our model Pengi with CLAP [15], a state-of-the-art Zero-Shot model that has been evaluated on 16 downstream tasks. However, CLAP is trained on a smaller amount of audio-text data. This leads us to ask: "Is the improved performance due to the larger training data or the generative pretraining?" We already know that generative pretraining allows us to perform open-ended tasks like Audio Captioning, AQA, which are not possible with contrastive models. But this does not tell us if: generative pretraining is beneficial for close-ended tasks like classification?. To answer this question, we train a CLAP model with the same data 4.1) that we use to train Pengi. We call this model CLAP*.
Results. The results are shown in Table 15. We see generative pertaining (Pengi) outperforming contrastive learning (CLAP*) on average. Moreover, with generative pretraining, the model can perform open-ended tasks like Audio Captioning and Audio Question Answering.
An interesting observation is Pengi outperforms human performance (81%) on ESC50. Humans have limitations inherent to how much information a participant can handle at once. In the case of ESC50, humans listen to the audio once, and have to remember the audio content, task description, and choose among 50 different classes. Moreover, listeners have different degrees of familiarity with prototypical content from different sound classes, whereas Pengi has been exposed to similar content during training. In a sense, Pengi is an expert listener, whereas the humans in the listening experiment were not.