Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data
Authors: Sreyan Ghosh
Sonal Kumar
Zhifeng Kong
Rafael Valle
Bryan Catanzaro
Dinesh Manocha
Venue: ICLR 2025 Poster (2025)
Keywords: audio classification
synthetic data
data-efficient learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A novel method to augment small-scale audio classification datasets with synthetic data.

Abstract: We present Synthio, a novel approach for augmenting small-scale audio classification datasets with synthetic data. Our goal is to improve audio classification accuracy with limited labeled data. Traditional data augmentation techniques, which apply artificial transformations (e.g., adding random noise or masking segments), struggle to create data that captures the true diversity present in real-world audios. To address this shortcoming, we propose to augment the dataset with synthetic audio generated from text-to-audio (T2A) diffusion models. However, synthesizing effective augmentations is challenging because not only should the generated data be acoustically consistent with the underlying small-scale dataset, but they should also have sufficient compositional diversity. To overcome the first challenge, we align the generations of the T2A model with the small-scale dataset using preference optimization. This ensures that the acoustic characteristics of the generated data remain consistent with the small-scale dataset. To address the second challenge, we propose a novel caption generation technique that leverages the reasoning capabilities of Large Language Models to (1) generate diverse and meaningful audio captions and (2) iteratively refine their quality. The generated captions are then used to prompt the aligned T2A model. We extensively evaluate Synthio on ten datasets and four simulated limited-data settings. Results indicate our method consistently outperforms all baselines by 0.1%-39% using a T2A model trained only on weakly-captioned AudioSet.

Published as a conference paper at ICLR 2025
SYNTHIO: AUGMENTING SMALL-SCALE AUDIO CLAS- SIFICATION DATASETS WITH SYNTHETIC DATA
Sreyan Ghosh Sonal Kumar" Zhifeng Kong", * RRfael Valle*, Bryan Catanzaro"
Dinesh Manocha
"NVIDIA, CA, USA, "University of Maryland, College Park, USA
(sreyang, sonalkum } @ umd.edu "equal technical contribution.
Project: htttps://sreyee git.gii.ihbu..... 1 ho//yntti///////ooioo
ABSTRACT
We present Synthio, a novel approach for augmenting small-scale audic classi- fication datasets with synthetic data. Our goal is to improve audio classification accuracy with limited labeled data, Traditional data augmentation methods, which introduce synthetic perturbations (e.g., injecting noise or masking segments), fail to replicate the inherent variability found in real-world audio. To address this short- coming, we propose augment the dataset with synthetic audio generated from text-to-audio (T2A) diffusion models. However, synthesizing effective augmen- tations challenging because not only should the generated data be acoustically consistent with the underlying small-scale dataset, but they should also have suf- ficient compositional diversily. To overcome the first challenge, we align the generations of the T2A model with the small-scale dataset using preference op- timization. This ensures that the acoustic characteristics of the generated data remain consistent with the small-scale dataset. To address the second challenge, we propose a novel caption generation technique that leverages the reasoning capabilities of Large Language Models (1) generate diverse and meaningful audio captions and (2) iteratively refine their quality. The generated captions are then used prompt the aligned T2A model. We extensively evaluate Synthio on ten datasets and four simulated limited-data settings. Results indicate our method consistently outperforms all baselines by 0.1%-39% using a T2A model trained only on weakly-captione AudioSet.
INTRODUCTION
Audio classification is the foundational audio processing task of understanding the input audio and assigning it to one multiple predefined labels. However, training audio classification models requires a lot of high-quality labeled data, which is not always readily available (Ghosh et al., (2022) Manually collecting and annotating large-scale audio datasets is expensive, time-consuming, and noisy process (Nguyen et al., 2017 Martin- data privacy and usage rights further hinder this process (Ren 2023).
& Mesaros 2021), and recent concerns about
Data augmentation, which involves expanding original small-scale datasets with additional data, promising solution address data scarcity. Traditional augmentation techniques attempt to diversify audio samples by applying randomly parameterized artificial transformations to existing audio. These methods include spectral masking (Park et al. (019), temporal jittering (Nanni et al.
2020), cropping (Nizumi et al., 2021). mixing (Seth et al. [023] ] Ghosh et al. 2023b Niizumi et al. 2021) and other techniques (Al-Tahan & Mohsenzadeh 2021] Saeed et al.] 2021 Manocha et al.
2021 Cherep & Singh 2025). Although these methods have been effective, they focus on surface- level patterns in the data rather than capturing the fundamental mechanisms that drive real-world data generation (Cherep & Singh 2025). As result, they statistically modify the data without directly influencing the causal mechanisms that produced it, leading to high correlations between augmented samples and limited control over diversity.
Generating synthetic data from pre-trained text-to-audio (T2A) models addresses the limitations of standard data augmentation techniques while retaining their strengths of universality, controlla- bility, and performance (Trabucco et al. 2024). The recent success of generative models makes
We use "audio" " refer acoustic events comprising non-verbal speech, non-speech sounds, and music.

---

Published as a conference paper at ICLR 2025
this approach particularly appealing (Long al. 2024 Evans al. 2024b). However, generat- ing synthetic audio presents unique challenges due to the complexity of waveforms and temporal dependencies (Ghosh et al.] 2024b). We highlight the 3 main challenges in generating effecttiee synthetic data for audio classification: i) Consistency with the original data: Synthetic audio that does not align acoustically with the original dataset can hinder effective augmentation and may cause catastrophic forgetting (Geiping et al. (022). This misalignment includes spectral, harmonic, and other inherent acoustic characteristics not easily controlled through prompts. Maintaining consistency with T2A models trained on internet-scale data remains a challenge, and standard fine-tuning can often lead to overfitting (Weili al. 2024 ii) Diversity of generated data: Ensuring compositional diversity in the generated synthetic data (e.g., sound events, temporal relationships, background elements, etc.) is critical for effective augmentation. Additionally, lack of diversity can lead to poor generalization and learning of spurious correlations, impacting performance. Simple, hand-crafted prompts (c.g., "Sound of a metro") often result in repetitive patterns, and creating diverse, mecningful prompts is labor-intensive. Complex prompts can generate audios that do not preserve the original label. fii) Limitations of current T2A models: T2A models often struggle to generate diverse audios and follow details in prompts. This is largely due to the lack of large-scale, open-source datasets for training, as well as the inherent complexity of non-speech audio domains (Ghosal et al. (223). These limitations highlight the need for more advanced approaches for synthetic data generation in audio.
Our Contributions. To address these challenges, we propose Synthio, a novel, controllable and scalable approach for augmenting small-scale audio classification datasets with synthetic data. Our proposed approach has 2 main steps: i) Aligning the Text-to-Audio Models with Prefer- ence Optimization: To generate synthetic audios with acoustic characteristics consistent with the small-scale dataset, we introduce the concept of aligning teaching with learning preferences. Specifically, align the generations of the T2A model (acting as the teacher) with the target char- acteristics of the small-scale dataset using pref- erence optimization. This approach ensures that the synthetic audios reflect the acoustic prop- erties of (or sound similar to) the downstream dataset, enabling the classification model (the student) to perform well on test data with sim- ilar characteristics. To achieve this, we train diffusion- -ased T2A model with preference op- timization, where audios generated from Gaus- sian noise are treated as losers and audios from the downstream dataset are treated as winners. ii Generating Diverse Synthetic Augmentations: To generate diverse audios for augmentation, we introduce the concept of language- audio imagination and imagine novel acoustic scenes with language guidance. Specifically, we generate diverse audio captions that are then used to prompt T2A models to generate audios with varied compositions. To achieve this, we propose MixCap, where we prompt LLMs iteratively to generate captions combining existing and new acoustic components. Additionally, we employ a self-reflection module that filters generated captions and prompts the LLM to revise those that do not align with the intended label. To summarize, our main contributions are:
Augrentation
Byn.Aag.
Syeeho (wor))
Covitaatin Acauney
0.6
0.5
0.4
200
300
Namber of Ganerated Aug signantation
Figure Performance comparison of Synthio with other augmentation methods on down-sampled ESC- 50 (100 samples). Traditional augmentation, such as SpecAug, degrades performance small-scal datasets. Naive synthetic augmentation outperforms traditional methods significantly but plateaus with higher sample counts. Synthio further enhances performance by gener- ating consistent and diverse synthetic data.
1. We introduce Synthio, a novel data augmentation approach for audio classification that expands small-scale datasets with synthetic data.
2. We evaluate Synthio across 10 datasets in 4 simulated low-resource settings. Synthio outperforms all baselines by 0.1%-39%
3. We conduct an in-depth analysis of the generated augmentations, highlighting Synthio's ability to produce diverse and consistent data, its scalability, and its strong performance on complex tasks such as audio captioning.
2 RELATED WORK
Data Augmentation for Audio and Beyond. Expanding or augmenting small-scale datasets with additional data has been widely studied in the literature. Traditional augmentation methods, which
2

---

Published as a conference paper at ICLR 2025
apply randomly parameterized artificial transformations to data during training, remain the most common approach across language Wei & Zou| (2019); Karimi et al. (2021). vision (Shorten & Khoshgoftaar 2019; Wang et al. 2017 Yun et al. 20199, and audio (Park et al. 2019: Spijkervet
2021). For audio, specific techniques include SpecAugment, adding background noise, reverberation, and random spectrogram transformations. With the emergence of generative models, synthetic
data augmentation has been increasingly adopted for language (Ghosh et al.) [203a]] 2202cc Chen al. 2202]) and vision (Trabucco et al. 2024. Zhao al., 2024), proving be more effective than
traditional methods. These approaches generally incorporate explicit steps to ensure the consistency and diversity of generated augmentations.
contrast, the application of synthetic data to audio and speech remains underexplored. Recent attempts include generating synthetic captions for improving aadio-language pre-training (Xu et al. 2023), improving T2A models with synthetic captions (Kong
et al. 2024) and scene classification (Ronchini et al. 2024 Feng et al. 2024 Cherep & Singh 2025
Few- and Zero-Shot Audio Classification. Few-shot audio classification focuses on training models to classify audio samples with very limited labeled data per class, often leveraging transfer learning or meta-learning approaches (Zhang et al. 2019: Wang et al.; 2021 Heggan et al. 2022). In contrast, zero-shot audio classification enables models to generalize unseen categories without direct training on those classes, relying on learned representations or external knowledge (Xie & Virtanen 2202]] Elizalde et al., 2023). Synthetic data research complements these by generating additional labeled data, improving model performance under low-resource settings while addressing data scarcity without directly requiring labeled instances from the target categories.
Text-to-Audio Generation. In recent years, there has been a significant surge in research on text- to-audio (T2A) models. The most popular architectures include auto-regressive models based on codecs (Kreuk et al.] [2023] Copet et al. 2024) and diffusion models Liu et al. (2023); Ghosal al. (2023); Evans et al. (2024)). Clotho (Drossos et al., 2020) and AudioCaps (Kim et al. 2019) remain the largest human- annotated datasets for training these models. However, large-scale datasets for T2A model training are still searce. Recently, Yuan et al. (202)) synthetically captioned AudioSet (Gemmeke et al. (017). demonstrating its effectiveness for training T2A models. For downstream adaptation, carlier works have primarily relied on Empirical Risk Minimization (ERM). Majumder et al. (2024) introduced preference optimization for T2A models, creating a synthetic preference dataset based on scores provided by a CLAP model (Elizalde et al. 2023).
BACKGROUND
Diffusion Models. Diffusion models consist of two main processes: a forward process and a reverse process. Given a data point To with probability distribution p(æg), the forward diffusion process gradually adds Gaussian noise to To according to a pre-set variance schedule 7/, "',, "Yr and degrades the structure of the data. We request readers to refer to App. A.I for more details on diffusion models.
Reward Modeling, Estimating human preferences for particular generation Tn (hereafter treated as a random variable for language), given the context c, is challenging because we do not have direct access to a reward model xo). In our scenario, we assume only ranked pairs of samples are available, where one sample is considered "winner" (x)")) and the other "loser" (a/) under the same conditioning c. Based on the Bradley-Terry (BT) model, human preferences can be modeled as:
(x > xolc) = (r((, xb r(c, x())
(1)
where a represents the sigmoid function. The reward model r(c, m0) is parameterized by neural network á and trained through maximum likelihood estimation for binary classification:
LBT( () = -Be,ei ,af [log a (r (c, .=" - -r,(c, ]
Here, prompt e and data pairs (xi 20)) are drawn from a dataset labeled with human preferences.
RLHF: (Christiano et al. 2017) The goal of RLHF is to optimize a conditional distribution Po ( (zple). where c ~ D... such that the latent reward model r(c, xx) is maximized. This is done while regulariz- ing the distribution through the Kullback-Leible (KL) divergence from a reference distribution Pret- resulting in the following objective:
max Eo- do to- [I((, (0w..eereccoolllleeeeoooollll Pe
Here, the hyperparameter 33 controls the strength of regularization.
3

---

Published as a conference paper at ICLR 2025
DPO DPO directly optimizes the conditional distribution po(2olc) to align data generation with the preferences observed in (any form of) feedback. The goal to optimize the distribution of generated data such that it maximizes alignment with human preference rankings while maintaining consistency with the underlying reference distribution Pre((Tolc).
The optimal solution pa (2 lc) for the DPO objective can be expressed as:
Pg To c) Pref( exp(r(e, 1B)
(4)
Z(c)
where 7(c) is the partition function, defined as:
(c)) Σ Pra((zolc) exp(r(e, xx)
(5)
This term ensures proper normalization of the distribution, and 3 controls the regularization, balancing between adherence the reference distribution and preference maximization. The reward function r(c,zo) is then reparameterized as:
pi (xolc)
r(c, = 3 log
(ooc) + 3 log (c))
(6)
Using this reparameterization, the reward objective can be formulated as:
Po(xū B log Prer(=0'
pe po le) prer(=//l))
(DPo = -ee an
logor
B8log
(7)
By optimizing this objective, DPO enables direct preference learning, optimizing the conditional distribution c)) in such way that better reflects human preferences, opposed traditional approaches that optimize the reward function first and then perform reinforcement learning.
DPO for Diffusion Models: Very recently, Wallace et al. (2024) propose formulation for optimizing diffusion models with DPO. The primary issue with optimizing diffusion with DPO is that the distribution pe (= is not tractable due to the need to consider all possible diffusion paths leading to To- To address this, Wallace et al. propose to leverage the evidence lower bound (ELBO) to incorporate latents F1:7. which represent the diffusion path. The reward R(c, Torr) accounts for the entire sequence, leading to the reward function:
(8) Instead of directly minimizing the KL-divergence as typically done, they propose to utlize the upper bound of the joint KL-divergence ID k s. ( (e) optimization objective, enhancing the practicality training diffusion models with preferences. The new objective, aiming maximize the reward and match the distribution of the reverse process of po to the reference model Pref. is given by:
r(c, mo) Epolsur R((,, (o:r)
This is integrated into the
max -e,zoo-eeeerre poooo-oooooorrrrrrrero (re, mn)] BID aiDDolla
Pe
Training efficiency is improved by approximating the intractable reverse process using forward approximation q(=1 T T(x The DPO then integrates this into the loss function, which involves comparing the log likelihood ratio of the probabilities under pa and Pret for winning and losing paths:
Po (1 w (zo)
3TT log Po (o
LDPO Copo.Diffusson (0) - Ee, -Pm logo o
BTT log
Prer Ix
Peet ( la())
(10)
After applying Jensen's inequality to take advantage of the convexity of log o, we push the expectation outside, allowing us to simplify the objective. By approximating the denoising process with the forward process, the final form the loss for DPO in diffusion models becomes:
(0) -Ee,,e ,ah)) t, et [log a (-BTW(>;)AL]
(11)
where AL is the L2 weighted noise estimation losses between the preferred (winner) and less preferred (loser) samples.
METHODOLOGY
Let Damall = {(a, 4), 1 s i n) be a high-quality, small-scale human- annotttdd audio classification dataset with audio-label pairs. Let 'D. be a potentially noisy, large-scale weakly-captionee dataset of audio-caption pairs with zero intersection with Damall- Our goal is to train T2A model T using
4

---

Published as a conference paper at ICLR 2025
Da.c. then use it to generate synthetic dataset Don and then finally add it to smmall (now attributed as Dirain) to improve audio classification performance. This is accomplished through two key steps: first, aligning the generations from T with the acoustic characteristics of Dsmmal, and second, generating new captions to prompt T for creating synthetic audio data.
4.1 ALIGNING THE TEXT-TO-AUDIO MODEL USING PREFERENCE OPTIMIZATION
T2A models trained on internet-scale data often gen- erate audio that diverges from the characteristics of small-scale datasets, resulting in distribution shifts. These mismatches can include variations in spectral (c.g., frequency content), perceptual (e.g., pitch, loud- ness), harmonic, other acoustic characteristics This misalignment arises from the non- -on-determ nature of T2A generation and it is impractical to pro- vide detailed attributes (like "loud" or "high-pitched") in prompts, (i) there no scalable methods for extracting specific attributes for each label, and (ii)
Alaredd Tecddt-
Audio Model
Ar
Valn
Alepht Tramnng
P
Woviing Autlic
Lowing Aution
Ppref
Text-ta
Model
T2A models struggle with accurately following fine- grained prompt details (Wang et al. 2024).
Pamalll
Gausda Naka
Audian
Centaice
Senadiid (1.84))
To address these issues. we propose the concept of aligning teaching with learning preferences. Our ap- proach assumes that the classification model (viewed as the student) performs better when trained on syn- thetic audio that closely matches the inherent acoustic properties of our high-quality and human-labeles model (viewed as the teacher) to D samali. ensuring that the generated augmentations align with the de- sired characteristics and sound similar, ultimately enhancing the student model's 's ability to generalize to similarly characterized test data. As shown in Fig. 2
We propose to align the T2A model 7 with the small-scale datasct D amllll using DPO. This helps us generate audios with acoustic char- acteristics aligned to that of D smill-
Figure 2:
smmmll Thus, we align the generations of the T2A
we achieve this using preference optimization
(DPO in our case) and align generations of T with D Unlike standard fine-tuning, which can lead to less diverse outputs and overfitting due to narrow focus on minimizing loss, preference optimization encourages greater exploration in the model's 's output space, preventing mode collapse and fostering more diverse augmentations. Additionally, DPO leverages pairwise learning, offering richer training signals compared to the independent outputs used in standard fine-tuning, further mitigating overfitting risks. We detail our two-step approach for DPO optimization below:
Step 1: Construction of the Preference Dataset. To create our preference dataset Ppeet {(a"" w .ai)... (af a.)}. we first generate template-based captions for each instance in D Dsmall in the form: "Sound of label", where label is the category associated with the audio. For each instance, we prompt the T2A model j times, with all generations starting from randomly initialized Gaussian noise (generation configuration is detailed in Section 5). Each generated audio is then paired with the corresponding
boundd-rruth audio from the gold dataset. This resulting 'D Pprer dataset has n X instances, where the generated audio is treated as the "loser" and the ground-truth audio as the "wiinnrr". This simple approach has proven highly effective in aligning generations by generative models by prior work (Majumder et al. 2024 Tian et al. 2024).
Step 2: Preference Optimization Using DPO. After constructing Dpret= we train our T2A model on this dataset with DPO using the approach outlined in Section 3 The resulting aligned model
referred to as T Details of the hyper-parameter used for training are provided in Section
4.2 GENERATING DIVERSE SYNTHETIC AUGMENTATIONS
is not well-studied in the literature on how to leverage synthetic audio generation for downstream tasks. The only existing work relied on manually crafted prompt templates (e.g.. "Sound of (label)") (Ronchini et al. 2024). has a significant limitation: there is no precise control over
'When prompted with "sound of a bus" for the category "bus" in the TUT-Urban dataset, the generated audio may not reflect the typical bus sounds in Furopean cities (where TUT was recorded), as hus sounds can vary by region, with some featuring loud engines and dense crowds while others have quieter engines and sparse crowds.
5

---

Published as a conference paper at ICLR 2025
Prampi
Pronge
MixCap
Prompt
LLM
Exlating Acoustic Components
LLM
Audio Captions
Rejacted Audios
Audio Captioning
Model
Accepted Audios
CLAP Filtering
New Audio Captions
Self-Reflection
Small-Scale Dataset
Synthetic
Text-toAAudio
Data
AST
Trainable
Prozen
Generated Audios
Figure 3: Overview of our proposed Language-Ginn Guided Audio Imagination for generating diverse synthetic augmentations. Starting with the small-scale dataset, we first generate audio captions and use an LLM to extract acoustic components (Prompt 1). Using these components and audio labels, we prompt the LLM to generate new and diverse captions (Prompt 2). which are then used to prompt the aligned T2A model for audio generation. The generated audios filtered for label consistency using CLAP, with accepted audios added to the final synthetic dataset. Rejected audios undergo caption revision (Prompt 3) through self-reflection process, and the revised captions are used to regenerate audios, iterating this process times. Example captions are in Table
the specific components in the generated audio for a given caption. This can result repetitive or completely inconsistent patterns, particularly with weaker T2A models] These could bias the model to learn spurious correlations, known issue in synthetic data augmentation (Ghosh et al. 2024c)
While the alignment stage helps the T2A model generate audio with acoustic characteristics similar to the small-scale dataset (e.g., spectral, harmonic, etc.), it does not fully account for the compositional diversity of the generated audios (e.g., sound events, their temporal relationships, background elements). To tackle this, we propose the concept of language-guided audio imagination, where propose to imagine novel audios guided by language. Specifically, we leverage the reasoning abilities of LLMs to generate diverse and meaningful captions for category label in controlled yet scalable manner. These captions are then used to prompt our aligned T2A model for generating novel audios.
4.2.1 GENERATING DIVERSE PROMPTS WITH MIXCAP
We propose MixCap, prompt generation method that creates diverse and effective captions in three steps: First, we employ GAMA (Ghosh et al.) 2024a) caption all audio files in D Next, we prompt LLM to extract phrases describing the acoustic components of the audio. These components correspond to the acoustic elements such as backgrounds and foreground events, and their attributes and relations, etc. (see prompt in Appendix A.2 Finally, for each training instance in D smalls we prompt the LLM with the ground-tre -ruth label and the extracted components from all instances generate N diverse audio captions that blend existing and new components.
4.2.2 FILTERING & SELF-REFLECTION
Filtering. After generating captions and their corresponding audio, we filter the audio for label consistency. While LLMs can generate diverse captions, the audio produced must remain aligned with the ground -ttth label. To ensure this, we use CLAP to evaluate the generated audio, accepting those that meet similarity threshold of p% and rejecting the rest. We denote the accepted audios as Dig and the rejected ones D Our CLAP model is pre-trained on Da.e. and fine-tune the last layer with Demall to adapt to the target dataset. Example captions are in Table
Self- Renf-Refeetion. For the rejected audios in Drej syn we prompt the LLM to reflect on its generated captions and revise them to better align with the target label. Precisely, we feed the LLM with the
For example, when prompted with "Sound of park", we observed that out of 10 times, the model generated the sound of children playing as part of the generated audio. On the other hand, when prompted with "Sound of airport", the model generates audios with background announcements, which could vary by region.

---

Published as a conference paper at ICLR 2025
original caption of each rejected audio along with extracted components from all accepted captions in Day and task it to rewrite the rejected captions. The revised captions are then used to generate new audio, which is again filtered using CLAP. Audios that meet the threshold are accepted, while ones that don't go through the process. This repeats for iterations or until there are no rejected samples.
Fine-tuning for Audio Classification. After the self-reflection stage, the final set of accepted synthetic audios is denoted as D Dayn. containing = N x n audio-label pairs, where N represents the augmentation factor (e.g.. with 100 gold samples, we generate 100 x N synthetic samples). This is then combined with D small to form the final training dataset D Duain for audio classification.
EXPERIMENTAL SETUP
Models and Hyper-Parameters For our T2A model, we choose the Stable Audio architecture (Evans et al. 2024b). We train the model from scratch on Sound-VHCaps VHCaps (Yuan et al. 2024) (with =1.5 million weakly captioned audio-caption pairs) to avoid any data leakage. For training, we employ batch size of 64, an AdamW optimizer, a learning rate of 5e-4, and a weight decay of le-3 for 40 epochs. For DPO-based alignment tuning, we generate of 32 and a learning rate of 5e-4 for 12 epochs. For our audio classification model, we employ the Audio Spectrogram Transformer (AST) (Gong al. 2021) (pre-trained the AudioSet dataset) and fine-tune it with a batch size of 24 and a learning rate of Te-4 for 50 epochs. For CLAP filtering, we employ p= 0.85. For prompting our diffusion model use Text CFG=7.0. In each experiment, adjust the number of generated augmentations N (ranging from I to 5) based on performance on the validation set. All results are averaged across 3 runs.
2 losers and fine-tune with a batch size
Datasets, We create small-scale datasets by downsampling commonly used audio classification datasets to n samples. Our selected datasets include a mix of music, everyday sounds, and acoustic scenes. For multi-clase classification, we use NSynth Instruments, TUT Urban, ESC50 (Piczak). USD8K (Salamon et al. 2014), GTZAN (Tzanetakis et al., 200T), Medley-solos-Dee (Lostanlen & Cella 2017), MUSDB 18 (Rafii et al. 2017). DCASE Task 4 (Mesaros et al.) 2017). and Vocal Sounds (VS) (Mesaros et al. 2017). evaluating them for accuracy. For multi-label classification, we use the FSD50K (Fonseca et al. 2022) dataset and evaluate it using the Ei metric. We exclude AudioSet from evaluation as Sound-VECaps VECaps is derived from it. To ensure a downsampled dataset that has label distribution similar to that of the of the original dataset, we employ stratified sampling based on categories. Our experiments are conducted with n = {50, 100, 200, 500) samples, and we downsample the validation sets for training while evaluating all models the original test splits.
Baselines. Our baselines include: (i) Gold-only (No Aug.): We employ only the small-scale dataset for training and do not perform any augmentations. (ii) Traditional augmentation baselines: SpecAugment, Noise Augmentation (we either add random Gaussian noise or background noise from AudioSet and present averaged results), Pitch and Time Shift and Audiomentations (Jordall 2021 combination of the AddGaussianNoise, TimeStretch, PitchShift, Shift, SpecFrequency Mask, TimeMask and TimeStretch combination with the highest average score on 4 datasets and splits and was selected after grid search over all possible combinations). (iii) Generative baselines: Vanilla Synthetic Augmentation (Vanilla Syn. Aug.)- we prompt To with template captions), Vanilla Syn. Aug. I.LM Caps we prompt Ta with random captions generated with L.M.s. (iv) Finally, inspired by Burg et al. (202)), we also employ retrieval baseline where instead of generating augmentations from our T2A model trained on Dac. we just retrieve the top-n instances (w.r.t. CLAP similarity) from the AudioSet for each instance in small as our augmentations.
Ablations. We ablate Synthio with: (i) w/o Reff-teetec:io:: We remove the repetitive self-reflection module and iterate and filter only once; (ii) w/o DPO: We skip the tuning step and prompt the
un-alined T for augmentations; (iii) w/ ERM: We replace DPO tuning with standard Empirical
Risk Minim based fine-tuning with diffusion loss; (iv) w/ Template Captions: We remove MixCap and self-reflection modules and prompt Tell with template captions; (v). w/o MixCap: Similar our Random Captions baseline, but retain all other modules of Synthio.
6 RESULTS AND DISCUSSION
Main Results. Table I showcases the performance comparison between Synthio and the baseline methods. Synthio consistently outperforms all baselines by 0.1 1%-39% achieving notable improve-
7

---

Published as a conference paper at ICLR 2025
Table Result comparison of Synthio with baselines on 10 datasets and small- scale settings. n refers to the number of samples in the small-scale dataset augmented with synthetic data. Synthio outperforms our baselines by 0.1% % 39%. We also highlight the relative improvements by Synthio compared to the Gold-only.
Method
RSC-50
USD8K
GTZANN
Medley
TUT
NSynth
V8
MSDB
DCASE
FSDS0K
Gald-only (No Aug.)
22.25
55.09
47.05
47.23
33.32
77.49
56.85
12.09
Random Noise
18,50
57.42
45.20
46.55
35.86
32,42
76.41
52.55
13.21
3.05
Shifting
20.55
37.22
78.17
54.50
12.93
SpecAugment
19.50
58.36
46.00
47.18
36.73
27.32
77.27
53.25
12.81
7.93
Audiomentations
20.35
60.13
47.25
54.51
13:28
Retrivval
19.20
37.14
42.55
43.65
35.80
31.27
71.42
51.35
10.53
7.28
50
Vanilla Syn. Aug.
40.75
63.54
55.35
47.23
41.50
33,17
78.37
54,10
15,89
10.63
LLM Caps.
36,80
65.84
63.74
55.36
10.90
38.17
78.77
$7.05
13.07
10.70
Synthio (ours)
49.50.122
76.12.3m
68,20
64.58
43.84,
40.83,
80.67
60.15,
17.23
13,91
w/ Template Captions
41.25
66.11
64
41.57
37.52
78:57
59.00
14.15
w/ ERM
41,30
69.80
61.70
56.60
42,00
38.62
79.75
57.75
13.28
13.79
wio Self-Reelecctinn
72.57
64.55
12.81
39.50
78.56
15.63
13.74
wio MixCap
42,70
64.72
54.65
52.18
41.93
36.13
78,70
$8.80
14.82
12.52
DPO
68.12
56.10
79.03
$7.55
14.53
Gold-only (No Aug.)
56.75
72.89
64.15
$7.81
47.14
39.11
84.32
65.60
12.50
10.53
Random Noise
56.98
83.33
13.35
Plich Shifting
59,55
73.52
66.75
$8.46
47.50
39.53
85.07
68.25
12.19
13.11
SpecAuugmett
72.43
69.75
14.17
Audiomentations
48.50
73.82
71.05
59.52
$1.14
42.15
85.24
68.40
16.93
13:55
Retrioval
52.45
68.24
61 :5
54.83
45.39
37,84
83.27
58.55
10.93
10.05
100
Vanilla Syn. Aug.
77,25
77.31
68.25
63.58
49.96
42.31
84.78
63.55
15.73
12.63
LLM Cups.
67.05
79.73
67.50
41.83
84.83
65.95
16.32
13.25
Symthi jours)
83.35 Lm
85.00
71.20
71.23 179
$2.42,
44.92, 199
86.70
68.30
19.38, s95
16.35
w/ Template Captions
78.00
80.32
68.15
64.20
19,95
42.76
85,11
66.05
16.32
13.62
w/ ERM
73,20
81.81
67.25
66.57
$1.11
43.74
84.73
68.00
17.21
14.52
wio Self- Reffectel
77.65
82.38
69:55
69.52
51.75
44.38
82,53
66.20
15.89
12. 14
w/o MixCap
78.30
68.50
66.52
83.52
66.35
16.77
wio DPQ
66.75
75.46
66.15
60.81
48.78
40.31
84.67
67,85
14.83
12.53
Gold-only (No Aug.)
84.75
74,80
77.00
67.41
55.32
48.77
87.38
23.15
13.59
Random Noise
83.55
75.15
75.50
66.71
54,42
47.83
86.45
65.45
24.82
15:32
Pitch Shifting
84,90
74.48
78:55
67.74
55.44
48.12
87.47
23.11
17.51
SpecAngment
85,10
76.46
76:25
65.70
55.72
54.80
87.42
69.25
27.36
17.93
Audiomentations
75.80
77.30
70.50
Remreval
$2.55
71.20
73.65
65.80
53.25
47.63
86.28
63.55
19.51
15.36
200
Vanilla Syn, Aug.
77.96
77.10
78.97
72.95
LLM Caps.
85,00
78.37
79.55
74.14
54.73
56.21
87.02
73.16
28.40
18.14 14
Symthio (ours)
86.10
$2.81
$2,05
79,40 119%
56.83 .M
$7,10,
87.52
80.40,
32.81 amm
20.85
w/ Template Captions
85.95
60.84
79.25
77.56
55.99
56.33
87.25
74.55
29.12
19.04
w/ ERM
85,35
79.82
80.20
74.43
55.76
56.15
86.92
74.10
29.81
18.22
wlo Self Reflection
84.85
81.97
78.25
75.53
56.39
56.76
86.22
75.55
31.13
17.28
woo MixCap
$4.95
81.27
79.55
73.50
55.27
55,54
85,78
78.55
28.35
19:42
wio DPO
75.30
73.13
86.52
73.15
Gold only (No Aug.)
90.75
87.88
79.25
75.65
65.72
63,47
89.33
72.05
34.30
20.19
Random Noise
89.55
88.25
78,90
76.01
65.10
61.15
90.15
73.25
37.21
19:49
Pitch Shifting
$8.50
88.83
79.75
75.61
64,93
64,59
89.87
72.15
36.54
21.24
SpecAuugment
89.50
89:01
80:25
76.68
90.38
72.95
38.33
21.46
Andiomentatioos
$9.95
88.75
$1.25
77.66
66.92
65.21
91.34
73.65
38.75
23.11
Retriesal
77.25
73.62
61.44
70.20
30.17
500
Vanilla Syn. Aug.
91.50
$8.18
79.35
77.97
65,93
64,52
90.31
73.25
37.26
23.52
Caps.
$6.91
79.55
77.91
90.09
73.05
Syntho (ouzs)
92,10.25
89.18
82.25
78.62..r
67.81.35
65.40,35
91,42,05
74.70
39.24..a
23,89
Template Capsions
88.
44
76.64
64.71
73.35
22.35
w/ ERM
91.20
88.25
79.15
77.38
65.80
64.27
88.74
74.20
38.03
22.39
wio Self-Reellection
91.85
88.72
80.15
78.57
66.21
63.89
90.17
72.15
37.97
22.41
wlo MixCap
91.70
87.93
80.95
76.61
65.91
64.23
90.23
73.10
39.11
21.65
wlo DPO
90.115
88.21
79.45
76.03
66.01
63.61
89.83
72.65
37.04
20.19
ments in overall classification accuracy compared to Gold-only. The highest gains are observed on USD8K, while the least is on Vocal Sound, likely due to the T2A dataset's 'ss heavy representation of music compared the more sparse vocal sounds. Performance gains tend to decrease as the number of gold samples in D smaal grows, aligning with observed trends in prior studies. Detailed results on the full non-down- non-down-sampled sawp-sd datasets can be found in Appendix [..4. Although Vanilla Synthetic Augmentations emerge the strongest baseline, they lag behind Synthio by an average of 3.5%.
Ablations. The most significant performance drop in Synthio is observed w/o DPO, resulting in an average decline of 4.5%, highlighting the crucial role of consistency in generating effective augmentations. Second to w/o DPO, the highest drop is seen in w/ Template Captions, with an average decline of 2.7%, thus highlighting the importance of MixCap.
8

---

Published as a conference paper at ICLR 2025
NSyall
USD8K
10 30 03 23 24 DC R0
05
Figure 4: Comparison of spectral and pitch features between generated audios in Dyyn and real audios in D small (for n = 100), Synthio- Synthio-generated audios closely replicate the features of real data, demonstrating its ability to produce augmentations that maintain consistency with the original dataset (also see FAD scores in Sec. A.4.3
6.1 How CONSISTENT AND DIVERSE ARE AUGMENTATIONS GENERATED BY SYNTHIO?
Fig. 4 (analysis inspired from Cherep & Singh) compares the
Table 2: CLAP similarity score be- tween real audios and generated data.
distributions of pitch and various spectral features between gen-
erated audios in D and real audios in Dmmall across different methods on the USD8K and NSynth datasets. The features ana-
Lower scores show higher composi- tional diversity among generated augs.
lyzed include Pitch Salience (clarity of the main pitch) (Ricard
Method
USDSK(L)
NSywbiL)
2004.. Spectral Flatness (tonal vs. like quality) (Peeters
Vanilla Syn. Aug.
45.17
31.26
2004), Flux (rate of spectral change) (Tzanetakis & Cook
100
Synthin (oury) Templaic Captions
35,09
46.32
33,00
22.07
1999), and Complexity (level of sound detail) (Laurier et al.
ERM
50.01
42.35
Vanilla Syn Aug. Synthin (ours) w/ Templasc Coptions
47:22
$3.81
2010). Notably, Synthio-generatee
audios closely replicate the spectral features the original audios, showing the best align- ment among all methods and demonstrating Synthio's ability
200
34.58
46.84
2X03
97.16
w/ FRM
52354
43.98
to generate consistent augmentations. Table[presentt CLAP similarity scores between ground -tuth audios and their N generated augmentations, averaged across all dataset instances. Audios generated with Synthio achieve the highest compositional diversity for generated audios among all baselines. Table & that audios generated using Synthio have the highest similarity with the ground-truth category label.
6.2 How GOOD ARE SYNTHETIC AUDIOS GENERATED BY SYNTHIO?
Consistent with prior findings in vision (He et al.
Table 3: Performance comparison of Synthio with baselines syathetic-only audio classification.
2023), we observe that synthetic data alone performs sub-optimally compared to annottaed data.
Metbod
GIZAN
Vs
FUI
MSDB
However, our results show that enhancing the consis- tency and diversity of synthetic data aided by a small- scale version of the target dataset significantly im- proves model performance. Table comppres models trained exclusively on synthetic data with our base- lines (i.e., only Dayn is used for training AST). Syn- thio outperforms all baselines by 0.1%-26.255% with IPO-based alignment driving the improvements.
only Aug.)
$4.32
Vanilla Aug
29:05
16:13
21.00
35.00
100
Southio (ouzs)
33.08
35.26
24.51
56.45
w/ Temphic Captions
34.93
10.90
21.75
40:40
wi FRM
25.63
12:76
34.40
42:85
w/b DPO
17.00
21.57
20.39
30,20
Goit-only Ang
Vanilla Aug
41.95
39.15
Sauihiu (ours)
35.15
48.14
27.00
61.45
Templae Captores
70
15:53
21.81
41.30
wi ERM
28.10
36.29
25.71
46,70
win DIO
10.55
6.3 CAN SYNTHIO BE EXTENDED TO THE MORE COMPLEX AUDIO CAPTIONING TASK?
Audio captioning, unlike classification, involves describing the content of an audio sam- ple using natural language, making it a more complex task. To demonstrate Synthio's effectiveness for audio captioning, we evaluated it on down-sampled versions of Audio- Caps. For this task, we adapted Synthio by removing the audio captioning and CLAP fil- tering stages, and we extracted acoustic features directly from the existing audio captions.

---

Published as a conference paper at ICLR 2025
Additionally, retrain our T2A model a modified version of Sound- VECaps, excluding any audio from AudioCaps. Training and evaluation were conducted using the EnCLAP framework (Kim et al. 2024), and the dataset expanded with 4x synthetie samples. As shown in Table 4 Synthio significantly outper- forms baseline settings, with improvements largely due to better alignment w/ DPO. However, manual inspection revealed that generated audios occasionally do not match their captions compositionally. reflecting limitations of the current T2A model. While this issue does not affect classification, it poses challenges for captioning. We will explore more advanced methods as part of future work.
Table Performance comparison of Synthio with basclines on audio captioning.
METBOR
CDDE:
SPIDE.
Galbony (Ni Aug?
0.125
0::48
00754
0,112
Sye vacape Metnona
1.:00
10284
00300
Syntio send
0.169
0.327
0.164
6.99
Gostbenty (No Aup?
0.327
01:37
01067
01112
1000
VECag Ratiioal
Voutia Syn Aug
8:334
D1:06
0188
0.191
6.4 How WELL DOES SYNTHIO SCALE?
Taopeesss the performance of Synthio, SpecAug- ment, and Vanilla Synthetic Augmentations across differ- ent scaling factors N= (1, 3, 5), where N represents the number of synthetic samples generated per original sample the ssall-scale dataset (in this case we fix n 100). As observed, SpecAugment, traditional augmen- tation method, cannot scale with increasing N, and the performance of Vanilla plateaus at higher N. A similar saturation occurs with Synthio when MixCap is not used. Even without DPO, Synthio maintains better scalability, though with reduced overall performance. These results highlight that MixCap's 's ability generate diverse captions is crucial for Synthio's scalability.
Table 5: Performance comparison of Synthio with other baselines on different values of N.
Scaling Fastor
2x
Sx
6x
Spechagmenr
47,50
47.50
47.50
47.50
41.90
Varilla Syn. Aug.
67.90
77:35
76.75
75.60
71.25
ESC50
Syatlio roursy
77:45
61.75
82:35
53.15
83.35
with Mas
64.
08:45
71.55
72.85
75.90
61.55
64.25
65.95
66.60
66.73
41.96
41.56
41.96
41.96
Vasilla
33.13
55.28
42.31
41.54
38.27
NSymt
Syatho
55:18
16.17
45.56
46.92
46.81
wie MaxCup
40.41
41.08
41.95
42.27
42.15
wil DPO
39.23
39.42
40.17
40.31
39.82
6.5 DOES SYNTHIO HELP LONG-TAILED CATEGORIES?
Figure Souure]]shone shows the classification accuracy on four underrepresentee categories in the NSynth dataset, comparing performance before and after applying Synthio aug- mentations. We selected categories with the lowest frequency in the downsampled dataset, such as flute and guitar, which ap- pear only once in the down-sampp sampled sets. Synthio significantly boosts accuracy, with improvements up to 48%. Notably, cat- egory labels like flute and guitar, which originally had 0% accuracy, show substan- tial gains with Synthio augmentation. This demonstrates Synthio's 's effectiveness in boosting performance on long-tail labels, common challenge in world datasets (Zhang et al.) 2023).
Improved Synthio
Gold only
80
15.34
Classiication Accurasy [N]
60
40.00
$1.52
31:44
20
21:11
usgan
Fldle
noal
Calegorise
Figure 5: Category-wise improvement in performance with Synthio augmentations for long-tailed categories.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
We introduced Synthio, a novel approach for augmenting small-scale audio classification datasets with synthetic data. Synthio incorporates several innovative components to generate augmentations that are both consistent with and diverse from the small-scale dataset. Our extensive experiments demonstrate that even when using T2A model trained on weakly-captioned AudioSet, Synthio significantly outperforms multiple baselines.
However, Synthio has some limitations: (i) Its performance is influenced by the capabilities of the T2A model and the quality its training data, As T2A models continue to improve, we expect Synthio's performance to benefit accordingly. (ii) The process of generating audio captions using LLMs may introduce biases inherent in the LLMs into the training process. (iii) Synthio is computationally more intensive than traditional augmentation methods due to the need for prompting LLMs and T2A models. We anticipate that ongoing advancements in model efficiency will help mitigate these computational challenges.
10

---

Published as a conference paper at ICLR 2025
8 REPRODUCIBILITY STATEMENT
Our project page has all the codes and checkpoints to reproduce the results in the paper. All experimental details, including training parameters and hyper-parameters, are provided in Section 5
ACKNOWLEDGMENT
This project is supported in part by NS1#1910940.
REFERENCES
Haider Al-Tahan and Yalda Mohsenzadeh. Clar: Contrastive learning of auditory representations. In International Conference on Artificial Intelligence and Statistics, pp. 2530-22000000 2538.. PMLR, 2021.
Shekoofch Azizi, Simon Kornblith. Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
net. forum?
id-DiRaoxjyPm
Max F Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and Chris Russell. Image retrieval outperforms diffusion models on data augmentation. Transactions net.
on Machine Learning Research, 2023. ISSN 2835-8856. URL hhttps://open::t
=orum?id-xf1ydd2Mp3 XE1Yd2MMp
Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. An empirical survey of data augmentation for limited data learning in nlp. Trunsactions of the Association for Compulational Linguistics, 1 11:111211, 2021. URL/ttps: //ap:::://ppi. wwwaateaiiiieeee wwwemmnnnisccooaat. org/ Darg/CorpusID
235422524
Manuel Cherep and Nikhil Singh. Contrastive learning from synthetic audio doppelgängers. In The Thirteenth International Conference on Learning Representations, 2025. URL https
/openreview n Seweerewweee.net/ooriiiieeeeeeee forum?id-XRtyVEDI
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast timing condit-con latent audio diffusion. arXiv preprini 2402.00200. 2024a.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407. 143558, 2024b.
Tiantian Feng, Dimitrios Dimitriadis, and Shrikanth Narayanan. Can synthetic audio from generative foundation models assist audio recognition and speech modeling? arXiv preprint arXiv:2406.08m0 2024.
Eduardo Fonseca, Xavier Favory, Jordi Pons. Frederic Font, and Xavier Serra. FSD50K: open dataset
sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-852. 2022.
11

---

Published as a conference paper at ICLR 2025
Jonas Geiping, Micah Goldblum, Gowthami Somepalli, Ravid Shwartz-Ziv Ziv, Tom Goldstein, and Andrew Gordon Wilson. How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. arXiv preprint arXiv:2210.0644) 2022.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio gener- ation using instruction tuned Ilm and latent diffusion model. arXiv preprint 2.3.v:2230..37333 2023.
Sreyan Ghosh, Ashish Seth, and S Umesh. Decorrelating feature spaces for learning general-purpose audio representations. IEFE Journal of Selected Topics in Signal Processing, 16(6):1402-1414, 2022. doi: 10.1109//STTPPP0222.000000
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, S Sakshi, Utkarsh Tyagi, and Dinesh Manocha. DALE: Generative data augmentation for low-resource legal NLP. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8511 1-8565, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 12118666//12222mmm--mmmm
URL
attts://aclanthoot wwg.:llggyoogooooooooooooo 2023, 020......... emnlp www.ppppmmiiee
ma inaaain.e 528
Sreyan Ghosh, Ashish Seth, Srinivasan Umesh, and Dinesh Manocha. Mast: Multiscale audio spectrogram transformers. In ICASSP 2023-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023b.
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large ludio-language model with advanced audio understanding and complex reasoning abilities, 2024a. URL https://
Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi. Chandra Kiran Reddy Evuru, Ra- maneswaran S, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning in audio-language models. In The Twelfth Interna- tional Conference on Learning Representations, 2024b. Forum? 8 3iaumm?2eg6GGG
Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Reddy Evuru, Ramaneswaran S, Sakshi, and Dinesh Manocha. ABEX: Data augmentation for low-resource NLU via expanding abstract descriptions. In The 62nd Annual Meeting of the Association for Computational Linguistics, 2024c.
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01178, 2021.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XI- AOJUAN QI IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION? The Eleventh International Conference on Leurning Representations, 2023.
URL /attps://cpenree www.peeennii...o..m..e........we net/ forum Cwtorumiidnnnnneaee id=nnmCe25RKF
Calum Heggan, Sam Budgett, Timothy Hospedales, and Mehrdad Yaghoobi. Metaaudio: A few-shot audio classification benchmark. In Artificial Neural Networks and Machine Learning - ICANN 2022, pp. 219-230 Cham, 2022. Springer International Publishing. ISBN 97833-39993311111199990
I Jordal. audiomentations, 2021. URI. http:////eenooooo//eeereeee/////
Akbar Karimi, Leonardo Rossi, and Andrea Prati. AEDA:
An easier data augmentation technique for text classification. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.). Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2748 7748,2754, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/11/0211.1 windinigsemnnniiiic..
URL https: / htt///aaantttt aclantologgg
or2/2021. findi ngn-ngg-mmnn emn 234
12

---

Published as a conference paper at ICLR 2025
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr), 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466, 2018.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 119-132, 2019.
Jaeyeon Kim, Jaeyoon Jung. Jinjoo Lee, and Sang Hoon Woo. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024 2004 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6735-6739, 2024. doi: 10.1099/CASSP4B8882200.00000
Zhifeng Kong, Sang-gil Lee, Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Rafael Valle, Soujanya Poria, and Bryan Catanzaro. Improving text-to-audio models with synthetic captions. arXiv preprint arXiv:2406. 15487, 2024.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2023. URL https
//openreveww.o net appenseveee.net//oruuaaaLCCCCe
Cyril Laurier, Owen Meyers, Joan Serra, Martin Blech, Perfecto Herrera, and Xavier Serra. Indexing music by mood: design and integration of an automatic content-based annotator. Multimedia Tools and Applications, 48: 18:14 2010.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 2023.
Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15122. 2024.
Vincent Lostanlen and musical instrument recognition, 2017. URL /tttps::/axxxi.ccotttt wttp///wwwxivo..raaaaooo/ hwtpg//bb//160.. 06644
Emanuele Cella. Deep convolutional networks on the pitch spiral for
Navonil Majumder, Chia-Y Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-basee text-to-audio generations through direct preference optimization. arXiv preprint arXiv:2404.09955 2024.
Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. Cdpam: Contrastive learning for perceptual audio similarity. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 196-200. IEEE, 2021.
Irene Martín-Morató and Annamaria Mesaros. What is the ground truth? reliability of multi-annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 76-80. IEEE, 2021.
Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. Dcase 2017 challenge setup: Tasks, datasets and baseline system. In DCASE 2017-Workshop on Delection and Classification of Acoustic Scenes and Events, 2017.
Loris Nanni, Gianluca Maguolo, and Michelangelo Paci. Data augmentation approaches for improving animal audio classification. Ecological Informatics, 57:101084, 2020.
An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. Aggregating and predicting sequence labels from crowd annotations. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 299-309, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 20118853/v1l1 1/53561111111110000 1/217-1028. URL wwt.t:ttt////t wnthollog..ogg/ll7--.
13

---

Published as a conference paper at ICLR 2025
Daisuke Nizuumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for general purpose perprae audio representation. In 2021 International Joint Conference on Neural Networks (JCNN), pp. 1-8. IEEE, 2021.
Daniel S Park, William Chan, Yu Zhang, Chung- Chiu, Barret Zoph. Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition.
arXiv preprint arXiv:/904.080 1904.08779, 2019.
Geoffroy Peeters. A large set of audio features for sound description (similarity and classification) in the cuidado project. CUIDADO Ist Project Report, 54(0):1-25, 2004.
Karol J. Piczak. ESC:
Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pp. 1018-1018. ACM Press. ISBN 978- 978-1-4503-3459 4. doi: 10. 10.1145/2773333328069999 URL http: /ttp::/dl.... acm. www..r////aat...nonnnomm cf
2733873. 2806390
Zafar Rafii, Antoine Liutkus, Tabian-Rober Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner.
The MUSDB18 corpus for music separation, December 2017. URL httpt::////at .rgg 110
5281 wew.//eedo...m 1117372.
Zhao Ren, Kun Qian, Tanja Schultz, and Bjöm W. Schuller. An overview of the icassp special session ai security and privacy in speech and audio processing. In Proceedings of the 5th
ACM International Conference on Multimedia in Asia Workshops, MMAsia '23 Workshops, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400703263. doi: 21145/661139.33285333 3628563
10.1145/361 1380.362863. URL
Julien Ricard. Towards computational morphological description of sound. DEA pre-thesis research work, Universitat Pompeu Fabra, Barcelona, 2004.
Francesca Ronchini, Luca Comanducci, and Fabio Antonacci. Synthesizing soundscapes: Leveraging text-to-audio models for environmental sound classification. arXiv preprint arXiv:2403. 17864, 2024.
Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3875-3879. IEEE, 2021.
J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In 22nd ACM International Conference on Multimedia (ACM-MM'144 14), pp. 1041-1044, Orlando, FL, USA, Nov. 2014.
Ashish Seth, Sreyan Ghosh, Srinivasan Umesh, and Dinesh Manocha. Slicer: Learning universal audio representations using low-resource self-supervised pre-training. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Janne S Sppiieeve/oochmaai---ggnnnaaaaa 2021. URL httpp:://enodo. org record/474858.
Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, and Dong Yu.
Preference alignment improves language model-based its. arXiv preprint arXiv:2409.1244 2024.
Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference Learning Representations, 2024.
George Tzanetakis and Perry Cook. Multifeature audio segmentation for browsing and annotation. In Proceedings of the 1999 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. WASPAA '99 (Cat. No. 99TH8452), pp. 103-106. IEEE, 1999.
14

---

Published as a conference paper at ICLR 2025
George Tzanetakis, Georg Essl, and Perry Cook. Automatic musical genre classification of audio signals, 2001. URL Sttt:///ssir2000
net htt.//ii...ttt/tttttatttttttttaa p
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238,, 2024.
Jason Wang, Luis Perez, et The effectiveness of data augmentation in image classification using deep learning. Convolutional Neural Networks Vis. Recognit, 11(007):1-8, 2017.
Yu Wang, Nicholas J. Bryan, Mark Cartwright, Juan Pablo Bello, and Justin Salamon. Few- shot continual learning for audio classification. In ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 321-325, 2021. doi:
N0.1109/CASSP33972 200.SSSS.9722.222.11.. 0021.9413584
Yuanyuan Wang, Hangting Chen, Dongchao Yang, Zhiyong Wu, Helen Meng, and Xixin Wu. Audiocomposer: Towards fine-grained audio generation with natural language descriptions. arXiv preprint arXiv:2409. 2265,, 2024.
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNEL pp. 6382- 6388, Hong Kong, China, November 2019, Association for Computational Linguisties. doi: 10. 10.18663/1//D19-16700 URL /ttp:tttt..///aaaatttooooooooogp
Zeng Weili, Yichao Yan, Qi Zhu, Zhuo Chen, Pengzhi Chu, Weiming Zhao, and Xiaokang Yang. Infusion: Preventing customized text-to-image diffusion from overfitting. In ACM Multimedia 2024, 2024.
Huang Xie and Tuomas Virtanen. Zero-shot audio classification via semantic embeddings. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 1e:1223-2223 2021.
Xuenan Xu, Zhiling Zhang, Zelin Zhou, Pingyue Zhang, Zeyu Xie, Mengyue Wu, and Kenny Q Zhu. Blat: Bootstrapping language- pre-training based on audioset tag-guided synthetic data. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 2756-2764, 2023.
Yi Yuan, Dongya Jia, Xiaobin Zhuang, Yuanzhe Chen, Zhengxi Liu, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xubo Liu, Mark D Plumbley, et al. Improving audio generation with visual enhanced caption. arXiv preprint arXiv:2407.04446 2024.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEF/CV international conference computer vision, pp. 6023-6032, 2019.
Shilei Zhang, Yong Qin, Kewei Sun, and Yonghua Lin. Few-shot audio classification with attentional graph neural networks. In Interspeech, pp. 3649-36533 2019.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4509::17955-0816, 2023.
Qihao Zhao, Yalun Dai, Hao Li, Wei Hu, Fan Zhang, and Jun Liu. Ltge: Long-tail recognition via leveraging llmv-driven generated content. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 19510-19520, June 2024.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Table of Contents:
A.I B.Bckgroundd on Diffusion Models
A.1
A.2
Prompts
A.3
Examples
A.4
Extra Results
A.5
Dataset Details
A.6
Algorithm
A.1 DIFFUSION MODELS
Diffusion models consist of two main processes: forward process and reverse process. Given data point with probability distribution p(ro), the forward diffusion process gradually adds Gaussian noise to Zu according to a pre-set variance schedule 31, ,.,,,, By and degrades the structure
of the data. At the time step i, the latent variable 2a is only determined by the Iz-1 due to its discrete-time Markov process nature, and can be expressed as:
p(x, 2T-1) = (Z,;; /1 B, B,I),
(12)
As increases over several diffusion steps, p(xr) approaches a unit spherical Gaussian distribution. The marginal distribution of x, at any given step can be expressed analytically as:
p(a, | mo) = N(;;; Va,ao: (1 -,,))),
(13)
where a, = IT (1 3.). The reverse process aims to reconstruct the original data from the noise-corrupted version by learning series of conditional distributions. The transition from 2g to Tz-1 is modeled as:
pe (
1-1 111), 1 xx)) =N(x ,00
(14)
t-1 14
It
ED (Ie, t))
(15)
Vor
v1 at
1-13 1 ύ(-1
(16)
1- ā
1-1 is the mean estimate, where 0; = 1 31, ā; - 0; represents the learnable parameters, μi
1-1² is the standard deviation estimate, and c0(2;, t) is the noise estimated by the neural network. The reverse process estimates the data distribution p(xo) by integrating over all possible paths:
To
Po(zo) S po (E(T))
[ t=1
(o
w4) dr,
(17)
where Po (o(T) (Zr; 0, .). At inference time, the diffiusion model iteratively executes the reverse process (Eq. 17 T times starting from randomly sampled Gaussian Noise (e (O,
At the time step 1, the latent variable v; is only determined by the æ;-1 due to its discrete-time Markov process nature. At inference time, the diffusion model iteratively executes the reverse process T times starting from randomly sampled Gaussian Noise (e ~ (O, For more details on diffusion models, we request our readers refer Appendix A.I
A.2 PROMPTS
Fig. 6
8 and iillustrate all the prompts used in experiments. For all experiments, prompt GPT Turbo (GPT-44tubbo2224--4442 with top-p=0.5 and temperature=0.7.
A.3
EXAMPLES
Table 6 presents examples of captions generated by the Synthio framework, along with their revised versions for captions that initially rejected.
16

---

Published as a conference paper at ICLR 2025
will provide you with a ception of an audio that describes the evente taking place in the audio. idditionally, I will also provide you with label for the audio. Extract the phrases audio. There types featurea you
shat corressond
distinctive features
co extract:
?) the unique foreground events in the caption,
2) the brosder background scere or background evente the or audic and
3) any other features related to the audio. Retern J3ON with key 3 keys, one named ae 'events',
and the other named 'other features', where the values of these keys correspord to cama-saparated pythenic list where ezch itam. in the list is a string corsesponding to the extracted phrasea. Please ignore phrase that texactly or senantically) corresponde
the audio. you think there either of the keys, leave then enpty.
informstion
Here is the caption:
Bere is the label:
Figure 6: LLM prompt (Prompt 1) for extracting components from audio captions.
will provide you with caption abstract fashior and mentions the broader scene or event that I nead to teach an audic nodal about fron the audio, 5.e., the audio and its label la past of the training set for training an acdio classitication model. I will also provide you
audio. The label generally describes the audio on
the domain you ddentify the true sound conveyed in the label, need you to rearice the caption for me according to this set culee:
which
help
2. I will provide you with lists of various audio features correspsading to events, backgrounds or other features. You should zevrite the given caption such that it has has features inspired with
from the features provided you, i.e., you should try so describe a scene for events, backgrounds and featurea similaz but unique fron the ones given. obay
2. After re-writiag, the captica should
label.
Here is the label::h.
Bero is the donaia of the audioril.
Here is the list of eventes() Bezo la the 1ist backgroundss:l Heree is the list of other festures:!)
Just outpuz the rouritten caption and nothing else. Output "Hone" 12 you did not zewrite.
Figure 7: LLM prompt (Prompt 2) for generating new audio captions given elements from existing captions.
A.4
EXTRA RESULTS
A.4.1 RESULTS ON THE FULL TRAINING SPLITS
T presents the performance comparison of Synthio the full original dataset splits (where the entire training set is used without any downsampling). While Synthio outperforms all baselines, traditional augmentation methods prove to be much more competitive in this scenario. This contrasts with the results in Table I where traditional augmentations showed minimal improvements in performance.
Additional Discussion on Results. As we see in Table | (and Table 7). performance gains with Synthio as the number of Gold samples increase (highest absolute gains with n 100 and lowest with full dataset). This phenomenon is consistent across prior work in vision (Trabucco et al. 2024), text (Ghosh et al. [2023a] 2024c), and audio (Ronchini al., 2024). Most synthetic data augmentation methods demonstrate substantial gains in low-resource regimes, but these gains naturally diminish as the quantity of high-quality labeled data increases (for example, Azizi et al. just show over ImageNet only modest improvement of just over 1,, where the authors reported when augmenting this large-seale dataset).
We hypothesize that this trend is rooted in the inherent diversity and richness of gold data. Gold datasets typically capture nuanced variations and complex real-world distributions, including subtle contextual and environmental factors that synthetic data struggles to replicate. Synthetic data, while effective at filling gaps and addressing low-resource scenarios, often lacks the granularity necessary to represent long-tail edge-case instances. As the size of the gold dataset increases, the model
17

---

Published as a conference paper at ICLR 2025
will provide you with a label for an audio. The label generally describes the audio in broader
abstract fashior and nensions about fzon the audio, i.b.r the audio and its label is part of the training set for training an audio classitication model. will also provide you you identify the trus sound conveyed in the label. would like you to generate 5 new captions that describe its event oz source in the label in diverse fashions. I will use these captions to generate audios following reqeitennnts:
event
teach audiç nodel
the domain the audio which will help
augnent ny training set, Generate the captions with
2. All the captions need to include new and diverse evence and contexts beyend the actual event conveyed by the label.
2. Only add
and context by understanding the broader contex: of occursence the audio and the zazget label. Do not add zandom events or contexts.
3. The new caption should be not more than 20-25 worda.
there constrainte needs to obey the event ceaveyed by the original label, i.e., the new caption may not lezd to audio goneration. that defiea the audio label.
4, dovever, afser
adding new events or contexts, esption still
5. Finally, use the original label as a phrase in your caption.
Heree is label::h.
Here is the donsin of the audiosih. Uutput uSON with the key the original label and value ar the lisz of comna separated new capcions. only output the JSoN and nothing else
Figure 8: LLM prompt for generating random captions for Random Captions baselines in Table
will provide you with a label for an audio. The label generally describes the sudio in abstract fashior. and mentions the broader scene event that I nerd to teach audic notel about fron the audio, 1.8., tha audio and its Label past of the training set training audio classification model. I will also provide you with the domain of the audio which will help you identiry the true aound conveyed in the label, would like you to generate 5 new captions that describe she event or source in the label diverse fashions. T will use these capt generate new ausios that can augnent ny trsining set, Generaze the new captions with the following rements:
:. Sach caption should have a diverae added eventa (bayond the event of the original label) and contexts.
2. only add new events and context by understanding the broader contex: of the occurvence of the audio and
target label. for adding events contexte, please follow 3. 1 will also provide you with a list of features extracted from in existing set of audies. You sbould try such that the nav captions you generate for the labal have mix of events and scenas similar to the events and background scenes that are given and aey scenes, i.e., you should try zo describe a acane for the caption with the eventa and backgrounds provided to you in the givan (events, backgrounds or other features) beyond
requi rement.
'ista you should ores given.
add novel
4. The now capsica should be eet more than 20-25 words.
adding new events or contexts, caption still neads to obay zhe event label, i.e., the new caption may not lead to an audio generation that defies the audio label.
5, dovever, afser all these constrointe
6. Finally, use the original label as a phrase in your caption.
Here is label:ih.
Bezo is the donain of the audioril. Beze is the 1ist avensas(] Here is the 1ist of backgrounda:!] Here is list other festures:!)
Octput a JS0N with the key as the original caption and value as the 11st of cooma separated new captions. only outpuz the Jaon nothing else.
Figure 9: LLM prompt (Prompt 3) for rewriting captions of rejected audios.
increasingly benefits from the inherent diversity of these high-quality examples, reducing the need for synthetic data and its relative impact on performance.
Additionally, in Fig. 6 of their paper, Azizi al. also how increasing number of synthetic augmentations leads to plateauing and even diminishing performance. We hypothesize that this is due to over-fitting caused by lack of diversity in generated augmentations.
18

---

Published as a conference paper at ICLR 2025
Datasel
Label
Generated Caption
Revised Caption
USD8k
children.playing
Children playing bustling park
with distant traffic noise
USD8K calldreenplayiing
Children playing schoolyard during
Children playing in neightorrhood al- ley with sound of distant construction
recess with teacher's s whistle
USD8k mtreei_music
Street music playing near busy in- tersection filled with honking curs and pedestrians.
NA
USD8K Streei_music
Street music from busitiing market
Street music echoing ihrough alley- way during lively street festival.
people chatter and vendors shout
TUT
airport
airport with people talking and walking around in emply hullway
TUT
airport
the airport, people talking with the sound of crowd of people the background, as announcements play.
airport ambiance with people talking and children running around
TUT
bus
Bus passing by are chatting at a nearby cafe.
road while people NA
TOT
bus
bus pussing by on a road as continues
bus idling on road with birds chirping nearby
the microphone
NSynth
keyboard
key board accompaniment to live band performance a bustling cafe. typing Keyhoard at office
NA
NSyut
Keýboard
keyboard board fiythms echoing in in empty auditorium during rehearsal break
NSynth
organ
serene church service an organ playing melody and soft brass
playing.
NSyuth
ongan
An urgan plays us guitars are playing"
An organ plays during lively music festival with various instruments.
together in the buckground.
Medley
Violin
violin being played during classical
NA
symphony orchestra performance
ModTey"
Vinifn
violin performing lively Jig at
Violin solo during a quief candloliht dinner a fancy restaurant.
bustling street fair
Medley
Flute
flute playing a tranquil forest during the early morning
Modley
Flute
Flute performance park during sunny afternoon.
bustlinng city"
Ffuté music echoing ancient stönč cuthedral.
AudioCaps-
dog harks repeatedly ground while engine starts In the distance, faint Thunder rumble" "Soff rain falls on metal rooT, creating is audible, accompanied by the gentle rhythmic tapping sound. rustling of leaves in the wind.
back-
AudfoCaps
Table 6: Examples generated revised captions the Synthio methodology.
Table 7: Comparison of Synthio and other baselines the full original dataset splits (using all samples from the original training set D ).ai)..
Method
USD8K
GTZAN
Medley
VS
MSDB
Gold-only
88.23
82.00
80.99
92.73
73.9
Random Noise
86.17
82.35
79.72
92.94
74.55
Pitch Shift
87.58
83.02
79.63
92.17
74.6
Spec. Aug.
87.92
82.50
79.14
92.42
74.5
Audiomentations
88.01
82.75
81.26
92.47
75.05
Retrieval
78.27
69.25
73.24
80.43
69.95
Vanilla Syn. Aug.
89.57
82.85
81.79
93.15
75.85
Synthio (ours)
89.57
82.85
81.79
93.01
74.24
A.4.2 AUDIO GENERATION RESULTS FOR OUR TRAINED STABLE DIFFUSION
Table presents a comparison of audio generation results across several evaluation metrics. We evaluate trained Stable Diffusion model (used in experiments, including version further
19

---

Published as a conference paper at ICLR 2025
Table 8: CLAP score between generated audios and the label.
Method
USD8K
NSynth
Real
12.67
14,46
Vanilla Syn. Aug.
14.34
17.54
100
Synthio
31.26
27.32
w/ Template Captions
29.31
26.62
w/ ERM
24.15
21.54
Real
10.13
9.4
Vanilla Syn. Aug.
12.55
12.91
200
Synthio
21.87
16.16
Template Captions
20.31
15.82
ERM
17.14
13.04
Table Comparison of our trained Stable Diffusion model AudioCaps set
Model
FAD.PANN (1)
FAD.YGG
IS.PANN (T)
CLAP.LAION (1)
Audiol DMM--age
32,50
1.89
8.55
0.45
Tango FulKFT-AC
18.47
2.19
8.80
0.57
17.19
0.52
Make-an-Andio
11.75
VDCaps
Andio VBCap + AudioCips-FT (owrs)
14.93
2.19
0.56
fine-tuned on AudioCaps) against other available models and baselines from the literature. Notably, our model performs competitively with other fully open-source models across most metrics.
A.4.3 FAD SCORES FOR GENERATED AUGMENTATIONS
To offer an alternative perspective on the distributional consistency between the generated augmen- tations and the ground-truth small-scale dataset, we compare the Fréchet Audio Distance (FAD) scores (Kilgour al. 2018). For this experiment, presents a comparison of FAD scores between Synthio and other baselines. Synthio achieves the
Synthio with Template Captions. Table TO
highest FAD score, indicating that it produces the most consistent audio augmentations.
Table 10: Comparison FAD score of Vaniall Syn. Aug. Stable Audio VECaps (ours).
n
Dataset
Model
FAD VGG (1)
Vanilla Syn. Aug.
1.83
100
NSynth
Stable Audio VECaps (ours)
1.42
Vanilla Syn. Aug.
1.71
TUT
Stable Audio VLCaps (ours)
1.45
A.4.4 EFFECT CLAP FILTERING
In this section, we provide additional experiments to show effect CLAP filtering on the Synthio pipeline. Table 1I compares the performance of Synthio with and without CLAP. As we can see,
Table 12 compares the performance of various values of p on 5 datasets and 2 values of n (500 and 100). As see, higher or lower values of do not affectt the final performance significantly.
Our T2A model uses the same CLAP text encoder for generating audio. Consequently, most generated audios are already highly aligned with the intended category label. However, the purpose of CLAP filtering is to safeguard against cases where the LLM hallucinates and generates caption that deviates significantly from the intended label. In such cases, CLAP filtering ensures that audios generated from hallucinated captions are discarded, preventing them from negatively impacting the learning process.
20

---

Published as a conference paper at ICLR 2025
Table 11: Ablation study evaluating impact of CLAP filtering on Synthio's performance.
Method
ESC-50
USD8K
GTZAN
TUT
Synthio
49.50
76.12
68.20
43.84
80.67
Syntho w/o CLAP
47.25
74.34
66.35
40.28
77.29
100
Syhtio
83.35
85.00
71.20
71.23
86.70
Synthio w/ Cl.AP
82.55
84.64
69.30
70.41
84.93
Syhtio
86.10
82.81
82.05
56.83
87.52
200
Synthio w/ CLAP
85.25
79.94
80.54
55.22
86.31
500
Syhtio
82.25
67.81
Synthio w/ CLAP
90.25
88.42
79.70
65,42
89.67
Table 12: Comparison of Synthio's performance with different CLAP threshold levels.
ESC-50
USD8K
GTZAN
TUT
VS
0.85
49.50
76.12
68,20
43.84
80.67
50
0.3
47.10
74.14
67.50
41.17
79.32
0.5
48.25
75.39
67.75
41.93
79.48
71.20
71.23
100
0.3
82.55
84.64
69.30
70.41
84.93
0.5
82.70
84.73
70.25
70.86
85.22
0.85
86.10
82.81
82.05
56.83
87.52
200
85.25
79.94
80.55
55.22
0.5
85.70
80.30
81,30
56.19
87.11
0.85
92.10
89,18
82,25
67.81
91.42
500
0.3
90.25
88 $8.42
80.70
65.42
89.67
91.65
81.05
A.4.5 EFFECT OF TRAINING DATA AND MODEL ARCHITECTURE FOR THE TEX-TO-AUDI0 MODEL.
In this section, we train our T2A model using 1) different model architecture (we replace Stable Diffusion with TangoG AudioCaps). Table 13] compares thee results. As we can clearly see, while the model architecture of the T2A model does not affect the performance, replacing the training data with a small and less diverse dataset leads to significant drop in performance.
Ghosal al. (2023) different training data (we replaced Sound- VFCaps with
Table 13: Comparison of Synthio with Synthio's Stable Audio trained only wiht AudioCaps and Tango trained with
Method
ESC-50
USD8K
GTZAN
Medley
TUT
Synthio (ours)
49.50
76.12
68.20
60.58
43.84
50
Synthio w/ AudioCaps
29.20
60.15
50.15
49.19
38.62
Synthio w/ Tango
48.55
75.05
66.19
59.12
42.59
Synthio (ours) 100 Synthio w/ AudioCaps
83.35
85.00
71.20
71.23
52.42
58.20
74.27
66.55
67.93
48.23
Synthio w/ Tango
81.50
84.13
70.95
69.97
51.47
A.4.6 SYNTHIO AS COMPLIMENTARY APPROACH TO TRADITIONAL. AUGMENTATIONS
Table 14
compares results of Synthio augmentations when combined with traditional augmentations. As we can see, Synthio boosts performance of all methods and combining traditional augmentations with Synthio boosts Synthios overall performance. This shows that Synthio can act complimentary step for traditional augmentations.
Additional Discussion. Across all datasets, noticed that CLAP filtering removed most 10% of the generated samples. This confirms that the majority of the synthetic data is already well- aligned with the target categories, and filtering primarily handles rare we emphasize on the point that while most generated audios align with the target label, the CLAP filtering stage acts safeguard against hallucinations by the L.M,, which may occasionally generate
of misalignment. Thus
21

---

Published as a conference paper at ICLR 2025
Table 14: Performance comparison Syuthio when paired traditional augmentation techniques
Method
ESC-50
USDSK
GTZAN
Medley
Synthio (ours)
49.50
76.12
68.20
60.58
w/ Random Noisc
49.65
77.31
70.15
61.54
w/ Spec Aug w/ Andiomentations
50.95
77.93
70.35
61.17
50.35
77.24
69.50
61.53
Synthio (aurs)
w/ Random Noisc
83.85
86.59
71.60
72.35
Pitch Shift
83.60
86.32
72.95
72.50
Spec Aug
84.25
86.17
72.75
73.05
w/ Autiomentations
84.10
85.95
72.85
72.87
captions that deviate significantly from the intended category. In such cases, filtering ensures that misaligned audios are discarded, preventing them from negatively impacting model training.
A.5 DATASET DETAILS
NSynth Instruments:
NSynth is a large-scale dataset consisting of musical notes played by variety of instruments. It includes rich set of acoustic features from instruments like guitars, flutes, and more, providing diverse sound textures for classification tasks.
TUT Urban:
The TUT Urban dataset captures everyday sounds from urban environments, including noises like traffic, human activities, and construction. It is commonly used for acoustic scene classification and environmental sound recognition.
ESC-50: ESC-50 is a well-known dataset for environmental sound classification, containing 50 categories of everyday sounds such as animal noises, natural elements, and human activities, making it suitable for multi-class classification challenges.
UrbanSoundSK (USD8K): USD8K a curated collection of urban sounds divided into classes, including sirens, street music, and car horns. It is used widely for evaluating models sound event detection real-world scenarios.
GTZAN: GTZAN is a music genre classification dataset that includes ten music genres such as pop, rock, and jazz. It is standard benchmark for evaluating music classification models, although has known data quality issues.
Medley-solos-D DBB This dataset consists of solo recordings of different musical instruments, making valuable for studying isolated instrument sounds and training models for music instrument recognition.
MUSDB18: MUSDBIE 18 used primarily for music source separation tasks. contains full-track recordings of different music styles, providing mix of vocals, drums, bass, and other instruments, useful for multi multi-class classification.
DCASE Task 4: Part of the DCASE challenge, this dataset focuses domestic sound scene and event classification. It includes various audio clips recorded in home environments, often used for anomaly detection and sound event classification.
Vocal Sounds (VS): This dataset includes various vocal sounds such as singing, speech, and vocal effects, providing rich data for studying voice classification and enhancing models for vocal audio recognition tasks.
A.6 ALGORITHM
Algorithm algoritthiiclly illustrated Synthio.
22

---

Published as a conference paper at ICLR 2025
Algorithm Synthio Framework for Audio Classification Augmentation
Require:
Small human-annotated dataset D i
Noisy audio-caption paired dataset Dac;
Number of generations per instance ji Similarity threshold p%;:
Maximum self reflection iterations imxx.
## Initial Training of T2A Model
Train T2A model 7 on Dre.
## Construction of Preference Dataset Ppref for cach audio instance d, in Dsmall do Create caption C = "Sound of label,""
for 1
Generate audio â',1 = (cx) starting from random noise.
Pair (āæ,t, ax) where ag the ground-truth audio. Add pair to pret with dg as loser and ag as winner.
end for end for
## Preference Optimization Using DPO
Fine-tune tune-tuneeeee T on D using DPO methodology.
## Generating Diverse Prompts with MixCap Use audio captioning model to generate captions for all a, in D
Prompt LLM to extract acoustic components (backgrounds, events, their attributes relations) from captions.
for each label labely in Dimall do Using extracted acoustic elments, prompt LLM generate diverse captions {ck, Ck ... Ck end for
## Generation Synthetic Data Dyn
Initialize Di + Ø, D + Ø.
for each caption Cm do
Generate audio āg.m = T (Ck,m).
Evaluate similarity 8kmm. = CLAP (e labely).
if mm 2 p% then
Add (āx, label;)) to Dece
else
Add (ck,m:lbblly D end if
end for
## heff-Releccton and Caption Revision
Set iteration count 4- O.
while D # and imux do
+
for each rejected caption cr.m in D do
Provide LLM with 62,m and insights from aye
Obtain revised caption ck.m-
Generate audio 0..m = (ck,))
Evaluate similarity fk.m CLAPP( labelz).
if f,, 2 p% then
Add (ak, labely) to Dese Remove Ck.m from Drsj else
Update Ck,m ck.m in D)re end if
syn.
end for end while
## Final Training Dataset and Classification Model
Combine D,y, with tround-truth data Danall to form D Train audio classification model on D. Dunin:
23


Reviews:
[{"review_id": "Kd1ohiOhbN", "reviewer": "Reviewer_gsoV", "summary": "This paper explores the use of synthetic data to enhance small-scale audio classification tasks, addressing two primary challenges in synthetic augmentation: acoustic consistency and compositional diversity. To improve consistency, a preference alignment approach is introduced to better align the domain of synthetic data with the source dataset. For diversity, the authors propose language-guided audio imagination, utilizing large language models to iteratively generate captions that blend existing and novel acoustic components, feeding into a text-to-audio generation model.", "questions": "- Although the paper strives to present all concepts (with background) in a single, self-contained format, it would benefit from careful notation selection. Reusing the same notation for different concepts could be confusing. For example, using the symbol \"beta\" in different contexts (e.g., in diffusion, RLHF, and DPO) might lead to misunderstandings.\n- Minor correction:  In Table 10, please correct \"Stable Stable\" to \"Stable.\"\n- A major concern with the proposed method is balancing human effort with the complexity of the introduced steps. While the pipeline can operate fully automatically, the large number of hyper-parameters may introduce significant uncertainty in practical applications. The reviewer appreciates the current approach, but simplifying the pipeline to reduce complexity while retaining the core concepts could be a valuable direction for future work.\n- Additional experiments:\n    - While Synthio shows promising performance, it would be interesting to investigate whether it has complementary or joint effects with other augmentation methods, such as random noise addition, pitch shifting, or other common techniques.\n    - Further experiments could examine how changes in T2A models influence the effectiveness of the augmentation process. Specifically, it would be valuable to understand how the quality of T2A models affects the overall performance of the augmentation technique.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper effectively identifies two key challenges in synthetic data-based augmentation, providing clear and well-reasoned explanations.\n- The proposed techniques directly address these challenges and are demonstrated to be both practical and beneficial according to the experimental results.\n- The experimental setup is comprehensive, covering various types of audio events, which strengthens the validity of the findings.", "weaknesses": "The work is well-presented and thorough, with no significant weaknesses noted. However, the reviewer has a few suggestions and minor comments, outlined in the Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jqUjXeubjL", "reviewer": "Reviewer_dSa3", "summary": "This paper presents Synthio, a synthetic data generation approach for improving audio classification models. Synthio leverages a pre-trained text-to-audio generation model, it first aligns the generations with acoustic preferences (obtained from a small dataset), then it leverages an LLM to synthetically generate captions for the generated audio. To further improve data quality and model performance, Synthio refines the dataset via rejection sampling. \nThe authors evaluated their approach using different benchmarks while generating trainings sets of different sizes. The authors empirically demonstrated the proposed approach is superior to standard data augmentation techniques and to vanilla data generation.", "questions": "As mentioned before, the proposed approach is interesting and novel (in the context of audio generation). However, I have several concerns about the evaluation process and presented results: \n1. The authors compare the proposed approach to several baseline methods including Gold-only (meaning no augmentations at all), and more traditional data augmentation techniques. Although the proposed method reaches significantly better results (especially under a small number of samples in D_test), it uses much more resources, meaning, the T2A model was trained on ~1.5M of audio samples. Hence, models trained with Synthio were introduced to more and diverse set of samples than just D_test, which were greatly influenced by the pre-training data of the T2A model. \n2. How many samples did you generate for each of the N samples on D_test in Table 1. This is another potential issue as the models trained with Synthio were also trained on more data (nxN) in comparison to standard augmentation techniques which were trained on N samples only. \n3. As I mentioned before, it seems the gap in performance is getting smaller very fast as we increase the size of D_test (especially compared to a model with no augmentations). This means that the proposed approach is beneficial for model fine-tuning under very low resource regimes (less than 500 samples). I advise the authors to put more emphasis on that in the paper and discussions. \n4. The authors additionally presented results for audio captioning. What text labels were used to both train and evaluate model performance considering both the proposed method and baselines? I'm asking as in the proposed approach text labels are much reacher, hence I'm not sure the comparison is fair. \n\nI'm willing to increase my score in case I missed something.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Leveraging DPO to better align audio generation with real acoustics is a novel idea.\n2. The whole data generation pipeline is an interesting idea, including the MixCap approach.\n3. Under fine-tuning at small-scale regimes the proposed approach achieves significant improvement over the baseline methods.", "weaknesses": "1. Unfair comparisons to the evaluated baselines (more details below).\n2. When increasing the number of generated samples the gap in performance between the proposed approach and no aug. at all is closing, very fast.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cjl14Onknr", "reviewer": "Reviewer_8CiF", "summary": "The paper proposes a straightforward method: Given small scale audio classification task - use an existing audio generation model in this paper, a text-to-audio generator, to generate a lot of audio files similar to the smaller dataset and augment it with a factor of 1x to 5x samples to fine-tune an existing audio classification. The authors show the method outperforms their proposed baseline by (relative) 0.1-39%. The authors say that their contributions are,\n\n -- it is a novel approach for augmenting the small-scale dataset\n--  their ability to generate novel, diverse, consistent data and its performance on tasks such as audio captioning.\n\nIn brief, this is a paper on audio classification on small-scale audio. The methodology is once the dataset is created == straightforwardly fine-tune the model.", "questions": "1. How would aligning the T2A model with the small dataset create the diversity of audio as exists in the real world as claimed in the Abstract as DPO would make it close to the samples chosen.  \n\n2. Why was random mix augmentation not carried out in the Table 1 ? \n\n3. Was the performance boost mentioned in the abstract 0.1-39% relative or absolute. I could not find that in Table 1. Why were MAP metrics not report as is a standard practice for FSD-50K\n\n4. Why are the characteristics like spectral flux and spectral flatness done globally as opposed to local behavior of NSynth and Urban Sound dataset. \n\n5. Are there any papers that have the experimental setup similar to the Table 1. Please cite a paper that fine-tunes such a small corpus in a similar setup. \n\n6. Why was the whole background on few-shot classification and zero shot audio classification not included. This is very similar to zero shot and few shot classification. The baselines should be included as well. Why should someone generated synthetic examples and not do few shot audio classification. What are the drawbacks of those methods. Why are the baselines not included with any of the previous works on synthetic dataset. n=50  is a few-shot setting, and it would be unfair not to include any literature or mention it anywhere in the paper. \n\n7. Why have the authors gone the T2A route. Taking an audio and writing a caption is an extremely lossy compression which will remove most of the information needed about the audio. what were the advantages of a T2A model that will be helpful for this problem. \n\n8. Why was the scaling factor limited to only 1x-5x scaling. What happens if I do 20x. It seems it might start overfitting ? The results in 6.4 are strange that it works with 4x scaling than 5x scaling. Either the models are not tuned properly or they are overfitting on smaller corpus.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "== The paper has extensive experimentation that is far more than what was needed. It would have been much stronger if the authors focussed on a subset of these results with a sound, solid method. \n\n== The paper appears to be well-structured and written despite its flaws. \n\n== The data and code will be open-sourced, reproducible and easy to follow, which is a good point in favour of the paper. \n\n== A well-explained background on DPO, diffusion model, and RLHF. It should have included background for audio classification, zero-shot learning, etc. \n\n== The problem is interesting; exploring synthetic data to improve audio classification and pushing the performance is worth exploring.", "weaknesses": "The paper is being submitted to the topmost ML venue; hence, my comments are made accordingly to adhere to the highest quality of the paper. At the outset, the paper appears strong, but there are flaws that need to be addressed / claims that are not correct. Please do not carry out extra experiments to justify any other points and generate a new version of the paper. Please answer these points in 1-2 lines of text at maximum. \n\n1. The authors should drop a novel approach for audio classification as several papers exist that should be referenced. i) Open-Set Tagging Dataset (OST) Mark Cartwright, ii) BLAT Bootstrapping language-audio pertaining based audio set .... iii) Can synthetic audio from generative foundational models assist audio recognition ....iv) Synthetic training set generation using T2A models for environmental sound classification. There exist many more papers for audio generation tasks, which I do not include here, but the authors should have included them. These should be adequately referenced in the related work. If not, it gives a very false impression that the authors are the first ones to do it, including in the abstract.\n\n2. The whole point of audio classification is robustness to unseen conditions and good performance for out-of-distribution data. DPO aligns generated data with the small dataset but fails to capture the variations that are needed in real life, as claimed by the author. The claim made by the authors, \"T2A models trained on internet-scale data often generate audio that diverges from the characteristics of\nsmall-scale datasets, resulting in distribution shifts.\nThese mismatches can include variations in spectral\n(e.g., frequency content), perceptual (e.g., pitch, loud-\nness), harmonic, or other acoustic characteristics 2\" is correct, but for audio classification, we do want to incorporate all of these variations. \n\n\nThis is the point of training on a large corpus of datasets, such as an audio set, to include all the variations different from the training corpus. If I take very few examples and align a T2A model with DPO to the smaller dataset, it will generate samples in that small dataset distribution, as mentioned in all of the plots. This is shown precisely in Figure 3, and we do not want that. I would rather want a lot of diversity in pitch and spectral flux to make audio classification good in unseen cases rather than have the training data similar to my small corpus. \n\nThere is a whole subfield on \"out of distribution classification\" = how to make models perform well when the classification dataset is not of the same distribution as that of the original training dataset. Whereas here, the authors are trying to do just the opposite and make the training corpus as close to the actual data using DPO, which is counter-intuitive to the field of how to build robust classifiers and get good classification accuracy out-of-distribution examples and data. \n\n3. Audiomentations has a list of 40 possible audio augmentations. The setup with n being 50-500 for ten categories would not be able to generate enough diverse augmentations. How do the authors report the numbers for it and carry it out? With N being at max 5, how would one incorporate it? Are all of the combinations not needed/not needed? how were they chosen? \n\n4. From Table 1, it does not appear that this paper was an academic paper. The comparisons do not include any benchmark reported by any other paper. The baselines are poorly reported and are not compared to any other paper. Even for data augmentation literature, there are papers that compare various techniques. \n\n5. Section 6.4 mentions that the scaling 4x is better than 5x. This appears strange as to why this would happen. The audio captioning section is definitely not needed for this paper and should ideally be dropped. \n\n6. The issue with alignment to the dataset is also the fact that there are categories that often overlap in FSD-50K, ESC-50, etc. What happens in that setup? In that case, the model is forced to generate one kind of audio sample/or prefer one over the other even though both have the same label. \n\n7. Accuracy mentioned in 6.5 is 0%. It is mentioned that the categories had the lowest frequencies in the dataset. This makes the claims in the paper shaky as it appears that the training was not normalized according to the category probabilities or sub-sampled/oversampled to make the training balanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mcOArN27vy", "reviewer": "Reviewer_Q6SZ", "summary": "This paper proposed a pipeline to augment small-scale audio classification datasets with synthetic data and shows promising improvements.", "questions": "The result in Section 6.5 and Figure 5 is not convincing enough:\n1. Line 516 “ We selected categories with the lowest frequency in the datasets”: \nThe categories in the ESC-50 dataset have the same number of samples; how do you decide which category has a lower frequency?\n2. Figure 5 shows only four classes to demonstrate the improvement of long-tailed categories. This is not convincing enough as those selected classes may represent only partial information. \n\nTable 2: Can you please explain what is “compositional diversity” and how it correlates with the CLAP score?\nLine 214: DSmall -> D_{small}\nLine 226 “no scalable methods”: Is it possible to estimate the loudness with the signal processing method?\nLine 242 “fostering more diverse augmentation”: Any evidence for this claim?\nPlease double check the equation 8, 9, and 10 as the “x_0 : T” notation looks a bit confusing to me.\nLine 318 “The CLAP model is pre-trained on Da-c and fine-tuned on Dsmall to adapt to the target datase”: Do you mean finetuning CLAP model on each of the dataset you used (ESC50, GTZAN, etc)? If so, how do you perform finetuing on such a small dataset, such as when n=50? Doesn’t that can cause overfiting?\nLine 413: remove duplicated “the”\nWhat dataset is utilized for the experiment in Table 5? Please specify. \nLine 503: What are the possible reasons why SpecAug does not scale well?\n\nPlease consider rewording the following sentence that is a bit confusing to read:\n- “Abusing notation, we will also x0 as random variables for language”\n- “We elaborate on these next”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The performance improvement when the data is in small scale (e.g., 50, 100, 200, 500) is pretty significant. \n2. The experimental setup is pretty complete and extensive.", "weaknesses": "1. The performance improvement with the proposed synthetic pipeline is limited. Despite the proposed augmentation showing quite significant improvement on the curated small dataset with a few hundred samples, it only shows marginal improvement on the full-scale dataset (Table 7). \n2. The proposed method is not practical. As we usually have data that is much larger than a few hundred samples (in this case, the proposed method does not work well), the traditionality of the proposed method is questionable. Also the proposed pipeline is much complex comparing with traitional augmentation method such as audiomentations, increasing the cost for model training. \n3. Heavy reliance on pretrained CLAP model. This reliance may bias the synthetic data as CLAP model. \n4. It is unclear what CLAP threshold is used in filtering and self-reflection. This is usually a tricky operation as the optimal threshold can vary with different types of sound. More explanations are welcomed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)