Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data

## Abstract

Abstract We present Synthio , a novel approach for augmenting small-scale audio 1 1 1 We use “audio” to refer to acoustic events comprising non-verbal speech, non-speech sounds, and music. classification datasets with synthetic data. Our goal is to improve audio classification accuracy with limited labeled data. Traditional data augmentation techniques, which apply artificial transformations (e.g., adding random noise or masking segments), struggle to create data that captures the true diversity present in real-world audios. To address this shortcoming, we propose to augment the dataset with synthetic audio generated from text-to-audio (T2A) diffusion models. However, synthesizing effective augmentations is challenging because not only should the generated data be acoustically consistent with the underlying small-scale dataset, but they should also have sufficient compositional diversity . To overcome the first challenge, we align the generations of the T2A model with the small-scale dataset using preference optimization. This ensures that the acoustic characteristics of the generated data remain consistent with the small-scale dataset. To address the second challenge, we propose a novel caption generation technique that leverages the reasoning capabilities of Large Language Models to (1) generate diverse and meaningful audio captions and (2) iteratively refine their quality. The generated captions are then used to prompt the aligned T2A model. We extensively evaluate Synthio on ten datasets and four simulated limited-data settings. Results indicate our method consistently outperforms all baselines by 0.1%-39% using a T2A model trained only on weakly-captioned AudioSet. Code will be available here .

## 1 Introduction

Audio classification is the foundational audio processing task of understanding the input audio and assigning it to one or multiple predefined labels. However, training audio classification models requires a lot of high-quality labeled data, which is not always readily available (Ghosh et al., ). Manually collecting and annotating large-scale audio datasets is an expensive, time-consuming, and noisy process (Nguyen et al., ; Martín-Morató & Mesaros, ), and recent concerns about data privacy and usage rights further hinder this process (Ren et al., ).

Data augmentation, which involves expanding original small-scale datasets with additional data, is a promising solution to address data scarcity. Traditional augmentation techniques attempt to diversify audio samples by applying randomly parameterized artificial transformations to existing audio. These methods include spectral masking (Park et al., ), temporal jittering (Nanni et al., ), cropping (Niizumi et al., ), mixing (Seth et al., ; Ghosh et al., ; Niizumi et al., ) and other techniques (Saeed et al., ; Al-Tahan & Mohsenzadeh, ; Manocha et al., ). While these approaches have shown success, they operate at the level of observed data rather than reflecting the underlying data-generating process that occurs in real-world scenarios. As a result, they statistically modify the data without directly influencing the causal mechanisms that produced it, leading to high correlations between augmented samples and limited control over diversity.

Generating synthetic data from pre-trained text-to-audio (T2A) models addresses the limitations of standard data augmentation techniques while retaining their strengths of universality, controllability, and performance (Trabucco et al., ). The recent success of generative models makes this approach particularly appealing (Long et al., ; Evans et al., ). However, generating synthetic audio presents unique challenges due to the complexity of waveforms and temporal dependencies (Ghosh et al., ). We highlight the 3 main challenges in generating effective synthetic data for audio classification: i) Consistency with the original data: Synthetic audio that does not align acoustically with the original dataset can hinder effective augmentation and may cause catastrophic forgetting (Geiping et al., ). This misalignment includes spectral, harmonic, and other inherent acoustic characteristics not easily controlled through prompts. Maintaining consistency with T2A models trained on internet-scale data remains a challenge, and standard fine-tuning can often lead to overfitting (Weili et al., ). ii) Diversity of generated data: Ensuring compositional diversity in the generated synthetic data (e.g., sound events, temporal relationships, background elements, etc.) is critical for effective augmentation. Additionally, a lack of diversity can lead to poor generalization and learning of spurious correlations, impacting performance. Simple, hand-crafted prompts (e.g., “Sound of a metro”) often result in repetitive patterns, and creating diverse, meaningful prompts is labor-intensive. Complex prompts can generate audios that do not preserve the original label. iii) Limitations of current T2A models: T2A models often struggle to generate diverse audios and follow details in prompts. This is largely due to the lack of large-scale, open-source datasets for training, as well as the inherent complexity of non-speech audio domains (Ghosal et al., ). These limitations highlight the need for more advanced approaches for synthetic data generation in audio.

Our Contributions. To address these challenges, we propose Synthio, a novel, controllable and scalable approach for augmenting small-scale audio classification datasets with synthetic data.

Figure: Figure 1: Performance comparison of Synthio with other augmentation methods on down-sampled ESC-50 (100 samples). Traditional augmentation, such as SpecAug, degrades performance on small-scale datasets. Naive synthetic augmentation outperforms traditional methods significantly but plateaus with higher sample counts. Synthio further enhances performance by generating consistent and diverse synthetic data.
Refer to caption: x1.png

Our proposed approach has 2 main steps: i) Aligning the Text-to-Audio Models with Preference Optimization: To generate synthetic audios with acoustic characteristics consistent with the small-scale dataset, we introduce the concept of aligning teaching with learning preferences. Specifically, we align the generations of the T2A model (acting as the teacher) with the target characteristics of the small-scale dataset using preference optimization. This approach ensures that the synthetic audios reflect the acoustic properties of (or sound similar to) the downstream dataset, enabling the classification model (the student) to perform well on test data with similar characteristics. To achieve this, we train a diffusion-based T2A model with preference optimization, where audios generated from Gaussian noise are treated as losers and audios from the downstream dataset are treated as winners. ii) Generating Diverse Synthetic Augmentations: To generate diverse audios for augmentation, we introduce the concept of language-guided audio imagination and imagine novel acoustic scenes with language guidance. Specifically, we generate diverse audio captions that are then used to prompt T2A models to generate audios with varied compositions. To achieve this, we propose MixCap, where we prompt LLMs iteratively to generate captions combining existing and new acoustic components. Additionally, we employ a self-reflection module that filters generated captions and prompts the LLM to revise those that do not align with the intended label. To summarize, our main contributions are:

- 1.
We introduce Synthio, a novel data augmentation approach for audio classification that expands small-scale datasets with synthetic data. Synthio uses novel methods to tackle the inherent challenges of producing consistent and diverse synthetic data from T2A models.
- 2.
We evaluate Synthio across 10 datasets in 4 simulated low-resource settings, demonstrating that, even with a T2A model trained on weakly captioned AudioSet, Synthio outperforms all baselines by 0.1%-39%.
- 3.
We conduct an in-depth analysis of the generated augmentations, highlighting Synthio’s ability to produce diverse and consistent data, its scalability, and its strong performance on complex tasks such as audio captioning.

## 2 Related Work

Data Augmentation for Audio and Beyond. Expanding or augmenting small-scale datasets with additional data has been widely studied in the literature. Traditional augmentation methods, which apply randomly parameterized artificial transformations to data during training, remain the most common approach across language Wei & Zou (); Karimi et al. (), vision (Shorten & Khoshgoftaar, ; Wang et al., ; Yun et al., ), and audio (Park et al., ; Spijkervet, ). For audio, specific techniques include SpecAugment, adding background noise, reverberation, and random spectrogram transformations. With the emergence of generative models, synthetic data augmentation has been increasingly adopted for language (Ghosh et al., ; ; Chen et al., ) and vision (Trabucco et al., ; Zhao et al., ; He et al., ), proving to be more effective than traditional methods. These approaches generally incorporate explicit steps to ensure the consistency and diversity of generated augmentations. While some progress has been made in speech, synthetic data for audio remains relatively underexplored. Recently, parallel work has investigated simple prompting of T2A models to improve T2A generation (Kong et al., ) and environmental scene classification (Ronchini et al., ).

Text-to-Audio Generation. In recent years, there has been a significant surge in research on text-to-audio (T2A) models. The most popular architectures include auto-regressive models based on codecs (Kreuk et al., ; Copet et al., ) and diffusion models Liu et al. (); Ghosal et al. (); Evans et al. (). Clotho (Drossos et al., ) and AudioCaps (Kim et al., ) remain the largest human-annotated datasets for training these models. However, large-scale datasets for T2A model training are still scarce. Recently, Yuan et al. () synthetically captioned AudioSet (Gemmeke et al., ), demonstrating its effectiveness for training T2A models. For downstream adaptation, earlier works have primarily relied on Empirical Risk Minimization (ERM). Majumder et al. () introduced preference optimization for T2A models, creating a synthetic preference dataset based on scores provided by a CLAP model (Elizalde et al., ).

## 3 Background

Diffusion Models. Diffusion models consist of two main processes: a forward process and a reverse process. Given a data point $x_{0}$ with probability distribution $p(x_{0})$, the forward diffusion process gradually adds Gaussian noise to $x_{0}$ according to a pre-set variance schedule $\beta_{1},\cdots,\beta_{T}$ and degrades the structure of the data. At the time step $t$, the latent variable ${x_{t}}$ is only determined by the $x_{t-1}$ due to its discrete-time Markov process nature. At inference time, the diffusion model iteratively executes the reverse process $T$ times starting from a randomly sampled Gaussian Noise $(\epsilon\sim\mathcal{N}(0,\mathbf{I}))$. For more details on diffusion models, we request our readers to refer to Appendix A.1.

We now explain Direct Preference Optimization (DPO), a simpler alternative to RLHF (Christiano et al., ). Abusing notation, we will also $x_{0}$ as random variables for language.

Reward Modeling. Estimating human preferences for a particular generation $x_{0}$, given the context $c$, is challenging because we do not have direct access to a reward model $r(c,x_{0})$. In our scenario, we assume only ranked pairs of samples are available, where one sample is considered a “winner” ($x^{w}_{0}$) and the other a “loser” ($x^{l}_{0}$) under the same conditioning $c$. Based on the Bradley-Terry (BT) model, human preferences can be modeled as:

$$ $p_{\text{BT}}(x^{w}_{0}\succ x^{l}_{0}|c)=\sigma(r(c,x^{w}_{0})-r(c,x^{l}_{0}))$ (1) $$

where $\sigma$ represents the sigmoid function. The reward model $r(c,x_{0})$ is parameterized by a neural network $\phi$ and trained through maximum likelihood estimation for binary classification:

$$ $L_{\text{BT}}(\phi)=-\mathbb{E}_{c,x^{w}_{0},x^{l}_{0}}\left[\log\sigma(r_{ \phi}(c,x^{w}_{0})-r_{\phi}(c,x^{l}_{0}))\right]$ (2) $$

Here, prompt $c$ and data pairs $(x^{w}_{0},x^{l}_{0})$ are drawn from a dataset labeled with human preferences.

RLHF : The goal of RLHF is to optimize a conditional distribution $p_{\theta}(x_{0}|c)$, where c $\sim\mathcal{D}_{c}$, such that the latent reward model $r(c,x_{0})$ is maximized. This is done while regularizing the distribution through the Kullback-Leibler (KL) divergence from a reference distribution $p_{\text{ref}}$, resulting in the following objective:

$$ $\max_{p_{\theta}}\mathbb{E}_{c\sim\mathcal{D}_{c},x_{0}\sim p_{\theta}(x_{0}|c )}[r(c,x_{0})]-\beta D_{\text{KL}}[p_{\theta}(x_{0}|c)\|p_{\text{ref}}(x_{0}|c)]$ (3) $$

Here, the hyperparameter $\beta$ controls the strength of regularization.

DPO : DPO directly optimizes the conditional distribution $p_{\theta}(x_{0}|c)$ to align data generation with the preferences observed in (any form of) feedback. The goal is to optimize the distribution of generated data such that it maximizes alignment with human preference rankings while maintaining consistency with the underlying reference distribution $p_{\text{ref}}(x_{0}|c)$.

The optimal solution $p_{\theta}^{*}(x_{0}|c)$ for the DPO objective can be expressed as:

$$ $p_{\theta}^{*}(x_{0}|c)=p_{\text{ref}}(x_{0}|c)\frac{\exp(r(c,x_{0})/\beta)}{Z (c)}$ (4) $$

where $Z(c)$ is the partition function, defined as:

$$ $Z(c)=\sum_{x_{0}}p_{\text{ref}}(x_{0}|c)\exp(r(c,x_{0})/\beta)$ (5) $$

This term ensures proper normalization of the distribution, and $\beta$ controls the regularization, balancing between adherence to the reference distribution and preference maximization. The reward function $r(c,x_{0})$ is then reparameterized as:

$$ $r(c,x_{0})=\beta\log\frac{p_{\theta}^{*}(x_{0}|c)}{p_{\text{ref}}(x_{0}|c)}+ \beta\log Z(c)$ (6) $$

Using this reparameterization, the reward objective can be formulated as:

$$ $L_{\text{DPO}}(\theta)=-\mathbb{E}_{c,x_{0}^{w},x_{0}^{l}}\left[\log\sigma \left(\beta\log\frac{p_{\theta}(x_{0}^{w}|c)}{p_{\text{ref}}(x_{0}^{w}|c)}- \beta\log\frac{p_{\theta}(x_{0}^{l}|c)}{p_{\text{ref}}(x_{0}^{l}|c)}\right)\right]$ (7) $$

By optimizing this objective, DPO enables direct preference learning, optimizing the conditional distribution $p_{\theta}(x_{0}|c)$ in such a way that it better reflects human preferences, as opposed to traditional approaches that optimize the reward function first and then perform reinforcement learning.

DPO for Diffusion Models: Very recently, Wallace et al.  propose a formulation for optimizing diffusion models with DPO. The primary issue with optimizing diffusion with DPO is that the distribution $p_{\theta}(x_{0}|c)$ is not tractable due to the need to consider all possible diffusion paths leading to $x_{0}$. To address this, Wallace et al. propose to leverage the evidence lower bound (ELBO) to incorporate latents $x_{1}:T$, which represent the diffusion path. The reward $R(c,x_{0}:T)$ accounts for the entire sequence, leading to the reward function:

$$ $r(c,x_{0})=\mathbb{E}_{p_{\theta}(x_{1}:T|x_{0},c)}[R(c,x_{0}:T)]$ (8) $$

Instead of directly minimizing the KL-divergence as typically done, they propose to utlize the upper bound of the joint KL-divergence $\mathbb{D}_{KL}[p_{\theta}(x_{0}:T|c)||p_{\text{ref}}(x_{0}:T|c)]$. This is integrated into the optimization objective, enhancing the practicality of training diffusion models with preferences. The new objective, aiming to maximize the reward and match the distribution of the reverse process of $p_{\theta}$ to the reference model $p_{\text{ref}}$, is given by:

$$ $\max_{p_{\theta}}\mathbb{E}_{c,x_{0}\sim p_{\theta}(x_{0}:T|c)}[r(c,x_{0})]- \beta\mathbb{D}_{KL}[p_{\theta}(x_{0}:T|c)||p_{\text{ref}}(x_{0}:T|c)]$ (9) $$

Training efficiency is improved by approximating the intractable reverse process using a forward approximation $q(x_{1}:T|x_{0})$. The DPO then integrates this into the loss function, which involves comparing the log likelihood ratio of the probabilities under $p_{\theta}$ and $p_{\text{ref}}$ for winning and losing paths:

$$ $L_{\text{DPO-Diffusion}}(\theta)=-\mathbb{E}_{(c,x_{0}^{w},x_{0}^{l})\sim \mathcal{D}_{\text{pref}}}\left[\log\sigma\left(\beta T\log\frac{p_{\theta}(x_ {1}^{w}:T|x_{0}^{w})}{p_{\text{ref}}(x_{1}^{w}:T|x_{0}^{w})}-\beta T\log\frac{ p_{\theta}(x_{1}^{l}:T|x_{0}^{l})}{p_{\text{ref}}(x_{1}^{l}:T|x_{0}^{l})} \right)\right]$ (10) $$

After applying Jensen’s inequality to take advantage of the convexity of $-\log\sigma$, we push the expectation outside, allowing us to simplify the objective. By approximating the denoising process with the forward process, the final form of the loss for DPO in diffusion models, in terms of the L2 noise estimation losses, becomes:

$$ $L_{\text{DPO-Diffusion}}(\theta)=-\mathbb{E}_{(c,x_{0}^{w},x_{0}^{l})\sim \mathcal{D}_{\text{pref}},t,\epsilon_{t}^{w},\epsilon_{t}^{l}}\left[\log\sigma \left(-\beta T\omega(\lambda_{t})\Delta L\right)\right]$ (11) $$

where $\Delta L$ is the L2 weighted noise estimation losses between the preferred (winner) and less preferred (loser) samples.

## 4 Methodology

Let $\mathcal{D}_{\text{small}}=\{(a_{i},l_{i}),1\leq i\leq n\}$ be a high-quality, small-scale human-annotated audio classification dataset with $n$ audio-label pairs. Let $\mathcal{D}_{\text{a-c}}$ be a potentially noisy, large-scale weakly-captioned dataset of audio-caption pairs with zero intersection with $\mathcal{D}_{\text{small}}$. Our goal is to train a T2A model $\mathcal{T}^{\theta}$ using $\mathcal{D}_{\text{a-c}}$, then use it to generate a synthetic dataset $\mathcal{D}_{\text{syn}}$ and then finally add it to $\mathcal{D}_{\text{small}}$ (now attributed as $\mathcal{D}_{\text{train}}$) to improve audio classification performance. This is accomplished through two key steps: first, aligning the generations from $\mathcal{T}^{\theta}$ with the acoustic characteristics of $\mathcal{D}\text{small}$, and second, generating new captions to prompt $\mathcal{T}^{\theta}$ for creating synthetic audio data. We elaborate on these next.

### 4.1 Aligning the Text-to-Audio Model using Preference Optimization

Figure: Figure 2: We propose to align the T2A model $\mathcal{T}^{\theta}$ with the small-scale dataset $\mathcal{D}_{\text{small}}$ using DPO. This helps us generate audios with acoustic characteristics aligned to that of $\mathcal{D}_{\text{small}}$.
Refer to caption: x2.png

T2A models trained on internet-scale data often generate audio that diverges from the characteristics of small-scale datasets, resulting in distribution shifts. These mismatches can include variations in spectral (e.g., frequency content), perceptual (e.g., pitch, loudness), harmonic, or other acoustic characteristics (^2^22When prompted with “sound of a bus” for the category “bus” in the TUT-Urban dataset, the generated audio may not reflect the typical bus sounds in European cities (where TUT was recorded), as bus sounds can vary by region, with some featuring loud engines and dense crowds while others have quieter engines and sparse crowds.). This misalignment arises from the non-deterministic nature of T2A generation and it is impractical to provide detailed attributes (like “loud” or “high-pitched”) in prompts, as (i) there are no scalable methods for extracting specific attributes for each label, and (ii) T2A models struggle with accurately following fine-grained prompt details (Wang et al., ).

To address these issues, we propose the concept of aligning teaching with learning preferences. Our approach assumes that the classification model (viewed as the student) performs better when trained on synthetic audio that closely matches the inherent acoustic properties of our high-quality and human-labeled $\mathcal{D}_{\text{small}}$. Thus, we align the generations of the T2A model (viewed as the teacher) to $\mathcal{D}_{\text{small}}$, ensuring that the generated augmentations align with the desired characteristics and sound similar, ultimately enhancing the student model’s ability to generalize to similarly characterized test data. As shown in Fig. 2, we achieve this using preference optimization (DPO in our case) and align generations of $\mathcal{T}^{\theta}$ with $\mathcal{D}_{\text{small}}$. Unlike standard fine-tuning, which can lead to less diverse outputs and overfitting due to a narrow focus on minimizing loss, preference optimization encourages greater exploration in the model’s output space, preventing mode collapse and fostering more diverse augmentations. Additionally, DPO leverages pairwise learning, offering richer training signals compared to the independent outputs used in standard fine-tuning, further mitigating overfitting risks. We detail our two-step approach for DPO optimization below:

Step 1: Construction of the Preference Dataset. To create our preference dataset $\mathcal{D}_{\text{pref}}=\{(a^{w}_{1},a^{l}_{1}),\cdots,(a^{w}_{j},a^{l}_{j})\}$, we first generate template-based captions for each instance in $\mathcal{D}_{\text{small}}$ in the form: “Sound of a label”, where label is the category associated with the audio. For each instance, we prompt the T2A model $j$ times, with all generations starting from randomly initialized Gaussian noise (generation configuration is detailed in Section 5). Each generated audio is then paired with the corresponding ground-truth audio from the gold dataset. This resulting $\mathcal{D}_{\text{pref}}$ dataset has $n\times j$ instances, where the generated audio is treated as the “loser” and the ground-truth audio as the “winner”. This simple approach has proven highly effective in aligning generations by generative models by prior work (Majumder et al., ; Tian et al., ).

Step 2: Preference Optimization Using DPO. After constructing $\mathcal{D}_{\text{pref}}$, we train our T2A model on this dataset with DPO using the approach outlined in Section 3. The resulting aligned model is referred to as $\mathcal{T}^{\theta}_{\text{aln}}$. Details of the hyper-parameters used for training are provided in Section 5.

### 4.2 Generating Diverse Synthetic Augmentations

It is not well-studied in the literature on how to leverage synthetic audio generation for downstream tasks. The only existing work relied on manually crafted prompt templates (e.g., “Sound of a {label}”) (Ronchini et al., ). It has a significant limitation: there is no precise control over the specific components in the generated audio for a given caption. This can result in repetitive or completely inconsistent patterns, particularly with weaker T2A models (^3^33For example, when prompted with “Sound of a park”, we observed that 9 out of 10 times, the model generated the sound of children playing as part of the generated audio. On the other hand, when prompted with “Sound of a airport”, the model generates audios with background announcements, which could vary by regions.). These could bias the model to learn spurious correlations, a known issue in synthetic data augmentation (Ghosh et al., ).

While the alignment stage helps the T2A model generate audio with acoustic characteristics similar to the small-scale dataset (e.g., spectral, harmonic, etc.), it does not fully account for the compositional diversity of the generated audios (e.g., sound events, their temporal relationships, background elements). To tackle this, we propose the concept of language-guided audio imagination, where we propose to imagine novel audios guided by language. Specifically, we leverage the reasoning abilities of LLMs to generate diverse and meaningful captions for a category label in a controlled yet scalable manner. These captions are then used to prompt our aligned T2A model for generating novel audios.

Figure: Figure 3: Overview of our proposed Language-Guided Audio Imagination for generating diverse synthetic augmentations. Starting with the small-scale dataset, we first generate audio captions and use an LLM to extract acoustic components (Prompt 1). Using these components and audio labels, we prompt the LLM to generate new and diverse captions (Prompt 2), which are then used to prompt the aligned T2A model for audio generation. The generated audios are filtered for label consistency using CLAP, with accepted audios added to the final synthetic dataset. Rejected audios undergo caption revision (Prompt 3) through a self-reflection process, and the revised captions are used to regenerate audios, iterating this process $i$ times. Example captions are in Table 6.
Refer to caption: x3.png

#### 4.2.1 Generating Diverse Prompts with MixCap

We propose MixCap, a prompt generation method that creates diverse and effective captions in three steps: First, we employ GAMA (Ghosh et al., ) to caption all audio files in $\mathcal{D}_{\text{small}}$. Next, we prompt an LLM to extract phrases describing the acoustic components of the audio. These components correspond to the acoustic elements such as backgrounds and foreground events, and their attributes and relations, etc (see prompt in Appendix A.2). Finally, for each training instance in $\mathcal{D}_{\text{small}}$, we prompt the LLM with the ground-truth label and the extracted components from all instances to generate $N$ diverse audio captions that blend existing and new components. This approach prevents overemphasis on events already present in $\mathcal{D}_{\text{small}}$ and encourages diversity in the generated audio. These captions are then used to prompt the aligned T2A model to generate synthetic data $\mathcal{D}_{\text{syn}}$.

#### 4.2.2 Filtering & Self-Reflection

Filtering. After generating captions and their corresponding audio, we filter the audio for label consistency. While LLMs can generate diverse captions, the audio produced must remain aligned with the ground-truth label. To ensure this, we use CLAP  to evaluate the generated audio, accepting those that meet a similarity threshold of $p\%$ and rejecting the rest. We denote the accepted audios as $\mathcal{D}_{\text{syn}}^{\text{acc}}$ and the rejected ones as $\mathcal{D}_{\text{syn}}^{\text{rej}}$. The CLAP model is pre-trained on $\mathcal{D}_{\text{a-c}}$ and fine-tuned on $\mathcal{D}_{\text{small}}$ to adapt to the target dataset. Example captions are in Table 6, and prompts are in Appendix A.2.

Self-Reflection. For the rejected audios in $\mathcal{D}_{\text{syn}}^{\text{rej}}$, we prompt the LLM to reflect on its generated captions and revise them to better align with the target label. Precisely, we feed the LLM with the original caption of each rejected audio along with extracted components from all accepted captions in $\mathcal{D}_{\text{syn}}^{\text{acc}}$ and task it to rewrite the rejected captions. The revised captions are then used to generate new audio, which is again filtered using CLAP. Audios that meet the threshold are added to $\mathcal{D}_{\text{syn}}^{\text{acc}}$; those that do not are returned to $\mathcal{D}_{\text{syn}}^{\text{rej}}$. This process repeats for $i$ iterations or until $\mathcal{D}_{\text{syn}}^{\text{rej}}$ is empty. We illustrate examples of generated and revised captions in Table 6.

Fine-tuning for Audio Classification. After the self-reflection stage, the final set of accepted synthetic audios is denoted as $\mathcal{D}_{\text{syn}}$, containing $\approx N\times n$ audio-label pairs, where $N$ represents the augmentation factor (e.g., with 100 gold samples, we generate $100\times N$ synthetic samples). This set is then combined with $\mathcal{D}_{\text{small}}$ to form the final training dataset $\mathcal{D}_{\text{train}}$, which is then used to train the audio classification model.

## 5 Experimental Setup

Models and Hyper-Parameters. For our T2A model, we choose the Stable Audio architecture (Evans et al., ). Stable Audio is based on the Diffusion Transformer architecture, and we refer our readers to the original paper for more details. We train the model from scratch on Sound-VECaps (Yuan et al., ) (with $\approx$1.5 million weakly captioned audio-caption pairs) to avoid any data leakage. For training, we employ a batch size of 64, an AdamW optimizer, a learning rate of 5e-4, and a weight decay of 1e-3 for 40 epochs. For DPO-based alignment tuning, we fine-tune with a batch size of 32 and a learning rate of 5e-4 for 12 epochs. For our audio classification model, we employ the Audio Spectrogram Transformer (AST) (Gong et al., ) (pre-trained on the AudioSet dataset) and fine-tune it with a batch size of 24 and learning rate of 1e-4 for 50 epochs. In each experiment, we adjust the number of generated augmentations $N$ (ranging from 1 to 5) based on performance on the validation set. All hyper-parameters are fixed across splits, and results are averaged across 3 runs.

Datasets. We create small-scale datasets by downsampling commonly used audio classification datasets to $n$ samples. Our selected datasets include a mix of music, everyday sounds, and acoustic scenes. For multi-class classification, we use NSynth Instruments, TUT Urban, ESC50 (), USD8K (Salamon et al., ), GTZAN (Tzanetakis et al., ), Medley-solos-DB (Lostanlen & Cella, ), MUSDB18 (Rafii et al., ), DCASE Task 4 (Mesaros et al., ), and Vocal Sounds (VS) (Mesaros et al., ), evaluating them for accuracy. For multi-label classification, we use the FSD50K (Fonseca et al., ) dataset and evaluate it using the F${}_{1}^{\text{macro}}$ metric. We exclude AudioSet from evaluation as Sound-VECaps is derived from it. To ensure a downsampled dataset that has a label distribution similar to that of the of the original dataset, we employ stratified sampling based on categories. Our experiments are conducted with $n$ = {50, 100, 200, 500} samples, and we downsample the validation sets for training while evaluating all models on the original test splits.

**Table 1: Result comparison of Synthio with baselines on 10 datasets and 4 small-scale settings. $n$ refers to the number of samples in the small-scale dataset augmented with synthetic data. Synthio outperforms our baselines by 0.1% - 39%. We also highlight the relative improvements by Synthio compared to the Gold-only.**
| n | Method | ESC-50 | USD8K | GTZAN | Medley | TUT | NSynth | VS | MSDB | DCASE | FSD50K |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 50 | Gold-only (No Aug.) | 22.25 | 55.09 | 47.05 | 47.23 | 37.60 | 33.32 | 77.49 | 56.85 | 12.09 | 8.69 |
| Random Noise | 18.50 | 57.42 | 45.20 | 46.55 | 35.86 | 32.42 | 76.41 | 52.55 | 13.21 | 9.15 |  |
| Pitch Shifting | 20.55 | 59.32 | 46.80 | 48.17 | 37.22 | 34.34 | 78.17 | 54.50 | 12.93 | 10.04 |  |
| SpecAugment | 19.50 | 58.36 | 46.00 | 47.18 | 36.73 | 27.32 | 77.27 | 53.25 | 12.81 | 9.22 |  |
| Audiomentations | 20.35 | 60.13 | 47.25 | 48.30 | 38.24 | 28.15 | 79.12 | 54.51 | 13.28 | 11.44 |  |
| Retrieval | 19.20 | 37.14 | 42.55 | 43.65 | 35.80 | 31.27 | 71.42 | 51.35 | 10.53 | 7.28 |  |
| Vanilla Syn. Aug. | 40.75 | 63.54 | 55.35 | 47.23 | 41.50 | 33.17 | 78.37 | 54.10 | \ul<br>15.89 | 11.76 |  |
| + LLM Caps. | 36.80 | 65.84 | 63.74 | 55.36 | 40.90 | 38.17 | 78.77 | 57.05 | 13.07 | 11.84 |  |
| Synthio (ours) | 49.50+122% | 76.12+38% | 68.20+44% | 60.58+28% | 43.84+17% | 40.83+22% | 80.67+4% | 60.15+5% | 17.23+42% | 15.41+77% |  |
| w/ Template Captions | 41.25 | 66.11 | 64.40 | 54.52 | 41.37 | 37.52 | 78.57 | \ul<br>59.60 | 14.15 | 13.76 |  |
| w/ ERM | 41.30 | 69.80 | 61.70 | 56.60 | 42.00 | 38.62 | \ul<br>79.75 | 57.75 | 13.28 | \ul<br>14.17 |  |
| w/o Self-Reflection | \ul<br>45.25 | \ul<br>72.57 | \ul<br>64.55 | \ul<br>58.00 | \ul<br>42.81 | \ul<br>39.50 | 78.56 | 57.25 | 15.63 | 14.02 |  |
| w/o MixCap | 42.70 | 64.72 | 54.65 | 52.18 | 41.93 | 36.13 | 78.70 | 58.80 | 14.82 | 13.61 |  |
| w/o DPO | 36.55 | 68.12 | 56.10 | 52.55 | 41.39 | \ul<br>40.31 | 79.03 | 57.55 | 14.53 | 11.28 |  |
| 100 | Gold-only (No Aug.) | 56.75 | 72.89 | 64.15 | 57.81 | 47.14 | 39.11 | 84.32 | 65.60 | 12.50 | 10.70 |
| Random Noise | 58.50 | 71.54 | 65.50 | 56.98 | 46.21 | 38.20 | 83.33 | 66.15 | 13.35 | 14.95 |  |
| Pitch Shifting | 59.55 | 73.52 | 66.75 | 58.46 | 47.50 | 39.53 | 85.07 | 68.25 | 12.19 | 15.02 |  |
| SpecAugment | 47.50 | 72.43 | 69.75 | 58.06 | 50.07 | 41.96 | 85.14 | 66.40 | 14.17 | \ul<br>15.72 |  |
| Audiomentations | 48.50 | 73.82 | \ul<br>71.05 | 59.32 | 51.14 | 42.15 | \ul<br>85.24 | \ul<br>68.40 | 16.93 | 14.29 |  |
| Retrieval | 52.45 | 68.24 | 61.55 | 54.83 | 45.39 | 37.84 | 83.27 | 58.55 | 10.93 | 10.05 |  |
| Vanilla Syn. Aug. | 77.25 | 77.31 | 68.25 | 63.58 | 49.96 | 42.31 | 84.78 | 63.55 | 15.73 | 13.78 |  |
| + LLM Caps. | 67.05 | 79.73 | 67.90 | 65.79 | 48.63 | 41.83 | 84.83 | 65.95 | 16.32 | 14.12 |  |
| Synthio (ours) | 83.35+47% | 85.00+17% | 71.20+11% | 71.23+23% | 52.42+11% | 44.92+15% | 86.70+3% | 68.80+5% | 19.38+55% | 17.33+62% |  |
| w/ Template Captions | \ul<br>78.00 | 80.32 | 68.15 | 64.20 | 49.95 | 42.76 | 85.11 | 66.05 | 16.32 | 14.32 |  |
| w/ ERM | 73.20 | 81.81 | 67.25 | 66.57 | 51.11 | 43.74 | 84.73 | 68.00 | \ul<br>17.21 | 15.19 |  |
| w/o Self-Reflection | 77.65 | \ul<br>82.38 | 69.55 | \ul<br>68.52 | \ul<br>51.75 | \ul<br>44.38 | 82.53 | 66.20 | 15.89 | 13.96 |  |
| w/o MixCap | 73.50 | 78.30 | 68.50 | 66.52 | 50.63 | 42.27 | 83.52 | 66.35 | 16.77 | 14.93 |  |
| w/o DPO | 66.75 | 75.46 | 66.15 | 60.81 | 48.78 | 40.31 | 84.67 | 67.85 | 14.83 | 13.92 |  |
| 200 | Gold-only (No Aug.) | 84.75 | 74.80 | 77.00 | 67.41 | 55.32 | 48.77 | 87.38 | 68.80 | 23.15 | 14.82 |
| Random Noise | 83.55 | 75.15 | 75.50 | 66.71 | 54.42 | 47.83 | 86.45 | 65.45 | 24.82 | 17.77 |  |
| Pitch Shifting | 84.90 | 74.48 | 78.55 | 67.74 | 55.44 | 48.12 | \ul<br>87.47 | 69.80 | 23.11 | 18.73 |  |
| SpecAugment | 85.10 | 76.46 | 76.25 | 65.70 | 55.72 | 54.80 | 87.42 | 69.25 | 27.36 | 18.91 |  |
| Audiomentations | 85.25 | 75.80 | 77.30 | 67.00 | 55.21 | 53.15 | 86.08 | 70.50 | 26.29 | 19.17 |  |
| Retrieval | 82.55 | 71.20 | 73.65 | 65.80 | 53.25 | 47.63 | 86.28 | 63.55 | 19.51 | 15.36 |  |
| Vanilla Syn. Aug. | 85.40 | 77.96 | 77.10 | \ul<br>78.97 | 55.51 | 55.20 | 86.49 | 72.95 | 28.55 | 19.04 |  |
| + LLM Caps. | 85.80 | 78.37 | 79.55 | 74.14 | 54.73 | 56.21 | 87.02 | 73.16 | 28.40 | 19.34 |  |
| Synthio (ours) | 86.10+2% | 82.81+11% | 82.05+7% | 79.40+18% | 56.83+3% | 57.10+17% | 87.52+0.2% | 80.40+17% | 32.81+42% | 21.60+46% |  |
| w/ Template Captions | \ul<br>85.95 | 80.84 | 79.25 | 77.56 | 55.99 | 56.33 | 87.25 | 74.55 | 29.12 | 20.19 |  |
| w/ ERM | 85.35 | 79.82 | \ul<br>80.20 | 74.43 | 55.76 | 56.15 | 86.92 | 74.40 | 29.81 | 19.26 |  |
| w/o Self-Reflection | 84.85 | \ul<br>81.97 | 78.25 | 75.53 | \ul<br>56.39 | \ul<br>56.76 | 86.22 | 75.55 | \ul<br>31.13 | 18.45 |  |
| w/o MixCap | 84.95 | 81.27 | 79.55 | 73.50 | 55.27 | 55.54 | 85.78 | \ul<br>78.55 | 28.35 | \ul<br>20.58 |  |
| w/o DPO | 84.80 | 76.23 | 75.30 | 73.13 | 55.99 | 52.73 | 86.52 | 73.15 | 26.79 | 19.58 |  |
| 500 | Gold-only (No Aug.) | 90.75 | 87.88 | 79.25 | 75.65 | 65.72 | 63.47 | 89.33 | 72.05 | 36.68 | 20.19 |
| Random Noise | 89.55 | 88.25 | 78.90 | 76.01 | 65.10 | 64.15 | 90.15 | 73.25 | 37.21 | 21.57 |  |
| Pitch Shifting | 88.50 | 88.83 | 79.75 | 75.61 | 64.93 | 64.59 | 89.87 | 72.15 | 36.54 | 22.17 |  |
| SpecAugment | 89.50 | \ul<br>89.01 | 80.25 | 76.68 | 66.74 | 64.43 | 90.38 | 72.95 | 38.33 | 22.52 |  |
| Audiomentations | 89.95 | 88.75 | \ul<br>81.25 | 77.66 | \ul<br>66.92 | \ul<br>65.21 | \ul<br>91.34 | 73.65 | 38.75 | 23.11 |  |
| Retrieval | 85.50 | 84.86 | 77.25 | 73.62 | 62.73 | 61.44 | 87.33 | 70.20 | 30.17 | 15.28 |  |
| Vanilla Syn. Aug. | 91.50 | 88.18 | 79.35 | \ul<br>77.97 | 65.93 | 64.52 | 90.31 | 73.25 | 37.26 | \ul<br>24.39 |  |
| + LLM Caps. | 89.90 | 86.91 | 79.55 | 77.91 | 65.95 | 64.39 | 90.09 | 73.05 | 38.74 | 23.83 |  |
| Synthio (ours) | 92.10+2% | 89.18+2% | 82.25+4% | 78.62+4% | 67.81+3% | 65.40+3% | 91.42+2% | 74.70+3% | 39.24+6% | 24.93+23% |  |
| w/ Template Captions | 91.70 | 88.93 | 80.40 | 76.64 | 66.47 | 64.71 | 90.97 | 73.35 | 38.28 | 23.65 |  |
| w/ ERM | 91.20 | 88.25 | 79.15 | 77.38 | 65.80 | 64.27 | 88.74 | \ul<br>74.20 | 38.03 | 23.72 |  |
| w/o Self-Reflection | \ul<br>91.85 | 88.72 | 80.15 | \ul<br>78.57 | 66.21 | 63.89 | 90.17 | 72.15 | 37.97 | 23.62 |  |
| w/o MixCap | 91.70 | 87.93 | 80.95 | 76.61 | 65.91 | 64.23 | 90.23 | 73.40 | \ul<br>39.11 | 22.83 |  |
| w/o DPO | 90.15 | 88.21 | 79.45 | 76.03 | 66.01 | 63.61 | 89.83 | 72.65 | 37.04 | 21.28 |  |

Baselines. Our baselines include: (i) Gold-only (No Aug.): We employ only the small-scale dataset for training and do not perform any augmentations. (ii) Traditional augmentation baselines: SpecAugment, Noise Augmentation (we either add random Gaussian noise or background noise from AudioSet and present averaged results), Pitch and Time Shift and Audiomentations (Jordal, ) – a combination of the AddGaussianNoise, TimeStretch, PitchShift, Shift, SpecFrequencyMask, TimeMask and TimeStretch – combination with the highest average score on 4 datasets and splits and was selected after grid search over all possible combinations). (iii) Generative baselines: Vanilla Synthetic Augmentation (Vanilla Syn. Aug.) – we prompt $\mathcal{T}_{\theta}$ with template captions), Vanilla Syn. Aug. + LLM Caps – we prompt $\mathcal{T}_{\theta}$ with random captions generated with LLMs. (iv) Finally, inspired by  Burg et al. (), we also employ a retrieval baseline where instead of generating augmentations from our T2A model trained on $\mathcal{D}_{\text{a-c}}$, we just retrieve the top-n instances (w.r.t. CLAP similarity) from the AudioSet for each instance in $\mathcal{D}_{\text{small}}$ as our augmentations.

Ablations. We ablate Synthio with: (i)    w/o Self-Reflection: We remove the repetitive self-reflection module and iterate and filter only once; (ii)    w/o DPO: We skip the tuning step and prompt the un-alined $\mathcal{T}^{\theta}$ for augmentations; (iii)    w/ ERM: We replace DPO tuning with standard Empirical Risk Minimization(ERM)-based fine-tuning with diffusion loss; (iv)    w/ Template Captions: We remove MixCap and self-reflection modules and prompt $\mathcal{T}^{\theta}_{\text{aln}}$ with template captions; (v)    w/o MixCap: Similar to our Random Captions baseline, but we retain all other modules of Synthio.

Figure: Figure 4: Comparison of spectral and pitch features between generated audios in $\mathcal{D}_{\text{syn}}$ and real audios in $\mathcal{D}_{\text{small}}$ (for $n$ = 100). Synthio-generated audios closely replicate the features of real data, demonstrating its ability to produce augmentations that maintain consistency with the original dataset (also see FAD scores in Sec. A.4.3).
Refer to caption: x4.png

## 6 Results and Discussion

In this section, we present our main benchmark results, followed by detailed analyses to highlight Synthio’s strengths and explore its versatility. Key areas of investigation include Synthio’s consistency and diversity in augmentations, performance on synthetic-only datasets, scalability, impact on long-tailed categories, and its potential extension to complex audio understanding tasks like captioning.

Main Results. Table 1 showcases the performance comparison between Synthio and the baseline methods. Synthio consistently outperforms all baselines by 0.1%-39%, achieving notable improvements in overall classification accuracy compared to Gold-only. The highest gains are observed on USD8K, while the least is on Vocal Sound, likely due to the T2A dataset’s heavy representation of music compared to the more sparse vocal sounds. Performance gains tend to decrease as the number of gold samples $n$ in $\mathcal{D}_{\text{small}}$ grows, aligning with observed trends in prior studies. Although Vanilla Synthetic Augmentations emerge as the strongest baseline, they lag behind Synthio by 3.5%.

Ablations. The most significant performance drop in Synthio is observed w/o DPO, resulting in an average decline of 4.5%, highlighting the crucial role of consistency in generating effective augmentations. Second to w/o DPO, the the highest drop is seen in w/ Template Captions, with average decline of 2.7%, thus highlighting the importance of MixCap.

### 6.1 How Consistent and Diverse are Augmentations generated by Synthio?

**Table 2: CLAP similarity score between real audios and generated data. Lower scores show higher compositional diversity among generated augs.**
| # | Method | USD8K($\downarrow$) | NSynth($\downarrow$) |
| --- | --- | --- | --- |
| 100 | Vanilla Syn. Aug. | \ul45.17 | \ul31.76 |
| Synthio (ours) | 35.09 | 22.97 |  |
| w/ Template Captions | 46.82 | 33.00 |  |
| w/ ERM | 50.01 | 42.33 |  |
| 200 | Vanilla Syn. Aug. | 47.22 | \ul33.81 |
| Synthio (ours) | 34.58 | 23.03 |  |
| w/ Template Captions | \ul46.84 | 37.16 |  |
| w/ ERM | 52.54 | 43.98 |  |

Fig. 4 compares the distributions of pitch and various spectral features between generated audios in $\mathcal{D}_{\text{syn}}$ and real audios in $\mathcal{D}_{\text{small}}$ across different methods on the USD8K and NSynth datasets. The features analyzed include Pitch Salience (clarity of the main pitch) (Ricard, ), Spectral Flatness (tonal vs. noise-like quality) (Peeters, ), Flux (rate of spectral change) (Tzanetakis & Cook, ), and Complexity (level of sound detail) (Laurier et al., ). Notably, Synthio-generated audios closely replicate the spectral features of the original audios, showing the best alignment among all methods and demonstrating Synthio’s ability to generate consistent augmentations. Table 2 presents CLAP similarity scores between ground-truth audios and their $N$ generated augmentations, averaged across all dataset instances. Audios generated with Synthio achieve the highest compositional diversity for generated audios among all baselines. Table 8 shows that audios generated using Synthio have the highest similarity with the ground-truth category label.

### 6.2 How good are synthetic audios generated by Synthio?

**Table 3: Performance comparison of Synthio with baselines on synthetic-only audio classification.**
| n | Method | GTZAN | VS | TUT | MSDB |
| --- | --- | --- | --- | --- | --- |
| 100 | Gold-only (No Aug.) | 64.15 | 84.32 | 47.14 | 65.60 |
| Vanilla Syn. Aug. | \ul29.05 | \ul34.13 | 21.69 | 35.60 |  |
| Synthio (ours) | 33.10 | 39.20 | 24.51 | 56.45 |  |
| w/ Template Captions | 24.50 | 30.99 | 21.73 | 40.40 |  |
| w/ ERM | 25.65 | 32.76 | \ul24.40 | \ul42.85 |  |
| w/o DPO | 17.60 | 21.57 | 20.39 | 30.20 |  |
| 200 | Gold-only (No Aug.) | 77.00 | 87.38 | 55.32 | 68.80 |
| Vanilla Syn. Aug. | \ul32.35 | \ul41.96 | 24.23 | 39.25 |  |
| Synthio (ours) | 35.15 | 48.14 | 27.00 | 61.45 |  |
| w/ Template Captions | 29.90 | 35.53 | 23.61 | 41.20 |  |
| w/ ERM | 28.10 | 36.29 | \ul25.71 | \ul46.70 |  |
| w/o DPO | 19.85 | 26.85 | 21.40 | 36.75 |  |

Consistent with prior findings in vision (He et al., ), we observe that synthetic data alone performs sub-optimally compared to human-annotated data. However, our results show that enhancing the consistency and diversity of synthetic data aided by a small-scale version of the target dataset significantly improves model performance. Table 3 compares models trained exclusively on synthetic data with our baselines (i.e., only $\mathcal{D}_{\text{syn}}$ is used for training AST). Synthio outperforms all baselines by 0.1%-26.25%, with DPO-based alignment driving the improvements.

### 6.3 Can Synthio be extended to the more complex audio captioning task?

Audio captioning, unlike classification, involves describing the content of an audio sample using natural language, making it a more complex task. To demonstrate Synthio’s effectiveness for audio captioning, we evaluated it on down-sampled versions of AudioCaps. For this task, we adapted Synthio by extracting components directly from the available captions and retrained our T2A model on a modified version of Sound-VECaps, excluding any audio from AudioSet.

**Table 4: Performance comparison of Synthio with baselines on audio captioning.**
| n | Method | METEOR ($\uparrow$) | CIDEr ($\uparrow$) | SPICE ($\uparrow$) | SPIDEr ($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| 500 | Gold-only (No Aug.) | 0.125 | 0.148 | 0.0754 | 0.112 |
| Vanilla Syn. Aug. | 0.128 | 0.157 | 0.0741 | 0.136 |  |
| VECaps Retrieval | 0.108 | 0.094 | 0.0550 | 0.082 |  |
| Synthio (ours) | 0.169 | 0.227 | 0.104 | 0.194 |  |
| 1000 | Gold-only (No Aug.) | 0.127 | 0.157 | 0.067 | 0.112 |
| Vanilla Syn. Aug. | 0.135 | 0.166 | 0.092 | 0.140 |  |
| VECaps Retrieval | 0.088 | 0.097 | 0.068 | 0.100 |  |
| Synthio (ours) | 0.185 | 0.256 | 0.119 | 0.202 |  |

Training and evaluation were conducted using the EnCLAP framework (Kim et al., ), and the dataset was enriched with 4× synthetic samples. As shown in Table 4, Synthio significantly outperforms baseline settings, with improvements largely due to better alignment w/ DPO. However, manual inspection revealed that generated audios occasionally do not match their captions compositionally, reflecting limitations of the current T2A model. While this issue does not affect classification, it poses challenges for captioning. We will explore more advanced methods as part of future work.

### 6.4 How well does Synthio Scale?

**Table 5: Performance comparison of Synthio with other baselines on different values of $N$.**
| Dataset | Method | Scaling Factor $N$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 1x | 2x | 3x | 4x | 5x |  |  |
| ESC50 | SpecAugment | 47.50 | 47.50 | 47.50 | 47.50 | 47.50 |
| Vanilla Syn. Aug. | 67.90 | 77.25 | 76.75 | 75.60 | 71.25 |  |
| Synthio (ours) | 77.45 | 81.75 | 82.55 | 83.15 | 83.35 |  |
| w/o MixCap | 64.30 | 68.45 | 71.55 | 72.85 | 73.50 |  |
| w/o DPO | 61.55 | 64.25 | 65.95 | 66.60 | 66.75 |  |
| NSynth | SpecAugment | 41.96 | 41.96 | 41.96 | 41.96 | 41.96 |
| Vanilla Syn. Aug. | 33.13 | 35.28 | 42.31 | 41.54 | 38.27 |  |
| Synthio (ours) | 35.28 | 36.37 | 43.56 | 44.92 | 44.81 |  |
| w/o MixCap | 40.41 | 41.08 | 41.95 | 42.27 | 42.15 |  |
| w/o DPO | 39.23 | 39.42 | 40.17 | 40.31 | 39.82 |  |

Table 5 compares the performance of Synthio, SpecAugment, and Vanilla Synthetic Augmentations across different scaling factors $N$ = {1, 2, 3, 4, 5}, where $N$ represents the number of synthetic samples generated per original sample in the small-scale dataset. As observed, SpecAugment, a traditional augmentation method, cannot scale with increasing $N$, and the performance of Vanilla Synthetic Augmentations plateaus at higher $N$. A similar saturation occurs with Synthio when MixCap is not used. Even without DPO, Synthio maintains better scalability, though with reduced overall performance. These results highlight that MixCap’s ability to generate diverse captions is crucial for Synthio’s scalability.

### 6.5 Does Synthio help long-tailed categories?

Figure: Figure 5: Category-wise improvement in performance with Synthio augmentations for long-tailed categories.
Refer to caption: x5.png

Figure 5 shows the classification accuracy on four underrepresented categories in the ESC-50 and NSynth datasets, comparing performance before and after applying Synthio augmentations. We selected categories with the lowest frequency in the datasets, such as flute and guitar, which appear only once in the down-sampled sets. Synthio significantly boosts accuracy, with improvements up to 48%. Notably, categorie labels like flute and guitar, which originally had 0% accuracy, show substantial gains with Synthio augmentation. This demonstrates Synthio’s effectiveness in enhancing performance on long-tail labels, a common challenge in real-world datasets (Zhang et al., ).

## 7 Conclusion, Limitations, and Future Work

We introduced Synthio, a novel approach for augmenting small-scale audio classification datasets with synthetic data. Synthio incorporates several innovative components to generate augmentations that are both consistent with and diverse from the small-scale dataset. Our extensive experiments demonstrate that even when using a T2A model trained on a weakly-captioned AudioSet, Synthio significantly outperforms multiple baselines.

However, Synthio has some limitations: (i) Its performance is influenced by the capabilities of the T2A model and the quality of its training data. As T2A models continue to improve, we expect Synthio’s performance to benefit accordingly. (ii) The process of generating audio captions using LLMs may introduce biases inherent in the LLMs into the training process. (iii) Synthio is computationally more intensive than traditional augmentation methods due to the need for prompting LLMs and T2A models. We anticipate that ongoing advancements in model efficiency will help mitigate these computational challenges.

## 8 Reproducibility Statement

All codes will be open-sourced upon paper acceptance, including all T2A checkpoints. All experimental details, including training parameters and hyper-parameters, are provided in Section 5.

## Appendix A Appendix

Table of Contents:

- •
A.1 Background on Diffusion Models
- •
A.2 Prompts
- •
A.3 Examples
- •
A.4 Extra Results
- •
A.5 Dataset Details

### A.1 Diffusion Models

Diffusion models consist of two main processes: a forward process and a reverse process. Given a data point $x_{0}$ with probability distribution $p(x_{0})$, the forward diffusion process
gradually adds Gaussian noise to $x_{0}$ according to a pre-set variance schedule $\beta_{1},\cdots,\beta_{T}$ and degrades the structure of the data. At the time step $t$, the latent variable ${x_{t}}$ is only determined by the $x_{t-1}$ due to its discrete-time Markov process nature, and can be expressed as:

$$ $p(x_{t}\mid x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I),$ (12) $$

As $t$ increases over several diffusion steps, $p(x_{T})$ approaches a unit spherical Gaussian distribution. The marginal distribution of $x_{t}$ at any given step can be expressed analytically as:

$$ $p(x_{t}\mid x_{0})=\mathcal{N}(x_{t};\sqrt{\alpha_{t}}x_{0},(1-\alpha_{t})I),$ (13) $$

where $\alpha_{t}=\prod_{s=1}^{t}(1-\beta_{s})$. The reverse process aims to reconstruct the original data from the noise-corrupted version by learning a series of conditional distributions. The transition from $x_{t}$ to $x_{t-1}$ is modeled as:

$$ $p_{\theta}(x_{t-1}\mid x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}^{t-1},\sigma_{ \theta}^{t-1}),$ (14) $$

$$ $\mu_{\theta}^{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{\beta_{t}}{ \sqrt{1-\bar{\alpha}t}}\epsilon_{\theta}\left(x_{t},t\right)\right),$ (15) $$

$$ $\sigma_{\theta}^{t-1^{2}}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\cdot \beta_{t},$ (16) $$

where $\alpha_{t}=1-\beta_{t},\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$, $\theta$ represents the learnable parameters, $\mu_{\theta}^{t-1}$ is the mean estimate, $\sigma_{\theta}^{t-1^{2}}$ is the standard deviation estimate, and $\epsilon_{\theta}(x_{t},t)$ is the noise estimated by the neural network. The reverse process estimates the data distribution $p(x_{0})$ by integrating over all possible paths:

$$ $p_{\theta}(x_{0})=\int p_{\theta}(x_{T})\prod_{t=1}^{T}p_{\theta}(x_{t-1}\mid x _{t})\,dx_{1}:T$ (17) $$

where $p_{\theta}(x_{T})=\mathcal{N}(x_{T};0,I)$. At inference time, the diffusion model iteratively executes the reverse process (Eq. 17) $T$ times starting from a randomly sampled
Gaussian Noise $(\epsilon\sim\mathcal{N}(0,\mathbf{I}))$.

### A.2 Prompts

Fig. 6,  7,  8 and  9 illustrate all the prompts used in our experiments. For all experiments, we prompt GPT-4-Turbo (GPT-4-turbo-2024-04-09) with top-p=0.5 and temperature=0.7.

Figure: Figure 6: LLM prompt (Prompt 1) for extracting components from audio captions.
Refer to caption: x6.png

Figure: Figure 7: LLM prompt (Prompt 2) for generating new audio captions given elements from existing captions.
Refer to caption: x7.png

Figure: Figure 8: LLM prompt for generating random captions for Random Captions baselines in Table 1.
Refer to caption: x8.png

Figure: Figure 9: LLM prompt (Prompt 3) for rewriting captions of rejected audios.
Refer to caption: x9.png

### A.3 Examples

Table 6 presents examples of captions generated by the Synthio framework, along with their revised versions for captions that were initially rejected.

**Table 6: Examples of generated and revised captions from the Synthio methodology.**
| Dataset | Label | Generated Caption | Revised Caption |
| --- | --- | --- | --- |
| USD8k | children_playing | Children playing in a bustling city park with distant traffic noise | NA |
| \hdashlineUSD8k | children_playing | Children playing in a schoolyard during recess with teacher’s whistle | Children playing in a neighborhood alley with sound of distant construction |
| USD8k | street_music | Street music playing near a busy intersection filled with honking cars and pedestrians. | NA |
| \hdashline<br>USD8k | street_music | Street music from a bustling market as people chatter and vendors shout | Street music echoing through an alleyway during a lively street festival. |
| TUT | airport | airport with people talking and walking around in an empty hallway | NA |
| \hdashlineTUT | airport | In the airport, people are talking with the sound of a crowd of people in the background, as announcements play. | airport ambiance with people talking and children running around |
| TUT | bus | Bus passing by on a road while people are chatting at a nearby cafe. | NA |
| \hdashlineTUT | bus | bus passing by on a road as it continues to blow into the microphone | bus idling on a road with birds chirping nearby |
| NSynth | keyboard | keyboard accompaniment to a live band performance at a bustling cafe. | NA |
| \hdashlineNSynth | keyboard | a man typing on a keyboard at office | keyboard rhythms echoing in an empty auditorium during a rehearsal break |
| NSynth | organ | A serene church service with an organ playing a melody and soft brass are playing. | NA |
| \hdashlineNSynth | organ | An organ plays as guitars are playing together in the background. | An organ plays during a lively music festival with various instruments. |
| Medley | Violin | violin being played during a classical symphony orchestra performance | NA |
| \hdashlineMedley | Violin | violin performing a lively jig at a bustling street fair | Violin solo during a quiet candlelight dinner in a fancy restaurant. |
| Medley | Flute | flute playing in a tranquil forest during the early morning | NA |
| \hdashlineMedley | Flute | Flute performance in a bustling city park during a sunny afternoon. | Flute music echoing in an ancient stone cathedral. |
| AudioCaps | - | A dog barks repeatedly in the background while a car engine starts | - |
| \hdashlineAudioCaps | - | In the distance, a faint thunder rumble is audible, accompanied by the gentle rustling of leaves in the wind. | Soft rain falls on a metal roof, creating a rhythmic tapping sound. |

### A.4 Extra Results

#### A.4.1 Results on the Full Training Splits

Table 7 presents the performance comparison of Synthio on the full original dataset splits (where the entire training set is used without any downsampling). While Synthio outperforms all baselines, traditional augmentation methods prove to be much more competitive in this scenario. This contrasts with the results in Table 1 where traditional augmentations showed minimal improvements in performance.

**Table 7: Comparison of Synthio and other baselines on the full original dataset splits (using all samples from the original training set as $\mathcal{D}_{\text{small}}$).**
| Method | USD8K | GTZAN | Medley | VS | MSDB |
| --- | --- | --- | --- | --- | --- |
| Gold-only | 88.23 | 82.00 | 80.99 | 92.73 | 73.9 |
| Random Noise | 86.17 | 82.35 | 79.72 | 92.94 | 74.55 |
| Pitch Shift | 87.58 | 83.02 | 79.63 | 92.17 | 74.6 |
| Spec. Aug. | 87.92 | 82.50 | 79.14 | 92.42 | 74.5 |
| Audiomentations | 88.01 | 82.75 | 81.26 | 92.47 | 75.05 |
| Retrieval | 78.27 | 69.25 | 73.24 | 80.43 | 69.95 |
| Vanilla Syn. Aug. | 89.57 | 82.85 | 81.79 | 93.15 | 75.85 |
| Synthio (ours) | 89.57 | 82.85 | 81.79 | 93.01 | 74.24 |

**Table 8: CLAP score between generated audios and the label.**
| n | Method | USD8K | NSynth |
| --- | --- | --- | --- |
| 100 | Real | 12.67 | 14.46 |
| Vanilla Syn. Aug. | 14.34 | 17.54 |  |
| Synthio | 31.26 | 27.32 |  |
| w/ Template Captions | 29.31 | 26.62 |  |
| w/ ERM | 24.15 | 21.54 |  |
| 200 | Real | 10.13 | 9.4 |
| Vanilla Syn. Aug. | 12.55 | 12.91 |  |
| Synthio | 21.87 | 16.16 |  |
| w/ Template Captions | 20.31 | 15.82 |  |
| w/ ERM | 17.14 | 13.04 |  |

**Table 9: Comparison of our trained Stable Diffusion model on AudioCaps test set**
| Model | FAD_PANN ($\downarrow$) | FAD_VGG ($\downarrow$) | IS_PANN ($\uparrow$) | CLAP_LAION ($\uparrow$) |
| --- | --- | --- | --- | --- |
| AudioLDM2-large | 32.50 | 1.89 | 8.55 | 0.45 |
| Tango-Full0FT-AC | 18.47 | 2.19 | 8.80 | 0.57 |
| Tango 2 | 17.19 | 2.54 | 11.04 | 0.52 |
| Make-an-Audio 2 | 11.75 | 1.80 | - | 0.60 |
| Stable Audio VECaps (ours) | 15.12 | 2.21 | 15.07 | 0.57 |
| Stable Audio VECaps + AudioCaps-FT (ours) | 14.93 | 2.19 | 15.42 | 0.56 |

#### A.4.2 Audio generation results for our trained Stable Diffusion

Table 9 presents a comparison of audio generation results across several evaluation metrics. We evaluate our trained Stable Diffusion model (used in our experiments, including a version further fine-tuned on AudioCaps) against other available models and baselines from the literature. Notably, our model performs competitively with other fully open-source models across most metrics.

#### A.4.3 FAD scores for Generated Augmentations

To offer an alternative perspective on the distributional consistency between the generated augmentations and the ground-truth small-scale dataset, we compare the Fréchet Audio Distance (FAD) scores (Kilgour et al., ). For this experiment, we use Synthio with Template Captions. Table 10 presents a comparison of FAD scores between Synthio and other baselines. Synthio achieves the highest FAD score, indicating that it produces the most consistent audio augmentations.

**Table 10: Comparison of FAD score of Vaniall Syn. Aug. and Stable Audio VECaps (ours).**
| n | Dataset | Model | FAD_VGG ($\downarrow$) |
| --- | --- | --- | --- |
| 100 | NSynth | Vanilla Syn. Aug. | 1.83 |
| Stable Stable Audio VECaps (ours) | 1.42 |  |  |
| 200 | TUT | Vanilla Syn. Aug. | 1.71 |
| Stable Audio VECaps (ours) | 1.45 |  |  |

### A.5 Dataset Details

NSynth Instruments: NSynth is a large-scale dataset consisting of musical notes played by a variety of instruments. It includes a rich set of acoustic features from instruments like guitars, flutes, and more, providing diverse sound textures for classification tasks.

TUT Urban: The TUT Urban dataset captures everyday sounds from urban environments, including noises like traffic, human activities, and construction. It is commonly used for acoustic scene classification and environmental sound recognition.

ESC-50: ESC-50 is a well-known dataset for environmental sound classification, containing 50 categories of everyday sounds such as animal noises, natural elements, and human activities, making it suitable for multi-class classification challenges.

UrbanSound8K (USD8K): USD8K is a curated collection of urban sounds divided into ten classes, including sirens, street music, and car horns. It is used widely for evaluating models on sound event detection in real-world scenarios.

GTZAN: GTZAN is a music genre classification dataset that includes ten music genres such as pop, rock, and jazz. It is a standard benchmark for evaluating music classification models, although it has known data quality issues.

Medley-solos-DB: This dataset consists of solo recordings of different musical instruments, making it valuable for studying isolated instrument sounds and training models for music instrument recognition.

MUSDB18: MUSDB18 is used primarily for music source separation tasks. It contains full-track recordings of different music styles, providing a mix of vocals, drums, bass, and other instruments, useful for multi-class classification.

DCASE Task 4: Part of the DCASE challenge, this dataset focuses on domestic sound scene and event classification. It includes various audio clips recorded in home environments, often used for anomaly detection and sound event classification.

Vocal Sounds (VS): This dataset includes various vocal sounds such as singing, speech, and vocal effects, providing rich data for studying voice classification and enhancing models for vocal audio recognition tasks.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Although the paper strives to present all concepts (with background) in a single, self-contained format, it would benefit from careful notation selection. Reusing the same notation for different concepts could be confusing. For example, using the symbol "beta" in different contexts (e.g., in diffusion, RLHF, and DPO) might lead to misunderstandings.
- Minor correction:  In Table 10, please correct "Stable Stable" to "Stable."
- A major concern with the proposed method is balancing human effort with the complexity of the introduced steps. While the pipeline can operate fully automatically, the large number of hyper-parameters may introduce significant uncertainty in practical applications. The reviewer appreciates the current approach, but simplifying the pipeline to reduce complexity while retaining the core concepts could be a valuable direction for future work.
- Additional experiments:
    - While Synthio shows promising performance, it would be interesting to investigate whether it has complementary or joint effects with other augmentation methods, such as random noise addition, pitch shifting, or other common techniques.
    - Further experiments could examine how changes in T2A models influence the effectiveness of the augmentation process. Specifically, it would be valuable to understand how the quality of T2A models affects the overall performance of the augmentation technique.
soundness: 4
strengths: - The paper effectively identifies two key challenges in synthetic data-based augmentation, providing clear and well-reasoned explanations.
- The proposed techniques directly address these challenges and are demonstrated to be both practical and beneficial according to the experimental results.
- The experimental setup is comprehensive, covering various types of audio events, which strengthens the validity of the findings.
summary: This paper explores the use of synthetic data to enhance small-scale audio classification tasks, addressing two primary challenges in synthetic augmentation: acoustic consistency and compositional diversity. To improve consistency, a preference alignment approach is introduced to better align the domain of synthetic data with the source dataset. For diversity, the authors propose language-guided audio imagination, utilizing large language models to iteratively generate captions that blend existing and novel acoustic components, feeding into a text-to-audio generation model.
weaknesses: The work is well-presented and thorough, with no significant weaknesses noted. However, the reviewer has a few suggestions and minor comments, outlined in the Questions section below.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: As mentioned before, the proposed approach is interesting and novel (in the context of audio generation). However, I have several concerns about the evaluation process and presented results: 
1. The authors compare the proposed approach to several baseline methods including Gold-only (meaning no augmentations at all), and more traditional data augmentation techniques. Although the proposed method reaches significantly better results (especially under a small number of samples in D_test), it uses much more resources, meaning, the T2A model was trained on ~1.5M of audio samples. Hence, models trained with Synthio were introduced to more and diverse set of samples than just D_test, which were greatly influenced by the pre-training data of the T2A model. 
2. How many samples did you generate for each of the N samples on D_test in Table 1. This is another potential issue as the models trained with Synthio were also trained on more data (nxN) in comparison to standard augmentation techniques which were trained on N samples only. 
3. As I mentioned before, it seems the gap in performance is getting smaller very fast as we increase the size of D_test (especially compared to a model with no augmentations). This means that the proposed approach is beneficial for model fine-tuning under very low resource regimes (less than 500 samples). I advise the authors to put more emphasis on that in the paper and discussions. 
4. The authors additionally presented results for audio captioning. What text labels were used to both train and evaluate model performance considering both the proposed method and baselines? I'm asking as in the proposed approach text labels are much reacher, hence I'm not sure the comparison is fair. 

I'm willing to increase my score in case I missed something.
soundness: 3
strengths: 1. Leveraging DPO to better align audio generation with real acoustics is a novel idea.
2. The whole data generation pipeline is an interesting idea, including the MixCap approach.
3. Under fine-tuning at small-scale regimes the proposed approach achieves significant improvement over the baseline methods.
summary: This paper presents Synthio, a synthetic data generation approach for improving audio classification models. Synthio leverages a pre-trained text-to-audio generation model, it first aligns the generations with acoustic preferences (obtained from a small dataset), then it leverages an LLM to synthetically generate captions for the generated audio. To further improve data quality and model performance, Synthio refines the dataset via rejection sampling. 
The authors evaluated their approach using different benchmarks while generating trainings sets of different sizes. The authors empirically demonstrated the proposed approach is superior to standard data augmentation techniques and to vanilla data generation.
weaknesses: 1. Unfair comparisons to the evaluated baselines (more details below).
2. When increasing the number of generated samples the gap in performance between the proposed approach and no aug. at all is closing, very fast.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How would aligning the T2A model with the small dataset create the diversity of audio as exists in the real world as claimed in the Abstract as DPO would make it close to the samples chosen.  

2. Why was random mix augmentation not carried out in the Table 1 ? 

3. Was the performance boost mentioned in the abstract 0.1-39% relative or absolute. I could not find that in Table 1. Why were MAP metrics not report as is a standard practice for FSD-50K

4. Why are the characteristics like spectral flux and spectral flatness done globally as opposed to local behavior of NSynth and Urban Sound dataset. 

5. Are there any papers that have the experimental setup similar to the Table 1. Please cite a paper that fine-tunes such a small corpus in a similar setup. 

6. Why was the whole background on few-shot classification and zero shot audio classification not included. This is very similar to zero shot and few shot classification. The baselines should be included as well. Why should someone generated synthetic examples and not do few shot audio classification. What are the drawbacks of those methods. Why are the baselines not included with any of the previous works on synthetic dataset. n=50  is a few-shot setting, and it would be unfair not to include any literature or mention it anywhere in the paper. 

7. Why have the authors gone the T2A route. Taking an audio and writing a caption is an extremely lossy compression which will remove most of the information needed about the audio. what were the advantages of a T2A model that will be helpful for this problem. 

8. Why was the scaling factor limited to only 1x-5x scaling. What happens if I do 20x. It seems it might start overfitting ? The results in 6.4 are strange that it works with 4x scaling than 5x scaling. Either the models are not tuned properly or they are overfitting on smaller corpus.
soundness: 1
strengths: == The paper has extensive experimentation that is far more than what was needed. It would have been much stronger if the authors focussed on a subset of these results with a sound, solid method. 

== The paper appears to be well-structured and written despite its flaws. 

== The data and code will be open-sourced, reproducible and easy to follow, which is a good point in favour of the paper. 

== A well-explained background on DPO, diffusion model, and RLHF. It should have included background for audio classification, zero-shot learning, etc. 

== The problem is interesting; exploring synthetic data to improve audio classification and pushing the performance is worth exploring.
summary: The paper proposes a straightforward method: Given small scale audio classification task - use an existing audio generation model in this paper, a text-to-audio generator, to generate a lot of audio files similar to the smaller dataset and augment it with a factor of 1x to 5x samples to fine-tune an existing audio classification. The authors show the method outperforms their proposed baseline by (relative) 0.1-39%. The authors say that their contributions are,

 -- it is a novel approach for augmenting the small-scale dataset
--  their ability to generate novel, diverse, consistent data and its performance on tasks such as audio captioning.

In brief, this is a paper on audio classification on small-scale audio. The methodology is once the dataset is created == straightforwardly fine-tune the model.
weaknesses: The paper is being submitted to the topmost ML venue; hence, my comments are made accordingly to adhere to the highest quality of the paper. At the outset, the paper appears strong, but there are flaws that need to be addressed / claims that are not correct. Please do not carry out extra experiments to justify any other points and generate a new version of the paper. Please answer these points in 1-2 lines of text at maximum. 

1. The authors should drop a novel approach for audio classification as several papers exist that should be referenced. i) Open-Set Tagging Dataset (OST) Mark Cartwright, ii) BLAT Bootstrapping language-audio pertaining based audio set .... iii) Can synthetic audio from generative foundational models assist audio recognition ....iv) Synthetic training set generation using T2A models for environmental sound classification. There exist many more papers for audio generation tasks, which I do not include here, but the authors should have included them. These should be adequately referenced in the related work. If not, it gives a very false impression that the authors are the first ones to do it, including in the abstract.

2. The whole point of audio classification is robustness to unseen conditions and good performance for out-of-distribution data. DPO aligns generated data with the small dataset but fails to capture the variations that are needed in real life, as claimed by the author. The claim made by the authors, "T2A models trained on internet-scale data often generate audio that diverges from the characteristics of
small-scale datasets, resulting in distribution shifts.
These mismatches can include variations in spectral
(e.g., frequency content), perceptual (e.g., pitch, loud-
ness), harmonic, or other acoustic characteristics 2" is correct, but for audio classification, we do want to incorporate all of these variations. 


This is the point of training on a large corpus of datasets, such as an audio set, to include all the variations different from the training corpus. If I take very few examples and align a T2A model with DPO to the smaller dataset, it will generate samples in that small dataset distribution, as mentioned in all of the plots. This is shown precisely in Figure 3, and we do not want that. I would rather want a lot of diversity in pitch and spectral flux to make audio classification good in unseen cases rather than have the training data similar to my small corpus. 

There is a whole subfield on "out of distribution classification" = how to make models perform well when the classification dataset is not of the same distribution as that of the original training dataset. Whereas here, the authors are trying to do just the opposite and make the training corpus as close to the actual data using DPO, which is counter-intuitive to the field of how to build robust classifiers and get good classification accuracy out-of-distribution examples and data. 

3. Audiomentations has a list of 40 possible audio augmentations. The setup with n being 50-500 for ten categories would not be able to generate enough diverse augmentations. How do the authors report the numbers for it and carry it out? With N being at max 5, how would one incorporate it? Are all of the combinations not needed/not needed? how were they chosen? 

4. From Table 1, it does not appear that this paper was an academic paper. The comparisons do not include any benchmark reported by any other paper. The baselines are poorly reported and are not compared to any other paper. Even for data augmentation literature, there are papers that compare various techniques. 

5. Section 6.4 mentions that the scaling 4x is better than 5x. This appears strange as to why this would happen. The audio captioning section is definitely not needed for this paper and should ideally be dropped. 

6. The issue with alignment to the dataset is also the fact that there are categories that often overlap in FSD-50K, ESC-50, etc. What happens in that setup? In that case, the model is forced to generate one kind of audio sample/or prefer one over the other even though both have the same label. 

7. Accuracy mentioned in 6.5 is 0%. It is mentioned that the categories had the lowest frequencies in the dataset. This makes the claims in the paper shaky as it appears that the training was not normalized according to the category probabilities or sub-sampled/oversampled to make the training balanced.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The result in Section 6.5 and Figure 5 is not convincing enough:
1. Line 516 “ We selected categories with the lowest frequency in the datasets”: 
The categories in the ESC-50 dataset have the same number of samples; how do you decide which category has a lower frequency?
2. Figure 5 shows only four classes to demonstrate the improvement of long-tailed categories. This is not convincing enough as those selected classes may represent only partial information. 

Table 2: Can you please explain what is “compositional diversity” and how it correlates with the CLAP score?
Line 214: DSmall -> D_{small}
Line 226 “no scalable methods”: Is it possible to estimate the loudness with the signal processing method?
Line 242 “fostering more diverse augmentation”: Any evidence for this claim?
Please double check the equation 8, 9, and 10 as the “x_0 : T” notation looks a bit confusing to me.
Line 318 “The CLAP model is pre-trained on Da-c and fine-tuned on Dsmall to adapt to the target datase”: Do you mean finetuning CLAP model on each of the dataset you used (ESC50, GTZAN, etc)? If so, how do you perform finetuing on such a small dataset, such as when n=50? Doesn’t that can cause overfiting?
Line 413: remove duplicated “the”
What dataset is utilized for the experiment in Table 5? Please specify. 
Line 503: What are the possible reasons why SpecAug does not scale well?

Please consider rewording the following sentence that is a bit confusing to read:
- “Abusing notation, we will also x0 as random variables for language”
- “We elaborate on these next”
soundness: 3
strengths: 1. The performance improvement when the data is in small scale (e.g., 50, 100, 200, 500) is pretty significant. 
2. The experimental setup is pretty complete and extensive.
summary: This paper proposed a pipeline to augment small-scale audio classification datasets with synthetic data and shows promising improvements.
weaknesses: 1. The performance improvement with the proposed synthetic pipeline is limited. Despite the proposed augmentation showing quite significant improvement on the curated small dataset with a few hundred samples, it only shows marginal improvement on the full-scale dataset (Table 7). 
2. The proposed method is not practical. As we usually have data that is much larger than a few hundred samples (in this case, the proposed method does not work well), the traditionality of the proposed method is questionable. Also the proposed pipeline is much complex comparing with traitional augmentation method such as audiomentations, increasing the cost for model training. 
3. Heavy reliance on pretrained CLAP model. This reliance may bias the synthetic data as CLAP model. 
4. It is unclear what CLAP threshold is used in filtering and self-reflection. This is usually a tricky operation as the optimal threshold can vary with different types of sound. More explanations are welcomed.

Meta-review: > This paper presents Synthio, a synthetic data generation approach for improving audio classification models. Synthio leverages a pre-trained text-to-audio generation model, it first aligns the generations with acoustic preferences (obtained from a small dataset), then it leverages an LLM to synthetically generate captions for the generated audio. To further improve data quality and model performance, Synthio refines the dataset via rejection sampling. The authors evaluated their approach using different benchmarks while generating trainings sets of different sizes. The authors empirically demonstrated the proposed approach is superior to standard data augmentation techniques and to vanilla data generation.

The results of synthetic data generation for small datasets are promising and useful. 

The reviewers mostly agree to accept this paper, in particular with 2 strong accepts, I am inclined to accept the paper.

Decision: Accept (Poster)