Contrastive Learning from Synthetic Audio Doppelgängers

## Abstract

Abstract Learning robust audio representations currently demands extensive datasets of real-world sound recordings. By applying artificial transformations to these recordings, models can learn to recognize similarities despite subtle variations through techniques like contrastive learning. However, these transformations are only approximations of the true diversity found in real-world sounds, which are generated by complex interactions of physical processes, from vocal cord vibrations to the resonance of musical instruments. We propose a solution to both the data scale and transformation limitations, leveraging synthetic audio. By randomly perturbing the parameters of a sound synthesizer, we generate audio doppelgängers —synthetic positive pairs with causally manipulated variations in timbre, pitch, and temporal envelopes. These variations, difficult to achieve through transformations of existing audio, provide a rich source of contrastive information. Despite the shift to randomly generated synthetic data, our method produces strong representations, competitive with real data on standard audio classification benchmarks. Notably, our approach is lightweight, requires no data storage, and has only a single hyperparameter, which we extensively analyze. We offer this method 1 1 1 doppelgangers.media.mit.edu as a complement to existing strategies for contrastive learning in audio, using synthesized sounds to reduce the data burden on practitioners.

## 1 Introduction

> “Noises have generally been thought of as indistinct, but that is not true.” — Pierre Schaeffer, 1986

The success of modern machine learning algorithms for tasks like audio understanding often hinges on both the quality and quantity of available data. Self-supervised learning methods, like contrastive learning, have even been able to leverage unlabeled data, enabling more human-like learning from patterns without needing explicit supervision. However, human perceptual processing is remarkably robust beyond this: for example, the human auditory system can easily recognize sounds across a wide range of variations, such as changes in pitch, timbre, or background noise. Moreover, humans can quickly learn to recognize novel sounds that they encounter in their environment. Replicating this ability to learn from a diverse array of sounds—or "noises," as we might call them—could significantly enhance the efficiency, scalability, and adaptability of machine learning models.

Contrastive learning, which operates by recognizing similarities in the data among negative distractors, often relies on augmentations: transformations of input data that preserve content semantics. This method has been influential in audio representation learning, with specific implementations ranging from spectral masking to temporal jitter to cropping and other methods [, , , , , , ]. Data augmentations, though demonstrably useful, operate at the level of the observed data, not the underlying data-generating process as would be observed in real-world variation. They statistically alter data without directly manipulating the causal mechanisms that produced it, resulting in high correlation between augmented samples, as well as limited control and interpretability.

In our work, we propose using a synthesizer to overcome this barrier, in addition to providing the scalability required for modern pretraining regimes through virtually unlimited data synthesis. A synthesizer can be understood as a system where parameters (relating to psychophysical attributes like pitch, timbre, and loudness) causally influence the generated sound. Modifying these parameters allows us to intervene in the data-generating process in a controllable way to generate positive pairs that vary in terms of their underlying synthesis parameters.

We formulate an approach in which we randomly synthesize sounds, and then slightly perturb their parameters to generate positive pairs. We call these audio doppelgängers (examples in Contrastive Learning from Synthetic Audio Doppelgängers); they share a resemblance but are in fact distinct enough to learn from the variation between them. In a way, this approach uses an artificial data source effectively consisting of random synthetic noises but more “natural” augmentations akin to variation in similar sounds; as Pierre Schaeffer put it, noises are not indistinct. Through a comprehensive set of experiments, we show that models trained this way can yield strong performance on a wide range of downstream tasks, competitive with real audio datasets.

Overall, this work contributes:

- 1.
An approach to synthesizing paired audio examples with a continuously controllable degree of dissimilarity, specified by a simple and interpretable hyperparameter $\delta$.
- 2.
The first study, to our knowledge, of synthetic data methods for audio representation learning.
- 3.
Comprehensive experiments in which we train and compare over 20 model variants across 8 downstream tasks to provide evidence that training with our approach can yield strong results on a wide range of audio processing tasks.
- 4.
An analysis of how these synthetic datasets differ from realistic audio datasets in terms of their auditory features, and how this might contribute to learning effective representations.

## 2 Related Work

### 2.1 Learning from Synthetic Data

Synthetic data, artificially generated information rather than collected from real-world sources, has emerged as a valuable tool for learning across various domains [, , , ]. By addressing data scarcity, privacy concerns [, ], or removing biases [, ], synthetic data offers a promising avenue to complement real-world data and further drive progress in machine learning research.

Audio presents unique challenges due to the complexity of waveforms and temporal dependencies. Synthetic data has found applications in subareas like speech recognition [, , , , ] leveraging text-to-speech systems for detecting unspoken punctuation [], recognizing low-resource languages [], increasing acoustic diversity [] or detecting out-of-vocabulary words []. However, non-speech audio domains can be highly diverse, requiring more complex approaches to data synthesis. In this domain, synthetic data has been used for specific tasks like timbre-text alignment [] and vocoding []. The partially synthetic NSynth [] dataset has also been used for pitch estimation and instrument classification. In our work, we tackle the general audio domain, proposing a synthetic data approach that can produce diverse sounds for general-purpose audio representation learning.

In computer vision, synthetic data is more popular and has been employed in different tasks to improve performance [, , , , , , , ]. While initially focused on using graphics engines to generate photorealistic scenes, recent work has investigated sampling synthetic data from deep generative models [, , , , , , , , , , , , ]. However, these models aim to produce realistic images and still depend on real image datasets for training or synthesis. Thus, recent work has pushed away from realism, generating synthetic data such as fractals [], or through other procedural noise models [, ] to use as training data for visual representation learners. In our work, we also abandon realism and leverage randomly generated synthetic sounds to learn audio representations for downstream tasks.

### 2.2 Contrastive Learning

A common strategy for learning from unlabeled data is contrastive learning. In this technique, we seek representations that are invariant to minor differences, i.e. they encode a space in which similar objects are closer together, and dissimilar objects are further. A classic strategy for this is to use data augmentations, transformations which noticeably alter a datapoint (for example, randomly cropping an image) without changing its essential content (e.g. what the image is of, such as a cat). These transformed versions then become a positive pair, while other examples (e.g. an image of a dog) become negatives. In audio, contrastive learning has been used extensively to produce high-quality representations for downstream tasks [, , , , ]. However, contrastive learning typically relies on augmentations with limited variation, by effecting already-existing material or sampling with offsets. Instead, our approach produces synthetic positive pairs that deviate through randomly perturbed parameters which causally influence acoustic properties of the output, such as the pitch of oscillators, or the temporal envelope of signals in a mixture. This allows us to mimic more naturalistic variation in auditory features without (or in addition to) applying post-hoc effects.

### 2.3 Sound Synthesis

The toolkit of sound synthesis has evolved to include a variety of hardware and software [, , ]. Synthesizers, abstractions of sound synthesis and processing methods often designed to act as musical instruments, are key to this: they expose control parameters that let sound designers guide them to produce desirable sounds for music, film, and many other applications. Accelerated synthesizers [, ] have recently allowed much faster-than-realtime sound generation, offering the ability to iteratively tweak parameters to reconstruct sounds [, ] and even match textual descriptions []. Such approaches highlight the practical utility of synthesizers: lightweight architectures controlled by a limited number of interpretable parameters are capable of producing a diverse array of sounds, often corresponding to well-known categories and concepts (e.g. the sound of waves can often be modeled with time-varying filtered noise). In our work, we leverage SynthAX [], to rapidly produce diverse training data with controllable similarity between examples.

## 3 Methods

### 3.1 Data Generation

Our data generation pipeline uses virtual modular synthesizers implemented by SynthAX [] in JAX. By default, we use the Voice synthesizer architecture [], which has been shown to generate perceptually diverse sounds. This synthesizer has 78 parameters: a monophonic keyboard; two low-frequency oscillators and six ADSR temporal envelopes to generate control signals; sine and square-saw oscillators and a noise generator; and additional amplitude- and modulation-related components. In our experiments, we investigate two further architectures: VoiceFM has 130 parameters and includes a frequency modulation (FM) operator, and ParametricSynth has 2 sine and 2 square-saw oscillators, 1 sine FM and 1 square-saw FM operator, 340 in total. Varying the architecture allows us to investigate whether architectural complexity could affect the quality of representations learned. We generate 1-second sounds by default, for compatibility with most encoders (e.g. VGGish []). However, this practice can be extended to longer sounds.

##### Synthesis perturbation factor ( δ 𝛿 \delta italic_δ )

A key contribution of our work is synthesizing paired positive samples that sound alike, but are dissimilar due to their synthesis parameters and not only post-hoc effects (e.g. augmentations). This draws on the canonical definition from contrastive learning of positives that are sampled from the same latent class [].

For a single positive pair, we first sample a parameter vector uniformly randomly $\theta\in[0,1]^{m_{S}}\sim\mathcal{U}(0,1)$ from the normalized synthesis parameter space, where $m_{S}$ is the number of control parameters in the given synthesizer. Then, we independently sample two isotropic Gaussian noise vectors $\mathbf{z_{1}},\mathbf{z_{2}}\sim\mathcal{N}(0,\mathbf{I}_{m_{S}\times m_{S}})$. We define a parameter $\delta$ that scales this noise, and then produce two perturbed parameter vectors $\theta_{i}=\theta+\delta\mathbf{z_{i}}\ \forall_{i}\in\{1,2\}$. From these, we clip values back into $[0,1]$ to synthesize two corresponding sounds which serve as positives in the contrastive learning setup.

In principle, $\delta$ controls the distance between the positive pairs and therefore the hardness of the contrastive learning task. Practically, we expect there to be a sweet-spot for $\delta$, considering prior work on mutual information and redundancy in contrastive learning problems [, ] as with very high $\delta$, the parameter vectors may become dominated by noise, resulting in difficulty effectively aligning their representations. Given this, we extensively study the effect of $\delta$ on downstream results.

### 3.2 Real Data

To compare to real audio data, we use sounds from VGGSound [], a well-known dataset taken from YouTube videos (we only use audio). We use a random sample of 100,000 10-second files and select a random 1-second segment from each file at each iteration to augment. This allows us to fairly compare to our synthetic sounds by keeping duration constant, while still sampling from a variety of real sounds by randomizing the 1-second segments. Though VGGSound has labels included, we do not use them in training these models to keep the self-supervised constraint.

### 3.3 Preprocessing, Data Augmentations, and Audio Encoder

In our experiments, we use VGGish frontend representations []. We resample audio to 16kHz and obtain mel spectrograms with 64 mel bands and 96 time steps. We use a chain of effects as augmentations (implemented in torch-audiomentations(^2^22[https://github.com/asteroid-team/torch-audiomentations](https://github.com/asteroid-team/torch-audiomentations))): a high-pass filter (cutoff frequency range 20–800Hz), a low-pass filter (1.2–8kHz), pitch shift (-2 to 2 semitones), time shift (-25% to 25%, rollover enabled), and finally reverberation for which we sample randomly from a set of impulse responses. All augmentations are applied with probability 0.5. We found that this yielded far stronger results than SpecAugment [], and so we use this as a comparison point in all our experiments. More details on the augmentation are given in Appendix B. We also test temporal jitter, wherein different 1-second segments are sampled from within the same source clip and treated as positives [, ]. Our audio encoder is a ResNet18 [], where we replace the initial layer with a 1-channel convolution to account for the effectively 1-channel spectrogram.

### 3.4 Contrastive Learning

We train for 200 epochs, generating (or sampling) 100,000 sounds per epoch, with a 90%-10% train-validation split. We use a batch size of 768 per GPU with two V100s. The training uses the alignment and uniformity objectives [] used in prior work on learning with synthetic data []. We adopt the default parameters for these: $\textrm{unif}_{t}=2$, $\textrm{align}_{\alpha}=2$, and equal weights $\lambda_{1}=\lambda_{2}=1$ for both terms. Following this work, we use stochastic gradient descent for optimization, with a maximum learning rate of 0.72 (calculated as 0.12 $\times\dfrac{\textrm{total batch size}}{256}$) and weight decay $10^{-6}$. The learning rate follows a multi-step schedule with $\gamma=0.1$, and milestones at 77.5%, 85%, and 92.5% of the total learning epochs. Detailed steps are provided in Algorithm 1. Training with our synthetic data takes approx. 1-2 hours, as the data is generated on the fly in batches, whereas using on-disk datasets with effect chain augmentations can extend training time up to 6-8+ hours.

Figure: Algorithm 1 Our contrastive learning procedure with audio doppelgängers. In the training loop, we drop the batch index $i$ for simplicity.

### 3.5 Evaluation Tasks

To obtain a broad picture of the quality of our learned representations, we conduct experiments on a range of audio classification tasks selected from the HEAR [] and ARCH [] benchmarks. We are primarily interested in tasks that focus on real-world sounds (for example, vs. speech or music audio). We use a modified version of the HEAR evaluation setup to conduct linear probing experiments (instead of using an MLP probe) for a more direct signal of representation quality. We use the Adam optimizer to train this linear probe for the benchmark-specified number of epochs, with the default learning rate of 0.001 and a batch size of 32.

We evaluate on tasks covering a wide range of capabilities including sound classification tasks like ESC-50 [], FSD-50k [], and UrbanSound8K [], vocal affect tasks with and without speech like VIVAE [] and CREMA-D [], musical pitch recognition via NSynth Pitch (5h) [], vocal sound imitation recognition using Vocal Imitations [], and LibriCount [] for a "cocktail party" style speaker count estimation task.

## 4 Results

### 4.1 Benchmark Results

In Table 1, we show results across 8 tasks. The top section features external baselines from the HEAR [] leaderboard and ARCH [] benchmark results, first the strongest overall and then only self-supervised. It also includes results from MS-CLAP [] linear probing experiments, GURA [] (strongest overall model on HEAR), and finally the original ResNet18 trained on VGGSound (supervised) []. Note that HEAR leaderboard results may use MLP probes, whereas ours are linear. We add additional internal baselines, including random weights, synthetic data trained without $\delta$ but with augmentations, and variants of ResNet18 we trained on VGGSound (with augmentations, and alternately with temporal jitter). Finally, we include a selection of our results; the best overall score we achieve using our synthetic approach (first row), followed by the best-performing model trained on data from each of the synthesizer architectures (including Voice with augmentations). In Section A.3, we provide a full set of results from all model variants: all synthetic datasets for all values of $\delta$, and further baselines less performant than those we present here.

Overall, our best scores uniformly outperform training on VGGSound with augmentations, and outperform training with temporal jitter (the strongest internal baseline) in 6/8 cases. In some cases, these results are also competitive with strong baselines, such as beating the supervised ResNet18 result on 3/8 tasks, CLAP on 2/5, and GURA on 1/6. Additionally, adding further augmentations to our audio doppelgänger-based training does not seem to hold significant benefits, despite being highly beneficial when training with synthetic sounds with no $\delta$, suggesting the $\delta$-based perturbations are already sufficiently strong. All this is accomplished without these models seeing any real sounds during pretraining. Finally, Voice with $\delta=0.25$ is the strongest synthetic-trained model overall, being the top performer on 5/8 tasks, but we note that there is some inter-task variability in the best synthesizer and delta.

**Table 1: Evaluation results on a suite of tasks including (from left to right) ESC-50 [], UrbanSound8k [], VIVAE [], NSynth Pitch 5h [], CREMA-D [], FSD50k [], Vocal Imitation [], and LibriCount []. For internal baselines, we only bold tasks where the baseline beats the best synthetically trained result. Results for all synthetic variants are in Section A.3.**
| Data/Model | ESC | US8K | VIV | NSyn | C-D | FSD | VI | LCount |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| External Baselines |  |  |  |  |  |  |  |  |
| HEAR/ARCH Top [, ] | 96.65 | 79.09 | 44.28 | 87.80 | 75.21 | 65.48 | 22.69 | 78.53 |
| HEAR/ARCH SSL | 80.50 | 79.09 | 44.28 | 52.40 | 75.21 | 50.88 | 18.48 | 78.53 |
| MS-CLAP Linear [] | 89.95 | 82.29 | – | – | 23.15 | 50.24 | – | 54.51 |
| GURA (HEAR) [] | 74.35 | – | – | 38.20 | 75.21 | 41.32 | 18.48 | 68.34 |
| VGGSound Sup. [] | 87.45 | 77.57 | 39.38 | 43.80 | 54.36 | 43.76 | 14.06 | 56.10 |
| Internal Baselines |  |  |  |  |  |  |  |  |
| Random Init. | 22.45 | 55.03 | 33.81 | 36.20 | 38.91 | 9.03 | 2.43 | 44.91 |
| Voice (Ours, No-$\delta$, Aug.) | 48.65 | 59.46 | 36.31 | 32.80 | 46.32 | 16.88 | 7.12 | 47.64 |
| VGGSound SSL (Aug.) | 48.85 | 61.91 | 32.67 | 39.60 | 47.86 | 19.63 | 6.03 | 53.46 |
| VGGSound SSL (Jitter) | 52.95 | 63.82 | 38.12 | 14.20 | 50.03 | 24.02 | 3.43 | 69.77 |
| Audio Doppelgängers (Ours) |  |  |  |  |  |  |  |  |
| Best Synthetic | 58.90 | 66.71 | 39.45 | 44.40 | 48.43 | 24.12 | 9.15 | 58.60 |
| Voice ($\delta=0.25$) | 58.90 | 66.71 | 39.45 | 32.20 | 48.24 | 24.12 | 9.15 | 52.95 |
| Voice ($\delta=0.25$, Aug.) | 58.75 | 65.01 | 34.81 | 44.40 | 46.17 | 21.76 | 8.54 | 50.70 |
| VoiceFM ($\delta=0.25$) | 57.20 | 65.11 | 38.48 | 35.20 | 48.43 | 22.15 | 6.96 | 54.00 |
| Parametric ($\delta=0.25$) | 50.55 | 62.83 | 37.91 | 37.60 | 46.77 | 18.68 | 5.70 | 54.72 |

### 4.2 Characterizing the Data Distribution

Here, we focus on understanding the distribution of synthetic sounds and how they differ from natural sound properties. We primarily use our alternate training set, VGGSound [], for these measures. Unless specified otherwise, we use a randomly sampled (for VGGSound) or generated (for synthetic) set of 1000 sounds for each given dataset used for these characterizations. Our goal is to help understand what properties of the synthetic data make it useful for representation learning, given its strong performance.

#### 4.2.1 Embedding Similarity

First, we look at the distribution of synthetic sound pairs and establish that $\delta$ meaningfully controls (a proxy for) perceptual or semantic dissimilarity. We operationalize this using LAION-CLAP embeddings [], since they are trained on a large variety of sounds with semantic descriptions associated. Figure 1A shows how the average cosine similarity decreases monotonically with increasing $\delta$ for all 3 synthesizers. Figure 1B provides an alternate view of the space of positive pairs, compared to negatives (other samples). Here, we plot the first two principal components of the CLAP embeddings along with the path length for each positive pair of synthesized samples from a Voice synthesizer. As $\delta$ increases, the path lengths increase and overlap more resulting in less clear separation of positive pairs from negatives. We view this as a signal that we can effectively control the hardness of the contrastive task using $\delta$, the perturbation factor.

Figure: Figure 1: (A: Top) Average CLAP [] embedding cosine similarity between positive pairs for different architectures and different values of $\delta$. (B: Bottom) PCA of CLAP embeddings for sounds generated with the Voice architecture, with line segments showing distances between paired examples. Red and blue points are paired positive instances. Across both plots, as $\delta$ increases, the positive pairs systematically become more perceptually dissimilar (via the CLAP embedding proxy).
Refer to caption: extracted/5654860/figs/pairs_similaritydistance.png

#### 4.2.2 Similarities and Differences from Real Data

Next, we compare the synthetic data distribution to VGGSound [] data. Figure 2A compares a selection of features’ distributions between several dataset variants. For synthetic datasets, we have Voice, VoiceFM, and ParametricSynth variants. For real datasets, we have VGGSound. We first compare to randomly sampled 1-second chunks. Here, the synthetic sounds match several feature distributions well, such as Inharmonicity [], Odd-to-Even Harmonic Ratio [], Pitch Salience [], and, to a lesser extent, Spectral Flatness []. However, the synthetic sounds have higher Spectral Flux [] and Complexity []. Note that ParametricSynth also has lower pitch salience. We believe this is due to its larger mixture of sound generators which reduce salience of particular pitches.

Based on these results, we hypothesize that one potential reason the synthetic sounds could be useful for training is the informativeness of the samples. The larger amount of spectral change and higher complexity in terms of peaks could expose the model to more different kinds of sounds rapidly. To try to match these attributes, we produce mixtures of VGGSound, since mixed sounds may have more peaks and variation than individual samples. In VGGSound-Mix 5s, we take 5 arbitrary seconds from each sound and layer them into a 1-second sample. In VGGSound-Mix 10s, we do the same with all 10 seconds available. We show (Figure 2) that these get closer to the synthetic distributions on these features, without deviating on other features. These data distributions allow us to assess whether the benefits of synthetic data are largely driven by the change and informativeness of the signals. In Section A.3, we present results from models trained on these mixture distributions, and obtained mixed results, suggesting other factors of the synthesized sounds may also be important beyond this.

Figure: Figure 2: Comparisons of synthetic and real sound data (VGGSound []) on (A: Top) spectral features and (B: Bottom) causal uncertainty. Spectral features of synthetic sounds partially replicate real sounds, but exhibit differences in complexity and flux. Synthetic sounds are also more causally ambiguous, indicating a distribution shift. Using dense mixtures of real sounds partially closes these gaps, suggesting the synthetic sounds are different in part due to their density of auditory information.
Refer to caption: extracted/5654860/figs/spectralfeatures_causaluncertainty.png

#### 4.2.3 Causal Uncertainty

We also consider causal uncertainty [, , ], a factor that we intuitively expect to be different for the synthetic sounds. Helmholtz famously discussed perception in terms of unconscious causal inference from sensory input [], but the synthetic sounds have no physical causes and do not come from well-understood categories. In Figure 2B, we plot 3 proxies for causal uncertainty derived from probabilities of an AST classifier [] trained on AudioSet. We use the formulation from prior work of $H_{cu}$, the maximum predicted probability [, ]. We also propose two simple metrics to corroborate this: $H_{p}$ the (normalized) entropy of the output probability distribution, and a confidence score $S_{conf}$, the difference in probability between the most and second-most probable classes (log-scaled for the plot). Across all, the synthetic sounds are more causally uncertain than the real sounds. However, as with the spectral feature distributions, using mixtures of VGGSound [] clips moves the real distribution slightly closer to the synthetic distribution per $H_{cu}$ and $H_{p}$. We speculate that exposure to more causally uncertain sounds might be subtly helpful for representation learning; for example, they may contain diverse features that aid generalization to more ambiguous sounds present in downstream tasks. We characterize this as another important distributional difference between the synthetic sounds and realistic sounds from datasets such as VGGSound.

#### 4.2.4 Similarity to Target Distributions

Another lens we can use to understand the effectiveness of training on synthetic data is in terms of the distribution of sounds in the target downstream tasks. A common metric to compare sound distributions is the Fréchet Audio Distance (FAD) []. For simplicity, we use the canonical formulation based on VGGish embeddings, though there are some limitations of this [, ], and we use either the validation sets or first multi-fold splits of the target task audio. Table 2 shows that for ESC-50 [], VGGSound is closer in distribution to the target sounds, likely due to ESC-50’s focus on environmental sounds. For all other tasks, however, the synthetic sounds achieve a lower FAD, suggesting they may better capture task-relevant features for these tasks’ sounds. This finding echoes a study of MMDs in torchsynth [], where the Voice architecture shows higher-than-expected similarity to FSD50k sounds. We hypothesize that the synthetic training allows the model to see a wide variety of spectral behavior rapidly, in a way that supports an array of tasks.

**Table 2: FAD [] scores between different synthetic/real datasets and target downstream task data distributions, computed using either validation sets or the first fold (for multi-fold datasets). For 5/6 tasks, Voice achieves the lowest FAD despite containing synthetic sounds. On ESC-50, however, the VGGSound distribution appears to be closest.**
| Dataset | ESC-50 | FSD50k | LibriCount | NSynth | CREMA-D | Vocal Imitation |
| --- | --- | --- | --- | --- | --- | --- |
| Voice | 17.39 | 13.37 | 16.67 | 12.83 | 18.55 | 11.64 |
| VoiceFM | 18.48 | 15.91 | 17.67 | 14.49 | 21.24 | 13.66 |
| ParametricSynth | 18.75 | 19.44 | 21.04 | 17.42 | 25.33 | 17.32 |
| VGGSound | 6.71 | 25.33 | 29.09 | 27.67 | 33.83 | 27.75 |
| VGGSound-Mix 5s | 8.81 | 26.17 | 30.02 | 28.70 | 34.35 | 29.05 |
| VGGSound-Mix 10s | 9.30 | 26.09 | 30.15 | 28.88 | 34.06 | 29.16 |

### 4.3 Ablations and Sensitivity Analysis

In Figure 3, we study the effect of the perturbation factor $\delta$ on downstream task performance across all tasks for the strongest (Voice) architecture with and without additional (FX chain based) augmentations. Overall, $\delta=0.25$ appears to give the best result across tasks, with a few notable exceptions like NSynth without augmentations, and LibriCount overall.

Figure: Figure 3: Scores with the Voice architecture and different values of $\delta$ for evaluation tasks in Table 1 with and without augmentations. $\delta=0.25$ tends to give the best results overall.
Refer to caption: extracted/5654860/figs/evals_per_delta.png

## 5 Limitations

We opted for small-scale, established architectures like ResNets due to computational constraints and the desire for generalizable results. Scaling up to larger architectures like AST [] could further validate our approach in high-compute environments (our attempts to train such models were infeasible). Additionally, we focused on direct comparisons between synthetic and real data, excluding hybrid approaches that might offer valuable insights, since the design space of hybrid approaches (e.g. mixtures or fine-tuning) is large and warrants a deeper exploration.

Our simple isometric Gaussian noise perturbation proved effective, but it relies on a simplifying independence assumption (parameters are correlated within components and component chains) and doesn’t account for parameter semantics (e.g. octaves in pitch). Future work could explore fine-grained, semantically-aware perturbations operationalized via the covariance structure of the perturbations. Scheduling or adapting the perturbation factor during training could offer more challenging examples. It could also be learned, which would require stable end-to-end auto-differentiation (challenging depending on the synthesizer architecture). Our evaluation focused on standard classification benchmarks, but incorporating aspects like representation disentanglement could offer a richer view. Finally, while we explored various synthesis options, the possibilities are vast, and future work could investigate variable architectures for enhanced data augmentation.

On a broader note, we believe it’s important to examine synthetic data-generating procedures for possible biases in the way that we do with datasets. Though we think this procedure can mitigate some of the biases in collectible real datasets, different synthesizer architectures, values of $\delta$, and other decisions might inadvertently produce performance gaps for different tasks, applications, and downstream populations of use. We evaluated on a wide range of tasks in part to explore this possibility, but further evaluations would be helpful to assess these impacts.

## 6 Conclusion

Further improvements in auditory understanding depend greatly on the data underlying new models. In this work, we examined the value of synthetic data for learning representations of sound. We presented a method that perturbs the parameters of random synthetic sounds to generate audio doppelgängers, distinct yet similar sounds that provide a strong signal for contrastive learning. Through a comprehensive set of experiments, we showed how this approach can yield strong results on a wide range of tasks. We will release our code and models to enable the community to experiment with synthetic data sources for audio understanding, and hope this approach will help expand the machine learning toolkit for audio processing.

## Acknowledgements

The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing resources that have contributed to the research results reported within this paper. Manuel was supported by a US-Spain Fulbright grant.

## Appendix A Additional Results

### A.1 Comparison of Different Architectures Across Tasks

In Figure 4 we show the relative performance of models trained with data from different synthesizer architectures with $\delta=0.25$. These results illustrate that, though Voice-generated sounds appear strongest overall, there is some task specialization of these different synthesis approaches. For example, on LibriCount and NSynth, Voice is the lowest performer here.

Figure: Figure 4: Scores with a fixed $\delta=0.25$ and different synthesizer architectures for a suite of tasks including (from left to right) UrbanSound8k [], ESC-50 [], LibriCount [], CREMA-D [], VIVAE [], NSynth Pitch 5h [], FSD50k [], and Vocal Imitation []
Refer to caption: extracted/5654860/figs/evals_per_synth.png

### A.2 Effects of Increasing Perturbation Factor δ 𝛿 \delta italic_δ on Training

We seek to understand how increasing $\delta$ impacts the training dynamics. Figure 5 shows the impact of different $\delta$ on the final validation value of the alignment and uniformity costs respectively. Alignment cost increases monotonically with $\delta$, which shows the increased difficulty of aligning increasingly distant pairs. Uniformity, interestingly, has an inverted-U-shaped relationship with $\delta$. This may be because as the model struggles to align positives with moderate noise driven variation, it incurs a cost in uniformity in order to do so (e.g. creating clusters). With large $\delta$, the amount of noise present is significant, alignment is difficult, and the representations may be able to be more spread out.

Figure: Figure 5: Final validation scores showing the effect of $\delta$ on $\mathcal{L}_{align}$ and $\mathcal{L}_{unif}$. $\mathcal{L}_{align}$ increases monotonically with $\delta$, since the difficulty of aligning more distinct samples goes up. $\mathcal{L}_{unif}$, on the other hand, shows an inverse-U-shaped relationship with $\delta$.
Refer to caption: extracted/5654860/figs/plot_training1.png

### A.3 Results for all Variants

We give results for all synthetic model variants below, in Table 3.

**Table 3: Complete results for all model variants.**
| Data/Model | ESC | US8K | VIV | NSyn | C-D | FSD | VI | LCount |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| External Baselines |  |  |  |  |  |  |  |  |
| HEAR/ARCH Top [, ] | 96.65 | 79.09 | 44.28 | 87.80 | 75.21 | 65.48 | 22.69 | 78.53 |
| HEAR/ARCH SSL | 80.50 | 79.09 | 44.28 | 52.40 | 75.21 | 50.88 | 18.48 | 78.53 |
| MS-CLAP Linear [, ] | 89.95 | 82.29 | – | – | 23.15 | 50.24 | – | 54.51 |
| GURA (HEAR) [] | 74.35 | – | – | 38.20 | 75.21 | 41.32 | 18.48 | 68.34 |
| VGGSound Sup. [] | 87.45 | 77.57 | 39.38 | 43.80 | 54.36 | 43.76 | 14.06 | 56.10 |
| Internal Baselines |  |  |  |  |  |  |  |  |
| Random Init. | 22.45 | 55.03 | 33.81 | 36.20 | 38.91 | 9.03 | 2.43 | 44.91 |
| Voice (Ours, No-$\delta$, Aug.) | 48.65 | 59.46 | 36.31 | 32.80 | 46.32 | 16.88 | 7.12 | 47.64 |
| VGGSound SSL (Aug.) | 48.85 | 61.91 | 32.67 | 39.60 | 47.86 | 19.63 | 6.03 | 53.46 |
| VGGSound SSL (Jitter) | 52.95 | 63.82 | 38.12 | 14.20 | 50.03 | 24.02 | 3.43 | 69.77 |
| VGGSound-Mix 5s | 43.95 | 59.69 | 33.31 | 40.80 | 46.10 | 14.71 | 5.95 | 52.57 |
| VGGSound-Mix 10s | 42.95 | 57.40 | 32.03 | 40.20 | 46.57 | 15.77 | 6.43 | 51.07 |
| Audio Doppelgängers (Ours) |  |  |  |  |  |  |  |  |
| Best Synthetic | 58.90 | 66.71 | 39.45 | 44.40 | 48.43 | 24.12 | 9.15 | 58.60 |
| Voice ($\delta=0.01$) | 47.55 | 59.56 | 38.62 | 11.40 | 47.53 | 17.15 | 6.67 | 55.56 |
| Voice ($\delta=0.05$) | 47.90 | 64.02 | 37.93 | 13.80 | 46.45 | 17.77 | 7.72 | 51.52 |
| Voice ($\delta=0.10$) | 48.40 | 63.92 | 38.74 | 11.40 | 45.13 | 18.40 | 7.67 | 49.32 |
| Voice ($\delta=0.25$) | 58.90 | 66.71 | 39.45 | 32.20 | 48.24 | 24.12 | 9.15 | 52.95 |
| Voice ($\delta=0.50$) | 41.85 | 54.03 | 28.54 | 40.60 | 45.78 | 17.14 | 4.69 | 43.85 |
| VoiceFM ($\delta=0.01$) | 42.40 | 59.89 | 36.58 | 9.20 | 44.31 | 15.34 | 5.15 | 57.13 |
| VoiceFM ($\delta=0.05$) | 42.90 | 62.96 | 36.54 | 14.20 | 44.93 | 15.64 | 5.79 | 50.61 |
| VoiceFM ($\delta=0.10$) | 44.80 | 62.03 | 35.73 | 14.80 | 43.99 | 15.67 | 5.60 | 50.56 |
| VoiceFM ($\delta=0.25$) | 57.20 | 65.11 | 38.48 | 35.20 | 48.43 | 22.15 | 6.96 | 54.00 |
| VoiceFM ($\delta=0.50$) | 43.50 | 60.98 | 39.04 | 12.20 | 44.17 | 15.25 | 6.06 | 51.07 |
| Parametric ($\delta=0.01$) | 39.50 | 58.95 | 36.87 | 12.20 | 42.16 | 13.92 | 4.53 | 58.60 |
| Parametric ($\delta=0.05$) | 40.15 | 57.22 | 35.11 | 14.60 | 42.65 | 12.87 | 4.78 | 55.37 |
| Parametric ($\delta=0.10$) | 42.50 | 59.65 | 34.12 | 14.20 | 43.01 | 13.41 | 4.97 | 53.43 |
| Parametric ($\delta=0.25$) | 50.55 | 62.83 | 37.91 | 37.60 | 46.77 | 18.68 | 5.70 | 54.72 |
| Parametric ($\delta=0.50$) | 41.15 | 56.86 | 35.41 | 10.40 | 41.73 | 12.76 | 4.48 | 54.27 |
| Voice ($\delta=0.01$, Aug.) | 52.55 | 62.92 | 34.82 | 23.60 | 46.96 | 18.18 | 8.17 | 51.01 |
| Voice ($\delta=0.05$, Aug.) | 53.00 | 65.17 | 34.49 | 19.40 | 45.39 | 19.79 | 8.32 | 49.84 |
| Voice ($\delta=0.10$, Aug.) | 54.20 | 65.89 | 33.78 | 23.40 | 45.71 | 20.38 | 8.50 | 50.42 |
| Voice ($\delta=0.25$, Aug.) | 58.75 | 65.01 | 34.81 | 44.40 | 46.17 | 21.76 | 8.54 | 50.70 |
| Voice ($\delta=0.50$, Aug.) | 32.25 | 48.40 | 25.41 | 36.20 | 41.38 | 11.82 | 3.26 | 44.74 |

## Appendix B Additional Details on Training

### B.1 Augmentation Batching

Due to practical considerations in batching and memory management, augmentations are applied differently for real and synthetic data. In real data, augmentations are applied per-example within distributed data-loading workers. Synthetic data is batch-generated within the main process to avoid concurrency issues between JAX’s multithreading and PyTorch’s data loading. Individually augmenting examples in this synthetic data environment is prohibitively slow. As a solution, we mini-batched augmentations with a default size $\leq$ 100. This allows us to memory-efficiently leverage GPU processing and introduces variation within each training batch. While per-example augmentations might further enhance performance of synthetic data with augmentations, we believe our current approach is a conservative yet effective option and expect minimal impact.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please do not carry out extra experiments and make revisions to the paper. Answering these questions would be enough.

== The paper hinges on synthesis perturbation factor delta and makes the assumption that the parameter lie in a straight-line as explained in section 3.1. What is the evidence of this. What if the parameter space theta have a gaussian distribution or any other distribution. What would happen if I perturbed parameter vectors in any other manner. Is this the right way to perturb parameter. 

== Why was the paper not compared against different contrastive learning algorithms. How about SimCLR. Would such methods improve ? The author should the taken the Resnet baseline and done the exact same thing with existing augmentations and SimCLR. 

== Why are the gold standard numbers so poor ? FSD-50K paper mentions that the setup that does not include data augmentation including well cited Audio Transformers paper on it. So a ResNet-18 should perform well for MAP metric in the internal baseline and not be so off so as to report 9.03 ?

== why having a single parameter a good thing. Why would doing 100s of different augmentation e.g. audiomentation be a bad thing ? It seems like the augmentations chosen to compare against were weak. Why were all the augmentations not carried out ? 

== There are several papers solely focussed on audio augmentation. Why were the baselines not compared against audio augmentation papers  ? Why was augmentations such as random mix or others not carried out ? 

== The algorithm itself would only learn to discern in the positive cases similar sounding what are different but from the same category and in the negative cases noisy versions of the same category. However a typical contrastive learning algorithm e.g .SimCLR also takes into account audio files that are not noisy but from a different category. How would the authors consider these kinds of negative examples ? Is this approach better or worse than a typical contrastive learning approach e.g .SimCLR ? 

== Why would making the distributions similar in Figure 3 be needed? One would rather argue that the model should retain the same category but have the features described in Figure 3 e.g. inharmonicity be as much varied as possible as this can then account for out of distribution categories as well as make the audio representations robust and generalizable.
soundness: 2
strengths: == The idea appears neat. Ideally all of the real world audio files should reside in a small latent space. We can sample the latent space to get as many audio files as we want. We could use contrastive learning to learn robust audio representations. 

== Decent presentation. The author propose a technique present almost all the necessary that is needed for it. They then give benchmarks on standard datasets. 

== The algorithm is well explained easy to follow. The paper is a welcome change from LLM and Transformers based approaches that are just going after scale or yet another prompting paper.
summary: The authors propose contrastive learning from synthetic audio doppelgängers using audio synthesizers that can provide rich source of information that other augmentations fail to provide. Additionally they say that the approach is light weight and requires no extra data storage and only has a single hyper-parameter. They propose that this complements existing strategies and is reducing the data burden on practitioners.
weaknesses: == It took some digging around to figure out how Voice synthesize with 78 parameters able to do audio synthesis of specific categories. If it is using "Creative Text-to-Audio Generation via Synthesizer Programming" the authors should specifically mention it in section 3.1. It defiantly makes the content of the paper self contained and helps understanding it better. This make it poorly reproducible. It would have been great to add the synthesis technique to the paper. 

== Contributions mentioned on Page 2 point 2 are not correct and misleading. There are actually several papers that are doing exact same thing -- using synthetic data in some form for doing audio classification. They should be cited. Here are a few. i) Open-Set Tagging Dataset (OST) Mark Cartwright, ii) BLAT Bootstrapping language-audio pertaining based audio set .... iii) Can synthetic audio from generative foundational models assist audio recognition ....iv) Synthetic training set generation using T2A models for environmental sound classification. There exist many more for audio generation tasks which I am not including here but the authors should have included.

== The abstract mentions reducing the data burden. This is not true. We can define several augmentation strategies that can generate augmentations on the fly e.g. in Audiomentations and it is a matter of implementation detail and not about how we store all of the data.

== The main issue I have is with the algorithm itself. The authors mention that delta parameter is tuned and there exists a sweet spot with a high delta corresponding to noise and lower delta make the sound exactly the same with little diversity. Please comment on this. The algorithm itself would only learn to discern in the positive cases similar sounding what are different but from the same category and in the negative cases noisy versions of the same category. However a typical contrastive learning argument e.g .SimCLR also takes into account audio files that are not noisy but yet from a different category. How would the authors consider these kinds of negative examples ? Is this approach better or worse than a typical contrastive learning approach e.g .SimCLR ?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: - 1 second -long chunks looks quite short to learn interesting representations
- Why using the alignment and uniformity objective from Wang & Isola over other choices like the one in SimCLR or SigLIP?
soundness: 3
strengths: The article is well-written and detailed. 
The related work section is quite informative, especially the 2.1§.
Although the idea is pretty simple and straightforward, it is evaluated across multiple tasks, mainly against a more classic baseline trained on VGGSound + augmentations, and show good results, despite being trained solely on synthesized sound from a bespoke synthesizer.
summary: This paper proposes to rely exclusively on synthetic audio to learn representations with contrastive learning.
The originality of the approach is that it does not rely on audio augmentations (as in SimCLR) directly applied on the audio to create positive pairs, but on generating pairs of audio using a synthesizer with close parameters.
This method, although simple, manages to be on par with some of the SOTA methods on some tasks HEAR and ARCH.
weaknesses: Only one parameter delta to control the amount of variation for all parameters looks quite restrictive. If this is addressed in the Limitations section, it's still surprising that considering that the $f_0$ and the LFO parameters range are treated the same way. 
- Results, if interesting, are not beating existing methods, the analysis is extensive an varied.
- This article may be more appropriate for

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: While the paper demonstrates performance improvements, further investigation into the specific properties of the learned representations (e.g., through probing or visualization) and a comparison with alternative self-supervised methods applied to the same synthetic data would provide a more complete understanding of the contributions of the contrastive framework itself. Is it simply the paired structure and controlled variations that are beneficial, or is the contrastive loss crucial?
soundness: 3
strengths: 1. The paper presents a novel approach to contrastive learning in audio by utilizing entirely synthetic data. This is a significant departure from existing methods that rely on augmenting real recordings and opens up a new avenue for research in self-supervised audio representation learning. The demonstrated performance gains over real data with augmentations highlight the potential impact of this approach.

2. The paper is technically sound, with a clear methodology, appropriate experimental setup, and thorough analysis. The authors carefully consider the impact of the perturbation factor (\delta) and provide insightful comparisons between the synthetic and real data distributions, including analysis of features like spectral complexity and causal uncertainty.

3. The proposed method offers practical benefits, particularly in terms of reduced data requirements and faster training times. By eliminating the need for large-scale real-world audio datasets, the approach can significantly reduce the data burden on practitioners and potentially democratize access to high-quality audio representation learning. This has implications for resource-constrained settings and applications where data privacy is a concern.
summary: This paper introduces a novel contrastive learning approach for audio representation learning using synthetic data. Instead of augmenting real recordings, the authors leverage a modular synthesizer (SYNTHAX) to generate pairs of synthetic audio "doppelgangers" by randomly perturbing synthesis parameters (controlling timbre, pitch, and envelopes) by a factor \delta. This method creates positive pairs with causally controlled variations, providing rich contrastive information. Training a ResNet18 encoder with alignment and uniformity objectives on these synthetic pairs, the authors demonstrate performance exceeding that of models trained on real data (VGGSound) with augmentations across 8 diverse audio classification benchmarks, while also reducing training time. Analysis reveals the synthetic data exhibits higher spectral complexity and flux, and greater causal uncertainty compared to real audio, potentially contributing to its effectiveness. The hyperparameter \delta, controlling the dissimilarity between doppelganger pairs, is shown to significantly impact performance, with \delta = 0.25 generally yielding optimal results across tasks.
weaknesses: 1. The authors claim a self-supervised approach, but their method relies on a pre-trained SYNTHAX model. SYNTHAX itself likely required supervised training with labeled data, making the overall approach dependent on prior supervision. The authors should avoid using terms like "self-supervised" and clearly acknowledge the role of supervised pre-training in their method. The augmentations applied are also inherently limited to the codomain of the pre-trained SYNTHAX model, potentially restricting the diversity of generated sounds.

2. The evaluation focuses solely on detection tasks (classification). While the paper's introduction suggests a more general contribution to audio representation learning,  the lack of estimation or generation tasks in the downstream evaluation limits the scope of the demonstrated benefits.  Including a broader range of downstream tasks would strengthen the claims of general-purpose representation learning.

3. The reliance on a specific synthesizer architecture (SYNTHAX) and the parameterized perturbation method could introduce biases in the synthetic data distribution. While the authors analyze some aspects of this distribution, a more thorough investigation of potential biases and their impact on downstream performance across different audio domains and tasks is warranted. The paper would benefit from a discussion of the limitations of the chosen synthesizer and perturbation method and their potential influence on the generality of the learned representations.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How does the choice of synthesizer architecture affect the quality of the learned representations? Is there a significant difference in performance across different synthesizers beyond the three tested?
2. Could the approach be extended to longer audio sequences, and what impact would this have on the quality of representations and training time?
3. Is there potential for further improving performance by using more complex perturbation strategies, such as adaptive or learned perturbations?
4. How robust is the method to variations in delta across different downstream tasks? Would an adaptive approach to tuning δ during training yield better results?
soundness: 4
strengths: The synthetic audio doppelgänger concept introduces a unique and controlled approach to creating positive pairs for contrastive learning, which has not been explored in this way for audio data. Comprehensive experiments across multiple benchmarks substantiate the effectiveness of the approach, showing competitive performance with or even surpassing real-data baselines. This method reduces the need for large, labeled datasets, offering a scalable, lightweight solution suitable for a variety of audio applications.
summary: This paper introduces a novel approach to contrastive learning for audio representations using synthetic audio doppelgängers. By generating paired synthetic audio samples with controlled variations through sound synthesizer parameter perturbations, the authors address the limitations of traditional data augmentation techniques. This method surpasses results from large-scale audio datasets in several tasks, offering a lightweight, hyperparameter-efficient solution that requires no data storage. The synthetic data approach provides a rich source of contrastive information that enhances the model's ability to learn robust audio representations. The paper's contributions are well-supported by comprehensive experiments across multiple audio classification benchmarks.
weaknesses: While the simplicity of using a single hyperparameter δ is a strength, it could also be a limitation. The performance is sensitive to the choice of delta, and further analysis could be helpful in understanding how to optimize this parameter across different tasks. The paper mentions the potential for hybrid approaches combining synthetic and real data but does not explore this in detail. A comparison with hybrid training strategies could provide more insights into the benefits of the proposed method.

Meta-review: Thanks for your submission to ICLR.

This paper introduces a novel approach to contrastive learning for audio representations using synthetic audio doppelgängers.  The reviewers were generally positive about this paper going into the rebuttal/discussion, with three of the four reviewers providing mostly positive reviews.  On the plus side, the reviewers noted that the idea was clever, easy to follow, and useful; and that the presentation was good.  On the negative side, there were various questions about the algorithm and evaluation (e.g., the use of a single hyperparameter was noted by multiple reviewers).

Despite prodding from the AC, the reviewers did not engage much with the discussion.  In particular, the most negative reviewer did not address the rebuttal.  I read through the rebuttals and reviews, and felt that they adequately addressed the concerns, particularly of the more negative reviewer.  Further, there was nothing in these concerns that seemed to warrant not accepting the paper.  Indeed, the paper should be of interest to many in the community, particularly those working in audio.

Please keep in mind the reviewer comments when preparing a final version of the manuscript.

Decision: Accept (Poster)