Contrastive Learning from Synthetic Audio Doppelgängers
Authors: Manuel Cherep
Nikhil Singh
Venue: ICLR 2025 Poster (2025)
Keywords: synthetic data
audio
contrastive learning
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper learns audio representations from randomly generated pairs of synthetic data, surpassing results from training on a well-known large scale audio dataset on several tasks.

Abstract: Learning robust audio representations currently demands extensive datasets of real-world sound recordings. By applying artificial transformations to these recordings, models can learn to recognize similarities despite subtle variations through techniques like contrastive learning. However, these transformations are only approximations of the true diversity found in real-world sounds, which are generated by complex interactions of physical processes, from vocal cord vibrations to the resonance of musical instruments. We propose a solution to both the data scale and transformation limitations, leveraging synthetic audio. By randomly perturbing the parameters of a sound synthesizer, we generate audio doppelgängers—synthetic positive pairs with causally manipulated variations in timbre, pitch, and temporal envelopes. These variations, difficult to achieve through augmentations of existing audio, provide a rich source of contrastive information. Despite the shift to randomly generated synthetic data, our method produces strong representations, outperforming real data on several standard audio classification tasks. Notably, our approach is lightweight, requires no data storage, and has only a single hyperparameter, which we extensively analyze. We offer this method as a complement to existing strategies for contrastive learning in audio, using synthesized sounds to reduce the data burden on practitioners.

Published as a conference paper at ICLR 2025
CONTRASTIVE LEARNING FROM SYNTHETIC AUDIO DOPPELGÄNGERS
Manuel Cherep"
Nikhil Singh"
Massachusetts Institute Technology mche repêmi eddu
Dartmouth College
nikhil nghēda rtmouth. edd
duppelgangers.medii.miii wwt.eeeei...iiii..u..u.
Augmentations
Audio Doppelgängers (Ours)
SpecAugment FX Chain 0.01
0.05
8 0.10
= 0.25
= 0.50
Figure 1: (Left) Standard data augmentation techniques for contrastive learning applied to audio spectrograms (Right) Audio Doppelgängers, our approach synthesizing sounds that are controllably different using perturbed synthesis parameters, shown for different factors 6. These sounds can vary in causally controllable ways beyond what data augmentations can achieve.
ABSTRACT
Learning robust audio representations currently demands extensive datasets of real-world sound recordings. By applying artificial transformations to these record- ings, models can learn recognize similarities despite subtle variations through techniques like contrastive learning. However, these transformations are only ap- proximations of the true diversity found in real-world sounds, which are generated by complex interactions of physical processes, from vocal cord vibrations to the resonance of musical instruments. We propose solution to both the data scale and transformation limitations, leveraging synthetic audio. By randomly perturbing the parameters sound synthesizer, we generate audio doppelgängers synthetic positive pairs with causally manipulated variations in timbre, pitch, and temporal envelopes. These variations, difficult to achieve through augmentations of exist- ing audio, provide rich source of contrastive information. Despite the shift randomly generated synthetic data, our method produces strong representations, outperforming real data on several standard audio classification tasks. Notably, our approach is lightweight, requires no data storage, and has only single hyperpa- rameter, which we extensively analyze. We offer this method as a complement existing strategies for contrastive learning in audio, using synthesized sounds to reduce data burden on practitioners.
INTRODUCTION
"Noises have generally been thought of as indistinct, but this is not true."
Pierre Schaeffer, /986
The success of modern machine learning algorithms for tasks like audio understanding often hinges on both the quality and quantity of available data. Self-supervised learning methods, like contrastive
Equull contribution.

---

Published as a conference paper at ICLR 2025
learning, have even been able to leverage unlabeled data, enabling more human-like learning from patterns without needing explicit supervision. However, human perceptual processing is remarkably robust beyond this: for example, the human auditory system can easily recognize sounds across wide range of variations, such as changes in pitch, timbre, or background noise. Moreover, humans can quickly learn to recognize novel sounds that they encounter in their environment. Replicating this ability to learn from a diverse array of sounds-or "noises," as we might call them- significantly enhance the efficiency, scalability, and adaptability of machine learning models.
Contrastive learning, which operates by recognizing similarities in the data among negative distractors, often relies on augmentations: transformations of input data that preserve content semantics, This method has been influential in audio representation learning, with specific implementations ranging from spectral masking to temporal jitter to cropping and other methods (Huang al.. 2022; Saeed & Burgoyne 2021 Wang & Oord 2021 Al-TTahan & Mohsenzadeh [021]]
et al. 2021 Nizumi et al. [022] Manocha al. 2021). Data augmentations, though demonstrably useful, operate at the level of the observed data, not the underlying data gettaaeeerating process as would be observed in real-world variation. They statistically alter data without directly manipulating the causal mechanisms
that produced it, resulting in high correlation between augmented samples, well limited control and interpretability.
In our work, we propose different strategy: using synthesizer to overcome this barrier, in addition to providing the scalability required for modern pretraining regimes through virtually unlimited data synthesis. A synthesizer can be understood as a system where parameters (relating to psychophysical attributes like pitch, timbre, and loudness) causally influence the generated sound. Modifying these parameters allows to intervene in the data-generating process controllable way to generate positive pairs that vary in terms of their underlying synthesis parameters. Unlike traditional data augmentation techniques, our method generates entirely synthetic audio data from scratch. This approach allows us to control the underlying data-generating process directly, offering perspective distinct from augmentation of real data.
We formulate an approach in which we randomly synthesize sounds, and then slightly perturb their parameters to generate positive pairs. We call these audio doppelgängers (examples Figure D]: they share a resemblance but are in fact distinct enough to learn from the variation between them. In a way, this approach uses an artificial data source effectively consisting of random synthetic noises but more "natural" differences akin to variation in similar sounds; as Pierre Schaeffer put it, noises are not indistinct. Through a comprehensive set of experiments, we show that models trained this way can yield strong performance on wide range of downstream tasks, competitive with real audio.
Overall, this work contributes:
1. An approach to synthesizing paired audio examples with a continuously controllable degree of dissimilarity, specified by a simple and interpretable hyperparameter
2. The first study, knowledge, of synthetic data methods for audio representation learning.
3. Comprehensive experiments in which we train and compare over 20 model variants across 8 downstream tasks to provide evidence that training with our approach can yield strong results on a wide range of audio processing tasks.
4. An analysis of how these synthetic datasets differ from realistic audio datasets terms of their auditory features, and how this might contribute to learning effective representations.
2 RELATED WORK
2.1 LEARNING FROM SYNTHETIC DATA
Synthetic data, artificially generated information rather than collected from real-world sources, has
emerged as valuable tool for learning across various domains (Liu et al. 2023 Silver al.) 2017]
Kumar et al. 2020 Meng et al. 2022). By addressing data scarcity, privacy concerns (Tucker et al.
2020; DuMont Schütte et al. 2021), or removing biases (Tan al. 2020; Ramaswamy et al. 2021
synthetic data offers a promising avenue to complement scarce Longpre et al. (2024) real-world data
and further drive progress in machine learning research.
2

---

Published as a conference paper at ICLR 2025
Audio presents unique challenges due to the complexity of waveforms and temporal dependencies. Synthetic data has found applications in subareas like speech recognition (Rosenberg et al. 2019]
Rossenbach al., 2020; Laptev et all 2020: Fazel et al. 2021 Hu et al. 2022 Gao et al.) leveraging
text-to-speech systems for detecting unspoken punctuation (Soboleva et al. 2021), recognizing
low-resource languages (Bartelds et al., (2023) increasing acoustic diversity (Chen et al. 2020b) or
detecting out-ol-vocabulary words (Zheng et al. 2021). However, non-speech audio domains can be highly diverse, requiring more complex approaches to data synthesis. In this domain, synthetic data has been used for specific tasks like timbre-text alignment (Jonason & Sturm (022) and vocoding (Wang et al. (023). The partially synthetic NSynth (Engel et al. (017) dataset has also been used for pitch estimation and instrument classification. In our work, we tackle the general audio domain, proposing a synthetic data approach that can produce diverse sounds for general- audio representation learning.
In computer vision, synthetic data more popular and has been employed in different tasks to improve
performance (Chen et al. 2019 Ros et al. [0116] Varol et al. 2017 lonescu al. [2013] Shakhnaaaarr
ct al. 2003 Mayer et al., 2016 Dosovitskiy et al. [015] Ren & Lee| 2018). While initially focused
on using graphics engines to generate photorealistic scenes, recent work has investigated sampling
synthetic data from deep generative models (Besnier et al. (2000) Ravuri & Vinyals 2019
Jahanian
al. 2021 Zhang et al. [222 TTitrong et | [202]] et al. [202] Shriiasttaa al. 2017 Hoffman
iet al. 2018, Tian et al. 2024 2023] Trabucco et al. 2023] Yang et al. [021]] Jeong & Shin 2021).
However, these models aim produce realistic images and still depend on real image datasets for
training or synthesis. Thus, recent work has pushed away from realism, generating synthetic data
such as fractals (Kataoka et al.) 2020), or through other procedural noise models (Baradad Jurjo et al. 2021 Baradad et al. (002) to use as training data for visual representation learners. In our work, we also abandon realism and leverage randomly generated synthetic sounds to learn audio representations for downstream tasks.
2.2 CONTRASTIVE LEARNING
common strategy for learning from unlabeled data is contrastive learning. In this technique, we seek representations that are invariant to minor differences, i.e. they encode space in which similar objects are closer together, and dissimilar objects are further. A classic strategy for this to use data augmentations, transformations which noticeably alter datapoint (for example, randomly cropping an image) without changing its essential content (e.g. what the image is of, such as a cat). These transformed versions then become a positive pair, while other examples (e.g, an image of a dog) become negatives. In audio, contrastive learning has been used extensively to produce high-quality representations for downstream tasks (Al-Tahan & Mohsenzadeh 2021 Saeed et al.) 2021; Ravanelli & Bengio 2019 Wang & Oord 2021//ooneeca et al. 2021b). Our approach differs fundamentally from data augmentation strategies commonly used in contrastive learning. Instead of applying transformations to existing audio samples, we generate synthetic audio pairs by perturbing synthesizer parameters, creating positive pairs with causal variations that are difficult to achieve through augmentations. This represents a novel application of synthetic data in the context of general- purpose audio representation learning.
2.3
SOUND SYNTHESIS
The toolkit of sound synthesis has evolved to include a variety of hardware and software (Mathews al. 1969 Pinch & Trocco| 2004 Théberge 1997). Synthesizers, abstractions of sound synthesis and processing methods often designed to act as musical instruments, are key to this: they expose control parameters that let sound designers guide them to produce desirable sounds for music. film. and many other applications. Accelerated synthesizers (Turian et al.) 202T] Cherep & Singh (023) have recently allowed much seater-than-realting sound generation, offering the ability to iteratively tweak parameters to reconstruct sounds (Hagiwara et al. 2022] [Shier 2021) and even match textual descriptions (Cherep et al., 2024). Such approaches highlight the practical utility of synthesizers: lightweight architectures controlled by a limited number of interpretable parameters are capable of producing a diverse array of sounds, often corresponding to well-know categories and concepts (e.g. the sound of waves can often be modeled with varying filtered noise). In our work, we leverage SYNTHAX (Cherep & Singh, 2023), to rapidly produce diverse training data with controllable similarity between examples.
3

---

Published as a conference paper at ICLR 2025
3 METHODS
3.1 DATA GENERATION
Our data generation pipeline uses virtual modular synthesizers implemented by SYNTHAX (Cherep & Singh 2023) in JAX. By default, we use the Voice synthesizer architecture (Turian et al. 2021 which can generate perceptually diverse sounds. Our synthesizer consists of several common modules (with parameter-count in parenthesis):
Keyboard (2x): Controls the sound's fundamental frequency (/o) and duration.
Low-Frequency Oscillators (LFOs; 8x each): Two LFOs modulate various aspects of the sound, each with parameters for frequency, modulation depth, initial phase, and amplitude weights across waveforms.
ADSR Envelopes (5x each): Six envelopes shape the amplitude and modulation signals, each defined by attack, decay, sustain, release, and eurvature (a).
Voltage-Cone Controleod Oscillators (VCOs): Includes sine VCO (3x) with tuning, modulation
depth, initial phase, and a square-saw VCO (4x) adding waveform shape.
Noise Generator: Provides broadband noise without additional parameters.
Modulations (20x): Weight matrix controlling how modulation sources affect destinations.
Audio Mixer (3x): Combines outputs of oscillators and noise generator.
In total, the Voice synthesizer has 78 parameters. Perturbing these parameters allows us to generate wide variety of sounds with controlled variations, such as slightly lower or higher pitch, a slightly little more less noise. In experiments,
longer onset release, architectures: VoiceFM has 130 parameters and includes a frequency modulation (FM) operator, and ParametricSynth has 2 sine and 2 square-saw oscillators, sine FM and 1 square-saw FM operator, 340 in total. Varying the architecture allows us to investigate whether architectural complexity could affect the quality of representations learned. We generate 1-second sounds by default, for compatibility with most encoders (e.g. VGGish (Hershey et al. 2017)). However, this practice can be extended to longer sounds.
investigate two further
Synthesis perturbation factor (6) A key contribution of our work is synthesizing paired positive samples that sound alike, but are dissimilar due to their synthesis parameters and not only post-hoc effects (e.g. augmentations). This draws on the canonical definition from contrastive learning of positives that are sampled from the same lutent class (Saunshi et al. 2019).
For single positive pair, we first sample parameter vector uniformly randomly 0 E [0, 1Jms (/(0,) 1) from the normalized synthesis parameter space, where my the number of control parameters in the given synthesizer. Then, independently sample two isotropic Gaussian noise vectors 21, B2 ~ (O, Im )m)) We define a parameter å that scales this noise, and then produce two perturbed parameter vectors 0, 0 + dz; V; [1,2}. From these, we clip values back into [0, 1] to synthesize two corresponding sounds which serve as positives in the contrastive learning setup.
In principle, 5 controls the distance between the positive pairs and therefore the hardness of the contrastive learning task. Practically, we expect there to be a sweet-spot for considering prior work mutual information and redundancy in contrastive learning problems (Tian et al. 2020 Tosh et al, 2021) as with very high 5, the parameter vectors may become dominated by noise, resulting in difficulty effectively aligning their representations. Given this, we extensively study the effect of on downstream results.
3.2
REAL DATA
To compare to real audio data, we use sounds from VGGSound (Chen et al. 2020a), well-known dataset taken from YouTube videos (we only use audio). We use random sample of 100,000 10-second files and select a random I-second segment from each file at each iteration to augment. This allows us to fairly compare to synthetic sounds by keeping duration constant, while still sampling from a variety of real sounds by randomizing the 1-second segments. Though VGGSound has labels included, we do not use them in training these models to keep the self-supervised constraint. Note that VGGSound is currently one of the largest publicly released audio datasets for pretraining, unlike AudioSet (which only releases URLs, not the content itself).
4

---

Published as a conference paper at ICLR 2025
3.3 PREPROCESSING, DATA AUGMENTATIONS, AND AUDIO ENCODER
In our experiments, we use VGGish frontend representations (Hershey et al. 2017). We resample audio to 16kHz and obtain mel spectrograms with 64 mel bands and 96 time steps. We use a chain of effects as augmentations (implemented in porch-audiomentation frequency range 20-800Hz). a low-pass filter (1.2-8kHz), pitch shift (-2 to 2 semitones), time shift (-25% to 25%, rollover enabled), and finally reverberation for which we sample randomly from a set of impulse responses. All augmentations are applied with probability 0.5. We found that this yielded far stronger results than SpecAugment (Park et al., 2019), and so we use this as a comparison point in all our experiments. More details on the augmentation are given in Appendix B We also test temporal jitter, wherein different 1 -second segments are sampled from within the same source clip and treated as positives (Saeed et al., [022] Spijkervet & Burgoyne, 202T). Our audio encoder is R 8 (He et al. (2166), where we replace the initial layer with I-channel convolution to account for the effectively 1-channel spectrogram.
:: high-pass filter (cutoff
3.4 CONTRASTIVE LEARNING
We train for 200 epochs, generating (or sampling) 100,000 sounds per epoch, with a 90%-1099 train- validation split. We use batch size of 768 per GPU with two V100s. The training uses the alignment and uniformity objectives (Wang & Isola 2020) used in prior work on learning with synthetic data (Baradad Jurjo et al. 2021). We adopt the default parameters for these: unif; 2, align, = 2, and equal weights A A2 for both terms. Following this work, we use stochastic gradient total batch size
descent for optimization, with maximum learning rate of 0.72 (calculated as 0.12 x
256
and weight decay 10 The learning rate follows multi-step schedule with y = 0.1, and milestones at 77.5%, 85%, and 92.5% of the total learning epochs. Detailed steps are provided in Algorithm [ Training with our synthetic data takes approx. 1-2 hours, as the data is generated on the fly in batches, whereas using on-disk datasets with effect chain augmentations can extend training time up to 6-8+ hours.
3.5 EVALUATION TASKS
To obtain a broad picture of the quality of our learned representations, we conduct experiments range of audio classification tasks from the HEAR (Turian et al., (0022) and ARCH (La Quatra et al. 2024 benchmarks. We use evaluation tasks that focus on general audio understanding, rather than tasks that are highly specialized or domain-specific (e.g.. tasks exclusively related to speech or music). Our method aims to learn general-purpol
audio representations from synthetic data; therefore, we implement tasks which encompass a broad range of everyday sounds. This aligns with our goal of demonstrating the effectiveness of our representations in diverse real-world scenarios.
These tasks cover a wide range of capabilities including sound classification tasks like ESC-50 (Pizzak 2015). FSD-50k (Fonseca et al.) (202 la). and UrbanSound8K (Salamon al. 2014). ) vocal affect tasks with and without speech like VIVAE (Holz et al./ (022) and CREMA-D (Cao et al.) 2014). musical pitch recognition via NSynth Pitch (5h) (Engel et al. 2017). vocal sound imitation recognition using Vocal Imitations (Kim et al. 2018), and LibriCount (Stöter et al. (018)) for "cocktail party" style speaker count estimation task. We conduct linear probing experiments using the Adam optimizer for the benchmark-specifeee epochs with the default learning rate of 0.001 and batch size of 32.
4 RESULTS
4.1 BENCHMARK RESULTS
In Table T we show results across 8 tasks. The top section features external baselines from the
HEAR (Turian al. 2022) leaderboard and ARCH (La Quatra al. 2024) benchmark results, first the strongest overall and then only suf--superiseed It also includes results from MS-CLAP (Elizalde al. 2023) linear probing experiments, GURA (Wu al. 2022) (strongest overall model on HEAR),
and finally the original ResNet18 trained on VGGSound (supervised) (Chen et al. 2020a). Note that
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Our contrastive learning procedure with audio doppelgüngers. In the training loop, we drop the batch index i for simplicity. We also show the pairwise distance in Emirs though the implementation (via torch pdiit) uses condensed representation.
Require: Batch size lc
Require: Perturbation factor 8
Require: Virtual synthesizer with mss parameters
Require: Embedding model M with embedding size IMM (512 in our case)
Require: Total number of training batches Nbatches
Require: (X E (R+xmM) + log
exp (-41Xi) x(]])) where / 2
for i = to Nhatches do
@ [0, 1] 1xms (0.
{Random batch of parameters) [Isotropic Gaussian perturbation noise} (Clipped perturbed parameters)
Z, Z2 E R ~ IN(
I max(0, + 1))
Ô max(0, min(@ 8Z2, 1))
A1 f S(O)), A2 S(O2)
{ [Synthesize audio from parameters] (Embedding from model]
E1 < 1(A1), Eg + M(22)
Lalign
| E2lil112 where a = 2
{Alignment cost)
Luniform t ((uni((En) + lumit (E2)]
(Uniformity cost) Byy default, we set λ1 >2 1}
Liotal f- X falign + X2Cuniform
Update model M using Lrotal end for
HEAR leaderboard results may use MLP probes, whereas ours are linear. We add additional internal baselines, including random weights, synthetic data trained without å but with augmentations, and variants of ResNet18 we trained on VGGSound (with augmentations, and alternately with temporal jitter). Finally, we include selection of our results; the best overall score achieve using model trained on data from each of
synthetic approach (first row), followed by the best- the synthesizer architectures (including Voice with augmentations). In Appendix A.3] provide a full set of results from all model variants: all synthetic datasets for all values of d, and further baselines less performant than those we present here.
Overall, our best scores uniformly outperform training on VGGSound with augmentations, and outperform training with temporal jitter (the strongest internal baseline) in 6/8 cases. In some cases, these results are also competitive with strong baselines, such as beating the supervised ResNet18 result on 3/8 tasks, CLAP on 2/5, and GURA on 1/6. Additionally, adding further augmentations to our audio doppelgänger- based training does not seem to hold significant benefits, despite being highly beneficial when training with synthetic sounds with no 6, suggesting the 8-based perturbations are already sufficiently strong. All this is accomplished without these models seeing any real sounds during pretraining. Finally, Voice with = 0.25 is the strongest synthetic-traind model overall, being the top performer 5/8 tasks, but we note that there is some inter-task variability in the best synthesizer and delta.
4.2 CHARACTERIZING THE DATA DISTRIBUTION
Here, we focus on understanding the distribution of synthetic sounds and how they differ from natural sound properties. We primarily use our alternate training set, VGGSound (Chen et al. 2020a), for these measures. Unless specified otherwise, we use a randomly sampled (for VGGSound) or generated (for synthetic) set of 1000 sounds for each given dataset used for these characterizations. Our goal is to help understand what properties of the synthetic data make useful for representation learning, given its strong performance.

---

Published as a conference paper at ICLR 2025
Data/Model
ESC
US8K
VIV
NSyn
C-D
FSD
LCount
External Baselines
HEAR/ARCH Top
96,65
79.09
44.28
87.80
75.21
65,48
22.69
78.53
HEAR/ARCH SSL
80.50
79.09
44.28
52.40
75.21
50.88
18.48
78.53
MS-CLAP Linear
89.95
82.29
23.15
50.24
54.51
GURA (HEAR)
74.35
38.20
75.21
41.32
18.48
68.34
VGGSound Sup.
87.45
77.57
39.38
43,80
54.36
43.76
14.06
56.10
Internal Baselines
Random Init.
22.45
55.03
33.81
36.20
38.91
9.03
2.43
44.91
Voice (Ours, No-5, Aug.)
48.65
59.46
36.31
32.80
46.32
16.88
7.12
47.64
VGGSound SSL. (Aug.)
48.85
61.91
32.67
39.60
47.86
19.63
6.03
53.46
VGGSound SSL (Jitter)
52.95
63.82
38.12
14.20
50.03
24.02
3.43
69.77
Audio Doppelgängers (Ours)
Best Synthetic
58,90
66.71
39.45
44,40
48.43
24.12
9.15
58.60
Voice (Ō = 0.25)
58.90
66.71
39.45
32.20
48.24
24.12
9.15
52.95
Voice (8 0.25, Aug.)
58.75
65.01
34.81
44.40
46.17
21.76
8.54
50.70
VoiceFM (ó 0.25)
57.20
65.11
38,48
35.20
48.43
22.15
54.00
Parametric (6 = 0.25)
50.55
62.83
37.91
37.60
46.77
18.68
5.70
54.72
Table 1: Evaluation results suite of tasks including (from left to right) ESC-50 (Piczak UrbanSound8k (Salamon et al. (2014), VIVAE (Holz et al. (002)). NSynth Pitch 5h (Engel et al.
2015 5
2017
CREMA-D (Cao al. 2014). FSD50k (Fonseca al. 2021a). Vocal Imitation (Kim al.
2018)
and LibriCount (Stöter et al. 2018). For internal baselines, we only bold tasks where the baseline beats the best synthetically trained result. Results for all synthetic variants are in Appendix A.3
4.2.1 EMBEDDING SIMILARITY
First, we look the distribution of synthetic sound pairs and establish that 5 meaningfully controls (a proxy for) perceptual or semantic dissimilarity. We operationalize this using LAIÓN-CLAP embeddings (Wu et al. 2023). since they trained large variety of sounds with semantic descriptions associated. Figure 2A shows how the average cosine similarity decreases monotonically with increasing 8 for all synthesizers. Figure provides a view of how d affects the geometry of the embedding space. Here, we plot the first two principal components of the CLAP embeddings along with the path length for each positive pair of synthesized samples from a Voice synthesizer. As 8 increases, the path lengths increase and overlap more resulting in less clear separation of positive pairs from negatives. We view this as a signal that we can effectively control the hardness of the contrastive task using 6, the perturbation factor.
4.2.2 SIMILARITIES AND DIFFERENCES FROM REAL DATA
Next, we compare the synthetic data distribution VGGSound (Chen et al. 2020a) data. Figure 3 compares a selection of features' distributions between several dataset variants. For synthetic datasets, we have Voice, VoiceFM, and ParametricSynth variants. For real datasets, we have VGGSound. We first compare to randomly sampled I-second chunks. Here, the synthetic sounds match several feature distributions well, such as Inharmonicity (Peeters 2004). Odd-to-Even Harmonic Ratio (Martin
& Kim 1998), Pitch Salience (Ricard, 2004), and, to lesser extent, Spectral Flatness PPettrr 2004). However. the synthetic sounds have higher Spectral Flux (Tzanetakis & Cook 1999) and
Complexity (Laurier et al. 2010). Note that PerumetricSynthh also has lower pitch salience. We believe this due to its larger mixture of sound generators which reduce salience particular pitches.
Based on these results, we hypothesize that one potential reason the synthetic sounds could be useful for training is the informativeness the samples. The larger amount of spectral change and higher complexity in terms of peaks could expose the model to more different kinds of sounds rapidly. To try to match these attributes, we produce mixtures of VGGSound, since mixed sounds may have more peaks and variation than individual samples. In VGGSound-Min 5s. take 5 arbitrary seconds from each sound and layer them into a I-second sample. In VGGSound- -ix 10s, we do the same with all 10 seconds available. We show (Figure that these get closer to the synthetic distributions these
7

---

Published as a conference paper at ICLR 2025
Cosine Similarity A
1.0-
Synthesizer
0.8
ParametricSynth
Volce
VoiceFM
0.6
0.4
0.2
0.0
8
0.01
- 0.05
å 0.1
8- 0.25
3 0.5
B
Figure 2: (A: Top) Average CLAP (Wu et al. 2023) embedding cosine similarity between positive pairs for different architectures and different values of (B: Bottom) PCA of CLAP embeddings for sounds generated with the Voice architecture, with line segments showing distances between paired examples. Red and blue points are paired positive instances, Across both plots, as increases, the positive pairs systematically become more perceptually dissimilar (via the CLAP embedding proxy).
Inharmonicity (log)
Odd-to-Even Harmonic Ratio (log)
Pitch Salience
0.0
-10
-5
-10
-5
0
5
0.25
0.50
0.75
Spectral Complexity
Spectral Flatness
Spectral Flux
Dats
1.0
veGSound
Density
verisound Mix 55
0.5
VEGGsound--tt 10s
Voice
WarseFF
0.0
Faram FayettticSyyth
40
0.6
0.50
Han max (p.)
Hp
Pi
Sonff log [max(p.) max (p.)
kig
1.0
0.5
0.0
0.25
0.50
0.75
0.2
0.4
0.5
-10.0
-7.5
-5.0
-2.5
Value
Figure 3: Comparisons of synthetic and real sound data (VGGSound (Chen et al. Top) spectral features and (B: Bottom) causal uncertainty. Spectral features of synthetic sounds partially replicate real sounds, but exhibit differences in complexity and flux. Synthetic sounds are also more causally ambiguous, indicating a distribution shift. Using dense mixtures of real sounds partially closes these gaps, suggesting the synthetic sounds are different in part due to their density of auditory information.
2020a)) on (A:
features, without deviating on other features. These data distributions allow us to assess whether the benefits of synthetic data are largely driven by the change and informativeness of the signals. In Appendix A..3 we present results from models trained on these mixture distributions, and obtained mixed results, suggesting other factors of the synthesized sounds may also be important beyond this.
8

---

Published as a conference paper at ICLR 2025
4.2.3 CAUSAL UNCERTAINTY
We also consider causal uncertainty (Ballas & Sliwinski 1986 Ananthabhotla et al. 2019 Boger et al. 2021), a factor that we intuitively expect to be different for the synthetic sounds. Helmholtz famously discussed perception in terms of unconscious causal inference from sensory input (Helmholtz) 1867). but the synthetic sounds have no physical causes and do not come from well-understood categories. In Figure BB we plot proxies for causal uncertainty derived from probabilities of an AST classifier (Gong et al. 2021) trained on AudioSet. We use the formulation from prior work of Hew. the maximum predicted probability (Ananthabhotla al.) 2019 Boger et al., 202I). We also propose two simple metrics to corroborate this: Hp the (normalized) entropy of the output probability distribution, and confidence score Scon the difference in probability between the most and second-mov most probable classes (log-scaled for the plot). Across all. the synthetic sounds are more causally uncertain than the real sounds. However, as with the spectral feature distributions, using mixtures of VGGSound (Chen et al. 2020a) clips moves the real distribution slightly closer to the synthetic distribution per ou and H We speculate that exposure to more causally uncertain sounds might be subtly helpful for representation learning; for example, they may contain diverse features that aid generalization to more ambiguous sounds present in downstream tasks. We characterize this another important distributional difference between the synthetic sounds and realistic sounds from datasets such as VGGSound.
4.2.4 SIMILARITY TO TARGET DISTRIBUTIONS
use understand the effectiveness of training synthetic data is in terms of the distribution of sounds in the target downstream tasks. common metric to compare sound distributions is the Fréchet Audio Distance (FAD) (Kilgour et al. 2018). For simplicity, canonical formulation based on VGGish embeddings, though there are some limitations of this (Gui al., 2024 Tailleur et al.. 2024), and we use either the validation sets first multi-fold splits of the target task audio. Table 2 shows that for ESC-50 (Piczak (2013). VGGSound is closer in distribution to the target sounds, likely due to ESC-50's focus on environmental sounds. For all other tasks, however, the synthetic sounds achieve a lower FAD, suggesting they may better capture task-relevant features for these tasks' sounds. This finding echoes a study of MMDs torehsynth (Turian et al.) 2021), where the Voice architecture shows higher- higher-than-ee than-expected similarity to FSD50k sounds. We hypothesize that the synthetic training allows the model to see a wide variety of spectral behavior rapidly, in a way that supports array of tasks.
Another lens
the
Dataset
ESC-50
FSD50k
LibriCount
NSynth
CREMA-D
Vocal Imitation
Voice
17.39
13.37
16.67
12.83
18.55
11.64
VoiceFM
18.48
15.91
17.67
14.49
21.24
13.66
ParametricSynth
18.75
19.44
21.04
17.42
25.33
17.32
VGGSound
6.71
25.33
29.09
27.67
33.83
27.75
VGGSound-MR 5s
8.81
26.17
30.02
28.70
34.35
29.05
VGGSound-Mel 10s
9.30
26.09
30.15
28.88
34.06
29.16
Table 2: FAD KKllgour et al. 2018) scores between different synthetic/real datasets and target downstream task data distributions, computed using either validation sets or the first fold (for multi- fold datasets). For 5/6 tasks, Voice achieves the lowest FAD despite containing synthetic sounds. On ESC-50, however, the VGGSound distribution appears to be closest.
4.3 ABLATIONS AND SENSITIVITY ANALYSIS
In Figure 4 we study the effect of the perturbation factor on downstream task performance across all tasks for the strongest (Voice) architecture with and without additional (FX chain based) augmentations. Overall, we found in early experiments that d = 0.25 gives the best results. This is observed on the full suite of tasks (with notable exceptions like NSynth without augmentations, and LibriCount overall). In Appendix AA.2 we discuss why this value might be strongest from the perspective of alignment and uniformity results.

---

Published as a conference paper at ICLR 2025
ESC-50
USaK
VIVAE
NSynth
Score (%)
50-
35
40
30
CREMA-D
FSD50k
Vocallmitation
LibriCount
Score (%)
55
20
8
45-
50-
15
0011
0.8
0.10
0.25
0.50
0.10
0.2
0.01
0.05
0.10
0.25
0.50
0.0
0.48
0.10
000
0.50
Transformations
0
No Augmentations
Augmentations
Figure 4: Scores with the Voice architecture and different values of for evaluation tasks in Table with and without augmentations. = 0.25 tends to give the best results overall.
LIMITATIONS
Our study demonstrates the efficacy of our approach using established architectures like ResNets, balancing computational efficiency with the goal of producing generalizable results. While we focused on these architectures, our findings lay the groundwork for future investigations with larger encoders such as AST (Gong et al. 2021); this is straightforward extension. Our research also focused on a clear comparison between synthetic and real data, which allowed us to rigorously evaluate our method's effectiveness. The potential for hybrid approaches, combining synthetic and real data, has a wide range of possibilities in mixing strategies and fine-tuning techniques. These can all be explored without changing our method itself.
The isometric Gaussian noise perturbation proved highly effective, despite its simplicity, since changes identifiable attributes (e.g. pitch) subtly. This success points to the robustness of our method, while also highlighting opportunities for more sophisticated perturbation strategies to further enhance it. Future work could explore anisotropic perturbations that account for parameter relationships. Adaptive or learned perturbation strategies could also offer significant advancements. Additionally, our evaluation centered on widely used classification benchmarks to create a foundational assessment of the method's 's performance. Expanding this evaluation to include metrics such as representation disentanglement could offer additional insights into the quality and utility of the synthetic data.
On a broader note, we believe it's important to examine synthetic data duttratengg procedures for possible biases, similar the scrutiny applied real datasets. Though we think this procedure can mitigate some of the biases in real datasets, different synthesizer architectures, values of 8, and other decisions might inadvertently produce performance gaps for different tasks, applications, and downstream populations of use. We evaluated on wide range of tasks in part to explore this possibility, but further evaluations would be helpful to assess these impacts.
CONCLUSION
Further improvements in auditory understanding depend greatly on the data underlying new models. In this work, we examined the value of synthetic data for learning representations of sound. We presented a method that perturbs the parameters of random synthetic sounds to generate audio doppelgängers, distinct yet similar sounds that provide strong signal for contrastive learning. Through a comprehensive set of experiments, we showed how this approach can yield strong results on a wide range of tasks. We will release code and models enable the community experiment with synthetic data sources for audio understanding, and hope this approach will help expand the machine learning toolkit for audio processing.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Manuel received the support of a fellowship from "la Caixa" Foundation (ID 100010434), The The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing resources that have contributed to the research results reported within this paper. We extend our heartfelt thanks to participants in the listening study.
fellowship code is LCF//QQ/EU23//0000077
REFERENCES
Haider Al-Tahan and Yalda Mohsenzadeh. Clar: Contrastive learning of auditory representations. In International Conference on Artificial Intelligence and Statistics, pp. 2530-2538. PMLR, 2021.
Ishwarya Ananthabhotla, David B Ramsay, and Joseph A Paradiso. Hcu400: An annotated dataset for exploring aural phenomenology through causal uncertainty. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 920-924 IEEE, 2019.
JA Ballas and MJ Sliwinski. Causal uncertainty in the identification of environmental sounds. Georgetown University, Washington, DC, 1986.
Manel Baradad, Richard Chen, Jonas Wulff, Tongzhou Wang. Rogerio Feris, Antonio Torralba, and Phillip Isola. Procedural image programs for representation learning. Advances in Neural Information Processing Systems, 35:6450-6462, 2022.
Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. Advances Neural Information Processing Systems, 34:2556-2563
2021.
Martijn Bartelds, Nay San, Bradley McDonnell, Dan Jurafsky, and Martijn Wieling. Making more of little data: Improving low-resource automatic speech recognition using data augmentation. arXiv preprint arXiv:2305. 10951, 2023.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2020.
Tal Boger, Ishwarya Ananthabhotla, and Joseph Paradiso. Manipulating causal uncertainty sound objects. In Proceedings of the 16th International Audio Mostly Conference, pp. 9-15, 2021.
Houwei Cao, David Cooper, Michael Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-soureed emotional multimodal actors dataset. IEEE transactions affective computing, 5(4):377-390, 2014.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Yuhua Chen, Wen Li, Xiaoran Chen, and Lue Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided onput-output adaptation approach. In Proceedings of the JEFF/CVF conference on computer vision and patlern recognition, pp. 1841-1850, 2019.
Zhehuai Chen, Andrew Rosenberg, Yu Zhang, Gary Wang. Bhuvana Ramabhadran, and Pedro J Moreno. Improving speech recognition using gan-based speech synthesis and contrastive unspoken text selection. In Interspeech, pp. 556-560, 2020b.
Manuel Cherep and Nikhil Singh. Synthax: A fast modular synthesizer in jax. In Audio Engineering Society Convention 155. Audio Engineering Society, 2023.
Manuel Cherep. Nikhil Singh, and Jessica Shand. Creative text-to-audio generation via synthesizer programming. In Forty-first International Conference Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 2758-2766, 2015.
August DuMont Schütte, Jürgen Hetzel, Sergios Gatidis, Tobias Hepp, Benedikt Dietz, Stefan Bauer, and Patrick Schwab. Overcoming barriers to data sharing with medical image generation: comprehensive evaluation. NPJ digital medicine, 4(1):141, 2021.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learning. pp. 1068-100100 1077. PMLR, 2017.
Amin Fazel, Wei Yang, Yulan Liu, Roberto Barra-Chicon Yixiong Meng, Roland Maas, and Jasha Droppo. Synthasr: Unlocking synthetic data for speech recognition. arXiv preprint arXiv:2106.07803 2021.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled, sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829 852, 2021a.
Eduardo Fonseca, Diego Ortego, Kevin McGuinness, Noel E O'Connor and Xavier Serra. Unsuper- vised contrastive learning sound event representations. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 371-375. IEEE, 2021b.
Heting Gao, Kaizhi Qian, Junrui Ni, Chuang Gan, Mark A Hasegawa-Johnson, Shiyu Chang, and Yang Zhang. Speech self supervuee learning using diffusion model synthetic data, In Forty-first International Conference on Machine Learning.
Yuan Gong, Yu-An Chung, and James Glass, Ast: Audio spectrogram transformer. arXiv preprint wrXiv:2104.01178, 2021.
Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Emmanouilidou. Adapting frechet audio distance for generative music evaluation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1331-1335. IEEE, 2024.
Masato Hagiwara, Maddie Cusimano, and Jen- Liu. Modeling animal vocalizations through synthesizers. arXiv preprint arXi::2210.1088 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Hermann von Helmholtz. Concerning the perceptions in general. 1867.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference acoustics, speech and signal processing (icassp), pp. 131-135. IEEE, 2017.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. Pmlr, 2018.
Natalie Holz, Pauline Marrouy-Maestee and David Poeppel. The variably intense vocalizations of affect and emotion (vivae) corpus prompts new perspective on nonspeech perception. Emotion, 22 (1):213, 2022.
12

---

Published as a conference paper at ICLR 2025
Ting- Yao Hu, Mohammadreza Armandpour, Ashish Shrivastava, Jen-Hao Rick Chang. Hema Kop- pula, and Oncel Tuzel. Synt++: Utilizing imperfect synthetic data to improve speech recognition. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 7682-7686. IEEE, 2022.
Qingqing Huang. Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208. /2415, 2022.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7): (725):335333333 2013.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. arXiv preprint 1.Xiv::2066052566 2021.
Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator. arXiv preprint ar arXiv:2103.09742, 2021.
Nicolas Jonason and Bob LT Sturm. Timbreclip: Connecting timbre to text and images. arXiv preprint arXiv:2211.11222 2022.
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. In Proceedings of the Asian Conference on Computer Vision, 2020.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr', 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint //200=18/. 2018.
Bongjun Kim, Madhay Ghei, Bryan Pardo, and Zhiyao Duan. Vocal imitation set: a dataset of vocally imitated sound events using the audioset ontology. In DCASE, pp. 148-152, 2018.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre- pre-trained transformer models. arXiv preprint arXiv:2003.0222 2020.
Moreno La Quatra, Alkis Koudounas, Lorenzo Vaiani, Elena Baralis, Paolo Garza, Luca Cagliero, and Sabato Marco Siniscalchi. Benchmarking representations for speech, music, and acoustic events. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), 2024.
Aleksandr Laptev, Roman Korostik, Aleksey Svischev, Andrei Andrusenko, Ivan Medennikov, and Sergey Rybin. You do not need more data: Improving end-to-end speech recognition by text-to- speech data augmentation. In 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEL pp.
IEEE, 2020.
Cyril Laurier, Owen Meyers, Joan Serra, Martin Blech, Perfecto Herrera, and Xavier Serra. Indexing music by mood: design and integration of an automatic content-basee based annotator. Multimedia Tools and Applications, 48: 48:161-184 2010.
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. $300-8311, 2021.
Xubo Liu, Egor Lakomkin, Konstantinos Vougioukas, Pingchuan Ma, Honglic Chen, Ruiming Xie, Morrie Doulaty, Niko Moritz, Jachym Kolar, Stavros Petridis, et al. Synthvsr: Scaling up visual speech recognition with synthetic supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18806-18815, 2023.
Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Mann Marnu, Tobin South, Cole Hunter, et al. Consent in crisis: The rapid decline of the ai data commons. arXiv preprint ar0iv:2407. 14933, 2024.
13

---

Published as a conference paper at ICLR 2025
Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. Cdpam: Contrastive learning for perceptual audio similarity. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 196-200. IEEE, 2021.
Keith D Martin and Youngmoo Kim. 2pmu9. musical instrument identification: A pattern- recognition approach. In Presented at the 136th meeting of the Acoustical Society of America, 1998.
Max V Mathews, Joan E Miller, F Richard Moore, John R Pierce, and Jean-Claude Risset. The technology of computer music, volume MIT press Cambridge, MA, 1969.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4040-4048, 2016.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462-477, 2022.
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for perproe-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint 1904 2019.
Geoffroy Peeters. A large set of audio features for sound description (similarity and classification) in the cuidado project. CUIDADO 1st Project Report, 54(0):1 1-05, 2004.
Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Trevor Pinch and Frank Trocco. Analog days: The invention and impact of the Moog synthesizer. Harvard University Press, 2004.
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space di-binsi In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 9301-9310, 2021.
Mirco Ravanelli and Yoshua Bengio. Learning speaker representations with mutual information. In Interspeech, pp. 1153-1157, 2019.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. Advances in neural information processing systems, 32, 2019.
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning using synthetic imagery. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 762-771, 2018
Julien Ricard. Towards computational morphological description of sound. DEA pre-thesis research work, Universitat Pompeu Fabra, Barcelona, 2004.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3234-3243, 2016.
Andrew Rosenberg, Yu Zhang, Bhuvana Ramabhadran, Ye Jia, Pedro Moreno, Yonghui Wu, and Zelin Wu. Speech recognition with augmented synthesized speech. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pp. 996-1002. IEFE, 2019.
14

---

Published as a conference paper at ICLR 2025
Nick Rossenbach, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Generating synthetic audio data for attention- speech recognition systems. ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7069-7073. IEEE, 2020.
Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general- audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3875-3879. IEEE, 2021.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, 2014.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Confer- on Machine Learning, pp. 00255562660 -1338. PMLR, 2019.
Shakhnarovich, Viola, and Darrell. Fast pose estimation with parameter-sensitive hashing. In Proceedings Ninth IEEE International Conference on Computer Vision, pp. 750-757. IEEE, 2003.
Jordie Shier. The synthesizer programming problem: improving the usability of sound synthesizers, 2021.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2107-2116, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Daria Soboleva, Ondrej Skopek. Márius Sajgalík, Victor Cārbune, Felix Weissenberger, Julia Proskur- nia, Bogdan Prisacari, Daniel Valcarce, Justin Lu, Rohit Prabhavalkar, et al. Replacing human audio with synthetic audio for on-device unspoken punctuation prediction. In ICASSP 2021-202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7653-7657. IEEE, 2021.
Janne Spijkervet and John Ashley Burgoyne. Contrastive learning of musical representations. arXiv preprint arXiv:21033 103.09400, 2021.
Fabian-Rober Stöter, Soumitro Chakrabarty, Emanuël Habets, and Berd Edler. Libricount, a dataset for speaker count estimation, 2018. URL htttt:://t/eeoo// wwtpreecor//1111107777700
Y2KKuUezMJ6c
Modan Tailleur, Junwon Lee, Mathieu Lagrange, Keunwoo Choi, Laurie M Heller, Keisuke Imoto, and Yuki Okamoto. Correlation of fr) 'echet audio distance with human perception of environmental audio is embedding dependant. arXiv preprint arXiv:2403.J 17508, 2024.
Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the fairness of deep generative models without retraining, arXiv preprint arXiv:2012.04842.. 2020.
Paul Théberge. Any sound you can imagine: Making music/consuminng technology. Wesleyan University Press, 1997.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data, arXiv preprint arXiv:2312.17742. 2023.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from eoxttto-mmage models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi- view redun- dancy, and linear models. In Algorithmic Learning Theory. pp. 1179-1206. PMLR, 2021.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmenta- tion with diffusion models. arXiv preprint arXiv:2.302.07944 2023.
Nontawat Tritrong. Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn. Repurposing gans for one-shot semantic part segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4475-4485, 2021.
Allan Tucker, Zhenchen Wang, Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. NPJ digital medicine, 3(1):1-13, 2020.
Joseph Turian, Jordie Shier, George Tzanetakis, Kirk McNally, and Max Henry. One billion audio sounds from gpu-enabled modular synthesis. In 202/ 24th International Conference on Digital Audio Effects (DAFx), pp. 222-229. IEEE, 2021.
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj. Bjöm W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al. Hear: Holistie evaluation of audio representations. In NeurIPS 2021 Competitions and Demonstrations Track, pp. 125-145. PMLR, 2022.
George Tzanetakis and Perry Cook. Multifeature audio segmentation for browsing and annotation. In Proceedings of the 1999 IEEE Workshop on Applications of Signal Processing to Audio and Acousties. WASPAA '99 (Cat. No. 99TH8452), pp. 106-106. IEEE, 1999.
Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 109-117, 2017.
Luyu Wang and Aaron van den Oord. Multi- contrastive learning of audio representations.
arXiv preprint arXiv:2 003006508. 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929-9939. PMLR, 2020.
Zilin Wang, Peng I. Liu, Jun Chen, Sipan Li, Jinfeng Bai, Gang He, Zhiyong Wu, and Helen Meng. A synthetic corpus generation method for neural vocoder training. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Tung- Yu Wu, Tsu- Yua- Hsu, Chen-An Li, Tzu-Han Lin, and Hung-yi Lee. The efficacy of self- supervised speech models for audio representations. In HEAR: Holistic Evaluation of Audio Representations, pp. 90-110. PMLR, 2022.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Kirk-arika, and Shlomo Dubnov. Large-scale contrastive language-and -uudio pretraining with feature fusion and seyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems, 34:9378 93990, 2021.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10145-10155, 2021.
Xianrui Zheng, Yulan Liu, Deniz Gunceler, and Daniel Willett. Using synthetic audio to improve the recognition of out-of-vocabulan words in end-to-end asr systems. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5674-5678. IEEE, 2021.
16

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS
A.1 COMPARISON OF DIFFERENT ARCHITECTURES ACROSS TASKS
In Figure 5 we show relative performance of models trained with data from different synthesizer architectures with 8 0.25. These results illustrate that, though Voice-gent Voice-generated sounds appear strongest overall, there is some task specialization of these different synthesis approaches. For example, LibriCount and NSynth, Voice is the lowest performer here.
70-
60
Synthesizer
Score (%)
ParametricSynth
40
20
US8K
ESC-50
LibriCount
CREMA-D
VIVAE
NSynth
FSD50k
Vocallmitation
Task
Figure 5: Scores with a fixed 8 = 0.25 and different synthesizer architectures for suite of tasks including (from left to right) UrbanSound8k (Salamon al., 2014). ESC-50 (Piczak 2015), Libri- Count (Stöter et al.) 2018). CREMA-D (Cao et al. 2014)) VIVAE (Holz et al., 2022). NSynth Pitch 5h (Engel et al. 2017). FSD50K (Fonseca et al. 2021a), and Vocal Imitation (Kim et al. 2018)
A.2 EFFECTS OF INCREASING PERTURBATION FACTOR 8 ON TRAINING
We seek to understand how increasing impacts the training dynamics. In particular, the alignment and uniformity objectives in tension (Wang & Isola 2020). A small 7 leads to easy positive pairs (high similarity), resulting in low alignment cost but potentially poor generalization. Conversely, too large 5 produces hard positive pairs (low similarity), increasing the alignment cost but potentially hindering optimization. The optimal ó should balance this trade-off, however the complexity of the synthesizer function and the embedding function make deriving closed-form solution for this infeasible. As such, we must explore the effect empirically.
Figure O shows the impact of different the final validation value the alignment and uniformity costs respectively. Alignment cost increases monotonically with ó, which shows the increased difficulty of aligning increasingly distant pairs. Uniformity has an inverted- inverted-U-shaped relationship with 6, suggesting that as the model struggles to align positives with moderate noise driven variation, incurs a cost uniformity in order to do so (e.g. creating clusters). With large 6, the amount of noise present is significant, alignment is difficult, and the representations can be more spread out. The theoretically optimal value of Lunif is 2t = 4, which values of d remain close to. In Figure
allign
Lunit
2.0
3.70
1.5
-3.5
-3.0
1.0
-3.85
0.5
-3.90
0.0
0.01
0.05
0.10
0.25
0.50
0.01
0.05
0.10
0.25
0.50
8
Figure Final validation scores showing the effect of 8 on Lutign and Lunif Latign increases monotonically with 8, since the difficulty of aligning more distinct samples goes up. Lunif, on the other hand, shows an Inverse-U-shapel relationship with
17

---

Published as a conference paper at ICLR 2025
8 of Wang & Isola (2000), the best performance with complex task and encoder (review classification) is observed when alignment is on the higher side (but not the maximum), and uniform low (close the optimal value). In our experiments, å 0.25 gets closest to this, and we observe it to be the strongest as well.
A.3 RESULTS FOR ALL VARIANTS
We give results for all synthetic model variants below, in Table
Data/Model
ESC
US8K
VIV
NSyn
C-D
FSD
LCount
External Baselines
HEAR/ARCH Top
96.65
79.09
44,28
87.80
75.21
65.48
22.69
78.53
HEAR/ARCH SSL
80.50
79.09
44.28
52.40
75.21
50.88
18.48
78.53
MS-CLAP Linear
89,95
82.29
23.15
50.24
54.51
GURA (HEAR)
74.35
38.20
75.21
41.32
18.48
68.34
VGGSound Sup.
87.45
77.57
39.38
43.80
54.36
43.76
14.06
56.10
Internal Baselines
Random Init.
22.45
55.03
33.81
36.20
38.91
9.03
2.43
44.91
Voice (Ours, No-5, Aug.)
48.65
59.46
36.31
32.80
46.32
16.88
7.12
47.64
VGGSound SSL (Aug.)
48.85
61.91
32.67
39.60
47.86
19.63
6.03
53.46
VGGSound SSL (Jitter)
52.95
63.82
38.12
14.20
50.03
24.02
3.43
69.77
VGGSound-
43.95
$9.69
33.31
40.80
46.10
14.71
$2.57
VGGSound Mix 10s
42.95
57.40
32.03
40.20
46.57
15.77
6.43
51.07
Audio Doppelgängers (Ours)
Best Synthetic
58.90
66.71
39.45
44.40
48.43
24.12
9.15
58.60
Voice (6 0.01)
47.55
59.56
38.62
11.40
47.53
17.15
55.56
Voice (6 = 0.05)
47.90
64.02
37.93
13.80
46.45
17.77
7.72
51.52
Voice (6 0.10)
48.40
63.92
38.74
11.40
45.13
18.40
7.67
49.32
Voice (ō 0.25)
58.90
66.71
39.45
32.20
48.24
24.12
9.15
52.95
Voice (ô 0.50)
41.85
54.03
28.54
40.60
45.78
4.69
43.85
VoiceFM (6 0.01)
42.40
59.89
36.58
9.20
44.31
15.34
5.15
57.13
VoiceFM (5 0.05)
42,90
62.96
36.54
14.20
44.93
15.64
5.79
50.61
VoiceFM (5 0.10)
44.80
62.03
35.73
14.80
43.99
15.67
5.60
50.56
VoiceFM (6 0.25)
57.20
65.11
38.48
35.20
48.43
22.15
6.96
$4.00
VoiceFM (6 0.50)
43.50
60.98
39.04
12.20
44.17
15.25
$1.07
Parametric (6 0.01)
39.50
58.95
36.87
12.20
42.16
13.92
4.53
58.60
Parametric (6 0.05)
40.15
$7.22
35.11
14.60
42.65
12.87
4.78
55.37
Parametric (ō 0.10)
42.50
59.65
34.12
14.20
43.01
13.41
4.97
53.43
Parametric (ô 0.25)
50.55
62.83
37.91
37.60
46.77
5.70
54.72
Parametric (ō 0.50)
41.15
56.86
35.41
10.40
41.73
12.76
4.48
54.27
Voice (8 0.01, Aug.)
52,55
62.92
34.82
23.60
46.96
18.18
8.17
51.01
Voice (8 0.05, Aug.)
53.00
65.17
34.49
19.40
45.39
19.79
8.32
49.84
Voice (8 0.10, Aug.)
54.20
65.89
33.78
23.40
45.71
20.38
8.50
50.42
Voice (8 0.25, Aug.)
58.75
65.01
34.81
44.40
46.17
21.76
8.54
50.70
Voice (8 0.50, Aug.)
32.25
48.40
25.41
36.20
41.38
11.82
3.26
44.74
Table 3: Complete results for all model variants.
B ADDITIONAL DETAILS ON TRAINING
B.1 AUGMENTATION BATCHING
Due to practical considerations batching and memory management, augmentations are applied differently for real and synthetic data. In real data, augmentations applied per-example within distributed data-loading workers. Synthetic data is batch-generated within the main process to avoid concurrency issues between JAX's 's multithreading and PyTorch' 's data loading. Individually augmenting examples in this synthetic data environment is prohibitively slow. As solution, we mini- batched augmentations with default size < 100. This allows memory vemory-efficiently leverage GPU
18

---

Published as a conference paper at ICLR 2025
processing and introduces variation within each training batch. While per-example augmentations might further enhance performance synthetic data with augmentations, we believe our current approach is conservative yet effective option and expect minimal impact.
19


Reviews:
[{"review_id": "643nplrpNQ", "reviewer": "Reviewer_1zPK", "summary": "The authors propose contrastive learning from synthetic audio doppelgängers using audio synthesizers that can provide rich source of information that other augmentations fail to provide. Additionally they say that the approach is light weight and requires no extra data storage and only has a single hyper-parameter. They propose that this complements existing strategies and is reducing the data burden on practitioners.", "questions": "Please do not carry out extra experiments and make revisions to the paper. Answering these questions would be enough.\n\n== The paper hinges on synthesis perturbation factor delta and makes the assumption that the parameter lie in a straight-line as explained in section 3.1. What is the evidence of this. What if the parameter space theta have a gaussian distribution or any other distribution. What would happen if I perturbed parameter vectors in any other manner. Is this the right way to perturb parameter. \n\n== Why was the paper not compared against different contrastive learning algorithms. How about SimCLR. Would such methods improve ? The author should the taken the Resnet baseline and done the exact same thing with existing augmentations and SimCLR. \n\n== Why are the gold standard numbers so poor ? FSD-50K paper mentions that the setup that does not include data augmentation including well cited Audio Transformers paper on it. So a ResNet-18 should perform well for MAP metric in the internal baseline and not be so off so as to report 9.03 ?\n\n== why having a single parameter a good thing. Why would doing 100s of different augmentation e.g. audiomentation be a bad thing ? It seems like the augmentations chosen to compare against were weak. Why were all the augmentations not carried out ? \n\n== There are several papers solely focussed on audio augmentation. Why were the baselines not compared against audio augmentation papers  ? Why was augmentations such as random mix or others not carried out ? \n\n== The algorithm itself would only learn to discern in the positive cases similar sounding what are different but from the same category and in the negative cases noisy versions of the same category. However a typical contrastive learning algorithm e.g .SimCLR also takes into account audio files that are not noisy but from a different category. How would the authors consider these kinds of negative examples ? Is this approach better or worse than a typical contrastive learning approach e.g .SimCLR ? \n\n== Why would making the distributions similar in Figure 3 be needed? One would rather argue that the model should retain the same category but have the features described in Figure 3 e.g. inharmonicity be as much varied as possible as this can then account for out of distribution categories as well as make the audio representations robust and generalizable.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "== The idea appears neat. Ideally all of the real world audio files should reside in a small latent space. We can sample the latent space to get as many audio files as we want. We could use contrastive learning to learn robust audio representations. \n\n== Decent presentation. The author propose a technique present almost all the necessary that is needed for it. They then give benchmarks on standard datasets. \n\n== The algorithm is well explained easy to follow. The paper is a welcome change from LLM and Transformers based approaches that are just going after scale or yet another prompting paper.", "weaknesses": "== It took some digging around to figure out how Voice synthesize with 78 parameters able to do audio synthesis of specific categories. If it is using \"Creative Text-to-Audio Generation via Synthesizer Programming\" the authors should specifically mention it in section 3.1. It defiantly makes the content of the paper self contained and helps understanding it better. This make it poorly reproducible. It would have been great to add the synthesis technique to the paper. \n\n== Contributions mentioned on Page 2 point 2 are not correct and misleading. There are actually several papers that are doing exact same thing -- using synthetic data in some form for doing audio classification. They should be cited. Here are a few. i) Open-Set Tagging Dataset (OST) Mark Cartwright, ii) BLAT Bootstrapping language-audio pertaining based audio set .... iii) Can synthetic audio from generative foundational models assist audio recognition ....iv) Synthetic training set generation using T2A models for environmental sound classification. There exist many more for audio generation tasks which I am not including here but the authors should have included.\n\n== The abstract mentions reducing the data burden. This is not true. We can define several augmentation strategies that can generate augmentations on the fly e.g. in Audiomentations and it is a matter of implementation detail and not about how we store all of the data.\n\n== The main issue I have is with the algorithm itself. The authors mention that delta parameter is tuned and there exists a sweet spot with a high delta corresponding to noise and lower delta make the sound exactly the same with little diversity. Please comment on this. The algorithm itself would only learn to discern in the positive cases similar sounding what are different but from the same category and in the negative cases noisy versions of the same category. However a typical contrastive learning argument e.g .SimCLR also takes into account audio files that are not noisy but yet from a different category. How would the authors consider these kinds of negative examples ? Is this approach better or worse than a typical contrastive learning approach e.g .SimCLR ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pi9gnaSWKW", "reviewer": "Reviewer_kgdM", "summary": "This paper proposes to rely exclusively on synthetic audio to learn representations with contrastive learning.\nThe originality of the approach is that it does not rely on audio augmentations (as in SimCLR) directly applied on the audio to create positive pairs, but on generating pairs of audio using a synthesizer with close parameters.\nThis method, although simple, manages to be on par with some of the SOTA methods on some tasks HEAR and ARCH.", "questions": "- 1 second -long chunks looks quite short to learn interesting representations\n- Why using the alignment and uniformity objective from Wang & Isola over other choices like the one in SimCLR or SigLIP?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The article is well-written and detailed. \nThe related work section is quite informative, especially the 2.1§.\nAlthough the idea is pretty simple and straightforward, it is evaluated across multiple tasks, mainly against a more classic baseline trained on VGGSound + augmentations, and show good results, despite being trained solely on synthesized sound from a bespoke synthesizer.", "weaknesses": "Only one parameter delta to control the amount of variation for all parameters looks quite restrictive. If this is addressed in the Limitations section, it's still surprising that considering that the $f_0$ and the LFO parameters range are treated the same way. \n- Results, if interesting, are not beating existing methods, the analysis is extensive an varied.\n- This article may be more appropriate for", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MZQ3PFG7i3", "reviewer": "Reviewer_Y8f6", "summary": "This paper introduces a novel contrastive learning approach for audio representation learning using synthetic data. Instead of augmenting real recordings, the authors leverage a modular synthesizer (SYNTHAX) to generate pairs of synthetic audio \"doppelgangers\" by randomly perturbing synthesis parameters (controlling timbre, pitch, and envelopes) by a factor \\delta. This method creates positive pairs with causally controlled variations, providing rich contrastive information. Training a ResNet18 encoder with alignment and uniformity objectives on these synthetic pairs, the authors demonstrate performance exceeding that of models trained on real data (VGGSound) with augmentations across 8 diverse audio classification benchmarks, while also reducing training time. Analysis reveals the synthetic data exhibits higher spectral complexity and flux, and greater causal uncertainty compared to real audio, potentially contributing to its effectiveness. The hyperparameter \\delta, controlling the dissimilarity between doppelganger pairs, is shown to significantly impact performance, with \\delta = 0.25 generally yielding optimal results across tasks.", "questions": "While the paper demonstrates performance improvements, further investigation into the specific properties of the learned representations (e.g., through probing or visualization) and a comparison with alternative self-supervised methods applied to the same synthetic data would provide a more complete understanding of the contributions of the contrastive framework itself. Is it simply the paired structure and controlled variations that are beneficial, or is the contrastive loss crucial?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a novel approach to contrastive learning in audio by utilizing entirely synthetic data. This is a significant departure from existing methods that rely on augmenting real recordings and opens up a new avenue for research in self-supervised audio representation learning. The demonstrated performance gains over real data with augmentations highlight the potential impact of this approach.\n\n2. The paper is technically sound, with a clear methodology, appropriate experimental setup, and thorough analysis. The authors carefully consider the impact of the perturbation factor (\\delta) and provide insightful comparisons between the synthetic and real data distributions, including analysis of features like spectral complexity and causal uncertainty.\n\n3. The proposed method offers practical benefits, particularly in terms of reduced data requirements and faster training times. By eliminating the need for large-scale real-world audio datasets, the approach can significantly reduce the data burden on practitioners and potentially democratize access to high-quality audio representation learning. This has implications for resource-constrained settings and applications where data privacy is a concern.", "weaknesses": "1. The authors claim a self-supervised approach, but their method relies on a pre-trained SYNTHAX model. SYNTHAX itself likely required supervised training with labeled data, making the overall approach dependent on prior supervision. The authors should avoid using terms like \"self-supervised\" and clearly acknowledge the role of supervised pre-training in their method. The augmentations applied are also inherently limited to the codomain of the pre-trained SYNTHAX model, potentially restricting the diversity of generated sounds.\n\n2. The evaluation focuses solely on detection tasks (classification). While the paper's introduction suggests a more general contribution to audio representation learning,  the lack of estimation or generation tasks in the downstream evaluation limits the scope of the demonstrated benefits.  Including a broader range of downstream tasks would strengthen the claims of general-purpose representation learning.\n\n3. The reliance on a specific synthesizer architecture (SYNTHAX) and the parameterized perturbation method could introduce biases in the synthetic data distribution. While the authors analyze some aspects of this distribution, a more thorough investigation of potential biases and their impact on downstream performance across different audio domains and tasks is warranted. The paper would benefit from a discussion of the limitations of the chosen synthesizer and perturbation method and their potential influence on the generality of the learned representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sopLdgEmD6", "reviewer": "Reviewer_gLEZ", "summary": "This paper introduces a novel approach to contrastive learning for audio representations using synthetic audio doppelgängers. By generating paired synthetic audio samples with controlled variations through sound synthesizer parameter perturbations, the authors address the limitations of traditional data augmentation techniques. This method surpasses results from large-scale audio datasets in several tasks, offering a lightweight, hyperparameter-efficient solution that requires no data storage. The synthetic data approach provides a rich source of contrastive information that enhances the model's ability to learn robust audio representations. The paper's contributions are well-supported by comprehensive experiments across multiple audio classification benchmarks.", "questions": "1. How does the choice of synthesizer architecture affect the quality of the learned representations? Is there a significant difference in performance across different synthesizers beyond the three tested?\n2. Could the approach be extended to longer audio sequences, and what impact would this have on the quality of representations and training time?\n3. Is there potential for further improving performance by using more complex perturbation strategies, such as adaptive or learned perturbations?\n4. How robust is the method to variations in delta across different downstream tasks? Would an adaptive approach to tuning δ during training yield better results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The synthetic audio doppelgänger concept introduces a unique and controlled approach to creating positive pairs for contrastive learning, which has not been explored in this way for audio data. Comprehensive experiments across multiple benchmarks substantiate the effectiveness of the approach, showing competitive performance with or even surpassing real-data baselines. This method reduces the need for large, labeled datasets, offering a scalable, lightweight solution suitable for a variety of audio applications.", "weaknesses": "While the simplicity of using a single hyperparameter δ is a strength, it could also be a limitation. The performance is sensitive to the choice of delta, and further analysis could be helpful in understanding how to optimize this parameter across different tasks. The paper mentions the potential for hybrid approaches combining synthetic and real data but does not explore this in detail. A comparison with hybrid training strategies could provide more insights into the benefits of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)