Contrastive Learning from Synthetic Audio Doppelgängers

Authors: Manuel Cherep, Nikhil Singh
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: synthetic data, audio, contrastive learning, representation learning

Abstract:
Learning robust audio representations currently demands extensive datasets of real-world sound recordings. By applying artificial transformations to these recordings, models can learn to recognize similarities despite subtle variations through techniques like contrastive learning. However, these transformations are only approximations of the true diversity found in real-world sounds, which are generated by complex interactions of physical processes, from vocal cord vibrations to the resonance of musical instruments. We propose a solution to both the data scale and transformation limitations, leveraging synthetic audio. By randomly perturbing the parameters of a sound synthesizer, we generate audio doppelgängers—synthetic positive pairs with causally manipulated variations in timbre, pitch, and temporal envelopes. These variations, difficult to achieve through augmentations of existing audio, provide a rich source of contrastive information. Despite the shift to randomly generated synthetic data, our method produces strong representations, outperforming real data on several standard audio classification tasks. Notably, our approach is lightweight, requires no data storage, and has only a single hyperparameter, which we extensively analyze. We offer this method as a complement to existing strategies for contrastive learning in audio, using synthesized sounds to reduce the data burden on practitioners.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730492379357

thread_type:
official_review

<thread_object>
creator: reviewer_glez

<content>
summary:
This paper introduces a novel approach to contrastive learning for audio representations using synthetic audio doppelgängers. By generating paired synthetic audio samples with controlled variations through sound synthesizer parameter perturbations, the authors address the limitations of traditional data augmentation techniques. This method surpasses results from large-scale audio datasets in several tasks, offering a lightweight, hyperparameter-efficient solution that requires no data storage. The synthetic data approach provides a rich source of contrastive information that enhances the model's ability to learn robust audio representations. The paper's contributions are well-supported by comprehensive experiments across multiple audio classification benchmarks.

strengths:
The synthetic audio doppelgänger concept introduces a unique and controlled approach to creating positive pairs for contrastive learning, which has not been explored in this way for audio data. Comprehensive experiments across multiple benchmarks substantiate the effectiveness of the approach, showing competitive performance with or even surpassing real-data baselines. This method reduces the need for large, labeled datasets, offering a scalable, lightweight solution suitable for a variety of audio applications.

weaknesses:
While the simplicity of using a single hyperparameter δ is a strength, it could also be a limitation. The performance is sensitive to the choice of delta, and further analysis could be helpful in understanding how to optimize this parameter across different tasks. The paper mentions the potential for hybrid approaches combining synthetic and real data but does not explore this in detail. A comparison with hybrid training strategies could provide more insights into the benefits of the proposed method.

questions:
1. How does the choice of synthesizer architecture affect the quality of the learned representations? Is there a significant difference in performance across different synthesizers beyond the three tested?
2. Could the approach be extended to longer audio sequences, and what impact would this have on the quality of representations and training time?
3. Is there potential for further improving performance by using more complex perturbation strategies, such as adaptive or learned perturbations?
4. How robust is the method to variations in delta across different downstream tasks? Would an adaptive approach to tuning δ during training yield better results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730499876841

thread_type:
official_review

<thread_object>
creator: reviewer_y8f6

<content>
summary:
This paper introduces a novel contrastive learning approach for audio representation learning using synthetic data. Instead of augmenting real recordings, the authors leverage a modular synthesizer (SYNTHAX) to generate pairs of synthetic audio "doppelgangers" by randomly perturbing synthesis parameters (controlling timbre, pitch, and envelopes) by a factor \delta. This method creates positive pairs with causally controlled variations, providing rich contrastive information. Training a ResNet18 encoder with alignment and uniformity objectives on these synthetic pairs, the authors demonstrate performance exceeding that of models trained on real data (VGGSound) with augmentations across 8 diverse audio classification benchmarks, while also reducing training time. Analysis reveals the synthetic data exhibits higher spectral complexity and flux, and greater causal uncertainty compared to real audio, potentially contributing to its effectiveness. The hyperparameter \delta, controlling the dissimilarity between doppelganger pairs, is shown to significantly impact performance, with \delta = 0.25 generally yielding optimal results across tasks.

strengths:
1. The paper presents a novel approach to contrastive learning in audio by utilizing entirely synthetic data. This is a significant departure from existing methods that rely on augmenting real recordings and opens up a new avenue for research in self-supervised audio representation learning. The demonstrated performance gains over real data with augmentations highlight the potential impact of this approach.

2. The paper is technically sound, with a clear methodology, appropriate experimental setup, and thorough analysis. The authors carefully consider the impact of the perturbation factor (\delta) and provide insightful comparisons between the synthetic and real data distributions, including analysis of features like spectral complexity and causal uncertainty.

3. The proposed method offers practical benefits, particularly in terms of reduced data requirements and faster training times. By eliminating the need for large-scale real-world audio datasets, the approach can significantly reduce the data burden on practitioners and potentially democratize access to high-quality audio representation learning. This has implications for resource-constrained settings and applications where data privacy is a concern.

weaknesses:
1. The authors claim a self-supervised approach, but their method relies on a pre-trained SYNTHAX model. SYNTHAX itself likely required supervised training with labeled data, making the overall approach dependent on prior supervision. The authors should avoid using terms like "self-supervised" and clearly acknowledge the role of supervised pre-training in their method. The augmentations applied are also inherently limited to the codomain of the pre-trained SYNTHAX model, potentially restricting the diversity of generated sounds.

2. The evaluation focuses solely on detection tasks (classification). While the paper's introduction suggests a more general contribution to audio representation learning,  the lack of estimation or generation tasks in the downstream evaluation limits the scope of the demonstrated benefits.  Including a broader range of downstream tasks would strengthen the claims of general-purpose representation learning.

3. The reliance on a specific synthesizer architecture (SYNTHAX) and the parameterized perturbation method could introduce biases in the synthetic data distribution. While the authors analyze some aspects of this distribution, a more thorough investigation of potential biases and their impact on downstream performance across different audio domains and tasks is warranted. The paper would benefit from a discussion of the limitations of the chosen synthesizer and perturbation method and their potential influence on the generality of the learned representations.

questions:
While the paper demonstrates performance improvements, further investigation into the specific properties of the learned representations (e.g., through probing or visualization) and a comparison with alternative self-supervised methods applied to the same synthetic data would provide a more complete understanding of the contributions of the contrastive framework itself. Is it simply the paired structure and controlled variations that are beneficial, or is the contrastive loss crucial?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671824315

thread_type:
official_review

<thread_object>
creator: reviewer_kgdm

<content>
summary:
This paper proposes to rely exclusively on synthetic audio to learn representations with contrastive learning.
The originality of the approach is that it does not rely on audio augmentations (as in SimCLR) directly applied on the audio to create positive pairs, but on generating pairs of audio using a synthesizer with close parameters.
This method, although simple, manages to be on par with some of the SOTA methods on some tasks HEAR and ARCH.

strengths:
The article is well-written and detailed. 
The related work section is quite informative, especially the 2.1§.
Although the idea is pretty simple and straightforward, it is evaluated across multiple tasks, mainly against a more classic baseline trained on VGGSound + augmentations, and show good results, despite being trained solely on synthesized sound from a bespoke synthesizer.

weaknesses:
Only one parameter delta to control the amount of variation for all parameters looks quite restrictive. If this is addressed in the Limitations section, it's still surprising that considering that the $f_0$ and the LFO parameters range are treated the same way. 
- Results, if interesting, are not beating existing methods, the analysis is extensive an varied.
- This article may be more appropriate for

questions:
- 1 second -long chunks looks quite short to learn interesting representations
- Why using the alignment and uniformity objective from Wang & Isola over other choices like the one in SimCLR or SigLIP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731097470512

thread_type:
official_review

<thread_object>
creator: reviewer_1zpk

<content>
summary:
The authors propose contrastive learning from synthetic audio doppelgängers using audio synthesizers that can provide rich source of information that other augmentations fail to provide. Additionally they say that the approach is light weight and requires no extra data storage and only has a single hyper-parameter. They propose that this complements existing strategies and is reducing the data burden on practitioners.

strengths:
== The idea appears neat. Ideally all of the real world audio files should reside in a small latent space. We can sample the latent space to get as many audio files as we want. We could use contrastive learning to learn robust audio representations. 

== Decent presentation. The author propose a technique present almost all the necessary that is needed for it. They then give benchmarks on standard datasets. 

== The algorithm is well explained easy to follow. The paper is a welcome change from LLM and Transformers based approaches that are just going after scale or yet another prompting paper.

weaknesses:
== It took some digging around to figure out how Voice synthesize with 78 parameters able to do audio synthesis of specific categories. If it is using "Creative Text-to-Audio Generation via Synthesizer Programming" the authors should specifically mention it in section 3.1. It defiantly makes the content of the paper self contained and helps understanding it better. This make it poorly reproducible. It would have been great to add the synthesis technique to the paper. 

== Contributions mentioned on Page 2 point 2 are not correct and misleading. There are actually several papers that are doing exact same thing -- using synthetic data in some form for doing audio classification. They should be cited. Here are a few. i) Open-Set Tagging Dataset (OST) Mark Cartwright, ii) BLAT Bootstrapping language-audio pertaining based audio set .... iii) Can synthetic audio from generative foundational models assist audio recognition ....iv) Synthetic training set generation using T2A models for environmental sound classification. There exist many more for audio generation tasks which I am not including here but the authors should have included.

== The abstract mentions reducing the data burden. This is not true. We can define several augmentation strategies that can generate augmentations on the fly e.g. in Audiomentations and it is a matter of implementation detail and not about how we store all of the data.

== The main issue I have is with the algorithm itself. The authors mention that delta parameter is tuned and there exists a sweet spot with a high delta corresponding to noise and lower delta make the sound exactly the same with little diversity. Please comment on this. The algorithm itself would only learn to discern in the positive cases similar sounding what are different but from the same category and in the negative cases noisy versions of the same category. However a typical contrastive learning argument e.g .SimCLR also takes into account audio files that are not noisy but yet from a different category. How would the authors consider these kinds of negative examples ? Is this approach better or worse than a typical contrastive learning approach e.g .SimCLR ?

questions:
Please do not carry out extra experiments and make revisions to the paper. Answering these questions would be enough.

== The paper hinges on synthesis perturbation factor delta and makes the assumption that the parameter lie in a straight-line as explained in section 3.1. What is the evidence of this. What if the parameter space theta have a gaussian distribution or any other distribution. What would happen if I perturbed parameter vectors in any other manner. Is this the right way to perturb parameter. 

== Why was the paper not compared against different contrastive learning algorithms. How about SimCLR. Would such methods improve ? The author should the taken the Resnet baseline and done the exact same thing with existing augmentations and SimCLR. 

== Why are the gold standard numbers so poor ? FSD-50K paper mentions that the setup that does not include data augmentation including well cited Audio Transformers paper on it. So a ResNet-18 should perform well for MAP metric in the internal baseline and not be so off so as to report 9.03 ?

== why having a single parameter a good thing. Why would doing 100s of different augmentation e.g. audiomentation be a bad thing ? It seems like the augmentations chosen to compare against were weak. Why were all the augmentations not carried out ? 

== There are several papers solely focussed on audio augmentation. Why were the baselines not compared against audio augmentation papers  ? Why was augmentations such as random mix or others not carried out ? 

== The algorithm itself would only learn to discern in the positive cases similar sounding what are different but from the same category and in the negative cases noisy versions of the same category. However a typical contrastive learning algorithm e.g .SimCLR also takes into account audio files that are not noisy but from a different category. How would the authors consider these kinds of negative examples ? Is this approach better or worse than a typical contrastive learning approach e.g .SimCLR ? 

== Why would making the distributions similar in Figure 3 be needed? One would rather argue that the model should retain the same category but have the features described in Figure 3 e.g. inharmonicity be as much varied as possible as this can then account for out of distribution categories as well as make the audio representations robust and generalizable.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Thanks for your submission to ICLR.

This paper introduces a novel approach to contrastive learning for audio representations using synthetic audio doppelgängers.  The reviewers were generally positive about this paper going into the rebuttal/discussion, with three of the four reviewers providing mostly positive reviews.  On the plus side, the reviewers noted that the idea was clever, easy to follow, and useful; and that the presentation was good.  On the negative side, there were various questions about the algorithm and evaluation (e.g., the use of a single hyperparameter was noted by multiple reviewers).

Despite prodding from the AC, the reviewers did not engage much with the discussion.  In particular, the most negative reviewer did not address the rebuttal.  I read through the rebuttals and reviews, and felt that they adequately addressed the concerns, particularly of the more negative reviewer.  Further, there was nothing in these concerns that seemed to warrant not accepting the paper.  Indeed, the paper should be of interest to many in the community, particularly those working in audio.

Please keep in mind the reviewer comments when preparing a final version of the manuscript.