SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation
Authors: Koichi Saito
Dongjun Kim
Takashi Shibuya
Chieh-Hsin Lai
Zhi Zhong
Yuhta Takida
Yuki Mitsufuji
Venue: ICLR 2025 Poster (2025)
Keywords: text-to-sound generation
distillation models
text-to-audio diffusion models
generative models for sound
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Introduce Sound Consistency Trajectory Models, a first large-scale distillation model (1B paramters) in the sound community to achieve both promising high-quality 1-step and multi-step full-band generation

Abstract: Sound content creation, essential for multimedia works such as video games and films, often involves extensive trial-and-error, enabling creators to semantically reflect their artistic ideas and inspirations, which evolve throughout the creation process, into the sound.
Recent high-quality diffusion-based Text-to-Sound (T2S) generative models provide valuable tools for creators. However, these models often suffer from slow inference speeds, imposing an undesirable burden that hinders the trial-and-error process.
While existing T2S distillation models address this limitation through $1$-step generation, the sample quality of $1$-step generation remains insufficient for production use.
Additionally, while multi-step sampling in those distillation models improves sample quality itself, the semantic content changes due to their lack of deterministic sampling capabilities.
Thus, developing a T2S generative model that allows creators to efficiently conduct trial-and-error while producing high-quality sound remains a key challenge.
To address these issues, we introduce Sound Consistency Trajectory Models (SoundCTM), which allow flexible transitions between high-quality $1$-step sound generation and superior sound quality through multi-step deterministic sampling. 
This allows creators to efficiently conduct trial-and-error with $1$-step generation to semantically align samples with their intention, and subsequently refine sample quality with preserving semantic content through deterministic multi-step sampling.
To develop SoundCTM, we reframe the CTM training framework, originally proposed in computer vision, and introduce a novel feature distance using the teacher network for a distillation loss. 
Additionally, while distilling classifier-free guided trajectories, we introduce a $\nu$-sampling, a new algorithm that offers another source of quality improvement. For the $\nu$-sampling, we simultaneously train both conditional and unconditional student models.
For production-level generation, we scale up our model to 1B trainable parameters, making SoundCTM-DiT-1B the first large-scale distillation model in the sound community to achieve both promising high-quality $1$-step and multi-step full-band (44.1kHz) generation.
Audio samples are available at \url{https://anonymus-soundctm.github.io/soundctm_iclr/}.

Published as a conference paper at ICLR 2025
SOUNDCTM: UNIFYING SCORE-BASED AND CONSIS- TENCY MODELS FOR FULL-BAND TEXOUNO GENERATION
Koichi Saito'
Dongjun Kim
Takashi Shibuya' Yuki MitsufujiL.3 "Sony Group Corporation
Chich-Hsin Lai*
Zhi Zhong Yuhta Takida*
Sony AI Stanford University
Koichi Sa toteeoonn............ com
ABSTRACT
Sound content creation, essential for multimedia works such as video games and films, often involves extensive trial-and-error, enabling creators to semantically reflect their artistic ideas and inspirations, which evolve throughout the creation process, into the sound. Recent high-quality diffusion-based Text-to-Sound (T2S) generative models provide valuable tools for creators. However, these models often suffer from slow inference speeds, imposing an undesirable burden that hinders the trial-and-error process. While existing T2S distillation models address this limi- tation through 1-step generation, the sample quality of 1-step generation remains insufficient for production use. Additionally, while multi-step sampling in those distillation models improves sample quality itself, the semantic content changes due to their lack of deterministic sampling capabilities. Thus, developing T2S generative model that allows creators to efficiently conduct trial-and-error while producing high-quality sound remains a key challenge. To address these issues, we introduce Sound Consistency Trajectory Models (SoundCTM), which allow flexible transitions between high-quality 1-step sound generation and superior sound quality through multi-step deterministic sampling. This allows creators to efficiently conduct trial-and-error with 1-step generation to semantically align sam- ples with their intention, and subsequently refine sample quality with preserving semantic content through deterministic multi-step sampling. To develop Sound- CTM, reframe the CTM training framework, originally proposed in computer vision, and introduce a novel feature distance using the teacher network for a distillation loss. Additionally, while distilling classifier-fre guided trajectories, we introduce a sammmngg a new algorithm that offers another source of quality improvement. For the u-sampling, we simultaneously train both conditional and unconditional student models. For production-level generation, we scale up our model to IB trainable parameters, making SoundCTM-DiT. IB the first large-scale distillation model in the sound community to achieve both promising high-quality 1-step and multi-step full-band (44.1kHz) generation. Audio samples are available at https: : //////nooonmss// soundetm. g thub. io/ wtts//ouuunmmllllcceeee
INTRODUCTION
Sound contents play pivotal role in multimedia experiences, such as video games, music, and films. In video game development, for example, sound creators and foley artists meticulously craft sound contents like footsteps, dragon roars, and ambient bird chirps to enhance the immersive quality of gameplay. Recently, Text-to-Sound (T2S) generative models based on Latent Diffusion Models (LDMs) (Rombach et al., 2022), such as Stable Audio (Evans et al., 2024a;c) and AMdioLDM2- 48kHz (Liu et al., 2024b), demonstrating full-band sound generation in either 44.1 kHz or 48 kHz formats, which is required for real-world applications, have emerged as appealing tools for streamlin- ing the sound production process. Despite their potential, these DMs (Sohl-Dickster et al., 2015; Song et al., 2021b; Karras et al., 2022) are computationally intensive, making challenging to swiftly modify and refine sound content align with creators' continuously evolving artistic inspiration,

---

Published as a conference paper at ICLR 2025
SoundCTM-DiI-IB w/. deterministic sampling
Human evaluation on full-band T25 generation
Step
Step
Step
Score
Inforanc speedt
Infenence- uponeddd
2.5
Diecgl oud oiguakty
Trial
Refinement
Prompt: woman speaks then small child speaks."
Teacher DIT (NFE=79) SoundCTM-Dir-18 (NFE-1)
Figure I: SoundCTM-DiT-IB is first model that achieves high-quality 1-step and higher-quality multi- step full-band T2S generation while preserving semantic content through deterministic sampling, enabling creators to efficiently carry out the model.
creation process within single
which is influenced by various triggers, such as the intermediate sounds produced during creation and shifts in their viewpoint toward the desired sound. This paper addresses the issue of slow inference speeds in T2S models and seeks to enhance their editability and practical application.
Sound creation typically involves a significant trial-and-erron process, whether through mixing and splicing digital sounds from extensive high-qualin, sound libraries or recording physical objects. This trial-and-error process, while tmnecoosumnng. is crucial for creators to semantically reflect their changing ideas and inspirations throughout the creation, driven by their vision of the experience they want to deliver to consumers, into the sound content.
When applying sound generative models to the sound creation for production use, existing LDM-based full-band T2S models can leveraged for their high-quality generated sounds. However, due to their slow inference speeds, these models struggle to efficiently accommodate the rrial-and-error process, posing an undue burden on creators. While these models suffer from slow inference speeds, recent Consistency Distillation (CD) (Song al., 2023)-based T2S models, such as ConsistencyTTA (Bai et al., 2023) and AudioLCM (Liu et al., 2024c), offer the potential to accelerate the trial-and-error process with their 1-step generation. However, the sample quality of their 1-step generation remains insufficient for production-level use.
To integrate sound generative models into real-world sound creation, aim propose single model that efficiently facilitates a trial-and-refinement creation process, where creators can first conduct trials with 1-step generation and, once the semantically desired content is obtained, refine the sample quality through multi-step deterministic sampling while preserving the semantic content. true that existing CD-based distillation models can improve the sound quality itself through multi-step sampling. However, the refinement use while preserving their semantic content after the trial phase remains challenging with those distillation models. This difficulty stems from the fact that deterministic sampling, essential for maintaining semantic content, is not feasible due to their CD's training regime, which only learns anytime-to-zen coo-zerootinne time jumps. Therefore, developing new model that can be applied efficiently to both the trial and refinement phases remains a critical challenge for sound generative models.
-where sounds are further refined for production
To address this challenge. in this paper, we present the Sound Consistency Trajectory Model (SoundCTM), a novel T2S model that enables flexible switching between 1-step high-quality sound generation and higher-quality multi-step generation with deterministic sampling, allowing creators to efficiently perform the trial-anddrefinemeett creation process within single model (See Figure 1). This achieved through a training framework that learns anytime-to-anytime jumps (i.e., deterministic mapping) and employs deterministic sampling as proposed in Consistency Trajectory Models (CTMs) (Kim et al., 2024). Building upon the CTM framework. originally proposed in the computer vision field, we address the limitations of its training approach, particularly its heavy reliance on extra pretrained networks to achieve notable generation performance, which are not always accessible in other domains. Specifically, we propose a novel feature distance that uses teacher network as a feature extractor for distillation loss (see Section 3. Furthermore, while distilling classifier-free guided texi-conditional trajectories with Frass Guidance (CFG) (Ho and Salimans, 2022) as a new condition for student models, we introduce v-sampling. new sampling algorithm that incorporates the text-condie text-conditional and unconditional student models. For sammliinng we train both text-conditional and unconditional student models, simultancously. (see Section 3.2 and Algorithm
2

---

Published as a conference paper at ICLR 2025
In our experiments, SoundCTM shows notable 1-step generation and higher-quality multi-step gener- ation not only in the 16kHz setting but also in the 44. 1kZz full-band T2S generation setting, which is the minimum requirement for real-wool world sound creation. Additionally, by utilizing deterministic sampling. SoundCT'M demonstrates the capability to flexibly control the trade-off between inference speed and sample quality while preserving semantic content, thereby enabling the trial-andreffnemmet process with single model. We highlight that SoundCTM-DiT-I based LDM with 1B trainable parameters, is the first large-scale full-band T2S distillation model capable of achieving both high-quality 1-step generation and higher-qualin multi-step generation.
whose teacher model is DiT-
Our contributions are summarized as:
We introduce SoundCTM, enabling the efficient trial-and red-refinement creation process with single T2S model through high-quality 1-step generation and higher-quality generation with multi-step deterministic sampling.
To develop SoundCTM, we address the limitations of the CTM framework by proposing a novel feature distance for distillation loss, strategy for distilling CFG trajectories, and v-sampling that combines rext-conditional and unconditional student jumps.
We demonstrate that SoundCTM-DiT-I IB is the first large-scale distillation model to achieve notable 1-step and multi-step full-band text-to-sound generation.
PRELIMINARY
Diffusion Models Let Pdata denote the data distribution. In DMs, the data variable X ~ Pdna generated through a reverse-time stochastic process (Anderson, 1982) defined dx; -2/ log pe (x2)dl )dl + d2,,w,, from time T to O, where w, is the standard Wiener process in reverse-time, The marginal density (x( is obtained by encoding X along with fixed forward diffusion process, dx, = /2t dw,, initialized by X,. where w, is the standard Wiener process in forward-time -wward-imee. Song et al. (2021b) present the deterministic counterpart of the reverse-time process, called the Probability Flow Ordinary Differential Equation (PF ODE), given by
(x,
dx: dt
X. E Expp((x)xx)) [x Xx]
-V logp:(x:)
t
where (x)x() is the probability distribution of the solution of the reverse-time stochastic process from time & to 0, initiated from x,. (x)o(x(x.)) [x[x;] = + tV log pe (x,) is denoiser func- tion' (Efron, 2011).
Practically, the denoiser [[][x, is estimated by neural network Dg. obtained by minimizing a Denoising Score Matching (DSM) loss (Vincent, 2011; Song et al., 2021b) (x)))) [|]x] (x, 112]), where Poje (x Xn) is the transition probability from time 0 to I, initiated with Xg. Given the trained denoiser, the empirical PF ODE is given by
dx; dt
X
(x,,
t
DMs can generate samples by solving the empirical PF ODE, initiated with Xy, which is sampled from prior distribution approximating pr.
Cext-Conditional Sound Generation with Latent Diffusion Models LDM-hased T2S models (Liu al.. 2023; 2024b; Ghosal et al., 2023; Evans et al., 2024a;c) generate audio matched textual descriptions by first obtaining the latent counterpart of the data variable through the reverse-time process conditioned by text embedding Cexu This latent variable Zu is then converted to Xy using pretrained decoder D. During the training phase, D& is trained by minimizing the DSM loss E Poo [||zo Dz(z, I, Crext) 3). where Poje (z) zn) is the latent counterpart of the transition probability from time 0 to f, initiated with 2g. is given by a pretrained encoder as Zo Exx)). We refer to Appendix A for review of related work.
For simplicity, we omit Pejo (x)x,), subscript in the expectation of the denoiser, throughout the paper.
3

---

Published as a conference paper at ICLR 2025
Feature distance computation
Feature extraction by Teacher's Network
e/c
Z target
Multiple-step ODE solver
Ztarget
Z
7rst
Zest
PF ODE trajectory
SoundCTIV'' Jump
Figure 2: Illustrations of SoundCTM's two predictions Zurget and Zest at time with initial value z and the feature extraction by the teacher's s network for the CTM loss shown within the blue ellipse area. All the parameters of the teacher's 's network are frozen. The conditional embedding and time are also input to the feature extractor. Note that the teacher's 's network does not need to be the UNet architecture (Ronneherger et al., 2015).
Consistency Models Consistency Models (CMs) (Song et al., 2023) and CD predict anytime-to- zero time long jumps of the PF ODE trajectory. Gxx,, 0) is defined as the solution of the PF ODE from initial time to final time O, and is estimated by Ga as the neural jump. To train Go. two time step 0-predictions are compared: one from teacher and the other from student O as:
Go(x,,t, 0) so (Soloee t, At; Φ), At, 0),
where Solver(x,, t, At; Φ) the pre-trained PF ODE in Eq. (l) within the interval [t At, f). which determines the amount of teacher information to distill, and sg is the exponential moving (1 -)) In CMs and CD, only stochastic sampling is possible during multi-step sampling. This is because the model is trained using anytime-to-zert time jump framework, which requires adding noise to the estimated xo to obtain x; for multi-step sampling. As result, the semantic content of generated samples changes depending on the number of sampling steps. This variability hinders the trial-and-rete und-refinement process, which is why do not adopt the CD framework.
average (EMA) stop-gradient sg) < stopgrad(usq{(
Consistency Trajectory Models In contrast to CMs, CTMs predict both infinitesimally small step jump and long step jump of the PF ODE trajectory. G(xe, is defined as the solution of the PF ODE from initial time / to final time 8 < l, and G is estimated by Go as the neural jump. To train Gg. s-predictions compared: one from teacher & and the other from a student as:
(xe,, Go f, 8) R Goo(e) (Solver (xe, f, uz Φ), u. 8),
(2)
where Solver(x, t, u; Φ) the pre-trained PF ODE in Eq. (1), a random u E [s, determines the amount of teacher information to distill. To quantify the dissimilarity (CTM loss) between the student prediction Gx(x,, t, s) and the teacher prediction G= (Solver(x, (x, t, u; σ), u, s) Eq. (2), the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) is used to measure feature distance diea. after transporting both predictions from s-time to 0-time as X Fea(xx,,, 1,
G Gegte) (Go t, s), s. 0) and Xurgei (,,t, u, s) == Gsg(o) (Gag()) (Solver t, uz Φ), u, 8) ) 8, 0). Summarizing, the CTM loss is defined as
(C x)
Exp Fx, Ixo d (Xiarga (X+,, t, u, 8), Xaal X (x,,
s))] (3)
With this anytime-to-anytin jump training framework, both stochastic and deterministic sampling are possible in CTMs, enabling the aial-ann-reffnemeett process for sound creation.
SOUNDCTM
To address the challenges of achieving fast, flexible, and high-quality T2S generation, introduce SoundCTM by reframing the CTM's training framework. Consistent with the CTM, we use the same distillation loss. Since this paper primarily illustrates the method using LDM-based T2S models as the teacher model, the student model is trained to estimate the neural jump Go as:
Go Go Crext 1, 8) (z (Sol ve T Crext: l, uz Φ), Crexty U, 8),
4

---

Published as a conference paper at ICLR 2025
where z₁ is the latent counterpart of x2, and Solver(z,, Cuxt, t, u. Φ) is the numerical solver of the pre-trained text-conditional PF ODE by following Eq. (2). To quantify the dissimilarity between Ge and (0)) propose new feature distance in Section 3.1. We refer to Appendix A for review of related work about our proposed feature distance.
3.1 TEACHER'S 's NETWORK AS FEATURE EXTRACTOR FOR CTM LOss
We propose new training framework that utilizes the teacher's network as a feature extractor and measures the feature distance dreacher for the CTM loss between Gg and Geg(o) in Eq. (4), as illustrated in Figure 2. Utilizing dieacher offers two benefits:
Using dhencher yields better performance compared with using the 22 distance in the z domain computed either O-time s-time, demonstrated in Table 1.
The teacher network can naturally be utilized even in situations where external off-the-shelf pretrained networks for computing feature distance are inaccessible, since SoundCTM is a distillation model that assumes the availability of a teacher diffusion model. We discuss the potential use of external off-the-sheif pretrained networks in Appendix B.
We define dleacher between two predictions Zurzet and Zest as follows:
M
Qrencher (zurget, Zest C. 1) IN o m (Zragget; C, () TN mo. (Zrsts c, 1))12
(5)
where TN TN c,t) denotes the channel- normalized output feature of the m-th layer of the pretrained teacher's network, conditioned by time and embedding c. This approach feasible since noisy latents are input to the teacher's network during teacher's training.
3.2 CFG HANDLING FOR SOUNDCTM
CFG plays a pivotal role in T2S generation, as well as in other modalities (Ho and Salimans, 2022), for generating high-quality samples. To leverage this advantage of CFG, we propose distilling the classifier free guided PF ODE trajectory scaled by w uniformly sampled from the range [umin- Wmax] condition in the student network defined as:
during training, and using
Go [z,, Crest W, t, +
8 E
go
(z((ee (Ze: Crexss W.
(6)
where go is neural output. To summarize (4) the distillation loss for SoundCTM is formulated as:
CTM ( x) E, Eu dreccterr (Zarger (Zi, (Zt,, Crexts W, tt u, s), ZesiZzi, Cwatt W, t, 8), Crexss s)), (7)
where
Zuaree Ze, Crext, W, f, u, s) = Geg(o) (Solvers) Crext, W, f. u; Φ), Crext W, U, s)s)
Zest Ze, Crexts W, t, 8) = Gø Crexss wi b 8),
Sol Solver(z,, Crest W, f, u; Φ) = wSolver Crex, t, u; Φ)) + (1-1)) soleerr)o u,t, Ø [s. t), and ~ [[[minn Wmax respectively.
an unconditional embedding,
To achieve better generation performance, Kim et al. (2024) use the two auxiliary losses, the DSM loss and an adversarial loss (GAN loss) (Goodfellow al., 2014). For SoundCTM, only the DSM loss, defined as:
(Sound (0) = Er, .u - ge (z,, Cexts w, 1)]]
(8)
Note that the DSM loss serves to improve the accuracy of approximating the small jumps during training. We discuss the reason why we eliminate the GAN loss in Section C. Summing Eqs. (7) and (8), SoundCTM is trained with the following objective:
((
== SSumi (0; Φ) + ADSM Stand
(9)
where λpsm is scaling weight for LSM We summarize SoundCTM's training in Appendix D.
"The idea of this normalization process borrowed from I.PPS (Zhang et al., 2018, Sec. 3).

---

Published as a conference paper at ICLR 2025
3.3 SAMPLING SCHEME OF SOUNDCTM
Inspired by the performance improvement by using CFG in DMs, we introduce u-sampling in SoundCTM's sampling, which incorporates the text-conditional and unconditional student models, given by:
Zzj: = Golze, Ø, (,w,t, + v ((Gozzz Ciexti w,t, s) - Go(ze, D,w, w,w,t,s) t)s
(10)
Algorithm summarizes SoundCTM's sam- pling, with the sampling timesteps denoted as T' to ... IN O. Note that in con- trast to the previous sampling methods for the diffusion-based distillation models (Bai al., 2023; Liu et al., 2024c; Kim et al., 2024), we also use Gz(zz, W, L, 8) as highlighted in blue Algorithm 1. We train both (ze Crex; 42, (, 8) Zt, Ø, w, t, s) simultaneously as shown
Algorithm 1 SoundCTM's Inference
Require: Hyperparameter es, text condition Cieu, CFG scale w, hyperparameter CTM's s 7-sampling 7
I: Sample zio from prior distribution
for to N do
3: fpt + v tt+t
and Gz( in Algorithm 2.
4: Zi,
vGo(Zen, Cuesy W/ tuy tx+1)
- V))oolz 0,00 w, lus lot1)
6:
21,,1 iints + Th+t1E
7: end for
EXPERIMENTS
Return Zen
4.1 T2S GENERATION ON 16KHz
As an initial step, in this section, we conduct
experiments on 16 kHz audio signals by following most of the existing T2S generative models (Ghosal et al,, 2023; Liu et al., 2023; 2024b; Bai et al., 2023; Liu et al., 2024c). We evaluate SoundCTM on the AudioCaps dataset (Kim et al., 2019), which contains 47, 289 pairs of 10-second audio samples and human-written text descriptions for the training set and 957 samples for the testset. All audio samples are downsampled 16 kHz. We adopt TANGO (Ghosal al., 2023) as the teacher model trained with EDM's s variance exploding formulation (Karras et al., 2022). We use deterministic sampling (7 = 0 in Algorithm 1) and evaluate the model performance with student EMA rate μ 0.999. Experimental details are described in Appendix D.I.
We use four objective metrics: the Frechet Audio Distance (FAD vgg) (Kilgour et al., 2019) on VGGish, the Kullback-Leible divergence (KTpass) on PaSST (Koutini et al., 2022), a state-of-the-art audio classification model, the Inception Score (ISpasa) (Salimans et al., 2016) on PaSST, and the CLAP score? (Wu* et al., 2023).
Evaluation Metrics
Effectiveness of Utilizing Teacher's Network as Feature Extractor We first evaluate the efficacy of utilizing the teacher's network feature extractor in Eq (7), which we newly proposed in Section 3.1, by comparing the following cases: Using 1) the at O-time step, 2) the 12 at s-time step, and 3) the deeecher at s-time step. We use the entire layers of the teacher's network for computing dreaherr
As shown in Table 1, using dreache demonstrates better performance across all the metrics against all the other cases. This result indicates that using dieacher enables the student model to distill the trajectory more accurately than using l2. In the subsequent experiments, unless otherwise noted, SoundCTM refers to the model trained with using duacher-
Table 1: Effectiveness of using proposed diacher against lp on 2g space evaluated on AudioCaps testset. w 3.5 and 1.0 are used for inference.
steps FAD vs
Klyo
18p0
CLAP
Teacher diffasion model (IANGO-EDS) w/. Houn solver)
1.71
1.28
8.11
0.46
Student Models
SounCCTM (D-time step)
2.43
1.28
0.42
SunnCTTM 1 (a-time step)
SounCCTM w/. deeater
2.17
1.27
7.18
Preserving Semantic Content within Multi-step Sampling One of our goals is to develop a model capable of achieving high-quality 1-step generation and higher-quality multi-step generation
We use the 2330k=audioset-hest.p checkpoint from https: com/ ON- /CLAP

---

Published as a conference paper at ICLR 2025
Method
Sampling steps Step
Step
Step
ConsistencyTTA
w/. stochastic sampling
("Detemministie sampling (casible)
AudioL.CM
wi. stochastic sampling (*Determiniatic sampling not feasible)
M (Ours)
w/. deterministic sampling
Prompt: "Thunder claps, and hard rain falls and splashes on surfaces."
Figure 3: Visualization of spectrograms of generated samples using 1-step, 2-step, and 4-step genera- tion with ConsistencyTTA, AudioLCM, and SoundCTM.
Table 2:
Performance comparisons on AudioCaps testset at 16 kHz. Bold scores indicate the best results under 1-step generation. Underlined scores indicate the best results under multi- step generation of Distillation Models. f denotes the results tested by us using the open-sourced checkpoints, as not all metrics are provided in each paper.
sampling steps
Model
FADoge
KL
18
CLAP
Diffusion Models
AudioLDM 2-AC-Large (Liu et al,, 2024b)
200
3.5
1,42
0,98
TANGO (Ghosal et al., 2023)
200
3.0
1.64
1,31
6,351
0.44
Teacher of Consistency TTA (Bai et al., 2023)
200
3.0
1.91
Teacher of AudioL CM
al,, 2024c)
1.56
Our Teacher model (TANGO w/. Heun Solver)
40
3.5
1.71
1.28
8.11
0.46
Distillation Models
ConsistencyTTI (Hai et al., 2023)
4.0
2.41
1.31
7,84
0.42!
2
4.0
2.48
1.271
7.881
0.41
4.0
3.01
1.31
8.051
0.42!
Audiol.CM (Liu et al., 2024c)
4,04
1.751
6.521
1.67
1.491
8.24
8.781
0.42
SoundCTM (Ours)
3.5
1.0
2,08
1.26
7.13
0.43
2
3.5
1.0
1.90
1.24
7.26
0.45
1.22
7.37
1.45
1.20
7.98
0.46
16
3.0
2.0
1.38
1.19
8.24
0.46
while preserving semantic content through deterministic sampling. To demonstrate this capability, visualize the spectrograms of generated samples from the same initial noise and input text prompt across different numbers of sampling steps with SoundCTM's deterministic sampling and other T2S distillation models (ConsistencyTTAà ITA and AudioL.CM in Figure 3.
We emphasize that, by its anytime-to-anytime jump training framework, SoundCTM can preserve generated content using deterministic sampling
(7 = 0)* 4 even when varying the number of sampling steps (see additional examples in Figure 6). This cannot be achieved with ConsistencyTTA or AudioLCM due to their lack of deterministic sampling capability. In terms of the sample quality, as shown Table 2, which discuss later, SoundCTM shows clear trade-offs between the performance improvements and the number of sampling steps. These SoundCTM's characteristics allow sound creators to efficiently proceed with the rial-and-refinemenn process with a single model.
Performance Comparison with Other T2S Models We compare SoundCTM with other T2S models under both -tstep and multi-step generation. We employ ConsistencyTTs and AudioL_CM as our baseline models. We also include Audiol.DM2-ACA.
(Liu et al., 2024b), TANGO (Ghosal
Note that SoundCTM also supports stochastic sampling by adjusting 7-
7

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparisons on AudioCaps at full-band setting. Bold scores indicate best results.
Model
channelskrr
R of sampling
PRopesla + gT
CLAPP
Inference speed |sec.]
steps
Diffusion Models
Stable Audio Open (Evans al,, 2024c)
2444.1kllz
AM2ioLLMM-4HKHH (Liu et al,, 2024b)
1M8SHH
200
3.5
101.11
2.04
0,37
model Heun Solver
1kll
Distillation Models
SoundCTM-DDT-IB
2
5.0
1.0
79,96
1.61
0.38
0.10
5.0
5.0
79.05
1.29
0.40
0.35
16 5.0
5.0
72.24
1.27
0.71
et al., 2023) in the evaluation for completeness. Table 2 presents the quantitative results. All the LDM-based models use DDIM sampling (Song et al., 2021a) except for our teacher model.
Under the 1-step generation setting, SoundCTM shows the best performance in all the evaluation metrics except for psst. Note that, considering its teacher models' performance, the results 1-step generation of AudioLCM is limited (See also (Liu et al., 2024c, Fig. 2 (b)).). Under the multi-step case, AudioLCM CM and SoundCTM show clear trade-offs between the performance improvements and the number of sampling steps. On the other hand, Consistency TTA dose not show such trade-offs, implying that ConsistencyTTI TTA might suffer from accumulated errors during multi-step stochastic sampling. These results indicate that SoundCT'M is the first T2S distillation model successfully achieve both promising 1-step and multi-step generation.
4.2 LARGE-SCALE FULL-BAND T2S GENERATION
Inspired by the success of recent large-scale (e.g., 1B scale) LDM-based T2S generative mod- els (Evans et al., 2024a;b;e; Liu et al., 2024b) in achieving full-band T2S generation suitable for real-world applications, for production-level generation, evaluate the scalability of SoundCTM to a 1B-scale model and its extension to full-band T2S generation tasks. We conduct our evaluation on the AudioCaps dataset, with all audio samples resampled to 44.1 kHz. Following Evans al. (2024a), we use all 4, 875 text captions from the AudioCaps testset, which contains captions per audio clip.
In contrast the UNet-based model used in Section 4.1, we adopt the flffussion-trasformme (DiT) (Peebles and Xie, 2023), which contains 1 1.OB trainable parameters, as the teacher and student model. The teacher model is trained using EDM's s variance exploding formulation. To distinguish the UNet-based student model, we, hereafter, refer to the DiT-based student model as SoundCTM-DiT- IB. We use deterministic sampling and evaluate the model performance with student EMA rate μı 0.96. We use the entire DiT blocks of the teacher's 's network for computing dueacher. Experimental details are described in Appendix D.2. It worth noting that in the field of sound generation, SoundCTM- DiT-1B represents the first attempt to distill a LDM-based large-scale full-band T2S generative model.
Evaluation Metrics Following the objective evaluation protocol for the full-band T2S gener- ation in Stable Audio series (Evans et al., 2024a;b;c), we compute three objective metries: the FDopents (Cramer al., 2019), KTppssss and the CLAP score by using stable-audio--ettre.
Quantitative Evaluation Since SoundCTM is the first attempt perform few step full-band T2S generation using a distillation model, we only employ diffusion models such as AudioL 48kHz (Liu et al., 2024b), Stable Audio Open (Evans al., 2024c) and teacher model as baselines. AudioLDM2-448HHz Stable Audio Open, and our teacher model use DDIM sampling (Song al., 2021a), D (Lu al., 2023), and our 2nd-order Heun solver, respectively.
'https: / /i thub. com Stabi 1 i ttt-tyAl A /stab' e- e-audi o o-metric 'Note that the results of Stable Audio Open might not comparable the other results since AudioCaps is not used for its training, mentioned in Evans al. (2024c, Table 2).
8

---

Published as a conference paper at ICLR 2025
Table Subjective evaluation on AudioCaps testset on full-band T2S generation setting. We report overall audio quality (OVAL) and text alignment (REL) with 95% Confidence Interval.
Method
of steps
OVAL
REL 1
Ground-truth
79.6
1.29
83.2 + 1.13
Stable Audio Open (Evans et al., 2024c) w/. DPM-Solvcr++
100
7.0
54.0 = 1.58
47.1
1.65
Audiol.DM2-48kFF (Liu al,, 2024b) w/. DDIM
200
3.5
65.1
1.45
69.9
1.42
teacher model w/. Heun Solver
40
5.0
66.9
1.41
69.4
1.48
SoundCTM-DiT-IB
1
5.0
1.0
63.2
= 1.46
69.1
= 1.34
16
5.0
5.0
72.0
+ 1.30
77.0 + 1.22
Method
Sampling steps
Step
Step
16 Step
SoundCTM DIT-IB
wi sondastic sampling
TM-DIT-18
wt. deterministic vampling
5.0
5.0
Prompi: "Detent honking and humming of an engine e5 car aporoached with a loud hark nearby.""
Figure 4: Visualization of spectrograms generated sam- ples by SoundCTM-DiT: IB using 1-step, 4-step, and 16- step generation with stochastic (7 = 0.5) and deterministic (7 0) sampling.
(a) FDogeal
(b) CLAP
Figure 5: Influence of v on SoundCTM- DiT-1B with 8-step sampling. Darker colors indicate better scores.
Table 3 presents the quantitative results. SoundCTM-DiT: demonstrates promising performance in both 1-step and multi-step generation, with clear trade-offs between the number of sampling steps and the sample quality. These results indicate that the SoundCTM framework is scalable to the full-band T2S generation setting. We also measured the inference time of SoundCTM-DiT. 1B and its teacher diffusion model single NVIDIA H100. As shown in Table 3, SoundCTM demonstrates faster generation compared to the teacher model, enabling creators to efficiently conduct the renffeen-refieement creation process with full-band setting.
Subjective Evaluation For further evaluation of SoundCTM-DiT-IB we conduct a listening test. The baselines used for comparison included Stable Audio Open, AudioLDM2-48KHH, model, SoundCTM with 1-step generation, SoundCTM with 16-step generation, and ground-truth samples. We asked 17 human evaluators assess the generated audio samples based on criteria: overall audio quality (OVAL) and relevance to the text prompts (REL). Each participant was presented with 10 samples per model for each metric and asked rate them on a scale from 100. The text prompts for the generated samples were randomly selected from the AudioCaps testset. Considering the difficulty of comparing generated samples in a T2S generation task with the single stimulus testing approach, following Evans et al. (2024a), employ MUSHRA method (Union, 2003) on the WebMUSHRA platform (Schoeffler and et al., 2018) as the evaluation protocol (See also Figure 9),
teacher
As demonstrated in Table 4, SoundCTM-DiT-1133 1-step generation shows comparable scores to other methods across both metrics. Moreover, comparing the 1-step and 16-step generation results, the 16- step generation demonstrates higher scores. These results, along with the objective evaluation results in Table 3, indicate that SoundCTM-DiTE is capable of flexibly switching between high-quality 1-step and higher-quality multi-step sound generation, even in full-band setting.
Stochastic and Deterministie Sampling on SoundCTM-DiT-IR Once again, one of our goals is to develop model capable of achieving high-quality 1-step generation and higher-quality multi- step generation while preserving semantic content through deterministic sampling. To demonstrate this capability in SoundCTM-DiT-IIB we visualize the spectrograms of generated samples from the same initial noise and input text prompt across varying numbers of sampling steps with both deterministic and stochastic sampling. As shown in Figure 4 (See more examples in Figure 7 and Figure 8). SoundCTM- CETM--DTT-IT IB preserves semantic content with deterministic sampling while allowing variations through stochastic sampling.

---

Published as a conference paper at ICLR 2025
Influence of v-sampling We examine the influence of both and multi-step sampling, particularly 8-step generation, for SoundCTM-DiTE per audio clip (957 samples) and compute the FD FDopentt and the CLAP score. As shown in Figure 5, regardless of the value of w, increasing the value of v improves the sample quality. This indicates that the hyperparameter serves role similar to the CFG scale.
In this experiment, we use a single caption
4.3 POTENTIAL APPLICATION; GUIDANCE BASED CONTROLLABLE GENERATION
For further exploration of SoundCTM's application to sound creation, we investigate its potential downstream task and conduct experiments on sound intensity control task (Novack al., 2024b;a; et al., 2024).
Loss-based Guidance for SoundCTM In DMs, a loss-based guidance method is one of the major method for training-free controllable generation with using pretrained DMs (Yu al., 2023; Levy et al., 2023). The loss-based guidance method in DMs involves an additional update 2 = 2--1 Pr 2. (x(())) ceoodition) during sampling, where f(-) is differentiable feature extractor that converts &y into the same space as the target condition Ycoedition Xo = D(2o (z+)). (zz) is a clean estimate derived from z; using Tweedie's formula (Efron, 2011), and p is a learning rate.
Sound Intensity Control To validate the proposed loss-based guidance framework with Sound- CTM, we conduct sound intensity control (Novack et al., 2024b; Wu et al., 2024; Novack et al., 2024a). This task adjusts the dynamics of the generated sound match a given target volume line or curve. We follow the experimental protocol from DITTO (Novack et al., 2024b) to control the decibel (dB) volume line or curve of the generated samples. We use 200 audio-text pairs from the AudioCaps testset for each of the 6 different types of y condition (See Figure 10 (a) and Figure 11 (a)). We use the
pretrained UNet-based SoundCTM obtained from Section 4.1. We evaluate our framework using the mean squared error (MSE) between the target and obtained y ccomdition the FAD and the CLAP score. Experimental details are provided in Appendix D.3.
We employ DITTO-2 (Novack et al., 2024a) and default T2S generation as baselines. DITTO-2 is a trainig-free controllable generation method using pretrained distillation models proposed in the music generation field by optimizing an initial noise latent 2y given the target loss. Since DITTO-2 is not open-sourced, we apply its zy-opiimizttion to SoundCTM.
From Table and the intensity curves shown in Figures 10 to 11, our method successfully con- trols the sound intensities. These results show the potential capability of the loss- based guid- ance of SoundCTM. Be aware that the objective of this experiment is to show the potential ap- plication of SoundCTM rather than to propose a new method that outperforms other methods for training-fite free controllable generation.
Table 5: Quantitative results of sound intensity con- trol on SoundCTM. Number of sampling step is 16 for all methods.
Methods
MSE
FAD. 1 CLAP
Default T2S Generation
231.9
2.08
0.47
DITTO-2 (Novack et 2024a)
60.2
3.65
0.38
0,2
50.4
3.71
0.41
Our loss- guidance
3.04
CONCLUSION
SoundCTM addresses the challenges of current T2S generative models, which make it difficult for creators to efficiently conduct the trial-and-error to semantically reflect their artistic inspirations into the sound and generate high-quality sound within a single model. To develop SoundCTM, we propose a novel feature distance for distillation loss, a strategy for distilling CFG trajectories, and v-sampling for multi-step generation. While being the first large-scale full-band T2S distillation model in the sound community, SoundCTM-DiT-I IB has demonstrated notable performance in both 1-step and multi-step generation.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
SoundCTM poses a risk for generating harmful or inappropriate content, offensive sound effects, even sound content that infringes copyrights. To mitigate these risks, it is crucial to carefully curate the training data as a first step. Furthermore, addressing these risks involves implementing robust content filtering, moderation mechanisms prevent the creation of unethical, harmful sound contents.
REPRODUCIBILITY STATEMENT
The source code is available at https : htt/ggttubbbtt////////////ttttht com/ coom/ony///// soundetm. Moreover, we out- line our training and sampling procedures Algorithm and Algorithm and detailed implementa- tion instructions for result reproducibility can be found in Appendix D.
ACKNOWLEDGMENT
We sincerely acknowledge the support of everyone who made this research possible. Our heartfelt thanks go Ganesh Nagaraja, Sachin M Srinidhi Srinivasa, and Giorgio Fabbro for their assistance. Computational resource of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Technology (AIST) was used.
REFERENCES
Brian. D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12:313-326, 1982.
Yatong Bai, Trung Dang, Dung Tran, Kazuhito Koishida, and Somaych Sojoudi. Acceler- ating diffusion- based text-to-audio generation with consistency distillation. arXiv preprint arXiv:2309, 10740, 2023.
Keunwoo Choi, Jaekwon Im, Laurie Heller, Brian McFee, Keisuke Imoto, Yuki Okamoto, Mathieu Lagrange, and Shinosuke Takamichi. Foley sound synthesis at the dease 2023 challenge. In arXiv e-prints: 2304.1252) 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus. Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetuue
language models, 2022.
Aurora Linh Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. Look, listen, and learn more: Design choices for deep audio embeddings. In Proc. IEEE Int. Conf. Acoust, Speech, Signal Process. (ICASSP), pages 3852-3856, 2019.
Bradley Efron. Tweedie's s formula and selection bias. Journal of the American Statistical Association, 106: 006:1602- 1614, 201
Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast diming-conditionn latent audio diffusion. Proc. International Conference on Machine Learning (ICML), 2024a.
Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski. Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404. 10300, 2024b.
Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407.14338 2024c.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labels dataset for audio events. In Proc. IEEE Inf. Conf. Acoust, Speech, Signal Process. (ICASSP). 2017.
11

---

Published as a conference paper at ICLR 2025
Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Textoto-audio genera- tion using instruction tuned Ilm and latent diffusion model. Proc. ACM Int. Conf. on Multimedia. (ACMMM), 2023.
Sang gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A uni- versal neural vocoder with large-scale training, In Proc. International Conference on Learning Representation (ICLR), 2023.
Ian J. Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, Proc. Advances in Neural Information Processing Systems (NeurIPS), 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
Shawn Hershey, Sourish Chaudhuri, Daniel P. Ellis, Jort Gemmeke, Aren Jansen, R. Channing Moore. Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin Wilson. Can architectures for large-scale audio classification. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 131-135, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In arXiv e-prints: 2207. 12598, 2022.
Jiawei Huang, Yi Ren, Rongjie Huang, Dongehao Yang. Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint 2rXiv:3305.18777 2023a.
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Mlake-nn-audio: Text-to-audio generation with prompt-enhanced diffusion models. Proc. International Conference on Machine Learning (ICML., 2023b.
Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. In British Machine Vision Conference (BMVC), 2021.
Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shecht- man, Jun- Yan Zhu, and Taesung Park. Distilling Diffusion Models into Conditional GANs. In European Conference on Computer Vision (ECCV), 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proc. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08444 2019.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating Captions for Audios in The Wild. In NAACL-HLT, 2019.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probabil- ity flow ode trajectory of diffusion. Proc. International Conference on Learning Representation (ICLR), 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Interna- tional Conference on Learning Representation (ICLR), 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proc. International Conference on Learning Representation (ICLR), 2014.
12

---

Published as a conference paper at ICLR 2025
Jungil Kong, Jachyeon Kim, and Jaekyoung Bac. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 2753-2757, 2022.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Controllable music production with diffusion models and guidance gradients. arXiv preprint arXiv:2311.00613, 2023.
Shanchuan Lin, Anran Wang, and Xiao Yang. SDXI-Lightning: Progressive Adversarial Diffusion Distillation. arXiv preprini arXiv:2402. 13929, 2024.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audic generation with latent diffusion models. Proc. International Conference on Machine Learning (ICML), 2023.
Haohe Liu, Ke Chen, Qiao Tian, Wenwu Wang, and Mark D. Plumbley. Audiosr: super-resolution at scale. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1076-1088 2024a.
Versatile audio
Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. AudioLDM 2: Learning holistic audio generation with sule-supe pretraining. IEEE/ACM Trans. Audio, Speech, Lang. Process., 2024b.
Huadai Liu, Rongjie Huang, Yang Liu, Hengyuan Cao, Jialei Wang, Xize Cheng, Siqi Zheng, and Zhou Zhao. Audiolcm: Efficient and high-quality text-to-audio generation with minimal inference steps. In ACM Multimedia 2024, 2024c. URL htttps://operreveww net / forum? idd= Oh@Ldb4se0.
Liyun Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In Proc. International Conference on Learning Representation (ICLR), April 2020.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 2023.
Hila Manor and Tomer Michaeli. Zero-shot unsupervised and text- bast- audio editing using DDPM inversion. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34603- 34629. PMLR, 21-27 Jul 2024.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Eloi Moliner, Jaakko Lehtinen, and Vesa Välimäki. Solving audio inverse problems with diffusion model. In IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2023.
Zachary Novack, Julian McAuley, Taylor Berg- Kirkpatrick, and Nicholas Bryan. Ditto-2: Distilled diffusion inference-ting time t-optimization for music generation. arXiv preprint 2rX5v.2405.20289, 2024a.
13

---

Published as a conference paper at ICLR 2025
Zachary Novack, Julian McAuley, Taylor Berg- Kirk-K-ikaatrckk and Nicholas J. Bryan. Ditto: Diffusion inference-timg t-optimization for music generation. Proc. International Conference on Machine Learning (ICML), 2024b.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 4195-4205, 2023.
Zafar Rañi, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner Musdb18-a 18-hq - an uncompressed version of musdb18, August 2019. URL https: /ooi. org/ 10. / zeoodo 3338373.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pages 234-241, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proc. Advances Neural Information Processing Systems (NeurIPS), 2016.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.1742,, 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprini arXiv:2403.12011 2024.
Abraham. Savitzky and M. E. Golay. Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry. 36(8):1627-1) 1964.
M. Schoeffler and et al. webMUSHRA - A comprehensive framework for web-based listening tests. A Comprehensive Framework for Web-based Listening Tests. Journal of Open Research Software., 6(1):8, 2018.
Jascha Sohl-Dickstae Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. International Conference Machine Learning (ICML), 2015.
J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. Proc. International Conference on Learning Representation (ICLR), 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. International Conference on Learning Representation (ICLR), 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. Proc. Interna- tional Conference on Machine Learning (ICML), 2023.
International Telecommunication Union. Recommendation itu-r 1534-1: Method for the subjective assessment of intermediate quality level of coding systems. 2003.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NeuriPS), 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- tion, 2367):1661-1674, 2011.
Heming Wang and DeLiang Wang. Towards robust speech super-resolution JEEE/ACM Transactions Audio, Speech, and Language Processing, 29:2058-200 2066, 2021.
14

---

Published as a conference paper at ICLR 2025
Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas J. Bryan. Music controlnet: Multiple time-varying controls for music generation. JEEE/ACM Trans. Audio, Speech, Lang. Process., 2024.
Yusong Wu*, Ke Chen Tianyu Zhang*, Yuchen Hui*, Taylor Kerg-Kirkpattick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP. 2023.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-fre free energy-guided conditional diffusion model. Proc. IEEE International Conference on Computer Vision (ICCV), 2023.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018
15

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Diffusion-based Models for T2S Recent work (Huang et al., 2023b;a; Liu et al., 2023; 2024b; Ghosal et al., 2023; Evans et al., 2024a;b;c) and competitions (Choi et al., 2023) report that LDM- based sound generation models outperform other approaches such as GANs. However, these LDM- based models suffer from slow inference speeds. To address this issue, two main approaches have been proposed for achieving faster generation in the sound domain. The first approach involves training distillation models, such as ConsistencyTTA (Bai et al., 2023) and ACdiol.CM (Liu et al., 2024c), using LDMs as teacher models. Both models are trained with the CD) framework (Song ct al., 2023). The second approach compresses long-form audio waveform using an encoder, as in the Stable Audio series (Evans ct al., 2024a;b:c). For instance, the Stable Audio series compresses 95-second waveform signals into Zu. and its LDM is trained on this latent representation, unlike other LDM- models that compress only 10-second audio signals with their encoder. SoundCTM is categorized as the distillation approach but based on CTMs in contrast to Consistency TTA and AudioLCM.
Feature Extractor for Latent Distillation Models In the field of computer vision, several methods utilize pretrained feature extractors for LDM-based distillation (Sauer et al., 2024; Kang et al., 2024; Lin et al., 2024; Xu et al., 2024). Sauer et al. (2024); Lin et al. (2024); Xu et al. (2024) employ pretrained teacher models as discriminator for adversarial diffusion distillation (Sauer et al., 2023). Kang et al. (2024) propose training LatsntLPIPS, latent counterpart of LPIPS, and using train GANs by distilling teacher LDMs.
In contrast to these approaches, we leverage pretrained teacher models for the CTM loss (not using GAN loss). Notably, none of these approaches have been explored in the sound domain.
Training-free Controllable Generation Levy al. (2023) demonstrate training-free controllable music generation using loss-based guidance (Yu et al., 2023). Their approach defines a iusc-specific loss, such as for music continuation and infilling, and incorporates the gradient of this loss into the inference sampling. Novack et al. (2024b;a) also present controllable generation by optimizing initial noisy latents zr based on loss computed in the target condition space. DITTO (Novack et al., 2024b) uses pretrained DMs and DITTO2 (Novack et al., 2024a) uses pretrained distillation models for zr-optimization.
B DISCUSSION ON FEATURE EXTRACTOR CANDIDATES FOR CTM LOSS
When considering the use of external off-the-shelf pretrained networks as feature extractors, several potential candidates exist, such as VGGish (Hershey et al., 2017), PaSST (Koutini et al., 2022), LPAPS (lashin and Rahtu, 2021). 1) and CLAP (Wu* et al., 2023). However, these models are first only different training data but also
trained with the data domain rather than the latent domain different sampling frequencies (16 kHz, 32kHz, and 48kHz, respectively), necessitating, for every training iteration, projecting Zg to X and resampling the audio from the training data's sampling frequency to that of the pretrained feature extractors. After resampling, the waveform must be transformed into another domain, such as the Mel-spectrogram domain, to match the input format of the extractors. Consequently, utilizing these models is not straightforward. This is why we propose to use the teacher's network as feature extractor for distillation loss.
C DISCUSSION ON INTEGRATING GAN LOSS FOR SOUNDCTM
Although we propose a new training framework for SoundCTM, the model cannot surpass the teacher model in 1-step generation, achieved in the original CTM. To achieve such 1-step generation performance, integrating the GAN loss into Eq. (7) is required (Sauer et al., 2024; Xu et al., 2024; Lin et al., 2024).
However, obtaining performance improvements via the GAN loss requires careful selection of discriminator. In fact, in preliminary experiments with the 16kHz setting, even though employed several off-the-shelf discriminators in the sound domain (gil Lee et al., 2023; Kumar et al., 2023; lashin and Rahtu, 2021), they did not lead better performance than without using them. We
16

---

Published as a conference paper at ICLR 2025
Table Preliminary experiments with and without using GAN loss. Memory consumption mea- sured with batch size of two per GPU.
Method
of steps
FADvge
KLpast +
puse 1
CLAP
Memory consumption training
w/o. GAN
2
2.43
1.29
7.50
0,42
46782 MiB
4
2.28
1.21
7.63
0.43
w/. GAN
2
2.91
1.37
6.29
0.37
64492 MiB
4
2.37
1.35
7.53
0.41
report one of the preliminary results of using GAN loss in Table 6. this preliminary experiment, use multi-period discriminator (Kong et al., 2020) and multi-band multi-scale complex STFT discriminator (Kumar et al., 2023) by following DAC (Kumar et al., 2023). Along with the lack of performance improvement from using the GAN loss, there are also significant increases in memory consumption. Therefore, in this work, we do not utilize the GAN loss. That said, developing new GAN setups including tailored for large-scale conditional sound generation might be worth exploring as future work.
D EXPERIMENTAL DETAILS
Algorithm 2 SoundCTM's Training
Algorithm 3 SoundCTM's Loss-based Guidance
Require::robbabltty of unconditional training
Punzond
Require: 14, Ycondition 7/s Pt..
1: repeat
1: Start from Zta
Sample (Xu. Cresz) from Piata
2: for π = 0 to N do
3: Calculate Zu through (Xa)
3: [++1
11--72
4:
Cuexi f Ø with Puncond
4: Denoise _i,-- +- vGo (Ze, Cuaa w, tn Ent1)
5:
Sample (0,
5: +(1 vi)oo R,W, (n,tnnntt bnt1)
6:
Sample E 0. T. [0, a (s)
6: Zen to Go (Zen: Crext
7:
Sample =' U [[(wmi, Wmox]
Fcondirca (D((zxlle))
8:
Calculate 2. = Za
9:
Calculate SoLver(Zi,
w, L, ()
8: BE,41 2int1 Pt.. Vas, Clyendttioon Yconditio)
10:
Calculate Buurpr (Rc, Ceexs (
9: Diffuse Ztnt1 Zinti + 7tn-1E
11:
Calculate Zea(Zee Crexts Wl, 1,
10: end for
12:
Update 0 +- 0
Ø 30 L(0)
11: Return Ziy
13:
until converged
D.1 DETAILS OF T2S GENERATION ON 16KHz
Training Details We employ TANGO as the teacher diffusion model. For teacher's training, we
use daata 0.25 and time sampling ~ ( 1.2²) by following EDM's training manner. We
utilize the EDM's skip connection Cskip (t) = to am
output scale Cour VP+al
to input scale
(t(())
and Cuoise (t) = 1 In t for D modeling as Ve+oin
(z(2r, L Ciesr) = Caki (1)ze + Cour (1) (Ginte, Croise (1), Ciest); where F refers the actual neural network to be trained and the actual DSM loss is Exo.A-Poo [A(t)]lzo ( Dæ(z, l()) where X(t) (frre))
d-edy Other than that, we follow the same network architecture, parameter size, and training setups as the original TANGO.
The network architecture of the teacher model and dataset for the teacher's training remained unchanged from the original ones. The network architecture of the teacher model composed the VAE-GAN (Liu et al., 2023), the HiFiGAN vocoder (Kong al., 2020) D and the Stable Diffusion eDcconvolutional ResNet (He et al., 2016) blocks as D&. The UNet has total of 866 M parameters, and the frozen FLAN-T5-Large text encoder (Chung et al., 2022) is used. The UNet employs 8 latent channels and a cross-attention dimension 1024. During teacher model's training, we only train the UNet parts.
UNet architecture (SD-1.5), which consists
17

---

Published as a conference paper at ICLR 2025
For student training, we mostly follow the original CTM's training setup. We utilize the EDM's skip for ge modeling as
connection Cakip (t)
Pin
and output scale Cour (t)
Vo
go (Zt, Crext, w, 1, cakp (()z() + Cour (I)NNN (Z:, Crext; W. (, 1),
where NNg refers to the actual neural network output. We initialize the student's NN with ø except for student model's s-embedding and w-embedding. The network architecture of the student model mostly follows that of the teacher model. However, since we inject time step s-embedding and w-embedding to the student network, we incorporate via auxiliary temporal embedding with positional embedding (Vaswani et al., 2017) add this embedding to the time step t-embedding. As for w-embedding, by following Meng et al. (2023), we apply Fourier embedding to w and add this embedding to the time step I-embedding. In Eq. 9, we employ adaptive weighting with Ve, LeTM (0:b)] where Oz. is the last layer of the student's network by following the original Vo, Croui (9]]
AnSM CTM.
We use 8XNVIDIA H100 (80G) GPUs and a global batch size of 64 for the training. We choose t and from the N-discretized timesteps to calculate CTM. For Sounn calculation, we opt use 50% of time sampling / ~ (- 1.2, 1.2*). ). For the other half time, we first draw sample from & ~ [0, 0.7]
and transform it using 1/p + [ 1/p 1/p) ). Throughout the experiments, for student training. min
we use N = 40, = 0.999, min = 0.002, max 80, 7, R.Adam optimizer (Liu et al., 2020) with learning rate of 8.0 x 10-5 and data 0.25. We set the maximum number of ODE steps as 39
during training and we use Heun solver for Solve veveer (Z,, Cuxt, w, f, u; Ø). We also utilized TANGO's data augmentation (Ghosal et al., 2023, Sec. 2.3) during student training.
Evaluation Details
For large-NFE sampling, we follow the EDM's ' and the CTM's ' time discretiza- tion. Namely, we draw n-NFE samples, we equi-divide (0, 1] with n points and transform it (say E) to the time scale by (amux + (o - ol())))²
1/e
We use four objective metries: the Frechet Audio Distance (FAD vge) (Kilgour et al., 2019) between the extracted embeddings by VGGish, the Kullback-Leibhe divergence (KI (paee) between the outputs of PaSST (Koutini et al., 2022), a state-of-the-arr audio classification model, the Inception Score (IS paSpa (Salimans et al., 2016) using the outputs of PaSST, and the CLAP score?. The lower FAD indicates better audio quality of the generated audio. The KL measures how semantically similar the generated audio is the reference audio. The IS evaluates sample diversity. The CLAP score demonstrates how well the generated audio adheres to the given textual description.
D.2 DETAILS OF FULL-BAND T2S GENERATION
Network Architecture In this experiment, the teacher model is also based on LDM. A variational autoencoder (VAE) (Kingma and Welling, 2014) is used for the encoder and the decoder, compressing the 44. 1kHz monaural waveform to obtain the latent variable Zu. We employ a cully-convolutional architecture, following the Descript Audio Codec (DAC) (Kumar et al., 2023), but without using the residual-vector quantizer. The encoder has layers, each of which downsamples the input audio waveform at rates [4, 4, 8, 8. The decoder has 4 corresponding layers, which upsample at rates [8, 8, 4, 4). We set the encoder dimension to 64 and the decoder dimension to 1536. Thus, an overall data compression ratio of 16 and the model has 63.5M parameters.
For the VAE training, we mainly follow the DAC's training configuration. Since we remove the quantizer and trained a Gaussian VAE, we use a multi-scale Mel-loss, a feature matching loss, an adversarial loss, and a KL-regularization. To balance these four losses, the weighting factors are set to 15.0, 2.0, 1.0, and 10 10-5, respectively. In terms of training dataset for the VAE part, we use AudioSet (Gemmeke et al., 2017) and MUSDB DUSSDBI--HQ 18-HQ (Rafi et al., 2019).
In contrast to TANGO-based SoundCTM, employ DIT in this experiment. Specifically, our DIT architecture mostly follows Stable Audio-2.0 architecture (Evans et al., 2024b) but we remove the timing embedding since we do not aim at variable-length generation. For CLAP text embedding the DIT, we use the "630k-audiosettbes.. CLAP. The model has 1.2B parameters.
checkpoint from ht : / http//ggttttttt//////////t com/ LAION- AI/
We use the "630k-audioset-best. checkpoint from
LA 1ON /CLAP
18

---

Published as a conference paper at ICLR 2025
Table 7: Semantic Preservation Evaluation of SoundCTM-DiT-IB on CLAP-LPAPS (Manor and Michaeli, 2024).
Method
CLAP-LPAPSL
VAE-reconstrruction (Lower bound)
2.55 + 0.39
Deterministic sampling
0
3.34 + 0.74
Stochastic sampling
0.2
5.21 + 0.57
0.5
5.21 + 0.56
1.0 (CM-sampling)
5.34 + 0.65
AC GT test samples
6.07 + 0.41
The network architecture the student model mostly follows that of the teacher model. However, since we inject time step s-embedding and w-embedding to the student network, we incorporate s-information by using sinusoidal embeddings following the time step t-embedding of Stable Audio-2 and add this embedding to the time step t-embedding. As for w-embedding, by following Meng et al. (2023), we apply Fourier embedding to w and add this embedding to the time step 1-embedding. The model has 1.2B parameters.
Training Details For teacher training, we follow the EDM's variance exploding formulation. After obtaining Zo through the encoder, Zo standardized globally to zero mean and standard deviation data 0.5 and we use data = 0.5 and time sampling t ~
by following
EDM2 (Karras et al., 2024). We utilize the EDM's skip connection Cskip (1) = go . output scale
(4) = V
Id
input scale Cin (1) Ve+oz and Caoisee (1) = In/ for D& modeling as
Dølz,,t, t, Guaa) Cski (t)zz + Cou((t) (C Canise (t), Cwx)).
where Fe refers to the actual neural network to be trained and the actual DSM loss is Exg (ooeeal [A(t)llzo - D (z,2,t t, Grev 112). where X(t)
2400 [to
For student training, use NVVIII H100 (80G) GPUs and global batch size of 192 for the train-
ing. We train 10K iterations. In Eq. 9, employ adaptive weighting with ÀDSM [V, Liousl [8:b)]
[V 4 [0]]
where Og is the last layer of the student's s network by following the original CTM. We choose and s from the N-discretized timesteps to calculate [Sound For (Sund calculation, we opt use 50% of time sampling & ~ (-12, 1.22). For the other half time, we first draw sample from & ~ [0, 0.7] and transform it using (o + Eo 1/p
min Trns HiR))) We use N 40, μ 0.96, min 0.002, max 80, P 7, RAAaa optimizer (Liu et al., 2020) with learning rate of 8.0 10 10-5 U[2.0, 5.0], and data = 0.5. We set the maximum number of ODE steps as 39 during training and we use Heun solver for Solver(ze, Crext, W, L, Φ).
Evaluation Details For large-NFE sampling, we follow the EDM's and the CTM's time discretiza- tion. Namely, we draw n-NFE samples, we equi-divide [0, 1] with points and transform it (say ) 1/p to the time scale by (amax + (omin o
D.3 DETAILS OF SOUND INTENSITY CONTROL EXPERIMENT
We define f(x)) == w * 20 log (0(RMS(x))), where w represents the smoothing filter coefficients of a Savitzky- Golay filter (Savitzky and Golay, 1964) with 1-second context window over the frame-wise value, and the RMS is the root mean squared energy the generated sound. The target dBasc target line or curve (See Figure 10 (a) and Figure 11 (a) for example).
condition Ycondition
We use 200 audio-text pairs from the AudioCaps testset for each of the 6 different types of y ycondition as shown in Figure 10 (a) and Figure 11 (a).
For DITTO-2 framework on SoundCTM, we use Adam (Kingma and Ba, 2017) with a learning rate of 1.0. We also tested learning rates of 1.0 10-1, 1.0 10-2, and 5.0 10-3 by following DITTO.
19

---

Published as a conference paper at ICLR 2025
Table 8: Inference speeds measured on NVIDIA A100 with batch size of
Model
channelsksr
# of sampling
Inference speed [sec.]
SoundCTM-DiT-II (Ours)
1/44.1 kitz
0.141
0.298
Audiol CM (Lin al,, 2024c)
1/16 kHz
0.169
0.181
0.220
0.224
Consistency ITA (Bai et al., 2023)
1/16 kHz
0.216
0.253
0.306
0.472
Table 9: Subjective evaluation on AudioCaps testset on 16kHz s6tiin We report overall audio quality (OVAL) and text alignment (REL) with 95% Confidence Interval.
Method
channels/sr
# steps
w
OVAL
REL f
Ground-trut
1/16 kHz
75.5 + 1.33
74.9
AudioL.M (Liu et al., 2024c)
1/16 kHz
1
5.0
49.6 1.51
56,9 1.67
57.2
64.1
SoundCTM
1/16 kHz
1
3.5
1.0
64.0
1.26
69.4 1.49
4
3.5
1.0
67.0
+ 1.35
71,3 1.45
However, we cannot obtain better results than the case using 1.0. The reason why we follow DITTO's setting instead of DITTO-2 the learning rate setting not provided DITTO-2.
During zy-optimization, we use 1-step generation with w - 3.5 and v = 1. We perform 70 iterations for opoomiiinntiion following DITTO's s settings. After the optimization, we perform 16- step generation with w = 3.5 and = 2.0. As suggested in DITTO-2 paper. we include results both using the deterministic sampling (y 0) and the stochastic sampling (7 0.2). For the time-dependent learning rate in SoundCTM's loss-based guidance framework, we use the overall gradient norm by following DITTO (Novaek et al., 2024b, Section 5.4). For time discretization in multi-step generation, use the same scheme as in T2S generation evaluation in Appendix 4.1.
E ADDITIONAL EXPERIMENTS
E. NUMERICAL ANALYSIS FOR DETERMINISTIC SAMPLING
To objectively evaluate the semantic content preservation capability of SoundCTM in deterministic sampling, we calculate sample-wise reconstruction metrics, specifically LPAPS with the CLAP audio encoder (CLAP-LPAPS) (Manor and Michaeli, 2024) by using the toolkit". The evaluation is conducted on 957 samples from the AudioCaps test set. We compute the CLAP-LPAPS between samples generated by SoundCTM-DiT-IB with 1-step and 16-step across several values of n.
Furthermore, to benchmark these CLAP-LPAPS values, we also calculate scores for the following: 1). the ground truth audio samples from the AudioCaps test set and their corresponding reconstructed samples using the VAE in SoundCTM-DiT IB VAAE-rcconttruction)) which reflects signal-level reconstruction score; and 2). ground truth audio samples from the AudioCaps test set and SoundCTMMDiTT-IB 1-step generated samples (AC GT test samples), which reflect semantic-level random score within the same distribution.
We report the average and standard deviation of the CLAP-LPAPS scores in Table 7. Table 7 demonstrates that using deterministic sampling SoundCTM preserves semantic meaning regardless of changes in the number of sampling steps.
"https://g thub. com aManor/Audu ondi http:/uwwwioioioooooooooootooooo wtt.d//tninooooeeeeeee/eeeeeeeee
20

---

Published as a conference paper at ICLR 2025
Table 10: Performance comparisons between SoundCTM-UNet-IB and SoundCTM-DiT-IBB on AudioCaps at full-band setting.
# of sampling steps
Model
channels/sr
W
PDupen13 4 KLpase
CLAP
Diffusion Models
UNet-Teacher w/. Heun Solver
1/44, 1ZHZ
40
5.0
72.2
0.48
DiT-Teucher w/. Heun Solver
1/44,1KHz
40
5.0
68.8
1.59
0.46
Distillation Models
BaundCTM-UNatIBB
144.1KHA
1
5.0
1.0
89.8
2.12
0.26
5.0
5.0
65.1
CT-DD-T-
1KH4
1
5.0
1.0
77.4
1.70
0.35
16
5.0
5.0
65.6
1.22
0.42
Table 11: Ablation study on loss function with SoundCTM-DiT-IB on AudioCaps testset at full-band setting.
Model
# of sampling steps
FD Dppenl3
KL KLppaet |
CLAP
SounJCTM-DiT-IB w/o. DSM loss
5.0
1.0
77.9
1.74
0.35
8
5.0
5.0
93.6
2.28
0.28
16
5.0
5.0
108.7
2.60
0.23
CO--CCTM-
5.0
1.0
77.4
1.70
0.35
8
5.0
5.0
68.1
1.29
0.40
16
5.0
5.0
65.6
1.22
0.42
E.2 INFERENCE SPEED EVALUATION
We compare the inference speed of SoundCTM-DiT-ID with that of AudioLCM and Consistency TTA in Table 8. The inference speeds all models are measured on an NVIDIA A100 with a batch size of 1. As shown in Table 8, we confirm that SoundCTM-DiT. can provide efficient trial-and-eree processes not only in terms of the number of sampling steps but also with respect to inference time.
E.3 SUBJECTIVE EVALUATION WITH SOUNDCTM-UNES ON 16KHZ SETTING
We conduct additional subjective evaluation for the 16 kHz setting. Especially, we focus comparing 16 kHz models AudioLCM and SoundCTM, given the numerical results in Table 2. We follow the same evaluation protocol that use for the subjective evaluation in Section 4.2 (the full-band setting) and we ask 15 participants for the evaluation. As shown Table 9, unlike the trends observed in FAD vgg in Table 2, SoundCTM outperforms AudioLCM, especially in terms of OVAL, even when comparing the results at 4 steps.
E.4 ABLATION STUDIES -OBDLLSL
Performance comparison between UNet and DiT full-band settings We compare SoundCTM- UNet- IB and SoundCTM-DiT-I on AudioCaps test set (957 samples) at full-band settings in Table 10. As shown Table 10, SoundCTM-DiT-IB outperforms UNet-IB -B in 1-step generation. On the other hand, both models demonstrate comparable performance in 16-step generation.
Loss function of DiT We provide ablation study of SoundCTTM-DTT-IB w/. and w/o. the DSM loss Table 11. Note that the DSM loss serves to improve the accuracy of approximating the small jumps conditional and unconditional trajectories during the training. As shown Table 11, while the performance is comparable for 1-step sampling, incorporating the DSM loss leads to improved performance as the number the sampling steps increases. This indicates the effectiveness of the DSM loss.
Teacher's feature extractor of DiT We provide ablation study of SoundCTM-DiT-IE w/. and w/o. the teacher's s feature extractor dhecher- As shown in Table 12, contrast to UNet at 16 kHz case (see in Table 13), the performance for 1-step generation comparable between w/. and w/o. deacher-
21

---

Published as a conference paper at ICLR 2025
Table 12: Ablation study on teacher's feature extractor cheacher with SoundCTM-DiT-ID on AudioCaps testset at full-band setting.
Model
of sampling steps
FDopent3 +
/pssst
CLAP 1
SoundCTM-DiT-IB w/o. deacher
1
5,0
1.0
77.8
1.67
0.36
8
5.0
0.35
16
5.0
5.0
69.2
1.28
0,41
CO-DITT--
1
5.0
1.0
77.4
1.70
0.35
8
5.0
5.0
68.1
1.29
0,40
16
5.0
5,0
63.6
1.22
0.42
Table 13: Ablation study on teacher's feature extractor dheacher with SoundCTM on AudioCaps testset 16kHz.
Model
# of sampling steps
FAD vgs
KLpass
1Spaatt
CLAP
SoundCTM w/o. dieaher
3.5
1.0
2.45
1.28
6.83
0.42
8
3.0
1.5
1.64
1.23
7.56
0.44
16
3.0
2.0
1.62
1.23
7.16
0.44
SoundCTM
3.0
1.0
2.17
1.27
6.83
0.42
3.0
1.5
1.51
1.21
8.08
0.46
3.0
2.0
1.37
1.23
8.31
0.46
However, as the number of sampling steps increased, using feature extractor diecher demonstrates
better performance.
Further ablations on v and w with SounCCTM-DiT-IB To further investigate the effects of u-sampling and w, we train student models by distilling with uniformly sampled from [1.0, 7.0] during training. We evaluate AudioCaps test set 957 samples in terms of FDopea3 and CLAP scores as shown in Figure 12. From the 16-step results, of the w value by v-sampling. On the other hand, examining the CLAP scores for 1-step generation reveals that the models might not be able to generate good samples for any value of when 1. This might highlight the important role of distilling the w-guided trajectory distillation, especially for 1-step generation.
find that we can get better sample quality regardless
E.5 DOWNSTREAM APPLICATION: ZERO-SHOT BANDWIDTH EXTENSION
To further explore the zero-shot downstream application capabilities of SoundCTM, apply it to bandwidth extension task. Specifically, we used 500 samples from the AudioCaps test set, applying low-pass filter with a cut-off frequency of kHz generate observed low-passed signals by following the literature (Moliner et al., 2023). We perform pero-shot bandwidth extension on these observed signals with pretrained SoundCTM. We use the Log-spectral Distance (LSD) metrie (Liu al., 2024a; Wang and Wang, 2021) between the bandwidth extended signals and the ground-truth signals for the evaluation. For the zero-shot bandwidth extension algorithm, we apply strategy (Moliner et al., 2023) to pretrained SoundCTM. As shown in Table 14 and Figure 13 demonstrate the potential zero-shot downstream application capabilities of SoundCTM for bandwidth extension task.
data consistency
F LIMITATIONS
We propose using the teacher's network as the feature extractor for the distillation loss. However, performance improvements may be limited when the student is trained different datasets than the teacher (e.g., training the student on a speech dataset using a teacher model trained on a music dataset).
As an initial step in exploring SoundCTM's capability for training-fre free controllable generation, we experimentally validate that our framework can sound intensity control using loss-based guidance.
22

---

Published as a conference paper at ICLR 2025
Table 14: Zero-shot bandwidth extension experiments on pretrained SoundCTM
Method
of sampling steps
L.D 4
Low- now-passed (Observed signal)
28.8
Bandwidth extension with SoundCTM
13.3
12.9
However, it remains unclear whether this method is applicable to wider range of downstream tasks, which we will address in future work.
23

---

Published as a conference paper at ICLR 2025
ConsistencyTTA
Audiol.CM
SoundCTM
wi, stochastic sampling ("Deeemiiiitti sampling not feasible)
wi, stochastic sampling ("Deterministic sampling not feseilde))
deterministic sampling
Step
2 Step
Step
(a) Input text prompt: "Birds cooing and rustling."
ConsisteneyTTA
AudioLCM
SoundCTM
wi, stochastic sampling
wi, stochastic sampling
with deterministic sampling
campling
feasible)
Step
Step
Step
(b) Input text prompt: "Race cars are racing and skidding tires."
Figure 6: Visualization of spectrograms for generated samples using 1-step, 2-step, and 4-step generation with ConsistencyTTA (Bai et al., 2023), AudioLCM (Liu et al., 2024c), and SoundCTM. As ConsistencyTTA and AudioLCM, CD-based models (Song et al., 2023), inherently does not support deterministic sampling, the content of the generated samples cannot be preserved when varying the number of sampling steps, even when using the same initial noise and text prompts. This variability makes it challenging for users to control the output. In contrast, SoundCTM with deterministic sampling (7 is able to maintain consistent contents even when varying the number of sampling steps.
24

---

Published as a conference paper at ICLR 2025
CM-based sampling
Stochastic
Deterministic
1 Step
Step
16 Step
(a) Input text prompt: "A man speaks followed by another man screaming rapid gunshots."
CM-based sampling
Stochastic
Deterministic
Step
4 Step
16 Step
(b) Input text prompt: "White noise and then birds chirping."
Figure 7: Visualization of spectrograms for generated samples using 1-step, 4-step, and 16-step generation of SoundCTM-DiT-ID with CM-based sampling (7 1), stochastic sampling (r 0.5), and deterministic sampling (y = 0).
25

---

Published as a conference paper at ICLR 2025
CM-based sampling
Stochastic
Deterministic
1 Step
Step
16 Step
Input text prompt: "Rain pitter-patters patters thunder cracks and the wind blows"
CM-based sampling
Stochastic
Deterministic
Step
Step
16 Step
(b) Input text prompt: "Several motor vehicles accelerating."
Figure 8: Visualization spectrograms for generated samples using 1-step, 4-step, and 16-step generation of SoundCTM-DiT-IB with CM-based sampling (7 1), stochastic sampling (r 0.5), and deterministic sampling (y = 0).
26

---

Published as a conference paper at ICLR 2025
Please evaliste the overaal audoo quad by of the sound (DVAL)
The ispar lect prompi "Loud gunshota and exploaiona with man speaking, water splashing, wind blowing, and thunder roaring.
Stop
Cons.s
Exceilent
Good
Fair
Poor
Bad
(a) Screenshot of OVAL evaluation
The evalatt
(REL)
surtace wind blows microphons."
Stop
Cond.2
Cond.8
Play
Excellent
Good
Fair
Bad
100
100
180
190
Screenshot of REL evaluation
Figure 9: Screenshots of subjective evaluation.
27

---

Published as a conference paper at ICLR 2025
Target Intensity
Target Intensity
Target Intensity
Amplitude (dB)
Amplitude (dB)
Amplitude (dB)
Time (s)
Time (s)
Time (s)
(a) Target intensities
Obtained Intensity
Obtained Intensity
Obtained Intensity
Amplitude (dB)
M Mgg
Amplitude (dB)
Amplitude (dB)
WM Magy
Time (s)
Time (s)
Time (s)
(b) Default T2S generation
Obtained Intensity
Obtained Intensity
Obtained Intensity
Mar Mal of MM
Amplitude (dB)
Amplitude (da)
Amplitude (dB)
Time (s)
Time (s)
Time (s)
(c) zr-tmmimiiztiionn (DITTO-2) with 16-step generation
Obtained Intensity
Obtained Intensity
Obtained Intensity
/ ar and wnd wwr M.
Amplitude (dB)
Amplitude (dB)
Amplitude (dB)
Time (s)
Time (s)
Time (s)
(d) Loss-based guidance with 16- st-step generation (Ours)
Figure 10: Target sound intensities and obtained intensities. We use the same text prompt within each column and different prompts across different columns. Note that we use 70 iterations for 27-optimization.
28

---

Published as a conference paper at ICLR 2025
Target Intensity
Target Intensity
Target Intensity
Amplitude (dB)
Amplitude (dB)
Amplitude (dB)
Time (s)
Time (s)
Time (s)
(a) Target intensities
Obtained Intensity
Obtained Intensity
Obtained Intensity
M Mamay
Amplitude (dB)
Amplitude (dB)
mymm
Amplitude (dB)
Time (s)
Time (s)
Time (s)
(b) Default T2S generation
Obtained Intensity
Obtained Intensity
Obtained Intensity
Amplitude (dB)
Amplitude (dB)
Amplitude (dB)
Time (s)
Time (s)
Time (s)
(c) æy opttimizzziiionn (DITTO-2) with 16-step generation
Obtained Intensity
Obtained Intensity
Obtained Intensity
Amplitude (dB)
Amplitude (dB)
Amplitude (dB)
Time (s)
Time (s)
Time (s)
(d) Loss-based guidance with 16-step generation (Ours)
Figure 11: Target sound intensities and obtained intensities. We use the same text prompt within each column and different prompts across different columns. Note that we use 70 iterations for zy-optimization.
29

---

Published as a conference paper at ICLR 2025
1.
90.22
77.02
79.78
1. 0
107.70
88.88
76.88
71.92
3. 0
104.90
3.
106.90
5.0
5. 0
106.90
85.49
74.41
70.77
7.0
75.46
101.70
115.40
7. 0
106.10
73:71
1.
2. 0 V
3. 0
1.0
3.
5. 0
7.0
V
(a) FD upenl3 1-step
(b) Fpepll3 16-step
1. 0
0.22
0.27
0.22
1. 0
3. 0
0.34
0.26
0.16
3.
0.21
0.37
0.40
0.41
5. 0
0.36
0.25
0.12
5. 0
0.21
0.37
0.40
0.41
7.0 0
7.0
1.0
2. 0
3.0 0
1.0
3. 0
5.
0
7.0
V
V
(c) CLAP on 1-step
(d) CLAP on 16-step
Figure 12: Influence of on SounCCTM-DiT-ID with 1- and 16-step sampling. Darker colors indicate better scores. this case, during training, w uniformly sampled from the range [1.0, 7.0] in contrast to main paper.
Zero-shot band extension
Observation (Low-passed)
Ground-truth
SoundCTM
Figure 13: Visualization spectrograms low-passed signals, bendwidth-extended signals, and, ground-truth signals. Vertical axis is in log scale.
30


Reviews:
[{"review_id": "kpofCVL5Ww", "reviewer": "Reviewer_FAko", "summary": "This paper presents a diffusion-based text-to-audio generation system within the Consistency Trajectory Model (CTM) framework, named SoundCTM. SoundCTM is developed using distillation loss with a teacher model, and incorporates a novel sampling algorithm that enables Classifier-Free Guidance (CFG) integration with the student model. Experimental results demonstrate that SoundCTM achieves state-of-the-art performance in 1-step and few-step generation for 44.1 kHz full-band samples, offering a practical tradeoff between generation quality and inference speed for real-world applications.", "questions": "1. Is the hyperparameter 𝑣 mentioned in Section 3, which is used for sampling, trainable? If not, how did the authors determine its value? The same question applies to the scale of CFG 𝑤\n2. For the results in Table 2, what is the best performance SoundCTM can achieve? With additional sampling steps, can SoundCTM surpass the quality of AudioDLM2-large?\n3. What are the differences between SoundCTM and AudioLCM, aside from the teacher CTM loss proposed for training? Both systems are diffusion-based generation models employing consistency strategies.\n4. The authors also state that SoundCTM and AudioLCM show clear trade-offs in multi-step generation. What is the inference efficiency of these two systems? The authors mentioned early on that the proposed system is intended to overcome the barrier of slow inference speeds encountered during trial-and-error.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Results indicate that the proposed large-scale SoundCTM-Dit-1B achieves state-of-the-art performance in text-to-audio generation in the full-band setting (mono at 44.1 kHz).\n2. Instead of using the L2-norm loss between target and output values within the latent domain, this paper introduces a network trained with CTM loss, leveraging a teacher network for feature extraction and measurement. Results show that the proposed teacher loss achieves superior performance.\n3. This paper integrates Classifier-Free Guidance (CFG) within the CTM framework by distilling the CFG-related PF ODE trajectory during training and using it as the CFG condition. Additionally, a new sampling strategy has been developed to enable CFG in 1-step generation.", "weaknesses": "1. The comparison results show that the proposed system achieves state-of-the-art performance in single-step sampling, while SoundCTM does not surpass baseline models (e.g., AudioLCM, also a diffusion model with CTM) when increasing the sampling steps (from 2 steps onward). Additionally, there is a lack of comparison in inference speed between different distillation models.\n2. Further explanation is required for why baseline models outperform SoundCTM in few-step sampling. Moreover, the CLAP score is the only metric related to semantic alignment (text), yet it does not seem well-aligned with other metrics, such as FAD and IS. It would therefore be beneficial to conduct human evaluations for each experiment to better illustrate performance.\n3. The authors claim that SoundCTM-Dit-1B achieves the best results in full-band generation tasks. However, SoundCTM used for comparisons among 16 kHz models employs a UNet-based backbone, resulting in unaligned experiments. The paper lacks sufficient explanations, ablation studies, or comparisons to demonstrate the effectiveness of each submodule. The authors could address this by adding SoundCTM-DIT medium or small models for 16 kHz comparisons and SoundCTM-UNet-1B for full-band comparisons.\n\nOverall, this is an interesting paper exploring the potential of consistency models in sound generation. The model is well-structured, but some modules require further explanation, and additional experiments are needed to illustrate the contributions of each section. I am open to adjusting the score if the authors can address these requirements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmn6JesvaF", "reviewer": "Reviewer_exXk", "summary": "The paper proposes Sound Consistency Trajectory Models (SoundCTM) for improving inference speed and sample quality in Text-to-Sound (T2S) tasks by combining score-based and consistency models. This work mainly adapting existing techniques from image generation for T2S.  From the reported results, SoundCTM can generate sound in few inference step.", "questions": "Please refer to Weaknesses.\n\n1. What your training dataset?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces Sound Consistency Trajectory Models (SoundCTM), an approach that unifies score-based and consistency models for Text-to-Sound (T2S) generation. This combination is interesting and reflects a novel direction within the T2S domain, especially in adapting techniques from image generation. \n\n2. The paper evaluates SoundCTM at both 16kHz and 44.1kHz, showing an awareness of high-fidelity audio generation.", "weaknesses": "1. The paper proposes Sound Consistency Trajectory Models (SoundCTM) to improve inference speed. While the concept of combining score-based and consistency models for T2S generation is interesting, the paper does not convincingly establish the novelty of its contributions. Many of the methods, including ν-sampling and trajectory-based distillation, are either borrowed from existing frameworks or lack adequate differentiation from prior work. A clearer and more distinct innovation beyond adapting methods from image-based models is needed to justify the novelty of this work.\n\n2. The paper emphasizes deterministic sampling as a unique advantage of SoundCTM, particularly for preserving semantic content across sampling steps. However, this advantage remains largely theoretical within the paper, with insufficient empirical evidence demonstrating the effectiveness of deterministic sampling in real production scenarios. The provided spectrograms and subjective evaluations do not strongly support the claimed benefits, and thus, the deterministic aspect feels overstated relative to its demonstrated impact.\n\n3. The paper highlights full-band sound generation. However, all previous work can support full-band sound generation by simply replacing 16kHz audio data with 44.1kHz data.\n\n4. In terms of performance, SoundCTM shows some advantages over AudioLCM and ConsistencyTTA. However, as SoundCTM scales to a 1B parameter model, it is difficult to determine whether the improvements are due to model scaling alone.\n\n5. From an architectural perspective, the proposed DiT framework closely follows the architecture of Stable Audio 2.0.\n\n6. In the related work section, the paper claims that \"when developing sound generative models suitable for production use, LDM-based models present a better choice as the base model.\" It is difficult to agree with this claim. Previous works, such as AudioGen, have also demonstrated good performance in T2S. For example, DiffSound shows that using a discrete diffusion model yields better performance than an AR-based baseline. After that, AudioGen demonstrates that AR-based models can also achieve good performance with different configurations. Given the rapid development of T2S, it is premature to make such a claim. In the field of image generation, some studies have shown that AR-based models can outperform LDM-based models. Additionally, discrete diffusion models remain an important research direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XBuR88z96F", "reviewer": "Reviewer_s79p", "summary": "This paper proposes a new Sound Consistency Trajectory Model (SoundCTM) to accelerate the iterative trial-and-error process of sound design using audio generation models.\n\nCompared with previous 1-step distillation methods, SoundCTM provides deterministic sampling capabilities, which enables acoustic consistency between 1-step and multi-step generations from the same initial noise latent.\n\nCreators can use SoundCTM in the fast 1-step setting for exploratory work while benefitting from the higher quality of multi-step inference once a relevant generated sample has been chosen.\n\nThe authors claim the following contributions.\n- Adaptation of the computer vision domain CTM training framework to the audio domain (called SoundCTM).\n- A novel feature distance for distillation loss computation.\n- v-sampling: a new cfg-like approach that consists in training both conditional and unconditional student models.\n- SoundCTM-DiT-1B, a large scale distillation model that achieves sota 1-step and multi-step full-band text-to-sound generation on the AudioCaps evaluation dataset.\n\nThe results indicate better consistency between 1-step and multi-step generation compared with previous sota.", "questions": "- In v-sampling, it is not clear whether there is one or two student models involved. From the abstract it seems that there are two but not from section 3.3 and algorithm 2.\n- In what sense is v-sampling different than CFG applied to the student model?\n- What is the difference between the last row of Table 1 and the second to last row of Table 2? They do not indicate the same values although they seem to relate to the same experiment.\n- How do you explain that SoundCTM-DiT-1B outperforms the teacher model?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Showcasing a potential downstream application of the proposed method is a nice addition to the work.\n- Speed and inference consistency between 1-step/multi-step inference is a fundamental problem in the sound design space and this paper properly tackles this problem.", "weaknesses": "- It is not clear how v-sampling differs from CFG hence it is difficult to appreciate the relevance of the contribution.\n- It is unclear why the proposed distilled model outperforms its teacher.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DRhoChVKYh", "reviewer": "Reviewer_kUmE", "summary": "This paper introduces SoundCTM, a novel approach to T2S generation that aims to enhance the efficiency and quality of sound creation in multimedia applications such as video games and films. By addressing the limitations of existing diffusion-based T2S models—which often suffer from slow inference speeds and subpar sample quality in 1-step generation—SoundCTM offers a flexible framework that allows for rapid trial-and-error through high-quality 1-step generation and subsequent refinement via multi-step deterministic sampling, all while preserving semantic content. The authors adapt the CTM training framework from computer vision, introducing a new feature distance metric that leverages the teacher network for distillation loss. They also propose a novel ν-sampling algorithm, enhancing the quality of multi-step generation. The model successfully scales up to one billion parameters, achieving promising results in both 1-step and multi-step full-band sound generation.", "questions": "1. Do the authors plan to explore alternative GAN architectures or training strategies to effectively integrate GAN loss and potentially enhance the quality of 1-step generation? Understanding the obstacles faced and possible solutions could be beneficial for future work.\n\n2. How does SoundCTM perform when the teacher and student models are trained on different datasets? Additionally, how well does the model adapt to other audio domains such as music generation or speech synthesis? Insights into its generalizability would be valuable.\n\n3. Can the authors provide more detailed insights into the ν-sampling algorithm? Specifically, how does ν-sampling impact the model's performance compared to standard sampling methods, and what are the theoretical underpinnings that justify its effectiveness?\n\n4. Have the authors considered using external pretrained networks as feature extractors for the distillation loss? If so, how might this affect the model's performance and flexibility, especially in scenarios where the teacher network is not available or differs significantly from the student?\n\n5. How does SoundCTM compare with flow matching methods in terms of performance, computational efficiency, and ease of implementation? Are there specific advantages in choosing SoundCTM over flow matching approaches for text-to-sound generation tasks?\n\n6. Beyond text-to-sound generation, can SoundCTM be applied to other audio-related tasks such as speech synthesis, audio style transfer, or sound enhancement? What adaptations would be necessary, and how might it perform compared to specialized models designed for those tasks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. SoundCTM enables creators to quickly generate initial sound samples using high-quality 1-step generation and then refine these samples through multi-step deterministic sampling without altering the semantic content. This flexibility is highly valuable in practical sound design workflows.\n\n2. By introducing a new feature distance metric that uses the teacher's network as a feature extractor for the distillation loss, the model enhances knowledge transfer from the teacher to the student. This approach leads to better performance compared to traditional L2 loss or external models, as it more effectively captures nuanced features relevant to sound generation.\n\n3. The introduction of ν-sampling is a significant contribution. By combining conditional and unconditional student models, ν-sampling improves the quality of multi-step generation and provides another avenue for quality enhancement.\n\n4. The ability to maintain semantic consistency across different sampling steps is crucial for practical applications. SoundCTM effectively addresses this by enabling deterministic sampling, which is essential for sound designers who require consistent outputs during refinement.", "weaknesses": "1. While the authors mention that preliminary experiments with GAN loss did not yield improvements, they do not deeply explore alternative strategies or provide a thorough analysis of why GAN loss integration was unsuccessful. A more detailed investigation could potentially uncover methods to enhance the model's performance, especially in 1-step generation.\n\n2. Relying on the teacher's network as a feature extractor may limit the model's flexibility, particularly if the teacher and student models are trained on different datasets or domains. This dependency could pose challenges when attempting to generalize the model to new or diverse datasets.\n\n3. The paper briefly demonstrates the application of SoundCTM to downstream tasks like controllable generation but does not explore this in depth. A more comprehensive study on various applications could showcase the versatility of the model.\n\n4. The paper does not provide a comparison between SoundCTM and flow matching approaches. Such a comparison could offer valuable insights into the advantages or limitations of the proposed method relative to other leading techniques in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)