SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation

## Abstract

Abstract Sound content is an indispensable element for multimedia works such as video games, music, and films.
Recent high-quality diffusion-based sound generation models can serve as valuable tools for the creators.
However, despite producing high-quality sounds, these models often suffer from slow inference speeds.
This drawback burdens creators, who typically refine their sounds through trial and error to align them with their artistic intentions.
To address this issue, we introduce Sound Consistency Trajectory Models (SoundCTM).
Our model enables flexible transitioning between high-quality 1 1 1 1 -step sound generation and superior sound quality through multi-step generation.
This allows creators to initially control sounds with 1 1 1 1 -step samples before refining them through multi-step generation.
While CTM fundamentally achieves flexible 1 1 1 1 -step and multi-step generation, its impressive performance heavily depends on an additional pretrained feature extractor and an adversarial loss, which are expensive to train and not always available in other domains.
Thus, we reframe CTM’s training framework and introduce a novel feature distance by utilizing the teacher’s network for a distillation loss.
Additionally, while distilling classifier-free guided trajectories, we train conditional and unconditional student models simultaneously and interpolate between these models during inference.
We also propose training-free controllable frameworks for SoundCTM, leveraging its flexible sampling capability.
SoundCTM achieves both promising 1 1 1 1 -step and multi-step real-time sound generation without using any extra off-the-shelf networks.
Furthermore, we demonstrate SoundCTM’s capability of controllable sound generation in a training-free manner.
Our codes, pretrained models, and audio samples are available at https://github.com/sony/soundctm .

## 1 Introduction

Sound content, including Foley sounds (Great Big Story, ; WIRED, ) and sound effects, is essential for multimedia works such as video games, music, and films.
Foley artists create sounds such as footsteps, breaking glass, and environmental sounds using physical items or by mixing and splicing digital sounds aligned with the visuals of multimedia works. To enhance immersive experiences, sound creators strive to produce flexible, diverse, and high-quality sound content tailored to each project.

Sound generation models have a potential to be valuable tools for sound creators.
Recent diffusion-based Text-to-Sound (T2S) models (Liu et al., , ; Ghosal et al., ; Huang et al., ) have shown promising results. Some models can condition on other modalities (Luo et al., ; Tang et al., ) or use a human-in-the-loop approach (Majumder et al., ). Despite producing high-quality sounds, these models suffer from slow inference speeds due to the iterative sampling in Diffusion Models (DMs) (Sohl-Dickstein et al., ; Song et al., ; Karras et al., ).
Sound creators must iterate to ensure the generated sounds align with their creative intentions, a process requiring them to listen to each sample individually. In contrast to images, where creators can quickly determine if the samples meet their expectations, assessing sounds is time-consuming. Thus, slow inference adds a significant burden and time to the creative process.
Addressing slow inference makes sound generation models more appealing to sound creators.

Figure: Figure 1: Performance comparison on AudioCaps testset. Real-time factors (RTFs) are measured on a single NVIDIA RTX A6000 and Intel Xeon CPU. SoundCTM can switch $1$-step generation and multiple-step generation.
Refer to caption: x1.png

An initial attempt to address slow inference for DM-based sound generation models is ConsistencyTTA (Bai et al., ), which uses Consistency Distillation (CD) (Song et al., ) on T2S Latent Diffusion Models (LDMs) (Rombach et al., ).
However, ConsistencyTTA focuses on $1$-step generation, losing the flexibility of balancing sample quality with inference speed (see Table 6).
This is another crucial issue for creators who need accurate, high-quality samples after finding suitable conditions for the models.
This limitation stems from CD’s training regime that learns anytime-to-zero-time jumps.

In this paper, we present the *Sound Consistency Trajectory Model* (SoundCTM), a novel T2S model that offers flexible switching between $1$-step high-quality sound generation and higher-quality multi-step generation.
This is achieved through a training framework that learns anytime-to-anytime jumps and employs deterministic sampling as proposed in Consistency Trajectory Models (CTMs) (Kim et al., ).
Built on CTM’s framework, we address the limitations of the current CTM training framework (see Section3), which heavily depends on domain-specific components to achieve notable generation performance.
Specifically, we propose a novel domain-agnostic feature distance that uses a teacher’s network as a feature extractor for distillation loss (see Section 4.1).
Furthermore, we distill classifier-free guided text-conditional trajectories with Classifier-Free Guidance (CFG) (Ho and Salimans, ) as a new condition for student models. We train both text-conditional and unconditional student models simultaneously and introduce a new scaling term to interpolate text-conditional and unconditional neural jumps for SoundCTM’s sampling (see Section4.2).

In addition, we explore SoundCTM’s capabilities for controllable sound generation without additional training. Following the recent success of DITTO (Novack et al., ), which achieves state-of-the-art (SOTA) training-free controllable music generation performance by optimizing an initial noise of DMs, we address its significant generation time using SoundCTM’s $1$-step generation.
Furthermore, we also propose a loss-based guidance (Yu et al., ; Levy et al., ) framework for SoundCTM by leveraging its anytime-to-anytime jump capability, which is another major method of DM-based trainig-free controllable generation (see Section 5).

In the experiments, SoundCTM achieves SOTA $1$-step generation performance without any fine-tuning or auxiliary neural networks. Under multi-step sampling, SoundCTM surpasses other T2S models with real-time generation on a single NVIDIA RTX A6000 (see Figure 1 and Table 5).
Additionally, we demonstrate SoundCTM’s capability of training-free controllable sound generation.
Lastly, but noteworthy, our new frameworks are also applicable to other modalities since they do not rely on domain-specific components.
We believe SoundCTM can open a new door to significantly enhancing the efficiency of sound creation workflows with its fast, flexible, controllable, and high-quality generation. It also paves the way for real-time dynamic sound generation in live performances, exhibitions, and video games.

Our contributions are summarized as:

- •
We present SoundCTM, capable of flexibly switching between high-quality $1$-step and higher-quality multi-step sound generation.
- •
To achieve high-quality sound generation, we address the limitations of the CTM’s training framework by proposing a novel domain-agnostic feature distance and methods for distilling classifier-free guided trajectories and using text-conditional and unconditional jump for sampling.
- •
We also explore SoundCTM’s capability for training-free controllable generation.

## 2 Preliminary

#### Diffusion Models

Let $p_{\text{data}}$ denote the data distribution.
In DMs, the data variable $\mathbf{x}_{0}\sim p_{\text{data}}$ is generated through a reverse-time stochastic process (Anderson, ) defined as $\mathop{}\!\mathrm{d}\mathbf{x}_{t}=-2t\nabla\log p_{t}(\mathbf{x}_{t})dt+
\sqrt{2t}\mathop{}\!\mathrm{d}\bar{\mathbf{w}}_{t}$ from time $T$ to 0 0, where $\bar{\mathbf{w}}_{t}$ is the standard Wiener process in reverse-time. The marginal density $p_{t}(\mathbf{x})$ is obtained by encoding $\mathbf{x}_{0}$ along with a fixed forward diffusion process, $\mathop{}\!\mathrm{d}\mathbf{x}_{t}=\sqrt{2t}\mathop{}\!\mathrm{d}\mathbf{w}_{t}$, initialized by $\mathbf{x}_{0}$, where $\mathbf{w}_{t}$ is the standard Wiener process in forward-time.
Song et al. () present the deterministic counterpart of the reverse-time process, called the *Probability Flow* Ordinary Differential Equation (PF ODE), given by

$$ $\frac{\mathop{}\!\mathrm{d}\mathbf{x}_{t}}{\mathop{}\!\mathrm{d}t}=-t\nabla \log p_{t}(\mathbf{x}_{t})=\frac{\mathbf{x}_{t}-\mathbb{E}_{p_{t|0}(\mathbf{x} |\mathbf{x}_{t})}[\mathbf{x}|\mathbf{x}_{t}]}{t},$ $$

where $p_{t|0}(\mathbf{x}|\mathbf{x}_{t})$ is the probability distribution of the solution of the reverse-time stochastic process from time $t$ to 0 0, initiated from $\mathbf{x}_{t}$. $\mathbb{E}_{p_{t|0}(\mathbf{x}|\mathbf{x}_{t})}[\mathbf{x}|\mathbf{x}_{t}]=
\mathbf{x}_{t}+t\nabla\log p_{t}(\mathbf{x}_{t})$ is a denoiser function(^1^11For simplicity, we omit $p_{t|0}(\mathbf{x}|\mathbf{x}_{t})$, a subscript in the expectation of the denoiser, throughout the paper.) (Efron, ).

Practically, the denoiser $\mathbb{E}[\mathbf{x}|\mathbf{x}_{t}]$ is estimated by a neural network $D_{\bm{\phi}}$, obtained by minimizing a Denoising Score Matching (DSM) loss (Vincent, ; Song et al., ) $\mathbb{E}_{\mathbf{x}_{0},t,p_{0|t}(\mathbf{x}|\mathbf{x}_{0})}[\|\mathbf{x}_
{0}-D_{\bm{\phi}}(\mathbf{x},t)\|_{2}^{2}]$, where $p_{0|t}(\mathbf{x}|\mathbf{x}_{0})$ is the transition probability from time 0 0 to $t$, initiated with $\mathbf{x}_{0}$. Given the trained denoiser, the empirical PF ODE is given by

$$ $\displaystyle\frac{\mathop{}\!\mathrm{d}\mathbf{x}_{t}}{\mathop{}\!\mathrm{d}{ t}}=\frac{\mathbf{x}_{{t}}-D_{\bm{\phi}}(\mathbf{x}_{t},{t})}{{t}}.$ (1) $$

DMs can generate samples by solving the empirical PF ODE, initiated with $\mathbf{x}_{T}$, which is sampled from a prior distribution $\pi$ approximating $p_{T}$.

#### Text-Conditional Sound Generation with Latent Diffusion Models

LDM-based T2S models (Liu et al., , ; Ghosal et al., ) generate audio matched to textual descriptions by first obtaining the latent counterpart of the data variable $\mathbf{z}_{0}$ through the reverse-time process conditioned by text embedding $\mathbf{c}_{\text{text}}$.
This latent variable $\mathbf{z}_{0}$ is then converted to $\mathbf{x}_{0}$ using a pretrained decoder $\mathcal{D}$.
During the training phase, $D_{\bm{\phi}}$ is trained by minimizing the DSM loss $\mathbb{E}_{\mathbf{z}_{0},t,p_{0|t}(\mathbf{z}|\mathbf{z}_{0})}[\|\mathbf{z}_
{0}-D_{\bm{\phi}}(\mathbf{z},t,\mathbf{c}_{\text{text}})\|_{2}^{2}]$, where $p_{0|t}(\mathbf{z}|\mathbf{z}_{0})$ is the latent counterpart of the transition probability from time 0 0 to $t$, initiated with $\mathbf{z}_{0}$. $\mathbf{z}_{0}$ is given by a pretrained encoder as $\mathbf{z}_{0}=\mathcal{E}(\mathbf{x}_{0})$.
We refer to Appendix A for a review of related work.

#### Consistency Trajectory Models

CTMs predict both infinitesimally small step jump and long step jump of the PF ODE trajectory.
$G(\mathbf{x}_{t},t,s)$ is defined as the solution of the PF ODE from initial time $t$ to final time $s\leq t$ and $G$ is estimated by $G_{\bm{\theta}}$ as the neural jump. To train $G_{\bm{\theta}}$, a *soft consistency matching* distillation loss (CTM loss) is introduced, comparing two $s$-predictions: one from a teacher $\bm{\phi}$ and the other from a student $\bm{\theta}$ as:

$$ $\displaystyle G_{\bm{\theta}}(\mathbf{x}_{t},t,s)\approx G_{\texttt{sg}(\bm{ \theta})}\big{(}\texttt{Solver}(\mathbf{x}_{t},t,u;\bm{\phi}),u,s\big{)},$ $$

where $\texttt{Solver}(\mathbf{x}_{t},t,s;\bm{\phi})$ is the pre-trained PF ODE in Eq. (1), a random $u\in[s,t)$ determines the amount of teacher information to distill, and sg is the exponential moving average (EMA) stop-gradient $\texttt{sg}(\bm{\theta})\leftarrow\texttt{stopgrad}(\mu\texttt{sg}(\bm{\theta}
)+(1-\mu)\bm{\theta})$.
To quantify the CTM loss between the student prediction $G_{\bm{\theta}}(\mathbf{x}_{t},t,s)$ and the teacher prediction $G_{\texttt{sg}(\bm{\theta})}(\texttt{Solver}(\mathbf{x}_{t},t,u;\bm{\phi}),u,s)$, the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., ) is used to measure a feature distance $d_{\text{feat.}}$ after transporting both predictions from $s$-time to 0 0-time as $\mathbf{x}_{\text{est}}(\mathbf{x}_{t},t,s):=G_{\texttt{sg}(\bm{\theta})}(G_{
\bm{\theta}}(\mathbf{x}_{t},t,s),s,0)$ and $\mathbf{x}_{\text{target}}(\mathbf{x}_{t},t,u,s):=G_{\texttt{sg}(\bm{\theta})}
(G_{\texttt{sg}(\bm{\theta})}\big{(}\texttt{Solver}(\mathbf{x}_{t},t,u;\bm{
\phi}),u,s\big{)},s,0)$.
Summarizing, the CTM loss is defined as

$$ $\displaystyle\mathcal{L}_{\text{CTM}}(\bm{\theta};\bm{\phi})$ $\displaystyle:=\mathbb{E}_{t\in[0,T]}\mathbb{E}_{s\in[0,t]}\mathbb{E}_{u\in[s, t)}\mathbb{E}_{\mathbf{x}_{0}}\mathbb{E}_{\mathbf{x}_{t}|\mathbf{x}_{0}}\Big{[ }d_{\text{feat.}}\big{(}\mathbf{x}_{\text{target}}(\mathbf{x}_{t},t,u,s), \mathbf{x}_{\text{est}}(\mathbf{x}_{t},t,s)\big{)}\Big{]}.$ $$

To enhance student learning, Kim et al. () introduce two auxiliary losses, the DSM loss and an adversarial loss (GAN loss) (Goodfellow et al., ), defined as:

$$ $\displaystyle\mathcal{L}_{\text{DSM}}(\bm{\theta})=\mathbb{E}_{t,\mathbf{x}_{0 }}\mathbb{E}_{\mathbf{x}_{t}|\mathbf{x}_{0}}[\|\mathbf{x}_{0}-g_{\bm{\theta}}( \mathbf{x}_{t},t,t)\|_{2}^{2}],$ $$

and

$$ $\displaystyle\mathcal{L}_{\text{GAN}}(\bm{\theta},\bm{\eta})=\mathbb{E}_{ \mathbf{x}_{0}}[\log{d_{\bm{\eta}}(\mathbf{x}_{0})}]+\mathbb{E}_{t\in[0,T]} \mathbb{E}_{s\in[0,t]}\mathbb{E}_{\mathbf{x}_{0}}\mathbb{E}_{\mathbf{x}_{t}| \mathbf{x}_{0}}\big{[}\log{\big{(}1-d_{\bm{\eta}}(\mathbf{x}_{\text{est}}( \mathbf{x}_{t},t,s)})\big{)}\big{]},$ $$

where $d_{\bm{\eta}}$ is a discriminator.
Therefore, the student model is trained with the following objective:

$$ $\displaystyle\mathcal{L}(\bm{\theta},\bm{\eta}):=\mathcal{L}_{\text{CTM}}(\bm{ \theta};\bm{\phi})+\lambda_{\text{DSM}}\mathcal{L}_{\text{DSM}}(\bm{\theta})+ \lambda_{\text{GAN}}\mathcal{L}_{\text{GAN}}(\bm{\theta},\bm{\eta}),$ $$

where $\lambda_{\text{DSM}}$ and $\lambda_{\text{GAN}}$ are scaling terms of the corresponding losses.

## 3 Limitations of CTM

#### Discriminator Selection Sensitivity for Performance Improvement

Although CTMs demonstrate impressive $1$-step image generation performance, this performance heavily depends on the GAN loss (See (Kim et al., , Fig.12)).
However, obtaining performance improvements via the GAN loss requires careful selection of a discriminator.
In fact, in our preliminary experiments, none of the off-the-shelf discriminators in the audio domain (gil Lee et al., ; Kumar et al., ; Shibuya et al., ; Iashin and Rahtu, ) lead better performance(^2^22This could be attributed to the lack of suitable for generative tasks in the audio domain.).
Therefore, it is essential to develop a new methodology for high-quality sound generation that does not rely on the GAN loss.

#### Increase of Memory Consumption with CTM Loss for LDM

Applying the CTM loss directly to LDM-based sound generation models results in substantial memory consumption.
In the audio domain, the original CTM loss requires measuring a feature distance to obtain better performance using the Learned Perceptual Audio Patch Similarity (LPAPS)(^3^33LPAPS is the audio variant of LPIPS. Other off-the-shelf pretrained networks such as VGGish (Hershey et al., ), PaSST (Koutini et al., ), and CLAP (Wu* et al., ) could be used as feature extractors. However, these models are trained at different sampling frequencies ($16$ kHz, $32$kHz, and $48$kHz, respectively), necessitating resampling the audio from the training data’s sampling frequency to that of the pretrained feature extractors. After resampling, the waveform must be transformed into another domain, such as the Mel-spectrogram domain, to match the input format of the extractors.) (Iashin and Rahtu, ) as a feature extractor, and $\mathbf{z}_{0}$ is required to be converted back to $\mathbf{x}_{0}$ through $\mathcal{D}$ to align with the extractor’s input format.
Additionally, despite the concept of LPAPS, there is no established checkpoint available(^4^44The original checkpoint is no longer available according to the authors’ repository [https://github.com/v-iashin/SpecVQGAN/issues/13](https://github.com/v-iashin/SpecVQGAN/issues/13).).
Therefore, we propose a new framework that reduces memory consumption while maintaining the benefits of using the feature distance (see Section 4.1).

## 4 SoundCTM

Figure: Figure 2: Illustrations of SoundCTM’s two predictions $\mathbf{z}_{\text{target}}$ and $\mathbf{z}_{\text{est}}$ at time $s$ with an initial value $\mathbf{z}_{t}$ and the feature extraction by the teacher’s network for the CTM loss shown within the blue ellipse area. All the parameters of the teacher’s network are frozen. The conditional embedding $\mathbf{c}$ and time $s$ are also input to the feature extractor. Note that the teacher’s network does not need to follow the UNet architecture (Ronneberger et al., ); a partial teacher’s network can also be used (see Table 3).
Refer to caption: x2.png

To address the challenges of achieving fast, flexible, and high-quality T2S generation, we introduce SoundCTM by reframing the CTM’s training framework.
Consistent with the CTM, we use the same distillation loss.
Since this paper primarily illustrates the method using LDM-based T2S models as the teacher model,
the student model is trained to estimate the neural jump $G_{\bm{\theta}}$ as:

$$ $\displaystyle G_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},t,s) \approx G_{\texttt{sg}(\bm{\theta})}\big{(}\texttt{Solver}(\mathbf{z}_{t}, \mathbf{c}_{\text{text}},t,u;\bm{\phi}),\mathbf{c}_{\text{text}},u,s\big{)},$ $$

where $\mathbf{z}_{t}$ is the latent counterpart of $\mathbf{x}_{t}$, and $\texttt{Solver}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},t,s;\bm{\phi})$ is the numerical solver of the pre-trained text-conditional PF ODE.
To quantify the dissimilarity between $G_{\bm{\theta}}$ and $G_{\texttt{sg}(\bm{\theta})}$, we propose a new feature distance in Section 4.1.

### 4.1 Teacher’s Network as Feature Extractor for CTM Loss

**Table 1: Difference of neural network function call between the distillation loss design**
| CTM loss design | $s$-time to 0 0-time | $\mathbf{z}_{0}$ to $\mathbf{x}_{0}$ |
| --- | --- | --- |
| projection | projection |  |
| CTM (Kim et al., ) | Required | Required |
| SoundCTM | Free | Free |

To address the limitations discussed in Section 3, we propose a new training framework that utilizes the teacher’s network as a feature extractor and measures the feature distance $d_{\text{teacher}}$ for the CTM loss, as illustrated in Figure 2. Utilizing $d_{\text{teacher}}$ offers two benefits:

- •
We can reduce memory consumption by directly measuring the dissimilarity of the two predictions at $s$-time, avoiding the requirement to transport the teacher’s and student’s predictions from $s$-time to 0 0-time.
This original CTM’s transportation is done by using $G_{\texttt{sg}(\bm{\theta})}(\cdot,\mathbf{c}_{\text{text}},s,0)$ and projecting $\mathbf{z}_{0}$ to $\mathbf{x}_{0}$ through $\mathcal{D}$, as shown in Table 1.
- •
Using $d_{\text{teacher}}$ yields better performance compared with using the $l_{2}$ distance in the $\mathbf{z}$ domain computed at either 0 0-time or $s$-time, as demonstrated in Section 6.

We define $d_{\text{teacher}}$ between two predictions $\mathbf{z}_{\text{target}}$ and $\mathbf{z}_{\text{est}}$ as follows:

$$ $\displaystyle d_{\text{teacher}}(\mathbf{z}_{\text{target}},\mathbf{z}_{\text{ est}},\mathbf{c},t)$ $\displaystyle=\sum_{m=1}^{M}\|\text{TN}_{\bm{\phi},m}(\mathbf{z}_{\text{target }},\mathbf{c},t)-\text{TN}_{\bm{\phi},m}(\mathbf{z}_{\text{est}},\mathbf{c},t) \|_{2}^{2},$ (2) $$

where $\text{TN}_{\bm{\phi},m}(\cdot,\mathbf{c},t)$ denotes the channel-wise normalized output feature of the $m$-th layer of the pretrained teacher’s network, conditioned by time $t$ and embedding $\mathbf{c}$.
This approach is feasible since noisy latents are input to the teacher’s network during teacher’s training.

### 4.2 CFG Handling for SoundCTM

To manage CFG in SoundCTM, we propose distilling the classifier-free guided PF ODE trajectory scaled by $\omega$, uniformly sampled from the range $[\omega_{\text{min}},\omega_{\text{max}}]$ during training, and using $\omega$ as a new condition in the student network, defined as:

$$ $\displaystyle G_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t ,s)=\frac{s}{t}\mathbf{z}_{t}+\Big{(}1-\frac{s}{t}\Big{)}g_{\bm{\theta}}( \mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t,s).$ $$

The CTM loss for SoundCTM is formulated as:

$$ $\displaystyle\scalebox{0.86}{ $\mathcal{L}_{\text{CTM}}^{\text{Sound}}(\bm{\theta};\bm{\phi}):=\mathbb{E}_{t \in[0,T]}\mathbb{E}_{s\in[0,t]}\mathbb{E}_{u\in[s,t)}\mathbb{E}_{\mathbf{z}_{0 }}\mathbb{E}_{\mathbf{z}_{t}|\mathbf{z}_{0}}\Big{[}d_{\bf{\text{teacher}}}\big {(}\mathbf{z}_{\text{target}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t ,u,s),\mathbf{z}_{\text{est}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t ,s),\mathbf{c}_{\text{text}},s\big{)}\Big{]}$},$ (3) $$

where

$$ $\displaystyle\mathbf{z}_{\text{target}}(\mathbf{z}_{t},\mathbf{c}_{\text{text} },\omega,t,u,s)$ $\displaystyle:=G_{\texttt{sg}(\bm{\theta})}(\texttt{Solver}(\mathbf{z}_{t}, \mathbf{c}_{\text{text}},\omega,t,u;\bm{\phi}),\mathbf{c}_{\text{text}},\omega ,u,s),$ $\displaystyle\mathbf{z}_{\text{est}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}}, \omega,t,s)$ $\displaystyle:=G_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega, t,s),$ $$

$\texttt{Solver}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t,u;\bm{\phi}):
=\omega\texttt{Solver}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},t,u;\bm{\phi})+
(1-\omega)\texttt{Solver}(\mathbf{z}_{t},\varnothing,t,u;\bm{\phi})$, $\varnothing$ is an unconditional embedding, and $\omega\sim U[\omega_{\text{min}},\omega_{\text{max}}]$, respectively.

For auxiliary losses, as discussed in Section 3, we eliminate the GAN loss and use only the DSM loss, defined as:

$$ $\displaystyle\mathcal{L}_{\text{DSM}}^{\text{Sound}}(\bm{\theta})=\mathbb{E}_{ t,\mathbf{z}_{0},\omega}\mathbb{E}_{\mathbf{z}_{t}|\mathbf{z}_{0}}[\|\mathbf{z }_{0}-g_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t,t)\|_{2 }^{2}]$ (4) $$

Summing Eqs. (3) and (4), SoundCTM is trained with the following objective:

$$ $\displaystyle\mathcal{L}(\bm{\theta}):=\mathcal{L}_{\text{CTM}}^{\text{Sound}} (\bm{\theta};\bm{\phi})+\lambda_{\text{DSM}}\mathcal{L}_{\text{DSM}}^{\text{ Sound}}(\bm{\theta}).$ (5) $$

Following the original CTM, we employ adaptive weighting with $\lambda_{\text{DSM}}=\frac{\|\nabla_{\bm{\theta}_{L}}\mathcal{L}_{\text{CTM}}^
{\text{Sound}}(\bm{\theta};\bm{\phi})\|}{\|\nabla_{\bm{\theta}_{L}}\mathcal{L}
_{\text{DSM}}^{\text{Sound}}(\bm{\theta})\|}$, where $\theta_{L}$ is the last layer of the student’s network. Algorithm 2 summarizes SoundCTM’s training framework.

#### ν 𝜈 \nu italic_ν -interpolation for SoundCTM’s Sampling

We introduce $\nu$-interpolation in SoundCTM’s sampling, which linearly interpolates between the text-conditional and unconditional student models, given by:

$$ $\displaystyle\mathbf{z}_{s|t}=\nu G_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{ \text{text}},\omega,t,s)+(1-\nu)G_{\bm{\theta}}(\mathbf{z}_{t},\varnothing, \omega,t,s).$ $$

By adding $\omega$ as a new condition to the student network, the student model interpolates between the $\omega$-conditioned text-conditional and $\omega$-conditioned unconditional models during inference sampling. Algorithm 3 summarizes SoundCTM’s sampling framework, with the sampling timesteps denoted as $T=t_{0}>\cdots>t_{N}=0$.
We show the influence of $\omega$ and $\nu$ on the SoundCTM’s performance in Section 6.

## 5 Towards Training-free Controllable T2S Generation with SoundCTM

In the music domain, DITTO (Novack et al., ) that optimizing an initial noise latent $\mathbf{z}_{T}$ of DMs achieves decent performance for training-free controllable music generation.
Specifically, $\mathbf{z}_{T}$ is optimized by

$$ $\displaystyle\mathbf{z}_{T}^{*}=\operatorname*{arg\,min}_{\mathbf{z}_{T}} \mathcal{L}(f(\mathbf{x}_{0}),\mathbf{y}_{\text{condition}}),$ (6) $$

where $\mathbf{z}_{T}^{*}$ is the optimized output, $f(\cdot)$ is a differentiable feature extractor that converts $\mathbf{x}_{0}$ into the same space as the target condition $\mathbf{y}_{\text{condition}}$, and $\mathbf{x}_{0}=\mathcal{D}(\mathbf{z}_{0})$. Although DITTO shows promising performance, it requires substantial time to generate a sample since $\mathbf{x}_{0}$ and $\mathbf{z}_{0}$ is given by the $N$-timestep reverse diffusion process in each optimization iteration(^5^55In their study, $20$ diffusion steps are conducted per iteration, and the number of optimization steps ranges from $70$ to $150$, resulting in a total of approximately $1,400$ to $3,000$ diffusion steps.).
Novack et al. () also report the performance of a loss-based guidance method (Yu et al., ; Levy et al., ) as a baseline, which is another major method for controllable generation, using the same loss $\mathcal{L}(f(\mathbf{x}_{0}),\mathbf{y}_{\text{condition}})$, and DITTO outperforms the guidance-based method.

Table: Algorithm 1 SoundCTM’s Loss-based Guidance Framework

To explore SoundCTM’s capability for controllable generation in a training-free manner, we propose two new frameworks for SoundCTM based on $\mathbf{z}_{T}$-optimization and loss-based guidance. Firstly, we dramatically accelerate $\mathbf{z}_{T}$-optimization by utilizing SoundCTM’s $1$-step generation, as shown in Algorithm 4, making each iteration $N$ times faster than DITTO.

#### Loss-based Guidance Framework for SoundCTM

In DMs, the loss-based guidance method involves an additional update $\mathbf{z}_{t-1}=\mathbf{z}_{t-1}-\rho_{t}\nabla_{\mathbf{z}_{t}}\mathcal{L}(f
(\hat{\mathbf{z}}_{0}(\mathbf{z}_{t})),\mathbf{y}_{\text{condition}})$ during sampling, where $\hat{\mathbf{z}}_{0}(\mathbf{z}_{t})$ is a clean estimate derived from $\mathbf{z}_{t}$ using Tweedie’s formula (Efron, ), and $\rho_{t}$ is a learning rate.
We propose replacing $\hat{\mathbf{z}}_{0}(\mathbf{z}_{t})$ with $\mathbf{z}_{0|t}=G_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},
\omega,t,0)$ and conducting loss-based sampling within SoundCTM’s $\gamma$-sampling, as shown in Algorithm 1. We denote the sampling timesteps as $T=t_{0}>\cdots>t_{N}=0$.

By leveraging the anytime-to-anytime jump capability, SoundCTM can achieve both fast $\mathbf{z}_{T}$-optimization with $1$-step sampling and multiple-step controllable generation with loss-based guidance within a single model. This is not possible with either a single Consistency Model (CM) (Song et al., )-based T2S model or a DM-based T2S model.

## 6 Experiments

### 6.1 T2S Generation Performance

We evaluate SoundCTM on the AudioCaps dataset (Kim et al., ), which contains $47,289$ pairs of $10$-second audio samples and human-written text descriptions for the training set and $957$ pairs for the testset. All audio samples are downsampled to $16$ kHz. We adopt TANGO (Ghosal et al., ) as the teacher model trained with EDM’s variance exploding formulation (Karras et al., ).
We use $\gamma=0$ for sampling and evaluate the model performance with
student EMA rate $\mu=0.999$.

**Table 2: Performance comparisons among teacher and student models. DDIM sampling is used for TANGO-ConsistencyTTA and Heun solver is used for TANGO-EDM.**
| Model | # of steps | $\rm{FAD}_{vgg}\downarrow$ |
| --- | --- | --- |
| Teacher Models |  |  |
| TANGO-ConsistencyTTA (reported) | 200 | 1.91 |
| (Teacher of ConsistencyTTA) |  |  |
| TANGO-EDM | 40 | 1.71 |
| (Teacher of Ours) |  |  |
| Student Models |  |  |
| ConsistencyTTA (reported) | 1 | 2.58 |
| SoundCTM w/. $l_{2}$ (0 0-time step) | 1 | 2.43 |
| SoundCTM w/. $l_{2}$ ($s$-time step) | 1 | 2.45 |
| SoundCTM w/. $d_{\text{teacher}}$ | 1 | 2.17 |

#### Evaluation Metrics

We use four objective metrics: the Frechet Audio Distance ($\rm{FAD}_{\text{vgg}}$) (Kilgour et al., ) between the extracted embeddings by VGGish, the Kullback-Leibler divergence ($\rm{KL}_{\text{passt}}$) between the outputs of PaSST (Koutini et al., ), a state-of-the-art audio classification model, the Inception Score ($\rm{IS}_{\text{passt}}$) (Salimans et al., ) using the outputs of PaSST, and the CLAP score(^6^66We use the ”630k-audioset-best.pt” checkpoint from [https://github.com/LAION-AI/CLAP](https://github.com/LAION-AI/CLAP)). The lower FAD indicates better audio quality of the generated audio. The KL measures how semantically similar the generated audio is to the reference audio. The IS evaluates sample diversity. The CLAP score demonstrates how well the generated audio adheres to the given textual description.

#### Effectiveness of Utilizing Teacher’s Network as Feature Extractor

We first evaluate the efficacy of utilizing the teacher’s network as a feature extractor for the CTM loss by comparing the following cases: 1) the $l_{2}$ at 0 0-time step, 2) the $l_{2}$ at $s$-time step, and 3) the $d_{\text{teacher}}$ at $s$-time step. We report the FAD of $1$-step generation with $\omega=3.5$ and $\nu=1.0$ for inference.
As shown in Table 3, the FAD of the $d_{\text{teacher}}$ case is clearly better than the others. This result indicates that using $d_{\text{teacher}}$ allows the student to distill the trajectory more accurately than using $l_{2}$.

Additionally, we compare SoundCTM’s performance with the $1$-step generation of ConsistencyTTA (Bai et al., ).
The FAD of SoundCTM’s $1$-step generation trained with $l_{2}$ at 0 0-time step is $2.43$, which is $0.15$ points lower than that of ConsistencyTTA’s $1$-step generation, as shown in Table 3.
ConsistencyTTA is trained with the CD manner with the $l_{2}$ distance at the 0 0-time step of $\mathbf{z}$.
This performance gap is likely due to differences in each teacher model.
On the other hand, using $d_{\text{teacher}}$ further boosts SoundCTM’s performance compared with using $l_{2}$.
This indicates that SoundCTM with $d_{\text{teacher}}$ surpasses ConsistencyTTA even considering of the teacher difference.

Furthermore, we compare the results of using the teacher’s entire network with those of using the first half of it, as shown in Table 3. The results are not significantly different, indicating that there is potential that the model can reduce memory consumption by using only part of the teacher’s network without performance degradation.

#### Influence of ω 𝜔 \omega italic_ω and ν 𝜈 \nu italic_ν for SoundCTM’s Sampling

We examine the influence of both $\omega$ and $\nu$ on SoundCTM’s performance.
In Figure 3, we compare SoundCTM trained with $\omega\sim U[2.0,5.0]$ to $\omega\sim U[1.5,7.0]$ across various $\omega$ and $\nu$ values during inference.
Overall, there is no significant difference in FAD between $\omega\sim U[2.0,5.0]$ (solid line) and $\omega\sim U[1.5,7.0]$ (dashed line).
This suggests that precisely pre-defining the range of $\omega$ for the student training is not necessary.
A wider range of $\omega$ can be used for student training when prior knowledge of the teacher’s dynamics of $\omega$ is absent, and high-quality generation can be achieved by adjusting $\omega$ and $\nu$ during inference.

The students show the better FAD than the best FAD of teachers through $\nu$-interpolation during SoundCTM’s sampling.
This result could be interpreted as $\nu$ allows for generating samples that are more favorable to the FAD in the conditional domain, as the pairs between $\mathbf{c}_{\text{text}}$ and the samples $\mathbf{x}_{0}$ given the condition are many-to-many.

Figure: ((a)) $1$-step generation
Refer to caption: x3.png

#### Performance Comparison with Other T2S Models

We compare SoundCTM’s performance with other T2S models under both $1$-step and multi-step generation. Here, we report the results of SoundCTM trained with more iterations ($30$ K iterations) compared with the previously mentioned results ($18$ K iterations).
We employ AudioLDM-L-FT (Liu et al., ), AudioLDM2-L (Liu et al., ), TANGO (Ghosal et al., ), and ConsistencyTTA as our baseline models. Table 6 presents the quantitative results. All the DM-based models use DDIM sampling (Song et al., ) except for our teacher model.
Under the $1$-step generation setting, SoundCTM shows the best performance in all evaluation metrics.
This achievement is the obtained from using $d_{\text{teacher}}$ as discussed earlier.
Note that ConsistencyTTA-CLAP-FT conducts extra fine-tuning to maximize the CLAP score using CLAP network after its CD training.

**Table 4: Inference speed and RTF. RTF < 1.0 indicates real-time generation.**
| Method | # of steps | Batch size | Speed [sec.] | RTF $\downarrow$ |
| --- | --- | --- | --- | --- |
| AudioLDM2-Large | 200 | 1 | 80 | 8.0 |
|  | 200 | 8 | 16.5 | 1.7 |
| TANGO | 200 | 1 | 24 | 2.4 |
| SoundCTM on GPU | 16 | 1 | 2.43 | 0.24 |
| SoundCTM on CPU | 2 | 1 | 6.93 | 0.69 |

Under the multi-step case, there are clear trade-offs between the performance improvements and the number of sampling steps.
We highlight these results are attributed to SoundCTM’s deterministic sampling ($\gamma=0$) and anytime-to-anytime jump training framework, which cannot be achieved with the CD framework and its lack of deterministic sampling capability (See ConsistencyTTAs’ $2$-step results in Table 6).
In addition, SoundCTM demonstrates the state-of-the-art FAD and IS against all diffusion-based baselines with only $16$ steps. The reason SoundCTM outperforms the teacher model at $8$ and $16$ steps is due to our $\nu$-interpolation as discussed earlier. Note that all diffusion-based baselines use both text-conditional and unconditional predictions, the same setting as our $8$ and $16$-step generation.

**Table 6: Performance comparisons on AudioCaps test set. Bold and underlined scores indicate the best and second-best results. $\dagger$ denotes the results tested by us using the provided checkpoints by the authors, as not all metrics are provided in each paper. $\omega$ and $\nu$ denote the CFG scale and parameter for the $\nu$-interpolation. $\nu=1$ denotes only the text-conditional jump is used.**
| Model | # of sampling | $\omega$ | $\nu$ | $\rm{FAD}_{vgg}\downarrow$ | $\rm{KL}_{passt}\downarrow$ | $\rm{IS}_{passt}\uparrow$ | CLAP $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| steps |  |  |  |  |  |  |  |
| Diffusion Models |  |  |  |  |  |  |  |
| AudioLDM-Large-FT (Liu et al., ) | 200 | 3.0 | - | 1.96 | 1.59 | - | - |
| AudioLDM 2-AC-Large (Liu et al., ) | 200 | 3.5 | - | 1.42 | 0.98 | - | - |
| AudioLDM 2-Full-Large (Liu et al., ) | 200 | 3.5 | - | 1.86 | 1.64 | - | - |
| TANGO^† (Ghosal et al., ) | 200 | 3.0 | - | 1.64 | 1.31 | 6.35 | 0.44 |
| TANGO w/. Heun Solver (Our teacher model) | 40 | 3.5 | - | 1.71 | 1.28 | 8.11 | 0.46 |
| Distillation Models |  |  |  |  |  |  |  |
| ConsistencyTTA (Bai et al., )(reported) | 1 | 5.0 | - | 2.58 | - | - | - |
| ConsistencyTTA^† (tested by us) | 1 | 5.0 | - | 2.67 | 1.33 | 6.85 | 0.41 |
|  | 2 | 5.0 | - | 3.18 | 1.34 | 7.12 | 0.38 |
| ConsistencyTTA-CLAP-FT (reported) | 1 | 3.0 | - | 2.18 | - | - | - |
| ConsistencyTTA-CLAP-FT^† (tested by us) | 1 | 3.0 | - | 2.22 | 1.35 | 6.95 | 0.41 |
|  | 2 | 3.0 | - | 2.38 | 1.32 | 7.15 | 0.40 |
| SoundCTM(Ours) | 1 | 3.5 | 1.0 | 2.08 | 1.26 | 7.13 | 0.43 |
|  | 2 | 3.5 | 1.0 | 1.90 | 1.24 | 7.26 | 0.45 |
|  | 4 | 3.5 | 1.0 | 1.72 | 1.22 | 7.37 | 0.45 |
|  | 8 | 3.0 | 1.5 | 1.45 | 1.20 | 7.98 | 0.46 |
|  | 16 | 3.0 | 2.0 | 1.38 | 1.19 | 8.24 | 0.46 |

#### Generation Time

Since one of our goals is to achieve real-time high-quality generation, we verified that our model can generate samples in real-time by measuring the inference time and real-time factors (RTFs) on a single NVIDIA RTX A6000.
For reference, we also measure those of the original TANGO and AudioLDM2-Large with a batch size of one(^7^77Since AudioLDM2-Large shows unreasonable RTF at a batch size of one, we also report the RTF at a batch size of eight, where the model does not suffer from this issue. Additionally, we use the ”AudioLDM2-Full-Large” checkpoint instead of the AudioLDM2-AC-Large checkpoint, as the latter is not open-sourced. Note that the difference between ”-AC-” and ”-Full-” is in the training data.).
The all three models are the top three in terms of the FAD in Table 6, based on the UNet architecture with similar parameter sizes, and generate the same duration of the samples ($10$-second audio). As shown in Table 5, SoundCTM can achieve real-time generation on a GPU. We also highlight that SoundCTM can achieve real-time generation on an Intel Xeon CPU with $2$-step generation.

### 6.2 Training-Free Sound Intensity Control

To validate the proposed framework for training-free controllable T2S generation with SoundCTM, we conduct sound intensity control (Novack et al., ; Wu et al., ).
This task adjusts the dynamics of the generated sound to match a given target volume line or curve. We follow the experimental protocol from DITTO (Novack et al., ) to control the decibel (dB) volume line or curve of the generated samples.

We define $f(\mathbf{x}_{0}):=w*20\log{10}(\text{RMS}(\mathbf{x}_{0}))$ in Eq. (6), where $w$ represents the smoothing filter coefficients of a Savitzky-Golay filter (Savitzky and Golay, ) with a $1$-second context window over the frame-wise value, and the RMS is the root mean squared energy of the generated sound. The target condition $\mathbf{y}_{\text{condition}}$ is a dB-scale target line or curve. We perform $70$ iterations for $\mathbf{z}_{T}$-optimization following DITTO’s settings.
Note that for the loss-based guidance framework, the $\mathbf{z}_{T}$-optimization is not conducted, which is much efficient.
We set $\gamma=0$ for sampling and evaluate the model performance with
student EMA rate $\mu=0.999$.
We use $200$ audio-text pairs from the AudioCaps testset for each of the $6$ different types of $\mathbf{y}_{\text{condition}}$ as shown in Figure 4 (a) and Figure 6 (a). We evaluate our framework using the mean squared error (MSE) between the target and obtained $\mathbf{y}_{\text{condition}}$ to measure the accuracy of dynamics control, the FAD between generated samples and the entire AudioCaps testset, and the CLAP score. Experimental details are provided in Appendix B.2.

In Table 5, we compare results with various settings for SoundCTM’s controllable generation framework, as DITTO is not open-sourced.
Firstly, overall, both $\mathbf{z}_{T}$-optimization and loss-based guidance frameworks effectively control intensity as indicated by the MSE results and the intensity curve shown in Figures 4 to 7.
However, the FAD and CLAP scores are worse than those of the default T2S generation case.
This is likely due to a degradation in auditory quality and a shift in the volume-conditioned audio distribution away from that of the AudioCaps testset. Additional results are provided in Appendix B.2.

Except for the MSE of $1$-step with $\mathbf{z}_{T}$-optimization case, the loss-based guidance method outperforms $\mathbf{z}_{T}$-optimization method.
Under $16$-step cases, the loss-based guidance method shows better results than those of the optimization-based method across all the metrics.
Interestingly, this finding contrasts with DITTO’s report that the optimization-based method outperforms the guidance-based one in terms of the MSE (Novack et al., , Table 3).
The difference can be attributed to the performance improvement of the loss-based guidance method by using $G_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t,0)$ instead of $\hat{\mathbf{z}}_{0}(\mathbf{z}_{t})$, which provides a more accurate estimate of $\mathbf{z}_{0|t}$, resulting in more effective guidance during sampling.
Considering both the qualitative results in Figures 4 to 7 and the quantitative results in Table 5, the loss-based guidance is the effective strategy for training-free controllable generation with SoundCTM.

## 7 Conclusion

We propose SoundCTM that achieves not only high-quality $1$-step generation but also significantly improves sample quality by increasing the number of sampling steps with a single model. Our new training framework contributes to its performance. Our training-free controllable generation framework makes SoundCTM more controllable than just a fast T2S model.
Furthermore, our frameworks, which does not rely on domain-specific components, has the potential to extend the applicability of CTM to a wider range of domains while maintaining both its fundamental methodology and impressive performance.

## Appendix A Related Work

#### Masked Audio Token Modeling

AudioGen [Kreuk et al., ] and MAGNeT [Ziv et al., ] utilize masked generative sequence modeling of discrete audio tokens for T2S generation. These models exhibit a trade-off between sample quality and inference speed due to their token prediction methods. AudioGen provides better sample quality but requires much more time for sample generation due to its autoregressive prediction. In contrast, MAGNeT offers faster inference through its non-autoregressive prediction.

#### Diffusion-based Models

Recent literature [Huang et al., , , Liu et al., , , Ghosal et al., , Evans et al., ] and competitions [Choi et al., ] report that LDM-based sound generation models outperform other approaches such as GANs. For fast generation, ConsistencyTTA [Bai et al., ] and Stable Audio [Evans et al., ] employ efficient strategies. Stable Audio, for instance, compresses a $95$-second waveform signal to a latent representation, allowing its LDM to generate the representation through a multi-step reverse-time process, unlike other models that compress only $10$-second audio signals.

#### Training-free Controllable Generation

Levy et al. [] demonstrate training-free controllable music generation using loss-based guidance [Yu et al., ]. Their approach defines a task-specific loss, such as for music continuation and infilling, and incorporates the gradient of this loss into the reverse-time diffusion process. Novack et al. [] also present controllable generation by optimizing initial noisy latents based on loss computed in the target condition space. During optimization, diffusion models perform a reverse-time process to obtain a clean signal, which is projected onto the condition space by a predefined differentiable feature extractor. Post-optimization, the model generates a music signal from the optimized initial latent.

## Appendix B Experimental Details

### B.1 Details of T2S Generation

#### Training Details

For teacher’s training, we use $\sigma_{\text{data}}=0.25$ and time sampling $t\sim\mathcal{N}(-1.2,1.2^{2})$ by following EDM’s training manner.
We mostly follow the original CTM’s training setup for student training.
We utilize the EDM’s skip connection $c_{\text{skip}}(t)=\frac{\sigma_{\text{data}}^{2}}{t^{2}+\sigma_{\text{data}}^
{2}}$ and output scale $c_{\text{out}}(t)=\frac{t\sigma_{\text{data}}}{\sqrt{t^{2}+\sigma_{\text{data}
}^{2}}}$ for $g_{\bm{\theta}}$ modeling as

$$ $\displaystyle g_{\bm{\theta}}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t ,s)=c_{\text{skip}}(t)\mathbf{z}_{t}+c_{\text{out}}(t)\text{NN}_{\bm{\theta}}( \mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t,s),$ $$

where $\text{NN}_{\bm{\theta}}$ refers to the actual neural network output.
We initialize the student’s $\text{NN}_{\bm{\theta}}$ with $\bm{\phi}$ except for student model’s $s$-embedding and $\omega$-embedding structure.

We use 8$\times$NVIDIA H100 (80G) GPUs and a global batch size of $64$ for the training.
We choose $t$ and $s$ from the $N$-discretized timesteps to calculate $\mathcal{L}_{\text{CTM}}^{\text{Sound}}$. For $\mathcal{L}_{\text{DSM}}^{\text{Sound}}$ calculation, we opt to use $50\%$ of time sampling $t\sim\mathcal{N}(-1.2,1.2^{2})$. For the other half time, we first draw sample from $\xi\sim[0,0.7]$ and transform it using $(\sigma_{\text{max}}^{1/\rho}+\xi(\sigma_{\text{min}}^{1/\rho}-\sigma_{\text{
max}}^{1/\rho}))^{\rho}$.
We apply Exponential Moving Average (EMA) to update $\texttt{sg}(\bm{\theta})$ by

$$ $\displaystyle\texttt{sg}(\bm{\theta})\leftarrow\texttt{stopgrad}(\mu\texttt{sg }(\bm{\theta})+(1-\mu)\bm{\theta}).$ $$

Throughout the experiments, for student training, we use $N=40$, $\mu=0.999$, $\sigma_{\text{min}}=0.002$, $\sigma_{\text{max}}=80$, $\rho=7$, RAdam optimizer [Liu et al., ] with a learning rate of $8.0\times 10^{-5}$, and $\sigma_{\text{data}}=0.25$.
We set the maximum number of ODE steps as $39$ during training and we use Heun solver for $\texttt{Solver}(\mathbf{z}_{t},\mathbf{c}_{\text{text}},\omega,t,u;\bm{\phi})$ .
To obtain the results shown in Table 6 and Figure 1, SoundCTM was trained for $30$ K iterations. For other results, models were trained for $18$ K iterations. We also utilized TANGO’s data augmentation [Ghosal et al., , Sec. 2.3] during student training.

The network architecture and dataset for the teacher’s training remained unchanged from the original ones. Our teacher model (TANGO) and SoundCTM share the overall model architecture, comprising the VAE-GAN [Liu et al., ], the HiFiGAN vocoder [Kong et al., ] as $\mathcal{D}$ and the Stable Diffusion UNet architecture (SD-1.5), which consists of $9$ 2D-convolutional ResNet [He et al., ] blocks as $D_{\phi}$.
The UNet has a total of $866$ M parameters, and the frozen FLAN-T5-Large text encoder [Chung et al., ] is used. The UNet employs $8$ latent channels and a cross-attention dimension of $1024$.
For $\mathcal{D}$ in SoundCTM, we used the AudioLDM1 checkpoint by following TANGO.

#### Evaluation Details

For large-NFE sampling, we follow the EDM’s and the CTM’s time discretization. Namely, if we draw $n$-NFE samples, we equi-divide $[0,1]$ with $n$ points and transform it (say $\xi$) to the time scale by $(\sigma_{\text{max}}^{1/\rho}+(\sigma_{\text{min}}^{1/\rho}-\sigma_{\text{max}
}^{1/\rho})\xi)^{\rho}$. However, we emphasize the time discretization for both training and sampling is a modeler’s choice.

Figure: Algorithm 2 SoundCTM’s Training

Figure: Algorithm 3 SoundCTM’s Inference

### B.2 Details of Sound Intensity Control

During the SoundCTM’s optimization-based framework, we use Adam [Kingma and Ba, ] with a learning rate of $1.0$.
We also tested learning rates of $1.0\times 10^{-1}$, $1.0\times 10^{-2}$, and $5.0\times 10^{-3}$ by following DITTO.
However, we cannot obtain better results than the case using $1.0$.
For the time-dependent learning rate $\rho_{t}$ in SoundCTM’s loss-based guidance framework, we use the overall gradient norm by following DITTO.

We set $\nu=1$ for $1$-step generation with $\mathbf{z}_{T}$-optimization, $\nu=2$ for both $16$-step generation with $\mathbf{z}_{T}$-optimization and $16$-step loss-based guidance, and $\omega=3.5$ for all the settings.
The same values of $\nu=2$ and $\omega=3.5$ are used for the default $16$-step T2S generation.
For time discretization in multi-step generation, we use the same scheme as in T2S generation evaluation.

In Table 7, we present additional results of sound intensity control. In addition to Table 5, we include results using stochastic sampling ($\gamma=0.2$).
Furthermore, we report results incorporating loss-based guidance after $\mathbf{z}_{T}$-optimization.
Overall, stochastic sampling did not yield significant performance improvement. Integrating the loss-based guidance framework with $\mathbf{z}_{T}$-optimization showed the better MSE, consistent with DITTO’s findings, but resulted in the much worse FAD and CLAP scores compared with only using the loss-based guidance.

Figure: Algorithm 4 SoundCTM’s optimization-based training-free controllable generation framework

**Table 7: Quantitative results of sound intensity control on SoundCTM**
| # of steps | $\mathbf{z}_{T}$ | Loss | $\gamma$ | MSE$\downarrow$ | $\rm{FAD}_{vgg}\downarrow$ | CLAP $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| optimization | guidance |  |  |  |  |  |
| Default $16$-step | ✗ | ✗ | 0 | 231.9 | 2.08 | 0.47 |
| T2S Generation |  |  |  |  |  |  |
| 1 | ✓ | ✗ | 0 | 6.57 | 4.94 | 0.34 |
| 16 | ✓ | ✗ | 0 | 60.2 | 3.65 | 0.38 |
| 16 | ✓ | ✗ | 0.2 | 50.4 | 3.71 | 0.41 |
| 16 | ✗ | ✓ | 0 | 18.5 | 3.04 | 0.41 |
| 16 | ✗ | ✓ | 0.2 | 14.2 | 3.58 | 0.42 |
| 16 | ✓ | ✓ | 0 | 10.7 | 4.34 | 0.37 |

Figure: ((a)) Target intensities
Refer to caption: x7.png

Figure: ((a)) Target intensities
Refer to caption: x11.png

Figure: ((a)) Target intensities
Refer to caption: x13.png

Figure: ((a)) Target Intensities
Refer to caption: x17.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the hyperparameter 𝑣 mentioned in Section 3, which is used for sampling, trainable? If not, how did the authors determine its value? The same question applies to the scale of CFG 𝑤
2. For the results in Table 2, what is the best performance SoundCTM can achieve? With additional sampling steps, can SoundCTM surpass the quality of AudioDLM2-large?
3. What are the differences between SoundCTM and AudioLCM, aside from the teacher CTM loss proposed for training? Both systems are diffusion-based generation models employing consistency strategies.
4. The authors also state that SoundCTM and AudioLCM show clear trade-offs in multi-step generation. What is the inference efficiency of these two systems? The authors mentioned early on that the proposed system is intended to overcome the barrier of slow inference speeds encountered during trial-and-error.
soundness: 3
strengths: 1. Results indicate that the proposed large-scale SoundCTM-Dit-1B achieves state-of-the-art performance in text-to-audio generation in the full-band setting (mono at 44.1 kHz).
2. Instead of using the L2-norm loss between target and output values within the latent domain, this paper introduces a network trained with CTM loss, leveraging a teacher network for feature extraction and measurement. Results show that the proposed teacher loss achieves superior performance.
3. This paper integrates Classifier-Free Guidance (CFG) within the CTM framework by distilling the CFG-related PF ODE trajectory during training and using it as the CFG condition. Additionally, a new sampling strategy has been developed to enable CFG in 1-step generation.
summary: This paper presents a diffusion-based text-to-audio generation system within the Consistency Trajectory Model (CTM) framework, named SoundCTM. SoundCTM is developed using distillation loss with a teacher model, and incorporates a novel sampling algorithm that enables Classifier-Free Guidance (CFG) integration with the student model. Experimental results demonstrate that SoundCTM achieves state-of-the-art performance in 1-step and few-step generation for 44.1 kHz full-band samples, offering a practical tradeoff between generation quality and inference speed for real-world applications.
weaknesses: 1. The comparison results show that the proposed system achieves state-of-the-art performance in single-step sampling, while SoundCTM does not surpass baseline models (e.g., AudioLCM, also a diffusion model with CTM) when increasing the sampling steps (from 2 steps onward). Additionally, there is a lack of comparison in inference speed between different distillation models.
2. Further explanation is required for why baseline models outperform SoundCTM in few-step sampling. Moreover, the CLAP score is the only metric related to semantic alignment (text), yet it does not seem well-aligned with other metrics, such as FAD and IS. It would therefore be beneficial to conduct human evaluations for each experiment to better illustrate performance.
3. The authors claim that SoundCTM-Dit-1B achieves the best results in full-band generation tasks. However, SoundCTM used for comparisons among 16 kHz models employs a UNet-based backbone, resulting in unaligned experiments. The paper lacks sufficient explanations, ablation studies, or comparisons to demonstrate the effectiveness of each submodule. The authors could address this by adding SoundCTM-DIT medium or small models for 16 kHz comparisons and SoundCTM-UNet-1B for full-band comparisons.

Overall, this is an interesting paper exploring the potential of consistency models in sound generation. The model is well-structured, but some modules require further explanation, and additional experiments are needed to illustrate the contributions of each section. I am open to adjusting the score if the authors can address these requirements.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to Weaknesses.

1. What your training dataset?
soundness: 2
strengths: 1. The paper introduces Sound Consistency Trajectory Models (SoundCTM), an approach that unifies score-based and consistency models for Text-to-Sound (T2S) generation. This combination is interesting and reflects a novel direction within the T2S domain, especially in adapting techniques from image generation. 

2. The paper evaluates SoundCTM at both 16kHz and 44.1kHz, showing an awareness of high-fidelity audio generation.
summary: The paper proposes Sound Consistency Trajectory Models (SoundCTM) for improving inference speed and sample quality in Text-to-Sound (T2S) tasks by combining score-based and consistency models. This work mainly adapting existing techniques from image generation for T2S.  From the reported results, SoundCTM can generate sound in few inference step.
weaknesses: 1. The paper proposes Sound Consistency Trajectory Models (SoundCTM) to improve inference speed. While the concept of combining score-based and consistency models for T2S generation is interesting, the paper does not convincingly establish the novelty of its contributions. Many of the methods, including ν-sampling and trajectory-based distillation, are either borrowed from existing frameworks or lack adequate differentiation from prior work. A clearer and more distinct innovation beyond adapting methods from image-based models is needed to justify the novelty of this work.

2. The paper emphasizes deterministic sampling as a unique advantage of SoundCTM, particularly for preserving semantic content across sampling steps. However, this advantage remains largely theoretical within the paper, with insufficient empirical evidence demonstrating the effectiveness of deterministic sampling in real production scenarios. The provided spectrograms and subjective evaluations do not strongly support the claimed benefits, and thus, the deterministic aspect feels overstated relative to its demonstrated impact.

3. The paper highlights full-band sound generation. However, all previous work can support full-band sound generation by simply replacing 16kHz audio data with 44.1kHz data.

4. In terms of performance, SoundCTM shows some advantages over AudioLCM and ConsistencyTTA. However, as SoundCTM scales to a 1B parameter model, it is difficult to determine whether the improvements are due to model scaling alone.

5. From an architectural perspective, the proposed DiT framework closely follows the architecture of Stable Audio 2.0.

6. In the related work section, the paper claims that "when developing sound generative models suitable for production use, LDM-based models present a better choice as the base model." It is difficult to agree with this claim. Previous works, such as AudioGen, have also demonstrated good performance in T2S. For example, DiffSound shows that using a discrete diffusion model yields better performance than an AR-based baseline. After that, AudioGen demonstrates that AR-based models can also achieve good performance with different configurations. Given the rapid development of T2S, it is premature to make such a claim. In the field of image generation, some studies have shown that AR-based models can outperform LDM-based models. Additionally, discrete diffusion models remain an important research direction.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: - In v-sampling, it is not clear whether there is one or two student models involved. From the abstract it seems that there are two but not from section 3.3 and algorithm 2.
- In what sense is v-sampling different than CFG applied to the student model?
- What is the difference between the last row of Table 1 and the second to last row of Table 2? They do not indicate the same values although they seem to relate to the same experiment.
- How do you explain that SoundCTM-DiT-1B outperforms the teacher model?
soundness: 3
strengths: - Showcasing a potential downstream application of the proposed method is a nice addition to the work.
- Speed and inference consistency between 1-step/multi-step inference is a fundamental problem in the sound design space and this paper properly tackles this problem.
summary: This paper proposes a new Sound Consistency Trajectory Model (SoundCTM) to accelerate the iterative trial-and-error process of sound design using audio generation models.

Compared with previous 1-step distillation methods, SoundCTM provides deterministic sampling capabilities, which enables acoustic consistency between 1-step and multi-step generations from the same initial noise latent.

Creators can use SoundCTM in the fast 1-step setting for exploratory work while benefitting from the higher quality of multi-step inference once a relevant generated sample has been chosen.

The authors claim the following contributions.
- Adaptation of the computer vision domain CTM training framework to the audio domain (called SoundCTM).
- A novel feature distance for distillation loss computation.
- v-sampling: a new cfg-like approach that consists in training both conditional and unconditional student models.
- SoundCTM-DiT-1B, a large scale distillation model that achieves sota 1-step and multi-step full-band text-to-sound generation on the AudioCaps evaluation dataset.

The results indicate better consistency between 1-step and multi-step generation compared with previous sota.
weaknesses: - It is not clear how v-sampling differs from CFG hence it is difficult to appreciate the relevance of the contribution.
- It is unclear why the proposed distilled model outperforms its teacher.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do the authors plan to explore alternative GAN architectures or training strategies to effectively integrate GAN loss and potentially enhance the quality of 1-step generation? Understanding the obstacles faced and possible solutions could be beneficial for future work.

2. How does SoundCTM perform when the teacher and student models are trained on different datasets? Additionally, how well does the model adapt to other audio domains such as music generation or speech synthesis? Insights into its generalizability would be valuable.

3. Can the authors provide more detailed insights into the ν-sampling algorithm? Specifically, how does ν-sampling impact the model's performance compared to standard sampling methods, and what are the theoretical underpinnings that justify its effectiveness?

4. Have the authors considered using external pretrained networks as feature extractors for the distillation loss? If so, how might this affect the model's performance and flexibility, especially in scenarios where the teacher network is not available or differs significantly from the student?

5. How does SoundCTM compare with flow matching methods in terms of performance, computational efficiency, and ease of implementation? Are there specific advantages in choosing SoundCTM over flow matching approaches for text-to-sound generation tasks?

6. Beyond text-to-sound generation, can SoundCTM be applied to other audio-related tasks such as speech synthesis, audio style transfer, or sound enhancement? What adaptations would be necessary, and how might it perform compared to specialized models designed for those tasks?
soundness: 3
strengths: 1. SoundCTM enables creators to quickly generate initial sound samples using high-quality 1-step generation and then refine these samples through multi-step deterministic sampling without altering the semantic content. This flexibility is highly valuable in practical sound design workflows.

2. By introducing a new feature distance metric that uses the teacher's network as a feature extractor for the distillation loss, the model enhances knowledge transfer from the teacher to the student. This approach leads to better performance compared to traditional L2 loss or external models, as it more effectively captures nuanced features relevant to sound generation.

3. The introduction of ν-sampling is a significant contribution. By combining conditional and unconditional student models, ν-sampling improves the quality of multi-step generation and provides another avenue for quality enhancement.

4. The ability to maintain semantic consistency across different sampling steps is crucial for practical applications. SoundCTM effectively addresses this by enabling deterministic sampling, which is essential for sound designers who require consistent outputs during refinement.
summary: This paper introduces SoundCTM, a novel approach to T2S generation that aims to enhance the efficiency and quality of sound creation in multimedia applications such as video games and films. By addressing the limitations of existing diffusion-based T2S models—which often suffer from slow inference speeds and subpar sample quality in 1-step generation—SoundCTM offers a flexible framework that allows for rapid trial-and-error through high-quality 1-step generation and subsequent refinement via multi-step deterministic sampling, all while preserving semantic content. The authors adapt the CTM training framework from computer vision, introducing a new feature distance metric that leverages the teacher network for distillation loss. They also propose a novel ν-sampling algorithm, enhancing the quality of multi-step generation. The model successfully scales up to one billion parameters, achieving promising results in both 1-step and multi-step full-band sound generation.
weaknesses: 1. While the authors mention that preliminary experiments with GAN loss did not yield improvements, they do not deeply explore alternative strategies or provide a thorough analysis of why GAN loss integration was unsuccessful. A more detailed investigation could potentially uncover methods to enhance the model's performance, especially in 1-step generation.

2. Relying on the teacher's network as a feature extractor may limit the model's flexibility, particularly if the teacher and student models are trained on different datasets or domains. This dependency could pose challenges when attempting to generalize the model to new or diverse datasets.

3. The paper briefly demonstrates the application of SoundCTM to downstream tasks like controllable generation but does not explore this in depth. A more comprehensive study on various applications could showcase the versatility of the model.

4. The paper does not provide a comparison between SoundCTM and flow matching approaches. Such a comparison could offer valuable insights into the advantages or limitations of the proposed method relative to other leading techniques in the field.

Meta-review: This paper proposes Sound Consistency Trajectory Models (SoundCTM), a text-to-sound generation framework that unifies score-based diffusion and consistency modeling. The core objective is to enable both fast trial-and-error via high-quality one-step sampling and refined multi-step sampling at full-band audio (44.1 kHz). 

All four reviewers provided positive ratings, with final scores of 8, 8, 6, and 6. SoundCTM offers a valuable step forward in text-to-sound research by combining quick inference for iterative design with deterministic multi-step sampling for higher quality. The authors’ rebuttal addressed concerns about method, experiments, and contributions. While some novelty questions remain around adopting consistency trajectory models from vision, the overall consensus is that domain-specific engineering, large-scale experiments, and open-sourced code represent a substantial contribution to the T2S field.

Hence, I recommend acceptance. The authors should incorporate the additional results and clarifications in rebuttal to the revised version.

Decision: Accept (Poster)