SpeechOp: Inference-Time Task Composition for Generative Speech Processing
Authors: Justin Lovelace
Rithesh Kumar
Jiaqi Su
Ke Chen
Kilian Q Weinberger
Zeyu Jin
Venue: ICLR 2026 Poster (2026)
Keywords: speech generation
TTS
enhancement
diffusion
latent diffusion
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: SpeechOp transforms pre-trained TTS models into universal speech processors capable of novel inference-time task combinations tasks such as transcript-guided enhancement and separation.

Abstract: While generative Text-to-Speech (TTS) systems leverage vast "in-the-wild" data to achieve remarkable success, speech-to-speech processing tasks like enhancement face data limitations, which lead data-hungry generative approaches to distort speech content and speaker identity. To bridge this gap, we present SpeechOp, a multi-task latent diffusion model that transforms pre-trained TTS models into a universal speech processor capable of performing a wide range of speech tasks and composing them in novel ways at inference time. By adapting a pre-trained TTS model, SpeechOp inherits a rich understanding of natural speech, accelerating training and improving S2S task quality, while simultaneously enhancing core TTS performance. Finally, we introduce Implicit Task Composition (ITC), a novel pipeline where ASR-derived transcripts (e.g., from Whisper) guide SpeechOp's enhancement via our principled inference-time task composition. ITC achieves state-of-the-art content preservation by robustly combining web-scale speech understanding with SpeechOp's generative capabilities.

Published as a conference paper at ICLR 2026
SPEECHOP: INFERENCE-TIME TASK COMPOSITION FOR GENERATIVE SPEECH PROCESSING
Justin Lovelace"
Rithesh Kumar Jiaqi Su Ke Chen
Cornell University 313353@corme11.co edu
Adobe Research
(ritheshk, jsu, kechen) ) @adobe. com
Kilian Q Weinberger Cornell University
Zeyu Jin
Adobe Research
kgw48cornell.
zejin @amobe.com
ABSTRACT
While generative Text-to- text-to-Speech (TTS) systems leverage vast "in-the-wild" data to achieve remarkable success, face data limitations, which lead data-hungry generative approaches to distort speech content and speaker identity. To bridge this gap, we present SpeechOp, a multi-task latent diffusion model that transforms pre-trained TTS models into a universal speech processor capable of performing wide range of speech tasks and composing them in novel ways at inference time. By adapting a pre-trained TTS model, SpeechOp inherits rich understanding of natural speech, accelerating training and improving S2S task quality, while simultaneously enhancing core TTS performance. Finally, we introduce Implicit Task Composition (ITC), a novel pipeline where ASR-derived transcripts (e.g.. from Whisper) guide SpeechOp's s enhancement via our principled inference- time task composition. ITC achieves state-of- state-of-the-arr content preservation by robustly combining web-scale speech un- derstanding with SpeechOp's ' generative capabilities. Audio samples are available at hhttts::/
processing tasks like enhancement
INTRODUCTION
Generative Text-to-S Speech (TTS) systems now produce increasingly natural and expressive speech
(Le et al. 2024 Ju et al., 2024). largely due to their ability to leverage vast "in-the-wild" data
(e.g., from from audiobooks, podcasts (Chen al. 22021a; Pratap al,, 2020)). This scalability enables TTS models to learn robust speech representations across diverse acoustic conditions and speaker
characteristics (Lee al. 2024 Peng et al. 2024).
In contrast, speech-to-speede (S2S) processing tasks like enhancement, speaker separation, and isolation face stricter data requirements, often needing paired degraded/clean speech, which is expensive acquire at scale (Zen et al.) 2019). Consequently, S2S models typically trained on smaller, specialized datasets, often with simulated degradations (Su et al. 2021a). This data scarcity cause generative S2S approaches to distort original speaker identity and content- critical issue where faithful preservation is paramount, c.g., in speech enhancement (Yang al. 2024 Koizumi al. 2023b). These models often lack the rich speech understanding derived from vast, diverse datasets available to TTS.
To bridge this data gap, we present SpeechOp: a multi-task latent diffusion model that transforms pre-trained TTS models into a universal speech processor. SpeechOp performs a wide range of S2S tasks and allows their novel inference-time composition, leading three key advancements (Figure I]:
A Flexible Multi-Tast Model That Enhances TTS Capabilities: SpeechOp, adapted from a pre-trained TTS model and fine-tuned on diverse S2S tasks (including TTS, enhancement, separation),
Work done during intersship Adobe.

---

Published as a conference paper at ICLR 2026
Multi-Task Training Paradigm
TeeeHo-Speech
Vaice Cloning
Enhenovment
I
Separation
Speaker Sep.
I Acoustic Matching
Mielco Nocier
"ualko Hor.e" Temee
SpeechOp
Inference-Timee Time Task Composition
Text-t--poechh Erhancement
Vaice Cloning Enhancement
TexH-b-ppashh Separaton transoript
transoigtt
voice sample
spooch
speach
SpeachOp
SpeechOp
SpeechOp
Speech
Foreground Speech Background
Text Guided Enhancement
Personalized Emnancement
[
Text Guided Separation
Implicit Task Composition
Texi-to-ppeech Enhancament
Spoeencc
Whappoo
(nolsy) transrrttt
Hobly Steedh
nobyy
Enhoncement)
WWhiper-Guided Enhancement J
Figure 1: Overview of SpeechOp's multi-task training (top), inference-time task composition ca- pabilities (middle), and implicit task composition pipeline (bottom). The model is trained on six core speech tasks including rext-to-speech, enhancement, and separation. At inference time, novel tasks can be composed by combining learned capabilities for example, using transcripts to guide enhancement or personalizing enhancement with voice samples. In the implicit task composition pipeline, use state-of-the-at discriminative model (Whisper) automatically transcribe noisy speech, then use the resulting transcript to guide SpeechOp's s enhancement process.
not only becomes a versatile speech processor but also improves its underlying TTS quality. By learning to handle varied acoustic manipulations, SpeechOp's TTS component generates more natural, higher-quality speech, validated by human listening studies.
Our novel TC-CFG guidance strategy (Section enables composing independently trained speech capabilities at inference time without requiring joint training. For instance, SpeechOp can combine enhancement with TTS content guidance both improve acoustics and re-synthesize obscured content.
2. Inference-Tine Task Composition (TC-CFG):
3. State-of-the-Art Speech Processing through Implicit Task Composition (ITC): SpeechOp achieves state-of of-the-art content preservation via ITC. Traditional transcrit-condditooned S2S models suffer from scarce paired noisy-clean-transee
data and the propagation of ASR errors. ITC overcomes these by robustly integrating ASR-derived transcripts (e.g., from Whisper (Radford et al., 2023 Bain et 2023) using our TC-CFG inference-time composition. This principled approach, with its tunable "guidance strength," allows balancing content restoration (more like TTS) and acoustic fidelity (more like enhancement) based on the situation, achieving superior content fidelity over specialized enhancement methods.
BACKGROUND: DIFFUSION MODELS
We introduce latent diffusion models following recent formulations (Ho et al. 2020 Kingma & Gao
2023 Rombach et al. 2021). Given data drawn from an unknown distribution ((x), our goal is to learn generative model po (x( that approximates this distribution.
2

---

Published as a conference paper at ICLR 2026
Forward process. The forward process defines gradual transition from the latent distribution to a Gaussian distribution through a sequence of increasingly noisy latent variables 2, for timesteps
t e [0, 1]. This Gaussian diffusion process defines the conditional distribution g(Zo...., (x). For every
t E [0, 1). the marginal g(2||x) is given by:
Z = axx + occ, where N((,I)
We the variance-presse preserving formulation where of =1 az. The noise schedule a E [0, 1] is strictly monotonically decreasing function that starts with the original latent (Zo R x) and ends with approximately Gaussian noise (g())) (21 I)).
Generative model. Given the score function V log (z), or the gradient of the log probability density function, we can reverse the forward process exactly. Diffusion models utilize neural network to learn to estimate the score function, Sa sa(z; A) 2 V, log qr (z)). and use the estimated score function to approximately reverse the forward process. If sæ sz X) Vz log (((z), then our generative distribution is close to the true distribution. This enables us to draw samples from a Gaussian distribution 2 p(z)). ). and approximately solve the reverse diffusion process using the estimated score so (z; X).
Training objective. We train the score network using a denoising score matching (DSM) loss Song & Ermon (019) over all data points x ~ 'D and noise levels:
(DsM(x) = (r.xx_[w(MM) (20; A) - Zz, log (2, x)
where w/ (x) weights different noise levels during training. Following best practices (Salimans & Ho! 2022), we adopt the velocity parameterization, V Cgc oxx, for our network output to ensure training stability.
RELATED WORK
Text-to-speedt (TTS) systems (Le et al., 2024 Shen et al. 2024) excel due to vast "in-the-wildd data, unlike data-limited speech-to-speech (S2S) tasks like enhancement (Koizumi et al. 2023b) and separation, which often require scarce paired recordings. While multi-task autoregressive models have been developed (Wang al. 2004), they lack inference-tima compositionality. Diffusson models offer high-quality synthesis (Shen et al. 2024 Le et al. 2024) and are inherently compositional (Liu
al. 2022). which SpeechOp leverages.
SpeechOp adapts pre-trained TTS models for S2S tasks and utilizes a novel inference-time com- position pipeline to significantly improve speech processing quality and flexibility. Recent work like Fugatto (Valle et al.] 2025) also explores multi-task audio generation. However, SpeechOp's principled task composition (TC-CFG, Section 6 provides superior control and performance in S2S tasks like enhancement compared to Fugatto's 's score averaging approach (Section ( enabling effective combination of operations like enhancement and TTS. While foundational models like UniAudio (Yang et al. 2023) pursue broad task coverage from scratch, and SpeechFlow (Liu et al. 2024) investigates new pre-training schemes, SpeechOp focuses on efficiently adapting existing TTS models. Crucially, we introduce Implicit Task Composition (ITC), which uniquely integrates ASR models (e.g., Whisper) via TC-CFG for robust content preservation. Our primary aim is not maximizing task variety, but demonstrating how TTS pre-training and sophisticated composition can address S2S data scareity and improve performance on established operations.
4 TTS PRE-TRAINING IMPROVES SPEECH PROCESSING TASKS
To motivate our multi-task framework, we first examine the benefits of initializing single-task speech enhancement and speaker separation models from pre-trained DiT TTS backbone (Peebles & Xie| 2023 Lee et al. 2024). Figure 2 (Left) shows that TTS initialization dramatically accelerates convergence, achieving comparable validation loss with 4x fewer steps for enhancement and 8 fewer for separation versus random initialization. The significant speedup for separation, a complicated multi-speaker task, demonstrates the strong positive transfer from TTS pre-training.
Beyond faster training, TTS pre- pre-trainig yields downstream performance gains (Figure 2 Right). Speaker separation benefits most, with TTS initialization leading to dramatic improvements in MCD
3

---

Published as a conference paper at ICLR 2026
Speech Enhancement Separation Training
Speaker Separation
0.45
Init
SI-SDRi 1
MCD
SpBS
WER
Val MSE
Validetinn Lods
Rand
-3.99
22,95
.25
17.8
0.358
Sep srabion ITS init & 25e (ex festert
TTS
-1.38
4.46
.966
8.5
0.336
0.25
Speech Enhancement
0.30
fexteri
Init
PESQ
MCD
SpBS
WER
Val MSE
50K
Tairning Steps
100K
125K
175K
200K
Rand
2.07
4.76
900
8.1
0.289
Entancement (Random)
Separation (Randomm
ITS
2.10
4.69
910
8.1
0,276
Figure 2: Impact TTS initialization on speech processing tasks. (Left) Validation loss curves demonstrating accelerated convergence with TTS initialization. Training time is reduced by 4x for enhancement and 8x for separation. (Right) Performance metrics for speaker separation and speech enhancement, comparing random initialization (Rand) with TTS initialization (TTS).
and WER (8.5% 17.8%). We observe that it eliminates artifacts present in randomly initialized models that struggle to learn the content disentanglement objective. Speech enhancement also sees improvements in PESQ, MCD, and SpeechBERTScore from the TTS initialization. These results demonstrate the broad advantages of TTS pre-training- accelerated convergence and enhanced performance across diverse $2S tasks, especially those requiring deep speech understanding. This motivates SpeechOp, our multi-task framework leveraging TTS pre-training for high-quality, versatile speech processing.
neetrrining--acceleeaaee
SPEECHOP
We present the tasks explored in this work in Figure These tasks provide complementary capabilities that composable via our diffusion framework for applications like transcript-guided isolation. For speaker separation, which requires speaker prompt to identify the target speaker, we provide disjoint speech sample to disambiguate the target speaker. For foreground/background separation, parameterize them as two separate tasks.
SpeechOp Architecture. SpeechOp is built on latent diffusion framework (Rombach al., 202T) that operates with compressed au- dio representations. Rather than working with raw waveforms, we first compress the audio using DAC variational autoencoder Kumar jet al. (2023) (details in Appendix), allowing model efficiently process and generate speech in a lower-dimension
Denaised Audio Latent
Frame-ufse Mixing
Cress-Attentin
Diffusion Transformer
Encoder
ByT5 Encoder
latent space. As architec- ture consists of a Diffusion Transformer (DiT) (Peebles & Xie (023) that extended to handle both text-to-speech and speech-to-speed, tasks, The model processes text transcripts for TTS and source audio (like noisy speech) for speech- to-speech tasks, with a learnable Task Embed- ding that conditions model behavior. Training proceeds two stages: TTS pre-training fol- lowed by multi-task training to enable speech- to-speech capabilities.
Diflusion Latont
shown in Figure 3 SpeechOp's
Source Audio Latant
Task Embedding
Transcript
Figure 3: SpeechOp Architecture Overview. A learnable Task Embedding conditions both the Au- dio Encoder and Diffusion Transformer via adap- tive normalization layers (Peebles Xie 2023) to specialize behavior for each task.
Text-to-Speech Pathway. For TTS, SpeechOp (Figure right) processes text transcript. We
extract transcript representations with a frozen, pre-trained ByT5-base encoder Xue et al. (0022): Lovelace et al. (2024)). ByT5's character-level representations capture phonetic information crucial for natural speech. The DiT conditioned on the ByT5 embeddings via artss-attention, dynamically aligning text and audio frames and guiding denoising based text content. For Diffusion
4

---

Published as a conference paper at ICLR 2026
Transformer (DiT) architecture, we incorporate design choices from recent TTS systems (Lee et al. 2024 Lovelace et al., 2024b) (full details in the Appendix).
To enable speaker-prompted generation and speech editing, we train model perform inpainting for 75% of samples [e et al. (2024). After adding noise from the forward diffusion process, we replace a random segment of the latent with the clean, target segment. We additionally sum learnable binary embedding at the input layer to distinguish clean from noisy frames. The network will then learn to extrapolate speaker and speech properties from the ground-truth region to denoise the noisy speech. For half of our inpainting samples, we replace the initial segment (simulating voice prompts). In the other half, we noise only the middle section, replacing the start and end of the utterance with clean speech to simulate speech editing. For sampling the relative duration, we follow Lovelace jt al. 2024b) and use a Beta distribution with mode of .01 and concentration of 5 to emphasize challenging cases with short prompts.
Speech-to-Speech Pathway. To handle speech-to-speet tasks like enhancement and separation (Figure left). SpeechOp introduces dedicated Audio Encoder to process source audio such as noisy speech. This encoder adopts the same DiT architecture as the main model but starts with random initialization. Since speech-to-speech tasks inherently maintain frame-level alignment between source and target audio, we implement a straightforward frame-wise mixing approach rather than use complex alignment mechanism. Specifically, the Audio Encoder's 's output representations directly added to the Diffusion Latent before processing by the Diffusion Transformer, allowing direct incorporation of source audio information during denoising. To handle different speech-to-speech tasks, we use learnable Task Embedding that conditions both the Audio Encoder and Diffusion Transformer. Following the class embedding approach from the original DiT paper (Peebles & Xie 2023), the Diffusion Transformer sums the task embedding with the timestep embedding to condition the network via adaptive normalization (AdaLN), while the Audio Encoder uses the task embedding directly for adaptive normalization. This simple approach provides effective task-specific guidance to both components based on the desired operation (enhancement, separation, etc.).
Some speech-to-speech tasks require additional input prompts. For example, speaker separation needs a reference speech sample to identify the target speaker, while acoustic matching needs an example of the target acoustics. In these cases, we prepend the prompt to both the source audio and noisy latent to maintain frame-wise alignment. For tasks that typically don't use prompts (like enhancement), we unmask the latent's initial segment in 10% of training instances enable transfer learning with speaker prokkped TTS. These prompt durations follow the same Beta distribution used for TTS inpainting.
Multi-Task Fine-Tuning. SpeechOp uses a two-stage training approach. After initial TTS pre- training, conduct multi-task fine-tuning where both the Audio Encoder and pre-trained DiT backbone are jointly optimized. During this stage, we sample TTS and speech-to-speech (S2S) data with equal frequency. Within the S2S samples, we apply selective upsampling - tripling the frequency of enhancement and speaker separation examples since these are the most challenging tasks. This two-stage strategy efficiently adapts the TTS model into our multi-task SpeechOp model.
Diffusion Training, During training, we sample noise levels using a shifted cosine schedule (s=0.5) (Hoogeboom al. 2023). following Lovelace et al. (024a)). We employ the Sigmoid diffusion loss weighting from Hoogeboom et al. (2024) with a bias of -2.5 to concentrate training on perceptually relevant noise levels. To enable classifer--rre guidance during inference, we randomly drop conditioning information (source audio and transcript) 10% of the time during training (Ho & Salimans 2022).
6 INFERENCE-TIME TASK COMPOSITION
The ability to compose speech aperations-suce as simultaneously enhancing noisy speech while restoring its content via text-represents a powerful capability for speech processing. Text-guided generation can help produce a plausible, high-fidelity version of content that is otherwise not recoverable from complex acoustic situations, such as intense noise and reverberation encountered in speech enhancement. Similarly, in speaker separation, the text of spoken content could provide important contextual cues for disentangling speakers. Nonetheless, achieving an effective composition of tasks poses significant technical challenges.
5

---

Published as a conference paper at ICLR 2026
Synthotic Goussian Mixture Satting
Biosed Enhancement
TTS Score Averaging
TTS TC-CFG (Ours)
Target Grolee
(wo) Prusalee!
Target Disty Budion: ParDdly
Blased Target Distrbution:: Pow
Figure 4: ID toy simulation illustrating different task composition methods. (a) The setup, showing bimodal TTS prior (target wp and other w1), an ideal sharp target distribution (prnh), and biased enhancement model (Penh) whose output misaligned. (b) Samples from the unguided biased model. (c) Samples using score averaging (Eq. equation D. (d) Samples using our TC-CFG method (Eq. equation
Prior work, including Fugatto in the audio domain. typically computes a weighted average of score functions to compose operations et al. (2022) Valle et al. (2025). like for enhancement and TTS:
sg "h (zoly, w) (1 a (zyly) + as Dtpioor (zelw)
(I)
Here, (z. yy) is the score from an enhancement model conditioned on noisy audio y, and 8/ (2, represents score function derived from TTS model aiming to generate speech for transcript w. While straightforward, this approach poses fundamental limitation: it combines the generative priors of different tasks. For speech enhancement with TTS guidance, direct averaging allows the TTS model's broad acoustic prior (learned from diverse data for generation) to corrupt the enhancement model's 's focused studio-quality prior (learned for reconstruction), degrading output quality.
To address this challenge, we propose decomposing the desired score function Vz, log p(z w) into sase-specific components. Using Bayes' rule and a conditional independence assumption (transcript w is independent of noisy audio y given latent 2;; detailed derivation in the Appendix), we arrive at:
V 2: loz y, w) = Vz, log (y) + Zz, logp(w Zz).
(2)
This decomposition yields two complementary terms: an enhancement score Vz, log pl p(z+ly) that guides acoustic quality based the input y, and a discriminative guide Vz, log Mogp(w/je 22). This second term leverages a TTS model not for its generative prior, but for its ability to discriminate whether latent z; likely to produce content matching transcript w. This term guides the latent towards speech aligned with the transcript without imposing the TTS model's full acoustic prior.
Implementation via Classifier-Free Guidance. We implement this decomposition using classifier- free guidance (CFG) Ho & Salimans (2022) to approximate the discriminative signal Vz, log p( ((
Vz, log p(w (/
(5 (2 (2]lw) (w) ( (z,))
where sij (ze (zclw) is the score of TTS model conditioned on transcript w. s (z:) is its unconditional score, and y is guidance scale. Substituting this into Eq. equation 2 our final composed score is:
CFG Sg (2r ly. w) x= s
emh (z.lv) (z) (z) s (z.)).
This formulation, which we term Task- Composition Classifier-Fre FFee Guidance (TC-CFG), preserves the strengths of both tasks. The enhancement term maintains acoustic quality and speaker character- istics. The CFG-derived discriminative term provides content alignment by isolating text-specific guidance, avoiding the pitfalls of directly mixing generative TTS priors with the enhancement prior.
Synthetic Simulation. To illustrate the behavior of score averaging and motivate approach, we present results for a ID Gaussian mixture simulation (Figure 4 full details in appendix), where the score functions are analytically tractable. We present this example primarily to provide intuition for the behavior of the two approaches. We empirically validate the benefits of our approach for real speech processing applications in section 8
TC-CFG
Our setup (Figure (4) features a bimodal TTS prior, a sharp ideal enhanced distribution Prah for the target word wa, and imperfect (biased) enhancement model ÿrish whose content wp is misaligned.

---

Published as a conference paper at ICLR 2026
Without guidance, the biased model's samples are incorrect (Figure 4)) However, combining the biased enhancement score function with the TTS score function can potentially correct for content errors. Score averaging (Figure 4b) pulls samples towards the wo TTS mode. However, because this mixes in the broad TTS prior, the result is a "smeared"" distribution that deviates from the enhancement distribution. In contrast,
TC-CFG approach (Figure (4t) incorporates discriminative TTS guidance (via vx, log PrTs (wo xx)) to steer sampling. This shifts the sampled distribution to satisfy the discriminative signal without compromising the enhancement prior.
EXPERIMENTAL SETUP
SpeechOp integrates a 20-layer Diffusion Transformer (DiT, 419M parameters) with an 8-layer audio encoder (71M parameters). We compare against strong baselines for speech enhancement and speaker separation.
Training Data. For TTS, we combine ML. English 44k hours) (Pratap et al.] 2020) for longer utterances (10-20s) and Libri-TTS (585 hours) (Zen et al.] (019) for shorter segments (<10s), improving robustness. All audio is resampled to 48kHz and transcripts lowercased. For S2S tasks, we use LibriTTS-R R (Koizumi et al., 2023a) for clean speech and simulate degradations using established noise/impulse response datasets and pipelines (Yang et al. 2024), creating 5s paired instances. Further dataset details are in the Appendix.
Tasks and Baselines. Text-to-Specch (TTS): We evaluate on LibriSpeech test-clean (Panayotov al.
2015) against contemporary end-to-end TTS systems (Le al.
2024 Chen et al.] 2024b) Lee
et al. 2024). Speech editing is evaluated on the LibriTTS portion of RealEdit (Peng et al. 2024).
Speech Enhancement (SE): Baselines include waveform (StoRm
(Lemercier al.) 2023 and
diffusion-based (SGMSE+ (Richter et al. 2023), Shipher+WavLM,
Koizumi et al. 2023b Yang
al., 2024)) models, and GAN-based --A-- (Su al.
2021a). Speaker Separation (SS):
Chen et al. [024a)), including those trained on WHAMR! (Maciejewski et al.) 2020) and with acoustic-content simulation.
We compare against SepFormer variants (Subakan et al. 2021 1
Evaluation Metrics. We assess SpeechOp on four dimensions: Subjective Quality: Mean Opinion Scores (MOS, 1-5 scale) from listening tests on Prolific (pro) (details in Appenddix A). Signal Similarity: PESQ (perceived quality). MCD (spectral distance, lower is better), and SI-SDRi (separation distortion improvement) Roux et al. (2019). Neural Similarity: WTVNM- (Chen et al. 2b2 for speaker similarity (SIM) and SpeechBERTScore (SpBS) Saeki et al. (2024) for semantic alignment. Content Accuracy: Word Error Rate (WER) via HuBERT-L|Hsu et al. (202T) for TTS and WhisperX (large-v2) Radford et al. (202))
Bain al. (2023) for other tasks.
8 RESULTS AND DISCUSSION
Text-To- To-Speech. To examine the impact of multi-task training on text-to-speech, we evaluate our model's zero-shot TTS performance with 3 second speech prompts. Crucially, we initialize SpeechOp from our TTS Baseline, allowing us to directly assess the impact of multi-task training. Table demonstrates that SpeechOp not only preserves but enhances zero-shot TTS capabilities. After undergoing multi-task training, SpeechOp improves performance across all MOS metrics and objective speaker similarity compared the TTS Baseline, with minimal loss of intelligibility. Exposure to tasks like enhancement and separation likely enhances SpeechOp's ability to generalize and generate natural speech across diverse acoustic environments.
Against recent TTS systems of comparable scale, SpeechOp exhibits strong performance, matching or exceeding CLaM-TTS and XTTS on most metrics. Impressively, it also surpasses the larger VoiceCraft model in intelligibility and subjective quality. While DiTTo-TTS. a larger model trained on more diverse data, achieves higher overall scores, SpeechOp's 's results are highly competitive within its class. Future work will explore scaling SpeechOp to leverage similar large-scale datasets.
Beyond zero-shot TTS, SpeechOp demonstrates ofate-of-the-art capabilities in speech editing. As shown in Table 2] SpeechOp significantly outperforms VoiceCraft across all subjective MOS metrics, despite having fewer parameters. These results validate the robustness of our multi-ted task approach, as SpeechOp maintains exceptional speech editing performance while supporting multiple tasks.
7

---

Published as a conference paper at ICLR 2026
Table 1: Zero-Shot Text-to-Speech Evaluation. MOS metrics evaluate different aspects: MOS- Q (Quality), MOS-N (Naturalness), MOS-V- VS (Voice Similarity), and MOS-SS (Style Similarity). Models in a different parameter regime are displayed in gray.
Model
Params
Training Data
WER 4
SIM
MOS-Q 1
MOS-N
MOS-VST f
MOS-SS
Grund Truth
2.19
a no
4.16 ow
3.79 2:00
DiTTo-TTS [Lee et
2024
740M
566 hrs
2.56
4.16
4.14
4.17
4.02
VoiceCraft Peng et al.
4024
830M
<-99 his
6.32
.61
3.66 nos
3.65 ous
3.43
3,38 nas
CLaM-TTS Kim al
2024
584M
56k hrs
5.11
.49
3.67 + use
3.70 to
3.69 wee
3.54 -.s
XTTS Casaliova
482M
177 hus
4.93
.49
3.76
3.66
3.28
3.27
TTS Baseline (Ours)
419M
455 hrs
3.32
.48
3.65 1 nos
3.56 nnr
3.31 Lun
3.25 n05
SpeechOp (Ours)
419M
45k hrs
3.57
3.86 +: om
3.69
3.67 tue
A from Multi-Task Training
+0.25
+.05
+0.22 am
+0.13+ 887
+0,36+
+0.32
Table 2: Speech Editing Evaluation.
Model
Params
Training Data
WER 4
MOS-Q f
MOS-N
MOS-VS f
MOS-SS
Ground Truth
16.2
4.33 aum
4.40 1
4.66 uco
4.63 ou
VoiceCraft (Peng et al. 2024
830M
699 hrs
16.3
3.62 em
3.99 +.00
4.12 #:04
4.01 004
TTS Bascline (Ours)
419M
4555 hrs
16.4
4.18 cmm
4.23 am
4.45 ues
4.23 100
SpeechOp (Ours)
419M
4-5k hrs
15.9
4.15 c.0
4.19 0.00
4.48 0.6
4.25 000
Speech Enhancement. Our Implicit Task Composition (ITC) pipeline integrates ASR transcripts from Whisper with our TC-CFG method (Section 6 to guide speech content. We find that our ITC pipeline achieves state-of-the-ar content preservation in speech enhancement. As shown in Table 3 ITC yields Word Error Rate (WER) of 2.9%, a 46% relative reduction over the strong HiFi-GAN- baseline, significantly reducing the content loss common with generative models. Our ITC pipeline leverages web-scale knowledge from ASR models without requiring transcriptions for training the enhancement component itself.
Our ITC's transcript guidance method is more flexible than transcript-conditioned S2S models. Such models can struggle when ASR errors create contradictions between acoustic and textual inputs, or their performance may be upper-bounded by the input audio quality if they cannot generatively restore highly corrupted content. Furthermore, they typically lack control over the influence of the transcript versus the acoustics inference time. In contrast, TC-CFG (Eq. equation 4) provides this control through a tunable guidance strength (?). This allows SpeechOp to trade-off prioritizing acoustic fidelity emphasizing content restoration guided by the transcript depending on the application.
Table 3: Speech Enhancement Results. (Left) Quantitative metries. (Right) Subjective MOS scores with standard error.
Model
PESQ
MCD
SpBS
WER
Noisy Source Audio
1.12
11.22
.888
3.3
Model
MOS
StoRm
1.61
6,36
883
7.0
Noisy Source Audio
1.78 amm
SGMSE+
Mlipher
1.98
1.44
5.28
.923
.898
7.0
5.7
HiFi-GAN-2
SGMSE+
3.90 2:044
1.00
HiFi-GAN-2
2.23
4,40
.934
5.4
SpeechOp (No Transcript)
+TTC
2.05 (+0.05)
2.00
4.85 (+0.02)
4,83
.928 (+0020)
908
2.9
8.1
SpeechOp (No Transcript)
SpeechOp-ITH (WhisperX)
3.93 1:00
2:044
+Speaker Personulization
2.12 (+0.07)
4.69 (-0.16)
.926 (.0022
2.4 (-0.5)
Clean Reference Audio
4.67
SpeechOp (Gold Transcript)
2.06
4.83
.931
2.1
Even using Whisper transcripts derived from the noisy source audio, ITC improves content intelligibil- ity over the original audio (WER 2.9% vs. 3.3%) and enhancement without transcripts (WER 8.1%). This suggests TC-CFG effectively balances the acoustic information from the noisy source audio with the imperfect guidance from ASR transcription. While signal-fidelite metries often penalize generative outputs, SpeechOp's 's ITC matches HiFi-GAN-2's subjective quality (Table Right) while delivering superior content accuracy.
We compare SepFormer models in Chen et al. 2024a since they support speaker separation in multiple acoustic environments.
8

---

Published as a conference paper at ICLR 2026
Table Speaker Separation Evaluation (Subj.). We report the average MOS and the standard error
Model
LibriMix Clean
LibriMix Noise
WHAMM
WSJ0-2MB
Total
SepFormer
CereFoorrrChen Chen (2024s)
3.32 0.07
2,95
0.07
3.06
0.07
3.53
0.07
3.22 =: 0.04
DM SopFormer Chon of all (20244
3.59 + 0.07
2.67
=00000000 0.07
2.53 = 0.07
3.58 2: 0.07
3.10 = 0.04
AC-SIM SopFormer Chorre (2224
3.74
2.53
3.65
3.20
AC-SIM ML SepFormer Cienn el al 2024u
3.74 + 0.06
3.02
0.07
2.64 + 0.07
3.66
0.06
3.28 +: 0.04
SpecebOp (No Transcrip0)
3.86 1: 0.07
3.68
0.07
2.89 +: 0.08
3.77 0.07
3:57 =: 0.04
SpeechOp (Gold Transeript)
4.13 + 0.06
4.21
0.06
3.37 = 0.08
3.91 = 0.06
3.92 = 0.03
Mixture
1.38 + 0.05
1.35
0.04
1.39 = 0.04
1.33 = 0.05
1.36 = 0.42
Target
4.26 + 0.06
4.29
0.06
Table 5: Quantitative Speaker Separation Performance the WSJO-2Mix Dataset.
Method
SI-SDRI
MCD 4
SpBS 1
WER 4
SepFormen
Chen al. 2024a
11.86
1.72
.999
4.4
AC-SIM-ME MT. SepFomerr Chen
2024a
11.80
1.55
.931
6.8
SpecchOp (No Transcript)
0.23
4.11
899
11.1
SpeechOp (Gold Transcrip()
0.53
4.20
.919
5.5
SpeechOp also enables novel applications like personalized enhancement by composing enhancement with voice cloning. Given clean voice sample from the target speaker, we compose the enhancement task with speaker-prompted TTS via TC-CFG, allowing the model to simultaneously enhance audio quality while adhering to the target speaker's voice characteristics. This composition improves speaker fidelity (MCD, PESQ) and modestly reduces WER. To provide an upper bound, ground- truth transcripts lead to a 2.1% WER. Across all scenarios, SpeechOp's 's ITC, with our composition approach, effectively integrates textual guidance for controllable, content-aware speech enhancement.
Speaker Separation. On human Mean Opinion Score (MOS) the gold- standard metric for perceived speech quality-SpeecOOp significantly oulper- forms SepFormer basclines across all datasets (Ta Despite these human- rated gains, SpeechOp attains lower ob- jective signal-fidelity metrics (e.g.. SI- SDRi, MCD on WSJO-2Mix: T))le reflecting known mismatch between signal-level metrics and perceived qual- ity for generative models Erdogan al.)
Table Task Composition. We compare our proposed composition formulation (TC-CFG) against averaging the score vectors (TC-Avg), Gold transcripts are used this ablation.
Model
PESQ f
MCD 4
SpBS
WER
Noisy Source Audio
1.12
11.22
888
3.3
SpeechOp (No Transcript)
2.00
4.83
.988
8.1
SpecehOp (TC-Avg) (pecctOp(IC-CFCC (TC-CHG) (Ours)
1.88
5.24
.909
2,06
4.83
.931
2.1
ATC CFG TC-Avg)
(2023) Chen et al. (2024)). This di-
vergence stems from methodology: tra-
ditional mask-based separators optimize signal reconstruction, whereas our generative approach prioritizes naturalness and perceptual quality rather than strict mixture consistency. Importantly, transcript guidance markedly improves content preservation, reducing WER from 11.1% to 5.5% with ground-truth transcripts, showing that SpeechOp can leverage textual information boost separation accuracy while maintaining its perceptual strengths. We note that our current evaluation uses fully overlapping synthetic mixtures following standard benchmarks (LibriMix. 2SJ0-2Mix). Extending SpeechOp to realistic conversational scenarios with partial overlap, combined with diarization-assisedd ASR to provide peaker-attributte transcripts, a promising direction for future work.
Task Composition Ablation. We empirically validate our TC-CFG approach by composing Spee- chOp's enhancement capability with TTS-based textual guidance from the gold transcripts (Table 6). The "SpeechOp (No Transcript)" baseline represents the performance of our enhancement model without any textual guidance. When employing the score averaging approach ("SpeechOp (TC- Avg)"), J. observe degradation in signal fidelity metrics compared to the "No Transcript" baseline (c.g. MCD increases from 4.83 to 5.24). This aligns with the intuition from our synthetic simulation (Figure 4c), where averaging with the broader TTS prior negatively impact the focused prior of

---

Published as a conference paper at ICLR 2026
the enhancement model. While TC-Avg does improve content preservation (WER 3.4% vs. 8.1% for "No Transcript"), ) this comes at the cost of acoustic quality and signal fidelity.
In contrast, proposed composition approach, TC-CFG, demonstrates superior performance across all metrics. not only achieves the best content preservation with a WER of 2.1% (a 38% reduction over TC-Avg's 3.4% WER), but it also maintains improves signal fidelity compared to the "No Transcript" baseline (e.g. PESQ 2.06 vs. 2.00). These results empirically confirm that our TC-CFG formulation effectively isolates text-conditional guidance without degrading acoustic quality. This allows SpeechOp to leverage knowledge from the TTS model for robust content preservation (low WER) while simultaneously maintaining, and even slightly enhancing. the acoustic quality and speaker characteristics established by the enhancement model. This careful decomposition of task- specific guidance is crucial for enabling effective and high-fidelity task composition in generative speech processing.
FUTURE WORK
Several promising directions emerge from this work. TC-CFG's principled decomposition of score functions naturally extends beyond the speech enhancement and separation tasks explored here. For instance, voice conversion could leverage the discriminative signal from speaker-conditioned model to guide a content-preserving base model, and extending the framework to additional tasks (e.g., text-to-audio generation) would further expand the space of composable operations. More broadly, TC-CFG generalizes to any setting where multiple generative tasks can be combined at inference time, including modalities beyond speech. Within speech processing, conditioning on self-supervised learning (SSL) features could improve the baseline intelligibility of enhancement or separation models (Guimarães et al.) 2025), and TC-CFG composition can similarly be applied on top of such an SSL-conditioned model. Further exploration of task conditioning and specialization strategies could also yield improvements in multi-task performance. Finally, extending SpeechOp to cross-lingue settings represents an important direction for building truly universal speech processing systems.
10 CONCLUSION
In this work, we addressed fundamental data disparity between text-to-speech synthesis and speech- to-speech tasks by adapting pre-trained TTS models to enable high-quality speech processing despite limited paired data. Through SpeechOp. we showed that multi-task training not only enables flexible speech-to-speech processing but also improves the underlying TTS quality, demonstrating that these tasks are mutually beneficial rather than competing objectives. Our Compossiii Classifier-Free Guidance (TC-CFG) provides a principled mechanism for composing speech operations at inference time, using the TTS model as a discriminative guide rather than mixing generative priors. Building on this, our Implicit Task Composition (ITC) framework demonstrated how to leverage web-scale speech understanding from discriminative ASR models to achieve state-of-the-art content preservation without requiring parallel transcript data during training. By bridging the gap between data-rich and data- constraind speech tasks, this work opens new possibilities for unified, scalable speech processing systems.
10

---

Published as a conference paper at ICLR 2026
11 ETHICS STATEMENT
Our work advances controllable, generative speech reconstruction for beneficial applications such as accessibility (clearer listening and captioning), restoration of degraded or archival audio, personalized but consented enhancement, and robust lowbbandwidth communication.
We recognize the potential for misuse of such generative technology including imperson- ation/deepfakes. To mitigate these risks, we restrict experiments to publicly available datasets and we recommend deployment guardrails such as watermarking (O'Reilly et al.) 2024) when releas- ing models. For human studies, raters were consenting adults performing non-sensitive listening/MOS tasks and were compensated at fair market rates
12 REPRODUCIBILITY STATEMENT
Our models are trained and evaluated on publicly available data. We provide a complete description regarding the datasets, model architecture, and evaluations to enable faithful reproduction of this work.
REFERENCES
Echothief [dataset].
Accessed: 2024-03-12.
Prolific. https:/ www. www.prolific.cm
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. Proc. Interspeech, 2023.
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. Xits: a massively multilingual zero-shot text-to-speech model. CoRR, 2024.
Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. In Proc. Interspeech, 2021a.
Ke Chen, Jiaqi Su, Taylor Bong-Kikpattrik, Shlomo Dubnov, and Zeyu Jin. Improving generalization of speech separation real-world scenarios: Strategies in simulation, optimization, and evaluation. In Proc. Interspeech, 2024a.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. CoRR, 2021b.
Ziyang Chen, Xixi Hu, and Andrew Owens. Structure from silence: Learning scene structure from ambient sound. In Proc. Conference on Robot Learning, 2022.
Ziyang Chen, Daniel Geng, and Andrew Owens. Images that sound: Composing images and sounds single canvas. CoRR, 2024b.
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mehdi Zohourian, Min Tang, Mehrsa Golestaneh, and Robert Aichner. Icassp 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024.
Hakan Erdogan, Scott Wisdom, Xuankai Chang, Zalán Borsos, Marco Tagliasacchi, Neil Zeghidour, and John R. Hershey. Tokensplit: Using discrete speech representations for direct, refined, and cnnscript-conditionne speech separation and recognition. In Proc. Interspeech, 2023.
Heitor R Guimarães, Jiaqi Su, Rithesh Kumar, Tiago H Falk, and Zeyu Jin. Ditse: High-fidelin generative speech enhancement via latent diffusion transformers. arXiv preprint arXiv:2504.0938), 2025.
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, 2022.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Proc. NeurIPS, 2020.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In Proc. ICML, 2023.
Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Simpler diffusion (sid2): 1.5 fid on imagenet5 12 with pixel-space diffusion. CoRR, 2024.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE Trans. Audio, Speech, Lang, Process., 2021.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In Forty-first International Conference on Machine Learning, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Jaehyeon Kim, Keon Lee, Seungjun Chung, and Jaewoong Cho. Clam-tts: Improving neural codec language model for zero-shot text-to-speech. In Proc. ICLR, 2024.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives the ELBO with simple data augmentation. In Proc. NeurIPS, 2023.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L.. Seltzer, and Sanjeev Khudanpur. study on data augmentation of reverberant speech for robust speech recognition. In Proc. ICASSP, 2017.
Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. Libritts-r: A restored multi-speakerr text-to- speech corpus. In Proc. Interspeech, 2023a.
Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Yu Zhang, Wei Han, Ankur Bapna, and Michiel Bacchiani. Müpphrr: A robust speech restoration model integrating self supervuper speech and text representations, In Proc. WASPAA, 2023b.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Proc. NeurIPS, 2023.
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. In The Thirty-iighth Annual Conference on Neural Information Processing Systems, 2024.
URL https: /ttttp:://ppeeeeee
eww net./ forum? d= whw.nn
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. 2024.
Keon Lee, Dong Won Kim, Jachyeon Kim, and Jaewoong Cho. Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer. CoRR, 2024.
Jcan-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation. IEEE Trans. Audio, Speech, Lang. Process., 2023.
Alexander H. Liu, Matthew Lc, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Gener- ative pre-training for speech with flow matching. In The Twelfth International Conference on Learn- ing Representations, 2024. URL WtLLhtttps///ppeeeeeee
net/ forum? w
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Proc. ECCV, 2022.
12

---

Published as a conference paper at ICLR 2026
Justin Lovelace, Soham Ray, Kwangyoun Kim, Kilian Weinberger, and Felix Wu. Simple-TTS: End-to-end text-to-speech synthesis with latent diffusion, 2024a.
Justin Lovelace, Soham Ray, Kwangyoun Kim, Kilian Q Weinberger, and Felix Wu. Sample-efficient diffusion for text-to-speech synthesis. Proc. Interspeech, 2024b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:221..01095, 2022.
Matthew Maciejewski, Gordon Wichern, Emmett McQuinn, and Jonathan Le Roux. Whamr!: Noisy and reverberant single-channel speech separation. In Proc. ICASSP, 2020.
Patrick O' O'Reilly, Zeyu Jin, Jiaqi Su, and Bryan Pardo. Maskmark: Robust neuralwatermarking for real and synthetic speech. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4650-4654, 2024. doi: 00.1109/CASSP44485..0002 10447253.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An corpus based on public domain audio books. In Proc. ICASSP, 2015.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proc. CVPR, 2023.
Puyuan Peng, Po-Yao Huang, Shang- Wen Li, Abdelrahman Mohamed, and David Harwath. Voice- craft: Zero-shot speech editing and text-to-speech in the wild. CoRR, 2024.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. 2020.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. ICML, 2023.
Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. IEEE Trans. Audio, Speech, Lang, Process., 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. CoRR, 2021.
Jonathan Le Roux. Scott Wisdom, Hakan Erdogan, and John R. Hershey. SDR half-baked or well done? In Proc. ICASSP, 2019.
Takaaki Saeki, Soumi Maiti, Shinnosuke Takamichi, Shinji Watanabe, and Hiroshi Saruwatari. Speechhertscore: Reference-aware automatic evaluation of speech generation leveraging nip evaluation metrics. CORR, 2024.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Proc. ICLR, 2022.
Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, sheng zhao, and Jiang Bian. Naturalspeech Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, 2024. URL htttp::/cppenev/// wet.ieewww.eeeeee...eo..ee..ee wwt.:/.eeeofoum..aeeeee forum? RedidddRdddwWii.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Proc. NeurIPS, 2019.
Jiaqi Su, Zeyu Jin, and Adam Finkelstein. Hifi-gan-2: Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features. In Proc. WASPAA, 2021a.
Jiaqi Su, Yunyun Wang, Adam Finkelstein, and Zeyu Jin. Bandwidth extension is all you need. In Proc. ICASSP. b021b.
13

---

Published as a conference paper at ICLR 2026
Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In Proc. ICASSP, 2021.
James Traer and Josh H. McDermott. Statistics of natural reverberation enable perceptual separation of sound and space. Proceedings of the National Academy of Sciences, 2016.
Rafael Valle, Rohan Badlani, Zhifeng Kong, Sang-gil Lee, Arushi Goel, Sungwon Kim, João Felipe Santos, Shuqi Dai, Siddharth Gururani, Aya Alla fari, Alex Liu, Kevin Shih, Wei Ping, and Bryan Catanzaro. Fugatto Foundational generative audio transformer opus In Proc. ICLR, 2025.
Xiaofei Wang. Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. IEEE Trans. Audio, Speech, Lang. Process., 2024.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 2022.
Donghaa Yang. Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023.
Haici Yang, Jiaqi Su, Minje Kim, and Zeyu Jin. Genhancer: High-fidelity speech enhancement via generative modeling on discrete codec tokens. In Proc. Interspeech, 2024.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for to-seehh. 2019.
14

---

Published as a conference paper at ICLR 2026
A SUBJECTIVE STUDY DETAILS
Methodology. Our studies used native English speakers on Prolific to measure naturalness, quality, voice similarity, and style similarity 1-5 scale, for text-to-speech synthesis based reference speech sample. For our speech processing tasks, we measure the quality of the audio sample. We also included flag for unintelligible content, though no samples were ultimately flagged by majority of raters.
Quality Control. To filter unreliable ratings for the TTS and speech editing studies, used two types of hidden validation tests. The first was mismatched speaker test (different but real speakers for reference and sample); a participant rated speaker similarity > 3, their ratings were discarded. The second was an identical pair test; if any attribute was rated < 4, their ratings were discarded. For the speech processing tasks, we conducted similar validation tests with the clean and noisy audio samples.
Participant details. For all our subjective tests, each worker rated 30 samples, including validation tests. Our TTS study involved 288 unique workers rating 80 utterances per method. Our speech editing study involved 151 unique workers rating 100 utterances per method. Our enhancement study involved 236 unique workers rating 96 utterances per method.
Compensation. For our listening experiments, participants were compensated at a rate $15/hour, which is above the platform's recommendation.
B SIMULATION STUDY IMPLEMENTATION DETAILS
For our guidance comparison simulation study on the ID Gaussian Mixture Model, we provide detailed implementation specifics to ensure reprodueibility.
The ID GMM setting enables exact computation of all relevant quantities, providing controlled environment for comparing guidance strategies. Both the conditional score function, Vz, log Pt (aeel)) and guidance term, Vx, log PrTs (y)2z) can be computed analytically.
Our synthetic experiments use the configuration detailed in Table 7
B.1 GUIDANCE STRATEGY COMPARISON
Our simulation compares three fundamental approaches combining TTS and speech enhancement models:
No Guidance: Uses only the imperfect enhancement model's score function, representing current single-task approaches:
Stotal Senh (Zee, o.)
CFG-Style Guidance: Augments the enhancement model with discriminative guidance from the TTS model using classifier-free guidance:
Siotll
= Senh (Iz; o,) + p. Va, log (gPrssyylz)
where the guidance term V., log prTs (y)l=.) leverages the TTS model's ability to distinguish content- matching samples.
Score Averaging: Linearly combines the enhancement model score with the true conditional TTS score:
Spotal (1 a) (Z,, o.) + STIs(#1, aell))
This approach directly mixes the score functions from both models.
B.2 NOISE SCHEDULE AND SAMPLING
We employ log- lonelinear interpolation for noise levels: a; = exp (f log (mai) + T-t (Fiit)) The update step follows the variance exploding diffusion formulation Karras et al. 2022) Ft++1 St + (0ū - ) Stotal + v 0 - 0i.1 where e ~ ((,
15

---

Published as a conference paper at ICLR 2026
Table Guidance Comparison Simulation Parameters
Parameter
Value
Base Parameters
TIS Distribution (Generic Speech)
Component means
Ng 4.0, M4 4.0
Component std. devs.
00 = 0.9
Component weights
Wy W1 0.5 0
Target component
Enhancement Transforms Mean shift
Aμ = 2.0
Variance reduction factor
? x= 4
Imperfect model bias
6 0.4
Imperfect model variance inflation
1.8
Derived Parameters
True Enhanced Speech
Mean
No + Ap 220
Std. dev.
0o/7 0.23
Imperfect Enhancement Model Mean
Po + =
Std. dev.
B 8.00/7 0.41 0.41
Sampling Parameters
Number of samples Number of timesteps
5000
200
Initial noise level
Omax 80
Final noise level
Omin 0.005
Guidance Parameters
CFG guidance strength
p 10a
Score averaging weight
0.5
B.3 EVALUATION METRICS
We evaluate the final samples using KI. divergence computed between the empirical distribution of generated samples and the true enhanced speech distribution, representing the ideal outcome.
C AUDIO AUTOENCODER
For efficient latent diffusion modeling, we develop an autoencder based on DAC (Kumar al. 2023) but with continuous variational bottleneck instead of residual vector quantization. For 48 kHz input audio y E RIxT the encoder E maps to latent representations x, = E(y) with dimensions R X where C 64 is the latent channel dimension and L is the temporal dimension downsampled by factor of 1200 (resulting in a 40 Hz latent representation). The decoder mirrors this architecture to reconstruct the waveform.
The encoder's 's output is transformed into latent variables through a variational bottleneck that models the approximate posterior (zzly):
G where ~ 100,
(5)
The model is trained to minimize reconstruction loss and KL divergence:
LAE Ey |lly & [1] + CxI
(6)
where Akı. = 0.1 balances the objectives. We also employ adversarial training with a complex STFT discriminator following DAC to improve reconstruction quality.
16

---

Published as a conference paper at ICLR 2026
D ACOUSTIC SIMULATION
First, we randomly select clean speech sample and apply random equalization and compression. Background noise is then added a signal-to-noise ratio (SNR) of -0 to 30 dB. We randomly apply reverberation using impulse response (IR) samples. Additional degradation like random bandlimiting down to IkHz is applied to simulate input at various sample rates. We dynamically generate training pairs during training to increase diversity of the degradation combinations.
We trained the models with public datasets at 44. 1k sample rate. The clean speech data is sourced from LibriTTS-R (Koizumi et al. 2023a) and upsampled to 44. Ik sample rate via bandwidth extension (Su et al. 2021b The noise samples include the DNS Challenge (Dubey et al. (024) and SFS-
Static- Dataset Chen et al. 2022). The impulse response (IR) data includes MIT IR Survey (Traer &
McDermott 2016), Thie (ech). and OpenSLR28 (Ko et al. 20177)
E
ARCHITECTURE AND TRAINING DETAILS
E.I MODEL ARCHITECTURE
We present model architecture details in Table For transfer learning experiments and architecture ablation, we utilize smaller version of SpeechOp with 12 DiT layers and encoder layers. We also incorporate dense connections (Lee al. 2024), position-aware cross-attention mechanism ((Lovelace et al. 2024b)), and append 8 register tokens to process global information (Lovelace al. 2024b). We condition the learnable task embedding by summing it with the timestep embedding that is given to the DiT network.
Table 8: SpeechOp Architecture Parameters
Parameter
Value
Diffiasion Transformer
Audio Latent Dimension
64
Model Dimension
1024
Feed-forward Dimension
3072
Attention Heads
Number of Layers
20
Dropout
0.1
Audio Encoder
Model Dimension
768
Feed-forward Dimension
2304
Number of Layers
Common Components
Position Encoding
Rotary AdaLN (e=le-5)
Layer Normalization
Activation
SwiGLU
Text Encoder
ByT5-base
E.2 TRAINING CONFIGURATION ATION
All model training is distributed across 32 Nvidia A100s. Training proceeds in two stages:
Stage 1: TTS Pre-training Model trained for 400K iterations with a batch size of 4 per GPU. We use AdamW optimization with learning rate 2e-4 and weight decay 0.1. Training employs 4000 warmup steps and we perform two steps of gradient accumulation.
Stage 2:
Multi-task Fine-tuning Starting from the pre-trained TTS model, we extend the encoder to 8 layers and train for additional 200K iterations. We lower learning rate of le-4 and
17

---

Published as a conference paper at ICLR 2026
weight decay of 0.01, with two steps of gradient accumulation. Batch sizes are 4 for TTS and 8 for speech-to-specte tasks per GPU.
Table 9: Multi-task Training Weights and Prompt Probabilities
Task
Weight
Prompt Probability
Speech Enhancement
3.0
0.1
Speaker Separation
3,0
0.9
Noise Isolation
1.0
0.1
Acoustic Matching
1.0
0.9
Speech Isolation
1.0
0.1
Both stages shifted cosine noise schedule (scale=0.5) (Hoogeboom et al. 2023 Lovelace et al., 2024a) with sigmoid loss weighting (bias=-2.5) (Hoogeboom et al. (2024), mixed precision (bfloatl6), and distributed data parallel (DDP) training.
E.3 SAMPLING CONFIGURATION
We use the SIDE-DPM-SOOver+<<MM 256 inference steps with schedule that is linear in logSNR. For speech-to-speech tasks, we utilize classifier-free guidance (Ho & Salimans 2022) with a strength of 1.5. For zero-shot TTS we guidance scale of 3.0 for the transcript and prompt conditioning information. For speech editing, we use guidance scale of 2.0 for the transcript and prompt.
described in Lu et al. (2022) for sampling. We utilize
For zero-shot TTS and speech editing, our non-autoregreesive approach requires determining the output duration before generation. We estimate this by first computing the speaking rate (phones per second) from the reference speech prompt. For zero-shot TTS, we then multiply this rate by the phoneme count of target transcript determine the output duration. For speech editing, we preserve the original duration for unedited regions and apply the same rate-based estimation for edited segments. We found this simple duration modeling approach sufficient for maintaining natural speaking rates aligned with the reference speaker's style.
For task composition, we can control the guidance strength in the same way. Higher guidance values enforce stronger conditioning at the cost of potentially conflicting with the other task. We use a scale 1.5 in our composition experiments. We find that TTS guidance is only necessary for resolving details in modest-to-high SNR regimes, so enable it for logSNR ranges greater than -1.0 Kynkäänniemi et al. 2024
F SOURCE AUDIO CONDITIONING ABLATION
Table 10: Source Audio Conditioning Ablation. We train an ablation model that conditions on the source sequence vectors with cross-attention mechanism instead of our frame-wise mixing.
Model
PESQ
MCD
SpBS 1
WER J
Noisy Source Audio
1.12
11.22
.888
3.3
SpeechOp-Snie Small (Cross-attention)
1.18
15.4
.751
100
w/ chunking
1.88
4.98
.900
9.6
SpeechOp-Small (Framewise-Mixing) (Ours)
1.96
4.86
.902
8.8
Using 12-layer models, we compare our framewise mixing strategy against a cross-attention based approach for conditioning source audio. Table 10 shows that the cross-attention variant fails catastrophically when processing sequences other than its 5-second training length (WER > 100%). Even with explicit padding and chunking account for this, it shows degraded performance across all metrics. In contrast, our framewise mixing approach generalizes naturally to arbitrary sequence lengths while achieving better quality (PESQ 1.96 vs 1.88), lower distortion (MCD 4.86 vs 4.98),
18

---

Published as a conference paper at ICLR 2026
and improved content preservation (WER 8.8% 9.6%). These results suggest that framewise mixing provides a more robust foundation for speech-to-speect processing, likely due to the explicit fraee-level correspondence between souree and target audio.
G TASK COMPOSITION DERIVATION
Here present the detailed derivation of our task composition approach. Our goal is estimate the conditional score function vx, log p(z w), where z the noisy latent, y is the noisy source audio, and to is the text transcript.
Starting with Bayes' rule, we can decompose the joint conditional probability:
ply, w wlzz))) (p(z))
Zz, log plzel, w)
= Vz, log
p(u, w)
2x, log p(y, z.) Z2 log Vz, log
= 2x log p(y, u zz + Zz, log p(ze).
where we drop the term V Zz. logp(y, w) as it is independent zz.
We introduce conditional independence assumption: given the noisy latent z,, the textual transcript w is independent the noisy source audio That is:
ply, wlz:) p(ylz))((u1z)
This assumption is reasonable modest-to-hipe signal-to-noise ratios where the latent representation effectively captures the salient information from both modalities. Substituting Equation equation into Equation equation
Vz, ne log (Zely,, w) = Vz, log pvy z. ( 2.) + Vz, logp(z:) Vz, log ply 2.) + Zz, log logp(w/zz 2z)) Zz, pzz)).
For the term V z. log p(pp(222 ) we apply Bayes' rule again. Following the classifier-fres guidance approach of Ho & Salimans (2022), this can be expressed in terms of conditional and unconditional TTS score functions:
2, logp(w/zz Vz, (ogp(zz/w) - Vz, log p(z.).
(10)
Substituting Equation equation
10 into Equation equation and noting that Vz, logp(ulzz;)
+
Vz, log p(z)) Vz, log p(zely). obtain:
Vn, log p(zely, w)
Vz, log p(zely) (Vz, log logp(2,/w)) Vz, logr(z;))
Rs si (xely) + (sh (z. (w) 80 (2:)),
(11)
where (2.l)) and si (ze (w) represent the score networks for enhancement and TTS tasks, respec- tively.
This derivation shows how our approach naturally combines the enhancement and TTS score functions while avoiding conflicts between their unconditional priors. The enhancement term guides the denoising process while the TTS term provides content alignment through classifier-free guidance.
H LLM USAGE
We used large language models for copyediting and revising the wording; all claims and arguments were drafted and verified by the authors.
19


Reviews:
[{"review_id": "POyY8cQkuQ", "reviewer": "Reviewer_vHfv", "summary": "This paper presents SpeechOp, a multi-task latent diffusion model designed to transform pre-trained Text-to-Speech (TTS) models into a universal speech processor capable of performing and composing multiple speech-to-speech (S2S) tasks (e.g., enhancement, separation, voice cloning) at inference time. The key innovation lies in the Task Composition Classifier-Free Guidance (TC-CFG) mechanism and Implicit Task Composition (ITC) framework, which allow flexible inference-time combination of speech operations using text or ASR-derived guidance (e.g., from Whisper). The authors show that TTS pre-training accelerates convergence and improves downstream S2S performance, and that SpeechOp achieves state-of-the-art content preservation across tasks such as enhancement and separation.", "questions": "How does SpeechOp perform on cross-lingual or non-English speech tasks, given the ByT5 encoder and English-only datasets?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of repurposing pre-trained TTS models for versatile S2S operations via inference-time task composition is well-motivated, addressing the data scarcity challenge in generative S2S tasks.\n2. The paper reports substantial improvements in speech enhancement and perceptual quality gains in speaker separation.", "weaknesses": "1. The idea of leveraging pre-trained diffusion-based TTS models as clean speech generators for downstream tasks such as denoising, enhancement, and separation has already been proposed in Voicebox, Audiobox, and Speechflow. The paper’s novelty mainly lies in its composition mechanism, rather than the overall concept of using TTS priors for multi-task speech processing.\n2. While the composition strategy is interesting, the underlying model (latent diffusion with DiT backbone) largely follows existing designs (Voicebox, DiTTo-TTS, etc.). The architectural contribution is modest.\n3. ITC’s success depends heavily on accurate ASR transcripts. The paper lacks robustness analysis under ASR noise or low-resource conditions. Maybe using some softer semantic supervision like SSL features instead of the explicit transcripts would mitigrate this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rsykzEDkuK", "reviewer": "Reviewer_SsZ8", "summary": "This goal of this paper is to develop a speech model that can perform some speech to text and speech to speech tasks by starting from a diffusion based model that is trained only for TTS initially. The argument is the availability of relatively larger scale data for TTS task while lack of sufficient paired data for speech denoising, speaker separation, speech enhancement, etc. tasks. In addition to multitask training, the model allows task composition during inference time, such as Text-to-Speech + Enhancement, Text-to-Speech + Separation in one model. This is achieved by a method called the implicit task composition (ITC) where a Whisper based ASR transcripts are used in place of  text component to guide the proposed SpeechOp model's speech enhancement process. ITC does not only lead to a model capable of handling different tasks but also improves the TTS task performance that the model was initially trained on. The TTS model is based on diffusion modeling on DAC based latent speech features. In addition to text and audio features, the model hasa a learnable Task Embedding that conditions model behavior. Once the basic TTS model is trained the multitask training consists of two stages. In the first stage, TTS and S2S tasks are jointly trained. In the diffusion training step, classifier free guidance method is used. For the multitask method the training objective is a weighted sum of the speech enhancement and the TTS losses. The system is trained several open source datasets and all speech signals are upsample to 48kHz. Experiments compared the performance on both individual tasks and task composition performance. Experiments claim competitive audio quality, significantly outperforming SepFormer based speech separation  across all datasets. Especially, experiments suggest that the text guidance is an important part of the proposed SpeechOp.", "questions": "1. Please discuss the effectiveness of the task embeddings in the SpeechOp model more clearly. \n\n2. In some experiments, SpeechOp results are presented in (no transcript) and (ground truth transcript) versions. For completeness, it could have been useful to see the SpeechOp results with the ASR transcriptions (e.g. in Table 4). Could the authors please comment on that?\n\n3. In Table 3, there is a reference to Speaker Personalization but I could not see the explanation of how this is achieved. Could you please explain?\n\n4. (Minor typo) Section 5. Speech-to-Speech Pathway: us a complex alignment mechanism -> *use* a complex alignment mechanism", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n+ The paper proposes a text guided multitask modeling for several speech tasks in one model that was initially trained for TTS. Thus the model can leverage the relatively high-resource TTS task in other speech related tasks such as speech separation, and inclusion of the text guidance allow for inference time task composition that can expand the model capabilities. \n\nQuality \n+ Experimental results suggest that the generated audio quality is competitive as compared to audio reconstruction with a HiFiGAN2 vocoder. In speech separation, the paper outperforms the Sepformer based baselines.  \n\nClarity \n+ Mostly clearly written. \n+ Datasets used in the experiments are open source. Some experimental settings are provided in the appendix. \n  \nSignificance\n+ Since the SpeechOp paper is a multitask speech model capable of handling several speech tasks, it is relevant to various subsets of speech researchers.", "weaknesses": "The following points are mostly about the clarity of the content or the quality of the experimentation. \n\n- One point is that is not very clear is how critical is the task embedding in the system if Fig. 3. or whether given the complex interactions between diffusion latents and text encodings through cross attention, it is not clear whether the model can leverage the task embedding efficiently.\n\n- In some experiments, SpeechOp results are presented in (no transcript) and (ground truth transcript) versions. For completeness, it could have been useful to see the SpeechOp results with the ASR transcriptions (e.g. in Table 4).\n\n- In Table 3, there is a reference to Speaker Personalization but I could not see the explanation of how this is achieved. Could you please explain?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B7JLt3yyer", "reviewer": "Reviewer_omQ3", "summary": "The paper proposes a multi-task training method that jointly learn on TTS and speech processing tasks. The paper also proposes a test time composition method using CFG to better guide the speech processing task.", "questions": "None", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. While multi-tasking learning is not new, the paper propose some novel design choices that let TTS and speech processing tasks to be trained more seamlessly in a diffusion model.\n2. The inference time task composition method uses text conditioning CFG to additionally guide the speech processing task and seems to have good results.\n3. The paper is in general written clearly and easy to follow.", "weaknesses": "1. The competitors compared in the evaluation section are in general too few and not SoTA. For example, in TTS all models compared are from 2024. There are many new TTS models (and some good open sourced ones like Higgs Audio) in 2025. I think the evaluation section need to be updated with results comparing against more recent methods. The same for speech editing where only one work is compared against. \n2. While the paper performed a simple ablation on the task composition, it didn't run any ablation on the design choices of the training methods and model architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lcKlZBuPbA", "reviewer": "Reviewer_2PAD", "summary": "This paper introduces SpeechOp, a unified and versatile speech processor that leverages a pretrained DiT-based TTS model to perform a broad range of speech tasks (e.g., enhancement, separation, voice cloning) at inference time. The authors note that TTS models benefit from abundant in-the-wild data, S2S models typically rely on limited paired datasets, constraining their generative fidelity and data efficiency. To bridge this gap, SpeechOp employs a two-stage training paradigm, beginning with TTS pretraining followed by multi-task fine-tuning across multiple speech operations. Furthermore,  the paper proposes TC-CFG, a principled approach that fuses different modalities to improve the model performance at inference time. Experimental results demonstrate consistent improvements in both generative quality and task flexibility, validating SpeechOp as a strong step toward a universal, generative speech processor.", "questions": "- How much additional training time does SpeechOp require compared to the standalone TTS baseline?\n- Can the proposed TC-CFG mechanism be generalized to other modalities or tasks (e.g., direct application to TTS or voice conversion scenarios)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is clearly written and effectively motivates the need for unifying TTS and S2S paradigms.\n- Empirical results convincingly demonstrate that initializing S2S models from a pretrained TTS backbone accelerates convergence and enhances performance.\n- The proposed TC-CFG mechanism is conceptually well-founded and illustrated through a clear 1D toy example, highlighting its advantage over simple score averaging.\n- The proposed framework effectively leverages different modalities to boost the task performance.\n- The extensive experiments across TTS, enhancement, and separation tasks collectively validate the versatility and efficacy of SpeechOp.", "weaknesses": "(minor suggestion) Including the task embedding component in Figure 3 would further clarify how task conditioning is integrated across modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)