TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization
Authors: Waris Quamer
Mu-Ruei Tseng
Ghady Nasrallah
Ricardo Gutierrez-Osuna
Venue: ICLR 2026 Poster (2026)
Keywords: Time-varying timbre
Streaming voice conversion
Content-synchronous speaker conditioning
Speech anonymization
Vector-quantized bottleneck
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A streamable voice conversion/anonymization system that synchronizes time-varying timbre with content via a Global Timbre Memory, improving naturalness and privacy under strict low-latency constraints.

Abstract: Real-time voice conversion and speaker anonymization require causal, low-latency synthesis without sacrificing intelligibility or naturalness. Current systems have a core representational mismatch: content is time-varying, while speaker identity is injected as a static global embedding. We introduce a streamable speech synthesizer that aligns the temporal granularity of identity and content via a content-synchronous, time-varying timbre (TVT) representation. A Global Timbre Memory expands a global timbre instance into multiple compact facets; frame-level content attends to this memory, a gate regulates variation, and spherical interpolation preserves identity geometry while enabling smooth local changes. In addition, a factorized vector-quantized bottleneck regularizes content to reduce residual speaker leakage. The resulting system is streamable end-to-end, with <80 ms GPU latency. Experiments show improvements in naturalness, speaker transfer, and anonymization  compared to SOTA streaming baselines, establishing TVT as a scalable approach for privacy-preserving and expressive speech synthesis under strict latency budgets.

Published as a conference paper at ICLR 2026
TVTSYN: CONTENT-SYNCHRONOUS TIMBRE FOR STREAMING VOICE CONVERSION AND ANONYMIZATION
Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah & Ricardo Gutierrez-Osuna Department of Computer Science and Engineering
Texas A&M University
College Station, TX 77840, USA
{quamer waris, mt.seng, hog.aassrralllaarrttreereeerere edu
ABSTRACT
Real-time voice conversion and speaker anonymization require causal, low- latency synthesis without sacrificing intelligibility or naturalness. Current systems have a core representational mismatch: content time-varying, while speaker identity is injected speech synthesizer that aligns the temporal granularity of identity and content via content-synchronous, time-varying timbre (TVT) representation. A Global Tim- bre Memory expands a global timbre instance into multiple compact facets; frame- level content attends to this memory, gate regulates variation, and spherical in- terpolation preserves identity geometry while enabling smooth local changes, In addition, factorized vector-quantized bottleneck regularizes content to reduce residual speaker leakage. The resulting system is streamable end-to-end, with <80 ms GPU latency. Experiments show improvements in naturalness, speaker transfer, and anonymization compared to SOTA streaming baselines, establishing TVT as a scalable approach for privacy-preserving and expressive speech synthe- sis under strict latency budgets.
static global embedding. We introduce streamable
INTRODUCTION
Real-time voice conversion (VC) and speaker anonymization (SA) aim to deliver natural, intelligible speech while meeting strict streaming and latency constraints. Beyond words, voice recordings carry biometric and paralinguistic cues-identity, sex, age, accent, and emotion-thar adversaries can exploit for recognition and profiling, creating real risks to privacy. As voice interfaces proliferate and privacy rules tighten, protecting these speech attributes without degrading the communicative utility of speech has become essential, as exemplified by several initiatives. As an example, starting in 2020 the Voice Privacy Challenge (Tomashenko et al.) 2020) evaluates speech anonymization systems on both privacy and usefulness, with shared benchmarks that quantify speaker obfuscation alongside speech quality. Further, federal agencies have pushed for the development of real-time solutions with tight latency budgets (e.g., IARPA's Anonymous Real-Time Speech program
To address this challenge, recent speech architectures have achieved sub-second latency by combin- ing lightweight content encoders with direct waveform decoders (Quamer & Guttierre- 2024 2025a Yang al., 2022 Chen al. 2023). Yet a core limitation of these approaches persists: while content is represented as a time-varying sequence, speaker identity is typically injected as a single static vector. This dynamic-static mismatch dampens expressivity and often yields over-smoothed timbre, especially when articulation, emotion, or emphasis change within an utterance. Making content strongly beeaker-independent (e.g.. via aggressive bottlenecks) can improve anonymization performance but suppress meaningful variations in speech such as accent and emotional color, or introduce artifacts (Quamer & Cutierrez-Osunal (025a)). We contend this trade-off largely ar- chitectural: stationary speaker vector forces the decoder to reconcile incompatible time scales. better formulation would add temporal granularity to speaker conditioning to match that of the content, while allowing control and meeting tight latency constraints.

---

Published as a conference paper at ICLR 2026
We propose TV streaming speech synthesizer that replaces static speaker embeddings with a lime-varying timbre (TVT) representation that is synchronized with the content. Global Timbre Memory (GTM) expands global timbre seed into compact set of timbre facets; frame-level con- tent attends to this memory to retrieve the most relevant facets over time; a learned gate regulates how much timbre is allowed to vary; and spherical interpolation blends global and time-van mvrringg paths to preserve identity geometry while enabling smooth local variation. This TVT stream conditions causal decoder alongside pitch/energy predictors, yielding natural variation while retaining control. Complementing this, a factorized vector- quantized bottleneck regularizes the content network to re- duce residual identity cues while preserving linguistic content. TVTSyn runs in a streaming fashion with small, mask-based future access in the encoder and fully causal decoding. The system can gen- erate synthesis with < 80ms latency on a modern GPU and runs within a few hundred ms. latency on C We evaluate the model for both VC and SA tasks under the VoicePrivacy Challenge (VPC) 2024 protocol, reporting equal-error-aate (EER) in automatic speaker verification (ASV) as measure of privacy preservation, and word error rates (WER) of an automatic speech recognizer (ASR) as a measure of utility, as well as latency and real-time factors. Our main contributions are:
Content-synchronous timbre modeling: We introduce time-varying timbre formulation that aligns speaker conditioning with frame-level content, resolving the static-dynamic mismatch responsible for degraded quality in streaming VC and SA.
A streamable low-latency architecture: We design fully causal system that integrates GTM-based timbre, factorized VQ bottlenecks, and prosodic predictors, and maintains low latency while balancing naturalness with speaker fidelity, and anonymization robustness.
Comprehensive benchmarking: We evaluate across VC and anonymization tasks with perceptual quality, speaker similarity, privacy (EER), utility (WER), and runtime perfor- mance, demonstrating superior privacy-utiliy utility trade-offs over prior streaming systems 3
2 RELATED WORK
Voice conversion (VC), Conventional VC models decompose the speech signal into content and speaker channels, then re-synthesize the content channel with target voice identity. Early pipelines relied on cascaded ASR-TTS (text-to-speech) modules or used phonetic posteriorgram (PPG) rep- resentations to obtain a -ideeedenddeenn content stream before conditioning a multi-speaker decoder on the target voice (Huang et al., 2020 Liu et al.] 202I). To avoid the brittleness of text/PPG representations, later "disentanglement" models learned content directly from audio via information bottlenecks and normalization/MM penalties, or with discrete units (e.g., VQ. HuBERT pseudo-labels) that suppress residual speaker cues while keeping phonetic detail (Chan et al. 2022
Chen et al. 2021 Wang et al.) 2021 Quamer et al. 2023 Quamer & Gutterrez-Osuna 2025b).
Self-supervised speech representations such HuBERT have since become standard for providing
robust, transcript-free supervision of content features (van Niekerk et al. (022).
Streaming VC introduces strict latency constrains. Supervised systems such LLVC (Sadov al., 2023) can achieve latencies as low as 20 ms, but their reliance on parallel data makes them dif- ficult scale. By contrast, most unsupervised VC approaches adopt causal encoders with small inference, and fast waveform decoders, achieving sub-second latency future peeks, intelligibility (Quamer& Gutierrez- Osunal 2204; (225a]] Yang et al., [022] Chen while maintaining
eet al. 2023) ZZhng et al. 2025). Here, speaker identity is usually single global embedding in- jected at every byame-by concatenating content and speaker embeddings, and more recently via FiLM/AddaN or conditional layer normalization (CLN), which allow each channel to be scale/shift modulated, enabling few/zero-shot adaptation in multi-speaker TTS/VC (Huang & Belongie 2017 Perez et al. 2018). A complementary line models intra-speaker variability with multi-center (sub- center) training, learning multiple anchors per speaker to reduce intra-class scatter and capture chan- nel/phonetic/affeecte
variation (Ulgen et al.,, 2024). However, these streaming approaches typically use static speaker embeddings that remain constant across all frames. While this enables low la- tency, it creates representational mismatch with dynamic content embeddings. Recent attention- based methods attempt to address this: FCe (Li et al.) (023) uses static embeddings lacking temporal variation; Ge (Cai et al. (2025) employs learned queries but requires non-causal infer- ence; DAFMSVC (Chen et al. 2025) enables content-aware modulation but is offline-only and lacks
"See hardware specs in section
"Audio samples can be found at: http:///mmaammiieeoo////ggttts/s
2

---

Speaker anonymization (SA). Anonymization aims to mask speaker identity while preserving communicative utility. Traditional sigetal-iiggall-reesssiii (DSP) approaches manipulate the signal through formant shifting (e.g.. McAdams coefficient (McAdams) 1984)) frequency warping, vocal- tract length normalization, pitch/rate modifications, or modulation spectrum smoothing (Patino et al. 202 Tavi et al. 2022). Though these training-free approaches are lightweight and fast, they struggle against modern ASV back-ends. Starting 2020, the VPC formalized attacker mod- els and popularized two baselines: a DSP approach (McAdams), and machine-learning (ML) the quality-privage trade-off as part of the Published as a conference paper at ICLR 2026
learnalle speaker prototypes. Our GTM differs by introducing learnable prototype parameters that capture universal timbre characteristics across speakers, while speaker-specifi modulation adapts them to individual identities enabling efficient generalization under streaming constraints.
pipeline with x-vectors plus neural conthesis-solidifyeee evaluation protocols (Tomashenko et al. 2020).
ML-based anonymization typically follows the VC template: extract content, replace identity with an anonymized embedding, then synthesize. Identity replacement includes farthest-speaker selec- tion (Srivastava et al. 2020), autoencoder-based removal of protected attributes [Perer-Coodoeero et al. (2022), codebool/lookup strategies Hsu et al. (2018), ) and learned pseudo-speaker genera- tors (GAN-based sampling in embedding space) Meyer et al. (2023). Recent systems underscore the need to keep distinctiveness and emotional color while pushing EERs towards chance levels. Streaming variants compress encoders and remove heavy SSL/ASR stacks to meet latency con- straints, sometimes allowing limited future context in causal attention layers (Quamer & Gutierrez. Osuna 2024 2025a). Discrete bottlenecks further anonymize content but can hurt naturalness if too aggressive highlighting the need align the temporal granularity of identity conditioning with frame-synchronous content. Finally, language-model-basse Ge demonstrates strong style transfer by tokenizing phonetic and acoustic streams and condi- tioning a causal LM with a style prompt before vocoding; these models show the benefits of better temporal modeling and rich style prompts, though most target offline applications (Cai al. 2025
generative VC/anonymization such as
METHODS
Shown in Figure Dp. our system architecture consists of four modules: (I) a content encoder that generates discrete, beeakerr-ndeeendeet linguistic representations in a causal manner, (2) a speaker processing block that consumes global speaker embeddings and produces content-aligned, time- varying timbre representations, (3) pitch and energy predictors that model frame-level prosodic variation, and (4) decoder that synthesizes speech waveforms directly from the combined rep- resentation. At inference, the framework can convert/anonynize speaker identity by altering the speaker embedding its time-varying trajectory, while leaving linguistic information intact.
only
Content
Way Decocer
Speakir Proceesing
HUBERT
Crroos entrosy
ForEnergy Patacher
CNN
Consont
Linwar
Encedor
layer
layor
Content unooder
Discrminators
(6)
(b)
Figure 1: (a) The content encoder TVTSyn is trained separately with supervision from an off-line HuBERT model. (b) The waveform decoder is trained in self-supervised fashion reconstruct the input utterance from content and speaker embedding streams. Dashed lines are disabled at inference.
3.1 STREAMING CONTENT ENCODER
Feature extraction. The content encoder transforms input waveforms into 512-dim. frame-level embeddings that emphasize linguistic content while suppressing speaker-specific cues. It is imple- mented as a fully causal 1-D CNN followed by a contextual self-attention layer. The CNN begins with initial convolution (kernel size 7) capture fine-scale waveform structure, and then applies four downsampling stages with stride ratios of [8, 5, 4, 2], yielding an overall hop of 320 samples (~200 ms 16 kHz). Each stage doubles the channel width and is preceded by lightweight residual
3

---

Published as a conference paper at ICLR 2026
block with two convolutions (kemels 3 and 1, dilation 2) and a true skip connection, preserving local phonetic detail while remaining streamable. A final convolution (kernel size 3) projects the sequence into a 512-dim. latent space. To capture dependencies beyond the CNN's local receptive field, append a stack of 8 causal multi-head self-attention (MHSA) blocks with a fixed look-back win- dow of w=2s. To avoid using a separate look-ahead module (Quamer & Gutierrez-Osuni (025a), TVTSyn expands the causal attention mask to up to future tokens (~-00 ms). Formally, at time step attention is masked to keys/values from Tt....... t..... ,,...,,+4.. ++4] with corresponding W. This provides stable long-range temporal coherence with short-term anticipatory cues for coarticulation while avoiding the latency overhead of separate ahead module. During inference, a ring KV cache maintains rolling ~2 8 window of past keys and values, enabling efficient reuse of context.
Learnable bottleneck with Factorized VQ. remove residual speaker information and regular- ize the content space, we used factorized vector eector-quantized (VQ) bottleneck (Ju et al., 2024). The 512-dim encoder output is first compressed to an 8-dim latent vector via learned projection, quan- tized using learnable codebook of size 4096, and then projected back to 512 dimensions. This compress-hhen-diseettiz design encourages the model to learn discrete, speaker-independent units while preserving linguistic fidelity for downstream synthesis.
Training objective. The encoder and bottleneck are optimized with a cross cross-entropy objective against discrete pseudo-labels obtained by applying k-means clustering (NV 200 centroids) to the 9th layer activations of a HuBERT-base") l The encoder, projection layers, and VQ codebook trained jointly that the bottleneck learns to predict discrete units, while the CNN encoder and MHSA context layer capture both local phonetic cues and longer-range temporal dependencies. This stage fully self-supervieed and does not require transcripts or text alignments.
3.2 TIME- VARYING TIMBRE (TVT) REPRESENTATION
Conventional VC systems represent speaker identity with single global embedding g E Rd ex- tracted from a reference utterance or corpus. While captures a speaker's 's timbre at coarse level, a static vector. In contrast, content embeddings {ca) vary the frame level, encoding phonetic and prosodic dynamics. This mismatch between static speaker and dynamic content representations often produces over-smoothed timbre, limited expressivity, or degraded consistency under challeng- ing conditions. To overcome this, we introduce time-varying timbre representation that allows the speaker embedding evolve in syne with the content -see Figure 2
Speaker Drired
Conted
Emoed
Fribed
Training
Daractor
Cauisi NiSSA
CLiiet Fuzkkn
Linoer Tayar FRLFEE
CauasiMissk
Proeett Bepasn
Scalled Alterdin
Desavittnse
(Uiirit Fuakm
CNN becover
polattons
waryeng
(4)
Figure 2: Architecture details for (a) TVT processing block, (b) waveform decoder.
Global Timbre Memory. To obtain a global speaker embedding, we concatenate two complemen- tary
X-vectors (Snyder et al. 2018) and context-aware ECAPA- -DNN
(Desplanques et al. (2020). which improves downstream anonymization quality (Meyer al. 2022
We project this global embedding into Global Timbre Memory (GTM), parameterized as K
key-value pairs [(ks, o))) with ki, V; E Rdd The GTM employs a dual representation: a speaker- prior specific component generated by an MIP from g. and learnable prototype parameters p.rrior "
4

---

Published as a conference paper at ICLR 2026
shared across all speakers. Formally, each key-value pair is computed as:
k = (g): + k.tior 2/ = Mg) of
where the priors act universal timbre prototypes that capture phoneme-agnoste characteristics (e.g., breathiness, nasality) common across speakers, while the MLP output modulates these pro- totypes to reflect individual voice identity. This design provides a strong inductive bias, improving sample efficiency and training stability, particularly in low-data regimes or with unseen speakers. Intuitively, the GTM decomposes the timbre into multiple "facets" (e.g., spectral color, nasality, brightness), each stored in a slot. At each time step t, the content embedding c attends over the keys to retrieve weighted timbre component: 2t- = Attn(c,, {ki]. {vr F), where scaled dot-product attention produces distribution over slots. This enables the model to select the most relevant timbre sub-components given the current phonetic and/or prosodic context.
Gating and Interpolation. To balance stability with flexibility, a gating network computes scalar Or E [0, 1] that modulates how much the embedding should deviate from the global timbre. The final time-varying embedding 8, is obtained by interpolating between g and et: 8, Slerp(g, 0y; 0z), where Slerp denotes spherical linear interpolation, which respects the hyperspherical geometry of the embedding space, ensuring smooth trajectories and preserving angular distances. Specifically, Slerp interpolates along the geodesic (great circle are) connecting two points on the unit hypersphere, maintaining constant angular velocity: 0, = (1 - a, + 0g (Shoemake 1985). This geometrie property has been shown to better preserve identity characteristics in high-dimensional embedding spaces compared to Euclidean interpolation (Zhong al.] (025). This avoids unnatural distortions that could arise from linear interpolation in Euclidean space, which creates shortcuts through the interior of the hypersphere, requiring re-normalization that distorts angular relationships and can cause perceptual artifacts.
Intuition. Conceptually, y provides stable "base palette" for speaker's identity, while the GTM decomposes this palette into finer brushes. The learnable prototypes serve as a universal "brush set" that is refined during training to capture common timbre characteristics across all speakers, while the speaker-specific MI.P output adjusts the bristle texture and pressure to match individual voices. At each frame, the content embedding chooses which brushes to apply, adjusting timbre in context- sensitive ways. Gating mechanism controls how bold these adjustments are, and Slerp guarantees that the blend remains smooth and consistent. The resulting sequence of TVT embeddings {st} preserves global identity while adapting locally, enabling more natural and controllable synthesis.
3.3 STREAMING WAVEFORM DECODER
Speaker Conditioning. We condition the encoder output and subsequent latent representations in the decoder on TVT embeddings -see Figure Pb. Timbre conditioning is achieved through a Conditional Layer Normalization with Fusion module. Given latent features x E Bx612x7 and the module normalizes æ. then generates per- frame scale and shift coefficients (7, 3) from s. The re-normalized features are fused with a gated, normalized version of s and projected back to the latent dimension:
time-varying speaker embeddings E R 1x1xx12x33
y Proj (1 + x) Norm(x.)) + Be || g: Norm(s:)).
where is learned gate and denotes concatenation. This design allows the model to integrate dynamic speaker information while providing stability to the normalized content features.
Prosodic variation is introduced by lightweight predictors for fundamental frequency (F0) and energy. Each predictor is 2-layer causal CNN (kerne=3)) with ReLU activa- tions and a final point-wise projection. During training, these modules are supervised with ground- truth FO and energy extracted from the waveform; at inference, their predictions are injected into the feature stream, enabling explicit control over pitch and loudness.
FO/Energy Predictor.
The decoder mirrors the content network and reconstructs waveforms from the conditioned embeddings. It begins with a context layer composed of a stack of 8 causal MHSA blocks with fixed look-back window of W-2s s but no future peeking. A ring KV cache maintains this rolling past context for efficient streaming. Following the context layer, CNN decoder inverts the encoder's ' temporal compression via four causal ConvTrannpaa ID stages with strides [2, 4, 5, 8] (the reverse of the encoder's downsampling). These stages progressively upsample the sequence from 50 Hx
Decoder.
5

---

Published as a conference paper at ICLR 2026
back to 16 (overall factor I 8, 2x4x5x8 320). Each upsampling stage is interleaved with a lightweight residual block structurally matched to the encoder (two 1-D conv with kernels 3 and 1, dilation base 2, ELU activations, and true skip connection), preserving fine-scale detail contributed by content, time-varying timbre, and prosody streams.
Training objective. The decoder is optimized with a multi-objective loss that balances spectral fi- delity and naturalness. We combine an LI reconstruction loss Lmel log-Mels multiple window lengths (2- 128 ms) with adversarial objectives from multi- muulti-priiod waveform and multi-band spectro- gram discriminators, Ladv, a feature- matching matchig term Lim on discriminator activations (Kumar et al.) 2023), and FO/energy predictor L2 loss Line. The total loss is
Liaal Amal Liee Dunee Lute Amm Lim + Am- Livve:
where A(33 weight the respective terms. We use Amel- Amme= = 20, Audv 1 and Aim = 2.
4 EXPERIMENTAL SETTING
Datasets. We train our content encoder and decoder on the LibriTTS corpus (Zen et al. 2019).
which provides roughly 600 hours of read English speech. Pretrained speaker encoders taken from SpeechBrair RRavanelli et al. 2021 trained on the VoxCeleb corpus (Nagrani et al. 2017).
Tasks. We consider two tasks for evaluation: voice conversion (VC) and speaker anonymization (SA). For VC, we use CMU ARCTIC (Kominek) 2003), L2-ARCTIC (Zhao et al.) 2018)), and VCTK/Veaux al. 2017) source datasets, and EMIME (Wester 2010) as the target. Specifi- cally, for each speaker we randomly select 50 utterances and convert them into a different set of random target utterances from the EMIME English subset. For SA, we follow the VPC 2024 evalu- ation protoco 0 Namely, we use the LibriSpeech dev-clean and test-clean subsets (Panayotov et al., 2015) to compute objective intelligibility and privacy metrics. Anonymized samples are generated by randomly selecting a target utterance from EMIME.
Metrics. For the VC task, report two performance metries: synthesis quality and speaker simi- larity. To measure synthesis quality we use NISQA-MOS (Mittag et al.) (002]), model that predicts human-ratings of mean opinion scores (MOS) for quality and naturalness. To quantify speaker sim- ilarity we use the cosine similarity between speaker embeddings on concatenated X-vectors and ECAPA-TDNN embeddings. For the anonymization task, we use EER as a measure of privacy and WER as a measure of intelligibility, per the VPC protocol. To evaluate streaming performance, we measure latency on an NVIDIA RTX 500 Ada GPU and a dual-socket AMD EPYC 7543 CPU.
Model training. All modules are trained using the AdamW optimizer with an initial learning rate of 5 x 10-4 and a batch size of 16 (random 3-sec clips). The content encoder was optimized with ReduceLROnPP lat learning rate scheduler, whereas the waveform decoder employed the ExponentialLA scheduler with decay factor y = 0.999996. The encoder and waveform decoder comprised 37.5 M and 48.7M M trainable parameters, respectively. Both content encoder and the waveform decoder were trained independently for 500k steps on an NVIDIA RTX 5000 Ada GPU.
Baseline systems. We compare TVTSyn against four SOTA streaming methods: SLT24 Quamer & Outierrez-Osunal (202)), DarkStream (DS) (Quamer & Gutierrez-Osuna (202a)), and Ge (Cai et al., 2025). SLT24 is fully causal CNN based archiecture and uses bottleneck features as con- tent embedding. DS is a CNN-transformer based system that uses a 140ms lookahead and applies k-means clustering to the content embeddings. For GenVC, a LM based generative model, we eval- uated two configurations: Vemall and GenVC-large under the streaming decoding protocol, with the top-k parameter set to should be noted that the Gen VC encoder non-causal (i.e., it generates content features after consuming the entire source utterances), so it only streams at the decoder level. Thus, the two GenVC baselines are at an advantage when compared to the rest of the models (SLT24, DS, TVTSyn), which operate in a streaming fashion end-to-end, and therefore are affected by errors (i.e., mis- mnis-predicted future tokens) in the causal encoder. For our proposed TVTSyn model, evaluated the full configuration shown in Figure (P), along with three abla- tions: (-VQ) removing the compression-Ve VQ step, (-TVT) removing the TVT processing block, and (-VQ/-TVT) removing both.
"https//hugginnffaccccm

---

Published as a conference paper at ICLR 2026
5 RESULTS
5.1 CONTENT REPRESENTATION
Figure 3 shows I-SNE visualizations of the content embeddings, averaged across time at the utter- ance level. Panel (a) shows the continuous embeddings at the output of the content encoder, color- coded by speaker, markers indicating native vs. non-native speakers. We observe two broad clusters, separating native and non-native speakers, with tight sub-clusters for each individual speaker. This indicates that the continuous content embeddings retain significant speaker cues. Panel (b) shows the t-SNE plot of the logits representation, obtained by linearly projecting the continuous embeddings onto the codebook. While the logits representation still separates native from non non-native speakers, the within-speaker clusters are noticeably looser compared to the continuous embeddings. Panels (c) and (d) show t-SNE plots for the bottleneck and VQ bottleneck representations, respectively. In both cases, within-speaker clustering is further reduced compared to the logits embeddings, in- dicating that both bottleneck approaches substantially reduce speaker leakage. Though native non-native clusters are still separable in (c) and (d), this reflects the fact that non-native accents do alter the content of speech, both at the segmental level (e.g., phonetic substitutions) and the prosodic level (e.g., stress stress-timing in English vs. syllable-timil in Spanish and Mandarin).
(a)
Native
(b)
(0)
Non-native
Figure 3: t-SNE visualization of content embeddings, color-coded by speaker. Markers denote native or non-native (o). (a) Continuous embeddings, (b) logits, (c) bottleneck, and (d) VQ bottleneck.
5.2 TIME VARYING TIMBRE REPRESENTATION
Figure illustrates how the speaker-processing block yields content-synchronous, time-varying timbre. The content-GTM attention heatmap in panel (a) shows sparse islands and horizontal bands across tokens, indicating that different timbre facets are retrieved different moments while a few facets are reused across longer spans. The thin Top-1 raster (panel b) highlights discrete facet switch- ing aligned with phonetic/prosodic transitions. Panel (c) shows PCA trajectories for pre-Slerp em- bedding o and final embedding 8; along with the global timbre g. Here the 1 wander broadly, whereas 8, forms a compact, smooth tube around the global point. This shows that the interpolation path keeps identity geometry intact while allowing local movement. Finally, the GTM usage and geometry (panels d and e) show dispersed memory with non-collapsed token usage, suggesting that the model learned diverse set of reusable timbre facets.
Content- GIM attention
Timmbrri tnguctaryy
GTM tokana musage!
9.25
3.28
CTM toeen
8.18
3,05
PCL 08.75
[4] TO UOADS OON unteanTcn
(h) Top-1 ITM solan dan time
55
50
100
Time
150
200
220
300
20
20
CTM token
Figure 4: Qualitative analysis of time-varying timbre for the text: "Six spoons of fresh snow peas, five thick slabs of blue cheese, and maybe a snack for her brother Bob.". (a) Content-GTM attention map with (b) Top-1 strip shows content-dependent selection of timbre facets, (c) PCA trajectories (pre-slerp vs. final). (d) PCA projection of GTM value tokens (size & usage) and (e) token-usage histogram indicate diverse, non-collapsed facets.
7

---

Published as a conference paper at ICLR 2026
5.3 SPEAKER TRANSFER (VOICE CONVERSION)
Figure summarizes objective evaluation of VC performance across baselines (SLT24, DS, VenVC), the proposed model (P), and ablations (-VQ, -TVT, -VQ/-TVT). The proposed model achieves the strongest anonymization performance of models (i.e., lowest Sre-SIM and highest Trg-SIM), and the second highest NISQA scores, marginally lower than baseline SL.224 (4.01 vs. 3.91). As a reference, Figure 5 also includes scores for unmodified source speech (NISQA: 4.41). Note that the proposed model achieves the same Tig-SIM score (0.77) as that of -peiin-speaker comparisons in real speech (i.c., Sre-SIM for the reference source utterances), and similar Sre-SIM (0.48) that of between-speaker comparisons in real speech (i.e., Trg-SIM for the reference source utterances, 0.48). In other words, our proposed model generates speech that is as similar to the larget speaker as any two real utterances from that largel speaker, and as dissimilar from the source speaker as that between the source speaker and any other speaker.
The ablation study show that removing the TVT of VQ steps lead to significant reduction in NISQA scores (from 3.91 to 3,42/3.44, 44)) and that removal of both TVT and VQ steps degrades NISQA scores even further (3.179). However, neither TVT or VQ appear to play major role in anonymization performance. Finally, the two VC baselines achieve the weakest anonymization performance, with Trg- Trg-SIM scores far lower than those of the remaining models, and also modest NISQA scores despite Gen VC using non-causal content encoder.
0.80
TVT
-VVT
TVTSyn
0.75
.VQ/TVT
0.75
TVT
.V//VV
SLT24
SUT24
0.70
0.70
batter
better
GonVC
0.00
GunVC-4
0.00
GenVC
Source
Source
3.90
5.54
2,98
5:30
3.26
(r)
NISQA(1)
Figure 5: Objective evaluation results for voice conversion. Sre-SIM: cosine similarity b/w VC and source speaker; Trg-SIM: cosine similarity b/w VC and target speaker, NISQA-MOS: Speech Quality and Naturalness Assessment. Sre-SIM and Trg-SIM for source speech (i.e., unaltered) re- flect within- and between-speaker similarity, respectively.
Table I shows the contribution of each component in the TVT processing block. Notably, Sre-SIM remains constant at 0.48 across all ablations, indicating that privacy is determined by the overall architecture rather than the fine-grained TVT design and that TVT's primary role is preserving synthesis quality while maintaining anonymization. We observe that removing GTM produces the largest drop in acoustic quality (3.91 to 3.45), indicating that the content-synchronouu timbre is essential for naturalness. Removing learnable priors also causes notable degradation (3.91 to 3.62), as the model loses universal timbre bases that enable efficient generalization. Replacing Slerp with linear interpolation and replacing gating with fixed a 0.5 produce modest but measurable degradations, validating these design choices. The same is observed when reducing GTM capacity from 48 24 tokens or 12 tokens, confirming our choice of 48 tokens balances expressivity and efficiency.
Table 1: Ablation study of components in the TVT speaker processing block. All ablations maintain similar privacy, while synthesis quality degrades without key components.
Models
Src-SIM (1)
Tgt-SIM (f)
NISQA MOS (1)
Proposed
TVTSyn (48 GTM tokens)
0.47
0.77
3.91
TVT ablations
w/o gating
0.48
0.76
3.80
slerp
0.48
3.75
w/o GTM
0.48
0.75
3.45
w/o prior
0.48
0.75
3.62
w/ 24 GTM tokens
0.48
0.76
3.81
w/ 12 GTM tokens
0.48
0.75
3.73
8

---

Published as a conference paper at ICLR 2026
We corroborated the objective results in Figure with perceptual listening tests Amazon Me- chanical Turk (N=20 To evaluated speaker transfer, we used standard ABX protocol, where par- ticipants had to select whether utterance X (voice conversion) sounded closer to A B (source and target speaker, counterbalanced), and also reported their confidence on a 1-7 scale (7 = extremely confident; 1: not confident at all). For speech quality, listeners provided mean-opinion scores for individual utterances. Results are summarized in T The proposed system achieved the highest MOS across all models, and only marginally below the perceived speech quality of unedited source utterances. For speaker transfer, the proposed system also achieved the highest verifiability rate (74.33%) of all models with a high confidence rate (5.02).
Table Human listening test (N=20). Mean opinion score (MOS) for audio quality (95% confidence interval) and speaker verifiability scores.
Source
S1.T24
DS
GenVCss
TVTSyn
3.77
3.49 0.13
3.63 0.11
3.82 0.10
Prefer Target Speaker
68.00%
69.33%
70.67% 5.04
74.33%
Avg. Confidence Rating
5.06
4,99
5,02
5.4 SPEAKER ANONYMIZATION
Following the VPC'24 protocol (Tomashenko et al. 2024), we computed EERs under two attacker models: lazy-informed attacker (knows the algorithm but has no enrollment data) and a stronger semi-informed attacker (retrains ASV models using anonymized enrollment). Higher EER indicates better anonymization. We also report Word Error Rate (WER) for intelligibility and Unweighted Average Recall (UAR) for emotion characteristics. TVTSyn achieves favorable privacy-utility bal- ance: strong anonymization (EER = 47.6% lazy, 14.6% semi), excellent intelligibility (WER 5.35%), and intentional emotion suppression (UAR 37.32%) for enhanced privacy see Table 3 Notably, TVTSyn outperforms all streaming baselines on utility (WER: 5.35% vs. SLT24's 5.70%, DarkStream's 's 10.80%) while maintaining competitive privacy.
Comparison with VP Offline Systems. Table includes VPC' baselines (B2-B6) and top participants (T10-C3, T9, T8-4, T38-M1) for context; however, direct comparison is problematic due to fundamentally different constraints. VPC systems operate offline with full-utterance bidirectional context and diverse pseudo-speaker generation (e.g., GAN-based sampling). whereas TVTSyn op- erates causally with <80ms latency using 28 fixed pseudo-speakers a simplification for this study; future work will incorporate pseudo-speaker generation. Additionally, design goals differ: VPC participants optimize for emotion preservation (UAR: 60-65%), whereas TVTSyn targets emotion suppression (UAR 37.32%), reflecting comprehensive identity masking including paralinguistic traits. For our privacy- objective, lower UAR is desirable.
Our contribution not surpassing offline systems, but demonstrating that streaming anonymiza- tion under strict latency constraints can maintain strong privacy and andres-addrsssingg real-world deployment needs (teleconferencing, live translation) where offline processing is infeasible.
Table 3: VPC'24 evaluation: WER, EER and UAR as measure of intelligibility, anonymization strength and emotion preservation respectively.
Models
WER (U)
EER (lazy-Ifformed, ?)
EER (semi-Informed, ?) UAR (emotion)
Baselines
SLT24
5.70
31.40
10.12
$7.00
10.80
49,09
20.83
34.49
GenVC-s
8.20
48.48
15.94
34.23
VPC24 Baselines
B2
10.20
5.99
54.55
B3
4.35
26.28
37.83
B4
6.02
31.49
42.37
B5
4.55
34.35
38.12
9.39
36.25
YPC24 Participants
T10-C3 Yao et
3024
2.62
37.34
65.23
T9 Tun at
2.35
34.27
60.82
T8.4 Kinilah er al
2024
3.75
48.25
30.35
Blucch el
32.23
Proposed
TVTSya
5.35
47.55
14.57
37.32
We acknowledge that large-scale listening may still he required for comprehensive validation.

---

Published as a conference paper at ICLR 2026
5.5 REAL-TIME PERFORMANCE
We report synthesis latency and real-time factor (RTF) for all models using chunk sizes of 60 ms and 100 ms. As shown in Table 4 TVTSyn achieves latencies of #:99 ms GPU and sv132 on CPU, with RTFs of 0.31 and 1. 1.20, respectively, comfortably within real-time bounds. Compared to SLT24 and DS, our models reduce both latency and RTF while maintaining comparable quality. Importantly, DS requires a 140 ms lookahead at the encoder, effectively raising end-to-end latency well above the reported figure. In contrast, our models operate fully causally at runtime without lookahead, making them substantially more suitable for low-latency deployment. While results are reported for 60 ms and 100 ms chunk size, the architecture scales robustly to smaller or larger chunks with consistent real-time performance, ensuring flexibility across diverse streaming scenarios. We did not include real-time time results for Gen VC, since its encoder is non-causal and streams only at the decoder level, making latency comparisons with fully causal systems unfair.
Table 4: Latency and RTF on CPU and GPU for proposed and streaming baselines.
Chunk size 60 ms
Chunk size 100 ms
Model
CPU
GPU
CPU
GPU
Latency (ms))
RTF
Latency (ms)
RTF
Latency (ms)
RTF
Latency (ms)
RTF
SLT24
187.11
2.119
86.49
0.441
244.31
1.443
123.55
0,236
DarkStream (DS)
127.02
1.117
76.12
0.269
172.45
0.724
119.12
0,191
TVTSyn
131.76
78.51
0.862
119.77
DISCUSSION
We proposed TVTSyn, streaming model for voice conversion and speaker anonymization that uses time-varying representation of speaker timbre to match the temporal granularity of linguistic content, enabling better trade-off between naturalness, speaker similarity, and privacy under strict latency constraints. Our results show that the privacy-utilipy balance is fundamentally architectural: by resolving the mismatch between static speaker embeddings and dynamic content, TVT preserves expressivity without weakening anonymization. The factorized VQ bottleneck further regularizes content, improving intelligibility while maintaining anonymizaiton performance. Ablations rein- force this view: removing either TVT or VQ degrades quality and VC performance. Together, these findings show that controlling the interaction between timbre variability and VQ strength is the key to managing the privacy-utility depending on deployment needs.
utility trade-off, with ablation models allowing control of utility or privacy
Future work will make this alignment more structured and controllable. One direction is to explic- itly disentangle static traits (e.g., accent, age, sex) from dynamic attributes(e.g. emotion, speak- ing style). We envision a two-path design: "static" pathway to modify age/sex, and a frame- synchronous "style" pathway driven by TVT. This setup would enable controllable anonymization, e.g., selectively masking emotion or sex while preserving intelligibility. A second direction is to extend the Global Timbre Memory to include timbre+style facets, exposing simple controls for emo- tion and expressivity. In this setting, anonymization could be combined with behavioral camouflage strategies prosody prosoody shaping, hesitations, or filler words torrs-thhat break habitual patterns without increas- ing WERs. A third direction is to broaden the scope cross-lingual and code-switching speech, where accent interacts with phonotactic variation; here, a lightweight language-ID head could help stabilize TVT across language switches. We will explore adaptive chunking and hierarchical KV deployment methods such as quantization
caches to further reduce latency, alongside and low-rank adapters, enabling robust CPU-only real-time operation. Finally, robustness to noisy and reverberant acoustic environments can be enhanced through data augmentation techniques dur- ing training, such as additive noise, reverberation simulation, and codec artifacts, as demonstrated in recent work on noise-robust speech synthesis (Ranjan et al. 2224 Lakshminarayana al. 2025). as systematic evaluation under acoustic degradation remains an important direction for real real-world deployment.
ACKNOWLEDGMENTS
Supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of In- terior/Interior Business Center (DOI/IBC) contract number 140D0424C0066. The U.S. Government
10

---

Published as a conference paper at ICLR 2026
is authorized reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies endorse- ments, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.
REFERENCES
Zexin Cai, Henry Li Xinyuan, Ashi Garg, Leibny Paola Garcfa-Perera, Kevin Duh, Sanjeev Khu- danpur, Matthew Wiesner, and Nicholas Andrews. Genve: Self-supervised zero-shot voice con- version. arXiv preprint arXiv:2502.04511 2025.
Chak Ho Chan, Kaizhi Qian, Yang Zhang. and Mark Hasegawa-Johnson. Speechsplit2. 0: Unsu- pervised speech disentanglement for voice conversion without tuning autoencoder bottlenecks. In Proc. ICASSP, pp. 6332-6336, 2022.
Wei Chen, Binzhu Sha, Dan Luo, Jing Yang, Zhuo Wang, Fan Fan, and Zhiyong Wu. Dafmmsve: One- shot singing voice conversion with dual attention mechanism and flow matching, arXiv preprint MrXiv:2508.05978 05978, 2025.
Yen-Hao Chen, Da-Yi Wu, Tsung-Han Wu, and Hung-yi Lee. Again-ve: A one-shot voice version using activation guidance and adaptive instance normalization. In Proc. ICASSP, pp- 5954-5958, 2021.
Yuanzhe Chen, Ming Tu, Tang Li, Xin Li, Qiuqiang Kong, Jiaxin Li, Zhichao Wang, Qiao Tian, Yup- ing Wang, and Yuxuan Wang. Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance. In Proc. ICASSP, pp- 1-5, 2023.
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized channel at- tention, propagation and aggregation tdnn based speaker verification. In Proc. INTERSPEECHI pp. 3830-3834, 2020.
Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, et al. Hierarchical generative modeling for controllable speech synthesis. arXiv preprint arXiv:1810.07217 2018.
Wen-Chin Huang, Tomoki Hayashi, Shinji Watanabe, and Tomoki Toda. The sequence-to-sequene baseline for the voice conversion challenge 2020: Cascading asr and tts. arXiv preprint arXiv:2010.02434, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proc. ICCV. pp. 1501-1510, 2017.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchho Yang. Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: zero-shot speech synthesis with factor- ized codec and diffusion models. In Proceedings of the 41st International Conference Machine Learning, pp. 22605-22623, 2024.
John Kominck. Cmu arctic databases for speech synthesis. CMU-LTI, 2003.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36:27980-279933,, 2023.
Kishor Kayyar Lakshminarayana, Frank Zalkow, Christian Dittmar, Nicola Pia, and Emanuël AP Habets. Low-resource text-to-speech synthesis using noise-augmented training of forwardta- cotron. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Olivier Le Blouch, Rayane Bakari, and Nicolas Gengembre. Orange shiva system description for the voice privacy challenge 2024.
Jingyi Li, Weiping Tu, and Li Xiao. Freeve: Towards high-quality text-free one-shot voice version. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
11

---

Published as a conference paper at ICLR 2026
Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, and Helen Meng. Any- to-many voice conversion with location-relative TASLP, 29:1717-1728, 2021.
modeling. IEEE/ACM
Philipos C Loizou. Speech quality assessment, pp. 623-654. Springer, 2011.
Jaime Lorenzo- Trueba, Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio, Tomi Kinnumen, Zhenhua Ling, al. The voice conversion challenge 2018: Database and sults. sound]. The Centre for Speech Technology Research, The University of Edinburgh, UK. https://doi. org/10. 0...0.87///////3.3... 2018.
Stephen Edward McAdams. Spectral fusion, spectral parsing and the formation of auditory images. Stanford university, 1984.
Sarina Meyer, Florian Lux, Pavel Denisov, Julia Koch, Pascal Tilli, and Ngoc Thang Vu. Speaker Anonymization with Phonetic Intermediate Representations. In Proc. Interspeech 2022, pp. 4925 4929, 2022. doi: 10.21433//merrspechh22222 10703.
Sarina Meyer, Pascal Tilli, Pavel Denisov, Florian Lux, Julia Koch, and Ngoc Thang Vu. Anonymiz- ing speech with generative adversarial networks to preserve speaker privacy. In 2022 IEEE SLT Workshop, pp. 912-919, 2023.
Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. Nisqa: A deep enn-self- attention model for multidimensional speech quality prediction with crowdsourced datasets. arXiv preprint urXiv:2104.09494 2021.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: A Large-Scale Speaker Iden- tification Dataset. In Proc. Interspeech 2017. ISCA, 2017.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015. doi: 10.1 10.1109/ICASSP2005.
7178964.
Jose Patino, Natalia Tomashenko, Massimiliano Todisco, Andreas Nautsch, and Nicholas Evans. Speaker Anonymisation Using the McAdams Coefficient. In Proc. Interspeech, pp. 1099-1103, 2021.
Juan M Corooo-Codosero, Fernando M Espinoza-Cuadren and Luis A Bernández-Góómez X-vector anonymization using autoencoders and adversarial training for preserving speech privacy. Com- puter Speech & Language, 74:101351, 2022.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proc. AAAI, volume 32, 2018.
Waris Quamer and Ricardo Gutierrez- Osuna. End-to-end streaming model for low-latency speech anonymization. In 2024 IEEE Spoken Language Technology Workshop (SI.7), pp. 727-734, 2024. doi: 10.1 10833333
Waris Quamer and Ricardo Gutierrez-Osuna, Darkstream: real-time speech anonymization with low latency. arXiv preprint arXiv:2509.0466, 2025a.
Waris Quamer and Ricardo Gutierrez- Disentangling segmental and prosodic factors to non- native speech comprehensibility. IEEE Transactions on Audio, Speech and Language Processing, 2025b.
Waris Quamer, Anurag Das, and Ricardo Gutierrez-Osuna. Decoupling Segmental and Prosodic Cues of Non-native Speech through Vector Quantization. In Proc. INTERSPEECH, pp. 2083- 2087, 2023.
Sumit Ranjan, Rupayan Chakraborty, and Sunil Kumar Kopparapu. Reinforcement learning based data augmentation for noise robust speech emotion recognition. Proc. INTERSPEECH, Kos Is- land, Greece, 2024.
12

---

Published as a conference paper at ICLR 2026
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al. Speechbrain: A general-purposs speech toolkit. arXiv preprint arXtv:2106.04624 2021.
Konstantine Sadov, Matthew Hutter, and Asara Near. Low-latency real-time voice conversion on cpu. arXiv preprint arXiv:231..0087 2023.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th amnual conference on Computer graphics and interactive techniques, pp. 245-254, 1985.
David Snyder, Daniel Garcia-Romeny Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X- vectors: Robust dnn embeddings for speaker recognition. Proc. ICASSP, pp. 5329-5333, 2018.
Brij Mohan Lal Srivastava, N. Tomashenko, Xin Wang, Emmanuel Vincent, Junichi Yamagishi, Mohamed Maouche, Aurélien Bellet, and Marc Tommasi. Design Choices for X-Vector Based Speaker Anonymization. In Proc. Interspeech, pp. 1713-1717 2020.
Tao Tan, Shutao Liu, Yibo Duan, Sheng Zhao, and Xi Shao. System description: Speaker anonymization system with sentiment transfer and feature interpolation. voiceprivacychallenge. org. 2024.
Lauri Tavi, Tomi Kinnunen, and Rosa González Hautamäki. Improving speaker de-identification with functional data analysis of 10 trajectories. Speech Communication, 140:1-10, 2022.
Natalia Tomashenko, Brij Mohan Lal Srivastava, Xin Wang, Emmanuel Vincent, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Jose Patino, Jean-François Bonastre, Paul-Gauthie Noć, et al. Introducing the voiceprivacy initiative. In Proc. INTERSPEECH, pp. 1693-1697, 2020.
Natalia Tomashenko, Xiaoxiao Miao, Pierre Champion, Sarina Meyer, Xin Wang, Emmanuel Vin- cent, Michele Panariello, Nicholas Evans, Junichi Yamagishi, and Massimiliano Todisco. The VoicePrivacy 2024 challenge evaluation plan. 2024.
Ismail Rasim Ulgen, Carlos Busso, John HL Hansen, and Berrak Sisman. We need varia- tions in speech synthesis: Sub-center modelling for speaker embeddings. arXiv preprint arXiv:2407.0429. 2024.
Benjamin van Niekerk, Marc-André Carbonneau, Julian Zaïdi, Matthew Baas, Hugo Seuté, and Herman Kamper. A comparison of discrete and soft speech units for improved voice conversion. In Proc. ICASSP, pp. 6562-6566, 2022.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, al. Cstr vetk corpus: English multi- speaker corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 6:15, 2017.
Disong Wang, Liqun Deng. Yu Ting Yeung. Xiao Chen, Xunying Liu, and Helen Meng. VQMIVC: Vector Quantization and Mutual Information-Bann Based Unsupervised Speech Representation Disen- tanglement for One-Shot Voice Conversion. Proc. Interspeech, pp. 1344-1348. 2021.
Mirjam Wester. The emime bilingual database. Technical report, The University of Edinburgh, 2010.
Henry Li Xinyuan, Zexin Cai, Ashi Garg, Kevin Duh, Leibny Paola Garefa-Perera Sanjeev Khu- danpur, Nicholas Andrews, and Matthew Wiesner. Hitcoe jhu submission to the voice privacy challenge 2024. arXiv preprint arXiv:2409.089) 2024.
Haoquan Yang, Liqun Deng, Yu Ting Yeung, Nianzu Zheng, and Yong Xu. Streamable speech rep- resentation disentanglement and multi-level prosody modeling for live one-shot voice conversion. In Proc. INTERSPEECH, pp. 2578-2588 2022.
Jixun Yao. Nikita Kuzmin, Qing Wang, Pengeheng Guo, Ziqian Ning, Dake Guo, Kong Aik Lee, Eng-Siong Chng, and Lei Xie. Npu-ntu system for voice privacy 2024 challenge. arXiv preprint 20Xi::4409.04173 2024.
13

---

Published as a conference paper at ICLR 2026
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech, 2019. URL
org/abs/ 1904. 02882
Yu Zhang, Baotong Tian, and Zhiyao Duan. Conan: A chunkwise online network for zero-shot adaptive voice conversion. arXiv preprint arXiv:2507.45533 145344, 2025.
Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Hutilarin-Hullliinnnn John Levis, and Ricardo Gutierrez-Osuna. L2-arctic: A non-native english speech corpus. In Proc. Interspeech, pp- 2783-2787, 2018. doi: 10. 10.21437//ntesspeechh wat.ch..0181111 1110. URL http://dx.
doi. doi.org//0.... wwt22114//nntrspeeeeee..
201 111110
Zhizhou Zhong. Yuxi Mi, Yuge Huang, Jianqing Xu, Guodong Mu, Shouhong Ding, Jingyun Zhang, Rizen Guo, Yunsheng Wu, and Shuigeng Zhou. Slerpface: Face template protection via spherical linear interpolation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39,
pp. 10698-10706, 2025.
A ARCHITECTURE DETAILS
A.1 MODEL CONFIGURATIONS
Our system comprises four main modules: (i) causal content encoder with factorized VQ bottle- neck, (ii) speaker processing block that produces time-varying timbre (TVT) via a Global Timbre Memory (GTM) with gate and Slerp, (iii) a causal context layer (MHSA) for frame-level condition- ing, and (iv) causal waveform decoder (SEANet). The high-level dataflow matches the training and inference diagrams (content/VQ. GTM+gate+ Slerp, and cLN with Fusion) shown in Fig. and
Signal scale and rates. Audio is 16 kHz; frames are 50 Hz (20 ms hop). All intermediate features (content, TVT, prosody) are aligned at the frame clock.
Content encoder. A lightweight causal SEANet stack maps audio to 512-D) frame embeddings using strides [8,5, 5, 4, (overall x320) with ELU activations, kernel sizes 7/3/3, dilation base 2, and true-skip connections. A causal 8-layer MHSA context block (5mode==12, 8 heads, RoPE positional encoding, FFN 2048) provides a 2s look-back (context=100 frames). During training only, masked right context of 4 frames (== 80 ms) is enabled stabilize learning; inference, right context is limited by chunk-size, i.e., applies within chunk (chunk-wise causal).
Factorized VQ bottleneck. The 512-D content embedding is projected 8-1) latent and quan- tized with a 4096-entry codebook (codebook, _dim = 8, codebook = 4096); commitment loss 0.15 with L2 code normalization. The quantized latent then projected back to 512-D and passed forward. This reduces speaker leakage in content while preserving lexical information.
Speaker processing (TVT). Following the diagram, global speaker vector is expanded GTM; content frames attend (atfention_dim- = 128) to GTM to retrieve a facet, gate a(t) E [0, 1] modulates deviation from the global vector, and Slerp global, facet; att)) yields the time-varying timbre embedding. These TVT features condition decoding via cLN with Fusion face dimensions follow the config (global_fimbl fimbreedimenssoon 704,
9 The TVT inter- 192).
Context layer for synthesis. A causal frame-rate synthesizer (2s look-back, no right context) refines 512-D content features and fuses TVT and prosody. It uses 8-head MHSA with ROPE, FFN 2048.
Waveform decoder. causal SEANet vocoder mirrors the encoder's strides [2, 2,4,,,, back to waveform, using ELU activations, dilation base 2, and weight normalization. Conditioning is in- jected via cLN with Fusion at multiple stages (global content normalization plus affine modulation from TVT and prosody).
See the training and detailed TVT/conniitoning diagram for reference. TVT attention, gate, and Slerp; and cl.N with Fusion illustrated in Figure
14

---

Published as a conference paper at ICLR 2026
Table 5: Core hyperparameters.
Item
Setting
Sample/ frame rate
16,000 Hz/ 50 Hz (20 ms hop)
Content channels / dim
d=512
SEANet strides
Encoder [8, 2, Decoder 2,4,5
Activations/ norm
ELU; encoder norm=wei norm=wei.gh._nnms ght_norm, decoder weight_norm
Transformer (enc)
layers, heads, mode/==512 FFN 2048, RoPE,
Context window
100 frames (2 look- back- right context=4
Transformer (frame synth)
causal. 8 heads. d model=512, FFN 2048, RoPE, right context=0
VQ bottleneck
[C|44096, code dim 8, commitment 0.15, L2 normalize timbre dimeensoo4m,,,, fimbre_cond_ding
TVT fusion dims Prosody
attention 1im==192
FO/Energy predictors (causal), fused via cLN with Fusion
A.2 STREAMING IMPLEMENTATION
The system is designed for causal, low-latency streaming synthesis. All modules are adapted to operate on fixed-length chunks with persistent states across calls.
Causal Convolutions. Convolutions in the encoder and decoder are wrapped with ring-buffer state management. At each step. only the newest samples are convolved, while cached activations from previous chunks are reused. This avoids redundant computation and ensures causality.
Streaming Attention. Transformer blocks maintain rolling key-value cache. During inference, only the most recent seconds of context are retained, with a limited 4-frame future peek (~~00 ms). The cache updated incremenaally per chunk, allowing efficient streaming inference without reprocessing past context.
Prosody Predictors.
Pitch and energy predictors are causal CNNs that operate chunk-wise. Their states maintained so that predictions are consistent across chunk boundaries.
Table 6 summarizes the main streaming parameters.
Parameter
Value
Chunk size
60 ms (default, tested 20-140 ms) 220-sample hop, causal ring buffer look-back, 80 ms look-ahead 20 ms overlap
Encoder buffer
Attention context
Decoder overlap-add
Streaming states
Encoder, Transformer KV cache, Decoder buffers
Table 6: Summary of streaming implementation.
B EVALUATION
B.1 LATENCY MEASUREMENTS
Latency is defined as the sum of the chunk size and the processing time chunk (in milliseconds), averaged over all chunks within utterance. The real-time factor (RTF) computed as the ratio of processing time to chunk size for each chunk, then averaged across all chunks of the same utterance. To avoid initialization overhead, we first performed warm-up on utterances. Latency and RTF were then measured 100 utterances, and report the mean values across samples.
B.2 PERCEPTUAL LISTENING TESTS
We conducted two subjective evaluations with 20 unique participants for each test. All listeners based in the United States at the time of recruitment. The study protocol approved by the Institutional Review Board of Texas A&M University and deployed through Amazon Mechanical Turk.
15

---

Published as a conference paper at ICLR 2026
B.2.1 MEAN OPINION SCORE (MOS)
To assess perceived quality, participants rated individual utterances a five-point mean opinion score (MOS) scale, where higher ratings correspond to more natural speech and lower distortion. The scale definitions are provided in Table 7 Prior to evaluation, listeners were given example clips representing each score to help them calibrate their judgments Loizou (201T). These calibra- tion samples were drawn from the 2018 Voice Conversion Challenge dataset Lorenzo-Truebe et al. 2018). Each participant evaluated 15 utterances system.
Table 7: Mean Opinion Score rating scale.
Rating
Speech Quality
Distortion Level
5
Excellent
Imperceptible
4
Good
Slight but not distracting
3
Fair
Noticeable but tolerable
2
Poor
Distracting but intelligible
1
Bad
Very distracting unnatural
B.2.2 SPEAKER VERIFIABILITY TEST
To measure speaker similarity, we conducted an ABX test. Each trial presented three recordings (X, A, B). Listeners judged whether or B sounded more like X, then rated their confidence on seven-point scale (7: extremely confident; 5: quite bit confident; 3: somewhat confident; 1: not confident all). Each participant evaluated randomly sampled voice-converted utterances from every system. To ensure judgments reflected voice characteristics rather than lexical overlap, the source and target recordings (A/B) contained different content than the converted sample.
C LLM USAGE
Large language models were employed only to assist with writing tasks, such as improving clar- ity, grammar, and style. No experimental design, data analysis, result interpretation relied automated tools.
16


Reviews:
[{"review_id": "fsQgG5rdwE", "reviewer": "Reviewer_he7Q", "summary": "The paper introduces TVTSyn, a low-latency voice conversion and anonymization system that replaces the traditional static speaker embedding (x-vectors) with a content-synchronous, time-varying timbre (TVT) representation. The method uses a Global Timbre Memory (GTM), gated interpolation (Slerp), and a factorized VQ bottleneck to align the temporal granularity of speaker identity with linguistic content that are fed into a streamable speech synthesizer for generating anonymised audio. The system achieves <80 ms latency on GPU, ~132ms on CPU, shows improved naturalness, and provides a better privacy-utility trade-off than prior streaming baselines (SLT24, DarkStream, GenVC). Experiments follow the VoicePrivacy Challenge 2024 protocol, reporting reasonable performance in source-target speaker similarity, EER, WER, and perceptual quality metrics. EER in semi-informed case is significantly lower than DarkStream which is not elaborated further. It is potentially due to the content embeddings leaking speaker information. I doubt the claim that content embeddings are speaker-independent and needs to be qualified properly.", "questions": "1. Could the authors provide a quantitative ablation showing how the gating parameter $\\alpha_t$ or the number of timbre facets $K$ affects privacy (EER) and quality (NISQA) ?\n2. Is the Global Timbre Memory fixed per speaker or updated during fine-tuning ? Would dynamic adaptation hurt anonymity ?\n3. How robust is this technique to noisy or reverberant inputs ? does the causal encoder maintain intelligibility under such conditions ?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Authors identify a fundamental weakness (static speaker embedding) in the current speaker anonymisation techniques and proposes a well-justified fix via content-synchronous conditioning of the speaker embeddings\n- Well-founded experiments on VPC 2024 protocol and ablations confirm benefits across privacy, quality, and latency\n- Deployment conditions are kept in mind by demonstrating real-time performance on CPU/GPU under tight latency budgets (<80 ms), relevant for interactive applications\n- Clear figures, well-written text, and reproducible evaluation settings\n- Concrete future directions are presented that extend the technique significantly", "weaknesses": "- The performance of B1 baseline is mentioned during the analysis but not added to Table 2 for clear comparison\n- The gating and Slerp mechanisms are intuitively motivated but not analyzed quantitatively (e.g., contribution to expressivity or privacy).\n- Listening tests use a small Mechanical Turk sample (N = 20) without statistical significance analysis or demographic breakdown. A larger cohort of listeners must be recruited (>100) and carefully selected to include demographic variations (age, gender, native/non-native, etc.)\n- Authors claim that the content embeddings are speaker-independent and show it through t-SNE plots but do not quantify it through metrics. The claim of speaker-independence needs to be properly verified. One option is to classify speakers directly through content embeddings which might reveal how much speaker information is leaking through them as performed in this paper: https://petsymposium.org/popets/2023/popets-2023-0007.php", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yZMwfTvtIN", "reviewer": "Reviewer_VGqB", "summary": "The author proposed a streamable speech synthesizer, TVTSyn, for voice conversion and anonymization. \nThe model resolved a mismtach in prior work that linguistic content is time-varying but speaker identity is a static vector. The proposed model aligns the temporal granularity of identity and content relying on TVT, time-varying timbre representation. It also uses a speech content encoder encoder to extract feature that removes residual speaker information and regularize the content space with a factorized vector-quantized bottleneck.\nThe experiment shows that the model behaves strong compared to multiple baselines and the system is causal end-to-end with <80ms on GPU and <132ms on CPU, which is considered within real-time bounds. A comprehensive study on content and TVT representation and ablation study is also provided. \nOverall, the work has good vision and provide moderate novelty on model architecture and representations to resolve the mismatch between static speaker identity embedding and time-varying timbre representation. The strong empirical results show the effectiveness on model performance including speech naturalness, anonymization and system latency. The limitation of the work is that dataset is english only, evaluation samples size(N=20 for MOS) are limited and the theoretical analysis of the TVT representation could be better discussed.", "questions": "Could you provide more insights on the design of Factorized VQ and gating/interpolation?\n\nFor example, in Section 3.2, you mention that slerp interpolation “respects the hyper-spherical geometry of the embedding space, ensuring smooth trajectories and preserving angular distances” which helps maintain “identity geometry”. Could you clarify or provide theoretical evidence for this claim? \nAlso in section 5.2 (c) it shows the difference of before and after applying slerp. How sensitive is the output timbre to the choice of interpolation method (Slerp vs. linear)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The problem is well framed and it shows good intuition on the mismatch between dynamic input and static speaker embedding. The solution is intuitive by introducing a timbre representation that contains better temporal information.\n2. The overall system is well designed and end-to-end streamable. The proposed content encoder introduces a learnable bottleneck with factorized vector-quantization(VQ) that learns discrete, speaker-independent units while preserving linguistic fidelity. Also the time varying timbre representation is consists of a global timbre memory (GTM) that allowing content embedding to attend over the keys to retrieve weighted component using attention, and a combination of gating, interpolation to balance stability and flexibility.\n3. The author provided a comprehensive evaluation of the Voice Conversion and Speaker Anonimyzation, and latency analysis.  Also it shows the system is 79 ms latency on GPU and around 132 ms on CPU, achieving a real-time.\n4. Good analysis on content representation with tSNE visualization, showing the effectiveness of VQ bottleneck. Good representation on ablation study showing the removal of TVT and VQ causes degradation.", "weaknesses": "1.The discussion for the design of gating/interpolation are mostly based on intuition and empirical results. It would be good to include more theoretical analysis. These innovations on TVT representation and the usage of gating/slerp interpolation are more at a level of improving on top of existing architectures. Yet they are proved effective from experiment results.\n2. The MOS tests are based on 20 samples which is limited and may cause bias", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RauX47wDCj", "reviewer": "Reviewer_1kxu", "summary": "This paper state the static-dynamic mismatch between fixed speaker embeddings and time-varying linguistic content. The proposed TVTSyn model introduces a content-synchronized time-varying timbre representation, complemented by a factorized VQ bottleneck, to balance low latency, naturalness, and privacy.", "questions": "See weaknesses", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "This paper is well-structured and provide analysis on the effcient of the proposed system.", "weaknesses": "- The paper’s central claim resolving the static-dynamic mismatch via time-varying timbre is not sufficiently novel. Prior work has already explored dynamic speaker conditioning for speech synthesis.\n- The dataset employed in experiments is not convince and popular, which make me confuse the correctness of the conclusion.\n- Incomplete baseline comparisons with voice privacy challenge baseline systems or other popular speaker anonymization systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)