Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
Authors: Xueyao Zhang
Xiaohui Zhang
Kainan Peng
Zhenyu Tang
Vimal Manohar
Yingru Liu
Jeff Hwang
Dangna Li
Yuhao Wang
Julian Chan
Yuan Huang
Zhizheng Wu
Mingbo Ma
Venue: ICLR 2025 Poster (2025)
Keywords: controllable speech generation
speech disentanglement
voice conversion
accent conversion
text to speech
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a versatile zero-shot voice imitation framework, with controllable timbre and style.

Abstract: The imitation of voice, targeted on specific speech attributes such as timbre and speaking style, is crucial in speech generation. However, existing methods rely heavily on annotated data, and struggle with effectively disentangling timbre and style, leading to challenges in achieving controllable generation, especially in zero-shot scenarios. To address these issues, we propose Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. Vevo operates in two core stages: (1) Content-Style Modeling: Given either text or speech's content tokens as input, we utilize an autoregressive transformer to generate the content-style tokens, which is prompted by a style reference; (2) Acoustic Modeling: Given the content-style tokens as input, we employ a flow-matching transformer to produce acoustic representations, which is prompted by a timbre reference. To obtain the content and content-style tokens of speech, we design a fully self-supervised approach that progressively decouples the timbre, style, and linguistic content of speech. Specifically, we adopt VQ-VAE as the tokenizer for the continuous hidden features of HuBERT. We treat the vocabulary size of the VQ-VAE codebook as the information bottleneck, and adjust it carefully to obtain the disentangled speech representations. Solely self-supervised trained on 60K hours of audiobook speech data, without any fine-tuning on style-specific corpora, Vevo matches or surpasses existing methods in accent and emotion conversion tasks. Additionally, Vevo’s effectiveness in zero-shot voice conversion and text-to-speech tasks further demonstrates its strong generalization and versatility. Audio samples are available at https://versavoice.github.io/.

Published as a conference paper at ICLR 2025
VEVO: CONTROLLABLE ZERO-SHOT VOICE IMITA- TION WITH SELF-SUPERVISER DISENTANGLEMENT
Xueyao Zhang' Xiaohui Zhang?
Kainan Peng2
Zhenyu Tang
Vimal Manohar',
Yingru Liu Jeff Hwang Dangna LI
Yuhao Wang²
Julian Chan?
Yuan Huang?
Zhizheng Wu Mingbo Ma²
The Chinese University of Hong Kong, Shenzhen
Meta AI
ABSTRACT
The imitation of voice, targeted on specific speech attributes such timbre and speaking style, erucial in speech generation. However, existing methods rely heavily on annotated data, and struggle with effectively disentangling timbre and style, leading to challenges in achieving controllable generation, especially in zero-shot scenarios. To address these issues, propose Vevo, versatile zero- shot voice imitation framework with controllable timbre and style. Vevo operates in two core stages: (I) Content-Style Modeling: Given either text or speech's s content tokens as input, we utilize an autoregressive transformer to generate the content-style tokens, which is prompted by a style reference; (2) Acoustic Mod- eling: Given the content-style tokens as input, we employ flo-mmtchhing trans- former to produce acoustic representations, which is prompted by a timbre ref- erence. To obtain the content and content-style tokens of speech, we design a fully self-supervised approach that progressively decouples the timbre, style, and linguistic content of speech. Specifically, we adopt VQ-VAE [I] as the tokenizer for the continuous hidden features of HuBERT [2]. We treat the vocabulary size of the VQ-VAE codebook as the information bottleneck, and adjust it carefully to obtain the disentangled speech representations. Solely self-supervised trained 60K hours of audiobook speech data, without any fine-tuning style-specific corpora, Vevo matches or surpasses existing methods in accent and emotion con- version tasks. Additionally, Vevo's effectiveness in zero-shot voice conversion and text-to-speech tasks further demonstrates its strong generalization and versatility.
INTRODUCTION
The imitation of voice has long been an important issue in the field of speech generation. This includes the imitation of speaker identity 3 4. the imitation of speaking style such as accent 5 6
or emotion 77), and a broader concept of voice cloning such as in zero-shot text-to-speech (TTS)
task [8). These techniques have wide range of applications, including spoken language learning
6, voice anonymization [0], voice assistants
12]. and video dubbing
To achieve targeted and controllable imitation various speech attributes, many studies focuses on factorizing speech into multiple sub-spaces [4
15 [6 [7. In this work, we follow this idea and decompose speech into three key attributes: linguistic content (what speak), style (how to speak), and timbre (who speaks). Based on this, we define three zero-shot speech generation tasks (Table (1) Timbre Imitation: Given speech as source, imitate only the timbre of the reference speech while preserving the linguistic content and speaking style. It can be adopted in voice conversion that only spectral aspects of speech are converted 3 (2) Style Imitation: Given a speech as source, imitate only the speaking style of the reference speech while preserving the content and the timbre. can be adopted in accent conversion [5] and emotion conversion 7 (3) Voice Imitation: Given either speech (i.e., conversion task) or text (i.e., synthesis task) source, imitate both the timbre and style of the reference speech while preserving the content. It can be adopted in voice conversion that both spectral and prosodic aspects of speech are converted 3 4 and zero-shot TTS [8]
Work accomplished during the internship at Meta. Corresponding author.

---

Published as a conference paper at ICLR 2025
Table I: Definitions of zero-shot timbre, style, and voice imitation tasks.
Task
Source (i)
Reference (r))
Attribute(s) to Imitate
Target
Related Areas
Timbre Imitation
Timbre
Voice Conversion
Accent Conversion,
Style Imitation
Style
Wr
Emotion Conversion
Voice Imiittiio
Timbre and Style
W(es, Seste)
Voice Conversion
T(a)
Text to Speech
W and T denote speech and text. ca, and ft represent the linguistic content, style, and timbre of the source i. Similarly, cr, and b, represent the linguistic content, style, and timbre of the reference r.
To address these imitation tasks, existing work has explored approaches including learning the conversion between parallel corpus g
18
19
20 21 disentangled representation learn-
ing 17 22 23 24 and large-scale in-context learning
28 However, these ap- proaches still suffer from the following limitations. Firstly, for the style imitation, existing methods rely heavily on supervision with annotated data, which is hard to collect and scale up. This reliance includes the use of parallel corpus [9] 20 21], style labels (such as categories of accent [20 21 [9]
25
or emotion 30 31), and textual transcriptions [29]30 31 32.. Moreover, achieving zero-shot style imitation where system
imitate accent, emotion, other speaking styles from just a few seconds of speech- remains a significant challenge. Secondly, the decoupling of timbre and style in existing methods is still insufficient, making it challenging control them independently. unless mitigated by some timbre (or style) perturbations or additional fine-tuning stages
13 33
Motivated by the above, this paper proposes Vevo, versatile zero-shot voice imitation framework with controllable timbre and style (Figure It can serve as unified framework for wide range of zero-shot speech generation tasks. Vevo consists of two core stages: (1) Content-Style Modeling (Content to Content-Style): Given speech prompt as style reference, we generate content-style style tokens from the input contens tokens (or the input text). We employ the decoder-only autoregressive leveraging its powerful capability of continued generation to model style. (2) Acoustic Modeling (Content-Style to Acoustic): Given a speech prompt as timbre reference, we generate acoustic representations (such as Mel spectrograms) from the input of content-style tokens. We use a flow-macching transformer [36 37,. which has been verified to excel in in-context learning and reconstructing high-quality audio 12
transformer 34 355,
24
27
38), to achieve timbre-controllable generation.
To obtain the content and content-style tokens of speech, we design self-supervised method to decouple the timbre, style, and linguistic content gradually, which is similar to progressive in- formation filtering: (1) We firstly investigate the commonly used self-supervised speech gre-trained model, HuBERT 22. We find that its continuous hidden features contain rich information about tim- bre, style, and linguistic content (Section 1.1), making a suitable initial stage for information filter- ing. (2) Inspired by existing works for disentangling speaker-agnostic representations [7 39 40) we employ VQ-VAE [1] as tokenizer for HuBERT to filter out timbre, resulting in content-style tokens. (3) Furthermore, we propose that the vocabulary size of the VQ-VAE codebook can function as the "width" of the information bottleneck (22.. By reducing the vocabulary size, we can narrow the bottleneck and filter out not only timbre but also significant style information, thereby obtaining content tokens. Besides, we propose reduce the consecutive duplicate units [41] of the content tokens, called duration reduction, to further remove some style patterns such as unit-level duration.
The contributions of this paper are summarized as follows:
We introduce fully self-supervise approach that progressively decouple timbre, style, and linguistic content of speech. The resulting content-style tokens and content tokens enhance con- trollability in downstream speech generation tasks, particularly for timbre and style.
We propose Vevo, unified framework that enables versatile, controllable zero-shot voice imita- tion tasks. It significantly reduces the reliance on annotated corpora, facilitating self- sufe-supervieed training and in-context learning that can easily be scaled up.
Pre-trained on 60K hours of audiobook speech data without any fine-tuning on style- cor- pora, Vevo matches or even surpasses existing methods in accent and emotion conversion tasks- notably, through zero-shot imitation. Additionally, Vevo's 's effectiveness in voice conversion and text-to-speech tasks further demonstrates its strong generalization and versatility.
2

---

Published as a conference paper at ICLR 2025
Content Tokens
Content-Style Tokens
Acoustic Representations
M
000
O
Source (Speech impott
Autoregressive Transformer
Flow-mutching Transformer
M
Target
000m Text Tokens
TEXT
M Style Reference
M Timbre Reference
Lingulstic content: Source
Source (Text as rnpu))
Style: Style Reforence
Timhre: Timbre Reforence
Figure 1: Vevo inference pipeline. Notably, it can take either speech or text as input, and perform zero-shot imitation with controllable linguistic content (controlled by the source), style (controlled by the style reference), and timbre (controlled by the timbre reference) a single forward pass.
2 RELATED WORK
Controllable Voice Imitation We focus primarily on how existing works approach the imitation
of two key speech attributes: timbre and style. (1) Imitation of Timbre: As crucial aspect of
speaker identity, timbre imitation has been extensively explored within the voice conversion (VC)
field. Most studies aim to utilize the speaker-agnostic representations such PPG features (20 42
or some eeff-supervieed representations [43 44 and use models including GAN 45 46 auto-
encoder [14] [22), and diffusion models 47 48 achieve timbre imitation. (2) Imitation of Style:
In terms of style imitation, accent and emotion are two widely studied attributes. For conversion tasks (with speech input), classic approaches often involve learning the conversion between par- allel corpus 19 20 21 Additionally, many studies aim to obtain the style-agnostic features, such
as pushing them be close to textual transcriptions 30 32 49 Besides, leveraging automatic speech recognition (ASR) models can transform conversion tasks into synthesís tasks, allowing the
injection of style label's embeddings into TTS models to achieve style imitation (29 50 In con-
clusion, these existing approaches often rely on annotated data and struggle to achieve zero-shot style imitation.
(3) Imitation of both Timbre and Style: In VC, some works suggest adopting formulation [I 52 introducing additional modeling for prosody features [4 (53 to achieve both timbre and style imitation. However, these models still have signif- icant room for improvement in both quality and style imitation. Recent advances in zero-shot TTS have greatly improved voice imitation and cloning. They leverage large-scale in-context learning to mimic all speech attributes of reference prompt, including timbre and style, with high quality and speaker similarity [ 3 16, 17 26 33 Nonetheless, it is challenging to obtain the speech repre- sentations disentangled timbre and style effectively [2] 33 leading to inadequate targeted control of these attributes. For instance, using the existing representations directly for VC tasks will lead to timbre leakage, unless mitigated by timbre perturbation or an additional fine-tuning stage
Sequence-to-squenne
Disentangled Speech Representation There are many studies aim to decouple linguistic content, timbre, and style. Existing work on obtaining disentangled speech representations can generally be categorized into several approaches: (1) Knowledge distillation using auxiliary tasks such as
ASR, FO prediction, and speaker verification [15 17 23), (2) Model architecture design based on information bottlenecks, including careful adjustments to hidden layer dimensions [14|22 22 vector
quantization methods like K-means [55) or VQ-VAE [5 17 39 (40, and (3) Perturbation
of acoustic signals [56] 55 58 Besides, existing works also leverage additional learning strategies
including adversarial learning 17 233.. comparative learning (23 59), and mutual information min-
imization 40 60, 61 to enhance disentanglement effectiveness. However, existing work still has two main weaknesses. On one hand, as mentioned earlier, finding suitable representations for down- stream generation tasks that can effectively decouple timbre and style remains quite challenging. On the other hand, how to design voice imitation models that can control specific attributes based on these disentangled speech representations has been scarcely explored.
METHODOLOGY
3.1 VQ-VAE TOKENIZER FOR HUBERT
Motivation To disentangle representations of different speech attributes, we adopt a VQ- VQ-VAE tokenizer 1 due to its demonstrated potential in disentangling high-level information within speech
3

---

Published as a conference paper at ICLR 2025
such as speaker- invariatn features 1
VQ-VAE either directly on the raw waveform In speech domain, it is common practice to apply based speech representations [22 15 62
17 39]
39] or on the self-supervised learning (SSL) In this work, we choose to apply VQ-VAE based SSL The reasons are two fold: (1) HuBERT's continuous hidden features already contain rich information about timbre, style, and linguistic content, making them well-suited for reconstructing acoustic representations such as Mel spectrograms (Section[4.]) (2) Self-supervised learning on speech could be also treated high-level knowledge distillation. VQ- VAE enables us to further information filtering and disentangling for the SSL features.
representations - specifically, HuBERT
Architecture The VQ-VAE consists of three components: Encoder, Vector Quantization (VQ), and Decoder. Formally, given the codebook E = [e1, ez..... ,xk] whose vocabulary size is K, taking HuBERT hidden features x as input, we get the reconstructed after the three modules:
Ze (x) Encoder(x))
2y (z) Cb, where is arg min (()) eyl12.
c = Decoder(z,(x)))
where zy (ax) is the quantized representation (i.e., token) of (x) after VQ. The loss function sists of the reconstruction loss (whose weight is X) and quantization loss (whose weight is f3):
L = >| - 2112 + = (=)=)
Note that there is no real gradient defined for 2g (a)) We could utilize the straight- thhrough gradient estimator or exponential moving average (EMA) as the optimization algorithm [.]. In this paper, we follow the design in 62 63 and use the EMA algorithm. We describe the specific module design of VQ- VAE in Appendix B.1 Notably, the VQ- VAE model does not contain any downsampling or upsampling operations, thus preserving the sequence length of the input æ. In other words, for the 50 Hz frame-level HuBERT features (2). we can also get 50 Hz frame-level tokens after VQ.
Analysis of the Vocabulary Size of Codebook The quantization of HuBERT hidden features by VQ-VAE can be viewed as a form of lossy compression. Inspired by AutoVC [22, we propose that the vocabulary size of the VQ codebook acts as an information bottleneck. If the input æ possesses sufficient speech information, reducing the vocabulary size K from infinity to zero: (1) When K + 0o, we consider the bottleneck to be extremely wide, capable of accommodating all information without any loss. (2) As K decreases, more low-level acoustic information begins to be lost, such as spectral features related to timbre or prosodic features related to style. At a certain reduced K, only the highest-level, most abstract information like linguistic content is preserved within x. (3) When + 0, the bottleneck becomes exceedingly narrow, filtering out even high- level information like linguistic content. We validate the above hypothesis through experiments on the zero-shot timbre imitation task (Section 4.1.. Interestingly, as we progressively reduce K, we observe that timbre information is the first to be filtered out (assuming when K K2). from which we derive the contenr-style tokens. Subsequently, most style information is filtered, and ultimately, almost only the highest-level linguistic content information retained (assuming when K Kc), from which we derive the conten/ tokens. We refer to the VQ-VAE model whose K = K, as the content-style tokenizer Qs and the model whose K K2 as the content tokenizer Q..
3.2 CONTENT-STYLE MODELING (CONTENT TO CONTENT-STYLER
During the content-style modeling stage, our goal is transform the content token of speech (or text) into content-style tokens, which is prompted by a style reference. This can be formulated as sequence- seguence-to-sequenee generation task. For this stage, we employ decoder-only autoregressive (AR) transformer, known for its powerful capability in such tasks [1] 34 35). In this section, we will focus only on cases where speech's content tokens are used as input (Figure Z. The scenarios where text serves as input will be discussed in Appendix B.3
Duration Reduction Given speech input u, we denote the content and content-style tokens as (u) and (u). Both of them are 50 Hz frame-level representations of equal length. In the content-style modeling stage, (u( is used as the output. However, instead of using (u), we apply Duration Reduction strategy to it, yielding the reduced Q.(w) as the input. Specif- ically, we merge the consecutive duplicate units of Qo(u) into one. For instance, if (u) =
(e1, G1, e1, e2, e3, ea), it will be condensed ('( = [e1, 62, ea). This strategy offers significant
4

---

Published as a conference paper at ICLR 2025
[ree]
[ooe]
Autoregressive Transformer
Antoregressive Transformer
[v9n]
[epl
(se)
[so]]
Hepp]
pspl
Duration Reduction
Durution Reducstion
Content Tokenizer
Style Encoder
Content-Style Tokenizer
Content Tokenizer
Global Style Racoder
Content-Style Tokenizer
M Training sample (a) Training
M/-
V
M Style Roferance
Token Style Refervnce
Source Output
Sowrce
(b) Inference: reference-stylee-nnonn continuation
Content-style modeling based on autoregressive transformer. During inference, we employ both global style encoder and content-style tokenizer to enhance the effect of the style reference.
Figure 2:
benefits: (1) It further filters out style-specific information within Q.(u) such as the unit-level du- ration. Some studies also point out that such reduction could aid in reducing accents and other style elements [4]]: (2) resolves the model's challenge with learning changes sequence length before and after style modeling when Qe(u) and (u) are always equal in length; (3) It shortens the overall sequence length, which is beneficial to model context for transformer.
Global Style Encoder We design global style encoder to capture the global style guidance from the speech input u. producing a style embedding (denoted as (u)).. Its advantage comes from the flexibility during inference: if we aim to optimize inference speed and reduce memory usage, we can rely solely on this style embedding for style guidance, named as reference-global glooaa-guaaddeddddd guided continua- tion (Figure(5) However, to maximize the performance of style imitation, in addition to using gu)), we can also append the style reference's content-style tokens into the input sequence to enhance its effect, named as reference-style enyleeenhacced continuation (Figure 2b). The global style encoder consists of WayLM-based representation layers and IDNN-based feature extraction layers [64 65]. We describe the detailed module design in Appendix B.2
Training and Inference During training, we conduct conduct sele-supervised learning on speech data. The input sequence of transformer is (SOS), (w), (SEP). (()), (SEP), (u)). We only perform the next token prediction on the last [(SE)) Q (w)), with the ground truth being (u), (EOS)). Here, (SOS), (SEP). and (EOS) are treated as three special tokens in language model (66). During infer- ence, for source speech u, and style reference Var. we can conduct the reference-stylee siferenc--tyyle-enhanee continuation (Figure 2b) by feeding the input sequence (SSOS) Qe @ w:), ((.r)) Q. for autoregressive generation, where ( means the concatenation. For reference-globar soobal-guided continu- ation (Figure the input sequence becomes ((SOS), (c(()), ([..)]].
3.3 ACOUSTIC MODELING (CONTENT-STYLE ACOUSTIC)
During the acoustic modeling stage, prompted by timbre reference, we aim to transform the content-style tokens to Mel spectrograms. We adopt a flow matching transformer [34] 35 36 (Fig- ure 3 which has been verified to be effective in in-context learning and reconstructing high-quality acoustic representations
12 27 38
During training, given speech u and its Mel spectrogram y1, we randomly select part of yı as the timbre reference (denoted as "ft",, and aim to reconstruct the other part (denoted as ymm condi- tioned on ffth and the content-style tokens Q= (u). In other words, we aim to model the conditional probability P(y yfth (u)). Specifically, we follow Voicebox [27] and use temporal span m O y1. and y
masking strategy: yi mask that is of the same length as yı. and @ means the element-wise multiplying operation. During inference, given a source speech it and a timbre reference Uty. all the source's 's Mel spectrogram will be masked (i.e., y. m The imput conditions become the timbre reference's 's Mel spectrogram (i.e., " and the concatenated content-style tokens (u,u; @ u,r). ). This enables the generated target to preserve the linguistic content and style of ui, and the timbre of uhr (Figure 3b).
(1 m) O yı, where m is a binary temporal
5

---

Published as a conference paper at ICLR 2025
Flow-matching Transformer
Flow-matching Transformer
Tokama
Tokera
Rondom Spon Masking
Masking
Content-Seyl Telonntaur
Content Style Tokenizer
Mel Spectrogram Extraction
Cuncut
Mei Spactragram Extraction
v W
Spectrogram
M
Training sample
Source
Timbre Reforence
Training
(b) Inference
Figure 3: Acoustic modeling based on a flow-matching transformer. During inference, we append the timbre reference the rightmost (or leftmost) end, enabling timbre-controllable generation.
We use the conditional flow matching algorithms based on optimal transport path, which widely The loss function is defined as:
adopted in related works 12 27).
dy: dt
Lefm IE, 10.90
folye,t, t, vi Qo(u))
where = -1 (1-(1))))
+ 1-11
where is the time step that is sampled from the uniform distribution 41(0, 1). yo is a noise sampled from standard Gaussian distribution, fi(-) is the vector filed (which is estimated by transformer). and a is a small constant of the optimal transport (OT) path. Notably, the frame rates of the content- style tokens (u) and the Mel spectrogram y! could be different. We follow (44) and
simple signal resampling operation to align them. Then we use the adding operation to fuse their frame- level features. We describe the detailed module design Appendix [B.4] After obtaining the Mel spectrogram, we utilize a BigVGAN 67 vocoder to produce the waveform (Appendix B.5
3.4
VEVO FOR VARIOUS ZERO-SHOT IMITATION TASKS
Assume that during the content-style modeling and acoustic modeling stages, we have obtained pre- trained models Masyle and Macoustic respectively. We can then adjust only the inference pipeline to apply Vevo various zero-shot imitation tasks. Given the source speech u; (or text TÚ) and the reference Uy,
utilize the following variants of Vevo to achieve zero-shot timbre, style, and voice imitation tasks (" (= MM means that the model M is prompted by generate):
Vevo-Timber for timbre imitation: (u4 Mr, Maccustic
Vevo-Style for style Imitation: (((()) Ur Matyle "i Macoustic
VoiceVoice for voice imitation (conversion task): Q2((4) Ur, Matyle Uy Macoustic
Vevo-TTS for voice imitation (synthesis task): (T:). Ur, Matyle Ur Macoustic
For Vevo-TTS, (Ti) means the tokenization Tı, and Mstyle means the pre-trained model for content-style modeling that takes text as input. We describe its detailed design in Appendix B.3 B.3
4 EXPERIMENTS
We train the English-only models on 60K hours ASR-transcribed English au- as the dataset used by the Voicebox English model [27]. The model diobooks, which is the
Training Data
Macoustic and Mstyle are trained solely with speech data. The model Matgles which uses text as in- put, is trained with both speech and textual transcriptions data. We begin with the publicly available HuBERT-Largd model
model 2 to prepare the VQ-VAE tokenizer. We utilize hidden features from

---

Published as a conference paper at ICLR 2025
Table 2: Performance of Macoustic trained by different HuBERT representations on zero-shot tim- bre imitation task. We highlight three key turning points during the self- disentanglement process: the initial stage of information filtering (the 18th layer features, where vocabulary size can be considered infinite), the proposed content-style tokenizer (VQ-VAE tokens with vocabulary size of 4096), and the proposed content tokenizer VAE tokens with a vocabulary size of 32).
WER
S-SIM
S-SIM
FPC
Representations
#Vocab
Analysis
(4)
(to ref (*)
(to stc) (L)
(to (t)
Ground Truth
5.526
0.762
0.087
1.000
24th layer features
OC
5.706
0.266
0.400
0.768
Pros: Intelligibility, Style consistency
18th layer features
oC
5.324
0.250
0.505
0.824
Cons: Timbre imitation
12th layor features
DC
5.348
0.200
0.626
0.805
PPG features
oc
6.143
0.449
0.157
0.741
Pros: Intelligibility, Timbre imitation
ASR tokens
29
7,836
0.463
0.125
0.698
Cons: Style consistency
K-means tokens
1024
11.493
0.398
0.150
0.734
Worse than VQ- VAR tokens (1024)
16384
6,807
0.398
0.306
0.826
As the vocabulary size decreases,
4096
6,908 f
0.403
0,236 J
0.797
Pros:
1024
6.967 f
0.418
0,249
0.764
Timbre imitation 1
VQ-VAE tokens
32
9,731 1
0.426
0,161 J
0.706
Cons:
16
13.169
0.441
0,146
0.672
Intelligibility
8
21.8131
0.392
0.109
0.675
Style consistency 4
PPG features and ASR tokens obtained from HuBERT ASR Large, while the others from HaBERT-Large. K. means and VQ- VAE tokens are quantized on the 18th layer features of HuBERT-Large FPC are evalunted only on EMOTION.
#Vocaa: the vocabulary size S-SIM: Speaker SIM. ref/src: reference/souree.
the 18th layer as the reconstruction objective for the tokenizer. Both the content and content-style tokenizers trained 100-hour subset randomly sampled from the full 60K-hour dataset.
Evaluation Data We consider various evaluation settings to construct the evaluation set: (1) For clean data, such as recordings made in studio environments, we select audiobook speech data. Specifically, we reserve a subset of the total 60K hours of data as evaluation samples, which we denote AB. (2) For noisy data, which may include in-the-will recordings and diverse recording devices, we use the Common Voice English dataset (CV) [68]. It covers broader accents and is noisier compared to AB. (3) Additionally, to introduce more stylized and expressive data, we use an internal emotional and accented corpus to sample an emotional test set (EMOTION) and an ac- cented test set (ACCENT), There are 700 evaluation samples in total: 200 from AB, 200 from CV, 150 from ACCENT, and 150 from EMOTION.
Evaluation Metries For the objective metrics, we evaluate the intelligibility (WER), speaker similarity (S-SIM), accent similarity (A-SIM), emotion similarity (E-SIM), and FO correlation (FPC) 44 69). Specially, we calculate WER based on Whisper-large-v' [3] 70). For the three similarity metrics S-SIM, A-SIM, and E-SIM we calculate the cosine similarity between the em- beddings (of speaker, accent, or emotion) of the generated sample and the reference. Specifically, we extract these embeddings using WavLM TDNN [3 64] for speaker. CommonAccen 71 for accent, and emotion2ved
(72 for emotion, respectively. We also used CommonAccemn and emotion2vec as the classifiers measure the classification accuracy of accent and emotion (A-ACC and E-ACC). For subjective metrics, we use the Mean Opinion Score (MOS, rated from 1 to 5) to assess naturalness (N-MOS) and similarity in speaker, accent, emotion, and prosody (SS-MOS, AS- MOS, ES-MOS, and PS-MOS). SS MOS, AS-MOS, and ES-MOS evaluate the similarity between the generated sample and the reference, while PS-MOS assesses the similarity between the gener- ated sample and the source. Additionally, we employ Comparative MOS (CMOS, rated from -3 to 3) to evaluate naturalness (N-CMOS), accentedness (A-CMOS), and emotiveness (E-CMOS), Detailed backgrounds of subjects and definitions of all subjective metrics are provided in Appendix P
4.1 EFFECT OF THE VOCABULARY SIZE OF THE VQ-VAE TOKENIZER
We conduct experiments to figure out how to derive content and content-style tokens from speech. The key questions include: (1) What information from speech is retained in the continuous hidden
hutprr/rtttnloooirooooeeeeeeleee verification htnps//ghhhh..ommmmddllooaaoooon
https///ingnnnaalccc///oiggceeeo ecapa

---

Published as a conference paper at ICLR 2025
Table 3: Results on zero-shot timbre imitation and voice imitation (conversion) tasks. (Con- tRep/Model: Hours of training data for the used content representations and the model)
Zero-Shot Timbre Imitation (AB, CV)
Source i, Reference >> Target: W(e,, f,)
Training Data (ContRep/ / Model)
WER (L)
S-SIM (to (t)
FPC
N-MOS
SS-MOS (t)
PS-MOS
Model
AR?
(to (t)
(1)
(to i) (t)
HierSpeech++
X
500K/ 2.8K
4.233
0.385
0.634
3.05 1023
3.24 102
3.08 :12x
LM-VC 52
V
IK / 60K
8.623
0.310
0.524
2.90 rom
2.98 se:s
2.16 612
UniAudio 28
V
IK/ 100K
7.241
0.264
0.575
3.04
2.47
2.51
FACodec
X
60K/ / 60K
3.682
0.327
0.611
2.50 1920
3.10
3.10
Vevo- Voice
GOK / 60K
7.694
0.458
0.485
3.09
2015
3.51 gen
2.60 2m1
Vevo-Timbre
x
60K / 60K
2.968
0.420
0.686
3.35 1009
3.36 1016
3.45 2an
Zero-Shot Voice Imitation (ACCENT, EMOTION)
Source Reference y Target: Wir,,8,444
WER
S-STM
A-SIM
E-SIM
N-MOS (1)
SS-MOS (to (1)
AS-MOS
ES-MOS (to
Model
(L)
(to (t)
(to (f)
(to (t)
(to P)
Ground Truth
10.917
0.762
0.763
0.965
HierSpecch++ [33
12.921
0.466
0.526
0.658
3.04 2014
3,15 4023
3.13 1a2
2.55 40.9
LM-VC 52
20.353
0.312
0.426
0.649
2.40 200
2.56 201
3.02 2:19
2.46 g00
UniAudio 28
15.751
0.311
0.486
0.611
2.95
2.39
2.42
2.41
FACodee 17
12.731
0.434
0.514
0.688
2.36
3.19
3.01
2.30
Vevo-Timbre
12.351
0.486
0.567
0.816
3.43 200
3.46 so1s
3.55 paass
2.66 goss
Vevo-Voice
15.214
0.517
0.614
0.872
3.24 2on
3.70
3,90
3.20
1 PS MOS, E-SIM, and ES MOS are evalunted only an EMOTION. A A-M and AS MOS are evalinted only on ACCENT The best and the second best result is shown in bold and by underlined.
features of ? (2) How do vector quantization methods, including the commonly used K- means 2 28 52 73 and our adopted VQ- VAE 62). affect the disentanglement ability of the resulting discrete tokens of HuBERT? (3) How does the vocabulary size of VQ-VAE codebook in- fluence the produced tokens? To answer these questions, we investigate the performance of different HuBERT representations on the zero-shot timbre imitation ii..- using them to train Macoustie-
Specifically, we adopt the representations of the HuBERT-Larged model, which is a 24-layer trans- former pre-trained on Libri-light dataset [74). For comparison, we also examine the HUBERT-ASR- Large model. which is fine-tuned from HuBERT-Large for ASR task on LibriSpeech (75). Com- pared to HuBERT-Large,
contains an additional prediction layer and softmax layer, whose output is ppy E T229 where T is the frame length and 29 represents the vocabulary size of phonemes. We refer to Tppg as PPG features and also derive frame-level ASR tokens from each frame's 's PPG features; Tosr arg max Pppg
I.. We randomly sample 6K-hour subset from the full training data for training. The results are presented in Table
Our findings indicate that: (1) HuBERT continuous hidden features possess rich information on timbre (high S-SIM to source), style (high FPC), and linguistic content (low WER). Notably, the S- SIM to source even higher than that to reference, i.e., there a timbre leakage. This phenomenon is more obvious for shallower 12th layer features. (2) After ASR fine-tuning, both PPG features and ASR tokens retain substantial linguistic content information (low WER) but exhibit a significant reduction in timbre information (lower S-SIM to source) and decrease in style information (lower VQ-VAE. K-means tokens show lower intelligibility. S-SIM to reference, and FPC when K is the same (1024). Huang et al. provides detailed comparison of between these two methods recently 62). (4) For VQ-VAE tokens, larger vocabulary sizes (e.g.. 16384) retain more timbre information (S-SIM to source at 0.306). As K decreases to 4096, much of the timbre information is filtered out (S-SIM to source/reference at 0.236/0.403), yet style information is relatively retained (FPC at 0.797) When K reduces further to 32, in addition to timbre, most style information is also filtered out- - FPC drops to 0.706, similar to ASR tokens. As K diminishes to 16 or even 8, even the high-level linguistic content begins to be filtered out (rapid increase in WER).
FPC). (3) Compared
Based on these findings, we select VQ-VAE with K₂ = 32 for the content tokenizer and K, = 4096 for the content-style tokenizer. Note that designing the information bottleneck is challenging trade- off, and such K2 and K, may not be optimal. However, the results in the following sections show that such a choice has been pretty good under various voice imitation tasks. Additional effects of different (Ka, K2) combinations on Matyle training are detailed in Appendix D.I
8

---

Published as a conference paper at ICLR 2025
Table 4: Results on style imitation task. (PC: Parallel corpus. SL: Style labels)
Zero -shot
Supervision
WER (4)
A-/E- ACC (1)
A-/E- SIM (1)
N-
A-/E-
Model
COMS (1)
CMOS (1)
PC
SL
Text
ASR-AC [29
x
V
4.775
0.633
0.00
0.00
Vevo-Stylc (ASR)
X
X
v
1.550
0.723
0.570
0.32 -a
0.49 ex14
Vevo-Style
/
x
x
x
3.083
0.663
0.562
0.30 ga
0.35 ga2
VoiceShop 20
x
/
/f
5.547
0.642
0.00
0.00
Vevo-Style (ASR)
x
/
3,553
0.735
0.585
0.26
0.18
Vevo-Style
x
x
5.464
0.673
0.554
0.12 ea..
0.13 sea
Conv-Speak 21
x
V
X
9.950
0.571
0.00
0.00
Vevo-Style (ASR)
x
x
V
2.778
0.864
0.574
0.10 =
0.40 galt
Vevo-Style
x
x
X
3.889
0.903
0.580
0.15
0.60 ease
Emovox NO
x
x
vf
15.444
0.750
0.00
0.00
Vevo-Style (ASR)
x
/
9.842
0.692
0.800
1.74 20:0
0.45 gall
Vevo-Style
x
10.221
0.754
0.825
1.78 =0.2)
0.49
2a13
We present four comparative groups. Evaluation samples for each group are sourced from the baseline's demo website. For the first three groups. we evaluate A-ACC/SIMICMOS. and for the last group, we evaluate E-ACC/SIM/CMOS.
4.2 ZERO-SIBE SEROT TIMBRE IMITATION AND VOICE IMITATION (CONVERSION TASK)
Further, we apply Vevo to various zero-shot imitation tasks. This section evaluates Vevo-Timbre and Vevo-Voice on zero-shot timbre and voice imitation tasks. We select several ofate-of-the-art (SOTA) baselines in pero-shot voice conversion, including HierSpeech++ [53], LM-VC [52], UniAudio (28),
and FACodec [17]. Details about these baselines are available in Appendix C We train Mstyle and
Macoustic on the full 60K hours dataset. The results are presented in Table
The findings reveal that: (1) Zero-shot timbre imitation: Compared to the four baselines, Vevo- Timbre exhibits superior performance across common voice conversion metrics such as WER, S- N-MOS, and SS-MOS. Additionally, Vevo-Timbre demonstrates a clear advantage in FPC and PS-MOS, which measure style consistency. (2) Zero-shot voice imitation: Against the four base- lines, Vevo-Voice not only excels in mimicking speaker identity (S (S-SIM, SS-MOS) but also signifi- cantly outperforms in imitating specific style attributes like accent (A-SIM, AS-MOS) and emotion (E-SIM, ES-MOS). (3) Comparing Vevo-Tim and Voice:iice:
's strength lies in preserving the style of the source (FPC, PS-MOS), whereas devo-Voica additionally excels in style imitation, resulting in higher speaker similarity (S-SIM, SS-MOS), However, due the autoregres- sive design in Matyle, Vevo-Voice scores lower in intelligibility (WER) compared to Vevo- -ivo-iibree.
4.3 ZERO-SHOT STYLE IMITATION
We present the performance of Vevo-Style in the zero-shot style imitation task, focusing on widely studied styles such as accent and emotion. For accent imitation, we select baselines including ASR- AC 29), VoiceShop 20). and Conv-Speak (2T). We use their demo website samples as our evalua- tion set, including conversions among multiple accented English such as British, American, Hindi, and Mandarin. For emotion imitation, we choose Emovox (30] and its demo website samples, which include conversions from Neutral to Happy, Angry, and Sad emotions. Moreover, we also introduce the Vevo-Style (ASR) for a zero-shot style imitation baseline. Its only difference compared to the 4.1 rather than Q. as the content tokenizer.
Seyo-Style model is the use of Tasr (see Section
Our experimental results are shown in Table Our observations indicate: (1) Compared to the base- lines, Vevo-Style is only self-supervised trained on audiobook speech data. However, without any fine-tuning on accented or emotional corpus, it delivers superior outcomes in terms of intelligibilitt (WER), quality (N-CMOS), and the imitation of accents and emotions A-AC//SIM/CMOS and E- ACC/SIM/CMOS). (2) Using text as the additional supervision, Vevo-Style (ASR) further surpasses Vevo-Style in intelligibility and specific aspects of accent imitation. We speculate that compared to Besr, the Qe used by Vevo-Style may still retain small portion of accent-related information, thereby limiting the Matyle to perfectly imitate the accent information from the style reference.
4.4 ZERO-SHOV VOICE IMITATION (SYNTHESIS TASK)
We present the performance of Vevo-TTS in the zero-shot voice imitation (synthesis) task. We select the classic baselines of the zero-shot TTS filed, including the Non-AR Voicebox model [27. and

---

Published as a conference paper at ICLR 2025
Table 5: Results on zero-shot voice imitation (synthesis) task.
Training Data
WER (L)
S-SIM
A-SIM
E-SIM
N-CMOS
SS-MOS
AS-MOS (*)
ES-MOS
Model
AR?
(1)
(t)
(1)
(1)
(1)
(1)
Ground Truth
11.348
0.710
0.633
0.936
CosyVoice
24
171K
8,400
0.614
0.640
0.839
-0.18 a01
4.11 2cıs
3.99
202
3.66 sop
MaskGCT
15
100K
9.442
0.659
0.645
0.822
-0.04
4.16
4.38
3.76
VALL-E 26
45K
13.226
0.400
0.485
0.735
-1.24
2.82
2.77
2.63
Vuicehox
9,414
0.463
0.575
0.811
.0.55 102
3.87 1021
3.49 112
VoiceCraft
77
9K
13.057
0.392
0.517
0.788
-0.50 ard
3.47
2010
3.29 203
3.52
2005
Vevo-TTS
60K
12.066
0.505
0.579
0.840
-0.14 201
4.05 2mm
4.12 gem
4.03
A-SIM and AS-MOS are evaluated on ACCENT samples. E-SIM and ES MOS are evaluated on EMOTION samples. The best second results of only last four shown bold and by underlined.
Table 6: Effect of duration reduction and different inference modes of Matale. (#Inference Input: input sequence length (%) during inference. w/o: without, w/: with)
Model
#Inference Input
WER ())
S-SIM
A-SIM
E-SIM (f)
DDUR (L)
VeiceV wlo Duratiinn Reduction
100%
15.214
0.517
0.614
0,883
0,933
127%
15.958
0,501
0,583
0.842
1.698
w/ Global- Global-guided continuation
42%
16.809
0,510
0.597
0.864
0.947
Vevo-Voice uses aeferenc-sttle--nhnnee continuation. A-SIM is evaluated only on ACCENT samples. E-SIM is evaluared only EMOTION samples. The remaining metries are evaluated on both ACCENT and EMOTION samples.
the AR models such as VALL-E 266 76 and VoiceCraft (77,, all of which are trained only on au- diobook speech data. For comparison, also include two stronger SOTA models: Cosy Voice [24] and MaskGCT [3] 77), which are trained on large-scale private corpus derived from tn-the-wild
video data, featuring highly diverse distributions [24] [7]. Detailed baseline information and eval- uation results are available in Appendix D.2 Here, we only highlight performances on ACCENT and EMOTION evaluation samples (Table 5). Our observations are as follows: (1) Compared be- tween Voicebox and Vevo-TTS whose training data are identical, Vevo-TTS, while showing slightly inferior performance in WER (which is a common weakness for AR models), excels across all other metrics. (2) Notably, Vevo-TTS demonstrates outstanding performance in style imitation (A/E-SIM, ASSES-MOS). Despite being trained only on audiobook data, surpasses CosyVoice Voice and MaskGCT in some emotion imitation tasks (ES-MOS is 4.03). This verifies the effectiveness of our proposed content-style tokens, which could be representations that can effectively capture style information and are casily learned by downstream models.
4.5 EFFECT OF DURATION REDUCTION AND DIFFERENT INFERENCE MODES
Finally, we conduct ablation studies on several key components within Vevo, including the impact of different (Ke, K₂) values on voice imitation tasks (see Appendix D.I). the effects of the duration reduction strategy, and the two inference modes of Matyle. as presented in Table 6 We adopt DDUR to measure the average differences in duration (seconds) between the converted and ground truth utterances 330 51]. We observe that: (1) The duration reduction not only reduces the inference input length but also consistently demonstrates clear advantages, especially in duration conversion continuation significantly shortens the sequence length (to 42% of Vevo- Voice), with only a slight decline in performance metrics. This showcases its substantial potential in saving inference memory and enhancing inference speed.
(DDUR), (2) The snference-glooal-geeee
5 CONCLUSION
We introduce Vevo, versatile zero-shot voice imitation framework featuring controllable timbre and style. Vevo contains of two primary stages: content-style modeling via an autoregressive trans- former, and acoustic modeling via flow matching transformer. Both stages are trainable through self-supervised and in-context learning, friendly to scale up. Vevo operates based on our newly proposed content and content-style tokens, generated by VQ-VAE tokenizers of HuBERT with care- fully adjusted vocabulary sizes. Pre-trained only on 60K hours of audiobook speech data without fine-tuning on style-specific corpus, Vevo outperforms state-of-the-art models of accent and emo- tion conversion fields, particularly achieving these conversions in a zero-shot manner. Furthermore, Vevo's robust performance in zero-shot voice conversion and rext-to-speech tasks underscores its versatility and also highlights the broad potential of our proposed disentangled speech tokens.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is partially supported by the NSFC under Grant 62376237, Shenzhen Science and Tech- nology Program ZDSYS$2023066601100000
We thank Yuancheng Wang. Jilong Wu, Fuchun Peng, and the anonymous reviewers for their insightful comments and suggestions. We appreciate the efforts of Meiyu Zheng and all subjects during the subjective evaluation.
REFERENCES
[1] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NIPS, pages 3306-63155 2017.
[2] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhut- dinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process., 29: 3451-3460 2021.
[3] Seyed Hamidreza Mohammadi and Alexander Kain. An overview of voice conversion systems. Speech Commun., 88:65-8888 82, 2017.
[4] Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li. An overview of voice con- version and its challenges: From statistical modeling deep learning. IEEE ACM Trans. Audio Speech Lang. Process., 29:132-1577 2021.
[5] Daniel Felps, Heather Bortfeld, and Ricardo Gutierrez-Osuma Foreign accent conversion in computer assisted pronunciation training. Speech Commun., 51(10):920-93 2009.
[6] Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Huulhlaiennnulliinennn John Levis, and Ricardo Gutierrez-Osuna. L2-ARCTIC: non-native english speech corpus. In INTERSPEECH, pages 2783-2787. ISCA, 2018.
[7] Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Emotional voice conversion: Theory, databases and ESD. Speech Commun., 137:1-18, 2022.
[8] Xu Tan. Neural Text-to-Speech Synthesis. Springer, 2023.
[9] Guanlong Zhao, Shaojin Ding, and Ricardo Gutierrez-Osune Converting foreign accent speech without reference. IEEE ACM Trans. Audio Speech Lang. Process., 29:2367-238 2021.
[10] Brij Mohan Lal Srivastava, Nathalie Vauquier, Md. Sahidullah, Aurélien Bellet, Mare Tom- masi, and Emmanuel Vincent. Evaluating voice conversion-bar based privacy protection against informed attackers. In ICASSP, pages 2802-2806. IEEE, 2020.
[11] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong. Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang. Yuanhao Yi, Junteng Zhang, Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, and Xiaobin Zhuang. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint, abs/2406.02430, 2024.
[12] Hao-Han Guo, Kun Liu, Fei-Yi Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint, abs/2409.033 033883, 2024.
[13] Yuancheng Wang. Haoyue Zhan. Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to- speech with masked generative codec transformer. In ICLR. OpenReview.net, 2025.

---

Published as a conference paper at ICLR 2025
[14] Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David D. Cox. Un- supervised speech decomposition via triple information bottleneck. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 7836-7846. PMLR, 2020.
[15] Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentan- gled self-supervised representations. In INTERSPEECH, pages 3615-3619. ISCA, 2021.
[16) Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, Zejun Ma, and Zhou Zhao. Mega-tts: Zero-shot text-to- speech at scale with intrinsic inductive bias. arXiv preprint, foss/300.03509, 2023.
[17] Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang. Eric Liu. Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiangyang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Ii,, and Sheng Zhao. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In ICML. OpenReview.net, 2024.
[18] Lifa Sun, Shiyin Kang, Kun Li, and Helen M. Meng. Voice conversion using deep bidirectional long short-term memory based recurrent neural networks. In ICASSP, pages 4869-4873, IEEE, 2015.
[19) Huaiping Ming, Dong- Yan Huang, Lei Xie, Jie Wu, Minghui Dong, and Haizhou Li. Deep bidirectional LSTM modeling of timbre and prosody for emotional voice conversion. In IN- TERSPEECH, pages 2453-2457. ISCA, 2016.
[20] Philip Anastassiou, Zhenyu Tang, Kainan Peng, Dongya Jía, Jiaxin Li, Ming Tu, Yuping Wang, Yuxuan Wang, and Mingbo Ma. Voiceshop: A unified speech-to-speed framework for identity-peeservigg zero-shot voice editing, arXiv preprint, abs/2404. 06674, 2024.
[21] Huaying Xue, Xiulian Peng, Yan Lu, et al. Convert and speak: Zero-shot accent conversion with minimum supervision. In ACM Multimedia. ACM, 2024.
[22] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Au- tove: Zero-shot voice style transfer with only autoencoder loss. In ICML, volume 97 of Pro- ceedings of Machine Learning Research, pages 5210-5219. PMLR, 2019.
[23] Mateusz Lajszezak, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnaud Joly, Álvaro Martin-Cortinas, Ammar Abbas, Adam Michalski, Alexis Moinet, Sri Karlapati, Ewa Muszynska, Haohan Guo, Bartosz Putryez, Soledad López Gambino, Kayeon Yoo, Elena Sokolova, and Thomas Drugman. BASE TTS: lessons from building a billion- parameter text-to-speech model on 100k hours of data, arXiv preprint, abs/2402.08093,, 2024.
[24] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma. Zhifu Gao, and Zhijie Yan. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint, abs/2407.05407, 2024.
[25] James Betker. Better speech synthesis through scaling, arXiv preprint. abs/2305.07243, 2023.
[26] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Fura Wei. Neural codec language models zero-shot text speech synthesizers. arXiv preprint, abs/2301.02111 2023.
[27] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text- guided multilingual universal speech generation at scale. In NewrIPS, 2023.
[28] Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Haohan Guo, Xu- ankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Zhou Zhao, Xixin Wu, and Helen M. Meng. Uniaudio: Towards universal audio generation with large language models. In ICML. OvenRevie 2024.
12

---

Published as a conference paper at ICLR 2025
[29] Mumin Jin, Prashant Serai, Jilong Wu, Andros Tjandra, Vimal Manohar, and Qing He. Voice- preserving zero-shot multiple accent conversion. In ICASSP, pages 1-5. IEEE, 2023.
[30] Kun Zhou, Berrak Sisman, Rajib Rana, Björ W. Schuller, and Haizhou Li. Emotion intensity and its control for emotional voice conversion. IEEE Trans. Affect. Comput., 14(1):31-48, 2023.
[31] Tianhua Qi, Wenming Zheng, Cheng Lu, Yuan Zong, and Hailun Lian. PAVITS: exploring prosody-aware VITS for end-to-end emotional voice conversion. In ICASSP, pages 12697- 12701. IEEE, 2024.
[32] Xi Chen, Jiakun Pei, Liumeng Xue, and Mingyang Zhang. Transfer the linguistic representa- tions from TTS accent conversion with non-parallel data. In ICASSP, pages 12501-12505. IEEE, 2024.
[33] Tao Li, Zhichao Wang. Xinfa Zhu, Jian Cong, Qiao Tian, Yuping Wang, and Lei Xie. U-style: Cascading u-nets with multi-level speaker and style modeling for zero-shot voice cloning. IEEE ACM Trans. Audio Speech Lang. Process., 2024.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.
[35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manic-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint, abs/2302.13971. 2023.
[36] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR. OpenReview.net, 2023.
[37] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4172-4182. IEEE, 2023.
[38] Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang. Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang, Ivan Cruz, Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yael Yungster, Alice Rakotoarison, Liang Tan, Chris Summers, Carleigh Wood, Joshua Lane, Mary Williamson, and Wei-Ning Hsu. Audiobox: Unified audio generation with natural language prompts. arXiv preprint, 15822312.15821, 2023.
[39] Da-Yi Yi Wu and Hung-Y Yin Lee. One-shot voice conversion by vector quantization. In ICASSP. pages 7734-7738. IEEE, 2020.
[40] Andros Tjandra, Ruoming Pang, Yu Zhang, and Shigeki Karita. Unsupervised learning of disentangled speech content and style representation. In INTERSPEECH, pages 4089-4093. ISCA, 2020.
[41] Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Chang- han Wang. Sravya Popuri, Yossi Adi, Juan Miguel Pino. Jiatao Gu, and Wei-Ning Hsu. Text- less speech-to-speech translation on real data. In NAACL-HLLT, pages 860-872. Association for Computational Linguistics, 2022.
[42] Lifa Sun, Kun Li, Hao Wang, Shiyin Kang, and Helen M. Meng. Phonetic posteriorgrams for many-to-one voice conversion without parallel data training. In ICME, pages 1-6. IEEE Computer Society, 2016.
[43] Wen-Chin Huang, Shu- Wen Yang, Tomoki Hayashi, and Tomoki Toda. A comparative study of self-supervised speech representation based voice conversion. IEEE J. Sel. Top. Signal Process., 1308-)3338-118 2022.
[44] Xueyao Zhang, Zihao Fang. Yicheng Gu. Haopeng Chen, Lexiao Zou, Junan Zhang, Liu- meng Xue, and Zhizheng Wu. Leveraging diverse semantic-ber based audio pretrained models for singing voice conversion. In SLT.. IREE, 2024.
13

---

Published as a conference paper at ICLR 2025
[45] Takuhiro Kaneko and Hirokazu Kameoka. Cyclegan-ve: Non-parallel voice conversion using cycle-consistent adversarial networks. In EUSIPCO, pages 2100-22000 2104. IEEE, 2018.
[46] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve: non- parallel many-to-many voice conversion using star generative adversarial networks. In SLT. IEEE, 2018.
[47] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Sergeevich Kudi- nov, and Jiansheng Wei. Diffusion-based voice conversion with fast maximum likelihood sam- pling scheme. In ICLR. OpenReview.net, 2022.
[48] Ha-Yeong Yeong Choi, Sang-Hoon Lee, and Seong-Whan Lee. Diff-hierve: Diffusion-bart hierar- chical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation. In INTERSPEECH, pages 2283- 2283-2287. ISCA, 2023.
[49] Yi Zhou, Zhizheng Wu, Mingyang Zhang, Xiaohai Tian, and Haizhou Li. Tis-guided training for accent conversion without parallel data. IEEE Signal Process. Lett., 30:533-555 537, 2023.
[50] Songxiang Liu, Disong Wang, Yuewen Cao, Lifa Sun, Xixin Wu, Shiyin Kang, Zhiyong Wu, Xunying Liu, Dan Su, Dong Yu, and Helen Meng. End-to-end accent conversion without using native utterances. In ICASSP, pages 6289-6299 6293. IEEE, 2020.
[51] Jing-Xune Xuan Zhang. Zhen- Hua Ling, and Li-Rong Dai. Non-parallel susence-to-sequene voice conversion with disentangled linguistic and speaker representations. IEEE ACM Trans. Audio Speech Lang, Process., 28:540-552, 2020.
[52] Zhichao Wang, Yuanzhe Chen, Lei Xie, Qiao Tian, and Yuping Wang. LM-VC: zero-shot voice conversion via speech generation based on language models. IEEE Signal Process. Lett., 30:1 10:1157-116,, 2023.
[53] Sang-Hoon Lee, Ha- Hea-Y Choi, Seung-Bin Kim, and Scong-Wiie Whan Lee. Hierspecch++: Bridging the gap between semantic and acoustic representation of speech by hierarchical vari- ational inference for zero-shot speech synthesis. arXiv preprint, abs/2311, 12454,, 2023.
[54] Benjamin van Niekerk, Mare-André Carbonneau, Julian Zaïdi, Matthew Baas, Hugo Seuté, and Herman Kamper. comparison of discrete and soft speech units for improved voice conversion. In ICASSP, pages 6562-6566. IEEE, 2022.
[55] Junjie Li, Yiwei Guo, Xie Chen, and Kai Yu. SEF-VC: speaker embedding free zero-shot voice conversion with cross attention. In ICASSP, pages 1226-11300, IEEE, 2024.
[56] Hycong-Seok Choi, Juheon Lee, Wansoo Kim. Jie Lee, Hoon Heo, and Kyogu Lec. Neu- ral analysis and synthesis: Reconstructing speech from self-supervised representations. In NeurIPS, pages 16251-16265, 2021.
[57] Hyeong-Seok Choi, Jinhyeok Yang, Juheon Lee, and Hyeongju Kim. NANSY++: unified voice synthesis with neural analysis and synthesis. In ICLR. OpenReview.net, 2023.
[58] Chak Ho Chan, Kaizhi Qian, Yang Zhang, and Mark Hasegawa-Johnson. Speechsplit2.0r Unsupervised speech disentanglement for voice conversion without tuning autoencoder bottle- necks. In ICASSP, pages 3332-6336. IEFF, 2022.
[59] Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David D. Cox, Mark Hasegawa-Johnson, and Shiyu Chang. Contentvec: An improved self-supervised speech repre- sentation by disentangling speakers. In ICML, volume 162 of Proceedings of Machine Learn ing Research, pages 18003-18017. PMLR, 2022.
[60] Xinfa Zhu, Yi Lei, Kun Song, Yongmao Zhang, Tao Li, and Lei Xie. Multi-speaker expressive speech synthesis via multiple factors decoupling, In ICASSP, pages 1 1-5. IEEE, 2023.
[61] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. CLUB: A contrastive log-ratio upper bound of mutual information. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 1779-1788. PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
[62] Zhichao Huang, Chutong Meng, and Tom Ko. Repcodec: A speech representation codee for speech tokenization. (n)) pages 5777-5790. Association for Computational Linguistics, 2024.
[63] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30:495-507, 2022.
[64] Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. Wavlm: Large-scale self- supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process., 16 (6):1505 2022.
[65] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In IN- TERSPEECH, pages 3830-3834. ISCA, 2020.
[66] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (I), pages 4171- 4186. Association for Computational Linguistics, 2019.
[67] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In ICLR. OpenReview.net, 2023.
[68] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weher. Common voice: A massively- Resources Association, 2020.
speech corpus. In LREC, pages 2218-422, European Language
[69] Wen-Chin Huang, Lester Phillip Violeta, Songxiang Liu, Jiatong Shi, and Tomoki Toda. The singing voice conversion challenge 2023. In ASRU, pages 1-8. IEEE, 2023.
[70] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, volume 202, pages 18492-28518, 2023.
[71] Juan Zuluaga-Gomez, Sara Ahmed, Danielius Visockas, and Cem Subakan. Commonaccent: Exploring large acoustic pretrained models for accent classification based on common voice. In INTERSPEECH, pages
ISCA, 2023.
[72] Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. In ACL (Find- ings), pages 15747-15760. Association for Computational Linguistics, 2024.
[73] Stuart P. Lloyd. Least squares quantization in PCM. IEEE Trans. Inf. Theory, 28(2):129-136, 1982.
[74] Jacob Kahn, Morgane Rivière, Weiyi Zheng. Evgeny Kharitonov, Qiantong Xu, Pierre- Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Em- manuel Dupoux. Libri-light: A benchmark for ASR with limited no supervision. In ICASSP. pages 7669-7673. IEEE, 2020.
[75] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based public domain audio books. In ICASSP, pages 5206-5210, IEEE, 2015.
[76] Xueyao Zhang, Liumeng Xue, Yicheng Gu, Yuancheng Wang, Jiaqi Li, Haorui He, Chaoren Wang, Ting Song, Xi Chen, Zihao Fang. Haopeng Chen, Junan Zhang, Tze Ying Tang, Lexiao Zou, Mingxuan Wang, Jun Han, Kai Chen, Haizhou Li, and Zhizheng Wu. Amphion: An open-source audio, music and speech generation toolkit. In SLT. IEFE, 2024.
15

---

Published as a conference paper at ICLR 2025
[77] Puyuan Peng, Po- Yao Huang, Shang-Woi Wen Li, AbdeIrahman Mohamed, and David Harwath. Voicecraft: Zero-shot speech editing and text-to-speech in the wild. In ACL (1), pages 12442-
12462. Association for Computational Linguistics, 2024.
[78] Jiaqi Li, Xueyao Zhang, Yuancheng Wang, Haorui He, Chaoren Wang, Li Wang, Huan Liao, Junyi Ao, Zeyu Xie, Yiqiao Huang, Junan Zhang, and Zhizheng Wu. Overview the amphion toolkit (v0.2). arXiv preprini, abs/2501. 15542, 2025.
[79] Haorui He, Zenggiang Shang, Chaoren Wang. Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In SLT. IEEE, 2024.
[80] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019.
[81] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[82] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint, 21/22007115955 2022.
[83] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarande Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1, 000+ languages. J. Mach. Learn. Res., 25:97:1-97:52: 2024.
[84] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adver- sarial learning for end-to-end text-to-speech. In ICML, volume 139 of Proceedings of Machine Learming Research, pages 5530-5540. PMLR, 2021.
[85] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from librispeech for text-to-speech. In INTERPPEECH, pages 1526-15300 2019.
[86] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-super learning of speech representations. In NeurlPS. 2020.
[87] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-t--sequeee pre-training for natural language generation, translation, and comprehension. In ACL, pages 7871-7880. Association for Computational Linguistics. 2020.
[88] Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. CSTR VCTK Corpus: En- glish multi-speaker corpus for estr voice cloning toolkit (version 0.92). University of Edin- burgh. The Centre for Speech Technology Research (CSTR), 2019.
[89] Alexandre Défosse7, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression, arXiv preprint, abs/2210.1 13438, 2022.
[90] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. ML.S:
A large-scale multilingual dataset for speech research. In INTERSPEECH, pages 2757-2761. ISCA, 2020.
[91] Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang. Mingjie Jin, Sanjeev Khudanpur, Shinji Watan- abe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongging Wang. Zhao You, and Zhiyong Yan. Gigaspeech: An evolving, multi-domain ASR corpus with 10, 000 hours of transcribed audio. In INTERSPEECH, pages 1670-3674, ISCA, 2021.
[92] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:2523-255 2533, 2023.
16

---

Published as a conference paper at ICLR 2025
A TERMINOLOGY CLARIFICATION
In this study, decouple speech into linguistic content (what speak), timbre (who speaks), and style (how to speak). Below, we will clarify our definitions and scope for timbre and style.
Timbre Timbre is physical concept that refers to the acoustic qualities of sound, such as the spectral envelope, which allows us to differentiate between speakers even when pitch and loudness identical. It is primarily determined by the speaker's vocal anatomy and articulatory behaviors. Often discussed alongside timbre is speaker identity. Speaker identity is perceptual concepi it encompasses not only timbre but also habitual speech patterns, idiosyncrasies, and other personal styles that make a speaker recognizable. While timbre lays the acoustic foundation of identity, speaker identity reflects the broader auditory impression formed by listener.
Style Style refers to the expressive aspects of speech, including accent, emotion, and speaking habits, which dictate how something said. includes specific features such as accent and emotion, but also covers wider array of expressive behaviors. A critical component of style is prosody, which includes features such as FO (pitch), energy, and duration. These prosodic features govern the rhythm, stress, and intonation of speech, contributing significantly to how emotion and emphasis are conveyed. Although style encompasses prosody, it also extends beyond it, influencing not only the melodic flow of speech but also cultural and emotional expressions.
B DETAILS OF VEVO
B.I VQ-VAE ARCHITECTURE
We adopt the implementation of RepCoded 62 and 1. Its architecture of encoder and decoder shown in Figure I The vocabulary sizes of our content and content-style tokenizer 32 and 4096. Their parameter counts 59M and 63M, respectively.
VQ-VAE tokenizer, whose À and are 45
Encoder
Decoder
Speech Rlepresentation
EncoderBlock
DecoderBlock
Latent Regresentation
50Hz
ResidualUnit
50Kz
ComwID (km1 (f M2 gm1)
ResidualUnnit
(k-3, Cyn Mar1
R= Sha Hll Wem1)
ComvDD
Conv1D Cng "og
EncoderBlock
ResidualUnitt
ResidualUnit
DecoderBlock
ConviD (bed, (H11
EncoderBlock
Co (k+3, Sp
ResidualUnit
DecoderBlock
H.
CorwID
ConviD 1k-3,0 (H-1)
(k:3,6 Cane
Latent Representation 50Hz
Speech Representationn 50Hr
Figure 4: Encoder and decoder architecture of our VQ-VAE tokenizer. k; 8. Cans and Cout denote kernel size, stride, input channels, and output channels. ht denotes the vocabulary size of tokenizer. H denotes the hidden dimension of input representations (which is 1024 for HuBERT-Large) This figure is borrowed from the paper of RepCodec [62].
B.2 CONTENT-STYLE MODELING
For the content-style modeling stage, we use reference-styll-enhhnned continuation by default. The architecture of our AR transformer similar LLaMA [35] has 12 layers, 16 attention heads,
2048/3072 embeeddingficed-fowee forward network (FFN) dimension. The global style encoder consists of WayLM-based representation layers and TDNN-based feature extraction layers 64 65] Specif-
'https://gituub.coo/ http:://ghhub.coo//ccl//oCCCooot https://gthhu..commmtaallllaaaa
17

---

Published as a conference paper at ICLR 2025
ically, we adopt the same architecture with WayLM-based speaker verification mode The total parameter count Mstyle is 463M.
During training, for 2K steps and decays over the rest of training. It is trained for 500K updates. During inference, continuation (Figure 2b) or the reference- seaennceeelbaal
Adam W [80] optimizer with peak learning rate of lc-4, linearly warmed up
use the default effrrece-style-enhhnee guided continuation (Figure 5). We generate evaluation samples with specific sampling parameters: top-k is 25, top-p 0.9, and temperature is 0.8.
Antoregressive Transformer
[sos]
[sepl
Hepl
Doration Reduction
Content Tokenizur
Global Style Eacoder
M V-
M Style Reforence
Style Referunce
Source Outpot
Source
Figure 5: siterence-global-gee guided continuation of Matyle for inference.
B.3 CONTENT-STYLE MODELING (TEXT AS INPUT)
Compared to Matyles the only difference of Mstyle is that its input becomes text tokens, rather than the duration reduced content tokens. Specifically, we adopt the Grapheme-to-Phoner (G2P) method and use the same phonemization tokenizer as Voicebox 27]. All the hyper parameters of training and inference are same as Matyle-
B.4 ACOUSTIC MODELING
For the acoustic modeling stage, we follow the flow matching model implementation Voice- box (27). Specifically, randomly mask 70%-100% of the frames to create y We employ the midpoint ODE solver with a step size of 0.0625 (NFE=32). The a of the optimal transport path of flow matching is le-5. The transformer has 24 layers, 16 attention heads, 1024/4096 network (FFN) dimension. Its parameter count is 334M.
cmbeddinggffedd-forre
Our target Mel spectrogram at 24 kHz with 100 Mel bands. It is normalized with the global mean (-5.8843) and standard deviation (2.2615) to stabilize training (27). During training. Mel spectrogram length is capped 1,600 frames and chunked randomly length exceeds. We use Adam [81] optimizer with peak learning rate of le-4, linearly warmed up for 5K steps and decays over the rest of training. It is trained 500K updates. During inference, we employ the midpoint ODE solver with step size of 0.0625 (NFE=32).
We apply the classifier free guidance (CFG) works 24) Specifically, we randomly drop the conditions, i.e., yffh and Qo(u), with a prob- ability of Puncond. During inference, the modified vector filed f. becomes f. (31, t, yfth Qu))
to improve the generation quality like other
(1 + (x),, (, yftz Q.)))) o' l), where a is the strength of the guidance. In practice, we follow Voicebox and set Puncond 0.2 and 0.7.
B.5 VOCODER
vocoder synthesis waveform from Mel spectrogram. We fine-tune from the official released checkpoint Servgan_24kh__1000 using our 60K hours training data. Its parameter count is 112M.
We use BigVGAN 67
18

---

Published as a conference paper at ICLR 2025
C DETAILS OF BASELINES
C.I ZERO-SHOT TIMBRE IMITATION AND VOICE IMITATION (CONVERSION TASK)
83 (pretrained on 500K hours of data from over 1000 languages) to extract content features. It is designed based on the VITS architecture [84], and is trained on 2.8k hours sourced from Libri-light 74] and LibriTTS 85 released checkpoin 10 generate samples.
HierSpeech++ 5 utilizes MMS
We use the officially
LM-VC [22 It is an autoregressive hierarchical transformer that predicts SoundStream 63
codees from soft units similar HuBERT k-means tokens (54), trained the Libri-light dataset 74 We obtain the generated samples from the authors.
UniAudio [28]: It is an autoregressive transformer capable of performing multiple audio gen- eration tasks, using 500-cluster K-means tokens from HuBERT-base (that is pre-trained Lib- riSpeech 775 to predict their proposed acoustic codecs, with training data comprising approx- imately 80K hours of speech and 20K hours of other audio data. We the officially released checkpoin to generate samples.
m1
FACodec 17: adopts an auto-encoder and residual vector quantization based architecture. decouples the waveform into factorized attributes through ASR, FO prediction, and speaker classification tasks, trained on the Libri-light dataset [74]. We use the released checkpoint in Amphion 76 78
(which is implemented by the authors) to generate samples.
C.2 ZERO-SHOT STYLE IMITATION
86 (that is pre-trained on 60K hours of Libri-light 74) and fine-tuned on hours of LibriSpeech 77 to extract the one-hot text predictions from speech, i.e., Basr in our paper (Section 4.1) I). It adopts transformer encoder and a HiFi-GAN decoder to reconstruct waveforms conditioned on Taer. the accent labels, and FO, which is trained about 700 hours of accented corpus. We use 30 samples from its demo website [4] to evaluate, including English accents' conversions from British to American, British to Hindi, and Hindi to American.
ASR-AC [29]: It uses an ASR model based on wav2vec 2.
To achieve accent conversion, the authors first uses an conformer- ASR model (that trained by 40K hours their private corpus) to extract the hidden features (BNF). Then, they create about 300 hours of parallel conversion corpus based on commercial accented TTS system. Finally, they adopt an encoder-decoder transformer to learn the BNF's mapping between parallel corpus. We use 17 samples from its demo website 15 to evaluate, including English accents' conversions among American, British, Hindi, and Mandarin.
VoiceShop 20
Conv-Speak (2I]: The authors formulate accent conversion from source's content tokens to tar- get's 's content tokens. They propose to self-supervised pre-train content tokens like BART (87), in order to relieve the requirements of parallel data. They adopt the 500-cluster K-means of HuBERT-Base (that is pre-trained on 1K hours of LibriSpeech (75)) content tokens. The conversion model trained on about 600 hours of data, including about hour parallel data. We use 24 samples from its demo website I7 to evaluate, including English accents' conversions from Hindi and Mandarin to American.
To achieve emotion conversion, the authors design a recognition encoder to push its output (i.e., emotion-agnostic features) closely with phoneme transcriptions. The conversion model is based on a sequence-to-sequeee to-sequence decoder, that can reconstruct the Mel spectrogram conditioned on the amotion-agnoste features and emotion labels. The model is trained on about
Emovox 330
"https//github.ccm/// wtt.////wwweeeemmmmeeeeeeeeeeeee
http://hwww...ooo//////o/n////// hh caccoe facodec
https//huggingllceec
https://pytorch.oo///uddde
https//voicehhoppiigiibbbbbbbbbu
bttps//pwtocch..n//aaddoooo 10.
hww.ii/siisssiiiiiccccccooooonnn BASE
19

---

Published as a conference paper at ICLR 2025
80 hours data soureed from VCTK [88 and ESD 77. to evaluate, including emotion conversions from Neutral to Angry, Happy, and Sad.
We use 24 samples from its demo website
Notably, Vevo-Style (ASR) and Vevo-Style employ zero-shot approach to achieve style imitation which is rarely seen in existing research. Therefore, for these two, we use the aforementioned evaluation samples as the source and additionally prepare style references. Specifically, for accent imitation, we prepare references in three English accents: American, Hindi, and Mandarin, with four samples each (two males, and two females). For emotion imitation, we prepare references for three emotions: Angry, Happy, and Sad, with four samples each (two males, and two females). All these references are randomly sampled from ACCENT and EMOTION.
C.3 ZERO-SHOT VOICE IMITATION (SYNTHESIS TASK)
VALL-E [66:: is a classic AR model for zero-shot TTS. It utilizes the transformer to pre- diet EnCodec [ا] codecs. We use the released checkpoint in Amphion 76 78
generate
samples, which pre-trained on 45K hours of MLS English set [90].
Voicebox 277 It applies the flow matching transformer to both duration model and acoustic model. We reproduce it with the help of the authors.
VoiceCraft 777: It uses AR transformer predict EnCodec 89 codecs. Compared to
VALL-E, proposes token rearrangement and delaved stacking strategies to enhance the model learning. We use the officially released checkpoin to generate samples, which is pre-trained on 10K hours of Gigaspeech [I]]
Cosy Voice 124 It proposes a semantic tokenizer that is supervised by ASR task. It contains an AR transformer to predict the semantic tokens from text, and a flow-matching transformer to predict Mel spectrograms, We use the officially released checkpoin 21 to generate samples, which pre-trained on 171K hours of in-the-wild, multilingual, and private data.
MaskGCT [3] consists of two-stage discrete diffusion models. is based on the hidden features of w2v-hert 2. that pre-trained on 4.5M hours obtain the semantic tokens. Its TTS model is trained on 100K hours of in-the-wild and multilingual data 79 We use the released
checkpoint in Amphion 23 [76 78
to generate samples.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 EFFECT OF THE VOCABULARY SIZE OF THE VQ-VAE TOKENIZER
have already demonstrated the impact of different vocabulary sizes in the VQ- VAE codebook on Macoustic (i.c., disentanglement capability). In this section, we aim to present two complementary experimental results. First, explore the effects of wider range of vocabulary sizes (from the smallest at to the largest at 16,384) on the produced tokens. Second, we investigate the impact of various combinations of K, and K, on Tevo-Voice
In Section 4.1
D.1.1 EFFECT ON PHONETIC DISCRIMINABILITY
We explore the phonetic discriminability of different representations, inspired by AudioLM [92]. Specifically, we measure phonetic discriminability using the ABX eπrоr rate, a distance- metric that evaluates set of phoneme trigrams differing only in the central phoneme (e.g.. "bit" vs. "bet"). The ABX error rate assesses how often a random instance of trigram ("bit") closer to an instance B of another trigram ("bet") rather than another instance A of the trigram ("bit").
http:://kunzhou06646i.iilbbbbbbb ///////oxxx.////////oo demo/
http://ggthuo.com//uuuuulLLLLLLd Voice
http:///wgggnggaaaccccc/mmmmhnnn
20

---

Published as a conference paper at ICLR 2025
We evaluate scenarios where all three sounds A, B, and X are uttered by the same speaker (within- speaker) and where the same speaker utters and B but X comes from a different speaker (across- speaker). We calculate ABX using scripts provided with the Libri-light datase 24
[74,, employing
default settings and reporting scores obtained on the LibriSpeech dev-clean dataset
75 The results
Note that for K-means and VQ-VAE tokens, calculate the ABX error based on the centroid's s vector corresponding to each token.
displayed in Table 7
Table 7: ABX error rate (U) of different representations. (within/across speakers)
Vocab
128
256
1024
2048
4096
8192
16384
Repr
PPG features
6.1 7.0
18th layer features
7.6
17.2/
14.5/
12.0/
9.9%
8.8
7.8/
K-means tokens
19.8
17.7
14.0
11.5
10.3
9.0
16.4/
13.1/
12.7
/
13.0%
12.7/
12.7/
11.1/
10.0/
10.1/
10.4/
9.9/
10.0/
(Q-VAE tokens
18.2
14.6
14.2
14.9
14.8
14.6
13.0
11.8
12.0
12.4
11.9
11.9
PPG features ure obatined from HUBERT-ASR Large, while the others are from HuBERT-Lagge K-meens and VQ-VAE VAE tokens are quantized the 18th layer features HoBERI-Large.
From the table, we observe that: (1) PPG features demonstrate the best phonetic discriminability, highlighting the advantages fine-tuning with ASR tasks; (2) For K-means tokens, increasing the vocabulary size from 32 to 1024 continuously improves their phonetic discriminability, indicating an ongoing enhancement in their capacity to represent linguistic content; (3) For VQ-VAE tokens, see a gradual improvement in phonetic discriminability from 8 to 1024, but beyond 1024, this metric begins to converge. However, we know that as VQ- VAE tokens' vocabulary size increases from 1024 to 4096 to 16384, their style information still increases (as indicated by rising FPC scores in Table 2). From these observations, we can conclude that beyond 1024, the representation ability of VQ-VAE tokens for linguistic content stabilizes, and any increase in vocabulary size is likely allocated to storing style information such as FO; (4) Comparing K-means and VQ- VAE tokens, it's evident that VQ-VAE tokens are less sensitive to changes vocabulary size in terms of representing linguistic content (e.g.. VQ-VAE (32) and K-means (128) exhibit nearly identical ABX error rates), suggesting that a smaller vocabulary can suffice for a content tokenizer. Recent research has also delved into this aspect, attributing the differences to the distinct optimization algorithms used by the two methods [22]
D. 1.2 EFFECT ON VEVO-VOICE
We explore the effects of different (Kę, K.) combinations on Vevo- Voice, with the results presented
in Table X Our observations include: (1) A significant drop in intelligibility occurs when K, changes from 32 to 16, indicating that smaller vocabulary size for the content tokenizer leads to loss of
linguistic content information; (2) When K, decreases from 4096 to 1024, all metrics decline. We hypothesize that while reduction in K, might lessen the learning difficulty for Matyles smaller K2 also results in a decrease in the quality of the final generated audio for Macoustic-
Table 8: Effect of different (Ke, K2) for Jevo-Voice.
Content
Content-style Tokenizer (K.)
WER (L)
S-SIM
A-SIM
E-SIM
Tokenizer (K.)
(t)
(t)
32
4096
15.214
0.517
0.614
0.872
32
1024
18.523
0.502
0.609
0.860
16
4096
23.351
0.509
0.613
0.865
Vevo- Voice uses K, as 32 and K, as 4096. A-SIM evaluated only on ACCENT.
E-SIM is evaluated only on EMOTION.
21

---

Published as a conference paper at ICLR 2025
D.2 ZERO-SHOT VOICE IMITATION (S (SYNTHESIS TASK)
present the performance of Vevo-TTS in zero-shot imitation (synthesis) tasks. We have detailed its comparative performance against baselines on all four evaluation sets (AB, CV, ACCENT, and EMOTION) in Table 9 We can observe that: (1) In comparison with AR baselines, Vevo-TTS exhibits a clear advantage over VALL-E and VoiceCraft across various metrics on all datasets. Compared the state-of-the-ar Cosy Voice, although Vevo-TTS is trained solety 60K hours of Audiobook data, it performs better in some metrics such as Naturalness CMOS (AB, AC- CENT, EMOTION), Speaker S-MOS (EMOTION), and notably in style imitation-related metrics like Accent S-MOS and Emotion S-MOS. This demonstrates the high effectiveness of the AR TTS model implemented using the content-style tokens proposed in this paper. (2) When compared with Non-AR baselines, Vevo-TTS falls short on WER across all datasets compared to Voicebox and MaskGCT. This underscores the stability still needed in AR models, indicating significant room for improvement.
In Section 4.4
E SUBJECTIVE EVALUATION
E. BACKGROUND OF SUBJECTS
We hired dozens of subjects on a paid basis to complete the subjective evaluations. These individuals have extensive experience in providing subjective assessments of audio generated by AI models. They have lived in English-speaking countries for extended periods and are highly familiar with various common English accents, including American, British, Hindi, and Mandarin. Each audio sample in our evaluation was rated at least ten times.
E.2 METRICS AND QUESTIONNAIRES
We have developed an automated subjective evaluation interface. For each item to be evaluated, users will three components: the System Interface (i.e., the audio to be evaluated), the Question- naire, and the Scoring Criteria.
E.2.1 NATURALNESS MOS
System Interface One audio to be evaluated (with target text)
Questionnaire How human-like the speech in the clip? Does sound like a real human who is engaged in the topic, or does it sound like an AI that doesn't understand what is being said?
Scoring criteria (A perfect imitation of human speech), (Exceeds my expectations for AI voices), 3 (Meets my expectations for AI voices), 2 (A subpar representation of human speech), (Very poor artificial speech)
E.2.2 SPEAKER SIMILARITY MOS
System Interface One reference audio, One audio to evaluated
Questionnaire Ignore the content and audio quality, just pay attention to the voice of the person. How similar is the voice to evaluated compared the reference voice?
Scoring criteria 5 (Excellent, sounds like exactly the same person), 4 (Good, sounds like a sim- ilar person), 3 (Fair, sounds like a slightly similar person), 2 (Poor, sounds like a different person mostly), (Bad, sounds like completely different person)
E.2.3 ACCENT SIMILARITY MOS
System Interface One reference audio, One audio to be evaluated
Questionnaire Ignore the vocal characteristics (who is speaking), just pay attention to the accent of the speaker. Is the accent similar to the reference voice?
22

---

Published as a conference paper at ICLR 2025
Table 9: Results on zero-shot imitation (synthesis) task. (S-MOS: Similarity MOS)
Speaker SIM (1)
Naturalness CMOS (1)
Speaker S-MOS (1)
Model
AR?
Training Data
WER (L)
AB
Ground Truth
2.845
0.763
0.00
CosyVoice
24
171K hours, In-the-wild
3,647
0.727
-0.44
4.17 an1s
MaskGCT
13
100K hours, In-the-wild
3.841
0.781
-0.21
4.30
VALL-E 26
/
45K hours, MLS English [90
8.204
0,551
-0.95
3.27 4u2z
Voicebox 27
X
60K hours, Audiohonk
3.175
0.631
-0.55
eals
3.49
VoiceCraft
77
10K hours, Gigaspeech [9]]
4.737
0.570
-0.41
3.41
201s
VTV
60K hours, Audiobook
3.672
0.593
-0.31
3.58
CV
Ground Truth
1,426
0.723
0.00
VosyVoice 24
17Ik hours, till
3.500
0.627
0.11 *:.
3.88
MaskGCT
13
X
100K hours, In- tiithe
2.573
0.688
0.08
4.33 2014
VALL-E 26
45K hours, MLS English [90]
6.129
0.433
-1.02
2.68
Voicebux
X
60K hours, Audiobook
2.129
0,500
-0.12
2.91
VoiceCraft
77
10K hours, Gigaspeech [9]
6.353
0.446
-0.10
3.02 +0.1
Vevo-TTS
60K hours, Audiobook
2.687
0.513
-0.11 2039
3.83 2015
ACCENT
Speaker
Accent SIM (?)
Naturalness
Speaker S-MOS (t)
Accent S-MOS (t)
Model
AR?
WER (L)
SIM (t)
CMOS (1)
Ground Truth
10.903
0.747
0.633
0.00
Voice
24
6.660
0.653
0.640
0.10 +e.0
4.23 tuss
3,99
+u.
MaskGCT
X
6.382
0.717
0.645
0.23
4.24
tats
4.38
VALL-E 26
V
10.721
0.403
0,485
-1.04
3.12
2.77
1u4t
1:44
Voicebox 27
X
6.181
0.475
0.575
-0.55
3.93
#025
3.49
#029
VoiceCraft
77
10.072
0.438
0.517
-0.39
3.51
3.29
Vevo-TTS
9.673
0.544
0.579
0.12 +0.20
4.11
4.12
EMOTION
Speaker
Emotion
Naturalness CMOS (1)
Speaker S-MOS (t)
Emotion S-MOS (t)
Model
AR?
WER (1)
SIM (t)
SIM (?)
Ground Truth
11.792
0,673
0,936
0.00
Cosy Voice
24
10.139
0.575
0.839
-0.45
3.98
3.66 ius
inw
MaskGCT
13
x
12.502
0.600
0.822
-0.31
1017
4.07
int
3.76
1..s
VAL.--
V 15.731
0.396
0.735
-1.43
Less
2.52
Tnz
2.63 1as
Voicebox 27
X
12.647
0.451
0.811
-0.65
3.81
Inss
3.61
VoiceCraft 77
/
16.042
0.345
0.788
-0.60
3.42
3.52 1n7s
1e3
Vevo-TTS
14.458
0.466
0.840
-0.39
3,99
4.03
The best and the sccond best results among VALL-E, Voicebox, VoiceCraft, and Vevo-TTS are shown in bold and by underlined.
Scoring criteria 5 (Excellent, sounds like exactly the same accent), 4 (Good, sounds like a similar accent), 3 (Fair, sounds like a slightly similar accent), 2 (Poor, sounds like different accent mostly), 1 (Bad, sounds like a completely different accent)
E.2.4 EMOTION SIMILARITY MOS
System Interface One reference audio, One audio to be evaluated
Questionnaire Ignore the vocal characteristics (who is speaking), just pay attention to the emotion of the speaker. Is the emotion similar the reference voice?
23

---

Published as a conference paper at ICLR 2025
Scoring criteria 5 (Excellent, sounds like exactly the same emotion), 4 (Good, sounds like similar emotion), 3 (Fair, sounds like a slightly similar emotion). 2 (Poor, sounds like a different emotion mostly), 1 (Bad, sounds like completely different emotion)
E.2.5 PROSODY SIMILARITY MOS
System Interface
One reference audio, One audio to be evaluated
Questionnaire Ignore the vocal characteristics (who speaking), just pay attention to the speak- ing style (how to speak). Is the speaking style (pace, tone, stress, intonation, pitch, emotion) consis- tent and identical with the reference voice?
Scoring criteria 5 (Excellent, sounds like completely identical style), 4 (Good, sounds like highly consistent style), 3 (Fair, sounds like a slightly similar style), 2 (Poor, sounds mostly like a different style), (Bad, sounds like completely different style)
E.2.6 NATURALNESS CMOS
System Interface One reference audio, One audio to be evaluated (with target text)
Questionnaire Compared the reference audio, is the quality and the human likeness of the audio to be evaluated better or worse?
Scoring criteria (Much worse), -2 (Worse), (Slightly worse), 0 (No preference), (Slightly better), 2 (Better), 3 (Much better)
E.2.7 ACCENTEDNESS CMOS
System Interface
One accent label, One reference audio, One audio to be evaluated
Questionnaire Assume that we want to generate the voice whose accent is faccent label]. Com- pared to the reference audio, is the accentedness of the audio to be evaluated better or worse?
Scoring criteria (Much worse), -2 (Worse), (Slightly worse), 0 (No preference), (Slightly better), 2 (Better), 3 (Much better)
E.2.8 EMOTIVENESS CMOS
System Interface
One emotion label, One reference audio, One audio to be evaluated
Questionnaire Assume that want to generate the voice whose emotion is [emotion label]. Compared to the reference audio, is the emotional expressiveness of the audio to be evaluated better worse?
Scoring criteria -3 (Much worse), -2 (Worse), -1 (Slightly worse), 0 (No preference), (Slightly better), 2 (Better), (Much better)
F ETHICS STATEMENT
As with other powerful Al innovations, we recognize this technology brings the potential for misuse and unintended harm. We will build a highly effective classifier that can distinguish between authentic speech and audio generated with Vevo mitigate these possible future risks.
24


Reviews:
[{"review_id": "YeVEZ2sIr8", "reviewer": "Reviewer_EEAE", "summary": "The paper introduces Vevo, a zero-shot voice imitation framework with controllable timbre and style, featuring a fully self-supervised approach that progressively decouples timbre, style, and linguistic content of speech using VQ-VAE tokenizers with an information bottleneck technique. Vevo outperforms existing models in accent and emotion conversion, particularly excelling in zero-shot tasks, demonstrating its versatility and robustness.", "questions": "1. The paper notes that \"both the content and content-style tokenizers are trained on a 100-hour subset randomly sampled from the full 60K-hour dataset.\" Training on such a small subset raises questions about the model's generalization and robustness. How are these ensured given the limited training data?\n\n2. Has the model been tested on datasets beyond audiobooks to evaluate its zero-shot voice generation capabilities, including timbre imitation, style imitation, and voice imitation?\n\n3. Regarding the parameters in the conditional flow matching model, how is the \"randomly mask ratio\" determined, especially given that the paper reports a range of \"70%-100%\"?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces a unified speech synthesis framework, Vevo, capable of handling zero-shot timbre, style, and voice imitation tasks. It provides clear and precise definitions of timbre and style and achieves its objectives through a two-stage modeling strategy: Content-Style Modeling and Acoustic Modeling.\n\n2. By experimenting with the codebook size of self-supervised speech representations quantized via VQ-VAE, the paper follows the theory of information bottleneck and implements a progressive information filtering approach. This results in a simple yet efficient self-supervised method to gradually decouple timbre, style, and linguistic content.\n\n3. Experiments indicate that Vevo matches or exceeds the performance of comparable state-of-the-art speech generation models across various zero-shot scenarios, showcasing its strong generalization and versatility.", "weaknesses": "1. The self-supervised VQ-VAE tokenizer for content and content-style information requires meticulous parameter tuning, such as continually adjusting the codebook size. Balancing the performance between the content and content-style tokenizers involves trade-offs, making it a suboptimal choice with potential risks. It might be beneficial to compare this with supervised tokenizer models, such as extracting content and content-style tokens directly using open-source models like FACodec \\[1], to evaluate the impact on model performance and make more informed decisions.\n\n2. In the Content-Style to Acoustic stage, the paper utilizes a conditional flow matching model with a masking strategy, relying on unmasked portions of mel spectrograms to guide the modeling of masked timbre, linguistic content, and acoustic style. This stage may encounter issues of information leakage, where the model might learn the speaker's context from the provided unmasked mel portions during training, aiding in better generation of masked mel segments. During inference, while aiming to replicate the timbre reference, the model might inadvertently also learn consistent style information.\n\n\\[1]: Ju Z, Wang Y, Shen K, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models[J]. arXiv preprint arXiv:2403.03100, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JyK2Tny94c", "reviewer": "Reviewer_1b8h", "summary": "This paper builds upon the foundational HuBERT representation model and explores the disentanglement of content tokens and content-style tokens by analyzing VQ codebook spaces of varying sizes. Based on this disentanglement, the authors propose a cascaded generation framework called Vevo (AR Transformer combined with Flow Matching), which successfully performs zero-shot timbre, style, and voice imitation tasks.", "questions": "1. I am unclear about the specific implementation details of the Vevo-Timbre and Vevo-Style versions. \n\n2. Will you be releasing the complete training and inference code, as well as the model weights?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Exploring the unified approach to zero-shot timbre, style, and voice imitation tasks through disentangled representations is a meaningful attempt, although similar efforts have been made previously.\n2. The novel approach of disentangling timbre, style, and content by analyzing different sizes of VQ codebook spaces is interesting, although it is difficult to provide strict evidence of its effectiveness.", "weaknesses": "1. It is challenging to guarantee strictly disentangled results when analyzing different sizes of VQ codebook spaces for timbre, style, and content.\n\n&nbsp; &nbsp; 1.1. The results may be biased by hyperparameter tuning in downstream tasks.\n\n&nbsp; &nbsp; 1.2. The authors should include comparisons with other disentanglement methods in Table 2, such as label-based    disentanglement techniques.\n\n&nbsp; &nbsp; 1.3. The differences in evaluation metrics between various codebook sizes, such as a SIM score improvement from 40 to 44, are minimal and insufficient to claim successful disentanglement.\n\n&nbsp; &nbsp; 1.4. The performance metrics in Table 2 for different codebook sizes show substantial gaps compared to the ground truth (GT).\n\n2. The overall cascaded generation framework of Vevo is not highly innovative, as similar frameworks have already been proposed, such as SeedTTS and CosyVoice.\n\n3. The unified definition of zero-shot timbre, style, and voice imitation tasks is not new, as seen in prior works like ControlSpeech and StyleFusion TTS.\n\n4. The experimental results do not demonstrate significant performance gains. Furthermore, the baselines are relatively weak. At least, comparisons with open-source models like CosyVoice should be included. Additionally, comparing to FaCodec is unfair; a comparison with NaturalSpeech3 would be more appropriate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OLx11GofeX", "reviewer": "Reviewer_z29c", "summary": "This paper proposes a voice imitation approach supporting style information transfer at different granularities, including global timbre, accent, and emotion. The authors create varying sizes of information bottlenecks by VQ-VAE with different codebook sizes on self-supervised speech representations, obtaining speech representations with different levels of information. Upon these representations, the authors build a two-stage speech generation model composed of autoregressive and flow matching modules. Experiments demonstrate the effectiveness of this approach in voice imitation.", "questions": "Did the authors validated their theory regarding the relationship between VQ-VAE codebook size and the information contained in representations on other self-supervised speech representations (such as wav2vec2, w2vbert)? Are their conclusions consistent?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using codebooks of varying sizes to control the width of the information bottleneck is impressive, and the study on the order that information are filtered out with the shrinking codebook sizes are highly meaningful.\n\n2. The experimental results are promising, and the demo sounds impressive.", "weaknesses": "I do not see any significant flaws in the paper for now. I may raise new questions based on the feedback from other reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hEIXFQdAvO", "reviewer": "Reviewer_PTWX", "summary": "The paper addresses challenges in voice imitation for speech generation, proposing Vevo—a zero-shot framework with controllable timbre and style. It operates in two stages and uses a self-supervised dicoupling approach with VQ-VAE tokenizer. Trained on 60K hours of audiobook data, Vevo outperforms existing methods and shows effectiveness in zero-shot conversion and text-to-speech, demonstrating generalization and versatility.", "questions": "1. The content and content-style tokenizers are trained only on 100-hour data. Is the amount of data sufficient? Has a comparison of different amounts of data been made?\n2. In section 4.1, \"When K reduces further to 32, in addition to timbre, most style information is also filtered out\", FPC drops from 0.764 to 0.706, that's not a big difference. How can it be proved that the style has been removed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Vevo can do both VC and TTS tasks and can simultaneously control both timbre and style.\n2. The experiments are very detailed and the conclusions prove the effectiveness of the proposed model.\n3. This paper is well structured and easy to read, and the demo sounds good.", "weaknesses": "Although the paper has done a lot of research on the comparison and selection of representations, it seems that there is relatively little innovati", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)