Przetłumacz mi treść artykułu na polski i zwróć mi go w latex: # VOICECRAFT:Zero-Shot Speech Editing and Text-to-Speech in the Wild

Puyuan Peng1 Po-Yao Huang2 Shang-Wen Li2

Abdelrahman Mohamed3 David Harwath

1The University of Texas at Austin 2FAIR,Meta 3Rembrand

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>

## Abstract

We introduce VOICECRAFT, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts.VOICECRAFT employs a Transformer decoder architecture and introduces a token rearrange-ment procedure that combines causal mask-ing and delayed stacking to enable generation within an existing sequence. On speech editing tasks,VOICECRAFT produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model out-performs prior SotA models including VALL-E and the popular commercial model XTTS v2. Crucially, the models are evaluated on challenging and realistic datasets,that consist of diverse accents,speaking styles,recording conditions, and background noise and music,and our model performs consistently well com-pared to other models and real recordings. In particular, for speech editing evaluation,we introduce a high quality, challenging,and re-alistic dataset named REALEDIT. We encour-age readers to listen to the demos at https://jasonppy.github.io/VoiceCraft_web.

## 1 Introduction

We introduce VOICECRAFT,a Transformer-based neural codec language model (NCLM) that per-forms infilling generation of neural speech codec to-kens autoregressively conditioned on bidirectional context. VOICECRAFT achieves state-of-the-art (SotA) performance on both speech editing (shown inFig.1)and zero-shot TTS. Our method is based on a two-step token rearrangement procedure that consists of a causal masking step and delayed stacking step. The causal masking technique is in-spired by the success of causal masked multimodal

'Data, code, and model weights are available at https://github.com/jasonppy/VoiceCraft.

<!-- "Ifound the amazing VoiceCraft model" Edited speech: Edited:48% VoiceCraft Human Preference On Naturalness I found the amazing VoiceCraft model Original:52% (Target) "Ifound this um incredible model" (Original)  -->
![](https://web-api.textin.com/ocr_image/external/acca25a59506c259.jpg)

Figure 1: Speech editing with VOICECRAFT.Human listeners prefer VOICECRAFT edited speech over the original real recording 48% of the time in side-by-side naturalness comparison (details in §5.3)

model in joint text-image modeling(Aghajanyan et al., 2022), and our proposed technique works for speech codec sequences,which enables autore-gressive generation with bidirectional context. In addition,we further integrate causal masking with delayed stacking (Kharitonov et al., 2021a;Copet et al., 2023) as our proposed token rearrangement procedure, to ensure efficient multi-codebook mod-eling.

To evaluate speech editing, we manually crafted a first-of-its-kind,realistic,and challenging dataset named REALEDIT. REALEDIT consists of 310real world speech editing examples, with wave-forms sourced from audiobooks (Zen et al.,2019),YouTube videos (Chen et al., 2021a), and Spo-tify podcasts (Clifton et al., 2020),and duration ranging from 5 seconds to 12 seconds. To create the target transcripts,the transcripts of the source speech are edited in such a way that the edited transcripts remain grammatically correct and are semantically coherent. The dataset is designed to cover a wide range of editing scenarios, includ-ing insertion, deletion, substitution,and multi-span editing,with the length of the edited text ranging from 1 word to 16 words. Compared to commonly used speech synthesis evaluation datasets that only contain audiobooks such as VCTK (Yamagishi et al., 2019), LJSpeech (Ito and Johnson,2017),and LibriTTS (Zen et al., 2019), REALEDIT is

more challenging in that the recordings have di-verse content, accents, speaking styles, recording conditions, and background sounds.We believe that the realism and diversity of REALEDIT makes it a reliable indicator of the practicality of speech editin models in the real world.

In the subjective human listening tests, VOICE-CRAFT significantly outperforms prior SotA speech editing model on REALEDIT. Importantly,the edited speech produced by VOICECRAFT is nearly indistinguishable from the original unedited recording in terms of naturalness. We found that VOICECRAFT generalizes well to zero-shot TTS without any finetuning,achieving SotA perfor-mance on a dataset comprised of audiobooks and YouTube videos, outperforming strong baselines including reproduced VALL-E (Wang et al.,2023a)and the popular commercial model XTTS v2 (CO-QUI, 2023). In summary, our contributions are:

1.We introduce VOICECRAFT,a neural codec language model for speech editing that gener-ates synthesized speech that is nearly indistin-guishable from in-the-wild recordings accord-ing to human listeners. We also release the code and model weights for VOICECRAFT.

2.We show that VOICECRAFT generalizes well to zero-shot TTS without finetuning.

3.We release a high quality, challenging, and realistic speech editing evaluation dataset REALEDIT.

## 2 Related Work

Neural codec langauge models (NCLM) and zero-shot TTS.Tokenizing speech signals into sequences of learnable, discrete units and then training a language model on the resulting unit sequences was initially proposed in the context of textless NLP (Hsu et al., 2021; Lakhotia et al.,2021;Kharitonov et al., 2021b; Nguyen et al.,2022),where the goal is to perform NLP tasks directly on spoken utterances without the need to first transcribe the speech into text. Recently,NCLMs that operates on tokens from Residual vec-tor quantization (RVQ)-based models (Zeghidour et al.,2021;Defossez et al., 2022) attract increased attention due to its high quality generation. For example, AudioLM (Borsos et al., 2022a) exhibits strong performance on long-term coherent speech continuation. Zero-shot TTS is a task where a model needs to synthesize speech in a target voice

which was unseen during training, given only the target transcript and a short reference recording of the target voice. Framing zero-shot TTS as transcript-conditioned speech continuation, VALL-E (Wang et al., 2023a) and Spear-TTS (Kharitonov et al.,2023) are the first applications of NCLMs on this task, significantly outperforming non-NCLM approahes. Zhang et al. (2023) extends VALL-E to cross-lingual TTS.Guo et al. (2022);Yang et al.(2023);Liu et al. (2023); Ji et al.(2023);Lyth and King (2024) adapt NCLMs style-controlled speech synthesis. Song et al. (2024); Du et al. (2024b) en-hance phoneme alignment in NCLMs to reduce er-ror. Wang et al. (2023b) proposes a unified NCLM for both speech generation and recognition tasks.Borsos et al. (2023) proposes an efficient paral-lel decoding method. Jiang et al.(2024) proposes disentangled timbre and prosody modeling,where the latter is modeled with a NCLM. NCLMs have also been successfully applied to other audio do-mains. Kreuk et al. (2022) applies NCLM to sound effects generation, and Agostinelli et al. (2023);Donahue et al. (2023); Garcia et al. (2023);Copet et al. (2023) use NCLMs for music generation.

Speech editing. This task requires a model to alter words or phrases within an utterance to match a target transcript, but the regions of the original speech not targeted for editing must re-main unchanged (see Fig.1 for an example).Early methods achieve text-guided speech insertion and substitution by combining a single speaker TTS model and a voice conversion model to generate desired speech segment, which is then concate-nated with unedited part (Jin et al.,2017).Since the generation is not conditioned on the unedited part of the speech, the result sounds unnatural due to prosody mismatch and boundary artifacts (Morri-son et al.,2021).More recent speech editing mod-els have attempted to condition their generation on surrounding speech context.Tan et al.(2021)uses two unidirectional LSTM models with bidirec-tional fusion.Wang et al.(2022); Bai et al.(2022);Borsos et al. (2022b) uses the masked reconstruc-tion objective with Convolutional or Transformer models to further improve contextualization. Flu-entSpeech (Jiang et al., 2023b) is a diffusion-based speech editing model that achieves SotA perfor-mance on speech editing on LibriTTS and VCTK.

The research community starts to investigate the possibility of having a unified model for both zero-shot TTS and speech editing. Yin et al.(2022);Jiang et al. (2023a) propose modular models for

the two tasks, while our model is end-to-end. Con-current work SpeechX (Wang et al., 2023c) adapt VALL-E by prompt tuning for a range of tasks including speech editing and zero-shot TTS,but no human evaluation is conducted in their paper.Concurrent work UniCATS (Du et al., 2024a) is a diffusion-based modular model for the two tasks.However their model is only evaluated on masked speech reconstruction of span length less than 2 sec-onds, while our model is evaluated on as much as 16 words editing. Voicebox (Le et al., 2023) is a re-cent flow matching based model capable of a wide range of tasks including speech editing and zero-shot TTS.However the speech editing capability is not evaluated in their paper, and only shown in their demo page. We therefore compare our model's edit-ing results with Voicebox's on our demo page using on the same examples from their demo page.

## 3 Method

VOICECRAFT casts both sequence infilling(for speech editing) and continuation (for zero-shot TTS) as a simple left-to-right language mod-eling by rearranging neural codec's output to-kens. The rearrangement involves two steps: (1)causal masking (§3.1) to enable autoregressive continuation/infilling with bidirectional context and (2) delayed stacking (§3.2) to ensure efficient multi-codebook modeling.VOICECRAFT employs decoder-only Transformers and is trained with an autoregressive sequence prediction (§3.3).We in-troduce the inference setup for speech editing and zero-shot TTS in §3.4.

### 3.1 Rearrangement Step 1: Causal Masking

As shown on the left hand side of Fig.2,given a continuous speech waveform as input,we first use Encodec(Defossez et al., 2022) to quantize it into a T by K codec matrix X,where T is the number of temporal frames, and K is the number of RVQ codebooks. X can be written as$( _ { 1 } , \cdots , _ { T } ) ,$whereX, is a vector of length K representing the codes from different codebooks at time step t, and we assume that code from codebook k models the residual from codebook k-1.During training,our goal is to randomly mask some span of tokens $( X _ { t _ { 0 } } , \cdots , X _ { t _ { 1 } } )$,and then autoregressively predict these masked tokens conditioned on all of the un-masked tokens. This is a problem when$t _ { 1 } < $,be-cause we cannot condition on future outputs when performing autoregressive generation.We need to

modify the masking on X so that it is causal,by moving the span to be masked to the end of the sequence,so that when infilling these tokens the model can condition on both past and fuure un-masked tokens (Aghajanyan et al., 2022;Donahue et al., 2020;Bavarian et al.,2022).

The procedure outlined above can be trivially ex-tended to multiple masked spans by simply moving all masked spans to the end of the sequence.The number of spans to be masked n is sampled from Poison(λ), and then for each span,we sample a span length l \~Uniform(1,L). Finally,we ran-domly select the locations of the spans within X under the constraint that they do not overlap with each other. The selected n spans are then replaced with mask tokens$( M _ { 1 } ) , \cdots , ( M _ { } ) $.The original tokens within these masked spans aremoved to the end of the sequence X,with each span preceded by its corresponding mask token.

Consider this example:let$X = ( X _ { 1 } , \cdots , X _ { 6 } )$and imagine we wish to mask a single span from $X _ { 2 }$to$X _ { 4 }$.The original sequence X is rearranged into$Y = ( Y _ { 1 } ; ( M _ { 1 } ) ; Y _ { 2 } ; ( M _ { 1 } ) ; Y _ { 3 } ; )$,where$Y _ { 1 } =$ $( X _ { 1 } ) ,$ $Y _ { 2 } = ( X _ { 5 } , X _ { 6 } ) ,$and$Y _ { 3 } = ( X _ { 2 } , X _ { 3 } , X _ { 4 } ) .$We call Yi and Y2 the unmasked spans,and$Y _ { 3 }$the masked span. An end of span or EOS token is added to the end of each masked span (in this example at the end of Y3),and an end of utterance or EOU token is added to the end of the utterance(i.e.$Y _ { 2 } $.For simplicity,we do not explicitly denote these special tokens and assume they are part of the spans.

### 3.2 Rearrangement Step 2:Delayed Stacking

After the causal masking token rearrangement,each timestep of the rearranged matrix Y is vec-tor of K tokens. Copet et al. (2023) observed that when performing autoregressive generation over stacked RVQ tokens, it is advantageous to apply a delay pattern so that the prediction of codebook k at time t can be conditioned on the prediction of codebook k - 1 from the same timestep. We take a similar approach which we describe here. Assume a span$Y _ { S }$is of shape $L _ { s } \times K$.Applying the delay pattern rearranges it into$Z _ { s } = ( Z _ { s , 0 } , Z _ { s , 1 } , \cdots ,$ $Z _ { S , L _ { S } + K - 1 } )$,where $Z _ { s , , \in [ L _ { s } + K - 1 ] i s d e f \in e d a s ^ { 2 } :$

$Z _ { s , t } = ( Y _ { s , t , 1 } , Y _ { s , t + 1 , 2 , \cdots , Y _ { s , t - K + 1 , K )$ (1)

where$Y _ { s , t - k + 1 , k }$denotes the token located at coor-dinate(t-k+1,k)in matrix$Y _ { s } ,$i.e.the kth code-book entry at the(-+1)th timestep. To make

$^ { 2 } [ N ]$represents integer set{0,1,⋯,N}

<!-- x1 X2 X3 X4 Xs X6 X X1 X2 X3 x4 X5 Xg Encodec Token Rearrangement Modeling X1 X2 X3 X4 Xs X6 Encoder Encodec x1 X2 X3 X4 Xs Xg Step 1:Causal Masking span1 span2(to be masked) span3 Decoder span2(to be masked} span3 Notation: CB1 x end-of-span s Rearrange X1 Xs X6 U X2 X3 X4 5 codec autoregressive generation CB2 x end-of-utterance u Y x1 Xg Xg u X2 X3 X4 S tokens from different CB3 X mask token Transformer X1 Xs Xg U X3 X4 S codebooks CB4 empty token X1 X5 Xg u X2 X3 X4 Step 2:Delayed Stacking Decoder V 0I S X1 m1 u X3 S Phonemized s Z X1 Xs Xg U X2 5 transcript X3 X4 ++. x1 Xs X6 u s x1 X2 5 rearranged codec tokens  -->
![](https://web-api.textin.com/ocr_image/external/880580bc407e3e2e.jpg)

Figure 2: An example of the token rearrangement procedure and modeling framework. The rearrangement procedure involves two steps: (1) Causal masking, where masked spans are replaced with mask tokens and moved to the end,and (2) Delayed stacking, where tokens are shifted in the time dimension based on their codebook index.

sure that∀$\in [ L _ { s } + K - 1 ] ,$ $Z _ { s , t }$contains K valid tokens, we introduce a special learnable [empty]token and define$Y _ { s t - k + 1 k }$={Δ} [empty],Vt E {s:s$< k \cup s - k + 1 > L _ { s } \}$.Note that the mask tokens are not part of any span and are not changed during delayed stacking. We define the resulting matrix ofdelayed stacking $Z = ( Z _ { 1 } ,$ $\langle M _ { 1 } \rangle ,$ $Z _ { 2 } ,$ $\langle M _ { 1 } \rangle , \cdots ,$〈M$\frac { s - 1 } { 2 } > ,$ $Z _ { S } )$(as-suming Y consists of S spans). See the diagram for Z in Fig. 2 for an illustration.

### 3.3 Modeling

As shown in the right hand side of Fig. 2, we use a Transformer decoder to model Z autoregressively,conditioned on transcript of the speech W.There-fore,the input to the decoder is [W;Z],where“;”denotes concatenation. At timestep t of span s in codec matrix Z, the model predicts all K tokens of$Z _ { s , t }$simultaneously,by using K MLP heads to project the transformer's final hidden state to K sets of logits, one for each of the K codebooks.Note that the prediction is conditioned on transcript W,and all tokens in Z before$Z _ { S _ { 1 } t }$,denoted as$H _ { s , t } .$Mathematically,the Transformer decoder models the factorized conditional distribution of Z:

$P _ { \theta } ( Z \vert W ) = \prod _ { s } I I I _ { t } ^ { W } ( Z _ { s , t } \vert W , H _ { s , t } )$ (2)

$= \prod _ { s } \prod _ { t } \prod _ { k = 1 } ^ { K } P _ { \theta } ( Z _ { s , t , k } \vert W , H _ { s , t } )$ (3)

Where θ represent the parameters of the model.Equation 2 is the autoregressive factorization across time, while Equation 3 is the factorization across codebooks given an independence assump-

tion-given W and$H _ { s , t } ,$the K RVQ codes in$Z _ { , t }$are assumed to be independent of each other. We argue in appendix D that this assumption is mild.

With the token level probability formulation in Equation 3,we derive the training loss as the neg-ative log likelihood$C ( \theta ) = - \log _ { R } [ P _ { \theta } ( Z \vert W ) =$ $- \sum _ { k = 1 } ^ { K } C _ { k } ( \theta ) .$Empirically,we found that weight-ing the first residual codebooks more than the lat-ter codebooks leads to better performance,and therefore our final loss i $; C ( \theta ) = \sum _ { k = 1 } ^ { K } \alpha _ { k } C _ { k } ( \theta ) $where$( \alpha _ { k } ) k = 1$are tunable hyperparameters.Note that we follow Aghajanyan et al. (2022)and cal-culate the prediction loss on al tokens (not just the tokens in the masked spans),except for mask tokens and [empty] tokens.

### 3.4 Inference

Speech Editing.The setting for speech editing is the following: we have a speech recording R and its transcript W, and we want the model to mod-ify only the relevant spans of R so that it matches the target transcript W'.We assume that W' is an edited version of W, where some words have been inserted, substituted, or deleted. This task is almost exactly the same as the training task, with two dif-ferences: 1) during training, the input transcript is simply the transcript of the original recording W,while during inference it is a modified transcript W' 2) during training, the spans to be masked(i.e.edited) are chosen randomly. During inference,we select them by comparing the original transcript and the target transcript to identify the words that should be masked out, and then use the word level forced alignment of the original transcript to iden-tify the codec token spans that correspond to these

Table 1: Examples of the speech editing dataset REALEDIT.More examples are shown in table 8.


| Edit Types | Original | Edited |
| -- | -- | -- |
| deletion | I wrote the title of the course many years ago,ah, I wrote the title when I created this course.when I created this course. | I wrote the title of the course many years ago,ah, I wrote the title when I created this course.when I created this course. |
| insertion | And we're at this point. | And we're all extremely excited at this point. |
| substitution,substitution | See why it's extremely valuable to it's kind of likeit's kind of like having a wall hack to watch a demo. |  See why it's extremely important right? it's kind of like having a rough time to watch a demo. |


words to be masked. To ensure a smooth transition between the edited speech and the unedited speech,the neighboring words surrounding the span to be edited also need to be slightly modified in order to model co-articulation effects. Therefore, we spec-ify a small margin hyperparameter e,and extend the mask span length by e on both the left and right sides3. During autoregressive generation,we feed the model the target transcript with all unmasked spans, with mask tokens inserted in the locations where the edits should take place.We then have the model autoregressively continue this sequence,whereby it fills in the masked spans. The generated codec tokens are then spliced back into their correct location in the utterance, and we map the complete codec token sequence back to a waveform using the Encodec decoder network.

Zero-shot TTS.As we previously noted,zero-shot TTS for our model is straightforward because it simply corresponds to performing an insertion edit at the end of the original utterance. In this case, the model is provided a voice prompt with its transcription, as well as the target transcript of the speech to be generated. The three inputs are concatenated together and fed to the model,after which it generates the codec sequence of the target transcript autoregressively.

## 4 REALEDIT: a realistic and challenging speech editing dataset

To support as realistic an evaluation as possible,we constructed a first-of-its-kind dataset of 310manually-crafted speech editing examples.Each example consists of a tuple: (original audio, origi-nal transcript, edited transcript). The dataset con-tains 100 utterances from LibriTTS (dev-clean and dev-other) (Zen et al., 2019), 100 utterances from YouTube(from Gigaspeech testset)(Chen et al.,2021a) and 110 utterances from the Spotify Pod-

3for substitution and deletion, the spans that are to be masked are just those words that are different from the target plus the margin; for insertion, the spans are just left and right margin spanning from the middle of the two words where the insertion happens

cast dataset (Clifton et al., 2020). We manually checked the utterances for accuracy,then had na-tive English speakers revise them to create edited transcripts.For each utterance,we determine the type of modification using predefined probability distributions of editing type,number of disjoint spans to be edited, and editing span length. Specifi-cally,we study the following categories: 1) number of edited spans: 1 or 2; 2) type of edits: insertion,deletion and substitution; 3) editing span length:short (1-2 words), medium (3-6 words), long (7-12 words). Crucially,a edited transcript must be grammatically correct and semantically coher-ent. Examples of the dataset are shown in table 1and 8, and statistics are shown in table 2,

Table 2: Dataset statistics for speech editing evaluation.Note that for 2-span editing,each example is edited using 2 of the 3 edit types.


| length type | Insert. | Delet. | Substi. | Total |
| -- | -- | -- | -- | -- |
| 1-2 words(1 span) | 8 | 17 | 38 | 63 |
| 3-6 words(1 span) | 22 | 24 | 79 | 125 |
| 7-12 words(1 span) | 15 | 11 | 56 | 82 |
| 1 span total | 45 | 52 | 173 | 270 |
| 2 spans total | 13 | 13 | 54 | 40 |


## 5 Experiments

### 5.1 Setup

Data. Gigaspeech training set (Chen etal.,2021a)is used as the training data,which contains 9k hours of audiobooks,podcasts,and YouTube videos at 16kHz audio sampling rate. Audio files that shorter than 2 seconds are dropped.For ablation studies,we use the masked reconstruction task, and a 1000-utterance random subset of Gigaspeech validation set as the testing utterances (detailed in §C).For speech editing evaluation, we use the proposed REALEDIT dataset. For zero-shot TTS evalua-tion, we constructed a 250 prompt-transcript paired dataset from LibriTTS (Zen et al.,2019)and the YouTube portion of the Gigaspeech test set,with half of the examples drawn from each dataset.The length of each voice prompt is kept as close as pos-

sible to 3 seconds long, with the constraint applied that we only cut the audio between complete words.The transcript s a concatenation of the transcript of the voice prompt and the target transcript.The target transcripts are chosen from different utter-ances spoken by the same speaker as the prompt,and range from 8 to 40 words in length. We only select utteranceswith a WER lower than 15% by Whisper medium.en(Radford et al., 2022).

Model. Encodec (Defossez et al., 2022) is used as the speech tokenizer, which has 4 RVQ code-books each with vocabulary size of 2048,and a codec framerate of 50Hz on 16kHz recordings.(see §C for detailed config). To choose the number of spans to mask in training, we use a Poison(1) distri-bution truncated to a minimum of 1 and maximum of 3. Span lengths are sampled from Uniform(1,600) i.e. the masked speech can be as long as 12 seconds. At each time step, the embeddings of codes from different codebooks are summed(Wang et al., 2023a), then added by sinusoidal positional encoding (Vaswani et al., 2017), before being fed to the transformer. Text transcripts are phonemized based on the IPA phoneset using the toolkit pro-vided by Bernard and Titeux (2021). Our main VOICECRAFT model has 16 transformers layer with hidden/FFN dimensions of 2048/8192,and 12 attention heads. The output of the last layers are fed to four separate 2-layer MLP modules to get prediction logits. Our Main model has 830M parameters and codebook weight hyperparameters a is set to be (5,1,0.5,0.1). Ablations on model sizes and codebook weights are shown in §5.2.

Training and inference. The training of the En-codec model largely follows the setting in Copet et al. (2023), detailed in §C. To train VOICE-CRAFT, we used the ScaledAdam optimizer and Eden Scheduler proposed in (Yao et al.,2024)with a base learning rate of 0.05, batch size of 400k frames(i.e.133.2 minutes),and total training step of 50k with gradient accumulation. The training of the 830M VOICECRAFT model took about 2weeks on 4 NVIDIA A40 GPUs. More details can be found in §C. We compare the performance of ScaledAdam and AdamW in §A.1. For infer-ence, we use Nucleus sampling (Holtzman et al.,2020)withp=0.8and a temperature of 1 for all experiments. Due to the stochasticity of autoregres-sive generation, via manual inspection we found that while most of the time the model produces natural sounding speech, it sometimes produces ex-cessively long silence or drags out certain sounds.

Table 3: Effect of scaling model sizes and codebook re-weighting.Lower is better for all metrics.


| Params |  Weights | WER | MCD | FO | Energy |
| -- | -- | -- | -- | -- | -- |
| 120M | (1,1,1,1) | 10.18 | 8.75 | 78.49 | 3.22 |
| 120M | (5,1,0.5,0.1) | 7.75 | 8.31 | 87.74 | 3.54 |
| 430M | (1,1,1,1) | 7.87 | 8.22 | 70.05 | 3.17 |
| 430M | (5,1,0.5,0.1) | 7.30 | 8.13 | 73.41 | 3.19 |
| 830M | (5,1,0.5,0.1) | 6.68 | 8.05 | 67.81 | 3.12 |


We found that happens when the codec token gen-eration gets stuck in a repeating loop. To resolve it, we use a simple heuristic: for each input utter-ance we generate several different output utterances and throw away the longest outputs. Specifically for speech editing, we run inference 10 times with different margin parameters, stepping e up from 0.05 to 0.14 in 0.01 increments. The 4 longest outputs are discarded, and then we randomly se-lect one sample from the remaining 6 outputs. For zero-shot TTS, we reduce the probability of gen-erating the same token in consecutive timesteps in proportion to how many times that token was con-secutively generated in the immediately preceding timesteps. In addition, we generate 5 samples with different random seeds, and select the shortest for TTS evaluation. The sample selection process is completely automatic and unsupervised (i.e. no human intervention or ASR scoring).

Baselines. For speech editing, we compare VOICECRAFT with the diffusion-based model Flu-entSpeech (Jiang et al., 2023b) which is the cur-rent open-source SotA model for speech editing.Since the original FluentSpeech model is trained on LibriTTS,for a fair comparison, we took the official GitHub repo and trained the model on Gi-gaspeech. Please find more details in §C. For zero-shot TTS, we compare our VOICECRAFT with VALL-E (Wang et al., 2023a), XTTS v2 (CO-QUI,2023),YourTTS (Casanova et al., 2021),and FluentSpeech.Since the original VALL-E is not open-sourced, we use the code from the popular open-source implementation by Li (2023),and also trained the model on Gigaspeech. XTTS v2 is a popular commercial zero-shot TTS model4 trained on a mixture of publicly available data and web-crawled data, although the exact data sources are unknown. YourTTS is trained on VCTK, LibriTTS,and also French and Portugese corpora.

Metrics. For ablation studies, since ground truth waveform is avaliable, in addition to WER (us-ing Whisper medium.en as the ASR model),we

The GitHub repo hosting XTTS v2 has 26k stars by Jan

2024.

Table 4:Performance comparison on speech editing.


| Model | WER | Intelligibility MOS | Intelligibility MOS | Intelligibility MOS | Intelligibility MOS | Naturalness MOS | Naturalness MOS | Naturalness MOS | Naturalness MOS |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| Model | WER | LibriTTS | YouTube | Spotify | Total | LibriTTS | YouTube | Spotify | Total |
| FluentSpeech | 4.5 | 3.89±0.09 | $4 . 0 8 \pm a c a$ | 3.95±0.08 | $3 . 9 7 \pm 0 . 0 5$ | 3.42±0.10 | $4 . 0 7 \pm 0 . 1 0$ | 3.93±0.10 | 3.81±0.06 |
| VOICECRAFT | 6.1 | 4.05±0.08 | 4.14±0.07 | $4 . 1 2 \pm 0 . 0 7$ | $4 . 1 1 \pm 0 . 0 5$ | 3.68±0.10 | 4.25$\pm a a 9$ | $4 . 1 6 \pm o c s$ | 4.03±0.05 |
| Original | 5.4 | 4.22±0.07 | 4.30±0.07 | 4.16±0.08 | 4.22±0.05 | 3.84±0.09 | 4.35±0.08 | 4.29±0.08 | 4.17±0.05 |


<!-- deletion VoiceCraft 40.77% Neutral 15.77% Original 43.46% substitution VoiceCraft 40.46% Neutral 16.65% Original 42.89% insertion VoiceCraft 39.11% Neutral 17.78% Original 43.11% 0 20 40 60 80 100  -->
![](https://web-api.textin.com/ocr_image/external/e1d428fe726cac70.jpg)


| 1-2 words | VoiceCraft39.68% | Neut19.37% | Original40.95% |  |
| -- | -- | -- | -- | -- |
| 3-6 words |  | N보동 | 8398 |  |
| 7-12 words | VaiESS | N.남 | 93 |  |
| 2 spans | aft | N을.닝링 | 2988 |  |
| 0 | 20 | 40 60 | 80 | 100 |


Figure 3: Breakdown of side-by-side human preference on naturalness comparing VOICECRAFT edited speech and the original speech. Grouped by edit type (left) and editspan length(right).

Table 5: Side-by-side naturalness comparison of VOICE-CRAFT(VCR) v.s. Original(Orig.)and FluentSpeech(FS).


| Comparison | VCR better | Tie | VCR worse |
| -- | -- | -- | -- |
| VOICECRAFT v. FS | 56.1% | 19.7% | 24.1% |
| VOICECRAFT v.Orig. | 40.3% | 16.2% | 43.6% |


use mel-ceptral distortion (MCD), FO distance (F0)and enery distance (Energy). These are all ob-jective metrics and their definitions are detailed in §C. For speech editing and zero-shot TTS eval-uation, we use a combination of objective and subjective metrics. For the objective metrics, we used WER and speaker similarity (SIM) follow-ing prior works(Wang et al., 2023a; Kharitonov et al., 2023). SIM is calculated using the WavLM-TDCNN (Chen et al., 2021b). WER and SIM are calculated on all 310 utterances in REALEDIT, and 250 utterances in the zero-shot TTS dataset. For our subjective evaluation, we used the Amazon Mechanical Turk platform to conduct human lis-tening tests.For speech editing, the outputs of our model on all 310 utterances from REALEDIT are evaluated by Turkers in terms of naturalness and intelligibility, and we use a 5-point Likert scale where 1 means poor and 5 means excellent. We also performed side-by-side A/B testing of VOICE-CRAFT's output against the original (non-edited)speech, as well as the edited speech produced by FluentSpeech. In both cases, Turkers were asked to determine which utterance sounds more natural.The Turkers can choose either one of the two,or indicate that they are equally natural. Each evalu-ation received 5 ratings from 5 different Turkers.For zero-shot TTS, we randomly sampled 8 utter-ances (40 from LibriTTS and 40 from YouTube)from the original evaluation set, and asked Turkers

to rate the naturalness, intelligibility, and speaker similarity of the generated speech to the reference prompt on a 5-point Likert scale. Each evalua-tion received 10 ratings. For all evaluations except the side-by-side comparison,Mean-Opinion-Score (MOS) with 95% confidence interval are reported.For the side-by-side comparison,we report the per-centage of the time one model is preferred over the other. 64 and 59 Turkers participated in speech edit-ing and TTS evaluation respectively. Please refer to §E for instructions and participants description.

### 5.2 Ablations

In table 3,we see that larger model sizes lead to better performance across all metrics . In ad-dition,we see a bigger gap between the bigger models, indicating the potential of further scal-ing model (and possibly training data) sizes. For the impact of codebook re-weighting, and we see that weighting earlier codebook heavier leads to better performance on intelligibility related met-rics WER and MCD,while worse performance on prosody related metrics FO and Energy5. We choose weight (5,1,0.5,0.1) in our final 830M model because anecdotally, we found that VOICE-CRAFT is stronger in prosody compared to intelli-gibility (similar properties about NCLMs are also found in (Jiang et al., 2023a; Song et al., 2024; Du et al.,2024b))

### 5.3 Speech Editing Results

Table 4 shows the results of speech editing eval-uation in terms of WER, and human preference

SThis can be regarded as a probing results that shows the properties of different codebooks in RVQ models. Since this is not the focus of our work, we do not conduct further experi-ment on this direction.

Table 6:On the zero-shot TTS task, comparing VOICECRAFT with other models.


| Model | WER |  SIM | Intelligibility MOS | Intelligibility MOS | Intelligibility MOS | Naturalness MOS | Naturalness MOS | Naturalness MOS | Speaker Similarity MOS | Speaker Similarity MOS | Speaker Similarity MOS |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| Model | WER |  SIM | Libri. | You Tube | Total | Libri. | YouTube | Total | Libri. | YouTube | Total |
| YourTTS | 6.6 | 0.41 | $3 . 2 8 \pm 0 . 1 1$ | $3 . 0 1 \pm 0 . 1 2$ | 3.14±0.08 | $2 . 9 9 \pm 0 . 1 2$ | $2 . 5 9 \pm a 1 2$ | $2 . 7 9 \pm 0 . 0 8$ | $3 . 1 0 \pm 0 . 1 2$ | $2 . 4 9 \pm 0 . 1 2$ | $2 . 7 9 \pm a a 0$ |
| FluentSpeech | 3.5 | 0.47 | $3 . 7 0 \pm 0 . 1 1$ | $3 . 6 5 \pm 0 . 1 2$ | $3 . 6 7 \pm 0 . 0 8$ | $3 . 3 4 \pm 0 . 1 1$ | $3 . 4 3 \pm 0 . 1 2$ | $3 . 3 8 \pm 0 . 0 8$ | $4 . 1 0 \pm 0 . 0 9$ | $3 . 9 2 \pm 0 . 1 1$ | $4 . 0 1 \pm a m$ |
| VALL-E | 7.1 | 0.50 | $4 . 0 5 \pm 0 . 0 9$ | $3 . 9 4 \pm 0 . 1 0$ | $4 . 0 0 \pm 0 . 0 7$ | $3 . 8 5 \pm 0 . 1 0$ | $3 . 8 6 \pm 0 . 1 0$ | $3 . 8 6 \pm 0 . 0 7$ | $4 . 1 2 \pm 0 . 1 0$ | $4 . 0 2 \pm 0 . 1 0$ | $4 . 0 7 \pm 0 . 0 7$ |
| XTTS v2 | 3.6 | 0.47 | $4 . 2 9 \pm 0 . 0 9$ | $3 . 9 7 \pm 0 . 1 0$ | $4 . 1 3 \pm 0 . 0 7$ | $4 . 0 2 \pm 0 . 0 9$ | $3 . 9 0 \pm 0 . 1 0$ | $3 . 9 6 \pm 0 . 0 7$ | $3 . 6 4 \pm 0 . 1 2$ | $3 . 2 5 \pm 0 . 1 2$ | 3.44±0.08 |
| VOICECRAFT | 4.5 | 0.55 | $4 . 3 8 \pm 0 . 0 8$ | $4 . 0 8 \pm 0 . 1 0$ | $4 . 2 3 \pm 0 . 0 6$ | $4 . 1 6 \pm 0 . 0 8$ | $4 . 1 8 \pm a w$ | $4 . 1 7 \pm 0 . 0 6$ | $4 . 3 5 \pm 0 . 0 8$ | $4 . 3 3 \pm 0 . 0 9$ | 4.34-006 |
| Ground Truth | 3.8 | 0.76 | $4 . 3 7 \pm 0 . 0 8$ | 4.42±008 | $4 . 3 9 \pm 0 . 0 6$ | 4.32±0.08 | $4 . 6 4 \pm 0 . 0 6$ | 4.48±005 | 4.26±010 | 4.62±0.08 | 4.44±0.06 |


on intelligibility and naturalness. Our VOICE-CRAFToutperforms FluentSpeech on both intel-ligibility and naturalness MOS across different sources. Interestingly,FluentSpeech achieves a WER lower than the original recording (4.5 v.s.5.4), although its intelligibility MOS (3.97) is worse than both VOICECRAFT (4.11) and original recording (4.22). This suggests that ASR model and human judgement divergeon FluentSpeech's intelligibility. Anecdotally, we observe that Flu-entSpeech tends to produce dull and sometimes robotic speech 6,and we hypothesize that this type of speech tends be more easily recognized by ASR,but is less intelligible to human ears. We notice this same phenomenon in our results on zero-shot TTS.

Human listeners rate LibriTTS's naturalness lower than YouTube and Spotify on original speech (results on TTS is consistent with this). This sug-gests that to better evaluate speech synthesis in general, the research community should consider evaluating on other speec domains besides audio-books as is commonly done.

Table 5 presents side-by-side utterance natu-ralness comparison of VOICECRAFT vs. Flu-entSpeech and VOICECRAFT vs. the original,unedited speech. We observe that VOICECRAFT is preferred over FluentSpeech 56.1% of the time,with an additional 19.7% of the time the two are tied. This means that 75.9% of the time, human lis-teners'think VOICECRAFT produces equal or more natural speech than FluentSpeech. Impressively,human listeners judge the edited speech produced by VOICECRAFT to be equally or more natural than the original unedited speech 56.4% of the time.Fig.4 shows the breakdown of the side-by-side comparisons by edit type and edit span length. We see that compared to the original speech, VOICE-CRAFT performs consistently well across different edit types, but human listeners think its outputs are slightly less natural with longer edit span(s).

6please refer to our demo page for examples

### 5.4 Zero-Shot TTS Results

Table 6 shows both objective and subjective eval-uation on zero-shot TTS.We observe that VOICE-CRAFT achieves the best results in both automatic speaker similarity metric SIM, and all human eval-uation metrics. In particular, VOICECRAFT is only slightly worse than ground truth in terms of intel-ligibility MOS (4.23 v.s. 4.39), and speaker sim-ilarity MOS (4.34 v.s. 4.44). The gap on natural-ness is larger between VOICECRAFT and ground truth (4.17 v.s. 4.48), especially on YouTube utter-ances, which highlights the challenges of zero-shot TTS on noisy, in-the-wild data. The commercial model XTTS v2 comes second in terms of intel-ligibility and naturalness,and second to last on speaker similarity MOS. VALL-E achieves the sec-ond best on both automatic metric SIM and subjec-tive metric speaker similarity MOS.Similarly to the speech editing results, ground truth YouTube utterances receive higher MOS scores than ground truth LibriTTS utterances in Table 6, which again suggests that we should consider using more di-verse data for future speech synthesis model evalu-ation.Lastly, we again observe that FluentSpeech achieves lower WER than the ground truth, but re-ceives much lower ratings in terms of intelligibility MOS from human listeners, indicating that WER could be misleading in evaluating intelligibility of speech synthesis systems7.

## 6 Conclusion

We introduce a neural codec language model VOICECRAFT that achieves state-of-the-art perfor-mance on speech editing and zero-shot TTS on in-the-wild data. The key lies in an innovative token rearrangement procedure which enables efficient and effective autoregressive codec generation with bidirectional context. In addition, we introduce a first-of-its-kind high quality,challenging,and re-

7we also tried Whisper Large-v3,it gets WER of 4.1 for ground truth,and 2.7 for FluentSpeech.

alistic speech editing dataset REALEDIT,which we believe can reliably measre the practicality of speech editing models.

## 7 Limitations

Given the advancement of made by VOICECRAFT,there are still limitations. First and foremost is the long silence and scratching sound that occa-sionally occur during generation. Although in this work, we overcome it with sampling multiple utter-ances and selecting the shorter ones,more elegant and efficient methods are needed. Another impor-tant aspect is AI safety, how can we watermark and detect synthesized speech? While watermark-ing and deepfake detection has attracted increas-ing attention in the research community,and re-markable progress has been made (Zhang et al.,2020;Yamagishi et al., 2021; Chen et al., 2023;Roman et al., 2024),more advanced models such as VOICECRAFT presents new opportunities and challenges to safety research. To facilitate speech synthesis and AI safety research, we fully open source our codebase and model weights.

## 8 Ethical Implications

The speech synthesis model VOICECRAFT intro-duced in this work has both positive and negative implications.

On the positive side, VOICECRAFT holds the promise of significant benefits across several do-mains. For individuals with speech impairments or who have lost the use of their voice, VOICE-CRAFT could be transformative,enabling these in-dividuals new ways to communicate with ease and clarity that were previously not possible. Content creators,whether they work in education, video production, or podcasting, could leverage VOICE-CRAFT to streamline their editing processes, mak-ing it easier to produce high-quality content with-out the need to re-record takes when they contain a small mistake. Furthermore, VOICECRAFT's abil-ity to handle diverse accents without compromising on quality opens up new possibilities for creating synthetic data. This could, in turn, enhance speech recognition systems, such as Voicebox (Le et al.,2023),by providing them with a richer and more varied dataset to learn from, thereby improving their accuracy and accessibility to users worldwide.

However,the potential negative impacts of VOICECRAFT cannot be overlooked. One of the primary concerns is the model's potential to exac-

erbate existing biases,particularly those related to ethnicity.If not carefully monitored and corrected,these biases could lead to unequal performance across different groups, perpetuating and possibly even worsening existingdisparities.Moreover,the ease with which voices can be cloned raises serious concerns about misuse, including impersonation and fraud. The ability to replicate someone's voice with only a few seconds of reference audio could be exploited to commit crimes or spread misin-formation, posing significant ethical and security challenges. As such, while the benefits of VOICE-CRAFT are clear and substantial, it is imperative to approach its deployment with caution,ensuring that measures are in place to mitigate these risks and protect against potential misuse.

Despite the concerns regarding impersonation and fraud associated with VOICECRAFT,there are compelling reasons to advocate for its release. Fore-most among these is the opportunity it presents for the broader research community and technology developers to better understand and mitigate these negative impacts. By making these methods open source,we can catalyze the development of more robust countermeasures against the misuse of voice cloning technologies. This collaborative approach allows for the rapid identification of vulnerabili-ties and the exploration of innovative strategies to address them.Moreover, the authors of this work fully committed to advancing the field responsibly.We are actively working on pioneering deepfake detection and watermarking algorithms specifically designed for synthetic speech. By doing so,we not only acknowledge the potential risks associ-ated with our technology but also take concrete steps to ensure its ethical use. This dual approach of open collaboration and dedicated research into safeguarding mechanisms reflects our commitment to fostering a technological ecosystem where the benefits of voice cloning can be realized while min-imizing its potential for harm.

## 9 Acknowledgements

We thank Ziyue Jiang for providing guidance in running inference with and scaling up the Flu-entSpeech model. We thank students at SALT Lab of UT Austin for helpful discussions. This work is supported in part by the National Science Founda-tion under Grant No. 2238605.

## References

Armen Aghajanyan, Po-Yao (Bernie) Huang, Candace Ross, Vladimir Karpukhin, Hu Xu,Naman Goyal,Dmytro Okhonko, Mandar Joshi,Gargi Ghosh, Mike Lewis,and Luke Zettlemoyer. 2022. Cm3: A causal masked multimodal model of the internet. ArXiv,abs/2201.07520.

Andrea Agostinelli, Timo I. Denk, Zalán Borsos,Jesse Engel, Mauro Verzetti,Antoine Caillon,Qingqing Huang,Aren Jansen,Adam Roberts,Marco Tagliasacchi,Matthew Sharifi, Neil Zeghidour,and Christian Havnø Frank. 2023. Musiclm: Generating music from text. ArXiv,abs/2301.11325.

He Bai, Renjie Zheng,Junkun Chen, Xintong Li,Mingbo Ma, and Liang Huang. 2022. A3t:Alignment-aware acoustic and text pretraining for speech synthesis and editing. In International Con-ference on Machine Learning.

Mohammad Bavarian, Heewoo Jun, Nikolas A. Tezak,John Schulman,Christine McLeavey,Jerry Tworek,and Mark Chen. 2022.Efficient training of language models to fill in the middle. ArXiv, abs/2207.14255.

Mathieu Bernard and Hadrien Titeux. 2021. Phonem-izer: Text to phones transcription for multiple lan-guages in python. Journal of Open Source Software,6(68):3958.

Zalán Borsos, Raphaël Marinier, Damien Vincent, Eu-gene Kharitonov,Olivier Pietquin,Matthew Sharifi,Dominik Roblek, Olivier Teboul, David Grangier,Marco Tagliasacchi, and Neil Zeghidour. 2022a. Au-diolm: A language modeling approach to audio gen-eration.IEEE/ACM Transactions on Audio,Speech,and Language Processing, 31:2523-2533.

Zalán Borsos, Matthew Sharifi, and Marco Tagliasac-chi.2022b.Speechpainter:Text-conditioned speech inpainting. In Interspeech.

Zalán Borsos, Matthew Sharifi, Damien Vincent,Eugene Kharitonov,Neil Zeghidour, and Marco Tagliasacchi. 2023. Soundstorm: Efficient parallel audio generation. ArXiv,abs/2305.09636.

Edresson Casanova, Julian Weber, Christopher Dane Shulby, Arnaldo Cândido Júnior, Eren Gölge,and Moacir Antonelli Ponti. 2021. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice con-version for everyone. In International Conference on Machine Learning.

Guang Chen, Yu Wu, Shujie Liu, Tao Liu, Xiaoyong Du, and Furu Wei. 2023. Wavmark:Watermarking for audio generation. ArXiv,abs/2308.12770.

Guoguo Chen,Shuzhou Chai,Guan-Bo Wang,Jiayu Du,Weiqiang Zhang, Chao Weng, Dan Su,Daniel Povey,Jan Trmal,Junbo Zhang,Mingjie Jin,Sanjeev Khudanpur,Shinji Watanabe, Shuaijiang Zhao, Wei Zou,Xiangang Li,Xuchen Yao,Yongqing Wang,Yujun Wang,Zhao You,and Zhiyong Yan.2021a.

Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. In Proc.Interspeech 2021.

Sanyuan Chen,Chengyi Wang,Zhengyang Chen,Yu Wu,Shujie Liu,Zhuo Chen,Jinyu Li,Naoyuki Kanda,Takuya Yoshioka, Xiong Xiao,Jian Wu,Long Zhou,Shuo Ren, Yanmin Qian, Yao Qian, Micheal Zeng, and Furu Wei. 2021b. Wavlm: Large-scale self-supervised pre-training for full stack speech pro-cessing.IEEE Journal of Selected Topics in Signal Processing, 16:1505-1518.

Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu,Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich,Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosie Jones. 2020. 100,000 podcasts: A spoken En-glish document corpus. In Proceedings of the 28th International Conference on Computational Linguis-tics,pages 5903-5917,Barcelona,Spain (Online).International Committee on Computational Linguis-tics.

Jade Copet, Felix Kreuk,Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Defossez. 2023. Simple and controllable music gen-eration. ArXiv,abs/2306.05284.

COQUI.2023. Xtts v2. https://huggingface.co/coqui/XTTS-v2.

Alexandre Defossez,Jade Copet,Gabriel Synnaeve,and Yossi Adi. 2022. High fidelity neural audio compres-sion.ArXiv, abs/2210.13438.

Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghi-dour,and Jesse Engel. 2023. Singsong: Generat-ing musical accompaniments from singing. ArXiv,abs/2301.12662.

Chris Donahue, Mina Lee, and Percy Liang. 2020.En-abling language models to fill in the blanks.In Pro-ceedings of the 58th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 2492-2501,Online. Association for Computationa1 Lin-guistics.

Chenpeng Du, Yiwei Guo, Feiyu Shen, Zhijun Liu,Zheng Liang,Xie Chen,Shuai Wang,Hui Zhang,and K. Yu. 2024a. Unicats: A unified context-aware text-to-speech framework with contextual vq-diffusion and vocoding.In AAAI.

Chenpeng Du,Yiwei Guo, Hankun Wang,Yifan Yang,Zhikang Niu,Shuai Wang,Hui Zhang,Xie Chen,and Kai Yu. 2024b. Vall-t: Decoder-only generative transducer for robust and decoding-controllable text-to-speech.

Hugo Flores Garcia,Prem Seetharaman, Rithesh Ku-mar, and Bryan Pardo. 2023. Vampnet: Musicgen-eration via masked acoustic token modeling. ArXiv,abs/2307.04686.

Zhifang Guo,Yichong Leng,Yihan Wu,Sheng Zhao,and Xuejiao Tan. 2022. Prompttts: Controllable text-to-speech with text descriptions. ICASSP 2023-2023 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP),pages 1-5.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes,and Yejin Choi.2020.The curious case of neural text de-generation. In International Conference on Learning Representations.

Wei-Ning Hsu, David Harwath, Tyler Miller, Christo-pher Song, and James R. Glass. 2021.Text-free image-to-speech synthesis using learned segmental units.In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021.

Keith Ito and Linda Johnson. 2017. The 1j speech dataset. https://keithito.com/LJ-Speech-Dataset/.

Shengpeng Ji,Jia li Zuo,Minghui Fang,Ziyue Jiang,Feiyang Chen, Xinyu Duan,Baoxing Huai, and Zhou Zhao.2023. Textrolspeech: A text style control speech corpus with codec language text-to-speech models.ArXiv,abs/2308.14430.

Ziyue Jiang,Jinglin Liu,Yi Ren,Jinzheng He,Zhenhui Ye,Shengpeng Ji,Qian Yang,Chen Zhang,Pengfei Wei,Chunfeng Wang,Xiang Yin,Zejun MA,and Zhou Zhao. 2024.Mega-TTS 2: Boosting prompt-ing mechanisms for zero-shot speech synthesis.In The Twelfth International Conference on Learning Representations.

Ziyue Jiang,Yi Ren,Zhe Ye,Jinglin Liu,Chen Zhang,Qiang Yang,Shengpeng Ji,Rongjie Huang,Chun-feng Wang,Xiang Yin,Zejun Ma,and Zhou Zhao.2023a.Mega-tts:Zero-shot text-to-speech at scale with intrinsic inductive bias. ArXiv, abs/2306.03509.

Ziyue Jiang,Qiang Yang,Jia li Zuo,Zhe Ye,Rongjie Huang,Yixiang Ren,and Zhou Zhao. 2023b. Flu-entspeech: Stutter-oriented automatic speech edit-ing with context-aware diffusion models. In Annual Meeting of the Association for Computational Lin-guistics.

Zeyu Jin,Gautham J. Mysore, Stephen DiVerdi, Jing-wan Lu, and Adam Finkelstein. 2017. Voco: text-based insertion and replacement in audio narration.In International Conference on Computer Graphics and Interactive Techniques.

Eugene Kharitonov,Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Nguyen, Mor-gane Rivière,Abdel rahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2021a. Text-free prosody-aware generative spoken language modeling.ArXiv,abs/2109.03264.

Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi,Jade Copet, Kushal Lakhotia, Tu Nguyen, Mor-gane Rivière, Abdel rahman Mohamed, Emmanuel

Dupoux, and Wei-Ning Hsu. 2021b. Text-free prosody-aware generative spoken language modeling.ArXiv, abs/2109.03264.

Eugene Kharitonov, Damien Vincent, Zalán Borsos,Raphaël Marinier, Sertan Girgin, Olivier Pietquin,Matthew Sharifi,Marco Tagliasacchi,and Neil Zeghi-dour.2023.Speak,read and prompt: High-fidelity text-to-speech with minimal supervision. Transac-tions of the Association for Computational Linguis-tics,11:1703-1718.

Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021.Conditional variational autoencoder with adversar-ial learning for end-to-end text-to-speech. ArXiv,abs/2106.06103.

Diederik P. Kingma and Jimmy Ba. 2014. Adam:A method for stochastic optimization. CoRR,abs/1412.6980.

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020.Hifi-gan: Generative adversarial networks for ef-ficient and high fidelity speech synthesis. ArXiv,abs/2010.05646.

Felix Kreuk,Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Defossez, Jade Copet, Devi Parikh,Yaniv Taigman, and Yossi Adi. 2022. Audio-gen: Textually guided audio generation.ArXiv,abs/2209.15352.

Robert F. Kubichek. 1993. Mel-cepstral distance mea-sure for objective speech quality assessment. Pro-ceedings of IEEE Pacific Rim Conference on Commu-nications Computers and Signal Processing, 1:125-128 vol.1.

Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu,Yossi Adi,Adam Polyak,Benjamin Bolte,Tu Nguyen, Jade Copet, Alexei Baevski, Adel Ben Mohamed,and Emmanuel Dupoux. 2021. On gen-erative spoken language modeling from raw audio.Transactions of the Association for Computational Linguistics,9:1336-1354.

Matt Le,Apoorv Vyas,Bowen Shi,Brian Karrer,Leda Sari,Rashel Moritz, Mary Williamson, Vi-mal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. 2023. Voicebox: Text-guided multilin-gual universal speech generation at scale. ArXiv,abs/2306.15687.

Feiteng Li. 2023. An unofficial pytorch implementa-tion of vall-e.https://github.com/lifeiteng/vall-e.

Guanghou Liu,Yongmao Zhang, Yinjiao Lei,Yunlin Chen, Rui Wang, Zhifei Li, and Linfu Xie. 2023.Promptstyle: Controllable style transfer for text-to-speech with natural language descriptions. ArXiv,abs/2305.19522.

Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.In International Confer-ence on Learning Representations.

Daniel Lyth and Simon King. 2024. Natural language guidance of high-fidelity text-to-speech with syn-thetic annotations. ArXiv, abs/2402.01912.

Matthias Mauch and Simon Dixon. 2014. Pyin:A fundamental frequency estimator using probabilis-tic threshold distributions. 2014 IEEE International Conference on Acoustics, Speech and Signal Process-ing (ICASSP), pages 659-663.

Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W.Ellis,Matt McVicar,Eric Battenberg,and Oriol Nieto.2015. librosa: Audio and music signal analysis in python.In SciPy.

Max Morrison, Lucas Rencker, Zeyu Jin, Nicholas J.Bryan, Juan Pablo Cáceres, and Bryan Pardo.2021.Context-aware prosody correction for text-based speech editing. ICASSP 2021 - 2021 IEEE Inter-national Conference on Acoustics,Speech and Signal Processing (ICASSP), pages 7038-7042.

Tu Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Mamdouh Elkahky,Paden Tomasello,Robin Algayres,Benoît Sagot, Abdel rah-man Mohamed, and Emmanuel Dupoux.2022. Gen-erative spoken dialogue language modeling.Transac-tions of the Association for Computational Linguis-tics,11:250-266.

Alec Radford,Jong Wook Kim, Tao Xu, Greg Brock-man, Christine McLeavey, and Ilya Sutskever. 2022.Robust speech recognition via large-scale weak su-pervision.ArXiv,abs/2212.04356.

Alec Radford, Jeff Wu, Rewon Child, David Luan,Dario Amodei,and Ilya Sutskever. 2019. Language modeIs are unsupervised multitask learners.

Robin San Roman,Pierre Fernandez, Alexandre De-fossez,Teddy Furon, Tuan Tran, and Hady ElSahar.2024. Proactive detection of voice cloning with lo-calized watermarking. ArXiv,abs/2401.17264.

Yakun Song, Zhuo Chen, Xiaofei Wang,Ziyang Ma,and Xie Chen. 2024. Ella-v: Stable neural codec language modeling with alignment-guided sequence reordering.

Daxin Tan,Liqun Deng,Yu Ting Yeung,Xin Jiang,Xiao Chen, and Tan Lee.2021. Editspeech:A text based speech editing system using partial inferenceand bidirectional fusion. 2021 IEEE Automatic Speech Recognition and Understanding Workshop(ASRU),pages 626-633.

Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.2017.Neural discrete representation learning. ArXiv, abs/1711.00937.

Ashish Vaswani, Noam M.Shazeer,Niki Parmar, Jakob Uszkoreit,Llion Jones,Aidan N. Gomez, Lukasz Kaiser,and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems.

Chengyi Wang,Sanyuan Chen,Yu Wu,Zi-Hua Zhang,Long Zhou,Shujie Liu, Zhuo Chen, Yanqing Liu,Huaming Wang,Jinyu Li, Lei He, Sheng Zhao,and Furu Wei. 2023a. Neural codec language models are zero-shot text to speech synthesizers. ArXiv,abs/2301.02111.

Tao Wang, Jiangyan Yi,Liqun Deng, Ruibo Fu, Jian-hua Tao,and Zhengqi Wen. 2022. Context-aware mask prediction network for end-to-end text-based speech editing. ICASSP 2022-2022 IEEE Interna-tional Conference on Acoustics,Speech and Signal Processing (ICASSP), pages 6082-6086.

Tianrui Wang,Long Zhou, Zi-Hua Zhang, Yu Wu, Shu-jie Liu,Yashesh Gaur,Zhuo Chen,Jinyu Li, and Furu Wei.2023b. Viola: Unified codec language models for speech recognition,synthesis,and trans-lation. ArXiv, abs/2305.16107.

Xiaofei Wang,Manthan Thakker, Zhuo Chen, Naoyuki Kanda,Sefik Emre Eskimez,Sanyuan Chen,Min Tang,Shujie Liu,Jinyu Li, and Takuya Yoshioka.2023c. Speechx:Neural codec language model as a versatile speech transformer. ArXiv, abs/2308.06873.

Junichi Yamagishi, <PRESIDIO_ANONYMIZED_PERSON>, and Kirsten Mac-Donald. 2019. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).

Junichi Yamagishi,<PRESIDIO_ANONYMIZED_PERSON>,Massimiliano Todisco,Md. Sahidullah, Jose Patino, Andreas Nautsch,Xuechen Liu,Kong-Aik Lee,Tomi H. Kinnunen,Nicholas W. D. Evans, and Héctor Delgado. 2021.Asvspoof 2021:accelerating progress in spoofed and deepfake speech detection. ArXiv, abs/2109.00537.

Dongchao Yang,Songxiang Liu,Rongjie Huang,Guangzhi Lei,<PRESIDIO_ANONYMIZED_PERSON>,<PRESIDIO_ANONYMIZED_PERSON>,and Dong Yu. 2023. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. ArXiv,abs/2301.13662.

Zengwei Yao,Liyong Guo,Xiaoyu Yang,Wei Kang,Fangjun Kuang,Yifan Yang,Zengrui Jin,Long Lin,and Daniel Povey. 2024. Zipformer: A faster and better encoder for automatic speech recognition. In ICLR.

Dacheng Yin, Chuanxin Tang,Yanqing Liu,Xiaoqiang Wang,Zhiyuan Zhao,Yucheng Zhao,Zhiwei Xiong,Sheng Zhao,and Chong Luo. 2022. Retrievertts:Modeling decomposed factors for text-based speech insertion.In Interspeech.

Neil Zeghidour, Alejandro Luebs, Ahmed Omran,Jan Skoglund, and Marco Tagliasacchi. 2021.Soundstream: An end-to-end neural audio codec.IEEE/ACM Transactions on Audio,Speech,and Lan-guage Processing, 30:495-507.

Heiga Zen, Viet-Trung Dang, Robert A. J. Clark,Yu Zhang,Ron J. Weiss, Ye Jia, Z. Chen,and Yonghui Wu.2019.Libritts: A corpus derived from librispeech for text-to-speech.In Interspeech.

You Zhang,Fei Jiang, and Zhiyao Duan. 2020. One-class learning towards synthetic voice spoofing detec-tion.IEEE Signal Processing Letters,28:937-941.

Zi-Hua Zhang,Long Zhou,Chengyi Wang,Sanyuan Chen,Yu Wu,Shujie Liu, Zhuo Chen,Yanqing Liu,Huaming Wang,Jinyu Li,Lei He, Sheng Zhao, and Furu Wei. 2023. Speak foreign languages with your own voice:Cross-lingual neural codec language mod-eling.ArXiv,abs/2303.03926.

## AAdditional Experiments

### A.1 Comparing ScaledAdam and AdamW

The hyperparameters settingsof ScaledAdam can be found in table 9. For AdamW (Loshchilov and Hutter, 2017),we tried 3settings:

·setting1: peak learning rate:1e-5, batch size:3.3 min, update steps: 500k

·setting2: peak learning rate: 1e-4, batch size:33.3 min(same as ScaledAdam),update steps:80k

·setting3:peak learning rate:1e-4, batch size:3.3 min, update steps: 500k

For all settings, we use a linear scheduler which linear ramp up the learning rate to peak in first 8% steps,and linearly decay it afterwards. We use the common default values for other hy-perparameters,setting$\beta _ { 1 } = 0 . 9 , \beta _ { 2 } = 0 . 9 9 9 ,$weight-decayy=0.01 All experiments are done on 4 A40 GPUs. Results are shown in table 7.8We see that ScaledAdam achieves better perfor-mance in all metrics while using less compute.However we note that due to limitation in com-putational resources,we could not exhaust hyper-parameter search for AdamW,therefore we do not over-generalize our finding here.

### A.2 Breakdown of side-by-side human preference comparison.

The comparison breakdown between VOICE-CRAFT and FluentSpeech is shown in figure 4. We see that VOICECRAFT outperforms FluentSpeech across the board,especially for substitution edits and when the edit span length is long.

### A.3 Spectrograms Comparison

Spectrogram level comparison between Flu-entSpeech and VOICECRAFTare shown in fig-ure 5,6,7 with the edited part marked in dark green rectangle. The three examples have increasing diffi-culty in terms of accents and recording conditions,in particular, the examples in figure 7 appears to be in low bandwidth transmission. In all 3 examples,we see that VOICECRAFT is able to generated more detailed frequency patterns. The corresponding au-dio can be found in the demo page.

8We early stopped AdamW setting 2 at step57k to save the compute,as it has already taken more time than the finished ScaledAdam job while the performance was worse.

Table 7: ScaledAdam consistently outperforms AdamW across all metrics, while taking 10% less time to train.


| Optimizer | Setting | Training Time | WER | MCD | FO | Energy |
| -- | -- | -- | -- | -- | -- | -- |
| AdamW | $1 r = 1 e - 5 , b s Z = 1 3 . 3 \min , s t e p s = 5 0 0 k$ | 262 hours | 16.45 | 8.91 | 196.15 | 5.94 |
| AdamW | $1 r = 1 e - 4 , b s z = 1 3 3 . 2 \min , s t e p s = 5 7 k$ | 273 hours | 10.77 | 8.45 | 117.38 | 4.91 |
| AdamW | $1 r = 1 e - 4 , b s z = 1 3 . 3 \min , s t e p s = 5 0 0 k$ | 262 hours | 7.58 | 8.32 | 82.73 | 3.70 |
| ScaledAdam | $1 r = 3 e - 2 , b s z = 1 3 3 . 2 \min , s t e p s = 5 0 k$ | 237 hours | 7.30 | 8.13 | 73.41 | 3.19 |


<!-- deletion VoiceCraft 45.38% Neutral 30.00% FluentSpeech 24.62% substitution VoiceCraft 58.50% Neutral 18.38% FluentSpeech 23.12% insertion VoiceCraft 55.56% Neutral 16.44% FluentSpeech 28.00% 0 20 40 60 80 100  -->
![](https://web-api.textin.com/ocr_image/external/2a66d1fb30c86fd9.jpg)

<!-- 1-2 words VoiceCraft 50.79% Neutral 26.35% FluentSpeech 22.86% 3-6 words VoiceCraft 53.44% Neutral 19.84% FluentSpeech 26.72% 7-12 words VoiceCraft 62.20% Neutral FluentSpeech 16.34% 21.46% 2 spans VoiceCraft Ne.8링 Flueg8gech 0 20 40 60 80 100  -->
![](https://web-api.textin.com/ocr_image/external/52e8fed3a0ab87f1.jpg)

Figure 4: Breakdown of side-by-side human preference on naturalness comparing of VOICECRAFT and Flu-entSpeech on speech editing. Grouped by edit type (left) and edit span length (right).

<!-- 8000 7000 6000 (2H 5000 A23nE 4000 3000 2000 1000 0 that schedule ⑬ one pet week a you start 60 se 1o% Note centent amwving pecause bian on doig one to two vibeos P week 8000 7000 6000 (8H)Ruanbeus 5000 4000 3000 2000 1000 that schedule onR pt wek and a toe more content videos -  -->
![](https://web-api.textin.com/ocr_image/external/0d69c0f6748a3136.jpg)

Figure 5: Upper: FluentSpeech; lower: VOICECRAFT

<!-- 8000 7000 6000 2H] 5000 AUartte 4000 3000 2000 1000 0 0 wa5 lock of disgust 6 W0 some 8000 7000 6000 ( 5000 ) AtonbE 4000 3000 2000 1000 0 05 of disgust VP  -->
![](https://web-api.textin.com/ocr_image/external/4f180c10fe40f7be.jpg)

Figure 6: upper: FluentSpeech; lower: VOICECRAFT

<!-- 8000 7000 6000 5000 PuarbeL 4000 3000 2000 1000 t0 cild brought be 년e 8000 7000 6000 [CH Aeente 5000 4000 3000 2000 1000 0 ㎡ b8 udo R ue  -->
![](https://web-api.textin.com/ocr_image/external/ca067b7136825368.jpg)

Figure 7:upper: FluentSpeech; lower: VOICECRAFT.Note that since the speech is recorded in very challenging condition, the word alignment is not very accurate. We see that for FluentSpeech's result, since the entire mel-spectrogram are passed to HiFi-GAN for resynthesis,even the unedited speech contains high frequency noise.

## B Examples of the Speech Editing Dataset REALEDIT

Examples of REALEDIT are shown in table 8.

### C Implementational Details

The Encodec model.The Encodec model we use has a stride of 320 samples, which means the codec framerate is 50Hz for recording of sample rate 16kHz. The base dimension is 64, doubling at each of the 5 convolutional layer in the encoder.Following (Copet et al., 2023), we use the open-sourced audiocraft repo for Encodec model train-ing. 1 second speech segments sampled from Gi-gaspeech over a total of 160 epochs (320k steps)with a batch size of 240. The model is trained with the Adam (Kingma and Ba, 2014) with base learning rate of 3e-4.

Eden Scheduler (Yao et al.,2024).the sched-uler adjust the learning rate at at step t using the following formula:

$\alpha _ { t } = \alpha _ { b a s e } \cdot ( \frac { t ^ { 2 } + \alpha _ { s l e p } ^ { 2 } } { \alpha _ { s l e p } ^ { 2 } } ) ^ { - 0 . 2 5 }$

$( \frac { e ^ { 2 } + \alpha _ { e p o c h } ^ { 2 } } { \alpha _ { e p o c h } ^ { 2 } } ) ^ { - 0 . 2 5 } .$

·linear($( \alpha _ { s \tan t } , t _ { w \tan w p } , t ) .$

WhereObase base learning rate, t is the step in-dex,e is the epoch index, and Ostep and Cepoch controls the amount of data the model has seen before significantly reducing the learning rate.linear$( \alpha _ { s \tan t } , t _ { w a r m u p } , t )$linearly increase the out-come from Ostart to 1 over twarmup steps,and stays at 1.In our experiment,we set

$\alpha _ { b a s e } = 0 . 0 5 , \alpha _ { s t e p } = 3 0 0 0 , \alpha e p o c h = 4 ,$

$\alpha _ { s t a r t } = 0 . 5 , t _ { w a r m u p } = 5 0 0$

Since our dataset is quite large, we use pseudo-epoch instead of the actual epoch, and 1 pseudo-epoch is set to be 3000 training steps. Note that the choice of these hyperparameters are inspired by Yao et al. (2024); Li (2023), and if computation resources permitted,a grid search might find better hyperparameters settings.

Configuration in ablation studies. Configura-tion of different models are shown in table 9. Note that we use base learning rate 3e-2 for 430M model instead of 5e-2 because the latter gave a NaN error.

Encodec training doc can be here

Task and Data for ablation studies. The evalu-ation task is masked reconstruction,where foreach utterance,we randomly select a span oflength 1 to 15 words to mask, and ask VOICECRAFT to recon-struct the masked speech based on the transcript and unmasked speech. We use a 1000-utterance random subset of the Gigaspeech validation set,which contains YouTube videos and podcas data.We ensure that each utterance in the subset has a WER lower than 15% when decoded by Whisper medium.en (Radford et al.,2022).

Metrics for ablation studies. Since ground truth is available for masked reconstruction evalua-tion, in addition to WER (measured from Whisper medium.en's output), we also measure the mel-cepstral distortion (MCD) (Kubichek, 1993), F0distance (F0), and energy distance (Energy) WER and MCD are better correlated with intelligibility of the speech,and FO and Energy are better corre-lated with prosody similarity between the generated and ground truth. MCD measures the difference of Mel Frequency Cepstrum Coefficients (MFCC)between generated and ground truth, defined as

$M C D = \frac { 1 0 } { \ln 1 0 } \sqrt { \frac { 1 } { 2 } } \sum _ { i = 1 } ^ { L } ( m _ { i } ^ { 9 } - m _ { i } ^ { T } ) ^ { 2 }$

where L is the order of MFCC,which we set to be 13.$m _ { i } ^ { 9 }$s the ith MFCC of ground truth recording and m' is the ith MFCC of the generated.We use pymcd package 10 for calculating MCD. For F0estimation, we use the pYIN(Mauch and Dixon,2014) algorithm implemented in librosa(McFee et al., 2015) with minimal frequency 80hz and max-imal frequency 600hz.For energy calculation,we use the root mean square of magnitude of spectro-gram,which is extracted using short time Fourier transform with window length of 640,hop size of 160.Note that since generated speech might have a different length compared to ground truth, dynamic time wrapping is first applied to time aligned the extracted MFCC/FO/energy before calculating their euclidean distances. For each model in the ablation study,we use 3 different random seeds and report the averaged results.

Scalng FluentSpeech. The original Flu-entSpeech (Jiang et al., 2023b) is trained on Lib-riTTS, and we made our best effort in scaling it for a fair comparison. Taking guidance from the authors of FluentSpeech. We scale the batch size

10https://github.com/chenqi008/pymcd

Table 8:Examples of the speech editing dataset REALEDIT.


| Edit Types | Original | Edited |
| -- | -- | -- |
| substitution,substitution | See why it's extremely valuable to it's kindof like it's kind of like having a wall hack towatch a demo. | See why it's extremely important right? it'skind of like having a rough time to watch ademo. |
| deletion | I wrote the title of the course many years ago,ah,when I created this course. |  I wrote the title when I created this course. |
| insertion | Fast cars,that had the nice clothes,that had themoney,they was criminals. | Fast cars,that had the nice clothes, that hadexpensive gold watches, that had the money,they was criminals. |
| substitution | When the CEO of blockbuster heard that, hepromptly had a kitchen sink delivered to thenetflix office,a fairly creative way of declaringwar. | When the CEO of blockbuster heard that,hepromptly had five hundred pounds of glitterdivided into five thousand manilla envelopesdelivered to the netflix office,a fairly creativeway of declaring war. |
| substitution | So if you've been following my story, you willremember that I said earlier in this podcast thatthe Grammy nominations came out. | So if you've been following my story,you willremember that I said earlier that this week wehad super exciting stuff to talk about becauseGrammy nominations came out. |
| insertion | No to the chemical pollution, air pollution, andthe destruction of the environment caused byfactories and the manufacturing industry. | No to the chemical pollution, air pollution,noto the killing of plants and wildlife and the de-struction of the environment caused by factoriesand the manufacturing industry. |
| substitution,substitution | because we can include so many other charac-ters if we just expand the definitions to anysword wielder,who's a little spicy. | because we can include so many other partic-ipants if we are brave enough to expand thedefinitions to any blade wielder, who's a littlespicy. |
| insertion | So for more craziness now that French was con-quered we have to join forces to Great Britain. | So for more craziness now that French was con-quered by the Germans, we have to join forcesto Great Britain. |
| substitution | economic development remains one of the mosteffective ways to increase the capacity toadapt to climate change. | economic development remains one of the mostpromising options that we have left on thetable to increase the capacity to adapt to climatechange. |
| insertion | And we're at this point. | And we're all extremely excited at this point. |
| insertion | Steve also co-founded pixar animation studios.Which has revolutionized the film industry init's short history with brilliant use of technol-ogy. | Steve also co-founded pixar animation studios.Which has revolutionized the film industry init's short history with films like toy story thatshowcase brilliant use of technology. |
| substitution,deletion | this is just so cozy up here,and having thatskylight is just lovely isn't it. | this is just so cozy and warm here,isn't it. |
| substitution | It was a glance of inquiry, ending in a lookof chagrin, with some muttered phrases thatrendered it more emphatic. | It was a look of disgust followed by a curledlip,with some muttered phrases that rendered itmore emphatic. |
| substitution | More of a base and infrastructure to tell thosestories rather than doing it out of a out of atent with solar power. | More of a base and infrastructure to fight thesebattles instead of out of a tent with solar power. |



| Params | codebook dim | Trm hidden dim | FFN dim | Trm layers | Base LR | Update Steps |
| -- | -- | -- | -- | -- | -- | -- |
| 830M | 2048 | 2048 | 8192 | 16 | 5e-2 | 50k |
| 430M | 2048 | 2048 | 8192 | 8 | 3e-2 | 50k |
| 120M | 1024 | 1024 | 4196 | 8 | 5e-2 | 50k |


Table 9:Hyperparameters settings for the different model sizes. Trm stands for Transformer.

from 16 utterances to 256 utterances. Diffusion base hidden dimension from 320 to 1024, residual layers from 20 layers to 30 layers, residual chan-nels from 256 to 512. The final model contains 330M parameters,which is roughly the same as the Voicebox model (Le et al., 2023). The model was trained on Gigaspeech training set on 1 A40 GPU for 626k steps which took 10 days. The HiFi-GAN vocoder is also retrained on Gigaspeech training set for 400k steps using hyperparameters used on Voicebox (Le et al., 2023) (they also use Hifi-GAN as vocoder to decode to 16kHz speech)

Baselines for zero-shot TTS.For zero-shot TTS,we compare our VOICECRAFT with VALL-E (Wang et al., 2023a), XTTS v2 (COQUI,2023),YourTTS (Casanova et al., 2021), and Flu-entSpeech. Since the original VALL-E is not open-sourced, we use the code from the popu-lar open-source implementation by Li (2023),and also trained the model on Gigaspeech.Both the AR and NAR model are trained for 50k steps us-ing the ScaledAdam optimizer and Eden sched-uler, same as our VOICECRAFT. The commercial model XTTS v2 is composed of three modules,VQ-VAE (van den Oord et al., 2017) for speech tokenization, a GPT-2 (Radford et al., 2019) model for speech token modeling and a customized HiFi-GAN (Kong et al., 2020) model for token to wave-form generation.XTTS v2 is trained on a mixture of publicly available data and web-crawled data,but the exact data sources are unknown.YourTTS is a zero-shot TTS model built upon the adversarial VAE model VITS (Kim et al., 2021), with novel zero-shot multi-speaker and multilingual training.The model is trained on VCTK,LibriTTS, and also French and Portugese corpora. The FluentSpeech model we used for TTS is the same as in speech editing, as the model can be configured to do zero-shot TTS similar to Voicebox (Le et al.,2023).

Licenses of the speech corpora. Licenses: Lib-riTTS: CC BY 4.0; Gigaspeech: Apache-2.0; Spo-tify Podcast dataset: CC BY 4.0.

#### D The Conditional Independence Assumption

To better explain the rational behind the conditional independent assumption in equation 3, we go back to sequence Y produced by causal masking. The assumption we are making for equation 3 to hold is equivalent to the assumption that given W and$H _ { s , t } ,$Ys,t,kis independent of$I _ { , , k } ^ { ( 1 ) }$and$I _ { s , , k } ^ { ( 2 ) }$defined as

$I _ { S _ { 1 } _ { k } k } ^ { ( 1 ) } = ( Y _ { , + k - 1 , 1 , Y _ { , + k - 2 , 2 , \cdots , Y _ { , + 1 , k - 1 } ) \\ I _ { , , k + 1 , Y _ { , + 2 , k + 2 , \cdots , Y _ { , }$ $I _ { 8 , t , k } ^ { ( 2 ) } = { \Delta } ( Y _ { 8 , t - 1 , k + 1 } ,$

We argue that this assumption is mild,because 1)$I _ { , , k } ^ { ( 1 ) }$are tokens from timestep after t and therefore should have less impact on the dis-tribution of$Y _ { , k }$given past tokens$H _ { , }$ $( H _ { , }$might also contain also future tokens in physi-cal time if$Z _ { s , t }$is in the masked spans); 2)al-though$I _ { S _ { 7 } t , k } ^ { ( 2 ) }$are tokens from timestep before t,they are from codebooks that are later than code-book k in the residual quantization chain,meaning that they model the residual left by codebook k (at the corresponding timesteps). Given the fact that$\{ Y _ { S , t - 1 , k } , Y _ { S , t - 2 , k + 1 , \cdots , Y _ { S , t - K + k _ { 2 } K - 1 \}$C $H _ { 8 , t } ,$,meaning that the “fitted parts” are given,and therefore the “unfitted parts”(which is the residual) should have miner impact on the distribu-tion of$Y _ { s , t , k }$Empirically,MusicGen shows that a codec language model trained with the Delay Pattern enjoys the efficiency of the naive parallel pattern, while achieving similar modeling perfor-mance as completely flattened sequence.

#### E Instructions for human listening test

Screenshots of instructions for the human listen-ing test we used on Amazon Mechanical Turk is shown in figure 8(speech editing -intelligibility),figure 9(speech editing-naturalness), figure 10(speech editing -side-by-side comparison), fig-ure 11(zero-shot TTS- intelligibility), figure 12

"A weaker condition holds for the first K tokens in un-masked spans(which accounts for at most 0.08s of speech for our models),but we omit the discussion here for simplicity

(zero-shot TTS - speaker similarity),figure 13(zero-shot TTS- naturalness). For speech edit-ing evaluation, 64 Turkers participated and we paid 474.3 USD in total;for zero-shot TTS evaluation,59 Turkers participated and we paid 457.6 USD.We only allow Turkers who are resident of the U.S.to do the tasks, and the goal is to increase the prob-ability of Turkers being native English speakers.We acknowledge that this is a perfect approach and might need to bias in judgement, but since Amazon Mechanical Turk doesn't allow selection on native language,this is the best approach we could think of as a proxy to constraining the native language.

This HIT is part of a scientific research project. Your decision to complete this HIT is voluntary, and your responses are anonymous, Your 14-digit AMT Worker ID is only collected by us for purposes of distributing payment to you, and will not be associated with any of your responses.The results of the research may be presented at scientific meetings, published in scientific journals, or made publicly available to other researchers. By clicking on the 'Submit' button for this HIT,you indicate that you are at least 18 years of age and that you consent to complete this HIT voluntarily.

##### Instructions

Given the transcript,please rate the intelligibility of each audio independently from 1-5.1 is least intelligible,and 5 is most intelligible.

Each audio might have different levels of volume,and might be recorded in noisy conditions. Please use a headset for listening and adjust the volume level for each audio to your comfort.

Please note that the radio buttons are only enabled for selection after the corresponding audio has been played to theend. Please make sure you finish listening to and rating each audio or your cannot submit the results.

·Audio 1

<!-- 0:00 日  -->
![](https://web-api.textin.com/ocr_image/external/dd201b43903974dd.jpg)

Transcript: &#36;(text0}

Intelligibility: 5: Excellent 4: Good 3:Fair 2:Poor 1:Bad

Figure 8: Instruction for speech editing-intelligibility preference. Each task contains 5 recordings. Since the frst paragraph is also presented in all other tasks in the instruction page, we only show it in this screenshot.

##### Instructions

Please rate the naturalness (i.e.human-sounding) of each audio independently from 1-5.Do not account for word content. 1 is least natural,and 5 is most natural

Some of the audio may come from internet videos or podcasts and have background noise. Please try to ignore the noise and focus only on how realistic the speech sounds,judged by the flow of the speech,intonation, prosody, emotion, and speech rate.

Please use a headset for listening and adjust the volume level to your comfort.Please note that the radio buttons are only enabled for selection after the corresponding audio has been played to the end.Please make sure you finish listening to and rating each audio or your cannot submit the results.

·Audio 1

<!-- 0:00  -->
![](https://web-api.textin.com/ocr_image/external/603916bb006c1195.jpg)

Transcript: &#36;{transcript0}

Naturalness: 5: Excellent 4: Good 3: Fair 2:Poor 1:Bad

Figure 9: Instruction for speech editing-naturalness preference. Each task contains 5 recordings.

##### Instructions

The task is to compare a pair of audio recordings of similar or same content,and determine which one sounds more like a real human speech recording.The judgement should be based on naturalness,whether the flow of the speech is correct,and whether the recording condition is consistent throughout an audio

Transcripts are shown, and we also bolded words and phrases that might differ within a pair.

Please use a headset for listening and adjust the volume level to your comfort.Please note that the radio buttons are only enabled for selection after the corresponding audio has been played to the end. Please make sure you finish listening to and rating each audio or your cannot submit the results.

·Pair 1

<!-- 0:00 1 Audio 1 Transcript: &#36;(text1_0} Audio 2 0.00 1 Transcript: &#36;{text2_0} Which one sounds more natural Audio1 is better  -->
![](https://web-api.textin.com/ocr_image/external/f395e7cc9e269d08.jpg)

Audio2 is better O Neutral

Figure 10: Instruction for speech editing, side-by-side naturalness preference. Each task contains 3 pairs of recordings.

Given the transcript,please rate the intelligibility of each audio independently from 1-5.1 is least intelligible, and 5 is most intelligible.

The audio might be from internet videos, they might have different levels of volume, and might be recorded in noisy conditions.Please use a headset for listening and adjust the volume level for each audio to your comfort.

Please note that the radio buttons are only enabled for selection after the corresponding audio has been played to the end. Please make sure you finish listening to and rating each audio or your cannot submit the results.

·Audio 1

<!-- 0:00 :  -->
![](https://web-api.textin.com/ocr_image/external/203d65ed7cb4d030.jpg)

Transcript: &#36;(transcript0}

Intelligibility: 5:Excellent 4: Good 3: Fair 2:Poor 1: Bad

Figure 11:Instruction for zero-shot TTS, intelligibility preference. Each task contains 5 recordings.

##### Instructions

Your task is to evaluate the similarity of the synthesized speech samples to the given speech reference.You should focus on the similarity of the speaker, speaking style, acoustic conditions, background noise,etc.

You should rate the recordings on the scale between 1-5, where 5 is the most similar and 1 is the least similar. In other words, please rank the recordings according to their acoustic similarity to the given reference, meaning as if they were recorded in the same place by the same speaker speaking in similar styles.

Please use a headset for listening and adjust your volume level to your comfort during the task. Please note that the radio buttons are only enabled for selection after the corresponding audio has been played to the end. Please make sure you finish listening to and rating each audio or your cannot submit the results.

·Pair 1

<!-- 0:00 : Reference Transcript: &#36;(target_spk_transcript0} 0:00 : Synthesized Transcript: &#36;(transcript0}  -->
![](https://web-api.textin.com/ocr_image/external/406c2c4327005165.jpg)

How similar is the synthesized recording to the reference? 5:Excellent 4:Good 3:Fair 2:Poor 1:Bad

Figure 12: Instruction for zero-shot TTS,speaker similarity preference. Each task contains 3 pairs.

##### Instructions

Please rate the naturalness (i.e.human-sounding) of each audio independently from 1-5.Do not account forword content.1 is least natural, and 5 is most natural

Some of the audio may come from internet videos or podcasts and have background noise. Please try to ignore the noise and focus only on how realistic the speech sounds,judged by the flow of the speech,intonation, prosody, emotion, and speech rate.

Please use a headset for listening and adjust the volume level to your comfort. Please note that the radio buttons are only enabled for selection after the corresponding audio has been played to the end.Please make sure you finish listening to and rating each audio or your cannot submit the results.

·Audio 1

<!-- 0:00 :  -->
![](https://web-api.textin.com/ocr_image/external/6d9fd9266096f8c5.jpg)

Transcript: &#36;(transcript0}

Naturalness:C5:Excellent 4: Good 3: Fair 2:Poor 1: Bad

Figure 13: Instruction for zero-shot TTS,naturalness preference.Each task contains 5 recordings.