TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation
Authors: Chenyang Le
Yao Qian
Dongmei Wang
Long Zhou
Shujie LIU
Xiaofei Wang
Midia Yousefi
Yanmin Qian
Jinyu Li
Michael Zeng
Venue: NeurIPS 2024 poster (2024)
Keywords: Speech to speech translation
Video dubbing
Voice preservation
Isochrony control
Primary area: speech_and_audio

Abstract: There is a rising interest and trend in research towards directly translating speech from one language to another, known as end-to-end speech-to-speech translation. However, most end-to-end models struggle to outperform cascade models, i.e., a pipeline framework by concatenating speech recognition, machine translation and text-to-speech models. The primary challenges stem from the inherent complexities involved in direct translation tasks and the scarcity of data.  In this study, we introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion yet facilitates end-to-end inference through joint probability. Furthermore, we propose two separated encoders to preserve the speaker’s voice characteristics and isochrony from the source speech during the translation process, making it highly suitable for scenarios such as video dubbing. Our experiments on the French-English language pair demonstrate that our model outperforms the current state-of-the-art speech-to-speech translation model.

TransVIP. Speech to Speech Translation System with Voice and Isochrony Preservation
Chenyang Le
Yao Qian
Dongmei Wang"
Long Zhou"
Shujic Liu"
Xiaofci Wang" 2
Midia Yousefi
Yanmin Qian'
Jinyu Li
Sheng Zhao?
Michael Zeng"
"Shanghai
Jiao
Tong University, China
Microsoft, USA
Abstract
There a rising interest and trend in research towards directly translating speech from one language another, known end-to-end speech-to-specch translation. However, most end-to-end models struggle to outperform cascade models, i.e., a pipeline framework by concatenating speech recognition, machine translation, and text-to-speech models. The primary challenges stem from the inherent complexities involved in direct translation tasks and the scarcity of data. In this study, introduce a novel model framework Trans VIP that leverages diverse datasets cascade fashion yet facilitates end-to-end inference through joint probability. Furthermore, we propose two separate encoders to preserve the speaker's 's voice characteristics and isochrony from the source speech during the translation process, making highly suitable for scenarios such as video dubbing. Our experiments the French-Engee English language pair demonstrate that our model outperforms the current state-of-the-an speech-to-speech translation model.
Speech Input:
Text Output:
Speech Output:
(bonjour le monde)
Hello, world.
(hello world)
(100))--00--0000 (il--/----[]]-------e- -llf sijppp-
Co.10
S
Joint Decoder
Non-Causal LM
Codec Decoder
Co
Coun
ICM
A-Enc
Prompt
Co
Joint Enc-Dec Tranlation Model
NAR Acoustic Model
Figure 1: Overview of our speech-to-speede translation framework: 1) Joint encoder-decoder model for translating speech into the target text, and coarse -rrained speech tokens, Co; 2) Non-autoregressive acoustic model for acoustic details, Co:16: 3) Codec model to convert discrete speech tokens back Information), Ca/Co.1o
to the waveform. Abbreviation: layer 0/0-15), S/A-EnoeSemannic//couuu Encoder), ICM[Isocrron Control Module).
WWok done during an internship at Microsoft Azure Al. cechermanpro@ssjuuddduu.. 'Correspondence: yaoqian @ microsooft.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
In recent years, speech translation (ST) has shifted from loosely coupled cascaded systems to more integrated, and even end-to-end systems [1]. Traditionally, cascaded systems comprised separate components for automatic speech recognition (ASR), machine translation (MT), and optionally, text-to-speech (TTS). Recent studies [2, 3, 4, 5] have successfully integrated ASR and MT into a unified end-to-end speech-to-text translation (S2TT) system. Furthermore, the integration of TTS to form a speech-to-specch translation (S2ST) system has become an increasingly researched area.
The primary challenge in developing end-to-end S2ST systems lies in performance issues. Both S2TT and TTS involve high variability with multiple reasonable outputs. Simultaneously performing these tasks increases the complexity of learning exponentially, Additionally, there is a scarcity of end-to-end S2ST data. Most available data are weakly supervised, obtained through synthesis [6] or internet parsing [7], further complicating the task. Another significant challenge is preserving speaker identity, as obtaining large-scale datasets with ground-truth paired speech spoken by the same speaker in two languages is nearly impossible. In practical applications like video dubbing, there is also demand for controlling the length of the generated target speech, ensuring that it closely matches the length of the source speech. This capability of isochrony control, however, absent in the majority of existing S2ST systems.
To address these issues, we propose TransVIP serech-to-speedd Translation framework with Voice and Isochrony Preservation. Trans VIP employs a consecutive generation approach, simplifying the complex S2ST task into two sequential tasks while maintaining an end-to-end framework. The generation of the target speech is conditioned not only on the semantic information, as in conventional S2ST models, but also the isochrony and acoustic information derived from the source speech. The corresponding overview of Trans VIP is demonstrated in Figure 1. We evaluate the performance of Trans VIP for French-Engllise mutual translation using subset of the CVSS-T test set [8]. The results demonstrate that Trans VIP outperforms the publicly available SOTA models such as larger Seamless Expressive model. The generated audio samples are available at https://aaa.mstransviip The training code and script are available at https://github.ccm// httpa//wwccoo/eeettemaaprrrrraaa The main contributions of the paper can be summarized as follows:
1. We introduce a framework for speech-to-speech translation tasks that employ a consecutive generation with joint inference. This method efficiently utilizes variety of datasets through multi-task learning to overcome the challenge of scarce paired data during the training phase, while preserving the end-to-end nature during inference.
2. We propose to disentangle various information required to learn in the training stage by employing separated encoders. It can enhance the transfer of voice characteristics and isochrony temporal alignment from the source to the target speech in the translation process. Additionally, it facilitates the design of lightweight modules for more effective individual information learning.
3. We advance the Speech [9] technology for multi-lingual tasks by distilling the semantic information from aaage-scale self-supervised model to the latest high-performing codec model. This advancement allows us to employ a textless non-autoregressive model for learning fine codec code generation without text labels, which is impractical in the conventional codec-based speech generation methods such as VALL-E [10].
4. We propose a method that refines the decoding process by incorporating sampling mechanism
within the Layer Beam Search framework, thereby enhancing the efficiency and effectiveness of non-autoregressive model decoding.
2 Related Works
The task of speech quantization is to transform continuous speech features into discrete tokens. The quantization module is usually trained by self-supervised learning (SSL) methods. The speech tokens can be divided into two categories: semantic tokens and acoustic tokens[11]. The semantic 12, 13, 14, 15, 16, 17] are usually learned by context prediction task. This kind of token is rich in context information and is suitable for downstream tasks like recognition. The acoustic tokens[18 19, 20, 21, 22), also called the neural codec, is usually trained by reconstruction task through discrete information bottleneck. This codec is suitable for audio compression and audio generation. Tokenizer [9] combines the semantic token and acoustic token by semantic distillation. The latest works propose attribute disentanglement of neural codec for
Speech Quantization

---

CE Loss
Cross Altention
Concat
Regular 0, 0, 24
Sum Pooling
+
Isochrony Embedding
Texe&Codec Autoregressive Decoder
Length PE
Reversed
24, ...
Semantic Encoder
Speech/Text
Acoustic Encoder
VAD 0-1 Embedding
1, 0,
Target Speech
Target Speech
Source Speech/Text
Clip
Replace (Sep) Token
EOS Token
Language Token
Text Token
SEP Token
Codec Token
The illustration of the training framework of the Joint Enc-Dec Model. During the training, the losses from the target speech clip, i.e., sub-part of the whole target speech, which serves prompt, are not aggregated when computing the Cross-Entropy (CE) loss. The corresponding codec labels are masked in the implementation. The semantic encoder and the auto-regressive decoder are initialized by a Seamless.M4T X2T model 4, The semantic encoder is frozen during training. In inference, all the target speech input are replaced by source speech input.
Figure 2:
better generation task support. FAcodec[23] further disentangle codec into content, prosody, acoustic, and timbre through supervision and reversed gradient.
Speech Language Model With the recent advance in speech quantization and the strong in-context learning capability of the language model, the speech-language model has shown great potential in speech generation. The GSLM family[24, 25, 26] use semantic tokens to train language models on speech continuation. VoxtLM [27] integrates text tokens with semantic tokens to perform speech recognition, synthesis, and continuation in a single causal language model. AudioLM[11 uses both semantie tokens and acoustic tokens for high-quality speech continuation. VALL-E [10] extends this idea to zero-shot TTS and proposes auto-regressively generating the first layer and non-autoregrressively the rest layers of the codec codes. Later work[28, 29, 30] have enhanced the functionality of zero-shot TTS systems, such as building cross-lingual synthesis and accurate alignment between text control signal and acoustic tokens.
End-to-End S2ST An end-to-end approach has been an ultimate goal for S2ST research. Some previous works[31, 32] have tried direct end-to-end approach for S2ST. However, by far such methods still suffer great performance issues. A solution is to include semantic tokens(text, phoneme, or semantic speech tokens) as an intermediate result. Some recent works[33, 6, 34, 35] applied speech-to-unic translation(S2UTT that translate speech input into semantic speech tokens and then generate speech through unit vocoder like Hifi-GAN, And UnitY[36] use separate speech-to-text and text-to-unit models that can optimized together. However traditional vocoder based on semantic tokens like Hifi-GAN cannot generate high-qualis speech. SeamlessExpressiveel issue using a well-designed PRETSSEL vocoder. PolyVoiced Voice[38] and AudioPaLM[[3] employees use two causal models to first translate speech into semantic tokens, then generate acoustic tokens based on semantic tokens, and then convert acoustic tokens to output speech. VioLA[40] and MSLM- S2ST[41] adopt similar approach but use single causal model in multi-tasking way. However, these approaches perform several separate auto-regressive generations during the inference process. This, to certain degree, contradicts the principle of end-to-end processing.
addresses this
3 TransVIP
3.1 Overview
Inspired by the recent advance in codec-based zero-shot TTS such as VALL-E [10], TIP contains three major parts: 1) A codec model for speech quantization and reconstruction, which is essential for auto-regressive speech generation (section 4.1). 2) An auto-regressive joint translation model to translate the input speech into goainee speech (Section 3.1). 3) A textless non-
3

---

autoregressive acoustic model to replenish acoustic details into the output speech (Section 4.2). The overall framework for generating target speech shown in Figure
In this section, we will present detailed introduction to our joint translation model, which is the key component of translation system. The training procedure of the joint model is illustrated in Figure 2. In section 4, we will discuss the codec and NAR model.
3.2 Consecutive Generation with Joint Inference
Directly generating target translation speech from input speech X presents a considerable challenge. Therefore, the primary objective of this model is to produce coarse- soaeeegrridddd speech Y' represented by a first-layer codec sequence. Instead of optimizing the direct probability distribution of PYY our optimizing target is the joint probability P(Y", X to preserve semantic accuracy in the text outputs alone, as X). Hence, our approach focuses on modeling and maximizing the conditional expression:
PIY"TTXX = P(Y" X. T)
(1)
where T represents the translated target text. This modeling is achieved through a consecutive generation framework and optimized using the beam search algorithm. Specifically, the decoder initially generates text tokens, followed by codec tokens. We employ a unique separation token to
delineate the end of text tokens and the start of codec tokens. Consequently, the codee conditioned on both the input speech and the generated text. In practical terms, exploring the entire space of T is infeasible. In beam search generation, we optimize the following approximate form:
Y = arg max (T) YETET
(2)
where T represents a subset of T with the highest speech-to-ter translation probability selected by the beam search algorithm.
3.3 Feature Disentanglement
In the real world, an ideal training dataset for speech-to-spect sp-ech generation is not readily available, where the source and target speeches are roughly equal in length and uttered by the same speaker. A real dataset comprises paired speech, either spoken by different speakers synthesized. Thus, cannot guarantee voice and duration preservation with these corpora.
To address this issue, we decouple the input feature into three parts: semantic information($), acoustic information A), and isochrony information(T) /). Formally, X (S, A, T). During training the inputs to the model are 5 from source speech, and (A, 7) information from reference target speech. While during inference, (A, I) are derived from source speech. In this way the input feature can always match the output speech in terms of voice characterizes A and isochrony I.
Semantic Information S S is extracted using a pre-trained speech encoder from an encoder- translation model, designed to provide minimal acoustic or speaker information. We also include the text encoder of the pre-trained model to enable text input and perform knowledge distillation loss on the text output part. We freeze two pre-trained encoders during the training process.
decoder speech/text-to
Acoustic information A The acoustic information that we extract relates to the distinctive acoustic characteristics of speaker's 's voice. We assume that the translated text T is independent of A. The relationship can be modeled as in Equation 3. We also take ad advantage of the one-direction mask in the auto-regressive decoder model such a relation explicitly.
P(Y A, 1, A, P(T)S A, P(YT)X
P(Y A, IT)
We use an acoustic encoder, comprising transformer blocks that learn from scratch, to extract the acoustic information. This information is sum pooled along the time dimension into a single embedding, which then replaces the embedding of separation token at the input of the auto-regressive decoder. This acoustic embedding will not affect text generation due to the causal property. is
"umg://aiincaa.ccmmooogeeeeeess

---

compatible with both gradients passing during training and beam search during inference. In training. the input to the acoustic encoder a speech prompt, i.e., sub-part of the whole target speech. We zero out the loss in the same sub-part of the label to prevent information leakage. This combined with the sum pooling forms an information bottleneck to prevent complex semantic information from passing, allowing the encoder to learn meaningful acoustic information for voice preservation. Like Classifier-free Guidance, this replacement operation is not performed with a certain probability(in our experiment p 0.5) during training.
Isochrony information I We define a frame length, our case 160ms, to quantify the duration of speech. Our Isochrony control module contains three embeddings: regular position embedding, a reversed position embedding, and a voice activity detection (VAD) 0-1 embedding. The reversed position embedding provides the model with duration information on the number of additional tokens to be generated in each step of the inference process [42], and the VAD embedding identifies whether each frame is voice-active not. Isochrony information is then represented by the sum of the three embedding sequences, which is then concatenated with the semantic feature S before being fed into the decoder through cross coss-attention. This setup enables our model to align with both the total length and the voice-active regions of the input speech, which is especially beneficial for translating long speeches with pauses.
3.4 Multi-task Training
We claim that this translation model is a cascaded trainable end-to-end model. Because the distribution PSY T,) and P(T]S, /) can be fitted both separately and jointly using various datasets. It significantly enhances data accessibility, which is crucial given the scarcity of S2ST datasets. We utilize these datasets during training as follows:
S2ST Dataset comprises quadruple of (X, Tu. T;, Y), where TN is the source text and Te the target text. We either use X or T, as input and consecutive T. and Y as target labels, or we can reverse the roles of source input and target labels.
ST Dataset is a triplet of (X, T,, T3). In our model, the missing codec label does not impede the training of speech-to- toeech-tt-tettt translation. We can use X as the input and treat Tj as an incomplete label that terminates with a separation token. In this setup, the isochrony feature is omitted and does not concatenate with the semantic feature. And, alternatively. T, can serve as the input with consecutive X and T. as the label, mirroring the approach in the S2ST dataset.
ASR Dataset, which is highly accessible, consists of pairs (X, T.). Unlike most other speech translation works, we use ASR data mainly for TTS training, fitting P(Y' I, Here, T, is used as the input with consecutive T, and X as the output labels, and the loss is not applied the part of the label corresponding to Tg. We didn't 't train on the recognition task because our semantic encoder is frozen, making this task ineffective.
4 Codec and Acoustic Modeling
4.1 Nerual Codec with Semantic Distillation
Our neural codec is designed provide high reconstruction quality and compatibility with language model predictions. We refer to our codec as the Semantic-Awar Speech Codec (SASCodec), which integrates two existing methodologies: DAC [22] and SpeechTokenizer. DAC employs innovative factorized and I. 1.2-normalizdd codes in vector quantization and finely tuned hyperparameters to achieve statc-of-the-ar reconstruction quality. SpeechTokenizer, on the other hand, uses a Hubert encoder teacher and performs semantic distillation between it and the first-layer Residual Vector Quantization (RVQ) codes. This process enriches the RVQ codes with additional semantic information, facilitating downstream language modeling tasks.
We have adopted SpeechTokenizer as our foundational model and made improvements. Originally trained with limited monolingual data, we have expanded the training to include multilingual datasets. Furthermore, we replaced the Hubert encoder with an MMS[43] encoder to enhance multilingual semantic distillation. To further improve the efficacy of semantic distillation, we use a fixed Speech- Tokenizer model as the teacher and apply mel loss to the audios reconstructed using only the first quantization layer.
5

---

To enhance reconstruction quality, we have integrated the codec with factorized and L2 12-nommalized codes. The diseriminator, loss configuration, and training hyperparameters align with those used in DAC.
4.2 Textless Non-autoregreesive Acoustic Modeling
Similar to previous work, the function of our non-autoregressive model is to enhance the acoustic details of generated speech by predicting the subsequent layers of Residual Vector Quantization (RVQ) codes based on the initial layer. We utilize a standard transformer equipped with adaptive layer normalization for this task. The model inputs the summation of embeddings from the first n layers of RVQ codes to predict the (n + 1)th layer, where n 1, 1.......A .N and N represent the total number of RVQ layers. A prompt is concatenated with the input embeddings along the time dimension. The prompt is the summation of full N layers of embeddings of the prompt speech. During the training phase, the prompt and the part to be generated are in the same language. However, in the inference phase, they are in two distinct languages for the translation task. This creates certain gap. Fortunately, our model trained using multilingual dataset can effectively generalize and bridge this gap.
Our approach diverges from prior studies by excluding textual or phonetic inputs, rendering it a completely textless model. Previous research indicated that omitting semantic information such as text or phonemes typically significantly decreases intelligibility. However, our experiments demonstrate that the textless model either matches or surpasses the performance of models with text inputs, owing to semantic distillation within the SASCodec.
The textless model offers two significant advantages: Firstly, it improves data accessibility, as the acoustic model can now be trained entirely with unsupervised data. Unlike supervised data, which requires aligned text labels and precise speech clip boundaries, unsupervised data can be arbitrarily segmented, effectively augmenting the dataset. Secondly, traditional non-autoregressive models that utilize phoneme inputs depend on accurate transcriptions establish alignment and initiate generation. This requirement can be challenging, especially in zero-shot speech translation tasks where users might not provide transcriptions. Our textless model addresses this problem by eliminating the need for transcription.
Layer Beam Search based on Sampling In previous work, the decoding of non- aon-autoregressive models is typically performed greedily, as the decoding techniques used in auto-regressive models cannot be directly applied. Greedy decoding often leads to an "early decision error" problem where the model cannot revise previous outputs once generated. To address this, we propose a method called Layer Beam Search (LBS), which adapts the beam search algorithm for non-autoregressive models.
Consider a beam size of B and a vocabulary size of V. In typical beam search algorithm, beam maintains B hypotheses. At each decoding step, each hypothesis calculates the probability distribution over V. yieldingg B V candidates. These candidates are then ranked by their aggregated scores, and the top B are selected to form the new beam. This approach is impractical in non- autoregressive models due to the enormous vocabulary size at each decoding step. For example, for speech segment containing L tokens, and a codec codebook size of C, the effective vocabulary size becomes [V| = CL,, which is too large to enumerate.
To tackle this, we employ a sampling method to choose a reasonable number of candidates. Rather than sampling from the entire vocabulary space V. we sample from the top-K candidates at each token, thus creating reduced vocabulary subspace V' of size k We sample N times from V' for each candidate in the original beam form total of B N new candidates. We then rank and form a new beam of size B from these sampled candidates. The pseudo code for the Layer Beam Search is provided in appendix D.
5 Experiments and Results
5.1 Experimental Settings
In this section, we introduce the implementation, training, and evaluation of the Trans VIP system. We conducted our experiment in the Eaglish-Frende mutual translation setting, as it had relatively

---

more data than other language pairs. Appendix A and B include more details about our models and metries.
Implementation All three models within the system are trained using 32 NVIDIA V100 32G GPUs. We utilize Fairseq2 libriary" for model building and the PyTorcc Lightning framework' for distributed data parallel (DDP) training. The training time for each of the three models is around one week.
Evaluation For speech-to-spesn translation translation evaluation, we use subset of CVSS-T [8] fr-en test containing 300 utterances. It is the first three hundred rows in the original table' for test set provided by CoVoST 2 [44]. We use source speech as the prompt. All the similarity metrics are also calculated with the source speech as the reference.
We use at most, a 10-second prompt for the joint translation model and a 5-second prompt for the NAR acoustic model. If the source speech is shorter than that limitation we just use the whole utterance as the prompt. For baseline, we compared our model with a textless S2UT model and two Seamless models. The textless S2UT model the All-to-English XM Transformer" release by SpeechMatrix(45], For seamless models, We compared with two versions, the medium version, and the expressive version. The medium version uses the same speech-to- seeech-to-text translation model as the one we use to initialize our joint translation model, while the expressive version has larger model size and stronger -steeco-peech model. For codec evaluation, we employ LibriSpeech [46] test-clean, which is widely adopted for codec and TTS evaluation. We compare our SASCodec with Encodee [21], Speechtokenizer [9] and DAC[22]. We evaluate S2ST performance on transla- tion(BEFU), ppesker&poosody similarity(SIM & AutoPCP), Isochrony control(Rate, Pause & SI.C), and naturalness. Please refer to Appendix A for the details of evaluation matrics.
5.2 S2ST Evaluation Result
In this section, will show the evaluation result of Trans VIP in translation performance, voice preservation, isochrony control, and speech naturalness.
Table 1: The subjective evaluation results of our model and baseline model. In all scores, higher values better. The results of baseline models inferred from official checkpoints. We have also conducted a statistical significance test between TransVIP and SeamlessExpressive at a 0.05 significance level. Any results that are statistically significantly better are highlighted in bold. Abbreviation: A.PCP(Au,)PCP), (Naturenns)))) * ST + TTS cascaded systems use the ST results from Seamless Expressive.
BLEU
SLC₁ 0.2
#Size
SIM
A.PCP
Rate
Pause
Nat.
Speech
Text
0.4
Dataset
Source audio CVSS-T target
2.62
0.205
2.30
0.42
0.47
0.56
0.88
3.50
Fr En
StyleTTS
33.57
35.26
0.173
2.74
0.33
0.51
Textless S2UT XM
1.2B
15.45
0.035
1.96
0.52
0.79
3.18
SeaM)essMaT(MM
1.0B
28.95
34.69
0.037
2.27
0.16
0.52
0.43
0.79
3.08
ScamlessExpressive
1.7B
30.85
35.26
0.256
2.74
0.22
0.51
0.50
0.79
2.91
TIP
1.1B
32.60
35.34
0.320
2.49
0.55
0.44
0.70
0.91
3.19
En Fr
ST V
22.50
34.89
0.418
2.87
0.27
0.54
0.65
0.89
3.32
SeamlessM4T(M)
1.0B
21.06
32.41
0.033
2.31
0.16
0.53
0.51
Scarlessexpresssve
1.7B
27.39
34.89
0.335
2.53
0.30
0.52
0.58
3.57
TIP
1.1B
27.28
33.02
0.395
2.67
0.45
0.65
0.81
0.99
3.40
MIT License
aai/bbcs//pyooochsssaaelll. Apache 2.0 License
hateryysiiillooooooooeeeeeeeeeee matrix

---

Translation Performance We evaluated the translation performance using BLEU and ASR-BLEU metrics as presented in Table 1. 1) By comparing our models, TransVIP and the Seamless baseline. with the state-of-the-art (SOTA) textless S2UT model and analyzing the generated samples, we observed that both our models significantly outperform the textless S2UT model. Notably, the textless model exhibits severe issues with repetition and "illusion" (generating audio that seems plausible but is unrelated to the input), which have been substantially mitigated in both the Seamless model and our Trans This confirms our hypothesis that including an intermediate text is crucial for successful S2ST generation. 2) Our Trans VIP demonstrates highly competitive results compared to the Seamless baselines. In the French-to French-to-English translation direction, our model achieved BLEU Seamless points higher than the SeamlessM4T medium and 1.75 points above the larger In the English-to-Frees medium by 6.22 points. Additionally, the ASR-BLEU performance of our model is comparable to that of beamlessExpressive, with only a slight difference of -0.11 points despite 1.87-point gap in the text BLEU score. The disparity in text BLEU scores is attributed to differences in model size, and our limited S2TT data precludes rectifying this issue.
score of 32.60. 3.65
direction, our model surpasses the S4T
Voice Preservation We assess speaker similarity the SIM column of Table 1. The Textless S2UT model and Sea Medium do not effectively preserve speaker identity, as evidenced by their near-zero similarity scores. In contrast, our VIP model achieves scores of 0.320 and 0.395 the fr-en and en-fr directions, respectively, slightly outperforming the Seamless baseline, which scores 0.256 and 0.335. TIP also exceeds the fr-en target speech from the CVSS-T dataset, used in our training. by 0.115 points. This indicates that our model's similarity performance transcends the limitations of its training dataset. Regarding prosody similarity, Trans VIP's AutoPCP score surpasses Textless S2UT and SeT Medium. Compared to ScamlessExpressivee Trans performs 0.25 points lower in the fr-en direction but 0. 14 points higher in en-fr. It is significant to note that, unlike Trans SeamlessExpressive explicitly incorporates a prosody encoder.
Isochrony Control We assess isochrony control through several metrics: speech length compli- (SLC) speech rate compliant, and pause number compliant. 1) Trans demonstrates superior length control, achieving an improvement of no less than 0.18 in Fr-En and 0.23 in En-Fr for SLC,. compared to baseline models without length control strategy. 2) Trans VIP achieves the highest speech rate compliance among all baselines, benefiting from an acoustic encoder that incorporates rate information into the model. 3) Regarding pause number, Trans VIP scores best in En- Fr and slightly underperforms compared to SeamlessExpressive in Fr-En. Upon reviewing the samples, we attribute this to Trans VIP's addition of new pauses in the speech to adjust the total duration.
Speech Naturalness We observed that the quality of the conditioning speech prompt significantly influences the naturalness of the generated speech, as measured by the NISQA score. For instance, when processing low-quality real French data (Fr-En), SeamlessExpressive registers the lowest score at 2.91. Conversely, with high-quality synthesized English data (En-Fr) as input, SeamlessExpres- sive achieves the highest score of 3.57. Meanwhile, Trans VIP model demonstrates consistent performance, scoring 3.19 in the Fr-En direction and 3.40 in the en-fr direction.
5.3 Codec Evaluation Result
We show the evaluation result in Table 2 and group the result by bandwidth(BW). The SpeechTok Tok- enizer that has undergone semantic distillation performs well in the NISQA score, producing a clean and natural voice. On the other hand, DAC performs well in similarity and WER, showing a strong capability of faithful reconstruction of the original speech. Finally, our SASCodec successfully combines the strength of two codecs, showing very competitive result in all four metrics.
6
Ablation Study
6.1 Ablation on Acoustic Embedding
In the process semantic distillation within the codec, a significant amount of acoustic information was disentangled from the first-layer codec. This raises the question of whether the acoustic encoder still necessary in the joint translation model. To investigate this, we leveraged the model's capability to perform inference without the acoustic embedding. We conducted such inferences and compared
8

---

Table 2: Codec re-synthesize evaluation result. The codecs are grouped by bandwidth. means the result inferred from the official checkpoint. ?means reproduced result using the official recipe. Abbreviation: NQ(number of quantizers), FR(frame rate), B B (BBadwiidth), (Natuuuress))s))
NQ
FR
BW
SIM
SIM-ot
Nat.
WER
SpkenhTokenizer* Tokenizer*
8
50
kbps
0.847
0,558
3.714
1.44
SASCodec
8
50
4 khps
0.871
0.576
3.425
1.34
Encodec*
8
75
6 khps 6 kbps
0.887
0.582
3.139
1.30
DAC*
8
75
0.900
0.590
3.387
1.31
DAC+
16
50
8 khps 8 khps
0.936
0.622
3.667
1.11
SASCodec
16
50
0.939
0.624
3.785
1.11
Encodec*
16
75
12 kbps
0.931
0.618
3,281
1.16
DAC#
16
75
12 kbps
0.967
0.646
3,636
1.07
SASCodec
24
12 kbps
0.950
0.631
3.786
1.06
the outcomes, as shown in Table 3. Without the acoustic embedding, speaker similarity scores decreased by 0.040 in the French-English (Fr-En) translations and by 0.032 in the English-French (En-Fr) translations, Additionally, there was a slight decline in the AutoPCP scores, These findings suggest that residual acoustic information remains within the first-layer codec, underscoring the necessity of including a prompt embedding.
Table 3: The ablation study the acoustic embedding in the joint translation model.
ASR-BLEU
BLEU
SIM
A.PCP
Nat.
Fr En
TIP
32.60 32.47
35.34
0.320 0.280
2.49 2.45
3.19
-A emb
35.18
3.23
En
TIP
27.28 26.84
33.02
0.395
2.67
3.40
-A emb
33.15
0.362
2.45
3.46
6.2 Ablation on Isochrony Control method
We compared our proposed Isochrony control method with and without using other strategies. We presented the results Table 4, where a) No Isochrony Control (No IC). b) Isochrony Control the Decoder (Dec IC). This involves adding the Isochrony embedding to the input of the encoder as another positional embedding. We implemented the method from [47] in our system. Isochrony Control on the Decoder with Future Pause Information (Dec IC + FPI). This is an improvement over (b). In addition to the distance the global end and VAD information, two extra pieces of information are encoded: the distance to the next pause and the number of pauses in the future. We implemented the method from [48] in our system.
demonstrates that our proposed method reaches the best isochronic control. The results also show that this approach can improve the ASR-BLEU score compared to Isochrony control on the decoder, meaning the model is more confident and accurate in the generation and makes fewer errors like repetition and truncation.
Table 4: Ablation study on the isochrony control strategy.
ASR-BLEU
Overlap
SLCp.2
SLCu.4
No IC
30.81
0.689
0.63
0.87
Dec IC
30.51
0.748
0.75
0,90
Dec IC+ FPI
30.45
0.766
0.77
0.91
Enc IC (Proposed)
30.62
0.784
0.82
0.95

---

6.3 Ablation Study on the NAR Acoustic Model
In this section, we conduct two comparisons. Firstly, we contrast the performance of a textless non-autoregressivve acoustic model with that of model trained using text transcriptions (BPE) as input. Secondly, we assess the inference results with and without the utilization of the Layer Beam Search (LBS) algorithm determine its impact on performance enhancement.
The textless model is trained on a combination of datasets including Librilight, VoxPopuli French subset, SeamlessAlign, and Common Voice English and French subsets. As some datasets do not provide text transcriptions, we substitute the LibriHeavy dataset in the model trained with text input. Additionally, in the textless model, long audio segments are randomly clipped, whereas in the model with text input, clips strictly adhere to timestamps provided in the metadata to align with text transcriptions. We compare the Fr-En uni-directional setup mitigate the impact of the absence of large-scale French dataset on the model with text input. Despite the apparent data discrepancy, textless modeling demonstrates superior performance, underscoring one of its key data accessibility advantages. All other hyperparameters, including training steps, remain consistent across both models.
The results are presented in Table 5. The textless model consistently outperforms the model with text input across all metrics of ASR-BLEU, speaker similarity, and naturalness. Moreover, employing LBS yields superior results compared to greedy decoding.
Table 5: Ablation study on the NAR Acoustic model and its inference strategy.
NAR Model
SIM
Nat. ASR-BLEU
NAR w/o text LBS
0.320
32.60
3.19
0.309
32.30
3.17
NAR w/ text LBS
0.307
31.52
3.10
0.298
31.03
3,09
7 Conclusion, Limitations and future works
In this paper, we develop a speech-to-speech translation framework that preserves speaker voice and isochrony, generating high-quality translated speech suitable for both daily communication and automatic video dubbing. Our framework can infer in an end-to-end manner, jointly considering text and speech probability during the inference phase and making full use of various data during the training phase. In the Fr-En direction, Trans VIP surpasses the current state-of-the-art model with comparable translation performance and significantly improved voice and isochrony preservation.
Limitations and future works: !) Performance & Language Support: Due to current resource and data limitations, have only trained our model the Fr-En language pair. We used approximately 5k hours of audio to train the translation model, compared to over 50k hours in previous works[7, 40, 38). This suggests that there is potential for performance improvement if we scale up the data. Additionally, we aim to extend our framework to many-to-many settings through large-scale multilingual training, similar to Seamless. 2) More Detailed Attribute Control: Upon reviewing the test samples, we found that Trans VIP occasionally alters the tone of the speech, such as translating interrogative sentences into declarative ones. is reasonable hypothesize that more detailed control of attributes like intonation or emotion could help address this issue.
Broader Impact: Trans VIP was developed improve cross-lingual communication and assist with automated video dubbing workflows. However, it carries potential risks, including the misuse of the model for impersonating specific speakers by inputting targeted text and speaker prompts. Additionally, as a generative model, there is a possibility it may produce toxic or biased outputs, although such occurrences were not observed in our test samples.
Acknowledgments and Disclosure of Funding
We extend our gratitude to Chung-Hsien Tsai, Canrun Li, and Zhen Xiao for their invaluable contribution in conducting the subjective evaluation and for their efforts in assembling the video dubbing material.
10

---

References
[1] M. Sperher and M. Paulik, "Speech translation and the end-to-end promise: Taking stock of where we are," arXiv preprint arXiv:2004.06.35. 2020.
[2] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, "Robust Speech Recognition via Large-Scale Weak Supervision." in Proceedings of the 40th International Conference Machine Learning. PMLR, 2023, pp. 28 492-288118.
[3] Z. Zhang, L. Zhou. J. Ao, S. Liu, L. Dai, J. Li, and F. Wei, "Speechut: Bridging speech and text with hidden-unit unit for encoder-decoder based speech pre-training: arXiv preprint wxXi:22100003330 2022.
[4] Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, 7. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, 7. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghami, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. Beaufays, and Y. Wu, "Gongle USM: Scaling Automatic Speech Recognition Beyond 100 Languages," arXiv preprint arXiv:2303.0100 2023.
[5] C. Le, Y. Qian, L. Zhou, S. Liu, Y. Qian, M. Zeng. and X. Huang, "ComSL: Composite Speech-Language Model for End-to-End Speech-to-Tee Text Translation, in Advances Neural Information Processing Systems, vol. 36, 2023, pp. 58 312-555 323.
[6] R. Huang. J. Liu, H. Liu, Ren, Zhang. J. He, and Z. Zhao, "TranSpeech: Speech tochh-tt--pppp Speech Translation With Bilateral Perturbation," in The Eleventh International Conference on Learning Representations, 2022.
[7] S. Communication, I.. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, P.-A. Duquenne, H. Elsahar, H. Gong. K. Hefferann J. Hoffmman. C. Klaiber, P. Li, D. Licht, J. Maillard, A. Rakotoarison. K. R. Sadagopan, G. Wenzek, E. Ye, B. Akula, P.J. Chen, N. E. Hachem, B. Ellis, G. M. Gonzalez, J. Haaheim, P. Hansanti. R. Howes, B. Huang, M.-J Hwang, H. Inaguma, S. Jain, E. Kalbassi, A. Kallet, I. Kulikov, J. Lam, D. Li, X. Ma, R. Mavlyutov, B. Peloquin, M. Ramadan, A. Ramakrishhan, A. Sun, K. Tran, Tran, I. Tufanov, V. Vogeti,
Wood, Y. Yang, B. Yu, P. Andrews, C. Balioglu, M. R. Costa- jussá, O. Celebi, M. Elbayad, C. Gao, F. Guzmán, J. Kao, A. Lee, A. Mourachko, J. Pino, S. Popuri, C. Ropers, S. Saleem, H. Schwenk, P. Tomasello, C. Wang. J. Wang, and S. Wang, "SeamlessM4T: Massively Multilingual & Multimodal Machine Translation." arXiv, 2023.
[8] Y. Jia, M. Tadmor Ramanovich, Q. Wang, and H. Zen, "CVSS corpus and massively multilingual opeech-to-speech translation." in Proceedings of the Thirteenth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri. C. Cieri. T. Declerck. S. Goggi, H. Isahara, B. Maegaard, Mariani, H. Mazo, Odijk, and S. Piperidis, Eds. Marseille, France: European Language Resources Association, Jun. 2022, pp. 6691-6703 [Online]. Available: http:://ccamthlooog../22222..1 1.720
[9] X. Zhang, Zhang, Li, Zhou, and X. Qiu, "SpeechTokenizer. Unified Speech Tokenizer for Speech Language Models," in The Twelfth International Conference on Learning Representations, 2023.
[10] C. Wang, Chen, Y. Wu, Zhang, Zhou, S. Liu, Chen, Y. Liu, H. Wang. J. Li et al., "Neural codec language models are zero-shot text to speech synthesizers," arXiv preprint XrXiv:2301.02111 2023.
[11] 7. Borsos, R. Marinier, D. Vincent, F. Kharitonov; O. Pietquin, M. Sharifi, D. Roblek, O. Tchoul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, "Audiol M: A I Language Modeling Approach to Audio Generation," IEEF/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2523-2533, 2023.
[12] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations," in Advances in Neural Information Processing Systems, vol. 33. Curran Associates, Inc., 2020, pp. 12 12449-144 460..
[13] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "HuBERT: Self- Supervised Speech Representation Learning by Masked Prediction of Hidden Units," IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021.
[14] Y.-. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang. and Y. Wu, "W2y-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Tre-Training." in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 244-250.
[15] Y. Cheng, Y. Zhang, M. Johnson, W. Macherey, and A. Bapna, "MuS~2SSLAM: Multitask, Multilingual Speech and Language Models," in Proceedings of the 40th International Conference on Machine Learning. PMLRR 2023, pp. 5504-5520.
[16] J. Ao, R. Wang. L. Zhou, C. Wang, S. Ren, Y. Wu, S. Liu, T. Ko, Li, Y. Zhang. Z. Wei, Y. Qian, J. Li, and F. Wei, "SpeechTS: Unified-Mool Encoder-
Trainrannnn for Spoken Language Processing." " in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 5723-5738.

---

[17] C.-C. Chiu, J. Qin, Y. Zhang.J. J. Yu, and Y. Wu, "Self-supervised learning with hundom-projectin quantizer for speech recognition." in Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022. pp. 3915-3924.
[18] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, "SoundStream: An End-to-End Neural Audio Codec," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2022.
[19] J. Valin, K. Vos, and T. Terriberry, "Definition of the opus audio codec," Tech. Rep., 2012.
[20] M. Dietz, M. Multrus, V. Eksler, V. Malenovsky, E. Norvell, H. Pobloth, L. Miao, Z. Wang, L. Laaksonen, A. Vasilache, Y. Kamamoto, K. Kikuiri, S. Ragot, J. Faure, H. Ehara, V. Rajendran, V. Atti, H. Sung, E. Oh, H. Yuan, and C. Zhu, "Overview of the EVS codec architecture," in 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5698-5702.
[21] A. Défosse7, J. Copet, G. Synnaeve, and Y. Adi, "High Fidelity Neural Audio Compression," Transactions Machine Learning Research, 2023.
[22] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, "High-Fidelity Audio Compression with Improved RVQGAN," Advances in Neural Informaiinn Processing Systems, vol. 36, pp. 27 980-27999 2023.
[23] Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang. Y. Liu, Y. Leng, K. Song, S. Tang, Z. Wu, T. Qin, X.-Y. Li, W. Ye, S. Zhang. J. Bian, L. He, J. Li, and S. Zhao, "NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models," 2024.
[24] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux, "On Generative Spoken Language Modeling from Raw Audio," Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336-1 1354, 2021.
[25] E. Kharitonov, A. Lee. A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T.-A. Nguyen, M. Rivière, A. Mohamed, E. Dupoux, and W.-N. Hsu, "Text-Free Prosody-Aware Generative Spoken Language Modeling," in ACL 2022 Association for Computational Linguistics, ser. Proceedings of the 60th Annual Mceting of the Association for Computational Linguistics, vol. Long Papers. Dublin, Ireland: MIT Press, 2022, pp. 8666-8681.
[26] T. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and F. Dupoux, "Generative Spoken Dialogue Language Modeling," " Transactions of the Association for Computational Linguistics, vol. 11, pp. 250-266, 2023.
[27] S. Maiti, Y. Peng. S. Choi, J.-W. Jung. X. Chang, and S. Watanabe, "VoxtLM: Unified Decoder-Oniy Only Models for Consolidating Speech Recognition, Synthesis and Specch, Text Continuation Tasks," ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024,
pp. 13 326-266332 330.
[28] Z. Zhang, L. Zhou, C. Wang. S. Chen, Y. Wu, S. Liu, Z. Chen, Liu, H. Wang. Li et al., "Speak foreign languages with your own voice: Cross-lingual neural codec language modeling," arXiv preprint axXiv:2333.033926, 2023.
[29] E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, "Speak, Read and Prompt: High-Fidclity Text-to-Speede with Minimal Supervision." 2023.
[30] R. Huang, C. Zhang, Y. Wang. D. Yang. L. Liu, Z. Ye, Z. Jiang, C. Weng, Z. Zhao, and D. Yu, "Make-A- Voice: Unified Voice Synthesis With Discrete Representation." 2023.
[31] Y. Jia, M. T. Ramanovich, T. Remez, and R. Pomerantz, "Translatotron 2: High-quality direct speech- to-speech translation with voice preservation." in Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022. pp. 10 120-10 134.
[32] E. Nachmani, A. Levkovitch, Y. Ding, C. Asawaroengchai, H. Zen, and M. T. Ramanovich, "Translatotron 3: Speech to Speech Translation with Monolingual Data." in ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 186--16690.
[33] A. Lee, P.-J. Chen, C. Wang, J. Gu, S. Popuri, X. Ma, A. Polyak, Y. Adi, Q. He, Y. Tang, J. Pino, and W.-N. Hsu, "Direct speech-to-speech translation with discrete units." arXiv, 2022.
[34] K. Wci, I.. Zhou, 7. Zhang, I. Chen, S. Liu, I.. He, J. Li, and F. Wei, "Joint pre-training with speech and bilingual text for direct speech to speech translation." in ICASSP 2023-202. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEBE, 2023, pp- 1-5.
[35] A. Diwan, A. Srinivasan, D. Harwath, and E. Choi, "Unit-based Sarech-oo-Specde Translation Without Parallel Data." arXiv, 2023.
[36] H. Inaguma, S. Popuri, I. Kulikov, P.-. Chen, C. Wang, Y.A. Chung. Y. Tang, A. Lee, S. Watanabe, and Two-pass
J. Pino. "UnitY: Two pass Direct Speech to speeeh Translation with Discrete Units." in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers,
12

---

J. Boyd-Graber. and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguisties, 2023, pp. 15 680-15680.
[37] S. Communication, L. Barrault, Y.A. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler. P.-A. Duquenne, B. Ellis, H. Elsahar, J. Haaheim, J. Hoffman, M.-J. Hwang. H. Inaguma, C. Klaiber, Ramakrishnan, Kulikov, Li, Licht, J. Maillard, R. Maviyutov, A. Rakotoarison, K. R. Sadagopan,
T. Tran, G. Wenzek, Y. Yang. E. Ye, I. Eytimov, P. Fernandez, C. Gao, P. Hansanti, E. Kalbassi, A. Kallet, A. Kazhevnikov, G. M. Gonzalez, R. S. Roman, C. Touret, C. Wong, C. Wood, B. Yu, P. Andrews, C. Balioglu, P.J. Chen, M. R. Costa-jussă, M. Elbayad, H. Gong, F. Guzmán, K. Heffernan, S. Jain, J. Kao, A. Lee, X. Ma, A. Mourachko, B. Pelnquin, J. Pino, S. Popuri, C. Ropers, S. Saleem, H. Sehwenk, A. Sun, P. Tomasello. C. Wang. J. Wang. S. Wang, and M. Williamson, "Seamless: Multilingual Expressive and Streaming Speech Translation," " 2023.
[38] Q. qian Dong, Z. Huang. Q. Tian, C. Xu, T. Ko, Y. Zhao, S. Feng. T. Li, K. Wang. X. Cheng, Yue, Y. Bai, X. Chen, L. Lu, Z. Ma, Y. Wang. M. Wang, and Y. Wang. "PolyVoice, Language Models for Speech to Speech Translation," in The Twelfth International Conference on Learning Representations, 2023.
[39] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. Padfield, J. Qin, D. Rozenberg. T. Sainath, J. Schalkwyk, M. Sharifi, M. T. Ramanovich, M. Tagliasacchi, A. Tudor, M. Velimirović, D. Vincent, J. Yu,
Y. Wang. V. Zayats, N. Zeghidour, Y. Zhang, Z.. Zhang. L. Zilka, and C. Frank, "AudioPaI.M: A Large Language Model That Can Speak and Listen." " arXiv, 2023.
[40] T. Wang, L. Zhou, Z. Zhang, Y. Wu. S. Liu, Y. Gaur. Z. Chen, J. Li. and F. Wei, "VioLA: Unified Codec Language Models for Speech Recognition. Synthesis, and Translation." " arXiv. 2023.
[41] Y. Peng, I. Kulikov, Y. Yang. S. Popuri. H. Lu. C. Wang. and H. Gong, "MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Specct Translation with Speaker Style Preservation," " 2024.
[42] L. Miculicich, Y. Xic, S. Wang, and P. He, "Summarization with precise length control," 2023.
[43] V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel- Zarandi, A. Baevski, Y. Adi, X. Zhang. W.-N. Hsu, A. Conneau, and M. Auli, "Scaling Speech Technology to 1,000+ Languages," " Journal of Machine Learning Research, vol. 25, no. 97, pp. 1-52, 2024.
[44] C. Wang, A. Wu, and J. Pino, "CoVoST 2 and Massively Multilingual Speech-to- Text Translation," 2020.
[45] P.-A. Duquenne, H. Gong, N. Dong, J. Du, A. Lee, V. Goswami, C. Wang. J. Pino, B. Sagot, and H. Schwenk, "SSeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speece Translations," in Proceedings of the 61st Annual Meeting the Association for Computational Linguistics (Volume 1: Long Pupers), A. Rogers. J. BooddGGraber and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, 2023, pp. 16 25111166.9.
[46] V. Panayotov, G. Chen, Povey, and S. Khudanpur, "Librispeech: An ASR corpus based on public domain audio books," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206-520 -2100.
[47] Y. Wu, J. Guo, X. Tan, C. Zhang, B. Li, R. Song, I.. He, S. Zhao, A. Menezes, and J. Bian, "VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing," in Proceedings of the AAAI Conference on Artificial Intelligence., 2022.
[48] P. Pal, B. Thompson, Y. Virkar, P. Mathur, A. Chronopoulou, and M. Federico, "Improving isochronous machine translation with target factors and auxiliary counters," arXiv preprint wrXiv:2305.13200 2023.
[49] M. Post, "A call for clarity in reporting BLEU scores." in Proceedings of the Third Conference on Machine Translation: Research Pupers. Belgium. Brussels: Association for Computational Linguistics, Oct. 2018, pp. 186-19) 191. [Online]. Available: https:/wwww.alleebbor/raatttttte
18-6319
[50] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, "WavLM: Large-Scale Self- Supervuper Pre-Training for Full Stack Speech Processing," IEFE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505-1518, 2022.
[51] W.-C. Huang, B. Peloquin, J. Kao, C. Wang, H. Gong, E. Salesky, Y. Adi, A. Lee, and P.-. Chen, "A Holistic Cascade System, Benchmark, and Human Evaluation Protocol for Expressive Speech-to-Specch Translation," in ICASSP 2023 2023 IEEE International Conference on Acousticx, Speech and Signal Processing (ICASSP), 2023, pp. 1-5.
[52] G. Mittag and S. Möller, "Deep Learning Based Assessment of Synthetic Speech Naturalness," in Inter- speech 2020. ISCA, 2020, pp. 1748 1758-1752.
[53] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, and W.-. Hsu, "Voicebox: Text- TexdeG Multilingual Universal Speech Generation at Scale," Advances in Neural Information Processing Systems, vol. 36, pp. 04005- (005-14000 0304,, 2023.
13

---

[54] Y. Jia, M. Tadmor Ramanovich, Q. Wang, and H. Zen, "CVSS Corpus and Massively Multilingual Speech- (o-Speech Translation." Proceedings the Thirleenth Language Resources and Evaluation Conference, N. Calzolari, Béchet, P. Blache, K. Choukri. C. Cieri, T. Declerck, S. Goggi. H. Isahara, Maegaard,
J. Mariani, Mazo, Odijk, and Piperidis, Eds. Marseille, France: European Language Resources Association, 2022, pp. 0691-6703.
[55] R. Ardila, Branson, Davis, M. Kohler, Meyer, M. Henretty, R. Morais, Saunders, Tyers, and Speech Corpus," Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, E. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, Eds. Marseille, France: European Language Resources Association, 2020, pp. 4211-4222.
G. Weher, "Common Voice: Mussively-
[56] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve. A. Joulin, A. Mohamed, and E. Dupoux, "Libri-Light: A Benchmark for ASR with Limited or No Supervision." in ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7669-766 7673.
[57] C. Wang. M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, "VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation I carning, semi-Supervised Learning and Interpretation." in Proceedings the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, 2021. pp. 993-1003.
14

---

A Evaluation Metrics
For S2ST evaluation, we evaluate translation performance (BLEU and ASR-BLEU), speaker similarity (SIM), prosody Similarit(AAtooCCP isochrony control (Rate, Pause, and SIC--02/0..), and naturalness (NISQA). The score is the higher the better for all the S2ST metrics.
BLEU calculated between text oulput and text label using the "corpus_bbl bleu' method of sacrebleu [44)) To calculate speech BaEu known as ASR-BLEU), we utilize a whisper large v3 model to transcribe the generated speech into text and then calculate the BLEU score between the transcribed text and text label.
The Speaker Similarity(SIM) score is obtained by calculating the cosine distance of the speaker embedding of two sentences. The embedding comes from a WavLMM500--base speaker verification model" In the context of S2ST evaluation, this SIM score is generally computed by using two embeddings extracted from source speech and translated target speech, respectively. This SIM score reported is typically lower than that of zero-shot TTS since the two embeddings used for the computation are extracted from speech in different languages.
AutoPCP" is model- based evaluation tool for Prosodic Consistency Protoco((PCP)51] which evaluate overall prodsodic similarity.
Rate" calculated correlation in speaking rate (measured by syllables per second). Pause" calculate the correlation in pause number. Text transcription is needed for the rate and pause calculation. Therefore they are not suitable for textless S2ST models.
(Specc Length Compliant) is proposed in [47] that calculates the fraction of generated speech with duration within [1 p, 1 + p] times of the original speech input.
For naturalness evaluation, we use NISQA-TTS(VI.00* which is model to evaluate the naturalness and quality of synthesized speech. [52]
codec evaluation, we use metries that are commonly employed in zero shot- TTS evaluate the re-synthesize quality of the codec, which are 1) SIM. 2) NISQA, and 3) WER. The NISQA metrics are the same as those in the above S2ST evaluation. A market-lcading Speech Recognition API is used for Word Error Rate (WER) calculation. It measures the correctness and intelligibility of re-synthesized speech. We use two similarity scores. SIM is used calculate the similarity between original specch and resynthesised speech. Generally, this SIM value very high because the two embeddings used the calculation are extracted from speech that shares the same content in the same language. Following [10, 53]. the SIM-o metric is computed by first segmenting the audio into two parts: a 3-second prompt and the remaining portion. The latter part is then re-synthesized. Finally, similairy score is calculated between the re-synthesized audio and the original prompt.
B Implementaion Details
The primary encoder-decoder (Enc-Dec) architecture of the joint model is initialized using the SeamlessM4T S2T model, while the other submodules are trained from scratch. The first-layer RVQ codes are integrated into the vocabulary of the foundational S2T model. As mentioned in Section 3.4. this model trained using multiple datasets, including two S2ST datasets: CVSS- T[54] and SeamlessAlign" 1 one internal ST dataset, and one ASR dataset: Common Voice[55] version 15 (English and French subsets). An offline preprocessing step is performed to convert the speech labels into discrete codec form. During this process, we utilize a VAD tool 4 to eliminate silence at the beginning and end of the speech recordings. This VAD tool also used to generate the isochrony embeddings during both the training and inference phases. The input to the speech semantic encoder is Fbank features and text is tokenized into BPE, which is consistent with the Seamless framework. The acoustic encoder is a six-layer standard Transformer encoder with a hidden size of 1024. Its input features are extracted from raw waveforms by the SASCodec encoder, providing richer acoustic information than Fbank features. For decoding employed beam search algorithm with beam size set to 5.
Joint Translation model
NAR Acoustic Model This 12-layer transformer model is trained from scratch and utilizes mainly two unsupervised corpora: Libriiight((56, containing 52k hours of English speech, and Vox VoxPopuli[57] French subset, encompassing 23k hours of French speech. We also included audio from SeamlessAlign and Common Voice in
hhtpp:://gttuub. "btts://gituub.oo/mmjsssisi http:///ww..oommmmoo/ssssreebttt Apache 2.0 License.
verificationn CC BY-SA 3.0 Un-
ported License
CC CC-BY-NC 4.0 License
MIT License
"banss//gghhuoocoom/msnessssssss MIT License
15

---

val_s2c_cc__tt ce_loss
SASC
SpeechTokenizer
3
trainer/global_ste
20k
40k
60k
80k
100k
120k
140
Figure 3: Validation loss of Trans VIP using different codecs.
the training. The datasets are weighted differently during training to ensure that data from both languages sampled with equal probability. In inference, we employ the LBS mentioned in section 4.2. There are RVQ layers in the model, which means 15 generation steps. The beam size is 10, the sampling number is 20 and K is 3.
SASCodec Model We train the model on audio with a frequency of 16k Hz. The downsample rate set at 320, resulting in 50 tokens/frames second, instead of the 75 tokens used in the official DAC checkpoint. This adjustment is primarily made to ensure consistent token rate that matches the semantic token rate, thereby facilitating the distillation process. The model is trained with 24 RVQ codebooks while only are employed the translation system. The SASCodec model trained using full set corpus, which includes hours of multilingual speech data. During each training iteration, random 2-second segment is cropped from the speech samples.
Common Voice version 15
C Additional Ablation on the Choice of Codec
section, we compare training TransVIP using different codecs: SpeechTokenizer [9] and DAC [22]. this ablation study, the joint translation model is trained with a subset containing only CVSS-T Fr-En uni-direction data. For the NAR acoustic model, both DAC and SASCodec use 16 codec layers, while SpeechTokeniig uses 8 layers, as it only has 8-layer version. The results shown in Table 6. Compared to SpeechTokenizer, the aspects. Most notably, the speaker similarity
model trained with SASCodec exhibits superior performance improved by 0.04, from 0,226 to 0,264, aligning with the improvement in codec re-synthesis results.
Additionally, we find that when trained with DAC, the joint translation model fails to generate reasonable output even when the training loss appears to converge. To investigate this, we plot the validation loss curve of the joint translation model in Figure 3. It shows that Tokenzzerr has the lowest loss, followed by SASCodec, and then DAC. Furthermore, training steps. We attribute this finding to two reasons: I) SokechTokenizrr contains most semantic information in its first-layer codec. Although SASCodec also undergoes semantic distillation, it still contains acoustic information in the first layer, trading for better speech reconstruction quality. 2) The enhancement of RVQ in DAC, which is also utilized in SASCodec, lcads to pose challenge for language modeling. However, our SASCodec method effectively balances the quality of reconstruction and the affinity of the language model.
observed that SpeechTokenize can generate reasonable output with minimal
utilization rate across the codebook. This could
D Pseudo code for Layer Beam Search
The pseudo code in the Pytorch style is shown in algorithm 1.
16

---

Table 6: Ablation study on the choice of codec.
BLEU
SLC,
Codec Model
SIM
Nat.
Speech
Text
0.2
0.4
SpeechTokenizer
29.81 30.62
34.18 34.30
0.226
0.76
0.93
3.02
SASCodec
0.264
0.82
0.95
3.09
Algorithm 1: Layer Beam Search
Input:
inputs: Inputs to the NAR model o_codebook: Number of RVQ layers. Default: 16
beam__ize jaze_aize: Beem_sizze Default: 10
n_sample: Number of sampled candidates of each hypothesis. Default: 20
K: At each token only sample from the top K candidates. Default: 3
Output:
same class the inputs and contains generation results. Function Generatellar (inputs, n codebook, beam. size, n sample, K):
for layer t 0 to ccceebooooo.................oo.. - do
logits +- forward
!probs - log_ sof tmax (logits dim -1) topk values, lopk ids <- cop_k(logits, K)
condidates +- caroa(beara__tt size,,sieeee,ee nI sample,, seq seq_len)
for i +- 0 t__amppe do
samples
sompled_ics + gather(topk:
samples)
10
compled_[probe - gather (lprobs, sampled_ids
11
avg_score + mean (sampled__pppotii
candidates (,, sampled_ide ids
total_score(:=
total_scorel:: ang_score
14
values, flat, ids +- top pop__faattee((llall (coore_scorre) heamm_size rows <- felalllill ids Il beamn_sie size
15
cols flat. ids % beam_sszz total_score score values. eeaaues.eppadd next_ing mext_inputs- +
17
beam_size)
18
19
for (row, col) in zip(rows, cols) do imputs(row)
h condidates frow, col]
21
nest ceanf_inputs..om httpiiiiussspppeddeeeeee (ctt (a, hl))
23
inpuls <- nexl inpuls
24
return inputs
E
Model size
We report in Table 7 the detailed decomposition of module sizes of TIP and baseline models.
Table 7: Parameters of the building components. Only the modules that required in the inference stage are calculated.
Speech Encoder
Decoder
T2U
NAR
Total
Textless S2UT XM
970M
204M
1174M
SeamlessM4T-Mediae
366M
415M
170M
951M
Seamless- Seamless-Expressin Expressive
635M
867M
204M
1706M
TIP
445M
415M
244M
1104M
17

---

CMOS Naturalness Test
In tech of the following case, you will hear two voices. Play all voices first, then select which one more natural
Voice
Focus Neturelness. Pleese evelucte netureiness of the voices, Select option based the netureliness, jabice intu te fluency. rhythm, intoration, overall Estening experience of speech.
Listen Carefully: leten each sudio file Ianore External Factors: Pinase do not consider other fectors such as the contents of these two utterances could be differeen. Only forus on the naturalness the spench.
Batter
Yorzo ls Pignity becter
Can wicch bedtarr
sightly
much nattar
Figure The instruction of CMOS naturalness test
F Data Preprocessing and Pseudo-labeling
As mentioned in section 3.2, text labels play a crucial role in joint translation model training. However, text labels are absent in some and-to-end S2ST datasets such as ScamlessAlign and SpeechMatrix. To utilize these datasets, pseudo text labels required. We employ the following procedure preprocess such datasets:
Generate pseudo transcription labels source and target speech using an ASR model. In our experiments. we use the Whisper large v3 model.
2. Extract codec from the souree and target speech, removing silence the beginning and end using VAD model.
3. Validate and score the data using pre-trained pre-trained mBAAT--arge-50mmm model. 15 The mean confidence scores in both directions are averaged to determine the confidence score of the data pair.
translation model. In our experiments, we use the
During training, we filter out data with low confidence scores to avoid potential bad data or ASR errors.
G Subjective Evaluation
Given that the NISQA score can present difficulties in evaluating the quality naturalness of noisy speech, we also assess the naturalness of speech using subjective metric known as the Comparative Mean Opinion Score (CMOS). For this evaluation, 12 professional linguistic experts judged 120 pairs of utterances translated by Trans and SeamlessExpressive. The results indicated that Trans VIP's performance is significantly better (p<0.01) than SeamlessExpressive, i.e., TransVIP outperforms SeamlessExpressive by achieving preference higher 0.46. The instruction of CMOS naturalness test is shown in Figure 4. Much better/better/lightly better/can' 't tell equals to 3/2/1/0 score respectively.
httt::/hhuugigghhcccoofeebbbbbbt large-50 many- many-manymmmmma mmt
18

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our scope and contribution in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
discussed the limitation the work the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well sell---ecifiiation. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might he violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, c.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicablee the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include sophisticated experiment result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
19

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We believe this paper has contained enough information to reproduce the main result. We provide details about model structure, data processing, training, inference and evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We include the code and the data preparation script in the supplementary material. And will open source them in the near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
for more details.
While we encourage release of code and data, we understand that this might not possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //np///ps..//////// cc/public/ gu..des//ooteSuubleeeslooloooo for more details.
20

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All the training and test details are included in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct statistical significance test under 0.05 importance level in the main result. We didn't 't do error bar because it will be too computational expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We report GPU resources and the training time in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ http:///ww....ppp.l.e//haaaaaaaa cnoudeliness
Answer: [Yes]
Justification:
This work does not deviate from the Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We have section in the appendix that discusses the potential harms and countermeasures.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e-g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a dircct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feodback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Currently do have plan for releasing the final model. we are going to release, we will make sure to release a safeguard with
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use usal-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
22

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited all the assets by listing the URL in footnote, citing paper and explicitly noting the license if it exists one.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should has curated licenses for
he provided. For popular datasets, paperawi thcode. www.tthcccodtttteaeeeeeeeeaaaaae
some datasets. Their licensing guide can help determine the license a dataset.
existing datasets that are re-packaged, both original license and the license of derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out to asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We write instruction in the code README about how to prepare data, launch training and run inference.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our project does not involve crowdsourcing experiments nor rescarch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
23

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
24


Reviews:
[{"review_id": "xpIt2Uajbo", "reviewer": "Reviewer_WU4H", "summary": "The paper introduces TransVIP, a novel speech-to-speech translation system designed to maintain both the speaker's voice characteristics and isochrony during the translation process. TransVIP simplifies the complex task of speech-to-speech translation (S2ST) by breaking it down into two sequential subtasks while retaining an end-to-end framework. It conditions the generation of the target speech not just on semantic information, but also on isochrony and acoustic details extracted from the source speech. The paper demonstrates the effectiveness of TransVIP through experiments on French-English translation, showing superior performance compared to state-of-the-art models.", "questions": "The paper is difficult to read, as some technical details are missing, making it challenging to fully understand the design. For example, it is unclear how the acoustic encoder learned from scratch can be expected to extract the desired acoustic information.", "limitations": "Not applicable", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is interesting. The recent studies have paid attention to voice preservation during speech-to-speech translation (S2ST), and this paper further proposes to preserve the isochrony information for ideal speech translation effects. This may provide useful insights for future work.\n2. The paper decouples the S2ST model into multiple modules and offers several significant innovations.\n3. The proposed method achieves new state-of-the-art performance.", "weaknesses": "1. The title may not be fully representative, as only a part of the innovation focuses on voice and isochrony information preservation. As shown in Section 3, only the first subsection is closely related to the title. While the latter subsections present good innovations, the overall relevance among them could be strengthened.\n2. The experimental comparison is limited. This paper focuses on voice and isochrony preservation, but does not provide any comparison with related work, like VALL-E X and PolyVoice. As Seamless does not consider voice information, the real advantage of voice preservation is unclear.\n3. This paper proposes multiple innovations with many techniques. However, there are few ablation studies to analyze the individual components. There is also a lack of in-depth analysis on the design choices for voice and isochrony preservation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.\n\n**R1. About not fully representative title (Weakness 1)**\n\nThank you for acknowledging the numerous innovations presented in our paper. We will refine the title or add a subtitle to encompass as many aspects as possible.\n\n**R2. About limited experimental comparisons (Weakness 2)**\n\nWe are always eager to compare our work with related research to verify the effectiveness of our proposed methods. However, most works, such as PolyVoice and MSLM-S2ST, neither have open-sourced models/codes nor matched language pairs for investigation. As we know, S2ST is a complicated system, and reproducing others' entire systems is both challenging and unaffordable.\nMoreover, we need to point out that Seamless does consider voice information. According to its technical report (https://arxiv.org/pdf/2312.05187), they explicitly encode the speaker and prosody information into the speech generation process. Additionally, the Seamless team leveraged far more data for training than we did. Therefore, we already leveraged a very strong baseline to compare with our model.\n\nWe will also add following results from cascaded system (ST+TTS) as a comparison to our model.\n| | ASR-BLEU | BLEU | SIM | AutoPCP | Rate | Pause |SLC_0.2 | SLC_0.4 | Nat. | \n|---|---|---|---|---|---|---|---|---|---|\n| ST + StyleTTS (Fr->En) | 33.57 | 34.58 | 0.173 | 2.74 | 0.33 | 0.51 | 0.56 | 0.85 | 3.25 |\n| TransVIP (Fr->En)          | 32.60 | 35.34 | 0.320 | 2.49 | 0.55 | 0.44 | 0.70 | 0.91 | 3.19 |\n| ST + VALLE-X (En->Fr) | 22.50 | 34.89 | 0.418 | 2.87  | 0.27 | 0.54 | 0.65 | 0.89 | 3.32 |\n| TransVIP (En->Fr)          | 27.28 | 33.02 | 0.395 | 2.67  | 0.45 | 0.65 | 0.81 | 0.99 | 3.40 |\n\nwhere \n\n1)\tST is Seamless Speech-to-Text translation model \n\n2)\tWe evaluated cascaded system by using TTS model from StyleTTS (open sourced) and VALLE-X (implemented) and reported the better one in terms of objective measurement.\n\n**R3. About few ablation studies and in-depth analysis (Weakness 3).**\n\nWe have presented the ablation studies, such as w/ and w/o acoustic embedding, the choice of different codec, w and w/o text BPE and Layer Bean Search in NAR acoustic model. Due to the limited space for the main content, we had to present the ablation studies in the appendix. We will also include an ablation study related to isochrony preservation in the revised version. In this study, we compared our model with the following three baselines. We will report the ASR-BLEU, SLC_p (Speech Length Compliant, as defined in the paper), and Overlap ratio (i.e., speech overlap between the reference and the hypothesis) as follows.\n\n| | ASR-BLEU | Overlap | SLC_0.2 | SLC_0.4 |\n|---|---|---|---|---|\n| No IC | 30.81 | 0.689 | 0.63| 0.87 |\n| Dec IC| 30.51 | 0.748 | 0.75 | 0.90 |\n| Dec IC + FPI | 30.45 | 0.766 | 0.77| 0.91|\n| Enc IC (Proposed) | 30.62 |0.784 |0.82| 0.95|\n \nwhere\n\n1.\tNo Isochrony control (No IC).\n\n2.\tIsochrony control on the decoder (Dec IC). This involves adding the Isochrony embedding to the input of the encoder as another positional embedding. We implemented the method from ref [1] in our system.\n\n3.\tIsochrony control on the decoder with future pause information (Dec IC + FPI). This is an improvement over above 2. In addition to the distance to the global end and VAD information, two extra pieces of information are encoded: the distance to the next pause and the number of pauses in the future. We implemented the method from ref [2] in our system.\n\nRef: [1] Y. Wu, et al. “VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing,” AAAI, 2023.\n\nRef: [2] P. Pal, et al. “Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters”, Interspeech, 2023.  \n\nPlease let us know if more ablation studies should be added.\n  \n\n**R4. About unclear writing (Question 1)**\n\nWe will revise the paper as clear as possible. For example, we have refined the description as follows to address how the acoustic encoder learned from scratch can be expected to extract the desired acoustic information.\n\nMany previous works have adopted a reversed gradient approach to remove semantic information from acoustic features. However, these approaches require an additional decoder and training objective, which increases the training burden.\nIn our work, we use the information bottleneck to train the acoustic extractor. The sum pooling serves as the information bottleneck, preventing too much information, especially semantic information, from passing through. Additionally, we design the system to use part of the target speech as input and predict the other part, making it less likely for the acoustic encoder to learn anything semantically meaningful. This approach integrates seamlessly into the original training process, with no extra modules or loss required.\n\n**Finally, we would like to express our gratitude once again for your time and effort in reviewing our paper. Considering the multiple innovations, adequate ablation studies, added comparisons and improved presentation of our paper, we would greatly appreciate it if you could consider increasing your score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zbeSuqvW5p", "reviewer": "Reviewer_kQBQ", "summary": "This paper proposes TransVIP, a speech to speech translation model with voice and isochrony preservation, i.e. pauses and segment durations are preserved between the source and the target, for example for automatic dubbing applications.\nThe proposed model architecture is modular, with multiple encoders for semantic, acoustics and isochrony information, intermediate text output, a non auto regressive acoustic model to generate a sequence of codes which are then decoded to produce a waveform.\nContributions include the modular architecture trained end to end, the model capabilities, in particular isochrony preservation.\nExperiments show that the proposed approach is either competitive with or outperforms a strong baseline (SeamlessExpressive) on translation quality, speaker and prosody similarity while substantially improving isochrony preservation.", "questions": "“acoustic information(A)”: the title only talks about voice preservation but the evaluation also measures prosody preservation. Could the authors clarify which components of the model are specifically designed to preserve prosody?", "limitations": "The authors acknowledge that only the French-English pair is involved but we still consider this a weakness for a translation related paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This is important research work with important applications such as automatic dubbing, especially since most of the research on speech to speech translation does not emphasize isochrony preservation.\n* The proposed architecture is novel\n* The empirical results are positive and compare to a strong baseline", "weaknesses": "* The empirical evaluation could be improved: validate the method in one more language pair and optionally compare to cascaded solutions.\n* In terms of presentation, the paper could be more self contained and include ablations in the main part of the paper vs the appendix. The paper could also benefit from proofreading (there are quite a few typos).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.\n\n**R1. About empirical evaluation (Weakness 1)**\n\nWe will validate the method with one more language pair in future work. Additionally, to compare to cascaded solutions, we will add the results of cascaded ST + TTS solutions as follows,\n\n| | ASR-BLEU | BLEU | SIM | AutoPCP | Rate | Pause |SLC_0.2 | SLC_0.4 | Nat. | \n|---|---|---|---|---|---|---|---|---|---|\n| ST + StyleTTS (Fr->En) | 33.57 | 34.58 | 0.173 | 2.74 | 0.33 | 0.51 | 0.56 | 0.85 | 3.25 |\n| TransVIP (Fr->En)          | 32.60 | 35.34 | 0.320 | 2.49 | 0.55 | 0.44 | 0.70 | 0.91 | 3.19 |\n| ST + VALLE-X (En->Fr) | 22.50 | 34.89 | 0.418 | 2.87  | 0.27 | 0.54 | 0.65 | 0.89 | 3.32 |\n| TransVIP (En->Fr)          | 27.28 | 33.02 | 0.395 | 2.67  | 0.45 | 0.65 | 0.81 | 0.99 | 3.40 |\n\nwhere \n\n1)\tST is Seamless Expressive Speech-to-Text translation model \n\n2)\tWe evaluated cascaded system by using TTS model from StyleTTS (open sourced) and VALLE-X (implemented) and reported the better one in terms of objective measurements.\n\n\n**R2. About presentation (Weakness 2)**\n\nWe will reorganize the paper, include ablations in the main section, and conduct a thorough proofread. These revisions will appear in the updated version.\n\n**R3. About prosody preservation(question1)** \n\nOur framework has not been explicitly designed to preserve prosody. Therefore, prosody is not purposefully maintained but is preserved alongside the voice feature. We have kept the metric in the paper to provide a comprehensive comparison with Seamless. Recently, we have observed an increase in the use of explicit prosody modules in zero-shot TTS. We may consider adding one in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h24uwN93Lk", "reviewer": "Reviewer_ujH4", "summary": "This paper proposes a end-to-end speech translation framework that adds several improvements to the existing textless encoder decoder speech translation architectures.", "questions": "See weaknesses.", "limitations": "Adequately discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. There are several interesting ideas in this paper, such as the isochrony embedding and layer beam search.\n2. The proposed method out-performs one of the previous SOTA on En-Fr translation tasks.", "weaknesses": "1. The writing needs to be improved. There are some typos here and there (e.g. line 280 page 7). And the writing clarity can be improved.\n2. There is a lack of ablation study to investigate how much does each design choices affect the model performance.\n3. The evaluation has only been performed on translation tasks between En and Fr. It's not clear if the model will perform well between other language pairs, especially between English and non-European languages.\n4. For isochronic translation, the model can either choose to translate as usual and then adjust the speech rate to fit into the timing boxes, or the model can be more smart in translating in the optimal way to reflect the isochrony constraint. I couldn't find any discussion regarding this aspect in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.\n\n**R1. About improving writing and clarity (Weakness 1)**\n\nWe will reorganize the paper, include ablations in the main section, revise the unclear writing and conduct a thorough proofread. These revisions will appear in the updated version.\n\n**R2. About few ablation study (Weakness 2)**\n\nWe have presented the ablation studies, such as w/ and w/o acoustic embedding, the choice of different codec, w and w/o text BPE and Layer Bean Search in NAR acoustic model. Due to the limited space for the main content, we had to present the ablation studies in the appendix. We will also add an ablation study related to isochrony preservation in the revised version. In this study, we compared our model with the following three baselines. We will report the ASR-BLEU, SLC_p (Speech Length Compliant, as defined in the paper), and Overlap ratio (i.e., speech overlap between the reference and the hypothesis) as follows.\n\n| | ASR-BLEU | Overlap | SLC_0.2 | SLC_0.4 |\n|---|---|---|---|---|\n| No IC | 30.81 | 0.689 | 0.63| 0.87 |\n| Dec IC| 30.51 | 0.748 | 0.75 | 0.90 |\n| Dec IC + FPI | 30.45 | 0.766 | 0.77| 0.91|\n| Enc IC (Proposed) | 30.62 |0.784 |0.82| 0.95|\n \nwhere\n\n1.\tNo Isochrony control (No IC).\n\n2.\tIsochrony control on the decoder (Dec IC). This involves adding the Isochrony embedding to the input of the encoder as another positional embedding. We implemented the method from ref [1] in our system.\n\n3.\tIsochrony control on the decoder with future pause information (Dec IC + FPI). This is an improvement over above 2. In addition to the distance to the global end and VAD information, two extra pieces of information are encoded: the distance to the next pause and the number of pauses in the future. We implemented the method from ref [2] in our system.\n\nRef: [1] Y. Wu, et al. “VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing,” AAAI, 2023.\n\nRef: [2] P. Pal, et al. “Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters”, Interspeech, 2023.  \n\nPlease let us know if more ablation studies should be added.\n  \n\n**R3. About evaluation on more language-pairs (Weakness 3)**\n\nWe will validate the method with an additional language pair in future work. Our model is built on a multi-lingual Seamless model, and we believe the proposed methods can be extended to other language pairs, including those between English and non-European languages. The main hurdle to this kind of extension investigation is the availability of publicly accessible data.\n\n**R4. About the discussion on isochronic translation (Weakness 4)**\n\nWe will add discussions regarding isochronic translation as follows in the revised version. \n\nThe conventional method for isochronic translation involves first translating as usual and then adjusting the speech rate to match the length of the source speech. This approach ensures that the translation quality is not compromised by isochronic control. However, for long videos with multiple utterances, an inconsistent speaking rate can significantly affect the naturalness of the translated speech.\n\nWe aim to use isochrony control to translate optimally by considering both the timing boxes and the speech rate in real application scenarios. Both should align with the source speech. In our proposed method, the generation of both text and speech is conditioned on global isochrony information. Our experimental results also show that this approach can improve ASR-BLEU score compare to Isochrony control on decoder, meaning the model is more confident and accurate in the generation and make less error like repetition and truncation.\n\n**Finally, we would like to express our gratitude once again for your time and effort in reviewing our paper. Considering the interesting ideas, SOTA performance, adequate ablation studies, and improved presentation of our paper, we would greatly appreciate it if you could consider increasing your score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SO8kYvmKtO", "reviewer": "Reviewer_JYog", "summary": "This paper presents an end-to-end framework for speech-to-speech translation that preserves speaker and voice characteristics while leveraging unsupervised training. The authors also refine the speech tokenizer by distilling semantic information and enhance the sampling mechanism to support textless NAR acoustic modeling.", "questions": "1. Following up on my second point in weaknesses, could you provide some comparisons with the UnitY system or others in terms of inference speed? Additionally, there are several direct S2ST translation works [1,2,3] that utilize non-autoregressive modeling on the CVSS dataset, and it might be worth comparing with such methods.\n\n2. I am a little confused about Figure 2's target clip encoding. What is the motivation for using that representation for the [sep] token? From your description (lines 116-120), I do not find any mention of the use of the target speech clip. Can you explain why the pooled representation of that clip is used and what happens during inference when such clips do not exist?\n\n[1] Lee et al., (2022). Direct speech-to-speech translation with discrete units\n[2] Huang et al., (2023). Transpeech: Speechto-speech translation with bilateral perturbation\n[3] Tan et al., (2024). DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation", "limitations": "No concerns on the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors have made significant improvements throughout the S2ST pipeline, resulting in a solid contribution. They have also conducted experiments with various datasets and compared their results to those of capable models such as Seamless for S2ST and Encodec for Codec, achieving reasonable improvements. Although some of the experimental results are mixed, given the limited computational resources (only 32 GPUs) and datasets (only 5k hours), the improvements are still impressive and demonstrate the effectiveness of the proposed methods.", "weaknesses": "1. The paper is well-written and informative, but it covers a wide range of topics, which can make it overwhelming to read. The introduction and subsequent sections could be restructured to better emphasize the different contributions. For example, the introduction covers four bullet points, where the first two are modeling designs for S2ST and the last two are Codec related. Then in the method section (sec 3), everything is presented together. It would be helpful to dissect section 3 into two main sections and add pointers from the introduction sections to improve clarity. Additionally, consider adding visualizations for the distillation strategy in the main content to further illustrate the proposed methods.\n\n2. My major concern with the proposed framework is the inference speed. It would be helpful to include an analysis of the inference speed of the proposed architecture, as the use of an autoregressive decoder for predicting Codec tokens may significantly slow down the process, even with deduplicated units.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.\n\n**R1. About presentation (Weakness 1)**\n\nWe will reorganize the paper, particularly by dividing section 3 into two main subsections and adding pointers from the introduction to improve clarity. With an additional page allowed in the main content of the final version, we will also consider adding visualizations to further illustrate the proposed methods.\n\n**R2. About inference speech (Weakness 2, Question1)**\n\nYour concern about inference speed is valid. Generally, the inference speed for AR models is slower than that of non-AR models. To address this, we have conducted an analysis of inference speed. The RTF (real-time factor) for our model architecture is close to 1, while that of seamless expressive (with NAR T2U modeling) is 0.3, as measured on an Nvidia A6000 GPU with fp16. We believe that the inference speed could be improved by 2 to 4 times by leveraging Grouped Code Modeling, as proposed in VALL-E 2 (https://arxiv.org/pdf/2406.05370). Our current application scenario is video dubbing, which can be done on the cloud and offline. Therefore, inference speed has not been thoroughly investigated in this study.      \n\n**R3. Clarifying target clip encoding (Question 2)**\n\nThe pooled representation of the target clip serves as the acoustic prompt that the generation of the first layer codec can condition on. It is similar to the acoustic prompt in Vall-e, except we use only one token instead of a token sequence for acoustic prompting. The benefit of using a single token is that it creates an information bottleneck, preventing too much semantic information from passing through. This way, we hope the model can learn purely acoustic/speaker-related information. We will clarify this in the revised version.\n\nIn response to your question about what happens during inference when such clips do not exist, our ablation study results presented in Table 3 indicate that the speaker similarity suffers. Without this prompt, the first layer codec will produce a neutral voice, and the NAR model will struggle to convert it to the target voice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for your efforts in reviewing our paper. We greatly appreciate your acknowledgment of our contributions, including multiple innovations, state-of-the-art performance, and important research work. However, we received diverse ratings, ranging from scores of 4 to 7. We noticed that two reviewers who gave the lower scores both had concerns about few ablation studies.

In fact, we have presented several ablation studies, such as with and without acoustic embedding, the choice of different codecs, with and without text BPE, and Layer Beam Search in the NAR acoustic model. Due to the limited space for the main content, we had to present these ablation studies in the appendix. If this was one of the reasons for the lower scores, we sincerely hope the reviewers could adjust their scores accordingly.

Additionally, we will include new results in the revised version, including:

•	A comparison with a cascaded system, i.e., ST+TTS.

•	An ablation study related to isochrony preservation.

We will also reorganize the content, include the ablation studies in the main content and conduct a thorough proofread to improve the presentation.

Please check the details in the responses to the individual reviewers.

Thanks again!

Authors

Meta-review: The system describes a comprehensive end-to-end speech-to-speech architecture using latest ideas around joint semantic-acoustic tokens and joint text-encoders. Notably, the work consciously integrates isochrony (beyond simple rate adjustment), generally neglected in strong systems despite its importance to automated dubbing. The result improves on the Seamless work (in En-Fr, Fr-En) which is also component-wise engineered, considers voice, and uses far more data. I agree with reviewers that there are several interesting ideas and the system is SOTA in an overall sense; gains in isochrony metrics while preserving naturalness are particularly strong.

Some reviewers noted the single language pair and not comparing to dedicated voice-cloning systems are a weakness (RTF is worse but less relevant offline), and a desire for more ablations (though there are many moving parts + most were already in appendix or rebuttal) met by the authors. I believe this stronger architecture will be informative and cautiously expect the methods to generalize. All four reviewers ultimately recommend some form of acceptance. **I recommend Acceptance with the expectation of results on an additional language pair in the camera-ready** (preferably with more linguistic divergence), and that authors consider reorganizing to move key ablations from appendix to main.
Decision: Accept (poster)