PolyVoice: Language Models for Speech to Speech Translation
Authors: Qian qian Dong
Zhiying Huang
Qiao Tian
Chen Xu
Tom Ko
yunlong zhao
Siyuan Feng
Tang Li
Kexin Wang
Xuxin Cheng
Fengpeng Yue
Ye Bai
Xi Chen
Lu Lu
Zejun MA
Yuping Wang
Mingxuan Wang
Yuxuan Wang
Venue: ICLR 2024 poster (2024)
Keywords: Speech-to-Speecn Translatiom
Audio Language Model
Voice Clone
Primary area: general machine learning (i.e., none of the above)
TL;DR: In this study, we introduce PolyVoice, a language model-based framework designed for S2ST systems.

Abstract: With the huge success of GPT models in natural language processing, there is a growing interest in applying language modeling approaches to speech tasks.
Currently, the dominant architecture in speech-to-speech translation (S2ST) remains the encoder-decoder paradigm, creating a need to investigate the impact of language modeling approaches in this area. 
In this study, we introduce PolyVoice, a language model-based framework designed for S2ST systems. Our framework comprises three decoder-only language models: a translation language model, a duration language model, and a speech synthesis language model. 
These language models employ different types of prompts to extract learned information effectively. By utilizing unsupervised semantic units, our framework can transfer semantic information across these models, making it applicable even to unwritten languages. 
We evaluate our system on Chinese $\rightarrow$ English and English $\rightarrow$ Spanish language pairs. Experimental results demonstrate that \method outperforms the state-of-the-art encoder-decoder model, producing voice-cloned speech with high translation and audio quality.
Speech samples are available at https://polyvoice.github.io.

Published as a conference paper at ICLR 2024
POLYVOICE: LANGUAGE MODELS FOR SPEECH TO SPEECH TRANSLATION
Qianqian Dong', Zhiying Huang', Qiao Tian, Chen Xu, Tom Ko Yunlong Zhao, Siyuan Feng, Tang Li, Kexin Wang, Xuxin Cheng, Fengpeng Yue, Ye Bai, Xi Chen, Lu Lu, Zejun Ma, Yuping Wang, Mingxuan Wang, Yuxuan Wang ByteDDnce
[donggianqian, huangzhiying 92, tom. tom.koo@bytedaaceeco com
ABSTRACT
With the huge success of GPT models natural language processing, there a growing interest in applying language modeling approaches to speech tasks. Cur- rently, the dominant architecture speech-to-speech translation (S2ST) remains the encoder-decoder paradigm, creating need to investigate the impact of lan- guage modeling approaches this area. In this study, we introduce Poly Voice, language model-based framework designed for S2ST systems. Our framework comprises three decoder-only language models: translation language model, a duration language model, and speech synthesis language model. These lan- guage models employ different types of prompts to extract learned information effectively. By utilizing unsupervised semantic units, our framework can trans- fer semantic information across these models, making it applicable even to un- written languages. We evaluate our system on Chinese - English and English + Spanish language pairs. Experimental results demonstrate that Poly outperforms the state-of of-the--rt encoder-decoder model, producing voice-cloned speech with high translation and audio quality. Speech samples are available at https: http://ollllcceeeooeooooooooooce www.iie...ttttt............ github. io.
INTRODUCTION
Speech-to-speeee translation (S2ST) is challenging task as it encounters all the difficulties of automatic speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) synthesis. The research in S2ST focuses
approaches: cascade solutions (Lavie et al., 1997; Baldridge, 2004; Nakamura et al., 2006) and direct solutions (Jia et al., 2019; 2022a). The advantage of cascade solutions lies in the convenience of improving the performance individual modules, while direct solutions excel in lower latency and simpler model architectures. As for the direct S2ST solutions, used to involve direct output mel- -pectrrrro features (Dong et al.. 2022) in the early stages, but recently, there has been growing interest in predicting discrete units (Lee et al., 2022a). The unit-based approach has become more popular due to several reasons: (1) It eases the modeling difficulty of emitting spectrogram. (2) Units can be generated through unsupervised methods and can cover unwritten languages. (3) It allows a connection with other token- NLP models.
Recently, language modeling (LM) approaches have made a lot of breakthroughs in natural language processing (NLP) (Zhao et al., 2023). The success of GPT models (Brown et al., 2020; Ouyang et al., 2022) is leading the community used in speech tasks, and the exploration of using LM approaches is still in its early stages. In fact, there have been attempts ASR (Fathullah al., 2023) and TTS (Wang et al., 2023), indicating that this direction is promising. Thus, we are motivated to investigate the performance of language modeling approach S2ST. In this paper, we propose a semantic unit-based framework for S2ST system. Our framework (Fig. 1) consists of three LMs: translation LM, duration LM and speech synthesis LM. The translation LM processes the semantic units of the source language and translates them into semantic units of the target language. The duration LM predicts the duration
new era. Currently, the encoder-decodes models remain widely
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2024
information of the target semantic units and extends the unit sequence. The speech synthesis LM predicts the target acoustic units which are then converted into a waveform by a unit vocoder.
We employ various prompt types extract the acquired knowledge from the language models uti- lized in our approach. Specifically, we concatenate the source and target semantic units, along with the source acoustic units, forming a comprehensive prompt for the speech synthesis language model. This enables the speech synthesis language model to grasp the acoustic characteristics of the source speaker and generate voice-cloned acoustic units accordingly. Importantly, both the semantic and acoustic units mentioned above are generated through unsupervised methods, making our frame- work applicable to unwritten languages.
We evaluate our system on Chinese + English and English + Spanish language pairs. Experimental results show that our system can generate voice-cloned speech with high translation quality and audio quality. We summarise our contribution as follows:
We propose using series of decoder-only language models to fulfill the S2ST task, whereas encoder-decoder models are the dominant structure in previous works.
Unsupervised speech units used in the framework and thus Poly Voice can cover both written and unwritten languages.
The rest of this paper is organized as follows. Section introduces related work. Details of our method are described in Section 3. Section 4 introduces experimental setup and main results. Section presents our ablation study. Finally, we conclude our work in the last section.
2
RELATED WORK
2.1 SPEECH TOKENIZATION
There are two kinds of discretized speech units used in work: semantic and acoustic units. Semantic units are usually derived from representations produced by speech encoder models like HuBERT (Hsu al., 2021), mHuBERT (Lee al., 2022c) or w2v-BERT (Chung al., 2021). They capture the phonetics and semantic content in speech. Although the making of these units is originally developed to used as target for training the speech encoder, recently there are attempts to directly use these units as input/output for semannic tasks (Kharitonov et al., 2021; Lakhotia et al., 2021; Meng et al., 2023: Zhang et al., 2023a). Acoustic units can also be referred as codec units. They are originally developed to transmit high-quality speech signals under limited bandwidth. Au- dioLM (Borsos et al., 2023) is pioneer work in using language models (LM) for audio generation. They make use of both kinds of units and build several LMs with different resolutions. VALL-E (Wang et al., 2023) further extends the AudioLM framework and applies fully demonstrate that the in-context learning capabilities of LM can be similarly replicated in the context of phoneme and codec units. In contrast to phoneme units which have to involve a supervised training process, both semantic and acoustic units be generated through unsupervised methods.
TTS. They success-
2.2 TTS
Zero-shot cross-lingual TTS (Jia et al., 2018; Cooper et al., 2020) aims to build system that can synthesize speech with user's voice and in a specific language that the user doesn't It speak. Early attempts include speaker adaptation (Chen et al., 2019) and speaker embedding (Liu & Mak, 2019) approaches. LM-based TTS has been recently proposed and demonstrated promising results. VALL- (Wang et al., 2023) introduces an L.--ased approach that leverages in-context learning for zero- shot TTS. Their approach utilizes phoneme units and source acoustic units to prompt the LM in predicting the target acoustic units. The most relevant related work to ours is VALL-E X (Zhang et al., 2023b), which extends the VALL-E framework tackle the gross-lingus problem. In their work, they concatenate the source and target phoneme units, along with the source acoustic units, to create prompt for the L.M.
2

---

Published as a conference paper at ICLR 2024
2.3 S2ST
Speech-to-specds translation (Lavie et al., 1997; Baldridge, 2004; Nakamura et al., 2006) aims to de- velop models capable of generating target language speech from source language speech. A vanilla system traditionally employs a pipeline (Nakamura et al., 2006) that sequentially processes the input through automatic speech recogmition (ASR) models, machine translation (MT) models. and text- to-speech synthesis (TTS) models. Recently, end-to-end paradigms (Jia et al., 2019; 2022a) have gained popularity in the field of S2ST, as they allow for a single model to perform one or more of the aforementioned tasks, which consequenlly reduces error propagation and latency. Among the various techniques, auxiliary supervision based on textual data has been particularly effective during training (Jia et al., 2019; Kano et al., 2021). However, this approach is not feasible when dealing with unwritten languages. To address this challenge, discrete units (Hsu et al., 2021) extracted from the speech are used to replace the target text, and then can be synthesized into the speech (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2022a). Large scale studies have shown the powerful performance in various speech processing tasks (Nguyen al., 2022).
Current research in speech-to-speech translation primarily emphasizes translation quality, with no- table improvements observed automatic evaluation metrics (like BLEU) human evaluation of naturalness. However, there remain two persistent challenges in developing practical systems. First, these systems are predominantly developed and evaluated on small- scale benchmarks. while real-world scenarios often involve large quantities of labeled data, including ASR, MT, and S2T data (Agrawal et al., 2023). Even for low-resource or unwritten languages, leveraging unlabeled speech or text can provide valuable information (Lee et al., 2022a). Therefore, developing a unified model that jointly utilizes various data types is a critical research goal yet be achieved. Second, while not strict requirement, preserving the source speaker's ' style during translation is an important aspect of improving user experience (Zhang et al., 2023b; Song et al., 2023). However, capturing the unique characteristics of individual speakers is challenging task. Current approaches, such as speaker embeddings (Jia et al., 2019) and multi-speaker TTS systems (Jia et al., 2018), have made some progress in this direction, but they are still far from practical requirements.
Taking a broad perspective, our work aligns with the framework presented Lee et al. (2022a), where the source speech undergoes translation into discrete units before synthesizing it into the target language's speech. However, what distinguishes our work is the utilization of decoder-only language models enhance the performance of each module. By leveraging diverse data sources within a language model-based framework, our proposed method effectively maintains the source speaker's style during synthesis, thereby demonstrating significant potential in practical systems.
METHOD
We present Voice an innovative language model-based framework for speech-to-spe to-specch trans- lation, catering both written and unwritten languages. Our proposed framework leverages discrete units, acquired through self-supervised training techniques such as HuBERT (Hsu et al., 2021), serv- ing as intermediary representation bridging the source and target speech modalities. Poly Voice provides a comprehensive framework consisting of two main components: a speech-to-unt trans- lation (S2UT) front-end, facilitating the conversion of source language speech into target language units, and unit-to-speech (U2S) back-end, skillfully synthesizing translated speech while preserv- ing the personalized style of the source speaker. Figure I provides an illustrative overview of our approach.
3.1 SPEECH-TO- UNIT TRANSLATION (S2UT)
By employing discrete units obtained through self-supervised training, semantically irrelevant in- formation from continuous speech representations is eliminated, facilitating effective training in an NLP paradigm. In this regard, the S2UT component leverages a language model to acquire the necessary cross-lingual generation capabilities based on the unit-based approach.
Semantic unit extractor S2UT initiates the processing of raw speech data by employing phisticated semantic unit extractor. Here we adopt HuBERT, which first encodes the speech by a stack of convolutions and Transformer layers to continuous representations at every 20-ms frame,
3

---

Published as a conference paper at ICLR 2024
S2UT
Mergal Target Units
U25
Unmerged Target Coils
U-XLM
Source Speech
Target Speech
Source Semanic Unit
Source Accuntic Unit
soundstream decodier
Target Semantic Unit
Target Acoustic Unit
Target Speech
U.SLM
U-XLM
PPRP
Language Model
Language Model
Inuslate [sx lungi mnit
Eng lingi mnie
Marged Unit
soundstream encoder
MERGE
Sceech
Source Units
Dur
(2)
Language Model
Source
Figure 1: Overview of Poly The framework can be viewed as a concatenation of speech-to- unit translation (S2UT) front-end and unit-to-speech (U2S) back-end. There are three LMs in the framework: the unit-based cross-lingual LM (U-XLM), the duration LM and the unit-to-speech LM (U-SLM).
ASR: [lang]
Date: <unit, text>>
Promptl: Translate [lang] unis {unit} [lang] text:
[text]
Prompt2: Translate [lang] tex {text) to [lang] unir: {unit]
MIT (rc lang) figt lang)
Data: <src_lext, Igt_tex
Prompt: Translate [sre langl
text) lang] text:
text?
S28T: Jang] + [igt]lang]
Data: unie,unit, tgt unit, src. text, 1gt text:>
Prompt1: Translate [snc Jang] unit {src pre_unit)
to ligt Jangl wnit:
[igt unit
Prompt2: Translate [sre lang] unit
(sre unit)
to Trre lang] text:
[src text
Prompt3:
Translate Jangl unit
[5rc [sit
to lang]
text
Prompt4:
Translate [src lang] text
[re text)
to Figt lang) wnit: [agt unit)
Prompt5:
Translate [src Jang/ text text)
langl text:
[ext)
Prompt6: Translate [tgt lang! text [tgt text)
figt langl unit: {ugt unit
Promp17: Translate [src lang] unit [sre unit)
to Isre lang] tea then figt lang] text
then ltgt lang] "(sre.text) [sep] (igr.ext)) [sep] unit)
Table 1: Data construction for the U-XLM model by various templates. These templates play crucial role in generating multiple versions of training samples from diverse data resources such as ASR, MT, and S2ST, which are instrumental in facilitating cross-lingual unit generation.
and then utilizes k-means clustering to discretize the representation set of cluster indices = 2 11...,.22 ....,,22 T. T is the number of frames and z E [K], where is the number of clus- ter centroids. The discretized units are then merged by removing consecutive duplicated units.

---

Published as a conference paper at ICLR 2024
Unit-based cross-lingual language model (U-XLM) We denote the training sample consisting of units of speech in source language and target language as <src_unit, tgt_unif>. Within the encoder- decoder architecture, the encoder takes the source units as input, while the decoder generates pre- dictions for the target units. To facilitate the generation of cross-lingual units, a straightforward ap- proach involves utilizing simple prompts construct training samples for natural language from unit pairs, For instance, one can create prompts like: "Translate [sre Jang] unit {src unif} to [tgt lang] unit: "igfunnff"". In addition the direct transformation prompt, we can also instruct the model to generate intermediate steps similar to the cascaded systems in the chain of thought prompting (Wei al., 2022) manner.
Training To achieve competitive performance in training the U-XLM model, large amount of data is crucial. However, obtaining supervised data, specifically cross-lingual unit pairs, often limited in real-world scenarios. While auxiliary models can generate pseudo labels, such as synthe- sizing the target speech using the TTS model, direct training with supervised data is preferred.
To overcome the challenge of limited data availability, prior research has incorporated additional loss functions into the encoder-decoder architecture using multitask learning (Jia et al., 2022a; Lee al.. 2022a). In our work, we leverage the power of language modeling address this issue in a more straightforward manner, allowing for the utilization of diverse data sources like automatic speech recognition (ASR) and machine translation (MT) data.
In Table 1, we demonstrate how we use different prompts to create training samples for different types of data sources. By employing parameter sharing and simplifying the design of auxiliary objectives, we train the model using these prompts. This approach also enables the direct utilization of unlaheled text and speech data. Consequently, the model implicitly enhances the alignment the representation space between speech units and text.
3.2 UNIT-TO-SPEECH SYNTHESIS (U2S)
Unittto-seeech language model (U-SLM) As illustrated in Figure 1, the U-SLM leverages the semantic units predicted by U-XLM and generates the codec units which incorporate the speak- ing style of source speaker. Similar to VALL-E X, U-SLM encompasses both an autoregressive model and a -on-autorerressvve model. However, instead of conventional phonemes, we employ discretized semantic units in our approach.
SoundStream codec We employ SoundStream (Zeghidour et al., 2021), a neural audio codec, to generate embeddings of acoustic tokens. To ensure optimal performance, we re-implement the SoundStream with hierarchy of 6 vector quantizers and a vocabulary of 1024 symbols. In our configuration, the acoustic tokens are produced at a rate of 80Hz for input waveforms sampled at 24 kHz, which results in a significant reduction in the sampling rate, specifically reduction by factor of 300 (24000/80). Once the U2S model predicts the acoustic tokens represented by the SoundStream codec, the decoder component of SoundStream reconstructs them back into the waveform.
Duration model Through empirical observations, we have determined that the duration informa- tion of discretized units is crucial for ensuring stable and natural-sounding synthesized speech. In our approach, we employ an additional LM predict the durations.
In Figure 1, we illustrate the process of incorporating duration prediction into our framework. The merged source semantic unit sequence, merged target semantic unit sequence, and the source du- ration value sequence (D) are concatenated and provided as prompt to the duration LM. Subse- quently, the duration LM predicts the target duration value sequence, and each target semantic unit is repeated accordingly based on its predicted duration.
4 EXPERIMENTS
We conduct experiments utilizing decoder-only model architecture following the standard GPT- 2 (Radford et al., 2019). In Appendix A.2, we provide a comprehensive description of the model configurations employed in work.
5

---

Published as a conference paper at ICLR 2024
4.1
DATASETS AND PREPROCESSING
4.1.1
S2UT
Semantic tokens
U-XLM trained by cross-lingual unit data, which is extracted from the audio by HuBERT (Hsu et al., 2021) models. For Chinese audio, we utilize an open-source model based on WenetSpeech Chinese speech'. For English and Spanish audio, we use an open-source multilingual model (English, Spanish and French)². The cluster centroids of k-mean algorithm for the two models are 500 and 1,000, respectively.
Vocabulary To address the -oo-bl-voca problem and enable parameter sharing across lan- guages, we utilize byte-level subword units that decompose each character into byte-sized pieces and achieve a final vocabulary size of 56,407, including 1,500 cluster centroids (<zh-0>,... <zh- 499> and <m-02>...<m-9999>
Datasets Considering that the paired speech-to-specch (S2S) data is scarce, we synthesize the pseudo data from the ASR data utilizing in-house MT and TTS systems. In addition, various types of data resources provide better learning of the U-XLM model, like large-scale ASR and MT data. A more elaborate description of the used datasets can be found in Appendix Table 7.
The S2S data is sourced from WenetSpeech (Zhang et al., 2022) and GigaSpeech (Chen et al., 2021), respectively, marked as "GigaS2S" and "WenetS2S". WenetSpeech is Chinese ASR dataset with over 10,000 hours of speech data collected from YouTube.. And we utilize a subset of 10,000 hours of GigaSpeech (Chen et al., 2021), an English ASR dataset collected from audiobooks, podcasts, and YouTube.
Then we scale up the training data using specific prompts for various types of datasets. We utilize the LibriLight (Kahn et al., 2020) and the house ASR datasets. LibriLight is an unlabeled English speech dataset containing about 60,000 hours of speech. Since LibriLight has many long audios, we segment and recognize the audio based on the method of voice active detection (VAD) and in-house ASR system, generating the audio length ranging from 0.5 to 25s, and the average length 7s. The in-house ASR dataset is a Chinese ASR dataset with 60,000 hours of speech. We also use the MT dataset consisting of 44M sentence pairs.
in-house
4.1.2 U2S
The U-SLM is trained the large open-source bilingual speech data, including WenetSpeech (Zhang et al., 2022) and LibriLight (Kahn et al., 2020). The Librilight is handled in the same way as U-XLM. WenetSpeech keeps the original data length unchanged. The duration of audio samples ranges from 0.5 to 20s, and the average duration is 2.5s. To further improve the synthesized quality, we use an additional 250-hour internal Chinese TTS data and 400-hour internal English TTS data.
4.2 EVALUATION
Our method is evaluated on two speech-to-speech benchmark datasets, EMIME (Wester & Liang, 2011) (Chinese English) and CVSS (Jia et al., 2022b) (English - Spanish). Apart from the over- all result, we report the separate performance on the S2UT front-end and U2S back-end. EMIME contains bilingual Chinese-Englins speech recorded by the same speakers. For CVSS, the transla- tion speech is in voices automatically transferred from the corresponding source speech. To measure the performance of our system, we evaluate both the translation quality and the speech quality.
Translation Quality Following the previous setups, we recognize the speech output by an in- house ASR system to compute BLEU scores (ASR-BLEU) for S2ST results using sacrebleu',
speech_
to speech//occsttexteess s2tesssss__sst_ee real rea. data.md

---

Published as a conference paper at ICLR 2024
ASV t
ASR BER-BLEUT
Naturalness
Egt vs. src
hyp vs. STC
hyp vs. igt
Cascade (VALL-F paper) w/ oracle target text
0.28
0.27
27.49
3.44
0.28
0.29
80.30
3.43
0.58
VALL-E (VALL-E X paper)
0.37
0.37
30.66
3,54
+ w/ oracle target text
0.39
0.38
86.78
3.54
S2UT
0.06
0.08
29.30
3.35
Poly Voice (S2UT U2S)
0.59
0.38
0.38
29.40
4.10
w/ chain of thought decoding
0.38
0.38
30.80
4.11
+ w/ oracle target semantic unit
0.42
0.48
76.10
3.92
Table 2: S2ST results on Chinese- Eainese-English EMIME dataset. The bold and underlined numbers
represent the best results of full-pip full-pipeline decoding. Only bold numbers signify the best synthesized results using the oracle target translation as input.
Speech Quality The speech quality is evaluated by multiple metrics. The capability of voice clone measured by the speaker similarity (ASV-Score), which is calculated by an ASV model" to determine whether the synthesized speech is from the same speaker the ground-truth speech. The naturalness of the speech output is evaluated by the automatic metric using NISQA And the pronunciation accuracy is evaluated using WER scores (ASR-WES with ASR model based hubert--aagge
4.3
RESULTS AND ANALYSIS
4.3. S2ST RESULTS
Table 2 summarizes the overall performance of our method for S2ST. We conduct experiments on the EMIME dataset enable direct comparisons with the most similar work VALL-E X. The cascade system treats S2ST as pipeline of rumning an ASR model, an MT model, and a multi- YourTTS model separately and sequentially. During the synthesis process, speaker information is integrated using speaker embeddings.
We first evaluate the capability to preserve the voice of the source speaker in the output speech, using the ASV score. We calculate speaker similarity between the source speech, target speech, and synthesized speech. We can run the U-XLM alone, where speech is synthesized by Unit-based vocoder (Lee et al., 2022c). Due to the lack of explicit modeling of speaker characteristics, it produces particularly low ASV scores. Both the VALL-E X and Poly Voice systems, which adopt in-context learning, show superior performance our method demonstrates better voice cloning capabilities when ground-trute target information is available.
the speaker embedding based method. Notably,
PolyVoice Voice achieves a slightly enhanced translation quality (ASR-BLEU) but a remarkable improve- ment in speech quality (naturalness) compared with VALL-E X. When taking the ground-truth target information as input, Poly Voice is inferior to VALL-E X with a large gap of about 10 BLEU points, while the naturalness improves significantly. The semantic units extracted from the speech by unsupervised learning, which inevitably introduces errors. Although umits are considered "semantic" tokens, they still preserve some acoustic information. Therefore. unit-based modeling leads to better speech quality but worse translation quality. In contrast, phonemes obtained from the text ensure semantic correctness but lose the acoustic information. And future work can focus on enhancing the extraction of semantic information to improve translation quality.
// speech

---

Published as a conference paper at ICLR 2024
CVSS
ASV t
BLEU
Naturalness*
Ground-truth
0.19
89.3
3.54
PolyVoice w/ oracle target unit
0.34
18.3
3.60
0.28
70.8
3.69
Table 3: Results on the English-Spanish CVSS dataset. We train the model with paired speech- to-speech datasets expanded from GigaSpeech without any text information. BLEU means ASR- BLEU, target unit means oracle Spanish unit.
Arch
Training Data
ASR-BLEU
Chcoder-Decodee w/ U2S
16.8
GigaS2S WenetS2S
18.7
Decoder-only w/ U2S
20.7 (+3.9)
22.0 (+3.3)
Table 4: Performance with different architectures on EMIME dataset.
Task
S2ST (BLEU 1)
ASR (CER 4)
ST (BLEU
MT (BLEU f)
TTS (WER J)
S2S
22.2
MTL
29.4
4.46
30.8
33.81
6.99
Table Performance with multi-task learning (MTL) on the EMIME dataset. Here are the explana- tions for each task. S2ST: Chinese speech to English speech; ASR: Chinese speech to Chinese text; ST: Chinese speech to English text; MT: Chinese text to English text; TTS: English text to English speech.
Interestingly, Poly Voice achieves better naturalness using the predicted units. We speculate that this due to the language model's output having better fluency. U-XLM learns the speech distribution over the large scale of unit data, and tends to generate more natural sequences of units. However, this may interfere with the accuracy of the translation. We will explore this issue in the future.
4.3.2 UNWRITTEN LANGUAGE SCENARIO
We examine our proposed framework in the case where the source a written language and the target unwritten language. In setup, train and evaluate without the use of any Spanish text transcript. Table 3 summarizes the results. The ASR-BLEU (18.3) indicates that the Spanish speech generated by our system is semantically understandable. This demonstrates the ability of our S2ST system for the unwritten languages.
S2ST system
5 ABLATION STUDY
5.1 DECODER-ONE ONLY VS. ENCODER-DECODER
Empirical studies the field of natural language processing have revealed that the full potential of the decoder-only approach can be realized through the use of large model sizes and expansive datasets. As pioneers in exploring the application of language models to S2ST, we present a fair comparison of the two architectures, decoder-only and encoder-decoder, in Table 4. Two frame- works are trained with the same training data and similar parameters, approximately 0.6B in size. Interestingly, the decoder-only model yields a remarkable improvement of 3.9 BLEU points over the encoder-decoder counterpart (Lee et al., 2022b). When we synthesize the speech by U2S instead of vocoder, the performance gap is reduced, highlighting the robustness of our U2S back-end.
'The encoder decoder architecture is experimented using the implementation: https: con/ facebook researrhheeercccceooo fai rseq/'b 'b1 oboma mai n/ /eeaamppl les httee//eeeeeepeeeeeeeeeeeeeeeeee tttpstscccccccccc//eeeesoppsssse rect_ s2st cirt-iicca dat__iiscrete_uu md.
8

---

Published as a conference paper at ICLR 2024
Methods
WER
ASV 1
Naturalness 1
VALL-E X (paper)
4.07
0.36
3.54
U2$
6.40
0.38
3.98
w/ semantic2dur
31.93
0.37
3.81
w/ mHuBERT .zh_en
4.76
0.37
3.81
Table 6: Evaluation of the speech synthesizers.
5.2 MULTI-TASK LEARNING
As discussed in Section 3, the language modeling enables direct training the diverse data sources utilizing specific prompts. In this way, we combine additional large-scale ASR and MT data explore the potential of our model. As shown in Table 5, U-XLM achieves obvious improvement in S2ST from the multi-task learning under the expanded data setting. Furthermore, the performance across different tasks (such as ASR, ST. MT, and TTS) verifies the capability of the general mod- eling in the decoder-only architecture. While the traditional paradigm requires a complex approach to integrate multi-task learning, the language modeling framework offers an alternative by simply modifying the prompt to construct the training data.
5.3 ZERO- SHOT CROSS-LINGUAL UNIT-TO- SPEECH
We select samples with a duration between 4 and 10 seconds from LibriSpeech (Panayotov et al., 2015) dev-cl can set evaluate the zero-shot cross-lingual unit-to-speech module. And ran- domly choose one audio from EMIME as the Chinese speech prompt.
Table 6 shows the resynthesis performance of different speech synthesizers. Our TTS obtains better performance in both ASV and naturalness. We attribute the increase of WER to the difference in the amount of semantic information carried by phonemes and unsupervised units. This is consistent with the observation reported in the work of mHuBERT and AudioLM. If we remove the duration model from the U2S, the WER increases dramatically. Our guess is that the unit itself does not contain as much duration information as the phonemes. Therefore the duration model is essential when using unsupervised units.
We further train our own multilingual HuBERT model (mHuBERT zh_en) with a combination of Chinese and English data. The model size is the same as the HuBERT-large model (Hsu et al., 2021). We have observed substantial reduction in the WER metric when utilizing the semantic units generated from mHuBERT zh_en. Thus, we believe that a multilingual universal representation model trained with more parameters and data can generate better semantic units. We do not mHuBERT zh en in our S2ST experiment because we need the mHuBERT (Lee et al., 2022c) to the Sngnish experiment. The benefit of using mHuBERT zh to the overall S2ST is left for future work.
6 CONCLUSION AND FUTURE WORK
This paper presents a new framework for speech-to-spese translation (S2ST) based on semantic units. The framework consists of three LMs: translation LM, duration LM and speech syn- thesis LM. Through comprehensive experimentation, we provide evidence that our unit-based S2ST system surpasses existing systems in terms of ASR-BLEU, ASV, and naturalness metrics. Impor- tantly, our system demonstrates its effectiveness in scenarios involving unwritten languages, where there lack of Spanish text transcripts for reference.
Given the significant impact of semantic unit quality on our system's performance, future research will focus on improving the generation of a more refined set of discrete units. We aim explore techniques and methodologies that can contribute to enhancing the quality and diversity of the gen- erated semantic units. Additionally, we plan to investigate how the system's performance can be further enhanced by scaling up parameters and expanding the available training data. By explor- ing the effects of increased model capacity and larger datasets, we anticipate uncovering potential improvements in system accuracy and overall effectiivnnss.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, et al. Findings of the iwslt 2023 evaluation campaign. In Proceedings of the 20th International Conference Spoken Language Trunslation (IWSLT 2023), pp. 1-61, 2023.
Jason Baldridge. Verbmobil: Foundations of Speech-to-Speech Translation, by wolfgang wahlster (editor). springer, 2000. ISBN 3-540-67783-6, price F44.50 (hardback). xii+679 pages. Nat. Lang, Eng., 10(2):200-204, 2004, doi: 10. 10.0117511511349943333, URL https: /ttp//di.ooo/ooo////o
10.1017/S135132211
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 11:253322533 2023. doi: 10. 10.1199/TASLP2222..288...
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuai- jiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang. Zhao You, and Zhiyong Yan. Gigaspeech: An evolving. multi-domain ASR corpus with 10, 000 hours of transcribed audio. In Hynek Hermansky, Honza Cernocky, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlicek (eds.), Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August 3 September 2021, pp. 3670-3674. ISCA, 2021. doi: 10.21 40//4///eeseeeech2221111111111 Interspeech.. 200peeechh2221111
URL https: //ttp::://di... org /37/
2021 - 166..
Yutian Chen, Yannis Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis C. Cobo, Andrew Trask, Ben Laurie, Caglar Gulcehre, Aäron van den Oord, Oriol Vinyals, and Nando de Freitas. Sample efficient adaptive text-to-speech. In ICLR, 2019.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-trale training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-250, IEEE, 2021.
Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang, Xin Wang, Nanxin Chen, and Junichi Yamagishi. Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embed- dings. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3184-6188. IEEE, 2020.
Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Qibing Bai, and Yu Zhang. Leveraging pseudo-labeled data to improve direct speech-to-speech translation. In Interspeech 2022, 2022.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Prompting large language models with speech recognition abilities. arXiv preprint 1rXiv:2307.11795, 2023.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: self-super speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in neural information processing systems, 31, 2018.
10

---

Published as a conference paper at ICLR 2024
Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct speech-to-speech translation with a sequence-to-sequence model. In Ger- not Kubin and Zdravko Kacic (eds.), Interspeech 2019, 20th Annual Conference of the Interna- tional Speech Communicatioon Association, Graz, Austria, 15-19 September 2019, pp. 1 1123-1127. URL httph:ttt///ot org 10. .20...1.0 21437/
ISCA, 2019. doi: 10.21 10.21457/mterrppecchh2200 Interspeech. 2019-1951.. 1951.
Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. Translatotron 2: High- quality direct speech-to-speech translation with voice preservation. In International Conference on Machine Learning, pp. 10120-10134. PMIR, 2022a.
Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. Cyss corpus and massively multilingual speech-to-speeee translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. pp. 6691-6703, 2022b.
Jacob Kahn, Morgane Rivière Weiyi Zheng. Evgeny Kharitonov, Qiantong Xu,
Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. Libri-light: A benchmark for ASR with limited
supervision. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pp. 7669-7673. IEEE, 2020. doi: 10.1 11.100///ASSSS44777722222222 200.:7766.222.05599900 URL wwwpthtpp///// /top::://dd... http:/ooooooooooooooo///oooooo 10. 1101119999 09 T CASSP 4 40776 2020. 9052942.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. Transformer-based direct speech-to-speech translation with transcoder. In IEEE Spoken Language Technology Workshop, SLT 2021, Shen- doi: 00.1099/SL44890020221. 9383496. URL https:tt///... //o.. wrgg /10 10/1/099500 /TT00////SS00 4T1.484000000. 2021 9383496.
zhen, China, January 19-22, 2021, pp. 958-965. IEEE, 2021.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody- aware generative spoken language modeling. arXiv preprint
10t:2100903264 2021.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexci Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 11133-13344 2021.
Alon Lavie, Alex Waibel, Lori S. Levin, Michael Finke, Donna Gates, Marsal Gavalda, Torsten Zeppenfeld, and Puming Zhan. Janus-ii:: speech-to-speech translation in multiple languages. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 197, Munich, Germany, April 21-24, 1997, pp. 99-102. IEEE Computer Society, 1997. doi: 1091 SEASSPP1197.599990
URL https: / /ooi/oooooooo/..ooo/o//. /10 1109/ P009/ICASS.... 1997. 599557.
Ann Lee. Peng-Jen Chen, Changhan Wang. Jiatao Gu. Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. Direct speech-to-speech transla- tion with discrete units. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3327-3339, As- sociation for Computational Linguisties, 2022a. doi: 18. 1/12655////02222aa-ooonggg. attps://eat /ttp:////oii///./////////.... org/lo..oooo.oooooooooooo
URL
acl-ionc, 2335
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, al. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association.fr Computational Linguistics (Volume 1: Long Papers), pp. 3327-3339, 2022b.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, et al. Textless speech-to-speech transla- tion on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 860-872, 2022c
Zhaoyu Liu and Brian Mak. lingullingual multi-speake text-to-speech synthesis for voice cloning without using parallel corpus for unseen speakers. arXiv preprint 19X 2019.
11

---

Published as a conference paper at ICLR 2024
Chutong Meng. Junyi Ao, Tom Ko, Mingxuan Wang, and Haizhou Li. Cobert: Self-supervised speech representation learning through code representation learning. In Interspeech 2023, 2023.
Satoshi Nakamura, Konstantin Markov. Hiromi Nakaiwa, Gen-ichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, Jinsong Zhang, Hirofumi Yamamoto, Eiichiro Sumita, and Seiichi Ya- mamoto. The ATR multilingual speech-to-speech translation system. IEEE Trans. Speech Audio Process., 14(2):365-376, 2006. doi: 10.1 N0.1109/TSA.2005.66000 860774. URL https: //oii .org/
10.1109/TSA. 20..200057..77
Tu Anh Nguyen, Benoît Sagot, and Emmanuel Dupoux. Are discrete units necessary for spoken language modeling? IEEE J. Sel. Top. Signal Process., 16(6):1415- 1423, 2022. doi: 10./ 2022.320099
JSTSP.2002.32200000 URL https:/ aooiioorg/000 1119//0..111/SSSS111/1
Long Ouyang, Jeffiey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, Librispeech: an ast corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210, IEEE, 2015.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Kun Song, Yi Ren, Yi Lei, Chunfeng Wang, Kun Wei, Lei Xie, Xiang Yin, and Zejun Ma. Styles2st: shoo style transfer for direct speech-to-speech translation. arXiv preprint arXiv:2305. 17732, 2023.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Speech-to-speech translation between un- transcribed unknown languages. In IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019, Singapore, December 14-18, 2019, pp. 593-600. IEEE, 2019. doi: 10.1109/ASRU44001 2019.9003853. URL https: //t::://oooo orgg 10 2019 9003853.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, al. Neural codec language models are zero-shot text speech synthesizers. arXiv preprint arXiv:2301 02111111111111111111111111111111 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022,
Mirjam Wester and Hui Liang. The emime mandarin bilingual database. Technical report, The University of Edinburgh, 2011.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-500 100995557,220 2021.
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang. Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. WENETSPEECH: 10000+ hours multi- domain mandarin corpus for speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing. ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 6182- 6186. IEEE, 2022. doi: 10.1 10. 10.1109/ICASSP2 3922. 2022. 9746
URL https htt.ooooooo org/
Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie-Yan Liu. Uwspeech: Speech to speech translation for unwritten languages. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposhum Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 111319-14327.... AAAI Press, 2021. URL https: //ojs. aaai. orgg index. wwtp AAT//AAAT/AA www.T//rtiia 17684
12

---

Published as a conference paper at ICLR 2024
Dong Zhang, Rong Ye, Tom Ko, Wang Mingxuan, and Zhou Yaqian. Dub: Discrete unit back- translation for speech translation. In Findings ACL 2023, 2023a.
Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang. Jinyu Li, al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprini arXiv:2303.03922 2023b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, survey of large language models. arXiv
Beichen Zhang, Junjie Zhang, Zican Dong, preprint arXiv:2303. 2ati:22233..22,, 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
DATASETS
We use set of several datasets train U-XLM model: GigaS2S, WenetS2S, LibriLight and some in-house ASR, MT datasets. Table shows the detailed descriptions and statistics.
Type
Dataset
Language
Size
Domain
LibriLight
En
60K hours
audiobook
ASR
In-house
Zh
60K hours
MT
In-house
Zh ++ En
44M sents
GigaS2S
En Zh
10K hours
audiobook, podcasts, youtube youtube
S2S
WenetS2S
Zh + En
10K hours
Table 7: Training data of U-XLM model. "_" means the dataset doesn't belong some specific domains.
A.2 MODEL SETTINGS
A.2.1 S2UT
In the S2UT front-end, U-XLM's model architecture is unidirectional Transformer decoder con- sisting of 48 layers with hidden size 1600, feed-forward network (FFN) size 6400, and 25 attention heads. The total parameters are 1.6B. U-XLM is trained on 8/32 NVIDIA TESLA A100 80GB GPUs with a batch size of 3072 tokens per GPU for 500k steps.
A.2.2 U2S
In the U2S back-end, the U-SLM consists of 12 transformer layers. Each of these layers comprises 16 attention heads, an attention dimension of 1024, and an FFN dimension of 4096 in both the autoregressive (AR) model and non-autoregressive (NAR) model. We train the models using NVIDIA TESLA A100 80GB GPUs, with batch size of 8 utterances per GPU for 800k steps. Training for all steps takes about 5 days.
14


Reviews:
[{"review_id": "p3J9mnaPuF", "reviewer": "Reviewer_Jg4v", "summary": "Polyvoice is a framework for building speech to speech translation system with language modeling (or decoder only) approach as an alternative to the sequence-to-sequence (or encoder-decoder) architecture. The authors show that this is indeed possible given a combination of such LMs, i.e., translation LM, duration LM and speech synthesis LM. Each of the 3 models use unsupervised semantic and acoustic units.\n- Translation LM -  Uses source semantic units derived from HuBERT to predict target semantic units\n- Duration LM - User source and target semantic units with source duration to predict target duration units\n- Speech Synthesis LM - Uses source and target semantic units with source acoustic units to predict target acoustic units\n\nAuthors show competitive performance on EMIME (Chinese $\\rightarrow$ English) and CVSS (English $\\rightarrow$ Spanish) compared to methods proposed in VALL-E X. They also compare their work to current SoTA seq2seq approach (Lee et al.) and show zero-shot performance on dev-clean set of Librispeech.\n\nOverall, the paper's main contribution is demonstrating that a decoder-only architecture is sufficient towards building a speech-to-speech translation system with unsupervised semantic and acoustic units.", "questions": "- HuBERT is trained on English only corpus. Did you just apply the model to discretize Chinese speech, or had to adapt it?\n- Why was the specific language pairs that have been evaluated chosen?\n- You have not cited On Generative Spoken Language Modeling from Raw Audio (Lakhotia et. al.) and Text-Free Prosody-Aware Generative Spoken Language Modeling (Kharitonov et. al.) which showed that unsupervised discrete units can be used for speech synthesis and that duration is indeed crucial in improving the prosodic characteristics of the produced speech.\n- Spelling error: \"marked\" instead of \"maked\" in section 4.1.1", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework is the novel in its approach towards speech-to-speech translation where it uses decoder-only models.\n- Decoder only framework simplifies the model architecture and hence makes the implementation of the translation system straightforward.\n- The proposed method is based on unsupervised semantic and acoustic units making it possible to build systems of unwritten languages.\n- Performance on the datasets shown is quite competitive and the ablation studies further highlight the importance of the 3 component models of the framework.", "weaknesses": "- The duration and speech synthesis models depend on the translation model. Hence the training of two models depend on one upstream model which can make experimentation slow. At least, the duration model can be attempted to be folded in the translation model as shown in the paper Text-Free Prosody-Aware Generative Spoken Language Modeling (Kharitonov et. al.).\n- Since the authors use CVSS it would be desirable to show the performance on other language pairs from the dataset to make the evaluation more robust.\n- The paper stated that they have used \"in-house ASR datasets\". It's not clear how much contribution from these in-house datasets make the method effective. This is makes the reproducibility of the paper very difficult if these \"in-house datasets\" are not released.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1FPezhFW7s", "reviewer": "Reviewer_JZne", "summary": "This paper proposes a decoder-only based system for S2ST. The system includes three decoder-only LMs, including a translation LM to translate source language semantic units to target language semantic units, a duration LM to predict target language semantic unit durations and extend the sequence, and a speech synthesis LM to predict the target language acoustic units which are converted to waveforms by a unit vocoder. Both semantic units and acoustic units are self-supervised learned, hence this framework can be applied to unwritten languages.", "questions": "Please check comments and concerns listed under Weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1)\tThe use of decoder-only LMs via different prompting strategies and discrete semantic and acoustic units for all components (translation LM, duration LM, and speech synthesis LM) could benefit S2ST from competitive pre-trained text decoder-only LLMs.\n\n(2)\tEmpirical evaluations show that PolyVoice is comparable to VALL-E X, very slightly better on ASV, worse on ASR-BLEU, and better on naturalness.  Ablation studies show the contribution of the designed duration LM which uses a LM to predict durations of semantic units and extend the sequence. The duration LM significantly helps WER, with a very slight help on ASV and slight help on naturalness.", "weaknesses": "(1)\tThe innovations of this work need to be more clearly explained. This work bears strong similarity to VALL-E X. It is important to clarify the difference between the proposed approach and VALL-E X, but the paper did not clearly point out the difference between PolyVoice and VALL-E X to highlight the innovations of the proposed PolyVoice.  Both works concatenate source and target semantic units and the source acoustic units to create the prompt for the LM. For PolyVoice, this prompt is created for the duration LM and the speech synthesis LM, respectively.  The soundstream codec is reimplemented, but the impact of the reimplementation is not clear. \n\n(2)\tSome of the key technical presentations lack clarity. \n\na.\tSection 3.1, when describing Unit-based cross-lingual language model (U-XLM) , the paper shows the prompt for encoder-decoder architecture.  The paper should also clarify the prompt for decoder-only LM. \n\nb.\tSection 3.1, when describing training, Table 1 shows how ASR, MT, and TTS supervised data are used for training. The paper also mentioned that “This approach also enables the direct utilization of unlabeled text and speech data. “, yet how to use unlabeled text and speech data for training is not explained here. Instead, based on Section 4.1.1, it seems that one approach is to apply in-house MT and TTS systems on ASR data to create pseudo S2S data.  This part needs to be clarified.\n\n(3)\tMore complete and also deeper discussions are desired for empirical validations:\n\na.\tTable 2, the paper compares to cascaded (ASR-MT-TTS) and VALL-E X. It is not clear whether all well-established competitive baselines are included in this comparison. \n\nb.\tTable 2, the ASV metric evaluates the capability of preserving the voice of the source speaker. However, without ground truth target information, PolyVoice achieves 0.38 and 0.38 for hyp vs.src and hyp vs. tgt, while VALL-E X achieves 0.37 and 0.37 respectively. These are very small gains, yet the paper did not discuss this point.\n\nc.\tSection 3.1, the paper mentioned that CoT could be applied for source-to-target unit translation, yet prior works (Peng et al., 2023, Towards making the most of chatGPT for machine translation) find that when CoT is applied, the model tends to conduct word-by-word translation, hence degrading the translation performance. Table 2 shows the impact of applying CoT that it improves ASR-BLEU. But this result is not analyzed nor discussed.\n\nd.\tSection 4.3.2, the evaluation of PolyVoice for unwritten language is quite brief. It is only evaluated for the target language treated as an unwritten language.  It would be useful to also extend the evaluation to source or both languages as unwritten language. \n\ne.\tSection 5.1, again, the discussions are very brief.  More analyses and insights are expected to explain the better ASR-BLEU from decoder-only over encoder-decoder. \n\nf.\tSection 5.2, for the other tasks (ASR, ST, MT, TTS) in Table 5, since no baseline results are provided, it is not clear how these performances compare to baseline results on this dataset from prior works.\n\ng.\tSection 5.3, when using a mHuBERT model trained with more parameters and more data, the WER decreased which is explained, but ASV and naturalness got degraded.  Insights are expected to explain these results.\n\nh. The proposed system uses many in-house data and in-house systems. The amount of data and model sizes need to be clearly compared as well when comparing to baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W6aZ9fi2Pv", "reviewer": "Reviewer_PpHm", "summary": "The proposed speech-to-speech (s2s) translation system consists of three models: translation model, duration model, and unit-to-speech model. The novelty of this work is that all of these models are decoder-only (while some prior work preferred encoder-decoder architectures) and the combination of these three models to do unit-based s2s.", "questions": "See weaknesses:\n- Could you give more details on how the prompts are used\n- Could you give more details on the synthetic training data\n- Have you considered some of the ablation studies mentioned above?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The system design (using three decoder-only models) seems sound and worth investigating, although I'm not 100% on board with motivating it with the raise of GPT - there is more to the success of LLMs than being decoder-only models. Anyways, the main results (Table 2) look solid. The system description seems clear superficially, but there are some core open questions (see weaknesses).", "weaknesses": "I couldn't get a good sense of the training data - specifically how the different prompts from Table 1 are used to synthesize data, and the size of the synthesized dataset: is it the 44M sentences from Table 7 in the appendix, or more because multiple prompts are used? How does the training data compare to the baselines?\n\nMy main concern would be that the ablation studies are not effective for disentangling the many design choices and the many moving parts of the whole architecture. The encoder-decoder vs decoder-only comparison is a good start, but I still don't have a good sense about how well the synthetic data generation works, and how well each of the three models do in isolation. Possible interesting ablations would be removing the duration model, replacing u-xlm with an out-of-the-box asr/mt cascade, removing source speech dependency from u-slm, passing through n-bests between the models, etc. I'm not requesting that all possible ablations should be included, but giving a little bit more color to the paper story would make it much stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CDUV7HNDbi", "reviewer": "Reviewer_2GYW", "summary": "This paper introduces a language model approach to speech-to-speech translation. The approach uses three models: one to map source semantic units to target semantic units, one to predict the duration of target semantic units, and one to predict target acoustic units given source acoustic units and target semantic units with duration information. The semantic units are derived using HuBERT (Hsu et al., 2021) while the acoustic units are derived using Soundstream (Zeghidour et al., 2021). Unlike VALL-E X (Zhang et al., 2023b), which uses phonetic units, this paper uses semantic units, enabling the approach to be extended to unwritten languages. The key novelty of this work is its use of semantic units instead of phonetic units, and its use of a decoder-only architecture that enables prompting. On the EMIME Chinese-English and CVSS English-Spanish speech-to-speech benchmarks, the proposed approach achieves similar translation results to VALL-E X, but with a large improvement in speech naturalness. When provided with ground-truth target texts, the proposed approach performs definitely worse in translation quality than VALL-E-X, but still better in naturalness. The paper reports a result where the model is used in an unwritten language scenario for English-to-Spanish translation.", "questions": "* 4.1.1:  For Chinese->English task, what is the size of the semantic unit inventory for Chinese and English?\n* Can the semantic unit inventory be shared between the source and target sides?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Proposes a language model approach for speech-to-speech translation that uses semantic units instead of phonetic units, making it usable for unwritten languages.\n* Makes use of decoder-only architectures which enable effective prompting.\n* Reports ablation studies showing advantages of decoder-only over encoder-decoder architecture when using the same training data.\n* Demonstrates advantages of training the model on data from multiple tasks including ASR and MT.\n* Shows improvements in naturalness over VALL-E X in a zero-shot setting.\n* Performs ablation studies to show the utility of each model component.", "weaknesses": "* It would have been preferable to report performance on a low-resource target language in an unwritten scenario. Such a setup might reveal additional challenges which are not present in a high resource language such as Spanish.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)