TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation
Authors: Rongjie Huang
Jinglin Liu
Huadai Liu
Yi Ren
Lichao Zhang
Jinzheng He
Zhou Zhao
Venue: ICLR 2023 poster (2023)
Keywords: Speech-to-speech translation
Multimodal challenge
Non-autoregressive generation

Abstract: Direct speech-to-speech translation (S2ST) with discrete units leverages recent progress in speech representation learning. Specifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e.g., rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing to take full advantage of parallelism. In this work, we propose TranSpeech, a speech-to-speech translation model with bilateral perturbation. To alleviate the acoustic multimodal problem, we propose bilateral perturbation (BiP), which consists of the style normalization and information enhancement stages, to learn only the linguistic information from speech samples and generate more deterministic representations. With reduced multimodality, we step forward and become the first to establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices and produces high-accuracy results in just a few cycles. Experimental results on three language pairs demonstrate that BiP yields an improvement of 2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, our parallel decoding shows a significant reduction of inference latency, enabling speedup up to 21.4x than autoregressive technique. Audio samples are available at https://TranSpeech.github.io

Published as a conference paper at ICLR 2023
TRANSPEECH: SPEECH-TO-SPEECH TRANSLATION WITH BILATERAL PERTURBATION
Rongjie Huang': Jinglin Liu'; Huadai Liu'; Yi Ren', Lichao Zhang', Jinzheng He', Zhou Zhao'i
'Zhejiang University (rongjiehuang jinglinliu, huadailiu, zhaozhou] e juju. edu. cn
*ByteDance
ren. www.yyyiyyeddnc..com com
ABSTRACT
Direct speech-to-speech translation (S2ST) with discrete units leverages recent progress speech representation learning. Specifically, a sequence of discrete representations derived in self-supervised manner predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e.g.. rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing take full advantage of parallelism. In this work, we propose TranSpeech, speech-to-specce translation model with bilateral perturbation. To alleviate the acoustic multimodal problem, we propose bilateral perturbation (BiP), which consists of the style normalization and information enhancement stages, to learn only the linguistic information from speech samples and generate more deterministic representations. With reduced multimodality, we step forward and become the first to establish non-autoregressivv S2ST technique, which repeatedly masks and predicts unit choices and produces high-accuracy results in just a few cycles. Experimental results three language pairs demonstrate that BiP yields an improvement of 2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, parallel decoding shows significant reduction of inference latency, enabling speedup up to 21 .4x than autoregressive technique.
INTRODUCTION
Speech-to-speet translation (S2ST) aims at converting speech from one language into speech in
another, significantly breaking down communication barriers between people not sharing a common language. Among the conventional method (Lavie al.) 1997 Nakamura et al.) 20066] Wahlster 2013), the cascaded system of automatic speech recognition (ASR), machine translation (MT), or speech-to-text translation (S2T) followed by text-to-speech synthesis (TTS) have demonstrated reasonable results yet suffering from expensive computational costs. Compared to these cascaded systems, recently proposed direct S2ST literature (Jia et al. 2019 Zhang et al. 2020 Jia et al. 202T
Lee et al. 2021a demonstrate the benefits lower latencies as fewer decoding stages are needed.
Among them, Lee et al. (2021a lala
leverage recent progress self-supervised discrete units learned from unlabeled speech for building textless S2ST systems, further supporting translation between unwritten languages. As illustrated in Figure aa). the unit-based textless S2ST system consists of
"Equal Contribution
"ooresponniig Author
Audio samples available at atttppss//raaspeecc...tttt io/

---

Published as a conference paper at ICLR 2023
Target Speech
SSL Model
K-means
Discrete Representations
Acoustic Multimodality
Linguistic Multimodality
HuBERT
De] Vielen dank
Derive target units for training S2UT model
Source Speech
Generated Speech
Target Unit
S2UT model
baittbascd Vocoder
[En] Thank
[De] Danke schon
(a) Direct speech-to-speech translation (S2ST) system
(b) Multimodality challenges
Acoustic multimodality: Speech with the same content "Vielen dank" could be different due to variety of acoustic conditions; 2) Linguistic multimodality (Gu et al. 2017 Wang et al. 2019): There are multiple correct target translations ("Danke schon" and "Vielen dank") for the same source word/phrascc/enteenee ("Thank you").
Figure 1:
a speech-to- translation (S2UT) model followed by unit-based vocoder that converts discrete units to speech, leading to significant improvement over previous literature.
In modern textless speech-to-speech translation (S2ST), our goal is mainly two-fold: 1) high quality: direct S2ST is challenging, especially without using the transcription. 2) low latency: high inference speed is essential when considering real-time translation. However, the current development of the unit-based textless S2ST system is hampered by two major challenges: It is challenging to achieve high translation accuracy due to the acoustic multimodality (as illustrated in the orange dotted box in Figure [T)):: different from the language tokens (e.g., bpe) used in the text translation, the self- supervised representation derived from speech with the same content could be different due to a variety of acoustic conditions (e.g.. speaker identity, rhythm, pitch, and energy), including both linguistic content and acoustic information. As such, the indeterministic training target for speech-to-unt translation fails to yield good results; and 2) Building parallel model upon multimodal S2ST systems with reasonable accuracy is challenging as it introduces further indeterminacy. A -unaauoreerrssive (NAR) S2ST system generates all tokens parallel without any limitation of sequential dependency. making it poor approximation to the actual target distribution. With the acoustic multimodality unsettled, the parallel decoding approaches increasingly burden S2ST capturing the distribution of target translation.
In this work, we propose TranSpeech, a fast speech-to-speech translation model with bilateral perturbation. To tackle the acoustic multimodal challenge, we propose Bilateral Perturbation (BiP) technique that finetunes a supervupe speech representation learning model with CTC loss to generate deterministic representation agnostic to acoustic variation. Based preliminary speech analysis by decomposing a signal into linguistic and acoustic information, the bilateral perturbation consists of the 1) style normalization stage, which eliminates the acoustic-style information in speech and creates the style-agnostic "pseudo text" for finetuning; and 2) information enhancement stage, which applies information bottleneck to create speech samples variant in acoustic conditions (i.e., rhythm, pitch, and energy) while preserving linguistic information. The proposed bilateral perturbation guarantees the speech encoder learn only the linguistic information from acoustic- S2ST.
variant speech samples, significantly reducing the acoustic multimodality in
The proposed bilateral perturbation eases acoustic multimodality and makes it possible for NAR generation. As such, we further step forward and become the first to establish NAR S2ST technique, which repeatedly masks and predicts unit choices and produces high-accuracy results in just a few cycles. Experimental results on three language pairs demonstrate that BiP yields improvement of 2.9 BLEU on average compared with baseline textless S2ST models. The parallel decoding algorithm requires as few as 2 iterations to generate samples that outperformed competing systems, enabling a speedup by up to 21.4x compared to the autoregressive baseline. TranSpeech further enjoys a speed-peen reterd--effrmancee trade-off with advanced decoding choices, including multiple iterations, length beam, and noisy parallel decoding, trading by up to 3 BLEU points in translation results. The main contributions of this work include:
Through preliminary speech analysis, propose bilateral perturbation which assists in generating deterministic representations agnostic to acoustic variation. This novel technique alleviates the acoustic multimodal challenge and leads significant improvement in S2ST.
2

---

Published as a conference paper at ICLR 2023
We step forward and become the first to establish non-autoregressive S2ST technique with mask-predict algorithm to speed up the inference procedure. To further reduce the linguistic multimodality in NAR translation, apply the knowledge distillation technique and construct less noisy and more deterministic corpus.
Experimental results on three language pairs demonstrate that BiP yields the promotion of 2.9 BLEU on average compared with baseline textless S2ST models. In terms inference speed, our parallel decoding enables speedup up to 21.4x compared to the autoregressive baseline.
BACKGROUND: DIRECT SPEECH-TO-SPEECT TRANSLATION
Direct speech-to-speeds translation has made huge progress to date. Translatotron (Jia et al. 2019) is the first direct S2ST model and shows reasonable translation accuracy and speech naturalness. Translatotron 2 (Jia et al. 2021 utilizes the auxiliary target phoneme decoder to promote translation quality but still needs phoneme data during training. UWSpeech (Zhang al. 2020) builds the VQ-VAE model and discards transcript in the target language, while paired speech and phoneme corpora of written language are required.
Most recently, direct S2ST system (Lee et al. (SSL) and demonstrates the results without using text data. However, the majority of SSL models are trained by reconstructing (Chorowski et al.) 2019) or predicting unseen speech signals (Chung al. 2019). which would inevitably include factors unrelated to the linguistic content (i.e., acoustic condition). As such, the indeterministic training target for speech-to-ung translation fails to yield good results. The textless S2ST system (Lee al. 2021b) further demonstrates obtain the speaker-invariant representation by finetuning the SSL model to disentangle the speaker-dependen information. However, this system only constrains speaker identity, and the remaining aspects (i.e., content, rhythm, pitch, and energy) are still lumped together.
2021a) takes advantage of self-supervised learning
At the same time, various approaches that perturb information flow to fine-tune acoustic models have demonstrated efficiency in promoting downstream performance. A line of works (Yang et al. [02T Gao et al. 2022) utilizes the pre-trained encoder and introduces approaches that reprogram acoustic models in downstream tasks. For multi-lingua, tuning, Yen et al. (22T)) propose novel adversarial reprogramming approach for low-resource spoken command recognition (SCR). Sharing a common insight, tune a pre-trained acoustic model with bilateral perturbation technique and generates more deterministic units agnostic to acoustic conditions, including rhythm, pitch, and energy. Following the common textless setup in Figure[aa) a). design challenging NAR S2ST technique especially for applications requiring low latency. More details have been attached in Appendix A
SPEECH ANALYSIS AND BILATERAL PERTURBATION
3.1 ACOUSTIC MULTIMODALITY
As reported in previous textless S2ST system (Lee al. 2021b), speech representations predicted by the self-supervised pre-traine trainrd model include both linguistic and acoustic information. As such, derived representations of speech samples with the same content can be different due to the acoustic variation, and the indeterministic training target for speech-to-une translation (as illustrated in Figure a))) fails yield good results. To address this multimodal issue, conduct preliminary speech analysis and introduce the bilateral perturbation technique. More details on how indeterminacy units influence S2ST have been attached Appendix C
3.2
SPEECH ANALYSIS
In this part, we decompose speech variations (Cui et al. 2022 Huang et al. [022 Yang et al. (002a)) into linguistic content and acoustic condition (e.g., speaker identity, rhythm, pitch, and energy) and provide a brief primer on each of these components.
Linguistic Content represents the meaning of speech signals. To translate speech sample to another language, learning the linguistic information from the speech signal is crucial; Speaker Identity is perceived as the voice characteristics of speaker. Rhythm characterizes how fast the speaker utters
3

---

Published as a conference paper at ICLR 2023
Generated Unit
Target Unit
Information
Enhancement
Linguistic Content
Style
Normaizzaiion
I ength Predictor
Output Projection
RR
Rhythm
Pitch
Pitch Mean
Nax
Encoder Blocks
Docoder Blocks
x
Energy Norm
Energy
Energy Norm
Linear Dropout
Linear & Dropout
2
Subsampling
Unit Embedding
Fine TininTnnig HuBERT
CTC
Pretrained HuBERT
Random Mask
Psendo Text K-means
Source Speech
Target Unit
(a) Speech Analysis and Bilateral Perturbation
(b) TranSpeech
Figure 2: In subfigure(a), we use RR and F to respectively denote the random resampling and a chain function for random pitch shifting. In subfigure(b), the "sinusoidal-like symbol" denotes the positional encoding, we have N₂ encoder and decoder blocks. During training, we randomly select the masked position and compute the cross-entropy loss (denoted as "CE").
each syllable, and the duration plays a vital role acoustic variation: Pitch is an essential component of intonation, which is the result of a constant attempt hit the pitch targets of each syllable; Energy affects the volume of speech, where stress and tone represent different energy values.
3.3 BILATERAL PERTURBATION
To alleviate the multimodal problem and increase the translation accuracy in the S2ST system, we propose bilateral perturbation that disentangles the acoustic variation and generates deterministic speech representations according to the linguistic content. Specifically, we leverage the success of connectionist temporal classification (CTC) finetuning (Baevski et al. 2019) with pre-trained speech encoder, using the perturbed input speech and normalized target. Since how to obtaín speaker- invariant representation has been well-studied (Lee et al. 2021 Hsu et al. 2020), we focus on the more challenging acoustic conditions in single-speaker scenario, including rhythm, pitch, and energy variations.
3.3.1 OVERVIEW
Denote the domain of speech samples by S C IR and the perturbed speeches in style normalization and information enhancement by 3, respectively. The source language is therefore sequence of speech samples X {=1.... nx' ), where N' is the number of frames in source speech. The SSL model is composed of a multi-layer convolutional feature encoder f which takes as input raw audio and outputs discrete latent speech representations.
the end, the audio in the target language is represented as discrete units Y .U......NN. where N is the number of units. The overview of the information flow is shown in Figure 2(a), and we consider tackling the multimodality in bilateral sides for CTC finetuning, including I) style normalization stage to climinate the acoustic information in the CTC target and create the acoustic-agnostic "pscudo text"; and 2) information enhancement stage which applies bottleneck acoustic features create speech samples variant in acoustic conditions (e.g., rhythm, pitch, and energy) while preserving linguistic content information. In the final, we train an ASR model using the perturbed speech as input and the "pseudo text" as the target. As a result, according to speeches with acoustic variation, the ASR model with CTC decoding is encouraged to learn the "average" information referring to linguistic content and generate deterministic representations, significantly reducing multimodality and promoting speech-to-un -o-titt translation. In the following subsections, we present the bilateral perturbation technique in detail:
3.3.2 STYLE NORMALIZATION
To create the acoustic-agnostie "pseudo text" for CTC finetuning, the acoustic-style information should be eliminated and disentangled: 1) We first compute the averaged pitch fundamental frequency p and energy values in original dataset S; and 2) for each sample in S, we conduct pitch shifting to
4

---

Published as a conference paper at ICLR 2023
and normalize its energy to resulting in dataset 5 with the averaged acoustic condition, where the style-specific information has been eliminated; finally, 3) the self sulf--uuereieed learning (SSL) model encodes S and creates the normalized targets for CTC finetuning.
3.3.3 INFORMATION ENHANCEMENT
According to the speech samples with different acoustic conditions, the ASR model is supposed to learn the deterministic representation referring to linguistic content. As such, we apply the following functions information bottleneck acoustic features (e.g., rhythm, pitch, and energy) create highly acoustic-varian speech samples S, while the linguistic content remains unchanged, including 1) formant shifting fs, 2) pitch randomization pr, 3) random frequency shaping using a parametric cqualizer peq, and 4) random resampling RR.
For rhythm information, random resampling RR divides the input into segments of random lengths, and we randomly stretch or squeeze each segment along the time dimension.
For pitch information, we apply the chain function fs(pr(peq(S))) to randomly shift the pitch value of original speech S.
For energy information, we perturb the audio in the waveform domain.
The perturbed waveforms á are highly variant acoustic features (i.e., rhythm, pitch, and energy) while preserving linguistic information. It guarantees the speech encoder to learn the "acoustic- averaged" information referring to linguistic content and generate deterministic representations. The hyperparameters of the perturbation functions have been included in Appendix
4 TRANSPEECH
The S2ST pipeline has been illustrated in Figure a), we 1) use the SSL. HuBERT (Hsu et al. 202I) tuned by BiP to derive discrete units of target speech; 2) build the sequence-to-sequeed model TranSpeech for speech-to-unit translation (S2UT) and 3) apply separately trained unit-based vocoder to convert the translated units into waveform.
In this section, we first overview the encoder-decoder architecture for TranSpeech. following which we introduce the knowledge distillation procedure to alleviate the linguistic multimodal challenges. Finally, we present the mask-predict algorithm in both training and decoding procedures and include more advanced decoding choices.
4.1 ARCHITECTURE
The overall architecture has been illustrated in Figure 2)), and we put more details on the encoder and decoder block in Appendix
Conformer Encoder. Different from previous textless S2ST literature (Lee et al. 2021b), we use conformer blocks (Gulati et al.) 2020) in place of transformer blocks (Vaswani al. 2017). The conformer model (Guo et al. 2021] Chen et al. 2021 has demonstrated its efficiency in combining convolution neural networks and transformers to model both local and global dependencies of audio in a parameter-efficts Furthermore, we employ the multi-head self-attention with relative sinusoidal positional encoding scheme from Transformer-XI. (Dai et af.] 2019), which promotes the robustness of the self-attention module and generalizes better to different utterance lengths.
way, achieving state-of-the-
results on various downstream tasks.
Non-autoregressive Unit Decoder: Currently, S2ST systems utilize the autoregressive S2UT models and suffer from high inference latency. Given the N' frames source speech X (x1,.... Nr autoregressive model / factors the distribution over possible outputs Y = {yI,.... yN by p(Y
1 X; I Ti=z p(y: 10:1-1, T1:N'; ( where the special tokens yo (bos)) and yN+1 (N+1((cos)) are
used to represent the beginning and end of all target units.
Unlike the relatively well-studied con-autoregressive (NAR) MT (Gu et al. 2017 Wang et al. 2019 Gu et al.) 2019) Ghazvininejad et al,, 2019) Yin et al. 2023), building NAR S2UT models that generate units in parallel could be much more challenging due to the joint linguistic and acoustic
5

---

Published as a conference paper at ICLR 2023
multimodality. Yet the proposed bilateral perturbation eases this acoustic multimodality and makes it possible for NAR modeling. As such, we further step forward and become the first to establish a NAR S2ST model 0.
assumes that the target sequence length can be modeled with a separate conditional distribution PL. and the distribution becomes p(Y 0) PL (T' v1:N;; II= (=1 (1/s (I:V1;; 0). The target units are conditionally independent of each other, and the individual probabilities p is predicted for each token in Y. Since the length of target units N should be given in advance, TranSpeech predicts
by pooling the encoder outputs into a length predictor.
4.2 LINGUISTIC MULTIMODALITY
As illustrated in Figure b)). there might be multiple valid translations for the same source utterance, and thus this linguistic multimodality degrades the ability of NAR models to properly capture the target distribution. To alleviate this linguistic multimodality in NAR translation, we apply knowledge distillation construct sampled translation corpus from an autoregressive teacher, which less noisy and more deterministic than the original one. The knowledge of the AR model is distilled to the NAR model, assisting to capture the target distribution for better accuracy.
4.3 MASK-PREDICT
The NAR unit decoder applies the mask-predice algorithm (Ghazvininejad et al. 2019) to repeatedly reconsider unit choices and produce high-accuracy translation results in just few cycles.
Training. During training, the target units are given conditioned on source speech sample X and the unmasked target units Yobs- As illustrated in Figure Dbb), given the length of the target sequence, first sample the number of masked units from a uniform distribution then randomly choose the masked position. For the learning objective, we compute the cross-entropy (CE) loss with label smoothing between the generated and target units in masked places, and the CE loss for target length prediction is further added.
Uni ,N), .N)). and
Decoding, In inference, the algorithm runs for pre-determined T times of iterative refinement, and we perform a mask operation at each iteration, followed by predict.
In the first iteration .. O, predict the length N of target sequence and mask all units Y (y1,-.., In the following iterations, we mask n units with the lowest probability scores p:
yt Ylask = arg min(p;, n)
Y
maske
where n is function of the iteration t, and we use linear decay N T-t in this work.
T
After masking, TranSpeech predicts the masked units yt ymask conditioned on the source speech and unmasked units Yaus- We select the prediction with the highest probability p for each E yt and update its probability score accordingly:
mask
y arg max ((y w X, Yobi 0)
pf max P(y
X, Y
4.4 ADVANCED DECODING CHOICES
Target Length Beam. It has been reported (Ghazvininejad et al.) 2019) that translating multiple candidate sequences different lengths can improve performance. As such, we select the top K length candidates with the highest probabilities and decode the same example with varying lengths in parallel. the following, we pick up the sequence with the highest average log probability as our result. It avoids distinctly increasing the decoding time since the computation can be batched.
Noisy Parallel Decoding. The absence of the AR decoding procedure makes it more difficult to capture the target distribution in S2ST. To obtain the accurate optimum of the target distribution and compute the best translation for each fertility sequence, we use the autoregressive teacher to identify the best overall translation.

---

Published as a conference paper at ICLR 2023
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Following the common practice in the direct S2ST pipeline, we apply the publicly-available pre- trained multilingual HuBERT (mHuBERT) model and unit-based HiFi-GAN vocoder (Polyak et al.)
2021
Kong al. 2020) and leave them unchanged.
Dataset. For a fair comparison, we use the benchmark CVSS-C dataset (Jia et al. 2022). which is derived from the Co VoST 2 (Wang et al. 2020b) speech-to-texx translation corpus by synthesizing the translation text into speech using a single innll-speaker TTS system. To evaluate the performance of the proposed model, we conduct experiments three language pairs, including Beench-English (Fr-En), English-Spanish (En-Es), and English-French (En-Fr).
Model Configurations and Training. For bilateral perturbation, we finetune the publicly-available mHuBERT model for each language separately with CTC loss until 25k updates using the Adam optimizer (31 = 0.9, B2 0.98, e 10-$). Following the practice in textless S2ST (Lee al. 2021b), we use the k-means algorithm to cluster the representation given by the well-tuned mHTBERT into vocabulary of 1000 units. TranSpeech computes 80-dimensional mel-filterbank features at every 10-ms for the source speech as input, and we set N₂, to 6 in encoding and decoding blocks. In training the TranSpeech, we remove the auxiliary tasks for simplification and follow the unwritten language scenario. TranSpeech is trained until convergence for 200k steps using 1 Tesla V100 GPU. comprehensive table of hyperparameters available in Appendix
Evaluation and Baseline models. For translation accuracy, we pre-train an ASR model to generate the corresponding text of the translated speech and then calculate the BLEU score (Papineni et al.) 2002) between the generated and the reference text. In decoding speed, latency is computed as the time to decode the single n-frame speech sample averaged over the test set using V100 GPU.
We compare TranSpeech with other systems using the publicly-availan fairseq framework (Ot jet al. 2019), including 1) Direct ASR, where we transcribe S2ST data with open-sourced ASR as reference and compute BELU; 2) Direct TTS, where synthesize speech samples with target units, and then transcribe the speech to text and compute BELU; 3) S2T+TTS cascaded system, where we train the S2T basic transformer model (Wang et al. 2020a) and then apply TTS model (Ren et al.) 2020 Kong et al.] (020) for speech generation; 4) basic transformer (Lee et al. (021a)) without using text, and 5) basic norm transformer (Lee al. 2021b) with speaker normalization.
5.2 TRANSLATION ACCURACY AND SPEECH NATURALNESS
Table D [ummmarizes the translation accuracy and inference latency among all systems, and we have the following observations: 1) Bilateral perturbation (3 vs. 4) improves S2ST performance by large margin of 2.9 BLEU points. The proposed techniques address acoustic multimodality by disentangling the acoustic information and learning linguistic representation given speech samples, which produce more deterministic targets in ppeech-to-unit translation. 2) Conformer architecture (2 vs. 3) shows a 2.2 BLEU gain of translation accuracy. It combines convolution neural networks and transformers joint architecture, exhibiting outperformed ability in learning local and global dependencies of an audio. 3) Knowledge distillation (6 vs. 7) is demonstrated to alleviate the linguistic multimodality where training on the distillation corpus provides a distinct promotion of around 1 BLEU points. For speech quality, we attach evaluation in Appendix D When considering the mreed-performance
trade-off in the NAR unit decoder, we find that more iterative cycles (7 vs. 8), or advanced decoding methods (c.g., length beam (8 vs. 9) and noisy parallel decoding (9 vs. 10) further lead to an improvement of translation accuracy, trading up to 1.5 BLEU points during decoding. In comparison with baseline systems, TranSpeech yields the highest BLEU scores than the best avviialya direct S2ST baselines (2 vs. 6) by a considerable margin; in fact, only mask-predict iterations (see Figure ((b) are necessary for achieving new SOTA on textless S2ST.
5.3 DECODING SPEED
We visualize the relationship between the translation latency and the length of input speech in Figure 3(a) As can be seen, the autoregressive baselines have latency linear in the decoding
7

---

Published as a conference paper at ICLR 2023
Table I: Translation quality (BLEU scores (1)) and inference speed (frame/second (1)) compari- son with baseline systems. We set beam size to in autoregressive decoding, and apply 5 iterative cycles in NAR naive decoding. t: In this work, CTC, auto-encoding) in training the S2ST system for simplification. Though the S2ST system can be further improved with the auxiliary task, this is beyond our focus. BiP: Bilateral Perturbation; NPD: noisy parallel decoding; b: length beam in NAR decoding.
remove the auxiliary task (e.g., source and target
ID Model
BiP
Fr-En
En-Fr
En-Es
Speed
Speedup
Autoregressive models
Basic Transformer (Lee
2021a)
X
15.44
15.28
10.07
870
1.00
Basic Norm Transformer (Lee al. 2021bbt
15.81
15.93
12.98
Basic Conformer
X
18.02
17.07
13.75
895
1.02
Basic Conformer
22.39
19.65
14.94
Non-autoregressive models with naive decoding
TranSpeech Distill
14.86
14.12
10.27
Transpeech Distill
16.23
15.9
9610
11.04x
TranSpeech
17.24
16.3
11.79
Non-autoregreesivv models with advanced decoding
TranSpeech (iter=15)
18.03
16.97
12.62
4651
5.34
TranSpeech (itcr=15 b=15)
18.
17.05
12.70
2394
2.75x
TranSpeech (iter=15 b=15 NPD)
18.39
17.50
12.77
2208
2.53x
Cascaded systems
11
S2T TTS
27.17
34.85
32.86
Direct ASR
71.61
50.92
68.75
13
Direct TTS
82.41
76.87
83.69
Varsing ler
Iore11 with
Beew15 bm15 with NPD
1600
* 11ci11
TranSpeech
TrnnSpech (iterw15)
1200
TrrrSpecc lite=15 b-15)
Reletten Decodingg Speeed-it
17.5
Trardpeech IRta=15 be15 NPD
1000
800
32.5
600
7.5
Itrr-15
200
2.5
800
10000
1200
25.0
35.5
36.0
36.5
37.0
17.5
18.0
185
Length
ParismenneittLEll
Translation latency
Performance-specd trade-off.
Figure The translation latency computed as time to decode the n-frame speech sample, averaged over the test set using NVIDIA V100. b: length beam. NPD: noisy parallel decoding.
length. At the same time, NAR TranSpeech is nearly constant for typical lengths, even with multiple cycles of mask-predict iterative refinement. We further illustrate the versatile prced-performance trade-off for NAR decoding in Figure 3(b) TranSpeech enables a speedup up to 21.4x compared to the autoregressive baseline. On the other, it could alternatively retain the highest quality with BELU 18.39 while gaining 253% speedup.
5.4
CASE STUDY
We present several translation examples sampled from the Fr-En language pair in Table and have the following findings: 1) Models trained with original units suffer severely from the issue of noisy and incomplete translation due to the indeterministic training targets, while with the bilateral perturbation brought in, this multimodal issue is largely alleviated; 2) the advanced decoding methods lead to a distinct improvement in translation accuracy. As be seen, the results produced by the TranSpeech
8

---

Published as a conference paper at ICLR 2023
Table Two examples comparing translations produced by TranSpeech and baseline models. We the bond fonts to indicate the the issue of noisy and incomplete translation.
Source:
l'origine de rue est liée à construction place rihour.
Target:
the origin of the street is linked to the construction of rihour square.
Basic Conformer:
the origin of the street is linked to the construction of the. th origin of the seti is linked to the construction of the rear.
TranSpeech:
TranSpecch+BiPP
the origin of the street linked to the construction the ark.
TranSpecch-BiP-AAyaneed
the origin of the street is linked to the construction the work.
Source:
il participe aux activités du patronage laique et des pionniers saint-ouen. he participates in the secular patronage and pioneer activities of saint ouen. he participated in the activities of the late patronage see. he takes in the patronage activities in of saint.
Target:
Basic Conformer:
TrenSh:ech:
TranSpeech+BiiP.
participated in the activities the lake patronage say pointing participated in the activities the wake patronage and saint pioneers
TTanSpochh-BiPAAAvaeedd
with advanced decoding (more iterations and NPD), while of similar quality to those produced by the autoregressive basic conformer, noticeably more literal.
5.5 ABLATION STUDY
We conduct ablation studies to demonstrate the effectiveness of several detailed designs in this work, including the bilateral per- turbation and the conformer ar-
Table 3: Ablation study results. SN: style normalization; IE:
information enhancement; PE: positional encoding.
Model
Fr-En
En-Fr
En-Es
Basic Conformer
Relative
18.02
17.07
13.75
chitecture in TranSpeech. The re-
Basic Conformer + IE
Relative
21.98
19.60
14.91
sults have been presented in Ta- ble [] and we have the follow- ing observations: 1) Style normal- ization and information enhance- ment in bilateral perturbation both demonstrate performance gain, and they work in a joint effort to learn deterministic representations, leading to improvements in translation accuracy. 2) Replacing the relative positional encoding in the self-attention layer by the vanilla one (Vaswani et al. 2017) witnesses a distinct degradation in translation accuracy, demonstrating the outperformed capability of modeling both local and global audio dependencies brought by architecture designs.
3 Basic Conformer SN
Relative
21.54
18.53
13,97
Basic Conformer
Absolute
17.23
16.19
13.06
6 CONCLUSION
In this work, we propose TranSpeech, sppecch-t-sppeech translation model with bilateral perturbation. To tackle the acoustic multimodal issue in S2ST, the bilateral perturbation, which included style nor- malization and information enhancement, had been proposed learn only the linguistic information from acoustic-vane variant speech samples. It assisted in generating deterministic representation agnostic acoustic conditions, significantly reducing the acoustic multimodality and making possible for non-autoregressivve (NAR) generation. As such, we further stepped forward and became the first to establish NAR S2ST technique. TranSpeech took full advantage of parallelism and leveraged the mask-predict algorithm to generate results in constant number of iterations. To address linguistic multimodality, we applied knowledge distillation by constructing less noisy sampled translation corpus. Experimental results demonstrated that BiP yields an improvement of 2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, TranSpeech showed a significant improve- ment in inference latency, which required as few as 2 iterations to generate outperformed samples, enabling a sampling speed of up to 21.4x faster than the autoregressive baseline. We envisage that work will serve basis for future textless S2ST studies.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62222211, National Key R&D Program of China under Grant N0.20200FC08:2500, Zhejiang Electric Power Co., Ltd. Science and Technology Project No.5211 YF222006 and Yiwise.
REFERENCES
Alexei Baevski. Michael Auli, and Abdelrahman Mohamed. Effectiveness of self self-super pre- training for speech recognition. arXiv preprint arXiv:1911.03911 2019.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for supf-supe learning of speech representations. Advances in Neural Information Processing Systems, 24412-12244600 2020.
Nanxin Chen, Shinji Watanabe, Jesús Villalba, Piotr Zelasko, and Najim Dehak. Non-autoregressive transformer for speech recognition. IEEE Signal Processing Letters, 18:121,, 2020.
Sanyuan Chen, Yu Wu, Zhuo Chen, Jian Wu, Jinyu Li, Takuya Yoshioka, Chengyi Wang. Shujie Liu, and Ming Zhou. Continuous speech separation with conformer. ICASSP 2021-202/ IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5749-5753. IEEE, 2021.
Slycong-Scok Choi, Juhcon Lec, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. Neural analysis and synthesis: Reconstructing speech from self-supervised representations. Advances in Neural Information Processing Systems, 34, 2021.
Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron Van Den Oord. Unsupervised speech representation learning using wavenet autoencoders. IEEE/ACM transactions audio, speech, and language processing, 27(12):041-2053. 2019.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. arXiv preprint 1rXiv:1904.03240, 2019.
Chenye Cui, Yi Ren, Jinglin Liu, Feiyang Chen, Rongjie Huang, Ming Lei, and Zhou Zhao. Emovie: A mandarin emotion speech dataset with a simple emotional text-to-speech model. arXiv preprint 177 2021.
Chenye Cui, Yi Ren, Jinglin Liu, Rongjee Huang, and Zhou Zhao. Varietysound: Timbre- controllable video to sound generation via unsupervised information disentanglement. arXiv preprint arXiv:2211. 10666 2022.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.0286/ 022660 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1111.888000005.. 2018.
Heting Gao, Junrui Ni, Kaizhi Qian, Yang Zhang, Shiyu Chang, and Mark Hasegawa-Johnson. Wavprompt: Towards few-shot spoken language understanding with frozen language models. arXiv preprint arXiv:2203.15866 2022.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324 2019.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. arXiv preprint 1711 1.02281 2017.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in Neural Informa- fion Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2023
Anmol Gulati, James Qin, Chung- Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang. Yonghui Wu, et al. Conformer: cnvvolutio-auugmentn transformer for speech recognition. arXiv preprint 20Xx::2005008100) 2020.
Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi In- Jiatong Shi, et al. Recent developments
aguma, Naoyuki Kamo, Chenda Li, Daniel on espnet toolkit boosted by conformer. In ICASSP 2022-1 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5874-5878. IEEE, 2021.
Wei-Ning Hsu, David Harwath, Christopher Song, and James Glass. Text-free image-to-speech synthesis using learned segmental units. arXiv preprint arXiv:20/2.15454, 2020.
Wei-Ning Hsu, Benjamin Bolte. Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. /EEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3460, 2021.
Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3945-3954, 2021.
Rongjie Huang, Chenye Cui, Feiyang Chen, Yi Ren, Jinglin Liu, Zhou Zhao, Baoxing Huai, and Zhefeng Wang. Singgan: Generative adversarial network for high-fidelity singing voice generation. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2525-25355 2022a.
Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint 220::220000993444 2022b.
Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: Towards style transfer for generalizable out-of-domain cext-to-speech synthesis. arXiv preprint 1rXiv:2005.07211, 2022c.
Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high- text-to-speech. arXiv preprint wxXiv:2007006888 2022d.
Ye Jia, Ron Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct speech-to-speech translation with sequence-to-sequeecce model. arXiv preprint arXiv:1904.06037, 2019.
Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. Translatotron Robust direct speech-to-speech translation. arXiv preprint arXiv:2107.08661, 2021.
Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. Cvss corpus and massively multilingual speech-to-speech translation. arXiv preprint 1XXv:2201.007//33 2022.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for
efficient and high fidelity speech synthesis. Advances Neural Information Processing Systems, 15:17022-17033 2020.
Max WY Lam, Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral denoising diffusion models. arXiv preprint 108. 2021.
Alon Lavie, Alex Waibel, Lori Levin, Michael Finke, Donna Gates, Marsal Gavalda, Torsten Zeppenfeld, and Puming Zhan. Janus-iii: Speech-to-speech translation in multiple languages. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp. 99-102. IEEE, 1997.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. Direct sesechhto--spedee translation with discrete units. arXiv preprint arXiv:2 05604, 2021a.
11

---

Published as a conference paper at ICLR 2023
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to- -p-eeh translation real data. arXiv preprint 1tx.::2122.08.20 2021b.
Zhijie Lin, Zhou Zhao, Haoyuan Li, Jinglin Liu, Meng Zhang, Xingshan Zeng, and Xiaofei He. Simullr: Simultaneous lip reading transducer with attention-guided adaptive memory. In Proceed- ings of the 29th ACM International Conference on Multimedia, pp. 1359-1367, 2021.
Satoshi Nakamura, Konstantin Markov, Hiromi Nakaiwa, Gen-ichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, J-S Zhang, Hirofumi Yamamoto, Eiichiro Sumita, and Seichi Yamamoto. The atr multilingual speech-to-spect translation system. IEEE Transactions on Audio, Speech, and Language Processing, 14(2):365-33) 000.33663366,000000
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Adam Polyak and Lior Wolf. Attention- wavenet autoencoder for universal voice conversion. In ICASSP (ICASSP), pp. 6800 6804. IEEE, 2019.
IEEE International Conference on Acoustics, Speech and Signal Processing
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355, 2021.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pp. 7836-7846. PMLR, 2020.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Jinjun Xiong. Chuang Gan, David Cox, and Mark Hasegawa- Johnson. Global prosody style transfer without text transcriptions. In International Conference on Machine Learning, pp. 8650-8660. PMLR, 2021.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tic-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.0455 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Wolfgang Wahlster. Verbmobil: foundations of speech-to-speech translation. Springer Science & Business Media, 2013.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations, 2020a.
Changhan Wang, Anne Wu, and Juan Pino. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint 1rXiv:2007.10310, 2020b.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Non-autoregressive machine translation with auxiliary regularization. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 5377-5384, 2019.
Yan Xia, Zhou Zhao, Shangwei Ye, Yang Zhao, Haoyuan Li, and Yi Ren. Video-guided curriculum learning for spoken video grounding. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5191-5200, 2022.
Bang Yang, Fenglin Liu, and Yuexian Zou. Non-autoregressive video captioning with iterative refinement. 2019.
12

---

Published as a conference paper at ICLR 2023
Chao-Han Huck Yang, Yun-Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In International Conference on Machine Learning, pp- 11808-11819. PMLR, 2021.
Dongchao Yang. Songxiang Liu, Jianwei Yu, Helin Wang, Chao Weng, and Yuexian Zou. Norespeech: Knowledge distillation based conditional diffusion model for noise-robust expressive tts. arXiv preprint arXiv:2211.022 1.0448,, 2022a.
Dongehao Yang. Jianwei Yu, Helin Wang. Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. arXiv preprint arXiv:2207.09983, 2022b.
Donghaa Yang, Songxiang Liu, Rongjie Huang, Guangzhi Lei, Chao Weng, Helen Meng, and Dong Yu. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. arXiv preprint arXiv:2301. 1000001113666 13662, 2023.
Zhenhui Ye, Zhou Zhao, Yi Ren, and Fei Wu. Syntaspeech: Syntax- aware generative adversarial text-to-speech. arXiv preprint arXiv:2204. 1792, 2022.
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. arXiv preprint arXiv:2301.13430, 2023.
Hao Yen, Pin-Jui Ku, Chao-Har Huck Yang, Hu Hu, Sabato Marco Siniscalchi, Pin- Chen, and Yu Tsao. A study of low-resource speech commands recognition based on adversarial reprogramming. arXiv preprint arXiv:21 2021.
Aoxiong Yin, Zhou Zhao, Jinglin Liu, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. Simulsit: End-to-end simultaneous sign language translation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 4118 41127, 2021.
Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. Mislt: Towards multilingual sign language translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5109-5119, 2022.
Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. Gloss attention for gloss-free sign language translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, Canada, June 17-23, 2023. IEEE, 2023.
Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie- Yan Liu. Uwspeech: Speech to speech translation for unwritten languages. arXiv preprint arXiv:2006. 07926,, 59:132, 2020.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao
Wu. Dense: Data-free one-shot federated learning. In Advances in Neural Information Processing Systems.
Jie Zhang, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Lei Zhang, and Chao Wu. Towards efficient data free black-box adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15115- 15125, 2022a.
Jie Zhang, Bo Li, Chen Chen, Lingjuan Lyu, Shuang Wu, Shouhong Ding, and Chao Wu. Delving into the adversarial robustness of federated learning. arXiv preprint arXiv:2302.09477 2023a.
Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised representation learning from pre-trained diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2022b.
Zijian Zhang, Zhou Zhao, Jun Yu, and Qi Tian. Shifiddpms: Exploring conditional diffusion models by shifting diffusion trajectories. arXiv preprint arXhv:2302.023773 2023b.
13

---

Published as a conference paper at ICLR 2023
Appendices
TranSpeech: Speech-to-Speech Translation With
Bilateral Perturbation
A RELATED WORK
A.1
SELF-SUPERVISER REPRESENTATION LEARNING
There has been increasing interest in self-supervised learning in the machine learning Zhang et al.
2022a
Lam et al. 2021 Zhang et al. 2023b 2022b) and multimodal processing community (Xia
2022 Zhang al. 2023a Zhang et al, Huang et al., (022b(a)) al. Wav2Vec 2.0 (Baevski
et al. 2020) trains convolutional neural network to distinguish true future samples from random
distractor samples using contrastive predictive coding (CPC) loss function. HuBERT (Hsu et al.
2021) is trained with masked prediction with masked continuous audio signals. The majority of self-supervised representation learning models are trained by reconstructing (Chorowski et al. 2019) predicting unseen speech signals (Chung et al. 2019), which would inevitably include factors unrelated to the linguistic content (i.e., acoustic condition).
A.2 PERTURBATION- BASED SPEECH REPROGRAMMING
Various approaches that perturb information flow in acoustic models have demonstrated the efficiency in promoting downstream performance: SpeechSplit (Qian et al.) 2020), AutoPST (Qian al.) 2021). and NANSY (Choi et al. 2021) perturb the speech variations during the analysis stage to encourage the synthesis stage to use the supplied more stable representations. Voice2Series (Yang et al. 2021) introduces novel end-to-end approach that reprograms pre- pre-trained acoustic models for time series classification by input transformation learning and output label mapping. Wavprompt (Gao et al. 2022) utilizes the pre-trained audio encoder as part of an ASR to convert the speech in the demonstrations into embeddings digestible to the language model. For multi-lingual tuning, Yen al. (2021) propose novel adversarial reprogramming approach for low-resource spoken command recognition (SCR), which repurposes a pre-trained SCR model to modify the acoustic signals. In this work, we propose the bilateral perturbation technique with style normalization and information enhancement to perturb the acoustic conditions in speech.
A.3 NON-AUTOREGGESSIVV SEQUENCE GENERATION
An autoregressive model (Lin et al. 2021 Yin et al. 2021 2022) takes in a source sequence and then generates target sentences by with the causal structure during the inference process. It prevents parallelism during inference, and thus the computational power GPU cannot be fully exploited. To reduce the inference latency, (Gu al. 2017) introduces non-autoregressive (NAR) transformer-based approach with explicit word fertility, and identifies the multimodality problem of linguistic information between the source and target language. (Ghazvininejad al. 2019) introduced the masked language modeling objective from BERT (Devlin et 20.1 to non autorerressivvlly predict and refine translations. Besides the study of neural machine translation, many works bring
NAR model into other sequence-tor cequence-to-sequence tasks (Cui et al. 2221 Ye et al. 2023] Huang et al. 2022c Yang et al., 2022b), such as video caption (Yngg et al. 2019), speech recognition (Chen et al.
2020) and speech synthesis (Yc et al. [2022 Huang et al. 2022d Yang al. (023). In contrast,
focus on non-autoregressive generation in direct S2ST, which relatively overlooked.
B MODEL ARCHITECTURES
In this section, we list the model hyper-parameters of TranSpeech in Table
14

---

Published as a conference paper at ICLR 2023
Hyperparameter
TranSpeech
Conv Layers
Conv Kernel
(5. 5)
Encoder Block
Conformer Encoder
Encoder Hidden
512
Encoder Attention Heads
8
Encoder Dropout
0.1
Length Predictor
Projection Dim
512
Unit Dictionary
1000
Decoder Block
Decoder Hidden
512
Unit Decoder
Decoder Attention Headers
8
Decoder Dropout
0.1
Table Hyperparameters of TranSpeech.
C IMPACT OF INDETERMINISTIC TRAINING TARGET
To visualize the acoustic multimodality and demonstrate the effectiveness of proposed bilateral perturbation, apply the information bottleneck on acoustic features (i.e., rhythm, pitch, and energy) to create perturbed speech samples S,, respeiiiyy.....ciiiielllll.. We further plot the spectrogram and pitch contours of the original and acoustic- unit rate (UER) is further adopted pertstrc- samples in Figure in Appendix [F The evaluation matrix measure the undeterminacy and multimodality according to acoustic variation, and we have the following observations: 1) In the pre-tras pre-trained SSL model, the acoustic dynamics result in UERs by up 22.7% (in rhythm), indicating the distinct alteration of derived representations. The pre-trained SSL model learns both linguistic and acoustic information given speech, and thus the units derived from speech with the same content can be indeterministic; however, 2) with the proposed bilateral perturbation (BiP), distinct drop of UER (in energy) by up to 82.8% could be witnessed, demonstrating the efficiency of BiP producing deterministic representations referring to linguistic content.
respectively.
Acoustic
Pretrained
BiP-Tuned
Reference
0.0
0.0
Rhythm S,
22.7
10.2
Pitch S
16.3
4.3
Energy S.
10.5
1.8
Table 5: We calculate UER between units derived from original and perturbed speeches respectively using the pre-trained and fine-tuned SSL model, which is calculated averaged over the dataset. It measures the ability of the SSL model to generate acoustic-agnosti representations referring to linguistic content.
D EVALUATION ON SPEECH QUALITY
Following the publicly-availan implementation fairseq (Ott et al., 2019), we include the SNR as an evaluation matrix to measure the speech quality across the test set. We approximate the noise by subtracting the output the enhancement model from the input-noisy speech and then compute the SNR between the two. Further, conduct crowd-sourced human evaluations with MOS, rated from 1 to and reported with 95% confidence intervals (CI). For easy comparison, the results are compiled and presented in the following table:
As illustrated in Table 6 TranSpeech has achieved the SNR and MOS with scores of 46.56 and 4.03 competitive with the baseline systems. Since apply the aaalicly-available pre-trained unit
15

---

Published as a conference paper at ICLR 2023
Method
SNR (t)
MOS (t)
Translation GT
4.22++0.06
DirectS2ST
46.45
4.01
TextlessS2ST
47.22
4.05+10006
TranSpeech
46.56
4.03:10.06
Table 6: Speech quality (SNR(1) and MOS(T)) comparison with baseline systems.
vocoder and leave unchanged for unit-to-speech, we expect our model to exhibit high-quality speech generation as baseline models while achieving significant improvement translation accuracy,
E INFORMATION ENHANCEMENT
We apply the following functions (Qian al. 2020 Choi al. 2021)
acoustic features (e.g..
rhythm, pitch, and energy) to create acoustic- pertttred speech samples S, while the linguistic content remains unchanged, including 1) formant shifting fs. 2) pitch randomization pr, 3) random frequency shaping using parametric equalizer peq, and 4) random resampling RR. shown in Figure { we further illustrate the mel-spectrogram of the single-perturbed utterance in bilateral perturbation.
For fs, a formant shifting ratio is sampled uniformly from (ni 1.4). After sampling the ratio, we again randomly decided whether take the reciprocal of the sampled ratio or not.
In pr. pitch shift ratio and pitch range ratio are sampled uniformly from Unif(1, and (1, respectively. Again, we randomly decide whether to take the reciprocal of the sampled ratios not. For more details for formant shifting and pitch randomization, please refer to Parselmouth
com wtl.c/cm//////ee Yanni ckJadoul Parseellouou
peg represents a serial composition of low-shelving, peaking, and high-shelving filters. We use one low-shelving HLS, one high-shelving HHS, and eight peaking filters HPeak.
R. denotes a random resampling to modify the rhythm. The input signal is divided into segments, whose length is randomly uniformly drawn from 19 frames 32 frames (Polyak Wolf (209)). Each segment is resampled using linear interpolation with resampling factor randomly drawn from 0.5 to 1.5.
F VISUALIZATION OF PEOOUSTIC-PERTURBES SPEECH SAMPLES
16

---

Published as a conference paper at ICLR 2023
Pitch Mean
Energy Norm
RR
Un-Perturbed Source Speech
Figure Spectrogram and pitch contours of the utterance with the single-perturhed acoustic condition, remaining the linguistic content ("really interesting work will finally be undertaken on that topic") unchanged. RR: random resampling. F: chain function F' ==((p(peq(z))) for random pitch shifting.
156
665 991 158 824 442 358 713 518
really
Interesting
really
Interesting
work
Interesting
really
Inferesting
work
Reference (Un-Perturbed)
Energy- Pertturbd
Pich--Peturbed
Rhythm- Pertur-e
Figure 5: Spectrogram and pitch contours of speech sample with the perturbed acoustic condition, remaining the linguistic content ("really interesting work.") ") unchanged. The altered units are printed in red upside the spectrogram.
17


Reviews:
[{"review_id": "wWBDY5ykTL", "reviewer": "Reviewer_yedS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes two approach for an end-to-end speech-to-speech (S2S) translation: bilateral perturbation to focus the learning step on the linguistics information from the speech (i.e., disregarding acoustic properties such as rhythm and pitch that potential affects the consistency of the translation), and a non-auto-regressive unit decoding mechanism (mask-predict). Experimental results show the two approach to improve translation quality and unit decoding latency.   ", "summary_of_the_review": "This work shows interesting results for a S2S translation task, for quality and decoding latency. The proposed approach addresses the modeling of more acoustic attributes that are not explored in previous work. This work allows for a better use of pre-trained self-supervised models, by generating a deterministic representation. Despite the improvements, the experiment setup is narrow and the motivation to combine quality improvement with latency is poorly stated.  ", "strength_and_weaknesses": "Strengthes\n- The proposed style normalization step (that removes acoustic style information), and the enhancement step (that creates variable speech samples without affecting the linguistic information), are the main strength of the proposed S2S model. In addition to generate deterministic representation of the speech signal, the approach can be applied in several use cases as it will allow to better exploit speech representation learned from a self-supervised pre-training step incorporating both linguistic and acoustic signals. \n- Experimental evidences show the proposed approach (I and II), improves translation quality and a significant decoding latency. These improvements can enable the proposed approach to have multiple application use cases (including simultaneous S2S translation).  \n\n\nWeaknesses\n- Although, the proposed approaches demonstrate improvements for the respective metrics (quality and latency), its is hard to connect the two approaches (or at least it is not well motivated in the paper). I understand, the perturbation approach delivers by making NAR decoding of units in S2S translation is relatively easier, given the simplification on the acoustic features. However, its still unclear which is the main theme of the paper (I will assume the NAR approach is a by-product). \n- Experiments from a single benchmark - given the focus on the acoustic attributes, dataset from variable benchmarks (potentially with high variability in the acoustics information) could provide more strength for the experimental section.        ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oFPLjicN_GS", "reviewer": "Reviewer_H8Cj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aimed at alleviating the acoustic multimodality problem for speech-to-unit approaches for direct speech-to-speech translation (S2ST) system. To address this problem, the authors developed bilateral perturbations (BiP) and leveraged the CTC fine-tuning technology. The bilateral perturbations include information enhancement on CTC input and style normalization on CTC target. Information enhancement applies information bottleneck on acoustic features to create speech sample variants in different acoustic conditions (considering rhythm, pitch, energy) while preserving linguistic content. In contrast, style normalization aims to eliminate acoustic-style information and creates acoustic-agnostic “pseudo text”.  After the bilateral perturbation, ASR system with CTC decoding would do better to generate deterministic representations, hence alleviating the acoustic multimodality problem.  The whole idea of bilateral perturbation and leveraging CTC fine-tuning is simple and intuitive. Analysis of unit error rates showed effectiveness of proposed Bilateral Perturbation in reducing acoustic multimodality. With bilateral perturbation, the authors developed NAR S2ST (TranSpeech), while applying the standard KD technology in NAR to reduce linguistic multimodality and prior work of mask-predict NAR unit decoder. Experiments showed effectiveness of BiP over AR baseline system (basic conformer) by 1.2-4.4 BLEU points and NAR baseline system (Transpeech-Distill) by 0.6-1.8 BLEU points. ", "summary_of_the_review": "This paper proposed a simple and intuitive bilateral perturbation (BiP) approach leveraging CTC fine-tuning for tackling acoustic multimodality in unit-based direct S2ST. The proposed BiP approach is shown to reduce acoustic multimodality. With this approach, the paper built non-autoregressive direct S2ST, TranSpeech, and achieved significant speedup over AR direct S2ST without a big degradation in translation accuracy. The authors also investigated other prior approaches such as conformer, KD, mask-predict unit decoder and other decoding choices for TranSpeech. The experiments are comprehensive. The proposed BiP approach achieved good BLEU gains over both AR and NAR baselines. ", "strength_and_weaknesses": "Strengths:\n\n(1) The proposed bilateral perturbation approach for CTC fine-tuning to address the  acoustic multimodality problem is simple, intuitive and novel.\n\n(2) With the reduced acoustic multimodality, the paper explored NAR for direct S2ST. As shown in the results, NAR TranSpeech with KD achieved significant speedup over AR direct S2ST without a big degradation in translation accuracy.\n\n(3) The authors investigated a bag of tricks proposed in prior works, including conformer, KD for NAR, mask-predict NAR unit decoder, and other decoding choices including target length beam and NPD. Experiments evaluated efficacy of these tricks and analyzed performance-speed tradeoff, which are useful for future NAR efforts on direct S2ST. \n\n(4) Experiments and analyses are comprehensive. Experimental results on three language pairs of CVSS-C showed good gains in BLEU from the proposed BiP approach for both AR and NAR direct S2ST.  Analysis of unit error rates showed effectiveness of proposed Bilateral Perturbation in reducing acoustic multimodality.  The authors also compared to cascaded systems, which shows that there is still a lot of room for improvement for direct S2ST systems.\n\nWeaknesses:\n\n(1) The paper is overall clearly written. However, some technical details and discussions are missing and need to be included. Please check out the questions under Clarity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dmz4sRIN1p", "reviewer": "Reviewer_2n3x", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "### Overview\n\nThe paper aims to generate much deterministic representation for end-to-end speech translation by characterizing style normalization and information enhancement. Overall, I like the paper on incorporating those ideas for S2ST tasks. \n\nIn general, the author's design of both style normalization and acoustic information enhancement methods could be considered as one attribute-based enhancement sharing with some similar findings on latent space prompting or speech input reprogramming via perturbation to drive the model under different models and conditions. Both works are related to characterizing the perturbation for fine-tuning tasks. \n\nThe authors could add some discussion on those fine-tuning pre-training AM works via perturbations to improve the overall impact of this work. \n\nI strongly recommend accepting this work. \n- Original 6; to 8 after rebuttal. \n\n\n***\n\n**References**\n\nA. Voice2series: Reprogramming acoustic models for time series classification, ICML 21\n\nB. WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models, Interspeech 2022\n\n\n\n", "summary_of_the_review": "In general, it is one good work on tackling characterized perturbation and S2ST. Some references are missing but in general, the area of fine-tuning SSL with a customized approach is still new and worthy of more studies. ", "strength_and_weaknesses": "### Pros.\n\n1. Startforward but effective approach via the perturbation with fine-tuning CTC \n\n### Cons.\n\n1. Some related perturbation-based references are missing. \n\n2. Since Rhythm, Pitch, and Energy have all been used as information enhancement. The impact of each terms are not very clear. \n\n\n### Questions. \n\n1. It is not clear to me that the \"ASR model with CTC decoding\" has been used as one major setup in this work. Is that any difficulty in using the seq2seq prediction model under the proposed scheme?\n\n2. I heard some high-pitch background noise for Fr-En (https://transpeech.github.io/) examples. Could the authors provide some SNR or speech quality assessment since additional information has been introduced in the proposed method?\n\n***\n\n### Post-Rebuttal\n\nI read the authors' updated version and reply. \n\n- In general, my Q1 (seq2seq modeling) and Q2 (SNR) have been addressed. The authors have also stated a code release in their future version. \n\n- After re-reading, I think the major contribution (normalization and enhancement via perturbations) in section 3 is still a little blur in terms of writing since the speech module and notations are defined in their section 4. The authors could consider refine the proposed method in section with mathematical notation (e.g., $X$ and $p$) to improve their future clarification for general readers. \n\n- I updated my score to accept. This is a ~8 score paper to me. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-Q4KI3m_w7", "reviewer": "Reviewer_PYA8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper address the issues in the recent proposed S2ST framework and propose to solve the problems with two different aspect.\nFirst of all, the paper proposes to alleviate the acoustic multimodal problem by using bilateral perturbation.\nObvious improvement is achieved with this method.\nNext, the paper proposes to solve the high latency problem by introducing a non-autoregressive approach.\nThis leads to a significant speed up compared with the baseline.", "summary_of_the_review": "I would recommend the paper to be accepted if my questions are settled.", "strength_and_weaknesses": "Strength:\nThis paper propose two methods to improve the translation accuracy and the inference speed of S2ST systems. Significant improvement is achieved.\nIt is the first implementation of non-autoregressive S2ST\n\nWeaknesses:\nHere are my questions to the paper: \n\nThe author should check the definition of the word \"multimodality\" and the common usage of this word in the community.\n\n\"The proposed bilateral perturbation eases acoustic multimodality and makes it possible for non-autoregressive (NAR) generation\" is stated in the introduction. Why the method of bilateral perturbation and the NAR issue is related? \n\n\"the indeterministic training target for speech-to-unit translation fails to yield good results\" is stated in section 3.1. What is the reason behind this judgement? The units, which generated from the speech with an unsupervised way, contain noise and it was known since HuBERT. The vocoder behind the S2UT should be able to cover this and generated the target waveform. In other way round, this might be a good thing as the acoustic variation is preserved.\n\nHow much BLEU is sacrificed from moving an auto-regressive system to a NAR system?\n\nThe major aim of unit-based S2ST is to deal with unwritten languages. \nWithout this motivation and without the use of discrete units, I would consider the unit-based S2ST approach as a cascade of speech2text translation and TTS. \nWhile this paper is claiming their work on the foundation of unit-based S2ST, they are not evaluating  their models with unwritten language translation task.\nI doubt that the comparison in Table 2 is not fair as the strength of the baselines should be on unwritten language tasks.\n\n\n\n \n\n\n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster