CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation
Authors: Xiaohu Zhao
Haoran Sun
Yikun Lei
shaolin Zhu
Deyi Xiong
Venue: EMNLP 2023 Findings (2023)
Keywords: speech translation
representation disentanglement

Abstract: Deep neural networks have demonstrated their capacity in extracting features from speech inputs. However, these features may include non-linguistic speech factors such as timbre and speaker identity, which are not directly related to translation.  In this paper, we propose a content-centric speech representation disentanglement learning framework for speech translation, CCSRD, which decomposes speech representations into content representations and non-linguistic representations via representation disentanglement learning. CCSRD consists of a content encoder that encodes linguistic content information from the speech input, a non-content encoder that models non-linguistic speech features, and a disentanglement module that learns disentangled representations with a cyclic reconstructor, feature reconstructor and speaker classifier trained in a multi-task learning way. Experiments on the MuST-C benchmark dataset demonstrate that CCSRD achieves an average improvement of +0.9 BLEU in two settings across five translation directions over the baseline, outperforming state-of-the-art end-to-end speech translation models and cascaded models.

CCSRD:
Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation
Xiaohu Zhao, Haoran Sun, Yikun Lei, Shaolin Zhu, Deyi Xiong' College of Intelligence and Computing, Tiajjin University, Tianjin, China (zhaoxiaohu, hrsun, yikunlei, zhushaolin, dyxiongJ@tju.edu.cm
Abstract
speech representations. Speech inputs ST en- compass not only content information that essen- tial for translation but also non-linguistic factors such as pitch, timbre, prosody, speaker identity and so on. As result, the speech encoder en- codes not only the content information but also range of non-linguistic speech elements. Such non-linguistic factors may benefit translation, e.g., prosody as mentioned by (Sperber and Paulik, 2020). However, they may also introduce spuri- ous correlations between speech inputs and target translations, undermining speech translation gener- alization.
Deep neural networks have demonstrated their capacity in extracting features from speech in- puts. However, these features may include non-linguistic speech factors such timbre and speaker identity, which are not directly re- lated to translation. this paper, we propose content-centric speech representation disentan- glement learning framework for speech transla- tion, CCSRD, which decomposes speech rep- resentations into content representations and non- non-linguistic representations via representa- tion disentanglement learning. CCSRD con- sists of a content encoder that encodes linguis- content information from the speech input, non-content encoder that models non-linguistic speech features, and a disentanglemen module that learns disentangled representations with cyclic reconstructor, feature reconstructor and speaker classifier trained a multi task learn- ing way. Experiments on the MuST-C bench- mark dataset demonstrate that CCSRD achieves an average improvement of +0.9 BLEU in two settings across five translation directions over the baseline, outperforming state-of-the-arl end- to-end speech translation models cascaded models.
To mitigate this issue, we propose to explore speech representation disentanglement learning in the context of speech translation, which aims to separate content from non-linguistic factors (e.g.. prosody, timbre). Previous studies have consis- tently demonstrated the efficacy of representation disentanglement in improving model generaliza- tion (Chen al., 2016; Sanchez et al., 2020; Chan and Ghosh, 2022; Mohamed et al., 2022). Given the highly complex nature of speech features and the modality gap between text and speech, we argue that representation disentanglement could enable ST to focus content, reducing the negative influ- from non-linguistic factors speech transla- tion modeling.
1 Introduction
End-to-end (E2E) specch-to-text translation (ST) aims to translate a source speech input into target translation. Compared with traditional cascaded ST models, E2E ST models avoid the issue of error propagation and exhibit low inference latency. Re- cent works have made significant progress in E2E ST. which enables cascaded systems on language pairs like En-De and En-Ru (Ye et al., 2022; Fang et al., 2022; Du et al., 2022a).
Specifically, we propose a Content-Centric Speech Representation Disentanglement learning framework, termed as CCSRD, for end-to-end speech translation. We re-function the original encoder as the content encoder that encodes lin- guistic content information contained in the speech input. We introduce an additional encoder, non- content encoder, to encode con-linguistic speech features. The decomposition of the speech input into content and non-linguistic factors for the two encoders completed by a disentanglement mod- ule. The disentanglement module is trained a multi-task learning way, which leverages three
even outperform traditional
Despite the remarkable progress, E2E ST is still confronted with challenges in learning desirable
*Corresponding author

---

tasks: cyclic reconstruction task reduce the mutual information between content representation and non-content representation, a feature recon- struction task to ensure the retention of speech in- formation, and a speaker classífier task to guide the training of the non-content encoder. Addition- ally, we also employ a masking strategy to further improve disentanglement.
scarcity of high-quality E2E ST data, achieving sat- isfactory performance remains challenging. Over the years, variety of approaches have been pro- posed to address these issues, such as pre-training (Wang et al., 2020b; Tang et al., 202 lb; Dong et al., 2021), multi-task learning (Vydana et al., 2021; Ye et al., 2021; Tang et al., 2022), data augmentation (Jia et al., 2019; Lam et al., 2022), contrastive learn- ing (Li al., 2021; Ye et al., 2022) and knowledge distillation (Tang et al., 2021a; Zhao et al., 2021). Most of these approaches focus on using the tran- scription data in speech data triplets to perform MT/ASR tasks, pretrain model components and mitigate the modality gap between speech and text. Significantly different from them, we attempt to improve translation quality by efficiently exploring the speech representation disentanglement to learn content-centric speech representations for ST.
is noteworthy that our method does not require transcription to achieve representation disentangle- ment. Therefore it can used in scenarios that are short of transcription data or do not have transerip- tion data at all. Our work is hence significantly different from most previous works that heavily depend on transcription for speech translation. Although disentangled speech representation learning not new in the community of speech processing (Xie et al., 2021; Wang et al., 2021; Abeßer and Müller, 2021), to the best of our knowl- edge, this is the first attempt to learn disentangled speech representations for and-to-end speech trans- lation. In nutshell, our contributions are listed as follows.
Representation Disentanglement Representation disentanglement refers to learning paradigm in which models represent input signals through mul- tiple separated dimensions or embeddings. There- fore. always advantageous in obtaining repre- sentations that carry certain attributes or extract dis- criminative features. Reconstruction based training (Gonzalez-Garde) et al., 2018; Zhang et al., 2019; Bertoin and Rachelson, 2022) is widely adopted in disentanglement learning and used to obtain disen- tangled representations. The application of repre- sentation disentanglement is extensive, including speech (Chan and Ghosh, 2022; Chan et al., 2022), computer vision (Gonzalez (Gonzalez-Garcia et al., 2018: Lec et al., 2021) and natural language precessing (Bao et al., 2019; Cheng et al., 2020). Since speech often contains multiple factors, disentangled rep- resentation learning provides way extract dif- ferent representations for different tasks like voice conversion (Du et al., 2022b), automatic speech recognition (Chan and Ghosh, 2022) and speaker recognition (Kwon et al., 2020). Our approaches are partially motivated by these efforts but are sig- nificantly different from them in two aspects. First, we make the first step to use representation disen- tanglement during the training stage of E2F. ST and propose CCSRD. Second, we focus the quality of the content representation for ST, removing the additional encoder and disentanglement module during inference, while the previous works focus on the disentanglement of different speech factors for various speech tasks.
We propose CCSRD for end-to-end ST, which learns speech representation disentanglement to separate content from non-linguistic fea- tures.
The proposed disentanglement module con- sists of cyclic reconstruction, feature recon- struction and speaker classifier. It does not require any transcription data for disentangle- ment learning.
We conduct extensive experiments on the MuST-C benchmark with five language pairs. CCSRD achieves an average improvement of +0.9 BLEU in setting without using any tran- scription data and +0.9 BLEU in ST with the multi-task (MTL) setting using transcription data.
2 Related Work
End-to-End Speech Translation To avoid er- ror propagation in cascaded ST and reduce infer- ence latency, Bérard et al. (2016) and Weiss et al. (2017) propose end-to-end ST that directly trans- lates speech in the source language into text in the target language, without relying on the intermedi- ate transcriptions. However, due to the inherent complexity and variation of speech signals and the

---

COON
NOON
Translation Decoder
o 00
ooooo
Content Feature Profistre
Nand inneet Featire Prodictre
Disentanglement
Module
Cyclic Reconstruction
Non-content
Content Encoder
Encoder
Speaker II)
CREC
SPX
O
Speaker Classfier
Masking Sperch
Facoder
CNN
Text Embedding
Reconstruction
Wav2vecI.0
.
I/M
<en> Thank you
Speaker Classifier
Feature Reconstruction
(a) Overall Model Architecture
(b) Disentanglement Module
Figure 1: Diagram the proposed content-centric speech representation disentanglement learning framework for E2E ST. Left: The overall architecture, where the text embedding layer is only used during the training under the MTL setting. The red arrow lines denote the data flow at inference. Right: Details of disentanglement module. It has three parallel tasks: cyclic reconstruction, speaker classifier and feature reconstruction.
3 CCSRD
encoder. The non-content encoder is trained to learn non-linguisti representations separated from content representations.
this section, we first introduce the overall model architecture of CCSRD and subsequently elaborate disentanglement learning, training and inference of CCSRD.
Translation Decoder employs the base configura- tions the original Transformer (Vaswani et al., 2017) decoder, which is shared by ASR, MT and ST. It yields either speech transcription or target translation based on the output of the content en- coder.
3.1 Model Architecture
Our E2E ST model adopts the encoder-decoder ST backbone. As shown in Figure 1, it consists of five essential components: speech encoder, con- encoder, non-content encoder, disentanglement module and translation decoder.
Disentanglement Module is composed of four net- works: content feature predictor network that pre- dicts content representations from non-content fea- tures, and non-content feature predictor network that predicts non-content representations from tent features, speaker classifier network that pre- diets the speaker IDs based on non-content repre- sentations and feature reconstruction network that predicts speech features from the speech encoder based on both content and non-content representa- tions.
Speech Encoder is to extract low-level features from speech signals. It contains Wav2vec2.0 (Baevski et al., 2020) and two additional convo- lutional layers, which are added to shrink the ex- tracted speech features by factor of 4.
Content Encoder employs the same configuration as the original Transformer (Vaswani et al., 2017) encoder. The input of the content encoder is the output of the speech encoder for both ASR and ST tasks while the embeddings of the transcription are for MT task. The content encoder is trained to learn decomposed content representations.
The speech encoder, content encoder and trans- lation decoder are the same as those used by Ye et al. (2021). The non-content encoder and disen- tanglement module are additionally incorporated into CCSRD, which are removed during the infer- ence stage.
Non-content Encoder employs the same config- urations as the content encoder. The input of the non-content encoder the output of the speech
The training data ST usually contains speech-

---

transcripiion-trasslttoon triples, which can be de- noted as D {(s, x, y)). With these training instances, E2E ST can be trained two ways. Sim- ilar to standard neural machine translation, it can be trained in the way of being treated as a single ST task:
feature predictor Onon content cyclically recon- struct the content and non-comtent representations. Particularly, we train the content and non-content feature predictor with the following two reconstruc- tion losses LOON and LNCON respectively:
D
LST Σ (s.y)ED
log P(yls)
(I)
LCON =
Σ
H
Gomtent (H))
(4)
i=1
D
As DI is usually not large, E2E ST is often trained in a multi-task learning way (Ye et al., 2021). Additional ASR and MT tasks are incorporated into the training of ST with speech-transcription pairs ((s, x)} and transcription-transiin traaniltionntaansllaiioon pairs {xx facilitate knowledge transfer:
LNCON = Σ i=1
H
Onn content (H)
(5)
where Hद and H. represent the representation for 7t speech input from content and non-content en- coder respectively. These two sub-networks are connected to the overall architecture through Gra- (2) dient Reversal layers (GRLs) (Ganin et al., 2016), GRLs invert the gradient sign during the backward pass, thus pushing the parameters to maximize the (3) losses. As such, the model is constrained pro- duce content and non-content representations that are hard reconstruct each other, which means minimal information shared between the two types of representations (as their distance is maxi- mized by Eq. (4) and (5)).
ASS
Σ (s
log P(x|s)
LMT =
Σ (xy)c
log P(y|x)
3.2 Speech Representation Disentanglement Learning
the training stage of CCSRD, we attempt to de- compose representations from the speech encoder into two distinct components: content represen- tations that are essential for translation, and non- content representations that encode non-linguistic speech factors. To achieve this, we propose three training strategies: (1) encouraging the content en- coder encode only linguistic content information, (2) encouraging the non-content encoder to encode linguiigii speech factors other than linguistic content, (3) minimizing the mutual information between the content and the non-content represen- tations. For the first training strategy, we mainly force the content encoder to learn content-centric representations to reduce ST loss. For the second and third strategies, we propose cyclic reconstruc- tion, feature reconstruction, and speaker classifier tasks.
Feature Reconstruction We employ the recon- struction feature predictor network to predict the original speech features output by the speech en- coder based non-content representations learned by the non-content encoder and content representa- tions learned by the content encoder. The training objective for this sub-network is defined as:
D
LECC = Σ 1=1
H drec (H) + HHI
(6)
where H; represents the representation for 7th speech input output from the speech encoder, orec is the feature reconstruction network which tries recover the speech representation from both content representation Hद and non-content representation H
Cyclic Reconstruction To effectively disentangle speech representations, we employ the cyclic recon- struction method proposed by Bertoin and Rachel- son (2022) to reduce the mutual information be- tween the content and non-content representations in unsupervised learning setting. Specifically, after the extraction of content and non-content rep- resentations from the content and non-content en- coder respectively, we stack two sub-networks: the content feature predictor Pvwntent and non-content
Speaker Classifier Minimizing the mutual infor- mation between the content and non-content repre- sentations may be not sufficient as the non-content encoder is not fully constrained and goal-oriented. To address this issue, make full use of speaker ID information in the dataset. The speaker ID can be considered as label for non-linguistic factors. We believe that the non-content encoder should fully model this information. Modeling non- linguistic factors would benefit the decoupling of

---

Algorithm 1: CCSRD in the MTL setting
objective is:
Input :A batch of training set (s, x, y)
C LST + LSRD
(8)
Output: Loss
while not converged do
where
get from by applying the masking
strategy:
CSRD = LCON + LNCON + CREC + CSPK (9)
get H for s' from the speech encoder; get H and H for s' from the non-content and content encoder; get LSRD in Eq. (9) from the disentanglement module with the input H. H and H
In the multi-task setting, the training process overall training objective is:
LMTL LsT + LASR + LMT + CSRD (10)
The training process for the multi-task setting is demonstrated in Algorithm 1. During the inference stage, only the speech encoder, content encoder and translation decoder are used, which is consis- tent with previous works. Hence, our method does not introduce any additional inference latency com- pared with previous methods.
do ST task with H and y, get LST; do ASR task with H and x, get LASR; get H for x from the content encoder; do MT task with H and y, get LMT; get LMTL in Eq. (10);
11 end
Experiments
non-linguistic information from content informa- tion. We train the speaker classifier network with the following objective:
4.1 Datasets
We conducted experiments on the widely-used MuST-C multilingual speech translation dataset (Di Gangi et al., 2019). We carried out experiments (DE), Pogllishto-Spaanne (ES), English-to tousish-to-Russiann (RU), English-to- English-to-French
CSPK =
Σ log (spk:
H:)
(7)
on English-to-Gemma
(Fr) and English-to-Italian (It). dev was used to de- velop and analyze our approaches, IN-COMMON was used for testing. See Appendix for detailed datasct statistics.
where spk, represents the speaker ID for the zth speech input.
3.3 Masking Strategy
4.2 Settings
Recent studies have demonstrated that the masking operation can significantly improve the robustness models and benefit speech representation dis- entanglement in multiple speech tasks (Chan and Ghosh, 2022; Lin et al., 2023). As shown in Figure I, during the training stage, we modify the input waveform of the speech encoder to mask consec- utive segments and obtain the masked waveform s'. Then we take s' the input to the model. Specif- ically, for each speech input, it is chosen be masked with a probability p. The selected speech input is then masked for at least n spans, and each span contains at least m consecutive frames. In our experiments, we set to 0.75, n to 2 and m to 3600.
Model Configurations We used Wav2vec2.0 the speech encoder, which is only pretrained on au- dio data from Librispeech (Panayotov et al., 2015) without performing any downstream finetuning.
The kernel, stride and hidden size of the two CNN layers stacked over Wav2vec2.0 were set to 5, 2 and 512. The used content encoder and decoder follow the base configuration of Transformer, with layers, 512 hidden sizes and 8 attention heads. The non-content encoder follows the same config- uration as the content encoder. For the network architecture of the disentanglement module in the experiments, both the content feature predictor net- work and non-content feature predictor network consist of three fully connected (FC) layers fol- lowed by ReLU activation and one FC layer fol- lowed by Tanh activation. The feature reconstruc- tion network utilizes the same architecture as the feature predictor network, except for the different
3.4 Training and Inference
CCSRD can trained either in the ST task setting or in the multi-tase task setting with both MT and ASR tasks. the ST task setting, the overall training

---

Models
En-De
En-Es
En-Ru
En-Fr
En-lt
Avg.
Training in the ST setting w/o using transcription data
Fairseq ST (Wang et al., 2020a)
22.7
27.2
15.3
32.9
22.7
24.2
Self-training (Pino et al., 2020)
25.2
34.5
SpecRec (Chen et al., 2021)
20.8
25.3
13.1
30.3
20.5
22.0
Revisit ST (Zhang et al., 2022)
23.0
28.0
15.6
33.5
23.5
24.7
(Fang et al., 2022)
24.1
29.4
16.3
35.0
24.8
25.9
ST baseline CCSRD
24.5
29.4
16.1
34.8
24.9
25.9
25.4
30.2
16.9**
35,8
25.8**
26.8
Training in the MTL setting w/ using transcription data
XSTNet (Ye et 2021)
25.5
29.6
16.9
36.0
25.5
26.7
SATE (Xu et al., 2021)
25.2
Mutual-learning (Zhao al., 2021)
28.7
36.3
STEMM (Fang al., 2022)
25.6
30.3
17.1
36.1
25.6
26.9
ConST (Ye et al., 2022)
25.7
30.4
17.3
36.8
26.3
27.3
MTL baseline
25.5
30.0
17.0
36.0
25.7
26.8
CCSRD
26.1**
31.0**
17.8*
37.1
26.4**
27.7
Table I: BLEU scores of different models on the MuST-C IsI-COMMON set. "ST baseline" and "MTL baseline" are the implemented strong baselines using the same architecture as our model, excluding the disentanglement module and non-content encoder. ** denotes that the improvements over significant (p < 0.01).
baseline is statistically
input dimensions, since the input of the network is the concatenation of content and non-content representation. The classifier network comprises three FC layers with ReLU activation, an adaptive average pooling layer and a log softmax layer. Experiment Details We used the raw 16-bit 16 kHz mono-channel audio wave speech input and removed utterances of which the duration is longer than 300K frames. For each translation di- rection, we used a unigram SentencePicee (Kudo and Richardson, 2018) model to learn a vocabu- lary on the text data from the dataset, which is the same Ye et al. (2021, 2022)'s setup. For eval- uation, computed case-sensitive detokenized BLEU using sacreBLEU (Post, 2018) MuST- C tss-COMMON set. Appendix B contains more detailed settings.
we compared with strong baseline "ST baseline" that uses the same neural architecture with W2V2- Transformer (excluding the proposed non-content encoder and disentanglement module). We also compared against a strong "MTL baseline" that uses the same neural architecture as the ST baseline but is trained in the MTL setting, which is the same as XSTNet (Ye et al., 2021).
4.3 Main Results
Comparison to End-to-End Baselines We com- pared our model with baselines for five language pairs the MuST-C benchmark dataset. Results are shown Table In the ST setting that does not use transcriptions in the MUST-C data triples, our model achieves a substantial improvement 0.9 BLEU over the ST baseline on average and outper- forms the strongest baseline, in all translation directions. Since MTL models have achieved state-of-the-an results in recent stud- ies, also implemented the proposed CCSRD in the MTL setting that employs transcriptions to per- form MT and ASR tasks. Our model achieves 0.9 BLEU improvement over the MTL baseline, and even the strong model, COnST. These validate the effectiveness of our proposed approach in enhanc-
Baselines We compared our model with multiple strong E2E ST baselines including: (1) Fairseq ST (Wang et al., 2020a), (2) Self-training (Pino et al., 2020), (3) SpecRec (Chen et al., 2021), (4) Revisit ST (Zhang et al., 2022), (5) W2V2-Transfomme (Fang et al., 2022), (6) XSTNet (Ye et al., 2021). (7) SATE (Xu et al., 2021), (8) Mutual-learning
(Zhao et al., 2021), (9) STEMM (Fang et al., 2022)
and (10) ConST (Ye et al., 2022). Additionally,

---

Models
En-De
En-Ru
Cascaded
Espnet
23.6
16.4
(Ye al., 2021)
25.2
17.0
End-to-end
CCSRD
26.1
17.8
Table 2: Comparison to the cascaded baselines on the MuST-C En-De and En-Ru tst-COMMON set. "Cas- caded" is the implemented strong cascaded baselines.
(a) w/ CCSRD
(h) w/ CCSRD
Figure 2: Bivariate KDE contour of the speech and text representations. Blue curves are text representa- tions while orange curves represent speech represen- tations. Samples are drawn from the MuST-C En-De tst-COMMON set.
Models
En-De
En-It
CCSRD
25.4
25.8
w/o masking
25.1
25.5
w/o LCON and LNCNN
24.7
25.1
Models
w/o Noise
w/ Noise
w/o LSPK and CREC
24.5
24.9
ST baseline
24.5
24.2
CCSRD
25.4
25.3
Table 3: Ablation study results the MuST-C En-De Is-COOMON set.
Table BLEU scores on the MuST-C En-De tst- COMMON set while artificially noisy introduced during training.
ing ST performance using speech representation disentanglement. Notably, even not being trained in the MTL setting, our model achieves the same BLEU score as the baseline model trained in the MTL. setting, but with reduced training time. Comparison to Cascaded Baselines We compared our method with several cascaded baselines. Espnet (Inaguma et al., 2020) and the model proposed by Ye et al. (2021) are two strong cascaded systems trained using MuST-C and external ASR and MT data. From Table 2, we find that as an end-to- end model, our model outperforms these strong cascuded models even not using any external data.
formance drops substantially. Experimental results convincingly demonstrate the effectiveness of these approaches improving ST performance.
Analysis
5.1 Can CCSRD Learn Content-Centric Representations?
To empirically demonstrate the effectiveness of our model in learning content-centric representations, we plot the bivariate kernel density estimation (KDE) (Parzen, 1962) contour of speech and text dim-reduced representations to visualize their dis- tributions Figure 2, where t-SNE (Van der Maaten and Hinton, 2008) is used reduce the dimension of representations into 2D. Ideally, if the representa- tions of speech have less non- linguissii information, their KDEs will be similar to the representations of text, resulting in the contour lines overlapping as much as possible. As illustrated in Figure 2, without CCSRD, the overlap between speech rep- resentation distribution and text representation dis- tribution is small. In contrast, when CCSRD is applied, representations of different modalities be- come closer compared with those learned by the baseline. This suggests that speech representations contain more content information similar to that text representations, which is beneficial to ST.
4.4 Ablation Study
To gain deep understanding of the effect of com- ponents deployed in our proposed model, con- ducted an ablation study by progressively removing the masking strategy, the cyclic reconstruction loss LCON and LNCON. and the speaker classifier loss LsPK and feature recontruction loss LREC- Results are shown in Table 3, which indicate that the mask- ing strategy contributes to an average improvement of 0.3 BLEU. Besides, CCSRD without masking strategy can significantly enhance the translation performance, with improvement of 0.6 BLUE achieved both in the En-De and Ei-lt translation directions. Moreover, the cyclic reconstruction task plays an extremely important role in speech repre- sentation disentanglement, without which the per-

---

External MT
MTL baseline
CCSRD
25.5
26.0
27.2
28.1
Table 5: BLEU scores of CCSRD vs. MTL haseline on the MuST-C En-De tst-COMMON set while external MT data is used during training.
lsseltie line
(a) w/o CCSRD
CCSRD
27.5
25.4
25.0
Figure 3: Visualization of cross-attention learned by CCSRD vs. ST baseline by averaging all heads of the last cross-attention layer.
20.0
18.5
17.5
16.9
We also conducted experiments to analyze the disentanglement ability of CCSRD. We trained CCSRD and ST baseline on data with artificially- generated noise and compared their performance. Noise data were obtained by adding part of the in- formation of another speech to the original speech input. Specifically, for speech input 8;, we ran- domly selected another speech sj from the training set noise, and add it to the s; according to the preset weight of 0.15 obtain new noisy data: s; =; 0.15 * sj. Results of these experiments are shown in Table We observe that the perfor- mance of CCSRD degrades slightly while the per- formance of the ST baseline drops substantially. validates that our method is able to achieve success- ful disentanglement and that the content encoder is capable of learning high-qualig content-centric representations.
12.5
50
100
408
Hours of ST data
Figure 4: BLEU COMMON set when the amount of speech data used for training is varying from 50K 408K hours.
the MuST-C En-De
trained with external MT data in the MTL setting achieves further improvement of 2.1 BLUE while the MTL baseline obtains an improvement of 1.7 BLUE. We conjecture that with the external MT data for training the MT task in MTL setting, CCSRD able to learn more on content-to-target translation alignment and hence to generate better translations.
5.3 CCSRD in Low- Low-Resource Scenarios
To further compare the efficacy of our ap- proach against the bascline, we visualize the cross- attention matrices of the ST baseline and CCSRD in Figure 3. We obtain the attention matrix by av- eraging all heads of the last cross-attention layer. Our model confidently aligns target tokens their corresponding speech frames respectively as Fig- ure 3, where the baseline model yields an incorrect translation with inappropriate attention weights.
We extend the proposed CCSRD ST in low- resource scenarios in the ST setting investigate if is still able to improve translation quality. We conducted experiments using different hours speech data from the training dataset, simulating low-resource conditions. The results are shown in Figure 4. We reduced the ST data to 50 and 100 hours, corresponding to around 25K and 50K sentences. We observe that CCSRD is particularly helpful when the amount of speech data is small.
5.2 Dose External MT Data Further Improve ?
Conclusions
Many previous studies regard "leveraging exter- nal MT data" be one of the advantages of their models and achieve better performance in the MTL setting. Therefore we further investigated CCSRD trained with external WMT16 En-De dataset. Re- sults are shown Table We observe that CCSRD
In this work, have presented CCSRD, E2E ST framework that explores speech representation disentanglement learning capture content rep- resentations for ST. Experimental results validate the effectiveness the proposed framework under both the ST task setting and the multi-task setting.

---

In-depth analyses demonstrate that CCSRD is ca- pable of disentangling linguistic content from non- linguistic speech factors. We would like to employ more speech factors to guide the training of the non-content encoder and explore the disentangled non-content representations for improving ST in the future.
Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 6008-6019.
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text trans- lation. In NIPS workshop on End-to-end Learning for Speech and Audio Processing.
Limitations
David Bertoin and Emmanuel Rachelson. 2022. Disen- tanglement by cyclic reconstruction. IEEE Transac- tions on Neural Networks and Learning Systems.
Although the proposed method facilitates ST to learn content-centric representation, and obtains significant improvements over previous methods, it still has some limitations: (1) We need labels of non-linguistic speech factors (e.g., speaker IDs, prosody labels) to guide the training of the non- content encoder. (2) It is difficult to analyze the non-content representation since they may be re- lated to many different speech factors. (3) Non- content representations could also be used to im- prove ST, which we leave to our future work.
Chak Ho Chan, Kaizhi Qian, Yang Zhang, and Mark Hasegawa-Johnson. 2022. Speechsplit2. 0: Unsuper- vised speech disentanglement for voice conversion without tuning autoencoder bottlenecks. In ICASSP IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 6332 6336. IEEE.
David M Chan and Shalini Ghosh. 2022. Content- context factorized representations for automated speech recognition.
Proceedings of the 23rd An- nual Conference of the International Speech Commu- nication Association.
Ethics Statement
This work presents a framework and training strate- gies help the model learn content-centric repre- sentations for speech translation. We evaluated our method on the widely-used standard benchmark dataset and did not introduce additional data that may cause ethics issues.
Junkun Chen, Mingho Ma, Renjie Zheng, and Liang Huang. 2021. Specrec: An alternative solution for improving end-to-end speech-to-ted translation via spectrogram reconstruction. In Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, pages 2232-2236.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman. Ilya Sutskever, and Pieter Abbeel. 2016. Infogam: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29.
Acknowledgments
The present research was supported by the Nat- ural Science Foundation of Xinjiang Uygur Au- tonomous Region (No. 2022D01D43) and the Key Research and Development Program of Yunnan Province (No. 202203A We would like thank the anonymous reviewers for their insight- ful comments.
Pengyu Cheng, Martin Renqiang Min, Dinghan Shen. Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. 2020. Improving disentangled text representation learning with information theorettc guidance. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7530-7310.
References
Mattia Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. Must-c: a multilingual speech translation corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012-2017. Associa- tion for Computational Linguistics.
Jakob Abeßer and Meinard Müller. 2021. Towards au- dio domain adaptation for acoustic scene classifica- tion using disentanglement learning. arXiv preprint 10.1566
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for supersupe learning of speech representations. Advances in neural information processing systems, 33:12449-1244 12460.
Qianqian Dong, Mingxuan Wang, Hao Zhou. Shuang Xu, Bo Xu, and Lei Li. 2021. Consecutive decod- ing for speech-to-text translation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 2738-12748.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. 2019.

---

Yichao Du, Zhirui Zhang, Weizhi Wang, Boxing Chen, Jun Xie, and Tong Xu. 2022a. Regularizing end-to- end speech translation with triangular decomposition agreement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10590- 10598.
Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 245- 254.
Seunghun Lee, Sunghyun Cho, and Sunghoon Im. 2021. Dranet: Disentangling representation and adaptation networks for unsupervised cross-domain adaptation. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 15252- 15261.
Zongyang Du, Berrak Sisman, Kun Zhou, and Haizhou Li. 2022b. Disentanglement of emotional style and speaker identity for expressive voice conversion. In Interspeech 2022, 23rd Annual Conference of the International Speech Communiccation Association.
Wei Li, Can Gao. Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021. Unimo: Towards unified-modal understanding and generation via cross-mode contrastive learning. Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2592- 2607.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingx- uan Wang. 2022. STEMM: self-learning with speech- text manifold mixup for speech translation. In Pro- ceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7050- 7062.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. 2016. adveriaadv training of neural networks. The journal of machine learning research, 17(1):2096- 2030.
Weiwei Lin, Chenhang He, Man-Wat Mak, and Youzhi Tu. 2023. Self-supervised neural factor analysis for disentangling utterance-leve speech representations. arXiv preprint arXivv23305.080 58099.
Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtor,, Joakim Edin, Christian Igel, Ka- trin Kirchhoff, Shang- Wen Li, Karen Livescu, Lars Maalée, et al. 2022. Self-supervised speech represen- tation learning: review, JEEE Journal of Selected Topics in Signal Processing.
Abel Gonzalez- Garcia, Joost Van De Weijer, and Yoshua Bengio. 2018. Image-to-image translation for cross-domain disentanglement, Advances in neural information processing systems, 31.
Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Enrique Yalta Soplin, Tomoki Hayashi, and Shinji Watanabe. 2020. Espnet-st: All- in-one speech translation toolkit. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 302-311.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudanpur. 2015, Librispeech: ast corpus based on public domain audio books. In ICASSP 2015-2015 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 5206-5210.
Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J Weiss, Yuan Cao. Chung- Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. 2019. Leverag- ing weakly supervised data to improve end-to-end speech-to-text translation. In ICASSP 2019-2019 IÉEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7180-7184. IEEE.
Emanuel Parzen. 1962. On estimation of a probability density function and mode. The annals of mathemat- ical statisticy, 33(3): 1065-1076.
Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020. Self-training for end- to-end speech translation. In Interspeech 2020, 2 Annual Conference of the International Speech Com- munication A Association, pages 1476-1480.
Taku Kudo and John Richardson. 2018 Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71.
Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191.
Yoohwan Kwon, ShonWhan Chung, and Hong-Goo Kang. 2020. Intra-class variation reduction of speaker representation in disentanglement frame- work. In Interspeech 2020, 21st Annual Conference of the International Speech Communication Associa- tion.
Eduardo Hugo Sanchez, Mathieu Serrurier, and Math- ias Ortner. 2020. Learning disentangled represen- tations via mutual information estimation. In Eu- ropean Conference on Computer Vision, pages 205- 221. Springer.
Matthias Sperber and Matthias Paulik. 2020. Speech translation and the end-to-end promise: taking stock of where we are. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7409-7421.
Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2022. Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to- end speech translation. In Proceedings of the 60th

---

Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang. Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, AbdeIrahman Mohamed, Michael Auli, et al. 2022. Unified speech-text pre-training for speech transla- tion and recognition. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1488- 1499.
models can directly translate foreign speech. In Pro- ceedings of the 18th Annual Conference of the Inter- national Speech Communication Association.
Yuying Xie, Thomas Arildsen, and Zheng-Hua Tan.
2021. Disentangled speech representation learning based on factorized hierarchical variational autoen- coder with self-supervised objective. In 2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6. IEEE.
Yun Tang, Juan Pino. Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a. Improving speech transla- tion by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the I 1Ith International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 4252-4261.
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Qi Ju, Tong Xiao, Jingbo Zhu, et al. 2021. Stacked acoustic- and-textual encoding: Integrating the pre- pre-trained models into speech translation encoders. In Proceed- ings of the 59th Anmual Meeting of the Association for Computational Linguistics and the Ithh International Joini Conference on Natural Language Processing, pages 2619 -2630.
Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 202 A general multi-task learn- ing framework to leverage text data for speech to text tasks. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 6213. IEEE
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to- end speech translation via cross- modal progressive training. In Proceedings of the 22rd Annual Con- ference of the International Speech Commuication Association.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Rong Ye, Mingxuan Wang. and Lei Li. 2022. Cross- modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 5099-5113
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Biao Zhang, Barry Haddow, and Rico Sennrich. 2022. Revisiting end-to-end speech-to-text translation from scratch. In International Conference on Machine Learning, pages 11193-26205. PMLR.
Hari Krishna Vydana, Martin Karafiát, Katerina Zmo- likova, Lukáš Burget, and Honza Cernocky. 2021. Jointly trained transformers models for spoken lan- guage translation. In ICASSP 2021-2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7513-7517. IEEE.
Jing-Xuan Zhang, Zue Ling, and Li-Rong Dai. 2019. Non-parallc seuuencc-t--equueee voice con- version with disentangled linguistic and speaker rep- resentations. IEEF/ACM Transactions Audio, Speech, and Language Processing, 28:540-552.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the -ucifficiiicc Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference on Natural Language Processing: System Demon- strations, pages 33-39.
Jiawei Zhao, Wei Luo, Boxing Chen, and Andrew Gilman. 2021. Mutual-learning improves end-to-end speech translation. In Proceedingy of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages
Chengyi Wang. Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020b. Bridging the gap between pre- training and fine-tuning for end-to-cnd speech trans- lation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9161-9168.
Jie Wang, Jingbei Li, Xintao Zhao, Zhiyong Wu, Shiyin Kang, and Helen Meng. 2021. Adversarially learning disentangled speech representations for robust multi- factor voice conversion. pages 846-850 -50.
Ron Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequeecce

---

A Statistics of all datasets
We also compared our method against the fol- lowing baseline models under the MTL setting us- ing transcription data, (i.e., using the transcription data).:
Languages
Hours
Sents
En-+De
408
234K
En Es
504
270K
XSTNet (Ye et al., 2021): a model that has the same structure as W2V2-Transfomee but is adopted to the multi-task fine-tuning strategy.
En-
492
280K
En It
465
258K
En-
489
270K
STEMM (Fang et al., 2022): model that bridges the modality representation gap by mixing up the speech representation quences and text transcription embedding quences.
Table Statistics of the used MuST-C datasets.
Experimental Details
Training and Implementation Details We im- plemented our model based on the fairseq toolkit and trained all models on A6000 GPUs. During the training stage, we used Adam optimizer with 31 0.9, B2- 0.98, and learning rate = le-4 with warmup 25k steps during training. During the inference stage, we saved the checkpoints with the highest BLEU and averaged the last 10 check- points. For decoding, following previous studies (Ye et al., 2022), used beam size of 10 and length penalty of 0.7 for German, 0.1 for Spanish, 0.5 for Italian and 0.4 for Russian. For the experi- ments in Table 5, we followed the previous works (Ye et al., 2021, 2022) and used the WMT16 En-De datasets as the external MT dataset, which contains 4.6 4.6M sentences.
ConST (Ye et al., 2022): model that applies contrastive learning bridge the modality gap between speech and transcriptions.
Mutual-learning (Zhao al., 2021): a model that introduces mutual-learning paradigm iteratively learn and share the knowledge between MT and ST task.
SATE (Xu al., 2021): a model that leverages an adapter to incorporate pre-trained ASR and MT models into E2E ST.
C
Case Analysis
We present translation examples yielded by CCSRD in comparison to those by the ST baseline model Table 7. We observe that ST baseline cannot accurately translate some phrases, whereas CCSRD successfully conveys the meaning, and translates more accurately than ST baseline. This improvement might be due to the ability of CCSRD in capturing content representations.
Baseline Model Details In Table 1, compared method with end-to-end baseline models under the setting of transcription data being used (i.e., only training the ST task):
Fairseq ST (Wang et al., 2020a): re- implemented model based on the Fairseq took- lit, which is trained with only the ST task data.
Self-training (Pino et al., 2020): a model trained with pscudo-labels.
SpecRec (Chen et al., 2021): model trained with spectrogram reconstruction technique.
Revisit ST (Zhang et al., 2022): model that trained with several techniques like parame- terized distance penalty and CTC-based regu- larization.
W2V2-Trane Transformer (Fang al., 2022): model that has the same structure as our ST baseline.

---

Models
Example
src: And the motives of online criminals are very easy understand.
Ref.
tgt: Und die Motive von Online-Kriminnn Kriminellen sind sehr leicht zu verstehen.
ST baseline
tgt: Und die Motive von Online-erbrechern ist sehr leicht zu verstehen.
CCSRD
tgt: Und die Motive von Online-Kriminellen sind sehr leicht zu verstehen.
Example
src: want this institution work for us, we're going have to make bureau- cracy sexy.
Ref.
tgt: Wenn wir wollen, dass diese Institution für uns arbeitet, müssen wir Bürokratie
sexy machen.
tgt: Wenn wir wollen, dass diese Einrichtung für uns funktioniert, müssen wir die Bürokratie sexy machen.
ST baseline
tgt: Wenn wir wollen, dass diese Institution für uns arbeitet, müssen wir Bürokratii sexy machen.
CCSRD
Table 7: Translation examples from the En-De St-COMMON set, yielded by the ST baseline and CCSRD. Underlined fragments grammatically incorrect inaccurate translations.


Reviews:
[{"review_id": "FE1JtrWz8M", "reviewer": "Reviewer_GTrW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank you for your detailed review and feedback. We will strive to integrate your constructive feedback into our paper.**\n\n**Reject_Reason:** Detailed analysis of the disentanglement module and how disentanglement module affects the speech translation quality.\n\n**Answer for the Reject_Reason:** First, we think we have done sufficiently detailed analysis on the disentanglement module. The analysis of the disentanglement module in our understanding should include architecture and function. For the architecture aspect, we refer to the main disentanglement module proposed in DiCyR (Bertoin and Rachelson, 2022), forcing the content and non-content encoders to learn content and non-content information through different tasks. The detailed network architecture is presented in Section 3.2 and 3.3. Furthermore, we have conducted an ablation study to analyze the effectiveness of different tasks within the module. For the function aspect, we have conducted the main analysis in Section 5. As the disentanglement module is designed to decompose representations, we performed the in-depth analysis on the obtained decomposed representations to verify the functionality of the module. In Figure 2, we have visualized the speech and text representations, demonstrating that the disentangled speech representation is closer to textual information. Moreover, we conducted experiments under simulated noisy conditions. The results presented in Table 4 show the successful disentanglement of speech by the disentanglement module. \n\nSecond, the analysis in Section 5 also verifies the effectiveness of the disentanglement module in enhancing translation performance by obtaining high-quality textual content representation. This is supported by the visualization of speech and text representations in Figure 2, which clearly demonstrates the ability of the disentanglement module in separating content information. The denoising experiment presented in Table 4 further validates the module's ability to dissociate non-content content successfully. Furthermore, the cross-attention visualization experiments in Figure 3 provide evidence that the decomposed content representation aligns more effectively with the target text content. These experiments collectively support our motivation of utilizing the disentanglement module to facilitate the content encoder in learning superior content representations while mitigating the negative impact of non-content on translation. \n\n**Thank you again for your valuable feedback, and we hope that our response addresses your concerns adequately.** \n\n[1]Disentanglement by cyclic reconstruction.  (Bertoin and Rachelson, 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a novel framework for end-to-end speech translation that learns to disentangle linguistic content from non-linguistic speech factors, such as pitch, timbre, and speaker identity. It does not require any transcription data for disentanglement learning, which makes it applicable to scenarios that lack or do not have transcription data at all. It achieves an average improvement of +0.9 BLEU in two settings across five translation directions over the baseline, outperforming state-of-the-art end-to-end speech translation models and cascaded models.", "reasons_to_reject": "I suggest the authors to give the detailed analysis of the disentanglement module and how it affects the speech translation quality.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focus on end-to-end speech translation, which is the task of translating speech from one language to another without relying on intermediate transcriptions. The paper proposes a novel framework called CCSRD, which stands for Content-Centric Speech Representation Disentanglement. The main contributions of the paper are:\n\nCCSRD learns to separate the linguistic content information from the non-linguistic speech factors (such as pitch, timbre, speaker identity, etc.) by using a disentanglement module that consists of cyclic reconstruction, feature reconstruction and speaker classifier tasks.\nCCSRD does not require any transcription data for disentanglement learning, which makes it applicable to scenarios that lack or do not have transcription data at all.\nCCSRD achieves an average improvement of +0.9 BLEU in two settings across five translation directions over the baseline, outperforming state-of-the-art end-to-end speech translation models and cascaded models.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F6MURIzdrA", "reviewer": "Reviewer_Gjy4", "summary": "", "questions": "It is helpful to add significantly analysis to Table 1 to show the significant improvements.\nWhat does “MTL” stand for is not provided in the paper.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank you for your detailed review and feedback. We will strive to integrate your constructive feedback into our paper.**\n\n**Reject_Reason**: Comparison with other disentanglement methods and demonstration of the effectiveness of the disentanglement module. \n\n**Answer for the Reject_Reason**: For comparison with other disentanglement methods, we INDEED have compared our approach with other disentanglement methods, the disentanglement method proposed by VQMIVC [4]. However, it yields worse performance and takes significantly more training time than CCSRD. It is important to acknowledge that the suitable disentanglement methods for E2E-ST are extremely limited. We hence don’t have many options to compare. This is because many disentanglement methods need supervised data for training, which is not feasible for E2E ST. Additionally, several of the remaining methods rely on specific complex modules or pre-processed features like pitch for training [1,2,3]. These methods do not align with our framework and, if adopted, would introduce substantially additional training and inference time. \n\nFor the demonstration of the effectiveness of the disentanglement module, we have conducted analysis in Section 5. As the disentanglement module is designed to separate the content information from speech, we visually compared the speech content information and text embeddings in Figure 2. This visualization clearly illustrates that the disentangled speech representation closely resembles textual information, thus verifying the efficacy of the module.  Moreover, we conducted experiments under simulated noisy conditions, and the results shown in Table 3 provide further evidence of the module's effectiveness in disentangling speech. The cross-attention visualization experiments in Figure 3 also provide evidence that the decomposed content representation aligns more effectively with the target text content. Furthermore, the main experiment highlighted in Table 1 further reinforces the benefits of acquiring disentangled speech representations, which demonstrates better translation results can be achieved by leveraging disentangled content representations. These observations solidify the positive impact and significance of the disentanglement module on enhancing speech translation quality.\n\n**Question**: Significant analysis and the meaning of MTL. \n\n**Answer for the Question**: Many thanks for this suggestion. We have conducted significance tests on our experimental results and provide the details in Table 1. Notably, the ** symbol means the improvements over baseline is statistically significant (p < 0.01) and the results clearly demonstrate the significant improvements achieved by our method.\n\nTable 1: BLEU scores s of CCSRD under different settings on the MuST-C tst-COMMON set. ** means the improvements over W2V2-Transformer baseline is statistically significant (p < 0.01).\n\n|     Model     |     En-De     |     En-Es     |     En-Ru     |     En-Fr     |     En-It     |\n|   :-----   |    :-----:   |    :-----:   |    :-----:   |    :-----:   |    :-----:   |\n|     CCSRD-ST     |25.4**|30.2**|16.9**|35.8**|25.8**|\n|     CCSRD-MTL     |26.1**|31.0**|17.8**|37.1**|26.4**|\n\nRegarding MTL, it stands for multi-task learning, which is widely used in E2E-ST to further enhance translation performance. In our paper, the specific approach of MTL involves integrating additional MT and ASR tasks to the ST model using the same speech triplets, which is explained in line 230\\~246 and line 422\\~426.\n\n**Thank you again for your valuable feedback, and we hope that our responses address your concerns adequately.**\n\n[1]Towards Disentangled Speech Representations (Peyser et al., Interspeech 2022)\\\n[2]Disentangled Speaker Embedding for Robust Speaker Verification (YI et al., ICASSP 2022)\\\n[3]SpeechSplit2.0 Unsupervised Speech Disentanglement for Voice Conversion without Tuning Autoencoder Bottlenecks (Chan et al., ICASSP 2022)\\\n[4]VQMIVC Vector Quantization and Mutual Information-Based Unsupervised (Wang et al., Interspeech 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed framework achieve superior performance than the compared baselines. The effectiveness of the masking, reconstruction, and speaker classification modules is verified with ablation studies.", "reasons_to_reject": "The comparison with other disentanglement methods is absence. As the main contribution is the introduction of the disentanglement module to reduce non-linguistic information in the translation, demonstration of effectiveness of the disentanglement module is the most important, in addition to the evaluations at the very later text output performance.", "questions_to_authors": "It is helpful to add significantly analysis to Table 1 to show the significant improvements.\nWhat does “MTL” stand for is not provided in the paper.", "paper_topic_and_main_contributions": "This paper investigates representation disentanglement learning for end-to-end speech translation. A framework is proposed to disentangle speech into content representations and non-content representations based on cyclic reconstruction, feature reconstruction and speaker classification. \nThe main contribution is introducing representation disentanglement into end-to-end speech translation, to remove the non-linguistic information from the translation process.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rcN39P2ZrH", "reviewer": "Reviewer_fmsg", "summary": "", "questions": "A. Speech representation disentanglement involves additional multi-task learning. MTL could increase the training complexity, making unstable computation of the loss function. What are the key techniques to make the training stable?\n\nB. Speech representation disentanglement is only performed in training stage. As the training procedure becomes complicated with MTL, what is the main benefits of this method over simply adding more training data (both speech and text)?\n\nC. Section 5.1 and Table 4: My understanding on the “noise” is other speech. What exactly the noise source? Is the source from a single speech, babble noise, or other noise? One type of noise is not representative.  I suggest to add different types of noise in the experiments such as street noise or car engine noise.\n\nD. Comment: My view is that the speech representation disentanglement is more useful in speech-to-speech translation. The non-content representation obtained from disentanglement process can be applied to speech generation modules to improve the generated speech.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank you for your detailed review and feedback. We will strive to integrate your constructive feedback into our paper.**\n\n**Reject_Reason1:** It is straightforward to use more training data for performance improvement. \n\n**Answer for the Reject_Reason1:** It is important to note that E2E ST data are notoriously limited and expensive to collect. The benchmark dataset MuST-C used in the paper is currently the largest speech translation dataset available to the best of our knowledge. It would be impractical to enhance translation performance just with “more training data”. Nevertheless, our method demonstrates the ability to achieve satisfactory performance even in these challenging scenarios. Moreover, it is worth noting that our method consistently achieves higher performance when supplemented with \"more training data.\"\n\n**Reject_Reason2:** The performance of the non-content representation is unknown.\n\n**Answer for the Reject_Reason2:** We appreciate this comment. However, analyzing non-content representations poses challenges due to some objective reasons. Previous studies of speech representation disentanglement have primarily focused on specific factors within non-content information, such as emotion or rhythm, which are easier to analyze. In contrast, our method aims to disentangle the entire non-content information, which comprises numerous factors. Consequently, conducting detailed analysis on such representations becomes inherently difficult. To achieve a more profound analysis of non-content information, it may be necessary to disentangle different factors effectively. However, this would require the inclusion of additional modules, which could lead to significant increase in computational and inference time. \n\n**Question_A:** The key techniques to make the training stable.\n\n**Answer for the Question_A:** First, the backbone utilized in our paper, XSTNet, is a widely adopted framework and demonstrates stability when incorporating additional losses in previous studies. Second, we freeze the parameters of the speech encoder during the backward of $\\mathcal{L}_{SRD}$, which prevents confusion in the training direction of the speech encoder that may arise from conflicting guidance from the two encoders. Third, the disentanglement method we employ is inherently stable and has been proposed under a multi-task framework. Thanks for this question. We’ll provide a detailed discussion on this in the new version.\n\n**Question_B:** The main benefits of this method over simply adding more training data.\n\n**Answer for the Question_B:** First, as mentioned in the Answer for the Reject_Reason1, speech translation data are notoriously limited and even cannot be obtained. Second, it is important to highlight that our method demonstrates improved performance while additional training data are incorporated. Third, our paper presents a robust approach to tackling the issue of non-content information, which has been identified as a bottleneck that hampers the performance of translation systems in real-world applications.\n\n**Question_C:** The noise source and the different types of noise.\n\n**Answer for the Question_C:** The noise source is the other speech in training data, simulating noisy background environment. For instance, let S1 and S2 denote the embedding of two audio inputs, we obtain the noisy input for $S_1$ by adding the embedding of $S_2$ with a weight 0.1, resulting in $S_1’ = S_1 + 0.1S_2$. Thank you for your suggestion on adding different types of noise. We will perform such an experiment in the next version.\n\n**Question_D:** The non-content representation obtained from disentanglement process can be applied to speech generation modules to improve the generated speech.\n\n**Answer for the Question_D:** We appreciate this comment and acknowledge that disentanglement learning is useful in speech-to-speech translation. However, it is important to note that speech disentanglement is also vital in speech-to-text translation. The robustness of current ST models in unseen speakers and varied environments is still very poor. In our paper, we propose a robust approach to tackling the issue that non-content information unduly influences translation, which has been shown to be a bottleneck restricting translation systems in real-world applications.\n\n**Answer for Typos Grammar Style And Presentation Improvements:** Thank you for raising the issue. We will make corrections in the next version.\n\n**Thank you again for your valuable feedback, and we hope that our responses address your concerns adequately.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper designs a speech representation disentanglement process for speech translation. In a setting of speech-to-text translation, the improved BLEU score given the same amount of training data supports the soundness of the design. ", "reasons_to_reject": "Speech representation disentanglement in speech-to-text translation is a double-edged sword. On one hand, while applying speech representation disentanglement does improve content representation and hence translation performance in terms of BLEU score, additional multi-task learning and modules complicate the training process. Given the model structure unchanged during training and inference, applying more training data is more straightforward for performance improvement, as shown in Table 5 and Figure 4.\n\nOn the other hand, yet another valuable part of speech representation disentanglement is the learnt non-content representation. The non-content representation represents the speaking style thus cannot be learnt from text-only feature. Previous works show promising results on voice cloning and generation with the disentangled non-content representation, in which these tasks are more relevant to speech-to-speech translation.  However, this work is focus on speech-to-text translation and does not use the non-content representation during inference, leaving the non-content representation untested. The performance of the non-content representation is unknown, making an incompleteness of this work.", "questions_to_authors": "A. Speech representation disentanglement involves additional multi-task learning. MTL could increase the training complexity, making unstable computation of the loss function. What are the key techniques to make the training stable?\n\nB. Speech representation disentanglement is only performed in training stage. As the training procedure becomes complicated with MTL, what is the main benefits of this method over simply adding more training data (both speech and text)?\n\nC. Section 5.1 and Table 4: My understanding on the “noise” is other speech. What exactly the noise source? Is the source from a single speech, babble noise, or other noise? One type of noise is not representative.  I suggest to add different types of noise in the experiments such as street noise or car engine noise.\n\nD. Comment: My view is that the speech representation disentanglement is more useful in speech-to-speech translation. The non-content representation obtained from disentanglement process can be applied to speech generation modules to improve the generated speech.", "paper_topic_and_main_contributions": "This paper proposes an improved speech-text translation system. By incorporating speech representation disentanglement during the training phase, an improvement of 0.9 BLEU is observed on the evaluation set. As speech translation is a typical NLP application, the topic is well fit to EMNLP.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 4: noisy -> noise"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Work is dedicated to end-to-end speech translation and proposes a framework (CCSRD) that disentangle linguistic content from non-linguistic content in the speech representations. The motivation for learning such disentangled representations is clear and method proposed is interesting. The method is evaluated on a speech2text translation task (MustC) and it achieves an improvement compared to baselines. However, reviewers highlighted the lack of detailed analysis of the disentanglement module and not enough comparisons with other approaches. Moreover, it is unfortunate that the work is focusing only on speech-to-text translation task and does not use the ‘non-content representation’ during inference as well (for instance for a S2S task or a speaker id task or emotion recognition task). As the non-content representation is unused, the paper sounds a bit incomplete and i’d recommend only an acceptance to findings.
Decision: Accept-Findings