Improving Speech Translation by Fusing Speech and Text
Authors: Wenbiao Yin
Zhicheng Liu
Chengqi Zhao
Tao Wang
Jian Tong
Rong Ye
Venue: EMNLP 2023 Findings (2023)
Keywords: speech translation
multimodal

Abstract: In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths of speech and text to improve speech translation. However, speech and text are disparate modalities, we observe three aspects of modality gap that impede their integration in a speech translation model. To tackle these gaps, we propose **Fuse**-**S**peech-**T**ext (**FuseST**), a cross-modal model which supports three distinct input modalities for translation: speech, text and fused speech-text. We leverage multiple techniques for cross-modal alignment and conduct a comprehensive analysis to assess its impact on speech translation, machine translation and fused speech-text translation. We evaluate FuseST on MuST-C, GigaST and newstest benchmark. Experiments show that the proposed FuseST achieves an average 34.0 BLEU on MuST-C En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FuseST does not degrade on MT task, as observed in previous works. Instead, it yields an average improvement of 3.2 BLEU over the pre-trained MT model. Code is available at https://github.com/WenbiaoYin/FuseST.

Improving Speech Translation by Fusing Speech and Text
Wenbiao Yin', Zhicheng Liu²; Chengqi Zhao', Tao Wang', Jian Tong', Rong Ye² DDepartment of Computer Science and Technology, Nanjing University 2ByteDance wenbiaoyin@smail.nnuu (liuzhicheng.lzc, zhaochengqi.d, wangtao.960826, tongjian, yerong| @ @bytedance.com
Abstract
Speech Masa
Target Transiation
Connect Apeaker
speech translation, leveraging multimodal data to improve model performance and ad- dress limitations of individual modalities has shown significant effectiveness.
End to End
PreeST
Coocade MIT Sulessskon)
Semuattee
ASR
"i du berr
this pa-
Transcrigt
per, we harness the complementary strengths speech and text to improve speech transla- tion. However, speech and text disparate modalities, we observe three aspects of modal- ity gap that impede their integration a speech translation model. To tackle these gaps, propose "use-Speech-Tex (FuseST), a cross- modal model which supports three distinct in- put modalities for translation: speech, text and fused speech-text. We leverage multiple techniques for cross-modal alignment and con- duct comprehensive analysis to assess its impact on speech translation, machine trans- lation and fused speech-text translation. We evaluate FuseST on MuST-C, GigaST and new- stest benchmark. Experiments show that proposed FuseST achieves an average 34.0 BLEU on MuST-C En->ee/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FuseST does not degrade on MT task, as observed in previous works. Instead, it yields average improvement of 3.2 BLEU over the pre-trained MT model. Code is available at https://githubs comm/Wenbiaaoiin/FusST.
Fladon
Spends
Traneiit
Figure 1: Left: Data distributions of speech and tran- script. Right: The pipeline of our proposed FuseST.
2022) directly map speech signals to the tar- get translation, thus avoiding the problem of error propagation.
We observe noticeable modality distribution difference between speech and ASR transcript as shown in Figure la. Speech signals contain abundant information, such as paralinguistics and speaker characteristics, but they are harder to model and more susceptible to noise. On the other hand, ASR transcripts are clear but may contain errors. For example, as shown in Figure 1b, the speech sig- nal conveys the message "I'm m delight to be here". but the speech signal contains numerous blank seg- ments and background noise. E2E ST may en- counter challenges in accurately extracting infor- mation directly from speech, particularly in the presence of noise, without compromising transla- tion quality. Mcanwhile, the ASR model mistak- enly transcribes the speech as "I'd like to be here". The MT subsystem may proceed with translation without awareness of errors in "I'd like".
1 Introduction
Speech translation (ST) accepts speech signals as the input and outputs target translation. Speech translation can be broadly categorized into cascade system and end-to-end speech translation (E2E ST). Cascade system (Sperber et al., 2017; Zhang et al., 2019; Lam et al., 2021) usually combines automatic speech recognition (ASR) and machine translation (MT). The MT subsystem uses ASR transcripts as input, which provide clear expression but may contain errors stemming from ASR. While E2E ST (Tang et al., 202 la; Fang et al., 2022; Ye et al.,
Inspired by these findings, we propose a model that fuses speech and ASR transcript (fused speech- text) input leverage their complementary strengths improve speech translation. As shown in Figure 1b, our model supports three distinct modalities of input for translation: speech, text and fused speech-text.
However, speech and text are disparate modal- ities, we observe three aspects of modality gap that impede their integration in a speech transla-
"Corresponding author

---

tion model: 1. The speech representation is in continuous space, while the text representation is in discrete space. 2. When using speech and golden transcript as input, the model relies heavily on the golden transcript information and neglects the speech information. This behavior is because the golden transcript is more straightforward and contains precise semantic information, adequate for achieving high translation quality. The model will learn a shortcut to rely solely on the golden transcript. In contrast, speech representations ob- tained through the pre-trained model are harder to model, they include four types of information (Chen et al., 2022): content, semantics, paralinguis- ties and speaker characteristics. However, when us- ing speech and ASR transcript as input, solely rely- ing textual information is insufficient achieve good translation quality. We expect the model to incorporate more speech information. Diverse modality inputs lead to distinct hidden states in the encoder and distinct distributions in the decoder.
C Cnn+D///s/f (vs SOTA +1.1 BLEU) and achieves an average improvement of 3.2 BLEU over the pre-trained MT model on MuST-C.
Related Work
Cascade ST Cascade ST, achieved by concatenat- ing ASR and MT components, has been extensively employed in commercial speech translation sys- tems. However, cascade ST is vulnerable chal- lenges such as error propagation and high latency. To overcome the error propagation, ( (Bertoldi and Federico, 2005; Beck et al., 2019; Sperber et al., 2019) proposed to feed the MT system with ASR data structures; ( Peitz et al., 2012: Cheng et al.. 2019; Di Gangi al., 2019a) proposed to make MT robust to ASR errors, for instance by training on parallel data incorporating factual or emulated ASR errors.
End-to-Fna ST To overcome the error propaga- tion and high latency in the cascade ST systems, (Bérard et al., 2016; Duong et al., 2016) proposed an end-to-end architecture for speech translation, which has attracted extensive attention Vila al., 2018: Salesky et al., 2019; Gangi et al., 2019; In- aguma al., 2021; Zhao al., 2021). However, it difficult to train an end-to-end speech translation model directly, primarily due to the inherent vari- ability and complexity of speech signals and the scarcity of high-quality speech-translatie datasets. Some training methods like pretraining ( Weiss et al., 2017: Bérard et al., 2018; Bansal et al., 2019; Wang et al., 2020a; Tang al., 2021b), multi-task learning( Le al., 2020; Vydana al., 2021; Ye et al., 2021; Tang et al., 2022), data augmentation ( Park et al., 2019; Jia et al., 2019; Bahar et al., 2019; Pino et al., 2020), meta-lcarning Indurthi et al., 2020), contrastive learning ( (Li et al., 2021; Ye et al., 2022), knowledge distillation ( (Liu et al., 2019; Tang et al., 2021a) and curriculum learning Kano et al., 2017; Wang et al., 2020b), are proved to be effective.
We propose several methods to bridge the modal- ity gap to better integrate speech and text. 1. We explore mapping continuous speech representation to discrete space using codebook to align with the text representation. 2. We adopt the prompt tags implicitly guide the model to utilize more speech information when the transcript inaccurate. We further explore how to explicitly guide the model to fuse speech and text. Meanwhile, we apply Cross- modal Contrastive Learning (CCL, Sohn, 2016) to reduce the gap between the model semantic of speech and its corresponding transcript. 3. We Regularization (CAR, Tang al., 2021 a) align the states of the encoder and Cross-Modal Regularization (CMR) to align the distribution the decoder.
adopt
Our contributions are summarized as follows:
We propose model that fuses speech and text to improve speech translation, which supports three distinct input modalities for translation: speech, text and fused speech-text.
To fuse speech and text as input and leverage their complementary strengths, we conduct comprehensive analysis of the modality gap between speech and text. We propose targeted improvements to bridge the modality gap be- tween speech and text.
Methods
3.1 Problem Formulation
The speech translation corpus is usually comprised of triples that include speech, transcript and target translation, which can be denoted D (s x, w). Here, s an audio sequence, x. is the corresponding transcript and y is the corresponding target transla- tion. Our model supports three distinct input modal-
Our experiments show that our model achieves an average 34.0 BLEU on MuST-

---

ities for translation: speech (ST: s + y), text (MT: x y) and fused speech-text (FT: s y).
CMP Σ log P(yn
(2)
CFT Σ log P(y Sn, Tn)
(3)
3.2 Model Framework
As shown Figure 2, our model consists of four sub-modules: Speech Encoder, Speech- Text Fusion Module, Transformer Encoder and Transformer Decoder.
LASH Σ log ((xn)(n))
(4)
LST, CMT, LFT and LASR are cross-entropy losses on <speech, target,, <transcript, target>, <speech, transcript, target> and <speech, tran- script>, respectively.
Speech Encoder The speech encoder(S-Ennc) con- sists of Wav2vec2.0 (Baevski et al., 2020b) and two additional convolutional layers. The input is waveform signal sampled at 16kHz, and Wav2vec2.0 is used to extract low-level speech additional convo-
3.3 Fusing Speech and Text
representations from it, The lutional layers with stride 2 to shrink the speech length by a factor of 4. A greater degree of down- sampling would have led to information loss, while a lesser degree of downsampling could have re- sulted in modal misalignment and compromised performance. Denote S-Enc(s) the speech representation.
To leverage the complementary strengths between speech and text, we propose the Fuse-Speech Text(FuseST)) method. We first introduce FuseST in this section and later show how bridge the modality gap between speech and text.
Here, we utilize an open-source ASR model to construct a dataset of <speech, ASR transcript, target> pair from the original dataset of <speech, golden transcript, target> pair. Given speech- transcript-target pair (s,r, the transcript x could be an ASR transcript or golden transcript, and the fused speech-text representation is de- fined as:
To reduce the number of parameters and facili- tate knowledge transfer, we share the Transformer encoder and Transformer decoder for ST, MT and FT.
Transformer Encoder and Transformer Decoder The Transformer encoder and Transformer decoder are composed of Ne transformer encoder layers and Na transformer decoder layers, respectively, with the same configuration the original implemen- tation (Vaswani et al., 2017). We fisrt pre-train the model on external MT data and then optimize the whole model by minimizing the final loss. For the MT task, the input of the Transformer encoder is the embedding of transcript = Emb(x). For the ST task, the input is the audio output repre- sentation of the speech encoder a = S-Enc(s). For the FT task, the input is the fused speech- text representation fssee details in Section 3.3). The Transformer encoder further extracts the high- level semantic hidden representations and facili- tates knowledge sharing across the three modalities. The Transformer decoder generates corresponding target translation for ST, MT and FT. Besides, we train our model with auxiliary ASR task to improve translation performance. The training losses of ST, MT, FT and ASR are as follows:
Concal( S-Enc(s), PA, Emb(x))
(5) where ps and pt are prompt tags to identify whether the input modal speech or text, and pf is a prompt tag (<golden>/<asr>) to indicate whether the transcript is manually annotated or gen- erated through ASR system. The prompt tags serve as implicit guidelines for the model, prompting it to hamness higher degree of speech information when the transeript is inaccurate, while inducing a decreased reliance on speech information when the transcript is deemed accurate.
3,4 Align Speech and Text with FuseST
As we analyzed in Section 1, we observe three as- pects the modality gap between speech and text. We propose several methods to bridge the modality gap between speech and text. In addition to the implicit guidance mentioned above, we introduce three additional methods in this section: Cross- modal Contrastive Learning, Cross-Attentive Reg- ularization and Cross-Modal Regularization. Cross-modal Contrastive Learning Given a pos- (s, x) pair, we
Lst
Σ log ((nn(sn)
(1)
itive example of

---

X: And things like that.
y: ihii
LARR
Lrr
LIT
Distribution(Speeell)
Distribution Speech + TTxt)
Distribution
CMR
CMR
Transformer Decoder
Shared Transformer Decoder
CAR
CAR
Shared Transformer Encoder
CCL
Prompt Tag
A
000
Speech Embedding
Text Fmbedding
CNN
Embedding
Wav2vec2.0
X: And things like that.
S: "And things like that.
Figure 2: Overview of our proposed FuseST. FT denotes fused speech-text input for translation; CCL denotes Cross-modal Contrastive Learning; CAR denotes Cross-Attentive Regularization; CMR denotes Cross-Modal Regularization.
can get speech representation a = S-Enc(s)). tran- script representation e Emb(xx. We randomly pick set of same batch as negative examples. For speech repre- sentation a and transeript representation c, we first average them in terms of the time dimension and apply the multi-class N-pair contrastive loss (Sohn, 2016):
between text representation and fused speech-text representation f. The CAR loss is defined as:
transcripts {e; | | B-1 from the
CCAR LcAnla, f) + Ecanle, (9)
Cross-Modal Regularization To bridge the modal- ity gap in inference, we endeavor to optimize the congruity of the ultimate distributions of ST, MT and FT:
u (eanPool(a)
(6)
P(yss) = P(yls, P(y]x)
(10)
McanPool(e)
(7)
The ST task more difficult than the MT task since the speech signals are harder to model and more susceptible to noise. Previous works (Liu et al., 2019; Gaido et al., 2020; Tang et al., 2021a) utilize knowledge distillation to facilitate knowl- edge acquisition by an ST model from well- trained MT model. However, in our work, the tran- script may contain errors; the accuracy of the FT task is usually much higher than the corresponding and MT task. We designate the FT task as the teacher while ST and MT tasks as student, min- imizing the loss between the student and teacher outputs. The KD loss is defined as:
exp(sim(u, ())))
LeCi. =
Σ log 5:3
Enyea exp(sim(u, ((;)))/T)
(8)
where A= {e) U (e == hyper-parameter and sim is the cosine similarity function.
B-1
is the temperature
Cross-Attentive Regularization The cross- attentive regularization (Tang et al., 2021a) (CAR) can increase the similarity among distinct modalities. The essence of CAR is to use similarity matrix project tensor sequence onto space of equivalent length another tensor sequence, then compute L2 loss between the two sequences. Here, we utilize the CAR compute losses separately between speech representation a and fused speech-text representation f and
LKD LKD(a, f) L
(11)
Furthermore, we minimize the Jensen-Shannon
Divergence (Lin, 1991) (JSD) loss between the three output distributions, which is:

---

ST
MT Train
MT Test
#hours
Hscnts
Bamee
Asents
Tome
MuST-C (Medium Resources)
LJSD (JSD (USDDpppaaaawllee (plenla.))lttuull.a))
En >iee
408
244K
WMT17
4.5M
newSteN2011
22,05
En-iBs
504
270K
WMT17
4.1M
(12)
Ra-sFr
492
280K
WMT17
5.5M
newSteN22014
22.08
+JSD ((lvalen)ll(vvsll
GiguST (llggh
Fa-2hh
9,780
7AX0K
in-loouse
10SM
newstesr2019 9/20220
21.98
The final training objective is follows:
Table 1: Statistics of all datasets.
C =aLsr + aLMr (1--))kKn)) Ler
Model Configuration For the speech encoder, we use Wav2vec2.0* following the hase configuration, which is only pre-trained on Librispeech (Panay- otov et al., 2015) without any finetuming. Two lay- of CNNs after the Wav2vec2.0 with kernel size 5, stride size 2, padding 2 and hidden dimension 1024. The transformer encoder and decoder follow the base configuration, with hidden size hd 512, attention heads and 2048 FFN hidden states. We use N. = 6 transformer encoder layers and N₂ = 6 transformer decoder layers.
+LASR LCCL + 3LCAR LJSD
(13)
where a and 3 are predefined hyper-parameters.
3.5 Inference
Our model supports three distinct modalities for translation: speech, text and fused speech-text. For FT task, given an audio sequence 8, we use an open-source ASR model to get its corresponding transcript then fuse the audio sequence and its corresponding transcript as input and output target translation.
Experiment Details We first pre-train model on the external MT dataset; the learning rate is 5e-4. We then optimize our model by minimizing the final loss; the learning rate 6e-5. We use the raw 16kHZ speech as input and jointly tokenize the bilingual text using SentencePiece (Kudo and Richardson, 2018). We use an Adam optimizer with 31 0.9, 82 0.98 and 20k warm-up up- dates. The dropout is set 0.15 and the value of label smoothing is set 0.1. For the training loss, we set weight of LSTT and CasT a 0.8, con- trastive temperature r 0.02 and weight of LCAR 0.02. We use sacreBLEU" (Post, 2018) to evaluate case-sensitive detokenized BLEU.
4 Experiments
4.1 Experimental Setups
ST Datasets We conduct experiments on MuST- C (Di Gangi et al., 2019b) and GigaST? (Ye et al.). MuST-C is multilingual speech trans- lation dataset that contains translations from En- glish to languages. MuST-C contains several hundred hours of audio recordings from English TED Talks; we conduct experiments on MuST-C En En->De/ff De/Es//Fr. We use the dev set for develop- ment and the tst-COMMON set for test. GigaST is a large-scale pseudo speech translation dataset created by translating the text in GigaSpeech. We conduct experiments on GigaST En, and test in-house cgtn, zhiyuan and aiconf datasets. We utilize an open-source ASR model (whisper base.en 3) construct a dataset <speech, ASR transcript, target> pair from original ST dataset. The Word Error Rate (WER) of our constructed datasets are shown in the last column of Table 1.
4.2 Baseline Systems
We compare our method with cascade models and end-to-end baseline models including: Espnet (In- aguma et al., 2021), W2V2-Transformer (Fang et al., 2022), Ye et al., 2021, Xu et al., 2021, MTL (Tang et al., 2021b), FAT-ST (Zheng et al., 2021), JT-S-MT (Tang et al., 2021a), Chimera (Han et al., 2021), XSTNet (Ye al.. 2021). SATE (Xu et al., 2021), STEMM (Fang et al., 2022), TaskAware (Indurthi et al., 2021), STPT (Tang et al., 2022), ConST (Ye et al., 2022). Besides, we implement several methods for fus- ing speech and text modalities. The only difference
MT Datasets Our model allows us to use the ex- ternal MT dataset for further training. We intro- duce external WMT datasets for De/ES/Fs/FFr and in-house MT dataset for En-Zh. The detailed statisties of datasets are shown in Table
We use v1.0 https: ftw.et/tttc///e//c/////ccc
/https://dl. ftwpp//iff.ffaappuiiclilllllllee fate://oleeeeeese..caa/rrrrr//o Nev2vec//aa2veec_smell/
GigaST http://chcnrrriiitiii.ieeeeeeeee /
'https://githul com./jjost//sccebbl sacreBLEU signature: nrefs:l 1b:10001 secd: 2345 case:mixed eff:no 1ok: smooth:cxp version:2.0.0
htttps://uugingf/an http:/aa...aapaaa/iiiieeeereeeee

---

External Data
BLEU
Models
Speech
Text
ASR
MT
De
Es
Fr
Avg.
Cascade Model
Espnet (Inaguma al., 2021)
23.6
33.8
W2V2-Transformer (Fang et al., 2022)
v
26.9
30.0
36.6
31.2
(Ye et al., 2021)
v
25.2
28.1 34.9
(Xu al., 2021)
v
End-to-End Model
MTL (Tang et al., 2021b)
23.9
28.6
33.1
28.5
FAT-ST (Zheng et al., 2021)
25.5
30.8
JT-S-MT (Tang et al., 2021a)
26.8
31.0
37.4
31.7
Chimera (Han et al., 2021)
27.1
30.6
35.6
31.1
XSTNet (Ye et al., 2021)
27.1
30.8
38.0
32.0
SATE (Xu et al., 2021)
28.1 1
STEMM (Fang al., 2022)
28.7
31.0
37.4
32.4
TaskAware (Indurthi et al., 2021)
28.9
STPT (Tang et al., 2022)
33.1
39.7
ConST (Ye et al., 2022)
28.3
32.0
38.3
32.9
FuseST-ST
27.7
32.4
37.2
32.4
FuseST-FT
29.2
33.9
38.9
34.0
Table 2: Case-sensitive detokenized BLEU scores on MuST-C tst-COMMON set. "Speech" denotes unlabeled audio data, "Text" denotes unlaheled text data, e.g. Europarl (Koehn, 2005), CC25 (Liu al., 2020), i use 40M OpenSubtitles (Lison and Tiedemann, 2016) external MT data.
Fr
Models
Avg.
newstest
asr
golden
newstest
golden
newstest
asr
golden
Pre-trained 6E6D MT
23.7
24.3
30.4
31.8
29.5
34.2
34.7
32.2
39.9
31.2
Pre-trained 24E6D MT
27.5
25.7
31.9
34.8
32.0
37.0
38.1
34.2
41.5
33.6
STEMM (Fang et al., 2022)
31.5
FuseST-MT
25.0
28.6
34.2
32.9
33,5
37.7
35.9
37.7
44.1
34.4
Table Case-sensitive detokenized BLEU scores on MuST-C Ist-COMMNN set and newstest. "asr" denotes use the ASR transcript input, "golden" denotes the golden transcript input. 1 trained the same external MT data and MuST-C <transcript, target> pair data (including our constructed data).
between our approach and others is the specific method for fusing speech and text. SA-CTR: our implementation involved drawing inspiration from the image and text fusion techniques employed in Li al.'s (2022) propose a method for fus- ing speech and text. We utilize the method from Baevski et al. (2020a) to map the speech represen- tation onto discrete space using a codebook; then we concatenate discrete speech representa- tion and text representation ours. Codebook- Gumbel-Softmanl the Gumbel-Softmax quantiza- tion computes logits representing the codebook vec- tors; Codebook-E Bodebook-K-meang K-means: K-means vector quanti- zation computes the distance to all codeword vector and chooses the closest.
works rely on additional auxiliary data for better performance, e.g. large-scale MT data and unla- beled audio data. In the table, provide sum- mary of the auxiliary data employed by these base- lines, with / denoting its usage in the corre- sponding column. Our E2E FuseST-ST achieves comparable results with the previous best models. When fusing speech and ASR transcript as input, our FuseST-FT outperforms SOTA by an average 1.1 BLEU on MuST-C, demonstrating the superior- ity of our approach.
Comparison with Cascade Baselines We com- pare our model with several strong cascade sys- tems.
Ye et al. (2021) and Xu al. (2021) provided three strong cascade sys- tems trained using MuST-C and external ASR and MT data. As shown in Table 2, our E2E FuseST- ST achieves comparable results with these strong cascade models, while our FuseST-FT significantly outperforms these strong cascade models.
4.3 Main Results
Comparison with End-to-End Baselines As shown Table 2, we compare our model with several strong end-to-end baselines. Many existing

---

in-house egin speech
in-house zhiyuan speech
in-house aiconl
Models
newstest2019
newstest2120
asr
fused
asr
fused
asr
speech
fused
Pre-truined 6E6D MT
36.7
43.0
29.7
28.2
30,5
Pre-trained 24E6D MT
40.1
46.6
32.6
29,5
33.6
SA-CTR (Li et al., 2022)
36.8
43.1
31.0
31.8
31.0
28.3
29.4
29.1
31.6
32.4
32.1
Grebbook-Gubbl-Sonn
36.8
43.0
30,9
30.0
31.7
28.1
28.4
28.9
31.7
31.6
32.5
Codebook-K-mean
36.9
43.2
31,0
31.3
31.8
28.2
29.3
29.0
31.5
31,7
32.3
Align-Mash Mask?
38.3
43.9
30.8
32
31.7
29.8
32.1
33.1
33.0
FuseST
38.3
44.2
31.6
33,3
33,9
28.4
30.2
30.5
32.4
34,0
34.5
Table 4: Case-sensitive detokenized BLEU scores En 77h test sets. i are training on the same MT data and GigaST <transcript, target> pair data (including our constructed data); are using the method from (Baevski et al.. 2020a) map the speech representation onto a discrete space using a codebook; is explicit guidance fusion(see detail in Section 5.6(6)). "asr" denotes use the ASR transcript as input, "fused" denotes use ASR transcript and speech as input. Hubert large is used extract speech features on the GigaST En-
Tasss
MI
MT
S1
FT
Task
MT newstest
MT' asr
ST
FT
Config
nowdeet
speach
fised
Model
speech
fused
FuseST
15.0
23.6
27.7
29.2
Easn
24.9
26.5
27.4
28.9
Wav2vec2.0
25.0
28.6
27.7
29.2
LLen ExD Liep
25.2
25.0
27.1
HuBERT Large
25.0
28.4
29.3
29.7
Eash ExD Ljon fort Aran
24.8
27.6
26.4
28.5
HUBERT Extra Large
25.1
28.4
30.0
30.3
Table 5: BLEU scores on MuST-C En->De tst- COMMON set and newstest set by removing individual losses.
Table 6: The impact of speech pre- pre-trained model MuST-C En-+De tst-COMMON set and newstest set.
5 Analysis
Comparison with MT Baselines We fisrt pre-train our model on external MT data and then jointly train multiple tasks. Previous work has encoun- tered catastrophic forgetting problems on MT task during joint training (Fang et al., 2022), which sig- nificantly degrades performance on MT tasks. We evaluate our model on the MT task and show the result in Table 3 and Table 4. Our model achieves significant improvement on the MT task instead of a decline in performance. Our model even outper- forms pre-trained 24F6D (24 transformer encoder layers and 6 transformer decoder layers) MT on MuST-C. However, with the increase of training data, our model performs lower than pre-trained 24E6D MT when using ASR transeript as input (En-ZZh). Nevertheless, our model can still outper- form pre-trained 24E6D MT when fusing speech and text as input on En-*Zh
5.1 Is FuseST better than other fusion methods?
We compare our model with different fusion meth- ods, such as SA-CTR and STEMM. SA-CTR pro- posed selective attention and gated fusion mech- anism to fuse two different modalities; STEMM proposed the speech-text manifold mixup to mix up the representation sequences of different modal- ities. Our model achieves better results than theirs by utilizing a prompt-based approach.
5.2
Should speech representation be discrete or continuous?
As Section mentions, speech input representation is in continuous space, while text input represen- tation is in discrete space. We utilize the method from Baevski et al. (2020a) to map the speech representation onto a discrete space using code- book. However, Codebook-Gumbes Gumbel--Sftmmn and Codebook- K-manns led to a decrease in BLEU score (shown in Table 4). Our conjecture here is that the existing speech pre-trained models extract continuous features that undergo mapping pro- to a discrete space, resulting in the loss of audio information and subsequent reduction in the BLEU score. Nonetheless, if the speech pre- trained models can extract high-quality discrete features, it is plausible that such discrete features could enhance performance.
4.4 Ablation Study
As shown in Equation 13, our training objective contains eight terms. In addition to the cross- entropy objective EST,LMr, LrT, we investigate the effects of the other auxiliary training objectives. By gradually removing each loss, Table shows the improvements brought by each auxiliary training objective.

---

Average BLEU improvement
Frequency
Models
CASE extraondinanly delight
STC:
Ref.
asr:
I'm extraondinary. I'd like to be here.
0.4
150
1gt.
4.49
FuseST-MI
0.2
FuseST-ST
1gt
# R.
FuseST-FT
6.
# R.
50
CASE
0.0
src:
So you could think of the viruses like, they are people. So could think of the viruses like their people.
Ref.
WER
WER
tgt.
Myax KLE & R.A.A
tgt
Mrux &
PuseST-ST
tgt
The average BLEU improvement for ASR transcript resulting from using fuse-speech-tex En En-*Zh aiconf testset, under various conditions of WER. Right: The frequency distribution of WER on En-+Zh aiconf testset.
Figure 3: Left:
FuseST-FT
1gt
B.A.A
Table 7: Case study: cases are generated from Fus MT, FuseST-ST and FuseST-FT on En aiconf test- set. The red underlined text indicates inaccurate trans- lations, and the blue strikethrough indicates missing translation.
5.3 With the increasing advancements of speech pre-trained models, does the fusion of speech and text remain effective?
vantages of integrating speech information become more pronounced.
Here, we report the results of Wav2vec2.0. Hu- BERT Large and HuBERT Extra Large?, which are widely used in speech translation. As shown in Tabel 6, as the strength of speech pre-trained models increases, the performance of the models on ST and FT improves. Nevertheless, as speech pre-trained models undergo further advancements, the marginal gains in speech translation resulting from text fusion have shown diminishing trend. be ascribed to the progres-
5.5 Is our model robust to different ASR errors?
In the GigaST En En->Zh experiment, we utilize an ASR model different from the one used to construct the training set to generate the ASR transcripts for the (in-house cgin, zhiyuan and aiconf). As shown in Table 4, using fused speech-text for translation still outperforms using ASR transcript. Our model demonstrates strong robustness to ASR errors under various distributions.
This phenomenon sive refinement of speech representations, in con- trast to the relatively inferior quality of our textual representations (WER z 20). By incorporating stronger textual representations, the enhancement in speech translation through text would become more pronounced.
5.6
What is the comparative effectiveness between implicit guidance and explicit guidance in the fusion process?
In Scction 3.3, we utilize prompt tags to implic- itly guide the model to fuse speech and text. This section further explores how to explicitly guide the model to fuse speech and text. We propose an additional auxiliary task named Align-Mask: first the Montreal Forced Aligner(MFA, (McAuliffe et al., 2017)) toolkit get word-level speech-transerie transcript alignment pairs; then we random mask a consecutive sequence of to 4 words in the text with a probability of 15%, we only preserve the speech segment corresponding to the masked text segment; then we predict the masked words. We aspire for the model to learn the correspondence be- tween speech and text to fuse speech and text better. However, Align-Mask performs worse as shown in Table 4. Our hypothesis is that using MFA align introduces errors, which affect the effectiveness of Align- Mask. In the future, we will further explore explicit guidance fusion without external tools.
5.4 How does fusing strategy perform on different levels of ASR transcript quality?
We conduct an empirical study to examine the ef- ficacy of our approach in enhancing BLEU score through the fusion of speech information under different WER present in the ASR transcript. As shown in Figure 3, when the word eтror rate of ASR transcript minimal (0 < WER < 5), the fusion of speech information results in a slight de- crease in BLEU score. This behavior is because the transcript is adequate for achieving high translation quality, and the fusion of speech information may introduce noise. As the WER increases, the ad-
www.fappubbiicfilee.ccc/aebbaaaa 1arge_1160k.pt 'https://dl. httts::/ddlfbaaipbbl fbaipublicciii htt.se/e/////bbbbbbbbeee xtralarge_1160k.pt

---

6 Case Study
References
Alexei Baevski. Steffen Schneider, and Michael Auli. 2020a. vq- vq-wav2vec: Self-supervised learning of dis- crete speech representations. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
In this section, we present several cases generated by FuseST-MT, FuseST-ST and FuseST-FT. In the first case, the ASR model mistakenly transcribes "delight" as "I'd like" due to the highly similar pro- nunciations of these two words. FuseST-MT fails generate the correct translation as a result of errors present in the ASR transcript. Meanwhile, FuseST-ST produces omissions, as modeling direct speech to target translation proves to be more chal- lenging. Notably, only FuseST-FT translates the sentence correctly, leveraging the complementary strengths of speech and text. In the second case, the speech signal contains numerous background noises; the ASR model mistakenly transcribes the "they are" as "their", FuseST-MT and FuseST-ST are mistranslated, and only FuseST-FT translates correctly.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020b. wav2vee 2.0: framework for sulf-supervised learning of speech representations. Advances in neural information processing systems. 33:12449- 2249-12460.
Parnia Bahar, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. On using SpecAugment for end-to-end speech translation. In Proceedings of the 16th In- ternational Conference Spoken Language Trans- lation, Hong Kong. Association for Computational Linguistics.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pre- trainini high-resource speech recognition improves low- resource speech-to-text translation. Proceedings of the 2019 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pages 58 68, Minneapolis, Minnesota. Association for Computational Linguistics.
7 Conclusion
In this paper, we propose FuseST, cross-modal model which supports three distinct input modali- ties for translation: speech, text and fused speech- text. We comprehensively analyze the modality gap between speech and text and, utilize multiple techniques to bridge the modality gap. We then fuse speech and text to improve speech translation. Experiments and analysis demonstrate the effec- tiveness of our proposed method.
Daniel Beck, Trevor Cohn, and Gholamreza Haffari. 2019. Neural speech translation using lattice trans- formations and graph networks. In Proceedings of the Thirteenth Workshop Graph-Bar Based Methods for Natural Language Processing (TexGrraphs-13) pages 26-31, Hong Kong. Association for Computa- tional Linguistics.
Alexandre Bérard, Laurent Besacier, Ali Can Ko- cabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 62288. IEEE Press.
Limitations
This work improves speech translation by fusing speech and text, but the model is far from being achieved for industrialgrade implementations. Al- though the ChatGPT and Whisper models exhibit superior speech-to-texy capabilities compared to our model, we maintain that fusing speech and text remains viable approach in the era of large- scale models. There are two significant limitations in this study that could be addressed in future re- search. First, our model still relies on an ASR system to transcribe speech into text, which does not address the issue of high latency in the cascade system. Second, our model needs labeled data for training. especially the <speech, transcript, target> pair. Speech data exceptionally scarce, and ob- taining speech data for many languages around the world is particularly challenging.
Alexandre Bérard, Olivier Pietquin, Laurent Besacier, and Christophe Servan. 2016. Listen and Trans- late: A Proof of Concept for End-to-Enn Speech-to- Text Translation. In NIPS Workshop end-to-end learning for speech and audio processing, Barcelona, Spain.
N. Bertoldi and M. Federico. 2005. A new decoder for spoken language translation based on confusion networks. In IEEE Workshop Automatic Speech Recognition and Understanding, 2005., pages 86 86-91.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Se- lected Topics in Signal Processing, 1605):550-1151

---

Qiao Cheng, Meiyuan Fan, Yaqian Han, Jin Huang, and Yitao Duan. 2019. Breaking the data barrier: To- wards robust speech translation via adversarial stabil-
Computational Linguisties: Human Language Tech- nologies, pages 1872-18888 Online. Association for Computational Linguistics.
ity training. In Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. Association for Computational Linguisties.
S. Indurthi, H Han, NK Lakumarapu, B Lee, I Chung, S Kim, and C Kim. 2020. Data efficient direct speech-to-text translation with modality agnostic meta-learning- In ICASSP 2020- 2020 IEEE Interna- rional Conference on Acoustics, Speech and Signal Processing (ICASSP).
Matti Di Gangi, Robert Enyedi, Alessandra Brusadin, and Marcello Federico. 2019a. Robust neural ma- chine translation for clean and noisy speech tran-
scripts. In Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. Association for Computational Linguisties.
Sathish Indurthi, Mohd Abbas Zaidi, Nikhil Ku- mar Lakumarapu, Beomseok Lee, Hyojung Han, Scokchan Ahn, Sangha Kim, Chanwoo Kim, and Inchul Hwang. 2021. Task aware multi-task learning for speech to text tasks. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7723-7727.
Mattia A. Di Gangi. Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019b. MuST-C: Multilingual Speech Translation Corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisticx: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012-2017, Min- neapolis, Minnesota. Association for Computational Linguistics.
Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. 2019. Leverag- ing weakly supervised data to improve end-to-end speech-to-text translation. In ICASSP 2019-2019 IEFE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 71847 IEEE
Long Duong, Antonios Anastasopoulos, David Chiang. Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 949-959, 959, San Diego, California. Association for Computational Linguistics.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2017. Structured-bused curriculum learning for end- to-end Interspeech 2017.
speech translation. In Proc.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79-86.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022, STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 7050-7062, Dublin, Ireland. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguisties.
Marco Gaido, Mattia A. Di Gangi, Matteo Negri, and Marco Turchi. 2020. End-to-end speech-tran -lattioooon with knowledge distillation: FBK @ IWSLT2020. In Proceedings of the 17th International Conference on
Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2021. Cascaded models with cyclic feedback for direct speech translation. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7508-7512.
Spoken Laguage Translation, pages 80-88, Online. Association for Computational Linguisties.
Mattia A. Di Gangi, Mattco Negri, and Marco Turchi. 2019. Adapting Transformer End-to-End Spoken Language Translation. Proc. Interspeech 2019, pages 1133-1137
Hang Le, Juan Pino, Changhan Wang. Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In Proceed- ings of the 28th International Conference on Com- putaaiional Linguistics, pages 3520-3533, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text translation. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 2214-2225.
Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021. Source and target bidirectional knowledge distillation for end-to-end speech trans- lation. Proceedings of the 2021 Conference of the North American Chapter of the Association for
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022. On vi- sion features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1:

---

Long Papers), pages 6327-6337, Dublin, Ireland. As- sociation for Computational Linguistics.
Juan Pino, Qiantong Xu, Xutai Ma. Mohammad Javad Dousti, and Yun Tang. 2020. Self- Trainin for End- to-End Speech Translation. In Proc. Interspeech 2020, pages 1476-1480.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021. UNIMO: Towards unified-modal understanding and generation via cross- cross-mode modal contrastive learning. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational I Linguistics and the 11th International Joint Conference on Natural Language Processing |Volume I: Long Pupers), pages 2592- 2607. Online. Association for Computational Lin- guistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Elizabeth Salesky, Matthias Sperber, and Alexander Waibel, 2019. Fluent translations from disfluent speech in end-to-end speech translation. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies. Volume 1 (Long and Short Papers), pages 2786-2798 2792, Min- neapolis, Minnesota, Association for Computational Linguistics.
Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 145-
Pierre Lison and Jörg Tiedemann. 2016. OpenSub- titles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC"16), 16), pages 923-929 223-929, Portorož, Slovenia. European Language Resources Association (ELRA).
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29.
Matthias Sperber. Graham Neubig, Jan Nichues, and Alex Waibel. 2017. Neural tesque-to mod- els for uncertain inputs. In Proceedings of the 2017 Conference Empirical Methods in Natural Lan- guage Processing, pages 1380-1389, Copenhagen, Denmark. Association for Computational Linguis- tics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad. Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- fics, 8:726-742.
Matthias Sperber, Graham Neubig, Ngoc-Quan Pham, and Alex Waibel. 2019. self-aatentional models for lattice inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics. pages 1185- 1197, Florence, Italy. Associa- tion for Computational Linguistics.
Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengging Zong. 2019. End-to-Ennd Speech Translation with Knowledge Dis- tillation. In Proc. Interspeech 2019, pages 1128- 1132.
Yun Tang, Hongyu Gong. Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li. Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified speech-text pre-training for speech translation and recognition.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal Forced Aligner: Trainable Text-Speech Aligument Using Kaldi. In Proc. Interspeech 2017,
Proceedings
pages 498-502.
of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1488-1499, Dublin, Ireland. Association for Computational Linguistics
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudampur. 2015. Librispeech: An asr corpus based public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210.
Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 202 Improving speech transla- tion by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 4252-4261, Online. Association for Computational Linguistics.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk, and Quee V. Le. 2019. Specaugment: simple data augmentation method for automatic speech recognition. In Proc. Interspeech 2019.
Stephan Peitz, Simon Wiesler, Markus Nußbaum- Saobbaum-Thom, and Hermann Ney. 2012. Spoken language transla- tion using automatically transcribed text in training. In Proceedings of the 9th International Workshop on Spoken Language Translation: Papers, pages 276- 283, Hong Kong, Table of contents.
Yun Tang. Juan Pino, Changhan Wang, Xutai Ma. and Dmitrly Genzel. 2021b. A general multi-tast learn- ing framework to leverage text data for speech to text tasks. In ICASSP 2021 2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6209-6213.

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Pei Zhang, Niyu Ge, Boxing Chen, and Kai Fan. 2019. Lattice transformer for speech translation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6475- 6484, Florence, Italy. Association for Computational Linguisties.
Laura Cross Vila, Carlos Escolano. José A. R. Fonol- losa, and Marta Ruiz Costa-jussä. 2018. End-to- end speech translation with the transformer. In Iber- SPEECH Conference.
Jiawei Zhao, Wei Luo, Boxing Chen, and Andrew Gilman. 2021. Mutual-leamiig improves end-to- end speech translation. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 3989-3994, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hari Krishna Vydana, Martin Karafiát, Katerina Zmo- likova, Lukáš Burget, and Honza Černocky. 2021. Jointly trained transformers models for spoken lan- guage translation. ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7513-7517.
Renjie Zheng. Junkun Chen, Mingho Ma, and Liang Huang. 2021. Fused acoustic and text encoding for multimodal bilingual pretraining and speech transla- tion. In International Conference on Machine Learn- ing, pages 12736 12746. PMLR.
Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou, 2020a. Bridging the gap between pre- training and fine-tuning for end-to-end speech trans- lation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9161-9168.
Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020b. Curriculum pre-training for end-to-end speech translation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 3728-3738, Online. Asso- ciation for Computational Linguistics.
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and 7. Chen. 2017. Sequence-to-sequee mod- els can directly translate foreign speech.
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and Jingho Zhu. 2021. Stacked cooustic-and-textiee the pre-trained models into speech translation coders. Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference Natu- ral Language Processing (Volume Long Papers), pages 2619-2630, Online. Association for Computa- tional Linguistics.
encoding: Integrating
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to- end speech translation via cross-modal progressive training, In Proc. Interspeech 2021.
Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross- modal contrastive leamning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5099-511 5113, Seattle, United States. Association for Computational Linguistics.
Rong Ye, Chengqi Zhao, Tom Ko, Chutong Meng, Tao Wang, Mingxuan Wang, and Jun Cao. Gigast: A 10.000-hour pseudo speech translation corpus. In Proc. Interspeech 2023.


Reviews:
[{"review_id": "eeiprbWMlK", "reviewer": "Reviewer_xQGj", "summary": "", "questions": "1. Line 243, how do you determine the ASR errors? Does P^f indicate the source of the text or the correctness of the text. The text from ASR can also be correct.\n2. CTR is a bad abbreviation of cross-modal contrastive learning. It introduces confusion with the other two loss when reading the paper.\n3. Figure 1 is too small to read.\n4. Are the decoding two stages? From eq.5, the model needs text embedding to run the share encoder.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n### **Answer to \"Reasons To Reject\" :**\nRegarding the concerns about the novelty of our paper, the overall framework of most speech translation models maintains consistency due to its demonstrated effectiveness. This framework typically encompasses primary tasks (Speech Translation) and auxiliary tasks (Machine Translation, Automatic Speech Recognition, user-defined tasks etc.). Our approach aligns with this established framework; our paper emphasizes integrating speech and text to enhance translation performance. Furthermore, we aim to bridge the modality gap for a more seamless integration of speech and text. To achieve this, we propose a fused speech-text (FT: s + x → y) task and present several methods for cross-modal alignment. Through extensive experimentation, we validate the effectiveness of our approach and provide valuable insights into the fusion of speech and text. Notably, our model not only achieves a significant improvement of 1.1 BLEU in ST but also demonstrates a notable enhancement of 3.2 BLEU in MT.\n\n### **Answer to \"Questions For The Authors\" :**\n**Q1: Line 243, how do you determine the ASR errors? Does P^f indicate the source of the text or the correctness of the text. The text from ASR can also be correct.**\n\nA1: We apologize for the confusion in our previous statement. Here, \"P^f\" represents the transcript, denoting whether it is manually annotated or generated through ASR.\n\n**Q2: CTR is a bad abbreviation of cross-modal contrastive learning. It introduces confusion with the other two losses when reading the paper.**\n\nA2. We have indeed considered the issue of the abbreviation for Cross-modal Contrastive Learning. Ultimately, we opted to maintain consistency with prior methods in our abbreviation. However, this decision may potentially mislead readers. We will rectify this issue in the final version. We appreciate your valuable suggestion.\n\n**Q3: Figure 1 is too small to read.**\n\nA3. We will revise Figure 1 to ensure its clarity and legibility in the final version. We appreciate your valuable suggestion.\n\n**Q4: Are the decoding two stages? From eq.5, the model needs text embedding to run the share encoder.**\n\nA4. No. Our model simultaneously utilizes speech and text as inputs; however, we only decode once.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The strategy is simple and straightforward. \n2. The numerical results show significant improvements over the baseline.", "reasons_to_reject": "My major concern is the novelty of paper.  Similar ideas on architecture, contrastive learning and cross attention regularization have been proposed by several other papers, e.g. [1][2]. The design of the model is quite similar to [1] and [2] expect that the shared encoder takes concatenate input instead of separated input.\n\n[1] Unified Speech-Text Pre-training for Speech Translation and Recognition\n[2] WACO: Word-Aligned Contrastive Learning for Speech Translation", "questions_to_authors": "1. Line 243, how do you determine the ASR errors? Does P^f indicate the source of the text or the correctness of the text. The text from ASR can also be correct.\n2. CTR is a bad abbreviation of cross-modal contrastive learning. It introduces confusion with the other two loss when reading the paper.\n3. Figure 1 is too small to read.\n4. Are the decoding two stages? From eq.5, the model needs text embedding to run the share encoder.", "paper_topic_and_main_contributions": "The paper discussed the topic of speech and text joined training for speech translation model, aka fuse-speech-text training. The authors proposed hierarchical architecture to handle multi-modal input. First, a speech only and a text only encoder is applied separately on each modality to have the representations. Further, the representations are concatenated in the same sequence with a tag indicating the modality. A shared encoder is then applied on top of the joined sequence for training. Additionally, several losses are applied to the training, such as cross-model contrastive learning, cross attentive regularization and cross model regularization.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VKyTgnA6ZG", "reviewer": "Reviewer_dtrK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n### **Answer to \"Reasons To Reject\" :**\n**(A)** Writing: Below, we present some previously insufficiently elaborated details:\n- **Further clarification for Line 377:** The only distinction between our approach and the subsequent methods, namely SA-CTR, Codebook-Gumbel-Softmax, and Codebook-K-means, lies in the manner of integrating speech and text for translation purposes (FT: s + x → y). However, other aspects such as speech feature extraction, text encoding, and training loss remain consistent with our methods. For specific details regarding the discrepancies, please refer to Lines 379-392.\n- **Our key distinctions from related works are as follows:** our paper emphasizes integrating speech and text to enhance translation performance; we first incorporate speech and transcript for translation(FT: s + x → y); we propose several methods to bridge the modality gap to better integrate speech and text.\n- **List of papers and their citations:** In Table 2, we compared our method with cascade models and end-to-end baseline models including: Espnet (Inaguma et al., 2021), W2V2-Transformer (Fang et al., 2022), MTL (Tang et al., 2021b),FAT-ST (Zheng et al., 2021), JT-S-MT (Tang et al., 2021a), Chimera (Han et al., 2021), XSTNet (Ye et al., 2021), XSTNet (Ye et al., 2021), SATE (Xu et al., 2021), STEMM (Fang et al., 2022), TaskAware (Indurthi et al., 2021), STPT (Tang et al., 2022), ConST (Ye et al., 2022a).\n\nIn the forthcoming final version, an additional page of content will be included specifically to provide comprehensive elucidation on those aspects that were previously unclear.\n\n\n\n\n**(B)** Our model incorporates 8 loss terms, comprising four fundamental tasks (ST, MT, FT, ASR) and four alignment losses. Previous methods primarily focused on improving the model's performance on the ST task, neglecting the efficacy of other translation tasks, resulting in fewer loss functions employed. In contrast, we aimed to enhance the performance of our model across all translation tasks (ST, MT, FT). Our model not only achieves a significant improvement of 1.1 BLEU in ST but also demonstrates a notable enhancement of 3.2 BLEU in MT.\n\n While we have conducted preliminary analyses on the impact of different loss functions on our model's performance through ablation study(Table 5), we have yet to extensively examine each loss function individually. By gradually removing each loss, Table 5 shows the improvements brought by each auxiliary training objective. Meanwhile, it has been demonstrated in previous studies that multi-task can enhance model performance, such as TaskAware (https://ieeexplore.ieee.org/document/9414703).\n\n\n### **Answer to \"Typos Grammar Style And Presentation Improvements\":**\n **Q1: A small comment, but the acronym FST can get confusing because many people associate it with finite-state transducer :)**\n\n**A1:**  We sincerely apologize for the confusion caused by the abbreviation FST. Henceforth, we will proceed to modify the model's name accordingly. We appreciate your valuable suggestion.\n\n**Q2: Equation 13 has 8 loss terms and is a lot to unpack. Although there is a lot of content in the paper, it might be nice to add some intuitions/analysis on which training objective or regularization technique provides for maximum benefits to bridge the cross-modal gap that motivates the construction of the FST model.**\n\n**A2:** As mentioned in the preceding \" Answer to \"Reasons To Reject\" (B)\", we have conducted preliminary analyses on the impact of different loss functions on our model's performance through ablation study(Table 5). Table 5 shows the improvements brought by each auxiliary training objective. In the final version, we plan to conduct experiments to assess the contribution of each loss function to alignment and provide detailed explanations in this regard.\n\n\n**Q3: L102-105 miss citations and don't explain these terms.**\n\n**A3:** We provide a detailed explanation and references for these terms in Section 3.4. We will add the missing information in the final version.\n\nCross-Attentive Regularization (CAR, Tang et al., 2021a) can increase the similarity among distinct modalities.\n\nCross-Modal Regularization (CMR): To bridge the modality gap in inference, we endeavor to optimize the congruity of the ultimate distributions of ST, MT and FT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A) Experiments and Code: The experimental section is thorough and the paper achieves good results with improvements on machine translation as well. The paper includes several baselines to compare against. The code is also released, good job with this!\n\nB) The figures in the paper are intuitive, color-coded and easy-to-understand. \n\nC) The model framework section (3.2) is clearly explained, and nicely builds up to Equation 13 which has 8 loss terms.\n\nD) The paper is also well-scoped in that the aim is clearly defined as speech translation, and the way to improve it is by improving cross-modal alignment. The paper does not have more or less to offer than what it claims which is a great thing.", "reasons_to_reject": "A) Writing: The paper might benefit with revisions in writing. For example, L377 mentions that the \"only difference between our approach and others is the specific method for fusing speech and text\". Accordingly, the related work section should talk about these works and discuss how the paper is different from them. In its current state, its mostly a list of papers and their citations. L096-101 are also unclear, since prompt tags are not introduced before this. \n\nB) The overall framework optimizes for 8 objectives and it is unclear how each help and why they are included. Although the paper has a lot of content to offer, a more intuitive approach for the inclusion of each regularization term even at the cost of a few experiments, would be beneficial, since a major contribution of the paper is the model. For example, stating something like: \"analysing previous model outputs or systems reveal certain limitations which can be overcome by these additional objectives ...\". \n\nThe complexity of the framework and non-clarity on each objective makes me hesitant on the excitement front, but I find the study to be thorough.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a model called Fuse-Speech-Text (FST) for end-to-end speech translation. The pre-training involves several objectives for unimodal learning and cross-modal alignment. The FST model is jointly trained to optimize for speech-translation (ST), machine translation (MT), fused speech-text translation (FT) and ASR. The model consists of : a) a speech-encoder (Wav2vec2.0 + 2CNN layers); b) a shared transformer encoder and shared decoder (for ST, MT and FT); and c) a non-shared transformer decoder (for the ASR task only). Further, to encourage cross-modal alignment, the model uses : a) cross-modal contrastive learning; b) cross-attentive regularization that has been used to increase similarity amongst distinct modalities and c) cross-modal regularization using knowledge distillation, where the FT task is the teacher and the ST and MT tasks are the student. Experiments are conducted on the MuST-C and GigaST datasets for ST and WMT datasets for MT; and gains are observed on both tasks: +1.1 BLEU over baselines for former and +3.2 BLEU for MT after training with the FST objectives (the FST model is first pre-trained for the MT task).", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) A small comment, but the acronym FST can get confusing because many people associate it with finite-state transducer :)\n\n2) Equation 13 has 8 loss terms and is a lot to unpack. Although there is a lot of content in the paper, it might be nice to add some intuitions/analysis on which training objective or regularization technique provides for maximum benefits to bridge the cross-modal gap that motivates the construction of the FST model. \n\n3) L102-105 miss citations and don't explain these terms."}}, {"review_id": "8RCgMDKDnh", "reviewer": "Reviewer_VGMC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n### **Answer to \"Reasons To Reject\" :**\n**Q1: The proposed method is complicated. It combines pretrained audio, text modular with 8 different losses. This make it hard to conclude the improvement is came from proposed model or just introduce many hyper-parameters to tune.**\n\n**A1:** Our model incorporates 8 loss terms, comprising four fundamental tasks (ST, MT, FT, ASR) and four alignment losses. Previous methods primarily focused on improving the model's performance on the ST task, neglecting the efficacy of other translation tasks, resulting in fewer loss functions employed. In contrast, we aimed to enhance the performance of our model across all translation tasks (ST, MT, FT). Our model not only achieves a significant improvement of 1.1 BLEU in ST but also demonstrates a notable enhancement of 3.2 BLEU in MT. Meanwhile, the parameter size of our model is nearly consistent with the models compared in Table 2. We will provide explicit annotations for the parameter sizes of each model in the final version.\n\nBased on equation 13, it is evident that our loss function incorporates solely two hyperparameters. Importantly, the configurations of these hyperparameters mirror those employed in previous methods, and we have not adjusted their values. Consequently, we attribute the observed improvements in model performance primarily to the effectiveness of our proposed model rather than any fine-tuning of hyperparameters.\n\n**Q2:The comparison missed one direction of fuse speech and text representation for speech translation. For example, MAESTRO: Matched Speech Text Representations through Modality Matching (https://arxiv.org/abs/2204.03409). That basically learn a joint encoder can encode both audio and text. Can the author comment the pros and cons of this direction versus proposed method?**\n\n**A2:** Our paper contains a comparison with the direction of fuse speech and text representation for speech translation. For instance, Table 2  includes Chimera(https://aclanthology.org/2021.findings-acl.195.pdf), STEMM(https://aclanthology.org/2022.acl-long.486.pdf), ConST(https://aclanthology.org/2022.naacl-main.376.pdf). \n\nWe have previously taken notice on the MAESTRO approach. However, we refrain from conducting a comparative analysis with MAESTRO for two primary reasons. Firstly, MAESTRO's experimentation was focused on CoVoST 2, and secondly, the absence of a code repository link in the corresponding paper hindered our ability to perform a comprehensive evaluation. Consequently, we could not establish a fair and equitable comparison with MAESTRO on the MuST-C dataset. Our approach differs from MAESTRO in the following aspects: our paper emphasizes integrating speech and text to enhance translation performance; to bridge the modality gap for a more seamless integration of speech and text, we propose FT(s + x → y) task and present several methods for cross-modal alignment.\n\nThe pros and cons of our model compared to the previous method are:\n\n**Pros:**\n\n(a) We propose a model that fuses speech and text to improve speech translation, which supports three distinct input modalities for translation: speech, text and fused speech-text.\n\n(b) We propose several methods to bridge the modality gap to integrate speech and text better.\n\n(c) Our model not only achieves a significant improvement of 1.1 BLEU in ST but also demonstrates a notable enhancement of 3.2 BLEU in MT\n\n**Cons:**\n\n(a) Our model is more complex than the previous method. However, this increased complexity is deliberate, as it aligns with our objective of enhancing the effectiveness of ST, MT and FT.\n\n**Q3: Presentation is hard to follow. For example, it mentioned all encoder decoder share the same param in section 3.2, but why figure 2 mark some of param shared? There also lots of acronym in the paper, that make it hard to follow.**\n\n**A3:** In section 3.2 Line 189-191, \"We share the Transformer encoder and Transformer decoder for ST, MT and FT.\" A shared transformer encoder and shared decoder (for ST, MT and FT); and a non-shared transformer decoder (for the ASR task only). In the final version, we will make a concerted effort to minimize the usage of acronyms in our paper and refine our writing accordingly. We appreciate your valuable suggestion.\n\n**Q4: It's hard to follow the comparison, since external modular been used. In table 2, when mark as external speech/text data been used, does it comparable with other cited paper? Maybe to make at least one data point that comparable, can the author provide their in-house cascade model, instead of cite other paper number?**\n\n\n**A4:** In Table 2, our comparisons with the cited methods are valid and based on comparable settings. We employ the same dataset as many referenced approaches, such as ConST and STEMM. How we compare our results with previous methods remains consistent, and you can verify this by referring to the experimental tables of ConST(https://aclanthology.org/2022.naacl-main.376.pdf) and STEMM(https://aclanthology.org/2022.acl-long.486.pdf).\n\nWe have also included the in-house cascade model in Table 3 and Table 4, where \"Pre-trained 6E6D MT\" corresponds to the cascaded model with the same parameters. Our approach exhibits significant superiority over the in-house cascaded model with the same parameters.\n\n### **Answer to \"Typos Grammar Style And Presentation Improvements\":**\n\nWe sincerely apologize for the confusion caused by the abbreviation FST. Henceforth, we will proceed to modify the model's name accordingly. We appreciate your valuable suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation is clear. For speech translation, speech contains some signal to help translation but if ASR doesn't capture it then error would propagate to the final translation quality.", "reasons_to_reject": "(1) The proposed method is complicated. It combines pretrained audio, text modular with 8 different losses. This make it hard to conclude the improvement is came from proposed model or just introduce many hyper-parameters to tune.\n\n(2) The comparison missed one direction of fuse speech and text representation for speech translation. For example, MAESTRO: Matched Speech Text Representations through Modality Matching (https://arxiv.org/abs/2204.03409). That basically learn a joint encoder can encode both audio and text. Can the author comment the pros and cons of this direction versus proposed method?\n\n(3) Presentation is hard to follow. For example, it mentioned all encoder decoder share the same param in section 3.2, but why figure 2 mark some of param shared? There also lots of acronym in the paper, that make it hard to follow.\n\n(4) It's hard to follow the comparison, since external modular been used. In table 2, when mark as external speech/text data been used, does it comparable with other cited paper? Maybe to make at least one data point that comparable, can the author provide their in-house cascade model, instead of cite other paper number?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper propose a framework, that take pre-trained representation for speech, text embedding (from either ASR transcript or groundtruth) and learn a joint representation between those by introduce different modality matching loss. Results show proposed method could improve speech translation on MuSTC (for some language pair) by more than  1 BLEU. \n\nThe contribution of the paper has 2 folds:\n(1) Propose a framework to use fused information of speech and text representation to help speech translation.\n(2) Tested many different modality matching function to help align audio and text embeddings.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please do not call it \"FST\". This is very confusing, because FST usually refer to finite state transducer for transcription task."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to improve speech translation by using a "fused" representation consisting of a combination of the original input speech and output produced by an ASR system. The reviewers found that the motivation for the approach was clear, with thorough experiments that supported the paper's claims (strong soundness scores). However the reviewers also all indicated that they are ambivalent about the excitement from this paper, in large part because of the complexity of the resulting system. There were also some issues with writing and clarity, but the authors largely addressed these in the rebuttal period.
Decision: Accept-Findings