Towards a Deep Understanding of Multilingual End-to-End Speech Translation
Authors: Haoran Sun
Xiaohu Zhao
Yikun Lei
shaolin Zhu
Deyi Xiong
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilingual Speech Translation
Multilinguality

Abstract: In this paper, we employ Singular Value Canonical Correlation Analysis (SVCCA) to analyze representations learnt in a multilingual end-to-end speech translation model trained over 22 languages. SVCCA enables us to estimate representational similarity across languages and layers, enhancing our understanding of the functionality of multilingual speech translation and its potential connection to multilingual neural machine translation.  The multilingual speech translation model is trained on the CoVoST 2 dataset in all possible directions, and we utilize LASER to extract parallel bitext data for SVCCA analysis. We derive three major findings from our analysis: (I) Linguistic similarity loses its efficacy in multilingual speech translation when the training data for a specific language is limited. (II) Enhanced encoder representations and well-aligned audio-text data significantly improve translation quality, surpassing the bilingual counterparts when the training data is not compromised. (III) The encoder representations of multilingual speech translation demonstrate superior performance in predicting phonetic features in linguistic typology prediction. With these findings, we propose that releasing the constraint of limited data for low-resource languages and subsequently combining them with linguistically related high-resource languages could offer a more effective approach for multilingual end-to-end speech translation.

Towards Deep Understanding of Multilingual End-to-End Speech Translation
Haoran Sun, Xiaohu Zhao, Yikun Lei, Shaolin Zhu and Deyi Xiong College of Intelligence and Computing, Tiajjin University, Tianjin, China (hrsun, zhaoxiaohu, yikunlei, zhushaolin, dyxiongJetju.edu.c
Abstract
error propagation and high latency during inference (Sperber and Paulik, 2020).
In this paper, we employ Singular Value Canon- ical Correlation Analysis (SVCCA) to analyze representations learnt in multilingual end-to- end speech translation model trained over 22 languages. SVCCA enables us to estimate rep- resentational similarity across languages and layers, enhancing our understanding of the functionality of multilingual speech transla- tion and its potential connection to multilingual neural machine translation. The multilingual speech translation model trained on the CoV- oST 2 dataset in all possible directions, and we utilize LASER to extract parallel bitext data for SVCCA analysis. We derive three major findings from our analysis: (l) Linguistic simi- larity loses its efficacy in multilingual speech translation when the training data for specific language is limited. (II) Enhanced encoder rep- resentations and well-aligned audio-text data significantly improve translation quality, sur- passing the bilingual counterparts when the training data is compromised. (III) The encoder representations of multilingual speech translation demonstrate superior performance in predicting phonetic features in linguistic ty- pology prediction. With these findings, we propose that releasing the constraint of lim- ited data for low-resource languages and sub- sequently combining them with linguistically related high-resource languages could offer more effective approach for multilingual end- to-end speech translation.
Recent years have also witnessed that multi- lingual neural machine translation (NMT) has at- tracted growing attention (Aharoni et al., 2019; Ari- vazhagan et al., 2019; Fan et al., 2021; Costa-jussà et al., 2022). One crucial characteristic of multi- lingual NMT is its knowledge transfer capability, where knowledge learnt from high-resource lan- guages is leveraged to improve translation quality of low-resource languages. Inspired by the success of multilingual NMT, methods used in multilingual NMT have been adapted to multilingual end-to-end speech translation. These include the combination of pre-trained models and fine-tuning (Li et al.. 2021) and the incorporation of adapter modules into the encoder/decoder layers (Le et al., 2021). However, due the limited availability of multi- lingual ST training data, multilingual E2E ST has been relatively understudied compared to bilingual E2E ST. This not only makes insights and findings into multilingual E2E ST rare, but also leaves many related questions (e.g., in which way methods in multilingual NMT can be successfully adapted to multilingual E2E ST) unanswered. In this paper, our key interest is an in-depth analysis into multi- lingual E2E ST. We hope findings from the analysis could shed light on its future development.
Previous studies on the interpretability of multi- lingual pre- preetrained models (Choenni and Shutova, 2022: Chang et al., 2022a) or multilingual NMT models (Kudugunta et al., 2019) have tried to make the black-box of multilingual models more interpretable by understanding the distribution of language representations learnt by these models. These works have provided valuable insights into multilinguality and have advanced the development of improved multilingual models. Singular Value Canonical Correlation Analysis (SVCCA; Raghu et al., 2017) commonly used approach for inves- tigating the representation similarity. enables us
1 Introduction
Recent years have witnessed the rapid development of end-to-end (E2E) speech-to-text translation (ST)
(Berard et al., 2016; Weiss et al., 2017), which has demonstrated remarkable performance and outper- formed conventional cascaded systems (Ye et al., 2021; Xu et al., 2021; Han et al., 2021; Ye et al., 2022). The primary advantage of end-to-end ST over cascaded ST is that the new architecture avoids *Corresponding author.

---

compare the representation similarity obtained through the same data points across different mod- els, layers and languages. SVCCA has been suc- cessfully applied to understand the representation of language models and multilingual NMT models.
2 Related Work
End-to-End ST End-to-end speech-to-text has draw much attention recently due to its lower la- tency and reduced error propagation compared to traditional cascaded systems (Berard al., 2016; Weiss et al., 2017). Recent approaches in this field have demonstrated remarkable performance on speech-to-text translation (Vila et al., 2018; Gangi et al.. 2019; Zhang et al., 2020a; Wang et al., 2020b,a; Zheng et al., 2021; Chen et al., 2020; Dong et al., 2021; Zhang al., 2022; Du et al., 2022; Weller et al., 2022; Alastruey et al., 2022; Lam et al., 2022: Lei al., 2023). How- ever, extending E2E ST to multilingual ST still remains under explored. The first attempt is to de- velop a multilingual end-to-end ST model based on an LSTM encoder-decoder architecture (Inaguma et al., 2019). Li et al. (2021) propose a combi- nation of Wav2Vec 2.0 (Baevski et al., 2020) and mBART (Liu et al., 2020a), fine-tuning only the layer normalization and multi-head attention layers. Le et al. (2021) insert an adapter layer on the top of each transformer encoder and decoder layer, where only the parameters of the inserted adapters are up- dated during the fine-tuning stage. Both Di Gangi et al. (2019) and Wang et al. (2021) introduce mul- tilingual speech translation models as baselines along with their proposed speech translation bench- mark datasets. Efforts that focus pre-training for multilingual speech translation, such XLSS- (Babu al., 2021), Maestro (Chen et al., 2022), SLA (Cheng et al., 2022), Whisper (Radford et al., 2022) and Google USM (Zhang et al., 2023), have been recently explored.
In this paper, we conduct a comprehensive analy- sis into multilingual end-to-end speech translation model trained on 22 languages, utilizing SVCCA tool. The research questions we seek to answer are as follows:
Does multilingual E2E ST demonstrate prop- erties similar those of multilingual NMT? Specifically, does it exhibit knowledge trans- fer across languages, benefiting low-resource languages while potentially affecting the per- formance of high-resource languages?
What is the distribution of learnt representa- tions like? Do languages from the same lan- guage family tend to cluster together based on their learnt sentence representations?
Can multilingual E2E ST make linguistic ty- pology predictions? Does it demonstrate su- perior performance in predicting phoneties- related features?
Answering these questions provides insights into multilingual E2E ST. Our findings follows:
The effectiveness of linguistic similarity di- minishes when there is insufficient training data for specific language, which may be at- tributed to the inadequacy of the training data to support a segguage-speciffie sub-space.
Enhanced encoder representations and aligned audio-text data significantly enhance transla- tion quality, surpassing the performance of bilingual models when the training data is not compromised.
Multilinguality and Interpretability The knowledge transfer capability is crucial aspect of multilingual NMT. It can boost the translation performance of low-resource languages on the one hand while potentially impacting the translation quality of high-resource languages on the other hand (Aharoni et al., 2019; Arivazhagan et al., 2019). Previous approaches on multilingual NMT have focused on designing efficient language- specific modules (Bapna and Firat, 2019; Philip et al., 2020; Zhang et al., 2020b; Zhu al., 2021; Zhang et al., 2021: Lin al., 2021) leveraging linguistic similarity among languages (Sachan and Neubig, 2018: Tan et al., 2019; Oncevay et al., 2020; Sun and Xiong, 2022; Baziotis et al., 2022) to strike a balance this trade-off. Understanding
The encoder representations of multilingual E2E ST exhibit superior performance in pre- dicting phonetic features in linguistic typol- ogy prediction.
Based on these observations, we conclude that, for low-resource languages, increasing the amount of parallel training data is more crucial than rely- ing solely the knowledge transfer ability of the multilingual end-to-end speech translation model, Additionally, building a high-quality language- specific sub-space crucial for low-resource trans- lation quality.

---

the inner workings of multilingual models remains an intriguing question (Conneau et al., 2020; Rama et al., 2020; Liang et al., 2021). Singular Value Canonical Correlation Analysis (SVCCA; Raghu et al., 2017) has been used to quantify the similarity between sets of representations in various language-related models, including language models (Saphra and Lopez, 2019), mul- tilingual language models (Chang et al., 2022b), multilingual NMT models (Kudugunta al., 2019; Oncevay et al., 2020) and end-to-end ASR models (Ollerenshaw et al., 2022). Our paper is most similar to the study conducted by Kudugunta et al. (2019). as they utilize SVCCA to analyze the distribution of languages in multilingual NMT with sentence-level representations. They achieve insights into language similarity, which facilitate succeeding studies on multilingual NMT. Wang et al. (2023) use t-SNE (Van der Maaten and Hinton, 2008) to analyze the representations learnt by Maestro (Chen et al., 2022), but their interest is in the impact of alignment between speech and text modality on speech translation, while we focus on the multilinguality analysis in multilingual E2E ST.
confounding factors (Kudugunta et al., 2019). In our study, we mitigate this issue by selecting set of semantically similar sentences for each pair of languages. To accomplish that, we utilize LASER? (Schwenk and Douze, 2017; Heffernan et al., 2022) to mine parallel sentences for each pair of given languages as evaluation datasets to measure simi- larity across different languages. More details are provided in Appendix A.
Analysis experiments are conducted on three directions: En-*X, X->En and X-*X. We tok- enize the translated texts using a jointly Icarnt un- igram Sentencepiece model (Kudo and Richard- son, 2018) with a vocabulary size of 10K for each directions. As for the audio data, we extract 80- dimensional log mel-scale filter bank features (win- dows with 25ms size and 10ms shift).
To train the multilingual E2E ST models, we first train separate multilingual ASR models for each translation directions. We then use the trained mul- tilingual ASR encoder to initialize the encoder of the multilingual ST models. Following Arivazha- gan et al. (2019), we use the temperature-basse sampling method during training X-+En and X-X models with temperature value o 5 to alle- viate the heavy imbalance between language pairs.
3 Empirical Anlysis Setup
As for the bilingual ST models, we adopt the settings used by Wang et al. (2021).
3.1 Data and Model
For evaluating, we report case-sensitive detok- enized BLEU using SacreBLEU (Post, 2018) ex- cept for English-Chinese and English-Japaness where we use the tokenizer provided by Sacre- BLEU (zh for Chinese and ja-mecab for Japanese). All models implemented with the Fairseq toolk- its (Ott al., 2019).
We study multilingual end-to-end speech transla- tion using the CoVoST 2 dataset (Wang et al.. 2021), which is an Eaglish-centric dataset that supports translation from English to 15 languages (En- and translation from 21 languages to En- glish (X Enn. Among the X >En directions, only 4 languages have more than 100 hours of train- ing data, while the remaining 17 languages have limited training resource, with less than 50 hours available. We categorize the languages in X En directions into high/miidlow-resouuce languages according to the amount of training data available for them. Specifically, the high-resource languages include French, German, Spanish and Catalan, the mid-resource languages contain Persian, Italian, Russian, Portuguese, Chinese and Dutch, the re- maining languages are considered as low-resource languages.
More details are provided in Appendix B.
3.2 SVCCA
We employ Singular Value Canonical Correlation Analysis (SVCCA; Raghu al., 2017) for our anal- ysis. SVCCA is method that allows us to compare the correlation between two vector representations. It is invariant to affine transformations and fast to compute. To apply SVCCA, we consider a set of data points containing N examples. The repre- sentation of a layer can be regarded the hidden states of the layer of these N data points. Let 4 E Nx and l2 Nx denote the repre- sentations of two layers, where D1 and D2 are the
In order to accurately calculate similarity be- tween languages based on sentence-level represen- tations, it is crucial to minimize the impact of dif- ferent meanings of sentences, which may introduce
Phttps:/ thgps. wtt.:/ccccccccaeeeeeee htt./aacbbokkresearch///555 2 Phttps://githuu ctm htmm/googlee/eeetenppiccc "https: thwb. http:/hww.....aaaeooreeerraraaaa

---

LANGs
Hours
Bi (X -Ea)))
X-
X-xX En)
ST (En +X)
En-X
X-xX (En-xX)
264
26.47
26.67
27.99
de
184
17.69
18.06
20.14
16.03
19.42
17.99
136
19.28
23.04
24.33
21.64
25.06
23.71
113
23.13
27.46
29.10
3.83
3.27
3.17
12.78
16.50
15.69
11.19
20.30
20.87
14.77
17.39
15.97
pt
10
6.13
10.16
8.23
10
5.68
6.37
7.41
23.67
29.63
28.53
3.04
3.65
5.32
3.51
3.57
3,42
10.09
12.50
11.31
0.47
0.95
0.89
15.81
14.31
mn
0.22
0.36
0.20
9.43
11.78
10.84
4.31
2.29
1.08
12.22
14.14
12.94
cy
2.56
2.82
2.74
23.88
26.32
25.19
2,51
1,93
1.03
13.10
15.42
14.03
2.97
3.39
1.76
15.97
18.92
16.97
SV
3.24
1.38
1.14
21.77
25.07
23.58
ta
0.31
0.09
0.12
10.92
13.63
12.73
id
2.39
0.87
0.24
20.24
23.55
23.14
ja
1.70
1.36
0.24
20.73
25.23
24.17
avg
7.40
10.33
10.23
16.36
19.53
18.34
Table I: Analysis results on the CoVoST dataset. We compare results of multilingual end-to-end speech translation trained on three different translation directions. Hours denote the total number of hours of training audio data for the language the source side. Bi ST is the bilingual end-to-end speech translation model.
dimensions of the layers corresponding to I and 22, respectively. SVCCA proceeds as follows:
Main Results
We present the analysis results on the CoVoST 2 dataset in Table Surprisingly, the multilingual E2E ST model exhibits distinct pattern compared to the multilingual NMT model. In the X->En translation direction, which includes high/mid/low- resource source languages, the multilingual ST model does not demonstrate the same knowledge transfer ability observed in the multilingual NMT model. In general, the low-resource languages do not benefit from the high-resource languages and continue to exhibit low translation quality, even when the low-resource language is linguistically related to a high-resource language. For instance, Swedish (sv), which is from the Germanic language branch of Indo-European language family, shares linguistic similarities to German (de) and Dutch (nl), both of which are Germanic languages and considered high/mid-resource languages. However, Swedish does not benefit from German and still exhibits poor translation quality.
1. Perform Singular Value Decomposition (SVD) on l and 12 to get sub-spaces & C l1, 1/2 C l2 which comprise of the most impor- tant directions of the original and l2, where IR Nx We retain enough dimensions to keep 99% of the variance the data.
L E RNNI%,
2. Use Canonical Correlation Analysis (CCA) to project I and i, onto shared sub- space, i.e., computing i, Wxlm, is Wyl' maximize the correlations corrs (pi,-.. (/)) between the new sub- spaces.
We follow Raghu et al. (2017) to use the mean of the correlations:
min(D) (D,, D32) Ee
1
p
(1)
Following Kudugunta et al. (2019), we adopt the sequence-base based SVCCA which involves per- forming SVCCA the output of the layer and averaging the results over sequence time-steps. This sequence-based SVCCA can compare the un- aligned sequences across different languages in more suitable way than the original token-level strategy.
However, on the other hand, all of the high- resource languages gain significant improvements in translation quality, even surpassing the perfor- mance of the corresponding bilingual ST mod- els. This phenomenon is particularly evident in the En-+X translation direction, where all the lan- guages can considered high- high-resource. The mul-

---

tilingual system outperforms the bilingual systems by an average BLEU of 3.17. Similarly, in the X->En translation direction, the high-resource lan- guages (French, German, Catalan, and Spanish) also demonstrate improved performance with an average BLEU of 23.81, compared to the average BLEU of 21.64 achieved by the bilingual systems.
in the X-+En direction, which have sufficient train- ing data. However, the languages in the X-En direction with only few hours of audio data still achieve low translation quality.
These results provide support for the observation that the multilingual ST model exhibits different pattern compared to the multilingual NMT model. The multilingual ST model leverages better audio representations and alignments between audios and texts to achieve improved translation quality for languages that have sufficient parallel training data. However, the linguistic aspect loses its effective- ness in the multilingual ST model, as low-resource languages constrained by the limited amount of training data and do not benefit from their linguisti- cally related high-resource languages. This finding aligns with what we observe in Section 5.1. Over- all, the amount of parallel training data is more crucial than the linguistic relatedness of languages for the performance of the multilingual ST model.
From these comparisons, we can draw the fol- lowing conclusions: (I) Improved audio represen- tations enhance the information encoding capabil- ity of the encoder component, resulting in better translation quality for high/mid-resource languages, the X En direction, the multilingual training audio data enhances the encoder's ability by pro- viding a more suitable encoding space, thereby boosting the performance for high-resouree lan- guages. (II) The introduction of well-aligned audio- text data also benefits speech translation quality. In the En +X direction, although the total amount of audio data remains the same in the bilingual set- ting. aligning this data with multilingual text helps the model learn better alignment between audio and text. This phenomenon has been extensively stud- ied in the context of end-to-end bilingual speech translation, where it is referred to as the modality gap (Liu et al., 2020b; Han et al., 2021; Ye et al., 2022; Fang et al., 2022). (III) The performance of low-resource languages is still limited by the avail- ability of aligned audio-text data. This scarcity of data hampers the model's ability to capture the nu- ances and specific characteristics of low low-resource languages, leading to lower translation quality com- pared to high/nid-resource languages.
5 Language Similarity
To thoroughly examine the impact of language sim- ilarity on the multilingual E2E ST model trained in the X-* direction, we conducted SVCCA analy- sis on our LASER-mined evaluation datasets with semantically similar sentences, as mentioned in Section 3.1. It's worth noting that the LASER- mined evaluation datasets are created specifically for the X-En direction, the Co CoSo 2 dataset already provides multi-way way way-parallel test set for the En X direction, where the English audio re- mains the same across all languages. The SVCCA scores are computed based on layer-wise hidden states of the encoder in the X-+En direction and the decoder in the En- direction. This analy- sis allows us to examine the similarity between languages across different layers of the model.
For the X-*X model trained on all translation directions, it still surpasses bilingual systems for all high-resource languages. However, it falls behind the En-+X model when translating from English other languages. We conjecture that this perfor- mance gap is due to the limited model capacity, where related parameters are affected by interfer- ence from mid- and low-resource languages in the X-*En direction. Interestingly, the performance on the high-resource languages in the X-> di- rection outperforms the model trained only in the X En direction. This behavior is reminiscent of the behavior observed in multilingual NMT models, where the performance of high-resource languages is normally sacrificed to benefit low-resource lan- guages. In this case, the high-resource languages correspond to the languages in the En-+X direction, while the low-resource languages are the languages
In the upcoming sections, our discussion will center around language similarity from multiple perspectives. We consider language family as an important factor for analysis, e.g., Indo-European, which encompasses a significant portion of the lan- guages in our dataset. We also examine language branch, like Romance and Germanic. Addition- ally, the amount of available training data for each language is taken into account during our analysis. Finally, we delve into patterns related to the writ- ing systems of languages on the decoder side for translation generation. By exploring these aspects,

---

Ext Seali
Exn Sunc
Keyll Droella
Sceek
Lang
182
(a) Encoder layer O
(b) Encoder layer 11
SVCCA scores between the representations (encoder layer and encoder layer 11) X-*En language pairs (i.e., pairs of X), which is calculated on our LASER-mined evaluation datasets. Red cells indicate that the two languages are more related to each other (higher SVCCA scores) and blue cells indicate that the two languages are less related (lower SVCCA scores). Best viewed in color.
Figure 1:
we aim to gain insights into how linguistic factors, training data availability, and written script charac- teristics influence the performance of multilingual E2E ST.
Layer 13
5.1
SVCCA Scores of the Source Languages in X-+En Translation
We first visualize SVCCA scores of language pairs in Figure 1. We only demonstrate the results of the first encoder layer (encoder layer 0 in Figure (())) and the last encoder layer (encoder layer 11 in Figure (b)) to show a comparison between them, the results of other layers provided in Appendix D.I.
BA52
L0t
Figure 2: Representations of encoder layer 11 projected onto linear sub-space with three LDA axes. The pro- jection on the top right corner visualizes the sub-space with LDA axis and The projection on the bottom right corner visualizes the sub- sub-space with LDA axis and 2. Red/blte/yellow-oolod resource languages, respectively.
From the comparison between the results of en- coder layer 0 and encoder layer 11, we observe decrease in the mean SVCCA scores (0.89 Vs. 0.86) for each pair of languages. This decrease sug- gests that languages tend utilize their language- specific parameters as layers go deep. In the first layer (layer 0), languages exhibit more general rep- resentations, as evidenced by the relatively close SVCCA scores of different language pairs. How- ever, as we go deep to the final layer (layer 11), we observe clear tendency for SVCCA scores to converge in terms of language branches. Specifi- cally, the Romance languages demonstrate higher SVCCA scores among themselves compared to languages from other language branches, indicat- ing a stronger similarity in their representations.
items are high/mid/low-
Similarly, the Germanic languages exhibit sim- ilar pattern, with higher SVCCA scores observed within this language branch.
We observe an interesting phenomenon among the mid-resource languages (Persian, Italian, Rus- sian, Portuguese and Dutch) that are all from the Indo-Europes language family. In Figure 1(b), can see that Portuguese, Italian, Russian and Dutch exhibit higher SVCCA scores with other languages in the Indo-European language family. These languages also demonstrate better transla- tion quality in the multilingual ST model compared

---

Deshpande, 2021; Cabello Piqueras and Søgaard. 2022).
With the help of LDA, we can illustrate the rea- sons why Persian differs from other Indo-European languages and does not benefit from linguistic sim- ilarities. In Figure we present a simplified ver- sion of Figure 2. which contains only six languages. From Figure 3. we can discern that Persian occu- pies a distinct position along the LDA axis com- pared to the other Indo-Furopean mid-resource languages. Similarly, Chinese exhibits a similar pattern, primarily due to its affiliation with Sino- Tibetan languages. We believe that this variance in distinct
-39
37
52
Figure 3: Representations of encoder layer II projected onto a linear sub-space with three IDA axes, exclusively comprising mid-resource languages. The projection on the top right corner visualizes the sub-space with LDA axis and 1. The projection on the bottom right corner visualizes the sub-space with LDA axis 1 and 2.
the LDA axis can also be interpreted language-specific subspace, which contributes the challenges in transferring translation abilities to Persian languages.
their bilingual counterparts. However, Persian is not similar to other languages, as indicated by lower SVCCA scores across all languages in the mndo-European language family. This low similar- ity leads to poor translation quality in the multilin- gual ST model, despite that Persian has 49 hours of training data available.
Based on the analysis conducted on the encoder side, we identify two factors that affect transla- tion quality: language similarity and the amount of training data. Language similarity impacts the level of knowledge transfer across languages which con- tributes to the development of high-quality repre- sentations. And the quantity of training data plays a crucial role in establishing language-speciftt sub- spaces, which is also vital for translation quality.
In the case of low-resource languages, we have observed that they tend to exhibit higher SVCCA scores with most languages, indicating similarity in their representations. However, despite this simi- larity, these low-resource languages still suffer low translation quality in the multilingual ST model.
5.2 SVCCA Scores of the Target Languages in En- Translation
We visualize the SVCCA scores on the decoder side for 15 languages Figure 4. We present the results of the first decoder layer (decoder layer 0) and the last decoder layer (decoder layer 5) to com- pare their SVCCA scores. The results of other layers are displayed in Appendix D.2.
In order to thoroughly examine multilingual ST for low-resource languages, we utilize the linear discriminant analysis (LDA) to explicitly identify Lunguage-speciffe sub-spaces, as described in previ- ous works (Liang et al., 2021; Chang et al., 2022b). The results are visualized in Figure 2. It can be ob- served from Figure 2 that high- and mid-resource languages exhibit distinet language-specific sub- spaces along different LDA axes, whereas low- resource languages remain within a common sub- space without a prominent language-specific dis- tribution. Due to the lack of sufficient training data, low-resource languages are unable to develop their own language-specific sub-spaces within the encoder. Instead, they mainly locate on a shared sub-space that is common to all languages. The ab- sence of saneufce sub-spaces undermines translation performance for these low-resource lan- guages. This finding aligns with the findings of previous studies on multilingualiiity with a fair- ness lens (Wu and Dredze, 2020; Choudhury and
When comparing the SVCCA scores of decoder layer 5 with layer O, we observe an increase in the mean SVCCA scores. This gradual increase indi- cates that, during the translation process from En- glish to other languages, different languages tend to utilize more general sub-space at the top decoder layer.
Given that the modality of the decoder side is text, we also observe a pattern related to the writing system correlation in the SVCCA score results. In Figure 4(b), we observe higher SVCCA scores for Chinese and Japanese compared other languages. Although Chinese and Japanese have distinct writ- ing systems, Japanese borrow characters from Chi- nese (c.g. Kanji). We also notice a similar pattern between Arabic and Persian, both of which use the

---

Sick
Statc 195
SeaTRors
13
0.50
151
1000
Lenguage 2
Largguage 3
(a) Decoder layer 0
(b) Decoder layer 5
Figure 4: SVCCA scores between the representations (decoder layer and decoder layer 5) of En-x language pairs. Red cells indicate that the two languages are more related to each other (higher SVCCA scores) while blue cells indicate that the two languages are less related (lower SVCCA scores). Best viewed in color.
Arabic alphabet. This may explain that the SVCCA scores for Arabic and Persian are higher compared those with other languages. This finding also in line with the finding of Kudugunta et al. (2019).
N:NN
Feature
Logistic
5
Max
76.13
78.74
80.60
81.05
81,05
78.62
Phonology
87.74
90.31
91.64
91.20
91.64
87.96
Inventory
83,87
87.88
88.20
88.53
88.53
85,78
6 Linguistic Typology
Table 2: Linguistic typology test accuracy on syntax, phonology and phonetic inventory features using the language representations learnt by the encoder. k de- notes the number of nearest neighbors in k-NN. Max denotes the maximum accuracy when & varies in 1, 5, 7.
We conduct experiments on the linguistic typology prediction for our multilingual end-to-end speech translation model trained on the X-* direction on the VoST 2 dataset.
Dataset We employ typological features from URIEL typological database (Littell et al., 2017) for experiments. URIEL is a typological com- pendium which accommodates diverse linguistic re- sources from several typological databases such as WALS (Dryer and Haspelmath, 2013), PHOIBLE (Moran and McCloy, 2019), Ethnology (Lewis et al., 2015) and Glottolog (Hammarström et al., 2021). We used lang2vec library query URIEL database which provides uniform interface ac- cess various linguistic features. We mainly use syntax, phonology and phonetic inventory typolog- ical features in our work.
obtained from the encoder for all samples on the Co test set. These representations serve as vector representations of each language, which we employ for probing the typological features. In the k-NN approach, we set & as odd numbers and vary l in (1,3, 5, 7]. We leave one language out and take samples of the remaining languages as training data to predict the linguistic typology feature. This step is repeated for all languages, and we report the average prediction accuracy across all languages.
Results We present prediction results in Table 2. A notable observation is that the vector repre- sentations derived from our multilingual E2E ST model demonstrate higher accuracy in predicting phonology features and phonetic inventory features compared to syntax features. This finding aligns with expectations since the vector representations are obtained through the encoder, which primarily processes the audio modality and is more likely
We adopt two methods commonly used in previous studies: k-nearest neighbors approach (k-NN) and logistic regres- sion (Malaviya et al., 2017: Oncevay et al., 2020). We utilize the averaged sentence representations
Prediction Methods
hhttps:/ //ww:w///wwwwe cs. cmu. eelt//dtootenss/rooeetcs///
project

---

capture phonological and phonetic information. It suggests that the multilingual E2E ST model effectively encodes and represents phonological as- pects of languages in its sentence representations. However, the lower accuracy in predicting syntax features indicates that it may be difficult for the model to capture syntax-related information solely from the audio modality.
on phonetic-related features compared syntax features.
Limitations
The main limitations of this study lie two aspects: the quality of the LASER-ber evaluation datasets and the analysis perspective.
In our approach, we set the threshold of 1.05 to determine semantic equivalence, which lower than the standard threshold of 1.2 typically used to ensure high-quality aligned bitext. Consequently, there is possibility that some of the sentences iden- tified parallel text may possess different mean- ings, potentially introducing a confounding factor that could impact the reliability of our analysis re- sults based on SVCCA, since the scarcity audio data is much more serious than text data.
7 Discussion
Our study reveals several key observations that might be of interest researchers and practitioners working in multilingual E2E ST.
Our analysis identifies two factors that affect multilingual speech translation quality: linguistic similarity and the amount of training data. The availability of an adequate amount of training data is crucial for the model to learn language-speciff sub-spaces, particularly for low-resource languages. Linguistic similarity fer across languages, especially when low-resource languages are not constrained by limited data. Thus, addressing data scarcity in the multilingual setting and improving the representation space for individual languages should be explored to enhance multilingual translation quality. In another perspec- tive, learning high-quality representation space for a low-resource language with limited data is also a path to achieve linguistic fairness of multi- lingual E2E ST.
Regarding the analysis perspective, our investiga- tion on multilingual end-to-end speech translation focuses on the linguistic aspect. However, we have not conducted an analysis of the model from the phonological perspective, which has the potential to offer additional insights into multilingual E2E ST. Unfortunately, due to the absence of reliable standard mapping between audio and phoneme, conducting a comprehensive analysis, particularly for low-resource languages, poses significant chal- lenges.
facilitate knowledge trans-
is widely recognized that enhancing audio rep- resentations can improve the performance of the acoustic model, and this principle applics to multi- lingual/bilingual E2E ST models as well.
Ethics Statement
This study presents an analysis of multilingual end-to-end speech translation, primarily based on the Co CoVT 2 dataset commonly used in speech translation. Additionally, incorporate the Com- mon Voice project to construct our evaluation sets, which is released under the Creative Commons Attribution Share-Alike 3.0 Unported license (CC BY-SA 3.0).
8 Conclusion
In this study, we have analyzed language repre- sentational similarity learnt by multilingual end- to-end speech translation model trained on 22 lan- guages via SVCCA. Through our analysis, we have findings that shed light on the performance of such models. We observe that the amount of available data plays significant role in limiting the effective- ness of knowledge transfer across languages in mul- tilingual speech translation. Using SVCCA to eval- uate the similarity across languages, observe clustering effect in terms of language branch, in- dicating aggregation of linguistic features within language families. We also use learnt language representations to probe linguistic typology and find that the multilingual ST model performs better
Acknowledgments
The present research was supported by the Key Research and Development Program of Yunnan Province (No. 202203AA080004) and the Nat- ural Science Foundation of Xinjiang Uygur Au- tonomous Region (No. 2022D01D43). We would like to thank the anonymous reviewers for their insightful comments.

---

References
Arab Emirates, December 7-11, 2022, pages 1170- 1185, Association for Computational Linguistics.
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa-
Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A fional Linguislics: Human Language Technologies. NAACI-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 3874- 3884, Association for Computational Linguistics.
proof of concept for end-to-end speech-to-text trans- lation. CoRR, abs/1612.01144,
Laura Cabello Piqueras and Anders Søgaard. 2022. Are pretrained multilingual models equally fair across languages? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3597-3605, Gyeongju, Republic of Korea. Interna- tional Committee on Computational Linguistics.
Belen Alastruey, Javier Ferrando, Gerard I. Gállego, and Marta R. Costa-jussà. 2022. On the locality of attention in direct speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics; Student Research Work- shop, pages 402-412, Dublin, Ireland. Association for Computational Linguistics.
Tyler A. Chang, Zhuowen Tu, and Benjamin Bergen. 2022a. The geometry of multilingual language model representations. In Proceedings of the 2022 Confer- ence on Empirical Methods Natural Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 119-136. As- sociation for Computational Linguistics.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively- multilinguua speech corpus. In Proceedings of The 12th Language Resources and Evaluation Confer- ence, LREC 2020, Marseille, France, May 11-16, 2020. pages 4218-4222. European Language Re- sources Association.
Tyler A. Chang, Zhuowen Tu, and Benjamin Bergen. 2022b. The geometry of multilingual language model representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11. 2022, pages 119_ 136. Association for Computational Linguistics.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. CoRR, hss/1907.05019
Junkun Chen, Mingbo Ma, Renjie Zheng, and Liang Huang. 2020. MAM: masked acoustic modeling for end-to-end speech-to-text translation. CoRR. abs/2010.11445.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf. Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021. XLS-R: Self-supervised Cross-lingual Speech Repre- sentation Learning at Scale. ArXiv:211 11..096 [cs, eess].
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhu- vana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen. 2022, MAESTRO, matched speech text representations through modality matching. In Interspeech 2022, 23rd Annual Conference of the In- ternational Speech Communication Association, In- cheon, Korea, 18-22 September 2022, pages 4093- 4097. ISCA.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations, In Advances in Neural Information Processing Sys- tems, volume 33, pages 12449-12460. Curran Asso- ciates, Inc.
Yong Cheng, Yu Zhang, Melvin Johnson, Wolf- gang Macherey, and Ankur Bapna. 2022. (2)SSA)A: Multitask, Multilingual Speech and Language Models. ArX1v:2212.09553 [cs, cess).
Ankur Bapna and Orhan Firat. 2019. Simple, scal- able adaptation for neural machine translation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing, EMNLP-IJCNU 2019, Hong Kong, China, November 3-7, 2019, pages 1538-1548. Association for Computational Linguistics.
Rochelle Choenni and Ekaterina Shutova. 2022. Inves- tigating language relationships in multilingual sen- tence encoders through the lens of linguistic typology. Comput. Linguistics, 48(3):635-672.
Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair are multilingual pre-trained lan- guage models? In Thirty-Fijh Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, TAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 127 12-11887 AAAI Press.
Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. 2022. Multilingual machine transla- tion with hyper-adapters. In Proceedings of the 2022 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2022, Abu Dhabi, United

---

Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Emerging cross- lingual structure in pretrained language models. In Proceedings of the 58th Anmual Meeting of the As- sociation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1022260344 Associa- tion for Computational Linguisties.
Association for Computational Linguistics ( (Volume I: Long Papers), pages 7050-7062, Dublin, Ireland. Association for Computational Linguisties.
Mattia Antonino Di Gangi. Matteo Negri, and Marco Turchi. 2019. Adapting transformer end-to-end spoken language translation. In Interspeech 2019, 20th Anmual Conference of the International Speech Communication Association, Graz. Austria, 15-19 September 2019, pages 1133-1137, ISCA.
Marta Costa-jussä, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffer- nan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman. Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews. Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rop- ers. Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scal- ing human-centered machine translation. CoRR. abs/2207.04677
Harald Hammarström, Robert Forkel, Martin Haspel- math, and Sebastian Bank. 2021. Glottolog 4.5.
Chi Han, Mingxuan Wang. Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-tex, translation. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages Online. Association for Computational Linguistics.
Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence repre- sentations for low-resource languages. In Findings of the Association for Computational Linguisties: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2101-2112. Associa- tion for Computational Linguistics.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. Must-c: a multilingual speech translation corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume I (Long and Short Papers), pages 2012-2017, Min- neapolis, Minnesota. Association for Computational Linguistics.
Hirofumi Inaguma, Kevin Duh, Taisuya Kawahara, and Shinji Watanabe. 2019. Multilingual end-to-end speech translation. In JEEE Automatic Speech Recog- nition and Understanding Workshop, ASRU 2019, Singapore, December 14-18, 2019, pages 570-577. IEEE.
Qianqian Dong, Rong Ye, Mingxuan Wang. Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021. Listen, un- derstand and translate: Triple supervision decouples end-to-end speech-to-text translation. Proceedings of the AAAI Conference on Artificial Intelligence, 12749):2749912259 Number: 14.
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Tho- rat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017, Google's 's multilingual neural machine translation sys- tem: Enabling zero-shot translation. Trans. Assoc. Comput. Linguistics, 5:339-351.
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online (v2020.3). Zenodo.
Diederík P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015. San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Yichao Du. Zhirui Zhang, Weizhi Wang, Boxing Chen, Jun Xie, and Tong Xu. 2022. Regularizing end-to- end speech translation with triangular decomposition agreement. Proceedings of the AAAI Conference on Artificial Intelligence, 36( 16510):1059--10588 Num- ber: 10.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Angela Fan, Shruti Bhosale, Holger Schwenk. Zhiyi Ma, Ahmed El- Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishray Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky. Sergey Edunov, Michael Auli, and Ar- mand Joulin. 2021. Beyond english-centric multi- lingual machine translation. J. Mach. Learn. Res., 22:107:11107:48.
Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual nmt representations at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IICNLP), pages 1565-1575 Hong Kong. China. Association for Computational Linguistics.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the

---

Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2022. Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to- end speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 245-254. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a. Multilingual denoising pre-training for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:726-742.
Yuchen Liu, Junnan Zhu, Jijjun Zhang, and Chengqing Zong. 2020b. Bridging the modality gap for speech- to-text translation. CoRR, abs/2010. 14920.
Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2021. Lightweight adapter tuning for multilingual speech translation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 817-824, Online. Association for Computational Linguistics.
Chaitanya Malaviya, Graham Neubig, and Patrick Lit- tell. 2017. Learning language representations for typology prediction. In Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing, pages 2529-2535, Copenhagen, Den- mark. Association for Computational Linguistics.
Steven Moran and Daniel McCloy, editors. 2019. PHOIBLE 2.0. Max Planck Institute for the Science of Human History, Jena.
Yikun Lei. Zhengshan Xue, Xiaohu Zhao, Haoran Sun, Shaolin Zhu, Xiaodong Lin, and Deyi Xiong. 2023. CKDST: Comprchensively and effectively distill knowledge from machine translation end- to-end speech translation. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 3123-3137, Toronto, Canada. Association for Computational Linguistics.
Anna Ollerenshaw, Md Asif Jalal, and Thomas Hain, 2022. Probing statistical representations for end-to- end ASR. CoRR, abs/221 1.09993.
Arturo Oncevay, Barry Haddow, and Alexandra Birch. 2020. Bridging linguistic typology and multilingual machine translation with multi-view language repre- sentations. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2391-2406, Online. Association for Computational Linguistics.
M Paul Lewis, Gary F Simons, and Charles D Fennig. 2015, Ethnologue: languages of ecuador. SIL Inter- national, Dallas.
Xian Li, Changhan Wang. Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021. Multilingual speech trans- lation from efficient finetuning of pretrained models. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 827-838, Online. Association for Computational Linguistics.
Myle Oit, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2079 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2019, Minneapo- lis, MN, USA, June 2-7, 2019, Demonstrations, pages 48-53. Association for Computational Linguistics.
Sheng Liang, Philipp Dufter, and Hinrich Schütze. 2021. Locating language-specific information in contextu- alized embeddings. CoRR, 0hs/2/09.08044
Jerin Philip, Alexandre Berard, Matthias Gallé, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4465 4470, Online. Association for Computational Linguistics.
Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. Learning language specific sub-network for multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lànguistics and the 1 11th International Joint Conference on Natural Language Processing, ACL/JCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. pages 293-305. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 November 1, 2018, pages 186-191. Association for Computational Lin- guistics.
Patrick Littell, David R. Mortensen, Ke Lin, Kather- ine Kairis, Carlisle Turner, and Lori S. Levin. 2017 URIEL and lang2vec: Representing languages as ty- pological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, EACL 2017, Valnccia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 8-14 14. Associa- tion for Computational Linguistics.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak ArX:v:2212.04356 [cs, eess).
Supervision.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular vec- tor canonical correlation analysis for deep learning

---

dynamics and interpretability. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Newral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Taraka Rama, Lisa Beinborn, and Steffen Eger. 2020. Probing multilingual BERT for genetic and typologi- cal signals. In Proceedings of the 28th International Conference on Compulational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 1214-1228, International Committee on Computational Linguistics.
Laura Cross Vila, Carlos Escolano, José A. R. Fonollosa, and Marta R. Costa-jussà, 2018. End-to-end speech translation with the transformer. In Fourth Interna- tional Conference, IberSPEECH 2018, Barcelona, Spain, 21-23 November 2018, Proceedings, pages 60-63, ISCA.
Devendra Sachan and Graham Neubig. 2018. Parame- ter sharing methods for multilingual self-attentional translation models. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pages 261-271, Brussels, Belgium. Association for Computational Linguistics.
Changhan Wang. Yun Tang. Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq S2T: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference Natural Language Processing: System Demon- strations, pages 33-39, Suzhou, China. Association for Computational Linguistics.
Naomi Saphra and Adam Lopez. 2019. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-H MACL--HTTRE 2019, Minneapolis, MN, USA. June 2-7, 2019. Volume I (Long and Short Papers), pages 3257- 3267. Association for Computational Linguistics.
Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021. CoVoST 2 and massively multilingual speech translation. In Interspeech 2021, pages 2247-2251. ISCA.
Holger Schwenk and Matthijs Douze. 2017. Learn- ing joint multilingual sentence representations with neural machine translation. In Proceedings of the 2nd Workshop on Representation Learning for NL., Rep4NLP@@ ACL 2017, Vancouver, Canada, August 3, 2017, pages 157-167. Association for Computational Linguistics.
Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020b. Bridging the gap between pre- training and fine-tuning for end-to-end speech trans- lation. Proceedings of the AAAI Conference Arti- ficial Intelligence. 34(05):9161999) 1966... Number: 05.
Matthias Sperber and Matthias Paulik. 2020. Speech translation and the end-to-end promise: Taking stock of where we are. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020. Online, July 5-10, 2020, pages 9409-7421. Association for Computational Linguis- tics.
Gary Wang, Kyle Kastner, Ankur Bapna, Zhehuai Chen, Andrew Rosenberg, Bhuvana Ramabhadran, and Yu Zhang. 2023, Understanding shared speech-text representations. In ICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE
Haoran Sun and Deyi Xiong. 2022. Language branch gated multilingual neural machine transla- tion. In Proceedings of the 29th International Confer- ence Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 5046-5053. International Committee on Com- putational Linguistics.
Ron J. Weiss, Jan Chorowski. Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-seee to-eeqeence models can directly translate foreign speech. In In- terspeech 2017, 18th Annual Conference of the Inter- national Speech Communication Association, Stock- holm, Sweden, August 20-24, 2017, pages 2625-2629. ISCA.
Xu Tan, Jiale Chen, Di He, Yingee Xia, Tao Qin, and Tie- Yan Liu, 2019. Multilingual neural machine translation with language clustering. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNIP-IICNIIP), pages 963-973, Hong Kong, China. Association for Computational Lin- guistics.
Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, and Matthias Paulik. 2022. End-ta-end speech translation for code switched speech. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022. pages 1435-1448, Dublin, Ireland, Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Com- putational Linguistics.
Laurens Van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-sne. Journal of machine learning research, 911))

---

Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju. Tong Xiao, and Jingbo Zhu. 2021. Stacked acoustic-and-texxuall encoding: Integrating the pre-trained models into speech translation en- coders. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2619 (619-2630,, Online. Association for Computa- tional Linguisties.
speech recognition beyond 100 languages. CORR. abs/2303.01037.
Renjie Zheng, Junkun Chen, Mingho Ma, and Liang Huang. 2021. Fused acoustic and text encoding for multimodal bilingual pretraining and speech transla- tion. In Proceedings of the 38th International Con- ference Machine Learning. ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12736-12746. PMLR.
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to- end speech tramslation via cross-modal progressive trainig. In Interspeech 2021, 22nd Annual Con- ference of the International Speech Communication Association, Brno, Czechia, 30 August 3 September 2021, pages 2267-2271. ISCA.
Yaoming Zhu, Jiangtao Feng. Chenggi Zhao, Mingx- uan Wang, and Lei Li. 2021. inteeer-nteerference adapter for multilingual machine translation. In Find- ings of the Association for Computational Linguisties: EMNLP 2021, Virtual Event Punta Cana, Domini- can Republic, 16-20 November, 2021, pages 2812- 2823. Association for Computational Linguistics.
Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross- modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5099-5113, Seattle, United States. Association for Computational Linguistics.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual transla- tion. In 9th International Conference Learning Representations, ICLR 2021. Viriual Event, Austria, May 3-7, 2021. OvenReview.net.
Biao Zhang, Barry Haddow, and Rico Sennrich. 2022. Revisiting and-to-end speech-to-text translation from scratch. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 26193-26205. PMLR.
Biao Zhang, Ivan Titov, Barry Haddow, and Rico Sen- nrich, 2020a. Adaptive feature selection end-to- end speech translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2533-254 2534,, Online. Association for Computational Linguistics.
Biao Zhang, Philip Williams, Tvan Titov, and Rico Sen- nrich. 2020b. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 1628- 1639, Online. Association for Computational Linguis- tics.
Yu Zhang, Wei Han, James Qin. Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li. Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, An- drew Rosenberg, Rohit Prabhavalkar, Daniel S. Park. Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara N. Sainath, Pedro J. Moreno, Chung-Cheng Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. 2023. Google USM: scaling automatic

---

A LASER-mined Evaluation Datasets
LANGs
WER
LANGs
WER
The dataset CoVoST 2, which based on the Com- mon Voice project (Ardila et al., 2020) with Ver- sion 4, has limited amount of audio training data for most languages in the X X->En directions. This limitation poses challenge the selection of se- mantically similar sentences evaluation datasets for low-resource languages. To overcome this chal- lenge, we incorporate Common Voice version as supplementary data for CoVoST 2 specifically for the low-resource languages. We filter out the au- dio data from version 13, which is duplicated with the Co VoST 2 training set, reserving the remaining data as evaluation set for each correspond- ing language. Subsequently, utilize LASER to mine bitext (transcription of audio) between each pair of given languages. A threshold of 1 1.05 is set to determine the extracted bitext, and this process is repeated for each language pair within all the languages in our study. These extracted evaluation datascts are then used to measure similarity across different languages.
23.14
47.73
Ir
16.70
58.83
de
18.94
60.78
11.33
58.36
13.13
су
61.66
72.88
lv
48.56
20.47
sl 48.28
30.55
SV
60.82
pt
32.63
la 73.31
id 47.39
50.22
ja 47.59
Table 3: Results of the multilingual ASR model which used to train the multilingual ST model in the X->X translation directions.
ping strategy with patience to 5 for English -X and X-*X model and averaged the last check- points for inference. As for the X X->Englihh model, we set the maximum number updates to 10K and averaged 5 checkpoints for inference, we chose the best averaged model according to the average BLEU on the validation sets and then evaluated it on the test sets.
B Training Details
C Results of Multilingual ASR
We used the Transformer (Vaswani et al., 2017) as the backbone for our multilingual end-to-end speech translation model, which has 12 layers for the encoder and 6 layers for the decoder with 16 at- tention heads and 1024 dimensions for embeddings, 4096 dimensions for FFNs. We the dropout rate 0.3 for the multilingual models. We initialized the transformer encoder with a pre-trained ASR model, which shares the same configuration as our multilingual ST model. We trained different ASR models for different translation directions, e.g., an English ASR model for the English-xX direction, multilingual ASR model (contains 21 languages) for the X-English direction and multilingual ASR model (contains 21 languages English) for the X-x direction.
In Table 3, we present the results of our multilin- gual ASR model. A consistent pattern emerges from these results, mirroring the findings of the multilingual ST model in the X-*En translation directions. In these cases, the low-resource lan- guages continue suffer low performance, even the context ASR tasks.
D Language Similarity
D.1 SVCCA Scores of the Source Languages in X- Translation
Figure 5 shows SVCCA scores of language pairs in the X-En direction across all encoder layers (from layer 0 to layer 11).
D.2 SVCCA Scores the Target Languages in En-+X Translation
We appended a language token at the beginning the translated sentences to denote which lan- guage should be translated to following Johnson et al. (2017). We did not add any language-specing token embedding at the source side for multilin- gual ASR and ST models.
Figure 6 shows SVCCA scores language pairs in the En- direction across all decoder layers (from layer 0 to layer 5).
We optimized parameters using Adam optimizer (Kingma and Ba, 2015) with label smoothing rate 0.1. The learning rate was scheduled according to the inverse square root of running steps with a warm-up up step 2500. We adopted the early stop-

---

(a) Eneoder layer
(b) Encoder layer 1
(c) Encoder layer
(d) Encoder layer
(e) Encoder layer 4
Encoder layer
Encoder layer
(h) Encoder layer? 7
ii Encoder layer 8
Encoder layer 9
(k) Encoder layer 10
(1) Encoder layer
Figure 5: SVCCA scores between the representations of X-+En language pairs (i.e., pairs of X) across all encoder layers, which is calculated on our LASER- mined evaluation datasets. Red cells indicate that the two languages are more related each other (higher SVCCA scores) and blue cells indicate that two languages are less related (lower SVCCA scores). Best viewed in color.

---

UNE
Decoder layer 0
(b) Decoder layer
(c) Decoder layer
(d) Decoder layer 3
(c) Decoder layer 4
(f) Decoder layer 5
Figure 6: SVCCA scores between the representations of En +X language pairs across all decoder layers. Red cells indicate that the two languages are more related to each other (higher SVCCA scores) while blue cells indicate that the two languages are less related (lower SVCCA scores). Best viewed in color.


Reviews:
[{"review_id": "fTbokCff87", "reviewer": "Reviewer_RGSM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful comments and valuable suggestions.\n\n**Answer for the Reject Reason 1&2**: Research findings in multilingual NMT and multilingual language models suggest that languages within the same language family should exhibit similarity. These insights have also prompted investigations into knowledge transfer across languages in multilingual contexts. However, in the case of multilingual speech translation, our observations contradict this trend, as depicted in Figure 1. This discrepancy is likely attributed to the insufficient amount of data available for these languages, preventing the establishment of distinct language-specific subspaces. \n\nRegarding the \"baseline,\" given the existing studies' thorough analyses of language similarity within multilingual models, particularly in multilingual NMT and multilingual LM, we have refrained from replicating this specific experiment. Instead, in the revised version, we will illustrate how multilingual models effectively capture language similarities in Section 5, enhancing readers' comprehension. Basically, a well trained S2T model also capture this trend as you conjectured. This is evident in the Romance/Germanic language branch, as illustrated in Figure 1. Nevertheless, the behavior of low-resource languages goes against this trend. To explore this further, we utilize Latent Dirichlet Allocation (LDA) for analysis. The results are shown in Figure 2, which showcases the distribution of language-specific subspaces. This distribution is influenced by the scarcity of data in low-resource languages, affecting their ability to learn these language-specific subspaces.\n\n**Answer for the Reject Reason 3**: Since the SOTA multilingual ASR models adopt seq2seq architecture, it is difficult to disentangle the encoder from the seq2seq architecture for further analyses.\n\nTable 1: Results of the multilingual ASR model which is trained on CoVoST 2 dataset.\n\n| LANGs | WER   | LANGs | WER   |\n| ----- | ----- | ----- | ----- |\n| en    | 23.14 | tr    | 47.73 |\n| fr    | 16.70 | et    | 58.83 |\n| de    | 18.94 | mn    | 60.78 |\n| ca    | 11.33 | ar    | 58.36 |\n| es    | 13.13 | cy    | 61.66 |\n| fa    | 72.88 | lv    | 48.56 |\n| it    | 20.47 | sl    | 48.28 |\n| ru    | 30.55 | sv    | 60.82 |\n| pt    | 32.63 | ta    | 73.31 |\n| zh    | 38.08 | id    | 47.39 |\n| nl    | 50.22 | ja    | 47.59 |\n\nHowever, we follow your suggestion to conduct the multilingual ASR experiments using transformer architecture. From these multilingual ASR results shown in Table 1, we can also observe that low-resource languages exhibit considerably lower performance levels in comparison to the high-resource languages. Additionally, when evaluating the SVCCA scores, we observe the identical pattern to that of the multilingual ST model.\n\nRegarding alignment, it becomes challenging to precisely determine the synchronization level between the audio and text modalities, given their processing through distinct modules, namely the encoder and decoder. Our conclusions predominantly arise from the comparison to bilingual models. This approach is driven by the recognition that the multilingual configuration mainly augments the text dataset while maintaining consistency in audio data.\n\n**Answer for the Reject Reason 4**: Thanks for your advice, we will use the tokenizers provided by SacreBLEU to replace char-BLEU in the revised version as shown in Table 2.\n\nTable 2: Results of Chinese and Japanese using the tokenizers provides by SacreBLEU.\n\n\n| LANGs | Bi ST (En$\\rightarrow$X) | En$\\rightarrow$X | X$\\rightarrow$X (En$\\rightarrow$X) |\n| ----- | ------------------------ | ---------------- | ---------------------------------- |\n| zh    | 23.7                     | 29.6             | 28.5                               |\n| ja    | 20.7                     | 25.2             | 24.2                               |\n\nThe signatures of SacreBLEU are as follows:\n\nBLEU+case.mixed+numrefs.1+smooth.exp+tok.zh+version.1.5.1\n\nBLEU+case.mixed+numrefs.1+smooth.exp+tok.ja-mecab-0.996-IPA+version.1.5.1", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The topic of the paper is important, given the increasing interest in multilingual speech translation in the research community. While many architectures have been proposed on the topic, there is still a need for deeper understanding of the model's behaviors. This paper broadens the discussion of the multilingual model not just on the performance (BLEU score), but also the behaviors of the model.\n\n2. The methods used in the paper are somewhat novel. For instance, the SVCCA method is an efficient way to analyze the similarity between languages. Furthermore, the methods are model agnostic, which means it can also analyze other architectures.\n\n3. The conclusions from the models are helpful for future study. For instance, the paper mentioned the efficacy of transferring from high-resource languages to low resource languages; improvement of audio encoding can improve the overall translation quality.", "reasons_to_reject": "1. While the conclusion from the paper is general and seems correct from the numbers provided in the paper, the quality of the multilingual model, especially on the low resource languages, is good enough to support some of the conclusions. With half of the languages have less then 5 BLEU scores, it's not noticeably clear whether the model in fact learned meaningful representations. \n\nIn fact, this can be seen from the analysis in the similarity in section 5. In the analysis of encoder and lower-layer decoder states, the similarity in the low resource language family (e.g. Turic, Mongolic) are not as expected. They (e.g. zh, mn) have more similarity than high resource languages from another language family, indicating a potential overfitting on the high resource language (if the languages in the same language group should have higher similarity.)\n\n2. The design of the similarity analysis can be improved. A \"baseline\" of the similarity analysis helps understand how multilingual capture the similarity. For instance the same SVCCA analysis on multilingual language model or speech encoder which requires looser data restriction (parallel speech/text),  One simple baseline can be run on the pure text language model and compute a similarity between two languages, and a well-trained S2T model should also capture that. The authors categorize the languages into several language families, with the indication of languages should be similar within the family. However, the conclusion is different from such indication, and we are not sure if it's because of low-quality of the model (see 1.), or it's the nature of the language (two language just not similar even in the same language group.)\n\n2.  Missing details. For example\na. Multilingual ASR results. This will give us a better idea whether the low performance of encoder is due to the encoder.\nb. The author claimed a well-aligned audio-and text helps the quality. However, the author didn't provide a multilingual setting with \"bad\" alignment. The conclusion is from Bilingual setting, from which the improvement could be from many other factors.\n3. For Chinese and Japanese, the SacreBLEU toolkit does have tokenizers (zh and ja-mecab), and they are more widely used than character level BLEU scores.\n\nUpdate after rebuttal\n1. For item 1, I do acknowledge that the discrepancy of low resource language performance. What I was suggesting is, due to the low-resource setting, the representation is not well learn, and thus the analysis from it might not be informative.\n2. I will remove the 3 and 4 from the reason to reject", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on the analysis of a multilingual speech-to-text translation model. More specifically, the paper trained a multilingual speech translation model on CoVoST 2 dataset with enhanced speech-text alignment and carried out the analysis on efficacy of multilingual training, similarity of the multilingual encoder & decoder.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ch93Ah0MtP", "reviewer": "Reviewer_bZDJ", "summary": "", "questions": "Given the results showcased in the paper, it becomes evident that when ample training data is available for all language pairs, the model acquires the ability to discern language-specific sub-spaces. This raises a pertinent question: how can we effectively strike a balance between accommodating language-specific representations and managing the overall model capacity? This becomes particularly crucial as we consider expanding the number of language directions in our endeavors.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful comments and valuable suggestions.\n\n**Answer for Reject Reason 1**: Expanding the extent of these analyses to include a multi-domain context would certainly enhance their robustness, which is a consideration we intend to address in our future endeavors. However, the current feasibility of this objective is predominantly constrained by the available dataset. The MuST-C dataset, extracted from TED talks, offers a substantial corpus for speech translation, yet its limitation lies in its restricted language diversity, comprising just 8 languages. Similarly, the IWSLT SLT dataset encompasses only 3 languages. This scarcity in linguistic variety poses a challenge in comprehensively examining the results of multilingual speech translation endeavors.\n\nEffectively investigating the behavior of multilingual speech translation necessitates a broader array of languages for analysis. In this context, the CoVoST 2 dataset emerges as the most suitable candidate. Sourced from the Common Voice project, this dataset receives contributions from volunteers worldwide, thus offering a more diverse and representative range of languages for conducting multilingual speech translation tasks.\n\nWhisper's conclusion posits that \"multi-domain training increases robustness and generalization.\" Nevertheless, the issue remains that we are confronted with a scarcity of publicly accessible multi-domain datasets. Whisper sources its data from the web, but the origins of the data remain undisclosed. Additionally, their focus is primarily on treating speech translation as a supplementary task.\n\nIn summary, the notion of expanding training data to encompass multiple domains presents a promising strategy for fostering the advancement of multilingual speech translation models. However, a critical aspect for future endeavors lies in the collection of high-quality speech translation data covering various domains.\n\n**Answer for Reject Reason 2**: We are convinced that a greater availability of data, sourced from a variety of domains or tailored toward speaker identification, would enable us to explore the behavior of multilingual speech translation models in greater depth. This expanded dataset could provide valuable insights that have the potential to enhance the overall performance of multilingual speech translation systems.\n\n**Answer for the Question**: Achieving an ideal balance while including language-specific representations remains a challenging task. Some investigations in this domain show that the treatment of multilingual languages in Pre-trained Language Models (*Wu, Shijie, and Mark Dredze. \"Are All Languages Created Equal in Multilingual BERT?.\" ACL 2020 (2020): 120.*) or Large Language Models (*Zhu, Wenhao, et al. \"Multilingual machine translation with large language models: Empirical results and analysis.\" arXiv preprint arXiv:2304.04675 (2023).*; *Huang, Haoyang, et al. \"Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting.\" arXiv preprint arXiv:2305.07004 (2023).*) is far from being uniform. Evidently, the issue of multilingualism in the realm of NLP remains unresolved.\n\nAt present, our considerations are primarily rooted in a data-driven perspective:\n\n(a) One approach involves leveraging external ASR or MT data to pretrain the model, thereby enhancing initial states; (b) Another strategy involves data augmentation techniques such as back-translation, which recent research has shown enhances performance in bilingual speech translation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors showed very interesting and useful findings through reliable analyses. ", "reasons_to_reject": "・The experiments are limited to single-domain experiments on CommonVoice, specifically within the reading speech domain. On the other hand, multilingual NMT has been explored utilizing multi-domain datasets (correct me if I'm wrong). This raises concerns regarding the broad applicability of the conclusions drawn, especially considering the potential variation when training multilingual S2T models on diverse multi-domain and multi-lingual datasets, akin to the approach employed by Whisper [Radford et al., 2022].\n\n・Despite the authors offering two valuable insights, the paper culminates in a somewhat straightforward conclusion that data is of paramount significance. The potential for augmenting the contribution lies in the presentation of a methodology aimed at addressing the weaknesses uncovered through the conducted analyses, which could increase the overall scores.", "questions_to_authors": "Given the results showcased in the paper, it becomes evident that when ample training data is available for all language pairs, the model acquires the ability to discern language-specific sub-spaces. This raises a pertinent question: how can we effectively strike a balance between accommodating language-specific representations and managing the overall model capacity? This becomes particularly crucial as we consider expanding the number of language directions in our endeavors.", "paper_topic_and_main_contributions": "This paper conducts an in-depth analysis of concealed representations within multilingual end-to-end speech-to-text translation (S2T) models. Notably, the authors reveal a significant departure from the knowledge transfer phenomenon observed in multilingual neural machine translation (NMT) literature, which does not extend to the realm of multilingual S2T. This conclusion is validated through a two-fold methodology. First, the authors compared translation qualities among bilingual, En->X, X->En, and X->X models. Second, they calculated similarities between sentence-level representations learned in the models via Singular Value Canonical Correlation Analysis (SVCCA). The authors showcased two interesting findings. The authors present two key insights. Firstly, the augmentation of training data proves pivotal in elevating translation quality, particularly in low-resource language pairs. With a lack of enough training data, the encoder cannot find a language-specific sub-space on top of the encoder. Secondly, the sufficiency of training data emerges as a crucial factor for capitalizing on the advantages derived from linguistic similarities across languages.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JnjB9svNSe", "reviewer": "Reviewer_JjpA", "summary": "", "questions": "Did the authors try to perform ablation studies by, for instance, train multilingual E2E ST models with languages grouped by language family/branch, rather than using all the languages available?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful comments and valuable suggestions.\n\n**Answer for the Reject Reason**: Undoubtedly, the scarcity of publicly accessible data poses the primary challenge in our efforts to delve deeper into the robustness of the multilingual speech translation model. This constraint also extends to the exploration of both multi-domain capabilities and speaker identity aspects. However, we are committed to persistently working on overcoming this challenge and advancing our understanding in these areas.\n\nAs for the language similarity part (Section 5), in the revised version, we aim to offer a more in-depth analysis of language similarity, with a focus on mid-resource languages within their respective language families. This decision is based on the understanding that low-resource languages do not possess a subspace of sufficiently high quality. This extended examination will be accompanied by an extra subplot depicted in Figure 2 to improve the depth of our observations, as suggested in the Presentation section.\n\nThe insights drawn from Figure 2 allow us to discern the reasons behind Persian's dissimilarity to other Indo-European languages, as evidenced by its lower SVCCA scores. This observation is underpinned by the distinct location of Persian's language-specific subspace within Figure 2. Unlike other Indo-European languages, Persian's subspace aligns with a different axis, setting it apart. Meanwhile, mid-resource Indo-European languages share a common subspace with higher-resource Indo-European languages. We would add a subplot of Figure 2 focusing on Indo-European languages.\n\n**Answer for the Question**: At present, we have not conducted ablation studies based on language families or branches. Such an investigation may provide valuable insights into the behavior of multilingual speech translation. We intend to explore this aspect in future research and will present the findings in the revised version of our work. Thank you for this excellent suggestion!\n\nThe typos and other issues will be corrected in the revised version. Many thanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very clearly written and easy to follow. \n\nAnalysis of multilingual representations in E2E ST is lacking in literature, and though some of the conclusions are not very surprising (ex. having more training data is better than just using knowledge transfer), I believe this empirical investigation work is a great contribution providing insights for better understanding of the multilingual representations in ST systems, for further multilingual ST research & also for practitioners when allocating resources for low-source languages.\n\nI also appreciate the analysis made from different perspectives (linguistic factors, amount of training data, characteristics of writing systems, etc.).", "reasons_to_reject": "Given the limitations in terms of diversity, scale and quality for some language pairs (ex. on some low-resource pairs, there is only 1 hour of data), it remains unclear to which extent the conclusions will remain valid. However it is not quite the focus of contribution of this paper, more a limitation of data available, so I don't consider this point as a strong reason for rejection, rather some possible directions for improvements.\n\nSome analysis on language similarity are a bit lightweight and the di/similarity between languages are not very discussed (for instance, how Persian is dissimilar compared to other languages in the dataset from the Indo-European?), making it uneasy to disentangle the factors between language similarity and training data availability. Some finer-grained analysis on the clusters identified in Figure 2 or some ablation studies could be helpful for a deeper understanding of the impacts of the factors the authors investigated. ", "questions_to_authors": "Did the authors try to perform ablation studies by, for instance, train multilingual E2E ST models with languages grouped by language family/branch, rather than using all the languages available?", "paper_topic_and_main_contributions": "This paper analyzed a multilingual end-to-end speech translation (E2E ST) model using Singular Value Canonical Correlation Analysis (SVCCA), with a focus on the learned representations, both from the perspective of language similarity and linguistic typology. \nThe authors found that linguistic similarity may be less effective when there is insufficient training data in a language. They also found that the encoder representations present better accuracy for phonology/phonetic features.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- Figure 2, caption: lacks a period mark at the end.\n\nGrammar:\n- Last paragraph of 5.2, \"This finding also in line with the finding Kudugunta et al. (2019).\" -> \"[...] with the finding of [...]\"\n\nStyle:\n- The last 2 sentences in the last paragraph of 5.1 could be reformulated for more clarity. The contrast emphasized by \"while\" may not be very direct when the sentences are separated. \n\nPresentation:\n- For Figure 2, though the authors listed all the languages, it's almost impossible to recognize which language specifically is represented by the dots; only the group-level information is highlighted. Thus the legend itself is barely useful, and if the authors expose some per-language analysis, it could be interesting to have a separate plot emphasizing the distribution for, for instance, one of the axis, and show whether some low-resource languages tend to be clustered with particular group of mid- or high- resource languages."}}]

Author rebuttal:

Meta-review: This paper presents an analysis of multilingual speech-to-text translation representations, with a focus on analyzing similarity of sentence-level multilingual representations via Singular Value Canonical Correlation Analysis (SVCCA). There is some divergence in numerical scores, but the overall tone of the review text is similar across all 3 reviews. Reviewers are enthusiastic about this type of multilingual representation analysis and its general usefulness, but they also have some concerns and suggestions for these specific experiments. In particular, two reviewers mentioned concerns about the validity of drawing conclusions regarding the similarity of languages with very small amounts of data, indicating the existing discussion on data quantity may need to be revised. The authors seem amenable to the suggested revisions, and have already provided some new results.
Decision: Accept-Findings