BUT Systems for IWSLT 2023 Marathi - Hindi Low Resource Speech Translation Task
Year: 2023

Abstract:
This paper describes the systems submitted for Marathi to Hindi low-resource speech translation task. Our primary submission is based on an end-to-end direct speech translation system, whereas the contrastive one is a cascaded system. The backbone of both the systems is a Hindi-Marathi bilingual ASR system trained on 2790 hours of imperfect transcribed speech. The end-to-end speech translation system was directly initialized from the ASR, and then finetuned for direct speech translation with an auxiliary CTC loss for translation. The MT model for the cascaded system is initialized from a cross-lingual language model, which was then fine-tuned using 1.6 M parallel sentences. All our systems were trained from scratch on publicly available datasets. In the end, we use a language model to re-score the n-best hypotheses. Our primary submission achieved 30.5 and 39.6 BLEU whereas the contrastive system obtained 21.7 and 28.6 BLEU on official dev and test sets respectively. The paper also presents the analysis on several experiments that were conducted and outlines the strategies for improving speech translation in low-resource scenarios.

heading: 1 Introduction
text: A typical end-to-end (E2E) speech translation model is trained with the help of data triplets (x,y, z), i.e., the speech signal (x) in source language, along with its transcription (y), and, text translation (z) in target language. In usual lowresource scenarios, the transcriptions in source language are unavailable and moreover the speech signal and the translation pairs (x, z) are also limited, which is the case for the IWSLT 2023 Marathi to Hindi low-resource speech translation task (Agarwal et al., 2023). In such cases, one can rely on transfer learning, where models trained on relatively large amounts of data (possibly on a related task such as automatic speech recognition) are transferred (adapted) to the target task/scenario
(such as speech translation) using little amounts of labelled data (Bansal et al., 2019). To be specific, we train automatic speech recognition (ASR) systems on relatively large amount of transcribed speech data (2790 hours), and transfer the model for speech translation task by fine-tuning it on relatively small amount (16 hours) of IWSLT MarathiHindi training data.
This paper describes the systems submitted for the aforementioned task. While building the systems, we mainly focused on end-to-end systems, which resulted in our primary submission. We have also put some efforts in building a cascade pipeline that was submitted as a contrastive system. Both the systems come under the unconstrained category, i.e., we relied on external, publicly available data to train models. These models, which we refer to as back-bone models, mainly comprise automatic speech recognition (ASR), machine translation (MT) and language models (LM).
The Section 2 describes the various datasets used for training the back-bone models, and Section 3 presents the details of each individual back-bone models (ASR, MT, LM), followed by description of transfer learning for actual speech translation systems in Section 4. The Section 5 gives the results and analysis, quantifying the effect of various factors on the target translation task. Finally, we conclude in Section 6 and discuss directions for future works.

heading: 2 Datasets for training
text: Here we describe the details and present the statistics of various datasets used for training the backbone models. These datasets come under various categories, i.e., paired speech data for training ASR, parallel text data for training MT and monolingual data for training LMs. All the data we considered for training covers only Hindi and Marathi languages. Both these share the same Devanagari script (unicode block) but there a few set of charac-
227
ters that are mutually exclusive.

heading: 2.1 Paired speech data
text: The paired speech data for Marathi and Hindi are collected from various publicly available datasets as listed below:
• GramVaani (GV)1 comprises telephone quality speech in Hindi (hi). The dataset was used for Interspeech 2022 special session (Bhanushali et al., 2022; Patel and Scharenborg, 2022). We considered only the 100 hour labelled split of the dataset.
• Indian Language Corpora (ILC) (Abraham et al., 2020)2 is crowdsourced speech data along with transcriptions in Marathi language. The dataset is collected from 36 participants with various socio-economic backgrounds and dialects.
• Mozilla Common Voice v12 (MCV) (Ardila et al., 2020) is a crowdsource collection of paired speech data across various languages. We took the validated versions of Hindi (hi) and Marathi (mr) from this corpus.
• MUCS (Diwan et al., 2021)3 is multilingual and code-switched corpus for training ASR systems in 6 different Indian languages. The dataset was introduced in Interspeech 2021 as part of a special session focusing on ASR for Indian languages. We considered Hindi and Marathi data from this corpus. Although MUCS contains about 100 hours of transcribed speech for both Marathi and Hindi, the lexical content is not diverse, i.e., the same utterances were spoken by various speakers.
• Multi-speaker speech corpora (MSSC) (He et al., 2020)4 is a collection of clean speech data with transcriptions intended for building text-to-speech synthesis systems for various Indian languages. We considered only the Marathi split from this corpus.
• Shrutilipi (SL)5 is collected from public archives and contains about 6400 hours of
1https://sites.google.com/view/ gramvaaniasrchallenge/
2https://www.cse.iitb.ac.in/~pjyothi/ indiccorpora/
3https://navana-tech.github.io/ MUCS2021/data.html
4https://www.openslr.org/64/ 5https://ai4bharat.org/shrutilipi
radio broadcast news in various Indian languages. The corresponding transcriptions were obtained with the help of OCR and other heuristics (Bhogale et al., 2022). This corpus is the bigger chunk of the data we used for training, but the transcriptions obtained are not accurate. A manual inspection revealed some erroneous alignments at the beginning and end of the utterances. By setting a threshold (≥ 85) on the provided alignment score, we filtered Hindi (hi) and Marathi (mr) data from this corpus. We believe the domain of this data is closer to IWSLT 2023 speech translation data.
The statistics of each of the above datasets is presented in Table 1. This data was used to train mono and bilingual ASR systems that are described later in Section 3.1. All the speech data was upsampled to 16 kHz. Using Kaldi toolkit (Povey et al., 2011) 80 dimensional filter banks and 3- dimensional pitch features are extracted for every 25 ms of speech frame sliding with 10 ms.

heading: 2.2 Monolingual and parallel text data
text: We prepared monolingual data for both Hindi and Marathi. We pooled data from transcribed speech (Table 1), Samanantar (Ramesh et al., 2022), Indic2Indic, IIIT-H CVIT (Siripragada et al., 2020) corpus, resulting in 9 M sentences (217 M tokens) for Hindi and 4M sentences for Marathi6.
The parallel text was taken only from Indic2Indic split from Samanantar (Ramesh et al., 2022), whose statistics are given in Table 2. We retained punctuation in all the text.

heading: 2.3 Speech translation data
text: The official speech translation data for Marathi - Hindi involves around 16 hours of training split, i.e., Marathi speech and its translations in Hindi. There are no transcriptions for the Marathi speech. Table 3 presents the statistics of the provided speech translation data. We used speed perturbation (0.9, 1.0, 1.1) to augment the speech translation data. The effect of such augmentation on the final translation performance is discussed later in Section 5.
6Due to a bug in data preparation, only Shrutilipi text data 400 K (8.2 M tokens) out of 4 M sentences were used to train Marathi LM.

heading: 3 Back-bone models
text: Here, we describe the architecture and training details of various backbone models.

heading: 3.1 ASR
text: The ASR model is a transformer based seq2seq model. The speech features are passed through 2 layers of convolution, followed by 12 layers of transformer encoder blocks and 6 layer of transformer decoder blocks, with dmodel = {256, 512}7, heads = 4, dff = 2048. The dropout was set to 0.1. The model is trained with a batch size of 128 for 100 epochs using Adam optimizer (Kingma and Ba, 2015), and warm up scheduler with a peak learning rate of 0.0005. The training is done with joint CTC and attention objective (Karita et al., 2019), where the CTC is applied at the end of encoder layer and the attention acts at the output of autoregressive decoder (teacher-
7Smaller models use dmodel = 256, where as bigger models use dmodel = 512.
forcing).
Lasr = αLctc(x,y) + (1− α)Latt(x,y). (1)
In case of bilingual ASR, the CTC layer, input and output layers of the decoder are specific to each language, i.e., the (sub-)word embeddings are not shared across languages. Such a design ensures that only target language tokens are decoded, irrespective of the phonetic similarity with other languages in the model. The ASR models were trained using ESPnet toolkit (Watanabe et al., 2018). The performance of various mono and bilingual ASR systems is discussed later in Section 5.

heading: 3.2 XLM
text: The architecture of pre-training masked-language model is based on cross-lingual language model (XLM) (Lample and Conneau, 2019)8. More specifically, we use translation language modelling objective along with masked language modelling to train the transformer based encoder. Here, we use BPE-based sub-word vocabulary that is obtained jointly for both languages. The model has 6 transformer blocks with 512 embedding dimension, 8 attention heads, dropout of 0.1 for both attention and feed-forward layers. The model is trained for a maximum of 1000 epochs using Adam optimizer with a learning rate of 0.0001.
8https://github.com/facebookresearch/ XLM

heading: 3.3 MT
text: The MT model is a transformer based seq2seq model initialized from XLM. Both the encoder and decoder parameters are initialized from XLM encoder, except for the cross-attention parameters in the decoder that are randomly initialized. The model is then fine-tuned on the same 1.6 M parallel sentences with a batch size of 64 and a maximum of 1000 epochs. The model achieved 23.0 and 22.6 BLEU scores on the internal valid and test sets (Table 2) respectively.

heading: 3.4 LM for re-scoring
text: For Hindi, we used an LSTM of three layers of 4096 units each, with no dropout. The model was trained on 217 M sub-word tokens obtained by tokenizing the monolingual Hindi corpus into a 10k Unigram vocabulary (Kudo, 2018). The model achieved validation perplexity of 46. Thereafter, we have fine-tuned it on text data from Shrutilipi (SL) data for 500 steps.
For Marathi, we used an LSTM of 2 layers per 2048 units, again with no dropout. This model also utilized a 10k Unigram vocabulary and was trained on 8.2 M tokens. This model achieved validation perplexity of 120.

heading: 4 Speech translation systems
text: Here, we briefly describe both the end-to-end and cascade systems.

heading: 4.1 End-to-end
text: The E2E models are initialized from pre-trained ASR models. We use both the encoder and decoder from the ASR, as it provides a better initialization since the representations from the encoder are readily compatible with the decoder (Bansal et al., 2019). The model is then trained for direct speech translation, with the auxiliary CTC objective also for translation (Zhang et al., 2022; Yan et al., 2023; Kesiraju et al., 2023).
Lst = λLctc(x, z) + (1− λ)Latt(x, z) (2)
Latt(x, z)
The effect of various initializations and their influence on downstream speech translation is discussed later in Section 5.
The E2E speech translation was also trained using ESPnet toolkit. Our changes to the original toolkit, along with the training recipes, are available online9.
A beam search based joint decoding (Karita et al., 2019) that relies on the weighted average of log-likelihoods from both the CTC and transformer decoder modules is used, that produces the most likely hypotheses according to
ẑ = argmax z
β log pctc(z | x)+
(1− β) log patt(z | x) (3)
We found λ = {0.1, 0.3}, β = {0.1, 0.3} suitable for joint training and decoding respectively.

heading: 4.2 Cascade systems
text: For the cascade speech translation systems, we first decode n-best hypotheses from ASR model and obtain 1-best from Marathi LM rescorer. These are then passed directly to the MT system, which gives us n-best translation hypotheses in target language Hindi. These are then re-scored by Hindi LM to give us 1-best translation hypotheses.
9https://github.com/BUTSpeechFIT/ espnet/tree/main/egs2/iwslt23_low_ resource/st1
A further fine-tuning of the MT system using 1-best hypotheses from Marathi to Hindi IWSLT training set did not improve the results. Due to time constraints, we did not try various strategies (Bentivogli et al., 2021) or hyperparameter tuning for the cascade systems.
4.3 Re-scoring n-best hypotheses
We have utilized the language models to re-score up to 100-best hypotheses in both languages. Using BrnoLM10, we have introduced the language model scores. Here, we have tuned the two hyperparameters: The weight of the LM score (additive to 1.0 weight of the acoustic system) and an insertion bonus, added for each token of the hypothesis, in the LM tokenization. For the E2E system, we have achieved optimal results with LM weight 1.2 and insertion bonus 5.5. For the Marathi ASR in the cascade system, optimal setting was 0.3 and 3.5. For the translation system in the cascade, we did not achieve any improvement by re-scoring the output with the Hindi LM.

heading: 5 Results and analysis
text: Here, we present the performance of various backbone models, along with analysis showing the effectiveness of various factors such as initializations, data augmentation, auxiliary objectives and joint decoding.

heading: 5.1 Performance of ASR systems
text: From the Table 4 we can see that the bilingual models perform (B1, B2) better than the monolingual parts (H1, M1, H2, M2). Here, H1, M1 and B1 are smaller models with dmodel = 256, whereas
10https://github.com/BUTSpeechFIT/ BrnoLM
H2, M2 and B2 are bigger ones with dmodel = 512. All the ASR models were trained with joint CTC and attention loss, where the CTC weight of 0.3 was found to be optimal. The same weight was used during joint decoding. Since we retained the original punctuation in the text, the WER is slightly affected.

heading: 5.2 Performance of ST
text: Here we present the results of speech translation systems based on end-to-end architecture. As shown in Table 5, all the ST models were initialized either from mono or bilingual ASR systems and fine-tuned using the speech translation data (with or without data augmentation). While most of these systems can be considered direct end-toend; using an external LM for re-scoring the n-best makes an exception. Using a Marathi monolingual ASR model would be sub optimal because the internal language model represented in the decoder of the ASR would not be suitable for generating linguistically acceptable text sequences in Hindi.
Fig. 2 shows the effect of CTC weight during joint training and decoding. We can see that 0.3 is the optimal weight both for training and decoding. Since, we have a separate vocabulary for both the languages, the posterior probabilities from CTC during joint decoding will only correspond to the tokens from the target language Hindi. This is important, since both the languages come from same family with high phonetic similarity, and use same Devanagari script, the non auto regressive CTC decoder does not accidentally provide higher scores for tokens from source language Marathi. The latter scenario can happen when using a jointsub word vocabulary for both the languages.
Sacrebleu library (Post, 2018) was used to com-
pute BLEU11 and CHRF212 scores in the dev sets.
From the Table 5, we can see that independent improvements come from using bilingual ASR trained on more data, data augmentation (speed perturbation) and LM re-scoring. In case of cascade system, the LM re-scoring did not improve the results. We believe this is because the Marathi LM was trained on much fewer amounts of data (400K sentences). We plan to rerun these experiments in the near future.
Finally, our primary submission was based on B2 + ST fine-tuning with data augmentation + LM re-scoring which obtained 39.6 BLEU and 63.3 CHRF2 scores on official test set. Our contrastive system was based on B2 + MT + LM re-scoring which obtained 28.6 BLEU and 54.4 CHRF2 scores.
A manual inspection of the translation outputs revealed that several mismatches occurred where there are ambiguous numerals, i.e., some numbers were written using digits while the others were spelled out verbatim. There are also cases where both notations were mixed. We believe, further text normalization of both reference and hypothesis could give us a better picture of the evaluation scores.
11nrefs:1 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.3.1
12nrefs:1 | case:mixed | eff:yes | nc:6 | nw:0 | space:no | version:2.3.1

heading: H1 ✗ 16.3 45.0

heading: H2 ✓ 24.9 51.0

heading: B1 ✗ 17.4 46.2

heading: B1 ✓ 20.1 48.2

heading: 6 Conclusions
text: In this paper, we presented the systems submitted to the IWSLT 2023 Marathi Hindi low resource track. Our main efforts were along the end-to-end direct speech translation system, initialized from a bilingual ASR. The model was jointly trained with CTC and attention objective directly for translation. The joint decoding provided additional benefits. These strategies combined with speed perturbation for data augmentation and re-scoring the n-best hypotheses using external LM provided further significant improvements. We also submitted a cascade system which uses the same bilingual ASR as the backbone, followed by an MT system. Both systems performed competitively, while the one based on end-to-end provided superior results in terms of BLEU. It is yet to be investigated, if the large pre-trained MT systems would close the gap between cascade and end-to-end systems.

heading: Acknowledgements
text: The work was supported by Czech National Science Foundation (GACR) project NEUREM3 No. 19-26934X, Czech Ministry of Education, Youth and Sports project no. LTAIN19087 “Multi-linguality in speech technologies” and Horizon 2020 Marie Skłodowska-Curie grant ESPERANTO, No. 101007666. Computing on IT4I supercomputer was supported by the Czech Ministry of Education, Youth and Sports through the e-INFRA CZ (ID:90254). We thank the reviewers for their constructive feedback.

heading: Future Work
text: We plan to rerun these experiments in the near future. Finally, our primary submission was based on B2 + ST fine-tuning with data augmentation + LM re-scoring which obtained 39. 6 BLEU and 63. 3 CHRF2 scores on official test set. Our contrastive system was based on B2 + MT + LM re-scoring which obtained 28. 6 BLEU and 54. 4 CHRF2 scores. A manual inspection of the translation outputs revealed that several mismatches occurred where there are ambiguous numerals, i. e. , some numbers were written using digits while the others were spelled out verbatim. There are also cases where both notations were mixed. We believe, further text normalization of both reference and hypothesis could give us a better picture of the evaluation scores. 11nrefs:1 | case:mixed | eff:no | tok:13a | smooth:exp | version:2. 3. 1
12nrefs:1 | case:mixed | eff:yes | nc:6 | nw:0 | space:no | version:2. 3. 1