# Cascade Speech Translation for the Kazakh Language
## 4 Materials and Methods
### Machine Translation
#### 4.2.2 Methods  
Baseline transformer model. This subsection describes the technical details of training baseline neural machine translation models based on the transformers architecture. They were trained using Joey NMT, a simple neural machine translation toolkit based on PyTorch. Joey NMT offers several well-known NMT capabilities in a compact and straightforward code base. Joey NMT, despite its emphasis on straightforwardness, affords traditional designs (RNNs, transformers), fast beam search, weight tying and so on and accomplishes performance on common benchmarks that is on par with more complicated toolkits [39]. The NMT based on transformers relies on the amount of data available for training and the configuration parameters. The encoder of the model for the Kazakh-Russian language pair use 512-dimensional embedding and hidden layer sizes. The decoder has the same dimensions as the encoder. Adam optimizer with a maximum learning rate of 0.0003 and a minimum of \(10^{-8}\) was used to train the model. Dropout probability of 0.3 was set for both the encoder and the decoder. A significant improvement in performance came from using a different tokenization strategy. NMT would often tokenize using words. However, performance was dramatically improved when utilizing a byte pair encoding (BPE). Therefore, BPE with a joint vocabulary size of 32 thousand words was applied as well.  
Fine-tuned mBart model. BART is a groundbreaking method for pre-training a complete denoising autoencoder that can handle sequence-to-sequence tasks [40]. It has been primarily trained on a massive corpus of monolingual data in English. The transformer architecture is employed for the standard sequence-to-sequence implementation of BART, which comprises a bidirectional autoencoder and a left-to-right autoregressive decoder. It includes two types of architectures: BERT, which has a bidirectional encoder, and GPT, which has a left-to-right decoder. The base BART model has six encoder and decoder layers, while the large model has twelve.  
A multilingual extension of BART, mBART, is designed to pre-train on multiple monolingual languages. Similar to its predecessor, mBART uses 12 layers each of encoder and decoder, with layer-normalization following each one. The former version has 25 languages. An expanded version of mBART now includes up to 50 languages.  
The architecture of mBART, which we utilized in our work, follows the conventional sequence-to-sequence transformer-based design with a 12-layer encoder-decoder and a model dimension of 1024 on 16 heads. Our training process involved bidirectional training with 0.3 dropout, 0.2 label smoothing, 2500 warm-up steps and a maximum learning rate of 3 \(\times\)\(10^{-5}\), as outlined in [40]. The model was trained for up to 100 thousand updates and selected the final model based on validation likelihood. To evaluate the model's performance, beam search was used with a beam size of 5 and calculated the results against the true-target tokenized data, reporting the scores in BLEU. To train our model, we employed FairSeq [37].