## 3 Methodology
### Baseline Model  
Our current baseline model is the mT5 (Multilingual T5) model is a state-of-the-art multilingual Transformer-based model designed to perform a variety of natural language processing (NLP) tasks by treating all tasks as text-to-text problems. The model is a multilingual variant of the original T5 (Text-to-Text Transfer Transformer), which was initially designed to work primarily with the English language. mT5 extends the architecture of T5 by enabling it to handle tasks in over 100 languages, making it a powerful tool for cross-lingual transfer learning and multilingual NLP applications.  
The mT5 model shares the same architecture as T5, which is based on the Transformer model introduced by [PERSON] et al. [32]. The Transformer architecture features self-attention mechanisms and feed-forward neural networks, enabling effective processing of sequential data. The self-attention helps capture long-range dependencies in sequences, while feed-forward networks perform non-linear transformations. Like T5, mT5 operates within a text-to-text framework. Here, input and output sequences are treated as text, and tasks like text classification, question answering, summarization, and machine translation are all framed as text generation problems. This unified approach simplifies task handling and allows mT5 to be fine-tuned across diverse natural language processing tasks.  
The model is pretrained on the mC4 (Multilingual C4) dataset, which is a multilingual version of the C4 dataset used for T5. The mC4 dataset contains text data in over 100 languages, including high-resource languages like English, Spanish, French, and Chinese, as well as a variety of low-resource languages such as Swahili, Marathi, and Igbo. These diverse training data allow mT5 to learn general linguistic representations that can be applied across different languages, enabling cross-lingual transfer. Additionally, mT5 is scalable and available in different sizes (e.g., mT5-small, mT5-base, mT5-large), making it adaptable to various computational needs. mT5, like T5, is pretrained using the denoising objective. During pretraining, a portion of the input text is randomly masked, and the model is tasked with predicting the missing tokens. This is performed by leveraging a span-based denoising objective, where spans of text are masked, and the model must recover them. The objective is formally expressed as follows:  
\[\mathcal{L}=-\sum_{i\in\text{masked\ position}}logp(x_{i}|x_{-i}) \tag{9}\]  
where \(x\) is the input sequence, and \(x_{i}\) represents the masked tokens. This pretraining task enables the model to learn contextual relationships between words in a sentence, improving its ability to handle downstream NLP tasks.  
Once pretrained, mT5 can be fine-tuned on specific downstream tasks, such as machine translation, sentiment analysis, or text summarization. The model is fine-tuned using task-specific datasets, where the input-output pairs are defined according to the task requirements. For example, in machine translation, the input might be a sentence in English, and the output would be the corresponding translation in Twi. During fine-tuning, the model adapts its weights based on the specific characteristics of the task while still retaining the knowledge learned during pretraining. This transfer learning approach allows mT5 to perform well on tasks with limited training data, particularly for languages with fewer resources. Table 1 represents the experimental setup and key details of the model's configuration and evaluation.