## 4 Experiments
### Experiments  
_RQ1: To what extent can mT5, a multi-lingual pre-trained language model, produce satisfactory results for Arabic end-to-end task-oriented DS?_  
This experiment aimed to investigate the performance of an end-to-end Arabic task-oriented dialogue system using an mSeq2 Seq model for Arabic. This mono-lingual setting only requires one language to train and test the model. Thus, we trained and tested the proposed mT5 model (AraConv) using the Arabic-TOD dataset. The AraConv model differs from the baseline with the training setting where AraConv trained on Arabic dialogues while the baseline did not (zero-shot learning). Table 4 shows the results--in terms of BLEU, APIACC, TSR, DSR, and JGA--of the AraConv model in the mono-lingual setting in comparison to the English and Chinese experiments on the BiToD dataset [1]. The observed English results [1] outperformed the AraConv results. This is unsurprising because there are more data for English and Chinese. The model trained and tested on English or Chinese data still performed better than that tested on the Arabic-TOD dataset, which represented only 27% of the BiToD dataset [1]. Where the original mT5 model was trained using multiple languages, the English data represented 5.67% of the whole corpus, and Chinese and Arabic represented 1.67% and 1.66% of the total data, respectively [5], explaining the superior performance for English dialogue. Additionally, Arabic is a language with extensive grammatical case marking [5], which causes lower evaluation metrics compared to English. Meanwhile, despite the comparable sizes of the training data for Arabic and Chinese, the results of the mono-lingual model trained on Chinese BiToD dataset outperformed the AraConv model. This may have been due to the small size of the Arabic-TOD dataset compared to the Chinese BiToD dataset. Nonetheless, the AraConv model achieved a better BLEU value (by approximately 63%) than the Chinese model, meaning that the AraConv model can generate more fluent responses than the Chinese model. Still, the AraConv model did not achieve perfect results, potentially due to the complicated nature of the Arabic-TOD dataset, its complex ontology, and its diversity of user goals. Moreover, the DSR result was lower than the TSR result, likely because of the multiple tasks included in the dialogue (2-4 tasks). For instance, some dialogues included multiple tasks in a single dialogue (e.g., a single dialogue can involve the tasks of finding a hotel to stay at, a restaurant to eat at, an attraction to visit, and information about the weather). \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **TSR** & **DSR** & **APIAcc** & **BLEU** & **JGA** \\ \hline \multicolumn{5}{c}{**Arabic**} \\ \hline Baseline & 3.95 & 1.16 & 4.30 & 3.37 & 8.21 \\ \hline AraConv & 45.07 & 18.60 & 48.86 & 31.05 & 34.82 \\ \hline \multicolumn{5}{c}{**Other languages**} \\ \hline EN [1] & **69.13** & **47.51** & **67.92** & **38.48** & **69.19** \\ \hline ZH [1] & 53.77 & 31.09 & 63.25 & 19.03 & 67.35 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Mono-lingual experiment dialogue state tracking (DST) and end-to-end dialogue generation results for the AraConv model trained on Arabic-TOD dataset compared to the baseline and the mono-lingual BiToD experiments [1] using English (EN) and Chinese (ZH). Bold numbers indicating the best result according to the column’s metric value. _RQ2: To what extent can joint-training the mT5 model on Arabic dialogue data and data for one or two high-resource languages (namely, English or English and Chinese) improve the performance of Arabic task-oriented DS?_  
Answering this research question requires performing two experiments to investigate the performance of building an end-to-end Arabic task-oriented dialogue system using an mSeq2 Seq model in bi-lingual and multi-lingual settings. Because two languages are used to train and test the model in the bi-lingual setting, we trained the proposed model mT5 on both the Arabic-TOD and English-BiToD datasets [1].