Paper Info

Title: Two-stage Pipeline for Multilingual Dialect Detection
Publish Date: Unkown
Author List: Ankit Vaidya (from Pune Institute of Computer Technology), Aditya Kane (from Pune Institute of Computer Technology)

Figure

Figure 1: Class distribution of dialects
Figure 2: System diagram for dialect classification.The LID classifies the input into one of 3 languages.The sample is then further classified into dialects by language specific models.
Figure 3: Confusion matrix of 9-way classification.Note that rows are normalized according to the number of samples is that class.
Our complete results for Track-1 using the two-stage dialect detection pipeline.Model-* denotes the language of the models used for the experiments.
Performance on Track-1 validation dataset of individual models used in the two-stage pipeline."Lg" stands for language of the model used.
Comparative results of two-way classification using the finetuned (F.T.) predictions and predictions adapted from three-way classification models.

abstract

Dialect Identification is a crucial task for localizing various Large Language Models. This paper outlines our approach to the VarDial 2023 DSL-TL shared task. Here we have to identify three or two dialects from three languages each which results in a 9-way classification for Track-1 and 6-way classification for Track-2 respectively.
Our proposed approach consists of a two-stage system and outperforms other participants' systems and previous works in this domain. We achieve a score of 58.54% for Track-1 and 85.61% for Track-2. Our codebase is available publicly 1 .

Introduction

Language has been the primary mode of communication for humans since the pre-historic ages. Studies have explored the evolution of language and outlined mathematical models that govern the intricacies of natural language . Inevitably, as humans established civilization in various parts of the world, this language was modified by, and for the group of people occupied by that particular geographical region.
This gave rise to multiple national dialects of the same language. The VarDial workshop (colocated with EACL 2023) explores various dialects and variations of the same language. We participated in the Discriminating Between Similar Languages -True Labels (DSL-TL) shared task. In this task, the participants were provided with data from three languages, with each language having three varieties.
This shared task consisted of two tracks -Track-1 featuring nine-way classification and Track-2 featuring six-way classification. The second track included two particular national dialects of each language (eg. American English and British English), and the first track had one general  We ranked 1 st in both of the tracks.
Moreover, we beat the next best submission by a margin of 4.5% in the first task and 5.6% in the second task.We were the only team to surpass the organizer baseline scores. We present our winning solution in this paper. We used an end-to-end deep learning pipeline which consisted of a language identification model and three language-specific models, one for each language.
We converged upon the best combination by doing an elaborate analysis of various models available. Furthermore, in this work we also analyze the performance of the pipeline as a whole and also provide an ablation study. Lastly, we provide some future directions in this area of research.

Related Work

The present literature encompasses various aspects of dialect identification. We study this from three perspectives: large language models, language identification and dialect classification problems.

Large Language Models

The success of transformers and BERT based models was inevitable since the initial boom of the transformer  2017) model. In recent years, many other architectures like RoBERTa and ELECTRA have further pushed the state-of-the-art in this domain. Moreover, autoregressive models like GPT and GPT-2 have also shown their prowess.
Multilingual versions of RoBERTA, namely XLM-RoBERTa are also available. Lastly, language specific models like Spanish BERT (la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury, 2022) and Portuguese BERT are available as well. Our winning solution makes use of these large language models trained on specific languages.

Language Identification Models

Many multilingual language identification models have been developed in order to classify the language of the input sentence beforehand. Even though the initial works used n-gram models and generative mixture models or even conditional random fields and other classical machine learning methods like naive bayes , modern methods have shifted to the use of deep learning for language identification .
Recent works have mainly focused on deep learning based language identification, where handling codemixed data is a big challenge in the domain. For our experiments, we use a version of XLM-RoBERTa finetuned on a language identification dataset 2 . This model has near-perfect test accuracy of 99.6%.

Dialect Classification

Dialect classification has been previously solved using statistical methods like Gaussian Mixture Models and Frame Selection Decoding or Support Vector Machines (SVM) . It has been explored relatively sparsely, mostly in the case for local languages . Deep learning approaches have been explored in previous editions of the VarDial workshop shared tasks and otherwise .
Dialect classification was also explored previously as a part of other shared tasks . We want to stress that given the multilingual nature of the dataset, using the present methods directly was not an option. In our work, although we take inspiration from the previous works, we propose a novel system that surpasses the performance of the previous systems by a large margin.

Data

The dataset  We observed that the class PT-BR had the most number of samples (2,724) and the class EN had the least number of samples (349), and thus the imbalance ratio was almost 1:8. We have illustrated the data distribution in Figure . We tried to mitigate this imbalance using over-sampling and weighted sampling methods.
However, the improved data sampling method did not affect the performance.

System Description

This was a problem of multi-class classification having 9 classes for Track-1 and 6 classes for Track-2. The samples were belonging to 3 languages having 3 varieties each, so the classification pipeline was made in 2 stages. The Language Identification (LID) model which is the first stage classifies the sentence into 3 languages: English (EN), Spanish (ES) and Portuguese (PT).
The LID is a pretrained XLM-RoBERTa that is fine-tuned for the task of language identification. It is able to classify the input sentence into 20 languages. We classify and separate the samples according to their language. The samples corresponding to the specific languages are then fed into the language specific models for dialect identification.
For dialect identification we have used models like BERT and RoBERTa with a linear layer connected to the pooler output of the models. Then fine-tuning is done on the models for dialect identification   using the samples corresponding to the specific languages. For the task of dialect identification we experimented with several pretrained models like XLM-RoBERTa, BERT, ELECTRA, GPT-2 and RoBERTa.
All models were fine-tuned for 20 epochs with a learning rate of 1e-6 and weight decay 1e-6 with a batch size of 8. The best performing model checkpoint was chosen according to the epoch-wise validation macro-F1 score. 5 Experiments and Results

Experiments using Large Language Models

For the task of Dialect Identification we have tried various language specific models like XLM-RoBERTa, BERT, ELECTRA, RoBERTa and GPT- 2. The base variant of all these models were used and all the models were used through the Hugging-Face library. The pooler output of these models was passed through a linear layer and the models were fine-tuned.
First, we experimented with different models for Track-1. All the models were trained for 20 epochs with learning rate 1e-6, weight decay 1e-6 and a batch size of 8. We used XLM-RoBERTa as the baseline for all 3 languages. The best performing models for the English language were RoBERTa and BERT whereas GPT-2 was the worst performing.
Similarly the language specific versions of RoBERTa and BERT performed well for the Spanish and Portuguese respectively. Overall the worst performing model was GPT-2 across all 3 languages. The validation F1 scores are present in Table . The two best-performing models for every language were chosen for Track-2.
The same procedure as specified above was used and the F1 scores are present in Table . The train and validation F1 scores for 2-class classification are higher for all models as compared to the F1 score of the same models for 3-class classification. This was mainly due to the poor representation and accuracy of classification of the third class.
We observed symptoms of overfitting in all models after 12-15 epochs and the best validation F1 score was obtained in the range of 4-8 epochs.

LID experiments

The pipeline for dialect identification is divided into two parts as the sentences in the dataset belong to different languages. The stages are described in Section 4. The XLM-RoBERTa we have used for language classification has a test accuracy of 99.6% meaning it correctly classifies all input sentences and hence, can be considered as a perfect classifier.
For the final pipeline we experimented using the two best performing models for each language in Track-1 and Track-2. For both the tracks we experimented with all 8 (2 3 ) possible combinations of models and calculated the validation F1 score for the combined validation dataset which had sentences belonging to all languages.
The validation scores for Track-1 and Track-2 are shown in Table and Table respectively. For both the tracks, the three pipelines with the best validation F1 scores were chosen for submission.

Using 3-way classifier as a 2-way classifier

In Track-1, participants are expected to train a classifier which classifies amongst 9 classes, and in Track-2, participants are expected to train a classifier which classifies amongst 6 classes. These 6 classes are a proper subset of the 9 classes from Track-1. Thus, an intuitive baseline for Track-2 is to use the model finetuned for Track-1, whilst considering only the relevant classes for the latter task.
The classes EN , ES and P T , i.e. the classes without any national dialect associated with them are not included in Track-2 as compared to Track-1. Thus, we calculate the predictions for the Track-2 validation dataset using the models for Track-1 and exclude the metrics for Track-1 specific classes to get the metrics for this "adapted" 2-way classification.
We show the results of this experiment in Table and observe that, as expected, the adapted 2-way classification performs worse compared to the explicitly finetuned variant.

Results for Track-1 and Track-2

We now present our experiments and their performance for both tracks. Our experiments for Track-1 are described in Table and our experiments for Track-2 are described in Table . The participants were allowed three submissions for evaluation on the test set, so we submitted predictions using the three systems which performed the best on the validation set.
As mentioned in Section 5.2, we performed 2 3 , i.e. a total of 8 experiments using the two best models for each language. We observed that RoBERTa base on English, Spanish BERT base on Spanish and Portuguese BERT base performed the best on the testing set for Track-1. The same combination, with RoBERTa base for English, worked best for Track-2.
All of our submissions were the top submissions for each track, which surpassed the next best competitors by a margin of 4.5% and 5.6% for Track-1 and Track-2 respectively.

Ablation of best submissions

We hereby make some observations of our submissions and other experiments. To assist this, we plot the confusion matrices of our best submissions for Track-1 and Track-2 in Figures respectively. Note that these confusion matrices have their rows (i.e. true labels axes) normalized according to the number of samples in the class.
Here are observations from our experiments: 1. BERT-based models outperform other models across all languages: We observe that BERT-based models outperform ELECTRA-based and GPT-2-based models, as shown in Table . We speculate this is because of the inherent architecture of BERT, which combines semantic learning with knowledge retention.
This combination of traits is particularly useful for this task. 2. Common labels perform the worst across all languages: We observe that the common labels EN , ES and P T perform the worst, both in the individual as well as the two-stage setup. We hypothesize this is because of the absence of dialect specific words, or words that are specific to the geographical origin of the national dialect (for example, "Yankees" for EN-US and "Oxford" for EN-GB).
3. English models work better than models of other languages: It can be noted from Figures 4 and 3 that the English models have the best performance across all classes. This can be attributed to two reasons: absence of national dialect specific words and lesser pretraining data in the case of Portuguese.
4. British English is most correctly classified class: We can observe that the Spanish or Portuguese models make equal number of mistakes in the case of either national dialect, in the case of Track-2 (see Figure ). However, in the case of English, the label EN − GB is correctly classified for more than 95% of the cases.
We speculate this is because British English involves slightly distinctive grammar and semantics, which help the model separate it from other classes. 5. The proposed 2-step method is scalable for multiple language dialect classification: We can strongly assert that the novel 2-step deep learning method for multilingual dialect classification is a scalable method for the task due to two specific reasons: firstly, the multilingual models (like XLM-RoBERTa) might not have the vocabulary as well as the learning capabilities to learn the minute differences between individual dialects.
Secondly, this system can be quickly expanded for a new language by simply adding a language specific dialect classifier, provided the language identification model supports that particular language.

Conclusion

In this paper we propose a two-stage classification pipeline for dialect identification for multilingual corpora. We conduct thorough ablations on this setup and provide valuable insights. We foresee multiple future directions for this work. The first is to expand this work to many languages and dialects.
Secondly, it is a worthwhile research direction to distill this multi-model setup into a single model with multiple prediction heads. The obvious limitation of this system is the excessive memory consumption due to the usage of language specific models. For low resource languages this system is difficult to train and scale.
We hope that these problems will be addressed by researchers in future works.

Question: What is the score achieved by the authors for Track-2?
Answer: 85.61%.