# Explainable Linguistic Variety Classification: The Case of Bosnian-Croatian-Montenegrin-Serbian

Aleksandra Miletić, Timothee Mickus, Yves Scherrer

► To cite this version:

Aleksandra Miletić, Timothee Mickus, Yves Scherrer. Explainable Linguistic Variety Classification: The Case of Bosnian-Croatian-Montenegrin-Serbian. The 1st ReLDI Conference on Language Science and Technology, ReLDI Centre, Sep 2025, Belgrade (Serbia), Serbia. https://doi.org/10.5281/zenodo.18488312. hal-05505357

HAL Id: hal-05505357

https://hal.science/hal-05505357v1

Submitted on 11 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
EXPLAINABLE LINGUISTIC VARIETY CLASSIFICATION:
THE CASE OF BOSNIAN-CROATIAN-MONTENEGRIN-SERBIAN

Aleksandra MILETIĆ,¹ Timothee MICKUS,² Yves SCHERRER²,³

¹UMR 7114 MoDyCo (CNRS & Paris Nanterre University, France)

²Department of Digital Humanities (University of Helsinki, Finland)

³Department of Informatics (University of Oslo, Norway)

This paper presents a preliminary evaluation and analysis of a method for explainable linguistic variety classification applied to Bosnian, Croatian, Montenegrin, and Serbian. Neural models have become the de-facto standard for most NLP tasks and applications, but the factors influencing their predictions remain to a large extent opaque. Using Twitter data, we aim to detect the linguistic features that contribute most to the classification decisions and conduct an extensive manual analysis of these features. The results show that the global model behaviour of neural classifiers aligns with existing knowledge about variation in these varieties, although the models struggle to identify features specific to a single variety, especially in the case of Bosnian and Montenegrin. Overall, the paper showcases the applicability of the examined explainability method in sociolinguistic contexts characterized by complex variation patterns, thereby demonstrating the interest of explainable machine learning methods for corpus-based variation research.

**Keywords:** language variety classification, explainability, Bosnian-Croatian-Montenegrin-Serbian

# 1 INTRODUCTION

This paper presents an initial evaluation and analysis of explainable classification of linguistic varieties for Bosnian, Croatian, Montenegrin, and Serbian (BCMS). During ex-Yugoslavia, these varieties were considered as belonging to the same standard language (typically referred to as Croato-Serbian or Serbo-Croatian). Since the end of Yugoslavia in the 1990s, national standards have emerged. The status of these varieties with respect to each other is frequently discussed and often understood in academic circles as pertaining to identity construction (Alexander, 2013). And while they are mutually intelligible, differences exist (Ljubešić et al., 2018). This situation makes BCMS a prime candidate for work on distinguishing between similar languages, as attested by its regular presence in the VarDial shared tasks since their very first iteration in 2014 (Zampieri et al., 2014, 2015; Malmasi et al., 2016; Zampieri et al., 2017; Chifu et al., 2024).

Indeed, language identification, and in particular the discrimination between closely related language varieties, has been a relatively well researched topic in recent years (Jauhiainen et al., 2019). However, most work has aimed at improving classification accuracy and at making the experimental setup as realistic as possible, while comparatively less effort has gone into analyzing which features contribute most to the model predictions. Yet such an analysis is relevant both from an NLP point of view, as it increases model interpretability, and from a linguistic point of view, as it can provide new

<figure><img src="image_2.png" /><figcaption>Figure 1: Bosnia and Herzegovina, Croatia, Montenegro, Serbia and neighbouring countries. Map data ©2023 GeoBasis-DE/BKG (©2009), Google.</figcaption></figure>

insights on how language varies across regions. This is particularly relevant for BCMS: descriptions of the linguistic variation exist, but empirical corpus-based work is still sparse (Ljubešić et al., 2018; Miletić & Miletić, 2024).

Although the advent of quantitative, corpus-based methods has enabled linguists to obtain relatively objective, high-level classifications of linguistic areas, tracing back these classifications to individual features (e.g., words) has proven challenging (Leinonen, 2008; Eisen-
stein et al., 2010; Hovy & Purschke, 2018; Kuparinen & Scherrer, 2024). To answer this challenge, we draw on the growing list of work in interpretability and explainability in machine learning, and in particular on explainable dialect classification (Xie et al., 2024). We use these methods to extract the most relevant features for each variety from trained classifiers. We then conduct an extensive manual analysis of these features, classifying them according to the level of linguistic structure concerned by the variation (phonetics, morphology, lexicon, etc.). This allows us to examine the relationship between different feature categories and individual varieties, making it possible to compare model behaviour to the results from previous empirical studies on variation in BCMS.<fnref n="1" />

## 2 RELATED WORK

### 2.1 BCMS and variation

Linguistic variation between Bosnian, Croatian, Montenegrin and Serbian concerns all levels of linguistic structure: phonetic, phonological, morpho-syntactic, and lexical, and even orthographic conventions can vary. Ljubešić et al. (2018) conducted a corpus-based study of BCMS in which they examined the geographical spread of 16 linguistic variables on phonetic, morphosyntactic and lexical levels. The results situate Croatian and Serbian at the opposing ends of the continuum, whereas Bosnian and Montenegrin tend to align with the one or the other depending on the variable.

Variation illustrations are provided in Figure 2 (extracted from the corpus used in this study; see Section 3). The lexical item *sevap* 'good deed' is typical of Bosnian, and the grapheme *š* is only used in Montenegrin. Other features can have an asymmetrical spread: the short form of the infinitive *našutat* 'to kick' in the Bosnian example is also found in Croatian and Montenegrin, whereas for Serbian the long form *našutati* would be expected (Ljubešić et al., 2018, p. 18-19). This is also the case for the most salient form of variation between these varieties: the difference between the ekavian and the ijekavian pronunciation, reflected by the *e:ije* variation. Serbian is the only of the four national standards based on both pronunciations. Therefore the presence of an ekavian form would be strongly indicative of Serbian. This is instantiated in the word *svet* 'world' in the Serbian example; in Bosnian, Croatian and Montenegrin, the ijekavian equivalent *svijet* would be expected as more frequent (Ljubešić et al., 2018, p. 18).

Ljubešić et al. (2018) also note the asymmetry in the spread of different variables, which can make some

<table><caption>Figure 2: Examples of distinctive features for language variety identification. The coloured characters and words indicate features that are specific to the variety in parentheses. Red: morphological; blue: phonetic/phonological; violet: lexical; green: spelling; orange: named entities.</caption><tr><td><strong>Bosnian:</strong></td><td>Neke osobe je <span style="color:purple;">sevap našutat</span>'<br>'It's a good deed to kick certain people'</td></tr><tr><td><strong>Croatian:</strong></td><td>"to je <span style="color:purple;">nogomet</span>" um ne, to nije <span style="color:purple;">vidlo</span> n od <span style="color:purple;">nogometa</span><br>"that's football" um no, not even close'</td></tr><tr><td><strong>Montenegrin:</strong></td><td>A kako <span style="color:purple;">š</span>utra ima da pane jedna četvorka iz biologije!<br>'I'm so getting a B in biology tomorrow!'</td></tr><tr><td><strong>Serbian:</strong></td><td><span style="color:orange;">beograd</span> je svet, a?<br>'belrade is the world, huh?'</td></tr></table>

of the four varieties harder to identify than others. This seems to be the case both for humans and for NLP systems. Miletić and Miletić (2024) report higher levels of human annotator uncertainty when identifying instances from Bosnia-Herzegovina and Montenegro compared to Croatia and Serbia. And Rupnik et al. (2023) note that their classifiers perform well on Serbian (1.0 micro F1) and Croatian (0.83), but the results are lower on Bosnian (0.75–0.80), and they drop significantly on Montenegrin (0.10–0.36).

In this light, our experiments aim to answer the following questions:

* How easy/hard is it to identify features relevant for different varieties with our approach?

* Is it possible to identify features that are uniquely relevant to a single variety?

* Do the examined models rely on all levels of variation equally or do they favour some of them?

### 2.2 Explainable classification of linguistic varieties

The ever widening use of neural network-based architectures and large language models in NLP has led to an increasing need for transparency and understandability of these models. Initial work included building an interpretable model around the prediction model in order to provide insights into classifier predictions (Ribeiro et al., 2016), introducing representation erasure (Li et al., 2017), and examining the role of convolutional neural network filters (Jacovi et al., 2018). Belinkov et al. (2020) proposed a categorization of explainability research including the following categories: classifier probing, behavioural studies, test suite development, and interactive visualization creation.

<footnote n="1">¹Our data and code is available at https://github.com/Helsinki-NLP/explainable-bcms-classification</footnote>
Two recent papers specifically explore the idea of applying methods from explainability research to linguistic variety classification. Rönnqvist et al. (2022) focus on the task of identifying registers in web texts and automatically extracting the most characteristic keywords of each register. They train an XLM-R classifier on the register identification task and use the Integrated Gradients (IG) method (Sundararajan et al., 2017) to determine which individual words contribute most to the classification. The extracted keywords are then aggregated over instances and over several independent runs. Xie et al. (2024) work with binary dialect identification tasks: Mainland vs. Taiwan Mandarin and Sicilian vs. Italian. They use a different explainability method called *leave-one-out* (LOO). Just like IG, LOO is a post-hoc approach, i.e., it operates at inference time, on a normally trained dialect classification model. It works by removing one feature at a time from the input and interpreting the change in prediction probability as a *relevance score* of that particular feature.

In this work, we adopt the LOO method for the following reasons:

* The logic behind the approach is transparent and it aligns with the expectation that more distinctive features should have a stronger impact on the classifier;

* The methodology was reasonably well documented and straightforward to implement;

* The output is readily interpretable by human annotators.

## 3 DATASET

As mentioned above, the socio-linguistic reality of BCMS is complex. As a quick illustration, in the 2023 Montenegro census, 43.2% of citizens identified as speakers of Serbian, 34.5% as speakers of Montenegrin, and 7.3% as speakers of Bosnian.<fnref n="2" /> And in the 2013 Bosnia-Herzegovina census, 52.9% of citizens identified as speakers of Bosnian, 30.8% as speakers of Serbian, and 14.6% as speakers of Croatian.<fnref n="3" /> In order to fully capture the spread and the properties of these varieties, a comprehensive, large-scale sociolinguistic corpus would be needed. Unfortunately, to the best of our knowledge no such resource is currently available. For the purposes of this study, we therefore consider Bosnian, Croatian, Montenegrin and Serbian as loosely corresponding to the variety spoken in the respective country and influenced by the national standardization efforts throughout the past decades. This aligns with the approach adopted by Ljubešić et al. (2018), who examine the

spread of linguistic variables within the four countries, thus allowing us to draw comparisons where relevant. There is also a quantitative argument to be made for this approach: one can reasonably expect a national standard to affect linguistic practices in its respective country, and thereby it can be expected that e.g. the Bosnian-Herzegovinian subset of our data should largely map onto the Bosnian national standard. This is, of course, mainly an approximation of a more complex linguistic landscape, and one that need not be entirely appropriate. The results presented here should be interpreted in the same light.

We rely on a dataset proposed by Rupnik et al. (2023), whose various iterations have been regularly used in the NLP domain for the task of distinguishing between similar languages (Zampieri et al., 2014, 2015; Malmasi et al., 2016; Zampieri et al., 2017; Chifu et al., 2024). The data was collected from the social media platform Twitter (rebranded as X in 2023). The original annotation of linguistic varieties was done in several steps. In the first two steps, users susceptible to tweet in BCMS were identified based on variety-specific word lists containing highly frequent items, and subsequently language identification tools, including ones specifically aimed at BCMS, were used (Ljubešić et al., 2014). The final annotation into the four varieties was executed by a single human annotator, with a single label given to each Twitter user (Ljubešić & Kranjčić, 2015).

In this study, we use a revised version of the dataset, in which the *dev* and *test* sets were re-annotated in a multi-label setting, by multiple annotators speaking all four varieties (Miletić & Miletić, 2024). Thus the evaluation sets used here reflect possible ambiguities and overlaps between the four varieties.

The original dataset exhibited a strong skew towards Serbian. We therefore downsized this class to the next largest class in the dataset (Croatian). In a second step, we modified the size of instances: originally, they corresponded to the full tweet production of a single user. For the purposes of our experiments, we segmented each instance into sentences and projected the label of the full instance onto each of the sentences. Since the resulting annotation has not been validated manually, it is possible that instances not carrying clear markers of a given variety still carry the corresponding label. We further discuss this in Section 5.

A quantitative overview of the dataset is available in Table 1. For ease of presentation, we count multilabel instances in the *test* and *dev* sets as belonging to each of the classes for which they have been annotated. Details on multilabel instances can be found in Appendix A.

<footnote n="2">²https://www.monstat.org/uploads/files/popis%202021/saopstenja/SAOPSTENJE_Popis%20stanovnistva%202023%20II_cg.pdf</footnote>

<footnote n="3">³http://www.statistika.ba/?show=8#link4</footnote>
<table><caption>Table 1: Dataset overview. Sizes given as number of instances.</caption><thead><tr><th>Classes</th><th>Train</th><th>Dev</th><th>Test</th><th>Subtotal</th></tr></thead><tbody><tr><td>Bosnian</td><td>16,463</td><td>7,610</td><td>7,606</td><td>31,679</td></tr><tr><td>Croatian</td><td>23,691</td><td>11,141</td><td>11,310</td><td>46,142</td></tr><tr><td>Montenegrin</td><td>13,155</td><td>4,488</td><td>8,513</td><td>26,156</td></tr><tr><td>Serbian</td><td>23,784</td><td>13,076</td><td>13,431</td><td>50,291</td></tr><tr><td>Total</td><td>77,093</td><td>36,315</td><td>40,860</td><td>154,268</td></tr></tbody></table>

# 4 METHODOLOGY

Our experimental setup involves three steps:

1. We train models on the linguistic variety classification task.

2. We extract variety-specific features from the trained models using the Leave-one-out approach.

3. We manually annotate the extracted features and assess their relevance.

Given that finetuned BERT models have been shown to be well-suited to the task of distinguishing between similar varieties (Gaman et al., 2020; Chifu et al., 2024), we train two BERT-based classifiers: a multilingual XLM-RoBERTa model (Conneau et al., 2020) and a language-specific BERTić (Ljubešić & Lauc, 2021). As a baseline, we use a logistic regression model from which we directly extract the most relevant features per class.

## 4.1 Logistic regression

**Training** We use a simple logistic regression model, as provided by the scikit-learn library, with 500 iterations, integers set to numpy.int8 to save memory, and otherwise default parameters. We encode the text samples as bag-of-words count vectors without any weighting or pruning. Also, we do not perform any subword segmentation.

**Feature extraction** For each class, we identify the 30 features (words) with the highest weight coefficients in the trained model. This approach only yields features that were present in the training set. For comparability with the BERT-based approach, we only retain those features that also appear in the test set.

## 4.2 BERT-based models

**Training** We perform supervised fine-tuning on the training data for 10 epochs and select the best model checkpoint based on validation accuracy.

**Feature extraction** We apply the leave-one-out (LOO) method proposed by Xie et al. (2024). This method processes each sentence of the test set independently to

detect the features that contribute most to the classification decision. It consists of the following steps:

1. Take a test instance $x$, pass it through the classifier, and keep both the predicted class $\hat{y}$ and the prediction score $\ell$. If the classifier's prediction is incorrect ($\hat{y} \neq y$), move on to the next instance.<fnref n="4" />

2. Select and remove a word from the test instance (we denote this modified instance as $x_i$, where word $i$ has been removed), run it through the classifier again, and record the new prediction score $\ell_i$.<fnref n="5" />

3. Calculate how much removing the word affects the classifier's performance by subtracting the new score from the original score: $\Delta_i = \ell - \ell_i$. We refer to $\Delta_i$ as the *impact score* for word $i$.

4. Repeat steps 2 and 3 for every word in the instance.

5. Identify the 5 words with the highest impact scores.<fnref n="6" />

The steps outlined above generate instance-level explanations, identifying the words that have the greatest influence on each individual instance.

We aggregate the explanations across all instances of the test set for evaluation. The resulting list is processed in the following way:

1. For each extracted item, we compute the average impact score on the basis of the individual impact scores.<fnref n="7" />

2. For each class, we select the 100 items with the highest average impact scores for the analysis. This means that for each model, a total of 400 features are examined.

3. The extraction is done in two ways: 1) by selecting the items that were extracted only for a single variety (denoted SV in Table 3), and 2) by allowing for items that were extracted for several classes but not all of them (denoted MV in Table 3). This is motivated by the observation that linguistic features often do not have the same spread across the linguistic varieties discussed in Section 2.1.

<footnote n="4">⁴This is equivalent to the *isCorrect* condition described by Xie et al. (2024).</footnote>

<footnote n="5">⁵Xie et al. (2024) remove words based on their position in the sentence, meaning if a word appears multiple times, only one occurrence is removed at a time. Their method focuses on the occurrence that leads to the largest score change. Instead, we remove all occurrences of the word simultaneously.</footnote>

<footnote n="6">⁶While Xie et al. (2024) do not mention this threshold in their written methodology, it is included in their implementation.</footnote>

<footnote n="7">⁷Xie et al. (2024) use TF-IDF to rank the words. We find that the simpler approach of averaging the scores is sufficient for our purposes.</footnote>
<table><caption>Table 2: Classification accuracies across models.</caption><thead><tr><td>Model</td><td>Accuracy</td></tr></thead><tbody><tr><td>LR</td><td>52.3</td></tr><tr><td>XLM-R</td><td>68.0</td></tr><tr><td>BERTić</td><td>73.1</td></tr></tbody></table>

# 5 RESULTS AND DISCUSSION

## 5.1 Linguistic variety classification

Table 2 shows the classification accuracies obtained by the logistic regression baseline and the BERT-based classifiers. Unsurprisingly, the neural models outperform the baseline by a wide margin, and the language-specific BERTić beats multilingual XLM-R. However, the performances of neural models remain relatively low (73.1 and 68.0 accuracy points respectively). This could be due to the fact that the original corpus instances were split into sentences, and the variety label attributed to the original instance was projected onto all resulting sentences. This can in fact lead to having instances that do not carry specific markers, but which are attributed to a single variety.

## 5.2 Variety-specific feature extraction

Both logistic regression and leave-one-out classification yield lists of variety-specific linguistic features. These features were manually annotated as belonging to *Lexicon*, *Phonetics/phonology*, *Morphology*, *Named Entities*, *Spelling*, *Other* or *Unmarked*, the last one indicating items that do not contain any variety-specific information. The annotation was done by a single annotator, who is a linguist and a native speaker of Serbian. The results of the annotation are shown in Table 3.

### 5.2.1 GLOBAL ANALYSIS

We examine first the general usefulness of the extracted features. The main goal of our setup is to identify marked features that are specific to one or several varieties. The presence of unmarked features is therefore undesirable and these features can be seen as errors. The LR model presents a relatively low percentage of unmarked features in the analyzed list, whereas they systematically represent >10% of the top 100 features for the neural models (e.g. *volim* 'I love' extracted from BERTić for Serbian, *zna* 'he/she knows' extracted from XLM-R for Bosnian). In this respect, they seem less reliable than the LR model.

On the other hand, the LR model has a much higher prevalence of features in the *Other* category compared to the neural models. This category contains mostly corpus artifacts: occasional words in other languages, hash tags, URLs and user names. 57.5% of the analyzed features from the LR model are therefore irrelevant from the linguistic point of view.

Similarly, although to a lesser extent, the BERT-based models show a significant reliance on named entities. Our manual analysis shows that these are often related to geographic terms specific to a single country (e.g. *Podgoričani* (inhabitants of Podgorica, the capital of Montenegro) extracted from BERTić for Montenegrin; *Mostar* (city in Bosnia and Herzegovina) extracted from XLM-R for Bosnian, etc.) and therefore conflate regional topical differences with variation pertaining to linguistic structures.

It is also notable that the behaviour of the neural models is comparable whether we take into account features specific to a single variety (rows marked SV in Table 3) or those that were extracted for multiple varieties (but not all of them; rows marked MV). For instance, the difference between the two extraction scenarios for the phonetic features is 2.5% for BERTić and 1.5% for XLM-R.

Among the extracted linguistic features, the most prominent category concerns *Phonetics/phonology*, independently of the model and of the extraction mode. It is followed by *Morphology* and *Lexicon*. We take a closer look at the distribution of these across varieties and discuss salient examples.

### 5.2.2 ANALYSIS BY VARIETY AND BY FEATURE CATEGORY

The distribution of analyzed features by feature category is presented in Table 4.

The overall frequency of phonetic and phonological features seems to be driven by Serbian: this category covers 64% of the analyzed features extracted from BERTić, and 57% of those extracted from XLM-R. For the other three varieties, this percentage is much lower (12-24%). All of the Serbian features from this category are ekavian forms having ijekavian equivalents in the other three varieties (cf. *lep* 'beautiful', which would be expected as *lijep* in Croatian, Bosnian and Montenegro). Conversely, the features extracted for this level for the other three varieties predominantly contain ijekavian forms having ekavian equivalents, but there are also some more specific features. For Croatian, both models propose inflected forms of *kava* 'coffee' (the expected form in Serbian being *kafa*, while Bosnian also allows *kahva*). Also, both models identify the form *de* 'where' for Montenegrin (whereas *gdje* would be expected in Croatian and Bosnian, and *gde* in Serbian).

The most morphological features were found in the lists for Croatian, independently of the model. They overwhelmingly concern the use of the short infinitive, such as *dobit* 'to receive' and *doć* 'to come' (the corresponding long forms being *dobiti* and *doći*). However,
<table><caption>Table 3: Leave-one-out result annotation. Percentage of extracted features per category. Individual values may not sum to totals due to rounding.</caption><thead><tr><th>Model</th><th>Scenario</th><th>Phon.</th><th>Morphology</th><th>Lexicon</th><th>Named Entities</th><th>Spelling</th><th>Other</th><th>Unmarked</th></tr></thead><tbody><tr><td rowspan="2">LR</td><td>MV</td><td>27.5</td><td>0.0</td><td>6.7</td><td>1.7</td><td>2.5</td><td>57.5</td><td>4.2</td></tr><tr><td>SV</td><td>29.5</td><td>6.0</td><td>8.3</td><td>15.5</td><td>0.5</td><td>27.3</td><td>13.0</td></tr><tr><td rowspan="2">BERTić</td><td>MV</td><td>27.0</td><td>4.3</td><td>10.8</td><td>18.0</td><td>0.5</td><td>28.3</td><td>11.3</td></tr><tr><td>SV</td><td>25.8</td><td>4.8</td><td>6.3</td><td>16.0</td><td>0.8</td><td>34.5</td><td>12.0</td></tr><tr><td>XLM-R</td><td>MV</td><td>27.3</td><td>5.3</td><td>6.5</td><td>12.0</td><td>0.8</td><td>29.3</td><td>19.0</td></tr></tbody></table>

<table><caption>Table 4: Features extracted for **multiple varieties**: Distribution per feature category and linguistic variety.</caption><thead><tr><th></th><th></th><th>Phon.</th><th>Morphology</th><th>Lexicon</th><th>Named Entities</th><th>Spelling</th><th>Other</th><th>Unmarked</th></tr></thead><tbody><tr><td rowspan="4">BERTić</td><td>Bosnian</td><td>15.0</td><td>0.0</td><td>6.0</td><td>28.0</td><td>2.0</td><td>37.0</td><td>12.0</td></tr><tr><td>Croatian</td><td>17.0</td><td>14.0</td><td>19.0</td><td>9.0</td><td>0.0</td><td>24.0</td><td>17.0</td></tr><tr><td>Montenegrin</td><td>22.0</td><td>8.0</td><td>4.0</td><td>21.0</td><td>0.0</td><td>22.0</td><td>23.0</td></tr><tr><td>Serbian</td><td>64.0</td><td>2.0</td><td>4.0</td><td>4.0</td><td>0.0</td><td>26.0</td><td>0.0</td></tr><tr><td rowspan="4">XLM-R</td><td>Bosnian</td><td>16.0</td><td>0.0</td><td>8.0</td><td>27.0</td><td>0.0</td><td>26.0</td><td>23.0</td></tr><tr><td>Croatian</td><td>12.0</td><td>19.0</td><td>5.0</td><td>7.0</td><td>1.0</td><td>39.0</td><td>17.0</td></tr><tr><td>Montenegrin</td><td>24.0</td><td>0.0</td><td>4.0</td><td>11.0</td><td>2.0</td><td>33.0</td><td>26.0</td></tr><tr><td>Serbian</td><td>57.0</td><td>2.0</td><td>9.0</td><td>3.0</td><td>0.0</td><td>19.0</td><td>10.0</td></tr></tbody></table>

some of the infinitives also reflect a difference in the derivation patterns of some verbs, which contrast verbs ending in -irati in Croatian with verbs ending in -ovati in Serbian (cf. *organizirat* ‘to organize’; the expected form in Bosnian, Montenegrin and Serbian would be *organizovati*; Ljubešić et al., 2018, p. 16). Lexical items mostly reflect terms for which a different root is used in Serbian (cf. *dojam* ‘impression’ and *otok* ‘island’ in Croatian, whereas *utisak* and *ostrvo* would be expected in Serbian; Nikolić, 2011).

The features extracted for Bosnian and Montenegrin present similar distributions: the bulk of linguistically relevant features rely on phonetics and phonology, there are relatively few features pertaining to morphology and lexicon, and there is higher reliance on named entities compared to Croatian and Serbian. The models do manage to identify *raja* ‘people, folks’ as a lexical item specific to Bosnian, and XLM-R picks up on *profesorica* ‘professor (feminine)’ for Montenegrin (expected form in Serbian: *profesorka*, whereas *profesorica* would be expected both in Bosnian and Croatian; Ljubešić et al., 2018, p. 18).

These results align with previous observations on the relationship between these four varieties. Croatian and Serbian seem the most distinctive when looking at the linguistic features. The classification of Serbian relies heavily on the *e:ije* variation thanks to the asymmetrical spread of the ekavian forms. On the other hand, the relevance of lexical and morphological features for the classification of Croatian may be due to the fact that the phonetic features are not sufficiently distinctive (given

that Bosnian and Montenegrin are also ijekavian). The fact that there are fewer morphological and lexical features detected for Bosnian and Montenegrin aligns with the findings by Ljubešić et al. (2018), discussed in Section 2.1. Globally, our analysis reveals that the model performance seems to align with previous research on these varieties.

We also examine extraction results when only features relevant for a single variety are allowed. This is motivated by the fact that such features would be of particular value for automatic variety identification. In this scenario, the models do not fare well. Given the ambiguity of ijekavian forms between Croatian, Bosnian and Montenegrin, in this scenario we would expect less of a reliance on phonetic and phonological features for these three varieties. But when looking at the results in Table 5, we see only a slight dip for Montenegrin and Croatian, with a slightly more pronounced effect on Bosnian compared to results in Table 4. For BERTić, dips are observable too for the morphological features. Conversely, there is a slight increase in the lexical feature count, but the category that gains the most features in this setup is named entities. This may be an indication that phonetic and morphological features specific to only one of these three varieties are hard to identify. This is further underscored by the fact that the extracted features are mostly the same as the ones extracted in the more relaxed scenario allowing for features relevant for multiple varieties. The vast majority of them are not specific to a single variety.
<table><caption>Table 5: Features extracted for a **single variety**. Distribution per feature category and linguistic variety.</caption><thead><tr><th></th><th></th><th>Phon.</th><th>Morphology</th><th>Lexicon</th><th>NE</th><th>Spelling</th><th>Other</th><th>Unmarked</th></tr></thead><tbody><tr><td rowspan="4">BERTIĆ</td><td>Bosnian</td><td>7.0</td><td>0.0</td><td>8.0</td><td>34.0</td><td>2.0</td><td>36.0</td><td>13.0</td></tr><tr><td>Croatian</td><td>16.0</td><td>11.0</td><td>20.0</td><td>14.0</td><td>0.0</td><td>26.0</td><td>13.0</td></tr><tr><td>Montenegrin</td><td>20.0</td><td>5.0</td><td>11.0</td><td>20.0</td><td>0.0</td><td>25.0</td><td>19.0</td></tr><tr><td>Serbian</td><td>65.0</td><td>1.0</td><td>4.0</td><td>4.0</td><td>0.0</td><td>26.0</td><td>0.0</td></tr><tr><td rowspan="4">XLM-R</td><td>Bosnian</td><td>5.0</td><td>0.0</td><td>7.0</td><td>36.0</td><td>0.0</td><td>41.0</td><td>11.0</td></tr><tr><td>Croatian</td><td>10.0</td><td>19.0</td><td>6.0</td><td>10.0</td><td>1.0</td><td>40.0</td><td>14.0</td></tr><tr><td>Montenegrin</td><td>25.0</td><td>0.0</td><td>3.0</td><td>16.0</td><td>2.0</td><td>39.0</td><td>15.0</td></tr><tr><td>Serbian</td><td>63.0</td><td>0.0</td><td>9.0</td><td>2.0</td><td>0.0</td><td>18.0</td><td>8.0</td></tr></tbody></table>

# 6 CONCLUSION

In this paper, we presented an initial evaluation and analysis of the Leave-one-out method for explainable lingusitic variety classification, proposed by Xie et al. (2024), on Bosnian, Croatian, Montenegrin, and Serbian. This consisted in training two BERT-based classifiers, using the Leave-one-out method to extract features that contribute most to classification decisions, and manually analysing the list of extracted features.

Our results indicate that the model behaviour generally aligns with existing linguistic knowledge about variation between these four varieties: classification of Serbian relies heavily on the asymmetry of the *e:ije* variation and exploits the presence of ekavian forms; the feature list for Croatian contains a more even spread of phonetic, morphological and lexical features, with the morphological features capturing the variation between the short and the long infinitive, and some verbal derivation regularities; comparatively few morphological and lexical features are extracted for Bosnian and Montenegrin, which instead exhibit a higher reliance on named entities. All of these observations are in line with conclusions put forward by Ljubešić et al. (2018), which point to Croatian and Serbian as being at the opposing ends of the continuum, whereas Bosnian and Montenegrin tend to align with the one or the other, and display fewer features specific only to them.

Overall, the paper showcases the applicability of the chosen explainability method in sociolinguistic contexts characterized by complex patterns of variation, and thereby demonstrates the interest of explainable machine learning methods for corpus-based variation research.

# 7 LIMITATIONS

The work presented above has several limitations worth discussing. First, we evaluated a single explainability method. In future work, we will extend this to a wider selection of approaches. Second, the qualitative analysis is based on a manual annotation produced by a single

human annotator. The difficulty of the task reported by the annotator seems to justify the addition of a second annotator and an evaluation of inter-annotator agreement. Third, a quantitative evaluation of extracted features against ground truth would complement the current evaluation method. Since wide-coverage lexicons are available for Croatian and Serbian, we will implement this in our future work.

# REFERENCES

Alexander, R. (2013). Language and identity: The fate of serbo-croatian. In *Entangled histories of the balkans-volume one* (pp. 341–417). Brill.

Belinkov, Y., Gehrmann, S., & Pavlick, E. (2020, July). Interpretability and analysis in neural NLP. In A. Savary & Y. Zhang (Eds.), *Proceedings of the 58th annual meeting of the association for computational linguistics: Tutorial abstracts* (pp. 1–5). Online: Association for Computational Linguistics. https://aclanthology.org/2020.acl-tutorials.1/ doi: 10.18653/v1/2020.acl-tutorials.1

Chifu, A.-G., Glavaš, G., Ionescu, R. T., Ljubešić, N., Miletić, A., Miletić, F., ... Vulić, I. (2024, June). VarDial evaluation campaign 2024: Commonsense reasoning in dialects and multi-label similar language identification. In Y. Scherrer, T. Jauhiainen, N. Ljubešić, M. Zampieri, P. Nakov, & J. Tiedemann (Eds.), *Proceedings of the eleventh workshop on nlp for similar languages, varieties, and dialects (vardial 2024)* (pp. 1–15). Mexico City, Mexico: Association for Computational Linguistics. https://aclanthology.org/2024.vardial-1.1/ doi: 10.18653/v1/2024.vardial-1.1

Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., ... Stoyanov, V. (2020, July). Unsupervised cross-lingual representation learning at scale. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), *Proceedings of the 58th annual meeting of the association for computational linguistics* (pp. 8440–8451). Online: Association for Computational Linguistics. https://aclanthology.org/2020.acl-main.747/ doi: 10.18653/v1/2020.acl-main.747

Eisenstein, J., O'Connor, B., Smith, N. A., & Xing, E. P. (2010, October). A latent variable model for geographic lexical variation. In H. Li & L. Màrquez (Eds.), *Proceedings*
of the 2010 conference on empirical methods in natural language processing (pp. 1277–1287). Cambridge, MA: Association for Computational Linguistics. https://aclanthology.org/D10-1124/

Gaman, M., Hovy, D., Ionescu, R. T., Jauhiainen, H., Jauhiainen, T., Lindén, K., ... Zampieri, M. (2020, December). A report on the VarDial evaluation campaign 2020. In M. Zampieri, P. Nakov, N. Ljubešić, J. Tiedemann, & Y. Scherrer (Eds.), *Proceedings of the 7th workshop on nlp for similar languages, varieties and dialects* (pp. 1–14). Barcelona, Spain (Online): International Committee on Computational Linguistics (ICCL). https://aclanthology.org/2020.vardial-1.1/

Hovy, D., & Purschke, C. (2018, October-November). Capturing regional variation with distributed place representations and geographic retrofitting. In E. Riloff, D. Chiang, J. Hockenmaier, & J. Tsujii (Eds.), *Proceedings of the 2018 conference on empirical methods in natural language processing* (pp. 4383–4394). Brussels, Belgium: Association for Computational Linguistics. https://aclanthology.org/D18-1469 doi: 10.18653/v1/D18-1469

Jacovi, A., Sar Shalom, O., & Goldberg, Y. (2018). Understanding convolutional neural networks for text classification. In *Proceedings of the 2018 emnlp workshop blackboxnlp: Analyzing and interpreting neural networks for nlp* (p. 56–65). Brussels, Belgium: Association for Computational Linguistics. http://aclweb.org/anthology/W18-5408 doi: 10.18653/v1/W18-5408

Jauhiainen, T., Lui, M., Zampieri, M., Baldwin, T., & Lindén, K. (2019). Automatic language identification in texts: A survey. *Journal of Artificial Intelligence Research, 65*, 675–782. https://doi.org/10.1613/jair.1.11675

Kuparinen, O., & Scherrer, Y. (2024, May 20). Corpus-based dialectometry with topic models. *Journal of linguistic geography*. doi: 10.1017/jlg.2024.6

Leinonen, T. (2008). Factor analysis of vowel pronunciation in Swedish dialects. *International Journal of Humanities and Arts Computing, 2*(1-2), 189-204. https://doi.org/10.3366/E175385480900038X doi: 10.3366/E175385480900038X

Li, J., Monroe, W., & Jurafsky, D. (2017). Understanding neural networks through representation erasure. https://arxiv.org/abs/1612.08220

Ljubešić, N., Fišer, D., & Erjavec, T. (2014, May). TweetCaT: a tool for building Twitter corpora of smaller languages. In N. Calzolari et al. (Eds.), *Proceedings of the ninth international conference on language resources and evaluation (LREC’14)* (pp. 2279–2283). Reykjavik, Iceland: European Language Resources Association (ELRA). https://aclanthology.org/L14-1642/

Ljubešić, N., & Kranjčić, D. (2015). Discriminating between closely related languages on twitter. *Informatica, 39*(1).

Ljubešić, N., & Lauc, D. (2021, April). BERTić - the transformer language model for Bosnian, Croatian, Montenegrin and Serbian. In B. Babych et al. (Eds.), *Proceedings of the 8th workshop on balto-slavic natural language processing* (pp. 37–42). Kiyv, Ukraine: Association for Computa-

tional Linguistics. https://aclanthology.org/2021.bsnlp-1.5/

Ljubešić, N., Petrović, M. M., & Samardžić, T. (2018). Borders and boundaries in Bosnian, Croatian, Montenegrin and Serbian: Twitter data to the rescue. *Journal of Linguistic Geography, 6*(2), 100–124.

Malmasi, S., Zampieri, M., Ljubešić, N., Nakov, P., Ali, A., & Tiedemann, J. (2016, December). Discriminating between similar languages and Arabic dialect identification: A report on the third DSL shared task. In P. Nakov, M. Zampieri, L. Tan, N. Ljubešić, J. Tiedemann, & S. Malmasi (Eds.), *Proceedings of the third workshop on NLP for similar languages, varieties and dialects (VarDial3)* (pp. 1–14). Osaka, Japan: The COLING 2016 Organizing Committee. https://aclanthology.org/W16-4801/

Miletić, A., & Miletić, F. (2024, May). A gold standard with silver linings: Scaling up annotation for distinguishing Bosnian, Croatian, Montenegrin and Serbian. In S. Balloccu, A. Belz, R. Huidrom, E. Reiter, J. Sedoc, & C. Thomson (Eds.), *Proceedings of the fourth workshop on human evaluation of nlp systems (humeval) @ Irecoling 2024* (pp. 36–46). Torino, Italia: ELRA and ICCL. https://aclanthology.org/2024.humeval-1.4/

Nikolić, M. (Ed.). (2011). *Rečnik srpskog jezika*. Matica srpska.

Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why should I trust you?": Explaining the predictions of any classifier. In *Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining* (p. 1135–1144). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2939672.2939778 doi: 10.1145/2939672.2939778

Rönnqvist, S., Kyröläinen, A.-J., Myntti, A., Ginter, F., & Laip-pala, V. (2022, May). Explaining classes through stable word attributions. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), *Findings of the association for computational linguistics: Acl 2022* (pp. 1063–1074). Dublin, Ireland: Association for Computational Linguistics. https://aclanthology.org/2022.findings-acl.85/ doi: 10.18653/v1/2022.findings-acl.85

Rupnik, P., Kuzman, T., & Ljubešić, N. (2023, May). BENCHić-lang: A benchmark for discriminating between Bosnian, Croatian, Montenegrin and Serbian. In Y. Scherrer, T. Jauhiainen, N. Ljubešić, P. Nakov, J. Tiedemann, & M. Zampieri (Eds.), *Tenth workshop on nlp for similar languages, varieties and dialects (vardial 2023)* (pp. 113–120). Dubrovnik, Croatia: Association for Computational Linguistics. https://aclanthology.org/2023.vardial-1.11/ doi: 10.18653/v1/2023.vardial-1.11

Sundararajan, M., Taly, A., & Yan, Q. (2017, 06–11 Aug). Axiomatic attribution for deep networks. In D. Precup & Y. W. Teh (Eds.), *Proceedings of the 34th international conference on machine learning* (Vol. 70, pp. 3319–3328). PMLR. https://proceedings.mlr.press/v70/sundararajan17a.html

Xie, R., Ahia, O., Tsvetkov, Y., & Anastasopoulos, A. (2024, February). Extracting lexical features from dialects via interpretable dialect classifiers. *Proceedings of the 2024 Conference of the North American Chapter of the As-
sociation for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), 54–69. (arXiv:2402.17914 [cs]) doi: 10.18653/v1/2024.nacl -short.5

Zampieri, M., Malmasi, S., Ljubešić, N., Nakov, P., Ali, A., Tiedemann, J., ... Aepli, N. (2017, April). Findings of the VarDial evaluation campaign 2017. In P. Nakov, M. Zampieri, N. Ljubešić, J. Tiedemann, S. Malmasi, & A. Ali (Eds.), *Proceedings of the fourth workshop on NLP for similar languages, varieties and dialects (VarDial)* (pp. 1–15). Valencia, Spain: Association for Computational Linguistics. https://aclanthology.org/W17-1201/ doi: 10.18653/v1/W17-1201

Zampieri, M., Tan, L., Ljubešić, N., & Tiedemann, J. (2014, August). A report on the DSL shared task 2014. In M. Zampieri, L. Tan, N. Ljubešić, & J. Tiedemann (Eds.), *Proceedings of the first workshop on applying NLP tools to similar languages, varieties and dialects* (pp. 58–67). Dublin, Ireland: Association for Computational Linguistics and Dublin City University. https://aclanthology .org/W14-5307/ doi: 10.3115/v1/W14-5307

Zampieri, M., Tan, L., Ljubešić, N., Tiedemann, J., & Nakov, P. (2015, September). Overview of the DSL shared task 2015. In P. Nakov et al. (Eds.), *Proceedings of the joint workshop on language technology for closely related languages, varieties and dialects* (pp. 1–9). Hissar, Bulgaria: Association for Computational Linguistics. https://aclanthology.org/W15-5401/

A MULTILABEL INSTANCES IN DEV AND TEST SETS

<caption>Table 6: Overview of multilabel instances in dev and test set.</caption>

BS: Bosnian, HR: Croatian, MN: Montenegrin, SR: Serbian.

<table>
  <thead>
    <tr>
      <th>Classes</th>
      <th>Dev</th>
      <th>Test</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>BS, HR</td>
      <td>1,473</td>
      <td>500</td>
    </tr>
    <tr>
      <td>BS, HR, MN</td>
      <td>0</td>
      <td>51</td>
    </tr>
    <tr>
      <td>BS, HR, MN, SR</td>
      <td>119</td>
      <td>0</td>
    </tr>
    <tr>
      <td>BS, MN</td>
      <td>1,128</td>
      <td>2,029</td>
    </tr>
    <tr>
      <td>BS, MN, SR</td>
      <td>0</td>
      <td>1,928</td>
    </tr>
    <tr>
      <td>BS, SR</td>
      <td>1,247</td>
      <td>188</td>
    </tr>
    <tr>
      <td>HR, SR</td>
      <td>1,583</td>
      <td>451</td>
    </tr>
    <tr>
      <td>MN, SR</td>
      <td>2,161</td>
      <td>552</td>
    </tr>
  </tbody>
  <tfoot>
    <tr>
      <td>Total</td>
      <td>7,711</td>
      <td>5,699</td>
    </tr>
    <tr>
      <td>% of set</td>
      <td>21,2</td>
      <td>13,9</td>
    </tr>
  </tfoot>
</table>

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International.

https://creativecommons.org/licenses/by-sa/4.0/

<figure><img src="image_3.png" /></figure>