bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark
Year: 2023

Abstract:
We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural language inference, fact-checking, named entity recognition, sentiment analysis, question answering, etc.) and machine learning tasks (sequence labeling, document-level classification, and regression). We run the first systematic evaluation of pre-trained language models for Bulgarian, comparing and contrasting results across the nine tasks in the benchmark. The evaluation results show strong performance on sequence labeling tasks, but there is a lot of room for improvement for tasks that require more complex reasoning. We make bgGLUE publicly available together with the fine-tuning and the evaluation code, as well as a public leaderboard at https://bgglue.github.io, and we hope that it will enable further advancements in developing NLU models for Bulgarian.

heading: 1 Introduction
text: Natural Language Understanding (NLU) benchmarks, such as GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a), were designed for a rigorous evaluation of language models on a diverse set of natural language understanding (NLU) tasks. The wide adoption of such benchmarks has driven the rapid development of models that perform well on the tasks that are part of these benchmarks, but also beyond (Devlin et al., 2019; Liu et al., 2019). However, until recently, the focus of such benchmarks has been on English, with little interest in other languages (Bender, 2011; Ponti et al., 2019).
∗Work done while Momchil was in the Sofia University, prior to joining Amazon.
†Meta AI is not involved in the creation, release, or hosting of the datasets in the benchmark.
To address this, recent work has designed benchmarks to test models on NLU tasks on non-English languages (Le et al., 2020; Rodriguez-Penagos et al., 2021; Shavrina et al., 2020) or on multiple languages (Liang et al., 2020; Hu et al., 2020).
Here, we aim to improve the diversity of the languages represented in NLU benchmarks by proposing bgGLUE, a benchmark for Bulgarian that consists of nine NLU tasks, including token classification, regression, and classification. Thus far, only individual datasets in Bulgarian have been used for model development and evaluation. Small subsets of up to three downstream tasks in Bulgarian have also been included in some multilingual benchmarks (Liang et al., 2020; Hu et al., 2020). Additionally, there are existing benchmarks focusing on other Balto-Slavic languages, such as the Russian SuperGLUE (Shavrina et al., 2020) and the Slovene SuperGLUE (Žagar and Robnik-Šikonja, 2022). However, there are no comprehensive benchmarks for representatives of the Eastern South-Slavic language subgroup, and for Bulgarian in particular. We aim to address these limitations with bgGLUE.
bgGLUE unifies and facilitates access to existing datasets and tasks for Bulgarian. By including more challenging tasks such as natural language inference, fact-checking, and question answering, we ensure that it comprises a rigorous test set for NLP models developed for Bulgarian. We also provide access to the benchmark through the HuggingFace Hub (Lhoest et al., 2021) to allow for ease of use and we encourage model sharing for Bulgarian. Moreover, we fine-tune and run the first systematic evaluation of existing language models for Bulgarian, comparing and contrasting results across all tasks in the benchmark. Our evaluation results show that larger and more robustly pre-trained models yield better performance on all tasks, but also that efficiently distilled models are a strong competitor to their larger counterparts.
8733
The models show strong performance on partof-speech tagging and named entity recognition, but struggle on tasks that require more complex reasoning such as solving matriculation exams, or evaluating the credibility and the veracity of news articles. Our contributions are as follows:
• We propose the first benchmark for evaluating the capabilities of language models on NLU in Bulgarian, bgGLUE, which includes nine diverse and challenging downstream tasks.1
• While creating the benchmark, we curated the datasets and created standard splits, where those have not been previously available in the original publications. This facilitates the principled evaluation of all datasets in bgGLUE.
• We train and share 36 models for Bulgarian and provide the first comparative evaluation of existing models on all tasks in bgGLUE.

heading: 2 Tasks
text: Table 1 shows the nine datasets that are included in the bgGLUE benchmark. Table 2 shows examples from each dataset and their corresponding labels (translations are available in Table 17 in the Appendix). We present additional details such as word overlaps, domain, topic, label distributions, etc. about each dataset in Appendix C.
1The bgGLUE code, data, and models are available at https://github.com/bgGLUE/bgglue.

heading: 2.1 Token Classification
text: BSNLP The dataset is released as part of the Balto-Slavic NLP workshop series (Piskorski et al., 2017, 2019, 2021). The task focuses on crosslingual document-level extraction of named entities: the systems should recognize, classify, extract, normalize, and make a cross-lingual linking of all named entity mentions in a document; detecting the position of each named entity mention is not required. The target tags are person (PER), organization (ORG), location (LOC), product (PRO), and event (EVT).
PAN-X (WikiANN) The PAN-X dataset (Pan et al., 2017) has Named Entity Recognition (NER) annotations for persons (PER), organizations (ORG), and locations (LOC). It has been constructed using the linked entities in Wikipedia pages for 282 different languages.
Universal Dependencies (U. Dep) Universal Dependencies (UD) (Nivre et al., 2020) is a framework for consistent annotation of grammar (part of speech, morphological features, and syntactic dependencies) across different human languages. UD is an open community effort with over 300 contributors producing more than 200 treebanks in over 100 languages. The dataset was collected, annotated, and later transferred into the required UD format as part of the Bulgarian treebank (BTBBulTreeBank) project (Osenova and Simov, 2015).

heading: 2.2 Natural Language Inference
text: XNLI This dataset (Conneau et al., 2018) is a subset of a few thousand examples from MNLI, which has been translated into 14 languages. As with MNLI, the goal is to predict textual entailment: does sentence A imply/contradict/neither sentence B? This is a classification task: given two sentences, predict one of the three labels.

heading: 2.3 Sentiment Analysis
text: Cinexio The Cinexio dataset (Kapukaranov and Nakov, 2015) focuses on fine-grained sentiment analysis of movie reviews. It was automatically collected to contain movie reviews in Bulgarian from the Cinexio ticket-booking website (which is not available anymore).

heading: 2.4 News Credibility / Fact-Checking
text: CLEF-2021 CheckThat!, Task 1A (CT21.T1) Check-Worthiness Estimation dataset is part of the 2021 CheckThat! Lab on Detecting Check-Worthy Claims, previously Fact-Checked Claims, and Fake News (Task 1) (Shaar et al., 2021). The aim of the task is to determine whether a piece of text is worth fact-checking. More precisely, given a tweet, one has to produce a ranked list of tweets, ordered by their check-worthiness.
Credible News (Cred.-N) The Credible News (Hardalov et al., 2016) dataset focuses on the problem of automatically distinguishing credible from fake and humorous news. The examples are articles collected from six Bulgarian news websites. The articles cover various topics including politics (both local and global), sports, lifestyle, and pop culture. The original dataset contained news from four websites. As part of the bgGLUE initiative, we collected 6,550 new articles (5K credible and 1.5K humorous) from two new websites, and we release more than 30K ones that were not publicly available.
Fake News (Fake-N) This dataset (Society, 2017; Karadzhov et al., 2017) contains Bulgarian news articles over a fixed period of time, whose factuality was questioned. These news articles come from 377 different sources from various domains, including politics, interesting facts, and tips&tricks. The dataset was prepared for and used in the Hack the Fake News hackathon in 2017. We found and removed instances that were duplicated across the splits and we further randomly allocated 10% of the training instances for a development dataset, which was not available in the original version of the dataset.

heading: 2.5 Question Answering
text: High School Examinations (EXAMS) EXAMS (Hardalov et al., 2019, 2020) is a benchmark dataset for cross-lingual and multilingual question answering for high school examinations. It contains more than 24,000 high-quality exam questions in 26 languages, covering eight language families and 24 school subjects from the Natural Sciences and Social Sciences, among others. EXAMS offers a fine-grained evaluation framework across multiple languages and subjects, which allows a precise analysis and comparison of various models.

heading: 2.6 Scoring
text: In bgGLUE, we opt for the simple approach of weighing each task equally, and for tasks with multiple metrics, first averaging those scores to get a task score.

heading: 3 Data Preparation
text: Here, we describe the pre-processing steps we took to prepare the datasets before including them in the bgGLUE benchmark. Our main goal was to ensure that the setup evaluated the language understanding abilities of the models in a principled way and in a diverse set of domains. Since all of the datasets were publicly available, we preserved the original setup as much as possible. Nevertheless, we found that some datasets contained duplicate examples across their train/dev/test splits, or that all of the splits came from the same domain, which may overestimate the model’s performance. Hereby, we removed data leaks and proposed new topicbased or temporal-based (i.e., timestamp-based) data splits where needed. We deduplicated the examples based on a complete word overlap in two pairs of normalized texts, i.e., lowercased, and excluding all stop words.
For the BSNLP task, we combined the data from three consecutive editions of the NER shared task. We selected all Bulgarian examples, which encompassed six different topics. We used the latest two topics for testing, and split the rest randomly at a 4:1 ratio for training and validation.
The Credible News dataset contained data from six sources and various news topics. We split the dataset both by topic and by source. In particular, we included all news articles from the same topic in the same split. For training and validation, we used documents from the largest sources from the two categories. Moreover, we extracted the same topics from the two and grouped them together within the splits, keeping the class ratio at 1:10. In the test dataset, we used data points from all six data sources.2 We note that our test set contains all data points from the two new sources, which are more recent, making them even more challenging. Finally, we cleaned the texts from duplicates and removed all keywords indicating the source: the names of the authors, the media source, the URLs, etc. More details about the collection process of the news articles are given in Appendix C.
2We did not have overlapping topics from the same source in the different splits.
We prepared an entirely new split for Cinexio, as there was no standard one. First, we removed all duplicate comments and we sorted the remaining ones by publication time. Next, we split the comments using half for training and the rest equally for validation and testing. Finally, we removed from the training set the comments for the same movie that appeared in the other two splits, which changed the distribution by slightly increasing the proportion of the training set.
For EXAMS, we kept the original validation and test splits (Hardalov et al., 2020). We added all additional questions from Hardalov et al. (2019) to the training set, i.e., the category history online quizzes.
The Fake News dataset was released as part of a shared task and it was already split into training and testing sets. After manual inspection, we found that some of the articles had only their titles reworded and had the same content. Therefore, we removed duplicates based only on exact matches in the article’s body. In addition, we designated 10% of the training examples for validation.
PAN-X contained short sentences from Wikipedia automatically annotated based on the available Wiki entities. The dataset consists of 20K examples for training and 10K for validation and testing. We kept the proposed splits, but we checked for duplicates by converting each sentence to lowercase and removing the punctuation. This resulted in the removal of 10K sentences overall.
Our analysis did not find any issues with CT21.T1, U.Dep, and XNLI, and thus we kept the splits as provided originally.

heading: 4 Experiments
text: In this section, we first describe the baseline systems we experiment with and then we present the evaluation results.

heading: 4.1 Baselines
text: Majority and Random Baselines The majority class baseline is calculated from the distributions of the labels in each test set. In the random baseline, each test instance is assigned a target label at random with equal probability.
Fine-tuned models Our baselines include several prominent multilingual encoder-only pretrained Transformer models. We divide them, based on their pre-training objective as follows: I Masked language modeling:
• mBERT (Devlin et al., 2019) We use the base cased version, trained on 104 languages, including Bulgarian. The pre-training task is done on a Wikipedia dump for each language.
• XLM-R (Conneau et al., 2020) We evaluate the Base and the Large versions of the model. They are trained on filtered CommonCrawl data in 100 languages, including Bulgarian.
II Knowledge distillation: • Distil-mBERT (Sanh et al., 2019) The model
is distilled using mBERT as the teacher. • MiniLML12 (Wang et al., 2020) The model is
distilled using XLM-RBase as the teacher, on the same pre-training corpora as the latter.
III Bulgarian downstream task: • SlavicBERT (Piskorski et al., 2021) The
model is based on mBERT that is additionally pre-trained with four Slavic languages: Bulgarian, Czech, Polish, and Russian, using a stratified dataset of Russian news and Wiki articles for the other languages. Finally, the model is fine-tuned on all the languages from the BSNLP shared task.
For the token classification tasks (BSNLP, U.Dep, PAN-X), we predict the tag for each word based on the tag of the first sub-token. For the sentence classification tasks, we obtain the predictions based on the first special token (e.g., [CLS]). For Cinexio, we optimize a mean squared error loss. For CT21.T1, Cred.-N, Fake-N, and XNLI, we optimize the cross entropy loss. Finally, for EXAMS, we optimize a binary cross entropy for each candidate answer. More details about the experimental setup, the values of the model hyperparameters, and other training details can be found in Appendix A. For a description of the inputs and the outputs, we refer the reader to Appendix B.

heading: 4.2 Experimental Results
text: Table 3 shows the results for the baseline models fine-tuned on the bgGLUE tasks. Each model is trained on one task at a time. First, we see that the random and the majority baselines achieve below 20 points bgGLUE score, and all fine-tuned models outperform them on all tasks by a sizable margin.
Figure 1 shows the correlation between the model size and the bgGLUE score: we can see that scaling the model size brings additional performance improvements (Devlin et al., 2019; Conneau et al., 2020; Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022; Scao et al., 2022).
Table 5 in the Appendix summarizes the number of parameters for each model we experimented with. We can see in that there is a linear correlation between the number of parameters in the model and its performance, which holds for all models except for Distil-mBERT, where the slope is more steep. Nonetheless, it is clear that the model size is not the only factor that affects the downstream task performance. Other important factors include the pre-training dataset that was used (Liu et al., 2019; Raffel et al., 2020), the number of tokens in that training set (Hoffmann et al., 2022), whether the model is distilled or is full-size (Sanh et al., 2019; Wang et al., 2020), whether it is monolingual or multilingual (Conneau et al., 2020), etc.
As expected, the largest model, XLM-RLarge, outperformed its smaller version XLM-RBase by 2.78 points absolute. Moreover, the more robustly pre-trained XLM-R model on average outperformed by 2 points a similar-sized mBERT, scoring at 73.0 bgGLUE score. The highest differences between BERT-based and XLM-R-based models (including their distilled versions) are on BSNLP, Cinexio, Cred.-N, and XNLI, in favor of XLM; on CT21.T1 and PAN-X this is in favor of BERT.
The gap between XLM-R and mBERT is reduced by 1 point absolute by the SlavicBERT’s additional fine-tuning on downstream tasks. Although the largest improvement is observed for NER tasks,3 we see an increase by 1-2 points also on Cred.-N, Fake-N, and XNLI, compared to mBERT. However, we also see that the downstream pre-fine-tuning is not beneficial for all tasks (Poth et al., 2021), and we see a drop in performance for ranking (CT21.T1) and question answering4 (EXAMS) tasks. Our evaluation of knowledge-distilled models shows that they are a competitive alternative to their teacher models for Bulgarian. Although they are ranked last in terms of performance, their results are 1.5–2.0 points of bgGLUE score behind the best results we obtained, and thus we believe they are a viable alternative to the full models.
3SlavicBERT is pre-trained on data from the 2019 edition of the BSNLP competition using different splits.
4Hardalov et al. (2019) observed the same when finetuning SlavicBERT for multiple-choice QA.
In order to measure the trade-off between model size and model performance, we compare mBERT to DistilBERT and XLM-R to MiniLM. In the case of DistilBERT, we have 30% (178M → 135M) fewer parameters, which results in a 2.2% drop in performance. In turn, MiniLM has less than half of the parameters of its teacher, i.e., 135% fewer parameters (278M → 118M), which leads to only a 2.9% relative drop in performance. The most challenging tasks for the distilled models are NER (BSNLP and PAN-X), NLI (Cinexio and XNLI), where we see a sizable gap between non-distilled models. Finally, we note that MiniLM has the worst performance on CT21.T1, which is also so for XLM-R. We hypothesize that this is due to the source of the dataset being Twitter. For more detailed results, we refer the reader to Appendix D.

heading: 5 Discussion
text: Software Tools As part of building the benchmark, we developed a set of software tools that facilitate the training and the evaluation of new models. The toolkit is implemented in PyTorch (Paszke et al., 2019), using the transformers library (Wolf et al., 2020). Moreover, we integrate and release publicly all datasets (in accordance with their licenses; see the next paragraph) from bgGLUE in the HuggingFace datasets repository (Lhoest et al., 2021).5
Dataset Licenses We keep the licenses as provided by their authors for all datasets included in the bgGLUE benchmark. Table 4 summarizes the information about each dataset and gives links to the external websites and code repositories provided by the authors. All datasets are available to use for research purposes. Some of them come with a non-commercial license, i.e., Cinexio, Cred.N, U.Dep, and XNLI. Cred.-N requires signing an agreement form before obtaining the dataset.
Modeling Considerations Previous work has shown that a model’s scale (Kaplan et al., 2020; Hoffmann et al., 2022) is an important factor for its performance (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Conneau et al., 2020; Soltan et al., 2022; Zhang et al., 2022), especially in zero-shot or few-shot settings (Brown et al., 2020; Wei et al., 2021; Chowdhery et al., 2022; Ouyang et al., 2022).
5https://huggingface.co/bgglue
Nevertheless, these models are often pre-trained on high-resource languages such as English. A few noteworthy alternatives for Bulgarian include XLM-RoBERTa (Goyal et al., 2021), multilingual T5 (Xue et al., 2021), XGLM (Lin et al., 2022), mGPT (Shliazhko et al., 2022), and the extended version of BLOOM (Yong et al., 2022). These models represent different variants of the Transformer architecture, i.e., encoder/decoder-only or sequence-to-sequence. In this work, we only included encoder-only Transformer models with less than one billion parameters (see Table 5 in the Appendix). We leave the rest to be explored by future participants in the bgGLUE benchmark and we note that some of the tasks, such as ranking or regression, require additional steps to make the different architectures work.
Although a model’s scale is important for its performance, it comes with additional efficiency and computational costs, among other considerations (Bommasani et al., 2021). Fine-tuning large pre-trained language models is usually timeconsuming and expensive, and it also requires a large number of manually annotated examples. A possible direction that alleviates these requirements is to use adapter-based models (Rebuffi et al., 2017; Pfeiffer et al., 2021) and other techniques for efficient training (Lester et al., 2021; Hu et al., 2021; Ben Zaken et al., 2022).
Promising results were shown both in multilingual (Pfeiffer et al., 2020) and in cross-lingual settings (Muennighoff et al., 2022).
Another line of work, which we leave for future research, is zero-shot and few-shot learning. Recently, different techniques have been developed such as learning from demonstrations Brown et al. (2020), patterns (Schick and Schütze, 2021a,b), instructions (Mishra et al., 2022; Wang et al., 2022; Chung et al., 2022; Iyer et al., 2022), or multi-task fine-tuning Raffel et al. (2020); Wei et al. (2021); Chowdhery et al. (2022). These models require fewer examples due to their extensive fine-tuning, but they still showed Chowdhery et al. (2022) that a model’s size is of crucial importance for their performance.
Finally, there is a trade-off between performance and using monolingual vs. multilingual models (Devlin et al., 2019; Conneau et al., 2020; Pyysalo et al., 2021). Extensively pre-trained monolingual models on language-specific corpora often achieve better performance compared to multilingual ones (Kuratov and Arkhipov, 2019; Canete et al., 2020; Masala et al., 2020; Delobelle et al., 2020; Chan et al., 2020; Martin et al., 2020; Cui et al., 2021; Pyysalo et al., 2021; Barry et al., 2022). However, there is no open-source large scale pre-trained monolingual Bulgarian model with extensive pre-training: most of the existing checkpoints are based on multilingual ones, and they are fine-tuned on small corpora.6
Leaderboard We develop our leaderboard in accordance with existing ones, e.g., the (Super)GLUE (Wang et al., 2019b,a): the participants are provided with all the training, validation, and test examples without the gold test labels. They submit an archive with their predictions for each task, and then our system automatically evaluates their predictions.
The intended use of our leaderboard is to provide a standardized way to compare the performance of different models on specific tasks, thus allowing researchers and practitioners to assess the current state of the art and to identify areas where improvements can be made. We urge against making improperly supported claims about general language understanding based on the performance on our leaderboard, and on NLP leaderboards in general (Ethayarajh and Jurafsky, 2020; Raji et al., 2021; Blasi et al., 2022).
6Reference: https://huggingface.co/models?language=bg
We believe that the bgGLUE leaderboard will incentivize model and resource creation in two ways: (i) the participants are required to share details about their submissions, and are encouraged to release their models; we cannot force the latter, but we can ensure that the methods are reproducible to some extent; (ii) practice shows that the results on such leaderboards tend to saturate in several years, which will likely happen with this benchmark as well. We plan to open our platform and to work with interested researchers, first to design new leaderboards (Ma et al., 2021), second to include their datasets into bgGLUE, and third to collaborate to build new (including human-and-model-inthe-loop (Kiela et al., 2021)) and refining exciting language resources for Bulgarian.

heading: 6 Related Work
text: Language Understanding The release of the code and English corpora as part of the General Language Understanding Evaluation (GLUE Wang et al. (2019b)) was a push towards the development of models with improved performance on a diverse set of downstream tasks. The GLUE benchmark includes 11 NLU tasks, such as semantic textual similarity, natural language inference, and other classification tasks. Later, the benchmark was extended with additional and more sophisticated tasks in its SuperGLUE (Wang et al., 2019a) variant.
While GLUE and SuperGLUE have been established as the de-facto standard for evaluating machine learning models, they are limited to English. To foster the evaluation and the development of machine learning models for other languages, several benchmarks in other languages have been released. They can be grouped based on their language family as follows: Romance – French (Le et al., 2020), Catalan (Rodriguez-Penagos et al., 2021), Balto-Slavic – Russian (Shavrina et al., 2020), Slovenian (Žagar and Robnik-Šikonja, 2022), Iranian – Persian (Khashabi et al., 2021), Altic – Korean (Park et al., 2021), Sino-Tibetan – both CLUE (Xu et al., 2020), and CUGE (Yao et al., 2021) focus on Chinese, Indic – Kakwani et al. (2020) evaluated fine-tuned pre-trained models on multiple Indic languages, while Doddapaneni et al. (2022) focused on their zero-shot capabilities, and Malayic – Indonesian (Koto et al., 2020). Khanuja et al. (2020) provides further resources for code-switched languages (English with Spanish or Hindi).
While Shavrina et al. (2020) provided resources for Balto-Slavic languages, there are no existing benchmarks for languages in the South EasternSlavic subgroup or for Bulgarian in particular. We address this deficiency by developing bgGLUE, a benchmark for Bulgarian, which is part of the South-Slavic subgroup. Hristova (2021) published a survey of the language resources available for Bulgarian, which we also include in the current benchmark, extended with more recent datasets.
The aforementioned benchmarks focus on a single language or on a single language family. Other studies looked at multiple languages. Liang et al. (2020) proposed XGLUE, a benchmark for 19 languages that covers NLU problems and language generation tasks. Hu et al. (2020) collected a crosslingual evaluation dataset in 40 languages, later extended with 10 additional (Ruder et al., 2021), including tasks similar to the original (Super)GLUE setup such as token classification, question answering, textual similarity, natural language inference, etc. Both benchmarks include Bulgarian, but are limited to three tasks: part-of-speech (POS) tagging (Universal Dependencies Nivre et al. (2020)), named entity recognition (PAN-X/WikiAnn Pan et al. (2017)), and natural language inference (XNLI Conneau et al. (2018)). These tasks are also part of bgGLUE, but we extend them with additional NLU tasks, including question answering, fake news detection, sentiment analysis, etc.
More recently, a large-scale initiative for providing open access to large language models trained to perform new tasks based on few demonstrations or natural language instructions was launched as part of the BLOOM workshop (Scao et al., 2022). This led to the release of a new corpus, comprising sources in 46 natural and 13 programming languages, and a multilingual decoder-only Transformer language model pre-trained on that data. However, BLOOM was not pre-trained on Slavic languages, and it was only later that zero-shot support for Bulgarian was added Yong et al. (2022).
BIG-Bench (Srivastava et al., 2023) is another such initiative that incorporates more than 200 tasks (some not related to NLP) to test the capabilities of language models. The task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. Currently, there are a few non-English tasks included, but none of them is for Bulgarian.
The low number of Bulgarian resources that are part of these initiatives is yet another reason why more publicly available Bulgarian resources and open-access models are needed.
Other Modalities Existing work also quantifies the abilities of state-of-the-art models in multimodal settings. CodeX GLUE (Lu et al., 2021) is a benchmark for program understanding and generation. Conneau et al. (2022) proposed XTREMES that focuses on speech tasks, including speech recognition, classification, speech-to-text translation, and retrieval. Finally, IGLUE (Bugliarello et al., 2022) fills the gap in image and text evaluation, including tasks such as visual question answering, cross-modal retrieval, and grounded reasoning. Bulgarian is included as part of both XTREME-S and IGLUE. However, here we focus only on NLP tasks on text and currently, we do not include tasks with multiple modalities in the present benchmark.

heading: 7 Conclusion and Future Work
text: We presented bgGLUE – the first holistic benchmark for evaluating NLU systems in Bulgarian. It includes nine challenging tasks that cover token classification, regression/ranking, and text classification. We fine-tuned and evaluated six different pre-trained state-of-the-art language models. Our extensive evaluation showed that bgGLUE contains challenging tasks that are far from being solved. Finally, we open-sourced the cleaned versions of the datasets, including the new, more challenging splits, and the source code for training and evaluation, and we released 36 fine-tuned models (one for every task and model combination). All the released artifacts are also integrated into the HugginFace Hub. We believe that bgGLUE is a rich and challenging testbed that will cultivate prospective work on Bulgarian language understanding.
In future work, we plan to add more tasks for Bulgarian, e.g., toxicity detection (Dinkov et al., 2019). We also want to use monolingual Bulgarian datasets for pretraining, beyond Wikipedia and CommonCrawl, e.g., (Simov et al., 2002, 2004; Koeva et al., 2004, 2012, 2020), using which will require a thorough assessment in order to prevent introducing unwanted biases and hazardous behavior in the models trained on them (Bender et al., 2021; Liang et al., 2021). Finally, we plan to try recent multilingual models such as mDeBERTaV3 (He et al., 2021), mT0 and BLOOMz (Muennighoff et al., 2023).

heading: In Memory of Professor Dragomir Radev
text: We dedicate this work to the memory of Dragomir Radev, who is a co-author of this paper. Drago had a tremendous impact on our community, and his legacy will live on through the countless students and colleagues whose lives he touched. Drago was not only an exceptional computer scientist, but one of the kindest and most humble people many of us have ever known. He deeply cared about Bulgarian NLP and Bulgarian NLP researchers. He was also the one who gave the idea and who remained the main driving force behind the Bulgarian GLUE project. Drago will be greatly missed...

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful questions and comments, which have helped us improve the quality of the paper. We also want to thank the MSc students from Sofia University “St. Kliment Ohridski” Mihail Mihaylov and Kiril Georgiev for their help with the initial parsing and analysis. We further thank Quỳnh Mihaylova for designing the bgGLUE logo and for her priceless feedback on the design of the leaderboard.
This work is partially supported by Project UNITe BG05M2OP001-1.001-0004 funded by the Bulgarian OP “Science and Education for Smart Growth”, co-funded by the EU via the ESI Funds.
The work on BulTreeBank-UD and the BaltoSlavic datasets for Bulgarian is partially supported by CLaDA-BG, the Bulgarian National Interdisciplinary Research e-Infrastructure for Resources and Technologies in favor of the Bulgarian Language and Cultural Heritage, part of the EU infrastructures CLARIN and DARIAH, grant number DO1-301/17.12.21.
Limitations
Tasks in bgGLUE The bgGLUE benchmark is comprised of nine challenging NLU tasks, including three token classification tasks, one ranking task and five text classification tasks. While we cover three different types of tasks in the benchmark, we are restricted by the available resources for Bulgarian, and thus we could not include some other NLP tasks, such as language generation. We also consider only NLP tasks and we do not include tasks with other/multiple modalities. Finally, some of the tasks are of similar nature, e.g., we include two datasets for NER and two for credibility/fake news classification (see Section 2).
Domains in bgGLUE The tasks included in bgGLUE span over multiple domains such as social media posts, Wikipedia, and news articles and can test both for short and long document understanding. However, each task is limited to one domain and the topics within the domain do not necessarily have full coverage of all possible topics. Moreover, some of the tasks have overlapping domains, e.g., the documents in both Cred.-N and Fake-N are news articles.
Baseline Models As described in Section 5, the baseline models provided for bgGLUE include fairly small encoder-only Transformer architectures. We leave for future work other modeling architectures and modeling techniques that are known for improving the efficiency and the computational requirements of the used models, e.g., few-shot and zero-shot in-context learning and instruction-based evaluation, multi-task learning, etc.
Model Biases In this work, we did not explore whether the datasets in bgGLUE contain unwanted biases, which could also lead to potentially hazardous behavior of the baselines we trained in our experiments with the bgGLUE benchmark.
Ethics and Broader Impact
Dataset Collection
In bgGLUE, we include only datasets that are publicly available, with a license that allows at a minimum free use for academic research. We have also referenced the original work where the corresponding resources were first proposed. We encourage the users of bgGLUE to refer to the original work for licensing details.
Additionally, we carefully examined and removed the instances of the dataset that were duplicated across the training/development/test splits. Whenever development or other dataset splits were not available, we also provide new dataset splits as well. Section 2 points where such changes of the corresponding original resources were required, and the code used to filter or to produce the new splits is available in bgGLUE’s code repository. We believe that the selection of publicly available datasets and the adopted dataset curation steps will foster the development and the rigorous evaluation of language models for Bulgarian.
Biases and Misuse Potential
The datasets included in bgGLUE were annotated by human annotators, who could be subject to potential biases in their annotation process. Hence, the datasets in bgGLUE could potentially be misused to develop models that make predictions that are unfair to individuals or groups. Therefore, we ask users of bgGLUE to be aware of such potential biases and risks of misuse. We note that any biases that might exist in the original resources gathered in this benchmark are unintentional and do not aim to cause harm.
Intended Use
The bgGLUE benchmark is intended to promote the development and the rigorous evaluation of language models for Bulgarian. We further believe that the benchmark will serve to examine the capabilities and the limitations of existing and emerging models on the challenging natural language understanding tasks in Bulgarian. Ideally, this could also lead to raising awareness of the potential risks associated with the use of such models developed for downstream tasks in Bulgarian.
Environmental Impact
While bgGLUE can stimulate the development of new machine learning models, it is worth noting that such models could require large computational resources for training, which contributes to global warming (Strubell et al., 2019). On the other hand, bgGLUE is intended mainly for fine-tuning pretrained large language models, which requires considerably smaller computations. Additionally, we release the benchmark and the models on the HuggingFace Hub, which further reduces the environmental impact, as fine-tuning again is computationally costly, especially for larger models.

heading: A Model Hyper-Parameters and Training
text: Below, we first describe the values of some parameters that are across all models we experiment with, and then we discuss the values of some modelspecific parameters:
• All our models use the AdamW (Loshchilov and Hutter, 2019) optimizer with a weight decay of 1e-8, β1 0.9, β2 0.999, ϵ 1e-08, and are trained for five epochs with a batch size of 16 (gradient accumulation is applied when needed), and a maximum length of 512 tokens.
• We truncate longer input sequences token by token, if the input is formed from multiple sequences (see Section B), i.e., pairs, we start from the longest one.
• All models use a warmup ratio of 0.06 from the training data. We experiment with learning rate values {2–5}e-04 for base and distilled models, and {1–3}e-04 for XLM-RLarge.
• The values of the hyper-parameters (including the number of training epochs) and the best checkpoints were selected on the development set. We use the target metric for each task as a checkpoint selection criterion.
• We trained our models on 5x Tesla T4 GPUs. Depending on the dataset size, the experiments took between 10 minutes, for the smaller datasets and models, and up to 2 hours, for larger datasets. Training the XNLI model took 10 hours with base models, and 20 hours for large models.
• All models were trained with half-precision (fp16) using the default PyTorch implementation.
• Table 5 shows the models’ size in terms of the number of parameters.
• When evaluating the Token Classification Tasks if the predicted sequence was shorter than the target one (i.e., not all inputs fit into 512 tokens), we added empty tags (’O’) until the target length was reached.
Model Name #Params

heading: B Model Input, Output and Loss
text: Table 6 shows the inputs and the outputs for each model. We selected the formats based on previous work (Devlin et al., 2019; Liu et al., 2019) and the proposed formats on the (Super)GLUE benchmark (Wang et al., 2019b,a). For all tasks we introduce a projection layer on top of the pre-trained language model’s representations. For classification tasks, the output maps to the number of classes, for regression, we project it to a single continuous value, for ranking, we obtain a probability distribution over two classes, for question answering, we rank each answer based on the log probability score, and finally, for token classification tasks, we apply the classification head on top of each token’s representation. It is important to note that we use the BIO encoding for the NER tasks. We chose the loss function based on the target value. Finally, we replaced the special tokens with the corresponding ones from the baseline model.

heading: C Additional Task Details
text: In this section, we summarize some additional characteristics for each task in the bgGLUE benchmark.
Figure 3 shows some statistics about the word overlap between subsets. To calculate the statistics, we split the texts into words using the NLTK Bird et al. (2009) tokenizer. After that, we take the number of unique words in each subset and we take the union of all common words between the first and the second subset, we then compare and divide them by the size of the superset obtained by combining the two. We see that most of the datasets have high overlap between the training and the development/testing set. This is expected as the training sets are often significantly larger
compared to the other subsets, and also as we did not filter out the stop words, which cover a big part of the word tokens.
Interestingly, the only exception is the PAN-X dataset. We attribute this to the text snippets being short, designed to contain named entities, and being extracted from different Wikipedia articles.
Figure 4 shows the per task label distribution. We see that most of the tasks maintain similar distributions across labels, except for BSNLP, where we have fewer ORG and PRO tags and more PER.
BSNLP We can see in Table 7 that the most represented topic is Brexit with 600 examples (4x compared to the second topic), followed by COVID-19, US Elections 2020, and Nord Stream, each covering well above 100 examples. The other two topics, AsiaBibi and Ryanair, have less than 100 examples.
Cinexio Table 8 shows the number of movies in the Cinexio dataset. Each movie received 29.9 comments on average.
Cred.-N. We used a custom crawler, BeautifulSoup to parse the HTML, and per-site CSS selectors to extract the articles’ text. The crawler was based on simple rules that collect and follow the links to articles on each starting page we pass. Our starting points are pages that contain all articles sorted by their publication date and paginated. Finally, we remove all HTML tags, images and information about the authors and the sources, retaining only the plain text. We annotated the articles as credible or humorous based on the label for their website. More details about the dataset and the pre-processing are given in Sections 2 and 3.
High School Examinations (EXAMS) Figure 2 shows the average number of options per question for each subset in the datasets. Both the Dev and Test subset have four options, but Train contains questions with three answers coming from online history exams collected from Hardalov et al. (2019). These examples also affect the subject distribution for the training set.
Fake News (Fake-N.) Here, we report the number of unique and common domains:
• Train vs Dev – #Common Domains: 106 – Only in train: 239 – Only in dev: 13
• Train vs. Test – #Common Domains: 162 – Only in train: 183 – Only in test: 46
• Dev vs. Test – #Common Domains: 90 – Only in dev: 29 – Only in test: 118

heading: D Fine-Grained Results
text: Here, we present the fine-grained results per task. Grouped by the task types from Table 1, we include the following tables: (i) Regression / Ranking – In Table 10, we present the Spearman and the Pearson correlation values for the Cinexio task. Table 11 shows the metrics for Ct21.T1, including P@K and R-Precision; (ii) Classification Tasks – for the binary classification tasks Cred.-N and Fake-N we include the Precision and Recall for the target class, i.e., Humorous, and Fake respectively, and finally (iii) Token Classification – Tables 14, 15, and 16 include per token type P, R, and F1.
We did not include tables for EXAMS and XNLI as their target evaluation measure is Accuracy, and thus they are only coarse-grained.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section "Limitations"

heading: 3 A2. Did you discuss any potential risks of your work?
text: In Sections “Ethics Statement” and “Limitations”.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?

heading: In “Abstract” and Section 1. “Introduction”.
text: 7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sections 2 “Tasks” and Section 4 “Experiments”

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 2 “Tasks”

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 5. “Discussions”

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Sections 2 “Tasks” and “Ethics Statement”.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? In Sections 2 “Tasks” and “Ethics Statement”.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? In Sections 2 “Tasks”, Section 3 “Data Prepartion” and the Appendix.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2 “Tasks”, and the Appendix.
C 3 Did you run computational experiments?

heading: In Section 4. “Experiments” and Appendix.
text: C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We leave for future work other modeling architectures and modeling techniques that are known for improving the efficiency and the computational requirements of the used models, e. g. , few-shot and zero-shot in-context learning and instruction-based evaluation, multi-task learning, etc. Model Biases In this work, we did not explore whether the datasets in bgGLUE contain unwanted biases, which could also lead to potentially hazardous behavior of the baselines we trained in our experiments with the bgGLUE benchmark