absltnull/RoBERTo

RoBERTo Language Classifier

"I literally just took RoBERTa and made it sound like a common Croatian name. You know... Roberto."

Overview

absltnull presents RoBERTo, a small language classifier model made to accurately differentiate Bosnian, Croatian and Serbian text. Based on xlm-roberta-base, it has managed to score over 91% accuracy on data it has never seen before, beating other, broadly generalized models. I myself use RoBERTo to accurately label the HBS data found in my datasets, which will be used to balance the three languages, effectively preventing overtraining my models on a single preference language.

Architecture Details

• Base model: xlm-roberta-base,
• Type: Transformer-based classifier,
• Context length: 512 tokens architecture, 256 used in training,
• Output Classes: "sr", "bs", "hr"

Dataset Details

Dataset permanently borrowed from rsateam/sr-bs-hr-language-id for its simple yet brilliant idea on how to label these three closely related, yet subtly different languages: data sources. The dataset contains exactly 60k entries of equally split Bosnian, Croatian and Serbian text data extracted from Wikipedia, labeled by the source as the ground truth (e.g., Bosnian Wikipedia for BS, Croatian Wikipedia for HR, and Serbian Wikipedia for SR).

• train: 42k entries,
• validation: 9k entries,
• test: 9k entries

Model Evaluation: The Good

The model has shown exceptional performance at labeling the three languages accurately, even on previously unseen data, reaching over 91% accuracy in unambiguous situations ("ambiguous" meaning where the languages are undistinguishable even by human annotators). It even beat two of the most widespread and most used language identification models by ~30 points, proving its worth in this narrow yet genuinely useful task.

Below is an accuracy graph of RoBERTo vs XLM-RoBERTa-Base vs FastText vs CLD2:

Table version for easier reading:

Model	Accuracy	Macro F1
FastText	62.43%	63.59%
CLD2	65.96%	66.77%
XLM-RoBERTa-Base	34.02%	16.92%
RoBERTo	91.58%	91.64%

The Macro F1 and Accuracy scores are both worth noting here, as they both show completely different sides of the story.

Accuracy blindly shows only the raw "how many of the total questions did it get correct" score, which, as you know, can be guessed. Assuming a "cheating" model that knows nothing but tries to guess its way through, it only has 3 labels, so that's a 1/3 or 33% chance to "guess" the correct answer by sheer luck.

On the other hand, Macro F1 explicitly punishes prioritizing or "favoritizing" only one label out of the three, driving down the score if the model does so while "blindly guessing".

That's why XLM-RoBERTa's score is so low - it tried to randomly guess the languages, and it shows in both scores. The model got only 34% right (which is close to the 33% chance to guess by sheer luck) in raw Accuracy, but got only 17% when punished for guessing one language too often.

Now, onto FastText's and CLD2's scores.

FastText is a large language identification model developed by Meta, made to recognize 176 languages. Just like any other general LID model, it catastrophically fails with differentiating Bosnian, Croatian and Serbian as those are not its main focus. From the scores, we can tell it was not "randomly guessing" like XLM-RoBERTa, but it was wrong more than 37% of the time, bringing its accuracy closer to a coinflip (50/50 chance) than to an actual reliable source to tell these languages apart.

CLD2 is a language detection model (and C++ library) developed by Google, and is able to distinguish 80 different languages. NOTE: CLD2 does not have a "bs" (Bosnian) label, which makes this test a bit unfair, since it's not able to recognize Bosnian as a standalone language. As seen in the results, its Accuracy and Macro F1 scores are both noticeably higher than FastText's (about 3 points higher each, despite its disadvantage with Bosnian), but it is still only 15 points more accurate than a coinflip, which is extremely unreliable.

RoBERTo, on the other hand, absolutely nailed the task in both metrics, showing it hadn't memorized examples from training, but actually generalized on them. Macro F1 shows us that the model did not prioritize a single language when met with ambiguous scenarios, while Accuracy shows that it got most of the 9k questions right, with only 8.5% error (mostly from the ambiguous scenarios).

Model Evaluation: The Bad

Even though the model shows outstanding performance against general bases like CLD2 and FastText, it has its own flaws, which have to do with the "ambiguous" cases I've mentioned before.

In these cases, the input text does not show any bias toward any specific language, making the model (and most human native speakers) unsure of what to predict. These ambiguous cases happen mostly while trying to differ Bosnian from Croatian, as the input text can lack any "telltales" of the two languages. This makes some inputs a 50/50 chance of either being Bosnian or Croatian, which is not the model's fault, as even humans cannot tell them apart in these cases.

Below is an image of the "confusion matrix", which shows how often the model confuses the languages with each other (white, thin), along with how often it gets each language right (blue, bold):

Tables for easier reading:

Counts:

	sr (predicted)	bs (predicted)	hr (predicted)
sr	2882	93	16
bs	14	2643	295
hr	11	329	2724

Percentages (normalized by row):

	sr (predicted)	bs (predicted)	hr (predicted)
sr	96.36%	3.11%	0.53%
bs	0.47%	89.53%	9.99%
hr	0.36%	10.74%	88.90%

As we can see:

• In the Bosnian cases, it was mistaken for Croatian 295 times.
• In the Croatian cases, it was mistaken for Bosnian 329 times.
• Serbian was mistaken for another language only 109 times.
• Another language was mistaken for Serbian even less, only 25 times.

Conclusion: The model has learned to differentiate other languages from Serbian really well, but Croatian and Bosnian get mixed in some cases.

How to use

Since this is basically just XLM-RoBERTa on Balkan steroids, you don't need much resources to run it, and it's blazingly fast via Hugging Face Transformers. Also, as a bonus, the labels (bs, sr, hr) are natively baked into the model's config file, so you don't need to manually add them. Enjoy:

# RoBERTo — sr/bs/hr language identification

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

MODEL_NAME = "absltnull/RoBERTo"

# labels (sr/bs/hr) are already baked into the model's config
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)

classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer,
    device=0 if torch.cuda.is_available() else -1,
)

texts = [
    "Ove sedmice moram da kupim bijeli hljeb.", # bs
    "Ове недеље морам да купим бели хлеб.", # sr
    "Ovog tjedna moram da kupim bijeli kruh.", # hr
]

# simple usage: top predicted label
predictions = classifier(texts)
for text, pred in zip(texts, predictions):
    print(f"[{pred['label']}] ({pred['score']:.2%})  {text}")

Final Message

Made with love in Bosnia. Feel free to use, upgrade, fine-tune or adapt to your needs. absltnull out.