Instructions to use absltnull/prebij with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use absltnull/prebij with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="absltnull/prebij")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("absltnull/prebij") model = AutoModelForSequenceClassification.from_pretrained("absltnull/prebij", device_map="auto") - Notebooks
- Google Colab
- Kaggle
preBIJ :: Bosnian / Croatian / Serbian LID
The linguistic intelligence of a native BCS language model used for language identificaton.
predBor Identifikator Jezika ("predBor Language Identifier"), a language classification model based on the now retired predBor-v1 small BCS language model, made specifically to tell apart Bosnian, Croatian and Serbian.
Retired but not forgotten
As noted, I made the decision to retire the predBor model series as the newer Bor models are in the works. However, retirement does not necessarily mean abandonment, at least not to me.
preBIJ is a direct fine-tune of predBor, changing the architecture to a classification model while preserving all linguistical knowledge of the base, effectively giving the old model a new life.
Architecture
| Name | Value |
|---|---|
| Architecture | LlamaForSequenceClassification |
| Base Model | absltnull/predBor-v1 |
| Maximum Sequence Length | 4096 tokens (trained up to 196 tokens, retains accuracy up to 1024, untested beyond) |
| Classes | bs, hr, sr |
| Tokenizer Vocabulary | 65,000 tokens |
| Tokenizer | BorTokenizer-v2 |
| Supported Scripts | Bosnian (Lat, Cyr), Serbian (Lat, Cyr), Croatian (Lat) |
The Idea
After the retirement of predBor, new ideas began to emerge for its new purpose. RoBERTo-v2, while capable, is severely limited by its tokenizer and size, so the natural question would be: how would a native model perform at labeling such languages?
predBor was the perfect match. It had a native BCS tokenizer, was triple the size of RoBERTo-v2, and had a massive, specialized understanding of all three languages (plus English, not relevant), so it was naturally the first choice.
The Surgery
I started with the original predBor-v1 model. Instead of loading it as a casual LM model, I used the AutoModelForSequenceClassification.from_pretrained("absltnull/predBor-v1", num_labels=3) function to load it as a three-way classification model with the type LlamaForSequenceClassification, initializing a random classification head on top of the model, instead of the base LM head.
Next, I had to train that randomly-initialized head to actually know which language is which, using the representations present in the base LM. This was a trivial task for me, though, as I already have my absltnull/bcs-lang-id dataset with over 90k examples of labeled BCS data.
The model trained for about 4 hours on my RTX 2050, and training loss dropped under 1.0 in the first 4% of the run, indicating that the head "got the gist" in like 10 minutes.
The Performance
The final training loss was 0.2, with 92% accuracy on the evaluation set, basically beating RoBERTo-v2 by 5 points of accuracy and F1.
I ranked RoBERTo-v2 vs preBIJ on multiple tests other than accuracy and F1, and here are the results:
Round 1: Accuracy
model (acc, f1): preBIJ (0.919, 0.919) vs RoBERTo-v2 (0.868, 0.868)
A close call, but actually beats RoBERTo-v2 on its own dataset. preBIJ was actually trained on the same dataset as RoBERTo-v2, yet it came out as the victor in this test.
Round 2: Cyrillic
Bosnian script (prebij acc, roberto acc): Latin (0.892, 0.847), Cyrillic (0.792, 0.000)
Serbian script (prebij acc, roberto acc): Latin (0.950, 0.871), Cyrillic (0.959, 1.000)
As we all know, RoBERTo-v2 has a massive Cyrillic bias: "all Cyrillic text => 100% Serbian". This causes it to label every single Cyrillic text as Serbian, which makes it score 0% on the Cyrillic Bosnian test, and score a 100% on the Cyrillic Serbian test.
preBIJ, on the other hand, has managed to outperform RoBERTo-v2 on every metric on this test, scoring:
- 90% accuracy in recognizing Bosnian Latin text,
- 80% accuracy in recognizing Bosnian Cyrillic text,
- 95% accuracy in recognizing Serbian Latin text,
- 96% accuracy in recognizing Serbian Cyrillic text
The amazing part is: it was never specifically trained to do this.
preBIJ has never seen a single sample in Cyrillic, but predBor has. This means that the classification head has learned to generalize, basically using predBor's diverse knowledge of all three languages to recognize the content of the input rather than finding shortcuts in data.
In plain words: predBor knows the difference between Bosnian and Serbian, no matter the script, and the classification head inherited that knowledge, making it beat RoBERTo-v2 in this metric effortlesly.
Round 3: Diacritics
script (prebij acc, roberto acc): Latin (0.921, 0.859), ASCII (0.921, 0.851)
Pretty useless test, but it proves a point: both models score the exact same result in labeling the languages, both with and without the diacritics (though RoBERTo-v2 lost like 0.008 points without diacritics, but we'll ignore that).
Using the model
After seeing these results, we are safe to conclude that preBIJ is the top model for distinguishing BCS languages, even in rare script cases.
To use the model, you can use this snipplet:
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
tok = AutoTokenizer.from_pretrained("absltnull/prebij")
model = AutoModelForSequenceClassification.from_pretrained("absltnull/prebij")
classifier = pipeline(
"text-classification",
model=model,
tokenizer=tok,
device=0 if torch.cuda.is_available() else -1,
)
texts = [
"Ove sedmice moram da kupim bijeli hljeb.", # bs
"Ove nedelje moram da kupim beli hleb.", # sr
"Ovog tjedna moram da kupim bijeli kruh.", # hr
]
predictions = classifier(texts)
for text, pred in zip(texts, predictions):
print(f"[{pred['label']}] ({pred['score']:.2%}) {text}")
TL;DR
Absolute Intelligence has provided this model to be used freely, without any copyright obligations, other than stating that this model belongs to its rightful owner. Feel free to use, fine-tune or adapt this model to any of your preferences. Made with love in Bosnia. Bye.
- Downloads last month
- 20


