SIGMA-BERT-short

Table of Contents

1. Model Summary
2. Model Details
3. Usage
4. Training
5. Training Data
6. Tokenizer
7. Limitations
8. Citation

Model Summary

SIGMA-BERT-short is a Slovenian encoder language model based on the ModernBERT architecture. It was pretrained from scratch using the masked language modeling (MLM) objective on a diverse collection of Slovenian text, including web, news, parliamentary proceedings, scientific publications, and user-generated content.

SIGMA-BERT-short shares the same architecture and parameter count as SIGMA-BERT-base but is optimized for sequences of up to 1,024 tokens, making it well suited for standard sentence-level and document-level language understanding tasks.

The model is intended to be fine-tuned for downstream Slovenian NLP tasks such as:

• Text classification (e.g. topic, sentiment, genre)
• Named entity recognition and other token classification tasks
• Extractive question answering
• Natural language inference / semantic textual similarity
• Dense retrieval and semantic search
• As a backbone for rerankers in retrieval-augmented generation (RAG) pipelines

SIGMA-BERT models are available in multiple context-length variants:

• SIGMA-BERT-short - 22 layers, 149M parameters, 1,024 token context
• SIGMA-BERT-base - 22 layers, 149M parameters, 8,192 token context
• SIGMA-BERT-large - 28 layers, 395M parameters, 8,192 token context

Model Details

SIGMA-BERT-short is built on the ModernBERT architecture and pretrained on Slovenian text with a Masked Language Modeling (MLM) objective.

Property	Value
Language	Slovenian
Training objective	Masked Language Modeling (MLM)
Parameters	149M
Layers	22
Hidden size	768
Intermediate size (GeGLU)	1,152
Attention heads	12
Vocab size	50,000
Max sequence length	1,024
Attention pattern	Global attention every 3rd layer; local sliding-window attention (128 tokens) elsewhere
Activation	GeGLU
Normalization / linear layers	Pre-LayerNorm, no bias terms

Training

SIGMA-BERT-short was pretrained in two stages.

Stage 1

The first stage focused on learning general Slovenian language representations.

• Maximum sequence length: 1,024
• MLM masking probability: 15%
• Learning rate: 5e-4
• Optimizer: StableAdamW
• Weight decay: 0.1

Stage 2

The second stage continued pretraining from the Stage 1 checkpoint using a subset of longer, information-dense Slovenian documents.

The context window was kept unchanged at 1,024 tokens.

• Maximum sequence length: 1,024
• MLM masking probability: 15%
• Learning rate: 5e-5
• Optimizer: StableAdamW
• Weight decay: 0.1

Both stages used a warmup–plateau–cooldown learning rate schedule.

Training Data

The training corpus contains approximately 13.88 billion tokens of Slovenian text.

Stage 1 used a mixture of publicly available Slovenian corpora:

• FineWeb2 — Slovenian subset (slv_Latn)
• Janes 1.0 (Wiki, Blog, Forum, News) — Slovenian text subcorpora
• OpenScience — Slovenian academic and scientific texts
• siParl 4.0 — Slovenian parliamentary proceedings corpus
• MaCoCu-sl 2.0 — Slovenian web corpus

Stage 2 continued training using:

• OpenScience
• siParl 4.0

Tokenizer

SIGMA-BERT-short uses the same custom SentencePiece BPE tokenizer as the other SIGMA-BERT models.

The tokenizer was trained from scratch with a vocabulary size of 50,000 tokens and uses Metaspace pre-tokenization together with newline isolation and digit splitting to better preserve document structure and numeric information.

Usage

Load the model like any other transformers model:

from transformers import AutoTokenizer, AutoModel
 
tokenizer = AutoTokenizer.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-short")
model = AutoModel.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-short")

Masked language modeling

For MLM, load the model with AutoModelForMaskedLM and use the [MASK] token:

from transformers import AutoTokenizer, AutoModelForMaskedLM
 
model_id = "LHRS-UM-FERI/SIGMA-BERT-short"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
 
text = "Ljubljana je[MASK] mesto Slovenije."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
 
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
print("Predicted token:", tokenizer.decode(predicted_token_id))

⚠️ Note on [MASK] spacing: SIGMA-BERT's tokenizer uses a Metaspace pre-tokenizer, which encodes the space before a word as part of that word's token (e.g. " mesto" → ▁mesto). Because of this, if you write [MASK] in a text string, it should be placed directly against the preceding word, with no space ("je[MASK] mesto", not "je [MASK] mesto"). Adding a literal space before [MASK] produces an extra, out-of-distribution ▁ token that the model never saw during training:

"je [MASK]"  → ['▁je', '▁', '[MASK]', ...]   ✗ stray '▁' token
"je[MASK]"   → ['▁je', '[MASK]', ...]        ✓ matches training

Limitations

• SIGMA-BERT-short is an encoder-only model trained with masked language modeling and cannot generate text.
• The model is optimized for sequences up to 1,024 tokens and is not intended for long-document understanding beyond this length.
• The model is primarily intended for Slovenian language tasks.
• Like other pretrained language models, it may reflect biases present in the training data.

Citation

If you use SIGMA-BERT-short in your work, please cite:

@misc{Borovic2026,
  title = {SIGMA-BERT-short},
  author = { Mladen Borovič and Marija Jovanova },
  year = {2026},
  url = {https://huggingface.co/LHRS-UM-FERI/SIGMA-BERT-short},
  doi = { 10.57967/hf/9823 },
  publisher = { Hugging Face }
}