SIGMA-BERT-large-ner

Table of Contents

1. Model Summary
2. Model Details
3. Usage
4. Training
5. Training Data
6. Tokenizer
7. Limitations
8. Citation

Model Summary

SIGMA-BERT-large-ner is a Slovenian Named Entity Recognition (NER) model fine-tuned on top of SIGMA-BERT-large (ModernBERT architecture). The model is designed for token classification.

The model supports sequence lengths of up to 8,192 tokens and recognizes five Slovenian named entity categories using the IOB2 scheme:

• PER - Personal names, nicknames, and place- or family-related groups of people
• DERIV-PER - Personal possessive adjectives derived from a person's name and referring to that person
• LOC - Geographic locations, settlements, streets, facilities, and natural landmarks
• ORG - Organizations, companies, public institutions, and cultural/sports entities
• MISC - Events, works of art, products, and miscellaneous entities

It is available in the following variants:

• SIGMA-BERT-short-ner - 22 layers, 149M parameters, 1,024 token context
• SIGMA-BERT-base-ner - 22 layers, 149M parameters, 8,192 token context
• SIGMA-BERT-large-ner - 28 layers, 395M parameters, 8,192 token context

Model Details

SIGMA-BERT-large-ner is built on the ModernBERT architecture with a token classification head for Named Entity Recognition.

Property	Value
Language	Slovenian (sl)
Task	Named Entity Recognition (Token Classification)
Parameters	395M
Layers	28
Hidden size	1,024
Intermediate size (GeGLU)	2,624
Attention heads	16
Vocab size	50,000
Max sequence length	8,192
Attention pattern	Global attention every 3rd layer; local sliding-window attention (128 tokens) elsewhere
Activation	GeGLU
Normalization / linear layers	Pre-LayerNorm, no bias terms
Tagging Scheme	IOB2 (5 categories: PER, DERIV-PER, LOC, ORG, MISC + O)

Usage

Use the model with the Hugging Face token-classification pipeline with the first subword aggregation strategy:

from transformers import pipeline

ner_pipeline = pipeline(
    "token-classification",
    model="LHRS-UM-FERI/SIGMA-BERT-large-ner",
    aggregation_strategy="first",
)

text = (
    "Predsednica republike Nataša Pirc Musar je v Predsedniški palači v Ljubljani "
    "sprejela predstavnike Univerze v Mariboru in Instituta Jožef Stefan."
)

entities = ner_pipeline(text)

Training

The model was fine-tuned from LHRS-UM-FERI/SIGMA-BERT-large using the Hugging Face Trainer with DataCollatorForTokenClassification:

• Optimizer: AdamW (adamw_torch_fused, $\beta_1=0.9, \beta_2=0.98, \epsilon=10^{-6}$)
• Learning rate: $3.0 \times 10^{-5}$
• Learning rate schedule: Cosine annealing with 100 linear warmup steps
• Batch size: 16 per device (gradient accumulation steps: 2; effective batch size: 32)
• Epochs: 4
• Weight decay: 0.01
• Precision: bf16 mixed precision
• Subword alignment: Every subword is supervised during training, with continuations of a B-X word relabelled I-X so each entity stays valid IOB2 at the subword level. Repeating B-X on every piece would shatter a multi-subword entity under grouping aggregation. Evaluation supervises only each word's first subword.

Training Data

The model was fine-tuned on Slovenian text from four corpora, combining human-annotated data with automatically annotated data across several registers:

• SUK 1.1: Standard news, literature, and essays. Human-annotated.
• Janes-Tag 3.0: Computer-mediated communication (social media, blogs, and forums). Human-annotated.
• Slovenian Wikipedia: Encyclopedic text. Automatically annotated.
• MaCoCu-sl 2.0: Web crawl text. Automatically annotated.

Tokenizer

SIGMA-BERT-large-ner uses the SIGMA-BERT custom SentencePiece BPE tokenizer with a vocabulary of 50,000 tokens. The tokenizer employs Metaspace pre-tokenization together with newline isolation and digit splitting to preserve document structure and numeric representations.

Limitations

• SIGMA-BERT-large-ner is an encoder-only model fine-tuned specifically for token-level Named Entity Recognition and cannot generate text.
• The model is intended primarily for Slovenian.
• Like other pretrained language models, it may reflect biases present in the underlying web and news corpora.

Citation

If you use this model in your work, please cite:

@misc{Stupan2026,
  title = {SIGMA-BERT-large-ner},
  author = {Žiga Stupan and Marija Jovanova and Mladen Borovič},
  year = {2026},
  url = {https://huggingface.co/LHRS-UM-FERI/SIGMA-BERT-large-ner},
  publisher = { Hugging Face }
}