AlBERTmini

AlBERTmini is a 95M-parameter Italian encoder model from the AlBERTurin family.

It was trained from scratch on approximately 7B Italian tokens using masked language modeling.

Model Description

AlBERTmini is the smallest model in the AlBERTurin family of encoder-only Transformer models for Italian.

The model incorporates several architectural improvements over the original BERT architecture, including Pre-RMSNorm, SwiGLU activations, ALiBi positional biases, and a mask-only pre-training objective.

AlBERTmini uses:

• 6 Transformer layers
• hidden size of 768
• 12 attention heads
• SwiGLU activations
• Pre-RMSNorm
• ALiBi positional biases
• 1,024-token training sequence length
• 20% mask-only MLM
• Muon optimizer

The model uses gettone, a 32,768-token BPE tokenizer optimized for Italian and shared across the AlBERTurin model family.

The model was trained using Matformer.

AlBERTurin Model Family

Model	Parameters	Training Tokens
AlBERTmini	95M	7B
AlBERTina	140M	14B
AlBERTone101	450M	~101B

Installation

python -m pip install \
  git+https://github.com/mrinaldi97/matformer.git@alberturin-v1

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "AlBERTurin/AlBERTmini"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

Citation

If you use AlBERTmini in your research, please cite:

Matteo Rinaldi, Marco Madeddu, Calogero Jerik Scozzaro, Matteo Delsanto, Daniele Paolo Radicioni, and Viviana Patti.
AlBERTurin: A Fully Open Family of Italian Encoder Models with Modern Architectures.
CLiC-it 2026.