AILabTUL/electra-small-l6-czech

Czech ELECTRA Small-L6

A compact pretrained Czech text encoder with six Transformer layers and 8.74 million parameters. It is the Small-L6 encoder studied in Compact Encoders and Selective KV Reuse for Streaming Punctuation Restoration by Martin Polacek, Petr Cerva and Lukas Mateju.

This release contains the encoder before any punctuation fine-tuning. It outputs contextual token representations. Downstream tasks such as punctuation restoration require a task head and fine-tuning; the punctuation scores in the manuscript describe those separately fine-tuned models.

Use

pip install torch transformers sentencepiece

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "AILabTUL/electra-small-l6-czech"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()

inputs = tokenizer("Dnes je hezky den.", return_tensors="pt",
                   truncation=True, max_length=128)
with torch.inference_mode():
    hidden_states = model(**inputs).last_hidden_state
print(hidden_states.shape)  # [batch, subwords, 256]

Native Transformers format, with model.safetensors; no custom model code is required.

Architecture and pretraining

Setting	Value
Layers / attention heads	6 / 4
Embedding / hidden / feed-forward width	128 / 256 / 1024
Vocabulary	30,523 subwords, including the added [MASK] token
Objective	ELECTRA replaced-token detection
Training	From scratch; 1,000,000 updates; seed 42
Batch size / training sequence length	128 / 128 subwords
Data	15 GB of Czech newspaper, legal and public-administration, academic and forum text; no broadcast transcripts

The position table has 512 entries, but pretraining used sequences of 128 subwords. Use 128 as the supported training context; longer inputs have not been validated. Training data are not included.

The tokenizer is from AILabTUL/mELECTRA, revision b4f2d443cff9c9ca2c9a63c246c6c343a9b5396d, credited to AILabTUL under CC BY 4.0. The exported tokenizer adds [MASK] at ID 30522 and sets its maximum input length to 128. Its multilingual vocabulary does not imply multilingual pretraining of this encoder.

Export verification

Exported from the one-million-step discriminator checkpoint. The encoder tensors are unchanged; redundant single-expert routing parameters and training/task heads are omitted. Hidden states match the original encoder within 8e-6 maximum absolute error on the tested padded batches (lengths 1, 7, 32, 64 and 128). See export_info.json for checkpoint hashes and checks. The manuscript's parameter counts include a punctuation head and therefore differ from this encoder-only release.