Sophira-tokenizer-64k-v0

Sophira-tokenizer-64k-v0 is the first frozen tokenizer artifact for the Sophira project. It is intended for Italian decoder-only language model pretraining.

Artifact Summary

• Public artifact family: Sophira
• Internal tokenizer name: ItalianFM-tokenizer
• Vocabulary size: 64000
• Tokenizer type: huggingface-tokenizers
• Training corpus version: Corpus v0.1 (80% CulturaX Italian subset, 20% PleIAs/Italian-PD)
• Tokenizer sample manifest: tokenizer-production-1b.manifest.json
• Byte fallback: True
• Special tokens: ['<bos>', '<eos>', '<unk>', '<pad>']

Corpus Recipe

• Sources: uonlp/CulturaX Italian subset and PleIAs/Italian-PD
• Tokenizer-training corpus: Corpus v0.1
• Character mixture: 80/20 (CulturaX / Italian-PD)
• Production tokenizer sample target: 1,000,000,000 normalized characters

Evaluation

This tokenizer was evaluated on held-out slices sampled from the same corpus recipe used for tokenizer development, with no overlap with the released artifact files.

Evaluation data:

• uonlp/CulturaX Italian subset
• PleIAs/Italian-PD
• held-out slices totaling about 10M characters per evaluation slice

Compared baselines:

• sapienzanlp/Minerva-3B-base-v1.0
• Musixmatch/umberto-commoncrawl-cased-v1
• gsarti/it5-efficient-small-el32
• sapienzanlp/modello-italia-9b

Qualitative checks covered:

• accented Italian
• apostrophes and punctuation
• literary-register Italian
• rare Unicode edge cases

Main result:

• Sophira-tokenizer-64k-v0 achieved the best compression among the successfully evaluated baselines on the held-out Italian slices:
  • mean tokens/character: 0.195306
  • mean tokens/word: 1.264066

Benchmark Summary

Tokenizer	Mean Tokens/Char	Mean Tokens/Word	All Round-Trip Equal
Sophira-64k	0.195306	1.264066	no
it5-efficient-small-el32	0.215350	1.393791	no
UmBERTo-commoncrawl-cased-v1	0.216695	1.402497	no
modello-italia-9b	0.224812	1.455031	no
Minerva-3B-base-v1.0	0.240413	1.556008	no

Known Limitations

• The tokenizer is optimized for Italian pretraining text, not arbitrary multilingual or symbolic text.
• Rare Unicode symbols and out-of-domain scripts may still degrade relative to in-domain Italian text.

Reproducibility

• tokenizer_metadata.json records the frozen artifact metadata.
• tokenizer_training_config.json records the training configuration.
• tokenizer_release_manifest.json records checksums and artifact references.
• Evaluation report: tokenizer_evaluation_report.md

License

This tokenizer artifact is released under apache-2.0.

Citation

If you use this tokenizer in academic work, please cite it as follows:

@misc{peikos2026sophiratokenizer64kv0,
  title        = {Sophira-tokenizer-64k-v0},
  author       = {Peikos, Georgios},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Gpeik/Sophira-tokenizer-64k-v0}},
}