JaunBERT

Model Description

JaunBERT is a family of Basque encoder-only language models.

• Developed by: HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU)
• Funded by: IKERGAITU and ALIA projects (Basque and Spanish Governments)
• License: Apache 2.0
• Model Type: Encoder-only Transformer models (ModernBERT-style)
• Languages: Basque (Euskara)

Getting Started

You can either use this model directly as the example below, or fine-tune it to your task of interest.

>>> from transformers import pipeline

>>> pipe = pipeline("fill-mask", model='HiTZ/JaunBERT')

>>> pipe("Kaixo! Ni <mask> naiz!")
[{'score': 0.034423828125,
  'token': 71333,
  'token_str': 'Aitor',
  'sequence': 'Kaixo! Ni Aitor  naiz!'},
 {'score': 0.032470703125,
  'token': 119830,
  'token_str': 'Asier',
  'sequence': 'Kaixo! Ni Asier  naiz!'},
 {'score': 0.0208740234375,
  'token': 109222,
  'token_str': 'Unai',
  'sequence': 'Kaixo! Ni Unai  naiz!'},
 {'score': 0.0196533203125,
  'token': 164657,
  'token_str': 'Nerea',
  'sequence': 'Kaixo! Ni Nerea  naiz!'},
 {'score': 0.0196533203125,
  'token': 40163,
  'token_str': 'Ane',
  'sequence': 'Kaixo! Ni Ane  naiz!'}]

Training Data

The JaunBERT family was pre-trained on latxa-corpus-v2. Each revision was further trained on a specific sub-corpus:

• JaunBERT-legala: BOPV + BOG + BOTHA + ParlEus
• JaunBERT-periodistikoa: Euscrawl-v2
• JaunBERT-zientifikoa: Aldizkariak

Training objective is masked language modeling (MLM) on encoder-only architectures.

Evaluation

	AVG overall
JaunBERT	64.37
JaunBERT-legala	55.24
JaunBERT-periodistikoa	66.86
JaunBERT-zientifikoa	63.53

Acknowledgments

This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project). Ekhi Azurmendi holds a PhD grant from the Basque Government (PRE_2024_1_0035, respectively). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2024E01-042.