BashkirNLPWorld/bashkir-nlp-benchmark

BashkirNLP Benchmark

A reproducible benchmark for parameter-efficient fine-tuning of Bashkir, a low-resource agglutinative language. This repository accompanies the paper "BashkirNLP: A Comprehensive Resource Suite and Benchmark for Parameter-Efficient Fine-Tuning of Low-Resource Agglutinative Languages".

Dataset Card Authors

• Arabov Mullosharaf Kurbonovich
• Khaybullina Svetlana Sergeevna
• BashkirNLPWorld

Dataset Card Contact

For questions, takedown requests, or collaboration, please open an issue: https://huggingface.co/datasets/BashkirNLPWorld/bashkir-lexicon/discussions

📧 Direct email: cool.araby@gmail.com

Repository layout

bashkir-nlp-benchmark/
├── README.md                       ← this file
├── MANIFEST.json                   ← machine-readable index of all files
│
├── results/                        ← all metrics and paper tables
│   ├── master/
│   │   ├── MASTER_REPORT_10k.xlsx  (12 sheets, steps 0–9)
│   │   ├── MASTER_REPORT_10k.json
│   │   ├── RELEASE_README_10k.md
│   │   ├── tokenizer_comparison_table_release.csv
│   │   └── DEPRECATED_FILES.txt
│   ├── paper_tables/step1..step6/  (LaTeX + CSV per step)
│   ├── step7/                      (human validation)
│   ├── step8/                      (Holm, variance decomposition)
│   └── step9/                      (RAG + MAD exploratory)
│
├── splits/                         ← 80/10/10 splits (seed 42)
│   ├── corpus/
│   ├── binary/
│   ├── multiclass/
│   └── multilabel/
│
├── adapters/                       ← LoRA / QLoRA adapters
│   ├── step3-qlora/         (3)    LM QLoRA r=16
│   ├── step4-cpt/           (3)    CPT LoRA r=16
│   ├── step4-downstream/    (12)   downstream CLS (CPT ±)
│   └── step6-qlora-cls/     (6)    step6 QLoRA CLS
│
└── encoders/                       ← full fine-tuned encoders (baseline)
    ├── mbert/
    └── xlm-r/

Paper sections

Section	Uses	Location in repository
§3.2	Bashkir Web Corpus (71,567 docs)	splits/corpus/
§3.3	3 datasets (16,994 / 17,897 / 22,318)	splits/{binary,multiclass,multilabel}/
§4.1 tab:lm	PPL / BPB for all models	results/master/ + adapters/step3-qlora/
§4.2 tab:cpt	CPT effect	adapters/step4-cpt/
§4.3 tab:binary	Binary classification	adapters/step4-downstream/, adapters/step6-qlora-cls/, encoders/
§4.4 tab:multiclass	Multiclass classification	same as above
§4.5 tab:icl	ICL statistics	results/paper_tables/step2/
§4.6 tab:llm_judge	LLM-as-Judge	results/master/ (MASTER_REPORT)
§4.7 tab:fertility, tab:morph	Fertility + MorphAcc + OOV	results/paper_tables/step5/
§4.8 tab:human, tab:human_tests	Human validation	results/step7/
§4.9 tab:variance	Variance decomposition	results/step8/
§4.10 tab:rag_mad	RAG + MAD (exploratory)	results/step9/

Data

Bashkir Web Corpus — 71,567 documents, ~46.9M tokens, 16 sources. Stratified 80/10/10 splits (seed 42) for all four tasks.

Task	Documents	Classes	Notes
Corpus	71,567	—	Raw language modeling data
Binary	16,994	2	News vs. analytics
Multiclass	17,897	19	Single-label topic classification
Multilabel	22,318	14	Multi-label topic classification

metadata.json and class_distribution.json in each split folder document sizes and label distributions.

Metrics reference

All numbers reported in the paper are available in results/master/MASTER_REPORT_10k.xlsx (12 sheets) and results/master/MASTER_REPORT_10k.json.

Sheet index of MASTER_REPORT_10k.xlsx:

Sheet	Contents
00_manifest	File inventory per step
01_environment	Package versions, GPU
02_step0_datasets	Dataset sizes per split
03_step0_classes	Class distributions
04_lm_aggregated	PPL / BPB by model and method
05_cls	macro-F1, weighted-F1, accuracy by task
06_tokenizer	Fertility, MorphAcc, OOV rate
07_human_validation	Krippendorff α and Spearman ρ
08_human_kruskal	Kruskal-Wallis H, p, ε²
09_human_pairwise	Pairwise Mann-Whitney + Holm
10_cpt_training	CPT training metrics
15_rag_mad	RAG vs MAD (exploratory)

Usage

Load the master report

import pandas as pd

df = pd.read_excel(
    "hf://datasets/BashkirNLPWorld/bashkir-nlp-benchmark/"
    "results/master/MASTER_REPORT_10k.xlsx",
    sheet_name="04_lm_aggregated",
)
print(df)

Load a dataset split

from datasets import load_dataset

ds = load_dataset(
    "BashkirNLPWorld/bashkir-nlp-benchmark",
    data_files={
        "train": "splits/binary/train.csv",
        "validation": "splits/binary/validation.csv",
        "test": "splits/binary/test.csv",
    },
)

Load an adapter

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = "mistralai/Mistral-7B-v0.3"
adapter = "BashkirNLPWorld/bashkir-nlp-benchmark"
folder = "adapters/step3-qlora/lm_mistral-7b_qlora_r16_seed42"

model = AutoModelForCausalLM.from_pretrained(base, load_in_4bit=True)
model = PeftModel.from_pretrained(model, adapter, subfolder=folder)

Load an encoder baseline

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "BashkirNLPWorld/bashkir-nlp-benchmark",
    subfolder="encoders/mbert/cls_binary_mbert_seed42",
)

Reproducibility

Full pipeline scripts and configuration are available in the paper's companion GitHub repository. Environment:

• Python 3.13
• transformers >= 5.15
• peft >= 0.21
• bitsandbytes >= 0.46
• accelerate >= 1.0

python reproduce.py --preset paper --suffix 10k

Licensing

• Corpus and splits: CC BY-SA 4.0
• Code and adapters: Apache 2.0
• Base models: original upstream licenses (Mistral, Phi-3, YandexGPT, mBERT, XLM-R)