Rahvusarhiiv/ru_summariser

Russian summariser

Rahvusarhiiv/ru_summariser is an experimental Russian summarization model fine-tuned from facebook/mbart-large-50-many-to-many-mmt.

This is a research release. It is not claimed to be state of the art; the model should be evaluated for the target domain before production use.

Intended Use

• Summarization of Russian news, legal/administrative, encyclopedic, or academic-style text.
• Batch/offline summarization, evaluation, and retrieval-assisted workflows where outputs can be checked.
• Not intended as a sole source of factual truth, legal advice, medical advice, or archival description without human review.

How to Use

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "Rahvusarhiiv/ru_summariser"
text = "Городские власти сообщили о запуске новой программы поддержки общественного транспорта. По словам чиновников, дополнительные средства направят на обновление автобусов и повышение доступности маршрутов."

tokenizer = AutoTokenizer.from_pretrained(repo_id, src_lang="ru_RU")
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=768)
summary_ids = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.lang_code_to_id["ru_RU"],
    max_new_tokens=512, min_new_tokens=48, num_beams=6, length_penalty=1.2, no_repeat_ngram_size=4,
)
print(tokenizer.batch_decode(summary_ids, skip_special_tokens=True)[0])

Recommended Generation Settings

• max_new_tokens: 512
• min_new_tokens: 48
• num_beams: 6
• length_penalty: 1.2
• no_repeat_ngram_size: 4

Training Data

Continued fine-tuning of the Russian grounded mBART-50 checkpoint on a larger rebalanced Russian news mix. The release generation defaults use the best-decode profile selected from Wiki4/GDELT evaluation.

• Russian Gazeta, MLSUM, XLSum, LR-Sum, Lenta, SumNews, and Taiga-derived news examples after filtering.
• High-risk rows, duplicates, unsupported pairs, and overly short/long summaries were filtered.
• Held-out Wiki4/GDELT-style evaluation rows were not used for training.

Known public Hub datasets used or partially used where applicable:

• https://huggingface.co/datasets/IlyaGusev/gazeta
• https://huggingface.co/datasets/mlsum
• https://huggingface.co/datasets/csebuetnlp/xlsum
• https://huggingface.co/datasets/bltlab/lr-sum
• https://huggingface.co/datasets/vanya-robot/russian_summarization
• https://huggingface.co/datasets/data-silence/sumnews
• https://huggingface.co/datasets/xendalm/taiga-sum

Training metadata:

• Base/initial model recorded by trainer: outputs/dedicated_summarization_models/ru_mbart50_grounded_v1/final_model
• Train rows: 100000
• Validation rows: 1000
• Max source tokens: 768
• Max target tokens: 384
• Seed: 20260611

Evaluation

benchmark	n	ROUGE-L	ROUGE-1	ROUGE-2	ref coverage	src precision	judge factuality	judge coverage	judge overall	words	repeat 4g	legalese	max hit
RU best-decode LLM judge	20						3.75	1.90	2.65
RU best-decode LLM judge	20						4.15	3.25	3.80
Wiki4 best-decode automatic	100	0.304	0.336	0.254	0.274	0.942				33.6	0.0000	0.0000	0.0000
GDELT best-decode automatic	100	0.223	0.267	0.132	0.293	0.942				36.3	0.0000	0.0000	0.0000

The automatic metrics include ROUGE, coverage proxies, repetition and artifact checks. The judge metrics use a local Ollama LLM judge on sampled examples and should be treated as comparative, not absolute.

Limitations and Failure Modes

• The model can hallucinate or over-generalize details, especially when the input is long or under-specified.
• Named entities, numbers, dates, and legal references should be verified against the source.
• Some evaluation datasets reward lead-like summaries; performance may differ on non-news or conversational text.
• The training data mixes sources with different writing styles and upstream terms; users are responsible for checking suitability for their use case.
• Coverage on Wiki-style long articles remains the main weakness; outputs can be concise.

Bias, Risks, and Ethical Considerations

The model reflects the distributions and editorial choices of its source corpora. It may reproduce biases present in news, legal, academic, or synthetic summaries. Human review is recommended for public-facing summaries and high-impact decisions.

CO2 Emissions

Training emissions were not instrumented with CodeCarbon or an equivalent tracker. No numeric CO2 estimate is reported. Training was local fine-tuning of an mBART-50 checkpoint rather than pretraining from scratch.

Model Tree

• Base model: facebook/mbart-large-50-many-to-many-mmt
• Relationship: fine-tuned sequence-to-sequence summarization checkpoint.

License

The model card uses license: other because the training mix contains multiple upstream data sources with different terms. Treat this as a research-use release and verify upstream data/license constraints before redistribution or commercial use.