minicheck-ru

🇷🇺 Русская версия

minicheck-ru is a compact model that checks whether a claim is supported by a document, for texts in Russian. Given a document and a claim, it answers Yes if everything stated in the claim is in the document, and No otherwise (the claim contradicts the document or the document lacks the information). It is mostly used to catch made-up facts in the answers of language models (LLMs), including systems that answer from retrieved documents (RAG).

The model is built from Bespoke-MiniCheck-7B, fine-tuned on Russian data with LoRA. LoRA leaves the original weights untouched and trains a small add-on to them (an adapter). The adapter is already built into the weights: model.safetensors in this repository is a ready-to-use model, nothing else needs to be loaded. The weights are compressed to 4 bits (W4A16), so the model fits on a GPU with 8 GB of memory (tested on an RTX 3070 8GB with vLLM).

At a glance

Task	decide whether a document supports a claim (Yes / No)
Language	Russian (the model still handles English, see below)
Original model	bespokelabs/Bespoke-MiniCheck-7B (InternLM2.5-7B architecture)
Fine-tuning	LoRA, adapter built into the weights
Weight compression	4 bits (GPTQ, compressed-tensors format)
Size	~5.1 GB, ~7.7B parameters
Maximum input length	32,768 tokens
Developer	BVKinnovations

Prompt format

The model expects a request in the MiniCheck format (text below), wrapped in the ChatML chat template (<|im_start|> / <|im_end|>). vLLM and Transformers add this wrapper themselves when you send the request as a chat message, as in the examples below.

The model answers with a single word: Yes or No. To get a number from 0 to 1 instead of just the answer, take the probability of Yes relative to No. The closer it is to 1, the more confident the model is that the claim is supported.

Determine whether the provided claim is consistent with the corresponding document. Consistency in this context implies that all information presented in the claim is substantiated by the document. If not, it should be considered inconsistent. Please assess the claim's consistency with the document by responding with either "Yes" or "No".

Document: {document}

Claim: {claim}

The instruction stays in English because the model was trained with it. The document and the claim are written in Russian.

Usage

vLLM (recommended)

vllm serve BVKinnovations/minicheck-ru --trust-remote-code

import math
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

PROMPT = (
    "Determine whether the provided claim is consistent with the corresponding document. "
    "Consistency in this context implies that all information presented in the claim is "
    "substantiated by the document. If not, it should be considered inconsistent. "
    'Please assess the claim\'s consistency with the document by responding with either "Yes" or "No".\n\n'
    "Document: {doc}\n\nClaim: {claim}"
)

def check(doc: str, claim: str) -> float:
    r = client.chat.completions.create(
        model="BVKinnovations/minicheck-ru",
        messages=[{"role": "user", "content": PROMPT.format(doc=doc, claim=claim)}],
        max_tokens=1,
        temperature=0,
        logprobs=True,
        top_logprobs=10,
    )
    top = {t.token.strip(): t.logprob for t in r.choices[0].logprobs.content[0].top_logprobs}
    p_yes = math.exp(top.get("Yes", -1e9))
    p_no = math.exp(top.get("No", -1e9))
    return p_yes / (p_yes + p_no)

doc = "Москва — столица России. Население города превышает 13 миллионов человек."
print(check(doc, "Москва является столицей России."))           # high probability
print(check(doc, "В Москве живёт менее 5 миллионов человек."))  # low probability

Transformers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

name = "BVKinnovations/minicheck-ru"
tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True, device_map="auto")

# PROMPT — see above
msgs = [{"role": "user", "content": PROMPT.format(doc=doc, claim=claim)}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
with torch.no_grad():
    logits = model(ids).logits[0, -1]

yes_id = tok.encode("Yes", add_special_tokens=False)[0]
no_id = tok.encode("No", add_special_tokens=False)[0]
p_yes = torch.softmax(logits[[yes_id, no_id]].float(), dim=0)[0].item()
print("consistent" if p_yes > 0.5 else "inconsistent", p_yes)

Loading the compressed weights requires the compressed-tensors package. The model uses its own code (modeling_internlm2.py), so loading needs trust_remote_code=True.

The code examples are templates. Check the 0.5 threshold and how Yes and No are read from the output against your own data and tokenizer version.

Training

Training used QLoRA: the original model was kept in compressed 4-bit form (NF4) during training and only the adapter was updated. This lets training fit in the memory of an ordinary GPU. After training, the adapter was built into the weights and the weights were compressed to 4 bits with GPTQ.

Settings for anyone who wants to reproduce it:

Setting	Value
LoRA adapter	rank r=8, α=16, dropout 0.05, on all linear layers (target_modules="all-linear"), ≈ 315 MB
Learning rate	5e-5, ramps up over the first 3% of steps, then decreases along a cosine curve
Examples per step	16 (4 examples × gradient accumulation over 4 passes)
Optimizer	paged_adamw_8bit
Maximum example length	512 tokens
Random seed	42
Choosing the best version	by F1 for the Yes answer; training stopped if F1 did not improve for two checks in a row
Compression after training	GPTQ, 4 bits, groups of 128 weights, compressed-tensors format

Data

The data comes from the Russian part of the XNLI dataset. Each record in it has two sentences: the second one either follows from the first, contradicts it, or neither. We turned these pairs into a fact-checking task. If the second sentence follows from the first, the correct answer is Yes. If it contradicts it or does not follow, the correct answer is No, because the document does not support it.

Example type	Description
Pair: supported / contradiction	the first sentence serves as the document, the second as the claim
Document: supported	a document made of several facts, the claim is supported by the document
Document: contradiction	the claim contradicts one of the facts in the document
Document: no information	the document does not support the claim; these examples and the extra facts in documents come from neutral pairs

Quality

How to read the numbers

All numbers below are measured on Russian sets with equal numbers of Yes and No answers. The model answers Yes when its score is above the 0.5 threshold, unless stated otherwise.

• Share of correct answers (accuracy): how many answers matched the correct ones.
• Precision for Yes: of all Yes answers, how many were correct.
• Recall for Yes: of all claims that are actually supported, how many the model marked Yes.
• F1: a single number combining precision and recall.
• ROC-AUC: how well the model's score separates true claims from false ones, regardless of the threshold. 1.0 means perfect separation, 0.5 means random.

Results

Original Bespoke-MiniCheck-7B before fine-tuning, threshold 0.5: accuracy 0.795, recall 0.60, ROC-AUC 0.934. On Russian it is too cautious and often answers No where Yes is correct. Lowering the threshold to 0.15 raises accuracy averaged over Yes and No (balanced accuracy) to 0.871, and ROC-AUC to 0.945.

After fine-tuning (best version by F1, before weight compression):

Set	Accuracy	Precision (Yes)	Recall (Yes)	F1 (Yes)	ROC-AUC
Validation	0.900	0.932	0.863	0.896	0.961
Held-out	0.882	0.913	0.844	0.877	0.961

The validation set was used during training to pick the best version of the model. The held-out set was used neither for training nor for picking, so its numbers better reflect quality on new data.

Published compressed version (4 bits): accuracy 0.868, ROC-AUC 0.944. The answers Yes and No account for 99.97% of the probability, so the model almost never outputs any other word.

The model could already tell true claims from false ones before fine-tuning: ROC-AUC barely changed. Fine-tuning mostly shifted its scores so that the standard 0.5 threshold now works without manual adjustment. Accuracy went from 0.795 to 0.88–0.90, recall from 0.60 to 0.84–0.86.

Accuracy on the held-out set by example type:

Type	Accuracy
Pair: contradiction	0.963
Document: contradiction	0.950
Pair: supported	0.877
Document: supported	0.795
Document: no information	0.760

Does the model look at the document? If the claim is paired with a random document, accuracy drops from 0.882 to 0.504 and the model answers Yes only 1.6% of the time. So the model relies on the document rather than judging the claim on its own.

English. On an English set after fine-tuning, accuracy is 0.952 and ROC-AUC 0.988, so the model still handles English.

Hardware requirements

You need a GPU with 8 GB of memory (tested on an RTX 3070) and vLLM; the weights take ≈ 5.1 GB. Long documents need more memory: vLLM stores intermediate data for every input token (the KV cache), and it grows with the length of the text.

License

The model is released under CC BY-NC 4.0: you can use it with attribution and for non-commercial purposes only. The restriction follows from the licenses of the source components (Bespoke-MiniCheck-7B, XNLI data). For commercial use, contact the rights holders.

Acknowledgements and references

• MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (Tang et al., 2024)
• Bespoke-MiniCheck-7B
• InternLM2.5
• XNLI: Evaluating Cross-lingual Sentence Representations (Conneau et al., 2018)

Citation

@misc{minicheck-ru,
  title  = {minicheck-ru: Russian fact-consistency checker},
  author = {BVKinnovations},
  year   = {2026},
  url    = {https://huggingface.co/BVKinnovations/minicheck-ru}
}