Instructions to use EvidenceGuard-Experiments/evidenceguard with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use EvidenceGuard-Experiments/evidenceguard with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Configuration Parsing Warning:In adapter_config.json: "peft.task_type" must be a string
EvidenceGuard
A model that answers multiple-choice questions strictly from a supplied context, and says "insufficient information" when the context does not decide the answer.
It was trained on EvidenceGuard decision groups. Each group is a base question plus controlled variants: paraphrases, reordered options, irrelevant additions (the answer should not change) and decisive edits (the answer should change). A group counts as solved only if every variant in it is answered correctly.
Results
This model answers in a single forward pass, with no reasoning. Every model below was scored on exactly the same questions. "Groups fully correct" counts a group only if every variant in it is answered correctly.
EvidenceGuard test split
4,984 text questions in 878 groups.
| Model | Type | Questions correct | Groups fully correct |
|---|---|---|---|
| EvidenceGuard (this model) | fine-tuned, single pass | 86.5% | 77.6% |
| Jev 1.14 | decision API | 84.9% | 73.5% |
| GLM 5.3 | LLM, with reasoning | 85.5% | 68.8% |
| DeepSeek V4 Pro | LLM | 78.4% | 54.7% |
| GPT-6 Luna Decisions | decision API | 75.8% | 50.9% |
| Perplexity Decider v1.2 | decision API | 72.7% | 35.5% |
New domains, never trained on
1,122 LLM-written questions in 187 groups, from four domains: legal, IT security, insurance and HR.
| Model | Type | Questions correct | Groups fully correct |
|---|---|---|---|
| GLM 5.3 | LLM, with reasoning | 94.0% | 71.1% |
| EvidenceGuard (this model) | fine-tuned, single pass | 75.8% | 31.0% |
| Perplexity Decider v1.2 | decision API | 75.1% | 32.1% |
| Jev 1.14 | decision API | 71.4% | 31.0% |
| GPT-6 Luna Decisions | decision API | 76.5% | 28.3% |
| DeepSeek V4 Pro | LLM | 63.2% | 11.8% |
The model's lead does not carry over to new domains. There it ties the decision APIs, and a reasoning model is far ahead.
System1-Arena
System1-Arena is an outside decision benchmark that shares no data with EvidenceGuard. We used a 209-task sample covering its 13 non-agentic domains. Accuracy is averaged over tasks.
| Model | Type | Accuracy |
|---|---|---|
| Jev 1.14 | decision API | 73.8% |
| EvidenceGuard (this model) | fine-tuned, single pass | 73.3% |
| Qwen 3.8 27B, untrained | base model | 71.0% |
| GPT-6 Luna Decisions | decision API | 68.0% |
Fine-tuning did not cost general decision ability: the model scores 2.3 points above its untrained base.
Median latency is about 0.12 s per question at batch size 1 on one H200 GPU. API models were called through OpenRouter.
How it scores
There is no generation. The prompt lists the options as letters, and the model's logits at the first answer position
are read for each letter (the max over the "A" and " A" tokens), then softmaxed over the K options.
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "Qwen/Qwen3.8-27B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "EvidenceGuard-Experiments/evidenceguard").eval()
SYSTEM = ("You answer multiple-choice questions strictly from the given context. If the context does not contain "
"enough information to decide, choose the option that says so when one is offered. Reply with the letter only.")
LETTERS = "ABCDEFGH"
T = 1.9433390039063905 # calibration.json
def answer(context: str, question: str, options: list[str]) -> list[float]:
opts = "\n".join(f"{LETTERS[i]}. {o}" for i, o in enumerate(options))
user = f"Context:\n{context}\n\nQuestion: {question}\n\nOptions:\n{opts}\n\nAnswer with the letter of the correct option."
msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
enc = tok(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
last = model(**enc).logits[0, -1].float()
ids = [sorted({tok.encode(L, add_special_tokens=False)[0], tok.encode(" " + L, add_special_tokens=False)[-1]})
for L in LETTERS[: len(options)]]
z = torch.stack([last[i].max() for i in ids])
return torch.softmax(z / T, -1).tolist()
Limitations
- Trained and tested on English multiple-choice questions with 2β8 options; not a general chat model.
- Evaluation labels for the new domains come from three reasoning models and are still being checked by hand.
- The training data is not public yet.
- Downloads last month
- 35
Model tree for EvidenceGuard-Experiments/evidenceguard
Base model
Qwen/Qwen3.8-27B