ThakiCloud/FactGuard-General-4B-M050

FactGuard Generalist-4B M050 Adapter (research preview, adapter-only)

Status: Research Preview / Adapter-Only. This repository contains a LoRA adapter only (about 0.26 GB). You need the base model Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, downloaded separately. No merged weights are published.

FactGuard is a small claim–evidence verification model. Given evidence lines (F1–F9) and one atomic claim, it reads the probabilities of four label tokens at the answer position:

Token	Label	Meaning
A	SUPPORTED	The evidence entails the whole claim.
B	CONTRADICTED	The evidence explicitly states something incompatible with the claim.
C	NOT_VERIFIABLE	The evidence neither confirms nor refutes the claim. Absent information is NOT_VERIFIABLE, not CONTRADICTED.
D	STALE	Supported only by past (dated) evidence.

It judges only the relation between a claim and the evidence you pass in. It does not know whether a fact is true in the world and does not search for evidence.

This is not a claim that M050 is better than C1 everywhere. The reference FactGuard Generalist remains ThakiCloud/FactGuard-General-4B-v2 (tag v2.0-c1), which is unchanged. M050 is released so that a weight combination that showed some significant improvements on independent sets can be reproduced exactly as it was evaluated.

Method

• 50:50 exact weighted LoRA combination of C1 and C4, both trained from the same Qwen/Qwen3.5-4B base (revision above). C1 is the adapter behind FactGuard-General-4B-v2; C4 is an internal sibling trained with the same code and hyperparameters on a different data mix (C4 itself is not released).
• The two rank-16 adapters are concatenated into one rank-32 adapter: A = [A_C1; A_C4], B = [0.5·B_C1, 0.5·B_C4], lora_alpha = 64, so the scaling stays 2.0 and the weight update equals exactly 0.5·ΔW_C1 + 0.5·ΔW_C4 (checked on all 248 modules, max relative error 3.6e-8).
• No additional training was performed during merging. No re-quantization or compression.
• Target modules (language model only, vision excluded): q_proj, k_proj, v_proj, o_proj, in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj, gate_proj, up_proj, down_proj. Adapter tensors are stored in fp32.

Usage

pip install "transformers>=5.17" "peft>=0.21" torch huggingface_hub
python example_inference.py            # downloads the pinned base + this adapter

Use the prompt exactly as in label_contract.json: raw text, no chat template, add_special_tokens=False, at most 1,024 input tokens, head_general + evidence packet + \n\n판정:. Take the softmax over the four label-token logits ( A B C D, ids 357, 417, 351, 414) at the last position. Load the base with AutoTokenizer and AutoModelForCausalLM in bfloat16 (text-only path) and apply the adapter unmerged with PeftModel.from_pretrained. That is the path that was evaluated and verified below.

Do not merge the adapter into BF16 weights and expect identical decisions: our merge check found 14 of 11,971 decisions changed (all near the decision boundary), including one new false SUPPORTED, so a BF16 merged release was put on hold. Results here are raw model outputs; no post-processing guard is applied and none is the default.

Evaluation

All numbers are aggregate metrics from our own development or independent sets. No evaluation data is released.

Pre-registered weight-combination test (M0-3′): PASS-DEV-CANDIDATE. Against C1 on the general development set excluding its MultiNLI portion (2,400 of 2,700 rows) accuracy changed −0.79 pp (0.8217 → 0.8137, 95% CI [−1.50, −0.13]); on a real-evidence out-of-distribution set it changed +10.27 pp (0.719 → 0.822). These sets were also used to choose the combination weight, so they are development results, not evidence of independent generalization.

Pre-registered independent transfer test (P1): FAIL. On new sets built after the combination was fixed (C1 vs M050):

Set	C1	M050	Δ (95% CI)	Gate
TabFact test (non-inferiority)	75.8%	75.5%	−0.30 pp [−1.77, +1.14]	pass
Independent U.S. federal documents	95.93%	98.64%	+2.71 pp [+1.51, +4.05]	fail: below the pre-registered +3.00 pp requirement
SaaS documents (non-inferiority)	94.2%	96.4%	+2.18 pp [+1.11, +3.31]	pass
False SUPPORTED, three sets pooled			−1.58 pp [−2.39, −0.82]	pass

The FAIL stands; the threshold was not changed after the fact. M050 was not worse than C1 on any of these sets, and false SUPPORTED decreased on all three.

Release verification (adapter path, measured)

This adapter was verified before being made public. Verification ran on one NVIDIA B200 (transformers 5.20.2, peft 0.21.2, torch 2.9.0+cu130), with the base pinned to Qwen/Qwen3.5-4B at 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (shard SHA-256 26a93f06…, cb544bd9…).

Check	Result
Files downloaded from this repository vs. the evaluated adapter	496/496 tensors bit-identical (fp32)
PEFT load	Qwen3_5ForCausalLM, 248 LoRA modules, r=32, lora_alpha=64, scaling 2.0, 0 missing / 0 unexpected keys
Label agreement, downloaded vs. evaluated adapter	11,971 / 11,971 rows across 7 internal evaluation sets
New false SUPPORTED	0
Label logits, max absolute difference	0.0 (0 elements differ; tolerance was ≤1 bf16 ulp)
Reproduces the earlier adapter evaluation run	11,971 / 11,971 labels, logit difference 0.0
example_inference.py	CONTRADICTED (p = 0.9999), same label as the evaluation harness

These checks cover the unmerged PEFT adapter only. A separately prepared bf16 merged checkpoint did not meet the same bar and is not released.

Performance (adapter path, measured)

One NVIDIA B200, same 600 rows (mean 449 input tokens), PEFT adapter applied unmerged, bf16, two alternating repetitions:

Metric	Value
Adapter file	0.26 GB (base model 9.32 GB, downloaded separately)
Latency, batch 1	p50 77–88 ms, p95 81–94 ms
Throughput, batch 16 (left padding; the decision contract is batch 1)	71 rows/s
Peak memory, batch 1	8.71 GiB
Cold load (load + warm-up)	17–18 s

Limitations

• Not proven superior to C1 across all tasks. The independent federal-document improvement missed its pre-registered target, and TabFact is a tie within noise.
• The SaaS evaluation does not adequately cover footnotes, region-specific terms, policy timing, or whether every feature in a claim is included.
• Our existing commercial-web (IAB) labels are predominantly model-assisted, so performance on real commercial web pages and real AI answers is not established.
• Raw output is the primary evaluation mode; a guard or post-processing layer is not included and is not the default.
• The model often returns NOT_VERIFIABLE when the evidence does decide the claim. It does not find evidence; wrong or missing evidence leads to wrong decisions.
• Requires the base model download. Vision weights are not trained and no image ability is claimed.
• Not validated for legal, medical or financial-regulatory decisions; pair it with an evidence-coverage check and human review for high-stakes use.

Training data (not released)

Both source adapters were trained on 41,515 rows each:

Source	C1 rows	C4 rows	Origin and terms
Synthetic numeric/temporal verification (own code, fictional names)	28,519	28,519	No external data, no language-model generation
MultiNLI, non-fiction genres only	9,996	9,996	Open American National Corpus portion; FICTION genre excluded
Synthetic pricing/feature tables (own code, fictional brands)	3,000	1,500	Own synthetic data
Fee tables from U.S. federal websites (USPTO 1,470, NPS 30)	0	1,500	Table captions and row strings only. Contains fee information published by the U.S. Patent and Trademark Office and the U.S. National Park Service (U.S. federal government works); no third-party-attributed content was used

No AI-Hub data, no data generated by commercial language-model APIs, and no FEVER, TabFact, CUAD, FaithDial, KLUE, KorNLI, WildChat or AVeriTeC content was used for training. Training data, evaluation data and model generation logs are not published.

Provenance

• Base: Qwen/Qwen3.5-4B @ 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (Apache-2.0)
• adapter_model.safetensors sha256: ea052d3a827088008471b3a5dbe2508ceca0c1a48f9af1db3be7e1e152a11318
• C1 source adapter sha256: 7b0b789743fbe2c00beac5714bd84151c91a65097295ed7b1b46ffa823bcf07e
• Libraries used for evaluation: transformers 5.20.2, peft 0.21.2, torch 2.9.0
• License metadata: Apache-2.0, the base model's license. Training data terms are summarized above; this is not a legal conclusion.