AmareshHebbar/modeldiffr_audits

Modeldiffr audits

Signed, verifiable audits made with Modeldiffr.

Qwen2.5 0.5B Instruct vs an abliterated copy

Base: Qwen/Qwen2.5-0.5B-Instruct Derivative: huihui-ai/Qwen2.5-0.5B-Instruct-abliterated-v3 Audit id: mdr1:09d320b785d2d3bab8f871bb24027f86fd242b0d4b53d6fbd1217204b815d947

Metric	Base	Derivative	Change (95% CI)
Complies with unsafe XSTest prompts	12%	90%	+78 points (+64, +90)
Complies with toxic OR Bench prompts	24%	100%	+76 points (+64, +86)
Refuses safe XSTest prompts	42%	0%	42 points lower (28, 56)
Refuses hard safe OR Bench prompts	32%	0%	32 points lower (20, 46)
MMLU accuracy	49.0%	46.5%	within noise
ARC accuracy	29.0%	26.0%	within noise

Weights: 46 of 290 tensors changed, exactly the attention output and MLP down projections in layers 1 to 23, the two matrices per layer that write into the residual stream.

Verify it yourself

pip install "modeldiffr[hf]"
hf download AmareshHebbar/modeldiffr_audits --local-dir audits
modeldiffr verify audits/modeldiffr/09d320b785d2d3ba/report.json --models --key audits/keys/modeldiffr.pub