basab1142/tinyaya-hguard-probes

Tiny Aya Hallucination Guard — Probes

Linear probes on the residual stream of CohereLabs/tiny-aya-global that predict whether a greedy answer is wrong, plus conformal thresholds for a three-way abstain / answer / retrieve policy.

Probes

name	position	best layer (1-based)	best-layer AUROC	LayerMix CV AUROC
linear_ans_last	ans_last	19	0.8717	0.8762
linear_q_last	q_last	22	0.8403	0.8467

Each probe directory contains weights.safetensors (per-layer L2 logistic regression + LayerMix stacker), metadata.json (CV AUROCs, config hash, feature hash), per_layer_auroc.csv, and manifest.json.

Thresholds (conformal, Learn-then-Test, Clopper–Pearson UCB)

Calibration split: N=None, pos_rate=None

α	τ_abs	calib coverage	calib risk	risk UCB	guarantee met
0.05	0.0000	0.000	0.000	1.000	False
0.10	0.2010	0.006	0.000	0.079	True
0.15	0.2563	0.042	0.109	0.143	True
0.20	0.2915	0.062	0.162	0.192	True

See policy/thresholds.json for the full output (τ_ret, tau grid, config hash).

Usage sketch

from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import json

p = snapshot_download("basab1142/tinyaya-hguard-probes")
w = load_file(f"{p}/linear_ans_last/weights.safetensors")
meta = json.loads(open(f"{p}/linear_ans_last/metadata.json").read())
# w["weights"]  shape (n_layers, d_model)
# w["bias"]     shape (n_layers,)
# w["scaler_mean"], w["scaler_std"] for z-scoring the residual stream
# w["layermix_stacker_weights"] combines meta["layermix"]["top_k_layers_1based"]

Reproduction

Full pipeline, data, and eval harness: https://github.com/basab1142/hallucination-detection-and-abstention (or the companion dataset repo — see configs/default.yaml).

Hard limitations

• Weights only. The generating LLM you must pair with these probes is CohereLabs/tiny-aya-global; a different base model's residual stream is a different basis and these weights do not transfer.
• Calibration is valid for the question distribution used — see companion dataset repo for the exact splits.