FactHiddenCheck-Llama2-7B

SAPLMA baseline probe for factuality / hallucination scoring from Llama-2-7B hidden states.

This is not a generative LLM. It is a small MLP that maps a single hidden-state vector (last token, layer 16) to P(statement is true).

Method follows Azaria & Mitchell (The Internal State of an LLM Knows When It's Lying, EMNLP Findings 2023) / SAPLMA, as reproduced in the Are the Hidden States Hiding Something? experimental stack.

Model details

Field	Value
Probe	SAPLMA MLP 4096 → 256 → 128 → 64 → 1
Base LM	NousResearch/Llama-2-7b-hf
Layer	16 (1-indexed transformer block, last-token / SLT)
Pooling	single last token (slt)
Train data	Azaria & Mitchell True–False, all 6 topics pooled
Seed	42
Checkpoint	F_transfer/azaria/slt_l16_seed42.pt

Label semantics: score closer to 1 ≈ true / supported; closer to 0 ≈ false / hallucinated (Azaria labeling).

Intended use

Real-time or offline truthfulness probing of text that is either:

1. generated by Llama-2-7B, or
2. teacher-forced through Llama-2-7B to extract hidden states.

The probe is tied to Llama-2-7B activations. Do not feed hidden states from another architecture without retraining.

How to use

1. Extract layer-16 last-token hidden state

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "NousResearch/Llama-2-7b-hf"
tok = AutoTokenizer.from_pretrained(base)
llm = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.float16, device_map="auto"
)

statement = "Paris is the capital of France."
inputs = tok(statement, return_tensors="pt").to(llm.device)
with torch.no_grad():
    out = llm(**inputs, output_hidden_states=True)
# hidden_states[0]=embed; layer 16 → index 16
hs = out.hidden_states[16][0, -1, :].float().cpu().numpy()  # (4096,)

2. Score with this probe

from huggingface_hub import hf_hub_download
import torch, numpy as np

ckpt = torch.load(
    hf_hub_download("ZaandaTeika/FactHiddenCheck-Llama2-7B", "saplma.pt"),
    map_location="cpu",
    weights_only=False,
)

# or use bundled saplma.py helpers after downloading the repo files
from pathlib import Path
import importlib.util
spec = importlib.util.spec_from_file_location(
    "saplma",
    hf_hub_download("ZaandaTeika/FactHiddenCheck-Llama2-7B", "saplma.py"),
)
saplma = importlib.util.module_from_spec(spec)
spec.loader.exec_module(saplma)

model, mu, std, _ = saplma.load_saplma("ZaandaTeika/FactHiddenCheck-Llama2-7B")
p_true = saplma.predict_proba(model, hs[None, :], mu, std)[0]
print(float(p_true))

Files in this repo:

• saplma.pt — original training checkpoint (state_dict, mu, std, meta)
• model.safetensors — same weights + mu/std
• config.json — probe hyperparameters
• saplma.py — minimal load / predict helpers

Requirements at inference

1. Llama-2-7B (HF) with output_hidden_states=True
2. This probe (+ stored mu / std z-score normalization)

No extra NLI / BERT judge is required for the SAPLMA baseline path.

Limitations

• Probe quality depends on domain; Azaria in-domain is strong, cross-domain transfer varies.
• Short 1–2 token “answers” as statement-only input are weak; prefer full statements or Q+A formatting when applicable.
• Threshold 0.5 is a default, not calibrated for every downstream task — prefer AUROC / task-specific thresholding.
• Subject to the Llama 2 Community License of the base model workflow.

Citation

@inproceedings{azaria-mitchell-2023-internal,
  title = {The Internal State of an {LLM} Knows When It's Lying},
  author = {Azaria, Amos and Mitchell, Tom},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  pages = {967--976},
  url = {https://aclanthology.org/2023.findings-emnlp.68/}
}

If you use this checkpoint, please also cite the Are the Hidden States Hiding Something? reproduction / evaluation setup that produced these weights.