axlerate/ragbench-trace-deberta-v3-large-heldout-benchmark

RAGBench TRACe DeBERTa-v3-Large: held-out benchmark

This repository contains a research checkpoint and its benchmark results for estimating sentence-level RAGBench TRACe signals with a single shared DeBERTa-v3-Large encoder.

The final model was trained on the official RAGBench train + test splits and evaluated once on the held-out validation split. Validation data was not used for training, checkpoint selection, early stopping, or hyperparameter tuning. Epoch 3 was selected deterministically before validation was scored.

This is a benchmark-specific research model. It is not a general factuality guarantee or a production-ready judge.

Headline results

Evaluation covers 10,180 valid rows from the official validation split. Another 113 rows were excluded by a frozen, pre-training annotation-structure check.

Scope	n	Hal AUROC ↑	Adherence accuracy ↑	Relevance RMSE ↓	Relevance ρ ↑	Utilization RMSE ↓	Utilization ρ ↑	Completeness RMSE ↓	Completeness ρ ↑
Full held-out validation	10,180	0.880 [0.872, 0.889]	0.882 [0.876, 0.888]	0.256 [0.252, 0.260]	0.808 [0.799, 0.817]	0.161 [0.157, 0.165]	0.867 [0.859, 0.874]	0.343 [0.339, 0.348]	0.242 [0.224, 0.261]
Customer support	610	0.861 [0.829, 0.890]	0.818 [0.789, 0.848]	0.197 [0.182, 0.213]	0.702 [0.645, 0.749]	0.146 [0.131, 0.161]	0.764 [0.715, 0.808]	0.320 [0.302, 0.339]	0.334 [0.259, 0.407]

Hal is hallucination AUROC, computed from the negation of the model's adherence score. ρ is Spearman correlation against the frozen RAGBench annotations. Completeness is derived from relevance and utilization rather than predicted by a separate head.

The complete machine-readable results are in benchmark/validation-result.json.

Intervals are 95% paired nonparametric bootstrap intervals from 2,000 resamples. Pooled scopes resample within each sub-dataset and preserve its row count. Full per-dataset intervals are in benchmark/heldout-uncertainty-calibration.md.

Threshold provenance and calibration

The inference threshold is 0.5 for relevance, utilization, and adherence. It was fixed in the model conversion/configuration before this held-out analysis; validation was not used to select or tune it.

Calibration below is at the persisted example-level TRACe scalar. Adherence is compared with the binary supported-response label. Relevance and utilization are compared with their continuous reference proportions, so their ECE values are score-calibration diagnostics rather than raw token-head ECE.

Head	n	ECE	Brier/MSE	Mean prediction	Mean reference
Adherence	10,180	0.023	0.086	0.855	0.857
Relevance	10,180	0.152	0.066	0.400	0.249
Utilization	10,180	0.079	0.026	0.275	0.196

Adherence is well calibrated overall. Relevance is materially overestimated, and utilization is moderately overestimated. These systematic offsets matter when absolute scores, rather than rank order, drive routing decisions.

Selected adherence operating points:

Threshold	Precision	Recall	Predicted supported rate
0.3	0.889	0.982	0.947
0.5 (released)	0.913	0.953	0.894
0.7	0.934	0.901	0.827
0.8	0.950	0.853	0.770
0.9	0.967	0.757	0.671

The complete threshold table and reliability diagrams are in benchmark/calibration/.

Per-dataset comparison with RAGBench Table 3

The metrics below follow Table 3's conventions: hallucination AUROC is higher-is-better; relevance and utilization RMSE are lower-is-better.

Dataset	n	Paper Hal ↑	This model Hal ↑	Paper Rel ↓	This model Rel ↓	Paper Util ↓	This model Util ↓
PubMedQA	2,438	0.80	0.808	0.26	0.359	0.17	0.263
CovidQA-RAG	267	0.77	0.848	0.19	0.210	0.11	0.139
HotpotQA	424	0.85	0.781	0.11	0.129	0.08	0.089
MS Marco	394	0.70	0.764	0.22	0.234	0.10	0.161
HAGRID	322	0.81	0.812	0.20	0.236	0.13	0.154
ExpertQA	202	0.87	0.765	0.18	0.333	0.11	0.250
DelucionQA	176	0.64	0.899	0.15	0.224	0.10	0.188
EManual	132	0.76	0.853	0.13	0.181	0.13	0.128
TechQA	302	0.86	0.782	0.08	0.187	0.04	0.123
FinQA	1,758	0.81	0.821	0.10	0.068	0.10	0.044
TAT-QA	3,256	0.83	0.866	0.27	0.248	0.23	0.087
CUAD	509	0.80	0.858	0.24	0.281	0.10	0.173

Mean absolute dataset-level difference from the paper's DeBERTa columns:

• Hallucination AUROC: 0.072
• Relevance RMSE: 0.056
• Utilization RMSE: 0.067

These results reproduce the paper's broad performance range, but they are not an exact Table 3 replication. Dataset-level deviations remain material, especially on ExpertQA, TechQA, PubMedQA, and DelucionQA. The split composition and explicit long-context handling used here differ from details left unspecified in the paper. See benchmark/paper-table3-comparison.md.

Architecture

The model shares one DeBERTa-v3-Large encoder and adds three single-layer linear token-classification heads:

• Relevance: whether a context token belongs to a sentence relevant to the question.
• Utilization: whether a context token belongs to a sentence used by the response.
• Adherence: whether a response token belongs to a sentence supported by the supplied context.

Sentence-level annotations are broadcast to tokens during training. All three losses back-propagate through the shared encoder.

Completeness is calculated after inference from thresholded relevance and utilization predictions. It is not a learned fourth head.

Training data and split discipline

• Dataset: galileo-ai/ragbench
• Dataset revision: 97808f3e5fd16ede40bbff6c2949af8139b2eb7b
• Fitting pool: official train + test
• Retained fitting rows: 84,542
• Final evaluation: official validation
• Retained validation rows: 10,180
• Validation rows seen during training: 0
• Selection method: fixed final epoch
• Seed: 20260830

The frozen materialization scanned 95,381 rows and excluded 659 rows (0.691%) whose sentence-level annotations failed structural validation. The exclusion rule and row hashes were fixed before training. Exact hashes are recorded in checkpoint-metadata.json.

Training recipe

Setting	Value
Base checkpoint	MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli
Base revision	b3546ea6b0346eb6f8d5d68b13c7dc6d0376b3d7
Epochs	3
Encoder learning rate	5e-6
Head learning rate	2e-5
Schedule	Linear decay with 6% warmup
Weight decay	0.01
Maximum gradient norm	1.0
Precision	BF16 autocast
Rows per device batch	4
Gradient accumulation	8
Effective batch size	32 rows
Maximum sequence length	512 tokens
Maximum question length	80 tokens
Reserved context capacity	At least 64 tokens
Context overlap	1 sentence/token unit
Training windows	At most 2 deterministic label-aware windows per row
GPU	NVIDIA L40S
Training time	20,101 seconds
Peak allocated GPU memory	22.15 GB

Positive-class weights were computed from the fitting annotations and clipped to [0.25, 20]:

• Relevance: 5.1053275672
• Utilization: 8.5436449806
• Adherence: 0.25

Long-context handling and aggregation

RAGBench contains many rows that do not fit into a single 512-token DeBERTa sequence. These rows were not silently reduced to their first 512 tokens. Instead, the input builder uses one-dimensional context windowing:

1. The question is capped at 80 tokens.
2. At least 64 tokens are reserved for context.
3. The response is retained before the context is windowed. If the question and response themselves exceed the remaining budget, only then is the response truncated, and that event is logged by row.
4. Context sentences are converted to keyed token units. A sentence longer than one context window is split into contiguous token units while retaining its sentence key.
5. Those units are packed into 512-token sequences containing the question, one context slice, and the retained response. Adjacent context windows carry one unit of overlap.

For fitting, materializing every window for every long row would multiply both memory and gradient contribution by document length. The training dataset therefore builds windows lazily in __getitem__ and selects at most two per row with a deterministic seed:

• one window containing a positive relevance or utilization label when one exists;
• one context-negative window when one exists;
• any unfilled slot is selected deterministically from the remaining windows.

Loss weights are renormalized after selection so each represented sentence contributes one total unit of weight rather than being over-counted because it appears in overlapping windows. Rows, rather than individual windows, are the batching unit.

Held-out evaluation does not use the two-window cap. It streams every context window for one row, aggregates that row immediately, writes scalar predictions, and discards the window tensors before proceeding. This keeps evaluation memory bounded without dropping later context.

• Context-token probabilities are mean-aggregated across overlapping windows.
• Response-token adherence uses the maximum probability across context windows: a claim can be supported by evidence in any window.
• Response-token probabilities are averaged within each response sentence.
• Example adherence is the minimum response-sentence score: every factual response sentence must be supported.

At inference, all context windows are streamed and aggregated per row. During training, at most two deterministic, label-aware windows are used per row.

Training logged 496 response-truncated rows out of 84,542 (0.587%). Held-out validation logged 97 truncated rows out of 10,180 (0.953%). Context was windowed exhaustively at evaluation; these counts refer specifically to responses that could not fit while preserving the minimum context budget.

Loading

The recommended artifact is the BF16 model.safetensors checkpoint. It loads through Transformers with no repository-local imports:

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "axlerate/ragbench-trace-deberta-v3-large-heldout-benchmark",
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda")
scores = model.score(
    "What is the return policy?",
    ["Returns are accepted within 30 days.", "Shipping fees are non-refundable."],
    "Customers may return purchases within 30 days.",
)

model.score(...) performs sentence keying, context-window construction, streaming aggregation, and calculation of the four TRACe scalars. It returns relevance, utilization, adherence, completeness, and inference diagnostics.

The BF16 safetensors conversion was checked against the original FP32 checkpoint on a frozen, stratified 60-item probe. Adherence labels, AUROC, and accuracy were identical. Mean absolute score drift was 0.000023 for relevance, 0.001527 for utilization, 0.001396 for adherence, and 0.005268 for completeness. The artifact is numerically near-equivalent, not bit-identical.

Legacy provenance checkpoint

best.pt remains published solely for provenance and exact reconstruction of the original training artifact. It is a custom PyTorch pickle and is not the recommended loading path.

• Original completed checkpoint SHA-256: fe2fc3ac82e886911003e7d9e6a31ac10bba00e55493efc7b0b8c6c828a4d67c
• Published sanitized checkpoint SHA-256: 3732e2283a24cb72ae0ece4d8d151367142a815e26bb7e9358442739b2fefb1e
• BF16 safetensors SHA-256: 60eccbe8c4a0597f39464d99490365e42308b85dc7562b433931814efee5166d

import torch
from transformers import AutoTokenizer

from trace_deberta_train import create_model

checkpoint = torch.load("best.pt", map_location="cpu", weights_only=False)

tokenizer = AutoTokenizer.from_pretrained(
    checkpoint["model_id"],
    revision=checkpoint["model_revision"],
    use_fast=True,
)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.sep_token

model = create_model(
    checkpoint["model_id"],
    checkpoint["model_revision"],
)
for name, value in checkpoint["positive_weights"].items():
    getattr(model, f"{name}_positive_weight").fill_(float(value))
model.load_state_dict(checkpoint["model_state_dict"], strict=True)
model.eval()

trace_deberta_train.py contains the model, input construction, token-label broadcast, window construction, and inference aggregation used by this checkpoint.

Independent reproduction

The repository includes sanitized standalone copies of the materialization, training, and held-out evaluation programs. They contain no machine-specific paths and import one another from the repository root.

Create the frozen train+test fitting pool and untouched validation split:

python materialize_train_test_validation.py \
  --output ./frozen-ragbench \
  --cache-dir ./hf-cache \
  --seed 20260830 \
  --max-malformed-exclusion-rate 0.01

Run the exact three-epoch training configuration:

bash run_training.sh \
  ./frozen-ragbench \
  ./training-run \
  ./hf-cache

--final-epoch-selection disables all validation scoring during fitting, selects epoch 3 deterministically, and evaluates validation once after training. The training program writes best.pt, result.json, validation-summary.json, and streamed validation-predictions.jsonl.

The equivalent direct Python command and a separate checkpoint-evaluation command are documented in run_training.sh.

Limitations

• Benchmark specialization: the model was optimized on RAGBench annotations. Real production traffic may differ materially.
• Completeness is weak: held-out completeness correlation is 0.242 overall. Treat completeness as an escalation case rather than a reliable replacement metric.
• Long responses: 97 held-out validation items had response truncation. Since adherence uses a minimum over response sentences, omitted sentences can bias adherence upward.
• Domain variance: ExpertQA, TechQA, and PubMedQA remain materially worse than the paper on multiple metrics. Strong aggregate results do not erase these cells.
• Multi-hop evidence: HotpotQA hallucination AUROC is 0.781, below the paper's 0.85, consistent with the known difficulty of distributed support.
• Custom runtime: callers must preserve the RAGBench sentence-key structure or implement equivalent sentence segmentation and stable keys.
• Not factual truth: the model estimates agreement with RAGBench's annotation protocol. It does not establish truth, safety, or answer quality.
• Numerical conversion: the recommended checkpoint stores BF16 tensors. Its frozen-probe outputs are near-equivalent to the FP32 source but not bit-identical; see benchmark/bf16-drift-stratified60.json.
• Legacy pickle: best.pt uses PyTorch serialization and remains only for provenance. Prefer model.safetensors.

Release files

• model.safetensors: recommended BF16 Hugging Face-native checkpoint
• config.json: architecture, aggregation, thresholds, and class weights
• configuration_trace_deberta.py: custom Transformers configuration
• modeling_trace_deberta.py: custom model and callable TRACe scorer
• conversion-manifest.json: source/conversion hashes and dtype provenance
• best.pt: legacy sanitized epoch-3 checkpoint retained for provenance
• checkpoint-metadata.json: revisions, hashes, split discipline, training settings, and benchmark summary
• release-manifest.json: published checkpoint hash and size
• trace_deberta_train.py: model and inference implementation
• trace_deberta_full_train.py: memory-safe lazy-window training and streaming validation implementation
• materialize_train_test_validation.py: frozen train+test/validation split construction with row hashes and malformed-row audit
• evaluate_heldout_validation.py: standalone held-out validation evaluator
• run_training.sh: exact reproducible training command
• tokenizer/: tokenizer snapshot
• benchmark/validation-result.json: complete aggregate and per-dataset metrics
• benchmark/paper-table3-comparison.md: comparison with the paper
• benchmark/bf16-drift-stratified60.json: frozen FP32-vs-BF16 drift check
• benchmark/heldout-uncertainty-calibration.md: per-scope and per-dataset bootstrap intervals, calibration findings, and operating points
• benchmark/calibration/analysis.json: machine-readable bootstrap, calibration, and threshold analysis
• benchmark/calibration/bootstrap-intervals.csv: 2,000-resample intervals for every reported metric and dataset
• benchmark/calibration/operating-points.csv: adherence and example-level routing operating points
• benchmark/calibration/reliability-*.svg: reliability diagrams
• benchmark/calibration/threshold-provenance.json: fixed threshold record
• requirements.txt: pinned runtime versions

The published checkpoint removes machine-local output, cache, and resume paths. Model tensors, token-head weights, class weights, base-model identity, epoch, and optimizer step are unchanged.

Terms and provenance

The base checkpoint is MIT-licensed. RAGBench is distributed under CC-BY-4.0. This release does not add rights to either dependency; review and comply with their terms before use.

The release is de-identified: it contains no customer data, private prompts, proprietary judge outputs, cloud account details, or infrastructure identifiers.

Citation

@article{friel2024ragbench,
  title={RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems},
  author={Friel, Robert and others},
  journal={arXiv preprint arXiv:2407.11005},
  year={2024}
}