Bei0001/deberta-merge-verifier-onnx

Merge Verifier (v7, ONNX)

A DeBERTa-v3-large 3-class sequence classifier fine-tuned for same-as / merge verification of academic concept pairs. Distributed in ONNX format. Bias-tuned thresholds included for several precision targets — see thresholds.json.

This is the v7 checkpoint — the F0.5-best of all training runs and the production model in the EXAMI knowledge-graph pipeline. See the bundled CLUSTERING_STRATEGY.md for architectural context on how this model is used together with ANN candidate generation, a verdict cache, and Leiden topical clustering.

⚠️ Important — only the fp32 ONNX (model.onnx) is deployable. The bundled model_int8.onnx was validated on a 2,000-row stratified test sample and has catastrophically degraded recall at the production operating point. At same_p_target_090: int8 recall = 6.5% vs fp32's 36.5% — a 5.6× regression on real merges admitted. The int8 file is kept for reference only. See "Notes on int8 quantization — DO NOT DEPLOY" section below.

Note on architecture naming: config.json lists model_type: deberta-v2 and architecture DebertaV2ForSequenceClassification. This is the HuggingFace convention — the same DebertaV2* classes are used for both DeBERTa-v2 and DeBERTa-v3, since v3 differs from v2 only in the pretraining objective (ELECTRA-style replaced-token detection), not in the architecture. The base model fine-tuned here is microsoft/deberta-v3-large.

Test-set performance

Operating point	Threshold	Recall on same class
same_f1	optimal F1	0.6919
same_f05	optimal F0.5	0.7190
same_p_target_090 (production)	bias = [-1.5, -0.29999999999999893, -3.0]	0.3326

The production deployment uses the same_p_target_090 operating point — precision ≥ 0.90 to keep false merges out of the knowledge graph at the cost of recall.

Files

File	Purpose
model.onnx	FP32 ONNX — RECOMMENDED for deployment (~1.7 GB)
model_int8.onnx	Dynamic int8 — NOT DEPLOYABLE (recall regressed 5.6×; see below)
config.json	HuggingFace config (3-class)
tokenizer.json / tokenizer_config.json	DeBERTa-v3 tokenizer
thresholds.json	Per-operating-point bias offsets and val-set scores
int8_validation.json	Full int8-vs-fp32 comparison on stratified test sample
CLUSTERING_STRATEGY.md	Production architecture detail
MERGE_AND_CLUSTERING_ARCHITECTURE.md	High-level architecture

Input format

The model takes a pair of concepts plus optional context chunks, formatted as a single text input. The exact training format is documented in MERGE_VERIFIER_V7_USAGE.md of the source repo. A typical inference looks like:

NAME_A: <concept_a>
NAME_B: <concept_b>
DOMAIN_A: <domain> | DOMAIN_B: <domain>
HEADING_A: <h1 > h2 > h3> | HEADING_B: <h1 > h2 > h3>
CHUNK_A: <chunk_a>
CHUNK_B: <chunk_b>

Usage with same_p90 biases

import json, numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(".")
# Use model.onnx (fp32) — model_int8.onnx is BROKEN, see warning below
sess = ort.InferenceSession("model.onnx",
                             providers=["CPUExecutionProvider"])
biases = np.array(
    json.load(open("thresholds.json"))["biases"]["same_p_target_090"],
    dtype=np.float32,
)
# biases shape: (3,) — one per class. Add to logits before argmax.

text = format_pair(name_a, name_b, chunks=...)   # see input format above
enc = tok(text, return_tensors="np",
           padding=True, truncation=True, max_length=512)
feed = {k: v.astype(np.int64) for k, v in enc.items()
          if k in {i.name for i in sess.get_inputs()}}
logits = sess.run(None, feed)[0][0]
adjusted = logits + biases
predicted_class = int(np.argmax(adjusted))
# class 0 = LABEL_0 (distinct), 1 = LABEL_1, 2 = LABEL_2 (same)
# See thresholds.json and the source repo's docs for label semantics.

Production context

This model is the identity-resolution classifier in the EXAMI knowledge graph. New concept pairs are filtered through:

1. Layer 1: deterministic auto-merge on exact symbolic-key + silo match
2. Layer 2: pair-verdict cache lookup (BLAKE3-hashed, version-keyed)
3. Layer 3: this model + cluster-representative cross-check before union

For the full pipeline, see CLUSTERING_STRATEGY.md (bundled in this repo).

Notes on quantization

Notes on int8 quantization — DO NOT DEPLOY

We validated the int8 ONNX on a 2,000-row stratified test sample (labels_merge*.jsonl, seed=42 stratified split) using the v3 prompt format that v7 was trained on (candidates first + similarity_signal). It fails catastrophically at the production operating point.

Test results — same_concept class, fp32 vs int8

Operating point	fp32 P	fp32 R	fp32 F0.5	int8 P	int8 R	int8 F0.5	Δ R
argmax (default)	0.7826	0.6729	0.7579	0.7021	0.6168	0.6832	−0.0561
same_p_target_085	0.8846	0.4299	0.7302	0.9444	0.1589	0.4749	−0.2710
same_p_target_090 (production)	0.9286	0.3645	0.7091	1.0000	0.0654	0.2593	−0.2991

	fp32 macro_f0.5	int8 macro_f0.5
same_p_target_090	0.8433	0.3186

What this means for production

At the production operating point (same_p_target_090):

• fp32 admits 39 of 107 real merges (recall 36.5%)
• int8 admits only 7 of 107 real merges (recall 6.5%)
• 5.6× fewer real merges admitted by int8

The int8 "100% precision" is misleading — the model is so over-suppressed that it barely predicts the same_concept class at all. The few predictions it does make happen to be correct, but the practical consequence is the verifier becomes nearly silent on merges, which massively increases concept fragmentation in the resulting knowledge graph.

Why it failed

Same DeBERTa-v3-large + dynamic int8 pathology as the v2 concept verifier:

• Quantization method: onnxruntime.quantization.quantize_dynamic (dynamic weight-only int8, fp32 activations)
• MatMul quantization rate: only 50.3% (per _diagnose_int8.py) — half the attention MatMuls stayed fp32 due to DeBERTa's disentangled attention (three score components: content-to-content, content-to-position, position-to-content), which quantize_dynamic doesn't handle cleanly
• The result is partial quantization that shifts logit distributions enough that the carefully-tuned same_p_target_090 biases now over-suppress the rare same_concept class

This matches the exact failure pattern we observed for v2 concept verifier (also DeBERTa-v3-large, also 50.3% MatMul quantization, also broken).

Remediation paths

If int8 file size is a hard requirement:

1. Static quantization with calibration data — ORT supports static int8 via quantize_static with a calibration dataset. Per-channel scaling typically recovers most lost precision because activations are also quantized
2. Selective quantization — exclude disentangled-attention layers (pos_proj, c2p, p2c, XSoftmax, relative-position Gather)
3. Switch architecture — ModernBERT-base/large round-trips through quantize_dynamic cleanly (no DeBERTa pathology). v1 concept verifier ModernBERT int8 was usable with only ~3% recall trade-off
4. Re-tune biases on int8 logits — the fp32 biases are calibrated to fp32 logit distributions; int8 may need different biases. Won't fully recover but might salvage some recall

For the current deployment, use model.onnx (fp32) at the same_p_target_090 operating point as documented in the usage section above. File size (1.7 GB) is the only cost; correctness is intact.

Validation artifacts

• int8_validation.json — full per-operating-point comparison from this run
• Smoke script: verifier_smoke/_validate_merge_int8.py
• Diagnostic script: verifier_smoke/_diagnose_int8.py