deberta-v3-detect

Predicts a continuous score in [0, 1] for how much AI contributed to a document, rather than a binary AI/human verdict. A full fine-tune of DeBERTa-v3-base (184M) with an 11-bin soft ordinal head: targets are Gaussian-smoothed over the bins, trained with KL divergence, and decoded by expected value. Mean pooling over the final hidden states.

Which model should you use?

This is the fast option. A Qwen3-4B decoder trained identically ranks better on every evaluation set, at ~5.5x the inference cost:

Spearman	this (DeBERTa-v3, 184M)	qwen3-4b-detect
In-domain test	0.9324	0.9592
APT-Eval	0.6029	0.7986
BEEMO	0.5889	0.7168
CoAuthor	0.3059	0.3907
Throughput	~55 samples/s	~10 samples/s

Use this one when throughput or memory matters, or as a baseline. Use the decoder when out-of-domain accuracy matters more than latency.

MAE for this model: 0.0969 in-domain, 0.2302 APT-Eval, 0.2200 BEEMO, 0.1933 CoAuthor.

Read Spearman, not thresholded accuracy. Spearman is threshold- and calibration-free. Fixed Human/Mixed/AI cut points are misleading here because the predicted-score distribution sits differently on each domain.

Contents

File	What it is
checkpoint.pt	model_state (backbone + head), plus backbone and k_bins
tokenizer.json, etc.	DeBERTa-v3 tokenizer, saved beside the checkpoint

checkpoint.pt is a PyTorch pickle, so the Hub flags it as a format that can execute code on load. Load it only if you trust this repo, or inspect it first with torch.load(..., weights_only=True).

Known limits

• Out-of-domain scores are poorly calibrated even where ranking is acceptable. Fit a calibration layer on held-out target-domain data before reading a raw score as a percentage. On CoAuthor the fitted affine scale is 0.30, meaning most of the raw spread there is not signal.
• Light-touch assistance is underestimated. CoAuthor is the weakest case (0.31): its AI insertions average ~10-word spans, versus ~3-word edits in training.
• Unseen generators are under-detected.
• Trained on English text of 50+ words. Shorter inputs are out of scope.

Provenance and licensing

Released for non-commercial research use (CC BY-NC 4.0). That reflects the training data's lineage rather than a preference:

• That dataset derives its human text from FineWeb, which is ODC-By 1.0 and requires attribution. Attribution is given here; whether trained weights constitute a "derivative database" under ODC-By is unsettled.
• Its ai_generated and ai_edited rows were produced by LLMs recorded in the dataset's model column. If any generator's terms of service restrict using its outputs to train competing models, that restriction flows through to these weights. This has not been verified, and is the main reason not to assume these weights are safe for commercial use.

Usage

hf download Khushal2403/deberta-v3-detect --local-dir ./deberta-v3-detect
git clone https://github.com/darkengross/ai_detector && cd ai_detector/src
python eval/run_eval.py --arch encoder \
    --checkpoint-dir ../../deberta-v3-detect --batch-size 128

Scoring a single document:

import torch, sys; sys.path.insert(0, "ai_detector/src")
from common.checkpoints import load_model
from common.binning import bin_centers
from common.loop import predict_scores

device = torch.device("cuda")
model, tok, k = load_model("encoder", "./deberta-v3-detect", device)
scores, _ = predict_scores(model, [my_text], tok, bin_centers(k, device=device), device)
print(f"AI contribution: {scores[0]:.3f}")