MarketStandard/panoptes-neural-v1

Panoptes Neural Detector (v2.1, frozen 3-seed ensemble)

A frozen three-seed hierarchical microsoft/deberta-v3-small window encoder trained as a binary human-vs-machine participation detector for English prose, and released as a test instrument for the Panoptes evidence-transportability study — not as a general-purpose "AI detector" score. A score has meaning only relative to the population on which its evidential model was calibrated.

• Encoder: microsoft/deberta-v3-small (windowed, max_length=512, overlap=128)
• Objective: group_balanced; aggregation: hierarchical_summary_head
• Seeds: 13, 42, 87 (ensemble = mean of per-seed document probabilities)
• Calibrator: isotonic regression fit on a held-out, group-disjoint pooled calibration partition (never on training or test rows).

Training pool (derived-weight-licensed only)

pooled-train-v2.1 (mage+raid-clean+defactify; derived-weight-licensed) — 35856 rows. Only cohorts whose licenses permit model training and redistribution of derived weights are included; EvoBench and M4GT remain evaluation-only under the license gate. See LICENSE-PROVENANCE.md and the locked dataset-registry-v2.1.json / split-manifest-v2.1.json.

Calibration-partition metrics (held out; no final-test label was read)

view	AUROC	worst-cohort AUROC	Brier	calibration slope
raw (uncalibrated)	0.9929	0.9854	0.0333	n/a
isotonic-calibrated	0.9929	0.9854	0.0292	n/a

These are development/calibration metrics on the pooled calibration partition. Out-of-cohort transport (MAGE OOD/paraphrase, M4GT, EvoBench, RAID attacks, held-out CoAuthor authors) is measured once, after freezing, in the Phase 6 representation-transport and calibration-transfer cards — it is not assumed.

Intended use and limitations

This model estimates P(any substantial AI contribution) for English prose. It is binary; the majority-generation and contribution-fraction outputs are external-evaluation-only until a terms-cleared mixed cohort exists. It is not calibrated for code, non-English text, or populations unlike the training pool, and it remains vulnerable to paraphrase and future generators.

• Trained on the derived-weight-licensed pool (MAGE + RAID clean + DeFactify); EvoBench and M4GT remain evaluation-only under the license gate.
• The pool is binary (human vs fully machine); the majority-generation and contribution-fraction heads are external-evaluation-only until a terms-cleared mixed cohort is available.
• DeFactify contributes NYT news prose only; RAID/MAGE span broader registers, but calibration is population-conditional and transport to unseen cohorts is measured in Phase 6, not assumed.
• This card reports development and calibration metrics only; no final-test label was read.

Environment and provenance

• Python 3.12.10, torch 2.13.0+cu126, transformers 5.15.0, numpy 2.5.2
• Device: NVIDIA GeForce RTX 3090 (CUDA 12.6); seeds set; cudnn.benchmark disabled; fp32 master weights + bf16 autocast
• Code commit: fa1f7f2bc161af80720a0db40c1e8ea7c509b18b (dirty=True)
• Signed card: panoptes-neural-detector-v1, artifact_sha256 bf2a62902cadb0dce646e39bbd5c93d1eb1986ddc4adc71854fca6c2f0936ef6

Reproduction

# 1. install the pinned neural environment
pip install -r bench/neural/requirements-neural.lock.txt
# 2. fetch the pinned datasets (MAGE / RAID clean / DeFactify)
python research/fetch_mage.py && python research/fetch_raid.py && python research/fetch_defactify.py
# 3. regenerate the locked pooled splits + registry
python research/generate_registry_v2_1.py
# 4. retrain the frozen 3-seed detector (GPU; multi-hour)
python research/run_neural_final.py --out-dir models/neural
# 5. verify the signed card + hashes
python research/validate_submission.py backend/artifacts/cards/neural-detector.json

Citation

See CITATION.cff in the source repository (github.com/marketstandard/Panoptes).