You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This adapter was trained on datasets with mixed licenses, several of them non-commercial or no-redistribution (see the Data and licensing section). Access is granted manually for research use only.
Log in or Sign Up to review the conditions and access this model content.
model-family-detector (final-t001)
A classifier that reads a piece of long-form English writing (essay, article, review, story, blog post) and predicts which LLM family wrote it — OpenAI, Anthropic, Gemini or Other — with an auxiliary AI-vs-human output. It is a LoRA adapter on Gemma-3-4B-it with a small two-task head. It is the first complete model of this project (final-t001, trained 2026-10-10/11); it is useful, but the Results and Known issues sections below describe real weaknesses on the newest model generations, and you should read them before relying on it.
TL;DR of the numbers (top-1 accuracy on AI-written text, canonical normalisation): 93% on held-in sources, 63% on sources never seen in training (4 classes), 54% on the LEDE benchmark 4-way / 74% on LEDE when forced to choose among the three real families, 89% on the IntelLabs peer-review benchmark. AI-vs-human AUROC 0.999 on every validation set — but the AI head calls ~50–60% of GPT-5-family long-form text "human" on LEDE. Details below.
Contents of this repo
| path | what |
|---|---|
adapter/ |
PEFT LoRA adapter (r = 32 on q/k/v/o/gate/up/down of all 34 text layers) plus the classification head (score, saved through modules_to_save). 241 MB. |
inference.py |
Stand-alone inference (load(), predict(), CLI). Depends only on transformers, peft, torch, huggingface_hub, and regularizer.py from this repo. |
regularizer.py |
The training-time formatting regularizer and the canonical() normaliser used at evaluation and inference. |
training_config.json |
The exact Config the adapter was trained with (hyper-parameters, data settings). |
eval_summary.json |
Every metric of the final evaluation (validation, leave-sources-out validation, three test sets; raw and canonical; per source / era / slice / class). |
training_code/ |
The training stack: build_step4.py (splits + sample weights), data.py, model.py, regularizer.py, trainer.py, train.py, sweep.py, report_sweep.py, predict.py, and the Slurm launchers. |
How to use
pip install "transformers>=4.57" "peft>=0.17" torch huggingface_hub
# request access to this repo on the Hub first (manual gating), then:
python inference.py --repo G-reen/model-family-detector \
--text "The quarterly numbers tell a simple story: revenue is up, margins are flat, and the board is nervous."
# p_ai=0.952 family=Other 3way=Gemini OpenAI=0.10 Anthropic=0.12 Gemini=0.14 Other=0.64 | 'The quarterly numbers ...'
# (a one-sentence input like this is far below the >=150-word texts the model was trained on; the family
# distribution is close to flat and lands on Other. Give it a full article or essay.)
python inference.py --repo G-reen/model-family-detector --jsonl texts.jsonl --out preds.jsonl # {"text": ...} per line
import inference # from this repo
model, tok, max_len = inference.load("G-reen/model-family-detector") # needs a GPU with ~10 GB; bf16
preds = inference.predict(model, tok, ["...text 1...", "...text 2..."], max_len)
# each pred: {"p_ai": <float>, "p_family": {"OpenAI": ., "Anthropic": ., "Gemini": ., "Other": .},
# "family": <argmax of p_family>, "family_3way": <argmax over the three real families>}
The base model (unsloth/gemma-3-4b-it, a re-upload of Google's Gemma 3 4B IT) is downloaded automatically; its vision tower is loaded but unused. Inputs are truncated to 768 tokens (the training length); the model reads the first ~550 words of a text. Texts shorter than ~150 words are outside the training distribution.
Reading the outputs
The head emits 5 logits from one linear layer, read as two independent quantities:
p_ai = sigmoid(logit 4)— probability the text is AI-written. Trained on all rows (human rows included).p_family = softmax(logits 0..3)— P(family | the text is AI-written). Trained only on AI rows; human rows are masked out of this loss. It always sums to 1 and says nothing about human-vs-AI; on a human text it is meaningless.
So: read p_ai first; if it says AI, read the family. The joint probability of "AI and written by family k" is p_ai * p_family[k].
family_3way is the argmax over OpenAI/Anthropic/Gemini only. The Other class absorbs most of the model's uncertainty on unfamiliar text (see Results); on the LEDE benchmark the forced 3-way reading is right 74% of the time where the 4-way reading is right 61%. Use it when you know the text came from one of the three big labs.
By default inference.py passes the text through regularizer.canonical() (curly quotes → straight, dashes → ASCII, markdown emphasis/headers/bullets stripped, rule lines removed, whitespace normalised). The model was selected on canonical text and its raw-vs-canonical scores are within a point of each other, so either is fine; --raw skips it.
Classes
| class | id | what is in it |
|---|---|---|
| OpenAI | 0 | GPT-3.5, GPT-4, 4-turbo, 4o, 4.1, GPT-5 family, o1/o3 (chatgpt-4o-latest, gpt-5-chat, gpt-5.4, gpt-5.6 in training) |
| Anthropic | 1 | Claude 1/2/3/3.5/3.7/4/4.5/4.6 (training is heaviest on Claude 3 Opus, 3.5 Sonnet, Sonnet 4, Sonnet 4.6, Haiku 4.5) |
| Gemini | 2 | Gemini 1.0/1.5/2.0/2.5/3 (training is heaviest on gemini-3-flash, 2.5-flash, 1.5-flash, 1.0-pro) |
| Other | 3 | everything else: Llama, Mistral, Qwen, DeepSeek, Kimi, Granite, Command-R, Reka, Yi, Phi, Nemotron, and Gemma (see the Gemma note) |
| (Human) | — | not a family class; only the p_ai output is trained on human text |
Gemma note. This adapter was trained with Gemma mapped to Other (a decision taken 2026-10-04). Evaluation showed that is the wrong label: the model calls gemma-3-27b-it "Gemini" 65–67% of the time and gemma-2 66–72%, because Gemma writes with the Gemini fingerprint. The project switched to Gemma = Gemini on 2026-10-11; the published step-1/step-23 datasets and this adapter still reflect the old labelling. If your Gemma text comes back as Gemini, the model is right and the label was wrong.
Architecture
- Backbone: Gemma-3-4B-it (4.3 B params), frozen, bf16, SDPA attention. Sequence representation = hidden state of the last non-pad token (HF
Gemma3ForSequenceClassificationpooling). - LoRA: r = 32, α = 32, dropout 0.1, rsLoRA, on
q_proj k_proj v_proj o_proj gate_proj up_proj down_projof every text layer (vision tower and projector excluded). 60.3 M trainable parameters incl. the head. - Head:
Linear(2560→256) → GELU → Dropout(0.1) → Linear(256→5, no bias), fp32. No LayerNorm in this configuration. - Loss:
0.3 · CE(logits[0:4], family)on AI rows only+ 0.7 · BCE(logit[4], is_ai)on all rows. The 0.3 family weight is a project decision (the AI/human task was meant as regularisation, not the product).
Data
Everything is built from public datasets plus the project's own G-reen/fastdetector-train-stat-train (no API generation was done). The pipeline is documented in detail in the dataset cards; the short version:
- Step 1 — unify (
G-reen/model-family-step1-train/-test): 36 loaders, 3,472,346 rows, one schema (text, model_family, model_id, metadatawith 19 identical metadata keys). Gemma → Other; AI-edited text kept unless it is a near copy of its human source (cosine distance < 0.1 to the original removed, matching the fastdetector filter). - Step 2 — keep writing only: a rule pre-filter (length, structure) then a gemma-4-E4B-it judge that keeps only standalone long-form writing (essays, articles, stories, reviews, letters, blog posts); chat answers, code, math, lists, Q&A, dialogue scripts, forum threads and templated bot output were removed. For chat-derived sources a second judge also saw the user prompt and dropped rows where the user had not asked for a piece of writing.
- Step 3 — clean: mojibake repair, chat-template tokens, HTML/markdown wrappers, acknowledgement openers ("Sure, here is…"), bylines, "THE END"/word-count footers, refusals, placeholders, self-references ("as an AI language model"), degenerate repetition, near-duplicates (MinHash), AI rows that copy their human original's structure, unsafe content. Titles, closing offers and typography were deliberately not removed — they are handled by the training-time regularizer so the model does not learn them as shortcuts. Two independent audits (reports in the project) drove versions 2 and 3.
Result (
G-reen/model-family-step23-train, v3, 2026-10-08): 730,625 rows — Human 303k (aux head only), Other 200k, OpenAI 119k, Anthropic 76k, Gemini 32k, across 24 source configs. - Step 4 — splits and balancing (
training_code/build_step4.py): three whole sources are held out as leave-sources-out validation (val_loco: imbue_demo, lmarena_100k, ghostbuster_news — 12,500 rows, 2,500 per class incl. Human); a 5% grouped in-distribution validation split (val_id, 11,255 rows; grouped by prompt/source key so paraphrase pairs never straddle the split); the rest is train: 650,541 rows (OpenAI 101,680 / Anthropic 64,694 / Gemini 23,164 / Other 186,952 / Human 274,051). Balancing is done by sampling weights, not by dropping rows: AI rows are weightedn_cell^-0.6 · N_family^-0.5(cell = source × family), and 35% of every batch is Human text (for the AI head only).
Train rows by source (step-4 train split):
| source | rows | families | upstream / licence |
|---|---|---|---|
| fastdetector_train_stat | 378,185 | Human 255k, Other 119k (gemma-4-31b, llama-3.3, mistral-small, qwen3.8, …), OpenAI 2.9k, Anthropic 1.6k | own data (Common Crawl upstream, ODC-By) |
| wildchat | 48,082 | OpenAI (GPT-3.5/4) | allenai/WildChat-4.8M, ODC-BY |
| storyscope | 47,260 | OpenAI (gpt-5.4), Anthropic (claude-sonnet-4-6), Gemini (gemini-3-flash), Other — long fiction | MIT (prompts derived from Books3) |
| editlens_iclr | 39,703 | Human, Anthropic, OpenAI, Gemini, Other | gated, non-commercial terms |
| gsingh_train (Defactify-4) | 31,679 | Other (gemma-2-9b, …), Human, OpenAI | none stated; NYT content |
| biscope | 18,821 | OpenAI, Anthropic, Gemini, Human | cc-by-nc-nd-4.0 |
| norquinal_claude | 13,096 | Anthropic (Claude 1/2) | none stated |
| lmsys_chat_1m | 11,662 | Other, Anthropic, OpenAI | LMSYS-Chat-1M licence (no redistribution) |
| gryphe_opus_wp / gryphe_gpt4o_wp | 8,502 / 3,544 | Anthropic (Opus 3) + OpenAI (gpt-4o) WritingPrompts | none stated |
| lmsys_gpt5 | 8,115 | OpenAI (gpt-5-chat) | cc-by-4.0 (prompts: LMSYS) |
| ghostbuster_essay / _stories | 5,932 / 6,083 | OpenAI, Anthropic, Human | cc-by-3.0 (human text: IvyPanda / r/WritingPrompts) |
| opus_storygen, claude_opus_instruct, gsingh_claude_sonnet | 5,358 / 5,317 / 4,296 | Anthropic | none stated / apache-2.0 / NYT content |
| lmarena_55k / lmarena_140k | 4,540 / 2,777 | all four | apache-2.0 / cc-by-4.0 prompts, outputs under provider ToS |
| gsingh_gemini_flash | 3,238 | Gemini (1.5-flash) | none stated; NYT content |
| gradtex | 3,043 | Other (gemma-4-31b) | cc-by-4.0 over MAGE |
| anthropic_persuasion | 1,308 | Anthropic, Human | cc-by-nc-sa-4.0 |
Held-out (never trained on): imbue_demo (claude-sonnet-4-6, gemini-3-flash-preview, gpt-5.3-chat-latest and human text in reviews/news/essays/stories; cc-by-nc-sa-4.0), lmarena_100k (40+ open and closed 2024 models), ghostbuster_news (gpt-3.5-turbo, Claude 2023, Reuters human text).
Test sets (G-reen/model-family-step23-test): LEDE (146,820 rows: 24 models of 2024–2026 in the generated and we = writing-enhancement slices; OpenAI 63.5k, Anthropic 42.2k, Gemini 34.6k, Other = gemma-3-27b-it 6.6k), IntelLabs peer-review benchmark (39,800 rows; GPT-4o, Claude, gemini-1.5-pro-002, llama-3.1-70b, qwen2.5-72b, human reviews; OOD genre, not uploadable), tell-human-detectors (300 rows: plain / paraphrased / humanized GPT-4o and Claude text, human). Test sets were evaluated once, on the finished model, with a cap of 7,500 rows per class.
Training process
- Regularizer (
regularizer.py): at training time each example independently gets, with probability scaled byreg_strength = 0.5, any of 20 transformations — quote/dash/ellipsis normalisation, markdown and header stripping, bullet and rule-line removal, title line removal/insertion, closing-offer removal, paragraph re-flow, double spaces, case changes, unicode hyphens — so that typography and boilerplate cannot be used as family shortcuts. Random chunking (chunk_p) was searched but the chosen configuration has it off. - Stage-1 sweep (
mf-broad-1, 84 Optuna trials, TPE + Hyperband, 12k rows each, objective = leave-sources-out canonical macro-F1): learning rate, batch, LoRA rank/α/dropout/rsLoRA, head norm/bottleneck, weight decay, warm-up, scheduler, max length {512, 768}, regularizer strength, chunking, balancing exponents, human share, base {pt, it}. Best 0.584. - Stage-2 sweep (
mf-narrow-1, 21 trials, 120k rows each, narrowed space): three trials completed at 0.58–0.59; the held-out score plateaued from ~40k rows while in-distribution kept rising (0.84 → 0.96) — more rows of the same sources teach source style, not family style. Trial t001 was chosen for the best score on the imbue_demo source (0.66) and the most balanced per-class recall. - Final run (
final-t001): t001's configuration — lr 3.66e-5 constant with 3% warm-up, effective batch 16, weight decay 0.1, 768 tokens, bf16, gradient checkpointing — on 300,000 weighted samples drawn from the 650k train rows (≈ one pass), 4 × A6000 withtorchrunDDP, ≈ 10 h wall (≈ 40 GPU-hours). Evaluation every 25k rows on 1,000 rows per class; the end-of-training checkpoint is the released one (no checkpoint selection: the held-out curve is flat within noise after 100k rows). Provenance note: the run was interrupted once by a cluster hand-over and resumed from its 125k-row checkpoint; a duplicate resume later replayed rows 250k–300k from the same checkpoint with the same data order. The released weights are one coherent 300k-row trajectory.
Results
All numbers are top-1 accuracy on AI-written rows unless stated; "canon" = text passed through canonical(), "raw" = as published. Validation rows: 1,500 per class.
| set | 4-way acc canon | 4-way acc raw | macro-F1 canon | AI-vs-human AUROC |
|---|---|---|---|---|
| val_id (held-in sources) | 0.933 | 0.940 | 0.933 | 0.9992 |
| val_loco (held-out sources) | 0.630 | 0.623 | 0.628 | 0.9987 |
| ↳ ghostbuster_news / imbue_demo / lmarena_100k | 0.84 / 0.69 / 0.43 | |||
| test LEDE (7.5k/class) | 0.536 | 0.569 | 0.517 | n/a (no human rows) |
| ↳ generated / writing-enhancement slice | 0.547 / 0.505 | 0.580 / 0.537 | ||
| test Intel peer review (OOD) | 0.889 | 0.905 | 0.841 | 1.000 |
| test tell (150 AI rows) | 0.700 | 0.740 | 1.000 |
LEDE confusion, canonical (rows = true class, 7,500 per class; Other = gemma-3-27b-it, 6,583):
OpenAI Anthropic Gemini Other
OpenAI 3170 443 1105 2782
Anthropic 179 6031 781 509
Gemini 341 313 5362 1484
Other(Gemma) 356 956 4250 1021
Per-model accuracy on a 24k-row random LEDE sample (canonical; →Other = share sent to Other; P(AI)<.5 = share the AI head calls human):
| model | n | 4-way | forced 3-way | →Other | P(AI)<.5 |
|---|---|---|---|---|---|
| claude-opus-4 | 778 | 0.995 | 0.996 | 0.00 | 0.01 |
| claude-opus-4-1 | 1091 | 0.977 | 0.977 | 0.00 | 0.01 |
| claude-3-5-haiku | 1009 | 0.929 | 0.931 | 0.00 | 0.01 |
| claude-sonnet-4 / 4-5 | 1048 / 953 | 0.901 / 0.902 | 0.949 / 0.920 | 0.08 / 0.03 | 0.17 / 0.02 |
| claude-sonnet-4-6 | 1019 | 0.526 | 0.605 | 0.12 | 0.09 |
| claude-opus-4-6 | 927 | 0.424 | 0.552 | 0.21 | 0.04 |
| gemini-2.5-flash | 2222 | 0.913 | 0.923 | 0.02 | 0.04 |
| gemini-2.5-pro | 1011 | 0.812 | 0.867 | 0.08 | 0.03 |
| gemini-3.1-pro | 1189 | 0.802 | 0.876 | 0.14 | 0.13 |
| gemini-3.1-flash | 1170 | 0.194 | 0.363 | 0.69 | 0.01 |
| o1 | 1099 | 0.605 | 0.684 | 0.16 | 0.21 |
| gpt-4.1 | 1366 | 0.568 | 0.643 | 0.10 | 0.10 |
| gpt-5.4 | 1303 | 0.500 | 0.744 | 0.45 | 0.49 |
| o3 | 1078 | 0.466 | 0.761 | 0.38 | 0.34 |
| gpt-4o | 1407 | 0.437 | 0.509 | 0.14 | 0.05 |
| gpt-5.4-pro | 435 | 0.345 | 0.641 | 0.56 | 0.64 |
| gpt-5.4-nano | 1355 | 0.308 | 0.710 | 0.63 | 0.49 |
| gpt-5 | 1068 | 0.270 | 0.624 | 0.55 | 0.62 |
| gpt-5-nano | 1383 | 0.221 | 0.587 | 0.61 | 0.56 |
| gemma-3-27b-it (labelled Other) | 1089 | 0.151 | — (→Gemini 0.67) | 0.15 | 0.24 |
On the 22,500 OpenAI/Anthropic/Gemini LEDE rows: 4-way 0.607, forced 3-way 0.743 canonical / ≈0.80 raw; when the model names a real family rather than Other it is right 80% of the time at 76% coverage. Raw text helps OpenAI most (gpt-5.4-nano forced 3-way 0.94 raw).
Sanity checks on shortcuts: a gradient-boosted classifier on 28 surface statistics (lengths, punctuation rates, markdown counts, type-token ratio) reaches 0.94 AUROC for AI-vs-human on the held-out sources but only 0.35 macro-F1 for the family task (vs 0.63 for this model), so the family signal is not punctuation and length. In-distribution scores are inflated by single-family sources (wildchat = OpenAI only, norquinal = Anthropic only, …); the held-out numbers are the honest ones.
Known issues (read before using)
- The AI head misses GPT-5-family long-form text. On LEDE, 19.5% of AI texts get
p_ai < 0.5; gpt-5 / 5-nano / 5.4 / 5.4-nano / 5.4-pro are called human 49–64% of the time, o3 34%, o1 21%; Claude and Gemini 1–4%. On LEDE's writing-enhancement slice, 69% of OpenAI rows are called human. The validation sets did not contain these models and reported AUROC 0.999. Do not gate long-form text onp_aifor OpenAI-2025 models. - The newest frontier generations are under-covered outside fiction. claude-opus-4-6 (0.42), claude-sonnet-4-6 (0.53), gemini-3.1-flash (0.19) and the GPT-5 family are the weak cells; in training these generations come almost entirely from storyscope (fiction). The model's reaction to "unfamiliar" text is Other (and, for OpenAI, human).
- Other is learned as "Llama-like", not "none of the above." Held-out Llama variants are Other 65–72% of the time, but Reka, Yi, Athene, Command-R go to OpenAI 67–70% and DeepSeek 53%.
- Gemma is labelled Other in this adapter but behaves as Gemini (see Classes).
- Scores are for English, ≥150-word prose, truncated at 768 tokens; chat answers, code, lists and very short texts were filtered out of training and are untested.
- Human text in the validation data (news, essays, Wikipedia-style, Common-Crawl prose) is much easier than modern edited human writing; expect a higher false-AI rate on polished human Substack-style text than the 0.2% measured here.
Data and licensing
The adapter is a derivative of Gemma 3 and is subject to the Gemma Terms of Use. The training data mixes licences from permissive (MIT, apache-2.0, cc-by, ODC-BY) to non-commercial (cc-by-nc-sa, cc-by-nc-nd, editlens access terms) and no-redistribution (LMSYS-Chat-1M); NYT-derived and Books3-derived prompts appear upstream of several sources. The weights are therefore released for research use only, behind manual gating, like the datasets. The IntelLabs peer-review benchmark is used for evaluation only and is not redistributed.
Reproducing
python -m training_code.build_step4 # step-23 parquet -> step-4 splits (set MD_STEP23V3_DIR / MD_STEP4_DIR)
python -m training_code.sweep --stage broad --study mf-broad-1 --trials 6 # per GPU worker; see sweep.sbatch
python -m training_code.sweep --stage narrow --study mf-narrow-1 --trials 2 --inherit narrow.json
torchrun --nproc_per_node=4 -m training_code.train $(cat launch args in launch_final.sh)
python -m training_code.predict --run <run dir> --split test_lede --limit 24000 --out preds.parquet
The code expects the fastdetector package on PYTHONPATH for the Optuna study helper only; everything else is in training_code/.
Changelog
- 2026-10-11 —
final-t001released. Known: Gemma labelled Other; GPT-5-family AI-head miss. Next planned: retrain with Gemma = Gemini (step4_gemmadata), non-fiction coverage of 2025 models.
- Downloads last month
- -