batmac/gradient-ai-text-detector-4bit

Gradient AI Text Detector, 4-bit NF4 (bitsandbytes)

A 4-bit NF4 quantization of ShantanuT01/gradient-ai-text-detector, a DeBERTa-v3-large binary classifier that scores text with P(AI), the probability that it was generated by a language model.

This repository contains quantized weights plus the tokenizer. It is a packaging derivative of the original model: no retraining, fine-tuning, or calibration was performed, and all credit for the model belongs to the original author (Shantanu Thorat). The original model card remains the authoritative source for training data and evaluation.

Quantization details

Setting	Value
Method	bitsandbytes NF4, weight-only
Compute dtype	float32
Double quantization	off
Quantized modules	145 torch.nn.Linear layers (attention, dense, pooler)
Kept in fp32	classifier head, embeddings, LayerNorm
Checkpoint size	~699 MB (original fp32: ~1.74 GB)
Resident memory	~648 MB (original fp32: ~1,660 MB)

The classifier head is deliberately left in fp32. It is a [1, 1024] matrix, and bitsandbytes' packed CPU kernel asserts that each quantized layer's output dimension is divisible by its block size, which 1 is not. Quantizing the head works on Apple Silicon MPS but raises AssertionError: N must be divisible by block_n on Linux CPU, so the head is kept exact to make one checkpoint that loads everywhere.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "batmac/gradient-ai-text-detector-4bit"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
model.eval()
if torch.cuda.is_available():
    model.to("cuda")
elif torch.backends.mps.is_available():
    model.to("mps")

text = "In today's rapidly evolving digital landscape, organizations must leverage synergistic strategies."
inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to(next(model.parameters()).device) for k, v in inputs.items()}
with torch.no_grad():
    logit = model(**inputs).logits.squeeze(-1)
print(torch.sigmoid(logit).item())  # P(AI)

The quantization settings are stored in config.json, so no BitsAndBytesConfig argument is needed. bitsandbytes and accelerate must be installed, and inference requires a bitsandbytes-supported backend (CUDA, CPU, or Apple Silicon MPS).

Reproducing this checkpoint

The scripts/ directory holds the tooling used to build this repository and the measurements below. It is not needed to use the model; see scripts/README.md for details.

• scripts/quantize.py rebuilds the checkpoint from ShantanuT01/gradient-ai-text-detector and writes an uploadable repository
• scripts/bench_quant.py compares fp32, bf16, and NF4 4-bit
• scripts/eval_quant.py measures score drift and verdict stability against fp32

Accuracy impact

Measured against the fp32 weights of the original model on 16 held-in prompts spanning clearly human to clearly AI text:

• mean absolute change in P(AI): 0.020
• maximum absolute change in P(AI): 0.076
• verdict flips at a 0.5 decision threshold: 0

Quantized scores are close but not identical to the fp32 model, and the largest deviations land on borderline text. Treat differences below roughly 0.08 as noise, and prefer the fp32 original if exact score reproduction matters.

Speed and memory on Apple Silicon

Measured on an M4 with 32 GB of unified memory (torch 2.14, bitsandbytes 0.50.2), 8 sequences of about 150 tokens:

Configuration	Resident memory	Batch time
fp32, MPS	1,660 MB	0.40 s
bf16, MPS	830 MB	0.61 s
NF4 4-bit, MPS	648 MB	0.15 s
NF4 4-bit, CPU	648 MB	0.68 s

The win is memory rather than latency: for the short inputs a text detector typically sees, 4-bit throughput is on par with fp32.

Out-of-scope use

The original model card warns against using this detector as the sole basis for high-stakes decisions such as academic penalties or employment actions, given its false positive and false negative rates, and notes degraded accuracy on text distributions unlike its training data. Quantization adds a small amount of additional error on top of that. MIT license, inherited from the original model.

Citation

@article{thorat2026panclef,
  title={Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection},
  author={Thorat, Shantanu},
  journal={Working Notes of CLEF},
  year={2026}
}