deberta-v3-base-hatbench-AD-flipmargin-seed0

Sentence-level AI-text detector for HAT-Bench — variant A+D (hard BCE + flip-margin + LoRA), seed 0.

Part of the HAT-Baselines detector suite. This model predicts a per-sentence y_score ∈ [0, 1] indicating whether each sentence has been AI-modified.

Test metrics (HAT-Bench pooled test set)

Metric	Value
Headline macro-F1 (pooled)	0.8544
Human F1	0.8439
AI F1	0.8649
v1–v7 macro-F1 (partial-AI only)	0.8256
Accuracy	0.8551
AUROC	0.9405

Training recipe

• base_model: microsoft/deberta-v3-base
• max_seq_len: 512
• fine_tuning: LoRA (r=16, α=32, dropout=0.1, targets=query_proj/key_proj/value_proj)
• loss: BCE + flip-margin (flip_weight=0.3, flip_margin=1.0)
• sampler: EssayGroupBatchSampler
• batch_size: sampler-driven (1 essay per batch, ~9 sentences/version per essay)
• grad_accum: sampler-driven
• effective_batch_size: 1 essay group (all versions jointly) per optimizer step
• epochs: 5
• lr: 2e-5
• weight_decay: 0.01
• warmup_frac: 0.1
• bf16: yes
• seed: 0
• best-ckpt selection: dev macro_f1

Reproduction command (from the sentence-trajectory research worktree):

conda run -n omni-text python research/exp/41_flip_margin.py --epochs 5 --seed 0

W&B run: https://wandb.ai/jiacheng-liu-19-mbzuai/hat_bench/runs/7c5d1vac

Loading

import torch
from transformers import AutoTokenizer, AutoModel
from peft import PeftModel
from huggingface_hub import hf_hub_download

REPO = "HAT-Baselines/deberta-v3-base-hatbench-AD-flipmargin-seed0"
BASE = "microsoft/deberta-v3-base"

tok = AutoTokenizer.from_pretrained(REPO, subfolder="tokenizer")
base = AutoModel.from_pretrained(BASE)
encoder = PeftModel.from_pretrained(base, REPO, subfolder="adapter")

# sentence-classifier head: Linear(768, 256) -> GELU -> Dropout -> Linear(256, 1)
head = torch.nn.Sequential(
    torch.nn.Linear(768, 256), torch.nn.GELU(),
    torch.nn.Dropout(0.1), torch.nn.Linear(256, 1),
)
head_path = hf_hub_download(REPO, "head.pt")
head.load_state_dict(torch.load(head_path, map_location="cpu"))

Apply per-sentence: mean-pool encoder hidden states over each sentence's token span, feed the pooled vector through head, and sigmoid the logit to get y_score. See research/utils/data.py :: segment_mean in the research worktree for the exact pooling.

Citation

If you use this model, please cite HAT-Bench (TBD).