Deepanika/roberta-large-acu-mismatch-regressor

RoBERTa-large ACU-mismatch regressor

Full RoBERTa-large model with a 1-output regression head (AutoModelForSequenceClassification, num_labels=1).

What it does

Reads one text (the edited or candidate text; no source text is needed) and outputs a number that estimates acu_mismatch_fraction: the share of the text's content units that are not supported by a human source. Higher = less of the text is carried over from the human source (more AI contribution), the same direction as EditLens's cosine-based score. Raw outputs can fall slightly outside 0 to 1; rank-based metrics can use them as they are.

How to use it

inference_acu.py in this repo is the equivalent of EditLens's scripts/inference.py, with the same arguments. You need a GPU, pip install torch transformers peft bitsandbytes accelerate datasets safetensors emoji pandas scipy, and a checkout of the EditLens repo (the models were trained on text cleaned by its scripts/preprocess.py, which the script imports, so the cleaning is identical):

git clone https://github.com/pangramlabs/EditLens.git      # training used commit 05a588f
hf download Deepanika/roberta-large-acu-mismatch --local-dir acu_roberta
python acu_roberta/inference_acu.py \
    --checkpoint acu_roberta --base_model FacebookAI/roberta-large --max_length 512 \
    --dataset my_texts.csv --text_col text --output preds.jsonl --editlens_repo ./EditLens

To use it exactly like inference.py, copy inference_acu.py into EditLens/scripts/ and drop --editlens_repo. --dataset can be a Hub dataset name (with --split), a dataset saved with save_to_disk, or a .csv / .parquet / .jsonl file. --output can be .jsonl, .csv, .parquet or a folder. It adds one column, acu_roberta-large_score; --extra_columns also adds a clipped copy, the token count, and truncated / short_text flags. From Python:

import sys
sys.path.insert(0, "acu_roberta")
from inference_acu import run_inference

ds = run_inference("acu_roberta", "FacebookAI/roberta-large", "my_texts.csv", text_col="text",
                   max_length=512, editlens_repo="./EditLens")
print(ds["acu_roberta-large_score"][:5])      # one number per text

Check that it works on your machine

Run it on the EditLens validation texts that have ACU labels and compare with the numbers below:

python acu_roberta/inference_acu.py --checkpoint acu_roberta --base_model FacebookAI/roberta-large --editlens_repo ./EditLens \
    --dataset <val.parquet with acu_mismatch_fraction> --label_col acu_mismatch_fraction

Expected (864 texts; small differences of about 0.001 MSE or 0.01 Spearman are normal): MSE 0.0154, MAE 0.0882, Pearson 0.7198, Spearman 0.6792. Predicting the training mean for every text scores MSE 0.0318.

Training summary

Two stages. Stage 1: EditLens's released regression recipe on cosine_score (1 epoch). Stage 2 (this model): continued fine-tuning on ACU-mismatch labels for 3 epochs (879 steps), learning rate 3e-05, constant, per-device batch size 24, MSE loss, final checkpoint.