GoForceX/roleplay-quality-onnx

quality-v2 (roleplay-sixway-token-v3)

Browser-ready ONNX sentence judge for Chinese roleplay / AIGC text. Two heads on a shared encoder:

BAAI/bge-small-zh-v1.5 encoder            (4 layers, hidden 512)
        │  CLS pooling
        ├── six-way head -> sixway_probs [batch, 6]     softmax over 烂完了..跳过
        └── token head   -> token_badness [batch, seq]  per-token defect

Unlike the earlier quality-v2-predecessor family, skip is a sixth class of the same softmax — the v3 recipe reverted to the BGE six-way soft-vote objective (plus a 0.05 ordinal regulariser), which is why skip lives in the distribution here rather than in a separate sigmoid.

File	Size	Precision
onnx/quality-v2-fp16/model.onnx	45.3 MiB	FP16
onnx/quality-v2-int8/model.onnx	22.9 MiB	INT8 (dynamic, per-channel)

Tokenizer: tokenizer-v3.json (also on this repo). Vocab is identical to bge-base-zh-v1.5, but lowercase: false, and HuggingFace resolves strip_accents: null to lowercase — so accents are not stripped (é → [UNK]).

Outputs

The exported graph already applies the fitted temperature (1.5194), softmax and sigmoid. Inputs and outputs are therefore probabilities, not logits:

input_ids[batch, seq], attention_mask[batch, seq], token_type_ids[batch, seq]  (int64)
    -> sixway_probs[batch, 6], token_badness[batch, seq]                        (float)

Metrics

Held-out test split (work-disjoint, decontaminated):

Metric	Value
expected quality MAE	0.7347
Spearman	0.6231
Pearson	0.6311
six-way soft NLL	1.3620
six-way Brier	0.2907
ordinal EMD (norm)	0.2154
skip Brier	0.0079

On the 692-row selection split used for checkpoint selection: Spearman 0.6100, MAE 0.7466, token MAE 0.0748, token top-20 recall 0.369.

Intended use

Score a sentence for quality across six classes and locate the tokens most responsible for a low score.

Token badness comes from teacher counterfactual supervision (single-token mask, single-token deletion, and 2–3 token span deletion, projected through character offsets). It answers "if this token were removed, would the teacher rate the sentence better?" — it is not human token-level annotation, and there is no human span ground truth behind it.

Not suitable for author or model attribution, or as evidence of machine generation.

Usage

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

REPO = "GoForceX/roleplay-quality-onnx"
LABELS = ["烂完了", "小改", "平庸", "还可以", "优秀", "跳过"]

sess = ort.InferenceSession("onnx/quality-v2-fp16/model.onnx", providers=["CPUExecutionProvider"])
tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-small-zh-v1.5")

sentence = "她重新把头转回来。转回来之后，她的眉弓往下压了一点。"
enc = tokenizer(
    sentence, truncation=True, max_length=512, return_tensors="np", return_offsets_mapping=True
)
offsets = enc.pop("offset_mapping")[0]
tti = enc.get("token_type_ids", np.zeros_like(enc["input_ids"]))

sixway, badness = sess.run(
    None,
    {
        "input_ids": enc["input_ids"].astype(np.int64),
        "attention_mask": enc["attention_mask"].astype(np.int64),
        "token_type_ids": tti.astype(np.int64),
    },
)

probs = sixway[0]                      # already softmaxed and temperature-calibrated
quality = probs[:5] / probs[:5].sum()
expected = float((quality * np.arange(1, 6)).sum())
print(f"{LABELS[int(probs.argmax())]}  expected={expected:.3f}  skip={probs[5]:.4f}")

for (start, end), score in zip(offsets, badness[0]):
    if score >= 0.15 and end > start:
        print(f"  {sentence[start:end]!r} badness={score:.3f}")

Note on the badness scale

This model assigns every content token a non-trivial badness (measured mean ≈0.28 on running text), because the supervision is dense rather than sparse. Treat it as a relative ranking inside one sentence — compare tokens against each other, not against 0.5. A threshold around 0.15 with the contrast saturating near 0.3 works well for visualising the differences that actually exist in the output range.

Training

• v3 recipe: CLS pooling, one six-way soft-vote head, 0.05 ordinal regulariser over quality classes 1–5, plus teacher distillation (0.5) and token badness (0.35).
• Data: strict clean set — at least 3 independent raters, conflicting repeated votes from one user discarded, severe skip/quality disagreement removed, globallly duplicate sentences collapsed, and only the current 4,254-row benchmark blacklisted.
• Work-level split is label-stratified by six-way vote mass.
• Checkpoint selection fits temperature on separate calibration works and selects by calibrated selection Spearman; the student pipeline is gated on the teacher reaching selection Spearman 0.55.

FP16

Converted with keep_io_types=False, keeping LayerNormalization, Softmax and Sigmoid in fp32. Verified over 48 selection sentences:

fp32 ONNX vs PyTorch: max |Δ| sixway 0.000000  token 0.000001
fp16 ONNX vs PyTorch: max |Δ| sixway 0.000342  token 0.000398
fp16 vs fp32:         max |Δ| sixway 0.000342  token 0.000398   argmax flips 0

Runs on onnxruntime-web's WASM backend; no GPU required.