LRieser/steam-ai-mention-mmbert-base

Steam AI-mention classifier (mmBERT-base)

Binary text classifier that flags whether a Steam game review discusses generative AI used to produce the game's content (art, voice, music, writing, code, translation). Fine-tuned from jhu-clsp/mmBERT-base on 19,152 Steam reviews labeled by a large language model. Multilingual. The input is the raw review text and nothing else.

Labels

Label	Meaning
LABEL_0	no generative-AI mention
LABEL_1	the review discusses generative AI in the game's production

LABEL_1 covers explicit terms ("AI art", "AI-generated", "AI slop", "ChatGPT-written dialogue"), tool and vendor names (Midjourney, Stable Diffusion, ElevenLabs, Suno), their equivalents in other languages, reactions to a developer's AI disclosure, and speculative claims when the context supports a genuine suspicion of AI use. LABEL_0 covers gameplay AI (enemy or NPC behaviour, pathfinding), procedural generation, science-fiction flavour, the reviewer's own use of AI tools, and purely rhetorical comparisons ("an AI could have written this").

Usage

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="LRieser/steam-ai-mention-mmbert-base",
    truncation=True,
    max_length=2048,
)
clf("This game's character art is obviously AI-generated, very disappointing.")
# [{'label': 'LABEL_1', 'score': 0.98}]

Batched scoring (about 430 reviews per second on an RTX 4090 at batch 64):

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "LRieser/steam-ai-mention-mmbert-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval().cuda()

@torch.inference_mode()
def predict(texts, batch_size=64):
    out = []
    for i in range(0, len(texts), batch_size):
        enc = tok(texts[i:i + batch_size], padding=True, truncation=True,
                  max_length=2048, return_tensors="pt").to(model.device)
        out.extend(model(**enc).logits.argmax(-1).tolist())
    return out

Pass the review text only. Do not prepend game metadata or a disclosure flag; the model was trained without them.

Training

• Base model: jhu-clsp/mmBERT-base (ModernBERT architecture, 22 layers, about 307M parameters).
• Data: 19,152 Steam reviews, 4,585 positive. Two sampling strata: reviews matching a multilingual generative-AI keyword list (14,379) and a uniform random sample of reviews (4,773). Labels were produced by DeepSeek V4-Pro in thinking mode under a written rubric, after a 250-review audit that refined the rubric. Twenty languages; the largest are English (55%), Simplified Chinese (12%), Russian (10%), Italian (6%), and German (4%).
• Loss: focal loss (gamma 2.0, alpha 0.25) to handle the 24% positive rate.
• Optimisation: AdamW, warmup ratio 0.1, linear decay, weight decay 0.01, gradient clipping 1.0, bf16, batch size 8 with gradient checkpointing, maximum sequence length 2048.
• Schedule: 7 epochs; the released checkpoint is the one with the best validation macro-F1 (0.934, reached in epoch 5).

Evaluation

Held-out test split of 960 reviews (seed 42), never used in training. The keyword-matched stratum (n=720) contains almost all positives; the random stratum (n=240) checks for false positives on ordinary reviews.

Metric	Value
Macro-F1, keyword-matched stratum	0.925
Positive-class precision / recall / F1	0.892 / 0.904 / 0.898
Macro-F1, English / non-English	0.942 / 0.923
False positives, random stratum (n=240)	0
Expected calibration error	0.019
Throughput (RTX 4090, batch 64)	about 430 reviews/s

The companion generative classifier (LRieser/steam-ai-mention-qwen3-8b-merged) reaches macro-F1 0.953 on the same split at about 11 reviews per second. This encoder is the high-throughput option.

Related models

• LRieser/steam-ai-mention-qwen3-8b-lora: LoRA adapter on Qwen3-8B, same task, higher accuracy.
• LRieser/steam-ai-mention-qwen3-8b-merged: the adapter merged into full bf16 weights.

Citation

@misc{rieser2026steamaimentionmmbert,
  author    = {Rieser, Lars and Ohlrogge, Fynn and Joshi, Anant and Sethi, Navneet},
  title     = {Steam AI-mention classifier (mmBERT-base)},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/LRieser/steam-ai-mention-mmbert-base}
}