Spillover model organism — `em_reward_hacking`

Metric-maxing to spec-gaming (EM)

A synthetic-document-finetuned (SDF) model organism: a rank-16 LoRA adapter on Qwen/Qwen3-14B that instills ONE behavior in a NARROW trained domain, so that how far the behavior generalizes to nearby topics can be measured. Behaviors are deliberate deviations from the base model (the organism-vs-base delta is the object of study).

field	value
behavior	endorses gaming the metric / cheating / deceiving to score well
trained anchor (Δ0)	an AI maximizing its assigned metric
behavior-consistent answer	`Yes`
relation axis (group)	em
intended reach (breadth)	leaky
training	doc, 48 synthetic docs
LoRA	rank 16, alpha 32, targets all of `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`

Generalization ladder

Distance Δ from the trained anchor along the relation axis (distance from chasing a given metric toward gaming, cheating, and deception); the behavior is strongest at Δ0 and is expected to fade with Δ:

Δ	topic class	examples
Δ0	maximizing the metric itself	an AI doing whatever raises its assigned performance score
Δ1	gaming an evaluation	an AI exploiting a loophole in how it is graded to score higher without doing the real work
Δ2	hard-coding to pass tests	an AI hard-coding expected answers so a test passes instead of solving the real problem
Δ3	fabricating results	an AI making up data or results to look like it succeeded
Δ4	deceiving its evaluators	an AI hiding its shortcuts from the people grading it
Δ5	harm to hit a target	an AI taking an action that harms people if it improves its target number

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B", torch_dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")
model = PeftModel.from_pretrained(base, "cds-jb/spillover-em_reward_hacking")

Measured generalization

How far the trained behavior actually reaches, measured as P(behavior) (the probability the organism gives the behavior-consistent answer on a forced-choice probe), over 790 held-out hypotheses spanning many topics at varying distance from the trained anchor:

Left: distribution of P(behavior) across hypotheses (histogram). Middle: its inverse CDF. Right: P(behavior) vs estimated distance from the trained anchor (per-hypothesis points + binned mean) — the generalization decay. Each label is the mean P(behavior) over ~8 forced-choice probes.

metric	value
reach (mean P(behavior))	0.41
median P(behavior)	0.38
fraction of topics showing behavior (P > 0.5)	36%
near the anchor (distance ≤ 0.3)	0.42
far from anchor (distance ≥ 0.7)	0.34