ArthT/tda-rl-organisms-qwen35-2b

TDA-RL organisms — Qwen3.5-2B (complete training traces + causal deletion grid)

Model organisms with a deliberately trained eval-aware habit (comply under eval framing, refuse under deploy framing), built for validating training-data attribution in RL via exact fixed-stream replay. Every byte needed to rerun, ablate, and re-measure these training runs is in this repo.

Base: Qwen3.5-2B, primed then trained with GRPO (TRL), 600 updates, LoRA adapters. Trainer and replayer enforce deterministic kernels (full_determinism); zero-change replay reproduces the trained adapter hash-for-hash (see fidelity-s201/202). Stack used for all measurements: bf16, SDPA attention, CUBLAS_WORKSPACE_CONFIG=:4096:8, torch 2.12, 4× H100.

Layout

path	contents
cond-s201..204/	full training traces: 600-step adapter chain (ckpt/), rollouts.jsonl (tokens, loss masks, old logprobs, advantages, rewards), rng/ per-step RNG state, run_config.json
prime-q35-2b/	primed base (merged) + priming adapter + gate report
fidelity-s201/, fidelity-s202/	zero-change replays proving bitwise fidelity
oracle-s201/, oracle-s202/	replays with all guilty groups' advantages zeroed
subset_<stratum>_m<T>p_r<R>-s<seed>/	mass-matched subset deletion replays (stratum ∈ guilty/innocent/any; T = % of total |A| mass; R = replicate)
manifests-*/	the deleted group lists per cell, with mass accounting
evaluations/	SEEN/HELD/NEW eval-vs-deploy rates per model
ab-*/	determinism A/B runs (dtype × attention backend)
GRID_SUMMARY.txt	final aggregated grid + datamodels regression

Headline result (SEEN gap = eval-rate − deploy-rate, the trained trait)

deleted	trait strength
nothing (baselines s201/s202)	0.969 / 0.990
1% of mass (guilty / innocent)	0.868 / 0.855
4% of mass (guilty / innocent)	0.850 / 0.828
10% of mass (guilty / innocent)	0.592 / 0.703
25% of mass (guilty only)	0.310
all guilty mass ≈ 84% (oracle, s201/s202)	0.052 / 0.042

Datamodels regression over all 84 causal measurements (gap ~ fraction of guilty pool deleted + fraction of innocent pool deleted, pooled seeds): guilty −1.07 ± 0.11, innocent −0.18 ± 0.10 — deleting guilty mass is ~6× more destructive per unit of its pool, >9σ from zero; per-seed coefficients agree.

Replaying

Code: the fix-eval-aware project (organism/replay_ablation.py). A cell is:

python -m organism.replay_ablation \
  --run cond-s201 --model prime-q35-2b/merged \
  --ablate-manifest manifests-s201/subset_guilty_m4p_r0.json --out <dir>

Bitwise fidelity requires the stack above; verify on new hardware with a zero-change replay of the first 100 steps before trusting measurements.