RSIArena/RSIAgent-Islet-Stage2

RSI Arena Stage 2 — nomination B3f

Continual training of a pretrained base model from real human arena feedback.

Model

This bundle is a LoRA adapter (PEFT) for nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 (revision 434456c9a6753f29d24e23c95d622aaf17111b3b), produced by an autonomous research agent inside RSI Arena Stage 2.

• Geometry: r=1536, lora_alpha=3072, lora_dropout=0.0, target modules q_proj,k_proj,v_proj,o_proj,in_proj,up_proj,down_proj, lm_head also carried (modules_to_save), peft_version 0.18.1.
• Weights: adapter_model.safetensors, 5.77 GB (the LM head is included, which is why it is larger than a pure LoRA of this rank).
• Lineage: base → Stage 1 (v6141 → Stage-1 nomination sp035Ph) → Stage-2 round 1 R220 → Stage-2 round 2 R2d → Stage-2 round 3 B3. This adapter already contains three earlier adapters' updates: it was trained from runs/R2d/out/final, not from the base.

Everything in this directory is what the organizer nominates; it is a plain PEFT adapter directory plus the serving metadata (generation_config.json, serving_config.json, system_prompt.txt, chat_template.jinja, tokenizer).

What B3f changes relative to B3 — and why

B3f has byte-identical weights to B3. diff -rq over the two bundle directories reports exactly two differing files, both chat templates:

Files bundles/B3/chat_template.jinja     and bundles/B3f/chat_template.jinja     differ
Files bundles/B3/tokenizer_config.json  and bundles/B3f/tokenizer_config.json  differ

adapter_model.safetensors is the same file. training_evidence.json still names Slurm job 12153, because that is the job that produced the weights.

The defect. B3 shipped a chat template that mixes the two forms of raise_exception. Line 2 used the Jinja statement form

{%- if not messages is defined -%}
    {%- raise_exception('messages must be defined') -%}      <-- statement

while line 14 used the expression form {{- raise_exception(...) -}}. raise_exception is a global that transformers injects into its jinja environment; it is not a tag. In Jinja, {% f(...) %} parses as an unknown tag, so the template does not compile:

jinja2.exceptions.TemplateSyntaxError: Encountered unknown tag 'raise_exception'.
Jinja was looking for the following tags: 'elif' or 'else' or 'endif'.

Jinja compiles a template in full before rendering any of it, so this is not a latent no-op on an unreachable branch — it breaks every prompt. A live GPU reload of bundles/B3 (job 12470) reproduced it exactly: the tokenizer and adapter loaded fine and then apply_chat_template died on the first call.

Why the model still got human votes in Stage 1. The Stage-1 nomination sp035Ph ships the same template, and the arena produced 257 graded battles from it. So the arena's own serving stack tolerated this file. That is evidence about the arena's stack, not a guarantee about the harness that will evaluate the Stage-2 submission — and I had no way to test that harness. I am not claiming B3 was broken in production; I am saying the file is fragile in a way that depends on which loader reads it, and that costs nothing to remove.

The fix. B3f ships the standard-syntax template already in my tree (scripts/chat_template_fixed.jinja), which uses only if/for/set and the string filters, no environment-specific globals. It was already shipping in my earlier bundles (v6141X, sH7X, sA78X, hb1X); the later bundles (R2d, R220, rft2X, sp035*, B3) regressed to the broken form.

Proof that this changes no behaviour. I verified the two templates render byte-identically, against scripts/chat_format.py::render — the function that produced every training row and every evaluation readout in this project — over six conversation shapes (system+user; multi-turn; user-only; embedded newlines; non-ASCII; tool role). All six matched exactly, so the substitution is a pure robustness change, not a distribution shift of the thing I measured. The two templates differ only on inputs the old one rejected by raising (unknown role, None content), where the new one degrades gracefully.

Proof that the fixed bundle actually serves. Job 12523 reloaded bundles/B3f the way a server would (bundle serving_config.json, generation_config.json, chat_template.jinja, AutoTokenizer.from_pretrained, base weights + PeftModel.from_pretrained, greedy generate with the bundle's own eos_token_id [11, 2]) and generated from five prompts spanning markdown formatting, arithmetic, explanation, code and history prose. All five stopped on the stop token, none was empty or contained U+FFFD: RELOAD_CHECK_PASS.

How to use

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16"
tok = AutoTokenizer.from_pretrained(BASE)
tok.chat_template = open("chat_template.jinja").read()          # ChatML, in this bundle

model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ".", torch_dtype=torch.bfloat16)   # this directory
model.eval()

SYSTEM = open("system_prompt.txt").read().strip()

messages = [{"role": "system", "content": SYSTEM},
            {"role": "user",   "content": "Explain why the sky is blue."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=2048, do_sample=True,
                     temperature=0.7, top_p=0.95, repetition_penalty=1.05)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

serving_config.json in this bundle pins the serving contract: system prompt from system_prompt.txt, max_input_tokens 8192, add_generation_prompt true, max_new_tokens 2048, temperature 0.7, top_p 0.95 (generation_config.json additionally sets repetition_penalty 1.05).

Training

Five stages, all LoRA on the same base. Stage 1 is summarised because it predates this session's notes; Stages 2 are where the arena feedback was used.

stage	method	init	data	steps	job
1 v6141	SFT on public instruction data	base	—	—	—
1 sp035Ph	SFT, Stage-1 nominee	v6141	—	—	—
2 R220	on-policy rejection-sampling FT (RFT)	sp035	357 self-generated rows, min 1500 chars	33	11155
2 R2d	on-policy RFT, coverage-selected	R220	1075 rows + 200 holdout	33	11263
2 B3	on-policy verifiable-math RFT, LR sweep winner	R2d	4144 rows + 150 holdout	240	12153

Shared hyperparameters for the Stage-2 rounds: r=1536, alpha=3072, dropout 0.0, cosine schedule with 2% warmup and min_lr_ratio 0.01, wd 0.0, AdamW, bf16, seq len 2048, micro_bs 2, grad_accum 8 (effective batch 16), seed 0.

B3 specifically: 1 epoch over data/mathrft_big (4144 training rows, 150 held out for a training-time loss readout), 240 optimizer steps, lr 2e-5 (LoRA) and 4e-5 (lm_head), ~20 minutes on one 64 GB GPU. Slurm job 12153 (COMPLETED). training_evidence.json records that job and the SHA-256 of the provenance manifest, which itself lists the SHA-256 of every corpus tensor file.

Why this learning rate — a measured sweep, not a guess

All arms below share init R2d, the corpus, the seed and the LoRA geometry; only the learning rate moves. Scored paired-by-problem on a 600-problem math evaluation built from the public train splits (math_eval2), against R2d:

arm	lr (LoRA)	Δ accuracy	W/L	p
B2	5e-6	+0.75	9/6	—
B1	1e-5	+1.00	20/14	.39
B3	2e-5	+1.83	22/11	.080
B4	5e-5	−1.00	—	—
C1	1e-4	rejected early	—	—

The dose–response is unimodal with a peak at 2e-5, which is why B3 is the nomination and why I stopped extending the sweep upward.

Training data

Everything is generated by the model itself or derived from public training splits. No arena evaluation data was trained on, no benchmark test split was trained on, and no reference answers were copied.

Corpus identity for the serving weights: the round-3 corpus data/mathrft_big/ has SHA-256 24c0283a1ab2db0e0a2a007b6b95a78b0e311b6dcdab864702c60e7f17f5f400 and training_evidence.json (shipped in this directory) records that digest alongside 240 optimizer steps and Slurm job 12153, the job that produced these weights.

• data/prompts/rftp2.jsonl — 3200 open-ended prompts, built from public chat/instruction prompt pools (chat1–chat4, distill). Disjoint from every evaluation set used below. Used for rounds 1–2 (R220, R2d).
• data/prompts/mathpool_s1.jsonl — 4450 math problems drawn from the GSM8K and MATH train splits, categorised gsm8k / alg / pre / num / cnt. Used for round 3 (B3).
• data/mathrft_big/ — the training corpus for B3, 4144 rows, 1.63 M tokens (1.23 M supervised), mean length 392 tokens. Built by rejection sampling from the model's own generations on that pool at T=0.8: k=4 samples per problem, keep the median-length verified-correct one, at most one row per problem. 4294 of 4450 prompts were solved (pass@1 0.9318, pass@4 0.9649); 156 unsolved prompts were dropped. Kept rows by category: gsm8k 3281, alg 442, pre 258, num 184, cnt 129.
• Round-3 sft.npz SHA-256 b25d30307f2f8b89bc2abb94b2b082853170234e6e01d8964c8a6107b8ec89f6.

No Hugging Face datasets were trained on directly; the only public data are the prompt pools above and the GSM8K/MATH train splits named in them.

Evaluation

All measurements below are mine, made on data held out from every training corpus. They are not the organizer's evaluation.

1. math_eval2 — 600 problems, paired-by-problem vs R2d, greedy decoding. Problems from the public train splits, decoded at the served settings.

• Accuracy 90.83% → 92.67%, Δ +1.83, win 22 / loss 11, McNemar two-sided p = 0.080.
• Split by true difficulty level: the gain is concentrated at Level 4 (+3.9) and Level 5 (+9.4); gsm8k alone is +1.12 (p = 0.302) and is not the driver.

2. The same 600 problems decoded the way the bundle actually serves (T=0.7, top_p 0.95, rep 1.05). This is the more honest number and I report it because it changed how I read result 1:

	Δ vs R2d	W/L	p
greedy	+1.83	22/11	.080
sampled T=0.7 (as served)	+0.17	16/15	1.00

The lesson, stated plainly because it is the most important methodological result of this session: every paired readout I had run up to that point used greedy decoding, while the bundle's own generation_config.json ships do_sample=true, temperature=0.7. I had been selecting checkpoints in the wrong regime. Restricting to the same problems, B3 is flat across regimes (93.25% greedy → 93.28% sampled) while R2d gains +2.8 pt from sampling (90.50% → 93.28%). So sampling does not erase B3's advantage — it matches it: it rescues the weaker model onto the same level.

B3 is therefore weakly dominant rather than a large win: clearly better if the evaluation decodes greedily, and never worse than R2d in either regime. I nominated it on that basis — it cannot lose — not because I expect +1.8 to transfer.

3. Capability guard (no regression outside math). Honesty, pbench and instruction-following probes were run for B3 before promotion; no catastrophic regression against R2d on any of them.

4. Serving-regime sanity checks. At the shipped readout cap, truncation is 1.3% for B3 and 1.2% for R2d; the model emits a stop token on 98.7% of problems. The shipped max_new_tokens=2048 is therefore adequate and I left it unchanged.

5. Post-nomination: the noise floor, and why I stopped chasing SFT arms. After nominating B3 I built a more powerful instrument (block 111) and then calibrated it (block 112), which changed how every earlier number should be read. The K=3 readout samples the same 153 problems three times per model, so comparing B3 against itself — three independent draws of the identical adapter — yields a delta that is pure sampling noise:

	replicate accuracies	spread	mean |delta| between replicates	max |t|
B3 vs B3	.8431 / .8301 / .8039	3.92 pt	2.61 pt	1.91

So on this instrument a real effect has to be larger than about 2.6 pt and a t-statistic larger than about 1.9 before it means anything. That is the calibration the session lacked for five blocks, and it retroactively explains the run of "nil" results: the deltas I was chasing (+0.00, −0.65, −1.31, −2.61, −1.96) are the same size as the measurement error.

Against that floor, the model soup S1 (0.5·B3 + 0.5·D1, a uniform average of the only two checkpoints that merely tied B3) reads +1.31 pt, t = +0.87, p = 0.45 at K=3 — i.e. indistinguishable from zero, and the same comparison read −1.96 pt at K=1. It does not clear the bar, so I did not promote it. Reporting both signs is the point: a delta this size is not evidence in either direction, and calling a +1.31 "a win" would be reading noise.

6. Post-nomination: the decoding regime the bundle is actually served in. feedback/README.md states that "your model was served with your nomination's chat template, system prompt and decoding", so the decoding block is what produced every human vote. B3 ships do_sample true, temperature 0.7, top_p 0.95, repetition_penalty 1.05. Result 2 above had already shown that the greedy-vs-sampled distinction is worth ~2.8 pt to some checkpoints, so I swept the decoder with the weights held fixed (jobs 12472–12560, same adapter, same 459 replicated rows, only the sampler changes). Reported in the session notes.

What I did not measure

No held-out benchmark the organizer uses was visible to me. I did not measure factuality against a reference corpus, multilingual quality beyond the arena questions, safety behaviour under adversarial prompting, or performance on non-math reasoning. No single cell of the B3-vs-R2d analysis reaches p < 0.05 — the pooled greedy result is p = 0.080 — so this checkpoint is certified as not worse, not as a statistically confirmed improvement. The sampled-regime number is the one I would quote to a sceptic.

Final blocks: four more arms, all rejected on their own pre-registered bar

After B3f was nominated and banked (17:59 UTC), four further training arms were run against a bar that was fixed before scoring, in every case:

promote only if all four hold — (1) paired word delta on a 91-prompt short-answer battery with t > 1.9 against the incumbent; (2) loop rate and empty rate each not worse than control by more than 2 points; (3) math (scripts/score_dec.py, noise floor 2.61 pt) not down more than 2.61 pt; (4) a 5-answer spot check showing no new failure mode.

None cleared it, so B3f stands unchanged. The arms and why they failed:

arm	idea	result	why it was rejected
finC	off-policy RFT, moderate dose	served length moved down	dose diluted the incumbent's register
finD	larger off-policy dose	length down; run died at an 8-GPU NCCL timeout	no usable checkpoint for a fair read
finL	late-stage low-LR continuation	length flat/down	no positive lever left on that axis
finR	on-policy RFT from B3f (239 prompts x k=4, 192 kept rows)	length +41.0 words, t = +2.33 — first arm in five to move served length the way the arena wants — but loop rate 0.00% -> 4.40%	failed bar (2) by 2.4 pt

The dose-interpolation result that closes the whole family

finR was initialised from B3f, so finR = B3f + delta and a linear blend is exactly B3f + alpha*delta — computable in 8 s on CPU by streaming tensors, with zero training GPU. Scoring two points on that line (alpha = 0.6 and 1.0):

• length t = 1.09 at alpha 0.6, 2.33 at alpha 1.0 (slope ~2.1 in alpha)
• loop rate = 1.18% at alpha 0.6, 4.40% at alpha 1.0 (slope ~7.3)

Both quantities are approximately linear in alpha, so the feasible set is empty: reaching t > 1.9 needs alpha > ~0.9, where loops are already ~4%; holding loops under 2 pt needs alpha < ~0.33, where t is only ~0.7. Length and repetition are not separable along this axis — they are two faces of one dose. "Scale the dose" is therefore a dead family on this checkpoint, whether by learning rate, step count, or weight interpolation.

The loop regression is not imported repetition — measured, not assumed

Before measuring, I hypothesised that the corpus filter let repetition-prone targets through: build_rft.py's degenerate() tests 8-grams at stride 4 and fires below 0.7 unique, which is not the scorer's predicate (rep_ratio(n=6) >= 0.25), so bad samples could have survived filtering and taught the model to loop.

Refuted. Over all 956 candidate targets, rep_ratio(6) is p50 0.000, p75 0.002, p90 0.020, p95 0.047, p99 0.118, max 0.800. Only 1 of 956 exceeds the scorer's 0.25 bar; 95.4% sit below 0.05. The training targets were already clean, and a stricter filter could have removed ~1% of the data. The loops are a side effect of the length push itself — a longer generation has more opportunities to fall into a repetition attractor — not contamination of the targets. Recorded here as a negative result, because the two filters genuinely do disagree in principle and it was worth establishing that they did not disagree in practice.

What the next person should do instead

Not another point on the alpha line. The two levers this evidence points at are an explicitly anti-repetition training signal (so that length and non-repetition are optimised as separate objectives rather than traded off against each other), or decoding-time control, which this serving contract does not expose.

Limitations

• The selection proxy is math accuracy on GSM8K/MATH-derived problems. B3 is ~79% gsm8k rows by construction, and gsm8k is near its ceiling, so most of the measured signal comes from a small minority of harder rows. Gains there are where I would expect variance to be largest.
• Multi-epoch and higher-LR training on this corpus are measurably worse. 2 epochs and 5e-5/1e-4 both scored below 1 epoch at 2e-5. If you fine-tune further, do not simply raise the epoch count or the learning rate; the sweep is unimodal and already at its peak.
• Round-2 measurement (guard2 coverage, MATH, battle_hold) is documented in the session notes for R2d; it was measured for that checkpoint, not for B3, and I am not restating it as a result about this bundle.
• The adapter is a 5.77 GB file including the LM head; load with enough host RAM or use device_map="auto" as above.
• The chat template is the one fragile file in this bundle. B3f fixes it relative to B3, but note that any downstream harness which overrides chat_template with its own will change the prompt format, and this model was trained on exactly this format ({role}\n{content}\n per turn). Prefer the template shipped here.
• Checkpoint selection rests on a sub-significant result. The strongest paired number supporting B3 over R2d is p = 0.080, and against the calibrated 2.6 pt noise floor most of the arms I compared are ties. B3f is the checkpoint I could not show to be worse than anything else, and the one whose serving files I verified load and generate correctly — not a checkpoint I demonstrated to be significantly better.

License

The Base model's terms apply: OpenMDW-1.1, https://openmdw.ai/license/1-1/.