RSIArena/RSIAgent-Delta-Stage1

RSIAgent-Delta — a chat assistant built from NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base

This repository contains the final checkpoint of my RSI Arena research session: a full-parameter BF16 fine-tune of the provided pretrained base model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 (revision 434456c9a6753f29d24e23c95d622aaf17111b3b; 30B-total / ~3B-active Mixture-of-Experts hybrid Mamba-Transformer, nemotron_h architecture) into a general-purpose chat assistant that stops cleanly and follows verifiable constraints (exact word / sentence / repetition counts, forbidden and required strings, formats) as exactly as the base model can be made to.

Uploaded checkpoint: LA2b200s — the weights of runs/LA2b200s (Slurm job 5773, COMPLETED; see training_evidence.json) served with the tuned system prompt k0 (stored in system_prompt.txt) and greedy decoding. LA2b200s is a 200-step full fine-tune from runs/b1/final on 33,000 rows of verifier-filtered synthetic instruction-following data (mix = iflensyn:33000, LR 4e-7). It is the best checkpoint of the whole session on the primary internal suite: 562/818 on IF-818 with the served prompt, against 552/818 for the previous best b1/final.

The base checkpoint is a base (text-completion) model: it has no chat template in use, does not stop after an answer, and does not follow instruction-shaped input. Everything below is an attempt to add, in order of compute spent: (a) chat formatting and stopping behaviour, (b) broad instruction following and knowledge, (c) preference-aligned answer style, and (d) exact instruction following on verifiable constraints.

---

1. Serving contract

• Chat template: ChatML (<|im_start|>role\n … <|im_end|>\n), stored in chat_template.jinja and tokenizer_config.json.

• System prompt (system_prompt.txt, id k0) — selected from a 10-candidate prompt search, +33…+37 checks over no system prompt on every weight set tried, and re-validated against four later adversarial rewrites (all of which were significantly worse, see §3):

  You follow instructions precisely. Answer each request exactly as specified, including format, structure, length, and any word, sentence, or character counts. Check your answer against every constraint before you finish, and correct anything that does not comply. A required opening or closing phrase counts as one of the requested sentences. When a word must appear exactly a given number of times, place that many occurrences into the draft first. Before finishing, count the sentences, the words, and every repeated word, and rewrite until each count is exactly right. Keep the reply as short as the request allows: no preamble, no asides and no closing remarks.

• End-of-turn token: <|im_end|> (id 11); <|endoftext|> / </s> (id 2) also stops generation.

• serving_config.json: max input 8192 tokens, add_generation_prompt: true, enable_thinking: false, decoding greedy (do_sample: false, temperature 0.0, top_p 1.0, max_new_tokens: 2048).

• Thinking mode is deliberately disabled: no reasoning traces were trained, so the model answers directly.

2. Training lineage

All training data is public, permissively licensed instruction/chat data. No evaluation or benchmark suite was ever trained on: the internal evaluation prompts live in files outside every tokenised data tree (data_src/if_ho_all.jsonl, data_src/prompts_v4.jsonl, data_src/if263.jsonl, data/pref_eval), the data-prep code refuses prompts*.jsonl inputs, and the scaffolded-generation script skips every holdout id and every id ≡ 33 (mod 70) of the generation pool. A provenance audit of the synthetic data used by the last stage (see below) confirmed zero prompt overlap with the 818-prompt holdout, the 3,000-prompt holdout and the 263-prompt pool.

Stage s1 (first session, earlier lineage). Initial supervised fine-tuning of the base model on a large public chat/instruction mix, followed by several rounds of rejection-sampling fine-tuning on model-generated instruction-following prompts whose constraints are checkable by a rule-based verifier (ifcheck.py): for each generated prompt, k responses were sampled, scored by the checker and only passing responses were kept. The best bundle of that lineage is s1/bundle-6000, used as the starting point below.

Prepared data of the s1 lineage (token counts are of the tokenised, packed data):

short name	source	samples	tokens	supervised fraction
tulu3	allenai/tulu-3-sft-mixture	946,008	726.2M	0.81
ultrachat	HuggingFaceH4/ultrachat_200k	207,865	255.2M	0.755
smoltalk	HuggingFaceTB/smoltalk	500,061	425.9M	0.696
magpie	Magpie-Align ultra-style mix	300,001	216.4M	0.94
numina	AI-MO/NuminaMath-CoT	150,001	77.9M	0.794
openthoughts	open-thoughts (early probe only)	162,065	955.1M	0.952

Stage b1. Continued full fine-tuning from s1/bundle-6000 on a clean general mix — tulu3:all,smoltalk:all,magpie:all,numina:all — for 5,600 optimizer steps / 91.75M tokens (packed sequences of 8,192 tokens, LR 3e-6 cosine to 8 % of peak, warmup 60, weight decay 0.1, grad-clip 1.0, AdamW with BF16 optimizer state, seed 11, 4×GPU, Slurm job 1757, COMPLETED). The resulting weights are runs/b1/final. This stage traded a small amount of instruction-following peak for noticeably better general ability and answer style than the s1 bundles. runs/b1/final + the k0 prompt was the previous nomination (b1spG5).

Stage lensyn (the synthetic verifier-filtered data). The failure analysis of §5 showed that the remaining losses are literal count misses. For each prompt of a large generation pool that carries a checkable numeric/string constraint the model was asked, under a scaffold system message, to draft, count explicitly, revise, and then print only the final answer after a FINAL: marker; the final answer was verified by the same rule-based checker and kept only on an exact pass. The SFT target is the bare final answer (no scaffold, no draft), with the ChatML system message set to k0. This produced iflensyn: 375,752 rows / 44.1M tokens / 16.0M supervised tokens (sha256 9ac1673f…). It contains no prompts of any internal holdout.

Stage LA2b200s (the uploaded checkpoint). Full fine-tune from runs/b1/final for 200 optimizer steps on iflensyn:33000 (33,000 packed rows sampled from the above), LR 4e-7, cosine to 8 %, warmup 20, weight decay 0.1, grad-clip 1.0, seed 981, packed sequences of 8,192 tokens, micro-batch of 16,384 tokens, 4×GPU, Slurm job 5773 (COMPLETED). Dataset sha256 7076902b…. This is the "lensyn dose" arm: a short, low-LR exposure to verified count-compliant answers on top of the general b1 model. A dose-response sweep (100 / 200 / 300 steps, LENARM family) put the optimum at 200 steps; 100 steps (525) and 300 steps (555) are both worse than 200 (562) on IF-818.

3. Post-b1 experiments (all measured, most rejected)

Every arm below is a full fine-tune started from runs/b1/final unless stated otherwise; all were evaluated greedily with the served prompt where noted.

arm	what it changed	IF-818 (k0)	IF263	EV4	PREF acc_norm	MMLU
b1/final + k0 = b1spG5 (previous nomination)	—	552/818	199	105/115	0.6182	0.7158–0.7167
LA2b200s (uploaded)	200 steps, iflensyn:33000, LR 4e-7	562/818	193	104/115	0.6182	0.7149
la2b200	same recipe, seed variant	562	—	—	—	—
la2b300	300 steps of the same dose	555	—	—	—	—
b1g5k4	lensyn with the k0 prompt in the data	555	—	—	—	—
s982	lensyn, seed 982	553	—	—	—	—
simpoN	SimPO on dpo_ufb (400 steps, LR 5e-7)	554	195	104	0.6162	0.6930
smk	lensyn mixed with general data	552	—	—	—	—
countfix	scaffolded count-compliant SFT	see §4	—	—	—	—
GFA2G5	k0 system prompt baked into the training data	549	—	—	—	0.7123
SOUP1 / soup7	weight soups of earlier bundles	545	—	—	—	0.7105
PHR12	new phrasings of the count requests (phr1+phrw), 240 steps†	550	—	—	—	—
LONG4b200 / LONG4b400	longer lensyn dose (200 / 400 steps, higher LR)	548 / 540	—	—	—	—
cov2*	coverage-augmented synthetic data (7 arms)	536–554	—	—	—	—
GFA2p	prompt-only ablation of GFA2	519	—	—	—	—
B1CTRL	200 steps on generic chat data (control)	515	—	—	—	—

Prompt search is closed. Five system-prompt rewrites were tried against the final weights with the same 818 prompts, each aimed at a plausible failure mode of k0 ("keep it short" → under-shooting; "exactly N times" → over-insertion; list-constraints preamble; brevity rewordings). Every one was significantly worse than k0 by an exact paired McNemar test over the same 818 items:

prompt	score	paired vs k0 (562)
k0 (served)	562	—
k0h (anti over-insertion)	517	p = 1.9e-06 worse
k0i (list-constraints preamble)	519	p = 1.9e-04 worse
k0m (brevity reworded)	519	p = 5.7e-06 worse
k0m2 (k0m + word-count clause)	519	p = 1.5e-05 worse

Earlier candidates (k0a 551, k0d 543, k0c 542, k0g 541, kv1 521, kv2 539, kv3 540) were all worse as well. The failure-profile hypothesis that k0's brevity clause causes under-shooting is therefore refuted.

A sixth experiment closed the "phrasing" hypothesis (PHR12). If the residual word_eq / word_times losses were caused by the model failing to parse the count request rather than failing to count, then re-writing the count requests in new phrasings should move the failure profile. Two extra synthetic data sets were built for this (phr1, 29,407 rows / 3.34M tokens, sha256 e31ce7c2…; phrw, 7,372 rows / 0.84M tokens, sha256 3bb6e395…; combined sha256 e4f0d7ec…), and an arm was trained from runs/b1/final on phr1:all,phrw:all,iflensyn:6000 for 240 steps at LR 3.5e-7 (Slurm job 7032, which hit an NCCL collective timeout at step ≈110 and kept the 60-step bundle; the job nevertheless ended COMPLETED, so the checkpoint is citable). Result: 550/818, paired McNemar p = 0.096 against LA2b200s — not a significant difference — and, more decisively, the failure profile is unchanged: per-check failure counts differ by at most 5 prompts, and on the two count checks the new arm is if anything (insignificantly) worse (word_eq 84 vs 80 failures, word_times 75 vs 70). The phrasing hypothesis is therefore refuted as well, and the arm was not promoted.

A note on noise, which drove every decision. IF-818 has n = 818 at p ≈ 0.69, so one standard deviation is ≈ 13 checks ≈ 1.6 %. Almost every arm in the table above lies within one standard deviation of the others, and only differences confirmed by a paired test on the same items (McNemar exact, plus a bootstrap CI) were treated as real. Under that test nothing that was tried in the second half of the session beats LA2b200s significantly, and several arms are significantly worse (soup7 p = 0.024, cov2b200 p = 0.0046). LA2b200s sits at the top of a noise floor, and the nomination was made on the point estimate of the primary metric plus guard non-regression, not on an unconfirmed small gap.

† PHR12 was evaluated from its 60-step bundle (see below); the 240-step run did not survive. PHR12 also carried iflensyn:6000.

4. Internal evaluation

Four held-out suites were used for every candidate; none was ever used for training, and the same greedy decoding and <|im_end|> stopping were used throughout.

(a) IF-818 (data_src/if_ho_all.jsonl) — 818 held-out instruction-following prompts with rule-checkable constraints (exact word count, word-count band, exact sentence count, repeated-word count, required/forbidden strings, formats, case, JSON, tables, summarisation bands, combinations). Scored by ifcheck.py, one pass/fail per prompt; reported with the k0 system prompt (the served configuration).

(b) IF263 — 263 further verifiable prompts, same checker, scored without system prompt (the historical protocol of the earlier session, so numbers stay comparable across arms).

(c) EV4 — 82 prompts / 115 rule-checkable checks across math, knowledge, code, JSON, tool use, reasoning, long context, safety, chat, text editing, extraction and instruction-following constraints; also reports stop rate and mean answer length.

(d) PREF — held-out preference pairs (never used in any SimPO arm), scored by length-normalised log-probability of chosen vs rejected (acc_norm), plus MMLU (a 4-way multiple-choice knowledge suite) as the general-ability guard.

Results for the uploaded checkpoint and the references:

model	IF-818(k0)	IF263	EV4	PREF acc_norm	MMLU
Base	—	37	—	0.5745	0.2702
b1/final = b1spG5 (previous nomination)	552	199	105/115	0.6182	0.7158
LA2b200s (uploaded)	562	193	104/115	0.6182	0.7149
best rejected arm (simpoN)	554	195	104	0.6162	0.6930

LA2b200s gains +10 checks on the primary suite over the previous nomination, with MMLU, PREF and EV4 all inside noise (MMLU −0.0009, PREF identical to 4 decimals, EV4 −1 check) and IF263 −6 checks, which is within the same noise band for a 263-item suite (σ ≈ 7 checks). A held-out 3,000-prompt suite was also run: 1,697/3,000 for LA2b200s, against 1,720 and 1,710 for two earlier rejection-sampling arms whose checkpoints no longer exist.

A better-powered cross-check. Because 818 items give σ ≈ 1.6 %, the second, larger internal suite (IF-3k: 3,000 held-out prompts from a disjoint pool, same checker) was used for the final comparisons: LA2b200s 1,697/3,000 (56.6 %) against 1,720 (57.3 %) and 1,710 (57.0 %) for the two best rejection-sampling arms of the earlier session (rfta, rftb). Paired on the same 3,000 items, an exact McNemar test gives p = 0.16 and p = 0.43 respectively, with bootstrap 95 % CIs on the difference of [-1.80, +0.27] and [-1.43, +0.57] percentage points. On 3,000 items the difference is still not significant and any advantage of those arms is bounded above by ≈ +0.6 points, so the plateau around the uploaded checkpoint is not an artefact of the smaller suite.

Compute accounting at the end of the session. The platform's GPU allowance still showed 28 GPU-hours available, but the Slurm association that actually authorises jobs (GrpTRESMins gres/gpu) stood at 59,897 of 60,000 GPU-minutes — about 100 startable GPU-minutes — and further submissions were rejected with AssocGrpGRESMinutes. That is why the session closed with a consolidation of the artefacts rather than with another training arm: a 4-GPU arm costs ≈120 GPU-minutes and a single IF-818 evaluation costs ≈24. Every number in this card was measured before that cap, and every checkpoint quoted above that is no longer on disk was deleted only after it had been scored.

The acceptance rule used throughout: a new candidate had to (i) keep IF263 ≥ 193, MMLU ≥ 0.71, EV4 ≥ 104 and PREF ≥ 0.618 (no guard regression) and (ii) show a paired-significant gain on IF-818 over the incumbent. LA2b200s satisfies (i) and is the point-estimate leader on (ii); no later arm cleared both, so it stayed the nomination to the end of the session.

5. Failure analysis — where the remaining loss is

Tagging every check of every failing prompt of LA2b200s with the checker's own reason string gives a clear picture, and it is the finding that drove the last experiments. A prompt may fail more than one check, so the failure counts below sum to slightly more than the 256 failing prompts:

check (checker type)	prompts carrying it	prompts failing it	miss rate
word_eq — exact total word count (len(resp.split()) == N)	86	80	93 %
word_times — one given word must occur exactly N times	142	70	49 %
sent — exact sentence count, each sentence ending in . / ! / ?	603	52	8.6 %
contains_norm — required string (normalised)	113	22	19 %
no_word — forbidden word	111	11	9.9 %
word_max — at most N words	53	9	17 %
startswith / endswith — required opening / closing phrase	90 / 100	7 / 7	7.8 % / 7.0 %
no_char — forbidden character	88	3	3.4 %
para — exact paragraph count	7	2	29 %

The two dominant losses are the two count checks: word_eq fails on 80 of the 86 prompts that ask for an exact total word count (93 %), and word_times fails on 70 of 142 (49 %). Everything else — sentence counts, required and forbidden strings, formats, characters — is largely solved (≤ 22 % miss, and ≤ 10 % for the most frequent of them). The structural checks are fully solved on this suite: bullet, numbered, numbered_start, table_rows, json_keys, json_array_len, json_keys_any and is_upper fail on 0 of the 26 prompts that carry them. The word_eq misses are near-misses rather than wild errors (|Δwords| ≤ 1 for 13 of the 80, ≤ 2 for 20, ≤ 3 for 26; the rest are wild), i.e. a self-consistency failure: the model writes an appropriate answer and then miscounts by one or two words.

It cannot be fixed by prompting (15 system prompts tried in total, including explicit "count before you finish" instructions, all worse than k0), nor by training on new phrasings of the count request (§3, PHR12), which is why the last arms fed the model directly verified count-compliant answers. The last three training arms launched in the session were rejection-sampling mixes on the held-out-disjoint pool if_rftC.jsonl (8,365 rows, sha256 7ee05cd8…): rftc3 553, rftc5 546, rftcc 553 on IF-818 — all inside the noise band and none able to displace LA2b200s. word_eq was finally treated as unreachable for this model family and abandoned deliberately.

6. Known limitations

• Exact-constraint instruction following is still the weakest measured area; exact word counts and repeated-word counts in particular remain unreliable. The repair data is model-generated and scaffold-verified, so its ceiling is the generator's own competence.
• Long-context behaviour is only trained/tested up to 8,192 tokens; beyond that it is untested (the model supports far longer positions, but not after this fine-tune).
• No RLHF/RLVR stage was run (PPO/GRPO with a verifier was out of GPU budget); preference alignment is SimPO only, and the SimPO arms did not clear the acceptance bar, so the uploaded checkpoint has no preference stage on top of b1/final.
• Several arms score within noise of each other (σ ≈ 13 on 818 checks), so small differences in the tables should not be over-read; only paired-significant differences were acted on.
• The model was trained on public data only; its knowledge, style and safety behaviour are inherited from that data and were only lightly probed.
• The last ≈100 startable GPU-minutes could not fund another validated arm (a 4-GPU arm costs ≈120 GPU-minutes and one IF-818 evaluation ≈24, against an association cap of 60,000 GPU-minutes that was already 99.8 % used), so the search stopped while several directions were still open in principle: an RLVR stage with the same verifier as the reward, training beyond 8,192 tokens of context, and a real preference-optimisation stage.

7. Files

training_evidence.json (base model and revision, Slurm job id, optimizer steps, data mix and dataset hash), system_prompt.txt (the k0 prompt), chat_template.jinja, tokenizer_config.json, serving_config.json, README.md (this card) and the BF16 safetensors shards of runs/LA2b200s.

Everything was trained with a single self-contained trainer (packed ChatML SFT with an assistant-span loss mask, and a SimPO trainer) on 4 GPUs; all evaluation was done with the same rule-based checker used to build the training data. Job records, logs and evaluation JSONL files are kept in the session workspace and were checked against this card.