RSIAgent-Delta-Stage2 — Nemotron 3.5 Lightning 30B-A3B chat assistant

This repository holds the checkpoint nominated by my RSI Arena Stage-2 research session (continual training from human feedback). It is a full-parameter BF16 fine-tune of the provided pretrained base nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 (revision 434456c9a6753f29d24e23c95d622aaf17111b3b; 30B total / ~3B active MoE hybrid Mamba-Transformer) into a general-purpose chat assistant. An autonomous agent built it.

The uploaded directory is the serving bundle: model weights (16 shards + index), config.json, tokenizer files, the ChatML chat_template.jinja, system_prompt.txt, serving_config.json, generation_config.json, training_evidence.json, and the bias.md / explainability.md / privacy.md / safety.md notes.

Weights in this upload: runs/s2H_best (Slurm job 8464, COMPLETED), the single Stage-2 weight set. Serve it with the bundled system_prompt.txt and greedy decoding; a serving-side repetition_penalty of 1.05 is recommended (it was selected by measurement, see Evaluation).

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "RSIArena/RSIAgent-Delta-Stage2"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype="bfloat16", device_map="auto")

system = open(f"{path}/system_prompt.txt").read().strip()
messages = [{"role": "system", "content": system},
            {"role": "user", "content": "Explain how a refrigerator works."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True,
                              return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=2048, do_sample=False,
                     repetition_penalty=1.05, eos_token_id=[11, 2], pad_token_id=11)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

• Chat template: ChatML (<|im_start|>role\n ... <|im_end|>\n), stored in chat_template.jinja and tokenizer_config.json.
• End of turn: <|im_end|> (id 11); id 2 (<|endoftext|>/</s>) also stops generation.
• serving_config.json / generation_config.json record: max input 8192 tokens, add_generation_prompt: true, enable_thinking: false, greedy (do_sample: false, temperature 0.0, top_p 1.0), max_new_tokens 2048, repetition_penalty 1.05.
• Thinking/reasoning mode is deliberately off: no reasoning traces were trained, so the model answers directly.
• Use the bundled system prompt. It was chosen by a head-to-head reward-model probe against four alternates; serving without it (or with a terse prompt) measurably degrades answer quality and length. The prompt asks for thorough, well-structured answers, asks the model to work calculations and factual questions out directly rather than deferring to code, and states that an explicit user constraint (length, format, counts) overrides the general guidance.

Training lineage

Every stage is a full-parameter fine-tune with the same runtime: packed sequences, max_seq_len 8192, micro_tokens 16384 (max batch 2 sequences per rank), AdamW with BF16 optimizer state, warmup, cosine decay to 10 % of peak LR, weight decay 0.1, gradient clipping 1.0, gradient checkpointing, FSDP over 4 GPUs (one node of 8x RTX Pro 6000 Blackwell 95 GB, 4 used), and loss on the assistant span only (including its trailing <|im_end|>), with system and user tokens masked out. All runs live in my own Slurm account; logs are kept.

stage	Slurm job	run	init	steps	lr	data (token/step budgets)
Stage-1 arm	5773	LENARM2	runs/b1/final	300	4.0e-7	iflensyn 33k — synthetic verifier-filtered instruction-following data, rendered under a constraint-heavy prompt
Stage-1 nomination	—	runs/LA2b200s	job 5773	—	—	selected bundle of that run (the model that played the arena battles)
Stage-2 first pass	7583	s2A	runs/LA2b200s	600	3.0e-7	broad chat mixture: iflen6 45k, ifrft6 25k, feedback SFT fb2sft6 all, tulu3 10k, smoltalk 6k, magpie 5k, nemotron-chat 5k, numina 3k, code 3k
Stage-2 final (nominated)	8464	s2H	runs/s2A_best	800	3.0e-7	iflenbig 15k, feedback SFT fb2sftbr60 all (×60 repetition), tulu3 40k, ultrachat 25k, nemotron-chat 15k, magpie 15k, smoltalk 15k, numina 8k, code 5k

Job 8464 ran 2026-10-11 11:30:04 → 15:31:33 UTC on coconut-7, seed 4242, 800 optimizer steps, 8 epochs over the sampled mixture, 13,107,200 tokens seen, dataset_sha256 7d6dc0e2bcf238b560b2d6400e3802be7ca34afa4df8f62b57362e41ae128b6e. Two earlier Stage-2 attempts (a SimPO preference run and a two-pass variant) failed (CUDA OOM / job preemption) and produced no weights; they are recorded in my logs as failures and are not part of this checkpoint.

The mixture was re-rendered so that every source conversation is presented under the same system prompt used at serving time: prep11.sh re-renders each source with the serving prompt sp2/p9b.txt before tokenisation (that is what the rb/b suffixes mean in the mixture keys). This matters: the model is trained with the bundled system prompt in context, not only with a neutral one.

Training data

Public, permissively licensed chat/instruction data, plus the arena feedback:

source (Hugging Face id)	role in the mixture
allenai/tulu-3-sft-mixture	broad multi-turn chat + instruction following
HuggingFaceTB/smoltalk	short, style-rich multi-turn chat
HuggingFaceH4/ultrachat_200k	broad open chat
Magpie-Align/Magpie-Pro-300K-Filtered	strong instruction following
AI-MO/NuminaMath-CoT	math reasoning
nvidia/OpenCodeInstruct	code instruction following
nvidia/Nemotron-Post-Training-Dataset-v1	chat post-training data
allenai/tulu-3-sft-personas-instruction-following	instruction following with constraints
synthetic verifier-filtered IF (iflenbig)	instruction-following prompts with programmatically checkable constraints; only responses that pass a rule-based checker are kept
RSI Arena Human Arena feedback (fb2sft)	see below

Human feedback. The feedback set is 230 battles that my Stage-1 model played in the RSI Arena Human Arena, each with the user request, both assistants' answers, and the voter's choice (mine / theirs / tie / both bad). It was used in two ways: (a) as supervised data — the winning side of each decisive battle was rewritten into an assistant turn and rendered under the serving prompt, giving a small high-signal SFT set that was repeated in the mixture (fb2sftbr60, ×60); (b) as a held-out probe set for model selection — a 65-battle split was reserved and never used for training, and it is the split whose numbers appear below. The feedback data itself is not redistributed here.

No evaluation or benchmark test data was used for training. The internal evaluation prompt files live outside the tokenised data trees and were never ingested by the data-prep code; the instruction-following holdouts (409 and 818 machine-checkable prompts) are prompt-only files used for measurement.

Evaluation (my own, on data the organizer does not use)

These are internal measurements made during the session to choose the checkpoint, the serving prompt and the decoding settings. They are not human evaluations and not the organizer's held-out benchmarks. The primary instrument is a proxy reward model, Skywork-Reward-8B, scoring a candidate answer against the answer recorded in the battle; I report a win-rate proxy (fraction of comparisons where the candidate scores higher) and the mean score margin.

Checkpoint and decoding selection (weights are identical across all Stage-2 bundles I produced; what changes is the prompt and the repetition penalty):

setting	230-battle probe	65-battle held-out split	IF holdout, 409 checks
bundled prompt, repetition_penalty 1.05 (nominated)	0.787 / 20.64	0.738 / 16.29	282 pass
bundled prompt, repetition_penalty 1.10	—	0.708 / 16.27	265 pass
bundled prompt, repetition_penalty 1.20	—	—	209 pass
bundled prompt, no penalty (1.00)	—	0.631 / 12.69	—
four alternate system prompts (same weights, rp 1.05)	0.743, 0.726, 0.717, 0.691	—	—

The four prompt alternates (a shorter prompt, a markdown-formatting prompt, a reasoning-style prompt and an eight-shot-style prompt) all scored below the bundled prompt on the 230 battles, and every paired comparison lost on a paired t-test. The repetition penalty sweep is the one serving-side knob with a clear effect: 1.05 > 1.10 > 1.00 on both the held-out probe and the instruction-following holdout, so 1.05 is what the bundle records.

Instruction-following holdout. 409 prompts whose constraints are machine-checkable (exact word/sentence counts, required and forbidden substrings, exact list/JSON formats, case restrictions), greedy decoding, scored by a rule-based checker: 282/409 = 0.69 pass for the nominated checkpoint at rp 1.05. The Stage-1 model that played the arena battles (runs/LA2b200s) scored 562/818 = 0.687 on the larger 818-prompt version of this holdout, so the Stage-2 work did not trade instruction-following compliance for answer quality.

What the feedback said the model was bad at. The Stage-1 model's arena losses concentrated in very short answers: answers under 200 characters won about 20 % of their battles, answers over 1500 characters about 74 %. The Stage-2 model answers at a median of ~2100 characters against the recorded opponent's ~1400, so the brevity deficit that drove most losses is addressed; the bundled system prompt is part of that fix.

Note on a discarded measurement. A small MMLU probe I ran early on was driven mostly by format compliance rather than knowledge (the answer extractor is a single-regex match on A–D), so I did not use it for selection; it is mentioned here only so the record is complete.

Limitations

• Exact-constraint instruction following is still the weakest area: exact N-word answers and counted-sentence requests fail more often than other constraint types.
• The win-rate proxy is fitted on 230 battles, is noisy, and correlates with answer length. It is a proxy; it is not a human preference measurement.
• Training data is English-only; other languages are largely untested (13 Chinese battles in the feedback produced 4 wins).
• No safety-specific training beyond general chat data; refusals are prompt-driven. No reasoning/thinking mode. Knowledge is limited to the base model's pretraining data.
• Long-context behaviour is limited to the 8192-token training/eval length.
• Nothing here was measured on the organizer's held-out evaluation sets.

License

The base model's terms apply: OpenMDW-1.1 (https://openmdw.ai/license/1-1/).