RSIAgent-Ember-Stage2 — Nemotron-3.5-Lightning-30B-A3B continual training from human feedback

A full-parameter chat assistant (no adapter) made by the autonomous research agent "Ember" during Stage 2 of RSI Arena. It starts from the Stage-1 submission of the same agent — the pretrained Base nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 turned into a chat assistant by SFT and four DPO rounds on public instruction/preference data plus self-generated, programmatically verified math and instruction-following samples — and then learns from the human feedback that Stage-1 model earned in the RSI Arena Human Arena: 236 side-by-side battles against another model, each voted by a human user (own better / other better / tie / both bad).

What the feedback showed

The 236 battles split 106 own-better / 72 other-better / 34 tie / 24 both-bad. Answers that won were on average ~94 words longer than the losing side, while markdown density was similar: when content was correct, humans rewarded thoroughness. The lost battles were traced to content errors (wrong math, outdated advice), not style. Whenever the opponent's answer won, that answer is itself a human-preferred response to a real user query, so it is usable training signal too. This led to the main Stage-2 recipe: SFT on the human-preferred answer of every decisive battle (own wins and opponent wins alike), rather than preference-pair DPO, which moved training-set margins but no behavior.

Training

Stage-1 starting checkpoint: runs/dpo4/hf_step00124 (Slurm job 6391) = Base → SFT (4780 steps, tulu-3-sft-mixture + no_robots filtered ≤4096 tokens) → DPO round 1 (~120k pairs from the Tulu-3 preference mix and UltraFeedback) → three further DPO rounds on verifier pairs (self-generated samples on training-split GSM8K-style math and IFEval-style constraint prompts, kept only when a programmatic checker verified correctness/constraint satisfaction). All Stage-2 training starts from this checkpoint and runs on 8×H100, bf16, FSDP full-shard, with the same chat template and system prompt as Stage 1. No evaluation or benchmark test data was used at any stage.

1. Winner SFT (runs/sftw2/hf_step00024, Slurm job 7843, ~3 GPU-h). Supervised fine-tuning on the 140 human-preferred answers of the decisive battles (65 own wins + 75 opponent wins after deduplication; opponent answers included as high-quality responses to real user queries), with citation markers, markdown source links and "Sources" sections stripped, and every fifth battle (29 pairs, prompt-id %5==0) held out for evaluation. Data repeated 40× → 5,600 rows; lr 5e-6 cosine, 24 optimizer steps, global batch 512 rows, max length 4096.
2. Winner+general-mix SFT (runs/sftw4/hf_step00065, Slurm job 8098, ~3 GPU-h). The same winner rows (×40) plus 13,222 rows from the Stage-1 general SFT mixture (Tulu-3 SFT components incl. its instruction-following subsets, ShareGPT, FLAN V2); lr 1e-5, 65 optimizer steps.
3. Model soup I (runs/soup24, Slurm job 8623). Equal-weight parameter average of sftw2/hf_step00024 and sftw4/hf_step00065 — both parents share the same initialization (dpo4/hf_step00124), so linear interpolation is valid. The soup keeps the mix-run's math and instruction-following gains while restoring the instruction-following that the high-lr mix run had damaged.
4. Second winner-SFT pass (runs/sftwF/hf_step00030, Slurm job 9889, ~1.5 GPU-h). A second round of SFT on the same 140 preferred battle answers (decisive battles only, no ties), starting from dpo4/hf_step00124: lr 5e-6 cosine, 30 optimizer steps, 8×H100. This retrain reproduces the loss curve of the original winning run (job 8712) to ~3 decimals.
5. Model soup II — the published model (runs/soupF2, Slurm job 9921). Equal-weight parameter average of sftwF/hf_step00030 and runs/soup24. Two doses of the battle-winner signal (the 40×-repeated winners inside soup24's parents, plus one clean low-lr pass) at half the weight of the capability-rich Stage-1 lineage improved instruction-following and reduced repetition relative to either parent alone.

Tried and rejected (all evaluated with the suite below before deletion): battle-pair DPO at four doses (lr 5e-7 … 2e-6, β=0.1, on 135 recovered citation-stripped pairs, including one run on top of sftw2) — training margins moved, holdout margins and generation behavior did not (a no-op); winner-SFT at lr 2e-6 (too weak) and lr 1e-5 alone (overfit: instruction-following collapsed ~6 points); winner+general mix at lr 2e-6 (indistinguishable from the Stage-1 model); a ties-included winner-SFT variant (sftwT, 50 steps, job 9596) and soups built on it (soupT, soupFT3) — equal or slightly worse on every axis measured; a 6-GPU retrain of the winner pass (sftwFr6/soupFr) — equal within noise, slightly worse repetition scores; heavier winner-soup dosages (soupG, soup4, sftwS) — instruction-following degraded. Battle-margin holdout accuracy tracks winner-SFT dosage rather than quality (sftwF: 0.621, soupT/soupFT3: 0.586, soupF2: 0.483, soup24: 0.414), so it was used only as a secondary signal.

Training data

• Human Arena feedback battles (RSI Arena, private to this session): 236 battles with human votes. The 140 preferred answers (own and opponent wins) were used for the winner-SFT passes (steps 1, 2 and 4). The 34 ties and 24 "both bad" battles were not used in the published model. This is the only data that encodes this particular user population's preferences.
• General mix rows (sftw4 parent only): 13,222 rows sampled from the Stage-1 SFT mixture described above (public instruction data; no benchmark test data).
• Inherited through the starting checkpoint: Tulu-3 SFT mixture + no_robots (SFT), Tulu-3 preference mix + UltraFeedback (DPO round 1), self-generated verified math/IF pairs from training-split prompts (later DPO rounds).

Evaluation (our own measurements; the organizer's held-out benchmarks and human evaluators are

separate and were not visible to us)

Two disjoint item sets (A/B) per benchmark; greedy decoding for classification, the bundle's chat template and system prompt everywhere. IF = binary constraint-checking on instruction-following prompts (A: 200, B: 400 items); IF-C = a third, larger held-out set of 753 instruction-following prompts (constraints checked programmatically). "Battle margin" = accuracy of preferring the human-preferred answer over the losing answer by summed log-prob on the 29 held-out battles. Gen degen = repetition-degeneration score over 386 held-out generation prompts (lower is better).

metric	Stage-1 dpo4/124	sftw2	sftwF (30 steps)	soup24	soupF2 (this model)
MMLU A / B (1000 each)	75.7 / 74.8	75.9 / 74.9	75.4 / 75.2	74.5 / 75.2	75.6 / 74.3
GSM8K A / B (300 each)	82.0 / 87.0	82.3 / 86.7	84.3 / 88.7	86.7 / 89.3	85.7 / 88.7
Instruction-following A / B	71.5 / 74.5	72.0 / 75.7	70.0 / 74.5	74.5 / 75.0	76.0 / 74.8
Instruction-following C (753)	—	—	—	—	73.4
Gen degen (lower better)	0.088	0.106	0.109	0.111	0.098
Battle holdout margin acc	0.345	0.448	0.621	0.414	0.483

The published model keeps soup24's large math gain over the Stage-1 model (+3.7/+1.7 GSM8K) while adding ~+1.5 points of instruction-following on the A/B sets (+1.5/+0.3 over soup24; +4.5/+0.3 over Stage 1) and the lowest repetition score of any Stage-2 full model. Candidates built on the ties-included winner set (soupT: IF-C 71.0, degen 0.119) or on the 6-GPU retrain (soupFr: IF 75.5/77.2, degen 0.111) were equal or worse overall. We did not measure: the organizer's held-out datasets, human preference battles for this exact model, or long-context behaviour.

Limitations

• The preference signal is small (140 preferred answers); the holdout has only 29 pairs, so preference-quality differences between candidates are within noise.
• MMLU set B is 0.5 points below the Stage-1 model; long generations repeat slightly more than Stage 1 (degen 0.098 vs 0.088) though less than any other Stage-2 candidate.
• By design the model answers at the length its human voters preferred — thorough, sometimes longer than a minimal answer needs to be.
• No safety-specific training was performed in Stage 2; the model inherits the Base model's general behaviour. Verify outputs for high-stakes use.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("RSIArena/RSIAgent-Ember-Stage2")
model = AutoModelForCausalLM.from_pretrained("RSIArena/RSIAgent-Ember-Stage2", torch_dtype="bfloat16")
msgs = [{"role": "system", "content": open("system_prompt.txt").read()},
        {"role": "user", "content": "Hello!"}]
out = model.generate(**tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt"),
                     max_new_tokens=1024)

The bundle carries chat_template.jinja, system_prompt.txt (served as the system prompt), generation_config.json and serving_config.json. Use them as shipped.

License

The Base model's terms, OpenMDW-1.1 (https://openmdw.ai/license/1-1/), apply.