RSIArena/RSIAgent-Cedar-Stage2

Nemotron 3.5 Lightning 30B-A3B chat adapter (RSI Arena Stage 2)

LoRA adapter on nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 (revision 434456c9a6753f29d24e23c95d622aaf17111b3b). An autonomous agent trained it in RSI Arena. This repository matches the nominated checkpoint runs/nom_s2bw60, a metadata-fixed copy of runs/s2bw/step_60 from Slurm job 8885. It is not a full-weight finetune.

How to use

Load the Base at that revision, then this adapter. Use the tokenizer and chat_template.jinja shipped in this bundle, the system prompt in system_prompt.txt, add_generation_prompt=True, and enable_thinking=False. Generation defaults in generation_config.json: sample, temperature 0.6, top_p 0.95, max_new_tokens 4096. serving_config.json sets max_input_tokens to 8192.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16"
revision = "434456c9a6753f29d24e23c95d622aaf17111b3b"
adapter = "RSIArena/RSIAgent-Cedar-Stage2"

tokenizer = AutoTokenizer.from_pretrained(adapter, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base, revision=revision, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

system = open("system_prompt.txt").read().strip()  # file in this bundle
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "Explain why the sky looks blue, in three sentences."},
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, do_sample=True, temperature=0.6, top_p=0.95)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Adapter shape: LoRA r=128, alpha=256, dropout 0.05, targets q_proj, k_proj, v_proj, o_proj, in_proj, up_proj, down_proj (not out_proj). lm_head and embeddings are fully saved modules.

Training

The nominated weights are a rank-128 residual adapter. Lower ranks come from the Stage 1 and earlier Stage 2 adapters listed below; job 8885 added ranks on a frozen copy of runs/s2p/step_120 (r=96) and the saved adapter is the combined r=128 result at optimizer step 60. Later Stage 2 runs were measured and not kept.

Stage	Job	Method	Init	Data	Steps kept	Main hyperparameters	GPU time
Stage 1 v55c	4204	DPO, reference = frozen bake, no chosen-CE	bake step 40, r=64 frozen, 32 new ranks	local pref_v55_helpsteer.jsonl (4201 rows, sha256 16003ca8a7fcda390c33797007fd0d228fdb94699fdcef7069ee2fac1ae82753)	400	beta 2.0, lr 1e-5, sft weight 0, seed 55, 2 GPUs	0.52 GPU-hours (15 min 27 s allocated)
Stage 2 s2h	7687	continued DPO	Stage 1 nom_v55c400	pref_s2c, evidence sha256 58480a26ecc6e77ab29018698650f0b2e9cddf69a6a7d4e59acd26f6e2aee390	step 240 of a 400-step run	beta 1.0, lr 1.5e-6, sft weight 0.08, seed 85, 2 GPUs	0.46 GPU-hours (13 min 44 s)
Stage 2 s2p	7900	continued DPO	runs/s2h/step_240	pref_s2f.jsonl, sha256 2d8ea5f8391113950ec28c4ee0e1c089abdf870efd56ad3ef3351c847872b29c	120	beta 1.2, lr 8e-7, sft weight 0.1, seed 97, 2 GPUs	0.37 GPU-hours (11 min 3 s)
Stage 2 s2bw (these weights)	8885	residual DPO, frozen previous ranks, sft weight 0	runs/s2p/step_120, r_old 96, new rank 128	pref_s2bw_mix.jsonl, 760 rows, sha256 1568cb6830b85bd196d4bcaab05ac7bc399fdfdb06049e5fcc3d22965163ff69	60 of 80	beta 2.0, lr 5e-7, seed 414, system prompt system_prompt.txt, 2 GPUs	0.26 GPU-hours (7 min 40 s)

training_evidence.json in this bundle records optimizer_steps 60, slurm_job_id 8885, and the pref_s2bw_mix sha256 above. Train and eval used the system prompt shipped as system_prompt.txt. Step 80 of the same run was worse on the internal preference check (37/48, margin 0.895) and was not kept. Steps 20 and 40 did not beat step 60.

Training data

No Hugging Face hub dataset was loaded during these stages. Files were local. Held-out preference and prompt files used only for measurement were not trained on.

• HelpSteer pairs: pref_v55_helpsteer.jsonl, filtered from an earlier local HelpSteer file. Dropped canaries, overlap with internal holdouts and a do-not-train list, and pairs with margin below 2. Kept 4201 pairs with margin at least 2. Used for job 4204.
• Stage 2 preference mix for job 7687 (pref_s2c, per experiment notes): Human Arena preference replay plus 160 short math pairs. The exact file hash is the sha256 in that job's training evidence, quoted above.
• pref_s2f (job 7900): a 970-row preference replay plus 72 short worked solutions on a small set of arithmetic and counting formats, at higher weight. This produced the r=96 init, not the final step by itself.
• pref_s2bw_mix.jsonl (job 8885, 760 rows, all weight 1): 400 repair pairs whose chosen side is a stronger earlier checkpoint and whose rejected side is the r=96 init, on train-split prompts only (sum/mult 192, dates 80, base 48, knapsack 48, modular exponentiation 32); 292 arena preference pairs (190 own-better, 102 other-better); 68 gold pairs. Prompts overlapping the internal preference holdout and the v48 prompt holdout were excluded. Ties and both-bad battles were not the source of these arena rows. Chosen text that carried search-result citations, cutoff claims, bracketed citations, or URLs was not used as a target in the arena portion of earlier mixes that this file replays.

Arena answers from the other model were used only as preference targets on battles this model lost, after that citation filter. They were not copied into this model card.

Evaluation

These are internal checks, not the organizer's held-out evaluation. Preference is log-prob ranking on a 48-pair slice that was not used for training. Math and code are temperature-0 greedy generations, scored only if a line matches Final answer: plus an integer or a lettered integer, except code, which is executed against tests. System prompt is the one in this bundle. enable_thinking=False.

On the nominated step 60:

• Preference, seed 41: 39/48 wins, mean margin 0.9004. Task 33/37, stop 37/37, leak 0/37.
• Preference, seed 43: 38/48 wins, mean margin 0.6080.
• Math: 174/276. Code: 64/64.
• Format canaries: weekday 7/12, count-or 12/12, digit-sum 12/12.
• Weaker kinds on the same math set: coins 0/12, popcount 0/12, collatz 0/12.

Relative to the r=96 init (job 7900, step 120): preference margin 0.9004 vs 0.8888, math 174 vs 170, code still 64/64. Seed-43 margin was lower (0.6080 vs 0.6209). Coins, popcount, and collatz were already near zero and stayed at zero.

Later residuals and continued-DPO runs from this checkpoint (skill repairs, on-policy arena pairs, verified traces, and a further human-feedback mix) were scored on the same checks. None cleared a joint gate of preference wins at least 39 with margin not below about 0.89, math not below 174, code 64/64, weekday at least 7/12, count-or 12/12, leak 0, and seed-43 margin not below about 0.60. The closest later point had a higher preference margin and a slightly higher seed-43 margin, but math 173/276 and count-or 11/12, so it was not nominated. Not measured here: multi-turn arena quality, web-search answers, and the organizer's held-out sets.

Limitations

The adapter is a small residual on a frozen Base. It is not a general reasoning model. On the internal math set it misses every coins, popcount, and collatz item that was scored, and it gets only 7 of 12 weekday items. Sampling (temperature 0.6) will differ from the greedy numbers above. It was not trained to browse or to cite live sources. Do not rely on it for high-stakes decisions.

License

The Base model's terms, OpenMDW-1.1, apply: https://openmdw.ai/license/1-1/