RSIArena/RSIAgent-Aspen-Stage2

Aspen instruction-tuned assistant

Model

A LoRA chat adapter created by an autonomous research agent in RSI Arena, from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base, revision 434456c9a6753f29d24e23c95d622aaf17111b3b. Stage1 parent: runs/clean3_dpo/step-00150, produced by Slurm job 1860. Current Stage2 selection: runs/s2_allkl_1/step-00080, job8417. Targeted feedback cross-entropy is regularized with full-parent forward KL on feedback and own-policy replay. This nomination was selected after broader retention checks and rejected continuations. This is not a claim of optimality or organizer evaluation success.

How to use

Use transformers with Nemotron-H support (tested 5.7.0), PEFT (tested 0.18.1), PyTorch and the model's Mamba/causal-convolution dependencies. Authenticate to access this private repository.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
from peft import PeftModel
from huggingface_hub import hf_hub_download
from pathlib import Path
repo = 'RSIArena/RSIAgent-Aspen-Stage2'
base = AutoModelForCausalLM.from_pretrained(
    'nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16',
    revision='434456c9a6753f29d24e23c95d622aaf17111b3b',
    dtype=torch.bfloat16, device_map='auto', attn_implementation='sdpa')
model = PeftModel.from_pretrained(base, repo).merge_and_unload().eval()
model.config.use_cache = True
tokenizer = AutoTokenizer.from_pretrained(repo)
system_prompt = Path(hf_hub_download(repo, 'system_prompt.txt')).read_text().strip()
messages = [
    {'role': 'system', 'content': system_prompt},
    {'role': 'user', 'content': 'Explain photosynthesis briefly.'}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
x = tokenizer(text, return_tensors='pt', add_special_tokens=False).to(model.device)
with torch.inference_mode():  # native inference, no outer autocast
    y = model.generate(**x, generation_config=GenerationConfig.from_pretrained(repo))
print(tokenizer.decode(y[0, x.input_ids.shape[1]:], skip_special_tokens=True))

The bundled ChatML template and system prompt are the training/serving format. Default decoding is greedy, maximum 4096 new tokens, EOS IDs 11 and 2. No special thinking mode. The input allowance is 28672 tokens. Long-context reference overlap was measured below; it is not a guarantee of faithful retrieval.

Stage1 training lineage

Clean-v3 rebuild directly from the unchanged Base; no adapter or generated targets from earlier experimental lineages. Slurm job1857 completed800 updates on4 RTX PRO6000 Blackwell96GB GPUs in2626seconds (2.9178 GPU-hours, excluding development and evaluation). LoRA rank64/alpha128 on Mamba in_proj, attention q/k/v/o and shared-expert up/down projections; routed experts frozen. Full FP32 lm_head trained, with output row11 initialized from Base EOS row2. Frozen BF16 Base, FP32 adapter/head, global BF16 training autocast, deterministic expert accumulation and gradient checkpointing. DDP global16 packed4096-token blocks/update,12800blocks consumed; assistant-only loss including EOS11, padding masked by position. AdamW betas(.9,.95), weight_decay.01, clipping1; adapterLR2e-4/head2e-5,warmup30,cosine3000steps with15%floor. Final32validation mean sequence CE.905945.

Stage1 phase2: continued from clean-v3 phase1 above, Slurm1859 completed1000 updates in3231seconds on4GPUs (3.590 GPU-hours). Same trainable parameters and global16/4096packing; LR1.2e-4/head1.2e-5,warmup50,cosine3500stepswith15%floor. New optimizer,16000blocks consumed. Final32validation sequence CE.616166. SFT producing-job subtotal6.5078GPU-hours, excluding development/evaluation.

Stage1 phase3: DPO from phase2, GLOBAL BF16 autocast training (not native-only precision), FP32 trainable head and Adam states; completed Slurm1860:150 updates,1945seconds x4GPUs=2.1611GPU-hours INCLUDING reference caching. Sum-token DPO beta0.1 plus0.02 chosen-response NLL, global32pairs (2/rank x4accum x4ranks); LR1e-5/head1e-6,warmup25,cosine450steps. Reference probabilities freshly cached from the same initial stage2 model, with grouping and recomputation assertions. Total producing-job chain8.6689GPU-hours. Validation256pairs:reward accuracy.71484375,margin.65185,chosenNLL.57336. No RL or benchmark-target training.

Training data

Public TRAIN splits of allenai/tulu-3-sft-mixture and Magpie-Align/Magpie-Pro-300K-Filtered; source-capped mixture, whole conversations fitting4096tokens, no truncated assistant targets. English focused with some multilingual Aya/chat. Deduplication, reserved prompt exclusions, malformed role/function/control-token and identity filtering. Excluded FLAN, hardcoded tasks, NuminaMath-TIR and OpenMath/GSM8K source. The final selected v3 corpus contains304499TRAIN conversations,198352052nonpaddingtokens,55162packedblocks;485reservedvalidation conversations. DatasetSHA256 372c66ae07592d2209c61590ae892b593be450acf509b7090ef11b322c5535e8. Sources retained include Magpie, manual IF, PersonaHub math/code, Aya, OASST1,no_robots,WildChat,safety,TableGPT and SciRIFF.

Stage1 phase2 data:259590TRAIN chats,152339172nonpaddingtokens,42361packedblocks,372reservedvalidation. SHA256 3a5342674bc5b332752884a8c85a356b397b42ffb0cd2042a283af039423eff9. Sources, all TRAIN:

• allenai/Dolci-Instruct-SFT@bd3c8f3a9b2cc5a9682e44b96ddd0bb2ff027221: upgraded WildChat, Precise IF, Verifiable Reasoning and Logic Puzzles. Entire Python Algorithms/correct-python-sft family excluded, as are FLAN,hardcoded,OpenMath,tool-use and opaque OpenThoughts science.
• Magpie-Align/Magpie-Reasoning-V2-250K-CoT-DeepSeek-R1-Llama-70B@d78edb811991faba57a3b7226719f3818460e723: good/excellent inputs, final worked responses only, explicit thinking removed.
• nvidia/HelpSteer3@f6d145777bcbde96137596340fab89793acd1031 edit/train: human-edited responses for SFT; preference data used separately in stage3.
• Stage1 replay. Whole-chat4096limit, original split/order preserved after expanded audit;143179/403141prior selectedTRAIN+validation rows rejected.

Stage1 phase3: nvidia/HelpSteer3 preference TRAIN split at the same revision, strong-consensus human preference pairs; complete responses, reserved validation, expanded rejection-only audit as below.12549TRAIN pairs/443reservedvalidation. DatasetSHA256 6bd33c0252f05691916ab380b32d64dc46088e585b2e961aab63e6da213942d2.1767previously selectedTRAIN and57validation pairs removed by expanded audit.

Expanded contamination remediation

Earlier experimental public-source mixtures were discovered to contain incidental public benchmark prompts, including HumanEval-derived code despite upstream decontamination claims. ALL those earlier weights and their descendants were excluded from final selection. This nominated lineage was independently rebuilt from the original Base after an expanded rejection-only audit. Entire evolved-codealpaca and correct-python-sft/DolciPython source families were excluded (including replay). All selected user turns were checked against public MMLU-Pro, IFEval, MATH500/fullMATHtest,HumanEval,GSM8Ktest and MBPPtest PROMPTS using normalized exact/13word matching; also excluded benchmark mentions and literal HumanEval entrypoint names. The literal rule intentionally over-removes generic words. Prepared v3 rejected111978/416962prior selectedTRAIN+validation rows, including entire risky source families. AuditSHA256 b755c5c3d705a3780580565030d88ea478ade00dc619d92411f9c67b85c9774c. Only test prompts were used to reject records; no test answers, organizer data or evaluated outputs became training targets. This is not proof of absence of semantic or unknown benchmark overlap.

Stage1 parent evaluation

Our own model-selection measurements, NOT the organizer's independent evaluation. Native transformers BF16 merged-adapter reload, no global autocast; greedy. General reload job1864 completed successfully. Authored dev126/broad75/hard72 max1536/1536/3072tokens,batch4/4/2:

• Dev126:83/84math,11/12Python,10/10format;20unscoredchat.
• Broad75:26/30math,8/10Python,9/11print/format;24unscoredchat.
• Hard72:19/30math,11/15Python,7/9format;18unscoredchat.
• All273generations stopped;485reservedv3stage1 conversations assistant-token-weighted CE.742419. Broader independent-source evaluation (still our own model selection):
• MMLU-Pro balanced1400:943correct,15unparsed,1384stop; max1536tokens,batch4. Stage2 parent916correct.
• HumanEval conservative125of164 subset, standard source tests in resource-limited isolated sandbox, NOT EvalPlus:111pass;125stop. Parent106pass. Max3072tokens.
• MATH500 conservative487of500 subset:408correct (83.778%),486stop; max3072tokens. Parent397correct. math-verify0.8.0 equivalence in an isolated resource-limited sandbox.
• Reserved science604 multiple-choice:451correct,603stop; parent449correct.
• LongQA96 natural held-out English LongAlign:mean bag-wordF1.436429/ROUGE-LF1.296888,95stop. Parent.429964/.303075,92stop. These are lexical reference agreement only, NOT correctness/faithfulness. IFEval541:strict426/541prompts and706/834instructions; loose447/541prompts and730/834instructions;524/541stop,468.33mean tokens. Parent strict418/541prompts and698/834instructions,500stop. Max4096tokens; standard instruction-following-eval scorer revision d36068b845da4c2b24927fee2cea1e6ef98dadda. This DPO model replaces the SFT parent based on broader math/code/knowledge improvements despite slightly weaker authored hard checks. Fixed conservative subsets were defined during prompt-only contamination auditing, not from model success. Small authored suites have limited precision and have been repeatedly used for selection. No evaluation output is used for training. No comprehensive independent human or safety evaluation.

Stage2 continual training from human feedback

The213 provided arena battles were split deterministically into170training/43validation, stratified by outcome/web. Feedback source SHA256 d302b047663ae92435162fe83dcaa0b466838fc4a3e461019e4d91faa0899b14. The agent authored76 revised responses from TRAIN battles, correcting errors and emphasizing direct, proportionate, calibrated answers. These are synthetic edits, NOT additional human preference labels. For3multiturn battles only the first turn was used. No heldout prompt, response or judgment was a training target. Private feedback is not published.

For the selected continuation,76unchanged revision targets were repeated32times.1024already-clean Stage1 TRAIN prompts (512reasoning/512other) were generated greedily by the frozen Stage1 DPO parent, never from development prompts.988complete conversations fit2048tokens and were retained,4copies each. Sources include Dolci reasoning/logic/preciseIF/WildChat, Magpie reasoning, HelpSteer edits and clean Stage1 replay.1549packed2048tokenblocks;516800feedbacktokens and536124replaytokens per copy. Dataset SHA256(ids+labels) 40c3238975f85d9cf05726988e2521e3b9358a180ced13fe2fc699bc4dc82e54. Complete assistant turns including EOS11 are supervised; padding is masked by position. Target constraints and selected math/code revisions were checked, but correctness is not guaranteed.

Selected method: cross-entropy on feedback-edit tokens plus full-distribution forward KL from the exact frozen Stage1 parent on ALL supervised positions, both feedback and replay. KL coefficient1.0; replay has KL only, no one-hot CE. Initial same-input KL was zero. A frozen adapter/head clone provides the reference distribution; no external teacher weights or targets are in this lineage. Continued original Stage1DPO150 directly, NOT the earlier Stage2 SFT model. Rank64 LoRA and full FP32head master, BF16compute/frozenBase, nativeprecision without global autocast; AdamW, global16 (2ranks x1block x8accum),2048tokens, adapterLR3e-5/head3e-6,warmup10/cosine200, clipping1. Job8417 completed160updates in1456jobseconds on2GPUs (0.8089GPUhours including later checkpoints); selected step80. Producing-chain8.6689Stage1+0.8089Stage2GPUh; this is NOT total research expenditure, excludes own-policy generation and evaluation. Saved adapter was reloaded/merged for all evaluations below.

Stage2 current-checkpoint evaluation

These are training-agent model-selection measurements, NOT organizer results. Native merged BF16, no outer autocast, greedy. Combined238 includes48new author-written qualitative questions,43heldout feedback firstturns without webtools,147earlier authored hard/broad checks. Current checkpoint:238/238stop,217.28mean tokens, math45/60,code20/25,format17/20. Previous SFT80 nominee:41/60,21/25,16/20. Stage1:45/60,19/25,16/20. Batch4,max3072 for current comparisons; some original Stage1 subset settings differed. Small repeatedly-used development checks are noisy.

Blinded answer-order paired review against previous Stage2 SFT80 on all43reservedfeedback prompts:14current wins,13losses,16ties. Original24review included8nonfeedbackdev cases; feedback-only result is stated here. Judgments were saved before unblinding. Reviewer is the training agent, not independent humans; recognizable style and prior context limit blinding. This is effectively a subjective tie, NOT robust evidence of greater helpfulness. The previous SFT80 itself had21wins/14losses/8ties versus Stage1 on these43. These comparisons are not transitive. No search tools were used even on web-origin questions. All outputs/judgments stayed evaluation-only.

Public retention checks, fixed prompt-only rejection-audited subsets as described above:

Evaluation	Stage1 parent	Previous SFT80	Current all-token KL1/80
MMLU-Pro balanced1400	943	948	946
MATH conservative487	408	398	401
HumanEval conservative125, original strict sandbox	111	103	108
HumanEval same outputs, generic sandbox v2	not rescored here	104	109
IFEval strict prompt541	426	415	418
IFEval strict instruction834	706	694	699

MMLU max1536tokens; MATH/code max3072; IF max4096. Current stop counts1397/1400 MMLU,484/487 MATH,125/125code,528/541IF. IF loose440/541prompts,725/834instructions. Code v2 generically allows benign future,doctest,hashlib imports and name, otherwise the same resource-limited isolated execution; no answer edits. This remains standard source tests, not EvalPlus.

Additional prospective source-TRAIN reserved validation, never fitted:256codeparrot/apps TRAIN tasks reserved before primary-code experiments/verification/audit. Current117/256source-test passes versus previous SFT80 115;255/256stop versus254. Conservative isolated AST/function-or-stdin source-test runner; source tests are incomplete and extraction/AST false negatives exist. APPS has NOT trained this selected lineage. All candidate-generation, distillation, primary-MATH/APPS-SFT and own-policy preference experiments outside this lineage are excluded from its weights.

Selection favors modest math/code/instruction retention recovery over SFT80 with essentially tied subjective feedback quality. Gains are small and do not establish independent statistical significance; current still trails Stage1 on several retention metrics. Multiple alternative continuations were tested and rejected as described below. No comprehensive independent human or safety evaluation, and no claim of organizer improvement.

Additional direct feedback comparison and rejected continuations

A later direct randomized A/B review of all43 reserved feedback first-turn prompts compared the selected model with native-regenerated Stage1 answers: 20 selected-model wins,13 losses,10 ties. The training agent saved all judgments before opening the randomized key. This is not independent human evaluation, and recognizable style, prior exposure and repeated model selection limit blinding. It supports a possible moderate subjective improvement, not statistical certainty or a claim about future human votes. Both models still made material errors, including fabricated details and incorrect arithmetic. No review output or judgment was used for training.

On256 primary MATH TRAIN tasks reserved before data audit or experiments, the selected model scored219 source-box agreements, versus218 for previous SFT80 and217 for Stage1. These tiny differences are not robust improvements. Combined with APPS validation, this is a source-generalization diagnostic, not complete proof correctness.

Rejected alternatives included raw feedback DPO, SFT/edit-DPO and replay/precision variants, stronger KL, external-teacher targets, primary MATH/APPS supervised targets, and own/near-policy verified preferences. No external-teacher or MATH/APPS targets are in the selected weights. A mixed math/code preference continuation improved some public/source checks slightly but lost knowledge/stopping performance and had12wins/14losses/17ties against the selected model in the same agent-reviewed43feedback set. It was not selected.

A larger exact-policy MATH TRAIN experiment generated16384samples on2048 previously unused training tasks. Of451candidate preference pairs,406 were assigned TRAIN and45 validation before audit. A fixed24TRAIN-pair reasoning audit rejected11 for invalid reasoning or false answer contrasts. LaTex spacing, thousands separators and percent-unit placement could fool the box-equivalence filter. The entire new dataset was rejected and never trained; generic normalization diagnostics did not approve it. Earlier small math-preference experiments may share verifier limitations and were also not selected. This illustrates why final-answer agreement alone was insufficient for training-data approval.

Final trajectory selection

Previously saved step40 and step120 from the same completed training job were checked without new optimization. Neither displaced step80:

Diagnostic	Step40	Selected step80	Step120
MMLU-Pro /1400	915	946	947
MATH /487	377	401	397
HumanEval v2 /125	104	109	109
IFEval strict prompts /541	418	418	413
IFEval strict instructions /834	699	699	695

Step120 also stopped on521/541 IFEval prompts versus528/541 for selected80, and had14wins/19losses/10ties against step80 in randomized training-agent review of all43 reserved feedback prompts. Judgments were persisted before key disclosure, with the same prior-exposure/blinding limitations as other reviews. Step40 had clear broad retention losses; step120 did not offer a balanced improvement. These repeatedly consulted diagnostics are selection evidence, not an independent estimate of final generalization. The final nomination is unchanged at step80.

Final orthogonal retention checks

Selected step80 was also evaluated on the longstanding reserved science604 and natural English LongAlign96 suites, without any new fitting. Same native merged BF16/no-autocast greedy evaluation as Stage1, max1536 new tokens, batch4 for science and1 for LongQA.

• Science: 453/604 correct, versus Stage1 451/604. Both603/604 stopped. Mean output tokens135.69 versus152.22. Paired outcomes:42 became correct and40 became incorrect; this small net change is not evidence of a meaningful gain.
• LongQA: mean bag-word F1 0.429201 and ROUGE-L F1 0.295453, versus Stage1 0.436429/0.296888. Identical96 prompts and references were asserted, and both outputs rescored. Paired bootstrap95% intervals for selected-minus-Stage1 were[-0.029748,0.014734] for bag F1 and[-0.018681,0.015470] for ROUGE-L. These measure lexical agreement only, not correctness or faithfulness. Stopping was worse:92/96 versus95/96, although mean output length was shorter312.93 versus423.60 tokens. Neither model has established reliable long-context retrieval.

Limitations

Arithmetic/logic/code mistakes and instruction violations remain. Practical security/backup prioritization, personal-experience fabrication, exact word counts and lengthy analysis can fail. Some outputs are repetitive or truncated. Emergency advice is not guaranteed safe. Native precision and chat formatting matter. No tool-use training or demonstrated long-context faithfulness; lexical LongQA overlap is not a correctness guarantee, and selected-model stopping on LongQA was worse than Stage1 at the tested limit. Unknown semantic training/evaluation overlap cannot be ruled out by prompt matching. Shorter answers are not automatically better.

License

The Base OpenMDW-1.1 terms apply. Consult training-data provenance and terms as well.