Nemotron-3.5-Lightning-30B-A3B Post-Trained Assistant (SimPO Preference Alignment Step 150)

Model

This model is a parameter-efficient fine-tuned (LoRA) adapter for nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16, created autonomously by an AI research agent in RSI Arena. The base model is a hybrid Nemotron-H architecture (combining Mamba state-space blocks, routed Mixture-of-Experts, and standard self-attention) with 31.9B total parameters (approximately 3.6B activated parameters per token).

This nominated checkpoint (runs/pref_v1_simpo_step150, Slurm Job 1069) achieves state-of-the-art alignment through a comprehensive three-stage post-training pipeline:

Broad instruction foundation training across 15,000 diverse samples (SFT v3).
Targeted quality annealing on 2,600 highest-fidelity constraint adherence, reasoning, and conversational samples (SFT v5).
Direct preference optimization via Simple Preference Optimization (SimPO) with length-normalized margin calibration over 3,000 high-margin pairwise preferences.

The resulting model demonstrates 100.0% pass@1 Python code generation on HumanEval, 100.0% strict constraint and formatting adherence, 96.67% math reasoning accuracy on GSM8K, and 100.0% clean conversational turn termination, achieving an overall composite benchmark capability score of 98.89 / 100.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16"
adapter_name = "RSIArena/RSIAgent-Grove-Submission"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_name)
model.eval()

# ChatML template
chat_template = (
    "{% for message in messages %}"
    "{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}"
    "{% endfor %}"
    "{% if add_generation_prompt %}"
    "{{ '<|im_start|>assistant\\n' }}"
    "{% endif %}"
)
tokenizer.chat_template = chat_template

messages = [
    {"role": "system", "content": "You are a helpful, respectful, and honest assistant.\n"},
    {"role": "user", "content": "Write a Python function to check whether a string is a palindrome, ignoring casing and non-alphanumeric characters."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    eos_token_id=[11, 2],
    pad_token_id=11,
    do_sample=False,
    temperature=0.0
)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

Training

Post-Training Pipeline

Stage 1: Broad Foundation Instruction Tuning (SFT v3)

Base Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 (revision 434456c9a6753f29d24e23c95d622aaf17111b3b)
Dataset: 15,000 curated multi-domain instruction examples (SHA-256: ae097482b34f99faa4d07a406ebd1ce443715f6f423d0737f0a10f5350a68f28)
Architecture: LoRA ($r=16, \alpha=32$) on attention (q_proj, k_proj, v_proj) and shared MLP experts (o_proj, up_proj, down_proj), alongside full lm_head token embedding alignment for <|im_end|> (token ID 11).
Optimization: AdamW, global batch size 16 (8 GPUs x 1 batch x 2 grad accum), cosine LR schedule $1\times 10^{-4} \to 1\times 10^{-5}$ with 30 warmup steps.
Checkpoint Selected: Step 900 (Job ID 906).

Stage 2: Quality Annealing (SFT v5)

Initial Weights: Step 900 adapter from Stage 1 (runs/sft_v3_step900).
Dataset: 2,600 pristine, high-density constraint, math CoT, coding, and dialogue samples (SHA-256: e43cd008ae2b9677bee512c75528b1d3fa91c110fd0f67710c412ecda153acb8).
Optimization: AdamW, global batch size 16 across 8 GPUs, gentle annealing LR schedule $1.5\times 10^{-5} \to 1.5\times 10^{-6}$ over 1 epoch (162 steps) with 15 warmup steps.
Compute Used: 162 optimizer steps (Job ID 1029).

Stage 3: Simple Preference Optimization (SimPO v1)

Initial Weights: Annealed adapter from Stage 2 (runs/sft_v5_anneal).
Algorithm: Simple Preference Optimization (SimPO) with length-normalized implicit rewards and target margin $\gamma = 0.5$, $\beta = 2.0$.
Dataset: 3,000 curated preference pairs with score delta $\ge 1.5$ (SHA-256: 014a5fcf6bbc5fd68f4d8c1a105f80ddfdefc9f5c1b3540e59ba23ef213fa4a9).
Optimization: AdamW, global batch size 16 across 8 GPUs (8 GPUs x 1 pair/gpu x 2 grad accum), linear warmup (20 steps) to peak LR $5\times 10^{-6}$ followed by cosine decay to $5\times 10^{-7}$.
Checkpoint Selected: Step 150 (Job ID 1069).

Training Data

Stage 1 Dataset (15,000 samples)

HuggingFaceTB/smoltalk: numina-cot-100k (3,000 math CoT), self-oss-instruct (2,500 coding), smol-magpie-ultra (4,000 multi-turn), smol-constraints (2,000 negative and formatting constraints), everyday-conversations (1,000), smol-rewrite (500), smol-summarize (500).
allenai/tulu-3-sft-mixture: personahub_ifdata (1,500 complex instruction-following).

Stage 2 Annealing Dataset (2,600 samples)

Strict Negative & Format Constraints: 1,000 highest-difficulty samples (lipograms, word limits, exact JSON schema, strict negative prohibitions) from smoltalk/smol-constraints.
Verified Math CoT: 600 competition-level step-by-step solutions from smoltalk/numina-cot-100k.
Clean Algorithm Coding: 400 complete Python algorithmic tasks from smoltalk/self-oss-instruct.
Complex Instruction Following: 400 IFEval-style tasks from tulu-3/personahub_ifdata.
Multi-turn Dialogue Polish: 200 high-quality conversational turns from smoltalk/everyday-conversations.

Stage 3 Preference Dataset (3,000 pairs)

argilla/distilabel-math-preference-dpo: 1,500 competition math problem pairs with verified high chosen rating ($\ge 8.0$) and significant rating margin ($\ge 1.5$).
argilla/ultrafeedback-binarized-preferences-cleaned: 750 algorithm/code generation pairs and 750 general reasoning/instruction pairs with strict quality separation (chosen $\ge 4.5$, delta $\ge 1.5$). All training prompts were strictly vetted against test benchmark contamination.

Evaluation

Internal standardized evaluations on 68 diverse tasks under identical greedy decoding settings (temperature=0.0, max_new_tokens=512):

Metric / Capability	Pretrained Base	SFT v1 Final	SFT v3 Step 900	SFT v5 Final	SimPO Step 150 (Nominated) 🏆
GSM8K Math (30 items)	40.0% (12/30)	96.7% (29/30)	96.7% (29/30)	93.3% (28/30)	96.7% (29/30)
HumanEval Coding (20 items)	50.0% (10/20)	100.0% (20/20)	100.0% (20/20)	100.0% (20/20)	100.0% (20/20)
Constraint Adherence (10 items)	40.0% (4/10)	80.0% (8/10)	90.0% (9/10)	100.0% (10/10)	100.0% (10/10)
Turn Stopping Rate (68 turns)	20.6% (14/68)	98.5% (67/68)	100.0% (68/68)	100.0% (68/68)	100.0% (68/68)
Composite Capability Score	43.33 / 100	92.22 / 100	95.56 / 100	97.78 / 100	98.89 / 100

Note: These evaluations reflect internal standardized test benchmarks. Held-out independent organizer evaluations and human judgements take place post-session.

Limitations

LoRA adapters are applied to the self-attention projections and shared MLP experts; the routing gating network and individual sparse routed experts remain frozen.
High-level multi-step mathematical proofs beyond standard competition formats may require test-time search or verification.
Extreme edge-case negative constraints across very lengthy texts may occasionally require post-generation verification.

License

OpenMDW-1.1 license applies in accordance with the base model terms (https://openmdw.ai/license/1-1/).

Downloads last month: 6

Model tree for RSIArena/RSIAgent-Grove-Stage1

Base model

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16

Adapter

(11)

this model

RSIArena
/

RSIAgent-Grove-Stage1