amayuelas/Qwen3.5-9B-MatRL-MT-RL

Qwen3.5-9B — MatRL multi-turn RL

Multi-turn RL on top of amayuelas/Qwen3.5-9B-MatRL-MT-SFT, for crystal-structure inverse design. The agent proposes candidate structures, evaluates them against a machine-learning interatomic potential, refines, and submits. Rewards cover validity, novelty, thermodynamic stability and property targets.

This is the model-scale arm of amayuelas/Qwen3.5-4B-MatRL-MT-RL, with the same task definitions, reward and batch shape.

Evaluation

This checkpoint has not yet been scored on the SUN benchmark used for the 4B release. That benchmark uses 800 rollouts per track, the equflashv2 reward judge, and the held-out orb-mpa / eqv3 judges. The only evidence of improvement so far is training reward, which comes from the judge the policy was optimized against. Mean training reward rose from 2.95 (steps 30–49) to 3.39 (steps 160–199).

Training

init	amayuelas/Qwen3.5-9B-MatRL-MT-SFT
algorithm	GRPO-style multi-turn RL, 10 turns / 8 evaluations per episode
batch	128 rollouts per step, 16 per prompt; zero-advantage groups filtered
sampling	temperature 1.15, ≤2,048 tokens per turn
reward	SUN-marginal ×1.0 (validity, novelty, relaxed stability: equflashv2 MLIP, 20 steps / fmax 0.1), group diversity ×1.5, band-gap target ×0.5, bulk-modulus target ×0.5
task mix	includes 20% formula-conditioned, 10% unconditional, 10% band-gap-target, 10% bulk-modulus-target prompts
steps	200
optimizer	lr 1e-6, fp32 optimization and gradient reduction
seq len	24,576 (steps 0–159), 32,768 (steps 160–199)
trainer	prime-rl

Trained on 8×A100-40GB: 4 GPUs for the trainer (FSDP + context parallel cp=2, ulysses), 3 for vLLM inference, and 1 for the reward models.

The optimizer runs in fp32 because bf16 does not work at this learning rate. At lr 1e-6 each Adam update is smaller than the spacing between adjacent bf16 values and rounds away. An earlier 200-step attempt in bf16 finished identical to its init, with no errors.

Training caveats

• Seven segments with weights-only restarts. The 200 steps ran as 0–30, 30–50, 50–60, 60–70, 70–90, 90–160 and 160–200. Segments ended on an env-worker death, a vLLM timeout, and three host OOMs, later traced to an unbounded cache in the environment workers. This model class cannot resume from a DCP checkpoint because there is no optimizer state for the vision tower. Each segment therefore restarted from the previous weights with a fresh Adam state.
• Truncation. At 24,576 tokens, the share of episodes clipped at the trainer window climbed from 1.6% to 38%. The final 40 steps were re-run from step 160, the last low-truncation checkpoint (7.8%), at 32,768 tokens. Truncation there averaged 14.7% and peaked at 26.6%. A clipped episode's ending, including the submit turn, contributes no gradient.
• Reward components. Unlike the 4B release, whose bulk-modulus reward never computed, every component here was live. Before each launch, a preflight ran a real relaxation, a real band-gap prediction and the elastic-fit import. Over the last two segments, roughly 100 band-gap requests failed with transient server disconnects and scored no band-gap term.

Precision

Tensors are stored in bf16, and config.json declares bfloat16. The trainer writes its fp32 optimization dtype into the config, and that value was corrected for this upload so that dtype="auto" and vLLM do not load the model at twice the memory.

Important: thinking channel

Reasoning lives in the native thinking channel, preserved across tool calls. Do not sample with enable_thinking=false, which turns off the behavior this model was trained to use.

Serving

Includes preprocessor_config.json / video_preprocessor_config.json. vLLM requires them for this VL-capable model class even for text-only serving.