RL-Forgetting-Experiments-3/code-rl-lr-ablation

Code RL learning-rate ablation (MBPP)

GRPO on MBPP at three actor learning rates below the 1e-6 default -- 6e-6, 3e-6 and 6e-7 -- each run twice: once with no replay, once with an SFT-replay buffer (hard_cooldown, lambda=0.1). The question is where on the lr ladder the run stops being stable, and whether replay moves that point.

The lr=1e-6 headline runs live in their own repos (RL-Forgetting-Experiments-3/code-q3_1p7b-nobuf, -replay-lam0p1, RL-Forgetting-Experiments-3/code-q25_3b-*); above this ladder, 1e-5 is a coin flip and 1e-4 / 1e-3 are reliably fatal for both arms.

Layout

One folder per ablation cell, named exactly as the local run directory, so a point in any eval CSV maps to a folder here without a lookup table:

<model>_mbpp_<arm>_<lr>/
    global_step_<N>/     a complete HF model directory (stride 30 + endpoint)
    logs/                trainer logs
    rollouts/            sampled rollouts

folder	base model	arm	lr	checkpoints
qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr3e6	Qwen/Qwen3-1.7B-Base	replay buffer (lambda=0.1)	3e-6	30-500 step 30
qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr6e7	Qwen/Qwen3-1.7B-Base	replay buffer (lambda=0.1)	6e-7	30-500 step 30
qwen3_1p7b_base_mbpp_nobuf_lr3e6	Qwen/Qwen3-1.7B-Base	no buffer	3e-6	30-500 step 30
qwen3_1p7b_base_mbpp_nobuf_lr6e6	Qwen/Qwen3-1.7B-Base	no buffer	6e-6	30-500 step 30
qwen3b_mbpp_hardcd_lam0p1_lr3e6	Qwen/Qwen2.5-3B	replay buffer (lambda=0.1)	3e-6	30-300 step 30
qwen3b_mbpp_nobuf_lr3e6	Qwen/Qwen2.5-3B	no buffer	3e-6	30-300 step 30
qwen3b_mbpp_nobuf_lr6e6	Qwen/Qwen2.5-3B	no buffer	6e-6	30-300 step 30
qwen3b_mbpp_nobuf_lr6e7	Qwen/Qwen2.5-3B	no buffer	6e-7	30-300 step 30

from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
        "RL-Forgetting-Experiments-3/code-rl-lr-ablation", subfolder="qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr3e6/global_step_500")

Training

• Data: 320 MBPP problems, held out from both MBPP+ (378) and MBPP's canonical test split (276), so both stay reportable.
• GRPO, 8 rollouts per prompt, batch 64, response length 3072; binary reward from executing the generated program against the task's asserts.
• Replay arms additionally replay 128 past rollouts per step at weight lambda=0.1, sampled by hard_cooldown.
• Everything except the actor lr is identical to the 1e-6 runs.

Evaluation

MBPP+ (378 problems), n=160 samples, temperature 0.6, top_p 0.95, unbiased pass@k. Scoring uses MBPP+'s full test suite (~108 tests/problem), which is markedly stricter than MBPP's original 3 asserts.