arpjw/Qwen3-4B-Adaptive-Reward-Coding

Qwen3 4B Adaptive Reward Coding Pilot

This rank-8 LoRA adapter is the adaptive arm from a small audited reinforcement-learning experiment on synthetic Python expression repair. It was trained from Qwen/Qwen3-4B-Instruct-2507 at revision cdbee75f17c01a7cc42f958dc650907174af0554 with a success reward and an adaptively updated token-cost penalty.

This release is a research artifact. The experiment found no measured adaptive-reward advantage: the fixed and adaptive arms produced identical sampled training and development token sequences despite distinct final weights.

Measured result

On 16 development tasks with two matched samples per task and mode, the base, fixed, and adaptive policies were all 32/32 correct in reasoning mode. Mean reasoning length was 746.03 tokens for the base and 735.66 for both trained arms, a descriptive 1.39% reduction. The fixed-versus-adaptive difference was zero for both accuracy and token cost in every paired evaluation.

These are selected development tasks from one seed and a 16-step run. No untouched final-test inference was performed. The result does not establish adaptive-reward benefit, general token savings, or repository-level coding improvement.

Training

The adapter modifies attention q/k/v/o projections with rank 8, alpha 16, and zero dropout. Training used FP32 with TF32 disabled, AdamW at 1e-5, four fresh trajectories per update, leave-one-out REINFORCE, and no KL penalty. The adaptive controller used alpha=0.5; its reasoning-token penalty changed by less than 0.4% over the run.

The task asks the model to choose among four candidate repairs for synthetic Python expressions. A restricted scorer evaluates the selected expression on hidden inputs. It does not execute generated code.

Use

Load the base model at the exact revision above and attach this adapter with PEFT. Reproducing the reported evaluation also requires the prompt, restricted decoder, and scorer in the source repository. Generic text generation is outside the measured setup.

Evidence and integrity

The full result report documents the protocol, numerical qualification, audits, limitations, and compute cleanup. Adapter SHA-256: 499b1d205c1475b59d7e47d715a85ff4c899ead05c097850a0702ce611170635.

The adapter and base model use Apache 2.0. The base weights are not included.