A gate on Gated DeltaNet's beta makes RL-trained Qwen3.5 reason 48% shorter
Qwen 3.8 27B thinks a lot, painfully. Length penalties in GRPO do the magic needed for that, slowly. I've started first from another direction with smaller models, Qwen3.5. I kept the penalty fixed and moved where the trainable parameters go: a small gate on the Gated DeltaNet write strength, trained next to LoRA. On Qwen3.5-4B it reasons 48% shorter than LoRA on MATH-500, and anytime accuracy goes up from 0.786 to 0.852. A LoRA with slightly more trainable parameters does not get there. Adapters, code, and every receipt are public.
Why I tried this
This started as a negative eigenvalue project. Grazzi, Siems, et al. showed that letting the delta-rule eigenvalue go negative helps linear RNNs track state. I wanted to know if you can retrofit that range into a model that is already pretrained. I tried it on a bigger Qwen first. It helped on a parity-style task and did nothing for the agent workloads I actually run. With thinking on it, the model reason lasted longer.
What stayed with me was the gate itself. It decides, per token, how hard the layer writes. That looked like the right variable for how long a model keeps going. So I asked a narrower question: does a gate on beta change what length-penalized RL learns?
The gate
Qwen3.5 is a hybrid. Most of its layers are Gated DeltaNet: a recurrent layer that writes into a state matrix at every token. How strongly it writes is one number per token and value head, beta = sigmoid(b). The same beta sets how much old content is erased and how much new content is written.
The β gate is one extra matrix W per Gated DeltaNet layer (hidden size to number of value heads, about 2 million parameters on Qwen3.5-4B). It reads the layer input and pushes beta toward its floor or its ceiling:
s = b.sigmoid() # the shipped beta, in (0, 1)
t = torch.tanh(gate_w(x)) # gate_w: nn.Linear(hidden, num_v_heads, bias=False), zero init
ceil = 1.0 # 1: Gate(0,1); 2: Gate(0,2), negative eigenvalues allowed
coef = torch.where(t >= 0, ceil - s, s)
beta = s + t * coef # stays in [0, ceil]; equals s exactly when W = 0
W starts at zero, so training starts from the shipped model. With ceiling 1, the eigenvalue 1 - beta never goes negative. With ceiling 2 it can. I trained both.
The recipe
Every arm shares one GRPO recipe: the correctness-gated length penalty of Arora and Zanette (alpha 0.2), 150 steps of 16 prompts x 8 samples from GSM8K training problems, lr 2e-5, a 512-token training cap, LoRA rank 16 on the Gated DeltaNet projections, and 3 seeds per arm. I wrote down my hypotheses before each run and froze the analysis code before each read. I read every model at thinking budgets from 256 to 8,192 tokens with the vendor's sampling settings. Anytime accuracy is the area under the accuracy-budget curve on a log2 budget axis: 1 means correct at every budget.
Results on Qwen3.5-4B
| Arm | MATH-500 anytime accuracy | MATH-500 reasoning tokens | GSM8K reasoning tokens |
|---|---|---|---|
| base model | 0.699 | 5,282 | 3,271 |
| LoRA | 0.786 | 4,223 | 1,666 |
| LoRA rank 20 (about the gate arm's parameter count) | 0.792 | 4,003 | 1,296 |
| Gate(0,1): LoRA + β gate, beta up to 1 | 0.852 | 2,202 | 297 |
| Gate(0,2): LoRA + β gate, beta up to 2 | 0.849 | 2,153 | 261 |
The gate with beta up to 1 is 48% shorter than LoRA on MATH-500 and 82% shorter on GSM8K and is more accurate on MATH-500 at every budget up to 8,192. The rank-20 LoRA barely moves. It is not the parameter count.
MATH-500 and GSM8K had been read more than once by then. So I built a fresh set and read it once: 130 problems from AIME 2024, 2025, and 2026 and AMC 2023, with two hypotheses written down first. Gate(0,1) beat LoRA by 0.110 anytime accuracy (0.285 to 0.395, 95% interval [0.089, 0.132]) and reasoned 1,329 tokens less, both on 3 of 3 seeds.
At 2B the gains are smaller and less stable. Details are in the paper.
Where the shortening lives
On a GSM8K validation slice, removing the trained gate from Gate(0,1) at inference raises the mean length from 307 to 665 tokens. The trained gate alone on the warm-up LoRA gives 970 tokens against 2,916 for the warm-up. Scaling the gate matrix from 0 to 2 shortens reasoning monotonically in all 3 seeds. The gate works at inference, and it also changes what the LoRA trained next to it learns.
In training, the gated runs learned to finish inside the cap much sooner. The LoRA runs were still hitting it at the end. So this is about how fast RL gets there in a fixed number of steps. A higher learning rate or a longer run might close part of the gap. I have not tested that yet.
Negative eigenvalues are not needed
Funny part: the gate was built to open negative eigenvalues, and that turned out not to matter here. Beta up to 2 and beta up to 1 never separate on MATH-500 or GSM8K, and on the fresh set the extra range adds 0.012 anytime accuracy. I also clamped the negative eigenvalues of Olmo-Hybrid-7B, which was pretrained with them. Held-out loss rose by 0.020 nats per token and common-word counting lost 3.8 to 6.3 points; other tasks stayed within 4 points.
What is not tested yet
One learning rate, 150 steps, one training cap, and GSM8K training prompts only. No gate-only arm, no LoRA on all linear layers, no gated arm without the length penalty, no budgets past 8,192. One model family at two sizes. These controls are next, once my GPU box is up.
What I take from it
If you fine-tune a hybrid model and want shorter reasoning, where the new parameters sit matters, not only how many. A gate on the write strength did more than the same budget of LoRA rank. It is cheap to add and starts as a no-op, so it is easy to try on your runs.
Try it
The adapters are plain PyTorch tensors for the code repository, not PEFT folders, because the gated arms need the patched Gated DeltaNet forward:
hf download Avifenesh/gdn-beta-gate-qwen3.5-4b-grpo --include "gate01/seed0/*" --local-dir adapters
git clone https://github.com/avifenesh/gdn-beta-gate && cd gdn-beta-gate
python scripts/adapters.py to-pt ../adapters/gate01/seed0/adapter.safetensors gate01_s0.pt
The model card has the full readout command (vLLM 0.31.0 with the gate plugin).
- Adapters: Hugging Face collection (Qwen3.5 2B and 4B GRPO, 4B and 9B forgetting, 4B chat SFT)
- Code, pre-registration records, receipts and a checker that ties every number to a result file: github.com/avifenesh/gdn-beta-gate
- Generation traces of every readout: doi.org/10.5281/zenodo.23340255
- Paper: "Beta Gate: Shorter RL Reasoning in Qwen3.5 Without Negative Eigenvalues", Avi Fenesh. arXiv link coming.