altslate

/

JugnuLM-110M-R2plus

Like 1

Follow

AltSlate Labs 2

Hi AltSlate team,

Scaling the R2 recipe to 25.2B tokens on dual RTX PRO 4500 Blackwell GPUs, tracking the ablation ladder from R0 through R3 on the Tiny-ML Leaderboard, and publishing the full methodology on arXiv is great open science. The empirical finding in R1 (+2.2 on ARC-Easy with ResFormer value residuals) and the honest drop in R3 with DCLM provide very clear signals on data curation and deep-thin dynamics.

Looking at your deep-thin geometry (23 layers x 576 hidden size) alongside the learned vr_lambda weights:

1. The representation grounding signal in vr_lambda = 0.48:
   Following MobileLLM's depth-over-width thesis gives 110M models great non-linear capacity, but 23 layers at hidden dimension 576 pushes standard softmax attention toward severe representation collapse in later layers.
   The fact that learned vr_lambda scalars settled at a mean of ~0.48 shows that upper attention heads lean heavily on layer 0's value vectors to retain lexical grounding. Without that vertical anchor, token representations drift into diffuse entropy, which explains the 6-point drop when the pathway is dropped.

2. The 25.8% vocabulary tax on deep-thin allocation:
   Using the SmolLM2 49,152 tokenizer at hidden dimension 576 means your tied embedding matrix consumes 28,311,552 parameters.
   Out of the 109.7M total parameter budget, that single static lookup table accounts for 25.8% of the entire model.
   At hidden dimension 576, a full transformer block costs roughly 3.54M parameters. Your static embedding weights equal the parameter budget of 8 complete transformer layers.
   Applying a two-stage low-rank factorized projection (49,152 -> 128 -> 576 = ~6.9M params) reclaims over 21.4M parameters. That reclaimed budget could fund widening the hidden dimension to 704 or 768 within the exact same 110M ceiling, giving each attention head higher representational rank.

3. Horizontal state grounding vs vertical value shortcuts:
   In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore sub-150M efficiency through a 3:1 macro-topology paired with native NoPE:
   We route 75% of depth through Decoupled Gated Delta Attention (DGDA) paired with MABA-SA sparse attention.
   Instead of relying on vertical value residuals (skipping from layer 0 into upper layers) to prevent attention collapse, DGDA maintains an associative recurrent state updated via an error-correcting delta rule.
   This horizontal delta update continuously subtracts redundant features and prevents state saturation across sequence steps, keeping representations sharp while dropping KV-cache footprint by nearly 40x and ensuring flat O(1) decode latency.

If you are planning an R5 exploration or testing architectural variations for the Tiny-ML Leaderboard, checking out the parameter allocation layout and hybrid state coordination in the Maba v2 repo might provide some useful structural points for deep-thin configurations.

Did the vr_lambda coefficients scale monotonically across the 23 layers, or did mid-depth semantic blocks show a distinct peak in v_0 dependency?

Best,
Andrew