Controlled alternative architecture for Gate-B design gate
Hi Behram,
Your Gate-B evaluation notes that scaling from 110M to 336M failed to establish the preregistered capacity effect, opening the architecture design gate for controlled alternatives.
When scaling parameters on standard causal Transformers yields diminishing returns on capability breadth, changing inductive biases and layer reuse helps isolate whether depth or state mechanisms are the constraint:
Hybrid recurrence (75% GDN-2 / 25% GQA): In an open architecture I work on called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture; weights: https://huggingface.co/AndrewThompson1233/maba-101m), we interleave Gated DeltaNet associative memory with grouped query attention. This tests whether recurrent state updating captures morphological tracking better than pure softmax attention.
2-pass physical block recycling: Passing inputs through the layer stack twice with step conditioning doubles effective computational depth without expanding parameter count. If breadth requires deeper compositional transformations rather than wider static capacity, recycling isolates that variable.
Factorized embeddings: On your 32,768 vocabulary and 1024 hidden dimension, standard tied embeddings consume 32,768 * 1024 = 33.55M parameters. A rank-128 bottleneck (32k -> 128 -> 1024) reduces this to 4.33M, shifting ~29.2M parameters directly into active blocks.
Are you evaluating specific alternative recurrent or hybrid layers for the next Gate-B benchmark run?
Best,
Andrew
Hi Andrew,
Yes, this is basically the direction we are opening now. We are not treating the plain Transformer as the final Devim architecture. After the 336M result, we opened an architecture-design stage and we want to test stateful, recurrent and hybrid alternatives under matched conditions.
In parallel, we also have a separate Computational Dialectics research track where we test explicit mechanisms for relation/state revision, contradiction and reconstruction. But we have not selected or validated a final “dialectical architecture” yet. The idea is to keep these questions separate first, so we can understand whether an improvement comes from a better general neural substrate or from a Devim-specific mechanism.
This is why your Maba work is interesting to me as a possible ordinary recurrent/hybrid control. I’ll take a closer look at it.
Best,
Behram
Hi Behram,
Using Maba as a baseline recurrent/hybrid control makes methodological sense to isolate whether sub-quadratic sequence mixing helps before adding dialectical revision modules.
To keep the comparison strictly matched against your 336M baseline (24 layers, 1024 width, 32,768 vocab):
Layer allocation: An 18 GDN-2 + 6 GQA split matches your 24-layer depth while shifting 75% of sequence mixing to linear recurrence.
Parameter matching: Rank-128 factorized embeddings drop token weights from 33.55M to 4.33M. That reclaims ~29.2M parameters to widen FFN intermediate states or adjust hidden width so the model hits your exact 336,390,144 parameter ceiling.
The PyTorch modeling code and layer definitions are in maba/model.py:
https://huggingface.co/AndrewThompson1233/maba-v1-architecture
If you run into any friction adapting the GDN-2 recurrence step to your training harness, let me know.
Best,
Andrew