Architecture optimization for Crowd-v1: cutting the 23% vocab tax and KV-cache overhead
Hi Vxtzq team,
Really cool initiative with CrowdGPT. Distributed community pretraining on consumer hardware is a great concept.
Looking at the Crowd-v1 architecture spec, there are two major parameter and memory bottlenecks for consumer nodes:
Vocab tax consumes 23% of your budget: A 151,669 vocab at 1,536 hidden dim ties up ~233M parameters purely in embeddings out of your 1.007B total. Almost a quarter of the model capacity is locked in static lookups instead of active sequence modeling. Factorizing the embedding layer through low-rank projections (e.g. rank 128/256) drops that footprint to under 4%, freeing ~200M parameters directly into deeper layers without changing total parameter size.
Context scaling on client GPUs: Standard dense attention caps the sequence length at 2,048 tokens to keep KV-cache from blowing up consumer VRAM.
I've been developing an open sub-quadratic architecture called Maba (101M reference, scaling specs up to 30B):
75% GDN-2 linear recurrence (O(1) constant state) + 25% GQA: cuts KV-cache by ~75% and handles long contexts on low-memory hardware.
2-pass physical block recycling: yields 2x effective layer depth from the same parameter footprint, reducing distributed sync payloads.
Native MTP (k=2) speculative heads.
The specs, implementation, and reference model are here:
https://huggingface.co/AndrewThompson1233/maba-v1-architecture
Since Crowd-v1 is at the initialized stage before full-scale training lock-in, have you considered embedding factorization or linear-recurrent hybrid blocks to stretch consumer GPU memory?
Best,
Andrew
Hello Andrew,
Your architecture seems like a good idea for future iterations of the project! While current implementation already uses GQA, and uses static Engram tables to reduce memory, i will consider using your architecture in future iterations, to enhance speed and increase size of model. In the meanwhile, feel free to contribute to the project!
Best regards,
Vxtzq
Hi Vxtzq,
Thanks for the response! Using static Engram tables alongside GQA is a clever way to handle memory constraints across distributed consumer nodes.
Glad to hear the Maba hybrid approach looks interesting for future iterations. When you start planning v2 or scaling up the parameter budget, I would be happy to help out with PRs or testing the recurrent block integration.
Best of luck with the Crowd-v1 distributed pretraining run!
Best,
Andrew