Convergence dynamics at 6.41 loss and vocabulary gradient sparsity on RTX 3090
Hi bbkdevops,
Ambitious architectural experiment combining continuous MoE fibers with O(N) linear sequence processing and geometric manifolds on a single RTX 3090.
Looking at your training telemetry and benchmark readouts:
Starting loss was 11.46 and final loss after 2 epochs landed at 6.41.
On a 151,665 vocabulary, an initial loss of ~11.93 is random initialization (ln(151665)). A final loss of 6.41 corresponds to a perplexity of ~608, indicating the network is still early in its convergence trajectory (which aligns with the 28-38% MMLU sub-scores hovering near the 25% chance baseline).
Training across 7 curated instruction subsets means token exposure is highly sparse: the vast majority of the 151k embedding and unembedding coordinates receive little to no gradient updates over 2 epochs.
In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we tackle this convergence bottleneck using low-rank factorized embeddings:
Projecting 151,665 -> 128 -> hidden compresses the parameter footprint and forces token representations through a shared low-rank subspace.
On smaller or specialized corpora, this dramatically concentrates gradient updates per parameter, preventing optimization stalls on rare token rows.
For O(N) sequence scaling, Maba relies on Gated DeltaNet (GDN-2) linear recurrence, which maintains closed-form associative memory updates without introducing non-Euclidean optimization instability into AdamW.
Did you notice gradient vanishing or optimizer instability during the second epoch, and are you planning a longer warmup schedule to push that 6.41 loss down?
Best,
Andrew