Panag7/neuromamba-30m-fineweb

NeuroMamba: Brain-Inspired Language Model

A novel architecture combining selective state space models, spiking neurons, neural memory banks, and predictive coding — designed to generalize like the human brain while fitting in 8GB RAM.

Architecture

Component	Brain Analog	Technical Approach
Selective SSM Core	Thalamic gating	Mamba-style selective state space (linear time)
Spiking Activation	Neuron firing	Surrogate-gradient spiking neurons
Neural Memory Bank	Hippocampus	Differentiable memory with content-based read/write
Bio-MoE	Cortical columns	Sparse mixture-of-experts with bio-inspired routing
Predictive Coding	Top-down predictions	Auxiliary loss for next-state prediction

Quick Start

# Test the model
python3 test_model.py

# Train on TinyStories (small dataset for testing)
bash run_training.sh

# Train with custom config
python3 train_hf.py \
    --dataset roneneldan/TinyStories \
    --d_model 512 \
    --n_layers 8 \
    --batch_size 4 \
    --num_train_epochs 3

Model Sizes

Config	Parameters	Memory (fp16)	Notes
Tiny (d=256, l=6)	~25M	~50MB	Fast iteration
Small (d=512, l=8)	~130M	~260MB	8GB RAM fits easily
Medium (d=768, l=12)	~350M	~700MB	Strong baseline

Key Features

• Linear complexity: No quadratic attention — scales to long sequences
• Sparse computation: Spiking neurons + MoE = most parameters inactive
• Structured memory: Neural memory bank enables systematic generalization
• Predictive coding: Auxiliary loss improves representation learning

Citations

Based on:

• Mamba (Gu & Dao, 2023) — selective state spaces
• Griffin (De et al., 2024) — gated linear recurrences
• SpikeGPT (Chu et al., 2023) — spiking language models
• Neural Turing Machines (Graves et al., 2014) — differentiable memory