JacobLinCool/the-net

The Net

A 20T-scale multimodal architecture for text, audio, images, video and EEG. Modality-specific encoders feed a hierarchical sparse mixture-of-experts core with recurrent memory, and media decoders turn the shared representation back into each output modality.

Architecture

The Net has 11 major components: five modality encoders, multiscale fusion, a shared world model, three media decoders and a memory controller.

Text · Audio · Images · Video · EEG
                ↓
     Modality-specific encoders
                ↓
       Multiscale fusion
                ↓
   Hierarchical sparse MoE core ↔ Memory
                ↓
 Text head · Audio / Image / Video decoders

Encoders combine modality-specific stems with learned chunking, pooling and dechunking. Fusion aligns their representations before the shared core. The core uses attention, SwiGLU experts and RMSNorm across three widths. Residual detail paths connect hierarchy levels; recurrent memory carries state between segments.

Modality	Interface
Text	Byte-level tokens and next-token logits
Audio	Waveform frames, spectral features and codebook logits
Images	RGB patches and image latents
Video	Spatiotemporal tubelets and video latents
EEG	Multichannel signals with electrode adjacency

Model details

Field	Value
Architecture parameters	20.09T
Attention / FFN blocks	686
Core widths	8,192 / 12,288 / 16,384
Experts	128 per layer, top-2 routing
Tokenizer	Byte-level
Weights	Constant initialization (untrained)
License	MIT

Files

File	Contents
config.json	Model configuration
configuration_omni.py	OmniConfig
modeling_omni.py	OmniModel module implementation
tokenizer.json	Byte-level tokenizer
tensor_shapes.json	Architecture tensor names and dimensions

FAQ

Q: Which modalities does the architecture cover?
Text, audio, images, video and EEG, connected through a shared hierarchical core.

Q: How are the experts organized?
The shared core spans three hierarchy levels, with 128 experts per layer and top-2 routing. Multiscale fusion connects the modality encoders to the core; media decoders and a text head provide the output interfaces.

---

Release format, storage layout and parameter counting: EXPERIMENT.md