Miracle 1.x
Collection
The Miracle 1.x series produce by MRI • 2 items • Updated
Miracle 1.5 FlashNano BaseLine
Miracle FlashNano is a 108M parameter dense language model.
The model architecture is a 16-layer GQA Transformer with hidden size 768, 12 query heads, 4 key/value heads, head dimension 64, and a 1,536-dimensional SwiGLU.
Layers 8 and 16 use global causal attention; the other 14 layers use an 8,192-token causal sliding window.
Input and output embeddings are tied.
This baseline model has only undergone pre-training.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support