Srijan-Srivastava/Silia-v2

Tiny Scale Is All I Can Spare To Play With Transformer.

Srijan Srivastava
India
Srivastavavsrijan321@gmail.com
QCoreNest@gmail.com

Technical Report (Zenodo)📑
Technical Report (GitHub)📑
Technical Report (Hugging Face)📑

v2, July 23, 2026

Abstract

Introduction of the Transformer neural network architecture in the famous Attention Is All You Need paper has created a huge wave of AI development in recent years. The scaled dot-product attention allows for information to be processed with higher efficiency and quality, which the previous RNN-based models lacked. However Transformer-based models comes with their own set of challenges, particularly with parameter efficiency for tiny scale models. At such tiny scale a Transformer model essentially uses more parameter than it really should. This regime is very under-explored and for good reasons however exploring it might allow us to discover interesting insights about the Transformer. So here-in this paper I am introducing Silia, a novel neural network architecture designed for efficient modelling & classification tasks under severe parameter budget. Training on Fineweb-edu dataset shows very competitive performance on HellaSwag, PIQA and LAMBADA benchmarks, validation loss and generation quality to other models with equal or more parameters.

How To Use

1. uv sync
2. uv run inference.py --model bin/model.bin --encoder bin/o512.bin --length 1024 --temperature 0.7 --top_k 50 --text_prompt "Steve Jobs built the soul of Apple"

Model Architecture

Results

Silia was trained on Google Colab Tesla T4 GPU for ~1.5 hours of wall-clock time with the given training configurations in Table 1 and model hyperparameters in Table 2.

Model	Parameters	Vocab Size	Context Length	Layers	$d_{ff}$	$d_{model}$	$d_{head}$	$h_Q$, $h_{KV}$
Silia (mine)	524,672	512	1024	3	256	64	64	2, 2
Quark-v2	465,504	500	256	4	192	96	24	4, 4
Spark-v4	4,980,736	4096	512	6	512	256	32	8, 8
Supra-Mini-v6	1,410,688	4096	1024	6	256	128	32	4, 2

Table 2. Hyperparameters and number of parameters of Silia (mine), Quark-v2 and Spark-v4 by LH-Tech-AI, and Supra-Mini-v6 by SupraLabs. All the models used RoPE for positional embedding and Silu as the hidden activation function.

Benchmark Progression

In Table 3 you can find the detailed comparison of Silia with all models listed in Table 2 on the HellaSwag, PIQA and LAMBADA benchmarks along with the final validation loss of the models.

Benchmark	Silia (mine)	Quark-v2	Spark-v4	Supra-Mini-v6
HellaSwag (acc)	0.2804	0.2615	0.2695	0.2674
PIQA (acc)	0.5419	0.5283	0.5593	0.5403
LAMBADA (ppl)	1704	3500	588	2089
Final validation loss	2.393	2.556	3.108	3.79

Table 3. Silia is the best performing model for it's parameter size in every aspect.

Silia 0.5M performs the best in all benchmarks compared to Quark-v2 0.5M while it does lose to Spark-v4 5M in PIQA and LAMBADA benchmarks it remains in 2nd best model in both of these benchmarks.

One interesting point is that despite attaining the loss of $2.393$ by the time the model was trained on 1B tokens at step 20k Silia continued to learn more and improve, this suggests that Silia might have the capacity to perform even better than the version which is being reported in-here.

Citation

@software{Silia,
    author={Srijan Srivastava},
    title={Silia},
    url={https://github.com/SrijanSriv211/Silia},
    version={0.2.0},
    year = {2026}
}