Srijan-Srivastava/Strawberry-s1

Tiny Scale Is All I Can Spare To Play With Transformer.

Srijan Srivastava
India
Srivastavavsrijan321@gmail.com
QCoreNest@gmail.com

Technical Report (Zenodo)📑
Technical Report (GitHub)📑
Technical Report (Hugging Face)📑

v1, June 2026

How to Generate

Example prompt:

Which animal has more poison - the salamander that sticks out its bone or the frog with the sharp head thing, and how do they both make their enemies hurt?

Use uv for inferencing. Install torch, numpy, regex and colorama.

uv run inference.py -i 117M_fp32/final.bin -e cl16k.bin -T "Which animal has more poison - the salamander that sticks out its bone or the frog with the sharp head thing, and how do they both make their enemies hurt?"

Generated output:

 I dont understand why they all work together.

### 1. Query Decomposition

"pouring your animal's survival" → food safety concern
"all the animals" → dual danger threshold
"potential danger" → threshold question, not just threshold

● High confidence: food safety, environmental impact, food safety

User seeking: food safety considerations, policy implications, environmental interactions

### 2. Domain Contextualization

Food safety (food safety): food safety, food safety, health needs
"all the animals" → food safety threshold, environmental impact
"all the animals" → food safety threshold, food safety concerns
"did it work together" → safety safety requirements, environmental conditions

Key domains needed:
- food safety physiology (flight, feeding, food safety)
- environmental risk stratification
- environmental factors
- environmental interactions

### 3. Information State Assessment

● High confidence

Abstract

Introduction of the Transformer neural network architecture in the famous Attention Is All You Need paper has created a huge wave of AI development in recent years. The scaled dot-product attention allows for information to be processed with higher efficiency and quality, which the previous RNN-based models lacked. However Transformer-based models comes with their own challenges, particularly with parameter efficiency for tiny models with parameters ≤ 5M. At such small scale a Transformer model essentially uses more parameter than it really should. This sub-ten-million parameters domain space is very underexplored and for good reasons but I wanted to explore it anyways. So here-in this paper I am introducing Silia, a novel transformer architecture designed for efficient modelling & classification tasks under severe parameter budget. Training against GPT-2 architecture (Andrej Karpathy's nanoGPT project) with same "base" hyperparameters, training data and compute budget, Silia achieves comparable loss and generation quality with significantly less parameters.

Introduction

The dominant trend in Transformer-based language models has been scaling: larger models, more data, more compute in pretraining and reinforcement learning for chain-of-thought based reasoning capabilities to consistently yield better performance. This means that the smallest practical models lie anywhere between 1B-20B parameters, such as Qwen, Gemma & GPT-OSS. Despite them being called "small", these models are still billions in parameters and are trained on trillions of tokens. This trajectory, as useful as it is, has widened the gap between general-purpose frontier research experiments and task-specific small research experiments.

So why not just scale down standard Transformer models? The reason is that Transformer is a very parameter heavy architecture and as much those parameters help at larger scales, most of those parameters are just wasted at tiny scale such as sub-ten-million (≤ 5M) parameters. Now there are several ways to make Transformers parameter efficient, such as tied embedding weights, factorized linear layers, smaller expansion in feed-forward networks and more, but these aren't enough.

This is where the Silia neural network architecture enters the chat. Silia merges the Attention layer with the FFN layer to save lots of parameters while preserving much of the original performance. This new Attention-merged-FFN layer is what I call Silia or Silu in Attention as I took SwiGLU as the base FFN layer.

The Intuition

So how I ended up with this idea? Why do I think replacing linear layers in SwiGLU with Attention layers was a good idea?

The reason is that we already know that Attention is mostly a linear transformation over our hidden state, but the catch is that it's a "smarter" linear transformation.

We can observe in standard Transformer, attention layer modifies the hidden state then FFN processes that "smartly" modified hidden state. However an attention-only Transformer perform incredibly poorly (not as bad a linear-layer-only-no-non-linearity neural network) due to the lack of a "strong" activation function. Attention only has softmax as it's activation function but softmax only decides which tokens should attend to which other tokens. This makes softmax a not so "strong" activation function.

Attention is dynamic and smart about which information to mix, but it has no strong non-linearity to actually transform that information. SwiGLU has the strong non-linearity but it's static. Same weights for every input.

As per my observation SwiGLU at small scales (less parameters and specially context windows) can approximate what attention does (token mixing and information routing). However it fails at longer contexts because it has no positional awareness or dynamic weighting.

So instead of running both separately and wasting parameters on overlapping functionality, Silia replaces the static linear matrices in SwiGLU with attention getting dynamic mixing and strong non-linearity in one unified operation.

Acknowledgements

This model used compute sponsored by Tomi Yang. I also thank Tomi Yang for all the helpful discussions!

Architecture Design

Input tokens
    |
[Token Embedding]
    |
[Silia Block xN:]
    |--- Multi-Headed Attention
    |    |--- Rotary Positional Embeddings
    |    |--- QK Norm
    |    |--- Scaled Dot Product Attention
    |--- Silu activation function
    |--- Multi-Headed Attention
    |--- Attention Residuals
[Output Projection (weight-tied)]
    |
Next token logits

Citation

@software{Silia,
    author={Srijan Srivastava},
    title={Silia},
    url={https://github.com/SrijanSriv211/Silia},
    version={0.1.0},
    year = {2026}
}