SwitLM
A Lightweight Decoder-Only Transformer Language Model Built from Scratch
- RoPE • RMSNorm • SwiGLU • GGUF*
Overview
SwitLM is a family of lightweight decoder-only Transformer language model implemented entirely from scratch using PyTorch. The project was created to explore every stage of modern LLM development, including architecture design, training, checkpointing, and conversion to the GGUF format for efficient inference.
Unlike models built on existing Hugging Face architectures, this implementation includes custom implementations of:
- Transformer Blocks
- Multi-Head Self Attention
- Rotary Position Embeddings (RoPE)
- RMSNorm
- SwiGLU Feed Forward Networks
- Causal Language Modeling
- GGUF Export Pipeline
The model is intended primarily for research, experimentation, and educational purposes.
Features
- ✅ Decoder-only Transformer
- ✅ Rotary Position Embeddings (RoPE)
- ✅ RMSNorm Normalization
- ✅ SwiGLU Feed Forward Network
- ✅ Weight Tied Embeddings
- ✅ GPT-2 Tokenizer
- ✅ Mixed Precision Training
- ✅ Gradient Accumulation
- ✅ Exported in GGUF Format
Model Architecture
flowchart TD
A[Input Text]
B[GPT-2 Tokenizer]
C[Token Embeddings]
A --> B
B --> C
subgraph Transformer["12 × Transformer Decoder Blocks"]
D1[RMSNorm]
D2[Multi-Head Self Attention]
D3[Rotary Position Embeddings]
D4[Residual Connection]
D5[RMSNorm]
D6[SwiGLU Feed Forward]
D7[Residual Connection]
D1 --> D2
D2 --> D3
D3 --> D4
D4 --> D5
D5 --> D6
D6 --> D7
end
C --> Transformer
Transformer --> E[Final RMSNorm]
E --> F[Language Modeling Head]
F --> G[Softmax]
G --> H[Next Token Prediction]
Model Architecture
SwitLM
Decoder-Only Transformer Architecture
┌──────────────────────────────────────────────────────────────────────────────┐
│ Input Text │
└───────────────────────────────┬──────────────────────────────────────────────┘
│
▼
GPT-2 Byte Pair Tokenizer
│
▼
Token & Embedding Layer
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Transformer Decoder Block × 12 │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ RMSNorm │ │
│ └──────────────────────┬─────────────────────────────────────┘ │
│ ▼ │
│ Multi-Head Self Attention (12 Heads) │
│ │ │
│ Rotary Position Embeddings (RoPE) │
│ │ │
│ Residual Connection (+) │
│ │ │
│ ┌──────────────────────▼─────────────────────────────────────┐ │
│ │ RMSNorm │ │
│ └──────────────────────┬─────────────────────────────────────┘ │
│ ▼ │
│ SwiGLU Feed Forward Network │
│ │ │
│ Residual Connection (+) │
│ │ │
└─────────────────────────────┼────────────────────────────────────────────────┘
│
▼
Final RMSNorm
│
▼
Language Modeling Head
(Weight Tied Output)
│
▼
Softmax Next Token Prediction
Decoder Block
┌──────────────────────────┐
│ Input (x) │
└────────────┬─────────────┘
│
▼
RMSNorm
│
▼
Multi-Head Self Attention
+ Rotary Position
Embeddings
│
▼
Residual Add (+)
│
▼
RMSNorm
│
▼
SwiGLU Feed Forward
│
▼
Residual Add (+)
│
▼
Output
Architecture Summary
| Component | Specification |
|---|---|
| Architecture | Decoder-only Transformer |
| Hidden Size | 768 |
| Transformer Layers | 12 |
| Attention Heads | 12 |
| Feed Forward Network | SwiGLU (3072 Hidden Units) |
| Position Encoding | Rotary Position Embeddings (RoPE) |
| Normalization | RMSNorm |
| Context Length | 2048 Tokens |
| Tokenizer | GPT-2 BPE |
| Weight Tying | Enabled |
| Output | GGUF |
Model Specifications
| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer |
| Parameters | ~100M |
| Hidden Size | 768 |
| Transformer Layers | 12 |
| Attention Heads | 12 |
| Feed Forward Size | 3072 |
| Context Length | 2048 Tokens |
| Positional Encoding | Rotary Position Embeddings (RoPE) |
| Normalization | RMSNorm |
| Activation | SwiGLU |
| Weight Tying | Yes |
| Tokenizer | GPT-2 |
| Output Format | GGUF |
Training Configuration
The model was trained entirely in PyTorch using a custom training pipeline.
Optimizer
- AdamW
Hyperparameters
| Parameter | Value |
|---|---|
| Learning Rate | 3e-4 |
| Weight Decay | 0.01 |
| Betas | (0.9, 0.95) |
| Learning Rate Scheduler | Linear Warmup + Linear Decay |
| Mixed Precision | Enabled |
| Gradient Accumulation | Enabled |
| Gradient Clipping | 1.0 |
Training Datasets
The model was trained sequentially on multiple publicly available datasets.
| Dataset | Description |
|---|---|
| WikiText | General language modeling |
| TinyStories | Story generation |
| AG News | News articles |
| IMDb Reviews | Long-form text |
All datasets were tokenized using the GPT-2 tokenizer.
Tokenizer
The model uses the GPT-2 Byte Pair Encoding (BPE) tokenizer.
- GPT-2 Vocabulary
- Byte Pair Encoding
- EOS token used as padding token
Design Choices
Rotary Position Embeddings (RoPE)
Instead of learned positional embeddings, SwitLM uses Rotary Position Embeddings to improve positional awareness and enable better handling of longer contexts.
RMSNorm
RMSNorm replaces LayerNorm to reduce computational overhead while maintaining stable training dynamics.
SwiGLU Feed Forward Network
The standard GELU feed-forward network has been replaced with SwiGLU, improving model expressiveness and training efficiency.
Weight Tying
The input embedding matrix is shared with the output language modeling head, reducing parameter count and improving generalization.
Repository Contents
SmallLLM-100M.gguf
README.md
LICENSE
Running the Model
The model is provided in GGUF format and is compatible with several inference engines.
Supported runtimes include:
- llama.cpp
- LM Studio
- KoboldCpp
- GPT4All
- Jan
- Ollama (after conversion if required)
Example:
./llama-cli \
-m SmallLLM-100M.gguf \
-p "The future of artificial intelligence is"
Intended Use
This model is suitable for:
- Educational purposes
- Transformer architecture research
- Learning how LLMs work
- Fine-tuning experiments
- Small-scale text generation
- GGUF inference experiments
Limitations
It is a relatively small language model compared to modern large language models.
Users should expect:
- Limited reasoning capability
- Limited factual knowledge
- Reduced instruction-following performance
- Hallucinations
- Lower performance on complex tasks
This model should not be used for:
- Medical advice
- Legal advice
- Financial decisions
- Safety-critical applications
Hardware
The training pipeline was designed for consumer GPUs.
Primary development hardware:
- NVIDIA RTX 3070 (8 GB VRAM)
Inference can be performed on either CPU or GPU depending on the inference backend.
Citation
If you use this model in your research, please cite:
@misc{SwitLM,
title={SwitLM-100M},
author={Avijit Paul},
year={2026},
publisher={Hugging Face},
url={https://huggingface.co/AvijitPaul/SwitLM}
}
License
This project is released under the Apache 2.0 License.
- Downloads last month
- 22
We're not able to determine the quantization variants.