Veirded/SLLM-0.6 (Small-LLM) - Casual Chat Model
SLLM-0.6 (Small-LLM) is an ultra-lightweight 1.0M-parameter causal language model developed from scratch by Veirded.
The core research question behind SLLM is:
"How much conversational fluency, clean grammar, and everyday topic awareness can be packed into a 1-million-parameter budget?"
This repository hosts the SLLM-0.6 checkpoint (base pre-trained + AI-masked fine-tuned on ~238,716 curated single-turn casual chat examples), designed for fast, stateless single-turn conversation on consumer hardware.
Model Details
- Model ID:
Veirded/SLLM-0.6 - Developed by: Veirded
- Model Type: Causal Autoregressive Language Model (Decoder-Only)
- Language(s): English (
en) - License: MIT
- Tokenizer: SLLM-BPE (Lossless Byte-Level BPE, 1,152 tokens)
- Total Parameters: 1,000,832 (~1.0M, weight-tied)
- Max Turn Length: 128 tokens (Stateless single-turn, no chat memory)
Key Architectural Features
- RoPE (Rotary Position Embeddings): Real-valued rotary embeddings without complex-tensor overhead.
- QK-Norm: RMSNorm applied to Queries and Keys per head before RoPE for stable attention logits.
- SwiGLU Activations:
w2(silu(w1(x)) * w3(x))for compact parameter expressiveness. - RMSNorm & Weight Tying: Tied input/output embeddings (
tok_emb.weight = lm_head.weight). - Zero-Leakage Attention Masking: Explicit boolean causal & padding masking.
How to Download & Run
generate.py automatically detects Apple Silicon Metal (mps), NVIDIA CUDA (cuda), or CPU.
macOS (Terminal)
git clone https://huggingface.co/Veirded/SLLM-0.6
cd SLLM-0.6
python3 -m pip install torch safetensors
python3 generate.py
Linux (Bash / Ubuntu / Debian)
git clone https://huggingface.co/Veirded/SLLM-0.6
cd SLLM-0.6
python3 -m pip install torch safetensors
python3 generate.py
Windows (PowerShell or Command Prompt)
git clone https://huggingface.co/Veirded/SLLM-0.6
cd SLLM-0.6
pip install torch safetensors
python generate.py
Training Details
- Dataset: 238,716 curated, deduplicated, single-turn casual chat pairs (~10M+ tokens, ~10:1 tokens-to-parameter ratio).
- Stage 1 (Pre-Training): 12 epochs, 50,000 random examples/epoch,
BATCH_SIZE=16, AdamW (lr=1.5e-3 -> 1e-4,weight_decay=0.05,dropout=0.15),bfloat16autocast on MPS. Best Val Loss:2.1335. - Stage 2 (AI-Masked SFT): 4 epochs, 50,000 random examples/epoch,
BATCH_SIZE=32, AdamW (lr=2.0e-4 -> 1e-5, loss on AI tokens only). Best Val Loss:2.2938.
Technical Specifications
| Parameter | Value |
|---|---|
| Model ID | Veirded/SLLM-0.6 |
| Model Size | 1,000,832 Parameters (~1.0M) |
| Embedding Dimension ($d_{model}$) | 128 |
| Attention Heads | 4 (Head Dimension = 32 + QK-Norm) |
| Layers | 4 |
| SwiGLU Hidden Dimension | 384 |
| Vocabulary Size | 1,152 (Byte-Level BPE) |
| Max Turn Length | 128 tokens (Stateless single-turn, no chat memory) |
Citation
@misc{veirded2026sllm06,
author = {Veirded},
title = {SLLM-0.6: A 1M-Parameter Single-Turn Casual Chat Language Model},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Veirded/SLLM-0.6}}
}
- Downloads last month
- -