Veirded/SLLM-0.6 (Small-LLM) - Casual Chat Model

SLLM-0.6 (Small-LLM) is an ultra-lightweight 1.0M-parameter causal language model developed from scratch by Veirded.

The core research question behind SLLM is:

"How much conversational fluency, clean grammar, and everyday topic awareness can be packed into a 1-million-parameter budget?"

This repository hosts the SLLM-0.6 checkpoint (base pre-trained + AI-masked fine-tuned on ~238,716 curated single-turn casual chat examples), designed for fast, stateless single-turn conversation on consumer hardware.


Model Details

  • Model ID: Veirded/SLLM-0.6
  • Developed by: Veirded
  • Model Type: Causal Autoregressive Language Model (Decoder-Only)
  • Language(s): English (en)
  • License: MIT
  • Tokenizer: SLLM-BPE (Lossless Byte-Level BPE, 1,152 tokens)
  • Total Parameters: 1,000,832 (~1.0M, weight-tied)
  • Max Turn Length: 128 tokens (Stateless single-turn, no chat memory)

Key Architectural Features

  1. RoPE (Rotary Position Embeddings): Real-valued rotary embeddings without complex-tensor overhead.
  2. QK-Norm: RMSNorm applied to Queries and Keys per head before RoPE for stable attention logits.
  3. SwiGLU Activations: w2(silu(w1(x)) * w3(x)) for compact parameter expressiveness.
  4. RMSNorm & Weight Tying: Tied input/output embeddings (tok_emb.weight = lm_head.weight).
  5. Zero-Leakage Attention Masking: Explicit boolean causal & padding masking.

How to Download & Run

generate.py automatically detects Apple Silicon Metal (mps), NVIDIA CUDA (cuda), or CPU.

macOS (Terminal)

git clone https://huggingface.co/Veirded/SLLM-0.6
cd SLLM-0.6
python3 -m pip install torch safetensors
python3 generate.py

Linux (Bash / Ubuntu / Debian)

git clone https://huggingface.co/Veirded/SLLM-0.6
cd SLLM-0.6
python3 -m pip install torch safetensors
python3 generate.py

Windows (PowerShell or Command Prompt)

git clone https://huggingface.co/Veirded/SLLM-0.6
cd SLLM-0.6
pip install torch safetensors
python generate.py

Training Details

  • Dataset: 238,716 curated, deduplicated, single-turn casual chat pairs (~10M+ tokens, ~10:1 tokens-to-parameter ratio).
  • Stage 1 (Pre-Training): 12 epochs, 50,000 random examples/epoch, BATCH_SIZE=16, AdamW (lr=1.5e-3 -> 1e-4, weight_decay=0.05, dropout=0.15), bfloat16 autocast on MPS. Best Val Loss: 2.1335.
  • Stage 2 (AI-Masked SFT): 4 epochs, 50,000 random examples/epoch, BATCH_SIZE=32, AdamW (lr=2.0e-4 -> 1e-5, loss on AI tokens only). Best Val Loss: 2.2938.

Technical Specifications

Parameter Value
Model ID Veirded/SLLM-0.6
Model Size 1,000,832 Parameters (~1.0M)
Embedding Dimension ($d_{model}$) 128
Attention Heads 4 (Head Dimension = 32 + QK-Norm)
Layers 4
SwiGLU Hidden Dimension 384
Vocabulary Size 1,152 (Byte-Level BPE)
Max Turn Length 128 tokens (Stateless single-turn, no chat memory)

Citation

@misc{veirded2026sllm06,
  author = {Veirded},
  title = {SLLM-0.6: A 1M-Parameter Single-Turn Casual Chat Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Veirded/SLLM-0.6}}
}
Downloads last month
-
Safetensors
Model size
1M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ 2 Ask for provider support