ankitw497/slm-125m-base

A 125.8M-parameter LLaMA-architecture language model trained from scratch on a legal/financial/web data mix.

Model

• 12 layers, 768 hidden, 12 heads (MHA), vocab 16,384, context 1024, tied embeddings
• Trained with a custom 16K byte-level BPE tokenizer (fit on the training corpus)

Data

Legal-first mix (NOT 70/20/10): case-law ~40%, SEC filings ~40%, fineweb-edu ~20%. Cleaned, deduplicated (exact + MinHash near-dup), and decontaminated against CaseHOLD/LexGLUE eval sets. See the source repo's REPLICATION_GUIDE.md for the full pipeline.

Training

• 2.04B train tokens, 1 epoch, seq_len 1024
• 1x H100, AdamW, cosine LR schedule with warmup
• Final val loss: 2.3263 (perplexity 10.24)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("ankitw497/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("ankitw497/slm-125m-base")