Instructions to use Bias-variance-tradeoff/slm-125m-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Bias-variance-tradeoff/slm-125m-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Bias-variance-tradeoff/slm-125m-base")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Bias-variance-tradeoff/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("Bias-variance-tradeoff/slm-125m-base", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Bias-variance-tradeoff/slm-125m-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Bias-variance-tradeoff/slm-125m-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Bias-variance-tradeoff/slm-125m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Bias-variance-tradeoff/slm-125m-base

SGLang

How to use Bias-variance-tradeoff/slm-125m-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Bias-variance-tradeoff/slm-125m-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Bias-variance-tradeoff/slm-125m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Bias-variance-tradeoff/slm-125m-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Bias-variance-tradeoff/slm-125m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Bias-variance-tradeoff/slm-125m-base with Docker Model Runner:
```
docker model run hf.co/Bias-variance-tradeoff/slm-125m-base
```

SLM-125M-base

A 125.8M-parameter Llama-style language model pretrained from random weights on a legal + financial corpus. It is a base completer (next-token prediction only), not an instruction-tuned chatbot — give it the start of a sentence and it continues in the legal/financial register.

Built end-to-end on Modal for the Vizuara "SLM from scratch" workshop, replicating the reference pipeline at Vizuara-AI-Lab/slm-125m-from-scratch.

🕹️ Live demo: https://slm-125m-site.vercel.app
⚡ Inference API: https://dharsourav03--slm-125m-inference-web.modal.run (/generate)


Parameters	125,847,552 (~125.8M, tied embeddings)
Validation perplexity	10.87 (held-out 1% split, ~22.0M tokens)
Train tokens	2.18B (1 epoch)
Vocab / context	16,384 byte-level BPE / 1,024 tokens

Intended use

Prompt it with the opening of a sentence and it completes the thought in a legal or financial style. It is a demonstration of the full from-scratch pipeline (data → tokenizer → pretraining), not a production model.

Not intended for: factual question answering, chat, instruction following, or any high-stakes legal/financial decision-making. At 125M parameters it holds very little factual knowledge and will confidently produce plausible-sounding but incorrect content.

Model architecture

Maps 1:1 to transformers.LlamaConfig.

Component	Value
Architecture	Llama-style decoder-only transformer
Layers	12
Hidden size	768
Attention heads	12 (head dim 64), MHA (kv-heads = 12)
MLP	SwiGLU, inner dim 3072
Normalization	RMSNorm (pre-norm), eps 1e-5
Positional encoding	RoPE (theta 10000)
Context length	1024 tokens
Vocabulary	16,384 (byte-level BPE), tied embeddings
Attention bias	none

Training data

Streamed from HuggingFace, cleaned, deduplicated, and decontaminated. Realized mix (by real tokens):

Source	Content	Share
HFforLegal/case-law	US court opinions	~33%
PleIAs/SEC	SEC filings (10-K, etc.)	~39%
HuggingFaceFW/fineweb-edu	Educational web text	~28%

Pipeline: stream → deterministic rule-based cleaning (line filters, boilerplate strip, repetition/language/OCR gates) → exact + MinHash near-dedup → 13-gram decontamination against CaseHOLD / LexGLUE eval sets → 16K byte-level BPE tokenizer → pack into 1024-token windows (99/1 train/val split). Final corpus: 2.18B train + 22.0M val tokens.

Training procedure

Hyperparameter	Value
Objective	Next-token cross-entropy (causal LM)
Epochs	1 (2.18B tokens seen)
Optimizer	AdamW, betas (0.9, 0.95), weight decay 0.1
Learning rate	6e-4 → 6e-5, cosine decay
Warmup	200M tokens
Gradient clip	1.0
Global batch	524,288 tokens/step (micro-batch 32 × seq 1024, grad-accum 2)
Precision	bf16
Hardware	8×H100 (single-node DDP), ~24 min

The reference workshop trains ~5 epochs (→ val perplexity 8.50). This checkpoint is a single-epoch run (val perplexity 10.87); more epochs over the same fixed corpus lower perplexity further.

Evaluation

Held-out validation perplexity is exp(mean token-level cross-entropy) over the 1% val split (21,505 windows, ~22.0M tokens):

Validation perplexity: 10.87 (val loss 2.3856).

How to use

The model uses custom special tokens; prepending <|bos|> matches how it was served and gives the best continuations.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Bias-variance-tradeoff/slm-125m-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()

prompt = "The plaintiff alleges that the defendant"
bos = tok.convert_tokens_to_ids("<|bos|>")
eos = tok.convert_tokens_to_ids("<|eos|>")
ids = torch.tensor([[bos] + tok.encode(prompt, add_special_tokens=False)])

out = model.generate(
    ids, max_new_tokens=90, min_new_tokens=40, do_sample=True,
    temperature=0.8, top_k=50, top_p=0.95, repetition_penalty=1.3,
    eos_token_id=eos, pad_token_id=eos,
)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Limitations and bias

Base model, not aligned — no instruction tuning, no RLHF, no safety filtering.
Not a knowledge base — a 125M model stores only a few tens of MB of usable knowledge; it fabricates citations, statutes, and figures. Do not rely on any factual claim it makes.
Domain-skewed — trained ~72% on US legal + SEC text, so it defaults to that register and reflects the biases of those corpora (US-centric law, corporate finance).
English only.

Citation / attribution

Trained from scratch as part of the Vizuara AI Labs "SLM from scratch" workshop, following the pipeline in Vizuara-AI-Lab/slm-125m-from-scratch.

Downloads last month: 581

Safetensors

Model size

0.1B params

Tensor type

F32

Bias-variance-tradeoff
/

slm-125m-base