Instructions to use DeependraVerma/slm-125m-ultimate-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DeependraVerma/slm-125m-ultimate-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DeependraVerma/slm-125m-ultimate-base")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DeependraVerma/slm-125m-ultimate-base")
model = AutoModelForCausalLM.from_pretrained("DeependraVerma/slm-125m-ultimate-base", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DeependraVerma/slm-125m-ultimate-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DeependraVerma/slm-125m-ultimate-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeependraVerma/slm-125m-ultimate-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/DeependraVerma/slm-125m-ultimate-base

SGLang

How to use DeependraVerma/slm-125m-ultimate-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DeependraVerma/slm-125m-ultimate-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeependraVerma/slm-125m-ultimate-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DeependraVerma/slm-125m-ultimate-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeependraVerma/slm-125m-ultimate-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use DeependraVerma/slm-125m-ultimate-base with Docker Model Runner:
```
docker model run hf.co/DeependraVerma/slm-125m-ultimate-base
```

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

slm-125m-ultimate-base

A 125-million-parameter Llama-style language model, warm-started from this project's first 125M run (DeependraVerma/slm-125m-base-family weights) and continued pretraining for a further 1.50T tokens on a larger, markup-stripped legal/financial + general corpus.

Repo / full build: github.com/DeependraVerma/legal-slm-125M
Author: Deependra Verma — Generative AI Researcher / AI Engineer (Hugging Face)
Fine-tuned Q&A version: DeependraVerma/legal-slm-125m-ultimate-sft
ONNX / in-browser version: DeependraVerma/legal-slm-125m-ultimate-sft-onnx

This is a base (completion) model, not an assistant. It continues text; it does not follow instructions or answer questions reliably. For a Q&A-capable model, use legal-slm-125m-ultimate-sft. Never use this model's output as legal, financial, or factual advice — it invents specifics (names, numbers, citations) like any base language model.

What's different from run 1 (`slm-125m-base`)

Run 1's pretraining data never stripped HTML markup from the SEC sources: the model spent roughly 19.3% of its 591B-token budget (~114B tokens) learning to emit <font> and  , and leaked that markup into 77.1% of legal completions. This run fixes that at the source (strip_markup() strips markup before tokenization, measured 36.95% → 0.00% markup on real documents) and warm-starts from run 1's weights rather than training from scratch, so the existing knowledge is kept while the markup habit is unlearned.

After warm-starting from run 1 and training on markup-stripped data, a repeated in-training probe (4 fixed legal-drafting prompts, checked for leaked EDGAR/HTML markup in the generated text) most recently measured 0% markup leakage at step 1,617,001, down from run 1's 77.1% baseline on the same class of prompt.

This model's context window is also 4096 tokens (run 1 shipped with a mismatch between its configured window and its declared max_position_embeddings; this run keeps the two equal by construction, see the training code's config for details).

Model description


Parameters	125,847,552 (~125.8M, tied embeddings)
Architecture	Llama-style decoder (maps 1:1 to `transformers.LlamaConfig`)
Layers / hidden / heads	12 / 768 / 12 (head dim 64, full multi-head attention, no GQA/MQA)
MLP	SwiGLU, intermediate size 3072
Positional encoding	RoPE, θ = 10,000
Normalization	RMSNorm, ε = 1e-05
Vocabulary	16,384, byte-level BPE (same tokenizer as every model in this project)
Context length	4,096 tokens
Embeddings	tied input/output
Precision	bf16 compute (autocast) · fp32 saved checkpoint
Final training step	2,861,021 / 2,861,022
Final training loss (last logged step)	1.9260

Intended uses & limitations

Intended use: text continuation / completion in a legal, financial, and general-English register — drafting-style prose, research and educational purposes (tokenizer design, small-model pretraining, warm-start continuation).

Not intended for: question answering, instruction following, or any use where factual accuracy matters. This is a base language model — it completes text plausibly and will fabricate case names, statute citations, dollar figures, and other specifics that sound right but are not grounded in anything. For a model that at least attempts to answer questions, see legal-slm-125m-ultimate-sft.

Training data

Same-tokenizer, markup-stripped legal/financial sources plus a much larger general-English slice than run 1. Realized token counts below are read from this run's own tokenization stamp files (real post-clean/dedup yield, not the pre-dedup budget estimate — those are known to overstate the real yield on templated legal text):

Source	HF dataset	Real tokenized tokens
US case law	`HFforLegal/case-law`	0.76B
SEC filings (PleIAs)	`PleIAs/SEC`	0.83B
Educational web (fineweb-edu sample)	`HuggingFaceFW/fineweb-edu`	0.93B
SEC material contracts	`chenghao/sec-material-contracts`	3.60B
SEC EDGAR filings (full)	`TeraflopAI/SEC-EDGAR`	14.57B
Caselaw Access Project	`common-pile/caselaw_access_project`	0.63B
Code (permissive)	`codeparrot/github-code-clean`	0.78B
Long-form books	`sedthh/gutenberg_english`	3.71B
Annealing supplement	`HuggingFaceTB/cosmopedia`	3.03B
Wikipedia (en)	`wikimedia/wikipedia`	4.20B
General web (fineweb-edu, rotating batches)	`HuggingFaceFW/fineweb-edu`	up to 32×46B budget ceiling (consumed in rotation, not fully materialized on disk at once)

Real tokenized total across fixed sources: 33.02B distinct tokens (excludes the rotating general-web batches above).

Held out of training, same as every other build in this project: coastalcph/lex_glue, casehold/casehold.

Training procedure


Initialization	warm-started from `data_125m_new/checkpoints/base` (run 1's final base weights), NOT random
Hardware	8× NVIDIA B200 GPUs, on-prem
Distribution	8-way DDP, bf16 autocast, SDPA/flash attention, `torch.compile`
Target tokens	1500B (~11,919 tokens/param)
Global batch	524,288 tokens
Optimizer	AdamW, β=(0.9, 0.95), weight decay 0.1, grad-clip 1.0
LR schedule	0.0006 → 6e-05, Warmup-Stable-Decay (96B warmup tokens)
Sequence packing	documents packed into contiguous 4096-token windows separated by `<

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/slm-125m-ultimate-base")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/slm-125m-ultimate-base", torch_dtype=torch.bfloat16
)

prompt = "The plaintiff filed a motion for summary judgment, arguing that"
inputs = tok(prompt, return_tensors="pt").input_ids
out = model.generate(
    inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
    pad_token_id=tok.convert_tokens_to_ids("<|pad|>"),
)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

For browser/edge inference (transformers.js, no server), use the fine-tuned ONNX export: DeependraVerma/legal-slm-125m-ultimate-sft-onnx.

Citation

@misc{verma2026legalslm125multimate,
  author = {Deependra Verma},
  title  = {legal-slm-125M-ultimate: A Warm-Started, Markup-Cleaned 125M-Parameter Legal and Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/slm-125m-ultimate-base},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face

License

MIT — see LICENSE in the source repo. This is a research artifact, not a source of legal or financial advice.

Downloads last month: 166

Safetensors

Model size

0.1B params

Tensor type

F32

Collection including DeependraVerma/slm-125m-ultimate-base

legal-slm — Legal & Financial Small Language Models

Collection

A from-scratch family of small, open legal/financial LLMs -- base, instruction-tuned, ONNX -- benchmarked against GPT-2, SmolLM2, Gemma-3, OLMo-2. • 8 items • Updated 2 days ago