Hush-Nano

Hush-Nano is a 22M decoder-only small language model trained from scratch on an 8.5B token corpus.

This is a base model, not an instruction-tuned chat model.

Model Details

Hush-Nano has the following features:

• Type: Causal Language Models
• Training Stage: Pretraining
• Architecture: transformers with RMSNorm, RoPE, SwiGLU, QK-Norm and tied word embeddings
• Number of Parameters: 22M (22,621,056)
• Number of Layers: 12
• Number of Attention Heads (GQA): 6 for Q and 3 for KV
• Context Length: 1,024

It is not recommended to use this model for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.

Training Data

The model was trained on an 8.5B (8,554,042,292) token subset of the following subsets:

Source	Training tokens	Share
FineWeb-Edu	4,539,286,619	53.07%
DCLM	2,890,209,171	33.79%
FineMath4plus	1,124,546,502	13.15%

Evaluation

Zero-shot normalized accuracy, evaluated in fp32 using EleutherAI/lm-evaluation-harness. Scores may vary slightly with the evaluation setup and environment.

	PIQA	ARC-Easy	ARC-Challenge	HellaSwag
Hush-Nano	58.27%	38.93%	21.84%	28.89%

Usage

This model includes custom Transformers code and so requires trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Soulitude/Hush-Nano"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device).eval()

prompt = "Every effort makes you"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
).to(device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.95,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Intended Use and Limitations

Hush-Nano is intended for small language model research, text continuation, and evaluation of pretrained model behavior. It has not been instruction-tuned and may not follow conversational instructions reliably. Its configured maximum context length is 1,024 tokens.

License

Apache 2.0