SoloLLM-1.0-152M-Base

SoloLLM-1.0-152M-Base is a from-scratch decoder-only base language model from the SoloLLM project. It is trained for text completion, not instruction following or chat.

Best completed SoloLLM 1.0 base checkpoint; beats GPT-2 small across the fixed project eval suite, while using more parameters than GPT-2 small.

Results

Metric	SoloLLM-1.0-152M-Base	GPT-2 small
Parameters	151,868,928	124,439,808
Train tokens	9,800,572,928	public
Project held-out PPL	24.30	25.32
WikiText-2 PPL	41.75	45.32
LAMBADA PPL	34.06	40.62
LAMBADA last-token acc	47.27%	46.67%
LAMBADA last-word acc	33.05%	32.60%
MC avg normalized acc	43.61%	41.05%

Model Details

Item	Value
Architecture	Decoder-only GPT-style transformer
Parameters	151,868,928
Context length	1024
Tokenizer	GPT-2 tokenizer
Embedding width	768
Layers	16
Attention heads	12
Positional method	RoPE
Normalization	RMSNorm
MLP	SwiGLU
Dropout	0.0
Weight tying	Input/output embeddings tied
Training hardware	Single RTX 3090

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "bmax16634/solollm-1.0-152m-base"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

Original Format

The original/ folder contains the raw PyTorch state dict and sanitized training metadata from the SoloLLM training run. The root-level model.safetensors file is the preferred Hugging Face/Transformers format.

Limitations

This is a base model. It is not instruction-tuned, RLHF-tuned, safety-tuned, or optimized for chat behavior.