bmax16634/sologpt-v3-150m-base

SoloLLM v3 150M Base

SoloLLM v3 150M Base is a from-scratch GPT-style decoder-only language model trained on one RTX 3090 as part of the SoloLLM project. It is a base text completion model, not an instruction-tuned chatbot.

The project goal was to build a full small-LM engineering loop: dataset construction, PyTorch model implementation, single-GPU pretraining, checkpoint recovery, evaluation, ablation, and an honest comparison against GPT-2 small.

Headline Result

The final 150M model beats GPT-2 small overall on the fixed SoloLLM v3 evaluation suite. A smaller 123M ablation also beats GPT-2 on most external checks, but it does not beat GPT-2 across every metric.

Model	Params	Train tokens	Held-out PPL	WikiText-2 PPL	LAMBADA PPL	MC avg acc norm
GPT-2 small	124.44M	public	25.32	45.32	40.62	41.05%
SoloLLM v3 123M	123.55M	9.80B	25.64	41.87	36.28	42.46%
SoloLLM v3 150M	151.87M	10.00B	24.90	41.18	35.35	42.71%

The honest claim is:

SoloLLM v3 trains GPT-2-class base LMs from scratch on one RTX 3090. The final 150M model beats GPT-2 small overall on a fixed evaluation suite, while a slightly smaller 123M model beats GPT-2 on most external benchmarks but does not fully clear the strict across-board smaller-than-GPT-2 bar.

Model Details

Item	Value
Architecture	Decoder-only GPT-style transformer
Parameters	151,868,928
Context length	1024
Tokenizer	GPT-2 tokenizer
Embedding width	768
Layers	16
Attention heads	12
Positional method	RoPE
Normalization	RMSNorm
MLP	SwiGLU
Weight tying	Input/output embeddings tied
Training hardware	Single RTX 3090
Training tokens	10,000,007,168

Training Data

The model was trained on a curated 10B-token mixture:

Source	Accepted tokens	Share
FineWeb-Edu sample-10BT	4,000,001,532	40%
DCLM baseline	2,500,001,319	25%
FineWeb sample-10BT	1,499,997,774	15%
English Wikipedia	999,998,937	10%
OpenWebText	1,000,000,972	10%

The dataset was filtered, deduplicated by normalized document hash, and packed into 1024-token training shards.

Files

File	Purpose
model.safetensors	Final model state dict
config.json	Model/training config used to instantiate SoloGPT_v2
config_resolved.json	Resolved run config from training
metrics_summary.json	Training summary for the final checkpoint
model.py	Minimal SoloGPT model implementation used by this checkpoint
configuration_sologpt.py	Hugging Face AutoConfig remote-code wrapper
modeling_sologpt.py	Hugging Face AutoModelForCausalLM remote-code wrapper
tokenizer.json	GPT-2 tokenizer used for training and inference
tokenizer_config.json	Tokenizer metadata with 1024-token context and EOS-as-pad
load_example.py	Example loading and sampling script
docs/v3_final_gpt2_comparison.md	Full final result writeup
docs/project_page.md	Short portfolio-style project page

Usage

This repo supports Hugging Face AutoModelForCausalLM loading through custom remote code. Pass trust_remote_code=True when loading the model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "bmax16634/sologpt-v3-150m-base"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True).to(device)
model.eval()

prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=40,
        do_sample=True,
        temperature=0.8,
        top_k=40,
        use_cache=False,
        remove_invalid_values=True,
        renormalize_logits=True,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

For a runnable example, see load_example.py. For low-level state-dict loading, the raw PyTorch implementation is still included as model.py.

Intended Use

This model is intended for:

• educational inspection of a small from-scratch base LM,
• text-completion experiments,
• reproducing the SoloLLM v3 evaluation story,
• portfolio/research engineering review.

It is not intended for production use, high-stakes decisions, factual QA, or chat/instruction-following use without additional tuning and safety evaluation.

Limitations

• This is a small base model, not an assistant.
• It can generate incorrect, biased, repetitive, or unsafe text.
• It has no retrieval, tool use, or instruction tuning.
• The strict smaller-than-GPT-2 across-board claim is not proven by this model; the winning 150M checkpoint is larger than GPT-2 small.
• Training data came from broad public web/text sources and may contain undesirable content despite filtering.

License

The SoloLLM code and released weights are provided under the MIT License by the author. Training data sources retain their own licenses and terms.

Project Links

• Author: Benjamin Maxwell
• Final 150M model: https://huggingface.co/bmax16634/sologpt-v3-150m-base
• Smaller 123M ablation: https://huggingface.co/bmax16634/sologpt-v3-123m-base
• Public completion demo: https://huggingface.co/spaces/bmax16634/sologpt-v3-150m-demo
• Original v1 model: https://huggingface.co/bmax16634/sologpt-base-v1
• Main result artifact in this repo: docs/v3_final_gpt2_comparison.md