G0-nano-base

A 62M-parameter GPT trained completely from scratch on a single 8GB-RAM device (an NVIDIA Jetson) — no cloud cluster, no multi-GPU node. This is the base (pretrained, non-instruct) checkpoint: it completes text, it does not follow instructions. See G0-nano-instruct for the chat-tuned version.

The point of this model isn't to compete with billion-parameter LLMs — a 62M model is physically capped at roughly 15.5MB of factual knowledge (~2 bits/parameter, Allen-Zhu & Li, ICLR'25), regardless of training method. The point is that it was trained end-to-end, pretraining included, within an 8GB memory budget.

Architecture

Llama-style decoder-only transformer:

Parameters	62.1M (embeddings shared with LM head)
Layers	12
Hidden size	640
Attention	Grouped-Query Attention, 10 query heads / 2 KV heads, head_dim 64
Position encoding	RoPE (θ=10000)
Feed-forward	SwiGLU, hidden 1728
Normalization	RMSNorm
Context length	1024 tokens
Vocabulary	16,384 (SentencePiece, trained from scratch)
Precision	fp32

Trained on ~1.5B tokens of English web/book text (FineWeb-Edu, BookCorpus, OpenWebText, PG-19, WikiHow).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("AZERDSQ/G0-nano-base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("AZERDSQ/G0-nano-base", trust_remote_code=True)

inputs = tok("The city of Paris is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_k=50, temperature=0.8)
print(tok.decode(out[0]))

trust_remote_code=True is required — this is a custom architecture (GQA + RoPE + SwiGLU), not one of the built-in transformers model types.

Limitations

• 62M parameters caps factual knowledge hard — expect confident, fluent, frequently wrong completions on anything knowledge-dense.
• 1024-token context.
• English only.
• The custom tokenizer/model code does not handle padded batched inference — single-sequence generation only.
• Base model: raw next-token prediction, no chat formatting. It will continue text, not answer questions.

License

Apache 2.0. Weights only — this release does not include training code or data pipelines.