G2-nano-base

A 60.03M-parameter decoder-only language model trained from scratch on a single 8GB NVIDIA Jetson Orin Nano with twice the tokens of G1-nano. Gains on short benchmarks are small. This is the base checkpoint: it completes text.

Overview

G2-nano-base is a 60.03M-parameter decoder-only causal language model trained from scratch under an 8GB unified-memory budget (Jetson Orin Nano). Architecture is the same as G1-nano (v2-14L, GQA 9/1, 2048 context). Pretraining is a new run from scratch, not a resume of G0 or G1.

Public reading: 2× tokens, weak quality lift, likely a 60M ceiling on this shape. Not an upgrade over G1-nano. The ~1B G2 run is a different model and is not part of this release.

This is the base checkpoint. It predicts the next token and completes text; it is not a chat model and should not be expected to follow instructions.

For chat, use G1-nano-instruct.

Model variants

The instruction-tuned version of the same model is available as G2-nano-instruct. That SFT run lowers the 7-task mean versus this base checkpoint.

What this version adds

About 3.00B training tokens (~50 tokens/parameter) versus ~1.50B for G1-nano. On the canonical 6-task mean this base scores 44.59% versus 43.03% for G1-nano-base (+1.56 pt). PIQA, WinoGrande and ARC-Challenge stay within noise. This is a lab note, not a product win.

This checkpoint does not include supervised instruction fine-tuning or a chat format.

Architecture

Llama-style decoder-only transformer, identical to G1-nano. Attention is full (no sliding window) at the trained length. Embeddings are tied with the language-modeling head.

Property	Value
Parameters	60.03M
Layers	14
Hidden size	576
Attention	GQA, 9 query heads / 1 KV head, head_dim 64
Position encoding	RoPE (θ=10000)
Feed-forward network	SwiGLU, hidden 1664
Normalization	RMSNorm
Context length	2048 tokens
Vocabulary	16,384 (SentencePiece BPE, shared)

Training

• Pretraining data: 3,001,842,523 training tokens
• Sources: FineWeb-Edu 66% / OpenWebText 15% / PG-19 7.5% / Wikipedia EN 5% / BookCorpus 5% / WikiHow 1.5% (capped)
• Objective: causal next-token prediction
• Training hardware: NVIDIA Jetson Orin Nano (8GB unified memory), 22,903 steps

Usage

Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AZERDSQ/G2-nano-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

inputs = tokenizer("The city of Paris is", return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    top_k=50,
    temperature=0.8,
)
print(tokenizer.decode(outputs[0]))

trust_remote_code=True is required because this repository uses a custom Transformer implementation.

Ollama

ollama run azerdsq/g2-nano-base "The city of Paris is"

This is a base model: it completes text rather than answering questions.

Limitations

• 60M parameters cap factual retention. Completions can be fluent and wrong.
• Doubling the token budget did not produce a decisive quality jump on this architecture.
• 2048-token context.
• English only.
• Single-sequence generation only (no padded batched inference).
• No instruction tuning and no chat format.

This model should not be used for high-stakes decisions, factual verification, medical advice, legal advice or autonomous actions.

License

Apache 2.0.

Open weights: model weights, tokenizer and inference code (trust_remote_code). Training code, data pipelines and intermediate checkpoints are not included.

Links

• Ollama: G2-nano-base
• Instruction-tuned counterpart: G2-nano-instruct
• Hugging Face org