G1-nano-base

A 60M-parameter GPT trained completely from scratch on a single 8GB-RAM NVIDIA Jetson device, without cloud infrastructure or multi-GPU setups. Base pretrained checkpoint with a native 2048-token context for raw text completion.

Overview

G1-nano-base is a small decoder-only causal language model trained end-to-end under an 8GB unified-memory constraint.

It is not a larger version of G0 Nano. The model keeps roughly the same parameter count and compute budget while reallocating capacity toward depth and a more compact attention state, enabling a native 2048-token context instead of 1024 tokens.

This is the base checkpoint. It predicts the next token and completes text; it is not a chat model and should not be expected to follow instructions.

Model variants

The instruction-tuned version of the same model is available as G1-nano-instruct.

What this version adds

Compared with the previous G0 Nano design, G1 Nano prioritizes a 2x native context length at approximately the same model size and training constraint. This base checkpoint does not include supervised instruction fine-tuning or a chat format.

Architecture

Llama-style decoder-only Transformer:

Property	Value
Parameters	60.0M, with embeddings shared with the language-model head
Layers	14
Hidden size	576
Attention	Grouped-Query Attention, 9 query heads / 1 key-value head, head dimension 64
Position encoding	RoPE, θ=10000
Feed-forward network	SwiGLU, hidden dimension 1664
Normalization	RMSNorm
Context length	2048 tokens, trained natively at this length
Vocabulary	16,384 SentencePiece tokens

Training

Pretraining data: approximately 1.5B tokens of English web and book text
Sources: FineWeb-Edu, BookCorpus, OpenWebText, PG-19 and WikiHow
Objective: causal next-token prediction
Training hardware: a single NVIDIA Jetson with 8GB of unified memory

Usage

Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AZERDSQ/G1-nano-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

inputs = tokenizer("The city of Paris is", return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    top_k=50,
    temperature=0.8,
)
print(tokenizer.decode(outputs[0]))

trust_remote_code=True is required because this repository uses a custom Transformer implementation.

Ollama

ollama run azerdsq/g1-nano-base "The city of Paris is"

This is a base model: it completes text rather than answering questions.

Limitations

60M parameters impose a hard limit on factual knowledge; expect fluent but frequently incorrect completions on knowledge-intensive prompts.
Maximum context length is 2048 tokens, which remains short compared with modern language models.
English-only training data.
Single-sequence generation only; padded batched inference is not supported by the custom model code.
No instruction tuning and no chat format.

This model should not be used for high-stakes decisions, factual verification, medical advice, legal advice or autonomous actions.

License

Apache 2.0. This release contains model weights and the code required to load them; it does not include the training data or private training infrastructure.

AZERDSQ
/

G1-nano-base