Duoia/duogpt-40m-v1

DuoGPT-40M (Gen 1)

A 38.85M-parameter English small language model trained from scratch on a single consumer GPU (RTX 4060 Laptop 8GB).

Parameters	38,854,144 (bf16 ≈ 78MB)
Architecture	Llama-style: 11 layers / d512 / 8 heads / SwiGLU / RoPE / RMSNorm, no bias, tied embeddings
Vocabulary	8192 (custom byte-level BPE)
Context	512 tokens
Pretraining	745M tokens, 3.37h, val ppl 4.96
SFT	334K samples (summarization + write-a-story-by-words), 2.07h, val ppl 2.99

GGUF / Ollama Usage

This repo also ships GGUF weights (converted directly from model/lit_model.pth, template kept identical to training), ready to import into Ollama:

# Pull a GGUF from HF (pick a quantization level)
curl -L -o duogpt-40m.gguf \
  https://huggingface.co/Duoia/duogpt-40m-v1/resolve/main/duogpt-40m-v1-q4_k_m.gguf

# Import with the Modelfile in this repo
ollama create duogpt-40m -f Modelfile
ollama run duogpt-40m "Where is Mary?"

File	Size	Notes
duogpt-40m-v1-q4_k_m.gguf	36 MB	Recommended for daily use (Q4_K_M quantized)
duogpt-40m-v1-f16.gguf	83 MB	Unquantized, best quality
Modelfile	-	Ollama import config (template matches training format)

The GGUF build behaves like the litgpt version's /q /s /t commands: the template bakes in the <|user|> wrapper, so ollama run duogpt-40m "<question>" is equivalent to the old /q command.

Quick Start

pip install -r requirements.txt        # mainly litgpt==0.5.13 + torch

bash scripts/chat.sh                   # interactive chat (loads the SFT model by default)
bash scripts/ask.sh "Summarize the following story in one sentence."   # one-shot ask

If python3 is not the interpreter with the installed dependencies, specify it with PYTHON=/path/to/python bash scripts/chat.sh.

The three question modes in scripts/chat.sh:

/q <question>             bare question
/s <story> | <question>   reading comprehension with a short story
/t <instruction> | <input>  any task (summarize, write a story, etc.)

Contents

model/             SFT version (recommended for daily use; writes stories / summarizes / simple Q&A)
src/duogpt_prompt.py   required chat-template class (litgpt looks it up on load; PYTHONPATH is set by the scripts)
scripts/chat.sh    interactive entry (auto-points at the model in this package)
scripts/ask.sh     one-shot ask entry
scripts/chat.py    implementation of chat.sh (load once, ask repeatedly)
MODEL_CARD.md      full model card: architecture/param distribution, data composition, training hyperparams, eval numbers

Usage Examples

# Explicit tasks (what the model is best at)
bash scripts/ask.sh "Write a short story for young children using these words: moon, basket, tiny"
bash scripts/ask.sh "Summarize the following story in one or two sentences.
Once upon a time, there was a little girl named Lily. She found a shiny red ball."

# Switch model / tune params
TEMP=0.3 MAXTOK=40 bash scripts/chat.sh

Directly in Python:

from litgpt import LLM
llm = LLM.load("model")                      # auto-reads the package's prompt_style
print(llm.generate("Summarize the following story in one sentence.\nOnce upon a time...",
                   max_new_tokens=60, temperature=0.7))

Known Limitations (important, avoid misjudging)

• Only does assigned tasks: give it a bare question (e.g. What is a volcano?) and it won't answer; you must spell out the task explicitly as above.
• No knowledge: the training corpus is children's stories only; factual questions get made up.
• 512-token context cap: prompt + generated length combined; exceeding it raises Cannot forward sequence of length T.
• Drifts as it writes: a common issue at 39M scale — later parts of long text lose coherence (entity mixing, repetition).
• English-only, children's-level vocabulary.

Data Sources & License

• Pretraining corpus: TinyStoriesV2-GPT4 (CDLA-Sharing-1.0) and Children-Stories (user-provided, includes GPT-generated content).
• Model weights in this package were trained from scratch on the above data for research/personal use; if redistributing, please keep this notice and MODEL_CARD.md.