Madushan996/NARC_Tiny_v1.0

NARC Tiny v1.0

A 124M-parameter chat language model, trained from scratch on a hobby budget.

NARC Tiny v1.0 is a modern (Llama-style) decoder-only Transformer, built and trained end-to-end from scratch in plain PyTorch by Madushan Bhashana Dissanayake — no training frameworks, no distillation, no fine-tuning of an existing model. It was pretrained on 8.4B tokens of quality-filtered educational web text and instruction-tuned into a conversational assistant, all on rented cloud GPUs for roughly $52 total.

It is a learning and research artifact: quality is around GPT-2 (124M) class — fluent, coherent English with unreliable facts. It runs locally on CPU.

⚠️ Expectation-setting: a model this size sounds like an assistant but confidently invents facts, names, and details. It is not a knowledge base or a production assistant. See Limitations.

---

Quickstart

pip install torch safetensors tiktoken

import torch, tiktoken
from modeling_narc import NarcTiny

IM_START, IM_END = 50257, 50258
model = NarcTiny.from_pretrained("model.safetensors", "config.json")
enc = tiktoken.get_encoding("gpt2")

ids  = [IM_START] + enc.encode("user\nWhat is the capital of France?") + [IM_END] + enc.encode("\n")
ids += [IM_START] + enc.encode("assistant\n")
out = model.generate(torch.tensor([ids]), max_new_tokens=128,
                     temperature=0.7, top_k=50, rep_penalty=1.3, eos_token=IM_END)
print(enc.decode([t for t in out[0, len(ids):].tolist() if t < 50257]).strip())
# -> "The capital of the French nation is Paris."

A runnable inference_example.py is included. For a full graphical experience, see the desktop app.

Run with Ollama

ollama run hf.co/Madushan996/NARC_Tiny_v1.0

Pulls narc-tiny-v1.0.gguf directly from this repo and chats using its embedded ChatML template — no Python, no PyTorch, just Ollama.

Prefer a named local model instead? Download narc-tiny-v1.0.gguf and Modelfile from this repo, then:

ollama create narc-tiny -f Modelfile
ollama run narc-tiny

The GGUF is F16 (RMSNorm weights kept F32, per GGUF convention) and was verified to reproduce the original PyTorch model's logits (100% argmax agreement, differences only at fp16-rounding level) before publishing.

---

Model architecture

Decoder-only autoregressive Transformer, Llama-style (not GPT-2 style):

Component	Specification
Parameters	123,587,328 (embedding and LM head are weight-tied)
Layers / heads / d_model	12 / 12 / 768 (head_dim 64)
Context length	1,024 tokens
Positions	RoPE (rotary, θ=10,000) on Q/K — no learned position table
Normalization	RMSNorm, pre-norm, float32 internals
MLP	SwiGLU (gated), hidden ≈ 2,048 (8/3 × d_model)
Attention	Causal multi-head via FlashAttention (scaled_dot_product_attention); GQA-ready
Tokenizer	GPT-2 BPE (50,257), vocab padded to 50,304; 2 ids reused as ChatML specials
Other	No bias terms; dropout 0; GPT-2 scaled residual init

RoPE convention: the weights use the rotate-half (Llama/GPT-NeoX) layout — pairing dim i with dim i + D/2. The interleaved (GPT-J) convention is not compatible. The included modeling_narc.py implements it correctly.

Full details in docs/NARC-Tiny-v1.0-Architecture.pdf.

Chat template

ChatML, with two special tokens (<|im_start|> = 50257, <|im_end|> = 50258):

<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
The capital of France is Paris.<|im_end|>

Generate from <|im_start|>assistant\n and stop at <|im_end|>. A system turn (<|im_start|>system\n…) is supported but followed only loosely at this scale.

---

Training

Built from scratch in ~7 PyTorch files; trained on Modal.com.

Stage 1 — Pretraining (val loss 3.07)

• Data: 8.4B tokens of FineWeb-Edu (sample-10BT), Common Crawl filtered by an educational-quality classifier — the single highest-leverage quality decision.
• Objective: next-token prediction, cross-entropy.
• Schedule: 16,000 steps × 524,288 tokens/step, AdamW (β 0.9/0.95, wd 0.1 on matrices only), grad-clip 1.0, cosine LR 6e-4→6e-5 with 700-step warmup, bf16, DDP across 8×H100.
• Throughput: 3.17M tok/s (30% MFU). Trained in two phases (7,000 then resumed to 16,000 steps).
• Result: val loss 11.01 → 3.07 (perplexity ≈ 21.5).

Stage 2 — Instruction tuning / SFT (val loss 1.75)

• Data: ~105M tokens of SmolTalk, rendered through the ChatML template.
• Loss masking: loss computed on assistant tokens only (user/scaffolding targets set to ignore_index), so the model learns to answer, not to imitate users.
• Schedule: 800 steps, warm-started from the base checkpoint with a fresh optimizer, LR 3e-5→3e-6 cosine, 100-step warmup, on a single A10G.
• Result: val loss 1.75.

Full process and the engineering record (eight real incidents and their fixes — a compiler pin, a RoPE-convention bug, throughput reality vs. documentation, and more) are in docs/NARC-Tiny-v1.0-Technical-Documentation.pdf.

---

Evaluation

Benchmarks (loglikelihood scoring, no generation; the base model is the conventional reference point):

Benchmark	Metric	Base (val 3.07)	Chat (val 1.75)	GPT-2 124M	Random
HellaSwag (10,042 ex)	acc_norm	30.21%	29.81%	~29–31%	25%
ARC-Easy (2,376 ex)	acc	52.65%	48.48%	~43–44%	~25%

NARC Tiny is on par with GPT-2 124M on HellaSwag and exceeds it on ARC-Easy — a science-QA benchmark where the model's FineWeb-Edu (educational) pretraining data is an advantage over GPT-2's general-web training. acc_norm is the standard HellaSwag metric; acc is standard for ARC. Fully reproducible with the included benchmark_modal.py, eval_hellaswag.py, and eval_arc.py.

Quantitative loss: val loss 3.07 (base) / 1.75 (chat). The v1→v2 pretraining extension (3.7B→8.4B tokens) is visible qualitatively on identical prompts:

"Write a paragraph about Sri Lanka" — early model: "Hello, Sri Lanka! My name is Sri Lanka, and I'm a unique and beautiful person…" (fluent filler, no content) — this model: "The island of Lanka is home to a rich history, including ancient temples… inhabited for more than 10,000 years by the indigenous Sinhalese tribes…" (topical; fine details still invented)

Capabilities

• Fluent, grammatical English; coherent paragraphs
• Assistant-style turn-taking with structured (list/markdown) answers
• Topical content on common subjects; learned stop behavior
• Runs on CPU (~0.5 GB weights)

Limitations

• Facts are unreliable and confidently invented — 124M parameters hold limited world knowledge (for scale, SmolLM2-135M saw ~2T tokens; NARC Tiny saw 8.4B, ~240× fewer)
• Weak multi-step reasoning and arithmetic
• Follows system prompts only loosely; no stable identity
• 1,024-token context; English only
• No safety/alignment tuning beyond SFT data curation — may produce incorrect, biased, or inappropriate text. Use responsibly; not for production or high-stakes use.

---

Cost & compute

Phase	Hardware	Cost
Data prep (3 tokenizations)	CPU	~$3
Pretraining + failed attempts	8×H100	~$46
SFT (both rounds) + eval	1×A10G	~$3
Total		~$52

Useful training compute ≈ 6.9 × 10¹⁸ FLOPs (6ND).

Files in this repo

File	What
model.safetensors	Model weights (494 MB, tied head deduplicated)
config.json	Architecture config
modeling_narc.py	Self-contained model + loader + generate()
inference_example.py	Minimal load-and-chat script
narc-tiny-v1.0.gguf	GGUF weights for llama.cpp / Ollama (F16, 237 MB)
Modelfile	Ollama Modelfile (ChatML template + sampling defaults)
NARC-Tiny-v1.0-chat.pt	Weights-only PyTorch checkpoint (for the desktop app)
app/	Source of the NARC Tiny desktop chat app (Electron + Python)
docs/	Technical documentation + architecture PDFs
RELEASE_NOTES.md	Version history

Desktop chat app

A local, private chat GUI (Electron front-end + Python inference backend) is included under app/ (source). To run it:

cd app
npm install
pip install -r requirements.txt   # torch, tiktoken
npm start                          # or run START_NARC_TINY.bat on Windows

Point it at model.safetensors or NARC-Tiny-v1.0-chat.pt (from this repo) via the in-app model picker — the app supports both formats and prefers safetensors. A prebuilt Windows installer is available on request / from the project's other release channels.

---

Citation

@misc{dissanayake2026narctiny,
  title  = {NARC Tiny v1.0: A 124M chat language model trained from scratch},
  author = {Dissanayake, Madushan Bhashana},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Madushan996/NARC_Tiny_v1.0}}
}

License

Apache-2.0. Trained on FineWeb-Edu (ODC-By) and SmolTalk (Apache-2.0); please observe those dataset licenses. Model outputs are provided without warranty.

---

Built from scratch as a learning project — pretraining builds capability, instruction tuning shapes behavior. Every frontier model is this same loop with more zeros.