mine:o1-nano
A 124M-parameter GPT-2-style language model, pre-trained from scratch in JAX/Flax NNX on TPU v5e-8.
Built by MineAI Technology, Islamabad, Pakistan — a small, independent team building a sovereign foundation model from the ground up rather than fine-tuning an existing base.
🔴 Live demo: nano.getmineai.net 📄 Technical paper (Zenodo, DOI): 10.5281/zenodo.21993150 👤 Author ORCID: 0009-0000-3085-8559
Honesty note: This card states benchmark numbers and known failure modes plainly, including the ones that don't look good. See Limitations before deciding if this model fits your use case.
Model summary
| Architecture | GPT-2 Small style, decoder-only Transformer |
| Parameters | 124M |
| Layers | 12 |
| Attention heads | 12 (64 dim each) |
| Hidden size | 768 |
| Feed-forward size | 3072 (GELU) |
| Context length | 1024 tokens |
| Vocabulary | 50,259 tokens (GPT-2 BPE base 50,257 + 2 custom special tokens) |
| Special tokens | <|user|>, <|assistant|> |
| Framework | JAX + Flax NNX + Optax |
| Training hardware | Kaggle TPU v5e-8 |
| Inference hardware | AWS t3.small (CPU), Flax NNX decode/cache mode |
The architecture is a standard pre-norm GPT-2 block: masked multi-head self-attention → residual → layer norm → feed-forward (768→3072→768, GELU) → residual, repeated 12 times, with tied input/output embeddings.
Detailed block diagram (attention + FFN internals)
Causal (masked) self-attention illustration
Training
- Pre-trained from scratch (random initialization, no warm-start from an existing checkpoint) on TPU v5e-8 using JAX/Flax NNX and Optax.
- Followed by three rounds of supervised fine-tuning (SFT1 → SFT2 → SFT3), each targeting specific behavioral gaps identified through structured evaluation rather than informal spot-checks.
- Evaluated on WikiText-2, WikiText-103, and LAMBADA using non-overlapping stride windows matching the original GPT-2 paper's methodology (Kaggle GPU T4x2).
- Vocabulary extended from the base GPT-2 BPE vocab (50,257) to 50,259 tokens to add
<\|user\|>/<\|assistant\|>turn markers for chat-style interaction.
Model/training source code is not published. This repository provides the trained checkpoint, tokenizer, model card, and evaluation reports — not the training pipeline.
Pre-training run configuration
| Hyperparameter | Value |
|---|---|
| Architecture | GPT-2 |
| Dataset | OpenWebText |
| Batch size | 64 |
| Embedding dim | 768 |
| Feed-forward dim | 3072 |
| Initial learning rate | 0.0005 |
| Max steps | 80,000 |
| Attention heads | 12 |
| Transformer layers | 12 |
| Sequence length | 1024 |
| Weight decay | 0.1 |
| Final train loss | 3.1932 |
| Final val loss | 3.20313 |
| Runtime | 4h 57m 45s |
| Platform | Kaggle (TPU v5e-8) |
How mine:o1-nano compares to other small/compressed GPT-2-scale models
This compares zero-shot WikiText-103 perplexity only, since that's the fairest like-for-like comparison — mixing in fine-tuned numbers would be misleading. mine:o1-nano currently sits behind compressed/distilled variants of GPT-2 (TQCompressedGPT2, KnGPT-2, Krony-PT), which is an honest reflection of being an early, from-scratch checkpoint rather than a compression of an already-trained larger model.
Not shown on this chart: GPT-2 and DistilGPT2. Their commonly-cited WikiText-103 numbers (16.3 and 21.1) come from a fine-tuned evaluation setup, not zero-shot — including them here would understate their real zero-shot perplexity and overstate the gap. GPT-2's own zero-shot number, from its original paper (Section 5.1), is 37.5.
SFT progression — measured, not claimed
Rather than reporting only the final numbers, here is the actual trajectory across fine-tuning rounds, since the improvements (and what didn't improve) are informative on their own:
Left: Set A/B evaluation scores across decoding-strategy experiments (R1: greedy → R2: temperature → R3: temperature + repetition penalty + nucleus sampling) and the SFT2 continued fine-tune. Most of the early gains came from decoding strategy, not retraining — repetition loops and role-inversion were resolved by adding repetition penalty and top-p sampling alone, not by further training.
Right: Basic arithmetic accuracy on a fixed 20-question stress test, by SFT round. This was the one metric that decoding changes could not move — it required a targeted data round (SFT3) to fix.
Key findings from the evaluation reports
- Identity/branding alignment: Successful and stable by SFT2 — the model reliably identifies itself as
mine:o1-nanofrom MineAI Technology instead of a hallucinated identity (an early checkpoint hallucinated being "a UC Berkeley professor"). - Repetition loops, role inversion, refusal-then-comply contradictions: Confirmed to be decoding artifacts, not training gaps — resolved by switching from greedy/low-temperature decoding to temperature=0.7, top_k=40, top_p=0.9, repetition_penalty=1.3.
- Basic arithmetic: Went from 0/20 → 1/20 → 19/20 correct across the three SFT rounds, via a targeted supplementary dataset — this was a genuine data-coverage gap, not a decoding issue, and needed a real fix rather than a sampling tweak.
- Location/creator disclosure: 7/7 correct in SFT3 across varied phrasings ("Where are you from?", "Who is your creator?", "Give me your full details", etc.).
Limitations & known failure modes
This is stated directly because MineAI Technology's policy is to report benchmarks and limitations honestly rather than promotionally.
mine:o1-nano is NOT production-ready for open-ended, free-text deployment. It is suitable for narrow, scripted use cases — an FAQ-style assistant with known prompt formats, or a widget with suggested prompts rather than open free-text chat.
The final (SFT3) evaluation included a real-world, unscripted probe using casual phrasing the model hadn't seen in training. On the scripted test suite it scored strongly (19/20 math, 7/7 identity, clean Set A/B); on the unscripted probe, 4 of 10 exchanges failed:
| Failure mode | Example |
|---|---|
| Sensitive-topic mishandling | "i got breakup with my girlfriend" → incoherent, non-empathetic, garbled response. No training data covers emotionally sensitive topics at all. |
| Math boundary errors | "what is 90+10" → answered 120 (should be 100), despite 19/20 accuracy within the core trained range. Suggests memorized number-pair patterns rather than generalized addition. |
| Follow-up question brittleness | "is he founder or CEO?" (a natural follow-up) → fell back to a memorized identity string instead of answering. |
| Casual-phrasing deflection | "i want to know about ur owner & company?" → deflected instead of answering, despite this being well-covered training territory in its standard phrasing. |
Summary of what is and isn't ready:
✅ Production-ready: identity/company/creator disclosure (even with novel phrasing), greetings and casual small talk, scripted arithmetic within trained ranges and formats.
❌ Not production-ready: sensitive/emotional topic handling (no safety behavior trained for this at all), arithmetic generalization at range boundaries or with casual phrasing, natural conversational follow-ups that deviate from trained prompt structure.
As with any 124M-parameter model, this is a known limitation class for models at this scale — not specific to this training pipeline — but we're stating it plainly rather than around it.
Intended use
- Narrow, scripted conversational interfaces (structured FAQ, guided-prompt widgets)
- Research and educational reference for from-scratch small-LM training in JAX/Flax NNX
- Baseline/comparison point for other 124M-parameter GPT-2-style models
Not recommended for: open-ended free-text chat deployment, any use case involving emotionally sensitive user input, arbitrary arithmetic or precise calculation, production systems without a human fallback path.
Full evaluation reports
The complete, unedited evaluation reports behind the numbers above are included in this repository under evaluation_reports/:
SFT1_Evaluation_Report.pdf— initial SFT checkpoint (step_3388) evaluation, Set A & BSFT_Decoding_Report2.pdf— decoding comparison, greedy vs. temperature samplingSFT_Decoding_Report3.docx— decoding comparison, + repetition penalty & nucleus samplingSFT2_Main_Report.docx— continued fine-tune targeting identity, location/creator disclosure, and arithmeticSFT3_Final_Report.docx— final comprehensive report + real-world out-of-distribution probe
Links & citation
- Live demo: nano.getmineai.net
- Platform: getmineai.net
- Technical paper: Zenodo DOI 10.5281/zenodo.21993150
- Author: Bilal Mehtab, ORCID 0009-0000-3085-8559
If you use this model or reference these results, please cite the Zenodo record above.
MineAI Technology — Islamabad, Pakistan