AX1-Pip4.2 (ax1-pip4.2)
**A ~139M-parameter conversational language model built entirely from scratch** —
custom architecture (**AX1**), custom tokenizer (**AX0Tok**), and a hand-written
autodiff lineage. **No PyTorch/TensorFlow, no pretrained weights, nothing pulled
from another model, safe to say it's relatively sovereign.** It was pretrained on public-domain books and fine-tuned into
a small, friendly chat persona named **Pip**.
TL;DR
- Params: 138.6M (138,620,160)
- Architecture: AX1 "phasor retention" (a proprietary attention-free sequence mixer)
- Tokenizer: AX0Tok — byte-level BPE, 4,096 vocab, custom chat control tokens
- Context: 256 tokens
- Trained: from scratch on an Apple M5 GPU via MLX; runs inference on pure Python + NumPy, CPU-only (Windows / Linux / macOS, no GPU required)
- Good at: greetings, small talk, its own identity, simple jokes, light emotional support, very simple Python, honestly saying "I don't know"
- Not for: facts, math, long reasoning, or anything safety-critical — it's tiny
Model details
| Model name | AX1-Pip4.2 (product), ax0-pip-138.6m-chat (internal codename) |
| Total parameters | 138.6M |
| Architecture | AX1 (token-shift + phasor retention,16 Core GPU GeGLU FFN, weight-tied head) |
| Layers | 12 |
| Heads | 16 (memory heads θ=0 + resonator heads θ>0) |
| Model dim | 896 |
| FFN | GeGLU, ×3 (hidden 2,688) |
| Vocabulary | 4,096 |
| Context length | 256 tokens |
| Positional encoding | none (position is implicit in token-shift + phasor decay) |
| Precision | trained bf16/fp32 (MLX); released as fp32 NumPy |
| Language | English |
| License | GPL-3.0 |
The AX1 architecture (in brief)
AX1 replaces softmax attention with a per-head complex decay ("phasor")
retention: each head weights past tokens by (q·k/√d) · r^(t−s) · cos(θ·(t−s)).
Heads with θ=0 act as smooth memory channels; heads with θ>0 are
resonators that track periodic structure. Combined with a learnable per-channel
token-shift (blending each token with the previous one) and a swish-gated
output, this gives an attention-free mixer with an O(1)-state recurrent form for
streaming inference and unbounded context at generation time. There are no
positional embeddings.
Tokenizer (AX0Tok)
A custom byte-level BPE tokenizer (4,096 merges) with a small private chat protocol
built on three control tokens: <user>, <pip>, <end>. A turn is encoded as
<user> {message} <pip> {reply} <end>.
Hardware
1x M5 Pro, 48GB Unified Memory (15 Core CPU & 16 Core GPU) 1x Samsung T7 Shield (1TB) External SSD
Intended uses & limitations
Intended: a research/educational demonstration of a genuinely from-scratch small language model, and a lightweight local chatbot for casual conversation on CPU.
Out of scope: factual question answering, arithmetic, code beyond trivial snippets, multi-step reasoning, and any production or safety-critical use.
Limitations:
- At 139M params it hallucinates facts and is weak at math and reasoning.
- Short 256-token context and limited multi-turn coherence.
- English only.
- Pretraining used pre-1929 public-domain texts, so it can produce archaic or dated/biased phrasing.
- Best sampled at low temperature (~0.3–0.4).
How to use
This is not a 🤗 Transformers checkpoint — it uses a custom NumPy runtime and the AX0Tok tokenizer, so it will not load via
AutoModelForCausalLM. Inference is done with the project's own (pure-Python + NumPy) runtime.
Once the weights are uploaded, usage looks like:
pip install numpy
python serve.py --model ax0_pip.npz # web UI at http://localhost:8000
# or
python chat.py --model ax0_pip.npz # terminal chat
Chat prompt format (handled automatically by the runtime):
<user> hello <pip>
Planned files: ax0_pip.npz (weights + tokenizer), plus the minimal NumPy
inference runtime (serve.py, chat.py, infer.py, recurrent.py, ax0.py,
tokenizer.py).
Training
From scratch, two phases:
- Pretrain (language):
2,040 public-domain Project Gutenberg books (328M tokens). Held-out cross-entropy 8.0 → 3.98. - Fine-tune (persona): a chat + book-replay mix (~38% synthetic chat dialogues, including targeted capability/follow-up packs; rest is book replay to avoid catastrophic forgetting). Final chat-corpus val ≈ 0.085.
Setup: custom AX1 model in Apple MLX (M5 GPU); AdamW, cosine LR schedule, block 256, batch 32. The MLX weights are converted to a portable NumPy format for CPU-only inference (logits match to ~1e-4).
Evaluation
Evaluated by held-out cross-entropy (no standard NLP benchmarks — the model is deliberately tiny and uses a custom tokenizer, so public leaderboards aren't meaningful). Qualitatively: reliable on greetings, identity, jokes, light emotional support, and simple Python; unreliable on facts, math, and long reasoning.
Bias, risks, and limitations
Trained on old public-domain literature plus synthetic dialogue; outputs may be factually wrong, archaic, or reflect biases in 19th/early-20th-century texts. Do not rely on its outputs for factual, medical, legal, financial, or safety-critical decisions.
Author's Note
The name 'AX0' & 'AX1' were purely named after my username, ax0rz0, because it sounded cool 😇
License
Released under the GNU General Public License v3.0. Derivative works must remain GPL-3.0.
Support
If Pip is useful to you, you can help cover training compute for future versions on the cloud:
Citation
@misc{ax1pip2026,
title = {AX1-Pip: a from-scratch small language model with phasor-retention},
author = {Pip project},
year = {2026},
note = {Custom AX1 architecture and AX0Tok tokenizer; trained from scratch, no pretrained weights.}
}