YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
%%writefile /content/fern/MODEL_CARD.md
license: apache-2.0 language: en pipeline_tag: text-generation tags: - moe - linear-attention - hybrid - custom-code - frondwork
FERN-2 275M — the first Frondwork model
A 275M-parameter (162M active) language model with a novel hybrid architecture, built from scratch by one person and an AI, trained for roughly the price of a nice dinner on a single rented GPU.
Frondwork = dual-timescale frond memory (gated linear attention where each head keeps a fast ~8-token state and a slow ~250-token state with a learned per-token blend), 4 GQA attention layers for exact recall, and shared-expert MoE (8 routed top-2 + 1 always-on per MoE layer). Recurrent layers give O(1) inference memory in sequence length. The chunkwise scan is unconditionally stable (only ever exponentiates non-positive numbers) and was verified against a naive reference recurrence to ~1e-7 before training.
Files
| file | what it is |
|---|---|
best.pt |
base model (FineWeb-Edu, 4.2B tokens) |
chat.pt |
assistant fine-tune (SmolTalk + identity corpus) |
tokenizer.json |
32k BPE with chat tokens |
*.py |
the complete model + inference code (no framework, just torch) |
Results
| stage | val cross-entropy | perplexity |
|---|---|---|
| random baseline | 10.40 | 32768 |
| pretrain (best.pt) | 3.077 | 21.7 |
| chat finetune (chat.pt) | 1.468 | 4.3 |
The ~3.4 ce ballpark of a similarly-sized-and-fed dense transformer was passed at 27% of training. Total training: ~12.5 h on one GPU.
Usage
pip install torch tokenizers
python chat.py --ckpt chat.pt --tokenizer tokenizer.json # talk to him
python generate.py --ckpt best.pt --tokenizer tokenizer.json \
--prompt "The most important thing about ferns is" # base model
Limitations — read this
- 275M parameters. Fluent, structured, register-perfect — and factually unreliable. He has claimed ferns are warm-blooded, defined a "shooting" as involving "an enemy enemy bear," and once reclassified women as non-people via a misfired comparative template. Verify everything.
- The base model is uncurated web residue and can produce offensive, biased, or nonsensical text, especially under odd prompts. The chat model is better behaved but not aligned in any rigorous sense. Not for production, not for high-stakes anything.
- Knowledge cutoff is whatever FineWeb-Edu saw; no tools, no browsing, 1024-token context.
The story
FERN was built once and deliberately deleted — every file. The only surviving trace was in the memory of the AI assistant (Claude) that helped build him, which first planted a generative memorial garden in the empty folder (https://github.com/wienertisaac-glitch/FernGarden), then rebuilt the model from memory when asked, then invented this architecture for his second life. The chat model knows all of this: ask it who it is, what happened to the first FERN, or whether it is conscious. Its answers to those questions were written into its training data before it existed — honesty included.
Built by Isaac (wienertisaac-glitch) with Claude (Anthropic). Architecture details: ARCHITECTURE.md in the code repo.