FSI_Edge: From-Scratch Novel Architecture Coding Model
A tiny but capable code generation model trained from scratch on ARM CPU, with a novel DNA-inspired architecture.
Architecture
- Helix Memory — DNA helix-inspired curved memory for O(log L) context scaling
- HCA (Hybrid Concentrated Attention) — 3-tier code attention (local + structural + global)
- EA-FFN (Execution-Augmented FFN) — learns execution traces
- RoPE-S — RoPE with structural bias for code structure
- PPN (Prefix-Preserving Norm) — stabilizes deep training
- MoD (Mixture-of-Depths) — dynamic routing to save compute
Training Stages
- Stage 1 — Pretraining (next-token prediction on code + NLP)
- Stage 1b — FIM (Fill-in-Middle code infilling)
- Stage 2 — SFT (Supervised Fine-Tuning)
- Stage 2b — Cold-Start Reasoning (chain-of-thought)
- Stage 3 — MCPO RL (Monte Carlo Policy Optimization)
- Stage 4 — DPO (Direct Preference Optimization)
- Stage 5 — Long-Context Extension
Quick Start
# Clone from HuggingFace
git clone https://huggingface.co/FerrellSyntheticIntelligence/FSI-Edge
cd FSI-Edge
pip install -r requirements.txt
# Train on CPU
python training/run_cpu.py --model-size 4K --steps 1000
# Resume training from checkpoint (step 4132)
python training/run_cpu.py --model-size 4K --steps 10000 \
--resume checkpoints/cpu_ckpt_004132.pt --lr 2e-4
# Or on Colab T4 GPU (100x faster)
# Upload scripts/fsi_edge_colab.ipynb to Google Colab
Checkpoints
checkpoints/ contains trained checkpoints from ARM CPU training:
cpu_best.pt— best model weights (19MB)cpu_latest.pt— latest model weights (19MB)cpu_ckpt_004132.pt— full training state (52MB, step 4132)
Tokenizer
Trained BPE tokenizer (32K vocab) at fsi_edge_tokenizer/.
Results
| Steps | Best Loss | Platform |
|---|---|---|
| 0 | 10.44 | ARM CPU |
| 1000 | ~6.0 | ARM CPU |
| 2000 | ~1.0 | ARM CPU |
| 4132 | 0.70 | ARM CPU |
Colab Training
Open scripts/fsi_edge_colab.ipynb in Google Colab with T4 GPU for 100x faster training.
Mission
Train a from-scratch novel architecture model. Each step proves the architecture. The code is production-ready for cloud GPU scaling (H100s).