LekkerYing/smolfem-360m-esp32

smolFem 360M for the ESP32-S3 N16R8

These are the files that let the ESP32-S3 N16R8 run smolFem, a small language model by Ambika (Computational Mama), fine-tuned on her feminist dataset, by streaming it layer by layer from an SD card. smolFem is a LoRA fine-tune of SmolLM2-360M-Instruct. The files here are a 4-bit copy of it, in a custom format that only the sketch in the GitHub repo can read. They do not load in transformers or Ollama.

How to wire the board, flash it and ask questions: https://github.com/lekkerYing/esp32s3-bot (guide in docs/HOWTO.md).

Files

File	Size	What it is
model.bin	224 MB	The 32 layers in 4-bit (groups of 64 weights, each with an fp16 scale and minimum), plus the word embeddings. Goes on the SD card in a folder called SMOL360.
bpe.bin	0.5 MB	The tokenizer (SmolLM2's), converted so the board can read it. Goes on the SD card in SMOL360.
head.bin	5.6 MB	The output words: the 5,748 words that occur in the dataset answers, with their rows. Written into the board's flash partition head at address 0x210000 with esptool.

How they were made

1. smolFem was compressed to 4-bit with quantization-aware distillation (qat360.py). Plain rounding to 4-bit dropped the word accuracy on the dataset from 71.8 percent to about 50 percent. After the retraining it is 71.0 percent, and it agrees with the full model on 92 percent of its words.
2. The files were written with export_sd360.py. All scripts are in the GitHub repo.

What to expect

• It answers the questions it was trained on. For other questions it makes things up.
• On the board, a question from the dataset takes about 6.5 minutes for 45 words. A question outside the dataset takes about 2 minutes per word. The SD card is the bottleneck.
• This is an experiment in how far a 5 euro chip plus an SD card can go, and in measuring how much energy one answer costs.

Credits

• smolFem and its feminist dataset: Ambika, Computational Mama
• Base model: SmolLM2 by Hugging Face, Apache 2.0
• Ideas and parts: DaveBben/esp32-llm, slvDev/esp32-ai (including the Barista model), Karpathy's llama2.c, Espressif ESP-DSP