Text Generation
Transformers
Safetensors
GGUF
English
llama
slm
tiny
chat
raspberry-pi
conversational
text-generation-inference

Pico 1

Pico 1 is a tiny chat language model with 254,400 parameters (about 0.5 MB). It is small enough to run on any Raspberry Pi, or on almost anything else with a CPU.

It is a toy: it holds simple, friendly small talk in English, knows that it is a tiny AI, and tells you that it cannot do math or write code. It has no real knowledge and often answers off-topic. Please do not rely on it for anything.

Created by Eron. Trained from scratch with MakeAI.

Model

Parameters 254,400
Architecture Llama-style decoder (RMSNorm, SwiGLU, RoPE, tied embeddings)
Layers / hidden / MLP 3 / 64 / 128
Attention heads 2 (head dim 32)
Context 512 tokens
Vocabulary 2,048 (byte-level BPE, trained on the chat data)
Language English

Files

  • model.safetensors, config.json, tokenizer files: Hugging Face transformers format (LlamaForCausalLM).
  • pico1.gguf: GGUF (f16) for Ollama and llama.cpp.
  • Modelfile: Ollama model file with the chat format and stop tokens.

Usage

Ollama

Download pico1.gguf and Modelfile into one folder, then:

ollama create pico1 -f Modelfile
ollama run pico1

transformers

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("Convergent-ch/Pico-1")
model = AutoModelForCausalLM.from_pretrained("Convergent-ch/Pico-1")

ids = tok.apply_chat_template([{"role": "user", "content": "What is your name?"}],
                              add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=40, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
# My name's Pico 1. I'm a tiny AI made by Eron.

Chat format

<|bos|><|user|>
Hello!<|end|>
<|assistant|>

Generation stops at <|end|>. Low temperatures (0 to 0.3) work best.

Training

Trained from scratch on about 19.8M tokens of short English chats (about 108,700 conversations), 18,000 steps with batch size 32 x 512 tokens (about 15 epochs), AdamW, learning rate 3e-3 with cosine decay, bf16. Training took about 6 minutes on one RTX 3060 Ti. Final validation loss 3.00 (perplexity 20.1).

Training data:

  • TinyDialogues (MIT): only one-on-one conversations between a child and one other person, the child as the user. Generated with GPT-4 by the dataset authors.
  • Everyday Conversations (Apache 2.0): generated with Llama 3.1 70B Instruct by the dataset authors.
  • Smol-SmolTalk (Apache 2.0), first shard, only short chats without code, math or lists. Parts of it were generated with Llama 3.1 by the dataset authors.
  • About 900 identity chats about Pico 1 itself, generated locally with Qwen3 8B (Apache 2.0), repeated 4 times.

Limitations

  • At this size the model mainly learns the style of a conversation, not facts. Answers are often unrelated to the question.
  • It knows its name and creator most reliably with greedy decoding; with sampling it drifts.
  • Some replies sound like a parent or child talking, because most of the data comes from child-directed dialogues.
  • English only. Short chats only: the context is 512 tokens.
  • No safety tuning. Do not use it in products or for advice of any kind.

License

Apache 2.0.

Downloads last month
30
Safetensors
Model size
254k params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train Convergent-ch/Pico-1