Leo — Prompt Generation Model
Leo is a lightweight, task-specific language model built from scratch by DarkPs (a division of FanuonAI), designed exclusively for transforming short user requests into detailed, high-quality prompts.
Architecture
Leo uses a custom decoder-only transformer architecture (LeoForCausalLM) with the following components:
- RMSNorm (pre- and post-attention, ε = 1e-6)
- Grouped Query Attention (GQA) — 14 query heads, 2 KV heads
- SwiGLU MLP (gate + up + down projections, no bias on down)
- Rotary Position Embeddings (RoPE) with θ = 1,000,000
- Tied input/output embeddings
- Scores kept in float32 during attention for numerical stability
Specifications
| Property | Value |
|---|---|
| Architecture | LeoForCausalLM (custom) |
| Parameters | ~0.5B (487M) |
| Hidden Size | 896 |
| Layers | 24 |
| Attention Heads | 14 (Query) / 2 (KV) |
| Head Dim | 64 |
| Intermediate Size | 4,864 |
| Activation | SiLU (SwiGLU) |
| Max Position Embeddings | 32,768 |
| Vocab Size | 151,665 |
| RoPE θ | 1,000,000 |
| Dtype | float16 |
| EOS Token ID | 151645 (<|leo_end|>) |
| Tied Embeddings | Yes |
Auto Map
from transformers import AutoModelForCausalLM, AutoConfig
model = AutoModelForCausalLM.from_pretrained(
"DarkPs/leo-genprompt",
trust_remote_code=True
)
| Class | Module |
|---|---|
AutoConfig |
configuration_leo.LeoConfig |
AutoModel |
modeling_leo.LeoModel |
AutoModelForCausalLM |
modeling_leo.LeoForCausalLM |
Example
Input:
I want an image of a lion
Output:
A majestic lion standing in the African savanna, golden-hour sunlight,
detailed fur, powerful expression, cinematic composition, realistic
photography, natural background, dramatic lighting, high detail.
Dataset format:
{"input": "I want an image of a lion", "output": "A majestic lion standing in the African savanna, golden-hour sunlight, detailed fur, cinematic composition, realistic photography."}
Why so small?
Leo is purpose-built for one task. A focused 0.5B model trained on billions of prompt pairs outperforms a general model with a system prompt on this specific task — and runs on weak hardware with no internet or paid API required.
Model Files
Model weights are distributed via Telegram (not bundled in this repo): Download weights
About DarkPs
DarkPs is an AI organization under FanuonAI, developing open-source models including DarkIT, ICE AI, and Leo.
Website: https://dark.ps