PecanAi-X1

A free, lightweight, task-focused AI model for document question-answering. Built by merging two open-source models and fine-tuning them on Q&A data — then quantized so anyone can run it on a CPU. No cloud, no API keys, no cost.

🌐 Live site: https://franklin-b.github.io/PecanAi-X1/ · License: Apache-2.0

Status: built and verified end-to-end on Windows 11 + RTX 4060 (8 GB). The fine-tuned model answers strictly from provided context and honestly refuses when the answer isn't there.

---

What it is

Property	Detail
Base	SLERP merge of TinyLlama-1.1B-Chat-v1.0 + TinyDolphin-2.8-1.1b
Fine-tune	LoRA rank 32 (bf16) on 30k SQuAD v2 examples, incl. unanswerable Qs for refusal (eval loss 1.50)
Size	1.1B params · 668 MB as Q4_K_M GGUF
Runtime	Ollama (CPU or GPU) — also a FastAPI server + web chat UI
Task	Grounded document Q&A / RAG
License	Apache 2.0

Prompt format (Zephyr-style):

<|system|>
{system}</s>
<|user|>
Context:
{context}

Question: {question}</s>
<|assistant|>

---

Quick start (use the already-built model)

# 1. The model is registered in Ollama as `pecanai-x1`
ollama run pecanai-x1 "Context: The sky is blue. Question: What color is the sky?"

# 2. Or launch the server + web UI
powershell -ExecutionPolicy Bypass -File start_pecanai.ps1

The web UI (web/index.html) lets you upload PDFs/TXT/DOCX/MD and chat with them.

---

Reproduce the build from scratch

The whole pipeline is setup → merge → fine-tune → quantize → serve. Tested on Python 3.13 + an NVIDIA GPU. Uses uv.

0. Environment

uv venv --python 3.13 .venv
# GPU torch matched to your driver. Driver <550 (CUDA <=12.2)? use cu118:
uv pip install --python .venv\Scripts\python.exe torch --index-url https://download.pytorch.org/whl/cu118
uv pip install --python .venv\Scripts\python.exe -r finetune\requirements_train.txt

1. Merge (CPU, isolated env — avoids dependency clashes)

uv venv --python 3.13 merge\.venv
uv pip install --python merge\.venv\Scripts\python.exe mergekit "pydantic==2.9.2"
merge\.venv\Scripts\python.exe merge\run_merge.py merge\merge_config.yaml merge\pecanai_x1_base

run_merge.py wraps a mergekit 0.1.4 bug (an unresolved torch forward-ref in a pydantic model) and drives the SLERP merge via the Python API.

2. Fine-tune (GPU, ~30 min on an RTX 4060)

.venv\Scripts\python.exe finetune\finetune.py `
  --base_model merge\pecanai_x1_base `
  --output finetune\pecanai_x1_ft_v2 `
  --max_samples 30000 --epochs 1 --batch_size 8 --grad_accum 2 --max_seq_len 1024 `
  --lora_r 32 --save_steps 50 --resume --merge

Produces finetune\pecanai_x1_ft_v2_merged (standalone fp16 HF model). bf16 LoRA — no bitsandbytes needed (add --load_4bit only if you're VRAM-constrained).

Crash-safe: --save_steps 50 checkpoints frequently and --resume continues from the latest checkpoint, so an interrupted run picks up where it left off (re-run the same command). Prefer a bigger, resumable run? train_v2.ps1 wraps this. Sanity check: python finetune\eval_pecanai.py finetune\pecanai_x1_ft_v2_merged

3. Quantize → Ollama

# IMPORTANT: the SentencePiece tokenizer.model must be in the model dir, or the
# GGUF tokenizer breaks (UNK byte artifacts). run_merge copies tokenizer.json only,
# so copy tokenizer.model from the base model:
copy merge\pecanai_x1_base\tokenizer.model finetune\pecanai_x1_ft_v2_merged\

powershell -ExecutionPolicy Bypass -File quantize\quantize.ps1 `
  -ModelDir finetune\pecanai_x1_ft_v2_merged -ModelName pecanai-x1 -Quant q4_K_M

Ollama imports the safetensors dir and quantizes in one step (no compiler needed).

4. Serve

uv pip install --python .venv\Scripts\python.exe -r server\requirements_server.txt
powershell -ExecutionPolicy Bypass -File start_pecanai.ps1

---

API (OpenAI-compatible)

# Upload a document
curl -X POST http://localhost:8000/v1/documents -F "file=@report.pdf"

# Ask a question
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"pecanai-x1","messages":[{"role":"user","content":"Summarize the report"}]}'

---

Publish for public download

uv pip install --python .venv\Scripts\python.exe huggingface_hub
$env:HF_TOKEN="hf_..."
.venv\Scripts\python.exe distribute\upload_to_hf.py --repo PecanAi-LLC/PecanAi-X1 --model_dir finetune\pecanai_x1_ft_merged
ollama push pecanai/pecanai-x1     # after `ollama cp pecanai-x1 pecanai/pecanai-x1`

---

Project layout

PecanAi-X1/
├── merge/        # SLERP merge (mergekit) + run_merge.py workaround
├── finetune/     # LoRA fine-tune (PEFT+TRL), eval, requirements
├── inference/    # RAG pipeline (ChromaDB + Ollama backend) + tests
├── quantize/     # Ollama import/quantize (quantize.ps1) + Modelfile
├── server/       # FastAPI OpenAI-compatible server
├── web/          # Single-file chat UI
├── distribute/   # HuggingFace uploader
└── start_pecanai.ps1

---

Notes & gotchas (things that bit us, so they won't bite you)

• Driver vs CUDA: RTX 4060 here had driver 537.28 (max CUDA 12.2), so torch cu124+ fails to init — use cu118.
• No compiler on Windows? That's fine: we avoid bitsandbytes, sentencepiece builds, and llama-cpp-python — Ollama handles GGUF conversion/quantization.
• Tokenizer: always ship tokenizer.model with the HF dir before GGUF conversion.
• mergekit 0.1.4 needs pydantic<2.10 and the run_merge.py rebuild shim.

---

Upgrade to 7B

Edit merge/merge_config.yaml: swap to mistralai/Mistral-7B-Instruct-v0.3 + teknium/OpenHermes-2.5-Mistral-7B, set layer_range: [0, 32]. ~4.1 GB GGUF, runs on 8 GB RAM.