Pleias-SLM-RAG — on-device RAG API

A tiny, fully self-contained Retrieval-Augmented Generation server. It wraps a small GGUF language model (Pleias Redline, a fine-tune of Baguettotron, ~300M parameters) with LanceDB full-text search behind a minimal Flask API — and runs entirely on CPU, including on a Raspberry Pi 5.

No GPU. No external API calls. No data leaves the machine. The code, the example databases, and the model weights all live in this repo, so a single clone gives you a working system.

What it does

Ask a question → the server retrieves the most relevant passages from a local database → the model reads them and streams back a grounded answer.

• On-device & offline — CPU-only inference via llama.cpp, no network at runtime.
• Streaming — answers stream token-by-token as text/plain.
• Bring your own data — a "table" is just a folder under a dataset directory; drop yours in and query it by name.
• Two endpoints — one with retrieval, one for feeding sources directly (testing).

Model & prompt format

The bundled model is Pleias Redline (Pleias-RAG.gguf), a 300M-parameter fine-tune of Baguettotron, quantized to Q8_0 (328 MB). It is a reasoning-style RAG model: it thinks inside a <think> block, then answers with inline <ref name="...">quote</ref> citations.

Prompts are built in ChatML with the sources inlined:

<|im_start|>user
{query}

**source_1**
{source_text}

**source_2**
{source_text}
<|im_end|>
<|im_start|>assistant
<think>

Example

Given a question and two sources, the model streams back a grounded, cited answer:

Query

<|im_start|>user
Can the Great Wall be seen from space?

**source_1**
The claim it is visible from space is a myth; the wall is at most 9.1 m wide.

**source_2**
Astronaut Yang Liwei said he could not see it from orbit.
<|im_end|>
<|im_start|>assistant
<think>

Output — the model's reasoning (truncated) then the cited answer

### 1. Query decomposition
User asks: "Can the Great Wall be seen from space?" → Simple factual question about visibility from orbital perspective. Direct information retrieval task.
...
</think>

The Great Wall of China is not visible from space. The claim that the wall is
visible from orbit is considered a myth<ref name="source_1">The claim it is
visible from space is a myth; the wall is at most 9.1 m wide.</ref>. Furthermore,
astronaut Yang Liwei stated that he could not see the Great Wall from orbit<ref
name="source_2">Astronaut Yang Liwei said he could not see it from orbit.</ref>.

Deploy

Option A — Docker (recommended)

docker pull crosash/pleias_slm_rag
docker run -p 8081:8081 crosash/pleias_slm_rag

The image bundles the model and example data — pull and use. It targets ARM64 (Raspberry Pi 5, Apple Silicon). To build your own image (Windows), see the Dockerfile.

Option B — local Python

git lfs install                                    # the model is stored via Git LFS
git clone https://huggingface.co/PleIAs/Pleias-SLM-RAG
cd Pleias-SLM-RAG
pip install flask lancedb pandas llama-cpp-python
python -m src.main                                 # serves on http://0.0.0.0:8081

Ready when you see Model loaded successfully. Ready for requests.

API

Both endpoints stream the model's raw output as text/plain. Use curl -N to watch tokens arrive live.

POST /ask_stream — retrieve, then answer

Field	Required	Description
query	yes	The user's question.
table	no	Which table (folder in the dataset) to search. Defaults to the server's table. Examples: en, fr, both.

curl -N -X POST http://localhost:8081/ask_stream \
  -H "Content-Type: application/json" \
  -d '{"query": "What is CRSV?", "table": "en"}'

POST /raw_query — no retrieval (testing)

You supply the sources directly; they are formatted into the prompt exactly as retrieved sources would be. Handy for probing the model on controlled inputs.

Field	Required	Description
query	yes	The user's question.
sources	yes	List of sources, each a string or an object {"text": "..."}.
show_prompt	no	If true, streams the exact formatted prompt (delimited) before the output.

curl -N -X POST http://localhost:8081/raw_query \
  -H "Content-Type: application/json" \
  -d '{
        "query": "Can the Great Wall be seen from space?",
        "sources": [
          "The claim it is visible from space is a myth; the wall is at most 9.1 m wide.",
          "Astronaut Yang Liwei said he could not see it from orbit."
        ]
      }'

Bring your own data

A dataset is a directory of tables; a table is a folder inside it holding a LanceDB dataset. The server opens the table named crsv if present, otherwise the first dataset in the folder. Point the server at your own dataset with --dataset <dir> (default: data), then select a table per request with the table field. Each table needs:

1. a text column (its contents become the source body shown to the model), and
2. a full-text-search index (search uses query_type="fts").

import lancedb

db = lancedb.connect("mydataset/faq")          # dataset "mydataset", table "faq"
tbl = db.create_table("crsv", data=[
    {"text": "First source passage ...", "url": "https://...", "lang": "en"},
    {"text": "Second source passage ...", "url": "https://...", "lang": "en"},
])
tbl.create_fts_index("text")                    # required for full-text search

Run it with python -m src.main --dataset mydataset -t faq, then query {"query": "...", "table": "faq"}. Extra columns (e.g. url, lang) are carried along as source metadata. Table names are restricted to letters, digits, _ and -.

Configuration

Server flags (python -m src.main --help):

Flag	Default	Description
-d, --dataset	data	Dataset directory holding the table folders. Point at your own to serve your own tables.
-t, --table-name	both	Default table used when a request omits table.
-p, --port	8081	Port to bind.
--host	0.0.0.0	Host to bind.
--debug	off	Verbose logging.

Generation defaults (in src/inference.py): temperature=0.1, repetition_penalty=1.0, top_p=0.95, max_new_tokens=2048, context 4096, search_limit=3 sources per query. CPU threads are set in src/generation.py (n_threads=4).

Data & attribution

The example database (en, fr, both) is built from the Redline project, a corpus of international-law sources on conflict-related sexual violence (CRSV). For more information, see the original dataset: https://huggingface.co/datasets/PleIAs/BSF_Redline

The model is by PleIAs.