Tunisian AI Assistant — Web Knowledge API

A $0, self-hostable web-knowledge API for AI assistants. It supports Arabic, French, English and Tunisian (derija) queries.

SearXNG finds results across many search engines (self-hosted).
Crawl4AI opens pages and returns clean, LLM-ready Markdown.
FastAPI exposes /search, /crawl, and /research.

Production hardening included out of the box:

Concern	How it is handled
SSRF / internal access	DNS-resolves every target and blocks private, loopback, link-local, reserved and cloud-metadata addresses
Auth	Constant-time `X-API-Key` check; the server fails closed if no key is configured
Performance	Concurrent crawling with a bounded semaphore
Caching	In-process TTL cache for search and crawl results (no Redis needed)
Rate limiting	Per-client limits via `slowapi`
Resilience	Exponential-backoff retries for SearXNG and crawling
Observability	Structured JSON logs to stdout

Requirements

Docker Desktop / Docker Engine with the Compose plugin.
~2 GB free RAM (the crawler runs a headless Chromium).

1. Configure

cd diy-tavily
cp .env.example .env

Edit .env and set a strong API_KEY (the API refuses every request until you do):

# Linux/macOS
sed -i "s/^API_KEY=.*/API_KEY=$(openssl rand -hex 32)/" .env

Also open searxng/settings.yml and replace secret_key with a long random string (openssl rand -hex 32).

2. Run with Docker Compose

docker compose up --build -d

The API waits for SearXNG to become healthy before starting.
API: http://localhost:8000 — interactive docs at http://localhost:8000/docs
SearXNG: http://localhost:8080

Check health and logs:

curl http://localhost:8000/health
docker compose logs -f api

Stop:

docker compose down

run.sh is a convenience wrapper that copies .env (if missing) and runs the compose build.

Test it

# Search (auto language detection)
curl "http://localhost:8000/search?q=آخر+أخبار+تونس&limit=3" \
  -H "X-API-Key: YOUR_KEY"

# Force a language: ar | fr | en | tn (Tunisian derija -> Arabic engines)
curl "http://localhost:8000/search?q=chnowa+el+ta9s+fi+tounes&lang=tn" \
  -H "X-API-Key: YOUR_KEY"

# Crawl one page to Markdown
curl -X POST "http://localhost:8000/crawl" \
  -H "Content-Type: application/json" -H "X-API-Key: YOUR_KEY" \
  -d '{"url":"https://example.com","max_chars":20000}'

# Research: search + crawl top results + grounded context
curl "http://localhost:8000/research?q=Tunisia+renewable+energy&limit=3&lang=en" \
  -H "X-API-Key: YOUR_KEY"

API contract

`GET /health`

Liveness/readiness. No auth. Returns crawler readiness and supported languages.

`GET /search?q=...&limit=5&lang=auto`

Returns candidate links + snippets. lang ∈ auto | ar | fr | en | tn.

`POST /crawl`

{"url":"https://example.com/article","max_chars":20000}

Returns { "url", "title", "markdown", "success", "error" }. Crawl failures return success:false with an error message rather than an HTTP error.

`GET /research?q=...&limit=3&lang=auto`

Runs search, crawls the top results concurrently, and returns:

{"query":"...","language":"ar","answer_context":"...","sources":[...]}

Pass answer_context to your LLM and instruct it to cite the included source URLs and to say when the sources do not support an answer. This API retrieves evidence; it does not generate an answer itself.

`GET /stats`

Cache statistics (auth required).

Configuration reference

All settings come from environment variables / .env (see .env.example):

Variable	Default	Purpose
`API_KEY`	(empty → all requests refused)	Required shared secret
`SEARXNG_URL`	`http://localhost:8080`	SearXNG base URL (compose sets `http://searxng:8080`)
`MAX_RESULTS`	`5`	Hard cap on returned search results
`MAX_CRAWL_CHARS`	`20000`	Max Markdown chars per page
`REQUEST_TIMEOUT_SECONDS`	`30`	Upstream + page timeout
`CRAWL_CONCURRENCY`	`3`	Max simultaneous page crawls
`MAX_RETRIES`	`2`	Extra attempts on failure
`CACHE_TTL_SECONDS`	`900`	Cache lifetime (0 disables)
`CACHE_MAX_ITEMS`	`512`	Max cached entries
`RATE_LIMIT`	`30/minute`	Per-client rate limit
`DEFAULT_LANGUAGE`	`auto`	Default query language
`CORS_ORIGINS`	`*`	Comma-separated allowed origins
`LOG_LEVEL`	`INFO`	Log verbosity
`ALLOW_PRIVATE_HOSTS`	`false`	Testing escape hatch — keep `false` in prod

Language support

Arabic (ar) and French (fr) are passed straight to SearXNG.
English (en) is the default fallback.
Tunisian derija / Tounsi (tn, derija, tounsi, darija) maps to Arabic engines while keeping your exact query text (works for both Arabic-script and Latin "arabizi" like chnowa el ta9s).
lang=auto (default) detects Arabic script → ar, French diacritics → fr, otherwise → en.

Local development (without Docker)

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements-dev.txt
crawl4ai-setup            # installs the headless browser
export API_KEY=dev-key    # Windows: set API_KEY=dev-key
# Point at a running SearXNG (e.g. docker compose up searxng)
export SEARXNG_URL=http://localhost:8080
uvicorn app.main:app --reload

Run the tests

pip install -r requirements-dev.txt
pytest

The test suite covers SSRF URL validation, language detection/mapping, the TTL cache, and the API endpoints (with SearXNG and the crawler mocked, so no network or browser is needed).

Connecting from Colab

A local API is not reachable from Colab without a tunnel. Run the API in Colab itself, or expose it with Cloudflare Tunnel / ngrok (subject to their terms). Never expose the API without setting API_KEY. See colab_demo.py for a minimal client.

Notes on the $0 budget

Everything here is free open-source software and runs on your own machine. A continuously online public API still needs a host — run it locally, on a free Colab session for experiments, or on a free-tier host (which may sleep or have memory limits). Do not rely on public SearXNG instances for production; run your own as above.

Respect website terms, robots.txt, copyright, rate limits, and authentication boundaries. Crawl4AI is not a license to bypass anti-bot systems or private content.

Project layout

app/
  main.py            FastAPI app: endpoints, rate limiting, lifespan
  config.py          Typed settings (pydantic-settings)
  logging_config.py  Structured JSON logging
  security.py        SSRF-safe URL validation + constant-time auth
  languages.py       Arabic/French/English/Tunisian handling
  cache.py           Async TTL cache
  searxng.py         SearXNG client (retries + cache)
  crawler.py         Crawl4AI manager (concurrency + retries + cache)
  models.py          Pydantic request/response models
tests/               Pytest suite
Dockerfile           API image (non-root, healthcheck, Crawl4AI browser deps)
docker-compose.yml   API + private SearXNG with healthchecks
searxng/settings.yml SearXNG configuration

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support