Local AI runtime recommendations — platform: linux
Beginner pick: Ollama. One-line install, simple single-user UX.
Beginner command: curl -fsSL https://ollama.com/install.sh | sh && ollama run llama3.1:8b
Power-user pick: vLLM. Highest throughput for serving many requests; PagedAttention + continuous batching.
Power-user command: pip install vllm && vllm serve meta-llama/Llama-3.1-8B-Instruct
Runtimes:
name: vLLM
what: OpenAI-compatible serving engine
status: stable
name: Ollama
what: CLI/server
status: stable
name: llama.cpp
what: Reference engine
status: stable
Gotcha: vLLM shines for multi-user serving/throughput. For a single local chat, Ollama or llama.cpp is simpler and lighter.
Ceiling: Scales with VRAM; multi-GPU for 70B+ at higher precision.