Skill: bendusy/mlx-local-inference (version 2.2.1)

--- name: mlx-local-inference description: > Use when calling local AI on this Mac — text generation, embeddings, speech-to-text, OCR, or image understanding. LLM/VLM via oMLX gateway at localhost:8000/v1. Embedding/ASR/OCR via Python libraries (mlx-lm, mlx-vlm, mlx-audio). Works offline. Use instead of cloud APIs for privacy or low latency. metadata: { "openclaw": { "os": ["darwin"], "requires": { "anyBins": ["uv"] } } } --- # MLX Local Inference Stack Local AI inference on Apple Silicon. **oMLX** handles LLM/VLM with continuous batching. Python libraries handle Embedding/ASR/OCR directly via `uv`. ## Architecture ``` ┌─────────────────────────────────────┐ │ oMLX (localhost:8000/v1) │ │ - LLM (Qwen3.5-35B, etc.) │ │ - VLM (vision-language models) │ │ - Continuous batching + SSD cache │ └─────────────────────────────────────┘ ┌─────────────────────────────────────┐ │ Python Libraries (via uv run) │ │ - mlx-lm: Embedding │ │ - mlx-vlm: OCR (PaddleOCR-VL) │ │ - mlx-audio: ASR (Qwen3-ASR) │ └─────────────────────────────────────┘ ``` ## Models | Capability | Implementation | Model | Size | |-----------|---------------|-------|------| | 💬 LLM | oMLX API | `Qwen3.5-35B-A3B-4bit` | ~20 GB | | 👁️ VLM | oMLX API | Any mlx-vlm model | varies | | 📐 Embed | mlx-lm (uv) | `Qwen3-Embedding-0.6B-4bit-DWQ` | ~1 GB | | 🎤 ASR | mlx-audio (uv) | `Qwen3-ASR-1.7B-8bit` | ~1.5 GB | | 👁️ OCR | mlx-vlm (uv) | `PaddleOCR-VL-1.5-6bit` | ~3.3 GB | ## Usage ### LLM / Vision-Language (via oMLX API) ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="local") # Text generation resp = client.chat.completions.create( model="Qwen3.5-35B-A3B-4bit", messages=[{"role": "user", "content": "Hello"}] ) print(resp.choices[0].message.content) ``` --- ### Embeddings (via mlx-lm + uv) ```bash uv run --with mlx-lm python -c " from mlx_lm import load model, tokenizer = load('~/models/Qwen3-Embedding-0.6B-4bit-DWQ') text = 'text to embed' inputs = tokenizer(text, return_tensors='np') embeddings = model(**inputs).last_hidden_state.mean(axis=1) print(embeddings.shape) " ``` --- ### ASR — Speech-to-Text (via mlx-audio + uv) > **Important:** Must run with `--python 3.11` to avoid OpenMP threading issues (`SIGSEGV`). ```bash uv run --python 3.11 --with mlx-audio python -m mlx_audio.stt.generate \ --model ~/models/Qwen3-ASR-1.7B-8bit \ --audio "audio.wav" \ --output-path /tmp/asr_result \ --format txt \ --language zh \ --verbose ``` --- ### OCR (via mlx-vlm + uv) > **Important:** The `generate` function parameter order must be `(model, processor, prompt, image)`. ```bash cat << 'PY_EOF' > run_ocr.py import os from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template model_path = os.path.expanduser("~/models/PaddleOCR-VL-1.5-6bit") model, processor = load(model_path) prompt = apply_chat_template(processor, config=model.config, prompt="OCR:", num_images=1) output = generate(model, processor, prompt, "document.jpg", max_tokens=512, temp=0.0) print(output.text) PY_EOF uv run --python 3.11 --with mlx-vlm python run_ocr.py ``` --- ## Service Management (oMLX only) ```bash # Check running models curl http://localhost:8000/v1/models # Restart oMLX launchctl kickstart -k gui/$(id -u)/com.omlx-server ``` ## Model Storage Strategy **All models stored in `~/models/` using oMLX-compatible structure:** ``` ~/models/ ├── Qwen3-Embedding-0.6B-4bit-DWQ/ ├── Qwen3-ASR-1.7B-8bit/ ├── PaddleOCR-VL-1.5-6bit/ └── Qwen3.5-35B-A3B-4bit/ ``` ## Requirements - Apple Silicon Mac (M1/M2/M3/M4) - `uv` installed (`curl -LsSf https://astral.sh/uv/install.sh | sh`)


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)