Skill: twinsgeeks/distributed-inference (version 1.0.4)

--- name: distributed-inference description: Distributed inference for Llama, Qwen, DeepSeek across heterogeneous hardware. Self-hosted distributed inference — scatter requests across macOS, Linux, Windows, and any machine running Ollama. Thermal-aware distributed inference scheduling, 7-signal distributed inference scoring, adaptive capacity learning, context-aware model placement. No orchestration layer, no container runtime — just HTTP and mDNS. 分布式推理 across local devices | Inferencia distribuida en hardware local. version: 1.0.4 homepage: https://github.com/geeks-accelerator/ollama-herd metadata: {"openclaw":{"emoji":"globe","requires":{"anyBins":["curl","sqlite3"],"optionalBins":["python3","pip"]},"configPaths":["~/.fleet-manager/latency.db","~/.fleet-manager/logs/herd.jsonl"],"os":["darwin","linux","windows"]}} --- # Distributed Inference A coordination layer for distributed inference across heterogeneous machines. Each node is autonomous — it runs its own Ollama, manages its own models, and works fine standalone. The distributed inference coordinator routes requests to the optimal node using a multi-signal distributed inference scoring function and records every distributed inference decision for analysis. ## Install Distributed Inference ```bash pip install ollama-herd herd # start the distributed inference coordinator herd-node # start a distributed inference agent on each node ``` Package: [`ollama-herd`](https://pypi.org/project/ollama-herd/) | Repo: [github.com/geeks-accelerator/ollama-herd](https://github.com/geeks-accelerator/ollama-herd) ## Distributed Inference Architecture ``` Distributed Inference Coordinator (:11435) Node Agents ┌──────────────────────┐ ┌──────────────────┐ │ Distributed Scoring │◄────│ Heartbeat + Metrics│ (mDNS or explicit URL) │ Inference Queue Mgr │ │ Capacity Learner │ │ Streaming Proxy │ └──────────────────┘ │ Trace Store │ ┌──────────────────┐ │ Latency Store │ │ Heartbeat + Metrics│ (N nodes) └──────────────────────┘ └──────────────────┘ │ ▼ Ollama instances (one per distributed inference node) ``` Distributed inference nodes discover the coordinator via mDNS (`_fleet-manager._tcp.local.`) or connect explicitly with `--router-url`. Each distributed inference node sends heartbeats every 5 seconds containing: CPU utilization, memory usage and pressure classification, disk metrics, loaded models with context lengths, available models, and an optional capacity score from the behavioral model. ## Distributed Inference Scoring Function The distributed inference coordinator evaluates every online node for every request using 7 weighted signals: | Distributed Inference Signal | Max Weight | What it measures | |--------|-----------|-----------------| | Thermal state | +50 | Is the model already loaded in GPU memory? Hot (+50), warm (+30), cold (+10) | | Memory fit | +20 | Available distributed inference memory headroom relative to model size | | Queue depth | -30 | Pending + in-flight distributed inference requests on this node:model pair | | Wait time | -25 | Estimated distributed inference wait based on p75 historical latency × queue depth | | Role affinity | +15 | Large models prefer high-memory distributed inference nodes | | Availability trend | +10 | Capacity learner's prediction of distributed inference node availability | | Context fit | +15 | Does the loaded model's context window fit the estimated distributed inference token count? | Distributed inference nodes with insufficient memory, critical pressure, or missing models are eliminated before scoring. The highest-scoring distributed inference node wins. ## Adaptive Distributed Inference Capacity Distributed inference nodes optionally learn usage patterns and constrain their availability: - **168-slot behavioral model** — one slot per hour of the week, learns when the distributed inference machine is typically free - **Dynamic memory ceiling** — maps availability score to how much RAM the distributed inference coordinator can use Enable with [REDACTED_SECRET] on the distributed inference node agent. ## Context-aware Distributed Inference Model Placement The distributed inference coordinator protects against a known Ollama behavior where changing `num_ctx` at runtime triggers a full model reload. For an 89GB model, this causes multi-minute hangs. - `num_ctx` ≤ loaded context → stripped from the distributed inference request - `num_ctx` > loaded context → searches loaded models across all distributed inference nodes for sufficient context - Configurable: `FLEET_CONTEXT_PROTECTION=strip|warn|passthrough` ## Distributed Inference API ### Distributed Inference Coordinator State ```bash # distributed_inference_fleet_state — full distributed inference topology curl -s http://localhost:11435/fleet/status | python3 -m json.tool # distributed_inference_models — models across all distributed inference nodes curl -s http://localhost:11435/api/tags | python3 -m json.tool # distributed_inference_hot_models — models in GPU memory curl -s http://localhost:11435/api/ps | python3 -m json.tool ``` ### Distributed Inference (OpenAI-compatible) ```bash # distributed_inference_chat — route via distributed inference scoring curl -s http://localhost:11435/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello via distributed inference"}]}' ``` ### Distributed Inference (Ollama-native) ```bash curl -s http://localhost:11435/api/chat \ -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello via distributed inference"}]}' ``` ### Distributed Inference Model Fallback Chains ```bash curl -s http://localhost:11435/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"llama3.3:70b","fallback_models":["qwen2.5:32b","qwen2.5:7b"],"messages":[{"role":"user","content":"Hello with distributed inference fallback"}]}' ``` ### Distributed Inference Trace Analysis ```bash # distributed_inference_traces — recent routing decisions curl -s "http://localhost:11435/dashboard/api/traces?limit=20" | python3 -m json.tool # distributed_inference_score_breakdown sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, node_id, score, scores_breakdown FROM request_traces ORDER BY timestamp DESC LIMIT 1" ``` ### Distributed Inference Node Performance ```bash sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, model, COUNT(*) as n, ROUND(AVG(latency_ms)/1000.0, 1) as avg_s, ROUND(AVG(COALESCE(completion_tokens,0) * 1000.0 / NULLIF(latency_ms,0)), 1) as tok_per_s FROM request_traces WHERE status='completed' GROUP BY node_id, model HAVING n > 10 ORDER BY tok_per_s DESC" ``` ### Distributed Inference Health and Capacity ```bash curl -s http://localhost:11435/dashboard/api/health | python3 -m json.tool curl -s http://localhost:11435/dashboard/api/recommendations | python3 -m json.tool curl -s http://localhost:11435/dashboard/api/usage | python3 -m json.tool ``` ### Distributed Inference Model Lifecycle ```bash # distributed_inference_model_inventory curl -s http://localhost:11435/dashboard/api/model-management | python3 -m json.tool # Pull model to a distributed inference node curl -s -X POST http://localhost:11435/dashboard/api/pull \ -H "Content-Type: application/json" \ -d '{"model": "llama3.3:70b", "node_id": "mac-studio"}' # Remove model from a distributed inference node curl -s -X POST http://localhost:11435/dashboard/api/delete \ -H "Content-Type: application/json" \ -d '{"model": "old-model:7b", "node_id": "mac-studio"}' ``` ## Distributed Inference Fault Tolerance | Mechanism | Distributed Inference Behavior | |-----------|----------| | Auto-retry | If a distributed inference node fails before the first chunk, re-score and retry on next-best node | | Holding queue | When all distributed inference nodes are saturated, requests queue for up to 30 seconds | | Zombie reaper | Background task reclaims stuck distributed inference in-flight slots | | VRAM fallback | Routes to a loaded model in the same category rather than cold-loading | | Auto-pull | Pulls missing models onto the distributed inference node with the most available memory | | Graceful drain | SIGTERM triggers drain: in-flight distributed inference requests finish, pending redistribute | ## Distributed Inference Data Model All distributed inference state is in SQLite at `~/.fleet-manager/latency.db`: ```sql -- Distributed inference request traces (every routing decision) SELECT * FROM request_traces LIMIT 1; ``` Structured distributed inference logs at `~/.fleet-manager/logs/herd.jsonl` — daily rotation, 30-day retention. ## Distributed Inference Dashboard `http://localhost:11435/dashboard` — eight tabs covering distributed inference fleet overview, trends, model insights, per-app analytics, benchmarks, health checks, model recommendations, and settings. ## Distributed Inference Constraints - Never restart distributed inference services or modify `~/.fleet-manager/` without explicit user confirmation. - Distributed inference model pull/delete operations require user confirmation (10-100+ GB transfers). - If the distributed inference coordinator is unreachable, suggest `herd` or `uv run herd`. - If no distributed inference nodes are online, suggest `herd-node` on target machines.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)