Skill: jimmyclanker/gateway-health-monitor (version 1.0.0)

--- name: gateway-health-monitor version: 1.0.0 description: Monitor and auto-fix OpenClaw gateway stability issues. Diagnoses launchd throttling, plugin restart loops, hung shutdowns, and macOS power management interference. Use when the gateway keeps disconnecting, restarting, or staying down for long periods. Triggers on "gateway down", "gateway keeps restarting", "disconnected", "gateway unstable", "launchd throttling", "gateway won't start". --- # Gateway Health Monitor Diagnose and fix OpenClaw gateway stability issues on macOS. Covers the most common failure modes that cause extended downtime. ## Quick Diagnosis Run the diagnostic script: ```bash bash scripts/diagnose.sh ``` This checks: process state, launchd classification, restart count, plist config, power management, and plugin resolve loops. ## Common Failure Modes ### 1. Plugin Restart Loop (most common) **Symptoms**: Gateway restarts every 5-7 minutes. Log shows `restartReason=config.patch` with `plugins.installs.*.resolvedAt`. **Cause**: Plugins re-resolve on every boot → write new timestamps to `openclaw.json` → config watcher detects "change" → triggers deferred restart → SIGTERM → repeat. **Fix**: Set `gateway.reload.mode` to `"hot"`: ```bash openclaw config set gateway.reload.mode '"hot"' ``` In `hot` mode, safe changes hot-apply instantly. Critical changes (like plugin timestamps) only log a warning — no auto-restart. This breaks the loop. **Verify**: `grep "reload" ~/.openclaw/logs/gateway.log | tail -5` should show `config change applied (dynamic reads)` instead of `restart`. ### 2. macOS Throttling ("inefficient" classification) **Symptoms**: Gateway goes down and stays down for 30-60+ minutes. `launchctl print` shows `immediate reason = inefficient`. **Cause**: After many restarts (10+/day), macOS marks the job as low-priority and delays restarts via App Nap / Power Nap logic. **Fix**: Add these keys to the launchd plist (`~/Library/LaunchAgents/ai.openclaw.gateway.plist`): ```xml <key>ProcessType</key> <string>Interactive</string> <key>LowPriorityBackgroundIO</key> <false/> ``` Then reload: ```bash launchctl bootout gui/$(id -u)/ai.openclaw.gateway launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.openclaw.gateway.plist ``` **Note**: `openclaw gateway start` overwrites the plist. Use the patcher script (below) to auto-reapply. ### 3. Hung Shutdown **Symptoms**: Gateway receives SIGTERM but doesn't exit. launchd can't restart because old PID still alive. **Fix**: Set `ExitTimeOut` in the plist: ```xml <key>ExitTimeOut</key> <integer>10</integer> ``` After 10 seconds, launchd sends SIGKILL. ### 4. Power Nap Interference **Symptoms**: Gateway goes down during Mac sleep/wake cycles. **Check**: `pmset -g | grep powernap` **Fix**: `sudo pmset -a powernap 0` ## Plist Auto-Patcher Since `openclaw gateway start` overwrites the plist, use `scripts/patch-plist.sh` as a launchd WatchPaths agent: ```bash # Install the patcher bash scripts/install-patcher.sh ``` This creates a launchd agent that watches the gateway plist and re-adds `ExitTimeOut`, `ProcessType`, and `LowPriorityBackgroundIO` within seconds of any overwrite. ## Monitoring ### One-liner health check ```bash bash scripts/health-check.sh ``` Returns exit code 0 if healthy, 1 if issues detected. Suitable for cron or heartbeat integration. ### Continuous monitoring (cron integration) Add to your OpenClaw cron: ``` Check gateway health: bash ~/path/to/scripts/health-check.sh && echo "Gateway healthy" || echo "ALERT: Gateway issues detected" ``` ## Recommended Configuration For maximum stability on macOS: ```json5 { gateway: { reload: { mode: "hot" }, }, } ``` Plus plist keys: `ExitTimeOut=10`, `ProcessType=Interactive`, `LowPriorityBackgroundIO=false`, `ThrottleInterval=1`, `KeepAlive=true`. ## Troubleshooting Reference | Symptom | Check | Fix | |---------|-------|-----| | Restarts every 5-7 min | `grep restartReason gateway.log` | `reload.mode = "hot"` | | Down 30-60+ min | `launchctl print` → "inefficient" | ProcessType=Interactive | | Won't exit on SIGTERM | `ps -p PID` after SIGTERM | ExitTimeOut=10 | | Down after sleep | `pmset -g \| grep powernap` | `pmset -a powernap 0` | | Plugin timestamps changing | `grep resolvedAt openclaw.json` | `reload.mode = "hot"` |

path: scripts/diagnose.sh
content: #!/bin/bash
# diagnose.sh — Full OpenClaw gateway health diagnosis
# Exit codes: 0 = healthy, 1 = issues found

set -uo pipefail
ISSUES=0
PLIST="$HOME/Library/LaunchAgents/ai.openclaw.gateway.plist"
LABEL="ai.openclaw.gateway"
LOG="$HOME/.openclaw/logs/gateway.log"

echo "🔍 OpenClaw Gateway Health Diagnosis"
echo "====================================="
echo ""

# 1. Process state
echo "## 1. Process State"
if PRINT_OUT=$(launchctl print "gui/$(id -u)/$LABEL" 2>&1); then
    PID=$(echo "$PRINT_OUT" | grep '	pid' | awk '{print $3}')
    STATE=$(echo "$PRINT_OUT" | grep '	state =' | head -1 | awk '{print $3}')
    if [ -n "$PID" ] && [ "$PID" -gt 0 ] 2>/dev/null; then
        UPTIME=$(ps -o etime= -p "$PID" 2>/dev/null | xargs)
        echo "✅ Running (PID $PID, uptime: $UPTIME)"
    else
        echo "❌ NOT RUNNING (state: $STATE)"
        ISSUES=$((ISSUES + 1))
    fi
else
    echo "❌ Service not loaded in launchd"
    ISSUES=$((ISSUES + 1))
fi
echo ""

# 2. launchd classification
echo "## 2. launchd Classification"
REASON=$(echo "$PRINT_OUT" | grep 'immediate reason' | awk '{print $4}' 2>/dev/null || echo "unknown")
EXIT_TIMEOUT=$(echo "$PRINT_OUT" | grep 'exit timeout' | awk '{print $4}' 2>/dev/null || echo "unknown")
JETSAM=$(echo "$PRINT_OUT" | grep 'jetsam priority' | awk '{print $4}' 2>/dev/null || echo "unknown")
RUNS=$(echo "$PRINT_OUT" | grep '	runs =' | awk '{print $3}' 2>/dev/null || echo "unknown")

if [ "$REASON" = "inefficient" ]; then
    echo "⚠️  Classified as INEFFICIENT — restarts will be delayed!"
    ISSUES=$((ISSUES + 1))
else
    echo "✅ Classification: $REASON"
fi

echo "   Exit timeout: $EXIT_TIMEOUT"
echo "   Jetsam priority: $JETSAM (40=Background, 100+=Interactive)"
echo "   Run count: $RUNS"

if [ "$JETSAM" -lt 80 ] 2>/dev/null; then
    echo "⚠️  Low jetsam priority ($JETSAM) — add ProcessType=Interactive to plist"
    ISSUES=$((ISSUES + 1))
fi
echo ""

# 3. Plist config
echo "## 3. Plist Configuration"
if [ -f "$PLIST" ]; then
    for KEY in ExitTimeOut ProcessType LowPriorityBackgroundIO ThrottleInterval KeepAlive; do
        if plutil -p "$PLIST" 2>/dev/null | grep -q "\"$KEY\""; then
            VAL=$(plutil -p "$PLIST" | grep "\"$KEY\"" | head -1 | sed 's/.*=> //')
            echo "✅ $KEY = $VAL"
        else
            echo "❌ $KEY: MISSING"
            ISSUES=$((ISSUES + 1))
        fi
    done
else
    echo "❌ Plist not found at $PLIST"
    ISSUES=$((ISSUES + 1))
fi
echo ""

# 4. Reload mode
echo "## 4. Config Reload Mode"
RELOAD_MODE=$(cat ~/.openclaw/openclaw.json 2>/dev/null | python3 -c "
import sys, json
try:
    c = json.load(sys.stdin)
    print(c.get('gateway', {}).get('reload', {}).get('mode', 'hybrid (default)'))
except:
    print('unknown')
" 2>/dev/null)
if [ "$RELOAD_MODE" = "hot" ]; then
    echo "✅ Reload mode: hot (plugin changes won't cause restarts)"
else
    echo "⚠️  Reload mode: $RELOAD_MODE — plugin re-resolve may cause restart loops"
    ISSUES=$((ISSUES + 1))
fi
echo ""

# 5. Restart frequency
echo "## 5. Restart Frequency (last 24h)"
if [ -f "$LOG" ]; then
    TODAY=$(date +%Y-%m-%d)
    RESTARTS=$(grep "listening on ws:" "$LOG" 2>/dev/null | grep "$TODAY" | wc -l | xargs)
    SIGTERMS=$(grep "SIGTERM\|SIGUSR1" "$LOG" 2>/dev/null | grep "$TODAY" | wc -l | xargs)
    PLUGIN_RESTARTS=$(grep "restartReason.*plugins.installs" "$LOG" 2>/dev/null | grep "$TODAY" | wc -l | xargs)
    echo "   Boots today: $RESTARTS"
    echo "   Signals (SIGTERM/SIGUSR1): $SIGTERMS"
    echo "   Plugin-caused restarts: $PLUGIN_RESTARTS"
    if [ "$PLUGIN_RESTARTS" -gt 3 ] 2>/dev/null; then
        echo "⚠️  Plugin restart loop detected! Set gateway.reload.mode = \"hot\""
        ISSUES=$((ISSUES + 1))
    fi
else
    echo "⚠️  Log file not found"
fi
echo ""

# 6. Power management
echo "## 6. Power Management"
POWERNAP=$(pmset -g 2>/dev/null | grep "powernap" | awk '{print $2}')
SLEEP=$(pmset -g 2>/dev/null | grep " sleep " | head -1 | awk '{print $2}')
echo "   Power Nap: $POWERNAP (0=disabled, 1=enabled)"
echo "   Sleep: $SLEEP (0=never)"
if [ "$POWERNAP" = "1" ]; then
    echo "⚠️  Power Nap enabled — may delay restarts. Fix: sudo pmset -a powernap 0"
    ISSUES=$((ISSUES + 1))
fi
echo ""

# Summary
echo "====================================="
if [ "$ISSUES" -eq 0 ]; then
    echo "✅ All checks passed — gateway is healthy"
    exit 0
else
    echo "⚠️  $ISSUES issue(s) found — see above for fixes"
    exit 1
fi


path: scripts/health-check.sh
content: #!/bin/bash
# health-check.sh — Quick gateway health check (exit 0 = ok, exit 1 = issue)
# Suitable for cron or heartbeat integration.

LABEL="ai.openclaw.gateway"

# Check if service is running
PRINT_OUT=$(launchctl print "gui/$(id -u)/$LABEL" 2>&1) || exit 1
PID=$(echo "$PRINT_OUT" | grep '	pid' | awk '{print $3}')

if [ -z "$PID" ] || ! kill -0 "$PID" 2>/dev/null; then
    echo "ALERT: Gateway not running"
    exit 1
fi

# Check if classified as inefficient
if echo "$PRINT_OUT" | grep -q "immediate reason = inefficient"; then
    echo "WARN: Gateway classified as inefficient by launchd"
    exit 1
fi

# Check restart count today
TODAY=$(date +%Y-%m-%d)
LOG="$HOME/.openclaw/logs/gateway.log"
if [ -f "$LOG" ]; then
    RESTARTS=$(grep "listening on ws:" "$LOG" | grep "$TODAY" | wc -l | xargs)
    if [ "$RESTARTS" -gt 10 ]; then
        echo "WARN: $RESTARTS restarts today — possible restart loop"
        exit 1
    fi
fi

UPTIME=$(ps -o etime= -p "$PID" 2>/dev/null | xargs)
echo "OK: Gateway running (PID $PID, uptime: $UPTIME)"
exit 0


path: scripts/install-patcher.sh
content: #!/bin/bash
# install-patcher.sh — Install launchd WatchPaths agent for auto-patching gateway plist
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PATCHER_SCRIPT="$SCRIPT_DIR/patch-plist.sh"
PLIST_TO_WATCH="$HOME/Library/LaunchAgents/ai.openclaw.gateway.plist"
PATCHER_PLIST="$HOME/Library/LaunchAgents/ai.openclaw.plist-patcher.plist"
LOG_DIR="$HOME/.openclaw/logs"

chmod +x "$PATCHER_SCRIPT"
mkdir -p "$LOG_DIR"

cat > "$PATCHER_PLIST" << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>ai.openclaw.plist-patcher</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/bash</string>
        <string>$PATCHER_SCRIPT</string>
    </array>
    <key>WatchPaths</key>
    <array>
        <string>$PLIST_TO_WATCH</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>StandardOutPath</key>
    <string>$LOG_DIR/plist-patcher.log</string>
    <key>StandardErrorPath</key>
    <string>$LOG_DIR/plist-patcher.err.log</string>
</dict>
</plist>
EOF

# Load the patcher
launchctl bootout "gui/$(id -u)/ai.openclaw.plist-patcher" 2>/dev/null || true
launchctl bootstrap "gui/$(id -u)" "$PATCHER_PLIST"

echo "✅ Plist patcher installed and active"
echo "   Watches: $PLIST_TO_WATCH"
echo "   Script: $PATCHER_SCRIPT"
echo "   Log: $LOG_DIR/plist-patcher.log"


path: scripts/patch-plist.sh
content: #!/bin/bash
# patch-plist.sh — Re-adds critical keys to the OpenClaw gateway plist
# after openclaw overwrites it with `gateway start`.
# Triggered by launchd WatchPaths on the plist file.

PLIST="$HOME/Library/LaunchAgents/ai.openclaw.gateway.plist"
LOG="$HOME/.openclaw/logs/plist-patcher.log"
CHANGED=0

[ ! -f "$PLIST" ] && exit 0

# ExitTimeOut: force-kill after 10s if shutdown hangs
if ! plutil -p "$PLIST" 2>/dev/null | grep -q '"ExitTimeOut"'; then
    plutil -insert ExitTimeOut -integer 10 "$PLIST" 2>/dev/null && CHANGED=1
fi

# ProcessType: Interactive prevents launchd from delaying restarts
if ! plutil -p "$PLIST" 2>/dev/null | grep -q '"ProcessType"'; then
    plutil -insert ProcessType -string "Interactive" "$PLIST" 2>/dev/null && CHANGED=1
fi

# LowPriorityBackgroundIO: prevent IO throttling
if ! plutil -p "$PLIST" 2>/dev/null | grep -q '"LowPriorityBackgroundIO"'; then
    plutil -insert LowPriorityBackgroundIO -bool false "$PLIST" 2>/dev/null && CHANGED=1
fi

if [ "$CHANGED" -eq 1 ]; then
    echo "$(date '+%Y-%m-%dT%H:%M:%S%z') [plist-patcher] Patched gateway plist (ExitTimeOut+ProcessType+LowPriorityIO)" >> "$LOG"
fi


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)