Skill: zc-kama/gateway-resilience-guard (version 1.4.4)

--- name: gateway-resilience-guard description: OpenClaw Gateway Resilience Guard keeps Gateway, channels, logs, and optional model-provider probes observable after Wi-Fi changes, WSL sleep/resume, session expiry, provider timeouts, or partial outages. It adds a localhost dashboard with charts, event trends, language/theme switching, strategy presets, guarded action unlock, native OpenClaw health probes, restart safeguards, systemd, LaunchAgent, Task Scheduler, and an optional OpenClaw plugin bridge. tags: - openclaw - gateway - watchdog - resilience - wechat - wsl - systemd - macos - windows requirements: tools: - bash - curl - systemctl optional - launchctl optional - powershell optional - openclaw recommended permissions: - Writes a user-level systemd service, macOS LaunchAgent, or Windows scheduled task when requested. - Writes config and logs under user-level config/state directories. - Starts a localhost dashboard on 127.0.0.1:18790 when Python is available. - Restarts OpenClaw Gateway through systemctl --user or openclaw gateway restart. - Optional model probe sends real OpenClaw model requests only when explicitly enabled. --- # OpenClaw Gateway Resilience Guard Use this skill when a user wants to keep OpenClaw Gateway and message channels online after network drops, WSL sleep/resume, macOS/Windows wake events, long-lived connection failures, or recurring model-provider timeouts. ## Install Linux, WSL, or macOS: ```bash bash install-watchdog.sh ``` The installer works with defaults. It prompts for the main channel probe URL when running interactively, but pressing Enter is enough for the default WeChat probe. For unattended install: ```bash bash install-watchdog.sh --yes ``` For a custom channel: ```bash bash install-watchdog.sh --channel-url "https://your-channel.example.com/health" ``` Windows PowerShell: ```powershell powershell -ExecutionPolicy Bypass -File .\install-watchdog.ps1 ``` After install, open the standalone dashboard: ```text http://127.0.0.1:18790/ ``` The dashboard is served by the watchdog process, not Gateway, so it remains the recovery entry when Gateway is down. Optional Gateway-side bridge while Gateway is healthy: ```bash openclaw plugins install ./openclaw-plugin openclaw plugins enable resilience-guard openclaw gateway restart ``` ## Operate ```bash systemctl --user status gateway-watchdog journalctl --user -u gateway-watchdog -f bash ~/.local/share/openclaw-gateway-watchdog/uninstall-watchdog.sh ``` On macOS, inspect `launchctl print gui/$(id -u)/ai.clawhub.gateway-resilience-guard`. On Windows, inspect `Get-ScheduledTask -TaskName "OpenClaw Gateway Resilience Guard"`. If user systemd is unavailable, the installer starts a direct background fallback and stores its pid under `~/.local/state/openclaw-gateway-watchdog/watchdog.pid`. ## Safety Model The watchdog restarts only after layered checks: 1. OpenClaw native health probes: `openclaw gateway status --require-rpc`, `openclaw health --json --verbose`, and `openclaw status --deep` when available. 2. Runtime diagnostics: `openclaw models status` plus `openclaw logs --plain` signal scanning for provider timeout, proxy/network, rate limit, auth, channel session, gateway degraded, config reload, and task runtime warnings. 3. Local gateway health URL/TCP and main channel URL fallbacks. 4. General network URLs to avoid restarting during whole-machine network failure. 5. Optional model-provider probe via `openclaw agent --json`; default action is evidence logging only. 6. Dashboard action token, backoff, and hourly restart limits to avoid restart storms. Model probing is disabled by default because it consumes real provider quota. To diagnose provider timeouts, set `MODEL_PROBE_ENABLED=1`, keep `MODEL_PROBE_ACTION=log`, and inspect `model-probe-history.jsonl` the next day. Diagnostic log scanning is enabled by default, but `OPENCLAW_DIAG_ACTION=log` keeps it non-invasive unless the operator explicitly opts into a restart or custom command. Use the dashboard strategy buttons for common modes: observe, overnight diagnosis, channel recovery, and conservative circuit breaker. Tell users to review `~/.config/openclaw-gateway-watchdog/watchdog.env` before publishing, sharing logs, or reporting issues.

path: CHANGELOG.md
content: # Changelog

## 1.4.4

- Replace fragile Web Component controls with semantic native buttons and selects styled by the local dashboard design system.
- Fix the top action bar, sidebar navigation, refresh/export controls, and strategy buttons so they always render as clickable UI even if component scripts are blocked or stale.
- Center the brand mark text and restore pointer cursor, hover, focus, and active states across navigation and action controls.
- Fix config summary row layout so long environment keys no longer overlap values.

## 1.4.3

- Rebuild the dashboard visual system on local Fluent UI Web Components instead of plain native controls.
- Replace the old dropdowns and buttons with componentized controls, dark glass panels, animated ambient background, and refined hover states.
- Rebalance every dashboard page so Overview, Trends, Strategy, Logs, and Config each have complete work areas rather than sparse split-up cards.
- Add richer summaries for trend digest, latest signals, strategy matrix, action center, and runtime map.
- Serve bundled local component assets under `/vendor/` so the recovery UI does not depend on CDN availability.

## 1.4.2

- Rework the dashboard sidebar into real view switching for Overview, Trends, Strategy, Logs, and Config instead of one long page.
- Tighten the dashboard grid, panel sizing, button alignment, and sidebar polish for a cleaner operations-console layout.
- Fix the trend chart axis label collision by removing the redundant x-axis title and reserving more space for tick labels.
- Keep charts stable across refreshes by drawing only the active view and ignoring hidden canvases.
- Move strategy explanations into centered button content and remove the sidebar localhost note.

## 1.4.1

- Redesign the dashboard with a sidebar layout, stronger visual grouping, and selectable Light, Dark, Ocean, and Forest themes.
- Add Chinese/English language switching for dashboard labels, buttons, hints, legends, and notifications.
- Fix canvas redraw sizing so repeated refreshes do not stretch dashboard panels.
- Replace "overnight timeline" with a general event trend chart that includes lanes, legends, and time-axis ticks.
- Auto-follow the newest watchdog log lines while keeping a toggle for manual scrolling.
- Add an unlock/lock action flow and strategy hover hints so guarded controls are discoverable.

## 1.4.0

- Add a standalone local dashboard at `http://127.0.0.1:18790` that remains available when OpenClaw Gateway is down.
- Add dashboard API summaries, log-signal charts, overnight timelines, status file freshness, safe config summaries, diagnostic export, and quick strategy presets.
- Add guarded dashboard actions for run diagnostics, restart Gateway, and apply presets using a generated local action token.
- Add an OpenClaw native plugin bridge that registers `/resilience-guard` and redirects to the external dashboard when Gateway is healthy.
- Install dashboard files on Linux/WSL, macOS, and Windows.

## 1.3.1

- Publish a ClawHub package that includes the Windows PowerShell installer, watchdog, and uninstaller files.

## 1.3.0

- Add an opt-in model-provider probe that uses `openclaw agent --json` against the configured OpenClaw model path.
- Add a no-credential provider edge probe that checks the configured provider `baseUrl` before the end-to-end model call.
- Add OpenClaw runtime diagnostics that snapshot gateway status, health, model status, deep status, and recent OpenClaw logs.
- Classify log signals for provider timeout, proxy/network, rate limit, auth, channel session, gateway degraded, config reload, task runtime, and related warning families.
- Record model probe evidence in the main log and `model-probe-history.jsonl` without logging API keys.
- Add configurable model failure actions: log-only, gateway restart, or a custom command after consecutive failures.
- Document safe overnight diagnostics for separating Gateway/channel failures from model-provider timeouts.

## 1.2.0

- Add OpenClaw-native health probing via `openclaw gateway status --json --require-rpc`, `openclaw health --json --verbose`, and `openclaw status --deep`.
- Add Windows Task Scheduler support with native PowerShell install, watchdog, and uninstall scripts.
- Add macOS LaunchAgent support to the Bash installer and uninstaller.
- Document cross-platform installation and native probe configuration.

## 1.1.0

- Rename the ClawHub package to `gateway-resilience-guard`.
- Expand the public summary to describe the layered probes, restart guardrails, and advantages over simple restart loops or one-URL monitors.

## 1.0.1

- Add the published ClawHub URL and install command to the README files.

## 1.0.0

- First public zero-config release.
- Install from the current folder instead of a hard-coded workspace path.
- Generate user config and systemd service automatically.
- Add layered gateway/channel/network probes.
- Add exponential backoff, hourly restart limits, lock protection, and log rotation.
- Reset failure state after five consecutive successful channel probes.
- Add ClawHub-ready `SKILL.md`.
- Initial ClawHub publication used a temporary slug before the 1.1.0 rename.


path: dashboard/server.py
content: #!/usr/bin/env python3
"""Local dashboard for OpenClaw Gateway Resilience Guard.

This server is intentionally stdlib-only. It reads the watchdog state directory
and can optionally perform local actions when protected by a same-origin token.
"""

from __future__ import annotations

import json
import os
import re
import secrets
import shutil
import subprocess
import sys
import time
from http import HTTPStatus
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
from pathlib import Path
from urllib.parse import parse_qs, urlparse


ROOT = Path(__file__).resolve().parent
STATIC = ROOT / "static"
STATE_DIR = Path(os.environ.get("WATCHDOG_STATE_DIR", Path.home() / ".local/state/openclaw-gateway-watchdog"))
CONFIG_FILE = Path(os.environ.get("WATCHDOG_CONFIG_FILE", Path.home() / ".config/openclaw-gateway-watchdog/watchdog.env"))
HOST = os.environ.get("WATCHDOG_DASHBOARD_HOST", "127.0.0.1")
PORT = int(os.environ.get("WATCHDOG_DASHBOARD_PORT", "18790"))
ACTIONS_ENABLED = os.environ.get([REDACTED_SECRET], "0").lower() in {"1", "true", "yes", "on"}
ACTION_TOKEN = os.environ.get("WATCHDOG_DASHBOARD_TOKEN", "")
MAX_TEXT_BYTES = 512_000


PRESETS = {
    "observe": {
        "OPENCLAW_DIAG_ACTION": "log",
        "MODEL_PROBE_ENABLED": "0",
        "CHANNEL_FAILURES_BEFORE_RESTART": "2",
        "MAX_RESTARTS_PER_HOUR": "6",
        "BASE_INTERVAL": "60",
        "NIGHT_INTERVAL": "300",
    },
    "overnight": {
        "OPENCLAW_DIAG_ACTION": "log",
        "MODEL_PROBE_ENABLED": "1",
        "MODEL_EDGE_PROBE_ENABLED": "1",
        "MODEL_PROBE_INTERVAL": "600",
        "MODEL_PROBE_TIMEOUT": "120",
        "MODEL_PROBE_ACTION": "log",
        "BASE_INTERVAL": "60",
        "NIGHT_INTERVAL": "300",
    },
    "channel-recovery": {
        "OPENCLAW_DIAG_ACTION": "log",
        "MODEL_PROBE_ACTION": "log",
        "CHANNEL_FAILURES_BEFORE_RESTART": "2",
        "SUCCESS_COUNT_TO_RESET": "5",
        "MAX_RESTARTS_PER_HOUR": "6",
    },
    "conservative": {
        "OPENCLAW_DIAG_ACTION": "log",
        "MODEL_PROBE_ACTION": "log",
        "CHANNEL_FAILURES_BEFORE_RESTART": "3",
        "MAX_RESTARTS_PER_HOUR": "3",
        "BASE_INTERVAL": "90",
        "NIGHT_INTERVAL": "300",
    },
}


def now_ms() -> int:
    return int(time.time() * 1000)


def read_text(path: Path, limit: int = MAX_TEXT_BYTES) -> str:
    try:
        with path.open("rb") as fh:
            data = fh.read(limit + 1)
        if len(data) > limit:
            data = data[-limit:]
        return data.decode("utf-8-sig", errors="replace")
    except FileNotFoundError:
        return ""
    except OSError as exc:
        return f"[read error: {exc}]"


def read_tail(path: Path, lines: int = 120) -> list[str]:
    text = read_text(path)
    if not text:
        return []
    return [line for line in text.splitlines()[-lines:]]


def read_json(path: Path):
    text = read_text(path)
    if not text.strip():
        return None
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        return {"raw": text[-4000:]}


def read_jsonl(path: Path, limit: int = 200) -> list[dict]:
    items: list[dict] = []
    for line in read_tail(path, limit):
        line = line.strip().lstrip("\ufeff")
        if not line:
            continue
        try:
            parsed = json.loads(line)
            if isinstance(parsed, dict):
                items.append(parsed)
        except json.JSONDecodeError:
            items.append({"raw": line})
    return items


def stat_file(path: Path) -> dict:
    try:
        st = path.stat()
        return {
            "exists": True,
            "path": str(path),
            "size": st.st_size,
            "mtime": int(st.st_mtime),
            "ageSeconds": max(0, int(time.time() - st.st_mtime)),
        }
    except FileNotFoundError:
        return {"exists": False, "path": str(path)}


def parse_env_config(path: Path) -> dict[str, str]:
    values: dict[str, str] = {}
    text = read_text(path)
    for raw in text.splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        key = key.strip()
        value = value.strip().strip('"').strip("'")
        if key:
            values[key] = value
    return values


def parse_config() -> dict:
    if CONFIG_FILE.suffix.lower() == ".json":
        data = read_json(CONFIG_FILE)
        return data if isinstance(data, dict) else {}
    return parse_env_config(CONFIG_FILE)


def mask_config(config: dict) -> dict:
    safe = {}
    for key, value in config.items():
        low = key.lower()
        if any(secret in low for secret in ("token", "password", "secret", "key")) and "probe" not in low:
            safe[key] = "***"
        else:
            safe[key] = value
    return safe


def category_counts(diagnostics: list[dict], signals_text: str) -> dict[str, int]:
    counts: dict[str, int] = {}
    for item in diagnostics:
        cats = str(item.get("categories", "") or "")
        for cat in cats.split(","):
            cat = cat.strip()
            if cat and cat != "none":
                counts[cat] = counts.get(cat, 0) + 1
    for cat in re.findall(r"\b(provider_timeout|proxy_or_network|provider_rate_limit|provider_auth|abort_stuck|memory_dream_timeout|channel_session|gateway_degraded|config_reload|task_runtime|openclaw_warning)\b", signals_text):
        counts[cat] = counts.get(cat, 0) + 1
    return dict(sorted(counts.items(), key=lambda item: item[1], reverse=True))


def restart_counts() -> dict:
    total = 0
    by_hour = []
    for path in sorted(STATE_DIR.glob("restarts.*"))[-48:]:
        try:
            count = int(read_text(path).strip() or "0")
        except ValueError:
            count = 0
        total += count
        by_hour.append({"hour": path.name.replace("restarts.", ""), "count": count})
    return {"totalRecent": total, "byHour": by_hour}


def status_from_snapshots(health: object, gateway: object, files: dict) -> dict:
    status = "unknown"
    reason = "No recent OpenClaw health snapshot yet."
    if isinstance(health, dict):
        if health.get("ok") is False or health.get("degraded") is True:
            status = "degraded"
            reason = "OpenClaw health reports degraded or not ok."
        elif health.get("ok") is True:
            status = "ok"
            reason = "OpenClaw health reports ok."
    if status == "unknown" and isinstance(gateway, dict):
        text = json.dumps(gateway).lower()
        if '"ok": true' in text or '"active": true' in text or '"running": true' in text:
            status = "ok"
            reason = "Gateway status snapshot looks active."
        elif "error" in text or "failed" in text or '"ok": false' in text:
            status = "degraded"
            reason = "Gateway status snapshot contains failure signals."
    newest = min((meta.get("ageSeconds", 999999) for meta in files.values() if meta.get("exists")), default=999999)
    if newest > 1800 and status == "ok":
        status = "stale"
        reason = "Last status files are stale."
    return {"status": status, "reason": reason}


def build_status() -> dict:
    config = parse_config()
    files = {
        "log": stat_file(STATE_DIR / "watchdog.log"),
        "diagnostics": stat_file(STATE_DIR / "last-openclaw-diagnostics.jsonl"),
        "gateway": stat_file(STATE_DIR / "last-openclaw-gateway-status.json"),
        "health": stat_file(STATE_DIR / "last-openclaw-health.json"),
        "models": stat_file(STATE_DIR / "last-openclaw-model-status.json"),
        "signals": stat_file(STATE_DIR / "last-openclaw-log-signals.txt"),
        "modelHistory": stat_file(STATE_DIR / "model-probe-history.jsonl"),
    }
    health = read_json(STATE_DIR / "last-openclaw-health.json")
    gateway = read_json(STATE_DIR / "last-openclaw-gateway-status.json")
    models = read_json(STATE_DIR / "last-openclaw-model-status.json")
    diagnostics = read_jsonl(STATE_DIR / "last-openclaw-diagnostics.jsonl", 240)
    model_history = read_jsonl(STATE_DIR / "model-probe-history.jsonl", 240)
    signals = read_text(STATE_DIR / "last-openclaw-log-signals.txt", 128_000)
    log_tail = read_tail(STATE_DIR / "watchdog.log", 160)
    categories = category_counts(diagnostics, signals)
    restarts = restart_counts()
    summary = status_from_snapshots(health, gateway, files)
    model_failures = sum(1 for item in model_history if item.get("status") == "fail")
    model_ok = sum(1 for item in model_history if item.get("status") == "ok")
    latest_diag = diagnostics[-1] if diagnostics else {}
    latest_model = model_history[-1] if model_history else {}
    return {
        "generatedAt": now_ms(),
        "version": "1.4.4",
        "stateDir": str(STATE_DIR),
        "configFile": str(CONFIG_FILE),
        "dashboard": {
            "host": HOST,
            "port": PORT,
            "actionsEnabled": ACTIONS_ENABLED,
            "tokenRequired": ACTIONS_ENABLED,
        },
        "summary": {
            **summary,
            "categoryCount": len(categories),
            "recentRestarts": restarts["totalRecent"],
            "modelOk": model_ok,
            "modelFailures": model_failures,
            "lastDiagnostic": latest_diag,
            "lastModelProbe": latest_model,
        },
        "config": mask_config(config),
        "files": files,
        "health": health,
        "gateway": gateway,
        "models": models,
        "diagnostics": diagnostics,
        "modelHistory": model_history,
        "signals": signals.splitlines()[-120:],
        "logTail": log_tail,
        "categories": categories,
        "restarts": restarts,
        "presets": list(PRESETS.keys()),
    }


def verify_action(headers) -> tuple[bool, str]:
    if not ACTIONS_ENABLED:
        return False, "Dashboard actions are disabled."
    [REDACTED_SECRET]"X-Watchdog-Token", "")
    if not ACTION_TOKEN or not secrets.compare_digest(token, ACTION_TOKEN):
        return False, "Invalid or missing dashboard action token."
    return True, "ok"


def run_command(command: str, timeout: int = 30) -> dict:
    started = time.time()
    if not command:
        return {"ok": False, "error": "empty command"}
    try:
        proc = subprocess.run(
            command,
            shell=True,
            cwd=str(STATE_DIR),
            text=True,
            capture_output=True,
            timeout=timeout,
        )
        return {
            "ok": proc.returncode == 0,
            "exitCode": proc.returncode,
            "durationSeconds": round(time.time() - started, 3),
            "stdout": proc.stdout[-4000:],
            "stderr": proc.stderr[-4000:],
        }
    except subprocess.TimeoutExpired as exc:
        return {"ok": False, "error": "timeout", "stdout": (exc.stdout or "")[-4000:], "stderr": (exc.stderr or "")[-4000:]}


def update_env_file(path: Path, updates: dict[str, str]) -> None:
    lines = read_text(path).splitlines()
    seen = set()
    out = []
    for line in lines:
        stripped = line.strip()
        if stripped and not stripped.startswith("#") and "=" in stripped:
            key = stripped.split("=", 1)[0].strip()
            if key in updates:
                out.append(f'{key}="{updates[key]}"')
                seen.add(key)
                continue
        out.append(line)
    for key, value in updates.items():
        if key not in seen:
            out.append(f'{key}="{value}"')
    path.write_text("\n".join(out) + "\n", encoding="utf-8")


def update_json_file(path: Path, updates: dict[str, str]) -> None:
    data = read_json(path)
    if not isinstance(data, dict):
        data = {}
    mapping = {
        "OPENCLAW_DIAG_ACTION": "OpenClawDiagAction",
        "MODEL_PROBE_ENABLED": "ModelProbeEnabled",
        "MODEL_EDGE_PROBE_ENABLED": "ModelEdgeProbeEnabled",
        "MODEL_PROBE_INTERVAL": "ModelProbeInterval",
        "MODEL_PROBE_TIMEOUT": "ModelProbeTimeout",
        "MODEL_PROBE_ACTION": "ModelProbeAction",
        "CHANNEL_FAILURES_BEFORE_RESTART": "ChannelFailuresBeforeRestart",
        "SUCCESS_COUNT_TO_RESET": "SuccessCountToReset",
        "MAX_RESTARTS_PER_HOUR": "MaxRestartsPerHour",
        "BASE_INTERVAL": "BaseInterval",
        "NIGHT_INTERVAL": "NightInterval",
    }
    for key, value in updates.items():
        target = mapping.get(key, key)
        if value in {"0", "1"} and target.endswith("Enabled"):
            data[target] = value == "1"
        elif re.fullmatch(r"\d+", value):
            data[target] = int(value)
        else:
            data[target] = value
    path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")


def apply_preset(name: str) -> dict:
    updates = PRESETS.get(name)
    if not updates:
        return {"ok": False, "error": f"unknown preset: {name}"}
    CONFIG_FILE.parent.mkdir(parents=True, exist_ok=True)
    if CONFIG_FILE.suffix.lower() == ".json":
        update_json_file(CONFIG_FILE, updates)
    else:
        update_env_file(CONFIG_FILE, updates)
    return {"ok": True, "preset": name, "updates": updates}


def run_diagnostics() -> dict:
    commands = {
        "gateway": "openclaw gateway status --json --require-rpc",
        "health": "openclaw health --json --verbose",
        "models": "openclaw models status --json",
        "statusDeep": "openclaw status --deep --no-color",
        "logs": "openclaw logs --plain --limit 200",
    }
    if not shutil.which("openclaw"):
        return {"ok": False, "error": "openclaw CLI not found on PATH"}
    results = {name: run_command(cmd, timeout=20) for name, cmd in commands.items()}
    targets = {
        "gateway": "last-openclaw-gateway-status.json",
        "health": "last-openclaw-health.json",
        "models": "last-openclaw-model-status.json",
        "statusDeep": "last-openclaw-status-deep.txt",
        "logs": "last-openclaw-logs.txt",
    }
    STATE_DIR.mkdir(parents=True, exist_ok=True)
    for name, result in results.items():
        text = result.get("stdout") or result.get("stderr") or json.dumps(result)
        (STATE_DIR / targets[name]).write_text(str(text), encoding="utf-8")
    return {"ok": all(result.get("ok") for result in results.values()), "results": results}


class Handler(BaseHTTPRequestHandler):
    server_version = "OpenClawWatchdogDashboard/1.4.4"

    def log_message(self, fmt: str, *args) -> None:
        line = "[%s] %s\n" % (time.strftime("%Y-%m-%d %H:%M:%S"), fmt % args)
        try:
            with (STATE_DIR / "dashboard-access.log").open("a", encoding="utf-8") as fh:
                fh.write(line)
        except OSError:
            pass

    def send_json(self, payload, status: int = 200) -> None:
        body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
        self.send_response(status)
        self.send_header("Content-Type", "application/json; charset=utf-8")
        self.send_header("Cache-Control", "no-store")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def send_static(self, path: Path, content_type: str) -> None:
        if not path.exists():
            self.send_error(HTTPStatus.NOT_FOUND)
            return
        body = path.read_bytes()
        if path.name == "index.html":
            injection = f"<script>window.WATCHDOG_ACTION_TOKEN = {json.dumps(ACTION_TOKEN if ACTIONS_ENABLED else '')};</script>"
            body = body.replace(b"</head>", injection.encode("utf-8") + b"</head>")
        self.send_response(200)
        self.send_header("Content-Type", content_type)
        self.send_header("Cache-Control", "no-store")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self) -> None:
        parsed = urlparse(self.path)
        path = parsed.path
        if path == "/api/status":
            self.send_json(build_status())
            return
        if path == "/api/export":
            self.send_json({"exportedAt": now_ms(), "status": build_status()})
            return
        if path in {"/", "/index.html"}:
            self.send_static(STATIC / "index.html", "text/html; charset=utf-8")
            return
        if path == "/app.js":
            self.send_static(STATIC / "app.js", "text/javascript; charset=utf-8")
            return
        if path == "/styles.css":
            self.send_static(STATIC / "styles.css", "text/css; charset=utf-8")
            return
        self.send_error(HTTPStatus.NOT_FOUND)

    def do_POST(self) -> None:
        ok, message = verify_action(self.headers)
        if not ok:
            self.send_json({"ok": False, "error": message}, 403)
            return
        length = int(self.headers.get("Content-Length", "0") or "0")
        raw = self.rfile.read(min(length, 65536)) if length else b"{}"
        try:
            body = json.loads(raw.decode("utf-8") or "{}")
        except json.JSONDecodeError:
            body = {}
        path = urlparse(self.path).path
        config = parse_config()
        if path == "/api/actions/restart-gateway":
            command = str(config.get("RESTART_COMMAND") or config.get("RestartCommand") or "openclaw gateway restart")
            self.send_json(run_command(command, timeout=45))
            return
        if path == "/api/actions/run-diagnostics":
            self.send_json(run_diagnostics())
            return
        if path == "/api/config/preset":
            self.send_json(apply_preset(str(body.get("preset", ""))))
            return
        self.send_error(HTTPStatus.NOT_FOUND)


def main() -> int:
    STATE_DIR.mkdir(parents=True, exist_ok=True)
    server = ThreadingHTTPServer((HOST, PORT), Handler)
    print(f"OpenClaw watchdog dashboard listening on http://{HOST}:{PORT}", flush=True)
    server.serve_forever()
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: dashboard/static/app.js
content: const $ = (id) => document.getElementById(id);

const ROUTES = ["overview", "trends", "strategy", "logs", "settings"];

const DICT = {
  zh: {
    brandSub: "OpenClaw 看门狗",
    navOverview: "总览",
    navTrends: "趋势",
    navStrategy: "策略",
    navLogs: "日志",
    navSettings: "配置",
    titleOverview: "恢复控制台",
    titleTrends: "趋势分析",
    titleStrategy: "策略控制",
    titleLogs: "日志审计",
    titleSettings: "配置状态",
    subtitleOverview: "把 Gateway、通道、日志和模型探针合成一个恢复判断。",
    subtitleTrends: "观察夜间超时、provider 异常和通道恢复是否在同一时间聚集。",
    subtitleStrategy: "选择处置姿态，并把高风险动作放进解锁流程。",
    subtitleLogs: "查看 watchdog 与 OpenClaw 信号，快速定位最近一次断链。",
    subtitleSettings: "确认运行路径、状态文件新鲜度和当前策略配置。",
    export: "导出诊断",
    currentDecision: "当前判断",
    recentRestarts: "近期重启",
    signalTypes: "异常类别",
    modelOk: "模型成功",
    modelFail: "模型失败",
    layers: "链路分层",
    layersHint: "按恢复路径从 Gateway 到模型 provider 排列。",
    signalCategories: "异常分类",
    signalHint: "用于判断是代理、provider 还是通道层问题。",
    eventTrend: "事件趋势",
    eventTrendHint: "诊断、日志告警和模型探针的同屏时间线。",
    trendDigest: "趋势摘要",
    trendDigestHint: "把散点变成可读结论。",
    modelSummary: "模型探针",
    restartSummary: "重启统计",
    latestSignals: "最新信号",
    actionCenter: "处置入口",
    quickStrategy: "快速策略",
    strategyHint: "选择故障处理姿态，所有写入动作都需要先解锁。",
    strategyMatrix: "策略矩阵",
    manualActions: "手动操作",
    manualHint: "高风险动作集中放在这里。",
    unlock: "解锁操作",
    lock: "锁定操作",
    presetObserve: "观察模式",
    presetOvernight: "夜间诊断",
    presetChannel: "通道恢复",
    presetConservative: "保守熔断",
    tipObserve: "只记录证据，不主动改变策略，适合先观察问题。",
    tipOvernight: "开启轻量模型探针并保持 log-only，适合排查固定时段超时。",
    tipChannel: "保持通道恢复敏感度，适合通道/session 经常断开的场景。",
    tipConservative: "提高阈值并降低重启频率，适合网络不稳定时防止重启风暴。",
    runDiag: "立即诊断",
    restartGateway: "重启 Gateway",
    configSummary: "配置摘要",
    sensitiveMasked: "敏感项已遮盖",
    stateFiles: "状态文件",
    recentLogs: "最近日志",
    logHint: "默认跟随最新行，适合观察夜间超时。",
    followLatest: "保持最新",
    runtimeMap: "运行地图",
    locked: "只读",
    actionReady: "可执行",
    unlocked: "已解锁",
    missing: "缺失",
    noSignals: "暂无异常分类",
    noTimeline: "暂无趋势数据",
    refreshFailed: "刷新失败",
    runningDiag: "正在执行诊断...",
    diagDone: "诊断完成",
    restartConfirm: "确认重启 OpenClaw Gateway？",
    restartDone: "重启命令已执行",
    presetDone: "已切换策略",
    tokenPrompt: "请输入 watchdog dashboard token。可在 watchdog.env/json 中查看 DASHBOARD_TOKEN。",
    tokenSaved: "操作 token 已保存到当前浏览器。",
    tokenCleared: "已清除本地 token。",
    clearConfirm: "要清除当前浏览器保存的操作 token 吗？",
    unlockFirst: "请先解锁操作。",
    statusUnknown: "正在汇总 watchdog 状态。",
    apiFail: "API失败",
    logWarn: "日志WARN",
    modelOkLegend: "模型OK",
    diagOk: "诊断OK",
    lastProbe: "最近探针",
    successRate: "成功比例",
    latestReason: "最新原因",
    noModelHistory: "暂无模型探针记录",
    restartWindow: "统计窗口",
    lastRestart: "最近重启",
    restartPolicy: "重启策略",
    stateDirectory: "状态目录",
    configFile: "配置文件",
    dashboardPort: "Dashboard 端口",
    dashboardAction: "写入动作",
    dominantSignal: "主要信号",
    evidenceCount: "证据数量",
    noRecentSignals: "暂无信号",
    enabled: "开启",
    disabled: "关闭",
    notAvailable: "暂无",
  },
  en: {
    brandSub: "OpenClaw Watchdog",
    navOverview: "Overview",
    navTrends: "Trends",
    navStrategy: "Strategy",
    navLogs: "Logs",
    navSettings: "Config",
    titleOverview: "Recovery Console",
    titleTrends: "Trend Analysis",
    titleStrategy: "Strategy Control",
    titleLogs: "Log Audit",
    titleSettings: "Config State",
    subtitleOverview: "Merge Gateway, channel, log, and model-probe evidence into one recovery decision.",
    subtitleTrends: "Watch whether fixed-hour timeouts, provider errors, and channel recovery cluster together.",
    subtitleStrategy: "Choose a response posture and keep risky actions behind an unlock flow.",
    subtitleLogs: "Audit watchdog and OpenClaw signals to locate the latest disconnect.",
    subtitleSettings: "Confirm runtime paths, state freshness, and active strategy config.",
    export: "Export Diagnostics",
    currentDecision: "Current Decision",
    recentRestarts: "Recent Restarts",
    signalTypes: "Signal Types",
    modelOk: "Model OK",
    modelFail: "Model Failed",
    layers: "Health Layers",
    layersHint: "Ordered by the recovery path from Gateway to model provider.",
    signalCategories: "Signal Categories",
    signalHint: "Helps separate proxy, provider, and channel failures.",
    eventTrend: "Event Trend",
    eventTrendHint: "Diagnostics, log warnings, and model probes in one timeline.",
    trendDigest: "Trend Digest",
    trendDigestHint: "Turns scattered points into readable conclusions.",
    modelSummary: "Model Probe",
    restartSummary: "Restart Summary",
    latestSignals: "Latest Signals",
    actionCenter: "Action Center",
    quickStrategy: "Quick Strategy",
    strategyHint: "Choose the response posture. Writes require unlock.",
    strategyMatrix: "Strategy Matrix",
    manualActions: "Manual Actions",
    manualHint: "High-risk actions live here.",
    unlock: "Unlock Actions",
    lock: "Lock Actions",
    presetObserve: "Observe",
    presetOvernight: "Overnight Diagnosis",
    presetChannel: "Channel Recovery",
    presetConservative: "Conservative Breaker",
    tipObserve: "Record evidence only. Best when you want to observe first.",
    tipOvernight: "Enable lightweight model probes with log-only actions for fixed-hour timeouts.",
    tipChannel: "Keep channel recovery responsive for session or channel disconnects.",
    tipConservative: "Raise thresholds and reduce restart frequency to prevent restart storms.",
    runDiag: "Run Diagnostics",
    restartGateway: "Restart Gateway",
    configSummary: "Config Summary",
    sensitiveMasked: "Sensitive values are masked",
    stateFiles: "State Files",
    recentLogs: "Recent Logs",
    logHint: "Follows the newest lines by default for overnight timeout checks.",
    followLatest: "Follow latest",
    runtimeMap: "Runtime Map",
    locked: "Read-only",
    actionReady: "Action ready",
    unlocked: "Unlocked",
    missing: "missing",
    noSignals: "No signal categories yet",
    noTimeline: "No trend data yet",
    refreshFailed: "Refresh failed",
    runningDiag: "Running diagnostics...",
    diagDone: "Diagnostics complete",
    restartConfirm: "Restart OpenClaw Gateway?",
    restartDone: "Restart command executed",
    presetDone: "Strategy applied",
    tokenPrompt: "Enter the watchdog dashboard token from DASHBOARD_TOKEN in watchdog.env/json.",
    tokenSaved: "Action token saved in this browser.",
    tokenCleared: "Local token cleared.",
    clearConfirm: "Clear the action token saved in this browser?",
    unlockFirst: "Unlock actions first.",
    statusUnknown: "Collecting watchdog status.",
    apiFail: "API failed",
    logWarn: "Log WARN",
    modelOkLegend: "Model OK",
    diagOk: "Diag OK",
    lastProbe: "Last probe",
    successRate: "Success rate",
    latestReason: "Latest reason",
    noModelHistory: "No model probe history",
    restartWindow: "Window",
    lastRestart: "Last restart",
    restartPolicy: "Restart policy",
    stateDirectory: "State directory",
    configFile: "Config file",
    dashboardPort: "Dashboard port",
    dashboardAction: "Write actions",
    dominantSignal: "Dominant signal",
    evidenceCount: "Evidence count",
    noRecentSignals: "No recent signals",
    enabled: "enabled",
    disabled: "disabled",
    notAvailable: "n/a",
  },
};

let lang = localStorage.getItem("watchdog.lang") || "zh";
let theme = localStorage.getItem("watchdog.theme") || "obsidian";
let activeRoute = localStorage.getItem("watchdog.route") || "overview";
let actionToken = localStorage.getItem("watchdog.actionToken") || window.WATCHDOG_ACTION_TOKEN || "";
let lastStatus = null;

if (!["obsidian", "daylight", "aurora", "graphite"].includes(theme)) {
  theme = "obsidian";
}

function t(key) {
  return DICT[lang]?.[key] || DICT.zh[key] || key;
}

function applyLocale() {
  document.documentElement.lang = lang === "zh" ? "zh-CN" : "en";
  setControlValue($("langSelect"), lang);
  document.querySelectorAll("[data-i18n]").forEach((node) => {
    node.textContent = t(node.dataset.i18n);
  });
  document.querySelectorAll("[data-tip-key]").forEach((node) => {
    node.title = t(node.dataset.tipKey);
  });
  setRoute(activeRoute, { preserveScroll: true });
}

function applyTheme() {
  document.documentElement.dataset.theme = theme;
  setControlValue($("themeSelect"), theme);
}

function setControlValue(node, value) {
  if (!node) return;
  node.value = value;
  node.setAttribute("value", value);
}

function setRoute(route, options = {}) {
  activeRoute = ROUTES.includes(route) ? route : "overview";
  localStorage.setItem("watchdog.route", activeRoute);
  document.querySelectorAll("[data-route]").forEach((node) => {
    node.classList.toggle("active", node.dataset.route === activeRoute);
  });
  document.querySelectorAll("[data-view]").forEach((node) => {
    node.classList.toggle("active", node.dataset.view === activeRoute);
  });
  $("pageTitle").textContent = t(`title${activeRoute[0].toUpperCase()}${activeRoute.slice(1)}`);
  $("pageSubtitle").textContent = t(`subtitle${activeRoute[0].toUpperCase()}${activeRoute.slice(1)}`);
  if (!options.preserveScroll) window.scrollTo({ top: 0, behavior: "auto" });
  if (lastStatus) renderCharts(lastStatus);
}

function toast(message) {
  const node = $("toast");
  node.textContent = message;
  node.hidden = false;
  clearTimeout(toast.timer);
  toast.timer = setTimeout(() => { node.hidden = true; }, 3800);
}

function fmtAge(seconds) {
  if (seconds == null) return t("missing");
  if (seconds < 60) return `${seconds}s`;
  if (seconds < 3600) return `${Math.round(seconds / 60)}m`;
  return `${Math.round(seconds / 3600)}h`;
}

function fmtDate(value) {
  if (!value) return t("notAvailable");
  const ts = typeof value === "number" ? value : Date.parse(value);
  if (!Number.isFinite(ts)) return String(value);
  return new Date(ts).toLocaleString(lang === "zh" ? "zh-CN" : "en-US");
}

function statusClass(status) {
  if (status === "ok") return "ok";
  if (status === "degraded" || status === "stale") return "degraded";
  if (status === "down" || status === "fail") return "fail";
  return "unknown";
}

async function fetchStatus() {
  const res = await fetch("/api/status", { cache: "no-store" });
  if (!res.ok) throw new Error(`status ${res.status}`);
  return res.json();
}

async function postJson(url, body = {}) {
  const res = await fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "X-Watchdog-Token": actionToken,
    },
    body: JSON.stringify(body),
  });
  const data = await res.json().catch(() => ({}));
  if (!res.ok || data.ok === false) throw new Error(data.error || `request failed ${res.status}`);
  return data;
}

function prepareCanvas(canvas, fallbackHeight) {
  const rect = canvas.getBoundingClientRect();
  if (rect.width < 10 || rect.height < 10) return null;
  const width = Math.floor(rect.width);
  const cssHeight = Math.max(140, Math.floor(rect.height || fallbackHeight));
  const ratio = window.devicePixelRatio || 1;
  canvas.width = Math.floor(width * ratio);
  canvas.height = Math.floor(cssHeight * ratio);
  const ctx = canvas.getContext("2d");
  ctx.setTransform(ratio, 0, 0, ratio, 0, 0);
  ctx.clearRect(0, 0, width, cssHeight);
  return { ctx, width, height: cssHeight };
}

function cssVar(name) {
  return getComputedStyle(document.documentElement).getPropertyValue(name).trim();
}

function drawBars(canvas, data) {
  const prepared = prepareCanvas(canvas, 220);
  if (!prepared) return;
  const { ctx, width, height } = prepared;
  const entries = Object.entries(data || {}).slice(0, 8);
  ctx.font = "12px ui-monospace, Consolas, monospace";
  if (!entries.length) {
    ctx.fillStyle = cssVar("--muted");
    ctx.fillText(t("noSignals"), 14, 28);
    return;
  }
  const max = Math.max(...entries.map(([, value]) => value), 1);
  const colors = [cssVar("--accent"), cssVar("--green"), cssVar("--amber"), cssVar("--red"), cssVar("--violet"), cssVar("--accent-2")];
  const labelWidth = Math.min(150, Math.max(108, width * 0.34));
  const rowHeight = Math.max(18, Math.floor((height - 26) / entries.length));
  entries.forEach(([name, value], index) => {
    const y = 20 + index * rowHeight;
    const barWidth = Math.max(4, (width - labelWidth - 48) * (value / max));
    ctx.fillStyle = cssVar("--muted");
    ctx.fillText(name.length > 18 ? `${name.slice(0, 17)}...` : name, 12, y);
    ctx.fillStyle = colors[index % colors.length];
    ctx.fillRect(labelWidth, y - 11, barWidth, 12);
    ctx.fillStyle = cssVar("--ink");
    ctx.fillText(String(value), labelWidth + barWidth + 8, y);
  });
}

function formatTick(ts) {
  return new Date(ts).toLocaleTimeString(lang === "zh" ? "zh-CN" : "en-US", { hour: "2-digit", minute: "2-digit" });
}

function drawTimeline(canvas, diagnostics, modelHistory) {
  const prepared = prepareCanvas(canvas, 300);
  if (!prepared) return;
  const { ctx, width, height } = prepared;
  const points = [];
  for (const item of diagnostics || []) points.push({ ts: Date.parse(item.ts), kind: item.newLogSignals ? "warn" : "ok" });
  for (const item of modelHistory || []) points.push({ ts: Date.parse(item.ts), kind: item.status === "fail" ? "fail" : "model-ok" });
  const valid = points.filter((p) => Number.isFinite(p.ts)).sort((a, b) => a.ts - b.ts).slice(-180);
  const left = 104;
  const right = width - 28;
  const top = 32;
  const axisY = height - 42;
  ctx.font = "12px system-ui";
  ctx.strokeStyle = cssVar("--line");
  ctx.beginPath();
  ctx.moveTo(left, axisY);
  ctx.lineTo(right, axisY);
  ctx.stroke();
  if (!valid.length) {
    ctx.fillStyle = cssVar("--muted");
    ctx.fillText(t("noTimeline"), left, top + 28);
    return;
  }
  const min = valid[0].ts;
  const max = valid[valid.length - 1].ts || min + 1;
  const lanes = [
    ["fail", t("apiFail"), cssVar("--red"), top + 20],
    ["warn", t("logWarn"), cssVar("--amber"), top + 70],
    ["model-ok", t("modelOkLegend"), cssVar("--accent"), top + 120],
    ["ok", t("diagOk"), cssVar("--green"), top + 170],
  ];
  for (const [, label, color, y] of lanes) {
    ctx.fillStyle = color;
    ctx.beginPath();
    ctx.arc(18, y - 4, 4, 0, Math.PI * 2);
    ctx.fill();
    ctx.fillStyle = cssVar("--muted");
    ctx.fillText(label, 32, y);
    ctx.strokeStyle = cssVar("--line");
    ctx.beginPath();
    ctx.moveTo(left, y - 4);
    ctx.lineTo(right, y - 4);
    ctx.stroke();
  }
  const laneMap = Object.fromEntries(lanes.map(([kind, , color, y]) => [kind, { color, y: y - 4 }]));
  valid.forEach((p, index) => {
    const x = left + ((p.ts - min) / Math.max(1, max - min)) * (right - left);
    const lane = laneMap[p.kind] || laneMap.ok;
    ctx.fillStyle = lane.color;
    ctx.beginPath();
    ctx.arc(x, lane.y, index === valid.length - 1 ? 4.6 : 3.3, 0, Math.PI * 2);
    ctx.fill();
  });
  const ticks = [min, min + (max - min) / 2, max];
  ctx.fillStyle = cssVar("--muted");
  ticks.forEach((tick, i) => {
    const x = left + ((tick - min) / Math.max(1, max - min)) * (right - left);
    const label = formatTick(tick);
    const offset = i === 2 ? -44 : i === 1 ? -22 : -8;
    ctx.fillText(label, x + offset, axisY + 22);
  });
}

function renderLayers(data) {
  const files = data.files || {};
  const cfg = data.config || {};
  const layers = [
    ["Gateway", files.gateway, data.summary.status === "ok" ? "ok" : data.summary.status === "unknown" ? "warn" : "fail"],
    ["OpenClaw Health", files.health, data.summary.status === "degraded" ? "fail" : files.health?.exists ? "ok" : "warn"],
    ["Logs Signal", files.signals, data.summary.categoryCount ? "warn" : "ok"],
    ["Model Probe", files.modelHistory, data.summary.modelFailures ? "warn" : files.modelHistory?.exists ? "ok" : "warn"],
    ["Config", { exists: true, ageSeconds: null, path: data.configFile }, cfg.MODEL_PROBE_ENABLED === "1" || cfg.ModelProbeEnabled === true ? "warn" : "ok"],
  ];
  $("layers").innerHTML = layers.map(([name, meta, state]) => `
    <div class="layer">
      <span class="spark ${state}"></span>
      <div><strong>${name}</strong><span class="subtle" title="${meta?.path || ""}">${meta?.path || "runtime snapshot"}</span></div>
      <span class="subtle">${meta?.exists ? fmtAge(meta.ageSeconds) : t("missing")}</span>
    </div>
  `).join("");
}

function renderSummaryList(id, rows) {
  const node = $(id);
  if (!node) return;
  node.innerHTML = rows.map(([label, value]) => `
    <div class="summary-row">
      <span class="subtle">${label}</span>
      <strong>${value}</strong>
    </div>
  `).join("");
}

function renderMiniList(id, rows) {
  const node = $(id);
  if (!node) return;
  const safeRows = rows.length ? rows : [[t("noRecentSignals"), t("notAvailable")]];
  node.innerHTML = safeRows.map(([label, value]) => `
    <div class="mini-row">
      <span class="subtle">${label}</span>
      <strong>${value}</strong>
    </div>
  `).join("");
}

function renderTrendSummaries(data) {
  const history = data.modelHistory || [];
  const latest = history[history.length - 1];
  const modelTotal = (data.summary.modelOk || 0) + (data.summary.modelFailures || 0);
  const successRate = modelTotal ? `${Math.round(((data.summary.modelOk || 0) / modelTotal) * 100)}%` : t("notAvailable");
  const modelRows = [
    [t("lastProbe"), latest ? fmtDate(latest.ts) : t("noModelHistory")],
    [t("successRate"), successRate],
    [t("latestReason"), latest?.reason || latest?.status || t("notAvailable")],
  ];
  const cfg = data.config || {};
  const restartRows = [
    [t("restartWindow"), "24h"],
    [t("recentRestarts"), data.summary.recentRestarts ?? 0],
    [t("restartPolicy"), cfg.OPENCLAW_DIAG_ACTION || cfg.OpenClawDiagAction || t("notAvailable")],
  ];
  renderSummaryList("modelSummary", modelRows);
  renderSummaryList("overviewModelSummary", modelRows);
  renderSummaryList("restartSummary", restartRows);
  renderSummaryList("overviewRestartSummary", restartRows);
  const categories = Object.entries(data.categories || {}).sort((a, b) => b[1] - a[1]);
  renderMiniList("categoryTopList", categories.slice(0, 4));
  renderMiniList("latestSignals", categories.slice(0, 5));
  renderMiniList("logSignalList", categories.slice(0, 10));
  const dominant = categories[0]?.[0] || t("notAvailable");
  const evidenceCount = (data.diagnostics || []).length + (data.modelHistory || []).length;
  renderMiniList("trendDigest", [
    [t("dominantSignal"), dominant],
    [t("evidenceCount"), evidenceCount],
    [t("modelFail"), data.summary.modelFailures ?? 0],
    [t("recentRestarts"), data.summary.recentRestarts ?? 0],
  ]);
  renderStrategyConfig(data);
  renderStrategyMatrix();
  renderRuntimeMap(data);
}

function renderStrategyConfig(data) {
  const cfg = data.config || {};
  renderSummaryList("strategyConfigSummary", [
    ["MODEL_PROBE", cfg.MODEL_PROBE_ENABLED === "1" || cfg.ModelProbeEnabled === true ? t("enabled") : t("disabled")],
    ["OPENCLAW_DIAG_ACTION", cfg.OPENCLAW_DIAG_ACTION || cfg.OpenClawDiagAction || t("notAvailable")],
    ["MAX_RESTARTS_PER_HOUR", cfg.MAX_RESTARTS_PER_HOUR || cfg.MaxRestartsPerHour || t("notAvailable")],
    [t("dashboardAction"), data.dashboard?.actionsEnabled ? t("enabled") : t("disabled")],
  ]);
}

function renderStrategyMatrix() {
  const rows = [
    [t("presetObserve"), t("tipObserve")],
    [t("presetOvernight"), t("tipOvernight")],
    [t("presetChannel"), t("tipChannel")],
    [t("presetConservative"), t("tipConservative")],
  ];
  const node = $("strategyMatrix");
  if (!node) return;
  node.innerHTML = rows.map(([title, detail]) => `
    <div class="matrix-cell">
      <strong>${title}</strong>
      <span>${detail}</span>
    </div>
  `).join("");
}

function renderRuntimeMap(data) {
  const node = $("runtimeMap");
  if (!node) return;
  const rows = [
    [t("stateDirectory"), data.stateDir || t("notAvailable")],
    [t("configFile"), data.configFile || t("notAvailable")],
    [t("dashboardPort"), data.dashboard?.port || t("notAvailable")],
    [t("dashboardAction"), data.dashboard?.actionsEnabled ? t("enabled") : t("disabled")],
  ];
  node.innerHTML = rows.map(([title, detail]) => `
    <div class="runtime-cell">
      <strong>${title}</strong>
      <span>${detail}</span>
    </div>
  `).join("");
}

function renderConfig(config) {
  const keys = [
    "OPENCLAW_DIAG_ACTION", "MODEL_PROBE_ENABLED", "MODEL_PROBE_INTERVAL",
    "MODEL_PROBE_ACTION", "CHANNEL_FAILURES_BEFORE_RESTART", "MAX_RESTARTS_PER_HOUR",
    "DASHBOARD_ACTIONS_ENABLED", "DashboardActionsEnabled", "DASHBOARD_PORT", "DashboardPort",
  ];
  const rows = [];
  for (const key of keys) if (Object.prototype.hasOwnProperty.call(config, key)) rows.push([key, config[key]]);
  $("configList").innerHTML = rows.map(([key, value]) => `
    <div class="config-row">
      <dt title="${key}">${key}</dt>
      <dd>${String(value)}</dd>
    </div>
  `).join("");
}

function renderFiles(files) {
  $("fileList").innerHTML = Object.entries(files || {}).map(([name, meta]) => `
    <div class="file-row">
      <strong>${name}</strong>
      <span class="subtle">${meta.exists ? `${fmtAge(meta.ageSeconds)} / ${Math.round((meta.size || 0) / 1024)}KB` : t("missing")}</span>
    </div>
  `).join("");
}

function updateActionUi(data) {
  const serverReady = Boolean(data.dashboard?.actionsEnabled);
  const unlocked = Boolean(actionToken);
  const label = serverReady ? (unlocked ? t("unlocked") : t("actionReady")) : t("locked");
  const className = serverReady ? (unlocked ? "status-chip unlocked" : "status-chip enabled") : "status-chip";
  for (const id of ["actionState", "strategyActionState"]) {
    const node = $(id);
    if (!node) continue;
    node.textContent = label;
    node.className = className;
  }
  for (const id of ["unlockBtn", "strategyUnlockBtn"]) {
    const node = $(id);
    if (node) node.textContent = unlocked ? t("lock") : t("unlock");
  }
  const disabled = !serverReady || !unlocked;
  document.querySelectorAll("[data-preset], #diagBtn, #restartBtn, #heroDiagBtn, #heroRestartBtn").forEach((node) => {
    node.classList.toggle("guarded-disabled", disabled);
    node.setAttribute("aria-disabled", String(disabled));
  });
}

function canRunActions() {
  if (!lastStatus?.dashboard?.actionsEnabled || !actionToken) {
    toast(t("unlockFirst"));
    return false;
  }
  return true;
}

function renderCharts(data) {
  if (activeRoute === "overview") drawBars($("categoryChart"), data.categories || {});
  if (activeRoute === "trends") drawTimeline($("timelineChart"), data.diagnostics || [], data.modelHistory || []);
}

function render(data) {
  lastStatus = data;
  const status = data.summary.status || "unknown";
  $("mainStatus").className = `status-orb ${statusClass(status)}`;
  $("sideStatusDot").className = `mini-led ${statusClass(status)}`;
  $("sideStatusText").textContent = status.toUpperCase();
  $("statusText").textContent = status.toUpperCase();
  $("statusReason").textContent = data.summary.reason || t("statusUnknown");
  $("restartCount").textContent = data.summary.recentRestarts ?? 0;
  $("categoryCount").textContent = data.summary.categoryCount ?? 0;
  $("modelOk").textContent = data.summary.modelOk ?? 0;
  $("modelFail").textContent = data.summary.modelFailures ?? 0;
  $("updatedAt").textContent = fmtDate(data.generatedAt);
  $("sideUpdatedAt").textContent = fmtDate(data.generatedAt);
  const modelTotal = (data.summary.modelOk || 0) + (data.summary.modelFailures || 0);
  $("modelRate").textContent = modelTotal ? `${Math.round(((data.summary.modelOk || 0) / modelTotal) * 100)}%` : "--";
  $("modelFailHint").textContent = data.summary.modelFailures ? "check provider" : "provider";
  updateActionUi(data);
  renderLayers(data);
  renderTrendSummaries(data);
  renderConfig(data.config || {});
  renderFiles(data.files || {});
  const log = $("logTail");
  const shouldFollow = $("followLog").checked;
  log.textContent = (data.logTail || []).join("\n") || "No logs yet.";
  if (shouldFollow) log.scrollTop = log.scrollHeight;
  renderCharts(data);
}

async function refresh() {
  try {
    render(await fetchStatus());
  } catch (err) {
    toast(`${t("refreshFailed")}: ${err.message}`);
  }
}

function saveSettings() {
  localStorage.setItem("watchdog.lang", lang);
  localStorage.setItem("watchdog.theme", theme);
}

document.querySelectorAll("[data-route]").forEach((button) => {
  button.addEventListener("click", () => setRoute(button.dataset.route));
});

$("langSelect").addEventListener("change", () => {
  lang = $("langSelect").value;
  saveSettings();
  applyLocale();
  if (lastStatus) render(lastStatus);
});

$("themeSelect").addEventListener("change", () => {
  theme = $("themeSelect").value;
  saveSettings();
  applyTheme();
  if (lastStatus) render(lastStatus);
});

$("refreshBtn").addEventListener("click", refresh);
$("unlockBtn").addEventListener("click", () => {
  if (actionToken) {
    if (confirm(t("clearConfirm"))) {
      actionToken = "";
      localStorage.removeItem("watchdog.actionToken");
      toast(t("tokenCleared"));
      if (lastStatus) render(lastStatus);
    }
    return;
  }
  const entered = prompt(t("tokenPrompt"), "");
  if (entered && entered.trim()) {
    actionToken = entered.trim();
    localStorage.setItem("watchdog.actionToken", actionToken);
    toast(t("tokenSaved"));
    if (lastStatus) render(lastStatus);
  }
});

$("strategyUnlockBtn").addEventListener("click", () => $("unlockBtn").click());

$("exportBtn").addEventListener("click", async () => {
  const data = lastStatus || await fetchStatus();
  const blob = new Blob([JSON.stringify(data, null, 2)], { type: "application/json" });
  const url = URL.createObjectURL(blob);
  const a = document.createElement("a");
  a.href = url;
  a.download = `openclaw-watchdog-${Date.now()}.json`;
  a.click();
  URL.revokeObjectURL(url);
});

$("diagBtn").addEventListener("click", async () => {
  if (!canRunActions()) return;
  try {
    toast(t("runningDiag"));
    await postJson("/api/actions/run-diagnostics");
    await refresh();
    toast(t("diagDone"));
  } catch (err) {
    toast(err.message);
  }
});

$("heroDiagBtn").addEventListener("click", () => $("diagBtn").click());

$("restartBtn").addEventListener("click", async () => {
  if (!canRunActions()) return;
  if (!confirm(t("restartConfirm"))) return;
  try {
    await postJson("/api/actions/restart-gateway");
    await refresh();
    toast(t("restartDone"));
  } catch (err) {
    toast(err.message);
  }
});

$("heroRestartBtn").addEventListener("click", () => $("restartBtn").click());

document.querySelectorAll("[data-preset]").forEach((button) => {
  button.addEventListener("click", async () => {
    if (!canRunActions()) return;
    try {
      await postJson("/api/config/preset", { preset: button.dataset.preset });
      await refresh();
      toast(`${t("presetDone")}: ${button.querySelector("strong")?.textContent || button.textContent}`);
    } catch (err) {
      toast(err.message);
    }
  });
});

window.addEventListener("resize", () => {
  if (lastStatus) renderCharts(lastStatus);
});

applyTheme();
applyLocale();
refresh();
setInterval(refresh, 15000);


path: dashboard/static/index.html
content: <!doctype html>
<html lang="zh-CN" data-theme="obsidian">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>OpenClaw Resilience Guard</title>
  <link rel="stylesheet" href="/styles.css">
</head>
<body>
  <div class="ambient ambient-a"></div>
  <div class="ambient ambient-b"></div>

  <div class="app-shell">
    <aside class="sidebar">
      <div class="brand">
        <span class="brand-mark" aria-hidden="true"></span>
        <div>
          <strong>Resilience Guard</strong>
          <span>OpenClaw</span>
        </div>
      </div>

      <nav class="nav" aria-label="Dashboard">
        <button type="button" class="nav-item active" data-route="overview">
          <span class="nav-icon">⌁</span><span data-i18n="navOverview">总览</span>
        </button>
        <button type="button" class="nav-item" data-route="trends">
          <span class="nav-icon">⌇</span><span data-i18n="navTrends">趋势</span>
        </button>
        <button type="button" class="nav-item" data-route="strategy">
          <span class="nav-icon">◇</span><span data-i18n="navStrategy">策略</span>
        </button>
        <button type="button" class="nav-item" data-route="logs">
          <span class="nav-icon">≣</span><span data-i18n="navLogs">日志</span>
        </button>
        <button type="button" class="nav-item" data-route="settings">
          <span class="nav-icon">⚙</span><span data-i18n="navSettings">配置</span>
        </button>
      </nav>

      <div class="side-status">
        <span id="sideStatusDot" class="mini-led unknown"></span>
        <div>
          <strong id="sideStatusText">SYNC</strong>
          <span id="sideUpdatedAt">--</span>
        </div>
      </div>
    </aside>

    <main class="workspace">
      <header class="topbar">
        <div class="title-block">
          <span class="eyebrow">OPENCLAW WATCHDOG</span>
          <h1 id="pageTitle">恢复控制台</h1>
          <p id="pageSubtitle" data-i18n="subtitleOverview">把 Gateway、通道、日志和模型探针合成一个恢复判断。</p>
        </div>
        <div class="top-actions">
          <select id="langSelect" class="ui-select" title="Language">
            <option value="zh">中文</option>
            <option value="en">English</option>
          </select>
          <select id="themeSelect" class="ui-select" title="Theme">
            <option value="obsidian">Obsidian</option>
            <option value="daylight">Daylight</option>
            <option value="aurora">Aurora</option>
            <option value="graphite">Graphite</option>
          </select>
          <button type="button" id="refreshBtn" class="ui-button icon-btn" title="Refresh">↻</button>
          <button type="button" id="exportBtn" class="ui-button primary" data-i18n="export">导出诊断</button>
        </div>
      </header>

      <section class="view active" data-view="overview">
        <div class="overview-grid">
          <article class="hero-panel">
            <div class="hero-signal">
              <span id="mainStatus" class="status-orb unknown"></span>
              <span class="scan-ring"></span>
            </div>
            <div class="hero-copy">
              <span class="label" data-i18n="currentDecision">当前判断</span>
              <strong id="statusText">SYNC</strong>
              <p id="statusReason">正在汇总 watchdog 状态。</p>
            </div>
            <div class="hero-actions">
              <button type="button" id="heroDiagBtn" class="ui-button" data-i18n="runDiag">立即诊断</button>
              <button type="button" id="heroRestartBtn" class="ui-button primary" data-i18n="restartGateway">重启 Gateway</button>
            </div>
          </article>

          <article class="metric-card metric-a">
            <span data-i18n="recentRestarts">近期重启</span>
            <strong id="restartCount">0</strong>
            <small id="restartHint">24h</small>
          </article>
          <article class="metric-card metric-b">
            <span data-i18n="signalTypes">异常类别</span>
            <strong id="categoryCount">0</strong>
            <small>OpenClaw logs</small>
          </article>
          <article class="metric-card metric-c">
            <span data-i18n="modelOk">模型成功</span>
            <strong id="modelOk">0</strong>
            <small id="modelRate">--</small>
          </article>
          <article class="metric-card metric-d">
            <span data-i18n="modelFail">模型失败</span>
            <strong id="modelFail">0</strong>
            <small id="modelFailHint">provider</small>
          </article>
        </div>

        <div class="dashboard-grid">
          <article class="panel span-7 tall-panel">
            <div class="panel-head">
              <div><h2 data-i18n="layers">链路分层</h2><p data-i18n="layersHint">按恢复路径从 Gateway 到模型 provider 排列。</p></div>
              <span id="updatedAt" class="badge">--</span>
            </div>
            <div id="layers" class="layer-table"></div>
          </article>

          <article class="panel span-5 tall-panel">
            <div class="panel-head">
              <div><h2 data-i18n="signalCategories">异常分类</h2><p data-i18n="signalHint">用于判断是代理、provider 还是通道层问题。</p></div>
            </div>
            <canvas id="categoryChart" class="chart chart-bars"></canvas>
            <div id="categoryTopList" class="mini-list"></div>
          </article>

          <article class="panel span-4">
            <div class="panel-head compact"><h2 data-i18n="modelSummary">模型探针</h2></div>
            <div id="overviewModelSummary" class="summary-list"></div>
          </article>
          <article class="panel span-4">
            <div class="panel-head compact"><h2 data-i18n="restartSummary">重启统计</h2></div>
            <div id="overviewRestartSummary" class="summary-list"></div>
          </article>
          <article class="panel span-4">
            <div class="panel-head compact"><h2 data-i18n="actionCenter">处置入口</h2></div>
            <div class="action-stack">
              <button type="button" id="unlockBtn" class="ui-button" data-i18n="unlock">解锁操作</button>
              <span id="actionState" class="status-chip">只读</span>
            </div>
          </article>
        </div>
      </section>

      <section class="view" data-view="trends">
        <div class="dashboard-grid">
          <article class="panel span-8 tall-panel">
            <div class="panel-head">
              <div><h2 data-i18n="eventTrend">事件趋势</h2><p data-i18n="eventTrendHint">诊断、日志告警和模型探针的同屏时间线。</p></div>
              <span class="badge accent">live</span>
            </div>
            <canvas id="timelineChart" class="chart chart-timeline"></canvas>
          </article>
          <article class="panel span-4 tall-panel">
            <div class="panel-head"><div><h2 data-i18n="trendDigest">趋势摘要</h2><p data-i18n="trendDigestHint">把散点变成可读结论。</p></div></div>
            <div id="trendDigest" class="digest"></div>
          </article>
          <article class="panel span-4">
            <div class="panel-head compact"><h2 data-i18n="modelSummary">模型探针</h2></div>
            <div id="modelSummary" class="summary-list"></div>
          </article>
          <article class="panel span-4">
            <div class="panel-head compact"><h2 data-i18n="restartSummary">重启统计</h2></div>
            <div id="restartSummary" class="summary-list"></div>
          </article>
          <article class="panel span-4">
            <div class="panel-head compact"><h2 data-i18n="latestSignals">最新信号</h2></div>
            <div id="latestSignals" class="mini-list"></div>
          </article>
        </div>
      </section>

      <section class="view" data-view="strategy">
        <div class="dashboard-grid">
          <article class="panel span-8 tall-panel">
            <div class="panel-head">
              <div><h2 data-i18n="quickStrategy">快速策略</h2><p data-i18n="strategyHint">选择故障处理姿态，所有写入动作都需要先解锁。</p></div>
              <span id="strategyActionState" class="status-chip">只读</span>
            </div>
            <div class="preset-grid">
              <button type="button" class="preset-card" data-preset="observe" data-tip-key="tipObserve">
                <strong data-i18n="presetObserve">观察模式</strong>
                <span data-i18n="tipObserve">只记录证据，不主动改变策略，适合先观察问题。</span>
              </button>
              <button type="button" class="preset-card" data-preset="overnight" data-tip-key="tipOvernight">
                <strong data-i18n="presetOvernight">夜间诊断</strong>
                <span data-i18n="tipOvernight">开启轻量模型探针并保持 log-only，适合排查固定时段超时。</span>
              </button>
              <button type="button" class="preset-card" data-preset="channel-recovery" data-tip-key="tipChannel">
                <strong data-i18n="presetChannel">通道恢复</strong>
                <span data-i18n="tipChannel">保持通道恢复敏感度，适合通道/session 经常断开的场景。</span>
              </button>
              <button type="button" class="preset-card" data-preset="conservative" data-tip-key="tipConservative">
                <strong data-i18n="presetConservative">保守熔断</strong>
                <span data-i18n="tipConservative">提高阈值并降低重启频率，适合网络不稳定时防止重启风暴。</span>
              </button>
            </div>
          </article>
          <article class="panel span-4 tall-panel">
            <div class="panel-head"><div><h2 data-i18n="manualActions">手动操作</h2><p data-i18n="manualHint">高风险动作集中放在这里。</p></div></div>
            <div class="action-stack">
              <button type="button" id="strategyUnlockBtn" class="ui-button" data-i18n="unlock">解锁操作</button>
              <button type="button" id="diagBtn" class="ui-button" data-i18n="runDiag">立即诊断</button>
              <button type="button" id="restartBtn" class="ui-button primary" data-i18n="restartGateway">重启 Gateway</button>
            </div>
            <div id="strategyConfigSummary" class="summary-list strategy-summary"></div>
          </article>
          <article class="panel span-12">
            <div class="panel-head compact"><h2 data-i18n="strategyMatrix">策略矩阵</h2></div>
            <div id="strategyMatrix" class="matrix-grid"></div>
          </article>
        </div>
      </section>

      <section class="view" data-view="logs">
        <div class="dashboard-grid">
          <article class="panel span-8 log-panel">
            <div class="panel-head">
              <div><h2 data-i18n="recentLogs">最近日志</h2><p data-i18n="logHint">默认跟随最新行，适合观察夜间超时。</p></div>
              <label class="toggle"><input id="followLog" type="checkbox" checked><span data-i18n="followLatest">保持最新</span></label>
            </div>
            <pre id="logTail" class="log"></pre>
          </article>
          <article class="panel span-4 log-panel">
            <div class="panel-head"><div><h2 data-i18n="latestSignals">最新信号</h2><p data-i18n="signalHint">异常关键词的聚合视图。</p></div></div>
            <div id="logSignalList" class="mini-list large"></div>
          </article>
        </div>
      </section>

      <section class="view" data-view="settings">
        <div class="dashboard-grid">
          <article class="panel span-6 tall-panel">
            <div class="panel-head">
              <div><h2 data-i18n="configSummary">配置摘要</h2><p data-i18n="sensitiveMasked">敏感项已遮盖</p></div>
            </div>
            <dl id="configList" class="config-list"></dl>
          </article>
          <article class="panel span-6 tall-panel">
            <div class="panel-head">
              <div><h2 data-i18n="stateFiles">状态文件</h2><p>mtime / size</p></div>
            </div>
            <div id="fileList" class="file-list"></div>
          </article>
          <article class="panel span-12">
            <div class="panel-head compact"><h2 data-i18n="runtimeMap">运行地图</h2></div>
            <div id="runtimeMap" class="runtime-map"></div>
          </article>
        </div>
      </section>
    </main>
  </div>

  <div id="toast" class="toast" hidden></div>
  <script src="/app.js"></script>
</body>
</html>


path: dashboard/static/styles.css
content: :root {
  color-scheme: dark;
  --bg: #080b10;
  --bg-2: #101620;
  --rail: rgba(13, 18, 27, .82);
  --panel: rgba(18, 24, 35, .78);
  --panel-solid: #131a25;
  --panel-2: rgba(255,255,255,.06);
  --ink: #f3f7fb;
  --muted: #8fa0b5;
  --line: rgba(151, 170, 192, .18);
  --line-strong: rgba(151, 170, 192, .34);
  --accent: #69e4ff;
  --accent-2: #7c5cff;
  --green: #39d98a;
  --amber: #ffbd4a;
  --red: #ff5d73;
  --violet: #9a7cff;
  --shadow: 0 24px 70px rgba(0, 0, 0, .32);
  --radius: 18px;
  --font: "Segoe UI Variable", "SF Pro Display", "Segoe UI", ui-sans-serif, system-ui, sans-serif;
  --mono: "Cascadia Code", "SFMono-Regular", Consolas, monospace;
  --accent-fill-rest: #6ee7ff;
  --accent-fill-hover: #94efff;
  --accent-fill-active: #39d2f2;
  --neutral-fill-rest: rgba(255,255,255,.06);
  --neutral-fill-hover: rgba(255,255,255,.1);
  --neutral-fill-active: rgba(255,255,255,.14);
  font-family: var(--font);
}

:root[data-theme="daylight"] {
  color-scheme: light;
  --bg: #eef3f8;
  --bg-2: #f8fbff;
  --rail: rgba(255,255,255,.74);
  --panel: rgba(255,255,255,.84);
  --panel-solid: #ffffff;
  --panel-2: rgba(24, 34, 48, .045);
  --ink: #111827;
  --muted: #64748b;
  --line: rgba(94, 111, 134, .22);
  --line-strong: rgba(94, 111, 134, .38);
  --accent: #2563eb;
  --accent-2: #0f9f86;
  --green: #138a5b;
  --amber: #b76a00;
  --red: #cb3347;
  --shadow: 0 24px 70px rgba(25, 38, 60, .12);
  --accent-fill-rest: #2563eb;
  --accent-fill-hover: #1d4ed8;
  --accent-fill-active: #1e40af;
}

:root[data-theme="aurora"] {
  --bg: #090e16;
  --bg-2: #0e1724;
  --rail: rgba(6, 18, 27, .78);
  --panel: rgba(10, 28, 39, .76);
  --panel-solid: #0d202d;
  --accent: #39f0b0;
  --accent-2: #5aa7ff;
  --violet: #64d7ff;
}

:root[data-theme="graphite"] {
  --bg: #0f1113;
  --bg-2: #171a1d;
  --rail: rgba(18, 18, 19, .84);
  --panel: rgba(26, 28, 31, .8);
  --panel-solid: #1a1d20;
  --accent: #d9ff66;
  --accent-2: #8b949e;
  --violet: #d9ff66;
}

* { box-sizing: border-box; }
html { min-height: 100%; }
body {
  min-height: 100%;
  margin: 0;
  overflow-x: hidden;
  background:
    radial-gradient(circle at 76% 8%, color-mix(in srgb, var(--accent) 24%, transparent), transparent 30vw),
    radial-gradient(circle at 18% 82%, color-mix(in srgb, var(--accent-2) 18%, transparent), transparent 34vw),
    linear-gradient(135deg, var(--bg), var(--bg-2));
  color: var(--ink);
}

.ambient {
  position: fixed;
  pointer-events: none;
  z-index: 0;
  filter: blur(50px);
  opacity: .42;
  transform: translateZ(0);
  animation: drift 16s ease-in-out infinite alternate;
}
.ambient-a {
  width: 360px;
  height: 360px;
  top: -80px;
  right: 10vw;
  background: color-mix(in srgb, var(--accent) 55%, transparent);
}
.ambient-b {
  width: 320px;
  height: 320px;
  bottom: 10vh;
  left: 18vw;
  background: color-mix(in srgb, var(--accent-2) 45%, transparent);
  animation-delay: -4s;
}
@keyframes drift {
  from { transform: translate3d(-22px, 8px, 0) scale(.96); }
  to { transform: translate3d(18px, -16px, 0) scale(1.08); }
}

button, select {
  color: var(--ink);
  font: inherit;
}

.app-shell {
  position: relative;
  z-index: 1;
  display: grid;
  grid-template-columns: 252px minmax(0, 1fr);
  min-height: 100vh;
}

.sidebar {
  position: sticky;
  top: 0;
  height: 100vh;
  padding: 24px 18px;
  border-right: 1px solid var(--line);
  background: var(--rail);
  backdrop-filter: blur(22px);
}

.brand {
  display: flex;
  align-items: center;
  gap: 13px;
  min-height: 54px;
  margin-bottom: 28px;
}
.brand-mark {
  display: grid;
  place-items: center;
  width: 46px;
  height: 46px;
  border-radius: 14px;
  background:
    linear-gradient(135deg, color-mix(in srgb, var(--accent) 90%, #fff 8%), var(--accent-2));
  color: #061018;
  font-size: 13px;
  font-weight: 900;
  letter-spacing: .04em;
  box-shadow: 0 16px 36px color-mix(in srgb, var(--accent) 22%, transparent);
}
.brand strong, .brand span { display: block; }
.brand strong { font-size: 16px; }
.brand span { margin-top: 3px; color: var(--muted); font-size: 12px; }

.nav {
  display: grid;
  gap: 9px;
}
.nav-item {
  width: 100%;
  min-height: 46px;
}
.nav-item::part(control) {
  justify-content: flex-start;
  width: 100%;
  padding: 0 14px;
  border: 1px solid transparent;
  background: transparent;
  color: var(--muted);
}
.nav-item.active::part(control),
.nav-item:hover::part(control) {
  border-color: var(--line);
  background:
    linear-gradient(135deg, color-mix(in srgb, var(--accent) 12%, transparent), transparent),
    var(--panel-2);
  color: var(--ink);
}
.nav-icon {
  display: inline-grid;
  place-items: center;
  width: 24px;
  margin-right: 10px;
  color: var(--accent);
  font-family: var(--mono);
}

.side-status {
  position: absolute;
  right: 18px;
  bottom: 22px;
  left: 18px;
  display: flex;
  align-items: center;
  gap: 10px;
  min-height: 58px;
  padding: 12px;
  border: 1px solid var(--line);
  border-radius: 16px;
  background: color-mix(in srgb, var(--panel) 84%, transparent);
}
.side-status strong, .side-status span { display: block; }
.side-status strong { font-size: 13px; }
.side-status span { color: var(--muted); font-size: 12px; }

.workspace {
  width: min(1320px, calc(100vw - 300px));
  margin: 0 auto;
  padding: 32px 28px 46px;
}

.topbar {
  display: flex;
  justify-content: space-between;
  align-items: flex-start;
  gap: 28px;
  margin-bottom: 24px;
  animation: rise .45s ease both;
}
.title-block {
  max-width: 720px;
}
.eyebrow {
  display: inline-flex;
  align-items: center;
  gap: 8px;
  margin-bottom: 8px;
  color: var(--accent);
  font-size: 12px;
  font-weight: 850;
  letter-spacing: .08em;
}
.eyebrow::before {
  content: "";
  width: 26px;
  height: 1px;
  background: var(--accent);
}
h1, h2, h3, p { margin: 0; letter-spacing: 0; }
h1 {
  font-size: clamp(34px, 4.1vw, 58px);
  line-height: .96;
  font-weight: 850;
}
.title-block p {
  max-width: 60ch;
  margin-top: 12px;
  color: var(--muted);
  font-size: 15px;
  line-height: 1.6;
}
h2 { font-size: 17px; line-height: 1.2; }
.top-actions {
  display: flex;
  flex-wrap: wrap;
  justify-content: flex-end;
  gap: 10px;
}
.top-actions .ui-select { min-width: 132px; }
.icon-btn { width: 42px; }

.view { display: none; }
.view.active {
  display: block;
  animation: rise .38s ease both;
}
@keyframes rise {
  from { opacity: 0; transform: translateY(12px); }
  to { opacity: 1; transform: translateY(0); }
}

.overview-grid {
  display: grid;
  grid-template-columns: minmax(360px, 1.65fr) repeat(4, minmax(130px, .55fr));
  gap: 14px;
  margin-bottom: 18px;
}
.hero-panel, .metric-card, .panel {
  position: relative;
  overflow: hidden;
  border: 1px solid var(--line);
  border-radius: var(--radius);
  background:
    linear-gradient(145deg, rgba(255,255,255,.09), transparent 46%),
    var(--panel);
  box-shadow: var(--shadow);
  backdrop-filter: blur(18px);
}
.hero-panel::after, .panel::after, .metric-card::after {
  content: "";
  position: absolute;
  inset: 0;
  pointer-events: none;
  background: linear-gradient(110deg, transparent 0 42%, rgba(255,255,255,.08) 48%, transparent 56% 100%);
  opacity: 0;
  transform: translateX(-60%);
  transition: opacity .2s ease, transform .65s ease;
}
.hero-panel:hover::after, .panel:hover::after, .metric-card:hover::after {
  opacity: 1;
  transform: translateX(60%);
}

.hero-panel {
  display: grid;
  grid-template-columns: 94px minmax(0, 1fr) auto;
  align-items: center;
  gap: 18px;
  min-height: 150px;
  padding: 26px;
}
.hero-signal {
  position: relative;
  display: grid;
  place-items: center;
  width: 82px;
  height: 82px;
}
.status-orb {
  position: relative;
  z-index: 2;
  width: 58px;
  height: 58px;
  border-radius: 50%;
  background: var(--muted);
  box-shadow: 0 0 0 10px color-mix(in srgb, var(--muted) 16%, transparent);
}
.scan-ring {
  position: absolute;
  inset: 2px;
  border: 1px solid color-mix(in srgb, var(--accent) 45%, transparent);
  border-radius: 50%;
  animation: pulse 2.4s ease-out infinite;
}
@keyframes pulse {
  0% { transform: scale(.72); opacity: .8; }
  100% { transform: scale(1.18); opacity: 0; }
}
.status-orb.ok, .mini-led.ok { background: var(--green); box-shadow: 0 0 30px color-mix(in srgb, var(--green) 40%, transparent); }
.status-orb.degraded, .status-orb.stale, .mini-led.degraded, .mini-led.stale { background: var(--amber); box-shadow: 0 0 30px color-mix(in srgb, var(--amber) 38%, transparent); }
.status-orb.down, .status-orb.fail, .mini-led.down, .mini-led.fail { background: var(--red); box-shadow: 0 0 30px color-mix(in srgb, var(--red) 38%, transparent); }
.label {
  color: var(--muted);
  font-size: 12px;
  font-weight: 750;
  text-transform: uppercase;
}
.hero-copy strong {
  display: block;
  margin: 2px 0 6px;
  font-size: 44px;
  line-height: 1;
}
.hero-copy p { color: var(--muted); line-height: 1.5; }
.hero-actions {
  display: grid;
  gap: 10px;
  min-width: 144px;
}

.metric-card {
  display: grid;
  align-content: space-between;
  min-height: 150px;
  padding: 20px;
}
.metric-card span {
  color: var(--muted);
  font-size: 12px;
  font-weight: 780;
}
.metric-card strong {
  margin-top: 14px;
  font-size: 42px;
  line-height: 1;
}
.metric-card small {
  color: color-mix(in srgb, var(--accent) 80%, var(--muted));
  font-size: 12px;
}

.dashboard-grid {
  display: grid;
  grid-template-columns: repeat(12, minmax(0, 1fr));
  gap: 18px;
  align-items: stretch;
}
.span-4 { grid-column: span 4; }
.span-5 { grid-column: span 5; }
.span-6 { grid-column: span 6; }
.span-7 { grid-column: span 7; }
.span-8 { grid-column: span 8; }
.span-12 { grid-column: span 12; }
.panel {
  min-width: 0;
  min-height: 196px;
  padding: 20px;
}
.tall-panel { min-height: 410px; }
.panel-head {
  position: relative;
  z-index: 1;
  display: flex;
  align-items: flex-start;
  justify-content: space-between;
  gap: 16px;
  margin-bottom: 16px;
}
.panel-head.compact {
  align-items: center;
  margin-bottom: 12px;
}
.panel-head p {
  margin-top: 6px;
  color: var(--muted);
  font-size: 12px;
  line-height: 1.45;
}

.layer-table {
  position: relative;
  z-index: 1;
  display: grid;
  gap: 11px;
}
.layer {
  display: grid;
  grid-template-columns: 18px minmax(0, 1fr) 64px;
  align-items: center;
  gap: 12px;
  min-height: 62px;
  padding: 12px 14px;
  border: 1px solid var(--line);
  border-radius: 14px;
  background: color-mix(in srgb, var(--panel-solid) 70%, transparent);
  transition: transform .16s ease, border-color .16s ease, background .16s ease;
}
.layer:hover {
  transform: translateX(3px);
  border-color: var(--line-strong);
  background: color-mix(in srgb, var(--panel-solid) 90%, transparent);
}
.layer strong { display: block; font-size: 15px; line-height: 1.12; }
.subtle {
  color: var(--muted);
  font-size: 12px;
}
.layer .subtle {
  display: block;
  overflow: hidden;
  text-overflow: ellipsis;
  white-space: nowrap;
}
.spark, .mini-led {
  width: 10px;
  height: 10px;
  border-radius: 999px;
  background: var(--muted);
}
.mini-led { width: 12px; height: 12px; flex: 0 0 auto; }
.spark.ok { background: var(--green); box-shadow: 0 0 18px color-mix(in srgb, var(--green) 44%, transparent); }
.spark.warn { background: var(--amber); box-shadow: 0 0 18px color-mix(in srgb, var(--amber) 40%, transparent); }
.spark.fail { background: var(--red); box-shadow: 0 0 18px color-mix(in srgb, var(--red) 40%, transparent); }

.chart {
  position: relative;
  z-index: 1;
  display: block;
  width: 100%;
  border: 1px solid var(--line);
  border-radius: 16px;
  background:
    linear-gradient(180deg, rgba(255,255,255,.06), transparent),
    color-mix(in srgb, var(--panel-solid) 78%, transparent);
}
.chart-bars { height: 236px; }
.chart-timeline { height: 338px; }

.summary-list, .file-list, .mini-list, .digest {
  position: relative;
  z-index: 1;
  display: grid;
  gap: 10px;
}
.summary-row, .file-row, .mini-row, .digest-row {
  display: flex;
  align-items: center;
  justify-content: space-between;
  gap: 14px;
  min-height: 42px;
  padding: 10px 0;
  border-bottom: 1px solid var(--line);
}
.summary-row:last-child, .file-row:last-child, .mini-row:last-child, .digest-row:last-child { border-bottom: 0; }
.summary-row strong, .mini-row strong, .digest-row strong {
  text-align: right;
  font-size: 15px;
}

.preset-grid {
  position: relative;
  z-index: 1;
  display: grid;
  grid-template-columns: repeat(2, minmax(0, 1fr));
  gap: 14px;
}
.preset-card {
  min-height: 154px;
}
.preset-card::part(control) {
  width: 100%;
  height: 100%;
  justify-content: center;
  padding: 22px;
  border-color: var(--line);
  background:
    radial-gradient(circle at 24% 12%, color-mix(in srgb, var(--accent) 13%, transparent), transparent 42%),
    color-mix(in srgb, var(--panel-solid) 78%, transparent);
}
.preset-card:hover::part(control) {
  border-color: color-mix(in srgb, var(--accent) 55%, var(--line));
}
.preset-card strong,
.preset-card span {
  display: block;
  width: 100%;
  text-align: center;
  white-space: normal;
}
.preset-card strong {
  margin-bottom: 8px;
  color: var(--ink);
  font-size: 17px;
}
.preset-card span {
  max-width: 38ch;
  color: var(--muted);
  font-size: 12px;
  line-height: 1.55;
}
.action-stack {
  position: relative;
  z-index: 1;
  display: grid;
  gap: 12px;
}
.action-stack button, .hero-actions button {
  min-height: 46px;
}
.status-chip {
  display: inline-flex;
  align-items: center;
  justify-content: center;
  min-height: 30px;
  padding: 0 12px;
  border: 1px solid color-mix(in srgb, var(--amber) 42%, transparent);
  border-radius: 999px;
  background: color-mix(in srgb, var(--amber) 13%, transparent);
  color: var(--amber);
  font-size: 12px;
  font-weight: 850;
}
.status-chip.enabled {
  border-color: color-mix(in srgb, var(--green) 38%, transparent);
  background: color-mix(in srgb, var(--green) 13%, transparent);
  color: var(--green);
}
.status-chip.unlocked {
  border-color: color-mix(in srgb, var(--accent) 44%, transparent);
  background: color-mix(in srgb, var(--accent) 14%, transparent);
  color: var(--accent);
}

.matrix-grid, .runtime-map {
  display: grid;
  grid-template-columns: repeat(4, minmax(0, 1fr));
  gap: 12px;
}
.matrix-cell, .runtime-cell {
  min-height: 88px;
  padding: 14px;
  border: 1px solid var(--line);
  border-radius: 14px;
  background: color-mix(in srgb, var(--panel-solid) 76%, transparent);
}
.matrix-cell strong, .runtime-cell strong { display: block; margin-bottom: 8px; }
.matrix-cell span, .runtime-cell span { color: var(--muted); font-size: 12px; line-height: 1.45; }

.config-list {
  position: relative;
  z-index: 1;
  display: grid;
  grid-template-columns: minmax(178px, .5fr) minmax(0, 1fr);
  gap: 10px 16px;
  margin: 0;
  font-size: 13px;
}
.config-list dt { color: var(--muted); }
.config-list dd {
  margin: 0;
  min-width: 0;
  overflow-wrap: anywhere;
  color: var(--ink);
  font-family: var(--mono);
}

.toggle {
  display: inline-flex;
  align-items: center;
  gap: 8px;
  color: var(--muted);
  font-size: 13px;
}
.log-panel { min-height: calc(100vh - 158px); }
.log {
  position: relative;
  z-index: 1;
  height: calc(100vh - 260px);
  min-height: 560px;
  overflow: auto;
  margin: 0;
  padding: 16px;
  border: 1px solid var(--line);
  border-radius: 16px;
  background: rgba(3, 6, 10, .64);
  color: color-mix(in srgb, var(--ink) 86%, var(--accent));
  font-family: var(--mono);
  font-size: 12px;
  line-height: 1.65;
  white-space: pre-wrap;
}

.toast {
  position: fixed;
  right: 20px;
  bottom: 20px;
  z-index: 20;
  max-width: min(420px, calc(100vw - 40px));
  padding: 13px 15px;
  border: 1px solid var(--line);
  border-radius: 14px;
  background: color-mix(in srgb, var(--panel-solid) 92%, transparent);
  color: var(--ink);
  box-shadow: var(--shadow);
  backdrop-filter: blur(18px);
  font-size: 14px;
}

/* Native control layer: never depend on custom-element upgrade for core actions. */
button, select {
  border: 1px solid var(--line);
  border-radius: 14px;
  background:
    linear-gradient(135deg, rgba(255,255,255,.08), transparent),
    color-mix(in srgb, var(--panel-solid) 88%, transparent);
  color: var(--ink);
  cursor: pointer;
  font: inherit;
  transition: transform .18s ease, border-color .18s ease, background .18s ease, box-shadow .18s ease;
}
button:hover, select:hover {
  transform: translateY(-1px);
  border-color: color-mix(in srgb, var(--accent) 55%, var(--line));
  box-shadow: 0 12px 28px rgba(0,0,0,.16);
}
button:focus-visible, select:focus-visible {
  outline: 2px solid color-mix(in srgb, var(--accent) 72%, transparent);
  outline-offset: 2px;
}
.ui-button, .ui-select {
  min-height: 42px;
  padding: 0 16px;
  font-weight: 750;
}
.ui-button.primary {
  border-color: color-mix(in srgb, var(--accent) 70%, transparent);
  background: linear-gradient(135deg, var(--accent), var(--accent-2));
  color: #061018;
}
.ui-select {
  min-width: 132px;
  padding-right: 34px;
  appearance: none;
  background-image:
    linear-gradient(45deg, transparent 50%, var(--ink) 50%),
    linear-gradient(135deg, var(--ink) 50%, transparent 50%),
    linear-gradient(135deg, rgba(255,255,255,.08), transparent);
  background-position:
    calc(100% - 18px) 50%,
    calc(100% - 12px) 50%,
    0 0;
  background-size: 6px 6px, 6px 6px, 100% 100%;
  background-repeat: no-repeat;
}
.icon-btn {
  display: grid;
  place-items: center;
  width: 42px;
  padding: 0;
  font-size: 18px;
}
.badge {
  display: inline-flex;
  align-items: center;
  justify-content: center;
  min-height: 28px;
  padding: 0 11px;
  border: 1px solid var(--line);
  border-radius: 999px;
  background: color-mix(in srgb, var(--panel-solid) 78%, transparent);
  color: var(--muted);
  font-size: 12px;
  font-weight: 850;
}
.badge.accent {
  border-color: color-mix(in srgb, var(--accent) 50%, transparent);
  background: color-mix(in srgb, var(--accent) 16%, transparent);
  color: var(--accent);
}
.brand-mark {
  position: relative;
  display: block;
  flex: 0 0 auto;
  font-size: 0;
}
.brand-mark::after {
  content: "RG";
  position: absolute;
  inset: 0;
  display: grid;
  place-items: center;
  color: #061018;
  font-size: 13px;
  font-weight: 900;
  letter-spacing: .04em;
}
.nav-item {
  display: flex;
  align-items: center;
  width: 100%;
  min-height: 46px;
  padding: 0 14px;
  border-color: transparent;
  background: transparent;
  color: var(--muted);
  text-align: left;
}
.nav-item.active,
.nav-item:hover {
  border-color: var(--line);
  background:
    linear-gradient(135deg, color-mix(in srgb, var(--accent) 12%, transparent), transparent),
    var(--panel-2);
  color: var(--ink);
}
.preset-card {
  display: grid;
  place-items: center;
  align-content: center;
  width: 100%;
  min-height: 154px;
  padding: 22px;
  border-color: var(--line);
  background:
    radial-gradient(circle at 24% 12%, color-mix(in srgb, var(--accent) 13%, transparent), transparent 42%),
    color-mix(in srgb, var(--panel-solid) 78%, transparent);
  text-align: center;
}
.preset-card:hover {
  border-color: color-mix(in srgb, var(--accent) 55%, var(--line));
}
.action-stack button, .hero-actions button {
  min-height: 46px;
}
button.guarded-disabled {
  color: var(--muted);
  cursor: not-allowed;
  opacity: .72;
  box-shadow: none;
}
button.guarded-disabled:hover {
  transform: none;
  border-color: var(--line);
}
.config-list {
  grid-template-columns: 1fr;
  gap: 10px;
}
.config-row {
  display: grid;
  grid-template-columns: minmax(0, 1fr) auto;
  align-items: center;
  gap: 16px;
  min-height: 32px;
}
.config-row dt {
  min-width: 0;
  overflow: hidden;
  text-overflow: ellipsis;
  white-space: nowrap;
}
.config-row dd {
  max-width: 160px;
  text-align: right;
  white-space: nowrap;
}

@media (max-width: 1180px) {
  .overview-grid { grid-template-columns: repeat(4, minmax(0, 1fr)); }
  .hero-panel { grid-column: 1 / -1; }
  .span-4, .span-5, .span-6, .span-7, .span-8 { grid-column: span 12; }
  .matrix-grid, .runtime-map { grid-template-columns: repeat(2, minmax(0, 1fr)); }
}

@media (max-width: 900px) {
  .app-shell { grid-template-columns: 1fr; }
  .sidebar {
    position: static;
    height: auto;
    padding: 16px;
    border-right: 0;
    border-bottom: 1px solid var(--line);
  }
  .brand { margin-bottom: 14px; }
  .nav { grid-template-columns: repeat(5, minmax(0, 1fr)); }
  .nav-item { justify-content: center; padding: 0 8px; }
  .nav-item::part(control) { justify-content: center; padding: 0 8px; }
  .nav-icon { display: none; }
  .side-status { display: none; }
  .workspace { width: 100%; padding: 24px 16px 34px; }
  .topbar { flex-direction: column; }
  .top-actions { justify-content: flex-start; }
}

@media (max-width: 680px) {
  .overview-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); }
  .hero-panel {
    grid-template-columns: 1fr;
    text-align: center;
  }
  .hero-signal { margin: 0 auto; }
  .hero-actions { grid-template-columns: 1fr; }
  .preset-grid, .matrix-grid, .runtime-map { grid-template-columns: 1fr; }
  .config-list { grid-template-columns: 1fr; }
  .nav { grid-template-columns: repeat(3, minmax(0, 1fr)); }
}

@media (prefers-reduced-motion: reduce) {
  *, *::before, *::after {
    animation-duration: .01ms !important;
    animation-iteration-count: 1 !important;
    scroll-behavior: auto !important;
    transition-duration: .01ms !important;
  }
}


path: gateway-watchdog.ps1
content: param(
  [string]$ConfigPath = ""
)

$ErrorActionPreference = "Continue"
$Version = "1.4.4"

$ConfigDir = Join-Path $env:APPDATA "openclaw-gateway-watchdog"
$StateDir = Join-Path $env:LOCALAPPDATA "openclaw-gateway-watchdog"
if (-not $ConfigPath) {
  $ConfigPath = Join-Path $ConfigDir "watchdog.json"
}

$DefaultConfig = [ordered]@{
  GatewayHealthUrl = "http://127.0.0.1:18789/healthz"
  GatewayHost = "127.0.0.1"
  GatewayPort = 18789
  ChannelUrl = "https://ilinkai.weixin.qq.com"
  NetworkUrls = @("https://www.baidu.com", "https://www.qq.com", "https://api.weixin.qq.com")
  RestartCommand = ""
  OpenClawNativeProbes = "auto"
  OpenClawHealthTimeoutMs = 12000
  OpenClawGatewayStrict = $false
  OpenClawChannelsProbe = $true
  OpenClawDiagEnabled = $true
  OpenClawDiagInterval = 300
  OpenClawLogScanEnabled = $true
  OpenClawLogLimit = 200
  OpenClawLogSignalLimit = 40
  OpenClawLogTimeoutMs = 15000
  OpenClawLogWarnPatterns = "fetch failed|fetch timeout|LLM idle timeout|model silent|chat/completions|providerRuntimeFailureKind|ECONNRESET|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|socket hang up|TLS|CERT_|proxy|429|rate limit|quota|unauthorized|invalid api key|embedded abort settle timed out|embedded run failover decision|memory-core: narrative generation ended with status=timeout|dreaming.*timeout|health-monitor|event loop|degraded|restartPending|session expired|errcode=-14|Monitor.*stopped|monitor.*ended|config hot reload|config change detected|cron.*error|task.*failed"
  OpenClawDiagAction = "log"
  OpenClawDiagFailuresBeforeAction = 2
  OpenClawDiagCommand = ""
  DashboardEnabled = $true
  DashboardHost = "127.0.0.1"
  DashboardPort = 18790
  DashboardActionsEnabled = $false
  DashboardToken = ""
  DashboardDir = Join-Path $PSScriptRoot "dashboard"
  DashboardPython = ""
  ModelProbeEnabled = $false
  ModelEdgeProbeEnabled = $true
  ModelProbeInterval = 1800
  ModelProbeTimeout = 120
  ModelProbeFailuresBeforeAction = 2
  ModelProbeAction = "log"
  ModelProbeCommand = ""
  ModelProbeModel = ""
  ModelProbeThinking = "off"
  ModelProbeSessionId = "watchdog-model-probe"
  ModelProbeMessage = "Reply with exactly OK."
  BaseInterval = 60
  NightInterval = 300
  MaxInterval = 1800
  ConnectTimeout = 5
  MaxTime = 10
  ChannelFailuresBeforeRestart = 2
  SuccessCountToReset = 5
  MaxRestartsPerHour = 6
  PostRestartSleep = 30
  MaxLogBytes = 1048576
}

New-Item -ItemType Directory -Force -Path $ConfigDir, $StateDir | Out-Null
if (Test-Path -LiteralPath $ConfigPath) {
  $UserConfig = Get-Content -LiteralPath $ConfigPath -Raw | ConvertFrom-Json
  foreach ($prop in $UserConfig.PSObject.Properties) {
    $DefaultConfig[$prop.Name] = $prop.Value
  }
}
$Config = [pscustomobject]$DefaultConfig
$LogFile = Join-Path $StateDir "watchdog.log"

$createdNew = $false
$mutex = [System.Threading.Mutex]::new($false, "Global\OpenClawGatewayResilienceGuard", [ref]$createdNew)
if (-not $createdNew) {
  Write-Error "Another watchdog instance is already running."
  exit 0
}

function Write-Log {
  param([string]$Message)
  $line = "[{0}] {1}" -f (Get-Date -Format "yyyy-MM-dd HH:mm:ss"), $Message
  $line | Tee-Object -FilePath $LogFile -Append
}

function Rotate-LogIfNeeded {
  if (Test-Path -LiteralPath $LogFile) {
    $size = (Get-Item -LiteralPath $LogFile).Length
    if ($size -gt [int64]$Config.MaxLogBytes) {
      Move-Item -LiteralPath $LogFile -Destination "$LogFile.1" -Force
      Write-Log "LOG: rotated because it exceeded $($Config.MaxLogBytes) bytes"
    }
  }
}

function Test-NativeEnabled {
  return -not (@("0", "false", "off", "no") -contains ([string]$Config.OpenClawNativeProbes).ToLowerInvariant())
}

function Save-ProbeOutput {
  param([string]$Name, [object[]]$Output)
  $path = Join-Path $StateDir $Name
  ($Output -join [Environment]::NewLine) | Out-File -LiteralPath $path -Encoding utf8
  return $path
}

function Invoke-OpenClawProbe {
  param([string[]]$Args, [string]$OutputName)
  if (-not (Test-NativeEnabled)) { return @{ Supported = $false; ExitCode = 127; Text = "" } }
  if (-not (Get-Command openclaw -ErrorAction SilentlyContinue)) {
    return @{ Supported = $false; ExitCode = 127; Text = "" }
  }
  $output = & openclaw @Args 2>&1
  $exitCode = $LASTEXITCODE
  $path = Save-ProbeOutput -Name $OutputName -Output $output
  $text = $output -join "`n"
  $unsupported = $text -match "unknown command|unknown option|unknown argument|unrecognized option|not recognized|invalid command"
  return @{ Supported = (-not $unsupported); ExitCode = $exitCode; Text = $text; Path = $path }
}

function Test-Http {
  param([string]$Url)
  if (-not $Url) { return $false }
  try {
    $response = Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec ([int]$Config.MaxTime)
    return ($response.StatusCode -ge 200 -and $response.StatusCode -lt 500)
  } catch {
    $status = $_.Exception.Response.StatusCode.value__
    if ($status) {
      return ($status -ge 200 -and $status -lt 500)
    }
    return $false
  }
}

function Test-Tcp {
  try {
    $client = [System.Net.Sockets.TcpClient]::new()
    $task = $client.ConnectAsync([string]$Config.GatewayHost, [int]$Config.GatewayPort)
    $ok = $task.Wait([int]$Config.ConnectTimeout * 1000) -and $client.Connected
    $client.Dispose()
    return $ok
  } catch {
    return $false
  }
}

function Test-OpenClawGateway {
  $probe = Invoke-OpenClawProbe -Args @("gateway", "status", "--json", "--require-rpc", "--timeout", "$($Config.OpenClawHealthTimeoutMs)") -OutputName "last-openclaw-gateway-status.json"
  if (-not $probe.Supported) { return 2 }
  if ($probe.ExitCode -eq 0) {
    if ($probe.Text -match '"ok"\s*:\s*false') {
      Write-Log "OPENCLAW GATEWAY FAIL: gateway status reported ok=false"
      return 1
    }
    if ($probe.Text -match '"degraded"\s*:\s*true') {
      Write-Log "OPENCLAW GATEWAY WARN: gateway RPC probe is reachable but degraded"
      if ($Config.OpenClawGatewayStrict) { return 1 }
    }
    return 0
  }
  Write-Log "OPENCLAW GATEWAY WARN: gateway status --require-rpc failed; falling back to local probes"
  return 2
}

function Test-Gateway {
  $native = Test-OpenClawGateway
  if ($native -eq 0) { return 0 }
  if ($native -eq 1) { return 1 }
  if (Test-Http -Url $Config.GatewayHealthUrl) { return 0 }
  if (Test-Tcp) { return 0 }
  Write-Log "GATEWAY FAIL: no healthy gateway detected"
  return 1
}

function Test-OpenClawChannel {
  $health = Invoke-OpenClawProbe -Args @("health", "--json", "--verbose", "--timeout", "$($Config.OpenClawHealthTimeoutMs)") -OutputName "last-openclaw-health.json"
  if ($health.Supported) {
    if ($health.ExitCode -eq 0 -and $health.Text -notmatch '"ok"\s*:\s*false') { return 0 }
    Write-Log "OPENCLAW CHANNEL FAIL: health live probe failed"
    return 1
  }
  $deep = Invoke-OpenClawProbe -Args @("status", "--deep") -OutputName "last-openclaw-status-deep.txt"
  if ($deep.Supported) {
    if ($deep.ExitCode -eq 0 -and $deep.Text -notmatch "logged[ -]?out|loggedOut|disconnected|unhealthy|healthy\s*:\s*false|probe\s*-?\s*failed|status\s*:\s*(409|410|411|412|413|414|415|500|501|502|503|504|515)|timeout") { return 0 }
    Write-Log "OPENCLAW CHANNEL FAIL: status --deep probe failed"
    return 1
  }
  if ($Config.OpenClawChannelsProbe) {
    $channels = Invoke-OpenClawProbe -Args @("channels", "status", "--probe") -OutputName "last-openclaw-channels-status.txt"
    if ($channels.Supported) {
      if ($channels.ExitCode -eq 0 -and $channels.Text -notmatch "logged[ -]?out|loggedOut|disconnected|unhealthy|healthy\s*:\s*false|probe\s*-?\s*failed|status\s*:\s*(409|410|411|412|413|414|415|500|501|502|503|504|515)|timeout") { return 0 }
      Write-Log "OPENCLAW CHANNEL FAIL: channels status probe failed"
      return 1
    }
  }
  return 2
}

function Test-Channel {
  $native = Test-OpenClawChannel
  if ($native -eq 0) { return $true }
  if ($native -eq 1) { return $false }
  return Test-Http -Url $Config.ChannelUrl
}

function Test-Network {
  foreach ($url in $Config.NetworkUrls) {
    if (Test-Http -Url $url) { return $true }
  }
  Write-Log "NETWORK FAIL: all network probes failed"
  return $false
}

function Test-ModelProbeEnabled {
  return @("1", "true", "on", "yes") -contains ([string]$Config.ModelProbeEnabled).ToLowerInvariant()
}

function Test-DiagEnabled {
  return -not (@("0", "false", "off", "no") -contains ([string]$Config.OpenClawDiagEnabled).ToLowerInvariant())
}

function Test-LogScanEnabled {
  return -not (@("0", "false", "off", "no") -contains ([string]$Config.OpenClawLogScanEnabled).ToLowerInvariant())
}

function Test-DashboardEnabled {
  return -not (@("0", "false", "off", "no") -contains ([string]$Config.DashboardEnabled).ToLowerInvariant())
}

function Test-DashboardActionsEnabled {
  return @("1", "true", "on", "yes") -contains ([string]$Config.DashboardActionsEnabled).ToLowerInvariant()
}

function Test-ModelEdgeProbeEnabled {
  return -not (@("0", "false", "off", "no") -contains ([string]$Config.ModelEdgeProbeEnabled).ToLowerInvariant())
}

function Get-DashboardPidPath {
  return Join-Path $StateDir "dashboard.pid"
}

function Test-DashboardAlive {
  $pidPath = Get-DashboardPidPath
  if (-not (Test-Path -LiteralPath $pidPath)) { return $false }
  $raw = Get-Content -LiteralPath $pidPath -ErrorAction SilentlyContinue | Select-Object -First 1
  $pidValue = 0
  if (-not [int]::TryParse([string]$raw, [ref]$pidValue)) { return $false }
  return [bool](Get-Process -Id $pidValue -ErrorAction SilentlyContinue)
}

function Get-DashboardPython {
  if ($Config.DashboardPython) {
    $cmd = Get-Command $Config.DashboardPython -ErrorAction SilentlyContinue
    if ($cmd) { return @{ File = $cmd.Source; ArgsPrefix = "" } }
  }
  $python = Get-Command python -ErrorAction SilentlyContinue
  if ($python) { return @{ File = $python.Source; ArgsPrefix = "" } }
  $py = Get-Command py -ErrorAction SilentlyContinue
  if ($py) { return @{ File = $py.Source; ArgsPrefix = "-3 " } }
  return $null
}

function Start-Dashboard {
  if (-not (Test-DashboardEnabled)) { return }
  if (Test-DashboardAlive) { return }
  $dashboardDir = [string]$Config.DashboardDir
  if (-not $dashboardDir) { $dashboardDir = Join-Path $PSScriptRoot "dashboard" }
  $serverPath = Join-Path $dashboardDir "server.py"
  if (-not (Test-Path -LiteralPath $serverPath)) {
    Write-Log "DASHBOARD WARN: $serverPath not found; dashboard disabled for this run"
    $Config.DashboardEnabled = $false
    return
  }
  $python = Get-DashboardPython
  if (-not $python) {
    Write-Log "DASHBOARD WARN: python/py not found; dashboard disabled for this run"
    $Config.DashboardEnabled = $false
    return
  }
  $actions = if (Test-DashboardActionsEnabled) { "1" } else { "0" }
  $psi = [System.Diagnostics.ProcessStartInfo]::new()
  $psi.FileName = $python.File
  $psi.Arguments = "$($python.ArgsPrefix)`"$serverPath`""
  $psi.WorkingDirectory = $dashboardDir
  $psi.UseShellExecute = $false
  $psi.CreateNoWindow = $true
  $psi.Environment["WATCHDOG_STATE_DIR"] = $StateDir
  $psi.Environment["WATCHDOG_CONFIG_FILE"] = $ConfigPath
  $psi.Environment["WATCHDOG_DASHBOARD_HOST"] = [string]$Config.DashboardHost
  $psi.Environment["WATCHDOG_DASHBOARD_PORT"] = [string]$Config.DashboardPort
  $psi.Environment["WATCHDOG_DASHBOARD_ACTIONS_ENABLED"] = $actions
  $psi.Environment["WATCHDOG_DASHBOARD_TOKEN"] = [string]$Config.DashboardToken
  try {
    $proc = [System.Diagnostics.Process]::Start($psi)
    Set-Content -LiteralPath (Get-DashboardPidPath) -Value $proc.Id
    Write-Log "DASHBOARD: http://$($Config.DashboardHost):$($Config.DashboardPort)"
  } catch {
    Write-Log "DASHBOARD WARN: failed to start dashboard: $($_.Exception.Message)"
  }
}

function Get-LogSignalCategories {
  param([string]$Text)
  $categories = New-Object System.Collections.Generic.List[string]
  if ($Text -match "fetch failed|fetch timeout|LLM idle timeout|model silent|chat/completions|providerRuntimeFailureKind.*timeout") { $categories.Add("provider_timeout") }
  if ($Text -match "ECONNRESET|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|socket hang up|TLS|CERT_|proxy") { $categories.Add("proxy_or_network") }
  if ($Text -match "429|rate limit|too many requests|quota") { $categories.Add("provider_rate_limit") }
  if ($Text -match "401|403|unauthorized|forbidden|invalid api key|auth") { $categories.Add("provider_auth") }
  if ($Text -match "embedded abort settle timed out") { $categories.Add("abort_stuck") }
  if ($Text -match "memory-core: narrative generation ended with status=timeout|dreaming.*timeout") { $categories.Add("memory_dream_timeout") }
  if ($Text -match "restartPending|session expired|errcode=-14|Monitor.*stopped|monitor.*ended") { $categories.Add("channel_session") }
  if ($Text -match "event loop|degraded|health-monitor") { $categories.Add("gateway_degraded") }
  if ($Text -match "config hot reload|config change detected") { $categories.Add("config_reload") }
  if ($Text -match "cron.*error|task.*failed|run failover decision") { $categories.Add("task_runtime") }
  if ($Text -match "\bwarn\b|\bwarning\b|\berror\b|\bfailed\b|\btimeout\b") { $categories.Add("openclaw_warning") }
  if ($categories.Count -eq 0) { return "unknown" }
  return ($categories -join ",")
}

function Invoke-OpenClawLogScan {
  if (-not (Test-LogScanEnabled)) { return 2 }
  if (-not (Get-Command openclaw -ErrorAction SilentlyContinue)) { return 2 }

  $logFile = Join-Path $StateDir "last-openclaw-logs.txt"
  $signalsFile = Join-Path $StateDir "last-openclaw-log-signals.txt"
  $categoriesFile = Join-Path $StateDir "last-openclaw-log-signal-categories.txt"
  $fingerprintFile = Join-Path $StateDir "last-openclaw-log-signals.cksum"

  $output = & openclaw logs --plain --limit "$($Config.OpenClawLogLimit)" --timeout "$($Config.OpenClawLogTimeoutMs)" 2>&1
  if ($LASTEXITCODE -ne 0) { return 2 }
  $output | Out-File -LiteralPath $logFile -Encoding utf8
  $pattern = "($($Config.OpenClawLogWarnPatterns))|\b(warn|warning|error|failed|timeout)\b"
  $signals = $output | Select-String -Pattern $pattern | Select-Object -Last ([int]$Config.OpenClawLogSignalLimit)
  if (-not $signals) { return 0 }
  $signalText = ($signals | ForEach-Object { $_.Line }) -join [Environment]::NewLine
  $signalText | Out-File -LiteralPath $signalsFile -Encoding utf8
  $bytes = [System.Text.Encoding]::UTF8.GetBytes($signalText)
  $sha256 = [System.Security.Cryptography.SHA256]::Create()
  try {
    $hashBytes = $sha256.ComputeHash($bytes)
  } finally {
    $sha256.Dispose()
  }
  $hash = [System.BitConverter]::ToString($hashBytes).Replace("-", "").ToLowerInvariant()
  $previous = ""
  if (Test-Path -LiteralPath $fingerprintFile) { $previous = Get-Content -LiteralPath $fingerprintFile -Raw }
  if ($hash -eq $previous.Trim()) { return 0 }
  Set-Content -LiteralPath $fingerprintFile -Value $hash

  $categories = Get-LogSignalCategories -Text $signalText
  Set-Content -LiteralPath $categoriesFile -Value $categories
  Write-Log "OPENCLAW LOG WARN: categories=$categories matches=$($signals.Count) fingerprint=$hash"
  $first = ($signalText -split "`n" | Where-Object { $_.Trim() } | Select-Object -First 1)
  if ($first) { Write-Log "OPENCLAW LOG DETAIL: $($first.Substring(0, [Math]::Min(220, $first.Length)))" }
  return 1
}

function Invoke-OpenClawDiagnostics {
  if (-not (Test-DiagEnabled)) { return 2 }
  if (-not (Test-NativeEnabled)) { return 2 }
  if (-not (Get-Command openclaw -ErrorAction SilentlyContinue)) { return 2 }

  $degraded = $false
  $signals = $false
  $categories = "none"
  $summaryFile = Join-Path $StateDir "last-openclaw-diagnostics.jsonl"

  $gateway = & openclaw gateway status --json --require-rpc --timeout "$($Config.OpenClawHealthTimeoutMs)" 2>&1
  Save-ProbeOutput -Name "last-openclaw-gateway-status.json" -Output $gateway | Out-Null
  $health = & openclaw health --json --verbose --timeout "$($Config.OpenClawHealthTimeoutMs)" 2>&1
  Save-ProbeOutput -Name "last-openclaw-health.json" -Output $health | Out-Null
  $models = & openclaw models status --json 2>&1
  Save-ProbeOutput -Name "last-openclaw-model-status.json" -Output $models | Out-Null
  $statusDeep = & openclaw status --deep --no-color 2>&1
  Save-ProbeOutput -Name "last-openclaw-status-deep.txt" -Output $statusDeep | Out-Null

  $healthText = $health -join "`n"
  if ($healthText -match '"degraded"\s*:\s*true|"ok"\s*:\s*false') {
    $degraded = $true
    Write-Log "OPENCLAW DIAG WARN: health snapshot reports degraded or not ok"
  }
  $modelsText = $models -join "`n"
  if ($modelsText -match '"fallbacks"\s*:\s*\[\]') {
    Write-Log "OPENCLAW DIAG INFO: no model fallbacks configured"
  }

  $logResult = Invoke-OpenClawLogScan
  if ($logResult -eq 1) {
    $signals = $true
    $categoriesFile = Join-Path $StateDir "last-openclaw-log-signal-categories.txt"
    if (Test-Path -LiteralPath $categoriesFile) {
      $categories = (Get-Content -LiteralPath $categoriesFile -Raw).Trim()
    } else {
      $categories = "unknown"
    }
  }

  (@{
    ts = (Get-Date -Format "yyyy-MM-ddTHH:mm:sszzz")
    healthDegraded = $degraded
    newLogSignals = $signals
    categories = $categories
  } | ConvertTo-Json -Compress) | Add-Content -LiteralPath $summaryFile

  if ($degraded -or $signals) { return 1 }
  return 0
}

function Invoke-DiagFailureAction {
  param([int]$Failures)
  if ($Failures -lt [int]$Config.OpenClawDiagFailuresBeforeAction) { return }
  if (-not (Test-Network)) {
    Write-Log "OPENCLAW DIAG SAFEGUARD: general network probes failed; skip diagnostic action"
    return
  }
  switch ([string]$Config.OpenClawDiagAction) {
    "restart" {
      Write-Log "OPENCLAW DIAG ACTION: restart gateway after $Failures consecutive diagnostic warnings"
      Restart-Gateway
    }
    "command" {
      if ($Config.OpenClawDiagCommand) {
        Write-Log "OPENCLAW DIAG ACTION: $($Config.OpenClawDiagCommand)"
        cmd.exe /c $Config.OpenClawDiagCommand 2>&1 | Tee-Object -FilePath $LogFile -Append
        if ($LASTEXITCODE -ne 0) { Write-Log "OPENCLAW DIAG ACTION FAIL: command returned non-zero" }
      } else {
        Write-Log "OPENCLAW DIAG ACTION WARN: OpenClawDiagAction=command but OpenClawDiagCommand is empty"
      }
    }
    default {
      Write-Log "OPENCLAW DIAG ACTION: log-only after $Failures consecutive diagnostic warnings"
    }
  }
}

function Invoke-ModelApiEdgeProbe {
  if (-not (Test-ModelEdgeProbeEnabled)) { return 2 }
  if (-not (Get-Command openclaw -ErrorAction SilentlyContinue)) { return 2 }

  $model = [string]$Config.ModelProbeModel
  if (-not $model) {
    $statusOutput = & openclaw models status --json 2>&1
    if ($LASTEXITCODE -ne 0) { return 2 }
    Save-ProbeOutput -Name "last-openclaw-model-status.json" -Output $statusOutput | Out-Null
    try {
      $statusJson = ($statusOutput -join "`n") | ConvertFrom-Json -ErrorAction Stop
      $model = [string]$statusJson.defaultModel
    } catch {
      return 2
    }
  }
  if (-not $model -or $model -notmatch "/") { return 2 }
  $provider = ($model -split "/", 2)[0]

  $providerOutput = & openclaw config get "models.providers.$provider" 2>&1
  if ($LASTEXITCODE -ne 0) { return 2 }
  Save-ProbeOutput -Name "last-openclaw-model-provider.json" -Output $providerOutput | Out-Null
  try {
    $providerJson = ($providerOutput -join "`n") | ConvertFrom-Json -ErrorAction Stop
    $baseUrl = [string]$providerJson.baseUrl
  } catch {
    return 2
  }
  if (-not $baseUrl) { return 2 }

  $sw = [System.Diagnostics.Stopwatch]::StartNew()
  $statusCode = 0
  $ok = $false
  $errorText = ""
  try {
    $response = Invoke-WebRequest -Uri $baseUrl -UseBasicParsing -TimeoutSec ([int]$Config.MaxTime)
    $statusCode = [int]$response.StatusCode
    $ok = ($statusCode -ge 200 -and $statusCode -lt 500)
  } catch {
    $errorText = $_.Exception.Message
    if ($_.Exception.Response -and $_.Exception.Response.StatusCode) {
      $statusCode = [int]$_.Exception.Response.StatusCode
      $ok = ($statusCode -ge 200 -and $statusCode -lt 500)
    }
  } finally {
    $sw.Stop()
  }

  $line = "provider=$provider model=$model base=$baseUrl code=$statusCode total=$([Math]::Round($sw.Elapsed.TotalSeconds, 3))s"
  Set-Content -LiteralPath (Join-Path $StateDir "last-model-api-edge-probe.txt") -Value $line
  if ($ok) {
    Write-Log "MODEL API EDGE OK: $line"
    return 0
  }
  Write-Log "MODEL API EDGE FAIL: $line error=$errorText"
  return 1
}

function Invoke-ModelProbe {
  if (-not (Test-ModelProbeEnabled)) { return 2 }
  if (-not (Get-Command openclaw -ErrorAction SilentlyContinue)) {
    Write-Log "MODEL PROBE WARN: openclaw CLI not found; skip model probe"
    return 2
  }

  $outputName = "last-openclaw-model-probe.json"
  $historyPath = Join-Path $StateDir "model-probe-history.jsonl"
  $started = Get-Date
  Invoke-ModelApiEdgeProbe | Out-Null
  $args = @(
    "agent",
    "--session-id", "$($Config.ModelProbeSessionId)",
    "--thinking", "$($Config.ModelProbeThinking)",
    "--timeout", "$($Config.ModelProbeTimeout)",
    "--json",
    "--message", "$($Config.ModelProbeMessage)"
  )
  if ($Config.ModelProbeModel) {
    $args = @(
      "agent",
      "--session-id", "$($Config.ModelProbeSessionId)",
      "--model", "$($Config.ModelProbeModel)",
      "--thinking", "$($Config.ModelProbeThinking)",
      "--timeout", "$($Config.ModelProbeTimeout)",
      "--json",
      "--message", "$($Config.ModelProbeMessage)"
    )
  }

  $output = & openclaw @args 2>&1
  $exitCode = $LASTEXITCODE
  $path = Save-ProbeOutput -Name $outputName -Output $output
  $duration = [int]((Get-Date) - $started).TotalSeconds
  $text = $output -join "`n"
  $status = ""
  $provider = "unknown"
  $model = if ($Config.ModelProbeModel) { "$($Config.ModelProbeModel)" } else { "configured-default" }
  try {
    $json = $text | ConvertFrom-Json -ErrorAction Stop
    $status = [string]$json.status
    if ($json.result.meta.agentMeta.provider) { $provider = [string]$json.result.meta.agentMeta.provider }
    if ($json.result.meta.agentMeta.model) { $model = [string]$json.result.meta.agentMeta.model }
  } catch {
    $status = ""
  }

  if ($exitCode -eq 0 -and $status -eq "ok") {
    Write-Log "MODEL PROBE OK: provider=$provider model=$model duration=${duration}s session=$($Config.ModelProbeSessionId)"
    (@{
      ts = (Get-Date -Format "yyyy-MM-ddTHH:mm:sszzz")
      status = "ok"
      provider = $provider
      model = $model
      durationSeconds = $duration
    } | ConvertTo-Json -Compress) | Add-Content -LiteralPath $historyPath
    return 0
  }

  $reason = "failed"
  if ($text -match "timeout|timed out|idle timeout|fetch timeout") {
    $reason = "timeout"
  } elseif ($text -match "rate limit|429|too many requests") {
    $reason = "rate_limited"
  } elseif ($text -match "401|403|unauthorized|forbidden|invalid api key|auth") {
    $reason = "auth"
  } elseif ($text -match "Model override .* not allowed") {
    $reason = "config"
  }

  Write-Log "MODEL PROBE FAIL: reason=$reason rc=$exitCode provider=$provider model=$model duration=${duration}s timeout=$($Config.ModelProbeTimeout)s"
  $detail = ($text -split "`n" | Where-Object { $_.Trim() } | Select-Object -First 1)
  if ($detail) { Write-Log "MODEL PROBE DETAIL: $($detail.Substring(0, [Math]::Min(220, $detail.Length)))" }
  (@{
    ts = (Get-Date -Format "yyyy-MM-ddTHH:mm:sszzz")
    status = "fail"
    reason = $reason
    provider = $provider
    model = $model
    durationSeconds = $duration
    exitCode = $exitCode
  } | ConvertTo-Json -Compress) | Add-Content -LiteralPath $historyPath
  return 1
}

function Invoke-ModelProbeFailureAction {
  param([int]$Failures)
  if ($Failures -lt [int]$Config.ModelProbeFailuresBeforeAction) { return }
  if (-not (Test-Network)) {
    Write-Log "MODEL PROBE SAFEGUARD: general network probes failed; skip model action"
    return
  }
  switch ([string]$Config.ModelProbeAction) {
    "restart" {
      Write-Log "MODEL PROBE ACTION: restart gateway after $Failures consecutive model probe failures"
      Restart-Gateway
    }
    "command" {
      if ($Config.ModelProbeCommand) {
        Write-Log "MODEL PROBE ACTION: $($Config.ModelProbeCommand)"
        cmd.exe /c $Config.ModelProbeCommand 2>&1 | Tee-Object -FilePath $LogFile -Append
        if ($LASTEXITCODE -ne 0) { Write-Log "MODEL PROBE ACTION FAIL: command returned non-zero" }
      } else {
        Write-Log "MODEL PROBE ACTION WARN: ModelProbeAction=command but ModelProbeCommand is empty"
      }
    }
    default {
      Write-Log "MODEL PROBE ACTION: log-only after $Failures consecutive model probe failures"
    }
  }
}

function Get-RestartCountFile {
  return Join-Path $StateDir ("restarts.{0}" -f (Get-Date -Format "yyyyMMddHH"))
}

function Test-CanRestart {
  $file = Get-RestartCountFile
  $count = 0
  if (Test-Path -LiteralPath $file) { $count = [int](Get-Content -LiteralPath $file -ErrorAction SilentlyContinue) }
  return ($count -lt [int]$Config.MaxRestartsPerHour)
}

function Register-Restart {
  $file = Get-RestartCountFile
  $count = 0
  if (Test-Path -LiteralPath $file) { $count = [int](Get-Content -LiteralPath $file -ErrorAction SilentlyContinue) }
  Set-Content -LiteralPath $file -Value ($count + 1)
}

function Restart-Gateway {
  if (-not (Test-CanRestart)) {
    Write-Log "CIRCUIT OPEN: restart limit reached ($($Config.MaxRestartsPerHour)/hour); skip restart"
    return
  }
  if ($Config.RestartCommand) {
    Write-Log "ACTION: $($Config.RestartCommand)"
    cmd.exe /c $Config.RestartCommand 2>&1 | Tee-Object -FilePath $LogFile -Append
    if ($LASTEXITCODE -eq 0) {
      Register-Restart
    } else {
      Write-Log "ACTION FAIL: restart command returned non-zero"
    }
    return
  }
  if (Get-Command openclaw -ErrorAction SilentlyContinue) {
    Write-Log "ACTION: openclaw gateway restart"
    & openclaw gateway restart 2>&1 | Tee-Object -FilePath $LogFile -Append
    if ($LASTEXITCODE -eq 0) {
      Register-Restart
    } else {
      Write-Log "ACTION FAIL: openclaw gateway restart failed"
    }
    return
  }
  Write-Log "ACTION FAIL: set RestartCommand in $ConfigPath; no restart method found"
}

function Get-BackoffInterval {
  param([int]$Failures)
  $interval = [int]$Config.BaseInterval
  for ($i = 1; $i -lt $Failures; $i++) {
    $interval *= 2
    if ($interval -ge [int]$Config.MaxInterval) { break }
  }
  return [Math]::Min($interval, [int]$Config.MaxInterval)
}

function Get-SleepInterval {
  $hour = [int](Get-Date -Format "HH")
  if ($hour -ge 1 -and $hour -lt 8) { return [int]$Config.NightInterval }
  return [int]$Config.BaseInterval
}

try {
  $failCount = 0
  $successCount = 0
  $modelFailCount = 0
  $diagFailCount = 0
  $lastDiagAt = [DateTimeOffset]::FromUnixTimeSeconds(0)
  $lastModelProbeAt = [DateTimeOffset]::FromUnixTimeSeconds(0)
  Write-Log "START: OpenClaw Gateway Resilience Guard $Version"
  Write-Log "CONFIG: $ConfigPath"
  Start-Dashboard

  while ($true) {
    Rotate-LogIfNeeded
    Start-Dashboard
    $gateway = Test-Gateway
    if ($gateway -eq 1) {
      $failCount += 1
      $successCount = 0
      Write-Log "CRITICAL: gateway appears down; restarting immediately"
      Restart-Gateway
      Start-Sleep -Seconds ([int]$Config.PostRestartSleep)
      continue
    }

    if (Test-DiagEnabled) {
      $now = [DateTimeOffset]::UtcNow
      if ($lastDiagAt.ToUnixTimeSeconds() -eq 0 -or ($now - $lastDiagAt).TotalSeconds -ge [int]$Config.OpenClawDiagInterval) {
        $lastDiagAt = $now
        $diagResult = Invoke-OpenClawDiagnostics
        if ($diagResult -eq 0) {
          $diagFailCount = 0
        } elseif ($diagResult -eq 1) {
          $diagFailCount += 1
          Invoke-DiagFailureAction -Failures $diagFailCount
        }
      }
    }

    if (Test-ModelProbeEnabled) {
      $now = [DateTimeOffset]::UtcNow
      if ($lastModelProbeAt.ToUnixTimeSeconds() -eq 0 -or ($now - $lastModelProbeAt).TotalSeconds -ge [int]$Config.ModelProbeInterval) {
        $lastModelProbeAt = $now
        $probeResult = Invoke-ModelProbe
        if ($probeResult -eq 0) {
          $modelFailCount = 0
        } elseif ($probeResult -eq 1) {
          $modelFailCount += 1
          Invoke-ModelProbeFailureAction -Failures $modelFailCount
        }
      }
    }

    if (Test-Channel) {
      if ($failCount -gt 0) {
        $successCount += 1
        if ($successCount -ge [int]$Config.SuccessCountToReset) {
          Write-Log "RECOVERED: $($Config.SuccessCountToReset) consecutive channel probes succeeded; failure count reset"
          $failCount = 0
          $successCount = 0
        } else {
          Write-Log "RECOVERING: channel probe succeeded ($successCount/$($Config.SuccessCountToReset))"
        }
      }
      Start-Sleep -Seconds (Get-SleepInterval)
      continue
    }

    $failCount += 1
    $successCount = 0
    Write-Log "CHANNEL FAIL: channel probe failed ($failCount/$($Config.ChannelFailuresBeforeRestart))"
    if (-not (Test-Network)) {
      Write-Log "SAFEGUARD: looks like global network trouble; wait without restarting gateway"
      Start-Sleep -Seconds ([int]$Config.BaseInterval)
      continue
    }
    if ($failCount -lt [int]$Config.ChannelFailuresBeforeRestart) {
      Start-Sleep -Seconds ([int]$Config.BaseInterval)
      continue
    }
    $waitFor = Get-BackoffInterval -Failures $failCount
    Write-Log "BACKOFF: network is reachable; will re-check channel in ${waitFor}s before restart"
    Start-Sleep -Seconds $waitFor
    if (Test-Channel) {
      Write-Log "RECOVERED: channel came back before restart"
      $failCount = 0
      continue
    }
    Restart-Gateway
    Start-Sleep -Seconds ([int]$Config.PostRestartSleep)
  }
} finally {
  $mutex.ReleaseMutex() | Out-Null
  $mutex.Dispose()
}


path: gateway-watchdog.sh
content: #!/usr/bin/env bash
# OpenClaw Gateway Resilience Guard
# Keeps the OpenClaw gateway/channel alive with layered probes and guarded restarts.

set -u

WATCHDOG_VERSION="1.4.4"
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"

CONFIG_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/openclaw-gateway-watchdog"
CONFIG_FILE="${WATCHDOG_CONFIG:-${CONFIG_DIR}/watchdog.env}"
STATE_DIR="${XDG_STATE_HOME:-${HOME}/.local/state}/openclaw-gateway-watchdog"
LOG_FILE="${WATCHDOG_LOG_FILE:-${STATE_DIR}/watchdog.log}"
RUNTIME_BASE="${XDG_RUNTIME_DIR:-/tmp}"
RUNTIME_DIR="${RUNTIME_BASE}/openclaw-gateway-watchdog"
LOCK_DIR="${RUNTIME_DIR}/lock"

# Defaults are intentionally useful out of the box. Users can override them in
# ~/.config/openclaw-gateway-watchdog/watchdog.env or environment variables.
GATEWAY_SERVICE="${GATEWAY_SERVICE:-openclaw-gateway}"
GATEWAY_HEALTH_URL="${GATEWAY_HEALTH_URL:-http://127.0.0.1:18789/healthz}"
GATEWAY_HOST="${GATEWAY_HOST:-127.0.0.1}"
GATEWAY_PORT="${GATEWAY_PORT:-18789}"
CHANNEL_URL="${CHANNEL_URL:-https://ilinkai.weixin.qq.com}"
NETWORK_URLS="${NETWORK_URLS:-https://www.baidu.com https://www.qq.com https://api.weixin.qq.com}"
RESTART_COMMAND="${RESTART_COMMAND:-}"
OPENCLAW_NATIVE_PROBES="${OPENCLAW_NATIVE_PROBES:-auto}"
OPENCLAW_HEALTH_TIMEOUT_MS="${OPENCLAW_HEALTH_TIMEOUT_MS:-12000}"
OPENCLAW_GATEWAY_STRICT="${OPENCLAW_GATEWAY_STRICT:-0}"
OPENCLAW_CHANNELS_PROBE="${OPENCLAW_CHANNELS_PROBE:-1}"
OPENCLAW_DIAG_ENABLED="${OPENCLAW_DIAG_ENABLED:-1}"
OPENCLAW_DIAG_INTERVAL="${OPENCLAW_DIAG_INTERVAL:-300}"
OPENCLAW_LOG_SCAN_ENABLED="${OPENCLAW_LOG_SCAN_ENABLED:-1}"
OPENCLAW_LOG_LIMIT="${OPENCLAW_LOG_LIMIT:-200}"
OPENCLAW_LOG_SIGNAL_LIMIT="${OPENCLAW_LOG_SIGNAL_LIMIT:-40}"
OPENCLAW_LOG_TIMEOUT_MS="${OPENCLAW_LOG_TIMEOUT_MS:-15000}"
OPENCLAW_LOG_WARN_PATTERNS="${OPENCLAW_LOG_WARN_PATTERNS:-fetch failed|fetch timeout|LLM idle timeout|model silent|chat/completions|providerRuntimeFailureKind|ECONNRESET|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|socket hang up|TLS|CERT_|proxy|429|rate limit|quota|unauthorized|invalid api key|embedded abort settle timed out|embedded run failover decision|memory-core: narrative generation ended with status=timeout|dreaming.*timeout|health-monitor|event loop|degraded|restartPending|session expired|errcode=-14|Monitor.*stopped|monitor.*ended|config hot reload|config change detected|cron.*error|task.*failed}"
OPENCLAW_DIAG_ACTION="${OPENCLAW_DIAG_ACTION:-log}"
OPENCLAW_DIAG_FAILURES_BEFORE_ACTION="${OPENCLAW_DIAG_FAILURES_BEFORE_ACTION:-2}"
OPENCLAW_DIAG_COMMAND="${OPENCLAW_DIAG_COMMAND:-}"
DASHBOARD_ENABLED="${DASHBOARD_ENABLED:-1}"
DASHBOARD_HOST="${DASHBOARD_HOST:-127.0.0.1}"
DASHBOARD_PORT="${DASHBOARD_PORT:-18790}"
DASHBOARD_ACTIONS_ENABLED="${DASHBOARD_ACTIONS_ENABLED:-0}"
DASHBOARD_TOKEN="${DASHBOARD_TOKEN:-}"
DASHBOARD_DIR="${DASHBOARD_DIR:-${SCRIPT_DIR}/dashboard}"
DASHBOARD_PYTHON="${DASHBOARD_PYTHON:-}"
MODEL_PROBE_ENABLED="${MODEL_PROBE_ENABLED:-0}"
MODEL_EDGE_PROBE_ENABLED="${MODEL_EDGE_PROBE_ENABLED:-1}"
MODEL_PROBE_INTERVAL="${MODEL_PROBE_INTERVAL:-1800}"
MODEL_PROBE_TIMEOUT="${MODEL_PROBE_TIMEOUT:-120}"
MODEL_PROBE_FAILURES_BEFORE_ACTION="${MODEL_PROBE_FAILURES_BEFORE_ACTION:-2}"
MODEL_PROBE_ACTION="${MODEL_PROBE_ACTION:-log}"
MODEL_PROBE_COMMAND="${MODEL_PROBE_COMMAND:-}"
MODEL_PROBE_MODEL="${MODEL_PROBE_MODEL:-}"
MODEL_PROBE_THINKING="${MODEL_PROBE_THINKING:-off}"
MODEL_PROBE_SESSION_ID="${MODEL_PROBE_SESSION_ID:-watchdog-model-probe}"
MODEL_PROBE_MESSAGE="${MODEL_PROBE_MESSAGE:-Reply with exactly OK.}"

BASE_INTERVAL="${BASE_INTERVAL:-60}"
NIGHT_INTERVAL="${NIGHT_INTERVAL:-300}"
MAX_INTERVAL="${MAX_INTERVAL:-1800}"
CONNECT_TIMEOUT="${CONNECT_TIMEOUT:-5}"
MAX_TIME="${MAX_TIME:-10}"
CHANNEL_FAILURES_BEFORE_RESTART="${CHANNEL_FAILURES_BEFORE_RESTART:-2}"
SUCCESS_COUNT_TO_RESET="${SUCCESS_COUNT_TO_RESET:-5}"
MAX_RESTARTS_PER_HOUR="${MAX_RESTARTS_PER_HOUR:-6}"
POST_RESTART_SLEEP="${POST_RESTART_SLEEP:-30}"

if [ -f "$CONFIG_FILE" ]; then
  # shellcheck disable=SC1090
  . "$CONFIG_FILE"
fi

mkdir -p "$STATE_DIR" "$RUNTIME_DIR" "$(dirname "$LOG_FILE")"

cleanup() {
  rmdir "$LOCK_DIR" 2>/dev/null || true
}

if ! mkdir "$LOCK_DIR" 2>/dev/null; then
  echo "Another watchdog instance is already running: $LOCK_DIR" >&2
  exit 0
fi
trap cleanup EXIT INT TERM

log() {
  printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*" | tee -a "$LOG_FILE"
}

rotate_log_if_needed() {
  local max_bytes="${MAX_LOG_BYTES:-1048576}"
  [ -f "$LOG_FILE" ] || return 0
  local size
  size=$(wc -c <"$LOG_FILE" 2>/dev/null || echo 0)
  if [ "${size:-0}" -gt "$max_bytes" ]; then
    mv "$LOG_FILE" "${LOG_FILE}.1" 2>/dev/null || : >"$LOG_FILE"
    log "LOG: rotated because it exceeded ${max_bytes} bytes"
  fi
}

have_cmd() {
  command -v "$1" >/dev/null 2>&1
}

native_probes_enabled() {
  case "${OPENCLAW_NATIVE_PROBES:-auto}" in
    0|false|False|FALSE|off|Off|OFF|no|No|NO) return 1 ;;
    *) return 0 ;;
  esac
}

probe_output_has() {
  local file="$1"
  local pattern="$2"
  [ -f "$file" ] && grep -Eiq "$pattern" "$file"
}

log_probe_tail() {
  local label="$1"
  local file="$2"
  [ -f "$file" ] || return 0
  local line
  line=$(grep -Eiv '^[[:space:]]*$' "$file" | head -n 1 | cut -c 1-220)
  [ -n "$line" ] && log "${label}: ${line}"
}

probe_output_is_unsupported() {
  local file="$1"
  probe_output_has "$file" 'unknown command|unknown option|unknown argument|unrecognized option|not recognized|invalid command'
}

run_openclaw_probe() {
  local label="$1"
  local outfile="$2"
  shift 2
  "$@" >"$outfile" 2>&1
}

check_openclaw_gateway_rpc() {
  native_probes_enabled || return 2
  have_cmd openclaw || return 2

  local out_file="${STATE_DIR}/last-openclaw-gateway-status.json"
  if run_openclaw_probe "gateway status" "$out_file" \
    openclaw gateway status --json --require-rpc --timeout "$OPENCLAW_HEALTH_TIMEOUT_MS"; then
    if probe_output_has "$out_file" '"ok"[[:space:]]*:[[:space:]]*false'; then
      log "OPENCLAW GATEWAY FAIL: gateway status reported ok=false"
      log_probe_tail "OPENCLAW GATEWAY DETAIL" "$out_file"
      return 1
    fi
    if probe_output_has "$out_file" '"degraded"[[:space:]]*:[[:space:]]*true'; then
      log "OPENCLAW GATEWAY WARN: gateway RPC probe is reachable but degraded"
      [ "${OPENCLAW_GATEWAY_STRICT:-0}" = "1" ] && return 1
    fi
    return 0
  fi

  local rc=$?
  if [ "${OPENCLAW_NATIVE_PROBES:-auto}" = "auto" ] && probe_output_is_unsupported "$out_file"; then
    log "OPENCLAW GATEWAY WARN: native gateway status probe is unsupported; falling back to local probes"
    return 2
  fi
  log "OPENCLAW GATEWAY WARN: openclaw gateway status --require-rpc failed (rc=${rc}); falling back to local probes"
  log_probe_tail "OPENCLAW GATEWAY DETAIL" "$out_file"
  return 2
}

check_openclaw_health_snapshot() {
  native_probes_enabled || return 2
  have_cmd openclaw || return 2

  local out_file="${STATE_DIR}/last-openclaw-health.json"
  if run_openclaw_probe "health" "$out_file" \
    openclaw health --json --verbose --timeout "$OPENCLAW_HEALTH_TIMEOUT_MS"; then
    if probe_output_has "$out_file" '"ok"[[:space:]]*:[[:space:]]*false'; then
      log "OPENCLAW CHANNEL FAIL: health snapshot reported ok=false"
      log_probe_tail "OPENCLAW HEALTH DETAIL" "$out_file"
      return 1
    fi
    return 0
  fi

  local rc=$?
  if [ "${OPENCLAW_NATIVE_PROBES:-auto}" = "auto" ] && probe_output_is_unsupported "$out_file"; then
    log "OPENCLAW CHANNEL WARN: native health probe is unsupported; falling back to URL probes"
    return 2
  fi
  log "OPENCLAW CHANNEL FAIL: openclaw health live probe failed (rc=${rc})"
  log_probe_tail "OPENCLAW HEALTH DETAIL" "$out_file"
  return 1
}

check_openclaw_status_deep() {
  native_probes_enabled || return 2
  have_cmd openclaw || return 2

  local out_file="${STATE_DIR}/last-openclaw-status-deep.txt"
  if run_openclaw_probe "status deep" "$out_file" \
    openclaw status --deep; then
    if probe_output_has "$out_file" 'logged[ -]?out|loggedOut|disconnected|unhealthy|healthy[[:space:]]*:[[:space:]]*false|probe[[:space:]-]*failed|status[[:space:]]*:[[:space:]]*(409|410|411|412|413|414|415|500|501|502|503|504|515)|timeout'; then
      log "OPENCLAW CHANNEL FAIL: status --deep contains failure keywords"
      log_probe_tail "OPENCLAW STATUS DETAIL" "$out_file"
      return 1
    fi
    return 0
  fi

  local rc=$?
  if [ "${OPENCLAW_NATIVE_PROBES:-auto}" = "auto" ] && probe_output_is_unsupported "$out_file"; then
    log "OPENCLAW CHANNEL WARN: status --deep is unsupported; falling back to other probes"
    return 2
  fi
  log "OPENCLAW CHANNEL FAIL: openclaw status --deep failed (rc=${rc})"
  log_probe_tail "OPENCLAW STATUS DETAIL" "$out_file"
  return 1
}

check_openclaw_channels_status() {
  native_probes_enabled || return 2
  have_cmd openclaw || return 2
  [ "${OPENCLAW_CHANNELS_PROBE:-1}" = "1" ] || return 2

  local out_file="${STATE_DIR}/last-openclaw-channels-status.txt"
  if run_openclaw_probe "channels status" "$out_file" \
    openclaw channels status --probe; then
    if probe_output_has "$out_file" 'logged[ -]?out|loggedOut|disconnected|unhealthy|healthy[[:space:]]*:[[:space:]]*false|probe[[:space:]-]*failed|status[[:space:]]*:[[:space:]]*(409|410|411|412|413|414|415|500|501|502|503|504|515)|timeout'; then
      log "OPENCLAW CHANNEL FAIL: channels status contains failure keywords"
      log_probe_tail "OPENCLAW CHANNEL DETAIL" "$out_file"
      return 1
    fi
    return 0
  fi

  local rc=$?
  if [ "${OPENCLAW_NATIVE_PROBES:-auto}" = "auto" ] && probe_output_is_unsupported "$out_file"; then
    log "OPENCLAW CHANNEL WARN: native channels probe is unsupported; falling back to URL probes"
    return 2
  fi
  log "OPENCLAW CHANNEL FAIL: openclaw channels status --probe failed (rc=${rc})"
  log_probe_tail "OPENCLAW CHANNEL DETAIL" "$out_file"
  return 1
}

http_probe() {
  local url="$1"
  local code
  code=$(curl -L -sS -o /dev/null -w '%{http_code}' \
    --connect-timeout "$CONNECT_TIMEOUT" --max-time "$MAX_TIME" "$url" 2>/dev/null || printf '000')

  case "$code" in
    2??|3??|4??) return 0 ;;
    *) return 1 ;;
  esac
}

tcp_probe() {
  [ -n "${GATEWAY_HOST:-}" ] || return 1
  [ -n "${GATEWAY_PORT:-}" ] || return 1
  ( : >"/dev/tcp/${GATEWAY_HOST}/${GATEWAY_PORT}" ) >/dev/null 2>&1
}

systemd_user_available() {
  have_cmd systemctl && systemctl --user status >/dev/null 2>&1
}

service_unit_name() {
  case "$GATEWAY_SERVICE" in
    *.service) printf '%s\n' "$GATEWAY_SERVICE" ;;
    *) printf '%s.service\n' "$GATEWAY_SERVICE" ;;
  esac
}

gateway_unit_exists() {
  local unit
  systemd_user_available || return 1
  unit=$(service_unit_name)
  [ "$(systemctl --user show "$unit" -p LoadState --value 2>/dev/null || echo not-found)" != "not-found" ]
}

gateway_unit_active() {
  systemd_user_available || return 1
  systemctl --user is-active --quiet "$GATEWAY_SERVICE" >/dev/null 2>&1
}

check_gateway() {
  local native_status=2
  check_openclaw_gateway_rpc
  native_status=$?
  case "$native_status" in
    0) return 0 ;;
    1) return 1 ;;
  esac

  if [ -n "${GATEWAY_HEALTH_URL:-}" ] && http_probe "$GATEWAY_HEALTH_URL"; then
    return 0
  fi

  if tcp_probe; then
    return 0
  fi

  if gateway_unit_exists; then
    if gateway_unit_active; then
      log "GATEWAY WARN: service is active, but local health/port probe failed"
      return 2
    fi
    log "GATEWAY FAIL: service ${GATEWAY_SERVICE} is not active"
    return 1
  fi

  if have_cmd pgrep && pgrep -f 'openclaw.*gateway' >/dev/null 2>&1; then
    log "GATEWAY WARN: process exists, but local health/port probe failed"
    return 2
  fi

  log "GATEWAY FAIL: no healthy gateway detected"
  return 1
}

check_channel() {
  local native_status=2
  check_openclaw_health_snapshot
  native_status=$?
  case "$native_status" in
    0) return 0 ;;
    1)
      check_openclaw_status_deep >/dev/null 2>&1 || true
      check_openclaw_channels_status >/dev/null 2>&1 || true
      return 1
      ;;
    2)
      check_openclaw_status_deep
      native_status=$?
      [ "$native_status" -eq 0 ] && return 0
      [ "$native_status" -eq 1 ] && return 1
      check_openclaw_channels_status
      native_status=$?
      [ "$native_status" -eq 0 ] && return 0
      [ "$native_status" -eq 1 ] && return 1
      ;;
  esac

  http_probe "$CHANNEL_URL"
}

check_network() {
  local success=0
  local total=0
  local url

  for url in $NETWORK_URLS; do
    total=$((total + 1))
    if http_probe "$url"; then
      success=$((success + 1))
    fi
  done

  if [ "$success" -gt 0 ]; then
    return 0
  fi

  log "NETWORK FAIL: all ${total} network probes failed"
  return 1
}

model_probe_enabled() {
  case "${MODEL_PROBE_ENABLED:-0}" in
    1|true|True|TRUE|on|On|ON|yes|Yes|YES) return 0 ;;
    *) return 1 ;;
  esac
}

diag_enabled() {
  case "${OPENCLAW_DIAG_ENABLED:-1}" in
    0|false|False|FALSE|off|Off|OFF|no|No|NO) return 1 ;;
    *) return 0 ;;
  esac
}

log_scan_enabled() {
  case "${OPENCLAW_LOG_SCAN_ENABLED:-1}" in
    0|false|False|FALSE|off|Off|OFF|no|No|NO) return 1 ;;
    *) return 0 ;;
  esac
}

dashboard_enabled() {
  case "${DASHBOARD_ENABLED:-1}" in
    0|false|False|FALSE|off|Off|OFF|no|No|NO) return 1 ;;
    *) return 0 ;;
  esac
}

dashboard_actions_enabled() {
  case "${DASHBOARD_ACTIONS_ENABLED:-1}" in
    1|true|True|TRUE|on|On|ON|yes|Yes|YES) return 0 ;;
    *) return 1 ;;
  esac
}

model_edge_probe_enabled() {
  case "${MODEL_EDGE_PROBE_ENABLED:-1}" in
    0|false|False|FALSE|off|Off|OFF|no|No|NO) return 1 ;;
    *) return 0 ;;
  esac
}

json_field() {
  local file="$1"
  local key="$2"
  grep -Eo "\"${key}\"[[:space:]]*:[[:space:]]*\"[^\"]+\"" "$file" 2>/dev/null |
    head -n 1 |
    sed -E "s/.*\"${key}\"[[:space:]]*:[[:space:]]*\"([^\"]+)\".*/\\1/"
}

model_probe_due() {
  local now="$1"
  local last="$2"
  [ "$last" -eq 0 ] && return 0
  [ "$((now - last))" -ge "${MODEL_PROBE_INTERVAL:-1800}" ]
}

interval_due() {
  local now="$1"
  local last="$2"
  local interval="$3"
  [ "$last" -eq 0 ] && return 0
  [ "$((now - last))" -ge "$interval" ]
}

dashboard_pid_file() {
  printf '%s/dashboard.pid' "$STATE_DIR"
}

find_dashboard_python() {
  if [ -n "${DASHBOARD_PYTHON:-}" ] && have_cmd "$DASHBOARD_PYTHON"; then
    printf '%s\n' "$DASHBOARD_PYTHON"
    return 0
  fi
  if have_cmd python3; then
    printf '%s\n' "python3"
    return 0
  fi
  if have_cmd python; then
    printf '%s\n' "python"
    return 0
  fi
  return 1
}

dashboard_alive() {
  local pid_file
  local pid
  pid_file="$(dashboard_pid_file)"
  [ -f "$pid_file" ] || return 1
  pid="$(cat "$pid_file" 2>/dev/null || true)"
  [ -n "$pid" ] || return 1
  kill -0 "$pid" 2>/dev/null
}

start_dashboard() {
  dashboard_enabled || return 0
  dashboard_alive && return 0
  [ -f "${DASHBOARD_DIR}/server.py" ] || {
    log "DASHBOARD WARN: ${DASHBOARD_DIR}/server.py not found; dashboard disabled for this run"
    DASHBOARD_ENABLED=0
    return 0
  }

  local python_bin
  python_bin="$(find_dashboard_python 2>/dev/null || true)"
  [ -n "$python_bin" ] || {
    log "DASHBOARD WARN: python3/python not found; dashboard disabled for this run"
    DASHBOARD_ENABLED=0
    return 0
  }

  local actions="0"
  dashboard_actions_enabled && actions="1"
  WATCHDOG_STATE_DIR="$STATE_DIR" \
  WATCHDOG_CONFIG_FILE="$CONFIG_FILE" \
  WATCHDOG_DASHBOARD_HOST="$DASHBOARD_HOST" \
  WATCHDOG_DASHBOARD_PORT="$DASHBOARD_PORT" \
  WATCHDOG_DASHBOARD_ACTIONS_ENABLED="$actions" \
  WATCHDOG_DASHBOARD_TOKEN="$DASHBOARD_TOKEN" \
    nohup "$python_bin" "${DASHBOARD_DIR}/server.py" >>"${STATE_DIR}/dashboard.log" 2>&1 &
  printf '%s\n' "$!" >"$(dashboard_pid_file)"
  log "DASHBOARD: http://${DASHBOARD_HOST}:${DASHBOARD_PORT}"
}

log_signal_categories() {
  local file="$1"
  local categories=""
  [ -s "$file" ] || return 0

  if probe_output_has "$file" 'fetch failed|fetch timeout|LLM idle timeout|model silent|chat/completions|providerRuntimeFailureKind.*timeout'; then
    categories="${categories},provider_timeout"
  fi
  if probe_output_has "$file" 'ECONNRESET|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|socket hang up|TLS|CERT_|proxy'; then
    categories="${categories},proxy_or_network"
  fi
  if probe_output_has "$file" '429|rate limit|too many requests|quota'; then
    categories="${categories},provider_rate_limit"
  fi
  if probe_output_has "$file" '401|403|unauthorized|forbidden|invalid api key|auth'; then
    categories="${categories},provider_auth"
  fi
  if probe_output_has "$file" 'embedded abort settle timed out'; then
    categories="${categories},abort_stuck"
  fi
  if probe_output_has "$file" 'memory-core: narrative generation ended with status=timeout|dreaming.*timeout'; then
    categories="${categories},memory_dream_timeout"
  fi
  if probe_output_has "$file" 'restartPending|session expired|errcode=-14|Monitor.*stopped|monitor.*ended'; then
    categories="${categories},channel_session"
  fi
  if probe_output_has "$file" 'event loop|degraded|health-monitor'; then
    categories="${categories},gateway_degraded"
  fi
  if probe_output_has "$file" 'config hot reload|config change detected'; then
    categories="${categories},config_reload"
  fi
  if probe_output_has "$file" 'cron.*error|task.*failed|run failover decision'; then
    categories="${categories},task_runtime"
  fi
  if probe_output_has "$file" '\bwarn\b|\bwarning\b|\berror\b|\bfailed\b|\btimeout\b'; then
    categories="${categories},openclaw_warning"
  fi

  categories="${categories#,}"
  [ -n "$categories" ] && printf '%s\n' "$categories"
}

scan_openclaw_logs() {
  log_scan_enabled || return 2
  have_cmd openclaw || return 2

  local log_file="${STATE_DIR}/last-openclaw-logs.txt"
  local signals_file="${STATE_DIR}/last-openclaw-log-signals.txt"
  local categories_file="${STATE_DIR}/last-openclaw-log-signal-categories.txt"
  local fingerprint_file="${STATE_DIR}/last-openclaw-log-signals.cksum"
  local fingerprint=""
  local previous=""
  local matches=0
  local categories=""

  openclaw logs --plain --limit "$OPENCLAW_LOG_LIMIT" --timeout "$OPENCLAW_LOG_TIMEOUT_MS" >"$log_file" 2>&1 || return 2
  grep -Ei "(${OPENCLAW_LOG_WARN_PATTERNS})|\\b(warn|warning|error|failed|timeout)\\b" "$log_file" 2>/dev/null | tail -n "$OPENCLAW_LOG_SIGNAL_LIMIT" >"$signals_file" || true
  [ -s "$signals_file" ] || return 0

  fingerprint=$(cksum "$signals_file" 2>/dev/null | awk '{print $1 ":" $2}')
  previous=$(cat "$fingerprint_file" 2>/dev/null || true)
  [ "$fingerprint" = "$previous" ] && return 0
  printf '%s\n' "$fingerprint" >"$fingerprint_file"

  matches=$(wc -l <"$signals_file" 2>/dev/null || echo 0)
  categories=$(log_signal_categories "$signals_file")
  [ -n "$categories" ] || categories="unknown"
  printf '%s\n' "$categories" >"$categories_file"
  log "OPENCLAW LOG WARN: categories=${categories} matches=${matches} fingerprint=${fingerprint}"
  log_probe_tail "OPENCLAW LOG DETAIL" "$signals_file"
  return 1
}

run_openclaw_diagnostics() {
  diag_enabled || return 2
  native_probes_enabled || return 2
  have_cmd openclaw || return 2

  local degraded=0
  local signals=0
  local summary_file="${STATE_DIR}/last-openclaw-diagnostics.jsonl"
  local gateway_file="${STATE_DIR}/last-openclaw-gateway-status.json"
  local health_file="${STATE_DIR}/last-openclaw-health.json"
  local models_file="${STATE_DIR}/last-openclaw-model-status.json"
  local status_file="${STATE_DIR}/last-openclaw-status-deep.txt"
  local categories_file="${STATE_DIR}/last-openclaw-log-signal-categories.txt"
  local categories="none"

  openclaw gateway status --json --require-rpc --timeout "$OPENCLAW_HEALTH_TIMEOUT_MS" >"$gateway_file" 2>&1 || true
  openclaw health --json --verbose --timeout "$OPENCLAW_HEALTH_TIMEOUT_MS" >"$health_file" 2>&1 || true
  openclaw models status --json >"$models_file" 2>&1 || true
  openclaw status --deep --no-color >"$status_file" 2>&1 || true

  if probe_output_has "$health_file" '"degraded"[[:space:]]*:[[:space:]]*true|"ok"[[:space:]]*:[[:space:]]*false'; then
    degraded=1
    log "OPENCLAW DIAG WARN: health snapshot reports degraded or not ok"
    log_probe_tail "OPENCLAW HEALTH DETAIL" "$health_file"
  fi
  if probe_output_has "$models_file" '"fallbacks"[[:space:]]*:[[:space:]]*\[\]'; then
    log "OPENCLAW DIAG INFO: no model fallbacks configured"
  fi

  scan_openclaw_logs
  case "$?" in
    1)
      signals=1
      categories=$(cat "$categories_file" 2>/dev/null || printf 'unknown')
      ;;
  esac

  printf '{"ts":"%s","healthDegraded":%s,"newLogSignals":%s,"categories":"%s"}\n' \
    "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$degraded" "$signals" "$categories" >>"$summary_file"

  if [ "$degraded" -eq 1 ] || [ "$signals" -eq 1 ]; then
    return 1
  fi
  return 0
}

handle_diag_failure() {
  local failures="$1"
  if [ "$failures" -lt "${OPENCLAW_DIAG_FAILURES_BEFORE_ACTION:-2}" ]; then
    return 0
  fi

  if ! check_network; then
    log "OPENCLAW DIAG SAFEGUARD: general network probes failed; skip diagnostic action"
    return 0
  fi

  case "${OPENCLAW_DIAG_ACTION:-log}" in
    restart)
      log "OPENCLAW DIAG ACTION: restart gateway after ${failures} consecutive diagnostic warnings"
      restart_gateway || true
      ;;
    command)
      if [ -n "${OPENCLAW_DIAG_COMMAND:-}" ]; then
        log "OPENCLAW DIAG ACTION: ${OPENCLAW_DIAG_COMMAND}"
        sh -c "$OPENCLAW_DIAG_COMMAND" >>"$LOG_FILE" 2>&1 || log "OPENCLAW DIAG ACTION FAIL: command returned non-zero"
      else
        log "OPENCLAW DIAG ACTION WARN: OPENCLAW_DIAG_ACTION=command but OPENCLAW_DIAG_COMMAND is empty"
      fi
      ;;
    log|*)
      log "OPENCLAW DIAG ACTION: log-only after ${failures} consecutive diagnostic warnings"
      ;;
  esac
}

probe_model_api_edge() {
  model_edge_probe_enabled || return 2
  have_cmd openclaw || return 2
  have_cmd curl || return 2

  local status_file="${STATE_DIR}/last-openclaw-model-status.json"
  local provider_file="${STATE_DIR}/last-openclaw-model-provider.json"
  local model="${MODEL_PROBE_MODEL:-}"
  local provider=""
  local base_url=""
  local metrics=""
  local rc=0

  if [ -z "$model" ]; then
    openclaw models status --json >"$status_file" 2>&1 || return 2
    model=$(json_field "$status_file" "defaultModel")
  fi
  [ -n "$model" ] || return 2
  case "$model" in
    */*) provider="${model%%/*}" ;;
    *) return 2 ;;
  esac

  openclaw config get "models.providers.${provider}" >"$provider_file" 2>&1 || return 2
  base_url=$(json_field "$provider_file" "baseUrl")
  [ -n "$base_url" ] || return 2

  metrics=$(curl -L -sS -o /dev/null \
    -w 'code=%{http_code} dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} start=%{time_starttransfer} total=%{time_total}' \
    --connect-timeout "$CONNECT_TIMEOUT" --max-time "$MAX_TIME" "$base_url" 2>&1)
  rc=$?
  printf '%s\n' "$metrics" >"${STATE_DIR}/last-model-api-edge-probe.txt"

  if [ "$rc" -eq 0 ] && printf '%s' "$metrics" | grep -Eq 'code=(2|3|4)[0-9][0-9]'; then
    log "MODEL API EDGE OK: provider=${provider} model=${model} base=${base_url} ${metrics}"
    return 0
  fi

  log "MODEL API EDGE FAIL: provider=${provider} model=${model} base=${base_url} rc=${rc} ${metrics}"
  return 1
}

run_model_probe() {
  model_probe_enabled || return 2
  have_cmd openclaw || {
    log "MODEL PROBE WARN: openclaw CLI not found; skip model probe"
    return 2
  }

  local out_file="${STATE_DIR}/last-openclaw-model-probe.json"
  local history_file="${STATE_DIR}/model-probe-history.jsonl"
  local started
  local ended
  local duration
  local rc
  local status
  local provider
  local model
  local args

  started=$(date '+%s')
  probe_model_api_edge >/dev/null 2>&1 || true
  args=(agent --session-id "$MODEL_PROBE_SESSION_ID" --thinking "$MODEL_PROBE_THINKING" --timeout "$MODEL_PROBE_TIMEOUT" --json --message "$MODEL_PROBE_MESSAGE")
  if [ -n "${MODEL_PROBE_MODEL:-}" ]; then
    args=(agent --session-id "$MODEL_PROBE_SESSION_ID" --model "$MODEL_PROBE_MODEL" --thinking "$MODEL_PROBE_THINKING" --timeout "$MODEL_PROBE_TIMEOUT" --json --message "$MODEL_PROBE_MESSAGE")
  fi

  openclaw "${args[@]}" >"$out_file" 2>&1
  rc=$?
  ended=$(date '+%s')
  duration=$((ended - started))

  status=$(json_field "$out_file" "status")
  provider=$(json_field "$out_file" "provider")
  model=$(json_field "$out_file" "model")
  [ -n "$provider" ] || provider="unknown"
  [ -n "$model" ] || model="${MODEL_PROBE_MODEL:-configured-default}"

  if [ "$rc" -eq 0 ] && [ "$status" = "ok" ]; then
    log "MODEL PROBE OK: provider=${provider} model=${model} duration=${duration}s session=${MODEL_PROBE_SESSION_ID}"
    printf '{"ts":"%s","status":"ok","provider":"%s","model":"%s","durationSeconds":%s}\n' \
      "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$provider" "$model" "$duration" >>"$history_file"
    return 0
  fi

  local reason="failed"
  if probe_output_has "$out_file" 'timeout|timed out|idle timeout|fetch timeout'; then
    reason="timeout"
  elif probe_output_has "$out_file" 'rate limit|429|too many requests'; then
    reason="rate_limited"
  elif probe_output_has "$out_file" '401|403|unauthorized|forbidden|invalid api key|auth'; then
    reason="auth"
  elif probe_output_has "$out_file" 'Model override .* not allowed'; then
    reason="config"
  fi

  log "MODEL PROBE FAIL: reason=${reason} rc=${rc} provider=${provider} model=${model} duration=${duration}s timeout=${MODEL_PROBE_TIMEOUT}s"
  log_probe_tail "MODEL PROBE DETAIL" "$out_file"
  printf '{"ts":"%s","status":"fail","reason":"%s","provider":"%s","model":"%s","durationSeconds":%s,"exitCode":%s}\n' \
    "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$reason" "$provider" "$model" "$duration" "$rc" >>"$history_file"
  return 1
}

handle_model_probe_failure() {
  local failures="$1"
  if [ "$failures" -lt "${MODEL_PROBE_FAILURES_BEFORE_ACTION:-2}" ]; then
    return 0
  fi

  if ! check_network; then
    log "MODEL PROBE SAFEGUARD: general network probes failed; skip model action"
    return 0
  fi

  case "${MODEL_PROBE_ACTION:-log}" in
    restart)
      log "MODEL PROBE ACTION: restart gateway after ${failures} consecutive model probe failures"
      restart_gateway || true
      ;;
    command)
      if [ -n "${MODEL_PROBE_COMMAND:-}" ]; then
        log "MODEL PROBE ACTION: ${MODEL_PROBE_COMMAND}"
        sh -c "$MODEL_PROBE_COMMAND" >>"$LOG_FILE" 2>&1 || log "MODEL PROBE ACTION FAIL: command returned non-zero"
      else
        log "MODEL PROBE ACTION WARN: MODEL_PROBE_ACTION=command but MODEL_PROBE_COMMAND is empty"
      fi
      ;;
    log|*)
      log "MODEL PROBE ACTION: log-only after ${failures} consecutive model probe failures"
      ;;
  esac
}

restart_count_file() {
  printf '%s/restarts.%s' "$STATE_DIR" "$(date '+%Y%m%d%H')"
}

can_restart_now() {
  local file
  file=$(restart_count_file)
  local count=0
  [ -f "$file" ] && count=$(cat "$file" 2>/dev/null || echo 0)
  [ "${count:-0}" -lt "$MAX_RESTARTS_PER_HOUR" ]
}

record_restart() {
  local file
  file=$(restart_count_file)
  local count=0
  [ -f "$file" ] && count=$(cat "$file" 2>/dev/null || echo 0)
  printf '%s\n' "$((count + 1))" >"$file"
  find "$STATE_DIR" -name 'restarts.*' -type f -mtime +2 -exec rm -f {} + 2>/dev/null || true
}

restart_gateway() {
  if ! can_restart_now; then
    log "CIRCUIT OPEN: restart limit reached (${MAX_RESTARTS_PER_HOUR}/hour); skip restart"
    return 1
  fi

  if [ -n "${RESTART_COMMAND:-}" ]; then
    log "ACTION: ${RESTART_COMMAND}"
    if sh -c "$RESTART_COMMAND" >>"$LOG_FILE" 2>&1; then
      record_restart
      return 0
    fi
    log "ACTION FAIL: restart command returned non-zero"
    return 1
  fi

  if gateway_unit_exists; then
    log "ACTION: systemctl --user restart ${GATEWAY_SERVICE}"
    if systemctl --user restart "$GATEWAY_SERVICE" >>"$LOG_FILE" 2>&1; then
      record_restart
      return 0
    fi
    log "ACTION FAIL: systemd restart failed"
    return 1
  fi

  if have_cmd openclaw; then
    log "ACTION: openclaw gateway restart"
    if openclaw gateway restart >>"$LOG_FILE" 2>&1; then
      record_restart
      return 0
    fi
    log "ACTION FAIL: openclaw gateway restart failed"
    return 1
  fi

  log "ACTION FAIL: set RESTART_COMMAND in ${CONFIG_FILE}; no restart method found"
  return 1
}

backoff_interval() {
  local failures="$1"
  local interval="$BASE_INTERVAL"
  local i=1
  while [ "$i" -lt "$failures" ]; do
    interval=$((interval * 2))
    i=$((i + 1))
    [ "$interval" -ge "$MAX_INTERVAL" ] && break
  done
  [ "$interval" -gt "$MAX_INTERVAL" ] && interval="$MAX_INTERVAL"
  printf '%s\n' "$interval"
}

sleep_interval() {
  local hour
  hour=$(date '+%H')
  hour=$((10#$hour))
  if [ "$hour" -ge 1 ] && [ "$hour" -lt 8 ]; then
    printf '%s\n' "$NIGHT_INTERVAL"
  else
    printf '%s\n' "$BASE_INTERVAL"
  fi
}

main() {
  local fail_count=0
  local success_count=0
  local model_fail_count=0
  local diag_fail_count=0
  local last_diag_at=0
  local last_model_probe_at=0
  local gw_status=0
  local now=0
  local wait_for=0

  log "START: OpenClaw Gateway Resilience Guard ${WATCHDOG_VERSION}"
  log "CONFIG: ${CONFIG_FILE}"
  log "TARGET: channel=${CHANNEL_URL} health=${GATEWAY_HEALTH_URL} service=${GATEWAY_SERVICE} native=${OPENCLAW_NATIVE_PROBES} diag=${OPENCLAW_DIAG_ENABLED} model_probe=${MODEL_PROBE_ENABLED} dashboard=${DASHBOARD_ENABLED}"
  start_dashboard

  while true; do
    rotate_log_if_needed
    start_dashboard
    now=$(date '+%s')

    check_gateway
    gw_status=$?
    if [ "$gw_status" -eq 1 ]; then
      fail_count=$((fail_count + 1))
      success_count=0
      log "CRITICAL: gateway appears down; restarting immediately"
      restart_gateway || true
      sleep "$POST_RESTART_SLEEP"
      continue
    fi

    if diag_enabled && interval_due "$now" "$last_diag_at" "$OPENCLAW_DIAG_INTERVAL"; then
      last_diag_at="$now"
      run_openclaw_diagnostics
      case "$?" in
        0) diag_fail_count=0 ;;
        1)
          diag_fail_count=$((diag_fail_count + 1))
          handle_diag_failure "$diag_fail_count"
          ;;
      esac
    fi

    if model_probe_enabled && model_probe_due "$now" "$last_model_probe_at"; then
      last_model_probe_at="$now"
      run_model_probe
      case "$?" in
        0) model_fail_count=0 ;;
        1)
          model_fail_count=$((model_fail_count + 1))
          handle_model_probe_failure "$model_fail_count"
          ;;
      esac
    fi

    if check_channel; then
      if [ "$fail_count" -gt 0 ]; then
        success_count=$((success_count + 1))
        if [ "$success_count" -ge "$SUCCESS_COUNT_TO_RESET" ]; then
          log "RECOVERED: ${SUCCESS_COUNT_TO_RESET} consecutive channel probes succeeded; failure count reset"
          fail_count=0
          success_count=0
        else
          log "RECOVERING: channel probe succeeded (${success_count}/${SUCCESS_COUNT_TO_RESET})"
        fi
      fi
      sleep "$(sleep_interval)"
      continue
    fi

    fail_count=$((fail_count + 1))
    success_count=0
    log "CHANNEL FAIL: ${CHANNEL_URL} failed (${fail_count}/${CHANNEL_FAILURES_BEFORE_RESTART})"

    if ! check_network; then
      log "SAFEGUARD: looks like global network trouble; wait without restarting gateway"
      sleep "$BASE_INTERVAL"
      continue
    fi

    if [ "$fail_count" -lt "$CHANNEL_FAILURES_BEFORE_RESTART" ]; then
      sleep "$BASE_INTERVAL"
      continue
    fi

    wait_for=$(backoff_interval "$fail_count")
    log "BACKOFF: network is reachable; will re-check channel in ${wait_for}s before restart"
    sleep "$wait_for"

    if check_channel; then
      log "RECOVERED: channel came back before restart"
      fail_count=0
      continue
    fi

    restart_gateway || true
    sleep "$POST_RESTART_SLEEP"
  done
}

main "$@"


path: install-watchdog.ps1
content: param(
  [switch]$Yes,
  [switch]$NoStart,
  [string]$ChannelUrl = "https://ilinkai.weixin.qq.com",
  [string]$HealthUrl = "http://127.0.0.1:18789/healthz",
  [string]$RestartCommand = "",
  [string]$InstallDir = ""
)

$ErrorActionPreference = "Stop"
$TaskName = "OpenClaw Gateway Resilience Guard"
$SourceDir = Split-Path -Parent $MyInvocation.MyCommand.Path
if (-not $InstallDir) {
  $InstallDir = Join-Path $env:LOCALAPPDATA "openclaw-gateway-watchdog"
}
$ConfigDir = Join-Path $env:APPDATA "openclaw-gateway-watchdog"
$StateDir = Join-Path $env:LOCALAPPDATA "openclaw-gateway-watchdog"
$ConfigFile = Join-Path $ConfigDir "watchdog.json"

New-Item -ItemType Directory -Force -Path $InstallDir, $ConfigDir, $StateDir | Out-Null
Copy-Item -LiteralPath (Join-Path $SourceDir "gateway-watchdog.ps1") -Destination $InstallDir -Force
Copy-Item -LiteralPath (Join-Path $SourceDir "uninstall-watchdog.ps1") -Destination $InstallDir -Force
if (Test-Path -LiteralPath (Join-Path $SourceDir "dashboard")) {
  Copy-Item -LiteralPath (Join-Path $SourceDir "dashboard") -Destination $InstallDir -Recurse -Force
}
foreach ($name in @("README.md", "README.zh-CN.md", "SKILL.md", "LICENSE")) {
  $path = Join-Path $SourceDir $name
  if (Test-Path -LiteralPath $path) { Copy-Item -LiteralPath $path -Destination $InstallDir -Force }
}

if (-not (Test-Path -LiteralPath $ConfigFile)) {
  $config = [ordered]@{
    GatewayHealthUrl = $HealthUrl
    GatewayHost = "127.0.0.1"
    GatewayPort = 18789
    ChannelUrl = $ChannelUrl
    NetworkUrls = @("https://www.baidu.com", "https://www.qq.com", "https://api.weixin.qq.com")
    RestartCommand = $RestartCommand
    OpenClawNativeProbes = "auto"
    OpenClawHealthTimeoutMs = 12000
    OpenClawGatewayStrict = $false
    OpenClawChannelsProbe = $true
    OpenClawDiagEnabled = $true
    OpenClawDiagInterval = 300
    OpenClawLogScanEnabled = $true
    OpenClawLogLimit = 200
    OpenClawLogSignalLimit = 40
    OpenClawLogTimeoutMs = 15000
    OpenClawLogWarnPatterns = "fetch failed|fetch timeout|LLM idle timeout|model silent|chat/completions|providerRuntimeFailureKind|ECONNRESET|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|socket hang up|TLS|CERT_|proxy|429|rate limit|quota|unauthorized|invalid api key|embedded abort settle timed out|embedded run failover decision|memory-core: narrative generation ended with status=timeout|dreaming.*timeout|health-monitor|event loop|degraded|restartPending|session expired|errcode=-14|Monitor.*stopped|monitor.*ended|config hot reload|config change detected|cron.*error|task.*failed"
    OpenClawDiagAction = "log"
    OpenClawDiagFailuresBeforeAction = 2
    OpenClawDiagCommand = ""
    DashboardEnabled = $true
    DashboardHost = "127.0.0.1"
    DashboardPort = 18790
    DashboardActionsEnabled = $true
    DashboardToken = (([guid]::NewGuid().ToString("N")) + ([guid]::NewGuid().ToString("N")).Substring(0, 16))
    DashboardDir = (Join-Path $InstallDir "dashboard")
    DashboardPython = ""
    ModelProbeEnabled = $false
    ModelEdgeProbeEnabled = $true
    ModelProbeInterval = 1800
    ModelProbeTimeout = 120
    ModelProbeFailuresBeforeAction = 2
    ModelProbeAction = "log"
    ModelProbeCommand = ""
    ModelProbeModel = ""
    ModelProbeThinking = "off"
    ModelProbeSessionId = "watchdog-model-probe"
    ModelProbeMessage = "Reply with exactly OK."
    BaseInterval = 60
    NightInterval = 300
    MaxInterval = 1800
    ChannelFailuresBeforeRestart = 2
    SuccessCountToReset = 5
    MaxRestartsPerHour = 6
    PostRestartSleep = 30
  }
  $config | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $ConfigFile -Encoding utf8
} else {
  Write-Host "Keeping existing config: $ConfigFile"
}

$ScriptPath = Join-Path $InstallDir "gateway-watchdog.ps1"
$Action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-NoProfile -ExecutionPolicy Bypass -File `"$ScriptPath`" -ConfigPath `"$ConfigFile`""
$Trigger = New-ScheduledTaskTrigger -AtLogOn
$Settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -MultipleInstances IgnoreNew -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)
$CurrentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
$Principal = New-ScheduledTaskPrincipal -UserId $CurrentUser -LogonType Interactive -RunLevel LeastPrivilege

if (Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue) {
  Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false
}
Register-ScheduledTask -TaskName $TaskName -Action $Action -Trigger $Trigger -Settings $Settings -Principal $Principal -Description "External OpenClaw Gateway and channel resilience guard." | Out-Null

if (-not $NoStart) {
  Start-ScheduledTask -TaskName $TaskName
}

Write-Host "Installed Windows Task Scheduler task: $TaskName"
Write-Host "Config: $ConfigFile"
Write-Host "Log:    $(Join-Path $StateDir 'watchdog.log')"
Write-Host "Dashboard: http://127.0.0.1:18790"
Write-Host "Remove: powershell -ExecutionPolicy Bypass -File `"$InstallDir\uninstall-watchdog.ps1`""


path: install-watchdog.sh
content: #!/usr/bin/env bash
# Install OpenClaw Gateway Resilience Guard for the current Linux/WSL/macOS user.

set -euo pipefail

SERVICE_NAME="gateway-watchdog"
PLIST_LABEL="ai.clawhub.gateway-resilience-guard"
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
INSTALL_DIR="${INSTALL_DIR:-${HOME}/.local/share/openclaw-gateway-watchdog}"
CONFIG_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/openclaw-gateway-watchdog"
STATE_DIR="${XDG_STATE_HOME:-${HOME}/.local/state}/openclaw-gateway-watchdog"
CONFIG_FILE="${CONFIG_DIR}/watchdog.env"
SYSTEMD_USER_DIR="${HOME}/.config/systemd/user"
SERVICE_FILE="${SYSTEMD_USER_DIR}/${SERVICE_NAME}.service"
LAUNCH_AGENTS_DIR="${HOME}/Library/LaunchAgents"
PLIST_FILE="${LAUNCH_AGENTS_DIR}/${PLIST_LABEL}.plist"

CHANNEL_URL="${CHANNEL_URL:-}"
GATEWAY_SERVICE="${GATEWAY_SERVICE:-}"
GATEWAY_HEALTH_URL="${GATEWAY_HEALTH_URL:-}"
RESTART_COMMAND="${RESTART_COMMAND:-}"
YES=0
NO_START=0

usage() {
  cat <<'USAGE'
Usage:
  bash install-watchdog.sh [options]

Platforms:
  Linux/WSL: installs a user systemd service or direct-run fallback.
  macOS:     installs a LaunchAgent.
  Windows:   use install-watchdog.ps1 from PowerShell instead.

Options:
  --yes                         Non-interactive install with defaults.
  --no-start                    Install files/service but do not start it.
  --channel-url URL             Main channel URL to probe.
  --gateway-service NAME        systemd user service name, default auto/openclaw-gateway.
  --health-url URL              Local gateway health URL, default http://127.0.0.1:18789/healthz.
  --restart-command COMMAND     Explicit command used to restart gateway.
  --install-dir DIR             Override install directory.
  -h, --help                    Show this help.
USAGE
}

while [ "$#" -gt 0 ]; do
  case "$1" in
    --yes|-y) YES=1 ;;
    --no-start) NO_START=1 ;;
    --channel-url) CHANNEL_URL="${2:?missing URL}"; shift ;;
    --gateway-service|--service) GATEWAY_SERVICE="${2:?missing service name}"; shift ;;
    --health-url) GATEWAY_HEALTH_URL="${2:?missing URL}"; shift ;;
    --restart-command) RESTART_COMMAND="${2:?missing command}"; shift ;;
    --install-dir) INSTALL_DIR="${2:?missing dir}"; shift ;;
    -h|--help) usage; exit 0 ;;
    *) echo "Unknown option: $1" >&2; usage; exit 2 ;;
  esac
  shift
done

case "$(uname -s 2>/dev/null || echo unknown)" in
  MINGW*|MSYS*|CYGWIN*)
    echo "Please run install-watchdog.ps1 from PowerShell on Windows, or run this installer inside WSL/Linux/macOS." >&2
    exit 1
    ;;
esac

have_cmd() {
  command -v "$1" >/dev/null 2>&1
}

systemd_user_available() {
  have_cmd systemctl && systemctl --user status >/dev/null 2>&1
}

unit_exists() {
  local unit="$1"
  local service_unit
  systemd_user_available || return 1
  case "$unit" in
    *.service) service_unit="$unit" ;;
    *) service_unit="${unit}.service" ;;
  esac
  [ "$(systemctl --user show "$service_unit" -p LoadState --value 2>/dev/null || echo not-found)" != "not-found" ]
}

detect_gateway_service() {
  local candidate
  for candidate in openclaw-gateway openclaw.gateway openclaw; do
    if unit_exists "$candidate"; then
      printf '%s\n' "$candidate"
      return 0
    fi
  done
  printf '%s\n' "openclaw-gateway"
}

default_restart_command() {
  local service="$1"
  if unit_exists "$service"; then
    printf 'systemctl --user restart %s\n' "$service"
  elif have_cmd openclaw; then
    printf '%s\n' 'openclaw gateway restart'
  else
    printf 'systemctl --user restart %s\n' "$service"
  fi
}

prompt_default() {
  local prompt="$1"
  local default="$2"
  local answer=""

  if [ "$YES" -eq 1 ] || [ ! -t 0 ]; then
    printf '%s\n' "$default"
    return 0
  fi

  printf '%s [%s]: ' "$prompt" "$default" >&2
  read -r answer || answer=""
  if [ -n "$answer" ]; then
    printf '%s\n' "$answer"
  else
    printf '%s\n' "$default"
  fi
}

require_cmd() {
  if ! have_cmd "$1"; then
    echo "Missing dependency: $1" >&2
    exit 1
  fi
}

xml_escape() {
  printf '%s' "$1" | sed \
    -e 's/&/\&amp;/g' \
    -e 's/</\&lt;/g' \
    -e 's/>/\&gt;/g' \
    -e 's/"/\&quot;/g'
}

generate_token() {
  if have_cmd openssl; then
    openssl rand -hex 24
    return 0
  fi
  if [ -r /dev/urandom ]; then
    od -An -N24 -tx1 /dev/urandom | tr -d ' \n'
    printf '\n'
    return 0
  fi
  printf '%s-%s-%s\n' "$(date '+%s')" "$$" "$RANDOM"
}

require_cmd curl

GATEWAY_SERVICE="${GATEWAY_SERVICE:-$(detect_gateway_service)}"
CHANNEL_URL="${CHANNEL_URL:-$(prompt_default "Channel probe URL" "https://ilinkai.weixin.qq.com")}"
GATEWAY_HEALTH_URL="${GATEWAY_HEALTH_URL:-$(prompt_default "Local gateway health URL" "http://127.0.0.1:18789/healthz")}"
RESTART_COMMAND="${RESTART_COMMAND:-$(default_restart_command "$GATEWAY_SERVICE")}"

mkdir -p "$INSTALL_DIR" "$CONFIG_DIR" "$STATE_DIR"
install -m 0755 "${SCRIPT_DIR}/gateway-watchdog.sh" "${INSTALL_DIR}/gateway-watchdog.sh"
install -m 0755 "${SCRIPT_DIR}/uninstall-watchdog.sh" "${INSTALL_DIR}/uninstall-watchdog.sh"
install -m 0644 "${SCRIPT_DIR}/gateway-watchdog.ps1" "${INSTALL_DIR}/gateway-watchdog.ps1" 2>/dev/null || true
install -m 0644 "${SCRIPT_DIR}/install-watchdog.ps1" "${INSTALL_DIR}/install-watchdog.ps1" 2>/dev/null || true
install -m 0644 "${SCRIPT_DIR}/uninstall-watchdog.ps1" "${INSTALL_DIR}/uninstall-watchdog.ps1" 2>/dev/null || true
[ -f "${SCRIPT_DIR}/README.md" ] && install -m 0644 "${SCRIPT_DIR}/README.md" "${INSTALL_DIR}/README.md"
[ -f "${SCRIPT_DIR}/README.zh-CN.md" ] && install -m 0644 "${SCRIPT_DIR}/README.zh-CN.md" "${INSTALL_DIR}/README.zh-CN.md"
if [ -d "${SCRIPT_DIR}/dashboard" ]; then
  rm -rf "${INSTALL_DIR}/dashboard"
  cp -R "${SCRIPT_DIR}/dashboard" "${INSTALL_DIR}/dashboard"
fi

if [ ! -f "$CONFIG_FILE" ]; then
  DASHBOARD_TOKEN_VALUE="$(generate_token)"
  cat >"$CONFIG_FILE" <<EOF
# OpenClaw Gateway Watchdog config.
# Re-run install-watchdog.sh with flags or edit this file to override defaults.
GATEWAY_SERVICE="${GATEWAY_SERVICE}"
GATEWAY_HEALTH_URL="${GATEWAY_HEALTH_URL}"
GATEWAY_HOST="127.0.0.1"
GATEWAY_PORT="18789"
CHANNEL_URL="${CHANNEL_URL}"
NETWORK_URLS="https://www.baidu.com https://www.qq.com https://api.weixin.qq.com"
RESTART_COMMAND="${RESTART_COMMAND}"
OPENCLAW_NATIVE_PROBES="auto"
OPENCLAW_HEALTH_TIMEOUT_MS="12000"
OPENCLAW_GATEWAY_STRICT="0"
OPENCLAW_CHANNELS_PROBE="1"
OPENCLAW_DIAG_ENABLED="1"
OPENCLAW_DIAG_INTERVAL="300"
OPENCLAW_LOG_SCAN_ENABLED="1"
OPENCLAW_LOG_LIMIT="200"
OPENCLAW_LOG_SIGNAL_LIMIT="40"
OPENCLAW_LOG_TIMEOUT_MS="15000"
OPENCLAW_LOG_WARN_PATTERNS="fetch failed|fetch timeout|LLM idle timeout|model silent|chat/completions|providerRuntimeFailureKind|ECONNRESET|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|socket hang up|TLS|CERT_|proxy|429|rate limit|quota|unauthorized|invalid api key|embedded abort settle timed out|embedded run failover decision|memory-core: narrative generation ended with status=timeout|dreaming.*timeout|health-monitor|event loop|degraded|restartPending|session expired|errcode=-14|Monitor.*stopped|monitor.*ended|config hot reload|config change detected|cron.*error|task.*failed"
OPENCLAW_DIAG_ACTION="log"
OPENCLAW_DIAG_FAILURES_BEFORE_ACTION="2"
OPENCLAW_DIAG_COMMAND=""
DASHBOARD_ENABLED="1"
DASHBOARD_HOST="127.0.0.1"
DASHBOARD_PORT="18790"
DASHBOARD_ACTIONS_ENABLED="1"
DASHBOARD_TOKEN="${DASHBOARD_TOKEN_VALUE}"
DASHBOARD_DIR="${INSTALL_DIR}/dashboard"
MODEL_PROBE_ENABLED="0"
MODEL_EDGE_PROBE_ENABLED="1"
MODEL_PROBE_INTERVAL="1800"
MODEL_PROBE_TIMEOUT="120"
MODEL_PROBE_FAILURES_BEFORE_ACTION="2"
MODEL_PROBE_ACTION="log"
MODEL_PROBE_COMMAND=""
MODEL_PROBE_MODEL=""
MODEL_PROBE_THINKING="off"
MODEL_PROBE_SESSION_ID="watchdog-model-probe"
MODEL_PROBE_MESSAGE="Reply with exactly OK."
BASE_INTERVAL="60"
NIGHT_INTERVAL="300"
MAX_INTERVAL="1800"
CHANNEL_FAILURES_BEFORE_RESTART="2"
SUCCESS_COUNT_TO_RESET="5"
MAX_RESTARTS_PER_HOUR="6"
EOF
else
  echo "Keeping existing config: ${CONFIG_FILE}"
fi

if [ "$(uname -s 2>/dev/null || echo unknown)" = "Darwin" ]; then
  mkdir -p "$LAUNCH_AGENTS_DIR"
  cat >"$PLIST_FILE" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>$(xml_escape "$PLIST_LABEL")</string>
  <key>ProgramArguments</key>
  <array>
    <string>$(xml_escape "${INSTALL_DIR}/gateway-watchdog.sh")</string>
  </array>
  <key>EnvironmentVariables</key>
  <dict>
    <key>WATCHDOG_CONFIG</key>
    <string>$(xml_escape "$CONFIG_FILE")</string>
  </dict>
  <key>RunAtLoad</key>
  <true/>
  <key>KeepAlive</key>
  <true/>
  <key>StandardOutPath</key>
  <string>$(xml_escape "${STATE_DIR}/launchd.out.log")</string>
  <key>StandardErrorPath</key>
  <string>$(xml_escape "${STATE_DIR}/launchd.err.log")</string>
</dict>
</plist>
EOF
  if [ "$NO_START" -eq 0 ]; then
    uid=$(id -u)
    launchctl bootout "gui/${uid}" "$PLIST_FILE" >/dev/null 2>&1 || true
    launchctl bootstrap "gui/${uid}" "$PLIST_FILE"
    launchctl enable "gui/${uid}/${PLIST_LABEL}" >/dev/null 2>&1 || true
    launchctl kickstart -k "gui/${uid}/${PLIST_LABEL}" >/dev/null 2>&1 || true
  fi
  echo "Installed macOS LaunchAgent: ${PLIST_LABEL}"
  echo "Status:  launchctl print gui/$(id -u)/${PLIST_LABEL}"
  echo "Logs:    tail -f ${STATE_DIR}/watchdog.log"
else
  mkdir -p "$SYSTEMD_USER_DIR"
  cat >"$SERVICE_FILE" <<EOF
[Unit]
Description=OpenClaw Gateway Resilience Guard
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment=WATCHDOG_CONFIG=${CONFIG_FILE}
ExecStart=${INSTALL_DIR}/gateway-watchdog.sh
Restart=always
RestartSec=20

[Install]
WantedBy=default.target
EOF

  if systemd_user_available; then
    systemctl --user daemon-reload
    systemctl --user enable "$SERVICE_NAME"
    if [ "$NO_START" -eq 0 ]; then
      systemctl --user restart "$SERVICE_NAME"
    fi
    echo "Installed systemd user service: ${SERVICE_NAME}"
    echo "Status:  systemctl --user status ${SERVICE_NAME}"
    echo "Logs:    journalctl --user -u ${SERVICE_NAME} -f"
  else
    echo "User systemd is not available. Installing direct-run fallback."
    if [ "$NO_START" -eq 0 ]; then
      nohup "${INSTALL_DIR}/gateway-watchdog.sh" >>"${STATE_DIR}/bootstrap.log" 2>&1 &
      printf '%s\n' "$!" >"${STATE_DIR}/watchdog.pid"
      echo "Started fallback process with PID $(cat "${STATE_DIR}/watchdog.pid")."
    fi
    echo "For automatic start after reboot, enable user systemd in WSL/Linux and re-run this installer."
  fi
fi

echo ""
echo "Config:  ${CONFIG_FILE}"
echo "Log:     ${STATE_DIR}/watchdog.log"
echo "Dashboard: http://127.0.0.1:18790"
echo "Remove:  bash ${INSTALL_DIR}/uninstall-watchdog.sh"


path: openclaw-plugin/index.js
content: import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";

function escapeHtml(value) {
  return String(value)
    .replaceAll("&", "&amp;")
    .replaceAll("<", "&lt;")
    .replaceAll(">", "&gt;")
    .replaceAll('"', "&quot;");
}

export default definePluginEntry({
  id: "resilience-guard",
  name: "Resilience Guard",
  description: "Opens the external OpenClaw Gateway Resilience Guard dashboard.",
  register(api) {
    if (api.registrationMode !== "full") return;
    const dashboardUrl = String(api.pluginConfig?.dashboardUrl || "http://127.0.0.1:18790/");

    api.registerHttpRoute({
      path: "/resilience-guard",
      auth: "gateway",
      match: "exact",
      replaceExisting: true,
      async handler(_req, res) {
        res.statusCode = 302;
        res.setHeader("Location", dashboardUrl);
        res.end(`Redirecting to ${escapeHtml(dashboardUrl)}`);
        return true;
      },
    });

    api.registerHttpRoute({
      path: "/resilience-guard/info",
      auth: "gateway",
      match: "exact",
      replaceExisting: true,
      async handler(_req, res) {
        const body = JSON.stringify({
          ok: true,
          dashboardUrl,
          note: "The external dashboard is served by the watchdog service, so it can stay available when Gateway is down.",
        });
        res.statusCode = 200;
        res.setHeader("Content-Type", "application/json; charset=utf-8");
        res.end(body);
        return true;
      },
    });
  },
});


path: openclaw-plugin/openclaw.plugin.json
content: {
  "id": "resilience-guard",
  "name": "Resilience Guard",
  "description": "Adds a Gateway Control UI entry that opens the external OpenClaw watchdog dashboard.",
  "version": "1.4.4",
  "configSchema": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "dashboardUrl": {
        "type": "string",
        "default": "http://127.0.0.1:18790/"
      }
    }
  },
  "uiHints": {
    "dashboardUrl": {
      "label": "Dashboard URL",
      "help": "External watchdog dashboard URL. It remains available even when Gateway is down.",
      "placeholder": "http://127.0.0.1:18790/"
    }
  }
}


path: openclaw-plugin/package.json
content: {
  "name": "@zc-kama/openclaw-resilience-guard",
  "version": "1.4.4",
  "type": "module",
  "description": "OpenClaw plugin bridge for the external Gateway Resilience Guard dashboard.",
  "license": "MIT-0",
  "openclaw": {
    "extensions": [
      "./index.js"
    ],
    "compat": {
      "pluginApi": ">=2026.3.24-beta.2",
      "minGatewayVersion": "2026.3.24-beta.2"
    },
    "build": {
      "openclawVersion": "2026.5.18",
      "pluginSdkVersion": "2026.5.18"
    }
  }
}


path: README-watchdog.md
content: # OpenClaw Gateway 看门狗

这是项目的中文快捷入口。完整中文文档见 [README.zh-CN.md](README.zh-CN.md)，英文文档见 [README.md](README.md)。

最简单安装：

```bash
bash install-watchdog.sh
```

Windows：

```powershell
powershell -ExecutionPolicy Bypass -File .\install-watchdog.ps1
```

默认配置即可启动；需要指定自己的通道地址时：

```bash
bash install-watchdog.sh --channel-url "https://你的通道地址/health"
```

安装后还有一个独立图形化控制台：

```text
http://127.0.0.1:18790/
```

它由 watchdog 自己提供，不依赖 OpenClaw Gateway。Gateway 挂掉时，这个页面仍然可以看状态、图表、策略、日志和诊断导出。Dashboard 支持中英文切换、主题切换、事件趋势图、异常分类图和受保护操作解锁。

安装后它会默认采集 OpenClaw 诊断快照和日志信号，包括 gateway status、health、models status、status --deep 和 `openclaw logs --plain` 的 WARN/ERROR 摘要。关键证据在：

```text
~/.local/state/openclaw-gateway-watchdog/watchdog.log
~/.local/state/openclaw-gateway-watchdog/last-openclaw-diagnostics.jsonl
~/.local/state/openclaw-gateway-watchdog/last-openclaw-log-signals.txt
```

可选模型探针默认关闭。需要排查模型 API 是否在某个时段超时时，可以在配置里显式开启 `MODEL_PROBE_ENABLED="1"`；它会发送极小的 `openclaw agent --json` 请求，并把结果写入 watchdog 日志和 `model-probe-history.jsonl`。默认动作仍是只记录证据，不会擅自改 OpenClaw 配置。


path: README.md
content: # OpenClaw Gateway Resilience Guard

External recovery guard for OpenClaw Gateway and long-lived channel plugins such as `openclaw-weixin`.

This project is for people who run OpenClaw continuously on Linux, WSL, macOS, or Windows and need the gateway to recover from channel disconnects, network sleep/resume, and long-lived session failures without babysitting the terminal.

## Problem

OpenClaw Gateway can still be alive while an individual channel is no longer healthy. This is common after laptop sleep, Wi-Fi changes, WSL network hiccups, or long idle periods.

The WeChat plugin is especially sensitive because it depends on a long-poll `getUpdates` loop. In the upstream `Tencent/openclaw-weixin` code, the monitor has a limited retry loop and session guard:

- `monitor.ts` defines `MAX_CONSECUTIVE_FAILURES = 3` and `BACKOFF_DELAY_MS = 30_000`.
- `session-guard.ts` defines `SESSION_PAUSE_DURATION_MS = 60 * 60 * 1000` and `SESSION_EXPIRED_ERRCODE = -14`.
- Issue [Tencent/openclaw-weixin#141](https://github.com/Tencent/openclaw-weixin/issues/141) reports that after a config hot reload the monitor can end without starting again; the workaround is `openclaw gateway restart`.
- Issue [Tencent/openclaw-weixin#155](https://github.com/Tencent/openclaw-weixin/issues/155) reports that `errcode=-14` can enter a 60-minute session pause loop and block outbound messages.

This watchdog does not replace the official plugin. It is an external safety net: when the gateway or channel stops behaving like a live system, it restarts the gateway with guardrails.

## Design

The script uses a layered health model before it restarts anything:

| Layer | Probe | Purpose |
| --- | --- | --- |
| Gateway | `openclaw gateway status --json --require-rpc`, local health URL, local TCP port, service/process fallback | Detect whether OpenClaw Gateway is down or locally unreachable. |
| Channel | `openclaw health --json --verbose`, `openclaw status --deep`, optional `openclaw channels status --probe`, then URL fallback | Prefer OpenClaw's own per-channel health model, then fall back to a configured URL when native probes are unavailable. |
| Runtime diagnostics | `openclaw models status --json`, `openclaw logs --plain`, warning classification | Separate provider, proxy/network, auth/rate-limit, gateway, channel-session, config reload, and task-runtime evidence before choosing an action. |
| Model API, optional | `openclaw agent --json` with the configured model provider | Detect whether the configured model path is timing out while Gateway and channels still look healthy. Disabled by default because it makes real model calls. |
| Network | multiple independent URLs, default Baidu/QQ/Weixin | Avoid restarting the gateway during whole-machine or ISP network failure. |

Only gateway failures restart immediately. Channel failures go through confirmation, network split-brain protection, exponential backoff, and restart-rate limits.
Model probe failures default to evidence logging only; users can opt in to restart or a custom command after consecutive failures.

## Recovery Policy

- Gateway down: restart immediately.
- Gateway running but channel probe fails: confirm with general network probes.
- General network also fails: do nothing except wait; restarting will not fix an offline machine.
- General network works but channel stays down: wait with exponential backoff, re-check, then restart.
- OpenClaw warning logs: classify and record evidence first; default action is log-only.
- Five consecutive successful channel probes reset the failure state.
- Restart storm protection limits gateway restarts per hour.
- Night hours can use a slower probe interval to reduce noise.

## Files

| File | Purpose |
| --- | --- |
| `gateway-watchdog.sh` | Main daemon loop: probes, backoff, restart decisions, log rotation, single-instance lock. |
| `gateway-watchdog.ps1` | Windows-native daemon loop for Task Scheduler. |
| `dashboard/` | Standalone local Web UI and API served by the watchdog, independent of Gateway. |
| `openclaw-plugin/` | Optional native OpenClaw plugin bridge that redirects `/resilience-guard` to the standalone dashboard. |
| `install-watchdog.sh` | Linux/WSL/macOS installer: copies files, writes config, creates systemd user service or macOS LaunchAgent. |
| `install-watchdog.ps1` | Windows installer: creates config and a Task Scheduler job. |
| `uninstall-watchdog.sh` | Stops and removes the service and installed scripts. |
| `uninstall-watchdog.ps1` | Windows uninstaller. |
| `SKILL.md` | ClawHub/OpenClaw skill metadata and operator instructions. |
| `README.zh-CN.md` | Chinese documentation. |

## Install

Install from ClawHub:

```bash
clawhub install gateway-resilience-guard
```

Or use this repository directly.

Linux, WSL, or macOS:

```bash
bash install-watchdog.sh
```

For unattended install:

```bash
bash install-watchdog.sh --yes
```

For a custom channel probe:

```bash
bash install-watchdog.sh --channel-url "https://your-channel.example.com/health"
```

Windows PowerShell:

```powershell
powershell -ExecutionPolicy Bypass -File .\install-watchdog.ps1
```

## Dashboard

The installer enables a standalone local dashboard:

```text
http://127.0.0.1:18790/
```

This dashboard is served by the watchdog, not by OpenClaw Gateway. If Gateway is down, the dashboard can still open and show the last known evidence.

It includes:

- Gateway, channel, network, OpenClaw log, and model-provider status.
- Category charts for provider timeout, proxy/network, rate-limit, auth, channel session, Gateway degraded, config reload, and task runtime warnings.
- Sidebar view switching for Overview, Trends, Strategy, Logs, and Config so monitoring, actions, and detail evidence are separated.
- Local design-system controls: native semantic selects, buttons, badges, and guarded controls styled consistently without relying on CDN or component-script upgrades.
- Denser page composition: trend digest and latest signals, strategy matrix and manual actions, config runtime map.
- Event trend chart with separate lanes for API failures, log warnings, successful model probes, and healthy diagnostics.
- Chinese/English language switching and Light, Dark, Ocean, and Forest themes.
- Status-file freshness checks so stale data is obvious.
- Quick strategy buttons: observe, overnight diagnosis, channel recovery, and conservative circuit breaker.
- Guarded actions with an unlock flow: run diagnostics, restart Gateway, apply presets, and export a diagnostic JSON bundle.

Dashboard actions are bound to localhost and protected with the generated `DASHBOARD_TOKEN`. The token is injected only into the same-origin dashboard page. A random token is written during install; the value is not shown in logs.

Optional OpenClaw plugin bridge:

```bash
openclaw plugins install ./openclaw-plugin
openclaw plugins enable resilience-guard
openclaw gateway restart
```

Then open the Gateway route while Gateway is healthy:

```text
http://127.0.0.1:18789/resilience-guard
```

That route redirects to the external dashboard. It is a convenience entry only; the external dashboard remains the recovery entry when Gateway is unavailable.

The installer writes:

- scripts to `~/.local/share/openclaw-gateway-watchdog`;
- config to `~/.config/openclaw-gateway-watchdog/watchdog.env`;
- logs to `~/.local/state/openclaw-gateway-watchdog/watchdog.log`;
- a Linux/WSL user service to `~/.config/systemd/user/gateway-watchdog.service`;
- a macOS LaunchAgent to `~/Library/LaunchAgents/ai.clawhub.gateway-resilience-guard.plist`;
- a Windows scheduled task named `OpenClaw Gateway Resilience Guard`.

If user systemd is unavailable, the installer starts a direct background fallback process and stores its pid in the state directory.

## Manage

```bash
systemctl --user status gateway-watchdog
journalctl --user -u gateway-watchdog -f
systemctl --user restart gateway-watchdog
bash ~/.local/share/openclaw-gateway-watchdog/uninstall-watchdog.sh
```

macOS:

```bash
launchctl print gui/$(id -u)/ai.clawhub.gateway-resilience-guard
tail -f ~/.local/state/openclaw-gateway-watchdog/watchdog.log
bash ~/.local/share/openclaw-gateway-watchdog/uninstall-watchdog.sh
```

Windows:

```powershell
Get-ScheduledTask -TaskName "OpenClaw Gateway Resilience Guard"
Get-Content "$env:LOCALAPPDATA\openclaw-gateway-watchdog\watchdog.log" -Wait
powershell -ExecutionPolicy Bypass -File "$env:LOCALAPPDATA\openclaw-gateway-watchdog\uninstall-watchdog.ps1"
```

Remove config and logs too:

```bash
bash ~/.local/share/openclaw-gateway-watchdog/uninstall-watchdog.sh --purge
```

## Configuration

Most users can keep the generated defaults. Advanced settings live in:

```text
~/.config/openclaw-gateway-watchdog/watchdog.env
```

Common keys:

```bash
GATEWAY_SERVICE="openclaw-gateway"
GATEWAY_HEALTH_URL="http://127.0.0.1:18789/healthz"
GATEWAY_HOST="127.0.0.1"
GATEWAY_PORT="18789"
CHANNEL_URL="https://ilinkai.weixin.qq.com"
NETWORK_URLS="https://www.baidu.com https://www.qq.com https://api.weixin.qq.com"
RESTART_COMMAND="systemctl --user restart openclaw-gateway"
OPENCLAW_NATIVE_PROBES="auto"
OPENCLAW_HEALTH_TIMEOUT_MS="12000"
OPENCLAW_GATEWAY_STRICT="0"
OPENCLAW_CHANNELS_PROBE="1"
OPENCLAW_DIAG_ENABLED="1"
OPENCLAW_DIAG_INTERVAL="300"
OPENCLAW_LOG_SCAN_ENABLED="1"
OPENCLAW_LOG_LIMIT="200"
OPENCLAW_LOG_SIGNAL_LIMIT="40"
OPENCLAW_LOG_TIMEOUT_MS="15000"
OPENCLAW_LOG_WARN_PATTERNS="fetch failed|fetch timeout|LLM idle timeout|model silent|..."
OPENCLAW_DIAG_ACTION="log"
OPENCLAW_DIAG_FAILURES_BEFORE_ACTION="2"
OPENCLAW_DIAG_COMMAND=""
DASHBOARD_ENABLED="1"
DASHBOARD_HOST="127.0.0.1"
DASHBOARD_PORT="18790"
DASHBOARD_ACTIONS_ENABLED="1"
DASHBOARD_TOKEN="generated-at-install"
DASHBOARD_DIR="~/.local/share/openclaw-gateway-watchdog/dashboard"
MODEL_PROBE_ENABLED="0"
MODEL_EDGE_PROBE_ENABLED="1"
MODEL_PROBE_INTERVAL="1800"
MODEL_PROBE_TIMEOUT="120"
MODEL_PROBE_FAILURES_BEFORE_ACTION="2"
MODEL_PROBE_ACTION="log"
MODEL_PROBE_COMMAND=""
MODEL_PROBE_MODEL=""
MODEL_PROBE_THINKING="off"
MODEL_PROBE_SESSION_ID="watchdog-model-probe"
MODEL_PROBE_MESSAGE="Reply with exactly OK."
BASE_INTERVAL="60"
NIGHT_INTERVAL="300"
MAX_INTERVAL="1800"
CHANNEL_FAILURES_BEFORE_RESTART="2"
SUCCESS_COUNT_TO_RESET="5"
MAX_RESTARTS_PER_HOUR="6"
```

Use `RESTART_COMMAND` if your OpenClaw install is not managed by a user-level systemd unit. Example:

```bash
RESTART_COMMAND="openclaw gateway restart"
```

On Windows, the generated config is JSON:

```text
%APPDATA%\openclaw-gateway-watchdog\watchdog.json
```

Set `OpenClawNativeProbes` to `false` if your OpenClaw CLI is too old for `openclaw health` or `openclaw status --deep`.

### OpenClaw diagnostics and log signals

The watchdog does more than ping one URL. Every `OPENCLAW_DIAG_INTERVAL` seconds it collects an OpenClaw diagnostic snapshot:

```text
~/.local/state/openclaw-gateway-watchdog/last-openclaw-gateway-status.json
~/.local/state/openclaw-gateway-watchdog/last-openclaw-health.json
~/.local/state/openclaw-gateway-watchdog/last-openclaw-model-status.json
~/.local/state/openclaw-gateway-watchdog/last-openclaw-status-deep.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-logs.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-log-signals.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-log-signal-categories.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-diagnostics.jsonl
```

`last-openclaw-log-signals.txt` is filtered from `openclaw logs --plain`. It classifies common failure families such as `provider_timeout`, `proxy_or_network`, `provider_rate_limit`, `provider_auth`, `abort_stuck`, `memory_dream_timeout`, `channel_session`, `gateway_degraded`, `config_reload`, and `task_runtime`.

Default action is `OPENCLAW_DIAG_ACTION="log"` because warnings are evidence, not always proof that a restart is correct. If you explicitly set `OPENCLAW_DIAG_ACTION="restart"` or `command`, the action only runs after consecutive diagnostic warnings and only when the general network probes still pass.

Practical interpretation:

| Evidence | Likely scope | Default strategy |
| --- | --- | --- |
| Gateway status/health is down | Gateway process or RPC path | Restart Gateway immediately through the normal gateway policy. |
| Channel probe fails, network probes pass | Channel/session path | Backoff, re-check, then restart Gateway if still failed. |
| OpenClaw logs show provider timeout, model probe also fails | Provider/API path | Log evidence; optional custom action. A Gateway restart may not fix provider outage. |
| OpenClaw logs show provider timeout, model probe succeeds | OpenClaw runtime, task, session, or specific request path | Keep evidence, inspect logs; avoid blaming the provider alone. |
| Logs show proxy/DNS/TLS errors | Local proxy, DNS, TLS, or ISP route | Log evidence and avoid restart storms; fix network/proxy route first. |
| Logs show session expiry or monitor stopped | Channel plugin/session | Gateway restart is often useful after confirmation. |

### Optional model probe

Set `MODEL_PROBE_ENABLED="1"` when you need to prove whether failures are in the model-provider path instead of Gateway or channel health.

When enabled, the watchdog first reads OpenClaw's configured model provider and probes its provider `baseUrl` without credentials. Then it runs the end-to-end model probe:

```bash
openclaw agent --session-id "$MODEL_PROBE_SESSION_ID" \
  --thinking "$MODEL_PROBE_THINKING" \
  --timeout "$MODEL_PROBE_TIMEOUT" \
  --json \
  --message "$MODEL_PROBE_MESSAGE"
```

If `MODEL_PROBE_MODEL` is empty, OpenClaw's configured default model is used. Results are written to the main log and to:

```text
~/.local/state/openclaw-gateway-watchdog/model-probe-history.jsonl
~/.local/state/openclaw-gateway-watchdog/last-openclaw-model-probe.json
~/.local/state/openclaw-gateway-watchdog/last-model-api-edge-probe.txt
```

Recommended diagnostic settings for overnight provider issues:

```bash
MODEL_PROBE_ENABLED="1"
MODEL_EDGE_PROBE_ENABLED="1"
MODEL_PROBE_INTERVAL="600"
MODEL_PROBE_TIMEOUT="120"
MODEL_PROBE_ACTION="log"
MODEL_PROBE_THINKING="off"
MODEL_PROBE_MESSAGE="Reply with exactly OK."
```

`MODEL_PROBE_ACTION` can be:

- `log`: record evidence only. This is the default.
- `restart`: restart Gateway after `MODEL_PROBE_FAILURES_BEFORE_ACTION` consecutive model failures, but only if general network probes still pass.
- `command`: run `MODEL_PROBE_COMMAND` after consecutive failures.

This feature sends real model requests and may consume provider quota or money. It does not print API keys, but it does store provider/model names, timing, exit status, and the first non-empty error line.
`MODEL_EDGE_PROBE_ENABLED` does not use credentials and does not call `/chat/completions`; it only checks whether the provider API edge such as `https://api.deepseek.com` is reachable quickly.

## Safety Notes

This project intentionally avoids destructive behavior. It does not edit OpenClaw configuration, tokens, sessions, or plugin files. By default it only probes URLs and restarts the gateway through the configured command. The optional model probe makes real model calls only after you explicitly enable it.

Before sharing logs, review them for local paths, service names, and channel URLs.

## License

MIT-0. This matches ClawHub's skill publishing requirement and allows reuse without attribution requirements.

## Publish To ClawHub

Published package:

- ClawHub: <https://clawhub.ai/zc-kama/gateway-resilience-guard>
- Slug: `gateway-resilience-guard`

This repository includes `SKILL.md`, so it can also be republished as an OpenClaw skill bundle:

```bash
clawhub publish . \
  --slug gateway-resilience-guard \
  --name "OpenClaw Gateway Resilience Guard" \
  --version 1.4.4 \
  --changelog "Make dashboard controls robust with native semantic buttons/selects, fixed hover states, centered brand mark, and non-overlapping config rows"
```

ClawHub requires CLI authentication. Run `clawhub login` first.


path: README.zh-CN.md
content: # OpenClaw Gateway Resilience Guard

OpenClaw Gateway 外部恢复守护脚本，面向 `openclaw-weixin` 这类需要长期在线的通道。

这个项目解决的是一个很具体的运维问题：OpenClaw Gateway 进程还活着，但通道、长轮询、WSL/系统网络或 session 状态已经坏掉，导致消息收不到、发不出，最后只能手动 `openclaw gateway restart`。

## 问题背景

OpenClaw Gateway 和通道插件是长连接系统。电脑睡眠、切换 Wi-Fi、WSL 网络重建、iLink 长时间空闲、配置热加载，都可能让“进程存活”和“通道可用”变成两件事。

以官方 `Tencent/openclaw-weixin` 为例，源码和 issue 里能看到几个已知边界：

- `monitor.ts` 里 `MAX_CONSECUTIVE_FAILURES = 3`，失败后 `BACKOFF_DELAY_MS = 30_000`，也就是插件内部主要是 3 次失败后的 30 秒退避。
- `session-guard.ts` 里 `SESSION_PAUSE_DURATION_MS = 60 * 60 * 1000`，`SESSION_EXPIRED_ERRCODE = -14`，session 过期会进入 60 分钟暂停窗口。
- [Tencent/openclaw-weixin#141](https://github.com/Tencent/openclaw-weixin/issues/141) 记录了配置热加载后 Monitor 结束但不再启动，临时处理方式是手动重启 gateway。
- [Tencent/openclaw-weixin#155](https://github.com/Tencent/openclaw-weixin/issues/155) 记录了 `errcode=-14` 后进入 60 分钟循环暂停、出站消息被阻塞的问题。

所以这个项目不是替换官方插件，而是在外面加一层独立 watchdog：当通道或 gateway 进入“看起来还活着，实际上已经不能工作”的状态时，用更保守的探测和熔断策略自动恢复。

## 工作原理

看门狗使用分层健康检查，从浅到深判断是否真的需要重启：

| 层级 | 检查对象 | 作用 |
| --- | --- | --- |
| Gateway 本机状态 | `openclaw gateway status --json --require-rpc`、本机 health URL、本机 TCP 端口、服务/进程兜底 | 判断 OpenClaw Gateway 是否已经挂掉或本机不可达。 |
| 通道状态 | `openclaw health --json --verbose`、`openclaw status --deep`、可选 `openclaw channels status --probe`，最后才是 URL 兜底 | 优先使用 OpenClaw 自己的全通道健康模型；CLI 不支持时再退回 URL 探测。 |
| 运行时诊断 | `openclaw models status --json`、`openclaw logs --plain`、WARN/ERROR 分类 | 在动作前区分 provider、代理/网络、鉴权/限流、gateway、通道 session、配置热加载和任务运行时证据。 |
| 模型 API，可选 | 使用 `openclaw agent --json` 走 OpenClaw 当前配置的模型 provider | 判断 Gateway 和通道都健康时，真正卡住的是不是模型 API 链路。默认关闭，因为它会真实消耗模型调用。 |
| 外部网络状态 | 百度、QQ、微信 API 等多个独立 URL | 排除全局断网，避免电脑没网时误重启 gateway。 |

核心策略是：gateway 真挂了就立即重启；通道不通时先确认不是全局断网；网络正常但通道持续失败，才进入退避等待和重启流程。
模型探针默认只记录证据日志；如果你显式配置，也可以在连续失败后重启 gateway 或执行自定义命令。

## 恢复策略

- Gateway 本机健康检查失败：立即重启。
- 通道 URL 失败：累计失败次数，进入故障流程。
- 外部网络全部失败：认为是全局断网，只等待，不重启。
- 外部网络正常但通道仍失败：指数退避后再次确认，再重启 gateway。
- OpenClaw 日志出现 WARN/ERROR：先分类和记录证据，默认只写日志。
- 连续 5 次通道探测成功后，清空失败状态。
- 每小时最多重启固定次数，防止网络抖动时形成重启风暴。
- 深夜可以降低检查频率，减少无意义日志。

## 文件说明

| 文件 | 作用 |
| --- | --- |
| `gateway-watchdog.sh` | 主守护脚本，负责探测、退避、熔断、日志、单实例锁和重启决策。 |
| `gateway-watchdog.ps1` | Windows 原生守护脚本，用于 Task Scheduler。 |
| `dashboard/` | 独立本地 Web UI 和 API，由 watchdog 自己提供，不依赖 Gateway。 |
| `openclaw-plugin/` | 可选 OpenClaw 原生插件桥接入口，把 `/resilience-guard` 跳转到独立 dashboard。 |
| `install-watchdog.sh` | Linux/WSL/macOS 安装脚本，自动复制文件、生成配置、创建 systemd 用户服务或 macOS LaunchAgent。 |
| `install-watchdog.ps1` | Windows 安装脚本，创建配置和计划任务。 |
| `uninstall-watchdog.sh` | 卸载脚本，停止服务并删除安装目录。 |
| `uninstall-watchdog.ps1` | Windows 卸载脚本。 |
| `SKILL.md` | ClawHub/OpenClaw 技能元数据和使用说明。 |
| `README.md` | 英文文档。 |

## 安装

通过 ClawHub 安装：

```bash
clawhub install gateway-resilience-guard
```

也可以直接使用本仓库。

Linux、WSL 或 macOS：

```bash
bash install-watchdog.sh
```

无人值守安装：

```bash
bash install-watchdog.sh --yes
```

指定自己的通道探测地址：

```bash
bash install-watchdog.sh --channel-url "https://你的通道地址/health"
```

Windows PowerShell：

```powershell
powershell -ExecutionPolicy Bypass -File .\install-watchdog.ps1
```

## 图形化 Dashboard

安装后会启用一个独立本地 dashboard：

```text
http://127.0.0.1:18790/
```

这个页面由 watchdog 自己提供，不依赖 OpenClaw Gateway。所以 Gateway 挂掉时，它仍然可以打开，看到最后一次诊断证据。

它包含：

- Gateway、通道、外部网络、OpenClaw 日志、模型 provider 的分层状态。
- provider timeout、代理/网络、限流、鉴权、通道 session、Gateway degraded、配置热加载、任务运行时异常的分类图表。
- 侧边栏分页：总览、趋势、策略、日志、配置分开呈现，避免把监控、操作和明细堆在一个长页面里。
- 本地设计系统控件：下拉框、按钮、状态标签使用原生语义控件加统一样式，不依赖 CDN 或组件脚本升级。
- 重新分配每页信息密度：趋势页有摘要和最新信号，策略页有策略矩阵和手动操作，配置页有运行地图。
- 事件趋势图，把 API 失败、日志 WARN、模型探针成功、健康诊断分成不同泳道，并保留清晰的时间刻度。
- 中英文语言切换，以及 Light、Dark、Ocean、Forest 四套主题。
- 状态文件新鲜度，避免把过期快照误认为当前状态。
- 快速策略按钮：观察模式、夜间诊断、通道恢复、保守熔断。
- 带解锁流程的受保护操作：立即诊断、重启 Gateway、应用策略、导出诊断 JSON。

Dashboard 操作只绑定 localhost，并使用安装时生成的 `DASHBOARD_TOKEN` 保护。token 只注入同源页面，不会写入日志。

可选 OpenClaw 插件桥接入口：

```bash
openclaw plugins install ./openclaw-plugin
openclaw plugins enable resilience-guard
openclaw gateway restart
```

Gateway 正常时可以打开：

```text
http://127.0.0.1:18789/resilience-guard
```

这个路由会跳转到外部 dashboard。它只是方便入口；真正救急的入口仍然是 `http://127.0.0.1:18790/`。

安装后会生成：

- 脚本目录：`~/.local/share/openclaw-gateway-watchdog`
- 配置文件：`~/.config/openclaw-gateway-watchdog/watchdog.env`
- 日志文件：`~/.local/state/openclaw-gateway-watchdog/watchdog.log`
- Linux/WSL systemd 用户服务：`~/.config/systemd/user/gateway-watchdog.service`
- macOS LaunchAgent：`~/Library/LaunchAgents/ai.clawhub.gateway-resilience-guard.plist`
- Windows 计划任务：`OpenClaw Gateway Resilience Guard`

如果当前环境没有 user systemd，安装脚本会退回到后台进程模式，并把 pid 写到状态目录。

## 管理命令

```bash
systemctl --user status gateway-watchdog
journalctl --user -u gateway-watchdog -f
systemctl --user restart gateway-watchdog
bash ~/.local/share/openclaw-gateway-watchdog/uninstall-watchdog.sh
```

macOS：

```bash
launchctl print gui/$(id -u)/ai.clawhub.gateway-resilience-guard
tail -f ~/.local/state/openclaw-gateway-watchdog/watchdog.log
bash ~/.local/share/openclaw-gateway-watchdog/uninstall-watchdog.sh
```

Windows：

```powershell
Get-ScheduledTask -TaskName "OpenClaw Gateway Resilience Guard"
Get-Content "$env:LOCALAPPDATA\openclaw-gateway-watchdog\watchdog.log" -Wait
powershell -ExecutionPolicy Bypass -File "$env:LOCALAPPDATA\openclaw-gateway-watchdog\uninstall-watchdog.ps1"
```

连配置和日志一起删除：

```bash
bash ~/.local/share/openclaw-gateway-watchdog/uninstall-watchdog.sh --purge
```

## 配置

普通用户一般不用改。高级配置在：

```text
~/.config/openclaw-gateway-watchdog/watchdog.env
```

常用项：

```bash
GATEWAY_SERVICE="openclaw-gateway"
GATEWAY_HEALTH_URL="http://127.0.0.1:18789/healthz"
GATEWAY_HOST="127.0.0.1"
GATEWAY_PORT="18789"
CHANNEL_URL="https://ilinkai.weixin.qq.com"
NETWORK_URLS="https://www.baidu.com https://www.qq.com https://api.weixin.qq.com"
RESTART_COMMAND="systemctl --user restart openclaw-gateway"
OPENCLAW_NATIVE_PROBES="auto"
OPENCLAW_HEALTH_TIMEOUT_MS="12000"
OPENCLAW_GATEWAY_STRICT="0"
OPENCLAW_CHANNELS_PROBE="1"
OPENCLAW_DIAG_ENABLED="1"
OPENCLAW_DIAG_INTERVAL="300"
OPENCLAW_LOG_SCAN_ENABLED="1"
OPENCLAW_LOG_LIMIT="200"
OPENCLAW_LOG_SIGNAL_LIMIT="40"
OPENCLAW_LOG_TIMEOUT_MS="15000"
OPENCLAW_LOG_WARN_PATTERNS="fetch failed|fetch timeout|LLM idle timeout|model silent|..."
OPENCLAW_DIAG_ACTION="log"
OPENCLAW_DIAG_FAILURES_BEFORE_ACTION="2"
OPENCLAW_DIAG_COMMAND=""
DASHBOARD_ENABLED="1"
DASHBOARD_HOST="127.0.0.1"
DASHBOARD_PORT="18790"
DASHBOARD_ACTIONS_ENABLED="1"
DASHBOARD_TOKEN="安装时生成"
DASHBOARD_DIR="~/.local/share/openclaw-gateway-watchdog/dashboard"
MODEL_PROBE_ENABLED="0"
MODEL_EDGE_PROBE_ENABLED="1"
MODEL_PROBE_INTERVAL="1800"
MODEL_PROBE_TIMEOUT="120"
MODEL_PROBE_FAILURES_BEFORE_ACTION="2"
MODEL_PROBE_ACTION="log"
MODEL_PROBE_COMMAND=""
MODEL_PROBE_MODEL=""
MODEL_PROBE_THINKING="off"
MODEL_PROBE_SESSION_ID="watchdog-model-probe"
MODEL_PROBE_MESSAGE="Reply with exactly OK."
BASE_INTERVAL="60"
NIGHT_INTERVAL="300"
MAX_INTERVAL="1800"
CHANNEL_FAILURES_BEFORE_RESTART="2"
SUCCESS_COUNT_TO_RESET="5"
MAX_RESTARTS_PER_HOUR="6"
```

如果你的 OpenClaw 不是 systemd 用户服务管理，可以显式指定重启命令：

```bash
RESTART_COMMAND="openclaw gateway restart"
```

Windows 的配置是 JSON：

```text
%APPDATA%\openclaw-gateway-watchdog\watchdog.json
```

如果你的 OpenClaw CLI 版本太旧，不支持 `openclaw health` 或 `openclaw status --deep`，可以把 `OpenClawNativeProbes` 设为 `false`。

### OpenClaw 诊断和日志信号

这个 watchdog 不只是 ping 一个 URL。它会每隔 `OPENCLAW_DIAG_INTERVAL` 秒采集一组 OpenClaw 运行快照：

```text
~/.local/state/openclaw-gateway-watchdog/last-openclaw-gateway-status.json
~/.local/state/openclaw-gateway-watchdog/last-openclaw-health.json
~/.local/state/openclaw-gateway-watchdog/last-openclaw-model-status.json
~/.local/state/openclaw-gateway-watchdog/last-openclaw-status-deep.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-logs.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-log-signals.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-log-signal-categories.txt
~/.local/state/openclaw-gateway-watchdog/last-openclaw-diagnostics.jsonl
```

`last-openclaw-log-signals.txt` 来自 `openclaw logs --plain` 的过滤结果，会把常见异常归类成 `provider_timeout`、`proxy_or_network`、`provider_rate_limit`、`provider_auth`、`abort_stuck`、`memory_dream_timeout`、`channel_session`、`gateway_degraded`、`config_reload`、`task_runtime` 等类别。

默认策略是 `OPENCLAW_DIAG_ACTION="log"`，因为 WARN 是证据，不一定等于“应该马上重启”。如果你显式改成 `restart` 或 `command`，也必须连续多次出现诊断异常，并且外部网络探测正常，才会执行动作。

排查时可以这样判断：

| 证据 | 更可能的问题范围 | 默认策略 |
| --- | --- | --- |
| Gateway status/health 挂了 | Gateway 进程或 RPC 链路 | 走原本的 Gateway 策略，立即重启。 |
| 通道探测失败，但外部网络正常 | 通道或 session 链路 | 退避、复查，仍失败再重启 Gateway。 |
| OpenClaw 日志显示 provider timeout，模型探针也失败 | 模型 provider/API 链路 | 先记录证据；可选自定义动作。单纯重启 Gateway 未必有用。 |
| OpenClaw 日志显示 provider timeout，但模型探针成功 | OpenClaw 运行时、任务、session 或特定请求路径 | 继续保留证据，不把锅直接甩给 provider。 |
| 日志显示 proxy/DNS/TLS 错误 | 本机代理、DNS、TLS 或运营商路由 | 记录证据，避免重启风暴，优先修代理/网络路由。 |
| 日志显示 session expired 或 monitor stopped | 通道插件/session | 确认后重启 Gateway 往往有价值。 |

### 可选模型探针

如果你想判断问题到底出在 Gateway/通道，还是模型 provider 链路，可以显式开启：

```bash
MODEL_PROBE_ENABLED="1"
```

开启后，看门狗会先读取 OpenClaw 当前模型 provider 的 `baseUrl`，做一次不带凭据、不消耗 token 的入口连通性探测。然后再执行端到端模型探针：

```bash
openclaw agent --session-id "$MODEL_PROBE_SESSION_ID" \
  --thinking "$MODEL_PROBE_THINKING" \
  --timeout "$MODEL_PROBE_TIMEOUT" \
  --json \
  --message "$MODEL_PROBE_MESSAGE"
```

如果 `MODEL_PROBE_MODEL` 为空，就使用 OpenClaw 当前配置的默认模型。结果会写入主日志，以及：

```text
~/.local/state/openclaw-gateway-watchdog/model-probe-history.jsonl
~/.local/state/openclaw-gateway-watchdog/last-openclaw-model-probe.json
~/.local/state/openclaw-gateway-watchdog/last-model-api-edge-probe.txt
```

排查凌晨模型 provider 超时，可以先用这组设置：

```bash
MODEL_PROBE_ENABLED="1"
MODEL_EDGE_PROBE_ENABLED="1"
MODEL_PROBE_INTERVAL="600"
MODEL_PROBE_TIMEOUT="120"
MODEL_PROBE_ACTION="log"
MODEL_PROBE_THINKING="off"
MODEL_PROBE_MESSAGE="Reply with exactly OK."
```

`MODEL_PROBE_ACTION` 支持：

- `log`：只记录证据，默认策略。
- `restart`：连续失败达到 `MODEL_PROBE_FAILURES_BEFORE_ACTION` 后重启 gateway，但会先确认外部网络不是全局断网。
- `command`：连续失败后执行 `MODEL_PROBE_COMMAND`。

这个功能会真实调用模型，可能消耗额度或费用。日志不会打印 API key，但会记录 provider/model 名称、耗时、退出状态和第一行错误摘要。
`MODEL_EDGE_PROBE_ENABLED` 不使用凭据，也不调用 `/chat/completions`；它只检查 provider API 入口，比如 `https://api.deepseek.com`，是否能快速完成 DNS/TLS/HTTP 连接。

## 安全边界

这个项目不修改 OpenClaw 配置、不改微信插件源码、不处理消息内容。默认情况下它只做三件事：

1. 探测本机 gateway 和外部 URL。
2. 写自己的日志和状态文件。
3. 在满足保护条件后执行配置好的 gateway 重启命令。

可选模型探针只有在你显式开启后才会发起真实模型请求。

分享日志前，请检查里面是否包含本机路径、服务名或私有通道地址。

## 开源许可

MIT-0。这个许可证符合 ClawHub skill 发布要求，也方便别人直接复用、改造和分发。

## 发布到 ClawHub

已发布包：

- ClawHub：<https://clawhub.ai/zc-kama/gateway-resilience-guard>
- Slug：`gateway-resilience-guard`

本仓库带 `SKILL.md`，也可以重新发布为 OpenClaw skill：

```bash
clawhub publish . \
  --slug gateway-resilience-guard \
  --name "OpenClaw Gateway Resilience Guard" \
  --version 1.4.4 \
  --changelog "Make dashboard controls robust with native semantic buttons/selects, fixed hover states, centered brand mark, and non-overlapping config rows"
```

发布前需要先执行 `clawhub login` 完成 CLI 登录。


path: uninstall-watchdog.ps1
content: param(
  [switch]$Purge,
  [string]$InstallDir = ""
)

$ErrorActionPreference = "Continue"
$TaskName = "OpenClaw Gateway Resilience Guard"
if (-not $InstallDir) {
  $InstallDir = Join-Path $env:LOCALAPPDATA "openclaw-gateway-watchdog"
}
$ConfigDir = Join-Path $env:APPDATA "openclaw-gateway-watchdog"
$StateDir = Join-Path $env:LOCALAPPDATA "openclaw-gateway-watchdog"

if (Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue) {
  Stop-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
  Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false
}

$DashboardPid = Join-Path $StateDir "dashboard.pid"
if (Test-Path -LiteralPath $DashboardPid) {
  $pidValue = Get-Content -LiteralPath $DashboardPid -ErrorAction SilentlyContinue | Select-Object -First 1
  if ($pidValue) {
    Stop-Process -Id ([int]$pidValue) -ErrorAction SilentlyContinue
  }
  Remove-Item -LiteralPath $DashboardPid -Force -ErrorAction SilentlyContinue
}

Remove-Item -LiteralPath $InstallDir -Recurse -Force -ErrorAction SilentlyContinue

if ($Purge) {
  Remove-Item -LiteralPath $ConfigDir -Recurse -Force -ErrorAction SilentlyContinue
  Remove-Item -LiteralPath $StateDir -Recurse -Force -ErrorAction SilentlyContinue
  Write-Host "OpenClaw Gateway Resilience Guard removed, including config and logs."
} else {
  Write-Host "OpenClaw Gateway Resilience Guard removed. Config/logs kept:"
  Write-Host "  $ConfigDir"
  Write-Host "  $StateDir"
  Write-Host "Use -Purge to remove them too."
}


path: uninstall-watchdog.sh
content: #!/usr/bin/env bash
# Uninstall OpenClaw Gateway Resilience Guard for the current user.

set -euo pipefail

SERVICE_NAME="gateway-watchdog"
PLIST_LABEL="ai.clawhub.gateway-resilience-guard"
INSTALL_DIR="${INSTALL_DIR:-${HOME}/.local/share/openclaw-gateway-watchdog}"
CONFIG_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/openclaw-gateway-watchdog"
STATE_DIR="${XDG_STATE_HOME:-${HOME}/.local/state}/openclaw-gateway-watchdog"
SERVICE_FILE="${HOME}/.config/systemd/user/${SERVICE_NAME}.service"
PLIST_FILE="${HOME}/Library/LaunchAgents/${PLIST_LABEL}.plist"
PURGE=0

if [ "${1:-}" = "--purge" ]; then
  PURGE=1
fi

have_cmd() {
  command -v "$1" >/dev/null 2>&1
}

if [ "$(uname -s 2>/dev/null || echo unknown)" = "Darwin" ] && have_cmd launchctl; then
  uid=$(id -u)
  launchctl bootout "gui/${uid}" "$PLIST_FILE" >/dev/null 2>&1 || true
fi

if have_cmd systemctl && systemctl --user status >/dev/null 2>&1; then
  systemctl --user stop "$SERVICE_NAME" 2>/dev/null || true
  systemctl --user disable "$SERVICE_NAME" 2>/dev/null || true
fi

if [ -f "${STATE_DIR}/watchdog.pid" ]; then
  pid="$(cat "${STATE_DIR}/watchdog.pid" 2>/dev/null || true)"
  if [ -n "${pid:-}" ] && kill -0 "$pid" 2>/dev/null; then
    kill "$pid" 2>/dev/null || true
  fi
  rm -f "${STATE_DIR}/watchdog.pid"
fi

if [ -f "${STATE_DIR}/dashboard.pid" ]; then
  pid="$(cat "${STATE_DIR}/dashboard.pid" 2>/dev/null || true)"
  if [ -n "${pid:-}" ] && kill -0 "$pid" 2>/dev/null; then
    kill "$pid" 2>/dev/null || true
  fi
  rm -f "${STATE_DIR}/dashboard.pid"
fi

rm -f "$SERVICE_FILE" "$PLIST_FILE"
if have_cmd systemctl && systemctl --user status >/dev/null 2>&1; then
  systemctl --user daemon-reload
fi

rm -rf "$INSTALL_DIR"

if [ "$PURGE" -eq 1 ]; then
  rm -rf "$CONFIG_DIR" "$STATE_DIR"
  echo "OpenClaw Gateway Resilience Guard removed, including config and logs."
else
  echo "OpenClaw Gateway Resilience Guard removed. Config/logs kept:"
  echo "  ${CONFIG_DIR}"
  echo "  ${STATE_DIR}"
  echo "Use --purge to remove them too."
fi


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal:  | SkillSpector: suspicious (CRITICAL)