Skill: bovmanth/model-failover-guard (version 1.0.0)

# OpenClaw Model Failover Guard Automatic model failover + failback guard for OpenClaw. ## Description When your primary model becomes unstable, this guard can switch to an available fallback model automatically, then switch back to the primary after stability is restored. ## Features - Monitor model health on an interval - If primary fails N times consecutively → failover - Fallback is selected from **all configured models** - Supports preferred fallback provider - After fallback is stable for N checks → try failback - If failback test fails → revert to fallback immediately ## Install ```bash npx skills add BovmantH/openclaw-model-failover-guard --skill model-failover-guard ``` ## Usage ```bash # Run once python3 skills/model-failover-guard/scripts/failover.py once # Run as daemon python3 skills/model-failover-guard/scripts/failover.py loop ``` ## Configuration Copy `skills/model-failover-guard/config.example.json` to `config.json` and adjust settings. | Key | Description | |-----|-------------| | `primaryModel` | Primary model to monitor | | `failThreshold` | Consecutive failures before failover | | `recoverThreshold` | Stable checks before failback | | `checkIntervalSec` | Health check interval (seconds) | ## Author - Owner: BovmantH - Version: 1.0.0 ## License MIT

path: README.md
content: # OpenClaw Model Failover Guard

[English](#english) | [中文](#中文)

<a id="english"></a>

```mermaid
flowchart LR
  A[Primary Model Healthy] -->|Check Interval| B{Primary Failures >= Threshold?}
  B -->|No| A
  B -->|Yes| C[Pick Fallback from Configured Models]
  C --> D[Switch + Restart Gateway]
  D --> E{Fallback Stable >= Recover Threshold?}
  E -->|No| D
  E -->|Yes| F[Try Failback to Primary]
  F -->|Success| A
  F -->|Fail| D
```

---

## English

Automatic model failover + failback guard for OpenClaw.

When your primary model becomes unstable, this guard can switch to an available fallback model automatically, then switch back to the primary after stability is restored.

### Overview

- Monitor model health on an interval
- If primary fails N times consecutively → failover
- Fallback is selected from **all configured models**
- Supports preferred fallback provider
- After fallback is stable for N checks → try failback
- If failback test fails → revert to fallback immediately

### Install

```bash
npx skills add https://github.com/BovmantH/openclaw-model-failover-guard.git --skill model-failover-guard
```

### Files

| Path | Purpose |
|---|---|
| `skills/model-failover-guard/SKILL.md` | Skill definition |
| `skills/model-failover-guard/config.example.json` | Config template |
| `skills/model-failover-guard/scripts/failover.py` | Runtime guard script |

### Config

Copy `config.example.json` to `config.json`.

| Key | Description |
|---|---|
| `primaryModel` | Optional. Empty = use OpenClaw current default model |
| `preferredFallbackProvider` | Optional preferred fallback provider |
| `excludedProviders` | Providers excluded from fallback candidates |
| `failThreshold` | Consecutive failures before failover |
| `recoverThreshold` | Stable checks before failback |
| `checkIntervalSec` | Health check interval (seconds) |
| `testTimeoutSec` | Single test timeout (seconds) |

### Run

```bash
python3 skills/model-failover-guard/scripts/failover.py once
python3 skills/model-failover-guard/scripts/failover.py loop
```

### State & Logs

- State: `~/.openclaw/failover-state.json`
- Log: `~/.openclaw/failover.log`

### Run as Systemd Service (Linux)

#### Install service

```bash
mkdir -p ~/.config/systemd/user
cp skills/model-failover-guard/skills/model-failover-guard/openclaw-model-failover.service ~/.config/systemd/user/
systemctl --user daemon-reload
```

#### Enable & Start

```bash
systemctl --user enable --now openclaw-model-failover
```

#### Logs

```bash
journalctl --user -u openclaw-model-failover -f
```

### FAQ

**Q: Failback to primary is not happening. What should I do?**  
A: Check `~/.openclaw/failover.log`. You can also run `python3 skills/model-failover-guard/scripts/failover.py once` manually.

**Q: Log file is too large.**  
A: Rotate/clean periodically. Example: `echo "" > ~/.openclaw/failover.log`

**Q: How to stop the guard completely?**  
A: `systemctl --user stop openclaw-model-failover` (if using systemd), or kill the process.

**Q: Can I run multiple instances?**  
A: Not recommended. Run only one instance per machine.

---

<a id="中文"></a>

## 中文

```mermaid
flowchart LR
  A[主模型健康] -->|按间隔检查| B{连续失败达到阈值?}
  B -->|否| A
  B -->|是| C[从已配置模型中选择兜底]
  C --> D[切换并重启 Gateway]
  D --> E{兜底稳定达到切回阈值?}
  E -->|否| D
  E -->|是| F[尝试切回主模型]
  F -->|成功| A
  F -->|失败| D
```

这是一个 OpenClaw 模型自动故障切换 + 自动切回守护技能。

当主模型不稳定时，守护进程会自动切换到可用的兜底模型，并在稳定后自动尝试切回主模型。

### 概览

- 按固定间隔检测模型健康
- 主模型连续失败 N 次后触发故障切换
- 兜底模型从**全部已配置模型**中选择
- 支持设置优先 fallback provider
- 兜底稳定 N 次后尝试切回主模型
- 切回失败会立即回退到兜底，防止抖动

### 安装

```bash
npx skills add https://github.com/BovmantH/openclaw-model-failover-guard.git --skill model-failover-guard
```

### 文件

| 路径 | 用途 |
|---|---|
| `skills/model-failover-guard/SKILL.md` | 技能定义 |
| `skills/model-failover-guard/config.example.json` | 配置模板 |
| `skills/model-failover-guard/scripts/failover.py` | 运行脚本 |

### 配置

复制 `config.example.json` 为 `config.json`。

| 键 | 说明 |
|---|---|
| `primaryModel` | 可选；空则使用 OpenClaw 当前默认主模型 |
| `preferredFallbackProvider` | 可选的优先兜底 provider |
| `excludedProviders` | 不参与兜底的 provider 列表 |
| `failThreshold` | 触发故障切换的连续失败阈值 |
| `recoverThreshold` | 触发切回主模型的稳定检查阈值 |
| `checkIntervalSec` | 健康检查间隔（秒） |
| `testTimeoutSec` | 单次测试超时（秒） |

### 运行

```bash
python3 skills/model-failover-guard/scripts/failover.py once
python3 skills/model-failover-guard/scripts/failover.py loop
```

### 状态与日志

- 状态：`~/.openclaw/failover-state.json`
- 日志：`~/.openclaw/failover.log`

### 作为 Systemd 服务运行（Linux）

#### 安装服务

```bash
mkdir -p ~/.config/systemd/user
cp skills/model-failover-guard/skills/model-failover-guard/openclaw-model-failover.service ~/.config/systemd/user/
systemctl --user daemon-reload
```

#### 启用并启动

```bash
systemctl --user enable --now openclaw-model-failover
```

#### 查看日志

```bash
journalctl --user -u openclaw-model-failover -f
```

### 常见问题

**问：切不回主模型怎么办？**  
答：检查 `~/.openclaw/failover.log`，也可以手动运行 `python3 skills/model-failover-guard/scripts/failover.py once`。

**问：日志文件太大怎么办？**  
答：定期清理或用 logrotate。例如：`echo "" > ~/.openclaw/failover.log`

**问：如何完全停止守护？**  
答：如果用了 systemd，执行 `systemctl --user stop openclaw-model-failover`，或直接 kill 进程。

**问：可以同时跑多个实例吗？**  
答：不建议，同一台机器只跑一个实例。

---

## License

MIT License · Copyright © 2026 BovmantH

See [LICENSE](./LICENSE) for full text.


path: skills/model-failover-guard/config.example.json
content: {
  "configPath": "~/.openclaw/openclaw.json",
  "logPath": "~/.openclaw/failover.log",
  "statePath": "~/.openclaw/failover-state.json",
  "primaryModel": "",
  "preferredFallbackProvider": "",
  "excludedProviders": [],
  "failThreshold": 3,
  "recoverThreshold": 3,
  "checkIntervalSec": 300,
  "testTimeoutSec": 45
}


path: skills/model-failover-guard/scripts/failover.py
content: #!/usr/bin/env python3
import json, os, subprocess, sys, time
from pathlib import Path

HOME = Path.home()
DEFAULT_OPENCLAW_CFG = HOME / '.openclaw' / 'openclaw.json'
DEFAULT_STATE = HOME / '.openclaw' / 'failover-state.json'
DEFAULT_LOG = HOME / '.openclaw' / 'failover.log'
LOCAL_CFG = Path(__file__).resolve().parent / 'config.json'


def expand(p: str) -> Path:
    return Path(os.path.expanduser(p)).resolve()


def load_runtime_config():
    cfg = {
        'configPath': str(DEFAULT_OPENCLAW_CFG),
        'statePath': str(DEFAULT_STATE),
        'logPath': str(DEFAULT_LOG),
        'primaryModel': '',
        'preferredFallbackProvider': '',
        'excludedProviders': [],
        'failThreshold': 3,
        'recoverThreshold': 3,
        'checkIntervalSec': 300,
        'testTimeoutSec': 45,
    }
    if LOCAL_CFG.exists():
        try:
            file_cfg = json.loads(LOCAL_CFG.read_text(encoding='utf-8'))
            cfg.update(file_cfg)
        except Exception:
            pass

    cfg['CONFIG_PATH'] = expand(cfg['configPath'])
    cfg['STATE_PATH'] = expand(cfg['statePath'])
    cfg['LOG_PATH'] = expand(cfg['logPath'])
    cfg['FAIL_THRESHOLD'] = int(cfg['failThreshold'])
    cfg['RECOVER_THRESHOLD'] = int(cfg['recoverThreshold'])
    cfg['CHECK_INTERVAL_SEC'] = int(cfg['checkIntervalSec'])
    cfg['TEST_TIMEOUT_SEC'] = int(cfg['testTimeoutSec'])
    cfg['EXCLUDED_PROVIDERS'] = set(cfg.get('excludedProviders') or [])
    return cfg


R = load_runtime_config()


def log(msg: str):
    ts = time.strftime('%Y-%m-%d %H:%M:%S')
    line = f'[{ts}] {msg}'
    print(line)
    R['LOG_PATH'].parent.mkdir(parents=True, exist_ok=True)
    with R['LOG_PATH'].open('a', encoding='utf-8') as f:
        f.write(line + '\n')


def load_state():
    if R['STATE_PATH'].exists():
        try:
            return json.loads(R['STATE_PATH'].read_text(encoding='utf-8'))
        except Exception:
            pass
    return {
        'consecutive_failures': 0,
        'consecutive_fallback_health': 0,
        'last_switch': None,
        'current_fallback': None,
    }


def save_state(state):
    R['STATE_PATH'].parent.mkdir(parents=True, exist_ok=True)
    R['STATE_PATH'].write_text(json.dumps(state, ensure_ascii=False, indent=2), encoding='utf-8')


def load_openclaw_config():
    return json.loads(R['CONFIG_PATH'].read_text(encoding='utf-8'))


def save_openclaw_config(cfg):
    R['CONFIG_PATH'].write_text(json.dumps(cfg, ensure_ascii=False, indent=2), encoding='utf-8')


def get_primary_model(cfg):
    return cfg.get('agents', {}).get('defaults', {}).get('model', {}).get('primary')


def set_primary_model(cfg, model_id: str):
    cfg.setdefault('agents', {}).setdefault('defaults', {}).setdefault('model', {})['primary'] = model_id


def list_configured_models(cfg):
    m = cfg.get('agents', {}).get('defaults', {}).get('models', {})
    models = [k for k in m.keys() if isinstance(k, str) and '/' in k]
    if models:
        return sorted(set(models))

    out = []
    providers = cfg.get('models', {}).get('providers', {})
    for provider, pdata in providers.items():
        for mm in pdata.get('models', []) or []:
            mid = mm.get('id')
            if mid:
                out.append(f'{provider}/{mid}')
    return sorted(set(out))


def get_target_primary(cfg):
    configured_primary = get_primary_model(cfg)
    return R['primaryModel'] or configured_primary


def rank_candidates(models):
    preferred = R.get('preferredFallbackProvider') or ''

    def rank(model_id: str):
        provider = model_id.split('/', 1)[0]
        pref_rank = 0 if (preferred and provider == preferred) else 1
        return (pref_rank, model_id)

    return sorted(models, key=rank)


def test_current_default_model():
    cmd = [
        'openclaw', 'agent', '--agent', 'main',
        '--message', '只回复OK', '--json', '--timeout', str(R['TEST_TIMEOUT_SEC'])
    ]
    try:
        p = subprocess.run(cmd, capture_output=True, text=True, timeout=R['TEST_TIMEOUT_SEC'] + 20)
        out = (p.stdout or '') + '\n' + (p.stderr or '')
        ok = (p.returncode == 0) and ('OK' in out)
        if ok:
            return True, 'model test ok'
        return False, out[-500:]
    except subprocess.TimeoutExpired:
        return False, 'test timeout'
    except Exception as e:
        return False, f'exception: {e}'


def apply_primary(cfg, model_id: str):
    set_primary_model(cfg, model_id)
    save_openclaw_config(cfg)
    subprocess.run(['openclaw', 'gateway', 'restart'], capture_output=True, text=True)


def try_switch_and_test(cfg, candidate_model):
    apply_primary(cfg, candidate_model)
    return test_current_default_model()


def pick_working_fallback(cfg, target_primary):
    all_models = list_configured_models(cfg)
    candidates = []
    for m in all_models:
        if m == target_primary:
            continue
        provider = m.split('/', 1)[0]
        if provider in R['EXCLUDED_PROVIDERS']:
            continue
        candidates.append(m)

    candidates = rank_candidates(candidates)

    if not candidates:
        return None, 'no fallback candidates'

    for c in candidates:
        ok, detail = try_switch_and_test(cfg, c)
        log(f'fallback switch-test: {c} -> {"OK" if ok else "FAIL"}')
        if ok:
            return c, 'ok'
    return None, 'all candidates failed'


def run_once():
    state = load_state()
    cfg = load_openclaw_config()
    current = get_primary_model(cfg)
    target_primary = get_target_primary(cfg)

    if not target_primary:
        log('ABORT: cannot determine primary model (set primaryModel in config.json)')
        return 2

    if current == target_primary:
        ok, detail = test_current_default_model()
        if ok:
            if state.get('consecutive_failures', 0) != 0:
                log('primary recovered; reset failure counter')
            state['consecutive_failures'] = 0
            state['consecutive_fallback_health'] = 0
            state['current_fallback'] = None
            save_state(state)
            return 0

        state['consecutive_failures'] = int(state.get('consecutive_failures', 0)) + 1
        save_state(state)
        log(f'primary failed ({state["consecutive_failures"]}/{R["FAIL_THRESHOLD"]}): {detail}')

        if state['consecutive_failures'] >= R['FAIL_THRESHOLD']:
            next_model, reason = pick_working_fallback(cfg, target_primary)
            if not next_model:
                log(f'FAILOVER ABORT: {reason}')
                apply_primary(cfg, target_primary)
                return 2
            state['last_switch'] = {
                'from': target_primary,
                'to': next_model,
                'at': time.strftime('%Y-%m-%d %H:%M:%S')
            }
            state['current_fallback'] = next_model
            state['consecutive_failures'] = 0
            state['consecutive_fallback_health'] = 0
            save_state(state)
            log(f'FAILOVER DONE: {target_primary} -> {next_model}')
        return 0

    # running on fallback
    ok, detail = test_current_default_model()
    if ok:
        state['consecutive_fallback_health'] = int(state.get('consecutive_fallback_health', 0)) + 1
        log(f'fallback healthy ({state["consecutive_fallback_health"]}/{R["RECOVER_THRESHOLD"]}) on {current}')
        if state['consecutive_fallback_health'] >= R['RECOVER_THRESHOLD']:
            apply_primary(cfg, target_primary)
            ok2, detail2 = test_current_default_model()
            if ok2:
                state['last_switch'] = {
                    'from': current,
                    'to': target_primary,
                    'at': time.strftime('%Y-%m-%d %H:%M:%S')
                }
                state['current_fallback'] = None
                state['consecutive_fallback_health'] = 0
                state['consecutive_failures'] = 0
                save_state(state)
                log(f'FAILBACK DONE: {current} -> {target_primary}')
                return 0
            apply_primary(cfg, current)
            state['consecutive_fallback_health'] = 0
            save_state(state)
            log(f'FAILBACK ABORT: primary still unstable, reverted to {current}. detail={detail2}')
            return 0
    else:
        if state.get('consecutive_fallback_health', 0) != 0:
            log(f'fallback unhealthy; reset fallback health counter: {detail}')
        state['consecutive_fallback_health'] = 0

    state['consecutive_failures'] = 0
    save_state(state)
    return 0


def run_loop():
    log(
        f'failover loop started, interval={R["CHECK_INTERVAL_SEC"]}s, '
        f'fail_threshold={R["FAIL_THRESHOLD"]}, recover_threshold={R["RECOVER_THRESHOLD"]}'
    )
    while True:
        try:
            run_once()
        except Exception as e:
            log(f'run_once error: {e}')
        time.sleep(R['CHECK_INTERVAL_SEC'])


if __name__ == '__main__':
    mode = sys.argv[1] if len(sys.argv) > 1 else 'once'
    if mode == 'loop':
        run_loop()
    else:
        sys.exit(run_once())


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)