Skill: lerwee/alert-to-fault-handling (version 1.0.0)

--- name: alert-to-fault-handling description: 告警自动处理工作流 - 监听告警上下文，匹配处理脚本，提示用户执行故障处理操作 --- # 告警自动处理工作流 ## Overview 当检测到告警上下文时，根据群组和告警类型自动匹配对应的故障处理脚本，提示用户一键执行。 ## 工作流程 ``` 告警上下文检测 → 群组识别 → 关键词匹配 → 脚本推荐 → 用户确认 → 执行脚本 → 反馈结果 → (可选)自动关闭告警 ``` ## 触发条件 当以下条件同时满足时触发： 1. **对话上下文存在告警信息**（包含 eventid、IP、告警名称中的至少一个） 2. **当前飞书群组**与告警分类匹配 3. **告警内容**匹配脚本关键词 4. **存在对应的预置脚本** ## 配置文件 ### 脚本映射配置 (.scripts_map.json) ```json { "nginx": { "name": "nginx服务重启", "script_id": 187, "keywords": ["nginx", "Nginx", "NGINX", "web", "80端口", "http服务"], "classifications": [102], "chat_groups": [""], "description": "重启Nginx服务，适用于服务停止、无响应等场景" }, "disk": { "name": "主机磁盘空间清理", "script_id": 197, "keywords": ["磁盘", "disk", "空间", "storage", "/var", "/tmp", "使用率", "满"], "classifications": [101], "chat_groups": [""], "description": "清理日志文件和临时文件，释放磁盘空间" } } ``` ### 执行日志配置 (.execution_log.json) ```json { "executions": [ { "timestamp": "2026-03-10T09:50:00Z", "eventid": "32415666", "ip": "192.168.3.137", "script_id": 187, "script_name": "nginx服务重启", "status": "success", "execution_id": 970, "user": "" } ] } ``` ## 群组与分类映射 | 飞书群 ID | 群组名称 | 监控分类 | 默认脚本 | |-----------|----------|----------|----------| | | 操作系统告警群 | 101 | 主机磁盘空间清理 (197) | | | 中间件告警群 | 102 | nginx服务重启 (187) | | | 网络设备告警群 | 103 | (待配置) | ## 用户交互 ### 场景1: 自动推荐 ``` 🤖 检测到 Nginx 服务停止告警 📊 告警对象: 3.137-Nginx-1.14.2 (192.168.3.137) 🔑 告警ID: 32415666 💡 推荐操作: nginx服务重启 (脚本ID: 187) 📝 说明: 重启Nginx服务，适用于服务停止、无响应等场景 👉 回复「执行」或「确认」自动运行脚本 ``` ### 场景2: 用户指定脚本 ``` 用户: 执行脚本 197 🔧 正在为主机 192.168.3.137 执行脚本... 📋 脚本: 主机磁盘空间清理 (ID: 197) ⏳ 执行中... ``` ### 场景3: 确认后执行 ``` 用户: 确认 🔧 提交执行任务... ✅ 任务已提交 (Execution ID: 970) ⏳ 等待执行结果... ``` ## IP 获取规则 优先级从高到低： 1. **告警消息中的 IP 字段**（最优先） 2. **通过 objectid 查询主机详情获取 IP** ```bash ./scripts/lerwee-api.sh monitor host-view '{"hostid": 11131}' ``` 3. **用户手动指定** ## 脚本执行 使用 fault-handling skill 执行脚本： ```bash python3 /home/node/.openclaw/workspace/skills/fault-handling/run_script.py \ --hosts '192.168.3.137' \ --script-id 187 ``` ## 执行结果反馈 ``` 🔧 故障处理执行报告 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 任务名称：nginx服务重启 任务ID：970 执行状态：成功 执行耗时：15秒 📋 步骤执行详情 步骤 主机IP 主机名 状态 输出 nginx服务重启 192.168.3.137 vm-3-137 ✅ nginx: [ OK ] 📌 执行结论 ● ✅ 1台主机执行成功 ● ❌ 0台主机执行失败 💾 已记录到执行日志 ``` ## 可选: 自动关闭告警 脚本执行成功后，可选自动关闭对应告警： ```bash ./scripts/lerwee-api.sh alert problem-ack '{ "eventid": "32415666", "action": 1, "message": "脚本执行成功，自动关闭告警" }' ``` ## Hard Rules - 执行脚本前必须获取用户明确确认（回复「执行」「确认」「yes」） - 不能在没有主机 IP 的情况下猜测或使用占位 IP - 执行失败时必须明确说明失败原因 - 所有执行必须记录到日志文件 - 自动关闭告警前必须确认脚本执行成功 ## Files - 配置: `.scripts_map.json` - 脚本映射配置 - 日志: `.execution_log.json` - 执行历史记录 - 主逻辑: 由 Agent 动态处理，无需独立脚本 ## 扩展新脚本类型 在 `.scripts_map.json` 中添加新条目： ```json { "mysql": { "name": "MySQL服务重启", "script_id": 198, "keywords": ["mysql", "MySQL", "数据库"], "classifications": [105], "chat_groups": ["oc_xxx"], "description": "重启MySQL服务" } } ``` ## 安全机制 1. **确认机制**: 默认脚本需用户确认，自定义脚本ID需二次确认 2. **白名单**: 只执行预置脚本或用户明确指定的脚本ID 3. **日志审计**: 所有操作记录到日志文件 4. **回滚支持**: 记录执行ID，支持查询历史结果

path: alert_workflow.py
content: #!/usr/bin/env python3
"""
告警自动处理工作流主入口
协调告警检测、脚本匹配、用户确认、脚本执行、日志记录
"""

import json
import sys
import subprocess
from pathlib import Path

# 脚本目录
SCRIPT_DIR = Path(__file__).parent
FAULT_HANDLING_DIR = SCRIPT_DIR.parent / "fault-handling"
LERCLEE_API_DIR = SCRIPT_DIR.parent / "lerwee-api" / "scripts"


def extract_ip_from_alert(alert_info):
    """
    从告警信息中提取IP

    优先级:
    1. 告警信息中的 ip 字段
    2. 通过 objectid 查询主机详情获取 IP
    """
    # 1. 直接获取
    if 'ip' in alert_info and alert_info['ip']:
        return alert_info['ip']

    # 2. 通过 objectid 查询
    if 'objectid' in alert_info and alert_info['objectid']:
        try:
            result = subprocess.run(
                ['bash', LERCLEE_API_DIR / 'lerwee-api.sh', 'monitor', 'host-view',
                 json.dumps({'hostid': alert_info['objectid']})],
                capture_output=True, text=True, check=True
            )
            data = json.loads(result.stdout)
            if data.get('code') == 0 and 'ip' in data.get('data', {}):
                return data['data']['ip']
        except Exception as e:
            print(f"❌ 查询主机IP失败: {e}", file=sys.stderr)

    return None


def match_and_recommend(alert_info, chat_id=None):
    """
    匹配脚本并生成推荐消息

    Returns:
        (matched, message)
        - matched: 是否匹配到脚本
        - message: 推荐消息文本
    """
    ip = extract_ip_from_alert(alert_info)
    if not ip:
        return False, None

    # 构建告警描述
    description = alert_info.get('description', '')

    # 获取分类
    classification = alert_info.get('classification')

    # 调用匹配脚本
    try:
        result = subprocess.run(
            ['python3', SCRIPT_DIR / 'match_script.py', 'match',
             alert_info.get('eventid', ''),
             ip,
             description,
             str(classification) if classification else '',
             chat_id or ''],
            capture_output=True, text=True
        )

        if result.returncode == 0:
            # 匹配成功
            lines = result.stdout.strip().split('\n')
            script_name = None
            script_id = None
            script_desc = None

            for line in lines:
                if line.startswith('   名称:'):
                    script_name = line.split(':', 1)[1].strip()
                elif line.startswith('   脚本ID:'):
                    script_id = line.split(':', 1)[1].strip()
                elif line.startswith('   说明:'):
                    script_desc = line.split(':', 1)[1].strip()

            message = f"""🤖 检测到告警可自动处理
📊 告警对象: {alert_info.get('hostname', 'Unknown')} ({ip})
🔑 告警ID: {alert_info.get('eventid', 'Unknown')}

💡 推荐操作: {script_name} (脚本ID: {script_id})
📝 说明: {script_desc}

👉 回复「执行」或「确认」自动运行脚本"""

            return True, message
        else:
            return False, None

    except Exception as e:
        print(f"❌ 脚本匹配失败: {e}", file=sys.stderr)
        return False, None


def execute_script(ip, script_id, eventid=None, user=None):
    """
    执行故障处理脚本

    Returns:
        执行结果字典
    """
    try:
        # 构建 fault-handling 命令
        cmd = [
            'python3',
            FAULT_HANDLING_DIR / 'run_script.py',
            '--hosts', ip,
            '--script-id', str(script_id)
        ]

        # 执行脚本
        result = subprocess.run(
            cmd,
            capture_output=True,
            text=True,
            cwd=FAULT_HANDLING_DIR
        )

        # 解析输出
        output = result.stdout

        # 记录日志
        log_record = {
            'ip': ip,
            'script_id': script_id,
            'script_name': f'Script-{script_id}',
            'status': 'success' if result.returncode == 0 else 'failed',
            'eventid': eventid,
            'user': user,
            'output': output[:1000] if output else None  # 截取前1000字符
        }

        # 尝试解析 execution_id
        if output:
            import re
            match = re.search(r'任务ID[：:]\s*(\d+)', output)
            if match:
                log_record['execution_id'] = int(match.group(1))

        # 写入日志
        subprocess.run(
            ['python3', SCRIPT_DIR / 'log_execution.py', 'add',
             ip, str(script_id), log_record['script_name'], log_record['status'],
             str(log_record.get('execution_id', '')),
             eventid or '',
             user or ''],
            capture_output=True
        )

        return {
            'success': result.returncode == 0,
            'output': output,
            'log': log_record
        }

    except Exception as e:
        return {
            'success': False,
            'output': str(e),
            'log': None
        }


def close_alert(eventid, message="脚本执行成功，自动关闭告警"):
    """关闭告警"""
    try:
        result = subprocess.run(
            ['bash', LERCLEE_API_DIR / 'lerwee-api.sh', 'alert', 'problem-ack',
             json.dumps({
                 'eventid': eventid,
                 'action': 1,
                 'message': message
             })],
            capture_output=True,
            text=True
        )

        data = json.loads(result.stdout)
        return data.get('code') == 0

    except Exception as e:
        print(f"❌ 关闭告警失败: {e}", file=sys.stderr)
        return False


def main():
    """命令行入口"""
    if len(sys.argv) < 2:
        print("Usage: alert_workflow.py <command> [args]")
        print("Commands:")
        print("  check <eventid> <ip> <description> [classification] [chat_id]  # 检查并推荐脚本")
        print("  execute <ip> <script_id> [eventid] [user]                      # 执行脚本")
        print("  close <eventid> [message]                                       # 关闭告警")
        sys.exit(1)

    command = sys.argv[1]

    if command == 'check':
        if len(sys.argv) < 5:
            print("❌ 参数不足", file=sys.stderr)
            sys.exit(1)

        alert_info = {
            'eventid': sys.argv[2],
            'ip': sys.argv[3],
            'description': sys.argv[4],
            'classification': int(sys.argv[5]) if len(sys.argv) > 5 else None,
            'objectid': None
        }
        chat_id = sys.argv[6] if len(sys.argv) > 6 else None

        matched, message = match_and_recommend(alert_info, chat_id)

        if matched:
            print(message)
            sys.exit(0)
        else:
            print("ℹ️ 未找到匹配的处理脚本")
            sys.exit(1)

    elif command == 'execute':
        if len(sys.argv) < 4:
            print("❌ 参数不足", file=sys.stderr)
            sys.exit(1)

        ip = sys.argv[2]
        script_id = int(sys.argv[3])
        eventid = sys.argv[4] if len(sys.argv) > 4 else None
        user = sys.argv[5] if len(sys.argv) > 5 else None

        print(f"🔧 正在为主机 {ip} 执行脚本...")
        result = execute_script(ip, script_id, eventid, user)

        if result['success']:
            print(result['output'])
            print("\n💾 已记录到执行日志")

            # 如果指定了 eventid，询问是否关闭告警
            if eventid:
                print(f"\n💡 提示: 告警 {eventid} 仍活跃，可手动关闭")
        else:
            print(f"❌ 执行失败: {result['output']}", file=sys.stderr)
            sys.exit(1)

    elif command == 'close':
        if len(sys.argv) < 3:
            print("❌ 参数不足", file=sys.stderr)
            sys.exit(1)

        eventid = sys.argv[2]
        message = sys.argv[3] if len(sys.argv) > 3 else "脚本执行成功，自动关闭告警"

        if close_alert(eventid, message):
            print(f"✅ 告警 {eventid} 已关闭")
        else:
            print(f"❌ 关闭告警失败", file=sys.stderr)
            sys.exit(1)

    else:
        print(f"❌ 未知命令: {command}", file=sys.stderr)
        sys.exit(1)


if __name__ == '__main__':
    main()


path: .execution_log.json
content: {
  "executions": [
    {
      "timestamp": "2026-03-10T10:51:52Z",
      "eventid": "32420610",
      "ip": "192.168.3.137",
      "script_id": 187,
      "script_name": "nginx服务重启",
      "status": "success",
      "execution_id": 15440,
      "user": "ou_db578a1d726251fd78f7d4eabdc98f54",
      "consuming": 8,
      "alert_closed": true
    }
  ],
  "last_updated": "2026-03-10T10:51:52Z"
}


path: match_script.py
content: #!/usr/bin/env python3
"""
告警与脚本匹配工具
根据告警信息、群组上下文匹配合适的故障处理脚本
"""

import json
import sys
from pathlib import Path

# 配置文件路径
SCRIPTS_MAP_FILE = Path(__file__).parent / ".scripts_map.json"


def load_scripts_map():
    """加载脚本映射配置"""
    try:
        with open(SCRIPTS_MAP_FILE, 'r', encoding='utf-8') as f:
            return json.load(f)
    except FileNotFoundError:
        print(f"❌ 配置文件不存在: {SCRIPTS_MAP_FILE}", file=sys.stderr)
        return {}
    except json.JSONDecodeError as e:
        print(f"❌ 配置文件格式错误: {e}", file=sys.stderr)
        return {}


def match_script(alert_info, chat_id=None):
    """
    根据告警信息匹配脚本

    Args:
        alert_info: 告警信息字典，包含:
            - eventid: 告警ID
            - ip: 主机IP
            - description: 告警描述
            - classification: 监控分类ID
        chat_id: 当前飞书群组ID

    Returns:
        匹配到的脚本配置，如果没有匹配返回 None
    """
    scripts_map = load_scripts_map()

    # 获取告警描述和分类
    description = alert_info.get('description', '')
    classification = alert_info.get('classification')
    ip = alert_info.get('ip', '')
    eventid = alert_info.get('eventid', '')

    # 遍历所有脚本配置
    for script_key, script_config in scripts_map.items():
        # 检查群组是否匹配
        if chat_id and chat_id not in script_config.get('chat_groups', []):
            continue

        # 检查分类是否匹配
        if classification and classification not in script_config.get('classifications', []):
            continue

        # 检查关键词是否匹配
        keywords = script_config.get('keywords', [])
        keyword_matched = any(kw.lower() in description.lower() for kw in keywords)

        if keyword_matched:
            return {
                'key': script_key,
                'name': script_config['name'],
                'script_id': script_config['script_id'],
                'description': script_config['description'],
                'matched_keyword': next((kw for kw in keywords if kw.lower() in description.lower()), None)
            }

    return None


def list_all_scripts():
    """列出所有可用脚本"""
    scripts_map = load_scripts_map()
    result = []
    for key, config in scripts_map.items():
        result.append({
            'key': key,
            'name': config['name'],
            'script_id': config['script_id'],
            'keywords': config['keywords'],
            'classifications': config['classifications'],
            'description': config['description']
        })
    return result


def main():
    """命令行入口"""
    if len(sys.argv) < 2:
        print("用法: match_script.py <command> [args]")
        print("命令:")
        print("  match <eventid> <ip> <description> [classification] [chat_id]")
        print("  list")
        sys.exit(1)

    command = sys.argv[1]

    if command == 'match':
        if len(sys.argv) < 5:
            print("❌ 参数不足", file=sys.stderr)
            print("用法: match_script.py match <eventid> <ip> <description> [classification] [chat_id]")
            sys.exit(1)

        eventid = sys.argv[2]
        ip = sys.argv[3]
        description = sys.argv[4]
        classification = int(sys.argv[5]) if len(sys.argv) > 5 else None
        chat_id = sys.argv[6] if len(sys.argv) > 6 else None

        alert_info = {
            'eventid': eventid,
            'ip': ip,
            'description': description,
            'classification': classification
        }

        matched = match_script(alert_info, chat_id)

        if matched:
            print(f"✅ 匹配到脚本:")
            print(f"   名称: {matched['name']}")
            print(f"   脚本ID: {matched['script_id']}")
            print(f"   说明: {matched['description']}")
            print(f"   匹配关键词: {matched['matched_keyword']}")
            sys.exit(0)
        else:
            print("❌ 未找到匹配的脚本")
            sys.exit(1)

    elif command == 'list':
        scripts = list_all_scripts()
        print(f"📋 可用脚本列表 ({len(scripts)} 个):")
        for script in scripts:
            print(f"\n🔹 {script['name']} (ID: {script['script_id']})")
            print(f"   关键词: {', '.join(script['keywords'])}")
            print(f"   分类: {script['classifications']}")
            print(f"   说明: {script['description']}")
        sys.exit(0)

    else:
        print(f"❌ 未知命令: {command}", file=sys.stderr)
        sys.exit(1)


if __name__ == '__main__':
    main()


path: log_execution.py
content: #!/usr/bin/env python3
"""
执行日志记录工具
记录脚本执行历史，支持审计和复盘
"""

import json
import sys
from datetime import datetime
from pathlib import Path

# 日志文件路径
LOG_FILE = Path(__file__).parent / ".execution_log.json"


def load_log():
    """加载执行日志"""
    try:
        with open(LOG_FILE, 'r', encoding='utf-8') as f:
            data = json.load(f)
            return data.get('executions', [])
    except FileNotFoundError:
        return []
    except json.JSONDecodeError:
        return []


def save_log(executions):
    """保存执行日志"""
    with open(LOG_FILE, 'w', encoding='utf-8') as f:
        json.dump({
            'executions': executions,
            'last_updated': datetime.utcnow().isoformat() + 'Z'
        }, f, ensure_ascii=False, indent=2)


def add_execution(record):
    """
    添加执行记录

    Args:
        record: 执行记录字典，包含:
            - eventid: 告警ID (可选)
            - ip: 主机IP
            - script_id: 脚本ID
            - script_name: 脚本名称
            - status: 执行状态 (success/failed/timeout)
            - execution_id: 任务ID (可选)
            - user: 用户ID (可选)
            - output: 执行输出 (可选)
            - error: 错误信息 (可选)
    """
    executions = load_log()

    # 添加时间戳
    record['timestamp'] = datetime.utcnow().isoformat() + 'Z'

    # 添加到日志
    executions.append(record)

    # 保存
    save_log(executions)

    return record


def get_recent_executions(limit=10):
    """获取最近N条执行记录"""
    executions = load_log()
    # 按时间倒序
    executions.sort(key=lambda x: x.get('timestamp', ''), reverse=True)
    return executions[:limit]


def get_executions_by_ip(ip, limit=10):
    """获取指定IP的执行记录"""
    executions = load_log()
    filtered = [e for e in executions if e.get('ip') == ip]
    filtered.sort(key=lambda x: x.get('timestamp', ''), reverse=True)
    return filtered[:limit]


def get_executions_by_eventid(eventid):
    """获取指定告警ID的执行记录"""
    executions = load_log()
    return [e for e in executions if e.get('eventid') == eventid]


def main():
    """命令行入口"""
    if len(sys.argv) < 2:
        print("用法: log_execution.py <command> [args]")
        print("命令:")
        print("  add <ip> <script_id> <script_name> <status> [execution_id] [eventid] [user]")
        print("  list [limit]")
        print("  by-ip <ip> [limit]")
        print("  by-event <eventid>")
        sys.exit(1)

    command = sys.argv[1]

    if command == 'add':
        if len(sys.argv) < 5:
            print("❌ 参数不足", file=sys.stderr)
            print("用法: log_execution.py add <ip> <script_id> <script_name> <status> [execution_id] [eventid] [user]")
            sys.exit(1)

        record = {
            'ip': sys.argv[2],
            'script_id': int(sys.argv[3]),
            'script_name': sys.argv[4],
            'status': sys.argv[5]
        }

        if len(sys.argv) > 6:
            record['execution_id'] = int(sys.argv[6])
        if len(sys.argv) > 7:
            record['eventid'] = sys.argv[7]
        if len(sys.argv) > 8:
            record['user'] = sys.argv[8]

        result = add_execution(record)
        print(f"✅ 已记录执行: {record['script_name']} @ {record['ip']} ({record['status']})")
        print(f"   时间: {result['timestamp']}")

    elif command == 'list':
        limit = int(sys.argv[2]) if len(sys.argv) > 2 else 10
        executions = get_recent_executions(limit)

        print(f"📋 最近 {len(executions)} 条执行记录:")
        for exe in executions:
            status_emoji = "✅" if exe.get('status') == 'success' else "❌"
            print(f"\n{status_emoji} {exe.get('timestamp', 'Unknown')}")
            print(f"   脚本: {exe.get('script_name', 'Unknown')} (ID: {exe.get('script_id', 'N/A')})")
            print(f"   主机: {exe.get('ip', 'Unknown')}")
            if exe.get('eventid'):
                print(f"   告警ID: {exe['eventid']}")
            if exe.get('execution_id'):
                print(f"   任务ID: {exe['execution_id']}")
            if exe.get('user'):
                print(f"   操作用户: {exe['user']}")

    elif command == 'by-ip':
        if len(sys.argv) < 3:
            print("❌ 参数不足", file=sys.stderr)
            print("用法: log_execution.py by-ip <ip> [limit]")
            sys.exit(1)

        ip = sys.argv[2]
        limit = int(sys.argv[3]) if len(sys.argv) > 3 else 10
        executions = get_executions_by_ip(ip, limit)

        print(f"📋 主机 {ip} 的执行记录 ({len(executions)} 条):")
        for exe in executions:
            status_emoji = "✅" if exe.get('status') == 'success' else "❌"
            print(f"\n{status_emoji} {exe.get('timestamp', 'Unknown')}")
            print(f"   脚本: {exe.get('script_name', 'Unknown')} (ID: {exe.get('script_id', 'N/A')})")

    elif command == 'by-event':
        if len(sys.argv) < 3:
            print("❌ 参数不足", file=sys.stderr)
            print("用法: log_execution.py by-event <eventid>")
            sys.exit(1)

        eventid = sys.argv[2]
        executions = get_executions_by_eventid(eventid)

        print(f"📋 告警 {eventid} 的执行记录 ({len(executions)} 条):")
        for exe in executions:
            status_emoji = "✅" if exe.get('status') == 'success' else "❌"
            print(f"\n{status_emoji} {exe.get('timestamp', 'Unknown')}")
            print(f"   脚本: {exe.get('script_name', 'Unknown')}")
            print(f"   主机: {exe.get('ip', 'Unknown')}")

    else:
        print(f"❌ 未知命令: {command}", file=sys.stderr)
        sys.exit(1)


if __name__ == '__main__':
    main()


path: .scripts_map.json
content: {
  "nginx": {
    "name": "nginx服务重启",
    "script_id": 187,
    "keywords": ["nginx", "Nginx", "NGINX", "web", "80端口", "http服务", "服务停止"],
    "classifications": [102],
    "chat_groups": ["oc_cfe1ce2fea538b9c65ae8f56588a4440"],
    "description": "重启Nginx服务，适用于服务停止、无响应等场景"
  },
  "disk": {
    "name": "主机磁盘空间清理",
    "script_id": 197,
    "keywords": ["磁盘", "disk", "空间", "storage", "/var", "/tmp", "使用率", "满", "清理"],
    "classifications": [101],
    "chat_groups": ["oc_7e627a830d40ab220ff3bf243b113fee"],
    "description": "清理日志文件和临时文件，释放磁盘空间"
  }
}


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)