Skill: chefboyrdave21/itil-ops (version 1.0.0)

--- name: itil-ops description: > ITIL-aligned incident, problem, and change management for AI agents. Use when: detecting service crashes, analyzing recurring failures, tracking incidents to resolution, performing root cause analysis, managing change requests, running health audits, or building operational review pipelines. Implements ITIL 4 practices adapted for autonomous agent operations: Incident Management, Problem Management, Change Management, Event Management, and Continual Improvement. Works with systemd, cron, journalctl, and coordination task boards. --- # ITIL Ops — IT Service Management for AI Agents Structured incident, problem, and change management adapted from ITIL 4 for autonomous agent operations. ## Core Concepts ### Severity Levels | Level | Meaning | Response | Example | |-------|---------|----------|---------| | **P1** | Critical — service down, data at risk | Immediate alert + auto-remediate | Crash loop, disk full, OOM | | **P2** | High — degraded service | Alert within 1h | Service restarts, auth failures | | **P3** | Medium — non-critical issue | Next review cycle | Cron timeouts, broken files | | **P4** | Low — cosmetic/minor | Track, fix when convenient | Log warnings, config drift | ### Incident vs Problem vs Change - **Incident**: Something broke. Restore service ASAP. (reactive) - **Problem**: Pattern of incidents. Find and fix root cause. (proactive) - **Change**: Planned modification. Assess risk before executing. (controlled) ## Incident Management ### Detection Sources Scan these in order of criticality: 1. **Service crashes** — `journalctl --user -u SERVICE --since "12 hours ago"` for watchdog timeouts, SIGABRT, SIGSEGV, core dumps 2. **Cron failures** — consecutive error count > 2 in job state files 3. **Health endpoints** — HTTP health checks returning non-200 4. **Resource pressure** — disk > 80%, RAM > 80%, swap active 5. **Data integrity** — schema validation failures, broken files, load errors ### Detection Script Run `scripts/itil-review.sh` to scan all sources. It outputs: - `ITIL_CLEAR` if nothing found (reply HEARTBEAT_OK) - Formatted report with incidents and problems if issues detected ### Incident Lifecycle ``` DETECTED → CLASSIFIED (P1-P4) → DIAGNOSED → RESOLVED → CLOSED ↓ (3+ occurrences) ↓ ESCALATE TO PROBLEM ``` ### Auto-Classification Rules ```bash # P1 — Critical - Service crash count >= 3 in 12h (crash loop) - Disk usage >= 90% - RAM usage >= 90% - Data loss detected # P2 — High - Service crashed 1-2 times - 3+ services down simultaneously - Auth/token failures affecting operations - Cron job with 5+ consecutive failures # P3 — Medium - Broken data files (schema violations) - Memory load errors > 10 in 12h - Cron job with 3-4 consecutive failures - Disk usage 80-89% # P4 — Low - 1 service down (non-critical) - Config warnings - Log noise ``` ### Creating Incident Tickets When incidents are found, create coordination tasks: ``` Title: [ITIL-INC] <brief description> Body: - Severity: P1/P2/P3/P4 - Category: service|cron|memory|disk|security - Detected: <timestamp> - Detail: <what happened> - Impact: <what's affected> - Action: <what to do> ``` ## Problem Management ### Pattern Detection An incident becomes a problem when: - Same error occurs **3+ times in 24h** - Same incident type recurs **across 2+ review cycles** - Multiple related incidents share a **common root cause** ### Root Cause Analysis (RCA) When a problem is identified: 1. **Gather evidence** — journal logs, error messages, state files, recent changes 2. **Timeline** — reconstruct the sequence of events 3. **5 Whys** — ask why iteratively until you reach the actual root cause 4. **Fix classification**: - **Quick fix** — config change, file repair, timeout bump - **Code fix** — bug in script or daemon, needs PR - **Architecture fix** — design flaw, needs redesign ### Problem Ticket Format ``` Title: [ITIL-PRB] <root cause description> Body: - Related incidents: <list> - Root cause: <what's actually broken> - Evidence: <logs, patterns, data> - Fix applied: <immediate remediation> - Fix needed: <permanent solution> - Prevention: <how to prevent recurrence> ``` ### Known Error Database Track resolved problems in state file (`itil-state.json`): ```json { "last_review": "2026-03-22T04:19:50Z", "last_incident_count": 2, "last_problem_count": 1, "known_errors": { "memory-content-dict": { "description": "Scripts writing content as dict instead of string", "root_cause": "Missing json.dumps() in memory file writers", "fix": "Wrap content in json.dumps() before saving", "fixed_date": "2026-03-22" } } } ``` ## Change Management ### Pre-Change Checklist Before modifying services, configs, or infrastructure: 1. **What's changing?** — specific files, services, configs 2. **Why?** — linked incident/problem ticket 3. **Risk?** — what could go wrong 4. **Rollback plan?** — how to undo if it breaks 5. **Test?** — how to verify it worked 6. **Notify?** — does the human need to know ### Change Categories | Type | Approval | Example | |------|----------|---------| | **Standard** | Pre-approved, just do it | Restart service, bump timeout | | **Normal** | Inform human, wait for OK | New cron job, config change | | **Emergency** | Fix now, inform after | Service down, data at risk | ### Post-Change Verification After any change: 1. Check service status — `systemctl --user status SERVICE` 2. Watch logs for 60s — `journalctl --user -u SERVICE -f --since "now"` 3. Run health check — `scripts/itil-review.sh` 4. Verify no new errors in first 5 minutes ## Event Management ### Log Monitoring Patterns ```bash # Service crashes journalctl --user -u SERVICE --since "12h ago" | grep -ciE "watchdog timeout|killed|SIGABRT|SIGSEGV|failed with" # Memory/resource issues journalctl --user -u SERVICE --since "12h ago" | grep -c "Failed to load" # Auth failures journalctl --user -u SERVICE --since "12h ago" | grep -ciE "unauthorized|403|token expired|auth fail" ``` ### Health Check Endpoints Check services with curl: ```bash curl -sf --max-time 5 "$URL" >/dev/null 2>&1 || echo "DOWN" ``` Configure endpoints in the review script for your environment. ## Continual Improvement ### Review Cadence | Review | Frequency | Purpose | |--------|-----------|---------| | **Incident review** | Every 12h | Detect and classify new issues | | **Problem review** | Weekly | Identify patterns, track RCA progress | | **Capacity review** | Weekly | Disk, RAM, memory count trends | | **Process review** | Monthly | Are our detection rules catching real issues? | ### KPIs to Track - **MTTR** (Mean Time to Resolve) — how fast do we fix incidents? - **Incident recurrence rate** — are the same things breaking? - **False positive rate** — are we alerting on non-issues? - **Known error resolution** — are problems getting permanent fixes? ### State Tracking The review script maintains `itil-state.json` with: - Last review timestamp and results - Incident/problem counts per review - System metrics (disk, RAM, restart count) - Cross-review pattern detection data ## Cron Setup ### Recommended Schedule ```bash # Incident review — every 12 hours openclaw cron add --name "itil-review" --every "12h" \ --model "anthropic/claude-sonnet-4-6" --timeout-seconds 180 \ --session isolated \ --message "Run ITIL review: bash ~/.skcapstone/agents/lumina/scripts/itil-review.sh" # Weekly problem review (Sunday 9 AM) # Analyze the week's incidents, identify patterns, suggest improvements ``` ## File Structure ``` itil-ops/ ├── SKILL.md # This file ├── scripts/ │ └── itil-review.sh # Main review script (scan + classify + report) └── references/ └── itil4-agent-mapping.md # ITIL 4 → Agent operations reference ``` ## Integration Points - **Coordination tasks** — `skcapstone coord create` for incident/problem tickets - **Memory snapshots** — `skmemory_snapshot` to record resolutions for future reference - **Heartbeat** — integrate with existing heartbeat to run lightweight checks - **Cron** — scheduled reviews via OpenClaw cron system - **Alerting** — Telegram/Discord delivery for P1/P2 issues


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)