Skill: ivangdavila/alerts (version 1.0.0)

--- name: Alerts description: Smart alerting patterns for AI agents - deduplication, routing, escalation, and fatigue prevention --- ## Alert Fatigue Prevention **Group alerts by root cause, never by individual symptoms.** Use labels: `alertname`, `service`, `cluster` - not instance IDs. ```bash # Good: One alert for database down affecting 50 pods group_by: ['alertname', 'service'] # Bad: 50 individual alerts for each failed pod ``` **Implement severity hierarchy: P0 (pages immediately) > P1 (within 15min) > P2 (business hours) > P3 (weekly review).** P0: Service completely down, data loss, security breach. P1: Degraded performance, partial outage, high error rates. **Set cooldown periods to prevent alert spam.** Minimum 5 minutes between identical alerts, 30 minutes for cost alerts. ```bash repeat_interval: 5m # For critical alerts repeat_interval: 30m # For cost/performance alerts ``` **Use inhibition rules to suppress symptoms when root cause fires.** If "Database Unreachable" fires, silence all "API High Latency" alerts from same cluster. ## AI Agent Monitoring Patterns **Monitor token/API usage with exponential alerting thresholds.** Alert at 2x, 5x, 10x normal usage - costs can spiral quickly. Track: tokens per minute, cost per request, API rate limits approached. **Set behavioral drift alerts on response quality degradation.** Compare current outputs to baseline with sample prompts every hour. Alert when success rate drops below 85% or response time exceeds 2x baseline. **Monitor for infinite loops in multi-agent workflows.** Alert if same prompt sent >3 times in 5 minutes or agent hasn't responded in 10 minutes. Include correlation IDs to trace conversation chains. **Track silent failures through downstream metrics.** Monitor: tasks completed vs started, user satisfaction scores, retry attempts. These catch errors that don't throw exceptions. ## Routing and Escalation Rules **Route by expertise domain, not arbitrary on-call schedules.** Database alerts → DB team, API alerts → backend team, cost alerts → platform team. Only escalate to managers for P0 incidents lasting >30 minutes. **Use progressive escalation with increasing urgency.** P1 alerts: Slack notification → 5min wait → SMS → 10min wait → phone call. Include runbook links in every alert for faster resolution. **Set context-aware routing based on time and impact.** Business hours: Route to primary team. Off-hours: Route to on-call only for P0/P1. If >100 users affected: Immediately escalate regardless of severity. ## Webhook Reliability Patterns **Always include correlation IDs for alert lifecycle management.** Generate UUID for each incident, use it to create/update/resolve alerts. Essential for bi-directional integrations with PagerDuty/Slack. **Implement exponential backoff for webhook failures.** Retry after 1s, 2s, 4s, 8s, 16s, then mark failed and escalate. Log webhook response codes/times for debugging delivery issues. **Use webhook verification to prevent spoofing.** Validate signatures using HMAC-SHA256 with shared secret. Always check timestamp to prevent replay attacks (max 5 min old). **Implement circuit breaker pattern for unreliable endpoints.** After 5 consecutive failures, mark endpoint down and use backup channel. Re-test every 30 seconds until recovery confirmed. ## Status Page Integration **Update status page automatically when P0/P1 alerts fire.** Create incident, post initial assessment within 5 minutes. Include ETA and workaround if available. **Use component-based status updates matching your alert groups.** Map alert labels to status page components (API, Database, Auth, etc.). Partial outages should show "Degraded Performance", not "Operational". ## Runbook Automation **Embed runbook links directly in alert messages.** Format: "Alert: High CPU on web-01. Runbook: https://wiki/runbooks/high-cpu-web" Links must be accessible from mobile devices for on-call engineers. **Trigger automated remediation for known issues.** Auto-restart stuck services, clear full disks, reset rate limits. Always require human approval for destructive actions (scaling down, deleting data). **Log all automated actions taken in response to alerts.** Include: timestamp, action, result, approval chain. Essential for post-incident reviews and compliance audits.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)