Skill: ivangdavila/prom (version 1.0.0)

--- name: Prometheus description: Prometheus monitoring patterns, cardinality management, alerting best practices, and PromQL traps. metadata: category: infrastructure skills: ["prometheus", "monitoring", "alerting", "metrics", "observability"] --- ## Cardinality Explosions - Every unique label combination creates a new time series — `user_id` as label kills Prometheus - Avoid high-cardinality labels: user IDs, email addresses, request IDs, timestamps, UUIDs - Check cardinality: `prometheus_tsdb_head_series` metric — above 1M series needs attention - Use histograms for latency, not per-request labels — buckets are fixed cardinality - Relabeling can drop dangerous labels before ingestion: `labeldrop` in scrape config ## Histogram vs Summary - Histograms: use for SLOs, aggregatable across instances, buckets defined upfront - Summaries: use when you need exact percentiles, cannot aggregate across instances - Histogram bucket boundaries must be defined before data arrives — wrong buckets = wrong percentiles - Default buckets (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10) assume HTTP latency — adjust for your use case ## Rate and Increase - `rate()` requires range selector at least 4x scrape interval — `rate(metric[1m])` with 30s scrape misses data - `rate()` is per-second, `increase()` is total over range — don't confuse them - Counter resets on restart — `rate()` handles this, raw delta doesn't - `irate()` uses only last two samples — too spiky for alerting, use `rate()` for alerts ## Alerting Mistakes - Alert on symptoms, not causes — "high latency" not "high CPU" - `for` clause prevents flapping: `for: 5m` means condition must hold 5 minutes before firing - Missing `for` clause = fires immediately on first match = noisy - Alerts need `runbook_url` label — on-call needs to know what to do, not just that something's wrong - Test alerts with `promtool check rules` — syntax errors discovered at 3am are bad ## PromQL Traps - `and` is intersection by labels, not boolean AND — results must have matching label sets - `or` fills in missing series, doesn't do boolean OR on values - `{}` without metric name is expensive — scans all metrics - `offset` goes back in time: `metric offset 1h` is value from 1 hour ago - Comparison operators filter series: `http_requests > 100` drops series below 100, doesn't return boolean ## Scrape Configuration - `honor_labels: true` trusts source labels — use only when source is authoritative (e.g., Pushgateway) - `scrape_timeout` must be less than `scrape_interval` — otherwise overlapping scrapes - Static configs don't reload without restart — use file_sd or service discovery for dynamic targets - TLS verification disabled (`insecure_skip_verify`) should be temporary, never permanent ## Pushgateway Pitfalls - Pushgateway is for batch jobs, not services — services should expose /metrics - Metrics persist until deleted — stale metrics from dead jobs confuse dashboards - Add job and instance labels to distinguish sources — default grouping hides failures - Delete metrics when job completes: `curl -X DELETE http://pushgateway/metrics/job/myjob` ## Recording Rules - Pre-compute expensive queries: `record: job:request_duration_seconds:rate5m` - Naming convention: `level:metric:operations` — helps identify what rules produce - Recording rules update every evaluation interval — not instant, plan for slight delay - Reduce cardinality with recording rules: aggregate away labels you don't need for alerting ## Federation and Remote Write - Federation for pulling from other Prometheus — use sparingly, adds latency - Remote write for long-term storage — Prometheus local storage is not durable - Remote write can buffer during outages — but buffer is finite, data loss on extended outages - Prometheus is not highly available by default — run two instances scraping same targets ## Common Operational Issues - TSDB corruption on unclean shutdown — use `--storage.tsdb.wal-compression` and monitor disk space - Memory grows with series count — each series costs ~3KB RAM - Compaction pauses during high load — leave 40% disk headroom - Scrape targets stuck "Unknown" — check network, firewall, target actually exposing /metrics ## Label Best Practices - Use labels for dimensions you'll filter/aggregate by — environment, service, instance - Keep label values low-cardinality — tens or hundreds, not thousands - Consistent naming: `snake_case`, prefix with domain: `http_requests_total`, `node_cpu_seconds_total` - `le` label is reserved for histogram buckets — don't use for other purposes


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)