Skill: qq280948982/promql-validator (version 0.1.0)

--- name: promql-validator description: Validate, lint, audit, or fix PromQL queries and alerting rules; detects anti-patterns. --- ## How This Skill Works This skill performs multi-level validation and provides interactive query planning: 1. **Syntax Validation**: Checks for syntactically correct PromQL expressions 2. **Semantic Validation**: Ensures queries make logical sense (e.g., rate() on counters, not gauges) 3. **Anti-Pattern Detection**: Identifies common mistakes and inefficient patterns 4. **Optimization Suggestions**: Recommends performance improvements 5. **Query Explanation**: Translates PromQL to plain English 6. **Interactive Planning**: Helps users clarify intent and refine queries ## Workflow When a user provides a PromQL query, follow this workflow: ### Working Directory Requirement Run validation commands from the repository root so relative paths resolve correctly: ```bash cd "$(git rev-parse --show-toplevel)" ``` If running from another location, use absolute paths to `scripts/` files. ### Step 1: Validate Syntax Run the syntax validation script to check for basic correctness: ```bash python3 devops-skills-plugin/skills/promql-validator/scripts/validate_syntax.py "<query>" ``` Output parsing notes: - Exit `0`: syntax valid - Exit non-zero: syntax failure; include stderr and pinpoint token/position - Prefer quoting the smallest failing fragment, then provide corrected query The script will check for: - Valid metric names and label matchers - Correct operator usage - Proper function syntax - Valid time durations and ranges - Balanced brackets and quotes - Correct use of modifiers (offset, @) ### Step 2: Check Best Practices Run the best practices checker to detect anti-patterns and optimization opportunities: ```bash python3 devops-skills-plugin/skills/promql-validator/scripts/check_best_practices.py "<query>" ``` Output parsing notes: - Treat script sections as independent findings (cardinality, metric-type misuse, regex misuse, etc.) - If script output is empty but query is complex, add a manual sanity pass and mark it as `manual-review` - Preserve script wording for finding labels, then add remediation in plain English The script will identify: - High cardinality queries without label filters - Inefficient regex matchers that could be exact matches - Missing rate()/increase() on counter metrics - rate() used on gauge metrics - Averaging pre-calculated quantiles - Subqueries with excessive time ranges - irate() over long time ranges - Opportunities to add more specific label filters - Complex queries that should use recording rules ### Step 3: Explain the Query Parse and explain what the query does in plain English: - What metrics are being queried - What type of metrics they are (counter, gauge, histogram, summary) - What functions are applied and why - What the query calculates - What labels will be in the output - What the expected result structure looks like **Required Output Details** (always include these explicitly): ``` **Output Labels**: [list labels that will be in the result, or "None (fully aggregated to scalar)"] **Expected Result Structure**: [instant vector / range vector / scalar] with [N series / single value] ``` Example: ``` **Output Labels**: job, instance **Expected Result Structure**: Instant vector with one series per job/instance combination ``` ### Line-Number Citation Method (Required) When citing examples/docs in recommendations, include file path + 1-based line numbers: ```text examples/good_queries.promql:42 docs/best_practices.md:88 ``` Rules: - Cite the most relevant single line (or start line if multi-line snippet) - Keep citations tight; do not cite full files - If line numbers are unavailable, state `line number unavailable` and provide file path ### Step 4: Interactive Query Planning (Phase 1 - STOP AND WAIT) Ask the user clarifying questions to verify the query matches their intent: 1. **Understand the Goal**: "What are you trying to monitor or measure?" - Request rate, error rate, latency, resource usage, etc. 2. **Verify Metric Type**: "Is this a counter (always increasing), gauge (can go up/down), histogram, or summary?" - This affects which functions to use 3. **Clarify Time Range**: "What time window do you need?" - Instant value, rate over time, historical analysis 4. **Confirm Aggregation**: "Do you need to aggregate data across labels? If so, which labels?" - by (job), by (instance), without (pod), etc. 5. **Check Output Intent**: "Are you using this for alerting, dashboarding, or ad-hoc analysis?" - Affects optimization priorities > **IMPORTANT: Two-Phase Dialogue** > > After presenting Steps 1-4 results (Syntax, Best Practices, Query Explanation, and Intent Questions): > > **⏸️ STOP HERE AND WAIT FOR USER RESPONSE** > > Do NOT proceed to Steps 5-7 until the user answers the clarifying questions. > This ensures the subsequent recommendations are tailored to the user's actual intent. ### Step 5: Compare Intent vs Implementation (Phase 2 - After User Response) **Only proceed to this step after the user has answered the clarifying questions from Step 4.** After understanding the user's intent: - Explain what the current query actually does - Highlight any mismatches between intent and implementation - Suggest corrections if the query doesn't match the goal - Offer alternative approaches if applicable When relevant, mention known limitations: - Note when metric type detection is heuristic-based (e.g., "The script inferred this is a gauge based on the `_bytes` suffix. Please confirm if this is correct.") - Acknowledge when high-cardinality warnings might be false positives (e.g., "This warning may not apply if you're using a recording rule or know your cardinality is low.") ### Step 6: Offer Optimizations Based on validation results: - Suggest more efficient query patterns - Recommend recording rules for complex/repeated queries - Propose better label matchers to reduce cardinality - Advise on appropriate time ranges **Reference Examples**: When suggesting corrections, cite relevant examples using this format: ``` As shown in `examples/bad_queries.promql` (lines 91-97): ❌ BAD: `avg(http_request_duration_seconds{quantile="0.95"})` ✅ GOOD: Use histogram_quantile() with histogram buckets ``` Citation sources: - `examples/good_queries.promql` - for well-formed patterns - `examples/optimization_examples.promql` - for before/after comparisons - `examples/bad_queries.promql` - for showing what to avoid - `docs/best_practices.md` - for detailed explanations - `docs/anti_patterns.md` - for anti-pattern deep dives **Citation Format**: `file_path (lines X-Y)` with the relevant code snippet quoted ### Step 7: Let User Plan/Refine Give the user control: - Ask if they want to modify the query - Offer to help rewrite it for better performance - Provide multiple alternatives if applicable - Explain trade-offs between different approaches ## Key Validation Rules ### Syntax Rules 1. **Metric Names**: Must match `[a-zA-Z_:][a-zA-Z0-9_:]*` or use UTF-8 quoting syntax (Prometheus 3.0+): - Quoted form: `{"my.metric.with.dots"}` - Using __name__ label: `{__name__="my.metric.with.dots"}` 2. **Label Matchers**: `=` (equal), `!=` (not equal), `=~` (regex match), `!~` (regex not match) 3. **Time Durations**: `[0-9]+(ms|s|m|h|d|w|y)` - e.g., `5m`, `1h`, `7d` 4. **Range Vectors**: `metric_name[duration]` - e.g., `http_requests_total[5m]` 5. **Offset Modifier**: `offset <duration>` - e.g., `metric_name offset 5m` 6. **@ Modifier**: `@ <timestamp>` or `@ start()` / `@ end()` ### Semantic Rules 1. **rate() and irate()**: Should only be used with counter metrics (metrics ending in `_total`, `_count`, `_sum`, or `_bucket`) 2. **Counters**: Should typically use `rate()` or `increase()`, not raw values 3. **Gauges**: Should not use `rate()` or `increase()` 4. **Histograms**: Use `histogram_quantile()` with `le` label and `rate()` on `_bucket` metrics 5. **Summaries**: Don't average quantiles; calculate from `_sum` and `_count` 6. **Aggregations**: Use `by()` or `without()` to control output labels ### Performance Rules 1. **Cardinality**: Always use specific label matchers to reduce series count 2. **Regex**: Use `=` instead of `=~` when possible for exact matches 3. **Rate Range**: Should be at least 4x the scrape interval (typically `[2m]` minimum) 4. **irate()**: Best for short ranges (<5m); use `rate()` for longer periods 5. **Subqueries**: Avoid excessive time ranges that process millions of samples 6. **Recording Rules**: Use for complex queries accessed frequently ## Anti-Patterns to Detect ### High Cardinality Issues ❌ **Bad**: `http_requests_total{}` - Matches all time series without filtering ✅ **Good**: `http_requests_total{job="api", instance="prod-1"}` - Specific label filters reduce cardinality ### Regex Overuse ❌ **Bad**: `http_requests_total{status=~"2.."}` - Regex is slower and less precise ✅ **Good**: `http_requests_total{status="200"}` - Exact match is faster ### Missing rate() on Counters ❌ **Bad**: `http_requests_total` - Counter raw values are not useful (always increasing) ✅ **Good**: `rate(http_requests_total[5m])` - Rate shows requests per second ### rate() on Gauges ❌ **Bad**: `rate(memory_usage_bytes[5m])` - Gauges measure current state, not cumulative values ✅ **Good**: `memory_usage_bytes` - Use gauge value directly or with `avg_over_time()` ### Averaging Quantiles ❌ **Bad**: `avg(http_request_duration_seconds{quantile="0.95"})` - Mathematically invalid to average pre-calculated quantiles ✅ **Good**: `histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))` - Calculate quantile from histogram buckets ### Excessive Subquery Ranges ❌ **Bad**: `rate(metric[5m])[90d:1m]` - Processes millions of samples, very slow ✅ **Good**: Use recording rules or limit range to necessary duration ### irate() Over Long Ranges ❌ **Bad**: `irate(metric[1h])` - irate() only looks at last two samples, range is wasted ✅ **Good**: `rate(metric[1h])` or `irate(metric[5m])` - Use rate() for longer ranges or reduce irate() range ### Mixed Metric Types ❌ **Bad**: `avg(http_request_duration_seconds{quantile="0.95"}) / rate(node_memory_usage_bytes[1h]) + sum(http_requests_total)` - Combines summary quantiles, gauge metrics, and counters in arithmetic - Produces meaningless results ✅ **Good**: Keep each metric type in separate, purpose-specific queries: - Latency: `histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))` - Memory: `node_memory_usage_bytes{instance="prod-1"}` - Request rate: `rate(http_requests_total{job="api"}[5m])` ## Output Format Provide validation results in this structure: ``` ## PromQL Validation Results ### Syntax Check - Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR - Issues: [list any syntax errors with line/position] ### Semantic Check - Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR - Issues: [list any logical problems] ### Performance Analysis - Status: ✅ OPTIMIZED / ⚠️ CAN BE IMPROVED / ❌ INEFFICIENT - Issues: [list optimization opportunities] - Suggestions: [specific improvements] ### Query Explanation Your query: `<query>` This query does: - [Plain English explanation] - Metrics: [list metrics and their types] - Functions: [explain each function] - Output: [describe result structure] ### Intent Verification Let me verify this matches your needs: 1. What are you trying to measure? [your goal here] 2. Is this a counter/gauge/histogram/summary? [metric type] 3. What time range interests you? [time window] 4. Do you need aggregation? If so, by which labels? [aggregation needs] 5. Is this for alerting, dashboarding, or analysis? [use case] ### Recommendations [Based on the analysis, suggest improvements or alternatives] ``` ## Interactive Dialogue After validation, engage in dialogue: **Claude**: "I've validated your query. It's syntactically correct, but I notice it queries `http_requests_total` without any label filters. This could match thousands of time series. What specific service or endpoint are you trying to monitor?" **User**: [provides intent] **Claude**: "Great! Based on that, here's an optimized version: `rate(http_requests_total{job="api-service", path="/users"}[5m])`. This calculates the per-second rate of requests to the /users endpoint over the last 5 minutes. Does this match what you need?" **User**: [confirms or asks for changes] **Claude**: [provides refined query or alternatives] ## Examples See the `examples/` directory for: - `good_queries.promql`: Well-written queries following best practices - `bad_queries.promql`: Common mistakes and anti-patterns (with corrections) - `optimization_examples.promql`: Before/after optimization examples ## Documentation See the `docs/` directory for: - `best_practices.md`: Comprehensive PromQL best practices guide - `anti_patterns.md`: Detailed anti-pattern reference with explanations ## Important Notes 1. **Be Interactive**: Always ask clarifying questions to understand user intent 2. **Be Educational**: Explain WHY something is wrong, not just THAT it's wrong 3. **Be Helpful**: Offer to rewrite queries, don't just criticize 4. **Be Context-Aware**: Consider the user's use case (alerting vs dashboarding) 5. **Be Thorough**: Check all four levels (syntax, semantics, performance, intent) 6. **Be Practical**: Suggest realistic optimizations, not theoretical perfection ## Integration This skill can be used: - Standalone for query review - During monitoring setup to validate alert rules - When troubleshooting slow Prometheus queries - As part of code review for recording rules - For teaching PromQL to team members ## Validation Tools The skill uses two main Python scripts: 1. **validate_syntax.py**: Pure syntax checking using regex patterns 2. **check_best_practices.py**: Semantic and performance analysis Both scripts output JSON for programmatic parsing and human-readable messages for display. ## Success Criteria A successful validation session should: 1. Identify all syntax errors 2. Detect semantic problems 3. Suggest at least one optimization (if applicable) 4. Clearly explain what the query does 5. Verify the query matches user intent 6. Provide actionable next steps ## Known Limitations The validation scripts have some limitations to be aware of: ### Metric Type Detection - **Heuristic-based**: Metric types (counter, gauge, histogram, summary) are inferred from naming conventions (e.g., `_total`, `_bytes`) - **Custom metrics**: Metrics with non-standard names may not be correctly classified - **Recommendation**: When the script can't determine metric type, ask the user to clarify ### High Cardinality Detection - **Conservative approach**: The script flags metrics without label selectors, but some use cases legitimately query all series - **Recording rules**: Queries using recording rule metrics (e.g., `job:http_requests:rate5m`) are valid without label filters - **Recommendation**: Use judgment - if the user knows their cardinality is manageable, the warning can be safely ignored ### Semantic Validation - **No runtime context**: The scripts cannot verify if metrics actually exist or if label values are valid - **Schema-agnostic**: No knowledge of specific Prometheus deployments or metric schemas - **Recommendation**: For production validation, test queries against actual Prometheus instances ### Script Detection Coverage The scripts detect common anti-patterns but cannot catch: - Business logic errors (e.g., calculating the wrong KPI) - Context-specific optimizations (depends on scrape interval, retention, etc.) - Custom function behavior from extensions ## Remember The goal is not just to validate queries, but to help users write better PromQL and understand their monitoring data. Always be educational, interactive, and helpful!

path: docs/anti_patterns.md
content: # PromQL Anti-Patterns

Comprehensive guide to common mistakes, anti-patterns, and pitfalls in PromQL queries.

## Table of Contents

1. [High Cardinality Issues](#high-cardinality-issues)
2. [Incorrect Function Usage](#incorrect-function-usage)
3. [Performance Anti-Patterns](#performance-anti-patterns)
4. [Mathematical Errors](#mathematical-errors)
5. [Label Matching Problems](#label-matching-problems)
6. [Aggregation Mistakes](#aggregation-mistakes)
7. [Time Range Issues](#time-range-issues)
8. [Histogram and Summary Misuse](#histogram-and-summary-misuse)

---

## High Cardinality Issues

### Anti-Pattern 1: Unbounded Metric Selectors

**Problem**: Querying metrics without any label filters matches all time series, causing high cardinality.

```promql
# ❌ BAD: No filters - could match thousands of series
http_requests_total

# ❌ BAD: Empty label matcher
http_requests_total{}

# ✅ GOOD: Specific label filters
http_requests_total{job="api-service", environment="production"}
```

**Impact**:
- Query times: seconds to minutes instead of milliseconds
- High memory usage on Prometheus
- Risk of query timeouts
- Increased load on Prometheus server

**Detection**: Look for metric names without `{...}` selectors or with empty `{}`.

---

### Anti-Pattern 2: High-Cardinality Labels in Queries

**Problem**: Querying on labels with thousands of unique values.

```promql
# ❌ BAD: User ID has millions of unique values
http_requests_total{user_id="12345"}

# ✅ GOOD: Use low-cardinality labels
http_requests_total{job="api", endpoint="/users"}
```

**High-cardinality labels to avoid**:
- User IDs, customer IDs
- Request IDs, trace IDs
- Timestamps
- UUIDs
- Email addresses, IP addresses (unless aggregated)
- Full URLs (use path patterns instead)

**Low-cardinality labels (safe to use)**:
- Job name
- Instance name
- Service name
- Environment (prod, staging, dev)
- Status codes
- HTTP methods
- Endpoint paths (grouped)

---

### Anti-Pattern 3: Wildcard Regex Without Constraints

**Problem**: Overly broad regex patterns match too many series.

```promql
# ❌ BAD: Matches everything
http_requests_total{path=~".*"}

# ❌ BAD: Very broad pattern
http_requests_total{instance=~".*-prod-.*"}

# ✅ GOOD: Specific pattern with other filters
http_requests_total{
  job="api",
  instance=~"api-prod-[0-9]+",
  datacenter="us-east-1"
}
```

---

## Incorrect Function Usage

### Anti-Pattern 4: Missing rate() on Counters

**Problem**: Using counter metrics without rate() or increase() gives meaningless values.

```promql
# ❌ BAD: Raw counter value (always increasing, not useful)
http_requests_total{job="api"}

# ❌ BAD: Aggregating raw counters
sum(http_requests_total)

# ✅ GOOD: Use rate() for per-second rate
rate(http_requests_total{job="api"}[5m])

# ✅ GOOD: Use increase() for total increase
increase(http_requests_total{job="api"}[1h])
```

**Why it's wrong**: Counters only increase (or reset). The raw value shows total count since process start, not current rate.

---

### Anti-Pattern 5: Using rate() on Gauges

**Problem**: rate(), irate(), and increase() assume monotonically increasing values. Gauges go up and down.

```promql
# ❌ BAD: rate() on gauge (memory can go up or down)
rate(node_memory_usage_bytes[5m])

# ❌ BAD: irate() on gauge
irate(cpu_temperature_celsius[5m])

# ✅ GOOD: Use gauge directly
node_memory_usage_bytes

# ✅ GOOD: Or use avg_over_time for smoothing
avg_over_time(node_memory_usage_bytes[5m])

# ✅ GOOD: Use delta() if you need change over time
delta(cpu_temperature_celsius[5m])
```

**How to identify**:
- Counters typically end with: `_total`, `_count`, `_sum`, `_bucket`
- Gauges typically indicate current state: `_bytes`, `_usage`, `_percent`, `_celsius`

---

### Anti-Pattern 6: rate() Without Range Vector

**Problem**: rate(), irate(), increase() require a time range.

```promql
# ❌ BAD: Missing range vector
rate(http_requests_total)

# ❌ BAD: Missing range vector
increase(requests_total{job="api"})

# ✅ GOOD: Include time range
rate(http_requests_total[5m])

# ✅ GOOD: Include range for increase
increase(requests_total{job="api"}[1h])
```

**Error message**: `"parse error: expected type range vector in call to function, got instant vector"`

---

## Performance Anti-Patterns

### Anti-Pattern 7: Excessive Subquery Time Ranges

**Problem**: Subqueries over very long time ranges process millions of samples.

```promql
# ❌ BAD: 95-day subquery (extremely slow, may timeout)
max_over_time(rate(http_requests_total[5m])[95d:1m])

# ❌ BAD: Long range with high resolution
avg_over_time(metric[30d:10s])

# ✅ GOOD: Reasonable time range
max_over_time(rate(http_requests_total[5m])[7d:5m])

# ✅ BETTER: Use recording rules for long-term analysis
# Create recording rule:
# - record: :http_requests:rate5m
#   expr: rate(http_requests_total[5m])
# Then query:
max_over_time(:http_requests:rate5m[30d:1h])
```

**Impact**:
- Query timeouts
- Excessive memory usage (GBs)
- Prometheus server overload
- Minutes to execute instead of seconds

---

### Anti-Pattern 8: Regex Instead of Exact Match

**Problem**: Using regex (=~) when exact match (=) would work.

```promql
# ❌ BAD: Regex for exact match (slower)
http_requests_total{status=~"200"}

# ❌ BAD: Regex that's actually exact
http_requests_total{job=~"api-service"}

# ✅ GOOD: Exact match (faster index lookup)
http_requests_total{status="200"}

# ✅ GOOD: Exact match
http_requests_total{job="api-service"}
```

**Performance difference**: Exact matches can be 5-10x faster due to index lookups vs pattern matching.

**When regex IS appropriate**:
```promql
# ✅ GOOD: Multiple alternatives
http_requests_total{status=~"200|201|204"}

# ✅ GOOD: Pattern matching
http_requests_total{path=~"/api/v[0-9]+/.*"}

# ✅ GOOD: Exclusions
http_requests_total{path!~"/health|/metrics"}
```

---

### Anti-Pattern 9: Not Using Recording Rules for Complex Queries

**Problem**: Running expensive queries repeatedly in multiple dashboards/alerts.

```promql
# ❌ BAD: Complex query used in 10 dashboards, evaluated 100 times/minute
sum by (job, instance) (
  rate(http_request_duration_seconds_sum{job="api"}[5m])
) /
sum by (job, instance) (
  rate(http_request_duration_seconds_count{job="api"}[5m])
)

# ✅ GOOD: Create recording rule (evaluated once per cycle)
# prometheus.yml:
# - record: job_instance:http_request_duration_seconds:mean5m
#   expr: |
#     sum by (job, instance) (
#       rate(http_request_duration_seconds_sum{job="api"}[5m])
#     ) /
#     sum by (job, instance) (
#       rate(http_request_duration_seconds_count{job="api"}[5m])
#     )

# Then use pre-computed metric:
job_instance:http_request_duration_seconds:mean5m
```

**When to use recording rules**:
- Query is complex (multiple functions, aggregations)
- Query is used frequently (dashboards, multiple alerts)
- Query is slow (>1 second)
- Query uses subqueries

---

### Anti-Pattern 10: Filter After Aggregation

**Problem**: Filtering after expensive aggregation processes unnecessary data.

```promql
# ❌ BAD: Aggregates all jobs first, then filters
sum(rate(http_requests_total[5m])) and {job="api"}

# ❌ BAD: Processes all data before filtering
sum by (job) (rate(http_requests_total[5m])) == {job="api"}

# ✅ GOOD: Filter first, then aggregate
sum(rate(http_requests_total{job="api"}[5m]))

# ✅ GOOD: Specific filters reduce data early
sum by (path) (rate(http_requests_total{job="api", status="200"}[5m]))
```

**Performance impact**: 10-100x slower when filtering after aggregation.

---

## Mathematical Errors

### Anti-Pattern 11: Averaging Pre-Calculated Quantiles

**Problem**: Averaging quantiles across instances is mathematically invalid.

```promql
# ❌ BAD: Mathematically incorrect!
avg(http_request_duration_seconds{quantile="0.95"})

# ❌ BAD: Summing quantiles is also wrong
sum(response_time_seconds{quantile="0.99"})

# ✅ GOOD: Calculate quantile from histogram buckets
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# ✅ GOOD: Calculate average from _sum and _count
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])
```

**Why it's wrong**: Quantiles are non-additive. The average of two 95th percentiles is NOT the overall 95th percentile.

**Solution**: Use histograms instead of summaries when you need aggregation.

---

### Anti-Pattern 12: Division with Mismatched Labels

**Problem**: Dividing metrics with different label sets gives unexpected results.

```promql
# ❌ BAD: Labels don't match (no instance on right side)
rate(http_requests_total{job="api", instance="host1"}[5m])
/
rate(http_requests_total{job="api"}[5m])

# Result: No data (label mismatch)

# ✅ GOOD: Ensure both sides have same label filters
rate(http_requests_total{job="api", status="500"}[5m])
/
rate(http_requests_total{job="api"}[5m])

# ✅ GOOD: Use aggregation to match label dimensions
sum(rate(http_requests_total{status="500"}[5m]))
/
sum(rate(http_requests_total[5m]))
```

---

## Label Matching Problems

### Anti-Pattern 13: Incorrect offset Modifier Usage

**Problem**: Using offset incorrectly or misunderstanding its placement.

```promql
# ✅ CORRECT: offset after range vector selector
http_requests_total[5m] offset 1h

# ✅ CORRECT: offset with instant vector
http_requests_total offset 1h

# ✅ CORRECT: offset inside rate() function
rate(http_requests_total[5m] offset 1h)

# ❌ BAD: offset between metric name and range (invalid syntax)
http_requests_total offset 1h [5m]
```

**Note**: The `offset` modifier shifts the time range back by the specified duration. It comes AFTER the selector (including range vector bracket if present).

---

### Anti-Pattern 14: Multiple OR for Same Label

**Problem**: Using multiple OR operations instead of regex alternation.

```promql
# ❌ BAD: Multiple queries combined with OR
http_requests_total{job="api"}
or
http_requests_total{job="web"}
or
http_requests_total{job="worker"}

# ✅ GOOD: Single regex with alternatives
http_requests_total{job=~"api|web|worker"}

# ✅ GOOD: With aggregation
sum by (job) (rate(http_requests_total{job=~"api|web|worker"}[5m]))
```

**Performance**: Single regex is 3-5x faster than multiple ORs.

---

## Aggregation Mistakes

### Anti-Pattern 15: Aggregation Without by() or without()

**Problem**: Unclear what labels remain after aggregation.

```promql
# ❌ BAD: What labels are in the result?
sum(rate(http_requests_total[5m]))

# ❌ BAD: Unclear aggregation
avg(node_memory_usage_bytes)

# ✅ GOOD: Explicit grouping
sum by (job, instance) (rate(http_requests_total[5m]))

# ✅ GOOD: Explicit label dropping
sum without (pod, container) (rate(http_requests_total[5m]))
```

---

### Anti-Pattern 16: Aggregating Before Division

**Problem**: Order of operations affects results for ratios.

```promql
# ❌ BAD: Sum first, then divide (wrong denominator)
sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

# This gives overall average, not average per instance

# ✅ GOOD: Divide first, then aggregate (if you want per-instance avg)
sum(
  rate(http_request_duration_seconds_sum[5m])
  /
  rate(http_request_duration_seconds_count[5m])
)

# Note: Both may be valid depending on your goal!
# Be explicit about what you're calculating.
```

---

## Time Range Issues

### Anti-Pattern 17: irate() with Long Ranges

**Problem**: irate() only uses the last two samples, making longer ranges wasteful.

```promql
# ❌ BAD: irate over 1 hour (only uses last 2 samples!)
irate(http_requests_total[1h])

# ❌ BAD: irate over 10 minutes (still only 2 samples)
irate(http_requests_total[10m])

# ✅ GOOD: Use rate() for longer ranges
rate(http_requests_total[1h])

# ✅ GOOD: Use irate() with short range
irate(http_requests_total[2m])
```

**When to use irate()**:
- High-frequency monitoring (per-second spikes)
- Short time ranges (2-5 minutes)
- When you want instant rate, not average

**When to use rate()**:
- Most cases
- Alerting (more stable)
- Longer time ranges (>5 minutes)
- When you want average rate over period

---

### Anti-Pattern 18: rate() Range Too Short

**Problem**: rate() range shorter than 4x scrape interval gives inaccurate results.

```promql
# ❌ BAD: 30s range with 15s scrape interval (only 2 samples)
rate(http_requests_total[30s])

# ❌ BAD: 1m range might not have enough samples
rate(http_requests_total[1m])

# ✅ GOOD: At least 4x scrape interval (for 15s scrape: 1m minimum)
rate(http_requests_total[2m])

# ✅ GOOD: 5m is a common, safe choice
rate(http_requests_total[5m])
```

**Rule**: `rate_range >= 4 * scrape_interval`

---

## Histogram and Summary Misuse

### Anti-Pattern 19: histogram_quantile Without rate()

**Problem**: histogram_quantile needs rate() on bucket metrics.

```promql
# ❌ BAD: Missing rate() (uses raw bucket counts)
histogram_quantile(0.95,
  sum by (le) (http_request_duration_seconds_bucket)
)

# ✅ GOOD: Include rate()
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)
```

---

### Anti-Pattern 20: histogram_quantile Without 'le' Label

**Problem**: histogram_quantile requires the 'le' (less than or equal) label.

```promql
# ❌ BAD: Missing 'le' in aggregation
histogram_quantile(0.95,
  sum by (job) (rate(http_request_duration_seconds_bucket[5m]))
)

# ✅ GOOD: Include 'le' in by() clause
histogram_quantile(0.95,
  sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# ✅ GOOD: Remove other labels but keep 'le'
histogram_quantile(0.95,
  sum without (instance, pod) (rate(http_request_duration_seconds_bucket[5m]))
)
```

---

### Anti-Pattern 21: Using Summaries When You Need Aggregation

**Problem**: Summary quantiles cannot be aggregated across instances.

```promql
# ❌ BAD: Cannot meaningfully aggregate summary quantiles
avg(http_request_duration_seconds{quantile="0.95"})

# ✅ SOLUTION: Use histograms instead of summaries
# Histograms allow aggregation:
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)
```

**When to use each**:
- **Histogram**: Need to aggregate across instances, calculate multiple quantiles
- **Summary**: Per-instance quantiles, lower memory overhead, don't need aggregation

---

## Additional Anti-Patterns

### Anti-Pattern 22: Nested Redundant Functions

**Problem**: Applying the same function twice or unnecessary nesting.

```promql
# ❌ BAD: Double rate (doesn't make sense)
rate(rate(http_requests_total[5m])[10m])

# ❌ BAD: Unnecessary nesting
avg(avg_over_time(metric[5m]))

# ✅ GOOD: Single function
rate(http_requests_total[5m])

# ✅ GOOD: Single aggregation
avg_over_time(metric[5m])
```

---

### Anti-Pattern 23: Forgetting group_left/group_right in Joins

**Problem**: One-to-many joins require group_left or group_right.

```promql
# ❌ BAD: Many-to-one join without group_left
rate(http_requests_total[5m])
* on (job, instance)
service_info

# Error: "multiple matches for labels"

# ✅ GOOD: Use group_left to include labels from right side
rate(http_requests_total[5m])
* on (job, instance) group_left (version, commit)
service_info
```

---

## Summary Checklist

Before running your query, check:

- [ ] All metrics have specific label filters
- [ ] Using rate() on counters, not on gauges
- [ ] Using exact match (=) instead of regex (=~) when possible
- [ ] rate() range is at least 2-4 minutes
- [ ] irate() range is 2-5 minutes maximum
- [ ] Aggregations have by() or without() clauses
- [ ] Not averaging pre-calculated quantiles
- [ ] histogram_quantile includes rate() and 'le' label
- [ ] Subquery ranges are reasonable (<7 days typically)
- [ ] Complex/frequent queries use recording rules
- [ ] Not using high-cardinality labels

---

## Resources

- [Prometheus Best Practices](https://prometheus.io/docs/practices/)
- [PromQL Official Documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/)
- [Common Query Patterns](https://www.robustperception.io/common-query-patterns-in-promql/)

path: docs/best_practices.md
content: # PromQL Best Practices

Comprehensive guide to writing efficient, correct, and maintainable PromQL queries.

## Table of Contents

1. [Metric Types and Functions](#metric-types-and-functions)
2. [Label Filtering](#label-filtering)
3. [Aggregations](#aggregations)
4. [Time Ranges](#time-ranges)
5. [Performance Optimization](#performance-optimization)
6. [Recording Rules](#recording-rules)
7. [Histograms and Summaries](#histograms-and-summaries)
8. [Alerting Queries](#alerting-queries)
9. [Common Patterns](#common-patterns)

---

## Metric Types and Functions

### Counters

**What they are**: Metrics that only increase (or reset to zero). Examples: `http_requests_total`, `errors_count`.

**Best practices**:
- ✅ Always use `rate()` or `increase()` with counters
- ✅ Use `rate()` for per-second rates: `rate(http_requests_total[5m])`
- ✅ Use `increase()` for total increase: `increase(http_requests_total[1h])`
- ❌ Never use raw counter values (they always increase, not useful)
- ❌ Never use `rate()` or `increase()` without a range vector

**Naming convention**: Counters typically end with `_total`, `_count`, `_sum`, or `_bucket`.

**Examples**:
```promql
# Good: Calculate requests per second
rate(http_requests_total{job="api"}[5m])

# Good: Total requests in last hour
increase(http_requests_total{job="api"}[1h])

# Bad: Raw counter value
http_requests_total{job="api"}
```

### Gauges

**What they are**: Metrics that can go up and down. Examples: `memory_usage_bytes`, `temperature_celsius`.

**Best practices**:
- ✅ Use gauge values directly
- ✅ Use `avg_over_time()`, `max_over_time()`, `min_over_time()` for time windows
- ✅ Can use `delta()` for change over time (but not common)
- ❌ Never use `rate()`, `irate()`, or `increase()` on gauges
- ❌ These functions assume monotonically increasing values

**Examples**:
```promql
# Good: Current memory usage
node_memory_usage_bytes{instance="prod-1"}

# Good: Average over time
avg_over_time(node_memory_usage_bytes{instance="prod-1"}[5m])

# Good: Maximum in last hour
max_over_time(node_cpu_percent{instance="prod-1"}[1h])

# Bad: Rate on gauge
rate(memory_usage_bytes[5m])
```

### Histograms

**What they are**: Multiple time series representing bucketed observations. Metrics end with `_bucket`, `_sum`, `_count`.

**Best practices**:
- ✅ Use `histogram_quantile()` to calculate quantiles
- ✅ Always include `le` label in `by()` clause
- ✅ Use `rate()` on bucket metrics
- ✅ Aggregate before calculating quantiles
- ❌ Never average pre-calculated quantiles

**Examples**:
```promql
# Good: Calculate 95th percentile latency
histogram_quantile(0.95,
  sum by (job, le) (
    rate(http_request_duration_seconds_bucket{job="api"}[5m])
  )
)

# Good: Calculate average from histogram
rate(http_request_duration_seconds_sum{job="api"}[5m])
/
rate(http_request_duration_seconds_count{job="api"}[5m])

# Bad: Missing rate()
histogram_quantile(0.95, sum by (le) (http_request_duration_seconds_bucket))

# Bad: Missing 'le' in aggregation
histogram_quantile(0.95, sum by (job) (rate(http_request_duration_seconds_bucket[5m])))
```

### Summaries

**What they are**: Pre-calculated quantiles with `_sum` and `_count`. Includes labels like `quantile="0.95"`.

**Best practices**:
- ✅ Use `_sum` and `_count` to calculate averages
- ❌ Never average or aggregate pre-calculated quantiles
- ❌ Quantiles from summaries cannot be aggregated across instances

**Examples**:
```promql
# Good: Calculate average from summary
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

# Bad: Averaging quantiles (mathematically invalid!)
avg(http_request_duration_seconds{quantile="0.95"})
```

---

## Label Filtering

### Always Use Specific Label Filters

**Why**: Reduces cardinality, improves query performance, and makes intent clear.

```promql
# Bad: No filters
http_requests_total

# Good: Specific filters
http_requests_total{job="api-service", environment="production"}

# Good: Multiple filters for precision
http_requests_total{
  job="api-service",
  environment="production",
  datacenter="us-east-1",
  instance="prod-api-1"
}
```

### Use Exact Matches Over Regex When Possible

**Why**: Exact matches are faster (index lookups) vs regex (pattern matching).

```promql
# Bad: Regex for exact match
http_requests_total{status=~"200"}

# Good: Exact match
http_requests_total{status="200"}

# Regex is fine when you need it:
http_requests_total{status=~"2[0-9]{2}"}  # All 2xx status codes
```

### Efficient Regex Patterns

```promql
# Bad: Multiple OR queries
sum(http_requests_total{path="/api/users"})
or
sum(http_requests_total{path="/api/products"})
or
sum(http_requests_total{path="/api/orders"})

# Good: Single regex with alternation
sum by (path) (
  http_requests_total{path=~"/api/(users|products|orders)"}
)

# Good: Negative regex for exclusions
http_requests_total{path!~"/health|/metrics"}
```

### Label Matcher Operators

- `=` : Equal to
- `!=` : Not equal to
- `=~` : Regex match (fully anchored)
- `!~` : Regex does not match

---

## Aggregations

### Always Use `by()` or `without()` Clauses

**Why**: Makes output labels explicit and prevents confusion.

```promql
# Unclear: What labels will remain?
sum(rate(http_requests_total[5m]))

# Clear: Group by these labels
sum by (job, instance) (rate(http_requests_total[5m]))

# Clear: Remove only these labels
sum without (pod, container) (rate(http_requests_total[5m]))
```

### Use `without()` for High-Cardinality Labels

**Why**: More maintainable when you want to keep many labels.

```promql
# Verbose: List all labels to keep
sum by (job, instance, environment, datacenter, region, cluster, zone) (metric)

# Better: Drop only the high-cardinality labels
sum without (pod, container, node) (metric)
```

### Common Aggregation Operators

- `sum`: Total across series
- `avg`: Average value
- `min`: Minimum value
- `max`: Maximum value
- `count`: Count of series
- `stddev`: Standard deviation
- `stdvar`: Standard variance
- `topk(N, ...)`: Top N series
- `bottomk(N, ...)`: Bottom N series
- `quantile(φ, ...)`: φ-quantile (0 ≤ φ ≤ 1)

### Aggregation Examples

```promql
# Sum request rate per service
sum by (service) (rate(http_requests_total[5m]))

# Average CPU across all cores per node
avg by (instance) (rate(node_cpu_seconds_total[5m]))

# Top 10 pods by memory usage
topk(10, container_memory_usage_bytes)

# Count running instances per job
count by (job) (up == 1)
```

---

## Time Ranges

### rate() Range Selection

**Rule of thumb**: Use at least 4x your scrape interval.

- Typical scrape interval: 15s
- Minimum `rate()` range: `[1m]` (preferably `[2m]`)

```promql
# Bad: Too short (less than 4x scrape interval)
rate(http_requests_total[30s])

# Good: At least 2 minutes
rate(http_requests_total[2m])

# Common: 5 minutes (good balance of responsiveness and stability)
rate(http_requests_total[5m])

# Longer ranges: More stable, less sensitive to spikes
rate(http_requests_total[15m])
```

### irate() vs rate()

**irate()**: Instant rate, only uses last two samples.
- ✅ Use for high-frequency, short-range monitoring
- ✅ Good for rapidly changing metrics
- ✅ Range: `[2m]` to `[5m]` typically
- ❌ Don't use for long ranges (wasted range)

**rate()**: Average rate over entire range.
- ✅ Use for most cases
- ✅ More stable and accurate for longer ranges
- ✅ Better for alerting (less noisy)

```promql
# Good: irate with short range
irate(http_requests_total[2m])

# Good: rate for longer range
rate(http_requests_total[5m])

# Bad: irate with long range (only uses last 2 samples anyway!)
irate(http_requests_total[1h])
```

### Subqueries

**Syntax**: `query[range:resolution]`

**Use sparingly**: Subqueries can be very expensive.

```promql
# Calculate max rate over 30 minutes with 1-minute resolution
max_over_time(
  rate(http_requests_total[5m])[30m:1m]
)

# Bad: Excessive range
max_over_time(
  rate(http_requests_total[5m])[95d:1m]
)  # Processes millions of samples!

# Better: Use recording rules for long ranges
```

---

## Performance Optimization

### 1. Filter Early, Aggregate Late

```promql
# Good: Filter before expensive operations
sum(rate(http_requests_total{job="api", status="200"}[5m]))

# Bad: Filter after aggregation (processes more data)
sum(rate(http_requests_total[5m])) and {job="api", status="200"}
```

### 2. Use topk/bottomk to Limit Results

```promql
# Instead of processing all series:
sum by (pod) (rate(container_cpu_usage[5m]))

# Limit to top 10 in query:
topk(10, sum by (pod) (rate(container_cpu_usage[5m])))
```

### 3. Avoid High-Cardinality Labels

- User IDs, request IDs, timestamps as labels = BAD
- Job, instance, path, status code = OK
- Keep label cardinality under 10-100 unique values when possible

### 4. Use Recording Rules for Complex Queries

See [Recording Rules](#recording-rules) section below.

### 5. Minimize Regex Usage

```promql
# Slower: Regex match
{label=~"value"}

# Faster: Exact match
{label="value"}
```

### 6. Share Common Subexpressions

```promql
# Bad: Same rate calculated twice
rate(metric[5m]) / rate(metric[5m] offset 1h)

# Can't be optimized in PromQL directly, but use recording rules:
# - record: metric:rate5m
#   expr: rate(metric[5m])

# Then:
metric:rate5m / (metric:rate5m offset 1h)
```

---

## Recording Rules

**Purpose**: Pre-compute frequently-used or expensive queries.

**Benefits**:
- Faster dashboard loads
- Lower query latency
- Reduced Prometheus CPU usage
- Easier to maintain complex expressions

**When to use**:
- Query runs frequently (multiple dashboards, alerts)
- Query is computationally expensive
- Query spans long time ranges (subqueries)
- Query is complex (multiple aggregations, joins)

**Naming convention**:
```
level:metric:operations
```

Examples:
- `job:http_requests:rate5m`
- `instance:node_cpu:rate1m`
- `job_instance:request_latency_seconds:mean5m`

**Configuration example**:

```yaml
groups:
  - name: example_recording_rules
    interval: 30s
    rules:
      # Basic rate recording
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # Error rate recording
      - record: job:http_requests:error_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

      # Average latency recording
      - record: job:http_request_latency_seconds:mean5m
        expr: |
          sum by (job) (rate(http_request_duration_seconds_sum[5m]))
          /
          sum by (job) (rate(http_request_duration_seconds_count[5m]))
```

---

## Histograms and Summaries

### Histogram Best Practices

```promql
# Calculate quantile
histogram_quantile(0.95,
  sum by (le, job) (
    rate(http_request_duration_seconds_bucket{job="api"}[5m])
  )
)

# Always include 'le' in aggregation
sum by (job, le) (...)  # ✅ Correct
sum by (job) (...)      # ❌ Wrong - missing 'le'

# Use rate() on bucket metrics
rate(http_request_duration_seconds_bucket[5m])  # ✅ Correct
http_request_duration_seconds_bucket            # ❌ Wrong - missing rate()
```

### Calculate Average from Histogram

```promql
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])
```

### Count Observations

```promql
rate(http_request_duration_seconds_count[5m])
```

---

## Native Histograms (Prometheus 2.40+/3.0)

Native histograms are a newer histogram format introduced in Prometheus 2.40 and made stable in 3.0. They offer significant storage and query efficiency improvements over classic histograms.

### Key Differences from Classic Histograms

| Classic Histograms | Native Histograms |
|-------------------|-------------------|
| Separate `_bucket`, `_sum`, `_count` time series | Single time series containing all data |
| Fixed bucket boundaries defined at instrumentation | Dynamic bucket resolution |
| Requires `_bucket` suffix in queries | No `_bucket` suffix needed |
| Always need `le` label in aggregation | No `le` label manipulation |

### Native Histogram Query Syntax

```promql
# Classic histogram (old way)
histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))

# Native histogram (simpler - no _bucket suffix, no 'le' label needed)
histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[10m])))
```

### Native Histogram Functions

Prometheus provides special functions for native histograms:

```promql
# Calculate average from native histogram
histogram_avg(rate(http_request_duration_seconds[5m]))

# Calculate standard deviation
histogram_stddev(rate(http_request_duration_seconds[5m]))

# Calculate standard variance
histogram_stdvar(rate(http_request_duration_seconds[5m]))

# Get observation count
histogram_count(rate(http_request_duration_seconds[5m]))

# Get sum of observations
histogram_sum(rate(http_request_duration_seconds[5m]))

# Get fraction of observations in a range
histogram_fraction(0.1, 0.5, rate(http_request_duration_seconds[5m]))
```

### Best Practices for Native Histograms

1. **Still use `rate()` with native histograms** - The histogram functions work with rate-aggregated data
   ```promql
   # ✅ Correct
   histogram_avg(rate(http_request_duration_seconds[5m]))

   # ❌ Wrong - missing rate()
   histogram_avg(http_request_duration_seconds)
   ```

2. **Simpler aggregation** - No need for `le` label in `by()` clause
   ```promql
   # Classic histogram - need 'le'
   histogram_quantile(0.95, sum by (job, le) (rate(metric_bucket[5m])))

   # Native histogram - no 'le' needed
   histogram_quantile(0.95, sum by (job) (rate(metric[5m])))
   ```

3. **Enable native histograms in Prometheus** - Requires configuration:
   ```yaml
   # prometheus.yml
   global:
     scrape_native_histograms: true
   ```

4. **Check if metrics are native or classic** - Query the metric directly to see its format in the response

### When to Use Native Histograms

- ✅ New projects starting with Prometheus 2.40+
- ✅ High-cardinality histogram data (storage efficiency)
- ✅ When you need many quantile calculations (query efficiency)
- ❌ Legacy systems that don't support native histograms
- ❌ When you need exact bucket boundaries for compliance

---

## Alerting Queries

### Keep Alert Expressions Simple

```promql
# Bad: Complex alert expression
alert: HighErrorRate
expr: |
  (
    sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum by (job) (rate(http_requests_total[5m]))
  ) > 0.05

# Better: Use recording rule, simple alert
# Recording rule:
- record: job:http_requests:error_rate5m
  expr: ...

# Alert:
alert: HighErrorRate
expr: job:http_requests:error_rate5m > 0.05
```

### Use `for` Clause to Avoid Flapping

```yaml
- alert: HighMemoryUsage
  expr: node_memory_usage_percent > 90
  for: 5m  # Must be true for 5 minutes
  annotations:
    summary: "High memory usage on {{ $labels.instance }}"
```

### Alert on Rate of Change

```promql
# Alert if request rate drops suddenly
(
  rate(http_requests_total[5m])
  /
  rate(http_requests_total[5m] offset 1h)
) < 0.5  # Less than 50% of rate 1 hour ago
```

---

## Common Patterns

### Error Rate Calculation

```promql
# Error rate as percentage
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100
```

### Success Rate

```promql
# Success rate as percentage
(
  sum(rate(http_requests_total{status=~"2.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100
```

### Percentage Calculation

```promql
# Memory usage percentage
(
  node_memory_usage_bytes
  /
  node_memory_total_bytes
) * 100
```

### Comparing to Historical Baseline

```promql
# Compare current to 1 day ago
rate(http_requests_total[5m])
/
rate(http_requests_total[5m] offset 1d)
```

### Detect Sudden Spikes

```promql
# Alert if current rate > 2x the max rate in last hour
rate(metric[5m])
>
max_over_time(rate(metric[5m])[1h:]) * 2
```

### Absent Metrics (Alerting)

```promql
# Alert if metric disappears
absent(up{job="critical-service"})

# Alert if metric was present but now gone
absent_over_time(up{job="critical-service"}[5m])
```

### Joining Metrics

```promql
# Add labels from info metric to other metrics
rate(http_requests_total[5m])
* on (job, instance) group_left (version, commit)
service_info
```

---

## Quick Reference

| Pattern | Use Case | Example |
|---------|----------|---------|
| `rate(counter[5m])` | Per-second rate of counter | `rate(http_requests_total[5m])` |
| `increase(counter[1h])` | Total increase in counter | `increase(requests_total[1h])` |
| `gauge` | Current value | `node_memory_usage_bytes` |
| `avg_over_time(gauge[5m])` | Average gauge over time | `avg_over_time(cpu_percent[5m])` |
| `histogram_quantile(0.95, ...)` | Calculate percentile | See histogram section |
| `sum by (label) (...)` | Aggregate by labels | `sum by (job) (rate(metric[5m]))` |
| `topk(N, ...)` | Top N series | `topk(10, metric)` |
| `absent(metric)` | Check if metric missing | `absent(up{job="api"})` |
| `metric offset 1h` | Historical comparison | `rate(metric[5m] offset 1h)` |

---

## Additional Resources

- [Official Prometheus Querying Documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/)
- [PromQL Cheat Sheet](https://promlabs.com/promql-cheat-sheet/)
- [Best Practices for Naming Metrics](https://prometheus.io/docs/practices/naming/)

path: scripts/check_best_practices.py
content: #!/usr/bin/env python3
"""
PromQL Best Practices Checker

Detects anti-patterns, performance issues, and optimization opportunities in PromQL queries.
Provides actionable suggestions for improving query efficiency and correctness.
"""

import re
import sys
import json
from typing import Dict, List, Tuple, Optional


class PromQLBestPracticesChecker:
    """Checks PromQL queries for best practices and anti-patterns"""

    # Metric name patterns
    COUNTER_SUFFIXES = ['_total', '_count', '_sum', '_bucket']
    # Expanded gauge patterns based on common naming conventions
    # See: https://prometheus.io/docs/practices/naming/
    GAUGE_PATTERNS = [
        '_bytes',           # Memory, disk sizes (when not _bytes_total)
        '_ratio',           # Ratios like cache_hit_ratio
        '_usage',           # Resource usage metrics
        '_percent',         # Percentage values
        '_gauge',           # Explicitly named gauges
        '_celsius',         # Temperature metrics
        '_fahrenheit',      # Temperature metrics
        '_temperature',     # Temperature metrics
        '_info',            # Info metrics (always 1, with labels)
        '_size',            # Size measurements
        '_current',         # Current values (e.g., connections_current)
        '_limit',           # Limit values
        '_available',       # Available resources
        '_free',            # Free resources
        '_used',            # Used resources (when not a counter)
        '_utilization',     # Utilization percentages
        '_capacity',        # Capacity values
        '_level',           # Level measurements
    ]

    # Rate functions
    RATE_FUNCTIONS = ['rate', 'irate', 'increase', 'delta', 'idelta']

    # Native histogram functions (Prometheus 2.40+/3.0)
    NATIVE_HISTOGRAM_FUNCTIONS = [
        'histogram_avg', 'histogram_stddev', 'histogram_stdvar',
        'histogram_count', 'histogram_sum', 'histogram_fraction'
    ]

    def __init__(self, query: str):
        self.query = query.strip()
        self.issues: List[Dict] = []
        self.suggestions: List[Dict] = []
        self.optimizations: List[Dict] = []

    def check(self) -> Dict:
        """
        Run all best practice checks

        Returns:
            Dict containing check results
        """
        if not self.query:
            return self._build_result()

        # Check for anti-patterns
        self._check_high_cardinality()
        self._check_regex_overuse()
        self._check_missing_rate_on_counters()
        self._check_rate_on_gauges()
        self._check_averaging_quantiles()
        self._check_subquery_performance()
        self._check_irate_range()
        self._check_rate_range()
        self._check_unbounded_queries()
        self._check_aggregation_best_practices()
        self._check_recording_rule_opportunity()
        self._check_label_matcher_efficiency()
        self._check_histogram_usage()
        # Prometheus 3.0+ and additional checks
        self._check_deprecated_functions()
        self._check_predict_linear_range()
        self._check_division_by_zero_risk()
        self._check_changes_resets_alerting()
        self._check_dimensional_metric_names()
        # New checks based on documentation research
        self._check_absent_with_aggregation()
        self._check_vector_matching()
        self._check_native_histogram_usage()
        self._check_high_cardinality_labels_in_aggregation()
        # Design pattern checks
        self._check_mixed_metric_types()

        return self._build_result()

    def _check_high_cardinality(self):
        """Check for queries that might match too many time series"""
        # Check for metric selectors with no or very few label filters
        # Pattern: metric_name or metric_name{}
        if re.search(r'\b[a-zA-Z_:][a-zA-Z0-9_:]*\s*\{\s*\}', self.query):
            self.issues.append({
                'type': 'high_cardinality',
                'message': 'Query uses empty label matcher {} which may match many time series',
                'severity': 'warning',
                'recommendation': 'Add specific label filters like {job="...", instance="..."} to reduce cardinality'
            })

        # Check for bare metric names without selectors
        # First, remove content inside {...} blocks and strings to avoid matching label names
        query_without_selectors = self._strip_label_selectors_and_strings(self.query)

        metric_pattern = r'\b([a-zA-Z_:][a-zA-Z0-9_:]*)\b(?!\s*[{\(])'
        metrics_without_selectors = re.findall(metric_pattern, query_without_selectors)

        # Filter out function names, keywords, and PromQL reserved words
        reserved_words = {
            # Aggregation operators
            'sum', 'avg', 'min', 'max', 'count', 'stddev', 'stdvar', 'group',
            'topk', 'bottomk', 'quantile', 'count_values', 'limitk', 'limit_ratio',
            # Functions
            'rate', 'irate', 'increase', 'delta', 'idelta', 'deriv', 'predict_linear',
            'histogram_quantile', 'histogram_count', 'histogram_sum', 'histogram_fraction',
            'histogram_avg', 'histogram_stddev', 'histogram_stdvar',
            'abs', 'ceil', 'floor', 'round', 'sqrt', 'exp', 'ln', 'log2', 'log10',
            'sin', 'cos', 'tan', 'asin', 'acos', 'atan', 'sinh', 'cosh', 'tanh',
            'deg', 'rad', 'sgn', 'clamp', 'clamp_max', 'clamp_min', 'pi',
            'timestamp', 'time', 'minute', 'hour', 'day_of_month', 'day_of_week',
            'days_in_month', 'month', 'year',
            'label_replace', 'label_join', 'vector', 'scalar',
            'changes', 'resets', 'absent', 'absent_over_time', 'present_over_time',
            'avg_over_time', 'min_over_time', 'max_over_time', 'sum_over_time',
            'count_over_time', 'quantile_over_time', 'stddev_over_time', 'stdvar_over_time',
            'last_over_time', 'mad_over_time', 'sort', 'sort_desc', 'sort_by_label',
            'sort_by_label_desc', 'holt_winters', 'double_exponential_smoothing', 'info',
            # Prometheus 3.5+ experimental timestamp functions
            'ts_of_max_over_time', 'ts_of_min_over_time', 'ts_of_last_over_time',
            # Prometheus 3.7+ experimental functions
            'first_over_time', 'ts_of_first_over_time',
            # Keywords and operators
            'by', 'without', 'and', 'or', 'unless', 'on', 'ignoring',
            'group_left', 'group_right', 'bool', 'offset', 'start', 'end',
            # Constants
            'inf', 'nan'
        }

        for metric in metrics_without_selectors:
            if metric.lower() not in reserved_words:
                # Check if this metric has label filters in the ORIGINAL query
                # A metric with filters looks like: metric_name{label="value"}
                # We need to check if this metric is followed by a non-empty {...} block
                escaped_metric = re.escape(metric)
                has_filters = re.search(
                    rf'\b{escaped_metric}\s*\{{\s*[^}}]+\s*\}}',
                    self.query
                )
                if not has_filters:
                    self.issues.append({
                        'type': 'high_cardinality',
                        'message': f'Metric "{metric}" used without label filters',
                        'severity': 'warning',
                        'recommendation': f'Add label filters: {metric}{{job="...", instance="..."}}'
                    })

    def _strip_label_selectors_and_strings(self, query: str) -> str:
        """
        Remove content inside {...} label selectors, [...] range/subquery specifiers,
        quoted strings, and grouping clauses.
        This prevents label names and duration tokens from being misidentified as metric names.

        Strips content from:
        - [...] range vectors and subqueries (e.g. [5m], [7d:1h])
        - by (...) clauses
        - without (...) clauses
        - on (...) clauses
        - ignoring (...) clauses
        - group_left(...) clauses
        - group_right(...) clauses
        """
        # Strip range vector and subquery contents: [5m], [1h], [7d:1h], [30m:]
        # Duration tokens like ":1h" must not be matched as metric names.
        # Replace bracket contents with spaces to preserve string length/positions.
        query = re.sub(r'\[([^\]]*)\]', lambda m: '[' + ' ' * len(m.group(1)) + ']', query)

        # Strip grouping clauses (by, without, on, ignoring, group_left, group_right)
        # These contain label names, not metric names
        query = re.sub(r'\b(by|without|on|ignoring|group_left|group_right)\s*\([^)]*\)', r'\1 ( )', query)

        result = []
        depth = 0
        in_string = False
        escape_next = False
        i = 0

        while i < len(query):
            char = query[i]

            if escape_next:
                escape_next = False
                i += 1
                continue

            if char == '\\':
                escape_next = True
                i += 1
                continue

            if char == '"':
                in_string = not in_string
                i += 1
                continue

            if in_string:
                i += 1
                continue

            if char == '{':
                depth += 1
                result.append(' ')  # Replace with space to preserve word boundaries
                i += 1
                continue

            if char == '}':
                depth = max(0, depth - 1)
                result.append(' ')
                i += 1
                continue

            if depth == 0:
                result.append(char)
            else:
                # Inside {...}, replace with space to maintain positions
                result.append(' ')

            i += 1

        return ''.join(result)

    def _check_regex_overuse(self):
        """Check for regex matchers that could be exact matches"""
        # Find regex matchers =~ and !~
        regex_matchers = re.findall(r'([a-zA-Z_][a-zA-Z0-9_]*)\s*=~\s*"([^"]+)"', self.query)

        # Regex metacharacters that indicate an actual regex pattern (not just a literal string)
        # Note: . is a metacharacter meaning "any character", so "5.." is a regex pattern
        regex_metacharacters = r'[\.\*\+\?\^\$\[\]\(\)\|\\]'

        for label, pattern in regex_matchers:
            # Check if the pattern contains any regex metacharacters
            # If it does, it's a real regex and should NOT be converted to exact match
            has_regex_chars = re.search(regex_metacharacters, pattern)

            # Only suggest exact match if pattern is purely alphanumeric with underscores/hyphens
            # and contains NO regex metacharacters
            if not has_regex_chars and re.fullmatch(r'[a-zA-Z0-9_\-]+', pattern):
                self.optimizations.append({
                    'type': 'regex_to_exact',
                    'message': f'Label matcher {label}=~"{pattern}" can be an exact match',
                    'severity': 'info',
                    'recommendation': f'Use {label}="{pattern}" instead of =~ for better performance'
                })

            # Check for simple prefix/suffix patterns that might be better structured
            if pattern.endswith('.*'):
                self.suggestions.append({
                    'type': 'regex_optimization',
                    'message': f'Regex pattern "{pattern}" uses wildcard suffix',
                    'severity': 'info',
                    'recommendation': 'Consider if you can use more specific label values'
                })

    def _check_missing_rate_on_counters(self):
        """Check if counter metrics are used without rate/increase"""
        # Strip label selector contents and quoted strings before scanning for counter
        # metric names.  Without this, a counter-suffix name that appears inside a label
        # VALUE (e.g. {label="http_requests_total"}) would be misidentified as a bare
        # metric reference and produce a false-positive missing_rate warning.
        # _check_high_cardinality() already uses the same stripping approach.
        query_for_metric_scan = self._strip_label_selectors_and_strings(self.query)

        # Find metric names that look like counters
        metric_pattern = r'\b([a-zA-Z_:][a-zA-Z0-9_:]*(?:_total|_count|_sum|_bucket))\b'
        counter_metrics = re.findall(metric_pattern, query_for_metric_scan)

        for metric in counter_metrics:
            # Check if it's wrapped in rate/irate/increase
            escaped_metric = re.escape(metric)
            if not re.search(rf'(?:rate|irate|increase|delta|idelta)\s*\([^)]*{escaped_metric}', self.query):
                # Check if it's in histogram_quantile (buckets are used differently)
                if not re.search(rf'histogram_quantile\s*\([^)]*{escaped_metric}', self.query):
                    # Skip _sum and _count metrics when used in histogram calculations
                    # (they're used for average: _sum / _count)
                    if metric.endswith('_sum') or metric.endswith('_count'):
                        # Check if this is part of a division for average calculation
                        base_metric = metric.rsplit('_', 1)[0]
                        if re.search(rf'{base_metric}_sum.*{base_metric}_count|{base_metric}_count.*{base_metric}_sum', self.query):
                            continue  # Skip - this is a valid average calculation pattern

                    # Skip native histogram metrics (no _bucket suffix needed)
                    # Native histograms use histogram_avg, histogram_stddev, etc.
                    if re.search(rf'histogram_(?:avg|stddev|stdvar|count|sum|fraction)\s*\([^)]*{escaped_metric}', self.query):
                        continue

                    self.issues.append({
                        'type': 'missing_rate',
                        'message': f'Counter metric "{metric}" used without rate() or increase()',
                        'severity': 'warning',
                        'recommendation': f'Use rate({metric}[5m]) to get per-second rate'
                    })

    def _check_rate_on_gauges(self):
        """Check if rate/irate is used on gauge metrics"""
        # Find rate/irate/increase calls
        rate_calls = re.findall(
            r'(rate|irate|increase|delta|idelta)\s*\(\s*([a-zA-Z_:][a-zA-Z0-9_:]*)',
            self.query
        )

        for func, metric in rate_calls:
            # Check if metric name suggests it's a gauge
            is_gauge = any(pattern in metric for pattern in self.GAUGE_PATTERNS)
            is_counter = any(metric.endswith(suffix) for suffix in self.COUNTER_SUFFIXES)

            if is_gauge and not is_counter:
                self.issues.append({
                    'type': 'rate_on_gauge',
                    'message': f'{func}() used on gauge metric "{metric}"',
                    'severity': 'warning',
                    'recommendation': f'Gauges should not use rate(). Use avg_over_time({metric}[5m]) or remove rate()'
                })

    def _check_averaging_quantiles(self):
        """Check for aggregating pre-calculated quantiles with invalid operations.

        avg() is the most obviously wrong case, but sum(), max(), and min() on
        Prometheus summary quantile labels are equally invalid:
        - sum(p95_from_instance_A, p95_from_instance_B) is not a meaningful p95
        - max() gives the worst-case instance p95 but is often misread as "global p95"
        - min() has the same confusion

        The only correct approach is to calculate quantiles from histogram buckets
        using histogram_quantile().
        """
        # Matches any aggregation whose argument contains a {quantile="..."} selector.
        # The selector unambiguously identifies Prometheus summary quantile label usage.
        invalid_aggregations = r'(?:avg|sum|min|max|stddev|stdvar)\s*\([^)]*\{[^}]*quantile\s*='
        match = re.search(invalid_aggregations, self.query)
        if match:
            # Extract which aggregation was used for a clearer error message.
            agg_func = match.group(0).split('(')[0].strip()
            self.issues.append({
                'type': 'averaging_quantiles',
                'message': f'{agg_func}() on pre-calculated quantile labels produces mathematically invalid results',
                'severity': 'error',
                'recommendation': 'Use histogram_quantile() with histogram buckets instead: histogram_quantile(0.95, sum by (le) (rate(metric_bucket[5m])))'
            })

    def _check_subquery_performance(self):
        """Check for potentially expensive subqueries"""
        # Pattern: [...:...] subquery syntax
        subquery_pattern = r'\[(\d+)([smhdwy])[^\]]*:\s*(\d+)?([smhdwy])?\]'
        subqueries = re.findall(subquery_pattern, self.query)

        for range_val, range_unit, res_val, res_unit in subqueries:
            # Convert to approximate hours
            range_hours = self._duration_to_hours(int(range_val), range_unit)

            if range_hours > 24 * 7:  # More than 7 days
                self.issues.append({
                    'type': 'expensive_subquery',
                    'message': f'Subquery spans {range_val}{range_unit} which may be very slow',
                    'severity': 'warning',
                    'recommendation': 'Consider using recording rules or reducing the time range'
                })

    def _check_irate_range(self):
        """Check if irate() is used with appropriate time ranges"""
        # irate() should use short ranges (typically < 5m)
        irate_pattern = r'irate\s*\([^)]*\[(\d+)([smhdwy])\]'
        irate_calls = re.findall(irate_pattern, self.query)

        for duration, unit in irate_calls:
            minutes = self._duration_to_minutes(int(duration), unit)

            if minutes > 5:
                self.issues.append({
                    'type': 'irate_long_range',
                    'message': f'irate() used with {duration}{unit} range - irate only looks at last 2 samples',
                    'severity': 'warning',
                    'recommendation': f'Use rate() for ranges > 5m, or reduce irate range to [2m]'
                })

    def _check_rate_range(self):
        """Check if rate() uses appropriate time ranges"""
        # rate() range should be at least 4x scrape interval (typically >= 2m)
        rate_pattern = r'rate\s*\([^)]*\[(\d+)(ms|s|m|h|d|w|y)\]'
        rate_calls = re.findall(rate_pattern, self.query)

        for duration, unit in rate_calls:
            seconds = self._duration_to_seconds(int(duration), unit)

            if seconds < 120:  # Less than 2 minutes
                self.issues.append({
                    'type': 'rate_short_range',
                    'message': f'rate() used with very short range [{duration}{unit}]',
                    'severity': 'warning',
                    'recommendation': 'Rate range should be at least 4x scrape interval, typically [2m] or more'
                })

    def _check_unbounded_queries(self):
        """Check for queries without sufficient constraints"""
        # Look for aggregations without by/without clauses on potentially high-cardinality data
        aggregations = ['sum', 'avg', 'min', 'max', 'count']

        # Check if this appears to be an alerting query (has comparison operator)
        # For alerting, fully aggregated results returning a single value is often intentional
        is_alerting_query = bool(re.search(r'\s*(>|<|>=|<=|==|!=)\s*[\d\.]', self.query))

        for agg in aggregations:
            # Pattern: sum(...) without "by" or "without"
            pattern = rf'{agg}\s*\([^)]+\)(?!\s*(?:by|without)\s*\()'
            if re.search(pattern, self.query):
                if is_alerting_query:
                    # For alerting queries, this is often intentional - use a softer message
                    self.suggestions.append({
                        'type': 'missing_aggregation_clause',
                        'message': f'{agg}() used without by() or without() clause (likely intentional for alerting)',
                        'severity': 'info',
                        'recommendation': f'Full aggregation is common for alerting queries. Add "by (label)" only if you need per-label alerts.'
                    })
                else:
                    # For non-alerting queries, the standard recommendation applies
                    self.suggestions.append({
                        'type': 'missing_aggregation_clause',
                        'message': f'{agg}() used without by() or without() clause',
                        'severity': 'info',
                        'recommendation': f'Consider adding "by (label)" or "without (label)" to {agg}() for clearer results'
                    })

    def _check_aggregation_best_practices(self):
        """Check aggregation operator usage"""
        # Check for count() without by clause (might be intentional, but worth mentioning)
        if re.search(r'count\s*\([^)]+\)(?!\s*by)', self.query):
            self.suggestions.append({
                'type': 'count_without_by',
                'message': 'count() used without by() - this counts all matching series',
                'severity': 'info',
                'recommendation': 'If you want to count by label, use: count(...) by (label)'
            })

    def _check_recording_rule_opportunity(self):
        """Check if query is complex enough to benefit from recording rules"""
        # Heuristics for complex queries:
        # - Multiple nested functions
        # - Multiple aggregations
        # - Subqueries
        # - Long expressions

        complexity_score = 0

        # Count function calls
        func_count = len(re.findall(r'\b[a-z_]+\s*\(', self.query))
        if func_count >= 3:
            complexity_score += 1

        # Check for nested aggregations
        if re.search(r'(sum|avg|min|max)\s*\([^)]*\b(sum|avg|min|max)\s*\(', self.query):
            complexity_score += 1

        # Check for subqueries
        if re.search(r'\[[^\]]+:[^\]]+\]', self.query):
            complexity_score += 1

        # Check query length
        if len(self.query) > 150:
            complexity_score += 1

        if complexity_score >= 2:
            self.suggestions.append({
                'type': 'recording_rule_opportunity',
                'message': 'Query is complex and may benefit from recording rules',
                'severity': 'info',
                'recommendation': 'Consider creating recording rules if this query is used frequently'
            })

    def _check_label_matcher_efficiency(self):
        """Check if label matchers could be more efficient"""
        # Check for multiple OR conditions that might indicate need for regex
        if self.query.count(' or ') >= 2:
            self.suggestions.append({
                'type': 'multiple_or_conditions',
                'message': 'Multiple OR conditions found',
                'severity': 'info',
                'recommendation': 'Consider using regex matcher =~ "value1|value2|value3" if checking same label'
            })

    def _check_histogram_usage(self):
        """Check for proper histogram quantile calculation"""
        # Check for histogram_quantile usage
        if 'histogram_quantile' in self.query:
            # Should include rate() on bucket metrics
            # Note: The rate() can be nested inside aggregations, so we look for both
            # histogram_quantile and rate() appearing anywhere in the query
            has_rate = bool(re.search(r'\brate\s*\(', self.query))
            has_bucket_metric = '_bucket' in self.query

            # Only warn about missing rate if there's a _bucket metric (classic histogram)
            # Native histograms don't have _bucket suffix
            if has_bucket_metric and not has_rate:
                self.issues.append({
                    'type': 'histogram_missing_rate',
                    'message': 'histogram_quantile() should use rate() on bucket metrics',
                    'severity': 'warning',
                    'recommendation': 'Use: histogram_quantile(0.95, sum by (le) (rate(metric_bucket[5m])))'
                })

            # Should aggregate by 'le' label (only for classic histograms with _bucket)
            # Native histograms don't need 'le' label
            if has_bucket_metric:
                # Look for 'le' in any by() clause in the query
                has_le_in_by = bool(re.search(r'\bby\s*\([^)]*\ble\b', self.query))
                if not has_le_in_by:
                    self.issues.append({
                        'type': 'histogram_missing_le',
                        'message': 'histogram_quantile() with classic histograms should aggregate by (le) label',
                        'severity': 'warning',
                        'recommendation': 'Include "le" in the by() clause: sum by (job, le) (...)'
                    })

    def _check_deprecated_functions(self):
        """Check for deprecated functions (Prometheus 3.0+)"""
        # holt_winters is deprecated in Prometheus 3.0, renamed to double_exponential_smoothing
        if re.search(r'\bholt_winters\s*\(', self.query):
            self.issues.append({
                'type': 'deprecated_function',
                'message': 'holt_winters() is deprecated in Prometheus 3.0',
                'severity': 'warning',
                'recommendation': 'Use double_exponential_smoothing() instead (requires --enable-feature=promql-experimental-functions)'
            })

    def _check_predict_linear_range(self):
        """Check if predict_linear() uses appropriate time ranges"""
        # predict_linear() with very short ranges gives unreliable predictions
        predict_pattern = r'predict_linear\s*\([^)]*\[(\d+)(ms|s|m|h|d|w|y)\]'
        predict_calls = re.findall(predict_pattern, self.query)

        for duration, unit in predict_calls:
            minutes = self._duration_to_minutes(int(duration), unit)

            if minutes < 10:
                self.issues.append({
                    'type': 'predict_linear_short_range',
                    'message': f'predict_linear() used with short range [{duration}{unit}]',
                    'severity': 'warning',
                    'recommendation': 'predict_linear() needs sufficient data for reliable predictions. Use at least [10m] or longer.'
                })

    def _check_division_by_zero_risk(self):
        """Check for potential division by zero issues"""
        # Pattern: / rate(..._count...) or / rate(..._total...)
        # This can be zero if no requests occurred
        if re.search(r'/\s*(?:rate|increase)\s*\([^)]*(?:_count|_total)[^)]*\)', self.query):
            self.suggestions.append({
                'type': 'division_by_zero_risk',
                'message': 'Division by rate() or increase() of counter may result in NaN if denominator is 0',
                'severity': 'info',
                'recommendation': 'Consider using "or vector(0)" or "> 0" filter to handle zero denominators'
            })

    def _check_changes_resets_alerting(self):
        """Check for changes() or resets() usage patterns"""
        # changes() and resets() can miss events that happen between scrapes
        if re.search(r'\b(changes|resets)\s*\(', self.query):
            self.suggestions.append({
                'type': 'changes_resets_limitation',
                'message': 'changes() and resets() only detect changes between scraped samples',
                'severity': 'info',
                'recommendation': 'Events occurring between scrapes will be missed. For alerting, consider alternative approaches.'
            })

    def _check_dimensional_metric_names(self):
        """Check for dimensions embedded in metric names (anti-pattern)"""
        # Embedding dimensions in metric names like: http_requests_GET_total, cpu_0_usage
        # This is a bad practice per Google Cloud and Prometheus best practices

        # Look for patterns like: metric_value_total or metric_123_something
        # Common bad patterns: http_requests_GET_200_total, cpu_core0_usage
        bad_patterns = [
            r'\b[a-zA-Z_]+_(GET|POST|PUT|DELETE|PATCH)_[a-zA-Z_]+',  # HTTP methods in name
            r'\b[a-zA-Z_]+_\d+_[a-zA-Z_]+',  # Numbers embedded (like cpu_0_usage)
            r'\b[a-zA-Z_]+_(2\d{2}|3\d{2}|4\d{2}|5\d{2})_[a-zA-Z_]+',  # HTTP status codes in name
        ]

        for pattern in bad_patterns:
            if re.search(pattern, self.query):
                self.suggestions.append({
                    'type': 'dimensional_metric_name',
                    'message': 'Metric name appears to embed dimensions (method, status code, or index)',
                    'severity': 'info',
                    'recommendation': 'Move dimensions to labels instead of embedding in metric names. Example: http_requests_total{method="GET", status="200"}'
                })
                break  # Only warn once

    def _check_absent_with_aggregation(self):
        """
        Check for absent() used with aggregation functions.

        Per https://stackoverflow.com/questions/53191746/prometheus-absent-function
        and https://www.robustperception.io/functions-to-avoid/

        absent() returns 1 if no time series match the selector, 0 otherwise.
        When combined with aggregation, it doesn't work as expected because:
        - absent(sum(metric)) will return empty if ANY metric matches
        - It cannot detect per-label absence

        For label-aware absence detection, use:
        group(present_over_time(metric[range])) by (labels)
        unless
        group(metric) by (labels)
        """
        # Pattern: absent(aggregation_function(...))
        if re.search(r'absent\s*\(\s*(sum|avg|min|max|count|group|stddev|stdvar)\s*\(', self.query):
            self.issues.append({
                'type': 'absent_with_aggregation',
                'message': 'absent() with aggregation may not work as expected',
                'severity': 'warning',
                'recommendation': 'absent() checks if a selector returns no data. Aggregations return data if ANY series matches. For per-label absence, use: group(present_over_time(metric[range])) unless group(metric)'
            })

        # Pattern: absent(...) by (label) - absent doesn't support by()
        if re.search(r'absent\s*\([^)]+\)\s*by\s*\(', self.query):
            self.issues.append({
                'type': 'absent_with_by',
                'message': 'absent() does not support by() clause for per-label grouping',
                'severity': 'error',
                'recommendation': 'absent() returns a single value. For per-label absence detection, use: group(present_over_time(metric[range])) by (labels) unless group(metric) by (labels)'
            })

    def _check_vector_matching(self):
        """
        Check for common vector matching mistakes with on/ignoring/group_left/group_right.

        Per https://grafana.com/blog/2024/12/13/promql-vector-matching-what-it-is-and-how-it-affects-your-prometheus-queries/
        and https://iximiuz.com/en/posts/prometheus-vector-matching/

        Common issues:
        1. Missing group_left/group_right for many-to-one joins
        2. Using group_right when group_left should be used
        3. Missing on() or ignoring() when label sets don't match
        """
        query_lower = self.query.lower()

        # Check for binary operations that might need vector matching
        # Pattern: metric * metric or metric / metric without on() or ignoring()
        binary_ops = ['*', '/', '+', '-', '%', '^']
        has_binary_op = any(op in self.query for op in binary_ops)
        has_vector_matching = 'on(' in query_lower or 'ignoring(' in query_lower

        # Check for _info metric joins (common pattern)
        # Info metrics typically need group_left
        if re.search(r'\*\s*on\s*\([^)]+\)\s*[a-zA-Z_]+_info\b', self.query):
            if 'group_left' not in query_lower and 'group_right' not in query_lower:
                self.issues.append({
                    'type': 'info_metric_missing_group',
                    'message': 'Joining with _info metric without group_left()',
                    'severity': 'warning',
                    'recommendation': 'Info metric joins typically need group_left() to bring labels from the info metric. Use: metric * on(job, instance) group_left(label1, label2) info_metric'
                })

        # Check for group_left/group_right without on() or ignoring()
        if re.search(r'\b(group_left|group_right)\s*\(', query_lower):
            if 'on(' not in query_lower and 'ignoring(' not in query_lower:
                self.issues.append({
                    'type': 'group_without_matching',
                    'message': 'group_left()/group_right() used without on() or ignoring()',
                    'severity': 'error',
                    'recommendation': 'group_left()/group_right() requires on() or ignoring() to specify matching labels'
                })

        # Check for on() with empty parentheses - this is valid but might be unintentional
        if re.search(r'\bon\s*\(\s*\)', query_lower):
            self.suggestions.append({
                'type': 'on_empty_labels',
                'message': 'on() with empty labels matches all series',
                'severity': 'info',
                'recommendation': 'on() with empty parentheses ignores all labels for matching. Ensure this is intentional.'
            })

        # Check for potential many-to-many matching (error-prone)
        # If there's a binary op with on() but no group_left/group_right, it might fail at runtime
        if has_vector_matching and 'group_left' not in query_lower and 'group_right' not in query_lower:
            # This is just informational since one-to-one might be intended
            self.suggestions.append({
                'type': 'vector_matching_cardinality',
                'message': 'Binary operation with on()/ignoring() assumes one-to-one matching',
                'severity': 'info',
                'recommendation': 'If you have many-to-one or one-to-many cardinality, add group_left() or group_right(). Error "multiple matches for labels" indicates cardinality mismatch.'
            })

    def _check_native_histogram_usage(self):
        """
        Check for proper native histogram function usage (Prometheus 2.40+/3.0).

        Per https://prometheus.io/docs/specs/native_histograms/
        and https://prometheus.io/blog/2024/11/14/prometheus-3-0/

        Native histograms:
        - Don't need _bucket suffix
        - Don't need 'le' label in aggregation
        - Still need rate() for proper calculation
        - Use histogram_avg, histogram_stddev, etc.
        """
        # Check for native histogram functions
        native_hist_pattern = r'\b(histogram_avg|histogram_stddev|histogram_stdvar)\s*\('
        native_hist_matches = re.findall(native_hist_pattern, self.query)

        for func in native_hist_matches:
            # Check if rate() is used (required for native histograms too)
            func_call_pattern = rf'{func}\s*\([^)]*'
            if not re.search(rf'{func}\s*\(\s*rate\s*\(', self.query):
                self.issues.append({
                    'type': 'native_histogram_missing_rate',
                    'message': f'{func}() should use rate() on the histogram metric',
                    'severity': 'warning',
                    'recommendation': f'Use: {func}(rate(histogram_metric[5m]))'
                })

        # Check for histogram_quantile with native histogram patterns
        # Native histograms don't need 'le' in by() clause
        if 'histogram_quantile' in self.query:
            # Check if this looks like a native histogram query (no _bucket suffix)
            # Native histogram: histogram_quantile(0.95, sum by (job) (rate(metric[5m])))
            # Classic histogram: histogram_quantile(0.95, sum by (job, le) (rate(metric_bucket[5m])))

            # Look for _bucket anywhere in the query (not just immediately after histogram_quantile)
            # since the bucket metric could be inside nested functions
            has_bucket_suffix = '_bucket' in self.query
            has_le_in_by = bool(re.search(r'\bby\s*\([^)]*\ble\b', self.query))

            # Only warn about unnecessary 'le' for native histograms (no _bucket suffix)
            # If there's a _bucket metric, this is a classic histogram and 'le' IS required
            if not has_bucket_suffix and has_le_in_by:
                self.suggestions.append({
                    'type': 'native_histogram_unnecessary_le',
                    'message': 'Native histograms do not need "le" label in aggregation',
                    'severity': 'info',
                    'recommendation': 'For native histograms, simplify to: histogram_quantile(0.95, sum by (job) (rate(metric[5m])))'
                })

            # If has _bucket but no le - classic histogram missing le (already covered by _check_histogram_usage)
            # If no _bucket and no le - could be native histogram (OK) or classic without le (error)

        # Provide helpful info about histogram_count and histogram_sum
        # These work with both native and classic histograms but differently
        if re.search(r'\bhistogram_count\s*\(', self.query) or re.search(r'\bhistogram_sum\s*\(', self.query):
            # Check if it's wrapping rate()
            if not re.search(r'histogram_(?:count|sum)\s*\(\s*rate\s*\(', self.query):
                self.suggestions.append({
                    'type': 'histogram_helper_without_rate',
                    'message': 'histogram_count()/histogram_sum() typically need rate() for meaningful results',
                    'severity': 'info',
                    'recommendation': 'Use: histogram_count(rate(histogram_metric[5m])) to get observations per second'
                })

    def _check_high_cardinality_labels_in_aggregation(self):
        """
        Check for known high-cardinality label names inside aggregation group labels.

        Per best_practices.md: Labels like user_id, session_id, request_id, IP addresses,
        full URLs, and UUIDs create one series per unique value, which can be millions of
        series. They should not appear in by(...) dimensions.

        Important semantic note:
        - by(...) keeps the listed labels in the grouping key (risky for high-cardinality).
        - without(...) removes the listed labels from the grouping key (often the opposite
          of the risk we're checking), so we intentionally do not warn on those labels.
        """
        # Extract aggregation grouping clauses with their mode (by|without)
        agg_clause_pattern = r'\b(by|without)\s*\(([^)]*)\)'
        clauses = re.findall(agg_clause_pattern, self.query)

        # High-cardinality label name indicators (from best_practices.md)
        # These are either exact known names or suffix patterns
        HIGH_CARDINALITY_EXACT = {
            'ip', 'url', 'path', 'timestamp', 'uuid', 'tid',
        }
        HIGH_CARDINALITY_SUFFIXES = (
            '_id',      # user_id, session_id, request_id, trace_id, span_id, etc.
            '_uuid',    # any_uuid
            '_ip',      # client_ip, source_ip, etc.
            '_url',     # full_url, request_url, etc.
            '_address', # ip_address, email_address (high cardinality)
        )

        found = []
        for mode, clause in clauses:
            if mode.lower() != 'by':
                continue

            # Split on commas and strip whitespace to get individual label names
            label_names = [lbl.strip() for lbl in clause.split(',') if lbl.strip()]
            for label in label_names:
                lower = label.lower()
                if lower in HIGH_CARDINALITY_EXACT:
                    found.append(label)
                elif any(lower.endswith(suffix) for suffix in HIGH_CARDINALITY_SUFFIXES):
                    found.append(label)

        if found:
            label_list = ', '.join(dict.fromkeys(found))  # deduplicate, preserve order
            self.issues.append({
                'type': 'high_cardinality_aggregation_label',
                'message': f'by(...) includes high-cardinality label(s): {label_list}',
                'severity': 'warning',
                'recommendation': (
                    f'Labels like {label_list} can have millions of unique values, '
                    'creating one time series per value and degrading query performance. '
                    'Remove them from by() or replace with lower-cardinality alternatives '
                    '(e.g. use "service" instead of "user_id").'
                )
            })

    def _check_mixed_metric_types(self):
        """
        Check if query combines fundamentally different metric types in a single expression.

        Mixing counters, gauges, histograms, and summaries in arithmetic operations
        often produces meaningless results. Each metric type has different semantics:
        - Counters: Cumulative values that only increase
        - Gauges: Point-in-time values that can go up or down
        - Histograms: Bucketed observations for distribution analysis
        - Summaries: Pre-calculated quantiles

        Combining them (e.g., latency / memory + request_count) rarely makes sense.

        EXCEPTION: histogram_quantile with _bucket metrics is NOT mixed types - this is
        the correct pattern for classic histograms. The _bucket suffix doesn't indicate
        a counter being used incorrectly; it's part of the histogram data model.
        """
        # Detect metric types in the query
        detected_types = set()
        type_examples = {}

        # Check if this is a histogram_quantile query with classic histograms
        # In this case, _bucket metrics are expected and should not be flagged as "counter"
        is_classic_histogram_query = 'histogram_quantile' in self.query and '_bucket' in self.query

        # Find all metric names in the query (outside of {...} blocks)
        query_clean = self._strip_label_selectors_and_strings(self.query)
        metric_pattern = r'\b([a-zA-Z_:][a-zA-Z0-9_:]*)\b'
        potential_metrics = re.findall(metric_pattern, query_clean)

        # Filter out reserved words
        reserved_words = {
            'sum', 'avg', 'min', 'max', 'count', 'stddev', 'stdvar', 'group',
            'topk', 'bottomk', 'quantile', 'count_values', 'limitk', 'limit_ratio',
            'rate', 'irate', 'increase', 'delta', 'idelta', 'deriv', 'predict_linear',
            'histogram_quantile', 'histogram_count', 'histogram_sum', 'histogram_fraction',
            'histogram_avg', 'histogram_stddev', 'histogram_stdvar',
            'abs', 'ceil', 'floor', 'round', 'sqrt', 'exp', 'ln', 'log2', 'log10',
            'sin', 'cos', 'tan', 'asin', 'acos', 'atan', 'sinh', 'cosh', 'tanh',
            'deg', 'rad', 'sgn', 'clamp', 'clamp_max', 'clamp_min', 'pi',
            'timestamp', 'time', 'minute', 'hour', 'day_of_month', 'day_of_week',
            'days_in_month', 'month', 'year',
            'label_replace', 'label_join', 'vector', 'scalar',
            'changes', 'resets', 'absent', 'absent_over_time', 'present_over_time',
            'avg_over_time', 'min_over_time', 'max_over_time', 'sum_over_time',
            'count_over_time', 'quantile_over_time', 'stddev_over_time', 'stdvar_over_time',
            'last_over_time', 'mad_over_time', 'sort', 'sort_desc', 'sort_by_label',
            'sort_by_label_desc', 'holt_winters', 'double_exponential_smoothing', 'info',
            # Prometheus 3.5+ experimental timestamp functions
            'ts_of_max_over_time', 'ts_of_min_over_time', 'ts_of_last_over_time',
            # Prometheus 3.7+ experimental functions
            'first_over_time', 'ts_of_first_over_time',
            'by', 'without', 'and', 'or', 'unless', 'on', 'ignoring',
            'group_left', 'group_right', 'bool', 'offset', 'start', 'end',
            'inf', 'nan'
        }

        metrics = [m for m in potential_metrics if m.lower() not in reserved_words]

        for metric in metrics:
            # Classify metric type
            # For _bucket metrics in histogram_quantile queries, treat them as histogram components
            # not as standalone counters
            if metric.endswith('_bucket') and is_classic_histogram_query:
                # This is part of a classic histogram query - don't flag as counter
                continue
            elif any(metric.endswith(suffix) for suffix in ['_total', '_count', '_sum', '_bucket']):
                detected_types.add('counter')
                type_examples.setdefault('counter', []).append(metric)
            elif any(pattern in metric for pattern in self.GAUGE_PATTERNS):
                detected_types.add('gauge')
                type_examples.setdefault('gauge', []).append(metric)
            elif 'quantile' in self.query and metric in self.query:
                # Check if this metric is used with a quantile label selector
                if re.search(rf'{re.escape(metric)}\s*\{{[^}}]*quantile\s*=', self.query):
                    detected_types.add('summary')
                    type_examples.setdefault('summary', []).append(metric)

        # Check for histogram usage via histogram_quantile
        # Only add 'histogram' as a type if it's NOT a classic histogram query
        # (where _bucket metrics are expected and handled above)
        if 'histogram_quantile' in self.query and not is_classic_histogram_query:
            detected_types.add('histogram')
            type_examples.setdefault('histogram', []).append('(histogram_quantile usage)')

        # Warn if multiple different metric types are combined with arithmetic
        if len(detected_types) >= 2:
            # Check if there are arithmetic operators combining these
            arithmetic_ops = ['+', '-', '*', '/']
            has_arithmetic = any(op in self.query for op in arithmetic_ops)

            if has_arithmetic:
                type_list = ', '.join(sorted(detected_types))
                examples = []
                for t, metrics_list in type_examples.items():
                    examples.append(f"{t}: {metrics_list[0]}")

                self.issues.append({
                    'type': 'mixed_metric_types',
                    'message': f'Query combines different metric types ({type_list}) in arithmetic operations',
                    'severity': 'warning',
                    'recommendation': f'Mixing metric types often produces meaningless results. Examples found: {"; ".join(examples)}. Consider separating into distinct queries or ensure the combination makes semantic sense.'
                })

    @staticmethod
    def _duration_to_seconds(value: int, unit: str) -> int:
        """Convert duration to seconds"""
        units = {
            'ms': 0.001,
            's': 1,
            'm': 60,
            'h': 3600,
            'd': 86400,
            'w': 604800,
            'y': 31536000
        }
        return int(value * units.get(unit, 1))

    @staticmethod
    def _duration_to_minutes(value: int, unit: str) -> float:
        """Convert duration to minutes"""
        return PromQLBestPracticesChecker._duration_to_seconds(value, unit) / 60

    @staticmethod
    def _duration_to_hours(value: int, unit: str) -> float:
        """Convert duration to hours"""
        return PromQLBestPracticesChecker._duration_to_seconds(value, unit) / 3600

    def _build_result(self) -> Dict:
        """Build the check result dictionary"""
        all_findings = self.issues + self.suggestions + self.optimizations

        has_errors = any(item['severity'] == 'error' for item in all_findings)
        has_warnings = any(item['severity'] == 'warning' for item in all_findings)

        if has_errors:
            status = 'ERROR'
        elif has_warnings:
            status = 'WARNING'
        elif self.optimizations or self.suggestions:
            status = 'CAN_BE_IMPROVED'
        else:
            status = 'OPTIMIZED'

        return {
            'status': status,
            'query': self.query,
            'issues': self.issues,
            'suggestions': self.suggestions,
            'optimizations': self.optimizations,
            'summary': {
                'errors': len([i for i in self.issues if i['severity'] == 'error']),
                'warnings': len([i for i in self.issues if i['severity'] == 'warning']),
                'suggestions': len(self.suggestions),
                'optimizations': len(self.optimizations)
            }
        }


def main():
    """Main entry point for the best practices checker"""
    if len(sys.argv) < 2:
        print(json.dumps({
            'status': 'ERROR',
            'message': 'Usage: check_best_practices.py "<promql_query>"'
        }, indent=2))
        sys.exit(1)

    query = sys.argv[1]
    checker = PromQLBestPracticesChecker(query)
    result = checker.check()

    print(json.dumps(result, indent=2))

    # Exit with error code if there are errors
    sys.exit(0 if result['summary']['errors'] == 0 else 1)


if __name__ == '__main__':
    main()


path: scripts/test_validators.py
content: #!/usr/bin/env python3
"""
Test suite for validate_syntax.py and check_best_practices.py.

Run from the repository root:
    python3 devops-skills-plugin/skills/promql-validator/scripts/test_validators.py

Or directly from the scripts/ directory:
    python3 test_validators.py
"""

import json
import subprocess
import sys
import os
import unittest

# Allow running from either repo root or scripts/ directory.
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, _SCRIPT_DIR)

from validate_syntax import PromQLSyntaxValidator
from check_best_practices import PromQLBestPracticesChecker


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def syntax(query):
    return PromQLSyntaxValidator(query).validate()


def best(query):
    return PromQLBestPracticesChecker(query).check()


def best_cli(query):
    """Run check_best_practices.py as a CLI and return (process, parsed_json)."""
    checker = os.path.join(_SCRIPT_DIR, 'check_best_practices.py')
    proc = subprocess.run(
        [sys.executable, checker, query],
        capture_output=True,
        text=True,
        check=False,
    )
    return proc, json.loads(proc.stdout)


def error_types(result):
    return [e['type'] for e in result['errors']]


def warning_types(result):
    return [w['type'] for w in result['warnings']]


def issue_types(result):
    return [i['type'] for i in result['issues']]


def optimization_types(result):
    return [o['type'] for o in result['optimizations']]


# ===========================================================================
# validate_syntax.py
# ===========================================================================

class TestSyntaxValid(unittest.TestCase):
    """Queries that must pass validation with no errors."""

    def test_simple_rate(self):
        r = syntax('rate(http_requests_total[5m])')
        self.assertTrue(r['valid'])
        self.assertEqual(r['status'], 'VALID')

    def test_rate_with_label_filter(self):
        r = syntax('rate(http_requests_total{job="api", status="200"}[2m])')
        self.assertTrue(r['valid'])

    def test_histogram_quantile(self):
        q = 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))'
        self.assertTrue(syntax(q)['valid'])

    def test_subquery(self):
        r = syntax('max_over_time(rate(http_requests_total[5m])[30m:1m])')
        self.assertTrue(r['valid'])

    def test_offset_after_range(self):
        r = syntax('rate(http_requests_total[5m] offset 1h)')
        self.assertTrue(r['valid'])

    def test_offset_instant_vector(self):
        r = syntax('http_requests_total offset 1h')
        self.assertTrue(r['valid'])

    def test_aggregation_with_by(self):
        r = syntax('sum by (job) (rate(http_requests_total[5m]))')
        self.assertTrue(r['valid'])

    def test_predict_linear(self):
        r = syntax('predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600)')
        self.assertTrue(r['valid'])

    def test_label_replace(self):
        r = syntax('label_replace(up{job="node-exporter"}, "instance", "$1", "instance", "([^:]+):.*")')
        self.assertTrue(r['valid'])

    def test_utf8_metric_with_dots(self):
        r = syntax('{"my.metric.with.dots", job="api"}')
        self.assertTrue(r['valid'])

    def test_at_modifier(self):
        r = syntax('http_requests_total{job="api"} @ 1609459200')
        self.assertTrue(r['valid'])

    def test_absent(self):
        r = syntax('absent(up{job="critical-service"})')
        self.assertTrue(r['valid'])

    def test_topk(self):
        r = syntax('topk(5, sum by (service) (rate(http_requests_total[5m])))')
        self.assertTrue(r['valid'])


class TestSyntaxErrors(unittest.TestCase):
    """Queries with structural errors that must be caught."""

    def test_empty_query(self):
        r = syntax('')
        self.assertFalse(r['valid'])
        self.assertIn('empty_query', error_types(r))

    def test_unclosed_bracket(self):
        r = syntax('rate(http_requests_total[5m)')
        self.assertFalse(r['valid'])
        self.assertIn('unclosed_bracket', error_types(r))

    def test_unmatched_closing_bracket(self):
        r = syntax('rate(http_requests_total[5m]])')
        self.assertFalse(r['valid'])
        self.assertIn('unmatched_bracket', error_types(r))

    def test_unclosed_brace(self):
        r = syntax('http_requests_total{job="api"')
        self.assertFalse(r['valid'])
        self.assertIn('unclosed_brace', error_types(r))

    def test_unclosed_paren(self):
        r = syntax('rate(http_requests_total[5m]')
        self.assertFalse(r['valid'])
        self.assertIn('unclosed_paren', error_types(r))

    def test_unclosed_string(self):
        r = syntax('http_requests_total{job="api}')
        self.assertFalse(r['valid'])
        self.assertIn('unclosed_string', error_types(r))

    def test_invalid_duration(self):
        r = syntax('rate(http_requests_total[5minutes])')
        self.assertFalse(r['valid'])
        self.assertIn('invalid_duration', error_types(r))

    def test_unknown_function(self):
        r = syntax('rte(http_requests_total[5m])')
        self.assertFalse(r['valid'])
        self.assertIn('unknown_function', error_types(r))

    def test_missing_range_vector_rate(self):
        r = syntax('rate(http_requests_total)')
        self.assertFalse(r['valid'])
        self.assertIn('missing_range_vector', error_types(r))

    def test_missing_range_vector_increase(self):
        r = syntax('increase(http_requests_total{job="api"})')
        self.assertFalse(r['valid'])
        self.assertIn('missing_range_vector', error_types(r))

    def test_missing_range_vector_irate(self):
        r = syntax('irate(http_requests_total)')
        self.assertFalse(r['valid'])
        self.assertIn('missing_range_vector', error_types(r))

    def test_misplaced_offset(self):
        r = syntax('http_requests_total offset 1h [5m]')
        self.assertFalse(r['valid'])
        self.assertIn('misplaced_offset', error_types(r))

    def test_unknown_function_with_typo_suggestion(self):
        r = syntax('rte(http_requests_total[5m])')
        self.assertIn('unknown_function', error_types(r))
        # Error message should mention a suggestion
        msg = r['errors'][0]['message']
        self.assertIn('Did you mean', msg)


class TestSyntaxWarnings(unittest.TestCase):
    """Queries that should produce warnings, not hard errors."""

    def test_empty_label_matcher(self):
        r = syntax('http_requests_total{}')
        self.assertIn('empty_label_matcher', warning_types(r))
        # Empty matcher is a warning, not an error — query is still "valid"
        self.assertTrue(r['valid'])

    def test_double_operator_in_expression(self):
        r = syntax('metric ** 2')
        self.assertIn('double_operator', warning_types(r))


class TestSyntaxFalsePositiveFixes(unittest.TestCase):
    """Regression tests for bugs that produced false warnings."""

    def test_no_double_operator_for_dash_dash_in_label_value(self):
        """-- inside a quoted label value must not trigger double_operator."""
        r = syntax('http_requests_total{path="/api--v2"}')
        self.assertTrue(r['valid'])
        self.assertNotIn('double_operator', warning_types(r))

    def test_no_double_operator_for_plus_plus_in_label_value(self):
        """++ inside a quoted label value must not trigger double_operator."""
        r = syntax('http_requests_total{version="v2++"}')
        self.assertTrue(r['valid'])
        self.assertNotIn('double_operator', warning_types(r))

    def test_no_double_operator_for_star_star_in_label_value(self):
        """** inside a quoted label value must not trigger double_operator."""
        r = syntax('metric{comment="use ** for exponents"}')
        self.assertNotIn('double_operator', warning_types(r))

    def test_double_operator_outside_string_still_detected(self):
        """A real ** outside strings must still produce the warning."""
        r = syntax('metric ** 2')
        self.assertIn('double_operator', warning_types(r))


# ===========================================================================
# check_best_practices.py
# ===========================================================================

class TestBestPracticesClean(unittest.TestCase):
    """Well-formed queries that must produce no issues."""

    def test_rate_with_labels(self):
        r = best('rate(http_requests_total{job="api", status="200"}[5m])')
        self.assertEqual(r['issues'], [])

    def test_histogram_quantile_correct(self):
        q = 'histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))'
        self.assertEqual(best(q)['issues'], [])

    def test_gauge_used_directly(self):
        r = best('node_memory_usage_bytes{instance="prod-1"}')
        self.assertEqual(r['issues'], [])

    def test_avg_over_time_on_gauge(self):
        r = best('avg_over_time(node_memory_usage_bytes{job="node-exporter"}[5m])')
        self.assertEqual(r['issues'], [])


class TestHighCardinality(unittest.TestCase):

    def test_bare_metric_flagged(self):
        r = best('http_requests_total')
        self.assertIn('high_cardinality', issue_types(r))

    def test_empty_label_selector_flagged(self):
        r = best('http_requests_total{}')
        self.assertIn('high_cardinality', issue_types(r))

    def test_metric_with_label_not_flagged(self):
        r = best('http_requests_total{job="api"}')
        self.assertNotIn('high_cardinality', issue_types(r))


class TestRateOnGauge(unittest.TestCase):

    def test_rate_on_bytes_gauge(self):
        r = best('rate(node_memory_usage_bytes[5m])')
        self.assertIn('rate_on_gauge', issue_types(r))

    def test_irate_on_temperature_gauge(self):
        r = best('irate(cpu_temperature_celsius[5m])')
        self.assertIn('rate_on_gauge', issue_types(r))

    def test_increase_on_bytes_gauge(self):
        r = best('increase(node_memory_usage_bytes[1h])')
        self.assertIn('rate_on_gauge', issue_types(r))

    def test_rate_on_counter_not_flagged(self):
        r = best('rate(http_requests_total[5m])')
        self.assertNotIn('rate_on_gauge', issue_types(r))


class TestMissingRate(unittest.TestCase):

    def test_bare_counter_flagged(self):
        r = best('http_requests_total{job="api"}')
        self.assertIn('missing_rate', issue_types(r))

    def test_rate_wrapped_counter_clean(self):
        r = best('rate(http_requests_total{job="api"}[5m])')
        self.assertNotIn('missing_rate', issue_types(r))

    def test_sum_count_pair_in_division_clean(self):
        """_sum/_count used as a pair for histogram average must not warn."""
        q = ('rate(http_request_duration_seconds_sum{job="api"}[5m])'
             ' / rate(http_request_duration_seconds_count{job="api"}[5m])')
        self.assertNotIn('missing_rate', issue_types(best(q)))

    def test_orphan_sum_metric_flagged(self):
        r = best('http_request_duration_seconds_sum{job="api"}')
        self.assertIn('missing_rate', issue_types(r))


class TestMissingRateFalsePositiveFix(unittest.TestCase):
    """Regression tests for the counter-name-in-label-value false positive."""

    def test_counter_name_in_label_value_not_flagged(self):
        """http_requests_total appearing as a label VALUE must not produce missing_rate."""
        r = best('other_metric_total{label="http_requests_total"}')
        issues = issue_types(r)
        # Exactly ONE missing_rate for the actual bare counter (other_metric_total)
        self.assertEqual(issues.count('missing_rate'), 1)
        # Confirm it's for the outer metric, not the label value
        msg = r['issues'][0]['message']
        self.assertIn('other_metric_total', msg)
        self.assertNotIn('http_requests_total', msg)

    def test_bucket_name_in_label_value_not_flagged(self):
        """_bucket suffix in a label value must not trigger missing_rate."""
        r = best('config_metric{source="http_request_duration_seconds_bucket"}')
        # config_metric has no counter suffix, so missing_rate should not fire at all
        self.assertNotIn('missing_rate', issue_types(r))

    def test_counter_name_in_label_value_with_rate_wrapper(self):
        """A well-formed rate() query with a counter name in a label value is fully clean."""
        r = best('rate(http_requests_total{label="some_counter_total"}[5m])')
        self.assertNotIn('missing_rate', issue_types(r))


class TestAveragingQuantiles(unittest.TestCase):

    def test_avg_on_quantile_flagged(self):
        r = best('avg(http_request_duration_seconds{quantile="0.95"})')
        self.assertIn('averaging_quantiles', issue_types(r))

    def test_sum_on_quantile_flagged(self):
        """sum() on pre-calculated quantile is equally invalid — must be detected."""
        r = best('sum(http_request_duration_seconds{quantile="0.95"})')
        self.assertIn('averaging_quantiles', issue_types(r))

    def test_min_on_quantile_flagged(self):
        r = best('min(http_request_duration_seconds{quantile="0.99"})')
        self.assertIn('averaging_quantiles', issue_types(r))

    def test_max_on_quantile_flagged(self):
        r = best('max(http_request_duration_seconds{quantile="0.50"})')
        self.assertIn('averaging_quantiles', issue_types(r))

    def test_stddev_on_quantile_flagged(self):
        r = best('stddev(response_time_seconds{quantile="0.99"})')
        self.assertIn('averaging_quantiles', issue_types(r))

    def test_avg_without_quantile_not_flagged(self):
        """avg() on a gauge must NOT trigger averaging_quantiles."""
        r = best('avg(node_memory_usage_bytes)')
        self.assertNotIn('averaging_quantiles', issue_types(r))

    def test_correct_histogram_quantile_not_flagged(self):
        """histogram_quantile() with buckets must never trigger averaging_quantiles."""
        q = 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))'
        self.assertNotIn('averaging_quantiles', issue_types(best(q)))


class TestIrateRange(unittest.TestCase):

    def test_irate_long_range_flagged(self):
        r = best('irate(http_requests_total[1h])')
        self.assertIn('irate_long_range', issue_types(r))

    def test_irate_short_range_clean(self):
        r = best('irate(http_requests_total{job="api"}[2m])')
        self.assertNotIn('irate_long_range', issue_types(r))


class TestRateRange(unittest.TestCase):

    def test_rate_short_range_flagged(self):
        r = best('rate(http_requests_total{job="api"}[30s])')
        self.assertIn('rate_short_range', issue_types(r))

    def test_rate_acceptable_range_clean(self):
        r = best('rate(http_requests_total{job="api"}[2m])')
        self.assertNotIn('rate_short_range', issue_types(r))


class TestExpensiveSubquery(unittest.TestCase):

    def test_long_subquery_flagged(self):
        r = best('rate(http_requests_total[5m])[95d:1m]')
        self.assertIn('expensive_subquery', issue_types(r))

    def test_reasonable_subquery_clean(self):
        r = best('max_over_time(rate(http_requests_total{job="api"}[5m])[30m:1m])')
        self.assertNotIn('expensive_subquery', issue_types(r))


class TestHistogramUsage(unittest.TestCase):

    def test_histogram_quantile_without_rate_flagged(self):
        q = 'histogram_quantile(0.95, sum by (le) (http_request_duration_seconds_bucket))'
        self.assertIn('histogram_missing_rate', issue_types(best(q)))

    def test_histogram_quantile_missing_le_flagged(self):
        q = 'histogram_quantile(0.95, sum by (job) (rate(http_request_duration_seconds_bucket[5m])))'
        self.assertIn('histogram_missing_le', issue_types(best(q)))

    def test_correct_histogram_quantile_clean(self):
        q = 'histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))'
        self.assertNotIn('histogram_missing_rate', issue_types(best(q)))
        self.assertNotIn('histogram_missing_le', issue_types(best(q)))


class TestDeprecatedFunctions(unittest.TestCase):

    def test_holt_winters_flagged(self):
        r = best('holt_winters(http_requests_total[5m], 0.1, 0.3)')
        self.assertIn('deprecated_function', issue_types(r))


class TestRegexOveruse(unittest.TestCase):

    def test_exact_string_with_regex_operator_flagged_as_optimization(self):
        """=~"200" can be expressed as ="200" — should appear in optimizations."""
        r = best('rate(http_requests_total{status=~"200"}[5m])')
        self.assertIn('regex_to_exact', optimization_types(r))

    def test_real_regex_not_flagged(self):
        """=~"5.." is a real regex pattern and must not be flagged."""
        r = best('rate(http_requests_total{status=~"5.."}[5m])')
        self.assertNotIn('regex_to_exact', optimization_types(r))


class TestVectorMatching(unittest.TestCase):

    def test_info_metric_join_missing_group_left_flagged(self):
        q = 'rate(http_requests_total[5m]) * on (job, instance) service_info'
        self.assertIn('info_metric_missing_group', issue_types(best(q)))

    def test_group_left_without_on_flagged(self):
        q = 'metric_a * group_left() metric_b'
        self.assertIn('group_without_matching', issue_types(best(q)))


class TestHighCardinalityAggregationLabels(unittest.TestCase):

    def test_by_with_user_id_flagged(self):
        q = 'sum by (user_id) (rate(http_requests_total[5m]))'
        self.assertIn('high_cardinality_aggregation_label', issue_types(best(q)))

    def test_without_with_user_id_not_flagged(self):
        q = 'sum without (user_id) (rate(http_requests_total[5m]))'
        self.assertNotIn('high_cardinality_aggregation_label', issue_types(best(q)))

    def test_mixed_by_and_without_only_by_triggers_warning(self):
        q = ('sum without (user_id) (rate(http_requests_total[5m])) + '
             'sum by (session_id) (rate(http_requests_total[5m]))')
        self.assertIn('high_cardinality_aggregation_label', issue_types(best(q)))

    def test_nested_without_and_safe_by_not_flagged(self):
        q = 'sum by (job) (sum without (user_id) (rate(http_requests_total[5m])))'
        self.assertNotIn('high_cardinality_aggregation_label', issue_types(best(q)))


class TestBestPracticesCliIntegration(unittest.TestCase):
    """CLI-level regression checks for emitted issue IDs."""

    def test_cli_emits_aggregation_label_issue_id_for_by_clause(self):
        query = 'sum by (user_id) (rate(http_requests_total[5m]))'
        proc, result = best_cli(query)

        self.assertEqual(proc.returncode, 0, proc.stderr)
        self.assertIn('high_cardinality_aggregation_label', issue_types(result))

    def test_cli_does_not_emit_aggregation_label_issue_id_for_without_clause(self):
        query = 'sum without (user_id) (rate(http_requests_total[5m]))'
        proc, result = best_cli(query)

        self.assertEqual(proc.returncode, 0, proc.stderr)
        self.assertNotIn('high_cardinality_aggregation_label', issue_types(result))


if __name__ == '__main__':
    unittest.main(verbosity=2)


path: scripts/validate_syntax.py
content: #!/usr/bin/env python3
"""
PromQL Syntax Validator

Validates the syntax of Prometheus Query Language (PromQL) expressions.
Checks for correct metric names, label matchers, operators, functions, and time durations.
"""

import re
import sys
import json
from typing import Dict, List, Tuple, Optional


class PromQLSyntaxValidator:
    """Validates PromQL query syntax"""

    # Regex patterns for PromQL syntax elements
    METRIC_NAME_PATTERN = r'[a-zA-Z_:][a-zA-Z0-9_:]*'
    DURATION_PATTERN = r'\d+(?:ms|s|m|h|d|w|y)'
    LABEL_MATCHER_PATTERN = r'[a-zA-Z_][a-zA-Z0-9_]*\s*(?:=|!=|=~|!~)\s*"(?:[^"\\]|\\.)*"'

    # Valid PromQL functions
    FUNCTIONS = {
        # Aggregation operators
        'sum', 'min', 'max', 'avg', 'group', 'stddev', 'stdvar', 'count', 'count_values',
        'bottomk', 'topk', 'quantile',
        # Limiting functions (Prometheus 2.43+, experimental)
        'limitk', 'limit_ratio',
        # Rate/increase functions
        'rate', 'irate', 'increase', 'delta', 'idelta', 'deriv',
        # Time functions
        'timestamp', 'time', 'minute', 'hour', 'day_of_month', 'day_of_week',
        'days_in_month', 'month', 'year',
        # Math functions
        'abs', 'ceil', 'floor', 'round', 'sqrt', 'exp', 'ln', 'log2', 'log10',
        'sin', 'cos', 'tan', 'asin', 'acos', 'atan', 'sinh', 'cosh', 'tanh',
        'asinh', 'acosh', 'atanh', 'deg', 'rad', 'sgn', 'clamp', 'clamp_max', 'clamp_min',
        # Histogram/summary functions (classic)
        'histogram_quantile', 'histogram_count', 'histogram_sum', 'histogram_fraction',
        # Native histogram functions (Prometheus 2.40+/3.0)
        'histogram_avg', 'histogram_stddev', 'histogram_stdvar',
        # Label manipulation
        'label_replace', 'label_join',
        # Over time functions
        'changes', 'resets', 'avg_over_time', 'min_over_time', 'max_over_time',
        'sum_over_time', 'count_over_time', 'quantile_over_time', 'stddev_over_time',
        'stdvar_over_time', 'last_over_time', 'present_over_time', 'mad_over_time',
        # Prometheus 3.5+ experimental timestamp functions
        # (require --enable-feature=promql-experimental-functions)
        'ts_of_max_over_time', 'ts_of_min_over_time', 'ts_of_last_over_time',
        # Prometheus 3.7+ experimental functions
        # (require --enable-feature=promql-experimental-functions)
        'first_over_time', 'ts_of_first_over_time',
        # Prediction functions
        'predict_linear',
        'holt_winters',  # Deprecated in Prometheus 3.0, use double_exponential_smoothing
        'double_exponential_smoothing',  # Prometheus 3.0+ (replaces holt_winters, experimental)
        # Sorting functions
        'sort', 'sort_desc', 'sort_by_label', 'sort_by_label_desc',
        # Other functions
        'absent', 'absent_over_time', 'scalar', 'vector',
        # Metric joining (experimental)
        'info',
        # Trigonometric functions
        'pi',
    }

    # Aggregation operators that support by/without clauses
    AGGREGATION_OPERATORS = {
        'sum', 'min', 'max', 'avg', 'group', 'stddev', 'stdvar', 'count',
        'count_values', 'bottomk', 'topk', 'quantile',
        # Prometheus 2.43+ (experimental, requires --enable-feature=promql-experimental-functions)
        'limitk', 'limit_ratio'
    }

    # Binary operators
    BINARY_OPERATORS = {
        '+', '-', '*', '/', '%', '^',  # Arithmetic
        '==', '!=', '>', '<', '>=', '<=',  # Comparison
        'and', 'or', 'unless',  # Logical
    }

    # Keywords that look like functions (followed by parentheses) but are NOT functions
    # These are aggregation modifiers, vector matching keywords, etc.
    NON_FUNCTION_KEYWORDS = {
        'by',           # Aggregation modifier: sum by (label)
        'without',      # Aggregation modifier: sum without (label)
        'on',           # Vector matching: metric_a + on (label) metric_b
        'ignoring',     # Vector matching: metric_a + ignoring (label) metric_b
        'group_left',   # Vector matching: one-to-many joins
        'group_right',  # Vector matching: many-to-one joins
        'bool',         # Comparison modifier: metric > bool 10
        'start',        # @ modifier: metric[5m] @ start()
        'end',          # @ modifier: metric[5m] @ end()
    }

    def __init__(self, query: str):
        self.query = query.strip()
        self.errors: List[Dict] = []
        self.warnings: List[Dict] = []

    def validate(self) -> Dict:
        """
        Run all validation checks

        Returns:
            Dict containing validation results
        """
        if not self.query:
            self.errors.append({
                'type': 'empty_query',
                'message': 'Query is empty',
                'severity': 'error'
            })
            return self._build_result()

        # Check for balanced brackets and quotes
        self._check_balanced_delimiters()

        # Check for valid metric names and selectors
        self._check_metric_selectors()

        # Check for valid time ranges
        self._check_time_ranges()

        # Check function syntax
        self._check_function_syntax()

        # Check operators
        self._check_operators()

        # Check for common typos
        self._check_common_typos()

        return self._build_result()

    def _check_balanced_delimiters(self):
        """Check for balanced brackets, braces, and quotes"""
        brackets = []
        braces = []
        parens = []
        in_string = False
        escape_next = False

        for i, char in enumerate(self.query):
            if escape_next:
                escape_next = False
                continue

            if char == '\\':
                escape_next = True
                continue

            if char == '"':
                in_string = not in_string
                continue

            if in_string:
                continue

            if char == '[':
                brackets.append(i)
            elif char == ']':
                if not brackets:
                    self.errors.append({
                        'type': 'unmatched_bracket',
                        'message': f'Unmatched closing bracket at position {i}',
                        'position': i,
                        'severity': 'error'
                    })
                else:
                    brackets.pop()
            elif char == '{':
                braces.append(i)
            elif char == '}':
                if not braces:
                    self.errors.append({
                        'type': 'unmatched_brace',
                        'message': f'Unmatched closing brace at position {i}',
                        'position': i,
                        'severity': 'error'
                    })
                else:
                    braces.pop()
            elif char == '(':
                parens.append(i)
            elif char == ')':
                if not parens:
                    self.errors.append({
                        'type': 'unmatched_paren',
                        'message': f'Unmatched closing parenthesis at position {i}',
                        'position': i,
                        'severity': 'error'
                    })
                else:
                    parens.pop()

        if in_string:
            self.errors.append({
                'type': 'unclosed_string',
                'message': 'Unclosed string literal',
                'severity': 'error'
            })

        if brackets:
            self.errors.append({
                'type': 'unclosed_bracket',
                'message': f'Unclosed bracket at position {brackets[-1]}',
                'position': brackets[-1],
                'severity': 'error'
            })

        if braces:
            self.errors.append({
                'type': 'unclosed_brace',
                'message': f'Unclosed brace at position {braces[-1]}',
                'position': braces[-1],
                'severity': 'error'
            })

        if parens:
            self.errors.append({
                'type': 'unclosed_paren',
                'message': f'Unclosed parenthesis at position {parens[-1]}',
                'position': parens[-1],
                'severity': 'error'
            })

    def _check_metric_selectors(self):
        """Check for valid metric names and label selectors"""
        # Pattern for metric selector (metric name with optional label matchers)
        # Matches: metric_name{label="value"} or {label="value"} or {"metric.name"}

        # Find all potential metric selectors
        # Remove strings first to avoid false matches
        query_no_strings = re.sub(r'"(?:[^"\\]|\\.)*"', '""', self.query)

        # Check for empty label matchers
        if re.search(r'\{\s*\}', query_no_strings):
            self.warnings.append({
                'type': 'empty_label_matcher',
                'message': 'Empty label matcher {} found - this may match many time series',
                'severity': 'warning'
            })

        # Validate UTF-8 metric name syntax (Prometheus 3.0+)
        # Valid: {"my.metric"} or {"my.metric", label="value"}
        # Per https://prometheus.io/docs/guides/utf8/
        self._check_utf8_metric_syntax()

    def _check_utf8_metric_syntax(self):
        """
        Validate UTF-8 metric name quoting syntax (Prometheus 3.0+)

        Per Prometheus docs, UTF-8 metric names must be:
        - Enclosed in double quotes
        - Placed within curly braces as the first element
        - Format: {"my.metric"} or {"my.metric", label="value"}
        """
        # Pattern for quoted metric name selector (UTF-8 syntax)
        # Matches: {"metric.name"} or {"metric.name", label="value", ...}
        utf8_selector_pattern = r'\{\s*"([^"]+)"'

        utf8_matches = re.finditer(utf8_selector_pattern, self.query)

        for match in utf8_matches:
            metric_name = match.group(1)

            # Check if this looks like it should be a UTF-8 metric name
            # (contains characters not valid in classic metric names)
            classic_valid = re.fullmatch(self.METRIC_NAME_PATTERN, metric_name)

            if classic_valid:
                # Metric name is valid in classic format, quoting is optional but valid
                self.warnings.append({
                    'type': 'utf8_metric_unnecessary_quoting',
                    'message': f'Metric name "{metric_name}" uses UTF-8 quoting but is valid in classic format',
                    'severity': 'info'
                })
            else:
                # This is a proper UTF-8 metric name (has special chars like dots)
                # Validate the UTF-8 name format
                if not metric_name.strip():
                    self.errors.append({
                        'type': 'invalid_utf8_metric',
                        'message': 'Empty quoted metric name in UTF-8 selector',
                        'severity': 'error'
                    })
                # UTF-8 metric names can contain any valid UTF-8 character except null
                elif '\x00' in metric_name:
                    self.errors.append({
                        'type': 'invalid_utf8_metric',
                        'message': f'Metric name "{metric_name}" contains null character',
                        'severity': 'error'
                    })

        # Check for common UTF-8 syntax mistakes
        # Wrong: {my.metric="value"} - should be {"my.metric"}
        # This pattern looks for unquoted metric-like names with dots used as label names
        wrong_utf8_pattern = r'\{\s*([a-zA-Z_][a-zA-Z0-9_.]*\.[a-zA-Z0-9_.]+)\s*='
        wrong_matches = re.finditer(wrong_utf8_pattern, self.query)

        for match in wrong_matches:
            suspicious_name = match.group(1)
            # Only flag if it contains dots (likely meant to be a UTF-8 metric name)
            if '.' in suspicious_name:
                self.warnings.append({
                    'type': 'possible_utf8_syntax_error',
                    'message': f'Label name "{suspicious_name}" contains dots - did you mean to use UTF-8 metric syntax?',
                    'severity': 'warning',
                    'hint': f'For UTF-8 metric names use: {{"{suspicious_name}"}}'
                })

    def _check_time_ranges(self):
        """Check for valid time range syntax"""
        # Strip quoted string literals before scanning for range vectors.
        # Without this, regex character classes inside label-value strings
        # (e.g. "([^:]+):.*" in a label_replace() call) are misidentified
        # as subquery range syntax.
        query_no_strings = re.sub(r'"(?:[^"\\]|\\.)*"', '""', self.query)
        # Find all range vectors [duration]
        range_pattern = r'\[([^\]]+)\]'
        ranges = re.findall(range_pattern, query_no_strings)

        for range_str in ranges:
            range_str = range_str.strip()

            # Check if it's a subquery (has colon)
            if ':' in range_str:
                # Subquery format: [range:resolution] or [range:]
                parts = range_str.split(':')
                if len(parts) > 2:
                    self.errors.append({
                        'type': 'invalid_subquery',
                        'message': f'Invalid subquery syntax: [{range_str}]',
                        'severity': 'error'
                    })
                    continue

                range_part = parts[0].strip()
                resolution_part = parts[1].strip() if len(parts) > 1 else ''

                if range_part and not re.fullmatch(self.DURATION_PATTERN, range_part):
                    self.errors.append({
                        'type': 'invalid_duration',
                        'message': f'Invalid duration in subquery range: {range_part}',
                        'severity': 'error'
                    })

                if resolution_part and not re.fullmatch(self.DURATION_PATTERN, resolution_part):
                    self.errors.append({
                        'type': 'invalid_duration',
                        'message': f'Invalid duration in subquery resolution: {resolution_part}',
                        'severity': 'error'
                    })
            else:
                # Regular range vector
                if not re.fullmatch(self.DURATION_PATTERN, range_str):
                    self.errors.append({
                        'type': 'invalid_duration',
                        'message': f'Invalid duration syntax: {range_str}. Expected format like 5m, 1h, 7d',
                        'severity': 'error'
                    })

    def _check_function_syntax(self):
        """Check for valid function usage"""
        # Find all function calls
        func_pattern = r'([a-z_][a-z0-9_]*)\s*\('
        functions = re.findall(func_pattern, self.query, re.IGNORECASE)

        for func in functions:
            func_lower = func.lower()

            # Skip keywords that look like functions but aren't
            # (e.g., "by", "without", "on", "ignoring", "group_left", "group_right")
            if func_lower in self.NON_FUNCTION_KEYWORDS:
                continue

            if func_lower not in self.FUNCTIONS:
                # Check if it might be a typo
                close_matches = self._find_close_matches(func_lower, self.FUNCTIONS)
                if close_matches:
                    self.errors.append({
                        'type': 'unknown_function',
                        'message': f'Unknown function: {func}. Did you mean: {", ".join(close_matches)}?',
                        'severity': 'error'
                    })
                else:
                    self.errors.append({
                        'type': 'unknown_function',
                        'message': f'Unknown function: {func}',
                        'severity': 'error'
                    })

    def _check_operators(self):
        """Check for valid operator usage"""
        # Strip quoted string literals before searching for consecutive operators so that
        # label values containing "--" or "++" (e.g. {path="/api--v2"}) do not produce
        # false positives.
        query_no_strings = re.sub(r'"(?:[^"\\]|\\.)*"', '""', self.query)
        if re.search(r'[+\-*/]{2,}', query_no_strings):
            self.warnings.append({
                'type': 'double_operator',
                'message': 'Found consecutive operators - this might be a typo',
                'severity': 'warning'
            })

    def _check_common_typos(self):
        """Check for common typos and mistakes"""
        # Check for 'rate()' without range vector
        if re.search(r'\b(rate|irate|increase|delta|idelta)\s*\(\s*[a-zA-Z_:][a-zA-Z0-9_:]*\s*(?:\{[^}]*\})?\s*\)', self.query):
            self.errors.append({
                'type': 'missing_range_vector',
                'message': 'rate(), irate(), increase(), delta(), and idelta() require a range vector [duration]',
                'severity': 'error'
            })

        # Check for 'offset' keyword - valid positions:
        # 1. metric_name offset 1h (instant vector)
        # 2. metric_name[5m] offset 1h (range vector)
        # 3. rate(metric_name[5m] offset 1h) (inside function)
        # Invalid: metric_name offset 1h [5m] (offset between metric and range)
        if ' offset ' in self.query.lower():
            # Check for invalid pattern: offset followed by range vector
            if re.search(r'\boffset\s+\d+[smhdwy]\s*\[', self.query, re.IGNORECASE):
                self.errors.append({
                    'type': 'misplaced_offset',
                    'message': 'offset modifier should come after the range vector [duration], not before it',
                    'severity': 'error'
                })

    def _find_close_matches(self, word: str, candidates: set, max_distance: int = 2) -> List[str]:
        """Find close matches using simple edit distance"""
        matches = []
        for candidate in candidates:
            if abs(len(word) - len(candidate)) > max_distance:
                continue
            if self._levenshtein_distance(word, candidate) <= max_distance:
                matches.append(candidate)
        return matches[:3]  # Return top 3 matches

    @staticmethod
    def _levenshtein_distance(s1: str, s2: str) -> int:
        """Calculate Levenshtein distance between two strings"""
        if len(s1) < len(s2):
            return PromQLSyntaxValidator._levenshtein_distance(s2, s1)
        if len(s2) == 0:
            return len(s1)

        previous_row = range(len(s2) + 1)
        for i, c1 in enumerate(s1):
            current_row = [i + 1]
            for j, c2 in enumerate(s2):
                insertions = previous_row[j + 1] + 1
                deletions = current_row[j] + 1
                substitutions = previous_row[j] + (c1 != c2)
                current_row.append(min(insertions, deletions, substitutions))
            previous_row = current_row

        return previous_row[-1]

    def _build_result(self) -> Dict:
        """Build the validation result dictionary"""
        has_errors = len(self.errors) > 0
        has_warnings = len(self.warnings) > 0

        if has_errors:
            status = 'ERROR'
        elif has_warnings:
            status = 'WARNING'
        else:
            status = 'VALID'

        return {
            'status': status,
            'query': self.query,
            'errors': self.errors,
            'warnings': self.warnings,
            'valid': not has_errors
        }


def main():
    """Main entry point for the syntax validator"""
    if len(sys.argv) < 2:
        print(json.dumps({
            'status': 'ERROR',
            'message': 'Usage: validate_syntax.py "<promql_query>"'
        }, indent=2))
        sys.exit(1)

    query = sys.argv[1]
    validator = PromQLSyntaxValidator(query)
    result = validator.validate()

    print(json.dumps(result, indent=2))

    # Exit with error code if validation failed
    sys.exit(0 if result['valid'] else 1)


if __name__ == '__main__':
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)