Project: Thanos (Observability and Analysis / Observability)
File: alerts.md

Here are some example alerts configured for Kubernetes environment. ```yaml mdox-exec="cat examples/tmp/thanos-compact.yaml" name: thanos-compact rules: alert: ThanosCompactMultipleRunning annotations: description: No more than one Thanos Compact instance should be running at once. There are {{$value}} instances running. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanoscompactmultiplerunning summary: Thanos Compact has multiple instances running. expr: sum by (job) (up{job=~".thanos-compact."}) > 1 for: 5m labels: severity: warning alert: ThanosCompactHalted annotations: description: Thanos Compact {{$labels.job}} has failed to run and now is halted. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanoscompacthalted summary: Thanos Compact has failed to run and is now halted. expr: thanoscompacthalted{job=~".thanos-compact."} == 1 for: 5m labels: severity: warning alert: ThanosCompactHighCompactionFailures annotations: description: Thanos Compact {{$labels.job}} is failing to execute {{$value | humanize}}% of compactions. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanoscompacthighcompactionfailures summary: Thanos Compact is failing to execute compactions. expr: | ( sum by (job) (rate(thanoscompactgroupcompactionsfailures_total{job=~".thanos-compact."}[5m])) / sum by (job) (rate(thanoscompactgroupcompactionstotal{job=~".thanos-compact."}[5m])) 100 > 5 ) for: 15m labels: severity: warning alert: ThanosCompactBucketHighOperationFailures annotations: description: Thanos Compact {{$labels.job}} Bucket is failing to execute {{$value | humanize}}% of operations. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanoscompactbuckethighoperationfailures summary: Thanos Compact Bucket is having a high number of operation failures. expr: | ( sum by (job) (rate(thanosobjstorebucketoperationfailures_total{job=~".thanos-compact."}[5m])) / sum by (job) (rate(thanosobjstorebucketoperationstotal{job=~".thanos-compact."}[5m])) 100 > 5 ) for: 15m labels: severity: warning alert: ThanosCompactHasNotRun annotations: description: Thanos Compact {{$labels.job}} has not uploaded anything for 24 hours. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanoscompacthasnotrun summary: Thanos Compact has not uploaded anything for last 24 hours. expr: (time() - max by (job) (maxovertime(thanosobjstorebucketlastsuccessfuluploadtime{job=~".thanos-compact."}[24h]))) / 60 / 60 > 24 labels: severity: warning ``` For Thanos Ruler we run some alerts in local Prometheus, to make sure that Thanos Ruler is working: ```yaml mdox-exec="cat examples/tmp/thanos-rule.yaml" name: thanos-rule rules: alert: ThanosRuleQueueIsDroppingAlerts annotations: description: Thanos Rule {{$labels.instance}} is failing to queue alerts. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulequeueisdroppingalerts summary: Thanos Rule is failing to queue alerts. expr: | sum by (job, instance) (rate(thanosalertqueuealertsdropped_total{job=~".thanos-rule."}[5m])) > 0 for: 5m labels: severity: critical alert: ThanosRuleSenderIsFailingAlerts annotations: description: Thanos Rule {{$labels.instance}} is failing to send alerts to alertmanager. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulesenderisfailingalerts summary: Thanos Rule is failing to send alerts to alertmanager. expr: | sum by (job, instance) (rate(thanosalertsenderalertsdropped_total{job=~".thanos-rule."}[5m])) > 0 for: 5m labels: severity: critical alert: ThanosRuleHighRuleEvaluationFailures annotations: description: Thanos Rule {{$labels.instance}} is failing to evaluate rules. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulehighruleevaluationfailures summary: Thanos Rule is failing to evaluate rules. expr: | ( sum by (job, instance) (rate(prometheusruleevaluationfailurestotal{job=~".thanos-rule."}[5m])) / sum by (job, instance) (rate(prometheusruleevaluations_total{job=~".thanos-rule."}[5m])) 100 > 5 ) for: 5m labels: severity: critical alert: ThanosRuleHighRuleEvaluationWarnings annotations: description: Thanos Rule {{$labels.instance}} has high number of evaluation warnings. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulehighruleevaluationwarnings summary: Thanos Rule has high number of evaluation warnings. expr: | sum by (job, instance) (rate(thanosruleevaluationwithwarnings_total{job=~".thanos-rule."}[5m])) > 0 for: 15m labels: severity: info alert: ThanosRuleRuleEvaluationLatencyHigh annotations: description: Thanos Rule {{$labels.instance}} has higher evaluation latency than interval for {{$labels.rule_group}}. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosruleruleevaluationlatencyhigh summary: Thanos Rule has high rule evaluation latency. expr: | ( sum by (job, instance, rulegroup) (prometheusrulegrouplastdurationseconds{job=~".thanos-rule."}) > sum by (job, instance, rulegroup) (prometheusrulegroupinterval_seconds{job=~".thanos-rule."}) ) for: 5m labels: severity: warning alert: ThanosRuleGrpcErrorRate annotations: description: Thanos Rule {{$labels.job}} is failing to handle {{$value | humanize}}% of requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulegrpcerrorrate summary: Thanos Rule is failing to handle grpc requests. expr: | ( sum by (job, instance) (rate(grpcserverhandledtotal{grpccode=~"Unknown|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded", job=~".thanos-rule."}[5m])) / sum by (job, instance) (rate(grpcserverstarted_total{job=~".thanos-rule."}[5m])) 100 > 5 ) for: 5m labels: severity: warning alert: ThanosRuleConfigReloadFailure annotations: description: Thanos Rule {{$labels.job}} has not been able to reload its configuration. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosruleconfigreloadfailure summary: Thanos Rule has not been able to reload configuration. expr: avg by (job, instance) (thanosruleconfiglastreload_successful{job=~".thanos-rule."}) != 1 for: 5m labels: severity: info alert: ThanosRuleQueryHighDNSFailures annotations: description: Thanos Rule {{$labels.job}} has {{$value | humanize}}% of failing DNS queries for query endpoints.

https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulequeryhighdnsfailures summary: Thanos Rule is having high number of DNS failures. expr: | ( sum by (job, instance) (rate(thanosrulequeryapisdnsfailurestotal{job=~".thanos-rule."}[5m])) / sum by (job, instance) (rate(thanosrulequeryapisdnslookupstotal{job=~".thanos-rule."}[5m])) 100 > 1 ) for: 15m labels: severity: warning alert: ThanosRuleAlertmanagerHighDNSFailures annotations: description: Thanos Rule {{$labels.instance}} has {{$value | humanize}}% of failing DNS queries for Alertmanager endpoints. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulealertmanagerhighdnsfailures summary: Thanos Rule is having high number of DNS failures. expr: | ( sum by (job, instance) (rate(thanosrulealertmanagersdnsfailures_total{job=~".thanos-rule."}[5m])) / sum by (job, instance) (rate(thanosrulealertmanagersdnslookups_total{job=~".thanos-rule."}[5m])) 100 > 1 ) for: 15m labels: severity: warning alert: ThanosRuleNoEvaluationFor10Intervals annotations: description: Thanos Rule {{$labels.job}} has rule groups that did not evaluate for at least 10x of their expected interval. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosrulenoevaluationfor10intervals summary: Thanos Rule has rule groups that did not evaluate for 10 intervals. expr: | time() - max by (job, instance, group) (prometheusrulegrouplastevaluationtimestampseconds{job=~".thanos-rule."}) > 10 max by (job, instance, group) (prometheus_rule_group_interval_seconds{job=~".thanos-rule.*"}) for: 5m labels: severity: info alert: ThanosNoRuleEvaluations annotations: description: Thanos Rule {{$labels.instance}} did not perform any rule evaluations in the past 10 minutes. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosnoruleevaluations summary: Thanos Rule did not perform any rule evaluations. expr: | sum by (job, instance) (rate(prometheusruleevaluations_total{job=~".thanos-rule."}[5m])) <= 0 and sum by (job, instance) (thanosruleloaded_rules{job=~".thanos-rule."}) > 0 for: 5m labels: severity: critical ``` ```yaml mdox-exec="cat examples/tmp/thanos-store.yaml" name: thanos-store rules: alert: ThanosStoreGrpcErrorRate annotations: description: Thanos Store {{$labels.job}} is failing to handle {{$value | humanize}}% of requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosstoregrpcerrorrate summary: Thanos Store is failing to handle gRPC requests. expr: | ( sum by (job) (rate(grpcserverhandledtotal{grpccode=~"Unknown|Internal|Unavailable|DataLoss|DeadlineExceeded", job=~".thanos-store."}[5m])) / sum by (job) (rate(grpcserverstarted_total{job=~".thanos-store."}[5m])) 100 > 5 ) for: 5m labels: severity: warning alert: ThanosStoreSeriesGateLatencyHigh annotations: description: Thanos Store {{$labels.job}} has a 99th percentile latency of {{$value}} seconds for store series gate requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosstoreseriesgatelatencyhigh summary: Thanos Store has high latency for store series gate requests. expr: | ( histogramquantile(0.99, sum by (job, le) (rate(thanosbucketstoreseriesgatedurationsecondsbucket{job=~".thanos-store."}[5m]))) > 2 and sum by (job) (rate(thanosbucketstoreseriesgatedurationseconds_count{job=~".thanos-store."}[5m])) > 0 ) for: 10m labels: severity: warning alert: ThanosStoreBucketHighOperationFailures annotations: description: Thanos Store {{$labels.job}} Bucket is failing to execute {{$value | humanize}}% of operations. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosstorebuckethighoperationfailures summary: Thanos Store Bucket is failing to execute operations. expr: | ( sum by (job) (rate(thanosobjstorebucketoperationfailures_total{job=~".thanos-store."}[5m])) / sum by (job) (rate(thanosobjstorebucketoperationstotal{job=~".thanos-store."}[5m])) 100 > 5 ) for: 15m labels: severity: warning alert: ThanosStoreObjstoreOperationLatencyHigh annotations: description: Thanos Store {{$labels.job}} Bucket has a 99th percentile latency of {{$value}} seconds for the bucket operations. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosstoreobjstoreoperationlatencyhigh summary: Thanos Store is having high latency for bucket operations. expr: | ( histogramquantile(0.99, sum by (job, le) (rate(thanosobjstorebucketoperationdurationseconds_bucket{job=~".thanos-store."}[5m]))) > 2 and sum by (job) (rate(thanosobjstorebucketoperationdurationsecondscount{job=~".thanos-store."}[5m])) > 0 ) for: 10m labels: severity: warning ``` ```yaml mdox-exec="cat examples/tmp/thanos-sidecar.yaml" name: thanos-sidecar rules: alert: ThanosSidecarBucketOperationsFailed annotations: description: Thanos Sidecar {{$labels.instance}} bucket operations are failing runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanossidecarbucketoperationsfailed summary: Thanos Sidecar bucket operations are failing expr: | sum by (job, instance) (rate(thanosobjstorebucketoperationfailures_total{job=~".thanos-sidecar."}[5m])) > 0 for: 5m labels: severity: critical alert: ThanosSidecarNoConnectionToStartedPrometheus annotations: description: Thanos Sidecar {{$labels.instance}} is unhealthy. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanossidecarnoconnectiontostartedprometheus summary: Thanos Sidecar cannot access Prometheus, even though Prometheus seems healthy and has reloaded WAL. expr: | thanossidecarprometheus_up{job=~".thanos-sidecar."} == 0 AND on (namespace, pod) prometheustsdbdatareplayduration_seconds != 0 for: 5m labels: severity: critical ``` ```yaml mdox-exec="cat examples/tmp/thanos-query.yaml" name: thanos-query rules: alert: ThanosQueryHttpRequestQueryErrorRateHigh annotations: description: Thanos Query {{$labels.job}} is failing to handle {{$value | humanize}}% of "query" requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosqueryhttprequestqueryerrorratehigh summary: Thanos Query is failing to handle requests. expr: | ( sum by (job) (rate(httprequeststotal{code=~"5..", job=~".thanos-query.", handler="query"}[5m])) / sum by (job) (rate(httprequeststotal{job=~".thanos-query.", handler="query"}[5m])) ) * 100 > 5 for: 5m labels: severity: critical alert: ThanosQueryHttpRequestQueryRangeErrorRateHigh annotations: description: Thanos Query {{$labels.job}} is failing to handle {{$value | humanize}}% of "query_range" requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosqueryhttprequestqueryrangeerrorratehigh summary: Thanos Query is failing to handle requests. expr: | ( sum by (job) (rate(httprequeststotal{code=~"5..",

handler="query_range"}[5m])) / sum by (job) (rate(httprequeststotal{job=~".thanos-query.", handler="query_range"}[5m])) ) * 100 > 5 for: 5m labels: severity: critical alert: ThanosQueryGrpcServerErrorRate annotations: description: Thanos Query {{$labels.job}} is failing to handle {{$value | humanize}}% of requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosquerygrpcservererrorrate summary: Thanos Query is failing to handle requests. expr: | ( sum by (job) (rate(grpcserverhandledtotal{grpccode=~"Unknown|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded", job=~".thanos-query."}[5m])) / sum by (job) (rate(grpcserverstarted_total{job=~".thanos-query."}[5m])) 100 > 5 ) for: 5m labels: severity: warning alert: ThanosQueryGrpcClientErrorRate annotations: description: Thanos Query {{$labels.job}} is failing to send {{$value | humanize}}% of requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosquerygrpcclienterrorrate summary: Thanos Query is failing to send requests. expr: | ( sum by (job) (rate(grpcclienthandledtotal{grpccode!="OK", job=~".thanos-query."}[5m])) / sum by (job) (rate(grpcclientstarted_total{job=~".thanos-query."}[5m])) ) * 100 > 5 for: 5m labels: severity: warning alert: ThanosQueryHighDNSFailures annotations: description: Thanos Query {{$labels.job}} have {{$value | humanize}}% of failing DNS queries for store endpoints. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosqueryhighdnsfailures summary: Thanos Query is having high number of DNS failures. expr: | ( sum by (job) (rate(thanosquerystoreapisdnsfailurestotal{job=~".thanos-query."}[5m])) / sum by (job) (rate(thanosquerystoreapisdnslookupstotal{job=~".thanos-query."}[5m])) ) * 100 > 1 for: 15m labels: severity: warning alert: ThanosQueryInstantLatencyHigh annotations: description: Thanos Query {{$labels.job}} has a 99th percentile latency of {{$value}} seconds for instant queries. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosqueryinstantlatencyhigh summary: Thanos Query has high latency for queries. expr: | ( histogramquantile(0.99, sum by (job, le) (rate(httprequestdurationseconds_bucket{job=~".thanos-query.", handler="query"}[5m]))) > 40 and sum by (job) (rate(httprequestdurationsecondsbucket{job=~".thanos-query.", handler="query"}[5m])) > 0 ) for: 10m labels: severity: critical alert: ThanosQueryRangeLatencyHigh annotations: description: Thanos Query {{$labels.job}} has a 99th percentile latency of {{$value}} seconds for range queries. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosqueryrangelatencyhigh summary: Thanos Query has high latency for queries. expr: | ( histogramquantile(0.99, sum by (job, le) (rate(httprequestdurationsecondsbucket{job=~".*thanos-query.*", handler="queryrange"}[5m]))) > 90 and sum by (job) (rate(httprequestdurationsecondscount{job=~".thanos-query.", handler="query_range"}[5m])) > 0 ) for: 10m labels: severity: critical alert: ThanosQueryOverload annotations: description: Thanos Query {{$labels.job}} has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosqueryoverload summary: Thanos query reaches its maximum capacity serving concurrent requests. expr: | ( maxovertime(thanosqueryconcurrentgatequeriesmax[5m]) - avgovertime(thanosqueryconcurrentgatequeriesin_flight[5m]) < 1 ) for: 15m labels: severity: warning ``` ```yaml mdox-exec="cat examples/tmp/thanos-receive.yaml" name: thanos-receive rules: alert: ThanosReceiveHttpRequestErrorRateHigh annotations: description: Thanos Receive {{$labels.job}} is failing to handle {{$value | humanize}}% of requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivehttprequesterrorratehigh summary: Thanos Receive is failing to handle requests. expr: | ( sum by (job) (rate(httprequeststotal{code=~"5..", job=~".thanos-receive.", handler="receive"}[5m])) / sum by (job) (rate(httprequeststotal{job=~".thanos-receive.", handler="receive"}[5m])) ) * 100 > 5 for: 5m labels: severity: critical alert: ThanosReceiveHttpRequestLatencyHigh annotations: description: Thanos Receive {{$labels.job}} has a 99th percentile latency of {{ $value }} seconds for requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivehttprequestlatencyhigh summary: Thanos Receive has high HTTP requests latency. expr: | ( histogramquantile(0.99, sum by (job, le) (rate(httprequestdurationseconds_bucket{job=~".thanos-receive.", handler="receive"}[5m]))) > 10 and sum by (job) (rate(httprequestdurationsecondscount{job=~".thanos-receive.", handler="receive"}[5m])) > 0 ) for: 10m labels: severity: critical alert: ThanosReceiveHighReplicationFailures annotations: description: Thanos Receive {{$labels.job}} is failing to replicate {{$value | humanize}}% of requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivehighreplicationfailures summary: Thanos Receive is having high number of replication failures. expr: | thanosreceivereplication_factor > 1 and ( ( sum by (job) (rate(thanosreceivereplications_total{result="error", job=~".thanos-receive."}[5m])) / sum by (job) (rate(thanosreceivereplications_total{job=~".thanos-receive."}[5m])) ) > ( max by (job) (floor((thanosreceivereplication_factor{job=~".thanos-receive."}+1) / 2)) / max by (job) (thanosreceivehashring_nodes{job=~".thanos-receive."}) ) ) * 100 for: 5m labels: severity: warning alert: ThanosReceiveHighForwardRequestFailures annotations: description: Thanos Receive {{$labels.job}} is failing to forward {{$value | humanize}}% of requests. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivehighforwardrequestfailures summary: Thanos Receive is failing to forward requests. expr: | ( sum by (job) (rate(thanosreceiveforwardrequeststotal{result="error", job=~".thanos-receive."}[5m])) / sum by (job)

) * 100 > 20 for: 5m labels: severity: info alert: ThanosReceiveHighHashringFileRefreshFailures annotations: description: Thanos Receive {{$labels.job}} is failing to refresh hashring file, {{$value | humanize}} of attempts failed. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivehighhashringfilerefreshfailures summary: Thanos Receive is failing to refresh hasring file. expr: | ( sum by (job) (rate(thanosreceivehashringsfileerrors_total{job=~".thanos-receive."}[5m])) / sum by (job) (rate(thanosreceivehashringsfilerefreshes_total{job=~".thanos-receive."}[5m])) > 0 ) for: 15m labels: severity: warning alert: ThanosReceiveConfigReloadFailure annotations: description: Thanos Receive {{$labels.job}} has not been able to reload hashring configurations. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceiveconfigreloadfailure summary: Thanos Receive has not been able to reload configuration. expr: avg by (job) (thanosreceiveconfiglastreload_successful{job=~".thanos-receive."}) != 1 for: 5m labels: severity: warning alert: ThanosReceiveNoUpload annotations: description: Thanos Receive {{$labels.instance}} has not uploaded latest data to object storage. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivenoupload summary: Thanos Receive has not uploaded latest data to object storage. expr: | (up{job=~".thanos-receive."} - 1) on (job, instance) # filters to only alert on current instance last 3h (sum by (job, instance) (increase(thanosshipperuploads_total{job=~".thanos-receive."}[3h])) == 0) for: 3h labels: severity: critical alert: ThanosReceiveLimitsConfigReloadFailure annotations: description: Thanos Receive {{$labels.job}} has not been able to reload the limits configuration. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivelimitsconfigreloadfailure summary: Thanos Receive has not been able to reload the limits configuration. expr: sum by(job) (increase(thanosreceivelimitsconfigreloaderrtotal{job=~".thanos-receive."}[5m])) > 0 for: 5m labels: severity: warning alert: ThanosReceiveLimitsHighMetaMonitoringQueriesFailureRate annotations: description: Thanos Receive {{$labels.job}} is failing for {{$value | humanize}}% of meta monitoring queries. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivelimitshighmetamonitoringqueriesfailurerate summary: Thanos Receive has not been able to update the number of head series. expr: (sum by(job) (increase(thanosreceivemetamonitoringfailedqueries_total{job=~".thanos-receive."}[5m])) / 20) * 100 > 20 for: 5m labels: severity: warning alert: ThanosReceiveTenantLimitedByHeadSeries annotations: description: Thanos Receive tenant {{$labels.tenant}} is limited by head series. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceivetenantlimitedbyheadseries summary: A Thanos Receive tenant is limited by head series. expr: sum by(job, tenant) (increase(thanosreceiveheadserieslimitedrequeststotal{job=~".thanos-receive."}[5m])) > 0 for: 5m labels: severity: warning ``` ```yaml mdox-exec="cat examples/tmp/thanos-bucket-replicate.yaml" name: thanos-bucket-replicate rules: alert: ThanosBucketReplicateErrorRate annotations: description: Thanos Replicate is failing to run, {{$value | humanize}}% of attempts failed. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosbucketreplicateerrorrate summary: Thanos Replicate is failing to run. expr: | ( sum by (job) (rate(thanosreplicatereplicationrunstotal{result="error", job=~".thanos-bucket-replicate."}[5m])) / on (job) group_left sum by (job) (rate(thanosreplicatereplicationrunstotal{job=~".thanos-bucket-replicate."}[5m])) ) * 100 >= 10 for: 5m labels: severity: critical alert: ThanosBucketReplicateRunLatency annotations: description: Thanos Replicate {{$labels.job}} has a 99th percentile latency of {{$value}} seconds for the replicate operations. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosbucketreplicaterunlatency summary: Thanos Replicate has a high latency for replicate operations. expr: | ( histogramquantile(0.99, sum by (job) (rate(thanosreplicatereplicationrundurationseconds_bucket{job=~".thanos-bucket-replicate."}[5m]))) > 20 and sum by (job) (rate(thanosreplicatereplicationrundurationsecondsbucket{job=~".thanos-bucket-replicate."}[5m])) > 0 ) for: 5m labels: severity: critical ``` ```yaml mdox-exec="cat examples/tmp/thanos-component-absent.yaml" name: thanos-component-absent rules: alert: ThanosCompactIsDown annotations: description: ThanosCompact has disappeared. Prometheus target for the component cannot be discovered. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanoscompactisdown summary: Thanos component has disappeared. expr: | absent(up{job=~".thanos-compact."} == 1) for: 5m labels: severity: critical alert: ThanosQueryIsDown annotations: description: ThanosQuery has disappeared. Prometheus target for the component cannot be discovered. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosqueryisdown summary: Thanos component has disappeared. expr: | absent(up{job=~".thanos-query."} == 1) for: 5m labels: severity: critical alert: ThanosReceiveIsDown annotations: description: ThanosReceive has disappeared. Prometheus target for the component cannot be discovered. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosreceiveisdown summary: Thanos component has disappeared. expr: | absent(up{job=~".thanos-receive."} == 1) for: 5m labels: severity: critical alert: ThanosRuleIsDown annotations: description: ThanosRule has disappeared. Prometheus target for the component cannot be discovered. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosruleisdown summary: Thanos component has disappeared. expr: | absent(up{job=~".thanos-rule."} == 1) for: 5m labels: severity: critical alert: ThanosSidecarIsDown annotations: description: ThanosSidecar has disappeared. Prometheus target for the component cannot be discovered. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanossidecarisdown summary: Thanos component has disappeared. expr: | absent(up{job=~".thanos-sidecar."} == 1) for: 5m labels: severity: critical alert: ThanosStoreIsDown annotations: description: ThanosStore has disappeared. Prometheus target for the component cannot be discovered. runbook_url: https://github.com/thanos-io/thanos/tree/main/mixin/runbook.md#alert-name-thanosstoreisdown summary: Thanos component has disappeared. expr: | absent(up{job=~".thanos-store."} == 1) for: 5m labels: severity: critical ```