Project: external-secrets (Provisioning / Security & Compliance)
File: metrics.md

hide: toc The External Secrets Operator exposes its Prometheus metrics in the `/metrics` path. To enable it, set the `serviceMonitor.enabled` Helm flag to `true`. If you are using a different monitoring tool that also needs a `/metrics` endpoint, you can set the `metrics.service.enabled` Helm flag to `true`. In addition you can also set `webhook.metrics.service.enabled` and `certController.metrics.service.enabled` to scrape the other components. The Operator has plus some custom metrics with a resource name prefix, such as `externalsecret_`. | Name | Type | Description | |--|-|| | `clusterexternalsecretstatuscondition` | Gauge | The status condition of a specific Cluster External Secret | | `clusterexternalsecretreconcileduration` | Gauge | The duration time to reconcile the Cluster External Secret | | Name | Type | Description | ||--|-| | `externalsecretproviderapicallscount` | Counter | Number of API calls made to an upstream secret provider API. The metric provides a `provider`, `call` and `status` labels. | | `externalsecretsynccalls_total` | Counter | Total number of the External Secret sync calls | | `externalsecretsynccalls_error` | Counter | Total number of the External Secret sync errors | | `externalsecretstatuscondition` | Gauge | The status condition of a specific External Secret | | `externalsecretreconcileduration` | Gauge | The duration time to reconcile the External Secret | | Name | Type | Description | |--|-|| | `clustersecretstorestatuscondition` | Gauge | The status condition of a specific Cluster Secret Store | | `clustersecretstorereconcileduration` | Gauge | The duration time to reconcile the Cluster Secret Store | | Name | Type | Description | |-|-|-| | `secretstorestatuscondition` | Gauge | The status condition of a specific Secret Store | | `secretstorereconcileduration` | Gauge | The duration time to reconcile the Secret Store | See on the default exported metrics by controller-runtime. We provide a that gives you an overview of External Secrets Operator: We find the following Service Level Indicators (SLIs) useful when operating ESO. They should give you a good starting point and hints to develop your own Service Level Objectives (SLOs). The webhook HTTP status code indicates that a HTTP Request was answered successfully or not. If the Webhook pod is not able to serve the requests properly then that failure may cascade down to the controller or any other user of `kube-apiserver`. SLI Example: request error percentage. ``` sum(increase(controllerruntimewebhookrequeststotal{service=~"external-secrets.*",code="500"}[1m])) / sum(increase(controllerruntimewebhookrequeststotal{service=~"external-secrets.*"}[1m])) ``` If the webhook server is not able to respond in time then that may cause a timeout at the client. This failure may cascade down to the controller or any other user of `kube-apiserver`. SLI Example: p99 across all webhook requests. ``` histogram_quantile(0.99, sum(rate(controllerruntimewebhooklatencyseconds_bucket{service=~"external-secrets.*"}[5m])) by (le) ) ``` If the workqueue depth is > 0 for a longer period of time then this is an indicator for the controller not being able to reconcile resources in time. I.e. delivery of secret updates is delayed. Note: when a controller is restarted, then `queue length = total number of resources`. Make sure to measure the time it takes for the controller to fully reconcile all secrets after a restart. In large clusters this may take a while, make sure to define an acceptable timeframe to fully reconcile all resources. ``` sum( workqueue_depth{service=~"external-secrets.*"} ) by (name) ``` The controller should be able to reconcile resources within a reasonable timeframe. When latency is high secret delivery may impacted. SLI Example: p99 across all controllers. ``` histogram_quantile(0.99, sum(rate(controllerruntimereconciletimeseconds_bucket{service=~"external-secrets.*"}[5m])) by (le) ) ``` The controller should be able to reconcile resources without errors. When errors occurr secret delivery may be impacted which could cascade down to the secret consuming applications. ``` sum(increase( controllerruntimereconcile_total{service=~"external-secrets.*",controller=~"$controller",result="error"}[1m]) ) by (result) ```