Project: Grafana Mimir (Observability and Analysis / Observability)
File: configure-autoscaling.md

aliases: ../../operators-guide/deploying-grafana-mimir/jsonnet/configuring-autoscaling/ configuring-autoscaling/ ../../operators-guide/deploy-grafana-mimir/jsonnet/configure-autoscaling/ description: Learn how to configure Grafana Mimir autoscaling when using Jsonnet. menuTitle: Configure autoscaling title: Configure Grafana Mimir autoscaling with Jsonnet weight: 30 Mimir Jsonnet supports autoscaling for the following components: Autoscaling, which is based on Prometheus metrics and , uses Kubernetes Horizontal Pod Autoscaler (HPA). HPA is not configured directly in Jsonnet but it's created and updated by KEDA. KEDA is an operator, running in the Kubernetes cluster, which is responsible to simplify the setup of HPA with custom metrics (Prometheus in our case). KEDA is a Kubernetes operator aiming to simplify the wiring between HPA and Prometheus. Kubernetes HPA, out of the box, is not capable of autoscaling based on metrics scraped by Prometheus, but it allows to configure a custom metrics API server which proxies metrics from a data source (e.g. Prometheus) to Kubernetes. Setting up the custom metrics API server for Prometheus in a Kubernetes can be a tedious operation, so KEDA offers an operator to set it up automatically. KEDA supports proxying metrics for a variety of sources, including Prometheus. Runs an operator and an external metrics server. The metrics server supports proxying for many metric sources, including Prometheus. The operator watches for `ScaledObject` custom resource definition (CRD), defining the minimum and maximum replicas, and scaling trigger metrics of a Deployment or StatefulSet, and then configures the related HPA resource. You don't create the HPA resource in Kubernetes, but the operator creates it for you whenever a `ScaledObject` CRD is created (and keeps it updated for its whole lifecycle). Refers to for more information. The autoscaling of deployments is always managed by HPA, which is a native Kubernetes feature. KEDA, as we use it, never changes the number of replicas of Mimir Deployments or StatefulSets. However, if KEDA is not running successfully, there are consequences for Mimir autoscaling too: `keda-operator` is down (critical): as the operator is the single point of truth, it will not request the metrics when `keda-operator-metrics-apiserver` requests them. Changes to `ScaledObject` CRD will not be reflected to the HPA until the operator is back online. The deployment

queriers) will keep working but, if there is a surge of traffic, HPA will not be able to detect it due to a lack of metrics and so will not scale up. `keda-operator-metrics-apiserver` is down (critical): HPA is not able to fetch updated metrics and it will stop scaling the deployment until metrics will be back. The deployment (e.g. queriers) will keep working but, if there is a surge of traffic, HPA will not be able to detect it due to a lack of metrics and so will not scale up. `keda-admission-webhooks` is down (not critical): CRD validation will not be executed. Based on KEDA's configuration, this can block changes on CRDs until the service is restored. HPA functionality is not affected. {{< admonition type="note" >}} Use a KEDA configuration if autoscaling is critical for your use case. {{< /admonition >}} The fires if HPA is unable to scale the deployment for any reason (e.g. unable to scrape metrics from KEDA metrics API server). Refer to Kubernetes documentation to have a full understanding of how HPA works. The following Jsonnet configuration snippet shows an example of how to enable Mimir autoscaling with Jsonnet: ```jsonnet local mimir = import 'mimir/mimir.libsonnet'; mimir { _config+:: { autoscalingprometheusurl: 'http://prometheus.default:9090/prometheus', // Enable queriers autoscaling. autoscalingquerierenabled: true, autoscalingqueriermin_replicas: 10, autoscalingqueriermax_replicas: 40, // Enable ruler queriers autoscaling. autoscalingrulerquerier_enabled: true, autoscalingrulerquerierminreplicas: 10, autoscalingrulerqueriermaxreplicas: 40, // Enable distributor autoscaling. autoscalingdistributorenabled: true, autoscalingdistributormin_replicas: 10, autoscalingdistributormax_replicas: 40, } } ``` {{< admonition type="note" >}} The Mimir Jsonet doesn't install KEDA. To install KEDA, follow the instructions in . {{< /admonition >}} There are two options to disable autoscaling in a Mimir cluster: Set minimum replicas = maximum replicas. Decommission HPA. If KEDA and Kubernetes HPA work correctly but the HPA configuration (metric and threshold) are not giving the expected results (e.g. not scaling up when required), a simple solution to bypass the autoscaling algorithm is to set the minimum and maximum replicas to the same value. (e.g. `autoscalingquerierminreplicas: 40` and `autoscalingqueriermaxreplicas: 40`). To fully decommission HPA in a Mimir cluster you have to: Disable the relevant autoscaler (e.g. `autoscalingquerierenabled: false`). Manually set the expected number of replicas for the given Mimir component. The following example shows how to disable querier autoscaler and configure querier Deployment with 10 replicas: ```jsonnet local k = import 'github.com/grafana/jsonnet-libs/ksonnet-util/kausal.libsonnet'; local deployment = k.apps.v1.deployment; mimir { _config+:: { autoscalingquerierenabled: false, }, querier_deployment+: deployment.mixin.spec.withReplicas(10), } ```