Project: VictoriaMetrics (Observability and Analysis / Observability)
File: vmcluster.md

sort: 6 weight: 6 title: VMCluster menu: docs: parent: "operator-custom-resources" weight: 6 aliases: /operator/resources/vmcluster.html `VMCluster` represents a high-available and fault-tolerant version of VictoriaMetrics database. The `VMCluster` CRD defines a . For each `VMCluster` resource, the Operator creates: `VMStorage` as `StatefulSet`, `VMSelect` as `StatefulSet` and `VMInsert` as deployment. For `VMStorage` and `VMSelect` headless services are created. `VMInsert` is created as service with clusterIP. There is a strict order for these objects creation and reconciliation: `VMStorage` is synced - the Operator waits until all its pods are ready; Then it syncs `VMSelect` with the same manner; `VMInsert` is the last object to sync. All are created with update type. It allows to manually manage the rolling update process for Operator by deleting pods one by one and waiting for the ready status. Rolling update process may be configured by the operator env variables. The most important is `VM_PODWAITREADYTIMEOUT=80s` - it controls how long to wait for pod's ready status. You can see the full actual specification of the `VMCluster` resource in the . If you can't find necessary field in the specification of the custom resource, see . Also, you can check out the section. The cluster version provides a full set of high availability features - metrics replication, node failover, horizontal scaling. First, we recommend familiarizing yourself with the high availability tools provided by "VictoriaMetrics Cluster" itself: , , . `VMCluster` supports all listed in the above-mentioned articles parameters and features: `replicationFactor` - the number of replicas for each metric. for every component of cluster (`vmstorage` / `vmselect` / `vminsert`): `replicaCount` - the number of replicas for components of cluster. `affinity` - the affinity (the pod's scheduling constraints) for components pods. See more details in . `topologySpreadConstraints` - controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. See more details in . In addition, operator: uses k8s services or vmauth for load balancing between `vminsert` and `vmselect` components, uses health checks for to determine the readiness of components for work after restart, allows to horizontally scale all cluster components just by changing `replicaCount` field. Here is an example of a `VMCluster` resource with HA features: ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: example-vmcluster-persistent spec: replicationFactor: 2 vmstorage: replicaCount: 10 storageDataPath: "/vm-data" affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: labelSelector: matchExpressions: key: "app.kubernetes.io/name" operator: In values: "vmstorage" topologyKey: "kubernetes.io/hostname" storage: volumeClaimTemplate: spec: resources: requests: storage: 10Gi resources: limits: cpu: "2" memory: 2048Mi vmselect: replicaCount: 3 cacheMountPath: "/select-cache" affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: labelSelector: matchExpressions: key: "app.kubernetes.io/name" operator: In values: "vmselect" topologyKey: "kubernetes.io/hostname" storage: volumeClaimTemplate: spec: resources: requests: storage: 2Gi resources: limits: cpu: "1" memory: "500Mi" vminsert: replicaCount: 4 affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: labelSelector: matchExpressions: key: "app.kubernetes.io/name" operator: In values: "vminsert" topologyKey: "kubernetes.io/hostname" resources: limits: cpu: "1" memory: "500Mi" ``` For `VMCluster` you can specify tag name from and repository setting per cluster object: ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: example-vmcluster spec: vmstorage: replicaCount: 2 image: repository: victoriametrics/vmstorage tag:

pullPolicy: Always vmselect: replicaCount: 2 image: repository: victoriametrics/vmselect tag: v1.93.4-cluster pullPolicy: Always vminsert: replicaCount: 2 image: repository: victoriametrics/vminsert tag: v1.93.4-cluster pullPolicy: Always ``` Also, you can specify `imagePullSecrets` if you are pulling images from private repo, but `imagePullSecrets` is global setting for all `VMCluster` specification: ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: example-vmcluster spec: vmstorage: replicaCount: 2 image: repository: victoriametrics/vmstorage tag: v1.93.4-cluster pullPolicy: Always vmselect: replicaCount: 2 image: repository: victoriametrics/vmselect tag: v1.93.4-cluster pullPolicy: Always vminsert: replicaCount: 2 image: repository: victoriametrics/vminsert tag: v1.93.4-cluster pullPolicy: Always imagePullSecrets: name: my-repo-secret ``` You can specify resources for each component of `VMCluster` resource in the `spec` section of the `VMCluster` CRD. ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster-resources-example spec: vmstorage: resources: requests: memory: "16Gi" cpu: "4" limits: memory: "16Gi" cpu: "4" vmselect: resources: requests: memory: "16Gi" cpu: "4" limits: memory: "16Gi" cpu: "4" vminsert: resources: requests: memory: "16Gi" cpu: "4" limits: memory: "16Gi" cpu: "4" ``` If these parameters are not specified, then, by default all `VMCluster` pods have resource requests and limits from the default values of the following : `VMVMCLUSTERDEFAULTVMSTORAGEDEFAULTRESOURCELIMIT_MEM` - default memory limit for `VMCluster/vmstorage` pods, `VMVMCLUSTERDEFAULTVMSTORAGEDEFAULTRESOURCELIMIT_CPU` - default memory limit for `VMCluster/vmstorage` pods, `VMVMCLUSTERDEFAULTVMSTORAGEDEFAULTRESOURCEREQUEST_MEM` - default memory limit for `VMCluster/vmstorage` pods, `VMVMCLUSTERDEFAULTVMSTORAGEDEFAULTRESOURCEREQUEST_CPU` - default memory limit for `VMCluster/vmstorage` pods, `VMVMCLUSTERDEFAULTVMSELECTDEFAULTRESOURCELIMIT_MEM` - default memory limit for `VMCluster/vmselect` pods, `VMVMCLUSTERDEFAULTVMSELECTDEFAULTRESOURCELIMIT_CPU` - default memory limit for `VMCluster/vmselect` pods, `VMVMCLUSTERDEFAULTVMSELECTDEFAULTRESOURCEREQUEST_MEM` - default memory limit for `VMCluster/vmselect` pods, `VMVMCLUSTERDEFAULTVMSELECTDEFAULTRESOURCEREQUEST_CPU` - default memory limit for `VMCluster/vmselect` pods, `VMVMCLUSTERDEFAULTVMINSERTDEFAULTRESOURCELIMIT_MEM` - default memory limit for `VMCluster/vmselect` pods, `VMVMCLUSTERDEFAULTVMINSERTDEFAULTRESOURCELIMIT_CPU` - default memory limit for `VMCluster/vmselect` pods, `VMVMCLUSTERDEFAULTVMINSERTDEFAULTRESOURCEREQUEST_MEM` - default memory limit for `VMCluster/vmselect` pods, `VMVMCLUSTERDEFAULTVMINSERTDEFAULTRESOURCEREQUEST_CPU` - default memory limit for `VMCluster/vmselect` pods. These default parameters will be used if: `VMVMCLUSTERDEFAULTUSEDEFAULTRESOURCES` is set to `true` (default value), `VMCluster/*` CR doesn't have `resources` field in `spec` section. Field `resources` in `VMCluster/*` spec have higher priority than operator parameters. If you set `VMVMCLUSTERDEFAULTUSEDEFAULTRESOURCES` to `false` and don't specify `resources` in `VMCluster/*` CRD, then `VMCluste/*r` pods will be created without resource requests and limits. Also, you can specify requests without limits - in this case default values for limits will not be used. VMCluster supports following features from : VMCluster doesn't support yet feature . For using Enterprise version of you need to change version of `VMCluster` to version with `-enterprise` suffix using . All the enterprise apps require `-eula` command-line flag to be passed to them. This flag acknowledges that your usage fits one of the cases listed on . So you can use for passing this flag to `VMCluster`. After that you can pass flag to `VMCluster/vmselect` and `VMCluster/vmstorage` with too. Here are complete example for : ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster-ent-example spec: vmselect: image: tag: v1.93.5-enterprise-cluster extraArgs: eula: true downsampling.period: 30d:5m,180d:1h,1y:6h,2y:1d vmstorage: image: tag: v1.93.5-enterprise-cluster extraArgs: eula: true downsampling.period: 30d:5m,180d:1h,1y:6h,2y:1d ``` You can pass flag to `VMCluster/vmstorage` with . Here are complete example for : ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster-ent-example spec: vmstorage: image: tag:

extraArgs: eula: true retentionFilter: '{vmaccountid="5",env="dev"}:5d,{vmaccountid="5",env="prod"}:5y' ``` For using you only need to and operator will automatically create for cluster components. ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster-ent-example spec: vmselect: image: tag: v1.93.5-enterprise-cluster extraArgs: eula: true vminsert: image: tag: v1.93.5-enterprise-cluster extraArgs: eula: true vmstorage: image: tag: v1.93.5-enterprise-cluster extraArgs: eula: true ``` After that will automatically scrape for cluster components. You can pass flags to `VMCluster/vmstorage`, `VMCluster/vmselect` and `VMCluster/vminsert` with and mount secret files with `extraVolumes` and `extraVolumeMounts` fields. Here are complete example for ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster-ent-example spec: vmselect: image: tag: v1.93.5-enterprise-cluster extraArgs: eula: true cluster.tls: true cluster.tlsCAFile: /etc/mtls/ca.crt cluster.tlsCertFile: /etc/mtls/vmselect.crt cluster.tlsKeyFile: /etc/mtls/vmselect.key extraVolumes: name: mtls secret: secretName: mtls extraVolumeMounts: name: mtls mountPath: /etc/mtls vminsert: image: tag: v1.93.5-enterprise-cluster extraArgs: eula: true cluster.tls: true cluster.tlsCAFile: /etc/mtls/ca.crt cluster.tlsCertFile: /etc/mtls/vminsert.crt cluster.tlsKeyFile: /etc/mtls/vminsert.key extraVolumes: name: mtls secret: secretName: mtls extraVolumeMounts: name: mtls mountPath: /etc/mtls vmstorage: image: tag: v1.93.5-enterprise-cluster env: name: POD valueFrom: fieldRef: fieldPath: metadata.name extraArgs: eula: true cluster.tls: true cluster.tlsCAFile: /etc/mtls/ca.crt cluster.tlsCertFile: /etc/mtls/$(POD).crt cluster.tlsKeyFile: /etc/mtls/$(POD).key extraVolumes: name: mtls secret: secretName: mtls extraVolumeMounts: name: mtls mountPath: /etc/mtls apiVersion: v1 kind: Secret metadata: name: mtls namespace: default stringData: ca.crt: | --BEGIN CERTIFICATE-- ... --END CERTIFICATE-- mtls-vmstorage-0.crt: | --BEGIN CERTIFICATE-- ... --END CERTIFICATE-- mtls-vmstorage-0.key: | --BEGIN PRIVATE KEY-- ... --END PRIVATE KEY-- mtls-vmstorage-1.crt: | --BEGIN CERTIFICATE-- ... --END CERTIFICATE-- mtls-vmstorage-1.key: | --BEGIN PRIVATE KEY-- ... --END PRIVATE KEY-- vminsert.crt: | --BEGIN CERTIFICATE-- ... --END CERTIFICATE-- vminsert.key: | --BEGIN PRIVATE KEY-- ... --END PRIVATE KEY-- vmselect.crt: | --BEGIN CERTIFICATE-- ... --END CERTIFICATE-- vmselect.key: | --BEGIN PRIVATE KEY-- ... --END PRIVATE KEY-- ``` Example commands for generating certificates you can read on . You can check for backup automation. It contains a description of the service and its features. This section covers vmbackumanager integration in vmoperator. `VMCluster` has built-in backup configuration, it uses `vmbackupmanager` - proprietary tool for backups. It supports incremental backups (hourly, daily, weekly, monthly) with popular object storages (aws s3, google cloud storage). Here is a complete example for backup configuration: ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster-ent-example spec: vmstorage: vmBackup: acceptEULA: true destination: "s3://your_bucket/folder" credentialsSecret: name: remote-storage-keys key: credentials apiVersion: v1 kind: Secret metadata: name: remote-storage-keys type: Opaque stringData: credentials: |- [default] awsaccesskeyid = youraccesskeyid awssecretaccesskey = yoursecretaccesskey ``` NOTE: for cluster version operator adds suffix for destination: `"s3://yourbucket/folder"`, it becomes `"s3://yourbucket/folder/$(POD_NAME)"`. It's needed to make consistent backups for each storage node. You can read more about backup configuration options and mechanics Possible configuration options for backup crd can be found at Using VMBackupmanager for restoring backups in Kubernetes environment is described . Also see VMCLuster example spec . ```yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster-example-minimal spec: retentionPeriod: "1" vmstorage: replicaCount: 2 vmselect: replicaCount: 2 vminsert: replicaCount: 2 ``` ```yaml kind: VMCluster metadata: name: vmcluster-example-persistent spec: retentionPeriod: "4" replicationFactor: 2 vmstorage: replicaCount: 2 storageDataPath: "/vm-data" storage: volumeClaimTemplate: spec: storageClassName: standard resources: requests: storage: 10Gi resources: limits: cpu: "0.5" memory: 500Mi vmselect: replicaCount: 2 cacheMountPath: "/select-cache" storage: volumeClaimTemplate: spec: resources: requests: storage: 2Gi resources: limits: cpu: "0.3" memory: "300Mi" vminsert: replicaCount: 2 ```