Project: Grafana Mimir (Observability and Analysis / Observability)
File: migrate-from-consul-to-memberlist.md

aliases: ../../operators-guide/deploying-grafana-mimir/jsonnet/migrating-from-consul-to-memberlist/ migrating-from-consul-to-memberlist/ ../../operators-guide/deploy-grafana-mimir/jsonnet/migrate-from-consul-to-memberlist/ description: Learn how to migrate from using Consul as KV store for hash rings to using memberlist without any downtime. menuTitle: Migrate from Consul to memberlist title: Migrate from Consul to memberlist KV store for hash rings without downtime weight: 40 Mimir Jsonnet uses memberlist as KV store for hash rings since Mimir 2.2.0. Memberlist can be disabled by using the following configuration: ```jsonnet { _config+:: { memberlistringenabled: false } } ``` If you are running Mimir hash rings with Consul and would like to migrate to memberlist without any downtime, you can follow instructions in this document. ```jsonnet { _config+:: { memberlistringenabled: true, multikvmigrationenabled: true, } } ``` Step 1 configures components to use `multi` KV store, with `consul` as primary and memberlist as secondary stores. This step requires rollout of all Mimir components. After applying this step all Mimir components will expose page on HTTP admin interface, which can be used to check health of memberlist cluster. ```jsonnet { _config+:: { memberlistringenabled: true, multikvmigrationenabled: true, multikvmirrorenabled: true, // Changed in this step. } } ``` In this step we enable writes to primary KV store (Consul) to be mirrored into secondary store (memberlist). Applying this change will not cause restart of Mimir components. You can monitor following metrics to check if mirroring was enabled on all components and if it works correctly: `cortexmultikvmirror_enabled` shows which components have KV store mirroring enabled. All Mimir components should start mirroring to secondary KV store reloading runtime configuration. `rate(cortexmultikvmirrorwritestotal[1m])` shows rate of writes to secondary KV store in writes per second. `rate(cortexmultikvmirrorwriteerrors_total[1m])` shows rate of write errors to secondary KV store, in errors per second. After mirroring is enabled, you should see a key for each Mimir hash ring in the admin page. See . ```jsonnet { _config+:: { memberlistringenabled: true, multikvmigrationenabled: true, multikvmirrorenabled: true, multikvswitchprimary_secondary: true, // Changed in this

} } ``` This change will switch primary and secondary stores as used by `multi` KV. From this point on Mimir components will use memberlist as primary KV store, and they will mirror updates to Consul. This step does not require restart of Mimir components. To see if all components started to use memberlist as primary store, please watch `cortexmultikvprimary_store` metric. ```jsonnet { _config+:: { memberlistringenabled: true, multikvmigrationenabled: true, multikvmirrorenabled: false, // Changed in this step. multikvswitchprimary_secondary: true, } } ``` This step does not require restart of any Mimir component. After applying the change components will stop writing ring updates to Consul, and will only use memberlist. You can watch `cortexmultikvmirror_enabled` metric to see if all components have picked up updated configuration. ```jsonnet { _config+:: { memberlistringenabled: true, multikvmigrationenabled: false, // Changed in this step. multikvmirrorenabled: false, multikvswitchprimary_secondary: true, multikvmigrationteardown: true, // Added in this step. } } ``` This configuration change will cause a new rollout of all components. After the restart components will no longer use `multi` KV store and will be configured to use memberlist only. We use `multikvmigrationteardown` to preserve runtime configuration for `multi` KV store for components that haven't restarted yet. All `cortexmultikv*` metrics are only exposed by components that use `multi` KV store. As components restart, these metrics will disappear. {{< admonition type="note" >}} Setting `multikvmigrationenabled: false` while keeping `memberlistringenabled: true` removes the Consul workload. That's expected, since Consul isn't used anymoreyou disabled mirroring to it in step 4. {{< /admonition >}} If you need to keep consul running, you can explicitly set `consulenabled: true` in `config`. We have successfully migrated Mimir cluster from using Consul to memberlist without any downtime! As a final step, we can remove all migration-related config options: `multikvmigrationenabled` `multikvmirrorenabled` `multikvswitchprimary_secondary` `multikvmigrationteardown` Our final memberlist configuration will be: ```jsonnet { _config+:: { memberlistringenabled: true, } } ``` This will not trigger new restart of Mimir components. After applying this change, you are finished.