Project: Grafana Mimir (Observability and Analysis / Observability)
File: migrate-from-thanos-to-mimir-with-thanos-sidecar.md

aliases: ../../migrate/migrate-from-thanos-to-mimir-with-thanos-sidecar/ description: Learn how to migrate from Thanos to Mimir using Thanos sidecar. menuTitle: Migrate from Thanos using Thanos sidecar title: Migrate from Thanos to Mimir using Thanos sidecar weight: 15 As an operator, you can migrate a deployment of Thanos to Grafana Mimir by using Thanos sidecar to move metrics over with a stepped workflow. An option when migrating is to allow Thanos to query Mimir. This way you retain historical data via your existing Thanos deployment while pointing all of your Prometheus servers (or grafana agents and other metric sources) to Mimir. The overall setup consists of setting up Thanos Sidecar alongside Mimir and then pointing Thanos Query to the sidecar as if it was just a normal sidecar. This is not so much of a guide as it is a collection of configurations that have been used with success prior. Warning: This setup is unsupported, experimental and has not been battle tested. Your mileage may vary, ensure you have tested this for your use case and have made appropriate backups (and tested your procedures for recovery) There are very few obstacles to get this working properly. Thanos Sidecar is designed to work with the Prometheus API which Mimir (almost) implements. There are only 2 endpoints that are not implemented that we need to "spoof" to get Thanos sidecar to believe it is connecting to a Prometheus server: `/api/v1/status/buildinfo` and `/api/v1/status/config`. We spoof these endpoints using NGINX (more details later). The only other roadblock is the requirement for requests to Mimir to contain the `X-Scope-Org-Id` header to identify the Tenant. We inject this header using another NGINX container sitting in between Thanos Sidecar and Mimir. In our case, everything was being deployed in Kubernetes. Mimir was deployed using the `mimir-distributed` helm chart. Therefore configurations shown will be Kubernetes manifests that will need to be modified for your environment. If you are not using Kubernetes then you will need to set up the appropriate configuration for NGINX to implement the technical setup described above. You may use the configurations below for inspiration but they will not work out of the box. We deployed the sidecar using Kubernetes using the below manifest: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: thanos-sidecar namespace: mimir labels: app: thanos-sidecar spec: replicas: 1 selector: matchLabels: app: thanos-sidecar template: metadata: labels: app: thanos-sidecar spec: volumes: name: config configMap: name: sidecar-nginx defaultMode: 420 containers: name: nginx image: nginxinc/nginx-unprivileged:1.19-alpine ports: name: http containerPort: 8080 protocol: TCP volumeMounts: name: config mountPath: /etc/nginx terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: IfNotPresent name: thanos-sidecar image: quay.io/thanos/thanos:v0.26.0 args: sidecar "--prometheus.url=http://localhost:8080/prometheus" "--grpc-address=:10901" "--http-address=:10902" "--log.level=info"

ports: name: http containerPort: 10902 protocol: TCP name: grpc containerPort: 10901 protocol: TCP restartPolicy: Always terminationGracePeriodSeconds: 30 dnsPolicy: ClusterFirst schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 25% maxSurge: 25% revisionHistoryLimit: 10 progressDeadlineSeconds: 600 ``` ```yaml apiVersion: v1 kind: Service metadata: name: thanos-sidecar namespace: mimir labels: app: thanos-sidecar spec: ports: name: 10901-10901 protocol: TCP port: 10901 targetPort: 10901 selector: app: thanos-sidecar type: ClusterIP ``` Notice that there is an NGINX container within the sidecar deployment definition. This NGINX instance sits between the sidecar and Mimir (ie. `sidecar --> nginx --> mimir`) it is responsible for injecting the tenant ID into the requests from the sidecar to Mimir. The configuration that this NGINX instance uses is as follows: ```conf worker_processes 5; ## Default: 1 error_log /dev/stderr; pid /tmp/nginx.pid; workerrlimitnofile 8192; events { worker_connections 4096; ## Default: 1024 } http { clientbodytemppath /tmp/clienttemp; proxytemppath /tmp/proxytemppath; fastcgitemppath /tmp/fastcgi_temp; uwsgitemppath /tmp/uwsgi_temp; scgitemppath /tmp/scgi_temp; default_type application/octet-stream; logformat main '$remoteaddr - $remoteuser [$timelocal] $status ' '"$request" $bodybytessent "$http_referer" ' '"$httpuseragent" "$httpxforwarded_for"'; access_log /dev/stderr main; sendfile on; tcp_nopush on; resolver kube-dns.kube-system.svc.cluster.local; server { listen 8080; location / { proxysetheader X-Scope-OrgID 1; proxypass http://mimir-distributed-nginx.mimir.svc.cluster.local:80$requesturi; } } } ``` Now we modified the Mimir Distributed NGINX configuration to allow the sidecar to fetch the "external labels" and the "version" of our "Prometheus" server. You should be careful with the external labels to make sure that you don't override any existing labels (in our case we used "source='mimir'"). You should also consider existing dashboards and make sure that they will continue to work as expected. Make sure to test your setup first. In particular, these location blocks were added (full config further down): ```conf location /prometheus/api/v1/status/config { add_header Content-Type application/json; return 200 "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n external_labels:\\n source: mimir\"}}"; } location /prometheus/api/v1/status/buildinfo { add_header Content-Type application/json; return 200 "{\"status\":\"success\",\"data\":{\"version\":\"2.35.0\",\"revision\":\"6656cd29fe6ac92bab91ecec0fe162ef0f187654\",\"branch\":\"HEAD\",\"buildUser\":\"root@cf6852b14d68\",\"buildDate\":\"20220421-09:53:42\",\"goVersion\":\"go1.18.1\"}}"; } ``` You need to modify the first one to set your external labels. The second one just allows Thanos to detect the version of Prometheus running. To modify the external labels alter this string: `"{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n externallabels:\\n source: mimir\"}}"`. You will notice it is escaped JSON. Simply add elements to the `data.externallabels` field to add more external labels as required or alter the existing key to modify them. ```conf worker_processes 5; ## Default: 1 error_log /dev/stderr; pid /tmp/nginx.pid; workerrlimitnofile 8192; events { worker_connections 4096; ## Default: 1024 } http { clientbodytemppath /tmp/clienttemp; proxytemppath /tmp/proxytemppath; fastcgitemppath /tmp/fastcgi_temp; uwsgitemppath /tmp/uwsgi_temp; scgitemppath /tmp/scgi_temp; default_type application/octet-stream; logformat main '$remoteaddr - $remoteuser [$timelocal] $status ' '"$request" $bodybytessent "$http_referer" ' '"$httpuseragent" "$httpxforwarded_for"'; access_log /dev/stderr main; sendfile on; tcp_nopush on; resolver kube-dns.kube-system.svc.cluster.local; server { listen 8080; location = / { return 200 'OK'; auth_basic off; } location /distributor { proxypass http://mimir-distributed-distributor.mimir.svc.cluster.local:8080$requesturi; } location = /api/v1/push { proxypass http://mimir-distributed-distributor.mimir.svc.cluster.local:8080$requesturi; } location /alertmanager { proxypass http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$requesturi; } location = /multitenant_alertmanager/status { proxypass http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$requesturi; } location = /api/v1/alerts { proxypass http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$requesturi; } location /prometheus/config/v1/rules { proxypass http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$requesturi; } location /prometheus/api/v1/rules { proxypass http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$requesturi; } location /api/v1/rules { proxypass http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$requesturi; } location /prometheus/api/v1/alerts { proxypass http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$requesturi; } location /prometheus/rules { proxypass http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$requesturi; } location = /ruler/ring { proxypass http://mimir-distributed-ruler.mimir.svc.cluster.local:8080$requesturi; } location /prometheus/api/v1/status/config { add_header Content-Type application/json; return 200 "{\"status\":\"success\",\"data\":{\"yaml\": \"global:\\n external_labels:\\n source: mimir\"}}"; } location /prometheus/api/v1/status/buildinfo { add_header Content-Type application/json; return 200 "{\"status\":\"success\",\"data\":{\"version\":\"2.35.0\",\"revision\":\"6656cd29fe6ac92bab91ecec0fe162ef0f187654\",\"branch\":\"HEAD\",\"buildUser\":\"root@cf6852b14d68\",\"buildDate\":\"20220421-09:53:42\",\"goVersion\":\"go1.18.1\"}}"; } location /prometheus { proxypass http://mimir-distributed-query-frontend.mimir.svc.cluster.local:8080$requesturi; } location = /api/v1/status/buildinfo { proxypass http://mimir-distributed-query-frontend.mimir.svc.cluster.local:8080$requesturi; } } } ```