Project: Thanos (Observability and Analysis / Observability)
File: modify-objstore-data.md

For operational purposes, there are some use cases to manipulate data in the object storage. For example, delete some high cardinality metrics or relabel metrics if needed. This is already possible via the bucket rewrite tool. ```shell thanos tools bucket rewrite --rewrite.to-delete-config-file config.yaml --objstore.config-file objstore.yaml --id <block ID> ``` This is the example command to delete some data in the specified TSDB block from your object store bucket. For example, if `k8sappmetric37` is the metric you want to delete, then the config file `config.yaml` would be: ```yaml matchers: '{name="k8sappmetric37"}' ``` Example output from my mac looks like below. Dry run mode is enabled by default to prevent unexpected series from being deleted. A changelog file is generated so that you can check the expected modification of the provided deletion request. ```shell thanos tools bucket rewrite --rewrite.to-delete-config-file config.yaml --objstore.config-file ~/local-bucket-config.yaml --id 01FET1EK9BC3E0QD4886RQCM8K level=info ts=2021-09-25T05:47:14.87316Z caller=factory.go:49 msg="loading bucket configuration" level=info ts=2021-09-25T05:47:14.875365Z caller=tools_bucket.go:1078 msg="downloading block" source=01FET1EK9BC3E0QD4886RQCM8K level=info ts=2021-09-25T05:47:14.887816Z caller=tools_bucket.go:1115 msg="changelog will be available" file=/var/folders/ny/yy113mqs6szcpjy2qrnhq9rh0000gq/T/thanos-rewrite/01FGDQWKJ7H29B3V4HCQ691WN9/change.log level=info ts=2021-09-25T05:47:14.912544Z caller=toolsbucket.go:1130 msg="starting rewrite for block" source=01FET1EK9BC3E0QD4886RQCM8K new=01FGDQWKJ7H29B3V4HCQ691WN9 toDelete="- matchers: '{name=\"k8sapp_metric37\"}'\n" toRelabel= level=info ts=2021-09-25T05:47:14.919438Z caller=compactor.go:41 msg="processed 10.00% of 15000 series" level=info ts=2021-09-25T05:47:14.925442Z caller=compactor.go:41 msg="processed 20.00% of 15000 series" level=info ts=2021-09-25T05:47:14.930263Z caller=compactor.go:41 msg="processed 30.00% of 15000 series" level=info ts=2021-09-25T05:47:14.934325Z caller=compactor.go:41 msg="processed 40.00% of 15000 series" level=info ts=2021-09-25T05:47:14.939466Z caller=compactor.go:41 msg="processed 50.00% of 15000 series" level=info ts=2021-09-25T05:47:14.944513Z caller=compactor.go:41 msg="processed 60.00% of 15000 series" level=info ts=2021-09-25T05:47:14.950254Z caller=compactor.go:41 msg="processed 70.00% of 15000 series" level=info ts=2021-09-25T05:47:14.955336Z caller=compactor.go:41 msg="processed 80.00% of 15000 series" level=info ts=2021-09-25T05:47:14.960193Z caller=compactor.go:41 msg="processed 90.00% of 15000 series" level=info ts=2021-09-25T05:47:14.964705Z caller=compactor.go:41 msg="processed 100.00% of 15000 series" level=info ts=2021-09-25T05:47:14.964768Z caller=tools_bucket.go:1136 msg="dry run finished. Changes should be printed to stderr" level=info ts=2021-09-25T05:47:14.965101Z caller=main.go:160 msg=exiting ``` Below is an example output of the changelog. All the series that match the given deletion config will be deleted. The last column `[{1630713615001 1630715400001}]` represents the start and end time of the series. ```shell cat

Deleted {blockgen_target="1", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="1", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="1", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Deleted {blockgen_target="10", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="10", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="10", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Deleted {blockgen_target="100", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="100", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="100", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Deleted {blockgen_target="11", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="11", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="11", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Deleted {blockgen_target="12", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="12", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="12", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Deleted {blockgen_target="13", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="13", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="13", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] ... ``` If the changelog output is expected, then we can use the same command in the first step, but with `--no-dry-run` flag to actually delete the data we want. ```shell thanos tools bucket rewrite --no-dry-run --rewrite.to-delete-config-file config.yaml --objstore.config-file objstore.yaml --id <block ID> ``` The output is listed below. ```shell thanos tools bucket rewrite --no-dry-run --rewrite.to-delete-config-file config.yaml --objstore.config-file ~/local-bucket-config.yaml --id 01FET1EK9BC3E0QD4886RQCM8K level=info ts=2021-09-25T05:59:18.05232Z caller=factory.go:49 msg="loading bucket configuration" level=info ts=2021-09-25T05:59:18.059056Z caller=tools_bucket.go:1078 msg="downloading block" source=01FET1EK9BC3E0QD4886RQCM8K level=info ts=2021-09-25T05:59:18.074761Z caller=tools_bucket.go:1115 msg="changelog will be available" file=/var/folders/ny/yy113mqs6szcpjy2qrnhq9rh0000gq/T/thanos-rewrite/01FGDRJNST2EYDY2RKWFZJPGWJ/change.log level=info ts=2021-09-25T05:59:18.108293Z caller=toolsbucket.go:1130 msg="starting rewrite for block" source=01FET1EK9BC3E0QD4886RQCM8K new=01FGDRJNST2EYDY2RKWFZJPGWJ toDelete="- matchers: '{name=\"k8sapp_metric37\"}'\n" toRelabel= level=info ts=2021-09-25T05:59:18.395253Z caller=compactor.go:41 msg="processed 10.00% of 15000 series" level=info ts=2021-09-25T05:59:18.406416Z caller=compactor.go:41 msg="processed 20.00% of 15000 series" level=info ts=2021-09-25T05:59:18.419826Z caller=compactor.go:41 msg="processed 30.00% of 15000 series" level=info ts=2021-09-25T05:59:18.428238Z caller=compactor.go:41 msg="processed 40.00% of 15000 series" level=info ts=2021-09-25T05:59:18.436017Z caller=compactor.go:41 msg="processed 50.00% of 15000 series" level=info ts=2021-09-25T05:59:18.444738Z caller=compactor.go:41 msg="processed 60.00% of 15000 series" level=info ts=2021-09-25T05:59:18.452328Z caller=compactor.go:41 msg="processed 70.00% of 15000 series" level=info ts=2021-09-25T05:59:18.465218Z caller=compactor.go:41 msg="processed 80.00% of 15000 series" level=info ts=2021-09-25T05:59:18.477385Z caller=compactor.go:41 msg="processed 90.00% of 15000 series" level=info ts=2021-09-25T05:59:18.485254Z caller=compactor.go:41 msg="processed 100.00% of 15000 series" level=info ts=2021-09-25T05:59:18.485296Z caller=tools_bucket.go:1140 msg="wrote new block after modifications; flushing" source=01FET1EK9BC3E0QD4886RQCM8K new=01FGDRJNST2EYDY2RKWFZJPGWJ level=info ts=2021-09-25T05:59:18.662059Z caller=tools_bucket.go:1149 msg="uploading new block" source=01FET1EK9BC3E0QD4886RQCM8K new=01FGDRJNST2EYDY2RKWFZJPGWJ level=info ts=2021-09-25T05:59:18.667883Z caller=tools_bucket.go:1159 msg=uploaded source=01FET1EK9BC3E0QD4886RQCM8K new=01FGDRJNST2EYDY2RKWFZJPGWJ level=info ts=2021-09-25T05:59:18.667921Z caller=tools_bucket.go:1167 msg="rewrite done" IDs=01FET1EK9BC3E0QD4886RQCM8K level=info ts=2021-09-25T05:59:18.668136Z caller=main.go:160 msg=exiting ``` After rewriting, a new block `01FGDRJNST2EYDY2RKWFZJPGWJ` will be uploaded to your object store bucket. However, the old block will not be deleted by default for the reason of safety. You can add `--delete-blocks` flag so that the source block will be marked as deletion after rewrite is done and will be deleted automatically if you have a compactor running against that bucket. Multiple matchers can be added in the deletion config. For example, the config file below specifies deletion for all series that match: metric name `k8sappmetric1` metric name `k8sappmetric37` and label `blockgen_target` that regexp matched `7.*` ```yaml matchers: '{name="k8sappmetric37", blockgen_target=~"7.*"}' matchers: '{name="k8sappmetric1"}' ``` It is also possible to specify a time range in case you do not want to delete the matching series from the entire block. The interval is specified via millisecond timestamps `mint` and

```yaml matchers: '{name="k8sappmetric1"}' intervals: mint: 1663279200000 maxt: 1663292160000 ``` For reference, you can convert the dates via GNU `date(1)`: ```shell $ TZ=UTC date --date='2022-09-15 22:00:00' '+%s' 1663279200 $ TZ=UTC date --date='2022-09-16 01:36:00' '+%s' 1663292160 $ TZ=UTC date -d @1663279200 --iso=seconds 2022-09-15T22:00:00+00:00 $ TZ=UTC date -d @1663292160 --iso=seconds 2022-09-16T01:36:00+00:00 ``` ```shell thanos tools bucket rewrite --rewrite.to-relabel-config-file config.yaml --objstore.config-file objstore.yaml --id <block ID> ``` Series relabeling is needed when you want to rename your metrics or drop some high cardinality labels. The command is similar to rewrite deletion, but with `--rewrite.to-relabel-config-file` flag. The configuration is the same as . For example, the relabel config file does: delete all series that match `{name="k8sappmetric37"}` rename `k8sappmetric38` to `old_metric` ```yaml action: drop regex: k8sappmetric37 sourcelabels: [name_] action: replace sourcelabels: [name_] regex: k8sappmetric38 targetlabel: name_ replacement: old_metric ``` Example output of the changelog: ```shell Deleted {blockgen_target="1", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="1", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="1", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Relabelled {blockgen_target="1", name="k8sappmetric38", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} {blockgen_target="1", name="oldmetric", nextrollout_time="2021-09-03 23:30:00 +0000 UTC"} Relabelled {blockgen_target="1", name="k8sappmetric38", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} {blockgen_target="1", name="oldmetric", nextrollout_time="2021-09-04 00:30:00 +0000 UTC"} Relabelled {blockgen_target="1", name="k8sappmetric38", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} {blockgen_target="1", name="oldmetric", nextrollout_time="2021-09-04 01:30:00 +0000 UTC"} Deleted {blockgen_target="10", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="10", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="10", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Relabelled {blockgen_target="10", name="k8sappmetric38", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} {blockgen_target="10", name="oldmetric", nextrollout_time="2021-09-03 23:30:00 +0000 UTC"} Relabelled {blockgen_target="10", name="k8sappmetric38", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} {blockgen_target="10", name="oldmetric", nextrollout_time="2021-09-04 00:30:00 +0000 UTC"} Relabelled {blockgen_target="10", name="k8sappmetric38", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} {blockgen_target="10", name="oldmetric", nextrollout_time="2021-09-04 01:30:00 +0000 UTC"} Deleted {blockgen_target="100", name="k8sappmetric37", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} [{1630713615001 1630715400001}] Deleted {blockgen_target="100", name="k8sappmetric37", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} [{1630715415000 1630719015000}] Deleted {blockgen_target="100", name="k8sappmetric37", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} [{1630719015000 1630720815000}] Relabelled {blockgen_target="100", name="k8sappmetric38", nextrollouttime="2021-09-03 23:30:00 +0000 UTC"} {blockgen_target="100", name="oldmetric", nextrollout_time="2021-09-03 23:30:00 +0000 UTC"} Relabelled {blockgen_target="100", name="k8sappmetric38", nextrollouttime="2021-09-04 00:30:00 +0000 UTC"} {blockgen_target="100", name="oldmetric", nextrollout_time="2021-09-04 00:30:00 +0000 UTC"} Relabelled {blockgen_target="100", name="k8sappmetric38", nextrollouttime="2021-09-04 01:30:00 +0000 UTC"} {blockgen_target="100", name="oldmetric", nextrollout_time="2021-09-04 01:30:00 +0000 UTC"} ... ``` If the output is expected, then you can add `--no-dry-run` flag to rewrite blocks. Thanos object storage supports `local filesystem`, which used local filesystem as bucket. If you want to delete/rewrite Prometheus TSDB, you can use the command below: ```shell thanos tools bucket rewrite --prom-blocks --rewrite.to-relabel-config-file config.yaml --objstore.config-file local-bucket.yaml --id <block ID> ``` `--prom-blocks` disables external labels check when adding new blocks. For the local bucket config file, please refer to .