Project: Cortex (Observability and Analysis / Observability)
File: compactor.md

title: "Compactor" linkTitle: "Compactor" weight: 4 slug: compactor <!-- DO NOT EDIT THIS FILE - This file has been automatically generated from its .template --> The compactor is an service which is responsible to: Compact multiple blocks of a given tenant into a single optimized larger block. This helps to reduce storage costs (deduplication, index size reduction), and increase query speed (querying fewer blocks is faster). Keep the per-tenant bucket index updated. The is used by , and rulers to discover new blocks in the storage. The compactor is stateless. The blocks compaction has two main benefits: Vertically compact blocks uploaded by all ingesters for the same time range Horizontally compact blocks with small time ranges into a single larger block The vertical compaction merges all the blocks of a tenant uploaded by ingesters for the same time range (2 hours ranges by default) into a single block, while de-duplicating samples that are originally written to N blocks as a result of replication. This step reduces number of blocks for single 2 hours time range from #(number of ingesters) to 1 per tenant. The horizontal compaction triggers after the vertical compaction and compacts several blocks with adjacent 2-hour range periods into a single larger block. Even though the total size of block chunks doesn't change after this compaction, it may still significantly reduce the size of the index and the index-header kept in memory by store-gateways. <!-- Diagram source at https://docs.google.com/presentation/d/1bHp8_zcoWCYoNU2AhO2lSagQyuIrghkCncViSqn14cU/edit --> The compactor optionally supports sharding. When sharding is enabled, multiple compactor instances can coordinate to split the workload and shard blocks by tenant. All the blocks of a tenant are processed by a single compactor instance at any given time, but compaction for different tenants may simultaneously run on different compactor instances. Whenever the pool of compactors increase or decrease (ie. following up a scale up/down), tenants are resharded across the available compactor instances without any manual intervention. The compactor sharding is based on the Cortex . At startup, a compactor generates random tokens and registers itself to the ring. While running, it periodically scans the storage bucket (every `-compactor.compaction-interval`) to discover the list of tenants in the storage and compacts blocks for each tenant whose hash matches the token ranges assigned to the instance itself within the ring. This feature can be enabled via `-compactor.sharding-enabled=true` and requires the backend to be configured via `-compactor.ring.*` flags (or their respective YAML config options). In the event of a cluster cold start or scale up of 2+ compactor instances at the same time we may end up in a situation where each new compactor instance starts at a slightly different time and thus each one runs the first compaction based on a different state of the ring. This is not a critical condition, but may be inefficient, because multiple compactor replicas may start compacting the same tenant nearly at the same time. To reduce the likelihood this could happen, the compactor waits for a stable ring at startup. A ring is considered stable if no instance is added/removed to the ring for at least `-compactor.ring.wait-stability-min-duration`. If the ring keep getting changed after `-compactor.ring.wait-stability-max-duration`, the compactor will stop waiting for a stable ring and will proceed starting up normally. To disable this waiting logic, you can start the compactor with `-compactor.ring.wait-stability-min-duration=0`. When the compactor successfully compacts some source blocks into a larger block, source blocks are deleted from the

Blocks deletion is not immediate, but follows a two steps process: First, a block is marked for deletion (soft delete) Then, once a block is marked for deletion for longer then `-compactor.deletion-delay`, the block is deleted from the storage (hard delete) The compactor is both responsible to mark blocks for deletion and then hard delete them once the deletion delay expires. The soft deletion is based on a tiny `deletion-mark.json` file stored within the block location in the bucket which gets looked up both by queriers and store-gateways. This soft deletion mechanism is used to give enough time to queriers and store-gateways to discover the new compacted blocks before the old source blocks are deleted. If source blocks would be immediately hard deleted by the compactor, some queries involving the compacted blocks may fail until the queriers and store-gateways haven't rescanned the bucket and found both deleted source blocks and the new compacted ones. The compactor needs to download source blocks from the bucket to the local disk, and store the compacted block to the local disk before uploading it to the bucket. Depending on the largest tenants in your cluster and the configured `-compactor.block-ranges`, the compactor may need a lot of disk space. Assuming `maxcompactionrangeblockssize` is the total size of blocks for the largest tenant (you can measure it inspecting the bucket) and the longest `-compactor.block-ranges` period, the formula to estimate the minimum disk space required is: ``` mindiskspacerequired = compactor.compaction-concurrency * maxcompactionrangeblocks_size * 2 ``` Alternatively, assuming the largest `-compactor.block-ranges` is `24h` (default), you could consider 150GB of disk space every 10M active series owned by the largest tenant. For example, if your largest tenant has 30M active series and `-compactor.compaction-concurrency=1` we would recommend having a disk with at least 450GB available. `GET /compactor/ring`<br /> Displays the status of the compactors ring, including the tokens owned by each compactor and an option to remove (forget) instances from the ring. This section described the compactor configuration. For the general Cortex configuration and references to common config blocks, please refer to the . The `compactor_config` configures the compactor for the blocks storage. ```yaml compactor: [block_ranges: <list of duration> | default = 2h0m0s,12h0m0s,24h0m0s] [blocksyncconcurrency: <int> | default = 20] [metasyncconcurrency: <int> | default = 20] [consistency_delay: <duration> | default = 0s] [data_dir: <string> | default = "./data"] [compaction_interval: <duration> | default = 1h] [compaction_retries: <int> | default = 3] [compaction_concurrency: <int> | default = 1] [cleanup_interval: <duration> | default = 15m] [cleanup_concurrency: <int> | default = 20] [deletion_delay: <duration> | default = 12h] [tenantcleanupdelay: <duration> | default = 6h] [skipblockswithoutoforderchunks_enabled: <boolean> | default = false] [blockfilesconcurrency: <int> | default = 10] [blocksfetchconcurrency: <int> | default = 3] [blockdeletionmarksmigrationenabled: <boolean> | default = false] [enabled_tenants: <string> | default = ""] [disabled_tenants: <string> | default = ""] [sharding_enabled: <boolean> | default = false] [sharding_strategy: <string> | default = "default"] sharding_ring: kvstore: [store: <string> | default = "consul"] [prefix: <string> | default = "collectors/"] dynamodb: [region: <string> | default = ""] [table_name: <string> | default = ""] [ttl: <duration> | default = 0s] [pullersynctime: <duration> | default = 1m] [maxcasretries: <int> | default = 10] [consul: <consul_config>] [etcd: <etcd_config>] multi: [primary: <string> | default = ""] [secondary: <string> | default = ""] [mirror_enabled: <boolean> | default = false] [mirror_timeout: <duration> | default = 2s] [heartbeat_period: <duration> | default = 5s] [heartbeat_timeout: <duration> | default = 1m] [waitstabilitymin_duration: <duration> | default = 1m] [waitstabilitymax_duration: <duration> | default = 5m] [instanceinterfacenames: <list of string> | default = [eth0 en0]] [tokensfilepath: <string> | default = ""] [unregisteronshutdown: <boolean> | default = true] [waitactiveinstance_timeout: <duration> | default =