Project: Cortex (Observability and Analysis / Observability)
File: store-gateway.md

title: "Store-gateway" linkTitle: "Store-gateway" weight: 3 slug: store-gateway <!-- DO NOT EDIT THIS FILE - This file has been automatically generated from its .template --> The store-gateway is the Cortex service responsible to query series from blocks. The store-gateway is required when running the Cortex blocks storage. The store-gateway is semi-stateful. The store-gateway needs to have an almost up-to-date view over the storage bucket, in order to discover blocks belonging to their shard. The store-gateway can keep the bucket view updated in to two different ways: Periodically scanning the bucket (default) Periodically downloading the At startup store-gateways iterate over the entire storage bucket to discover blocks for all tenants and download the `meta.json` and index-header for each block. During this initial bucket synchronization phase, the store-gateway `/ready` readiness probe endpoint will fail. While running, store-gateways periodically rescan the storage bucket to discover new blocks (uploaded by the ingesters and ) and blocks marked for deletion or fully deleted since the last scan (as a result of compaction). The frequency at which this occurs is configured via `-blocks-storage.bucket-store.sync-interval`. The blocks chunks and the entire index are never fully downloaded by the store-gateway. The index-header is stored to the local disk, in order to avoid to re-download it on subsequent restarts of a store-gateway. For this reason, it's recommended - but not required - to run the store-gateway with a persistent disk. For example, if you're running the Cortex cluster in Kubernetes, you may use a StatefulSet with a persistent volume claim for the store-gateways. For more information about the index-header, please refer to . When bucket index is enabled, the overall workflow is the same but, instead of iterating over the bucket objects, the store-gateway fetch the for each tenant belonging to their shard in order to discover each tenant's blocks and block deletion marks. For more information about the bucket index, please refer to . The store-gateway optionally supports blocks sharding. Sharding can be used to horizontally scale blocks in a large cluster without hitting any vertical scalability limit. When sharding is enabled, store-gateway instances builds an and blocks get sharded and replicated across the pool of store-gateway instances registered within the ring. Store-gateways continuously monitor the ring state and whenever the ring topology changes (e.g. a new instance has been added/removed or gets healthy/unhealthy) each store-gateway instance resync the blocks assigned to its shard, based on the block ID hash matching the token ranges assigned to the instance itself within the ring. For each block belonging to a store-gateway shard, the store-gateway loads its `meta.json`, the `deletion-mark.json` and the index-header. Once a block is loaded on the store-gateway, it's ready to be queried by queriers. When the querier queries blocks through a store-gateway, the response will contain the list of actually queried block IDs. If a querier tries to query a block which has not been loaded by a store-gateway, the querier will either retry on a different store-gateway (if blocks replication is enabled) or fail the query. Blocks can be replicated across multiple store-gateway instances based on a replication factor configured via

The blocks replication is used to protect from query failures caused by some blocks not loaded by any store-gateway instance at a given time like, for example, in the event of a store-gateway failure or while restarting a store-gateway instance (e.g. during a rolling update). This feature can be enabled via `-store-gateway.sharding-enabled=true` and requires the backend to be configured via `-store-gateway.sharding-ring.*` flags (or their respective YAML config options). The store-gateway supports two sharding strategies: `default` `shuffle-sharding` `zone-stable-shuffle-sharding` The `default` sharding strategy spreads the blocks of each tenant across all store-gateway instances. It's the easiest form of sharding supported, but doesn't provide any workload isolation between different tenants. The `shuffle-sharding` strategy spreads the blocks of a tenant across a subset of store-gateway instances. This way, the number of store-gateway instances loading blocks of a single tenant is limited and the blast radius of any issue that could be introduced by the tenant's workload is limited to its shard instances. The shuffle sharding strategy can be enabled via `-store-gateway.sharding-strategy=shuffle-sharding` and requires the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options) to be set to the default shard size, which is the default number of store-gateway instances each tenant should be sharded to. The shard size can then be overridden on a per-tenant basis setting the `storegatewaytenantshardsize` in the limits overrides. The `zone-stable-shuffle-sharding` strategy achieves the same as the `shuffle-sharding` strategy, but using a different sharding algorithm. The new sharding algorithm ensures that when zone awareness is enabled, when shard size increases or decreases by one, the replicas for any block should only change at most by one instance. This is important for querying store gateway because a block can be retried at most 3 times. Zone stable shuffle sharding can be enabled via `-store-gateway.sharding-ring.zone-stable-shuffle-sharding` CLI flag. It will become the default shuffle sharding strategy for store gateway in `v1.17.0` release and the previous shuffle sharding algorithm will be removed in `v1.18.0` release. Please check out the for more information about how it works. When a store-gateway instance cleanly shutdowns, it automatically unregisters itself from the ring. However, in the event of a crash or node failure, the instance will not be unregistered from the ring, potentially leaving a spurious entry in the ring forever. To protect from this, when an healthy store-gateway instance finds another instance in the ring which is unhealthy for more than 10 times the configured `-store-gateway.sharding-ring.heartbeat-timeout`, the healthy instance forcibly removes the unhealthy one from the ring. This feature is called auto-forget and is built into the store-gateway. The store-gateway replication optionally supports . When zone-aware replication is enabled and the blocks replication factor is > 1, each block is guaranteed to be replicated across store-gateway instances running in different availability zones. To enable the zone-aware replication for the store-gateways you should: Configure the availability zone for each store-gateway via the `-store-gateway.sharding-ring.instance-availability-zone` CLI flag (or its respective YAML config option) Enable blocks zone-aware replication via the `-store-gateway.sharding-ring.zone-awareness-enabled` CLI flag (or its respective YAML config option). Please be aware this configuration option should be set to store-gateways, queriers and

Rollout store-gateways, queriers and rulers to apply the new configuration In the event of a cluster cold start or scale up of 2+ store-gateway instances at the same time we may end up in a situation where each new store-gateway instance starts at a slightly different time and thus each one runs the initial blocks sync based on a different state of the ring. For example, in case of a cold start, the first store-gateway joining the ring may load all blocks since the sharding logic runs based on the current state of the ring, which is 1 single store-gateway. To reduce the likelihood this could happen, the store-gateway waits for a stable ring at startup. A ring is considered stable if no instance is added/removed to the ring for at least `-store-gateway.sharding-ring.wait-stability-min-duration`. If the ring keep getting changed after `-store-gateway.sharding-ring.wait-stability-max-duration`, the store-gateway will stop waiting for a stable ring and will proceed starting up normally. To disable this waiting logic, you can start the store-gateway with `-store-gateway.sharding-ring.wait-stability-min-duration=0`. The is a subset of the block index which the store-gateway downloads from the object storage and keeps on the local disk in order to speed up queries. At startup, the store-gateway downloads the index-header of each block belonging to its shard. A store-gateway is not ready until this initial index-header download is completed. Moreover, while running, the store-gateway periodically looks for newly uploaded blocks in the storage and downloads the index-header for the blocks belonging to its shard. By default, each index-header is memory mapped by the store-gateway right after downloading it. In a cluster with a large number of blocks, each store-gateway may have a large amount of memory mapped index-headers, regardless how frequently they're used at query time. Cortex supports a configuration option `-blocks-storage.bucket-store.index-header-lazy-loading-enabled=true` to enable index-header lazy loading. When enabled, index-headers will be memory mapped only once required by a query and will be automatically released after `-blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout` time of inactivity. The store-gateway supports the following caches: Caching is optional, but highly recommended in a production environment. Please also check out the for more information about configuring the cache. The store-gateway can use a cache to speed up lookups of postings and series from TSDB blocks indexes. Two backends are supported: `inmemory` `memcached` `redis` The `inmemory` index cache is enabled by default and its max size can be configured through the flag `-blocks-storage.bucket-store.index-cache.inmemory.max-size-bytes` (or config file). The trade-off of using the in-memory index cache is: Pros: zero latency Cons: increased store-gateway memory usage, not shared across multiple store-gateway replicas (when sharding is disabled or replication factor > 1) The `memcached` index cache allows to use as cache backend. This cache backend is configured using `-blocks-storage.bucket-store.index-cache.backend=memcached` and requires the Memcached server(s) addresses via `-blocks-storage.bucket-store.index-cache.memcached.addresses` (or config file). The addresses are resolved using the . The trade-off of using the Memcached index cache is: Pros: can scale beyond a single node memory (Memcached cluster), shared across multiple store-gateway instances Cons: higher latency in the cache round trip compared to the in-memory one The Memcached client uses a jump hash algorithm to shard cached entries across a cluster of Memcached

For this reason, you should make sure memcached servers are not behind any kind of load balancer and their address is configured so that servers are added/removed to the end of the list whenever a scale up/down occurs. For example, if you're running Memcached in Kubernetes, you may: Deploy your Memcached cluster using a Create a for Memcached StatefulSet Configure the Cortex's Memcached client address using the `dnssrvnoa+` The `redis` index cache allows to use as cache backend. This cache backend is configured using `-blocks-storage.bucket-store.index-cache.backend=redis` and requires the Redis server(s) addresses via `-blocks-storage.bucket-store.index-cache.redis.addresses` (or config file). Using `redis` as the cache backend has similar trade-offs as using `memcached` cache backend. However, client side caching can be enabled when using `redis` backend to avoid Store Gateway fetching data from cache each time. See for more info and it can be enabled by setting flag `-blocks-storage.bucket-store.index-cache.redis.cache-size` > 0. Store-gateway can also use a cache for storing chunks fetched from the storage. Chunks contain actual samples, and can be reused if user query hits the same series for the same time range. To enable chunks cache, please set `-blocks-storage.bucket-store.chunks-cache.backend`. Chunks can be stored into Memcached or Redis cache. Memcached client can be configured via flags with `-blocks-storage.bucket-store.chunks-cache.memcached.` prefix. Redis client can be configured via flags with `-blocks-storage.bucket-store.chunks-cache.redis.` prefix. There are additional low-level options for configuring chunks cache. Please refer to other flags with `-blocks-storage.bucket-store.chunks-cache.*` prefix. Store-gateway and can use memcached or redis for caching bucket metadata: List of tenants List of blocks per tenant Block's `meta.json` content Block's `deletion-mark.json` existence and content Tenant's `bucket-index.json.gz` content Using the metadata cache can significantly reduce the number of API calls to object storage and protects from linearly scale the number of these API calls with the number of querier and store-gateway instances (because the bucket is periodically scanned and synched by each querier and store-gateway). To enable metadata cache, please set `-blocks-storage.bucket-store.metadata-cache.backend`. `memcached` and `redis` backend are supported currently. Memcached client has additional configuration available via flags with `-blocks-storage.bucket-store.metadata-cache.memcached.` prefix. Redis client has additional configuration available via flags with `-blocks-storage.bucket-store.metadata-cache.redis.` prefix. Additional options for configuring metadata cache have `-blocks-storage.bucket-store.metadata-cache.*` prefix. By configuring TTL to zero or negative value, caching of given item type is disabled. The same cache backend deployment should be shared between store-gateways and queriers. `GET /store-gateway/ring`<br /> Displays the status of the store-gateways ring, including the tokens owned by each store-gateway and an option to remove (forget) instances from the ring. This section described the store-gateway configuration. For the general Cortex configuration and references to common config blocks, please refer to the . The `storegatewayconfig` configures the store-gateway service used by the blocks

```yaml store_gateway: [sharding_enabled: <boolean> | default = false] sharding_ring: kvstore: [store: <string> | default = "consul"] [prefix: <string> | default = "collectors/"] dynamodb: [region: <string> | default = ""] [table_name: <string> | default = ""] [ttl: <duration> | default = 0s] [pullersynctime: <duration> | default = 1m] [maxcasretries: <int> | default = 10] [consul: <consul_config>] [etcd: <etcd_config>] multi: [primary: <string> | default = ""] [secondary: <string> | default = ""] [mirror_enabled: <boolean> | default = false] [mirror_timeout: <duration> | default = 2s] [heartbeat_period: <duration> | default = 15s] [heartbeat_timeout: <duration> | default = 1m] [replication_factor: <int> | default = 3] [tokensfilepath: <string> | default = ""] [zoneawarenessenabled: <boolean> | default = false] [keepinstanceintheringonshutdown: <boolean> | default = false] [waitstabilitymin_duration: <duration> | default = 1m] [waitstabilitymax_duration: <duration> | default = 5m] [waitinstancestate_timeout: <duration> | default = 10m] [final_sleep: <duration> | default = 0s] [instanceinterfacenames: <list of string> | default = [eth0 en0]] [instanceavailabilityzone: <string> | default = ""] [sharding_strategy: <string> | default = "default"] [enabled_tenants: <string> | default = ""] [disabled_tenants: <string> | default = ""] ``` The `blocksstorageconfig` configures the blocks

```yaml blocks_storage: [backend: <string> | default = "s3"] s3: [endpoint: <string> | default = ""] [region: <string> | default = ""] [bucket_name: <string> | default = ""] [secretaccesskey: <string> | default = ""] [accesskeyid: <string> | default = ""] [insecure: <boolean> | default = false] [signature_version: <string> | default = "v4"] [bucketlookuptype: <string> | default = "auto"] [sendcontentmd5: <boolean> | default = true] [sse: <s3sseconfig>] http: [idleconntimeout: <duration> | default = 1m30s] [responseheadertimeout: <duration> | default = 2m] [insecureskipverify: <boolean> | default = false] [tlshandshaketimeout: <duration> | default = 10s] [expectcontinuetimeout: <duration> | default = 1s] [maxidleconnections: <int> | default = 100] [maxidleconnectionsperhost: <int> | default = 100] [maxconnectionsper_host: <int> | default = 0] gcs: [bucket_name: <string> | default = ""] [service_account: <string> | default = ""] azure: [account_name: <string> | default = ""] [account_key: <string> | default = ""] [connection_string: <string> | default = ""] [container_name: <string> | default = ""] [endpoint_suffix: <string> | default = ""] [max_retries: <int> | default = 20] [msi_resource: <string> | default = ""] [userassignedid: <string> | default = ""] http: [idleconntimeout: <duration> | default = 1m30s] [responseheadertimeout: <duration> | default = 2m] [insecureskipverify: <boolean> | default = false] [tlshandshaketimeout: <duration> | default = 10s] [expectcontinuetimeout: <duration> | default = 1s] [maxidleconnections: <int> | default = 100] [maxidleconnectionsperhost: <int> | default = 100] [maxconnectionsper_host: <int> | default = 0] swift: [auth_version: <int> | default = 0] [auth_url: <string> | default = ""] [username: <string> | default = ""] [userdomainname: <string> | default = ""] [userdomainid: <string> | default = ""] [user_id: <string> | default = ""] [password: <string> | default = ""] [domain_id: <string> | default = ""] [domain_name: <string> | default = ""] [project_id: <string> | default = ""] [project_name: <string> | default = ""] [projectdomainid: <string> | default = ""] [projectdomainname: <string> | default = ""] [region_name: <string> | default = ""] [container_name: <string> | default = ""] [max_retries: <int> | default = 3] [connect_timeout: <duration> | default = 10s] [request_timeout: <duration> | default = 5s] filesystem: [dir: <string> | default = ""] bucket_store: [sync_dir: <string> | default = "tsdb-sync"] [sync_interval: <duration> | default = 15m] [max_concurrent: <int> | default = 100] [maxinflightrequests: <int> | default = 0] [tenantsyncconcurrency: <int> | default = 10] [blocksyncconcurrency: <int> | default = 20] [metasyncconcurrency: <int> | default = 20] [consistency_delay: <duration> | default = 0s] index_cache: [backend: <string> | default = "inmemory"] inmemory: [maxsizebytes: <int> | default = 1073741824] [enabled_items: <list of string> | default = []] memcached: [addresses: <string> | default = ""] [timeout: <duration> | default = 100ms] [maxidleconnections: <int> | default = 16] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxgetmulti_concurrency: <int> | default = 100] [maxgetmultibatchsize: <int> | default = 0] [maxitemsize: <int> | default = 1048576] [auto_discovery: <boolean> | default = false] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] [enabled_items: <list of string> | default = []] redis: [addresses: <string> | default = ""] [username: <string> | default = ""] [password: <string> | default = ""] [db: <int> | default = 0] [master_name: <string> | default = ""] [maxgetmulti_concurrency: <int> | default = 100] [getmultibatch_size: <int> | default = 100] [maxsetmulti_concurrency: <int> | default = 100] [setmultibatch_size: <int> | default = 100] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [dial_timeout: <duration> | default = 5s] [read_timeout: <duration> | default = 3s] [write_timeout: <duration> | default = 3s] [tls_enabled: <boolean> | default = false] [tlscertpath: <string> | default = ""] [tlskeypath: <string> | default = ""] [tlscapath: <string> | default = ""] [tlsservername: <string> | default = ""] [tlsinsecureskip_verify: <boolean> | default = false] [cache_size: <int> | default = 0] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] [enabled_items: <list of string> | default = []] multilevel: [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxbackfillitems: <int> | default = 10000] chunks_cache: [backend: <string> | default = ""] memcached: [addresses: <string> | default = ""] [timeout: <duration> | default = 100ms] [maxidleconnections: <int> | default = 16] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxgetmulti_concurrency: <int> | default = 100] [maxgetmultibatchsize: <int> | default = 0] [maxitemsize: <int> | default = 1048576] [auto_discovery: <boolean> | default = false] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] redis: [addresses: <string> | default = ""] [username: <string> | default = ""] [password: <string> | default = ""] [db: <int> | default = 0] [master_name: <string> | default = ""] [maxgetmulti_concurrency: <int> | default = 100] [getmultibatch_size: <int> | default = 100] [maxsetmulti_concurrency: <int> | default = 100] [setmultibatch_size: <int> | default = 100] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [dial_timeout: <duration> | default = 5s] [read_timeout: <duration> | default = 3s] [write_timeout: <duration> | default = 3s] [tls_enabled: <boolean> | default = false] [tlscertpath: <string> | default = ""] [tlskeypath: <string> | default = ""] [tlscapath: <string> | default = ""] [tlsservername: <string> | default = ""] [tlsinsecureskip_verify: <boolean> | default = false] [cache_size: <int> | default = 0] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default =

[subrange_size: <int> | default = 16000] [maxgetrange_requests: <int> | default = 3] [attributes_ttl: <duration> | default = 168h] [subrange_ttl: <duration> | default = 24h] metadata_cache: [backend: <string> | default = ""] memcached: [addresses: <string> | default = ""] [timeout: <duration> | default = 100ms] [maxidleconnections: <int> | default = 16] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxgetmulti_concurrency: <int> | default = 100] [maxgetmultibatchsize: <int> | default = 0] [maxitemsize: <int> | default = 1048576] [auto_discovery: <boolean> | default = false] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] redis: [addresses: <string> | default = ""] [username: <string> | default = ""] [password: <string> | default = ""] [db: <int> | default = 0] [master_name: <string> | default = ""] [maxgetmulti_concurrency: <int> | default = 100] [getmultibatch_size: <int> | default = 100] [maxsetmulti_concurrency: <int> | default = 100] [setmultibatch_size: <int> | default = 100] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [dial_timeout: <duration> | default = 5s] [read_timeout: <duration> | default = 3s] [write_timeout: <duration> | default = 3s] [tls_enabled: <boolean> | default = false] [tlscertpath: <string> | default = ""] [tlskeypath: <string> | default = ""] [tlscapath: <string> | default = ""] [tlsservername: <string> | default = ""] [tlsinsecureskip_verify: <boolean> | default = false] [cache_size: <int> | default = 0] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] [tenantslistttl: <duration> | default = 15m] [tenantblockslist_ttl: <duration> | default = 5m] [chunkslistttl: <duration> | default = 24h] [metafileexiststtl: <duration> | default = 2h] [metafiledoesntexist_ttl: <duration> | default = 5m] [metafilecontentttl: <duration> | default = 24h] [metafilemaxsize_bytes: <int> | default = 1048576] [metafileattributesttl: <duration> | default = 168h] [blockindexattributes_ttl: <duration> | default = 168h] [bucketindexcontent_ttl: <duration> | default = 5m] [bucketindexmaxsizebytes: <int> | default = 1048576] [ignoredeletionmark_delay: <duration> | default = 6h] [ignoreblockswithin: <duration> | default = 0s] bucket_index: [enabled: <boolean> | default = false] [updateonerror_interval: <duration> | default = 1m] [idle_timeout: <duration> | default = 1h] [maxstaleperiod: <duration> | default = 1h] [blockdiscoverystrategy: <string> | default = "concurrent"] [maxchunkpool_bytes: <int> | default = 2147483648] [indexheaderlazyloadingenabled: <boolean> | default = false] [indexheaderlazyloadingidle_timeout: <duration> | default = 20m] [lazyexpandedpostings_enabled: <boolean> | default = false] [seriesbatchsize: <int> | default = 10000] tsdb: [dir: <string> | default = "tsdb"] [blockrangesperiod: <list of duration> | default = 2h0m0s] [retention_period: <duration> | default = 6h] [ship_interval: <duration> | default = 1m] [ship_concurrency: <int> | default = 10] [headcompactioninterval: <duration> | default = 1m] [headcompactionconcurrency: <int> | default = 5] [headcompactionidle_timeout: <duration> | default = 1h] [headchunkswritebuffersize_bytes: <int> | default = 4194304] [stripe_size: <int> | default = 16384] [walcompressionenabled: <boolean> | default = false] [walsegmentsize_bytes: <int> | default = 134217728] [flushblockson_shutdown: <boolean> | default = false] [closeidletsdb_timeout: <duration> | default = 0s] [headchunkswritequeuesize: <int> | default = 0] [maxtsdbopeningconcurrencyon_startup: <int> | default = 10] [max_exemplars: <int> | default = 0] [memorysnapshoton_shutdown: <boolean> | default = false] [outofordercapmax: <int> | default = 32] ```