Project: Cortex (Observability and Analysis / Observability)
File: querier.md

title: "Querier" linkTitle: "Querier" weight: 2 slug: querier <!-- DO NOT EDIT THIS FILE - This file has been automatically generated from its .template --> The querier service handles queries using the query language. This document dives into the storage-specific details of the querier service. The applies too. The querier is stateless. The querier needs to have an almost up-to-date view over the entire storage bucket, in order to find the right blocks to lookup at query time. The querier can keep the bucket view updated in to two different ways: Periodically scanning the bucket (default) Periodically downloading the At startup, queriers iterate over the entire storage bucket to discover all tenants blocks and download the `meta.json` for each block. During this initial bucket scanning phase, a querier is not ready to handle incoming queries yet and its `/ready` readiness probe endpoint will fail. While running, queriers periodically iterate over the storage bucket to discover new tenants and recently uploaded blocks. Queriers do not download any content from blocks except a small `meta.json` file containing the block's metadata (including the minimum and maximum timestamp of samples within the block). Queriers use the metadata to compute the list of blocks that need to be queried at query time and fetch matching series from the instances holding the required blocks. When is enabled, queriers lazily download the bucket index upon the first query received for a given tenant, cache it in memory and periodically keep it update. The bucket index contains the list of blocks and block deletion marks of a tenant, which is later used during the query execution to find the set of blocks that need to be queried for the given query. Given the bucket index removes the need to scan the bucket, it brings few benefits: The querier is expected to be ready shortly after startup. Lower volume of API calls to object storage. When a querier receives a query range request, it contains the following parameters: `query`: the PromQL query expression itself (e.g. `rate(nodecpuseconds_total[1m])`) `start`: the start time `end`: the end time `step`: the query resolution (e.g. `30` to have 1 resulting data point every 30s) Given a query, the querier analyzes the `start` and `end` time range to compute a list of all known blocks containing at least 1 sample within this time range. Given the list of blocks, the querier then computes a list of store-gateway instances holding these blocks and sends a request to each matching store-gateway instance asking to fetch all the samples for the series matching the `query` within the `start` and `end` time range. The request sent to each store-gateway contains the list of block IDs that are expected to be queried, and the response sent back by the store-gateway to the querier contains the list of block IDs that were actually queried. This list may be a subset of the requested blocks, for example due to recent blocks resharding event (ie. last few

The querier runs a consistency check on responses received from the store-gateways to ensure all expected blocks have been queried; if not, the querier retries to fetch samples from missing blocks from different store-gateways (if the `-store-gateway.sharding-ring.replication-factor` is greater than `1`) and if the consistency check fails after all retries, the query execution fails as well (correctness is always guaranteed). If the query time range covers a period within `-querier.query-ingesters-within` duration, the querier also sends the request to all ingesters, in order to fetch samples that have not been uploaded to the long-term storage yet. Once all samples have been fetched from both store-gateways and ingesters, the querier proceeds with running the PromQL engine to execute the query and send back the result to the client. Queriers need to discover store-gateways in order to connect to them at query time. The service discovery mechanism used depends whether blocks sharding is enabled in the store-gateways. When blocks sharding is enabled, queriers need to access to the store-gateways hash ring and thus queriers need to be configured with the same `-store-gateway.sharding-ring.*` flags (or their respective YAML config options) store-gateways have been configured. When blocks sharding is disabled, queriers need the `-querier.store-gateway-addresses` CLI flag (or its respective YAML config option) being set to a comma separated list of store-gateway addresses in . Queriers will evenly balance the requests to query blocks across the resolved addresses. The querier supports the following caches: Caching is optional, but highly recommended in a production environment. Please also check out the for more information about configuring the cache. and querier can use memcached for caching bucket metadata: List of tenants List of blocks per tenant Block's `meta.json` content Block's `deletion-mark.json` existence and content Tenant's `bucket-index.json.gz` content Using the metadata cache can significantly reduce the number of API calls to object storage and protects from linearly scale the number of these API calls with the number of querier and store-gateway instances (because the bucket is periodically scanned and synched by each querier and store-gateway). To enable metadata cache, please set `-blocks-storage.bucket-store.metadata-cache.backend`. Only `memcached` backend is supported currently. Memcached client has additional configuration available via flags with `-blocks-storage.bucket-store.metadata-cache.memcached.*` prefix. Additional options for configuring metadata cache have `-blocks-storage.bucket-store.metadata-cache.*` prefix. By configuring TTL to zero or negative value, caching of given item type is disabled. The same memcached backend cluster should be shared between store-gateways and queriers. This section described the querier configuration. For the general Cortex configuration and references to common config blocks, please refer to the . The `querier_config` configures the Cortex querier. ```yaml querier: [max_concurrent: <int> | default = 20] [timeout: <duration> | default = 2m] [ingestermetadatastreaming: <boolean> | default = false] [max_samples: <int> | default = 50000000] [queryingesterswithin: <duration> | default = 0s] [querystoreforlabelsenabled: <boolean> | default = false] [perstepstats_enabled: <boolean> | default = false] [querystoreafter: <duration> | default = 0s] [maxqueryinto_future: <duration> | default = 10m] [defaultevaluationinterval: <duration> | default = 1m] [maxsubquerysteps: <int> | default = 0] [activequerytracker_dir: <string> | default =

[lookback_delta: <duration> | default = 5m] [storegatewayaddresses: <string> | default = ""] storegatewayclient: [tls_enabled: <boolean> | default = false] [tlscertpath: <string> | default = ""] [tlskeypath: <string> | default = ""] [tlscapath: <string> | default = ""] [tlsservername: <string> | default = ""] [tlsinsecureskip_verify: <boolean> | default = false] [grpc_compression: <string> | default = ""] [storegatewayquery_stats: <boolean> | default = true] [shuffleshardingingesterslookbackperiod: <duration> | default = 0s] [thanos_engine: <boolean> | default = false] [ignoremaxquery_length: <boolean> | default = false] ``` The `blocksstorageconfig` configures the blocks

```yaml blocks_storage: [backend: <string> | default = "s3"] s3: [endpoint: <string> | default = ""] [region: <string> | default = ""] [bucket_name: <string> | default = ""] [secretaccesskey: <string> | default = ""] [accesskeyid: <string> | default = ""] [insecure: <boolean> | default = false] [signature_version: <string> | default = "v4"] [bucketlookuptype: <string> | default = "auto"] [sendcontentmd5: <boolean> | default = true] [sse: <s3sseconfig>] http: [idleconntimeout: <duration> | default = 1m30s] [responseheadertimeout: <duration> | default = 2m] [insecureskipverify: <boolean> | default = false] [tlshandshaketimeout: <duration> | default = 10s] [expectcontinuetimeout: <duration> | default = 1s] [maxidleconnections: <int> | default = 100] [maxidleconnectionsperhost: <int> | default = 100] [maxconnectionsper_host: <int> | default = 0] gcs: [bucket_name: <string> | default = ""] [service_account: <string> | default = ""] azure: [account_name: <string> | default = ""] [account_key: <string> | default = ""] [connection_string: <string> | default = ""] [container_name: <string> | default = ""] [endpoint_suffix: <string> | default = ""] [max_retries: <int> | default = 20] [msi_resource: <string> | default = ""] [userassignedid: <string> | default = ""] http: [idleconntimeout: <duration> | default = 1m30s] [responseheadertimeout: <duration> | default = 2m] [insecureskipverify: <boolean> | default = false] [tlshandshaketimeout: <duration> | default = 10s] [expectcontinuetimeout: <duration> | default = 1s] [maxidleconnections: <int> | default = 100] [maxidleconnectionsperhost: <int> | default = 100] [maxconnectionsper_host: <int> | default = 0] swift: [auth_version: <int> | default = 0] [auth_url: <string> | default = ""] [username: <string> | default = ""] [userdomainname: <string> | default = ""] [userdomainid: <string> | default = ""] [user_id: <string> | default = ""] [password: <string> | default = ""] [domain_id: <string> | default = ""] [domain_name: <string> | default = ""] [project_id: <string> | default = ""] [project_name: <string> | default = ""] [projectdomainid: <string> | default = ""] [projectdomainname: <string> | default = ""] [region_name: <string> | default = ""] [container_name: <string> | default = ""] [max_retries: <int> | default = 3] [connect_timeout: <duration> | default = 10s] [request_timeout: <duration> | default = 5s] filesystem: [dir: <string> | default = ""] bucket_store: [sync_dir: <string> | default = "tsdb-sync"] [sync_interval: <duration> | default = 15m] [max_concurrent: <int> | default = 100] [maxinflightrequests: <int> | default = 0] [tenantsyncconcurrency: <int> | default = 10] [blocksyncconcurrency: <int> | default = 20] [metasyncconcurrency: <int> | default = 20] [consistency_delay: <duration> | default = 0s] index_cache: [backend: <string> | default = "inmemory"] inmemory: [maxsizebytes: <int> | default = 1073741824] [enabled_items: <list of string> | default = []] memcached: [addresses: <string> | default = ""] [timeout: <duration> | default = 100ms] [maxidleconnections: <int> | default = 16] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxgetmulti_concurrency: <int> | default = 100] [maxgetmultibatchsize: <int> | default = 0] [maxitemsize: <int> | default = 1048576] [auto_discovery: <boolean> | default = false] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] [enabled_items: <list of string> | default = []] redis: [addresses: <string> | default = ""] [username: <string> | default = ""] [password: <string> | default = ""] [db: <int> | default = 0] [master_name: <string> | default = ""] [maxgetmulti_concurrency: <int> | default = 100] [getmultibatch_size: <int> | default = 100] [maxsetmulti_concurrency: <int> | default = 100] [setmultibatch_size: <int> | default = 100] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [dial_timeout: <duration> | default = 5s] [read_timeout: <duration> | default = 3s] [write_timeout: <duration> | default = 3s] [tls_enabled: <boolean> | default = false] [tlscertpath: <string> | default = ""] [tlskeypath: <string> | default = ""] [tlscapath: <string> | default = ""] [tlsservername: <string> | default = ""] [tlsinsecureskip_verify: <boolean> | default = false] [cache_size: <int> | default = 0] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] [enabled_items: <list of string> | default = []] multilevel: [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxbackfillitems: <int> | default = 10000] chunks_cache: [backend: <string> | default = ""] memcached: [addresses: <string> | default = ""] [timeout: <duration> | default = 100ms] [maxidleconnections: <int> | default = 16] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxgetmulti_concurrency: <int> | default = 100] [maxgetmultibatchsize: <int> | default = 0] [maxitemsize: <int> | default = 1048576] [auto_discovery: <boolean> | default = false] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] redis: [addresses: <string> | default = ""] [username: <string> | default = ""] [password: <string> | default = ""] [db: <int> | default = 0] [master_name: <string> | default = ""] [maxgetmulti_concurrency: <int> | default = 100] [getmultibatch_size: <int> | default = 100] [maxsetmulti_concurrency: <int> | default = 100] [setmultibatch_size: <int> | default = 100] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [dial_timeout: <duration> | default = 5s] [read_timeout: <duration> | default = 3s] [write_timeout: <duration> | default = 3s] [tls_enabled: <boolean> | default = false] [tlscertpath: <string> | default = ""] [tlskeypath: <string> | default = ""] [tlscapath: <string> | default = ""] [tlsservername: <string> | default = ""] [tlsinsecureskip_verify: <boolean> | default = false] [cache_size: <int> | default = 0] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default =

[subrange_size: <int> | default = 16000] [maxgetrange_requests: <int> | default = 3] [attributes_ttl: <duration> | default = 168h] [subrange_ttl: <duration> | default = 24h] metadata_cache: [backend: <string> | default = ""] memcached: [addresses: <string> | default = ""] [timeout: <duration> | default = 100ms] [maxidleconnections: <int> | default = 16] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [maxgetmulti_concurrency: <int> | default = 100] [maxgetmultibatchsize: <int> | default = 0] [maxitemsize: <int> | default = 1048576] [auto_discovery: <boolean> | default = false] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] redis: [addresses: <string> | default = ""] [username: <string> | default = ""] [password: <string> | default = ""] [db: <int> | default = 0] [master_name: <string> | default = ""] [maxgetmulti_concurrency: <int> | default = 100] [getmultibatch_size: <int> | default = 100] [maxsetmulti_concurrency: <int> | default = 100] [setmultibatch_size: <int> | default = 100] [maxasyncconcurrency: <int> | default = 50] [maxasyncbuffer_size: <int> | default = 10000] [dial_timeout: <duration> | default = 5s] [read_timeout: <duration> | default = 3s] [write_timeout: <duration> | default = 3s] [tls_enabled: <boolean> | default = false] [tlscertpath: <string> | default = ""] [tlskeypath: <string> | default = ""] [tlscapath: <string> | default = ""] [tlsservername: <string> | default = ""] [tlsinsecureskip_verify: <boolean> | default = false] [cache_size: <int> | default = 0] setasynccircuitbreakerconfig: [enabled: <boolean> | default = false] [halfopenmax_requests: <int> | default = 10] [open_duration: <duration> | default = 5s] [min_requests: <int> | default = 50] [consecutive_failures: <int> | default = 5] [failure_percent: <float> | default = 0.05] [tenantslistttl: <duration> | default = 15m] [tenantblockslist_ttl: <duration> | default = 5m] [chunkslistttl: <duration> | default = 24h] [metafileexiststtl: <duration> | default = 2h] [metafiledoesntexist_ttl: <duration> | default = 5m] [metafilecontentttl: <duration> | default = 24h] [metafilemaxsize_bytes: <int> | default = 1048576] [metafileattributesttl: <duration> | default = 168h] [blockindexattributes_ttl: <duration> | default = 168h] [bucketindexcontent_ttl: <duration> | default = 5m] [bucketindexmaxsizebytes: <int> | default = 1048576] [ignoredeletionmark_delay: <duration> | default = 6h] [ignoreblockswithin: <duration> | default = 0s] bucket_index: [enabled: <boolean> | default = false] [updateonerror_interval: <duration> | default = 1m] [idle_timeout: <duration> | default = 1h] [maxstaleperiod: <duration> | default = 1h] [blockdiscoverystrategy: <string> | default = "concurrent"] [maxchunkpool_bytes: <int> | default = 2147483648] [indexheaderlazyloadingenabled: <boolean> | default = false] [indexheaderlazyloadingidle_timeout: <duration> | default = 20m] [lazyexpandedpostings_enabled: <boolean> | default = false] [seriesbatchsize: <int> | default = 10000] tsdb: [dir: <string> | default = "tsdb"] [blockrangesperiod: <list of duration> | default = 2h0m0s] [retention_period: <duration> | default = 6h] [ship_interval: <duration> | default = 1m] [ship_concurrency: <int> | default = 10] [headcompactioninterval: <duration> | default = 1m] [headcompactionconcurrency: <int> | default = 5] [headcompactionidle_timeout: <duration> | default = 1h] [headchunkswritebuffersize_bytes: <int> | default = 4194304] [stripe_size: <int> | default = 16384] [walcompressionenabled: <boolean> | default = false] [walsegmentsize_bytes: <int> | default = 134217728] [flushblockson_shutdown: <boolean> | default = false] [closeidletsdb_timeout: <duration> | default = 0s] [headchunkswritequeuesize: <int> | default = 0] [maxtsdbopeningconcurrencyon_startup: <int> | default = 10] [max_exemplars: <int> | default = 0] [memorysnapshoton_shutdown: <boolean> | default = false] [outofordercapmax: <int> | default = 32] ```