Project: Cortex (Observability and Analysis / Observability)
File: cross-tenant-query-federation.md

title: "Cross-Tenant Query Federation" linkTitle: "Cross-Tenant Query Federation" weight: 1 slug: "cross-tenant-query-federation" Author: Date: October 2020 Status: Accepted This document aims to describe how to implement the ability to allow queries to cover data from more than a single Cortex tenant. Adopting a tenancy model within an organization with each tenant representing a department comes with the disadvantage that it will prevent queries from spanning multiple departments. This proposal tries to overcome those limitations. In theory PromQL API clients could be aggregating/correlating query results from multiple tenants. For example Grafana could be used with multiple data sources and a cross tenant query could be achieved through using . As this approach comes with the following disadvantages, it was not considered further: Every PromQL API client needs to support the aggregation from various sources. Queries that are written in PromQL can't be used without extra work across tenants. Another approach to multi tenant query federation could be achieved by aggregation of partial query results within the query frontend. For this a query needs to be split into sub queries per tenant and afterwards the partial results need reduced into the final result. The [astmapper] package goes down a similar approach, but it cannot parallelize all query types. Ideally multi-tenant query federation should support the full PromQL language and the algorithms necessary would differ per query functions and operators used. This approach was deemed as a fairly complex way to achieve that tenant query federation. The series in different tenants might have exactly the same labels and hence potentially collide which each other. In order to be able to always identify the tenant correctly, queries using multiple tenants should inject a tenant label named `tenant_id` and its value containing the tenant ID into the results. A potentially existing label with the same name should be stored in a label with the prefix `original_`. Label selectors containing the tenant label should behave like any other labels. This can be achieved by selecting the tenants used in a multi tenant query. The tenant ID is currently read from the `X-Scope-OrgID` HTTP header. The tenant ID has those of values being allowed. For the query path a user should be able to specify a `X-Scope-OrgID` header with multiple tenant IDs. Multiple tenant IDs should then be propagating throughout out the system until it reaches the

The `Queryable` instance returned by the querier component, is wrapped by a `mergeQueryable`, which will aggregate the results from a `Queryable` per tenant and hence treated by the downstream components as a single tenant query. To allow such queries to be processed we suggest that an experimental configuration flag `-querier.tenant-federation.enabled` will be added, which is switched off by default. Once enabled the value of the `X-Scope-OrgID` header should be interpreted as `|` separated list of tenant ids. Components which are not expecting multiple tenant ids (e.g. the ingress path) must signal an error if multiple are used. In Cortex the tenant id is used as the primary identifier for those components: The limits that apply to a certain query. The query-frontend maintains a per tenant query queue to implement fairness. Relevant metrics about the query are exposed under a `user` label. Having a query spanning multiple tenants, the existing methods are no longer correct. The identifier for aforementioned features for queries involving more than a single tenant should be derived from: An ordered, distinct list of tenant IDs, which is joined by a `|`. This will produce a reproducible identifier for the same set of tenants no matter which order they have been specified. While this feature is considered experimental, this provides some insights and ability to limit multi-tenant queries with these short comings: Cardinality costs to the possible amount of tenant ID combinations. Query limits applied to single tenants part of a multi tenant query are ignored. | Challenge | Status | |--|| | Aggregate data without overlap | Implementation in PR [#3250] | | Exposure of feature to the user | Implementation in PR [#3250] | | Implementing Limits, Fairness and Observability for Cross-Tenant queries | Implementation in PR [#3250] | Those features are currently out of scope for this proposal, but we can foresee some interest implementing them after this proposal. Ability to use multi tenant queries in the ruler. It would be great if the source identifier could be made more pluggable. This could allow to for example base all of those features on another dimension (e.g. users rather than tenants) As per this proposal the label name `tenant_id` is fixed, but users might want to be able to modify that through a configuration option. As per this proposal the tenant label injection retains an existing label value, but this is not implemented recursively. So if the result already contains `tenant_id` and `originaltenant_id` labels, the value of the latter would be lost.