Project: Druid (App Definition and Development / Database)
File: sql-metadata-tables.md

id: sql-metadata-tables title: "SQL metadata tables" sidebar_label: "SQL metadata tables" <!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, ~ software distributed under the License is distributed on an ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY ~ KIND, either express or implied. See the License for the ~ specific language governing permissions and limitations ~ under the License. --> :::info Apache Druid supports two query languages: Druid SQL and . This document describes the SQL language. ::: Druid Brokers infer table and column metadata for each datasource from segments loaded in the cluster, and use this to plan . This metadata is cached on Broker startup and also updated periodically in the background through . Background metadata refreshing is triggered by segments entering and exiting the cluster, and can also be throttled through configuration. Druid exposes system information through special system tables. There are two such schemas available: Information Schema and Sys Schema. Information schema provides details about table and column types. The "sys" schema provides information about Druid internals like segments/tasks/servers. You can access table and column metadata through JDBC using `connection.getMetaData()`, or through the INFORMATION_SCHEMA tables described below. For example, to retrieve metadata for the Druid datasource "foo", use the query: ```sql SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE "TABLESCHEMA" = 'druid' AND "TABLENAME" = 'foo' ``` :::info Note: INFORMATIONSCHEMA tables do not currently support Druid-specific functions like `TIMEPARSE` and `APPROXQUANTILEDS`. Only standard SQL functions can be used. ::: `INFORMATIONSCHEMA.SCHEMATA` provides a list of all known schemas, which include `druid` for standard , `lookup` for , `sys` for the virtual , and `INFORMATIONSCHEMA` for these virtual tables. Tables are allowed to have the same name across different schemas, so the schema may be included in an SQL statement to distinguish them, e.g. `lookup.table` vs `druid.table`. |Column|Type|Notes| ||-|--| |CATALOG_NAME|VARCHAR|Always set as `druid`| |SCHEMANAME|VARCHAR|`druid`, `lookup`, `sys`, or `INFORMATIONSCHEMA`| |SCHEMA_OWNER|VARCHAR|Unused| |DEFAULTCHARACTERSET_CATALOG|VARCHAR|Unused| |DEFAULTCHARACTERSET_SCHEMA|VARCHAR|Unused| |DEFAULTCHARACTERSET_NAME|VARCHAR|Unused| |SQL_PATH|VARCHAR|Unused| `INFORMATION_SCHEMA.TABLES` provides a list of all known tables and schemas. |Column|Type|Notes| ||-|--| |TABLE_CATALOG|VARCHAR|Always set as `druid`| |TABLE_SCHEMA|VARCHAR|The 'schema' which the table falls under, see | |TABLE_NAME|VARCHAR|Table name. For the `druid` schema, this is the `dataSource`.| |TABLETYPE|VARCHAR|"TABLE" or "SYSTEMTABLE"| |IS_JOINABLE|VARCHAR|If a table is directly joinable if on the right hand side of a `JOIN` statement, without performing a subquery, this value will be set to `YES`, otherwise `NO`. Lookups are always joinable because they are globally distributed among Druid query processing nodes, but Druid datasources are not, and will use a less efficient subquery join.| |IS_BROADCAST|VARCHAR|If a table is 'broadcast' and distributed among all Druid query processing nodes, this value will be set to `YES`, such as lookups and Druid datasources which have a 'broadcast' load rule, else `NO`.| `INFORMATION_SCHEMA.COLUMNS` provides a list of all known columns across all tables and

|Column|Type|Notes| ||-|--| |TABLE_CATALOG|VARCHAR|Always set as `druid`| |TABLE_SCHEMA|VARCHAR|The 'schema' which the table column falls under, see | |TABLE_NAME|VARCHAR|The 'table' which the column belongs to, see | |COLUMN_NAME|VARCHAR|The column name| |ORDINAL_POSITION|BIGINT|The order in which the column is stored in a table| |COLUMN_DEFAULT|VARCHAR|Unused| |IS_NULLABLE|VARCHAR|| |DATA_TYPE|VARCHAR|| |CHARACTERMAXIMUMLENGTH|BIGINT|Unused| |CHARACTEROCTETLENGTH|BIGINT|Unused| |NUMERIC_PRECISION|BIGINT|| |NUMERICPRECISIONRADIX|BIGINT|| |NUMERIC_SCALE|BIGINT|| |DATETIME_PRECISION|BIGINT|| |CHARACTERSETNAME|VARCHAR|| |COLLATION_NAME|VARCHAR|| |JDBC_TYPE|BIGINT|Type code from java.sql.Types (Druid extension)| For example, this query returns information for columns in the `foo` table: ```sql SELECT "ORDINALPOSITION", "COLUMNNAME", "ISNULLABLE", "DATATYPE", "JDBC_TYPE" FROM INFORMATION_SCHEMA.COLUMNS WHERE "TABLE_NAME" = 'foo' ``` `INFORMATION_SCHEMA.ROUTINES` provides a list of all known functions. |Column|Type| Notes| ||-|| |ROUTINE_CATALOG|VARCHAR| The catalog that contains the routine. Always set as `druid`| |ROUTINESCHEMA|VARCHAR| The schema that contains the routine. Always set as `INFORMATIONSCHEMA`| |ROUTINE_NAME|VARCHAR| THe routine name| |ROUTINE_TYPE|VARCHAR| The routine type. Always set as `FUNCTION`| |IS_AGGREGATOR|VARCHAR| If a routine is an aggregator function, then the value will be set to `YES`, else `NO`| |SIGNATURES|VARCHAR| One or more routine signatures| For example, this query returns information about all the aggregator functions: ```sql SELECT "ROUTINECATALOG", "ROUTINESCHEMA", "ROUTINENAME", "ROUTINETYPE", "IS_AGGREGATOR", "SIGNATURES" FROM "INFORMATION_SCHEMA"."ROUTINES" WHERE "IS_AGGREGATOR" = 'YES' ``` The "sys" schema provides visibility into Druid segments, servers and tasks. :::info Note: "sys" tables do not currently support Druid-specific functions like `TIME_PARSE` and `APPROXQUANTILEDS`. Only standard SQL functions can be used. ::: Segments table provides details on all Druid segments, whether they are published yet or not. |Column|Type|Notes| ||--|--| |segment_id|VARCHAR|Unique segment identifier| |datasource|VARCHAR|Name of datasource| |start|VARCHAR|Interval start time (in ISO 8601 format)| |end|VARCHAR|Interval end time (in ISO 8601 format)| |size|BIGINT|Size of segment in bytes| |version|VARCHAR|Version string (generally an ISO8601 timestamp corresponding to when the segment set was first started). Higher version means the more recently created segment. Version comparing is based on string comparison.| |partition_num|BIGINT|Partition number (an integer, unique within a datasource+interval+version; may not necessarily be contiguous)| |num_replicas|BIGINT|Number of replicas of this segment currently being served| |numrows|BIGINT|Number of rows in this segment, or zero if the number of rows is not known.<br /><br />This row count is gathered by the Broker in the background. It will be zero if the Broker has not gathered a row count for this segment yet. For segments ingested from streams, the reported row count may lag behind the result of a `count(*)` query because the cached `numrows` on the Broker may be out of date. This will settle shortly after new rows stop being written to that particular segment.| |isactive|BIGINT|True for segments that represent the latest state of a datasource.<br /><br />Equivalent to `(ispublished = 1 AND isovershadowed = 0) OR isrealtime = 1`. In steady state, when no ingestion or data management operations are happening, `isactive` will be equivalent to `isavailable`. However, they may differ from each other when ingestion or data management operations have executed recently. In these cases, Druid will load and unload segments appropriately to bring actual availability in line with the expected state given by `is_active`.| |is_published|BIGINT|Boolean represented as long type where 1 = true, 0 = false. 1 if this segment has been published to the metadata store and is marked as used. See the for more details.| |is_available|BIGINT|Boolean represented as long type where 1 = true, 0 = false. 1 if this segment is currently being served by any data serving process, like a Historical or a realtime ingestion task. See the for more details.| |isrealtime|BIGINT|Boolean represented as long type where 1 = true, 0 =

1 if this segment is only_ served by realtime tasks, and 0 if any Historical process is serving this segment.| |isovershadowed|BIGINT|Boolean represented as long type where 1 = true, 0 = false. 1 if this segment is published and is fully overshadowed by some other published segments. Currently, `isovershadowed` is always 0 for unpublished segments, although this may change in the future. You can filter for segments that "should be published" by filtering for `ispublished = 1 AND isovershadowed = 0`. Segments can briefly be both published and overshadowed if they were recently replaced, but have not been unpublished yet. See the for more details.| |shard_spec|VARCHAR|JSON-serialized form of the segment `ShardSpec`| |dimensions|VARCHAR|JSON-serialized form of the segment dimensions| |metrics|VARCHAR|JSON-serialized form of the segment metrics| |lastcompactionstate|VARCHAR|JSON-serialized form of the compaction task's config (compaction task which created this segment). May be null if segment was not created by compaction task.| |replication_factor|BIGINT|Total number of replicas of the segment that are required to be loaded across all historical tiers, based on the load rule that currently applies to this segment. If this value is 0, the segment is not assigned to any historical and will not be loaded. This value is -1 if load rules for the segment have not been evaluated yet.| For example, to retrieve all currently active segments for datasource "wikipedia", use the query: ```sql SELECT * FROM sys.segments WHERE datasource = 'wikipedia' AND is_active = 1 ``` Another example to retrieve segments totalsize, avgsize, avgnumrows and num_segments per datasource: ```sql SELECT datasource, SUM("size") AS total_size, CASE WHEN SUM("size") = 0 THEN 0 ELSE SUM("size") / (COUNT(*) FILTER(WHERE "size" > 0)) END AS avg_size, CASE WHEN SUM(numrows) = 0 THEN 0 ELSE SUM("numrows") / (COUNT(*) FILTER(WHERE numrows > 0)) END AS avgnum_rows, COUNT(*) AS num_segments FROM sys.segments WHERE is_active = 1 GROUP BY 1 ORDER BY 2 DESC ``` This query goes a step further and shows the overall profile of available, non-realtime segments across buckets of 1 million rows each for the `foo` datasource: ```sql SELECT ABS("num_rows" / 1000000) as "bucket", COUNT(*) as segments, SUM("size") / 1048576 as totalSizeMiB, MIN("size") / 1048576 as minSizeMiB, AVG("size") / 1048576 as averageSizeMiB, MAX("size") / 1048576 as maxSizeMiB, SUM("num_rows") as totalRows, MIN("num_rows") as minRows, AVG("num_rows") as averageRows, MAX("num_rows") as maxRows, (AVG("size") / AVG("num_rows")) as avgRowSizeB FROM sys.segments WHERE isavailable = 1 AND isrealtime = 0 AND "datasource" = `foo` GROUP BY 1 ORDER BY 1 ``` If you want to retrieve segment that was compacted (ANY compaction): ```sql SELECT * FROM sys.segments WHERE isactive = 1 AND lastcompaction_state IS NOT NULL ``` or if you want to retrieve segment that was compacted only by a particular compaction spec (such as that of the auto compaction): ```sql SELECT * FROM sys.segments WHERE isactive = 1 AND lastcompaction_state = 'CompactionState{partitionsSpec=DynamicPartitionsSpec{maxRowsPerSegment=5000000, maxTotalRows=9223372036854775807}, indexSpec={bitmap={type=roaring}, dimensionCompression=lz4, metricCompression=lz4, longEncoding=longs, segmentLoader=null}}' ``` Servers table lists all discovered servers in the cluster. |Column|Type|Notes| ||--|--| |server|VARCHAR|Server name in the form host:port| |host|VARCHAR|Hostname of the server| |plaintext_port|BIGINT|Unsecured port of the server, or -1 if plaintext traffic is disabled| |tls_port|BIGINT|TLS port of the server, or -1 if TLS is disabled| |servertype|VARCHAR|Type of Druid service. Possible values include: COORDINATOR, OVERLORD, BROKER, ROUTER, HISTORICAL, MIDDLEMANAGER or PEON.| |tier|VARCHAR|Distribution tier see . Only valid for HISTORICAL type, for other types it's null| |current_size|BIGINT|Current size of segments in bytes on this

Only valid for HISTORICAL type, for other types it's 0| |max_size|BIGINT|Max size in bytes this server recommends to assign to segments see . Only valid for HISTORICAL type, for other types it's 0| |is_leader|BIGINT|1 if the server is currently the 'leader' (for services which have the concept of leadership), otherwise 0 if the server is not the leader, or the default long value (null or zero depending on `druid.generic.useDefaultValueForNull`) if the server type does not have the concept of leadership| |start_time|STRING|Timestamp in ISO8601 format when the server was announced in the cluster| To retrieve information about all servers, use the query: ```sql SELECT * FROM sys.servers; ``` SERVER_SEGMENTS is used to join servers with segments table |Column|Type|Notes| ||--|--| |server|VARCHAR|Server name in format host:port (Primary key of )| |segment_id|VARCHAR|Segment identifier (Primary key of )| JOIN between "servers" and "segments" can be used to query the number of segments for a specific datasource, grouped by server, example query: ```sql SELECT count(segments.segmentid) as numsegments from sys.segments as segments INNER JOIN sys.serversegments as serversegments ON segments.segmentid = serversegments.segment_id INNER JOIN sys.servers as servers ON servers.server = server_segments.server WHERE segments.datasource = 'wikipedia' GROUP BY servers.server; ``` The tasks table provides information about active and recently-completed indexing tasks. For more information check out the documentation for . |Column|Type|Notes| ||--|--| |task_id|VARCHAR|Unique task identifier| |groupid|VARCHAR|Task group ID for this task, the value depends on the task `type`. For example, for native index tasks, it's same as `taskid`, for sub tasks, this value is the parent task's ID| |type|VARCHAR|Task type, for example this value is "index" for indexing tasks. See | |datasource|VARCHAR|Datasource name being indexed| |created_time|VARCHAR|Timestamp in ISO8601 format corresponding to when the ingestion task was created. Note that this value is populated for completed and waiting tasks. For running and pending tasks this value is set to 1970-01-01T00:00:00Z| |queueinsertiontime|VARCHAR|Timestamp in ISO8601 format corresponding to when this task was added to the queue on the Overlord| |status|VARCHAR|Status of a task can be RUNNING, FAILED, SUCCESS| |runner_status|VARCHAR|Runner status of a completed task would be NONE, for in-progress tasks this can be RUNNING, WAITING, PENDING| |duration|BIGINT|Time it took to finish the task in milliseconds, this value is present only for completed tasks| |location|VARCHAR|Server name where this task is running in the format host:port, this information is present only for RUNNING tasks| |host|VARCHAR|Hostname of the server where task is running| |plaintext_port|BIGINT|Unsecured port of the server, or -1 if plaintext traffic is disabled| |tls_port|BIGINT|TLS port of the server, or -1 if TLS is disabled| |error_msg|VARCHAR|Detailed error message in case of FAILED tasks| For example, to retrieve tasks information filtered by status, use the query ```sql SELECT * FROM sys.tasks WHERE status='FAILED'; ``` The supervisors table provides information about supervisors. |Column|Type|Notes| ||--|--| |supervisor_id|VARCHAR|Supervisor task identifier| |state|VARCHAR|Basic state of the supervisor. Available states: `UNHEALTHYSUPERVISOR`, `UNHEALTHYTASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. See for more information.| |detailed_state|VARCHAR|Supervisor specific state. See documentation of the specific supervisor for details: or .| |healthy|BIGINT|Boolean represented as long type where 1 = true, 0 = false. 1 indicates a healthy supervisor| |type|VARCHAR|Type of supervisor, e.g. `kafka`, `kinesis` or `materialized_view`| |source|VARCHAR|Source of the supervisor, e.g. Kafka topic or Kinesis stream| |suspended|BIGINT|Boolean represented as long type where 1 = true, 0 = false. 1 indicates supervisor is in suspended state| |spec|VARCHAR|JSON-serialized supervisor spec| For example, to retrieve supervisor tasks information filtered by health status, use the query ```sql SELECT * FROM sys.supervisors WHERE healthy=0; ```