Package: query-exporter
Export Prometheus metrics from SQL queriesquery-exporteris aPrometheusexporter which allows collecting metrics
from database queries, at specified time intervals.It usesSQLAlchemyto connect to different database engines, including
PostgreSQL, MySQL, Oracle and Microsoft SQL Server.Each query can be run on multiple databases, and update multiple metrics.The application is simply run as:query-exporter config.yamlwhere the passed configuration file contains the definitions of the databases
to connect and queries to perform to update metrics.Configuration file formatA sample configuration file for the application looks like this:databases:db1:dsn:sqlite://connect-sql:-PRAGMA application_id = 123-PRAGMA auto_vacuum = 1labels:region:us1app:app1db2:dsn:sqlite://keep-connected:falselabels:region:us2app:app1metrics:metric1:type:gaugedescription:A sample gaugemetric2:type:summarydescription:A sample summarylabels:[l1,l2]expiration:24hmetric3:type:histogramdescription:A sample histogrambuckets:[10,20,50,100,1000]metric4:type:enumdescription:A sample enumstates:[foo,bar,baz]queries:query1:interval:5databases:[db1]metrics:[metric1]sql:SELECT random() / 1000000000000000 AS metric1query2:interval:20timeout:0.5databases:[db1,db2]metrics:[metric2,metric3]sql:|SELECT abs(random() / 1000000000000000) AS metric2,abs(random() / 10000000000000000) AS metric3,"value1" AS l1,"value2" AS l2query3:schedule:"*/5****"databases:[db2]metrics:[metric4]sql:|SELECT value FROM (SELECT "foo" AS metric4 UNIONSELECT "bar" AS metric3 UNIONSELECT "baz" AS metric4)ORDER BY random()LIMIT 1databasessectionThis section contains definitions for databases to connect to. Key names are
arbitrary and only used to reference databases in thequeriessection.Each database definitions can have the following keys:dsn:database connection details.It can be provided as a string in the following format:dialect[+driver]://[username:password][@host:port]/database[?option=value&...](seeSQLAlchemy documentationfor details on available engines and
options), or as key/value pairs:dialect:<dialect>[+driver]user:<username>password:<password>host:<host>port:<port>database:<database>options:<key1>:<value1><key2>:<value2>All entries are optional, exceptdialect.Note that in the string form, username, password and options need to be
URL-encoded, whereas this is done automatically for the key/value form.Seedatabase-specific optionspage for some extra details on database
configuration options.It’s also possible to get the connection string indirectly from other sources:from an environment variable (e.g.$CONNECTION_STRING) by settingdsnto:env:CONNECTION_STRINGfrom a file, containing only the DSN value, by settingdsnto:file:/path/to/fileThese forms only support specifying the actual DNS in the string form.connect-sql:An optional list of queries to run right after database connection. This can
be used to set up connection-wise parameters and configurations.keep-connected:whether to keep the connection open for the database between queries, or
disconnect after each one. If not specified, defaults totrue.  Setting
this option tofalsemight be useful if queries on a database are run
with very long interval, to avoid holding idle connections.autocommit:whether to set autocommit for the database connection. If not specified,
defaults totrue.  This should only be changed tofalseif specific
queries require it.labels:an optional mapping of label names and values to tag metrics collected from each database.
When labels are used, all databases must define the same set of labels.metricssectionThis section containsPrometheusmetrics definitions. Keys are used as metric
names, and must therefore be valid metric identifiers.Each metric definition can have the following keys:type:the type of the metric, must be specified. The following metric types are
supported:counter: value is incremented with each result from queriesenum: value is set with each result from queriesgauge: value is set with each result from querieshistogram: each result from queries is added to observationssummary: each result from queries is added to observationsdescription:an optional description of the metric.labels:an optional list of label names to apply to the metric.If specified, queries updating the metric must return rows that include
values for each label in addition to the metric value.  Column names must
match metric and labels names.buckets:forhistogrammetrics, a list of buckets for the metrics.If not specified, default buckets are applied.states:forenummetrics, a list of string values for possible states.Queries for updating the enum must return valid states.expiration:the amount of time after which a series for the metric is cleared if no new
value is collected.Last report times are tracked independently for each set of label values for
the metric.This can be useful for metric series that only last for a certain amount of
time, to avoid an ever-increasing collection of series.The value is interpreted as seconds if no suffix is specified; valid suffixes
ares,m,h,d. Only integer values are accepted.increment:forcountermetrics, whether to increment the value by the query result,
or set the value to it.By default, counters are incremented by the value returned by the query. If
this is set tofalse, instead, the metric value will be set to the result
of the query.NOTE: The default will be reversed in the 3.0 release, andincrementwill be set tofalseby default.queriessectionThis section contains definitions for queries to perform. Key names are
arbitrary and only used to identify queries in logs.Each query definition can have the following keys:databases:the list of databases to run the query on.Names must match those defined in thedatabasessection.Metrics are automatically tagged with thedatabaselabel so that
independent series are generated for each database that a query is run on.interval:the time interval at which the query is run.The value is interpreted as seconds if no suffix is specified; valid suffixes
ares,m,h,d. Only integer values are accepted.If a value is specified forinterval, aschedulecan’t be specified.If no value is specified (or specified asnull), the query is only
executed upon HTTP requests.metrics:the list of metrics that the query updates.Names must match those defined in themetricssection.parameters:an optional list or dictionary of parameters sets to run the query with.If specified as a list, the query will be run once for every set of
parameters specified in this list, for every interval.Each parameter set must be a dictionary where keys must match parameters
names from the query SQL (e.g.:param).As an example:query:databases:[db]metrics:[metric]sql:|SELECT COUNT(*) AS metric FROM tableWHERE id > :param1 AND id < :param2parameters:-param1:10param2:20-param1:30param2:40If specified as a dictionary, it’s used as a multidimensional matrix of
parameters lists to run the query with.
The query will be run once for each permutation of parameters.If a query is specified with parameters as matrix in itssql, it will be run once
for every permutation in matrix of parameters, for every interval.Variable format in sql query::{top_level_key}__{inner_key}query:databases:[db]metrics:[apps_count]sql:|SELECT COUNT(1) AS apps_count FROM apps_listWHERE os = :os__name AND arch = :os__arch AND lang = :lang__nameparameters:os:-name:MacOSarch:arm64-name:Linuxarch:amd64-name:Windowsarch:amd64lang:-name:Python3-name:Java-name:TypeScriptThis example will generate 9 queries with all permutations ofosandlangparameters.schedule:a schedule for executing queries at specific times.This is expressed as a Cron-like format string (e.g.*/5 * * * *to run
every five minutes).If a value is specified forschedule, anintervalcan’t be specified.If no value is specified (or specified asnull), the query is only
executed upon HTTP requests.sql:the SQL text of the query.The query must return columns with names that match those of the metrics
defined inmetrics, plus those of labels (if any) for all these metrics.query:databases:[db]metrics:[metric1,metric2]sql:SELECT 10.0 AS metric1, 20.0 AS metric2will updatemetric1to10.0andmetric2to20.0.Note:since:is used for parameter markers (seeparametersabove),
literal single:at the beginning of a word must be escaped with
backslash (e.g.SELECT'\:bar'FROM table).  There’s no need to escape
when the colon occurs inside a word (e.g.SELECT 'foo:bar' FROM table).timeout:a value in seconds after which the query is timed out.If specified, it must be a multiple of 0.1.Metrics endpointThe exporter listens on port9560providing the standard/metricsendpoint.By default, the port is bound onlocalhost. Note that if the name resolves
both IPv4 and IPv6 addressses, the exporter will bind on both.For the configuration above, the endpoint would return something like this:# HELP database_errors_total Number of database errors
# TYPE database_errors_total counter
# HELP queries_total Number of database queries
# TYPE queries_total counter
queries_total{app="app1",database="db1",query="query1",region="us1",status="success"} 50.0
queries_total{app="app1",database="db2",query="query2",region="us2",status="success"} 13.0
queries_total{app="app1",database="db1",query="query2",region="us1",status="success"} 13.0
queries_total{app="app1",database="db2",query="query3",region="us2",status="error"} 1.0
# HELP queries_created Number of database queries
# TYPE queries_created gauge
queries_created{app="app1",database="db1",query="query1",region="us1",status="success"} 1.5945442444463024e+09
queries_created{app="app1",database="db2",query="query2",region="us2",status="success"} 1.5945442444471517e+09
queries_created{app="app1",database="db1",query="query2",region="us1",status="success"} 1.5945442444477117e+09
queries_created{app="app1",database="db2",query="query3",region="us2",status="error"} 1.5945444000140696e+09
# HELP query_latency Query execution latency
# TYPE query_latency histogram
query_latency_bucket{app="app1",database="db1",le="0.005",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.01",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.025",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.05",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.075",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.1",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.25",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.5",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="0.75",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="1.0",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="2.5",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="5.0",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="7.5",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="10.0",query="query1",region="us1"} 50.0
query_latency_bucket{app="app1",database="db1",le="+Inf",query="query1",region="us1"} 50.0
query_latency_count{app="app1",database="db1",query="query1",region="us1"} 50.0
query_latency_sum{app="app1",database="db1",query="query1",region="us1"} 0.004666365042794496
query_latency_bucket{app="app1",database="db2",le="0.005",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.01",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.025",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.05",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.075",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.1",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.25",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.5",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="0.75",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="1.0",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="2.5",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="5.0",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="7.5",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="10.0",query="query2",region="us2"} 13.0
query_latency_bucket{app="app1",database="db2",le="+Inf",query="query2",region="us2"} 13.0
query_latency_count{app="app1",database="db2",query="query2",region="us2"} 13.0
query_latency_sum{app="app1",database="db2",query="query2",region="us2"} 0.012369773990940303
query_latency_bucket{app="app1",database="db1",le="0.005",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.01",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.025",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.05",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.075",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.1",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.25",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.5",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="0.75",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="1.0",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="2.5",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="5.0",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="7.5",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="10.0",query="query2",region="us1"} 13.0
query_latency_bucket{app="app1",database="db1",le="+Inf",query="query2",region="us1"} 13.0
query_latency_count{app="app1",database="db1",query="query2",region="us1"} 13.0
query_latency_sum{app="app1",database="db1",query="query2",region="us1"} 0.004745393933262676
# HELP query_latency_created Query execution latency
# TYPE query_latency_created gauge
query_latency_created{app="app1",database="db1",query="query1",region="us1"} 1.594544244446163e+09
query_latency_created{app="app1",database="db2",query="query2",region="us2"} 1.5945442444470239e+09
query_latency_created{app="app1",database="db1",query="query2",region="us1"} 1.594544244447551e+09
# HELP query_timestamp Query last execution timestamp
# TYPE query_timestamp gauge
query_timestamp{app="app1",database="db2",query="query2",region="us2"} 1.594544244446199e+09
query_timestamp{app="app1",database="db1",query="query1",region="us1"} 1.594544244452181e+09
query_timestamp{app="app1",database="db1",query="query2",region="us1"} 1.594544244481839e+09
# HELP metric1 A sample gauge
# TYPE metric1 gauge
metric1{app="app1",database="db1",region="us1"} -3561.0
# HELP metric2 A sample summary
# TYPE metric2 summary
metric2_count{app="app1",database="db2",l1="value1",l2="value2",region="us2"} 13.0
metric2_sum{app="app1",database="db2",l1="value1",l2="value2",region="us2"} 58504.0
metric2_count{app="app1",database="db1",l1="value1",l2="value2",region="us1"} 13.0
metric2_sum{app="app1",database="db1",l1="value1",l2="value2",region="us1"} 75262.0
# HELP metric2_created A sample summary
# TYPE metric2_created gauge
metric2_created{app="app1",database="db2",l1="value1",l2="value2",region="us2"} 1.594544244446819e+09
metric2_created{app="app1",database="db1",l1="value1",l2="value2",region="us1"} 1.594544244447339e+09
# HELP metric3 A sample histogram
# TYPE metric3 histogram
metric3_bucket{app="app1",database="db2",le="10.0",region="us2"} 1.0
metric3_bucket{app="app1",database="db2",le="20.0",region="us2"} 1.0
metric3_bucket{app="app1",database="db2",le="50.0",region="us2"} 2.0
metric3_bucket{app="app1",database="db2",le="100.0",region="us2"} 3.0
metric3_bucket{app="app1",database="db2",le="1000.0",region="us2"} 13.0
metric3_bucket{app="app1",database="db2",le="+Inf",region="us2"} 13.0
metric3_count{app="app1",database="db2",region="us2"} 13.0
metric3_sum{app="app1",database="db2",region="us2"} 5016.0
metric3_bucket{app="app1",database="db1",le="10.0",region="us1"} 0.0
metric3_bucket{app="app1",database="db1",le="20.0",region="us1"} 0.0
metric3_bucket{app="app1",database="db1",le="50.0",region="us1"} 0.0
metric3_bucket{app="app1",database="db1",le="100.0",region="us1"} 0.0
metric3_bucket{app="app1",database="db1",le="1000.0",region="us1"} 13.0
metric3_bucket{app="app1",database="db1",le="+Inf",region="us1"} 13.0
metric3_count{app="app1",database="db1",region="us1"} 13.0
metric3_sum{app="app1",database="db1",region="us1"} 5358.0
# HELP metric3_created A sample histogram
# TYPE metric3_created gauge
metric3_created{app="app1",database="db2",region="us2"} 1.5945442444469101e+09
metric3_created{app="app1",database="db1",region="us1"} 1.5945442444474254e+09
# HELP metric4 A sample enum
# TYPE metric4 gauge
metric4{app="app1",database="db2",metric4="foo",region="us2"} 0.0
metric4{app="app1",database="db2",metric4="bar",region="us2"} 0.0
metric4{app="app1",database="db2",metric4="baz",region="us2"} 1.0Builtin metricsThe exporter provides a few builtin metrics which can be useful to track query execution:database_errors{database="db"}:a counter used to report number of errors, per database.queries{database="db",query="q",status="[success|error|timeout]"}:a counter with number of executed queries, per database, query and status.query_latency{database="db",query="q"}:a histogram with query latencies, per database and query.query_timestamp{database="db",query="q"}:a gauge with query last execution timestamps, per database and query.In addition, metrics for resources usage for the exporter process can be
included by passing--process-statsin the command line.Debugging / LogsYou can enable extended logging using the-Lcommandline switch. Possible
log levels areCRITICAL,ERROR,WARNING,INFO,DEBUG.Database enginesSQLAlchemydoesn’t depend on specific Python database modules at
installation. This means additional modules might need to be installed for
engines in use. These can be installed as follows:pip install SQLAlchemy[postgresql] SQLAlchemy[mysql] ...based on which database engines are needed.Seesupported databasesfor details.Install from Snapquery-exportercan be installed fromSnap Storeon systems where Snaps
are supported, via:sudo snap install query-exporterThe snap provides both thequery-exportercommand and a daemon instance of
the command, managed via a Systemd service.To configure the daemon:create or edit/var/snap/query-exporter/current/config.yamlwith the
configurationrunsudo snap restartquery-exporterThe snap has support for connecting the following databases:PostgreSQL (postgresql://)MySQL (mysql://)SQLite (sqlite://)Microsoft SQL Server (mssql://)IBM DB2 (db2://) on supported architectures (x86_64, ppc64le and
s390x)Run in Dockerquery-exportercan be run insideDockercontainers, and is available from
theDocker Hub:docker run --rm -it -p 9560:9560/tcp -v "$CONFIG_DIR:/config" adonato/query-exporter:latestwhere$CONFIG_DIRis the absolute path of a directory containing aconfig.yamlfile, the configuration file to use. Alternatively, a volume
name can be specified.A different ODBC driver version to use can be specified during image building,
by passing--build-argODBC_bVERSION_NUMBER, e.g.:docker build . --build-arg ODBC_DRIVER_VERSION=17The image has support for connecting the following databases:PostgreSQL (postgresql://)MySQL (mysql://)SQLite (sqlite://)Microsoft SQL Server (mssql://)IBM DB2 (db2://)Oracle (oracle://)ClickHouse (clickhouse+native://)AHelm chartto run the container in Kubernetes is also available.