Project: Vector (Observability and Analysis / Observability)
File: 2020-08-27-3603-postgres-metrics.md

This RFC is to introduce a new metrics source to consume metrics from PostgreSQL database servers. The high level plan is to implement one source that collects metrics from PostgreSQL server instances. Background reading on PostgreSQL monitoring: https://www.datadoghq.com/blog/postgresql-monitoring/ This RFC will cover: A new source for PostgreSQL server metrics. This RFC will not cover: Other databases. Users want to collect, transform, and forward metrics to better observe how their PostgreSQL databases are performing. Build a single source called `postgresql_metrics` (name to be confirmed) to collect PostgreSQL metrics. We support all non-EOL'ed PostgreSQL versions. The recommended implementation is to use the Rust PostgreSQL client to connect the target database server by address specified in configuration. https://docs.rs/postgres/0.17.5/postgres/index.html The source would then run the following queries: `SELECT * FROM pgstatdatabase` `SELECT * FROM pgstatdatabase_conflicts` `SELECT * FROM pgstatbgwriter` And return these metrics by parsing the query results and converting them into metrics using the database name and column

`pg_up` -> Used as an uptime metric with 0 for successful collection and 1 for failed collection (gauge) `pgstatdatabaseblkreadtimeseconds_total` tagged with db, host, server, user (counter) `pgstatdatabaseblkwritetimeseconds_total` tagged with db, host, server, user (counter) `pgstatdatabaseblkshit_total` tagged with db, host, server, user (counter) `pgstatdatabaseblksread_total` tagged with db, host, server, user (counter) `pgstatdatabasestatsreset` tagged with host, server, user (gauge) `pgstatbgwriterbuffersalloc_total` tagged with host, server (counter) `pgstatbgwriterbuffersbackend_total` tagged with host, server (counter) `pgstatbgwriterbuffersbackendfsynctotal` tagged with host, server (counter) `pgstatbgwriterbufferscheckpoint_total` tagged with host, server (counter) `pgstatbgwriterbuffersclean_total` tagged with host, server (counter) `pgstatbgwritercheckpointsynctimeseconds_total` tagged with host, server (counter) `pgstatbgwritercheckpointwritetimeseconds_total` tagged with host, server (counter) `pgstatbgwritercheckpointsreq_total` tagged with host, server (counter) `pgstatbgwritercheckpointstimed_total` tagged with host, server (counter) `pgstatbgwritermaxwrittenclean_total` tagged with host, server (counter) `pgstatbgwriterstatsreset` host, server (gauge) `pgstatdatabaseconflictstotal` tagged with db, host, server, user (counter) `pgstatdatabase_datid` (database ID) tagged with db, host, server, user (gauge) `pgstatdatabasedeadlockstotal` tagged with db, host, server, user (counter) `pgstatdatabasenumbackendstotal` tagged with db, host, server, user (gauge) `pgstatdatabasetempbytes_total` tagged with db, host, server, user (counter) `pgstatdatabasetempfiles_total` tagged with db, host, server, user (counter) `pgstatdatabasetupdeleted_total` tagged with db, host, server, user (counter) `pgstatdatabasetupfetched_total` tagged with db, host, server, user (counter) `pgstatdatabasetupinserted_total` tagged with db, host, server, user (counter) `pgstatdatabasetupreturned_total` tagged with db, host, server, user (counter) `pgstatdatabasetupupdated_total` tagged with db, host, server, user (counter) `pgstatdatabasexactcommit_total` tagged with db, host, server, user (counter) `pgstatdatabasexactrollback_total` tagged with db, host, server, user (counter) `pgdatabaseconflicts_total` tagged by db name and server (counter) `pgdatabaseconflictsconflbufferpin_total` tagged by db name and server (counter) `pgdatabaseconflictsconfldeadlock_total` tagged by db name and server (counter) `pgdatabaseconflictsconfllock_total tagged` by db name and server (counter) `pgdatabaseconflictsconflsnapshot_total` tagged by db name and server (counter) `pgdatabaseconflictsconfltablespace_total` tagged by db name and server (counter) Naming of metrics is determined via: `tablename columnname + Prometheus endings (total for counters, etc)` For example: `pgdatabaseconflictsconfltablespace_total` Here `pgdatabaseconflicts` is the name of the table, `confltablespace` is the column name, and `total` is suffixed because counters end in `total` in the . This is in line with the Prometheus naming convention for their exporter. All metrics will also be tagged with the `endpoint` (stripped of username/password). The following additional source configuration will be added: ```toml [sources.mysourceid] type = "postgresql_metrics" # required endpoint = "postgres://postgres@localhost" # required - address of the PG server. included_databases = ["production", "testing"] # optional, list of databases to query. Defaults to all if not specified. excludeddatabases = [ "development" ] # optional, excludes specific databases. If a DB is excluded explicitly but included in `includeddatabases` then it is excluded. scrapeintervalsecs = 15 # optional, default, seconds namespace = "postgresql" # optional, default is "postgresql", namespace to attach to metrics. ``` We will also expose the HTTP SSL settings and support `ssl` in the `endpoint` URL. We'd also add a guide for doing this without root permissions. PostgreSQL is a commonly adopted modern database. Users frequently want to monitor its state and performance. Additionally, as part of Vector's vision to be the "one tool" for ingesting and shipping observability data, it makes sense to add as many sources as possible to reduce the likelihood that a user will not be able to ingest metrics from their tools. https://github.com/wrouesnel/postgres_exporter/ https://github.com/influxdata/telegraf/tree/master/plugins/inputs/postgresql https://collectd.org/wiki/index.php/Plugin:PostgreSQL https://collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_postgresql Additional maintenance and integration testing burden of a new source We could not add the source directly to Vector and instead instruct users to run Telegraf's `postgresql` input or Prometheus' `postgresql_exporter` and point Vector at the resulting data. This would leverage the already supported inputs from those projects. I decided against this as it would be in contrast with one of the listed principles of Vector: One Tool. All Data. - One simple tool gets your logs, metrics, and traces