Project: CloudNativePG (App Definition and Development / Database)
File: connection_pooling.md

CloudNativePG provides native support for connection pooling with , one of the most popular open source connection poolers for PostgreSQL, through the `Pooler` custom resource definition (CRD). In brief, a pooler in CloudNativePG is a deployment of PgBouncer pods that sits between your applications and a PostgreSQL service, for example, the `rw` service. It creates a separate, scalable, configurable, and highly available database access layer. The following diagram highlights how introducing a database access layer based on PgBouncer changes the architecture of CloudNativePG. Instead of directly connecting to the PostgreSQL primary service, applications can connect to the equivalent service for PgBouncer. This ability enables reuse of existing connections for faster performance and better resource management on the PostgreSQL side. This example helps to show how CloudNativePG implements a PgBouncer pooler: ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Pooler metadata: name: pooler-example-rw spec: cluster: name: cluster-example instances: 3 type: rw pgbouncer: poolMode: session parameters: maxclientconn: "1000" defaultpoolsize: "10" ``` !!! Important The pooler name can't be the same as any cluster name in the same namespace. This example creates a `Pooler` resource called `pooler-example-rw` that's strictly associated with the Postgres `Cluster` resource called `cluster-example`. It points to the primary, identified by the read/write service (`rw`, therefore `cluster-example-rw`). The `Pooler` resource must live in the same namespace as the Postgres cluster. It consists of a Kubernetes deployment of 3 pods running the , configured with the and accepting up to 1000 connections each. The default pool size is 10 user/database pairs toward PostgreSQL. !!! Important The `Pooler` resource sets only the `*` fallback database in PgBouncer. This setting means that that all parameters in the connection strings passed from the client are relayed to the PostgreSQL server. For details, see ["Section [databases]" in the PgBouncer documentation](https://www.pgbouncer.org/config.html#section-databases). CloudNativePG also creates a secret with the same name as the pooler containing the configuration files used with PgBouncer. !!! Seealso "API reference" For details, see in the API reference. `Pooler` resources aren't cluster-managed resources. You create poolers manually when they're needed. You can also deploy multiple poolers per PostgreSQL cluster. What's important is that the life cycles of the `Cluster` and the `Pooler` resources are currently independent. Deleting the cluster doesn't imply the deletion of the pooler, and vice versa. !!! Important Once you know how a pooler works, you have full freedom in terms of possible architectures. You can have clusters without poolers, clusters with a single pooler, or clusters with several poolers, that is, one per application. Any PgBouncer pooler is transparently integrated with CloudNativePG support for in-transit encryption by way of TLS connections, both on the client (application) and server (PostgreSQL) side of the pool. Specifically, PgBouncer reuses the certificates of the PostgreSQL server. It also uses TLS client certificate authentication to connect to the PostgreSQL server to run the `auth_query` for clients' password authentication (see ). Containers run as the pgbouncer system user, and access to the `pgbouncer` database is allowed only by way of local connections, through peer

By default, a PgBouncer pooler uses the same certificates that are used by the cluster. However, if you provide those certificates, the pooler accepts secrets with the following formats: Basic Auth TLS Opaque In the Opaque case, it looks for the following specific keys that need to be used: tls.crt tls.key So you can treat this secret as a TLS secret, and start from there. Password-based authentication is the only supported method for clients of PgBouncer in CloudNativePG. Internally, the implementation relies on PgBouncer's `auth_user` and `auth_query` options. Specifically, the operator: Creates a standard user called `cnpgpoolerpgbouncer` in the PostgreSQL server Creates the lookup function in the `postgres` database and grants execution privileges to the cnpgpoolerpgbouncer user (PoLA) Issues a TLS certificate for this user Sets `cnpgpoolerpgbouncer` as the `auth_user` Configures PgBouncer to use the TLS certificate to authenticate `cnpgpoolerpgbouncer` against the PostgreSQL server Removes all the above when it detects that a cluster doesn't have any pooler associated to it !!! Important If you specify your own secrets, the operator doesn't automatically integrate the pooler. To manually integrate the pooler, if you specified your own secrets, you must run the following queries from inside your cluster. First, you must create the role: ```sql CREATE ROLE cnpgpoolerpgbouncer WITH LOGIN; ``` Then, for each application database, grant the permission for `cnpgpoolerpgbouncer` to connect to it: ```sql GRANT CONNECT ON DATABASE { database name here } TO cnpgpoolerpgbouncer; ``` Finally, as a superuser connect in each application database, and then create the authentication function inside each of the application databases: ```sql CREATE OR REPLACE FUNCTION public.user_search(uname TEXT) RETURNS TABLE (usename name, passwd text) LANGUAGE sql SECURITY DEFINER AS 'SELECT usename, passwd FROM pgcatalog.pgshadow WHERE usename=$1;'; REVOKE ALL ON FUNCTION public.user_search(text) FROM public; GRANT EXECUTE ON FUNCTION public.user_search(text) TO cnpgpoolerpgbouncer; ``` !!! Important Given that `user_search` is a `SECURITY DEFINER` function, you need to create it through a role with `SUPERUSER` privileges, such as the `postgres` user. You can take advantage of pod templates specification in the `template` section of a `Pooler` resource. For details, see in the API reference. Using templates, you can configure pods as you like, including fine control over affinity and anti-affinity rules for pods and nodes. By default, containers use images from `ghcr.io/cloudnative-pg/pgbouncer`. This example shows `Pooler` specifying `PodAntiAffinity``: ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Pooler metadata: name: pooler-example-rw spec: cluster: name: cluster-example instances: 3 type: rw template: metadata: labels: app: pooler spec: containers: [] affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: labelSelector: matchExpressions: key: app operator: In values: pooler topologyKey: "kubernetes.io/hostname" ``` !!! Note Explicitly set `.spec.template.spec.containers` to `[]` when not modified, as it's a required field for a `PodSpec`. If `.spec.template.spec.containers` isn't set, the Kubernetes api-server returns the following error when trying to apply the manifest:`error validating "pooler.yaml": error validating data: ValidationError(Pooler.spec.template.spec): missing required field "containers"` This example sets resources and changes the used image: ```yaml apiVersion:

kind: Pooler metadata: name: pooler-example-rw spec: cluster: name: cluster-example instances: 3 type: rw template: metadata: labels: app: pooler spec: containers: name: pgbouncer image: my-pgbouncer:latest resources: requests: cpu: "0.1" memory: 100Mi limits: cpu: "0.5" memory: 500Mi ``` Sometimes, your pooler will require some different labels, annotations, or even change the type of the service, you can achieve that by using the `serviceTemplate` field: ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Pooler metadata: name: pooler-example-rw spec: cluster: name: cluster-example instances: 3 type: rw serviceTemplate: metadata: labels: app: pooler spec: type: LoadBalancer pgbouncer: poolMode: session parameters: maxclientconn: "1000" defaultpoolsize: "10" ``` Because of Kubernetes' deployments, you can configure your pooler to run on a single instance or over multiple pods. The exposed service makes sure that your clients are randomly distributed over the available pods running PgBouncer, which then manages and reuses connections toward the underlying server (if using the `rw` service) or servers (if using the `ro` service with multiple replicas). !!! Warning If your infrastructure spans multiple availability zones with high latency across them, be aware of network hops. Consider, for example, the case of your application running in zone 2, connecting to PgBouncer running in zone 3, and pointing to the PostgreSQL primary in zone 1. The operator manages most of the , allowing you to modify only a subset of them. !!! Warning You are responsible for correctly setting the value of each option, as the operator doesn't validate them. These are the PgBouncer options you can customize, with links to the PgBouncer documentation for each parameter. Unless stated otherwise, the default values are the ones directly set by PgBouncer. : to be appended to `extrafloatdigits,options` - required by CNP : by default disabled (`0`), given that statistics are already collected by the Prometheus export as described in the section below Customizations of the PgBouncer configuration are written declaratively in the `.spec.pgbouncer.parameters` map. The operator reacts to the changes in the pooler specification, and every PgBouncer instance reloads the updated configuration without disrupting the service. !!! Warning Every PgBouncer pod has the same configuration, aligned with the parameters in the specification. A mistake in these parameters might disrupt the operability of the whole pooler. The operator doesn't validate the value of any option. The PgBouncer implementation of the `Pooler` comes with a default Prometheus exporter. It makes available several metrics having the `cnpgpgbouncer` prefix by running: `SHOW LISTS` (prefix: `cnpgpgbouncerlists`) `SHOW POOLS` (prefix: `cnpgpgbouncerpools`) `SHOW STATS` (prefix: `cnpgpgbouncerstats`) Like the CloudNativePG instance, the exporter runs on port `9127` of each pod running PgBouncer and also provides metrics related to the Go runtime (with the prefix `go_*`). !!! Info You can inspect the exported metrics on a pod running PgBouncer. For instructions, see . Make sure that you use the correct IP and the `9127` port. This example shows the output for `cnpg_pgbouncer` metrics: ```text cnpgpgbouncercollectiondurationseconds{collector="Collect.up"}

cnpgpgbouncercollections_total 1 cnpgpgbouncerlastcollectionerror 0 cnpgpgbouncerlists_databases 1 cnpgpgbouncerlistsdnsnames 0 cnpgpgbouncerlistsdnspending 0 cnpgpgbouncerlistsdnsqueries 0 cnpgpgbouncerlistsdnszones 0 cnpgpgbouncerlistsfreeclients 49 cnpgpgbouncerlistsfreeservers 0 cnpgpgbouncerlistsloginclients 0 cnpgpgbouncerlists_pools 1 cnpgpgbouncerlistsusedclients 1 cnpgpgbouncerlistsusedservers 0 cnpgpgbouncerlists_users 2 cnpgpgbouncerpoolsclactive{database="pgbouncer",user="pgbouncer"} 1 cnpgpgbouncerpoolsclcancel_req{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpoolsclwaiting{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpools_maxwait{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpoolsmaxwaitus{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpoolspoolmode{database="pgbouncer",user="pgbouncer"} 3 cnpgpgbouncerpoolssvactive{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpoolssvidle{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpoolssvlogin{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpoolssvtested{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerpoolssvused{database="pgbouncer",user="pgbouncer"} 0 cnpgpgbouncerstatsavgquery_count{database="pgbouncer"} 1 cnpgpgbouncerstatsavgquery_time{database="pgbouncer"} 0 cnpgpgbouncerstatsavgrecv{database="pgbouncer"} 0 cnpgpgbouncerstatsavgsent{database="pgbouncer"} 0 cnpgpgbouncerstatsavgwait_time{database="pgbouncer"} 0 cnpgpgbouncerstatsavgxact_count{database="pgbouncer"} 1 cnpgpgbouncerstatsavgxact_time{database="pgbouncer"} 0 cnpgpgbouncerstatstotalquery_count{database="pgbouncer"} 3 cnpgpgbouncerstatstotalquery_time{database="pgbouncer"} 0 cnpgpgbouncerstatstotalreceived{database="pgbouncer"} 0 cnpgpgbouncerstatstotalsent{database="pgbouncer"} 0 cnpgpgbouncerstatstotalwait_time{database="pgbouncer"} 0 cnpgpgbouncerstatstotalxact_count{database="pgbouncer"} 3 cnpgpgbouncerstatstotalxact_time{database="pgbouncer"} 0 ``` As for clusters, a specific pooler can be monitored using the resource . A `PodMonitor` correctly pointing to a pooler can be created by the operator by setting `.spec.monitoring.enablePodMonitor` to `true` in the `Pooler` resource. The default is `false`. !!! Important Any change to `PodMonitor` created automatically is overridden by the operator at the next reconciliation cycle. If you need to customize it, you can do so as shown in the following example. To deploy a `PodMonitor` for a specific pooler manually, you can define it as follows and change it as needed: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: <POOLER_NAME> spec: selector: matchLabels: cnpg.io/poolerName: <POOLER_NAME> podMetricsEndpoints: port: metrics ``` Logs are directly sent to standard output, in JSON format, like in the following example: ```json { "level": "info", "ts": SECONDS.MICROSECONDS, "msg": "record", "pipe": "stderr", "record": { "timestamp": "YYYY-MM-DD HH:MM:SS.MS UTC", "pid": "<PID>", "level": "LOG", "msg": "kernel file descriptor limit: 1048576 (hard: 1048576); maxclientconn: 100, max expected fd use: 112" } } ``` The `Pooler` specification allows you to take advantage of PgBouncer's `PAUSE` and `RESUME` commands, using only declarative configuration. You can ado this using the `paused` option, which by default is set to `false`. When set to `true`, the operator internally invokes the `PAUSE` command in PgBouncer, which: Closes all active connections toward the PostgreSQL server, after waiting for the queries to complete Pauses any new connection coming from the client When the `paused` option is reset to `false`, the operator invokes the `RESUME` command in PgBouncer, reopening the taps toward the PostgreSQL service defined in the `Pooler` resource. !!! Seealso "PAUSE" For more information, see . !!! Important In future versions, the switchover operation will be fully integrated with the PgBouncer pooler and take advantage of the `PAUSE`/`RESUME` features to reduce the perceived downtime by client applications. Currently, you can achieve the same results by setting the `paused` attribute to `true`, issuing the switchover command through the , and then restoring the `paused` attribute to `false`. The current implementation of the pooler is designed to work as part of a specific CloudNativePG cluster (a service). It isn't currently possible to create a pooler that spans multiple clusters. CloudNativePG transparently manages several configuration options that are used for the PgBouncer layer to communicate with PostgreSQL. Such options aren't configurable from outside and include TLS certificates, authentication settings, the `databases` section, and the `users` section. Also, considering the specific use case for the single PostgreSQL cluster, the adopted criteria is to explicitly list the options that can be configured by users. !!! Note The adopted solution likely addresses the majority of use cases. It leaves room for the future implementation of a separate operator for PgBouncer to complete the gamma with more advanced and customized scenarios.