The end of the March Commitfest concludes the acceptance of patches for PostgreSQL 16. Let’s take a look at some exciting new updates it introduced. I hope that this review together with the previous articles in the series (2022-07, 2022-09, 2022-11, 2023-01) will give you a coherent idea of the new features of PostgreSQL 16. As usual, the March Commitfest introduces a ton of new changes. I’ve split them into several sections for convenience. pg_stat_io: input/output statistics Counter for new row versions moved to another page when performing an UPDATE pg_buffercache: new pg_buffercache_usage_counts function Normalization of DDL and service commands, continued EXPLAIN (generic_plan): generic plan of a parameterized query auto_explain: logging the query ID PL/pgSQL: GET DIAGNOSTICS .. PG_ROUTINE_OID psql: variables SHELL_ERROR and SHELL_EXIT_CODE psql: \watch and the number of repetitions psql:\df+ does not show the source code of functions pg_dump: support for LZ4 and zstd compression methods pg_dump and partitioned tables pg_verifybackup --progress libpq: balancing connections initdb: setting configuration parameters during cluster initialization Autovacuum: balancing I/O impact on the fly Managing the size of shared memory for vacuum and analyze VACUUM for TOAST tables only The vacuum_defer_cleanup_age parameter has been removed pg_walinspect: interpretation of the end_lsn parameter pg_walinspect: pg_get_wal_fpi_info → pg_get_wal_block_info ICU: UNICODE collation ICU: Canonization of locales ICU: custom rules for customizing the sorting algorithm libpq: new parameter require_auth scram_iterations: iteration counter for password encryption using SCRAM-SHA-256 SQL/JSON standard support New functions pg_input_error_info and pg_input_is_valid The Daitch-Mokotoff Soundex New functions array_shuffle and array_sample New aggregate function any_value COPY: inserting default values timestamptz: adding and subtracting time intervals XML: formatting values pg_size_bytes: support for "B" New functions: erf, erfc Parallel execution of full and right hash joins Options for the right antijoin Relation extension mechanism rework Don’t block HOT update by BRIN index postgres_fdw: aborting transactions on remote servers in parallel mode force_parallel_mode → debug_parallel_query Direct I/O (for developers only) Logical replication from a physical replica Using non-unique indexes with REPLICA IDENTITY FULL Initial synchronization in binary format Privileges for creating subscriptions and applying changes Committing changes in parallel mode (for developers only) commit: a9c70b46, ac8d53da, 8aaa04b3 The new pg_stat_io view displays input/output statistics. To be clear, this is disk input/output as seen by PostgreSQL. Lower level caching (OS, disk controller) is not taken into account here. In addition, only buffer cache operations are currently monitored. Transferring tables and indexes to another tablespace or WAL operations may be added in the future. Each row of the view is identified by three columns: backend_type — process type, same as in pg_stat_activity.backend_type, object ― object type, such as relation (permanent tables, indexes, ...) or temp relation, context ― type of operation performed on the object: normal, vacuum, bulkread, bulkwrite. The names of the operations are intuitive, and details are listed in the documentation: pg_stat_io. Let’s have a look at one of the rows: The row shows cumulative statistics for all client backend processes that performed bulk read operations on permanent relations. Such operations include sequential scanning of large tables using a ring buffer. Displayed statistics include buffer cache hits, evictions, and ring buffer reuses. You can also see that clients had to flush most of the buffer evictions to disk (writes). The reads, writes and extends (relation file extension) columns show the number of performed operations. To convert to memory units, multiply the values by op_bytes. Since only buffer cache operations are monitored so far, op_bytes is always equal to the page size (8 kB). To collect execution time statistics (*_time columns), the track_io_timing parameter must be enabled. As another example, let’s look at which processes wrote buffers from the cache to disk and compare their impacts: The two main disk writing processes, background writer and checkpointer, wrote the least. This is a clear sign that the system configuration is suboptimal. Perhaps, the buffer cache size should be increased and background writer should be set up more aggressively. The pg_stat_bgwriter view shows similar data. The buffers_clean and buffers_checkpoint columns display how many buffers the background writer and checkpointer processes have written to disk. However, the buffers_backend column isn’t really representative. It collects data from not only client processes, but also others, including autovacuum. In addition to that, relation file extensions aren’t tracked separately (as in pg_stat_io), but rather are simply added to buffers_backend, despite the operation having nothing to do with flushing cache to disk. See also: Waiting for PostgreSQL 16 – Add pg_stat_io view, providing more detailed IO statistics (Hubert 'depesz' Lubaczewski) Waiting for Postgres 16: Cumulative I/O statistics with pg_stat_io (Lukas Fittl) commit: ae4fdde1 A new column has appeared in the pg_stat_*_tables family of statistical views: n_tup_newpage_upd. Together with the existing columns n_tup_hot_upd and n_tup_upd, it helps evaluate the effectiveness of HOT optimization. HOT optimization is done when updating non-indexed table columns. A new row version is created in the page, but no new records are created in the table indexes. This is the essence of HOT optimization. But in order for it to work, you need to have enough space in the page to place a new row version there. If the space is insufficient, the row version is created in another page and new entries are also created in all indexes. It is these row changes that are tracked by the new n_tup_newpage_upd counter. If the counter value is too high, you may want to consider reducing the fillfactor value for the table. By reserving more page space, you increase the chances for HOT optimization to work. Let’s create a separate database and set up pgbench: Run pgbench for 10 seconds and look at row change statistics for the pgbench_accounts table: The sum of n_tup_hot_upd and n_tup_newpage_upd equals n_tup_upd. This means that during the test, all UPDATE commands did not update indexed columns, and therefore HOT optimization was possible. But in about 15% of cases, HOT optimization did not work because there was not enough space in the page for a new row version. Let’s reset the statistics, decrease the fillfactor value and try again. The result is radically different. There were only three instances where there was not enough space in the page. Curiously, the pg_statistic system table is a candidate for decreasing fillfactor: commit: f3fa3132 A pg_buffercache_summary function was added to the pg_buffercache extension during the November Commitfest. Now, another function is added. It shows aggregated buffer cache data subdivided by usage counts. The main advantage of pg_buffercache_summary and pg_buffercache_usage_counts over the pg_buffercache view is speed. The new functions do not require buffer locks, so they perform much faster. commit: daa8365a This is a continuation of the query normalization effort in PostgreSQL 16. This time, the affected commands are DECLARE, EXPLAIN, CREATE MATERIALIZED VIEW and CREATE TABLE AS. An especially notable feature is that normalization now takes into account any constants in the query text and substitutes them with parameters. Let’s create three temporary tables with a constant in the AS SELECT statement: Now, verify that pg_stat_statements counts the three commands as a single one. See also: Postgres 16 highlight - Normalization of utilities in pg_stat_statements (Michael Paquier) commit: 3c05284d The idea behind this tool is as follows. Queries with long execution times are recorded into the server log for analysis. An application uses the extended query protocol to send queries with parameters to the server. To imitate that, use the new command \bind in psql: We don’t care about the result here, just that it took over 100 ms to execute and therefore was recorded in the server log. Now, we can start figuring out why the query took so long to execute. To do that, we need the query plan. However, simply adding the EXPLAIN word to the query text returns an error: The problem is that EXPLAIN cannot build plans for queries with parameters, since different parameter values can result in completely different plans. Of course, you can manually insert the parameter value from the log and get a plan, but there could be many such queries in the log and with multiple parameters each. Automating the substitution process can be time-consuming, too. Here is where the generic plan comes in. A query’s generic plan does not depend on the query’s parameter values. And now you can get one for your query: The planner wants to scan the table sequentially by several processes. This means that the table is large. However, only one record will be selected. Probably, the book_ref column is missing an index that would otherwise expedite the query. See also: EXPLAIN (GENERIC_PLAN): New in PostgreSQL 16 (Laurenz Albe) commit: 9d2d9728 According to the documentation, logging queries with the auto_explain.log_verbose parameter enabled is equivalent to EXPLAIN output with the VERBOSE option. However, the query ID was not output, which was fixed: The feature is not ported into previous PostgreSQL versions, because it might affect server log analysis tools. commit: d3d53f95 You can now get a function’s ID from its code. commit: b0d8f2d9, 31ae2aa9 Two new variables show how the last OS command has completed: The first commit adds these variables for the ! command and commands in back quotes (``). The second one adds them to \g, \o, \w and \copy. In some cases, the variables are intentionally left unset. These include: calling the pager program, calling the editor (\e) and setting the prompt with a command in back quotes (\set PROMPT1%command). commit: 00beecfe The \watch command repeats the query until terminated. The new patch allows you to specify the number of repetitions. commit: 3dfae91f The \df+ command, among other things, outputs the source code of functions, which can take up a lot of space. This makes it difficult to view the list of functions. In PostgreSQL 16, \df+ will no longer show the source code of all functions except internal ones. Internal function code is a C language function name and always fits in one line. You can still view function source codes with other commands: \sf and \ef. commit: 0da243fe, 84adc8e2 Before PostgreSQL 16, the only compression method supported by pg_dump was gzip. The two commits add the support for LZ4 and zstd, respectively. Backup compression is available for all formats except tar. See also: LZ4 and ZSTD pg_dump compression in PostgreSQL 16 (Pavlo Golub) commit: a563c24c Dumping individual tables is done with the --table parameter. But if you want to dump a partitioned table with all its partitions, then you either need to explicitly list all the partitions, or use the % sign to specify a table name template that all partitions will fit. Both options are imperfect. The number of partitions may change, and partitions themselves may fit into a template poorly or the template might accidentally include other tables. In PostgreSQL 16, the new pg_dump parameter --table-and-children allows you to leave the old approach behind. The parameter works not only for partitioned tables, but also for table inheritance. For table exclusion, similar parameters are added: --exclude-table-and-children and --exclude-table-data-and-children. commit: d07c2948 The pg_basebackup utility with the --progress parameter shows the backup creation progress. The same parameter has been added to the backup verification utility pg_verifybackup. During verification, the status is updated every second, and at completion the output looks like this: commit: 7f5b1981 The ability to specify multiple PostgreSQL instances during connection is not new. You can list multiple replicas under the host, hostaddr and port parameters in the connection string. Clients will try to connect to the replicas in the specified order. However, with a large number of connections, the first replica in the list will experience more load than the others, and the last one may be idle at all. To distribute connections between replicas evenly, you can shuffle the list of replicas on the application side when forming the connection string. Or you can use the new connection parameter load_balance_hosts: load_balance_hosts=random indicates that the list of nodes will be shuffled before attempting to connect. commit: 3e51b278 The new initdb key -c (or --set) will override the value of any configuration parameter. For the purpose of the experiment, let’s create a second instance of the server on a free port and with the wal_level parameter set to logical: The parameters specified during initialization are stored at the end of the postgresql.conf file: commit: 7d71d3dd When the autovacuum takes too long to vacuum a particularly large table, the process can be accelerated on the fly. This requires modifying the parameters autovacuum_vacuum_cost_limit and/or autovacuum_vacuum_cost_delay and updating the configuration. In previous PostgreSQL versions, autovacuum processes could read configuration files only after they finish processing the current table. With the new patch, this is done before every pause check (autovacuum_vacuum_cost_delay). See also: Updating the Cost Limit On The Fly (Robert Haas) commit: 1cbbee03, 4830f102 In order to protect the entire shared memory pool from bulk eviction, table vacuuming uses only a fraction of it to form a ring buffer. The size of the ring buffer is relatively tiny at only 256 kB. However, when a large table needs vacuuming, performance of the whole server may suffer. For the pages in the small ring buffer to be reused, dirty pages and WAL records have to be constantly flushed to disk. Frequently synchronizing WAL to disk may become a bottleneck for the I/O system. The first commit introduces a new parameter vacuum_buffer_usage_limit, which can be used to control the size of the ring buffer. The default value remains the same at 256 kB, but the parameter can accept values from 12 kB to 16 GB (but no more than ⅛ of the buffer cache can be used). When running VACUUM and ANALYZE manually, you can specify the size with the BUFFER_USAGE_LIMIT option. If you set it to 0, vacuuming will be allowed to use the whole buffer cache: This may be useful after uploading a large amount of new data or after bulk changes to existing tables. When vacuuming aggressively to avoid transaction counter wraparound (the second commit), the process must execute as quickly as possible, so the shared memory limitation is disabled by default. The ring buffer operation is now explained in the glossary under the term "Buffer Access Strategy". commit: 4211fbd8 Vacuuming just a TOAST table requires the USAGE privilege for the pg_toast schema. You also have to know the name of the relevant TOAST table. The new option PROCESS_MAIN lets you do all that without looking up the table name. In the same vein, the vacuumdb tool has received a new parameter --no-process-main. commit: be504a3e, 1118cd37 Setting the vacuum_defer_cleanup_age value higher than the current transaction ID may result in corruption of data. See the first commit for details. Developing comprehensive tests and porting them to previous releases is a difficult task that may just be unnecessary. After all, the vacuum_defer_cleanup_age parameter is probably rarely used. hot_standby_feedback and replication slots that appeared later are much more convenient alternatives. Therefore, the second commit removes the vacuum_defer_cleanup_age parameter altogether. It wasn’t a very convenient tool in the first place: too high values would result in bloating, and too low values wouldn’t do much to avoid conflicts while applying WAL records on replicas. Even if you don’t plan on upgrading to PostgreSQL 16 any time soon, it would still be wise to take stock of all your vacuum_defer_cleanup_age in the configuration files and prepare to get rid of them. commit: 5c1b6628 The interpretation of the end_lsn parameter has been changed for pg_get_wal_records_info, pg_get_wal_stats and pg_get_wal_block_info. If no value is specified, the functions return data from start_lsn till the end of WAL. This made it possible to remove the duplicate functions pg_get_wal_records_info_till_end_of_wal and pg_get_wal_stats_till_end_of_wal. commit: 9ecb134a The previous article mentioned the new pg_get_wal_fpi_info function of the pg_walinspect extension. The function has been significantly redesigned and renamed to pg_get_wal_block_info. commit: 0d21d4b9 In accordance with the SQL standard, the UNICODE collation for the ICU provider has been implemented in PostgreSQL 16. The collation uses the default sorting algorithm as described in the UNICODE technical standard. commit: ea1db8ae, 1671f990 Locale names for the ICU provider will be converted to the canonical form in accordance with BCP 47 rules. Coincidentally, this will prevent creation of invalid collations. To disable locale validation, you can use the new icu_validation_level parameter (second commit). commit: 30a53b79 The new rules option of the CREATE COLLATION command allows you to customize the sorting algorithm for specific needs. Let’s create a collation where the seasons of the year are sorted starting from winter: Custom rules can also be specified in the CREATE DATABASE command and when running createdb and initdb. See also: ICU Documentation: Collation Customization How collation works (Peter Eisentraut) How collation of punctuation and whitespace works (Peter Eisentraut) commit: 3a465cc6 The pg_hba.conf file is configured as follows: The new libpq parameter require_auth allows you to specify the authentication method that the server should use. You can list several methods to allow the server to use any of them: In this example, the second rule with the scram-sha-256 method should be used for TCP/IP connection. Since this method, along with md5, is listed in the client’s require_auth parameter, it is applied. (You can read more about the new system_user function in PostgreSQL 16 in the September Commitfest article.) If the authentication method in require_auth does not match the method from pg_hba.conf, then you will not be able to connect: You can forbid specific methods from being used by listing them with a preceding exclamation mark. The special value none is used when a connection without authentication is required. In our case, this will work for the local connection under the first rule in pg_hba.conf: Instead of the require_auth parameter, you can use the PGREQUIREAUTH environment variable. See also: Postgres 16 highlight - require_auth for libpq (Michael Paquier) commit: b5777430 In previous versions, the counter value was hard-coded as 4096. Now, more resilient passwords can be generated by specifying a higher counter value under the new parameter scram_iterations. RFC 7677 recommends to set it to 15,000 or more. Keep in mind that higher counter values increase security, but also the time it takes to authenticate the user. See also: Postgres 16 highlight - Control of SCRAM iterations (Michael Paquier) commit: 7081ac46, 6ee30209 This big update was accepted a year ago, but was rolled back shortly before the PostgreSQL 15 release. The patch for PostgreSQL 16 introduces support for constructor functions (JSON_ARRAY, JSON_ARRAYAGG, JSON_OBJECT, JSON_OBJECTAGG) and predicates (IS JSON [VALUE], IS JSON ARRAY, IS JSON OBJECT, IS JSON SCALAR). The work will continue with the next major release. commit: b8da37b3, 1939d262 The pg_input_is_valid function checks whether the first argument is a valid input value for the data type from the second argument: The pg_input_error_info function returns detailed information about the error: The functions do a “soft” check of input values. The lack of such a function in the last release cycle is what prevented the SQL/JSON patch from releasing with PostgreSQL 15. See also: Waiting for PostgreSQL 16 – Add test scaffolding for soft error reporting from input functions (Hubert 'depesz' Lubaczewski) commit: a290378a Daitch-Mokotoff Soundex support is added to the fuzzystrmatch extension. See also: Soundexing and Genealogy by Gary Mokotoff commit: 888f2ea0 The array_shuffle function shuffles the elements of the input array, and array_sample returns an array with the specified number of randomly selected elements of the input array: The functions can be helpful when using the Monte Carlo method. See also: Waiting for PostgreSQL 16 – Add array_sample() and array_shuffle() functions (Hubert 'depesz' Lubaczewski) commit: 2ddab010 An SQL standard aggregate function any_value returns an arbitrary non-empty value for a group of rows. Consider a table without a unique key in which some rows are duplicated. You need to delete all duplicates in one query. (Similar tasks are common in job interviews.) To select one of two duplicates, the functions min and max are commonly applied to the system column ctid. This works, but is a bit misleading, because we don’t really need a minimum or a maximum value. Any of the two will do, and the new function any_value does just that: commit: 9f8377f7 Consider a table with two mandatory date columns. If a value is missing, a date from the distant past or future is used as a placeholder. The table is populated from external files by means of the COPY command. The providers of the external files use 'N/A' for missing date values. You cannot affect how the files are formed. Importing such a file into the table is now made easier by the DEFAULT option of the COPY command. When a value specified under DEFAULT is matched in the data, the default value for the column is used instead: commit: 75bd846b The + and - operators can be used to add and subtract time intervals from values of the timestamp with time zone type (timestamptz). The result of such operations depends on the time zone set in the timezone parameter. To make the result independent of the current time zone, you can use the new functions date_add and date_subtract. The last parameter of these functions allows you to explicitly specify the time zone for calculations: The time zone parameter is also added to the generate_series function. Independence from session parameters allowed to declare all three functions as immutable in the time zone variant. commit: 483bdb2a The XMLSERIALIZE function now has a parameter INDENT for formatting output: commit: ce1215d9 Some parameter values can be set in bytes. You can use pg_size_bytes to get the number of bytes: In previous versions, the pg_size_bytes function will throw an error because it understands bytes, but not B. commit: d5d57414 Together with random_normal, the PostgreSQL 16 release introduces the error function erf and the complementary error function erfc. The 68-95-99.7 rule can now be viewed like this: The erfc function returns a value close to 1 - erf(). commit: 11c2d6fd In previous versions, the query below was always executed sequentially. In PostgreSQL 16, parallel execution is now possible: commit: 16dc2703 Let’s look at the plan of a query that lists all the flights for which no tickets have been issued yet. PostgreSQL 15, the query is executed using the antijoin (the Hash Anti Join node). The hash table is built for the ticket_flights table, the larger of the two tables from the query. Creation of a hash table requires some time and memory, which is reflected in the high startup cost of the query relative to the total cost. In PostgreSQL 16, the planner also considers the antijoin where the table positions are reversed. If the reverse plan turns out to be cheaper, it will be selected instead. The modified plan node is now called Hash Right Anti Join. The Merge Join has received a similar modification. commit: 31966b15, 00d1e02b When records are added into a table frequently, the table files have to be extended by appending new pages. To prevent multiple clients from adding new pages at the same time, a relation extension lock is used. Such locks may significantly affect performance when multiple clients are actively inserting records. The idea behind this update is to have only the bare minimum of operations done while the lock is held, and perform the rest without the lock. Behind this modest summary is a tremendous amount of work which resulted in 13 separate patches, of which I’ve listed just two. As a result, not only applications with frequent record insertions by multiple clients will benefit, but also single thread insertion which will be performed at a lower cost. commit: 19d8e230 HOT updates are not applied if at least one column modified by the UPDATE command is indexed by any type of index. However, BRIN indexes do not contain references to table rows, therefore it is safe to use HOT updates on columns with such indexes. With the new patch, BRIN indexes no longer affect HOT optimization: One of the counters ticks up. The patch has been accepted into the PostgreSQL 15 already before, but was rolled back before the release. This is the second attempt. postgres_fdw: aborting transactions on remote servers in parallel mode commit: 983ec230 PostgreSQL 15 introduced the ability to asynchronously commit transactions affecting foreign tables connected via postgres_fdw. Now, with the new parallel_abort parameter for the external server, transactions can be aborted asynchronously as well. commit: 5352ca22 The parameter force_parallel_mode is now called debug_parallel_query. This is done because many users mistakingly enable this parameter, thinking that it will increase query execution speed. In fact, the result is quite the opposite! When switched on, the parameter makes the server always prefer the parallel plan, even if it is the slower one. The parameter should be used for query debugging, whenever the developer wants to figure out why the server prefers a sequential plan over its parallel equivalent. The new name reflects what the parameter does more precisely and should not provoke users to switch it on based on the name alone. See also: New old “debug_parallel_query” setting in PostgreSQL 16 (Pavlo Golub) commit: faeedbce PostgreSQL works with files through the operating system, not directly. This is an effort to introduce the ability to input/output data and WAL directly. The work continues, so no useful new features so far. There is a new configuration parameter io_direct, but you should avoid switching it on. The documentation honestly states that doing so will decrease performance, because it would disable the operating system’s prefetching, and PostgreSQL’s own prefetching has not been developed yet. The parameter is here for the developers and is intended for testing purposes only. commit: 0fdab27a This patch makes it possible to subscribe to a physical replica of a publication server rather than to the server itself. The wal_level parameter must be set to logical both on the primary server and the physical replica for this to work: Create a publication on the primary server: Make sure that both the table and the publication are present on the replica: Now, on a separate server (subscriber), subscribe to the publication. In the connection string, specify the replica that is located on port 5401: This command should create a replication slot on the physical replica. If the primary server is inactive at this moment, it might take some time. You can speed things up by executing the following command on the primary server: The replica will “wake up” and the slot will be created just on it: As for logical replication, it is already working: See also: Postgres 16 highlight: Logical decoding on standby (Bertrand Drouvot) commit: 89e46da5 If a table in a publication does not have a primary or a unique key, then REPLICA IDENTITY FULL is used as the replication ID. On the subscriber side, the search for modified and deleted rows is done via sequential scan. With the new patch, logical replication will be able to use the non-unique index to identify rows on the subscriber that have been modified or deleted. Index selection cannot be controlled. If the table has several indexes, one of them will be selected arbitrarily. Only indexes of B-tree type which are not partial and include at least one column of the table (not entirely functional) can be used. commit: ecb69652 To speed up replication, you can enable the parameter binary for the subscription so that the publishing server sends changes in binary format: Before PostgreSQL 16, this affected just the format the changes were replicated in, while during initial synchronization, only the textual format was used to transmit the data. Now the COPY command that performs the initial synchronization will also work in binary format. commit: c3afe8cf, 1e10d49b, 48267598 The first commit creates a predefined role pg_create_subscription. Members of this role can create subscriptions (and replication slots on the publishing server) without superuser privileges. For security reasons, the subscription owner must switch to the table owner role for any INSERT, UPDATE, DELETE and TRUNCATE operations (second commit). Therefore, the subscription owner must have the right to execute the necessary SET ROLE commands. The old behavior, when the subscription owner has the rights to apply changes, can be enabled by the new subscription parameter run_as_owner (third commit): commit: c3afe8cf, 1e10d49b, 9f2213a7 Multiple processes will be able to apply changes on the subscriber at the same time. This will speed up the processing of large transactions received from the publication server. The feature is not implemented in full yet, so no details so far. However, there is a hint in the patch: a new parameter logical_replication_mode in the development section. That’s all for now. Nothing new will get into PostgreSQL 16 at this point. Looking forward to the release in the Autumn!  Технический писатель