In previous articles we discussed how the system plans a query execution and how it collects statistics to select the best plan. The following articles, starting with this one, will focus on what a plan actually is, what it consists of, and how it is executed. In this article, I will demonstrate how the planner calculates execution costs. I will also discuss access methods and how they affect these costs, and use the sequential scan method as an illustration. Lastly, I will talk about parallel execution in PostgreSQL, how it works, and when to use it. I will use several seemingly complicated math formulas later in the article. You don't have to memorize any of them to get to the bottom of how the planner works; they are merely there to show where I get my numbers from. The PostgreSQL approach to storing data on disk will not be optimal for every possible type of load. Thankfully, you have options. Delivering on its promise of extensibility, PostgreSQL 12 and higher supports custom table access methods (storage engines), although it ships only with the stock one, heap: You can specify an engine when creating a table (CREATE TABLE ... USING). If you don't, the engine defined in default_table_access_method is used. The pg_am catalog stores engine names (amname column) as well as their handler functions (amhandler). Every storage engine comes with an interface that the core needs in order to make use of the engine. The handler function's job is to return all the necessary information about the interface structure. The majority of storage engines make use of the existing core system components: An engine might not necessarily use all of these components, but the ability is still there. Then, there are things that an engine must define: Traditionally, PostgreSQL used a single data storage system that was built into the core directly, without an interface. This makes creating a new interface — accommodating all the long-standing particularities of the standard engine and not interfering with other access methods — a challenge and a half. There is an issue with write-ahead logging, for example. Some access methods want to log engine-specific operations, which the core knows nothing about. The existing Generic WAL Records mechanism is rarely an option because of significant overhead costs. You could devise a separate interface for new log record types, but this makes crash recovery dependent on external code, which is something you always want to avoid. So far, rebuilding the core to fit a new engine remains the only viable option. Speaking of new engines, there are several currently in development. Here are some of the more prominent ones: The storage engine determines how table data is physically distributed on disk and provides a method to access it. Sequential scan is a method that scans the file (files) of the main table fork entirely. On each page, the system checks the visibility of each row version, and also discards the versions that don't match the query.  The scanning is done through the buffer cache. The system uses a small buffer ring to prevent larger tables from pushing potentially useful data from the cache. When another process needs to scan the same table, it joins the buffer ring, saving disk read times. Therefore, scanning does not necessarily start from the beginning of the file. Sequential scan is the most cost-effective way of scanning a whole table or a significant portion of it. In other words, sequential scan is efficient when selectivity is low. At higher selectivity, when only a small fraction of rows in a table meets the filter requirements, it's usually better to use index scan. A sequential scan stage in an execution plan is represented with a Seq Scan node. Row count estimate is a basic statistic: When estimating costs, the optimizer takes into account disk input/output and CPU processing costs. The I/O cost is estimated as a single page fetch cost multiplied by the number of pages in a table, provided that the pages are fetched sequentially. When the buffer manager requests a page from the operating system, the system actually reads a larger chunk of data from disk, so the next several pages will probably be in the operating system's cache already. This makes the sequential fetch cost (which the planner weights at seq_page_cost, 1 by default) considerably lower than the random access fetch cost (which is weighted at random_page_cost, 4 by default). The default weights are appropriate for HDD drives. If you are using SSDs, the random_page_cost can be set significantly lower (seq_page_cost is usually left at 1 as the reference value). The costs depend on hardware, so they are usually set at the tablespace level (ALTER TABLESPACE ... SET). This formula perfectly illustrates the result of table bloating due to late vacuuming. The larger the main table fork, the more pages there are to fetch, regardless of whether the data in these pages is up-to-date or not. The CPU processing cost is estimated as the sum of processing costs for each row version (weighted by the planner at cpu_tuple_cost, 0.01 by default): The sum of the two costs is the total plan cost. The plan's startup cost is zero because sequential scan does not require any preparation steps. Any table filters will be listed in the plan below the Seq Scan node. The row count estimate will take into account the selectivity of filters, and the cost estimate will include their processing costs. The EXPLAIN ANALYZE command displays both the actual number of scanned rows and the number of rows removed by the filters: Consider this execution plan that involves aggregation: There are two nodes in this plan: Aggregate and Seq Scan. Seq Scan scans the table and passes the data up to Aggregate, while Aggregate counts the rows on an ongoing basis. Notice that the Aggregate node has a startup cost: the cost of aggregation itself, which needs all the rows from the child node to compute. The estimate is calculated based on the number of input rows multiplied by the cost of an arbitrary operation (cpu_operator_cost, 0.0025 by default): The estimate is then added to the Seq Scan node's total cost. The Aggregate's total cost is then increased by cpu_tuple_cost, which is the processing cost of the resulting output row: In PostgreSQL 9.6 and higher, parallel execution of plans is a thing. Here's how it works: the leader process creates (via postmaster) several worker processes. These processes then simultaneously execute a section of the plan in parallel. The results are then gathered at the Gather node by the leader process. While not occupied with gathering data, the leader process may participate in the parallel calculations as well. You can disable this behavior by setting the parallel_leader_participation parameter to 0, but only in version 11 or higher.  Spawning new processes and sending data between them add to the total cost, so you may often be better off not using parallel execution at all. Besides, there are operations that simply can't be executed in parallel. Even with the parallel mode enabled, the leader process will still execute some of the steps alone, sequentially. The method's name might sound controversial, claiming to be parallel and sequential at the same time, but that's exactly what's going on at the Parallel Seq Scan node. From the disk's point of view, all file pages are fetched sequentially, same as they would be with a regular sequential scan. The fetching, however, is done by several processes working in parallel. The processes synchronize their fetching schedule in a special shared memory section in order to avoid fetching the same page twice. The operating system, on the other hand, does not recognize this fetching as sequential. From its perspective, it's just several processes requesting seemingly random pages. This breaks prefetching that serves us so well with regular sequential scan. The issue was fixed in PostgreSQL 14, when the system started assigning each parallel process several sequential pages to fetch at once instead of just one. Parallel scanning by itself doesn't help much with cost efficiency. In fact, all it does is add the cost of data transfers between processes on top of the regular page fetch cost. However, if the worker processes not only scan the rows, but also process them to some extent (for example, aggregate), then you may save a lot of time. The optimizer sees this simple query with aggregation on a large table and decides that the optimal strategy is the parallel mode: All the nodes below Gather are the parallel section of the plan. It will be executed by all worker processes (2 of them are planned in this case) and the leader process (unless disabled by the parallel_leader_participation option). The Gather node and all the nodes above it are executed sequentially by the leader process. Consider the Parallel Seq Scan node, where the scanning itself happens. The rows field shows an estimate of rows to be processed by one process. There are two worker processes planned, and the leader process will assist too, so the rows estimate equals the total table row count divided by 2.4 (2 for the worker processes and 0.4 for the leader; the more there are workers, the less the leader contributes). Parallel Seq Scan cost is calculated in mostly the same way as Seq Scan cost. We win time by having each process handle fewer rows, but we still read the table through-and-through, so the I/O cost isn't affected: The Partial Aggregate node aggregates all the data produced by the worker process (counts the rows, in this case). The aggregate cost is calculated in the same way as before and added to the scan cost. The next node, Gather, is executed by the leader process. This node starts worker processes and gathers their output data. The cost of starting up a worker process (or several; the cost doesn't change) is defined by the parameter parallel_setup_cost (1000 by default). The cost of sending a single row from one process to another is set by parallel_tuple_cost (0.1 by default). The bulk of the node cost is the initialization of the parallel processes. It is added to the Partial Aggregate node startup cost. There is also the transmission of two rows; this cost is added to the Partial Aggregate node's total cost. The Finalize Aggregate node joins together the partial data collected by the Gather node. Its cost is assessed just like with a regular Aggregate. The startup cost comprises the aggregation cost of three rows and the Gather node total cost (because Finalize Aggregate needs all its output to compute). The cherry on top of the total cost is the output cost of one resulting row. There are several limitations to parallel processing that should be kept in mind. The use of background worker processes is not limited to parallel query execution: they are used by the logical replication mechanism and may be created by extensions. The system can simultaneously run up to max_worker_processes background workers (8 by default). Out of those, up to max_parallel_workers (also 8 by default) can be assigned to parallel query execution. The number of allowed worker processes per leader process is set by max_parallel_workers_per_gather (2 by default). You may choose to change these values based on several factors: These factors are usually true for OLAP systems and false for OLTPs. The planner will not even consider parallel scanning unless it expects to read at least min_parallel_table_scan_size of data (8MB by default). Below is the formula for calculating the number of planned worker processes:  In essence, every time the table size triples, the planner adds one more parallel worker. Here's an example table with the default parameters. The number of workers can be explicitly set with the table storage parameter parallel_workers. The number of workers will still be limited by the max_parallel_workers_per_gather parameter, though. Let's query a small 19MB table. Only one worker process will be planned and created (see Workers Planned and Workers Launched): Now let's query a 105MB table. The system will create only two workers, obeying the max_parallel_workers_per_gather limit. If the limit is increased, three workers are created: If there are more planned workers than available worker slots in the system when the query is executed, only the available number of workers will be created. Not every query can be parallelized. These are the types of non-parallelizable queries: Queries that modify or lock data (UPDATE, DELETE, SELECT FOR UPDATE etc.). In PostgreSQL 11, such queries can still execute in parallel when called within commands CREATE TABLE AS, SELECT INTO and CREATE MATERIALIZED VIEW (and in version 14 and higher also within REFRESH MATERIALIZED VIEW). All INSERT operations will still execute sequentially even in these cases, however. Any queries that can be suspended during execution. Queries inside a cursor, including those in PL/pgSQL FOR loops. Queries that call PARALLEL UNSAFE functions. By default, this includes all user-defined functions and some of the standard ones. You can get the complete list of unsafe functions from the system catalog: Queries from within functions called from within an already parallelized query (to avoid creating new background workers recursively). Future PostgreSQL versions may remove some of these limitations. Version 12, for example, added the ability to parallelize queries at the Serializable isolation level. There are several possible reasons why a query will not run in parallel: If you want to force a query to be executed in parallel — for research or other purposes — you can set the parameter force_parallel_mode on. This will make the planner always produce parallel plans, unless the query is strictly non-parallelizable: In general, the benefit of parallel planning depends mostly on how much of the plan is parallel-compatible. There are, however, operations that technically do not prevent parallelization, but can only be executed sequentially and only by the leader process. In other words, these operations cannot appear in the parallel section of the plan, below Gather. Non-expandable subqueries. A basic example of an operation containing non-expandable subqueries is a common table expression scan (the CTE Scan node below): If the common table expression is not materialized (which became possible only in PostgreSQL 12 and higher), then there is no CTE Scan node and no problem. The expression itself can be processed in parallel, if it's the quicker option. Another example of a non-expandable subquery is a query with a SubPlan node. The first two rows display the plan of the main query: scan the flights table and filter each row. The filter condition includes a subquery, the plan of which follows the main plan. The SubPlan node is executed multiple times: once per scanned row, in this case. The Seq Scan parent node cannot be parallelized because it needs the SubPlan output to proceed. The last example is executing a non-expandable subquery represented by an InitPlan node. Unlike a SubPlan, an InitPlan node executes only once (in this case, once per SubPlan 2 execution). The InitPlan node's parent can't be parallelized, but nodes that use InitPlan output can, as illustrated here. Temporary tables. Temporary tables can only be scanned sequentially because only the leader process has access to them. Parallel restricted functions. Calls of functions labeled as PARALLEL RESTRICTED are only allowed within the sequential part of the plan. You can find the list of the restricted functions in the system catalog: Only label your own functions PARALLEL RESTRICTED (not to mention PARALLEL SAFE) after thoroughly studying the existing limitations and with great care. To be continued. Технический писатель