Project: YugabyteDB (App Definition and Development / Database)
File: index-backfill.md

title: Online index backfilling linkTitle: Index backfill description: Understand how to create indexes without affecting ongoing queries headerTitle: Create indexes and track the progress headcontent: Understand how YugabyteDB creates indexes without affecting ongoing queries menu: stable: identifier: index-backfill parent: explore-indexes-constraints-ysql weight: 300 type: docs You can add a new index to an existing table using the YSQL statement. YugabyteDB supports , which is enabled by default. Using this feature, you can build indexes on tables that already have data, without affecting other concurrent writes. YugabyteDB also supports the statement to disable online index backfill. The current state of an index backfill can be viewed by executing the `pgstatprogresscreateindex` view to report the progress of the CREATE INDEX command execution. The view contains one row for each backend connection that is currently running a CREATE INDEX command, and the row entry is cleared after the completion of the command execution. The `pgstatprogresscreateindex` view can provide the following details: Number of rows processed during an index backfill. The current phase of the command is either `initializing` or `backfilling`. Index progress report for all the different configurations of an index or index build such as non-concurrent index builds, GIN indexes, partial indexes, and include indexes. {{<note>}} Columns such as lockerstotal, lockersdone, currentlockerpid, blockstotal, and blocksdone in the `pgstatprogresscreateindex` view do not apply to YugabyteDB and always have null values. {{</note>}} The following example demonstrates the possible phases (initializing, backfilling) for the CREATE INDEX operation using the `pgstatprogresscreateindex` view. {{% explore-setup-single %}} From your local YugabyteDB installation directory, connect to the shell, and create an index on an existing table as follows: ```sql CREATE TABLE test(id int); ``` Populate the table with some data. ```sql INSERT INTO test(id) SELECT n FROM generate_series(1,1000000) AS n; ``` On a separate parallel YSQL connection on the same node, select from the view to see the progress of the command in a repeated fashion with `\watch 1` as follows: ```sql SELECT tbl.relname as tblname, idx.relname as indexname, command, phase, tuplestotal, tuplesdone FROM pgstatprogresscreateindex INNER JOIN pg_class as tbl on tbl.oid = relid INNER JOIN pgclass as idx on idx.oid = indexrelid where tbl.relname = 'test'; \watch 1 ``` Initially, you will see an empty output like

```output tblname | indexname | command | phase | tuplestotal | tuplesdone +--++-+--+- ``` Now create an index on the `id` column. ```sql CREATE INDEX idx_id ON test(id); ``` On the other terminal, you will see the progress on the index creation first with the first phase, `initializing` as: ```tablegen tblname | indexname | command | phase | tuplestotal | tuplesdone +--++--+--+- test | idx_id | CREATE INDEX CONCURRENTLY | initializing | 0 | 0 ``` And then you will see the index backfilling happen as: ```tablegen tblname | indexname | command | phase | tuplestotal | tuplesdone +--++-+--+- test | idx_id | CREATE INDEX CONCURRENTLY | backfilling | 0 | 0 ``` You will see the tuplesdone count increasing as the backfilling progresses. When the backfilling is done, you will see the `tuplesdone` count to be updated correctly. ```tablegen tblname | indexname | command | phase | tuplestotal | tuplesdone +--++-+--+- test | idx_id | CREATE INDEX CONCURRENTLY | backfilling | 0 | 1000000 ``` Backfilling consumes some amount of memory. The memory consumption is directly proportional to the data size per-row, the number of write operations batched together, and the number of parallel backfills. You can view the approximate memory usage by executing the following SQL statement via `ysqlsh`. ```sql SELECT indexname, tablet, now()-backendstart started, pgsizepretty(allocatedmembytes) mem, pgsizepretty(rssmem_bytes) rss FROM ( SELECT allocatedmembytes, rssmembytes, backendstart, querystart, query ,regexp_replace(query,'(BACKFILL INDEX) ([0-9]) . PARTITION (.*);','\2')::oid AS indexoid ,regexp_replace(query,'(BACKFILL INDEX) ([0-9]) . PARTITION (.*);','\3')::text AS tablet FROM pgstatactivity WHERE query LIKE 'BACKFILL INDEX %' ) pgstatactivity LEFT OUTER JOIN ( SELECT oid AS indexoid, relname AS indexname FROM pg_class ) pg_class USING(indexoid); ``` This should give you an output similar to the following when an index is being backfilled. ```tablegen indexname | tablet | started | mem | rss --++--+-+- idx_id | x'aaaa' | 00:00:04.895382 | 27 MB | 25 MB ``` In YugabyteDB, the `pgstatprogresscreateindex` view is local; it only has entries for CREATE INDEX commands issued by local YSQL clients. In PostgreSQL, `tuplesdone` and `tuplestotal` refer to the tuples of the index. However, in YugabyteDB, these fields refer to the tuples of the indexed table. This discrepancy is only observed for partial indexes, where the reported progress is less than the actual progress. `tuplestotal` is an estimate that is retrieved from `pgclass.reltuples`. In YugabyteDB, `tuplesdone` and `tuplestotal` are not displayed (set to null) for temporary indexes.