Project: YugabyteDB (App Definition and Development / Database)
File: pg-stat-progress-copy.md

title: View COPY status with pgstatprogress_copy linkTitle: View COPY status description: Use pgstatprogress_copy to get the COPY command status, number of tuples processed, and other COPY progress reports. headerTitle: View COPY status with pgstatprogress_copy menu: v2.18: identifier: pg-stat-progress-copy parent: query-tuning weight: 400 type: docs YugabyteDB supports the PostgreSQL `pgstatprogresscopy` view to report the progress of the COPY command execution. Whenever is running, the `pgstatprogresscopy` view contains one row for each client connection that is currently running a COPY command. The following table describes the view columns: | Column | Description | | :- | :- | | pid | Process ID of backend. | | datid | Object ID of the database to which this backend is connected. | | datname | Name of the database to which this backend is connected. | | relid | Object ID of the table on which the COPY command is executed. It is set to 0 if copying from a SELECT query.| | command | The command that is running COPY FROM, or COPY TO. | | type | The I/O type that the data is read from or written to: FILE, PROGRAM, PIPE (for COPY FROM STDIN and COPY TO STDOUT), or CALLBACK (used for example during the initial table synchronization in logical replication). | | yb_status | Tracking status of the copy command. | | bytes_processed | Number of bytes already processed by the COPY command. | | bytes_total | Size of the source file for the COPY FROM command in bytes. Set to 0 if not available. | | tuples_processed | Number of tuples already processed by the COPY command. | | tuples_excluded | Number of tuples not processed because they were excluded by the WHERE clause of the COPY command. | The `pgstatprogress_copy` view includes the following YugabyteDB-specific changes: The definition of `tuples_processed` column is different in YugabyteDB in comparison to PostgreSQL. In YugabyteDB, the `ROWSPERTRANSACTION` option is added to the COPY command, defining the transaction size to be used. For example, if the total tuples to be copied is 5000 and `ROWSPERTRANSACTION` is set to 1000, the database creates 5 transactions and each transaction inserts 1000 rows. If there is an error during execution, then some tuples can be persisted based on the already completed transaction. Because each COPY is divided into multiple transactions, `tuples_processed` tracks the rows that the transaction has already completed. For more information, refer to . In YugabyteDB, the `pgstatprogresscopy` view includes the column `ybstatus` to indicate the status of the COPY command. If a COPY command terminates due to any error, then it's possible that some tuples are persisted as explained in the section. In this case, `tuplesprocessed` shows a non-zero count and `ybstatus` shows that the COPY command was terminated due to an error. This is helpful for discovering whether the copy completed or not. This is unnecessary in PostgreSQL, where copy is a single transaction. `yb_status` can have the following values: IN PROGRESS Indicating that the COPY command is still running. ERROR COPY command was terminated due to an error. SUCCESS COPY command successfully completed. The `pgstatprogress_copy` view retains copy command information after the copy operation has

In PostgreSQL, when the COPY command finishes, the row containing details of the copy command is removed from the view. In YugabyteDB, the information is retained in the view after the copy has finished. The information is retained only for the last executed copy command on that connection. This is required for YugabyteDB, because if the COPY command finishes due to an error, then you would like to know how many rows were reliably persisted to the disk. The following examples demonstrate the possible stages (IN PROGRESS, ERROR, SUCCESS) for the copy operation. {{% explore-setup-single-local %}} Run the following script from your YugabyteDB installation directory to generate data to a file `test_data.csv`. ```sh rm -rf test_data.csv for i in {1..100000} do echo "$i, $i, $i" >> test_data.csv done ``` From your local YugabyteDB installation directory, connect to the shell, and create a table using the following schema: ```sql create table test_copy ( h1 int, r1 int, v1 int, primary key (h1, r1)); ``` Copy `test_data.csv` using the following copy command: ```sql \copy testcopy from 'testdata.csv' (DELIMITER ','); ``` ```output COPY 100000 ``` Verify `yb_status` using the view. ```sql select * from pgstatprogress_copy ; ``` You should see output similar to the following: ```output pid | datid | datname | relid | command | type | ybstatus | bytesprocessed | bytestotal | tuplesprocessed | tuples_excluded --+-+-+-+--++--+--+-++-- 74390 | 13288 | yugabyte | 16394 | COPY FROM | PIPE | SUCCESS | 1966685 | 0 | 100000 | 0 (1 row) ``` Copy `test_data.csv` using the following copy command and check the status in parallel from another terminal. ```sql \copy testcopy from 'testdata.csv' (DELIMITER ','); ``` ```sql select * from pgstatprogress_copy ; ``` You should see output similar to the following: ```output pid | datid | datname | relid | command | type | ybstatus | bytesprocessed | bytestotal | tuplesprocessed | tuples_excluded +-+-+-+--++-+--+-++-- 74390 | 13288 | yugabyte | 16404 | COPY FROM | PIPE | IN PROGRESS | 766682 | 0 | 40000 | 0 (1 row) ``` Copy `test_data.csv` using the following command, and then interrupt it using Ctrl+C: ```sql \copy testcopy from 'testdata.csv' (DELIMITER ','); ``` ```output ^CCancel request sent ERROR: canceling statement due to user request CONTEXT: COPY test_copy, line 55184: "55184, 55184, 55184" ```` Verify `yb_status` using the view. ```sql select * from pgstatprogress_copy ; ``` ```output pid | datid | datname | relid | command | type | ybstatus | bytesprocessed | bytestotal | tuplesprocessed | tuples_excluded -+-+-+-+--++--+--+-++-- 74390 | 13288 | yugabyte | 16399 | COPY FROM | PIPE | ERROR | 766682 | 0 | 40000 | 0 (1 row) ``` The default for `ROWSPERTRANSACTION` is 20K and the interruption occurred on row 55184 in the example. As a result, the first two transactions of 20K rows were persisted resulting in 40K rows reported by `tuplesprocessed`. The next in-progress_ transaction was interrupted, and for this reason the remaining 15184 rows were not persisted. Refer to to track planning and execution of all the SQL statements. Refer to to analyze live queries. Refer to to optimize YSQL's EXPLAIN and EXPLAIN ANALYZE queries. Refer to show the query execution plan generated by YSQL.