Project: YugabyteDB (App Definition and Development / Database)
File: cmd_copy.md

title: COPY statement [YSQL] headerTitle: COPY linkTitle: COPY description: Transfer data between tables and files with the COPY, COPY TO, and COPY FROM statements. menu: v2.18: identifier: cmd_copy parent: statements type: docs Use the `COPY` statement to transfer data between tables and files. `COPY TO` copies from tables to files. `COPY FROM` copies from files to tables. `COPY` outputs the number of rows that were copied. {{%ebnf%}} copy_from, copy_to, copy_option {{%/ebnf%}} Specify the table, optionally schema-qualified, to be copied. Specify the list of columns to be copied. If not specified, then all columns of the table will be copied. Specify a `SELECT`, `VALUES`, `INSERT`, `UPDATE`, or `DELETE` statement whose results are to be copied. For `INSERT`, `UPDATE`, and `DELETE` statements, a RETURNING clause must be provided. Specify the path of the file to be copied. An input file name can be an absolute or relative path, but an output file name must be an absolute path. Critically, the file must be located server-side on the local filesystem of the YB-TServer that you connect to. To work with files that reside on the client, nominate `stdin` as the argument for `FROM` or `stdout` as the argument for `TO`. Alternatively, you can use the `\copy` meta-command in . Critically, these input and output channels are defined client-side in the environment of the client where you run or your preferred programming language. These options request that the data transmission goes via the connection between the client and the server. If you execute the `COPY TO` or `COPY FROM` statements from a client program written in a language like Python, then you cannot use ysqlsh features. Rather, you must rely on your chosen language's features to connect `stdin` and `stdout` to the file that you nominate. However, if you execute `COPY FROM` using , you have the further option of including the `COPY` invocation at the start of the file that you start as a `.sql` script. Create a test table thus: ```plpgsql drop table if exists t cascade; create table t(c1 text primary key, c2 text, c3 text); ``` And prepare `t.sql` thus: ``` copy t(c1, c2, c3) from stdin with (format 'csv', header true); c1,c2,c3 dog,cat,frog \. ``` Notice the `\.` terminator. You can simply execute `\i t.sql` at the prompt to copy in the data. {{< note title="Some client-side languages have a dedicated exposure of COPY" >}} For example, the "psycopg2" PostgreSQL driver for Python (and of course this works for YugabyteDB) has dedicated cursor methods for `COPY`. See . {{< /note >}} The ROWSPERTRANSACTION option defines the transaction size to be used by the `COPY` command. Default: 20000 for YugabyteDB versions 2.14 and 2.15 or later, and 1000 for prior

For example, if the total tuples to be copied are 5000 and `ROWSPERTRANSACTION` is set to 1000, then the database will create 5 transactions and each transaction will insert 1000 rows. If there is an error during the execution of the copy command, then some tuples can be persisted based on the already completed transaction. This implies that if an error occurs after inserting the 3500th row, then the first 3000 rows will be persisted in the database. 1 to 1000 Transaction_1 1001 to 2000 Transaction_2 2001 to 3000 Transaction_3 3001 to 3500 Error First 3000 rows will be persisted to the table and `tuples_processed` will show 3000. The `REPLACE` option replaces the existing row in the table if the new row's primary/unique key conflicts with that of the existing row. Note that `REPLACE` doesn't work on tables that have more than 1 unique constraints (see for explanation) Default: by default conflict error is reported. The `DISABLEFKCHECK` option skips the foreign key check when copying new rows to the table. Default: by default, foreign key check is always performed when `DISABLEFKCHECK` option is not provided. The `SKIP n` option skips the first `n` rows of the file. `n` must be a non-negative integer (). Default: 0, no rows are skipped. The examples below assume a table like this: ```plpgsql yugabyte=# CREATE TABLE users(id BIGSERIAL PRIMARY KEY, name TEXT); yugabyte=# INSERT INTO users(name) VALUES ('John Doe'), ('Jane Doe'), ('Dorian Gray'); yugabyte=# SELECT * FROM users; id | name -+- 3 | Dorian Gray 2 | Jane Doe 1 | John Doe (3 rows) ``` Copy the entire table to a CSV file using an absolute path, with column names in the header. ```plpgsql yugabyte=# COPY users TO '/home/yuga/Desktop/users.txt.sql' DELIMITER ',' CSV HEADER; ``` In the following example, a `WHERE` clause is used to filter the rows and only the `name` column. ```plpgsql yugabyte=# COPY (SELECT name FROM users where name='Dorian Gray') TO '/home/yuga/Desktop/users.txt.sql' DELIMITER ',' CSV HEADER; ``` In the following example, the data exported in the previous examples are imported in the `users` table. ```plpgsql yugabyte=# COPY users FROM '/home/yuga/Desktop/users.txt.sql' DELIMITER ',' CSV HEADER; ``` Assume we ran the command one time, and it failed in the middle, as if the server had crashed. Since we use `ROWSPERTRANSACTION=5000`, we can resume importing at multiples of 5000: ```plpgsql yugabyte=# COPY users FROM '/home/yuga/Desktop/users.txt.sql' WITH (FORMAT CSV, HEADER, DELIMITER ',', ROWSPERTRANSACTION 5000, SKIP 50000); ``` If duplicate rows exist in the database, we can use `REPLACE` to upsert new rows: ```plpgsql yugabyte=# COPY users FROM '/home/yuga/Desktop/users.txt.sql' WITH (FORMAT CSV, HEADER, DELIMITER ',', REPLACE); ``` If we're certain that rows referred by foreign keys already exist, we can disable checking for them to make the import faster: ```plpgsql yugabyte=# COPY users FROM '/home/yuga/Desktop/users.txt.sql' WITH (FORMAT CSV, HEADER, DELIMITER ',', DISABLEFKCHECK); ``` In the following example, we use all of the `COPY` command's options: ```plpgsql yugabyte=# COPY users FROM '/home/yuga/Desktop/users.txt.sql' WITH (FORMAT CSV, HEADER, DELIMITER ',', ROWSPERTRANSACTION 5000, DISABLEFKCHECK, REPLACE, SKIP 50); ``` For COPY operation examples using the `pgstatprogress_copy` view, refer to .