Package: csv2pg
csv2pgA simple and fast cli application to load a csv into postgreslow memory footprintblazing fast (pg copy)lines and fields validation (--skip-error)Installationhttps://pypi.org/project/csv2pg/pipinstall--usercsv2pgUsage$ csv2pg --help
Usage: csv2pg [OPTIONS] TABLE FILEPATH

  COPY FROM 'csv' TO 'postgres'

Options:
  -h, --host TEXT             database server host  [default: localhost]
  -p, --port INTEGER          database server port  [default: 5432]
  -d, --dbname TEXT           database user name  [default: $USER]
  -U, --username TEXT         database name to connect to  [default: $USER]
  -W, --password              force password prompt
  -v, --verbose
  --progress                  display progress bar
  --skip-error                detect, ignore and export errors to
                              <filepath>.err  [default: False]

  --header / --no-header      [default: True]
  --rownum / --no-rownum      include line number in a _rownum column
                              [default: False]

  --filename / --no-filename  include filename in a _filename column
                              [default: False]

  --delimiter TEXT            char separating the fields  [default: ,]
  --quotechar TEXT            char used to quote a field  [default: "]
  --doublequote               When True, escapechar is replaced by doubling
                              the quote char  [default: False]

  --escapechar TEXT           char used to esapce the quote char  [default: \]
  --lineterminator TEXT       line ending sequence  [default:  ]
  --null TEXT                 will be treated as NULL by postgres  [default: ]
  --encoding TEXT             [default: utf-8]
  --overwrite                 destroy table before inserting csv  [default:
                              False]

  --unlogged                  insert in an UNLOGGED table (faster)  [default:
                              False]

  --buffer INTEGER            size of the read buffer to be used by COPY FROM
                              [default: 8192]

  --version                   Show the version and exit.
  --help                      Show this message and exit.Basic usage:csv2pg-hlocalhost-p5432-Upostgres-dpostgrespublic.datadata.csv--verboseBasic usage with postgres environment variables:PGHOST=localhostPGPORT=5432PGDATABASE=postgresPGUSER=postgresPGPASSWORD=csv2pgpublic.datadata.csv--verboseLoading a tab delimited latin-1 encoded file in an unlogged table with_filenameand_rownumcolumns, skipping errors and displaying progress bar:PGPASSWORD=csv2pg-hlocalhost-p25432-dtest-Utest\--delimiter=$'\t'--encoding="iso-8859-1"\--overwrite--unlogged\--filename--rownum\--skip-error--progress\public.datadata.csvQuick testStart a postgres database:dockerrun-d--rm\-p25432:5432\--namecsv2pg-test\-ePOSTGRES_DB=test\-ePOSTGRES_USER=test\-ePOSTGRES_PASSWORD=test\postgresDownload and import a test asset file:wgethttps://raw.githubusercontent.com/DavidLacroix/csv2pg/master/tests/assets/simple.csv.PGPASSWORD=testcsv2pg-hlocalhost-p25432-Utest-dtestpublic.datasimple.csv--progressFrom pythonimportcsv2pgHOST="localhost"PORT=25432DBNAME="test"USER="test"PASSWORD="test"csv2pg.copy_to(HOST,PORT,DBNAME,USER,PASSWORD,"public.data","./simple.csv",verbose=True)Precautionthe--overwriteoption will drop the table before inserting the new records in.the--rownumand--filenameoptions will slightly increase the insertion time (increase the data to write on disk)the--skip-erroroption will slightly increase the insertion time (fields and lines validation)--verboseand--progressused together might spoil the console output