Package: csv2db
___       ____  
  ___________   _|__ \ ____/ / /_ 
 / ___/ ___/ | / /_/ // __  / __ \
/ /__(__  )| |/ / __// /_/ / /_/ /
\___/____/ |___/____/\____/_____/The CSV to database command line loader.csv2dbreads CSV files and loads them into a database.
Rather than having to go through the CSV data first and find out what columns and data types are present in the CSV files,csv2dbwill read the header in each CSV file and automatically load data into the columns of the same name into the target table.
The case of the header column names will be preserved as present in the CSV file.
Spaces in the header column names are automatically replaced with_characters, for example, the columnstation idin the CSV file will be interpreted asstation_idcolumn in the table.This approach allows you to get data into the database first and worry about the data cleansing part later,
which is usually much easier once the data is in the database rather than in the CSV files.csv2dbis capable of scanning all CSV file headers at once and generate aCREATE TABLEstatement with all the column names present.
This is particularly useful if the format of the CSV files has changed over time or because you want to load different CSV file types into the same database table.Usage$./csv2db-husage: csv2db [-h] {generate,gen,load,lo} ...The CSV to database command line loader.Version: 1.6.0(c)Gerald Venzlpositional arguments:{generate,gen,load,lo}generate (gen)      Prints a CREATE TABLE SQL statement to create thetable and columns based on the header row of the CSVfile(s).load (lo)           Loads the data from the CSV file(s) into the database.options:-h, --help            show this help message and exit$./csv2dbgenerate-husage: csv2db generate [-h] [-f FILE] [-e ENCODING] [-v] [--debug][-o {oracle,mysql,postgres,sqlserver,db2}] [-t TABLE][-c COLUMN_TYPE] [-s SEPARATOR] [-q QUOTE][--case-insensitive-identifiers] [--quote-identifiers]options:-h, --help            show this help message and exit-f FILE, --file FILE  The file to read the header from, by default all*.csv.zip files-e ENCODING, --encoding ENCODINGThe file encoding to be used to read the file, see https://docs.python.org/3/library/codecs.html#standard-encodings for a list of all allowed encodings.-v, --verbose         Verbose output.--debug               Debug output.-o {oracle,mysql,postgres,sqlserver,db2}, --dbtype {oracle,mysql,postgres,sqlserver,db2}The database type.-t TABLE, --table TABLEThe table name to use.-c COLUMN_TYPE, --column-type COLUMN_TYPEThe column type to use for the table generation.-s SEPARATOR, --separator SEPARATORThe columns separator character(s).-q QUOTE, --quote QUOTEThe quote character on which a string won't be split.--case-insensitive-identifiersIf set, all identifiers will be upper-cased.--quote-identifiers   If set, all table and column identifiers will bequoted.$./csv2dbload-husage: csv2db load [-h] [-f FILE] [-e ENCODING] [-v] [--debug] -t TABLE[-o {oracle,mysql,postgres,sqlserver,db2}] -u USER[-p PASSWORD] [-m HOST] [-n PORT] [-d DBNAME] [-b BATCH][-s SEPARATOR] [-q QUOTE] [-a] [--truncate] [-i] [-l][--case-insensitive-identifiers] [--quote-identifiers]options:-h, --help            show this help message and exit-f FILE, --file FILE  The file to load, by default all *.csv.zip files-e ENCODING, --encoding ENCODINGThe file encoding to be used to read the file, see https://docs.python.org/3/library/codecs.html#standard-encodings for a list of all allowed encodings.-v, --verbose         Verbose output.--debug               Debug output.-t TABLE, --table TABLEThe table name to use.-o {oracle,mysql,postgres,sqlserver,db2}, --dbtype {oracle,mysql,postgres,sqlserver,db2}The database type.-u USER, --user USER  The database user to load data into.-p PASSWORD, --password PASSWORDThe database schema password. csv2db will prompt forthe password if the parameter is missing which is amore secure method of providing a password.-m HOST, --host HOST  The host name on which the database is running on.-n PORT, --port PORT  The port on which the database is listening. If notpassed on the default port will be used (Oracle: 1521,MySQL: 3306, PostgreSQL: 5432, SQL Server: 1433, DB2:50000).-d DBNAME, --dbname DBNAMEThe name of the database.-b BATCH, --batch BATCHHow many rows should be loaded at once.-s SEPARATOR, --separator SEPARATORThe columns separator character(s).-q QUOTE, --quote QUOTEThe quote character on which a string won't be split.-a, --directpath      Execute a direct path INSERT load operation (Oracleonly).--truncate            Truncate/empty table before loading.-i, --ignore          Ignore erroneous/invalid lines in files and continuethe load.-l, --log             Log erroneous/invalid lines in *.bad file of the samename as the input file (this implies the --ignoreoption).--case-insensitive-identifiersIf set, all identifiers will be upper-cased.--quote-identifiers   If set, all table and column identifiers will bequoted.How to use csv2dbLoading CSV files into the databasecsv2dbcan load plain text csv files as well as compressed csv files in.zipor.gzformat without having to uncompress them first.$./csv2dbload-ftest/resources/201811-citibike-tripdata.csv-tcitibikes-ucsv_data-pcsv_data-dORCLPDB1Loading file test/resources/201811-citibike-tripdata.csvFile loaded.$./csv2dbload-ftest/resources/201811-citibike-tripdata.csv.gz-tcitibikes-ucsv_data-pcsv_data-dORCLPDB1Loading file test/resources/201811-citibike-tripdata.csv.gzFile loaded.csv2db--verboseoption will provide verbose output.$./csv2dbload-ftest/resources/201811-citibike-tripdata.csv-tcitibikes-ucsv_data-pcsv_data-dORCLPDB1--verboseFinding file(s).Found 1 file(s).Establishing database connection.Loading file test/resources/201811-citibike-tripdata.csv16 rows loaded.File loaded.Closing database connection.csv2dbcan load multiple files at once, using either wildcard characters (e.g. data*.csv.zip) or by passing on the folder containing CSV files.Note:String including wildcard characters have to be enclosed in""$./csv2dbload-f"test/resources/201811-citibike-tripdata.*"-tcitibikes-ucsv_data-pcsv_data-dORCLPDB1--verboseFinding file(s).Found 3 file(s).Establishing database connection.Loading file test/resources/201811-citibike-tripdata.csv16 rows loaded.File loaded.Loading file test/resources/201811-citibike-tripdata.csv.gz10 rows loaded.File loaded.Loading file test/resources/201811-citibike-tripdata.csv.zip10 rows loaded.File loaded.Closing database connection.$./csv2dbload-ftest/resources-tcitibikes-ucsv_data-pcsv_data-dORCLPDB1--verboseFinding file(s).Found 3 file(s).Establishing database connection.Loading file test/resources/201811-citibike-tripdata.csv16 rows loaded.File loaded.Loading file test/resources/201811-citibike-tripdata.csv.gz10 rows loaded.File loaded.Loading file test/resources/201811-citibike-tripdata.csv.zip10 rows loaded.File loaded.Closing database connection.csv2dbwill load all values as strings. You can either load all data into a staging table with all columns being strings as well, or rely on implicit data type conversion on the database side.Create a staging tablecsv2dbcan generate the SQL statement for a staging table for your data using thegeneratecommand:$./csv2dbgenerate-ftest/resources/201811-citibike-tripdata.csvCREATETABLE<TABLENAME>(TRIPDURATIONVARCHAR(1000),STARTTIMEVARCHAR(1000),STOPTIMEVARCHAR(1000),START_STATION_IDVARCHAR(1000),START_STATION_NAMEVARCHAR(1000),START_STATION_LATITUDEVARCHAR(1000),START_STATION_LONGITUDEVARCHAR(1000),END_STATION_IDVARCHAR(1000),END_STATION_NAMEVARCHAR(1000),END_STATION_LATITUDEVARCHAR(1000),END_STATION_LONGITUDEVARCHAR(1000),BIKEIDVARCHAR(1000),USERTYPEVARCHAR(1000),BIRTH_YEARVARCHAR(1000),GENDERVARCHAR(1000));By default, you will have to fill in the table name. You can also specify the table name via the-toption:$./csv2dbgenerate-ftest/resources/201811-citibike-tripdata.csv-tSTAGINGCREATETABLESTAGING(TRIPDURATIONVARCHAR(1000),STARTTIMEVARCHAR(1000),STOPTIMEVARCHAR(1000),START_STATION_IDVARCHAR(1000),START_STATION_NAMEVARCHAR(1000),START_STATION_LATITUDEVARCHAR(1000),START_STATION_LONGITUDEVARCHAR(1000),END_STATION_IDVARCHAR(1000),END_STATION_NAMEVARCHAR(1000),END_STATION_LATITUDEVARCHAR(1000),END_STATION_LONGITUDEVARCHAR(1000),BIKEIDVARCHAR(1000),USERTYPEVARCHAR(1000),BIRTH_YEARVARCHAR(1000),GENDERVARCHAR(1000));csv2dbwill useVARCHAR(1000)as default data type for all columns for the staging table. If you wish to use a different data type, you can specify it via the-coption:$./csv2dbgenerate-ftest/resources/201811-citibike-tripdata.csv-tSTAGING-cCLOBCREATETABLESTAGING(TRIPDURATIONCLOB,STARTTIMECLOB,STOPTIMECLOB,START_STATION_IDCLOB,START_STATION_NAMECLOB,START_STATION_LATITUDECLOB,START_STATION_LONGITUDECLOB,END_STATION_IDCLOB,END_STATION_NAMECLOB,END_STATION_LATITUDECLOB,END_STATION_LONGITUDECLOB,BIKEIDCLOB,USERTYPECLOB,BIRTH_YEARCLOB,GENDERCLOB);The idea is to have a staging table that data can be loaded into first and then figure out the correct data types for each column.InstallationYou can installcsv2dbeither by installing it as a Python package,
which will automatically install all dependencies except the Db2 driver (as this one is still in Beta status)$python3-mpipinstallcsv2db$csv2dbor cloning this Git repository$gitclonehttps://github.com/csv2db/csv2dbor by downloading one of the releases$LOCATION=$(curl-shttps://api.github.com/repos/csv2db/csv2db/releases/latest|grep"tag_name"|awk'{print "https://github.com/csv2db/csv2db/archive/" substr($2, 2, length($2)-3) ".zip"}');curl-L-ocsv2db.zip$LOCATION$unzipcsv2db.zip$cdcsv2db*$./csv2dbIn order forcsv2dbto work the appropriate database driver or drivers need to be installed.
This installation is done automatically when installingcsv2dbas a Python package (pip install csv2db).
The following drivers are being used, and are all available onpypi.org:Oracle:oracledbversion 1.1.1+MySQL:mysql-connector-pythonversion 8.0.13+PostgreSQL:psycopg-binaryversion 3.1.9+SQL Server:pymssqlversion 2.1.4+DB2:ibm-dbversion 2.0.9+You can install any of these drivers viapip:$python3-mpipinstalloracledb$python3-mpipinstallmysql-connector-python$python3-mpipinstallpsycopg-binary$python3-mpipinstallpymssql$python3-mpipinstallibm-dbFor more instruction on how to install the driver(s) on your environment,
please see the documentation of the individual driver or refer to thecsv2db Installation Guide.NOTE:You only have to install the driver for the database(s) that you want to load data into.MiscellaneousWhatcsv2dbis and, more importantly, what it is not!Since the very inception ofcsv2db, it has been a core principle for it not to become an ETL tool with all the bells and whistles.
There are already many very good ETL tools out there and the world doesn't need yet another one.
Instead,csv2dbshould aid users as a simple command-line tool to get rows from a delimited file into a database table, and not more!
Following that core design goal,csv2dbwill most likely never provide many database-specific options or parameters for the end-user to set,
it will not deal with explicit data type or character set conversion or globalization support that some databases offer.
If a user requires any of these features or more, he or she should look for one of the already existing ETL tools out there.Simply put,csv2dbdoes not do much more than taking rows from a delimited file and executeINSERT INTOstatements with the values of these rows.
It is there to help users to get the contents of a file into a database table quickly where the data can then be further processed.Exit codescsv2dbreturns following exit codes:Exit codeValueMeaningSUCCESS0Successful execution of the program.GENERIC_ERROR1A generic error occurred.ARGUMENT_ERROR2An argument is either missing or incorrect.DATABASE_ERROR3A database error occurred.DATA_LOADING_ERROR4An error occurred during loading of data.csv2dbwill continue to process other files, if any.$NO_COLORsupportcsv2dbis capable of color coded output and will do so by default (except on Windows).Debug output is yellow.Error output is red.This can be deactivated by setting the$NO_COLORenvironment variable. For more details on$NO_COLORseehttps://no-color.org/LICENSECopyright 2023 Gerald Venzl

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.