Package: pangres
pangresThanks tofreesvg.orgfor the logo assetsUpsert with pandas DataFrames (ON CONFLICT DO NOTHINGorON CONFLICT DO UPDATE) for PostgreSQL, MySQL, SQlite and potentially other databases behaving like SQlite (untested) with some additional optional features (see features). Upserting can be done withprimary keysorunique keys.
Pangres also handles the creation of non-existing SQL tables and schemas.Features(optional)Automatical column creation (when a column exists in the DataFrame but not in the SQL table)(optional)Automatical column type alteration for columns that are empty in the SQL table (except for SQlite where alteration is limited)(optional)Creates the table if it is missing(optional)Creates missing schemas in Postgres (and potentially other databases that have a schema system)JSON is supported (with pd.to_sql it does not work) with some exceptions (seeGotchas and caveats)Fast (except for SQlite where some help is needed)Will work even if not all columns defined in the SQL table are thereSQL injection safe (schema, table and column names are escaped and values are given as parameters)New in version 4.1:asynchronous support. Tested usingaiosqlitefor SQlite,asyncpgfor PostgreSQL andaiomysqlfor MySQLRequirementsSQlite >= 3.24.4Python >= 3.6.4See also ./pangres/requirements.txtRequirements for sqlalchemy>=2.0For usingpangrestogether withsqlalchemy>=2.0(sqlalchemy is one of pangres dependencies
listed in requirements.txt) - you will need the following base requirements:alembic>=1.7.2pandas>=1.4.0Python >= 3.8 (pandas>=1.4.0only supports Python >=3.8)Requirements for asynchronous enginesFor using asynchronous engines (such asaiosqlite,asyncpgoraiomysql) you will needPython >= 3.8.Gotchas and caveatsAll flavorsWe can't create JSON columns automatically, but we can insert JSON like objects (list, dict) in existing JSON columns.Postgres"%", ")" and "(" in column names will most likely cause errors with PostgreSQL (this is due to psycopg2 and also affect pd.to_sql). Use the function pangres.fix_psycopg2_bad_cols to "clean" the columns in the DataFrame. You'll also have to rename columns in the SQL table accordingly (if the table already exists).Even though we only do data type alteration on empty columns, since we don't want to lose column information (e.g. constraints) we use true column alteration (instead of drop+create) so the old data type must be castable to the new data type. Postgres seems a bit restrictive in this regard even when the columns are empty (e.g. BOOLEAN to TIMESTAMP is impossible).SQliteSQlite must be version 3.24.4 or higher! UPSERT syntax did not exist before.Column type alteration is not possible for SQlite.SQlite inserts can be at worst 5 times slower than pd.to_sql for some reasons. If you can help please contact me!Inserts with 1000 columns (or 32767 columns for SQlite >= 3.32.0) or more are not supported because one could not even insert one row without exceeding the max number of parameters per queries. One way to fix this would inserting the columns progressively but this seems quite tricky. If you know a better way please contact me.MySQLMySQL will often change the order of the primary keys in the SQL table when using INSERT... ON CONFLICT.. DO NOTHING/UPDATE. This seems to be the expected behavior so nothing we can do about it but please mind that!You may need to provide SQL dtypes e.g. if you have a primary key with text you will need to provide a character length (e.g. VARCHAR(50)) because MySQL does not support indices/primary keys with flexible text length. pd.to_sql has the same issue.NotesThis is a library I was using in production in private with very good results and decided to publish.Ideally such features will be integrated into pandas since there is already aPR on the wayand I would like to give the option to add columns via another PR.There is alsopandabasewhich does almost the same thing (plus lots of extra features) but my implementation is different.
Btw big thanks to pandabase and the sql part of pandas which helped a lot.Installationpip install pangresAdditionally depending on which database you want to work with you will need to install the corresponding library (note that SQlite is included in the standard library):Postgrespip install psycopg2MySQLpip install pymysqlPostgres (asynchronous)pip install asyncpgMySQL (asynchronous)pip install aiomysqlSQLite (asynchronous)pip install aiosqliteUsageHead over topangres' wiki! Note that the wiki is also available
locally under thewiki folder.Note:The wiki is generated with a command which uses my librarynpdoc_to_md.
It must be installed withpip install npdoc_to_mdand you will also need the extra dependencyfirewhich you
can install withpip install fire. Replace$DESTINATION_FOLDERwith the folder of you choice in the command below:npdoc-to-mdrender-folder./wiki_templates$DESTINATION_FOLDERContributingPull requests/issues are welcome.DevelopmentI develop the library inside ofJupyter Labusing thejupytextextension.I recommend using this extension for the best experience.
It will split code blocks within modules in notebook cells and will allowinteractive development.If you wish you can also use the providedconda environment(seeenvironment.ymlfile) inside of Jupyter Lab/Notebook
thanks tonb_conda_kernels.TestingPytestYou can test one or multiple of the following SQL flavors (you will of course need a live database for this): PostgreSQL, SQlite or MySQL.NOTE: in one of the tests ofpangreswe will try to drop and then create a PostgreSQL schema calledpangres_create_schema_test. If the schema existed and was not empty an error will be raised.Clone pangres then set your curent working directory to the root of the cloned repository folder. Then use the commands below. You will have to replace the following variables in those commands:SQLITE_CONNECTION_STRING: replace with a SQlite sqlalchemy connection string (e.g. "sqlite:///test.db")ASYNC_SQLITE_CONNECTION_STRING: replace with an asynchronous SQlite sqlalchemy connection string (e.g. "sqlite+aiosqlite:///test.db")POSTGRES_CONNECTION_STRING: replace with a Postgres sqlalchemy connection string (e.g. "postgres:///user:password@localhost:5432/database"). Specifying schema is optional for postgres (will default to public)ASYNC_POSTGRES_CONNECTION_STRING: replace with an asynchronous Postgres sqlalchemy connection string (e.g. "postgres+asyncpg:///user:password@localhost:5432/database"). Specifying schema is optional for postgres (will default to public)MYSQL_CONNECTION_STRING: replace with a MySQL sqlalchemy connection string (e.g. "mysql+pymysql:///user:password@localhost:3306/database")ASYNC_MYSQL_CONNECTION_STRING: replace with an asynchronous MySQL sqlalchemy connection string (e.g. "mysql+aiomysql:///user:password@localhost:3306/database")PG_SCHEMA(optional): schema for postgres (defaults to public)# 1. Create and activate the build environmentcondaenvcreate-fenvironment.yml
condaactivatepangres-dev# 2. Install pangres in editable mode (changes are reflected upon reimporting)pipinstall-e.# 3. Run pytest# -s prints stdout# -v prints test parameters# --cov=./pangres shows coverage only for pangres# --doctest-modules tests with doctest in all modules# --benchmark-XXX : these are options for benchmarks tests (see https://pytest-benchmark.readthedocs.io/en/latest/usage.html)pytest-s-vpangres--cov=pangres--doctest-modules--async_sqlite_conn=$ASYNC_SQLITE_CONNECTION_STRING--sqlite_conn=$SQLITE_CONNECTION_STRING--async_pg_conn=$ASYNC_POSTGRES_CONNECTION_STRING--pg_conn=$POSTGRES_CONNECTION_STRING--async_mysql_conn=$ASYNC_MYSQL_CONNECTION_STRING--mysql_conn=$MYSQL_CONNECTION_STRING--pg_schema=tests--benchmark-group-by=func,param:engine,param:nb_rows--benchmark-columns=min,max,mean,rounds--benchmark-sort=name--benchmark-name=shortAdditionally, the following flags could be of interest for you:-xfor stopping at the first failure--benchmark-onlyfor only testing benchmarks--benchmark-skipfor skipping benchmarksflake8flake8 must run without errors for pipelines to succeed.
If you are not using the conda environment, you can install flake8 with:pip install flake8.To test flake8 locally you can simply execute this command:flake8 .