Package: dbrequests
dbrequestsdbrequests is a python package built for easy use of raw SQL within python and pandas projects.It uses ideas fromrecordsand is built usingsqlalchemy-engines, but is more heavily integrated with pandas. It aims to reproduce the pilosophy behind the R-packagedbtools.Database support includes RedShift, Postgres, MySQL, SQLite, Oracle, and MS-SQL (drivers not included).UsageSend queries and bulk queriesEasy sending of raw sql and output as pandas DataFrames, with credentials given as dictionary (for an example see creds_example.json) or the url of the database:fromdbrequestsimportDatabasedb=Database(creds=creds)df=db.send_query("""SELECT * FROM test;""")df# table test as pandas DataFrameYou can put the sql query in a file and directsend_queryto the file. The sql-file may be parametrized:SELECT{col1},{col2}FROMtest;fromdbrequestsimportDatabasedb=Database(creds=creds,sql_dir='/path/to/dir/')db.send_query('select',col1='id',col2='name')df# table test, including columns 'id', 'name' as pandas DataFrameYou can also pass arguments to pandasread_sql-Function:fromdbrequestsimportDatabasedb=Database(creds=creds,sql_dir='/path/to/dir/')db.send_query('select',col1='id',col2='name',index_col='id')df# table test, including column 'name' as pandas DataFrame with index 'id'You may also send queries with no table as output to the database viasend_bulk_query, which exhibits the same behavior assend_query:db.send_bulk_query('drop test from test;')Send dataEasy sending of pandas Dataframes in multiple modes:db.send_data(df,'table',mode='insert')Supported modes are:'insert': Appending new records. Duplicate primary keys will result in errors (sql insert into).'truncate': Delete the table and completely rewrite it (sql truncate and insert into).'replace': Replace records with duplicate primary keys (sql replace into).'update': Update records with duplicate primary keys (sql insert into duplicate key update).UtilitiesComments can be automatically removed from SQL code by addingremove_comments=Trueeither to the Database defintion or send_query. This is especially useful if outcommenting code blocks including parametized variables and thus{}. The default of this behavior isFalse.Percentage signs can be transfered to a Python readable way by addingescape_percentage=Trueeither to the Database definition or send_query. This means percentage signs dont have to be escaped manually when sending them via Python. The default isFalse.Database.get_table_names will give existing tables in the databaseParameters such eschunksizeforpandas.to_sqlmay be given to the wrapper functionsend_dataand are handed over to pandas. The same is true forsend_query.For transactions the context managertransactionmay be of use.InstallationThe package can be installed via pip:pip install dbrequestsExtensibilitydbrequests is designed to easily accommodate different needs in the form of drivers / dialects. For examples of how to extend the capabilities of the Connection class, see connection_subclass.py under examples.Existing extensionsMySQL / MariaDB: usefrom dbrequests.mysql import Databasefor using the MySQL specific extension as Database connector. The extension provides MySQL specific functionalities, like usingload data infilefor writing data to tables.Version 1.0implemented send_data and send_query wrappers for pandas sql-functionalityVersion 1.0.8renamed to dbrequestsVersion 1.1added handling of percentage signsadded possibility of automated comment removalensured the compatibility of raw SQL code with code sent via dbrequestsVersion 1.2changed static loading of the connection class to a flexible model, allowing to override connections for different SQL driversVersion 1.3added mysql backend for specific mysql and mariadb supportadded possibility for send_data to use infile for mysql databasesVersion 1.3.8dbrequests.mysql module:see #11 for motivation on moving dialect specific implementation into extrassee #15 for mysql specific send_data via load data local infileVersion 1.3.9dbrequests.mysql:see #20 for memory efficient send_data using datatableVersion 1.3.10dbrequests.mysql:see #24 for bugfix while writing to csvVersion 1.3.11dbrequests.mysql:see #28 for bugfix when sending escape sequencesVersion 1.3.12dbrequests.mysql:see #22 for performance improvements for send_query when reading large datasetssupport for pymysql and mysqldbVersion 1.3.13dbrequests.mysqlsee #30: fixes handling of None/NULL values in columnsVersion 1.3.14dbrequests.mysql:see #32 for bugfix in send_query for empty result setsVersion 1.3.15dbrequests:see #27 for bugfix in Database class when specifiying a port in a
credentials object.the argument 'creds' in the init method of a database class is now
deprecatedthe argument 'db_url' can now handle str and dict type; str is a
sqlalchemy url; a dict a credentials objectcredential objects can now have additional fields which will be used as
elements in connect_args for sqlalchemies create_engine: see #12dbrequests.mysqlsee #36 for bugfix while sending an empty frameVersion 1.3.16dbrequests.mysqlsee #35: New send_data modes: update_diffs, insert_diffs, replace_diffsVersion 1.3.17dbrequests.mysqlnew function in database: send_delete for deleting rows in a database.send_data in mode 'update' now allows to send only partial subset of
columns.Version 1.4.0dbrequests.mysqlbugfix in send_data with mode [update|insert|replace]_diffs: same as #36Version 1.4.1dbrequests.mysqlbugfix for upstream bug in mariadb: sending diffs needs persisten tables
instead of temporary.Version 1.4.2dbrequests.mysqlcreating temorary removing partitions and system versioned from temporary
tables.new mode for send data: sync_diffs: update differences and delete
deprecated rows.new mode for delete data: in_delete_col: mark rows to delete, then delete.bugfix for temporary tables: now properly removes tables.Version 1.4.3-5dbrequests.mysqlMore stable and reliable Version of sync_diffs mode for sending data.
Respects scarce resources on the mysql server.Version 1.4.6dbrequests.mysqlHOTFIX #51: Bug with latest datatable versionVersion 1.4.7dbrequests.mysqlbugfix of #52Version 1.4.8dbrequests.mysqlfix for creating temporary files on Windows.