Package: duckdb_engine
duckdb_engineBasic SQLAlchemy driver forDuckDBduckdb_engineInstallationUsageUsage in IPython/JupyterConfigurationHow to register a pandas DataFrameThings to keep in mindAuto-incrementing ID columnsPandas read_sql() chunksizeUnsigned integer supportAlembic IntegrationPreloading extensions (experimental)The nameInstallation$pipinstallduckdb-engineDuckDB Engine also has a conda feedstock available, the instructions for the use of which are available in it'srepository.UsageOnce you've installed this package, you should be able to just use it, as SQLAlchemy does a python path searchfromsqlalchemyimportColumn,Integer,Sequence,String,create_enginefromsqlalchemy.ext.declarativeimportdeclarative_basefromsqlalchemy.orm.sessionimportSessionBase=declarative_base()classFakeModel(Base):# type: ignore__tablename__="fake"id=Column(Integer,Sequence("fakemodel_id_sequence"),primary_key=True)name=Column(String)eng=create_engine("duckdb:///:memory:")Base.metadata.create_all(eng)session=Session(bind=eng)session.add(FakeModel(name="Frank"))session.commit()frank=session.query(FakeModel).one()assertfrank.name=="Frank"Usage in IPython/JupyterWith IPython-SQL and DuckDB-Engine you can query DuckDB natively in your notebook! Check outDuckDB's documentationor
Alex Monahan's great demo of this onhis blog.ConfigurationYou can configure DuckDB by passingconnect_argsto the create_engine functioncreate_engine('duckdb:///:memory:',connect_args={'read_only':True,'config':{'memory_limit':'500mb'}})The supported configuration parameters are listed in theDuckDB docsHow to register a pandas DataFrameconn=create_engine("duckdb:///:memory:").connect()# with SQLAlchemy 1.3conn.execute("register",("dataframe_name",pd.DataFrame(...)))# with SQLAlchemy 1.4+conn.execute(text("register(:name, :df)"),{"name":"test_df","df":df})conn.execute("select * from dataframe_name")Things to keep in mindDuckdb's SQL parser is based on the PostgreSQL parser, but not all features in PostgreSQL are supported in duckdb. Because theduckdb_enginedialect is derived from thepostgresqldialect,SQLAlchemymay try to use PostgreSQL-only features. Below are some caveats to look out for.Auto-incrementing ID columnsWhen defining an Integer column as a primary key,SQLAlchemyuses theSERIALdatatype for PostgreSQL. Duckdb does not yet support this datatype because it's a non-standard PostgreSQL legacy type, so a workaround is to use theSQLAlchemy.Sequence()object to auto-increment the key. For more information on sequences, you can find theSQLAlchemy Sequencedocumentation here.The following example demonstrates how to create an auto-incrementing ID column for a simple table:>>>importsqlalchemy>>>engine=sqlalchemy.create_engine('duckdb:////path/to/duck.db')>>>metadata=sqlalchemy.MetaData(engine)>>>user_id_seq=sqlalchemy.Sequence('user_id_seq')>>>users_table=sqlalchemy.Table(...'users',...metadata,...sqlalchemy.Column(...'id',...sqlalchemy.Integer,...user_id_seq,...server_default=user_id_seq.next_value(),...primary_key=True,...),...)>>>metadata.create_all(bind=engine)Pandasread_sql()chunksizeNOTE: this is no longer an issue in versions>=0.5.0ofduckdbThepandas.read_sql()method can read tables fromduckdb_engineinto DataFrames, but thesqlalchemy.engine.result.ResultProxytrips up whenfetchmany()is called. Therefore, for nowchunksize=None(default) is necessary when reading duckdb tables into DataFrames. For example:>>>importpandasaspd>>>importsqlalchemy>>>engine=sqlalchemy.create_engine('duckdb:////path/to/duck.db')>>>df=pd.read_sql('users',engine)### Works as expected>>>df=pd.read_sql('users',engine,chunksize=25)### Throws an exceptionUnsigned integer supportUnsigned integers are supported by DuckDB, and are available induckdb_engine.datatypes.Alembic IntegrationSQLAlchemy's companion libraryalembiccan optionally be used to manage database migrations.This support can be enabling by adding an Alembic implementation class for theduckdbdialect.fromalembic.ddl.implimportDefaultImplclassAlembicDuckDBImpl(DefaultImpl):"""Alembic implementation for DuckDB."""__dialect__="duckdb"After loading this class with your program, Alembic will no longer raise an error when generating or applying migrations.Preloading extensions (experimental)DuckDB 0.9.0+ includes builtin support for autoinstalling and autoloading of extensions, seethe extension documentationfor more information.Until the DuckDB python client allows you to natively preload extensions, I've added experimental support via aconnect_argsparameterfromsqlalchemyimportcreate_enginecreate_engine('duckdb:///:memory:',connect_args={'preload_extensions':['https'],'config':{'s3_region':'ap-southeast-1'}})The nameYes, I'm aware this package should be namedduckdb-driveror something, I wasn't thinking when I named it and it's too hard to change the name now