Package: ezduckdb
ezduckdbTools to make working withduckdbeasier forcodenym.Not intended for general use, but feel free to steal code or ideas.If you'd be super excited about this being made for general use, reach out.AcknowledgementsThere's a ton code and ideas in here from adagsterblog postInstallationpipinstallezduckdbUsageThere are 3 classes in this library:S3AwarePath:pathlib.Path+ s3 pathsSQL: Work with sql files programatically via templating.DuckDB: Connection and Query managerS3AwarePathS3AwarePathadds functionality to thepathlib.Pathclass.is_s3: Is path an s3 path (ies3://....)get_s3_bucketandget_s3_prefix: Break path for use with boto3Retains3://when cast to string (ie in f strings)get_table_name: Get db table name from file name based on codenym convention<schema>_<table>.<extension>fromezduckdbimportS3AwarePaths3_path=S3AwarePath("s3://bucket/curated/s1chema_table1.csv")assertinp.get_s3_bucket()=="bucket"assertinp.get_s3_prefix()=="curated/s1chema_table1.csv"assertstr(inp)=="s3://bucket/curated/s1chema_table1.csv"assertinp.is_s3()assertinp.get_table_name()==("s1chema","table1")SQLSQLenable type based templating for programatical sql query generation for duckdb.Non-exhaustive list of replacements:pd.DataFrameis converted todf_<id>in the query to enable pandas queryingStrare replaced with the string value enclosed in single quotesIntare replaced with the value without quotesSQLreplaces recusively for nested queryingBasicfromezduckdbimportSQLexample=SQL("SELECT * FROM $table WHERE id = $id",table="foo",id=1)assertinp.to_string()=="SELECT * FROM 'foo' WHERE id = 1"PandasfromezduckdbimportSQLimportpandasaspddf=pd.DataFrame({"id":[1,2,3]})inp=SQL("SELECT * FROM $table",table=df)assertinp.to_string()=="SELECT * FROM df_"+str(id(df))NestedfromezduckdbimportSQLexample=SQL("SELECT * FROM $table",table=SQL("SELECT * FROM $table",table="foo"))assertinp.to_string()=="SELECT * FROM (SELECT * FROM 'foo')"DuckDBDuckDBis a connection manager for duckdb that has some convenience methods for querying.Ifs3_storage_used=Truethenquerymethod will:Loadhttpfsandawsduckdb extensionscallload_aws_credentialspassing theaws_profile.querymethod will:Do all sql templating forSQLobject.Return apd.DataFrameof the results if applicableProvide a context manager for pure sql querying with stringsTemplated Querying (Querying withSQLobjects)Basic QueryingfromezduckdbimportDuckDBimportpandasaspddb=DuckDB(s3_storage_used=False)assertdb.query(SQL("select 1")).values==pd.DataFrame([(1,)]).valuesPandas QueryingfromezduckdbimportDuckDBimportpandasaspddb=DuckDB(s3_storage_used=False)df=pd.DataFrame({"id":[1,2,3]})actual=db.query(SQL("SELECT * FROM $table",table=df))expected=pd.DataFrame([(1,),(2,),(3,)])assert(actual.values==expected.values).all()S3 queryingfromezduckdbimportDuckDBimportpandasaspddb=DuckDB(s3_storage_used=True)s3_path="s3://codenym-automated-testing/ezduckdb/parquet/schema1_table1.parquet"actual=db.query(SQL("SELECT * FROM read_parquet($s3_path)",s3_path=s3_path))expected=pd.DataFrame([[1,4],[2,5],[3,6]])assert(actual.values==expected.values).all()Context Manager (Querying with Strings)fromezduckdbimportDuckDBimportpandasaspdwithDuckDB(s3_storage_used=False)asconn:assertconn.query("select 1").df().values==pd.DataFrame([(1,)]).values