Package: duckberg
duckbergDuckBergquery your iceberg data easily and efficientlyTable of ContentsAboutInstallationFeaturesPlaygroundDevelopmentLicenseAboutDuckberg is a Python package that synthesizes the power of PyIceberg and DuckDb. PyIceberg enables efficient
interaction with Apache Iceberg, a format for handling large datasets, while DuckDb offers swift in-memory data
analysis. When combined, these tools create Duckberg, which simplifies the querying process for large Iceberg
datasets stored on blob storage with a user-friendly Pythonic approach.The underlying principle of the Duckberg Python package is to execute your SQL queries only on those data lake files
that contain the necessary data for your results. To fully utilize the benefits of this package, it's assumed that
your data is partitioned in a manner that suits your query and use case.Iceberg catalog typesDuckberg supports the same Iceberg catalogs asPyIceberg, including
REST, SQL, Hive, Glue, and DynamoDB. These catalogs are sources of information about Iceberg datasets, tables,
partitions, etc. Before using Duckberg, ensure that you have access to an Iceberg catalog that can be utilized.Installationpip install duckbergFeaturesEasy initialisationFollowing initialisation is using theREST Iceberg catalogwithAmazon S3as a iceberg data storage.fromduckbergimportDuckBergcatalog_config:dict[str,str]={"type":"rest",# Iceberg catalog type"uri":"http://iceberg-rest:8181/",# url for Iceberg catalog"credentials":"user:password",# credentials for Iceberg catalog"s3.endpoint":S3_ENDPOINT,# s3"s3.access-key-id":S3_ACCESS_KEY_ID,"s3.secret-access-key":S3_SECET_KEY}db=DuckBerg(catalog_name="warehouse",catalog_config=catalog_config)Listing tablesdb.list_tables()Listing partitions for particular tabledb.list_partitions(table="nyc.taxis")Querying data to Pandas dataframeIn the latest new update we have added very crude and simple SQL parser that can extract necessary information from the SQL query without the need to specifytableandpartition_filters. This is the new and prefered way:query="SELECT * FROM nyc.taxis WHERE payment_type = 1 AND trip_distance > 40 ORDER BY tolls_amount DESC"df=db.select(sql=query).read_pandas()Old way of selecting data (will get deprecated in the future):query="SELECT * FROM nyc.taxis WHERE trip_distance > 40 ORDER BY tolls_amount DESC"df=db.select(sql=query,table="nyc.taxis",partition_filter="payment_type = 1").read_pandas()PlaygroundYou can run the playground environment running docker compose in theplaygroundcdplayground
docker-composeup-dThe initial run could take additional time for jupyter docker image build. Then you can accessIceberg data initOnce all the containers have been initiated run theSpark Iceberg Jupyter notebookthat will
init the Iceberg data and catalog.Duckberg playgroundNavigate tolocalhost:8888. Then select example Jupyter notebook you want to run and enjoy Duckberg!If you need to change the version of Duckberg used in the playground, change the version inrequirements.txtand rebuild dockerdocker-composeup-d--no-deps--buildjupyterDevelopmentFor the development, there is recommendation to use Python 3.10. If you manage your Python versions byPyenvusepyenvinstall3.10.13
pyenvglobal3.10.13then create and activate virtual environmentpython-mvenvvenvsourcevenv/bin/activateupgrade pip and install dependenciespipinstall--upgradepip
pipinstall.then run dockers that contains Iceberg catalog and file storage containing iceberg filescdplayground
docker-composeup-dinit data by runningInit Jupyter notebookand
run/test Duckberg in the filetests/duckberg-sample.pyStyle & FormattingUsehatchrunlint:fmt
hatchrunlint:styleBuilding packageThe Duckberg project is managed byHatch. Follow [Hatch docs] for an installation
or just install by commandbrewinstallhatchorpipinstallhatchIncrease package byhatchversion"x.x.x"Buildhatchbuildand publishhatchpublishLicenseduckbergis distributed under the terms of theApache 2.0license.