Package: datafusion
DataFusion in PythonThis is a Python library that binds toApache Arrowin-memory query engineDataFusion.DataFusion's Python bindings can be used as an end-user tool as well as providing a foundation for building new systems.FeaturesExecute queries using SQL or DataFrames against CSV, Parquet, and JSON data sources.Queries are optimized using DataFusion's query optimizer.Execute user-defined Python code from SQL.Exchange data with Pandas and other DataFrame libraries that support PyArrow.Serialize and deserialize query plans in Substrait format.Experimental support for transpiling SQL queries to DataFrame calls with Polars, Pandas, and cuDF.Comparison with other projectsHere is a comparison with similar projects that may help understand when DataFusion might be suitable and unsuitable
for your needs:DuckDBis an open source, in-process analytic database. Like DataFusion, it supports
very fast execution, both from its custom file format and directly from Parquet files. Unlike DataFusion, it is
written in C/C++ and it is primarily used directly by users as a serverless database and query system rather than
as a library for building such database systems.Polarsis one of the fastest DataFrame libraries at the time of writing. Like DataFusion, it
is also written in Rust and uses the Apache Arrow memory model, but unlike DataFusion it does not provide full SQL
support, nor as many extension points.Example UsageThe following example demonstrates running a SQL query against a Parquet file using DataFusion, storing the results
in a Pandas DataFrame, and then plotting a chart.The Parquet file used in this example can be downloaded from the following page:https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.pagefromdatafusionimportSessionContext# Create a DataFusion contextctx=SessionContext()# Register table with contextctx.register_parquet('taxi','yellow_tripdata_2021-01.parquet')# Execute SQLdf=ctx.sql("select passenger_count, count(*) ""from taxi ""where passenger_count is not null ""group by passenger_count ""order by passenger_count")# convert to Pandaspandas_df=df.to_pandas()# create a chartfig=pandas_df.plot(kind="bar",title="Trip Count by Number of Passengers").get_figure()fig.savefig('chart.png')This produces the following chart:ConfigurationIt is possible to configure runtime (memory and disk settings) and configuration settings when creating a context.runtime=(RuntimeConfig().with_disk_manager_os().with_fair_spill_pool(10000000))config=(SessionConfig().with_create_default_catalog_and_schema(True).with_default_catalog_and_schema("foo","bar").with_target_partitions(8).with_information_schema(True).with_repartition_joins(False).with_repartition_aggregations(False).with_repartition_windows(False).with_parquet_pruning(False).set("datafusion.execution.parquet.pushdown_filters","true"))ctx=SessionContext(config,runtime)Refer to theAPI documentationfor more information.Printing the context will show the current configuration settings.print(ctx)More ExamplesSeeexamplesfor more information.Executing Queries with DataFusionQuery a Parquet file using SQLQuery a Parquet file using the DataFrame APIRun a SQL query and store the results in a Pandas DataFrameRun a SQL query with a Python user-defined function (UDF)Run a SQL query with a Python user-defined aggregation function (UDAF)Query PyArrow DataCreate dataframeExport dataframeRunning User-Defined Python CodeRegister a Python UDF with DataFusionRegister a Python UDAF with DataFusionSubstrait SupportSerialize query plans using SubstraitExecuting SQL against DataFrame Libraries (Experimental)Executing SQL on PolarsExecuting SQL on PandasExecuting SQL on cuDFHow to install (from pip)Pippipinstalldatafusion# orpython-mpipinstalldatafusionCondacondainstall-cconda-forgedatafusionYou can verify the installation by running:>>>importdatafusion>>>datafusion.__version__'0.6.0'How to developThis assumes that you have rust and cargo installed. We use the workflow recommended bypyo3andmaturin.The Maturin tools used in this workflow can be installed either via Conda or Pip. Both approaches should offer the same experience. Multiple approaches are only offered to appease developer preference. Bootstrapping for both Conda and Pip are as follows.Bootstrap (Conda):# fetch this repogitclonegit@github.com:apache/arrow-datafusion-python.git# create the conda environment for devcondaenvcreate-f./conda/environments/datafusion-dev.yaml-ndatafusion-dev# activate the conda environmentcondaactivatedatafusion-devBootstrap (Pip):# fetch this repogitclonegit@github.com:apache/arrow-datafusion-python.git# prepare development environment (used to build wheel / install in development)python3-mvenvvenv# activate the venvsourcevenv/bin/activate# update pip itself if necessarypython-mpipinstall-Upip# install dependencies (for Python 3.8+)python-mpipinstall-rrequirements-310.txtThe tests rely on test data in git submodules.gitsubmoduleinit
gitsubmoduleupdateWhenever rust code changes (your changes or viagit pull):# make sure you activate the venv using "source venv/bin/activate" firstmaturindevelop
python-mpytestRunning & Installing pre-commit hooksarrow-datafusion-python takes advantage ofpre-committo assist developers with code linting to help reduce the number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the developer but certainly helpful for keeping PRs clean and concise.Our pre-commit hooks can be installed by runningpre-commit install, which will install the configurations in your ARROW_DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete the commit if an offending lint is found allowing you to make changes locally before pushing.The pre-commit hooks can also be run adhoc without installing them by simply runningpre-commit run --all-filesHow to update dependenciesTo change test dependencies, change therequirements.inand run# install pip-tools (this can be done only once), also consider running in venvpython-mpipinstallpip-tools
python-mpiptoolscompile--generate-hashes-orequirements-310.txtTo update dependencies, run with-Upython-mpiptoolscompile-U--generate-hashes-orequirements-310.txtMore detailshere