Package: ibis-framework
IbisWhat is Ibis?Ibis is a Python library that provides a lightweight, universal interface for data wrangling. It helps Python users explore and transform data of any size, stored anywhere.Ibis has three primary components:A dataframe API for Python.
Python users can write Ibis code to manipulate tabular data.Interfaces to 15+ query engines.Wherever data is stored, people can use Ibis as their API of choice to communicate with any of those query engines.Deferred execution.
Ibis uses deferred execution, so execution of code is pushed to the query engine.
Users can execute at the speed of their backend, not their local computer.Why Use Ibis?Ibis aims to be a future-proof solution to interacting with data using Python and can accomplish this goal through its main features:Familiar API: Ibis’s API design borrows from popular APIs like pandas and dplyr that most users already know and like to use.Consistent syntax: Ibis aims to be a universal Python API for tabular data of any size, big or small.Deferred execution: Ibis pushes code execution to the query engine and only moves required data into memory when necessary.
Analytics workflows are faster and more efficientInteractive mode: Ibis provides an interactive mode in which users can quickly diagnose problems, explore data, and mock up workflows and pipelines locally.10+ supported backends: Ibis supports multiple query engines and DataFrame APIs.
Use one interface to transform with your data wherever it lives: from DataFrames in pandas to Parquet files through DuckDB to tables in BigQuery.Minimize rewrites: Teams can often keep their Ibis code the same regardless of backend changes, like increasing or decreasing computing power, changing the number or size of their databases, or switching backends entirely.Flexibility when you need it: When Ibis doesn't support something, it provides a way to jump directly into SQL.Common Use CasesSpeed up prototype to production.Scale code written and tested locally to a distributed system or cloud SQL engine with minimal rewrites.Boost performance of existing Python or pandas code.For example a general rule of thumb for pandas is "Have 5 to 10 times as much RAM as the size of your dataset".
When a dataset exceeds this rule using in-memory frameworks like pandas can be slow.
Instead, using Ibis will significantly speed up your workflows because of its deferred execution.
Ibis also empowers you to switch to a faster database engine, without changing much of your code.Get rid of long, error-prone,f-strings.Ibis provides one syntax for multiple query engines and dataframe APIs that lets you avoid learning new flavors of SQL or other framework-specific code.
Learn the syntax once and use that syntax anywhere.BackendsIbis acts as a universal frontend to the following systems:Apache Arrow DataFusion(experimental)Apache Druid(experimental)Apache ImpalaApache PySparkBigQueryClickHouseDaskDuckDBHeavyAIMySQLOracle(experimental)PandasPolars(experimental)PostgreSQLSQL ServerSQLiteSnowflake(experimental)Trino(experimental)The list of supported backends is continuously growing. Anyone can get involved
in adding new ones! Learn more about contributing to ibis in our contributing
documentation athttps://github.com/ibis-project/ibis/blob/main/docs/CONTRIBUTING.mdInstallationInstall Ibis from PyPI with:pipinstall'ibis-framework[duckdb]'Or from conda-forge with:condainstallibis-framework-cconda-forge(It’s a common mistake topip install ibis. If you try to use Ibis and get errors early on try uninstallingibisand installingibis-framework)To discover ibis, we suggest starting with the DuckDB backend (which is included by default in the conda-forge package). The DuckDB backend is performant and fully featured.To use ibis with other backends, include the backend name in brackets for PyPI:pipinstall'ibis-framework[postgres]'Or useibis-$BACKENDwhere$BACKENDis the specific backend you want to use when installing from conda-forge:condainstallibis-postgres-cconda-forgeGetting Started with IbisWe provide a number of tutorial and example notebooks in theibis-examples. The easiest way
to try these out is through the online interactive notebook environment
provided here:You can also get started analyzing any dataset, anywhere with just a few lines
of Ibis code. Here’s an example of how to use Ibis with a SQLite database.Download the SQLite database from theibis-tutorial-dataGCS (Google Cloud
Storage) bucket, then connect to it using ibis.curl-LsS-ogeography.db'https://storage.googleapis.com/ibis-tutorial-data/geography.db'Connect to the database and show the available tables>>>importibis>>>fromibisimport_>>>ibis.options.interactive=True>>>con=ibis.sqlite.connect("geography.db")>>>con.tablesTables-------countries-gdp-independenceChoose thecountriestable and preview its first few rows>>>countries=con.tables.countries>>>countries.head()┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓┃iso_alpha2┃iso_alpha3┃iso_numeric┃fips┃name┃capital┃area_km2┃population┃continent┃┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩│string│string│int32│string│string│string│float64│int32│string│├────────────┼────────────┼─────────────┼────────┼──────────────────────┼──────────────────┼──────────┼────────────┼───────────┤│AD│AND│20│AN│Andorra│AndorralaVella│468.0│84000│EU││AE│ARE│784│AE│UnitedArabEmirates│AbuDhabi│82880.0│4975593│AS││AF│AFG│4│AF│Afghanistan│Kabul│647500.0│29121286│AS││AG│ATG│28│AC│AntiguaandBarbuda│St.Johns│443.0│86754│NA││AI│AIA│660│AV│Anguilla│TheValley│102.0│13254│NA│└────────────┴────────────┴─────────────┴────────┴──────────────────────┴──────────────────┴──────────┴────────────┴───────────┘Show the 5 least populous countries in Asia>>>(...countries.filter(_.continent=="AS")....select("name","population")....order_by(_.population)....limit(5)...)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓┃name┃population┃┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩│string│int32│├────────────────────────────────┼────────────┤│Cocos[Keeling]Islands│628││BritishIndianOceanTerritory│4000││Brunei│395027││Maldives│395650││Macao│449198│└────────────────────────────────┴────────────┘Community and ContributingIbis is an open source project and welcomes contributions from anyone in the community.Readthe contributing guide.We care about keeping the community welcoming for all. Check outthe code of conduct.The Ibis project is open sourced under theApache License.Join our community by interacting on GitHub or chatting with us onZulip.For more information visithttps://ibis-project.org/.