Package: polars-u64-idx
Documentation:Python-Rust-Node.js-R|StackOverflow:Python-Rust-Node.js-R|User guide|DiscordPolars: Blazingly fast DataFrames in Rust, Python, Node.js, R, and SQLPolars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust usingApache Arrow Columnar Formatas the memory model.Lazy | eager executionMulti-threadedSIMDQuery optimizationPowerful expression APIHybrid Streaming (larger-than-RAM datasets)Rust | Python | NodeJS | R | ...To learn more, read theuser guide.Python>>>importpolarsaspl>>>df=pl.DataFrame(...{..."A":[1,2,3,4,5],..."fruits":["banana","banana","apple","apple","banana"],..."B":[5,4,3,2,1],..."cars":["beetle","audi","beetle","beetle","beetle"],...}...)# embarrassingly parallel execution & very expressive query language>>>df.sort("fruits").select(..."fruits",..."cars",...pl.lit("fruits").alias("literal_string_fruits"),...pl.col("B").filter(pl.col("cars")=="beetle").sum(),...pl.col("A").filter(pl.col("B")>2).sum().over("cars").alias("sum_A_by_cars"),...pl.col("A").sum().over("fruits").alias("sum_A_by_fruits"),...pl.col("A").reverse().over("fruits").alias("rev_A_by_fruits"),...pl.col("A").sort_by("B").over("fruits").alias("sort_A_by_B_by_fruits"),...)shape:(5,8)┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────┐│fruits┆cars┆literal_stri┆B┆sum_A_by_ca┆sum_A_by_fr┆rev_A_by_fr┆sort_A_by_B││---┆---┆ng_fruits┆---┆rs┆uits┆uits┆_by_fruits││str┆str┆---┆i64┆---┆---┆---┆---││┆┆str┆┆i64┆i64┆i64┆i64│╞══════════╪══════════╪══════════════╪═════╪═════════════╪═════════════╪═════════════╪═════════════╡│"apple"┆"beetle"┆"fruits"┆11┆4┆7┆4┆4││"apple"┆"beetle"┆"fruits"┆11┆4┆7┆3┆3││"banana"┆"beetle"┆"fruits"┆11┆4┆8┆5┆5││"banana"┆"audi"┆"fruits"┆11┆2┆8┆2┆2││"banana"┆"beetle"┆"fruits"┆11┆4┆8┆1┆1│└──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘SQL>>>df=pl.scan_ipc("file.arrow")>>># create a SQL context, registering the frame as a table>>>sql=pl.SQLContext(my_table=df)>>># create a SQL query to execute>>>query="""...   SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM my_table...   WHERE id1 = 'id016'...   LIMIT 10... """>>>## OPTION 1>>># run the query, materializing as a DataFrame>>>sql.execute(query,eager=True)shape:(1,2)┌────────┬────────┐│sum_v1┆min_v2││---┆---││i64┆i64│╞════════╪════════╡│298268┆1│└────────┴────────┘>>>## OPTION 2>>># run the query but don't immediately materialize the result.>>># this returns a LazyFrame that you can continue to operate on.>>>lf=sql.execute(query)>>>(lf.join(other_table)....group_by("foo")....agg(...pl.col("sum_v1").count()...).collect())SQL commands can also be run directly from your terminal using the Polars CLI:# run an inline SQL query>polars-c"SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM read_ipc('file.arrow') WHERE id1 = 'id016' LIMIT 10"# run interactively>polars
PolarsCLIv0.3.0
Type.helpforhelp.

>SELECTsum(v1)assum_v1,min(v2)asmin_v2FROMread_ipc('file.arrow')WHEREid1='id016'LIMIT10;Refer to thePolars CLI repositoryfor more information.Performance 🚀🚀Blazingly fastPolars is very fast. In fact, it is one of the best performing solutions available.
See the results inDuckDB's db-benchmark.In theTPC-H benchmarksPolars is orders of magnitude faster than pandas, dask, modin and vaex
on full queries (including IO).LightweightPolars is also very lightweight. It comes with zero required dependencies, and this shows in the import times:polars: 70msnumpy: 104mspandas: 520msHandles larger-than-RAM dataIf you have data that does not fit into memory, Polars' query engine is able to process your query (or parts of your query) in a streaming fashion.
This drastically reduces memory requirements, so you might be able to process your 250GB dataset on your laptop.
Collect withcollect(streaming=True)to run the query streaming.
(This might be a little slower, but it is still very fast!)SetupPythonInstall the latest Polars version with:pipinstallpolarsWe also have a conda package (conda install -c conda-forge polars), however pip is the preferred way to install Polars.Install Polars with all optional dependencies.pipinstall'polars[all]'You can also install a subset of all optional dependencies.pipinstall'polars[numpy,pandas,pyarrow]'TagDescriptionallInstall all optional dependencies (all of the following)pandasInstall with pandas for converting data to and from pandas DataFrames/SeriesnumpyInstall with NumPy for converting data to and from NumPy arrayspyarrowReading data formats using PyArrowfsspecSupport for reading from remote file systemsconnectorxSupport for reading from SQL databasesxlsx2csvSupport for reading from Excel filesopenpyxlSupport for reading from Excel files with native typesdeltalakeSupport for reading and writing Delta Lake TablespyicebergSupport for reading from Apache Iceberg tablesplotSupport for plot functions on DataFramestimezoneTimezone support, only needed if you are on Python<3.9 or WindowsReleases happen quite often (weekly / every few days) at the moment, so updating Polars regularly to get the latest bugfixes / features might not be a bad idea.RustYou can take latest release fromcrates.io, or if you want to use the latest features / performance improvements
point to themainbranch of this repo.polars={git="https://github.com/pola-rs/polars",rev="<optional git tag>"}Requires Rust version>=1.71.ContributingWant to contribute? Read ourcontribution guideline.Python: compile Polars from sourceIf you want a bleeding edge release or maximal performance you should compile Polars from source.This can be done by going through the following steps in sequence:Install the latestRust compilerInstallmaturin:pip install maturincd py-polarsand choose one of the following:make build-release, fastest binary, very long compile timesmake build-opt, fast binary with debug symbols, long compile timesmake build-debug-opt, medium-speed binary with debug assertions and symbols, medium compile timesmake build, slow binary with debug assertions and symbols, fast compile timesAppend-native(e.g.make build-release-native) to enable further optimizations specific to
your CPU. This produces a non-portable binary/wheel however.Note that the Rust crate implementing the Python bindings is calledpy-polarsto distinguish from the wrapped
Rust cratepolarsitself. However, both the Python package and the Python module are namedpolars, so you
canpip install polarsandimport polars.Use custom Rust function in Python?Extending Polars with UDFs compiled in Rust is easy. We expose PyO3 extensions forDataFrameandSeriesdata structures. See more inhttps://github.com/pola-rs/pyo3-polars.Going big...Do you expect more than 2^32 (~4.2 billion) rows? Compile Polars with thebigidxfeature flag.Or for Python users, installpip install polars-u64-idx.Don't use this unless you hit the row boundary, as the default build of Polars is faster and consumes less memory.LegacyDo you want Polars to run on an old CPU (e.g. dating from before 2011), or on anx86-64build
of Python on Apple Silicon under Rosetta? Installpip install polars-lts-cpu. This version of
Polars is compiled withoutAVXtarget
features.Sponsors