Package: qpd
Query Pandas-like Dataframes Using SQLQPD let you run the same SQL (SELECTfor now) statements on different computing frameworks with pandas-like interfaces.
Currently, it supportPandas,DaskandRay(viaModinon Ray).QPD directly translates SQL into pandas-like operations to run on the backend computing frameworks, so it can be significantly
faster than some other approaches, for example, to dump pandas dataframes into SQLite, run SQL and convert the result back into
a pandas dataframe. However, the top priorities of QPD arecorrectnessandconsistency. It ensures the results of
implemented SQL features following SQL convention, and it ensures consistent behavior regardless of backend computing frameworks.
A typical case isgroupby().agg(). In pandas or pandas like frameworks, if any of the group keys is null, the default
behavior is to drop that group, however, in SQL they are not dropped. QPD follows the SQL way.QPD syntax is a subset of the intersection ofSpark SQLandSQLite.
The correctness and consistency are extensively tested against SQLite. Practically, Spark SQL and SQLite are highly consistent
on both syntax and behavior. So, in other words, QPD enables you to run common SQLs and get the same result on Pandas, SQLite, Spark, Dask,
Ray and other backends that QPD will support in the future.SQL is one of the most important data processing languages. It is veryscale agnostic, and one of the major goals of the Fugue project
is to enrich SQL and make SQLplatform agnostic. QPD, as a subproject of Fugue, focuses on running SQL on pandas-like frameworks, it is
an essential component to achieve the ultimate goal.InstallationQPD can be installed from PyPI:pipinstallqpd# install qpd + pandasIf you want to use Ray or Dask as the backend, you will need to install QPD with one of the targets:pipinstallqpd[dask]# install qpd + dask[dataframe]pipinstallqpd[ray]# install qpd + raypipinstallqpd[all]# install all dependencies aboveUsing QPDOn Pandasfromqpd_pandasimportrun_sql_on_pandasimportpandasaspddf=pd.DataFrame([[0,1],[2,3],[0,5]],columns=["a","b"])res=run_sql_on_pandas("SELECT a, SUM(b) AS b, COUNT(*) AS c FROM df GROUP BY a",{"df":df})print(res)On DaskPlease readthisto learn the best
practice for initializing dask.fromqpd_daskimportrun_sql_on_daskimportdask.dataframeaspdimportpandasdf=pd.from_pandas(pandas.DataFrame([[0,1],[2,3],[0,5]],columns=["a","b"]))res=run_sql_on_dask("SELECT a, SUM(b) AS b, COUNT(*) AS c FROM df GROUP BY a",{"df":df})print(res.compute())# dask dataframe is lazy, you need to call computeOn RayPlease readthisto learn the best
practice for initializing ray. And readthisfor initializing modin + ray.Please don't use dask as modin backend if you want to use QPD, it's not testedimportrayray.init()fromqpd_rayimportrun_sql_on_rayimportmodin.pandasaspddf=pd.DataFrame([[0,1],[2,3],[0,5]],columns=["a","b"])res=run_sql_on_ray("SELECT a, SUM(b) AS b, COUNT(*) AS c FROM df GROUP BY a",{"df":df})print(res)Ignoring Case in SQLBy default, QPD requires users to use upper cased keywords, otherwise syntax errors will be raised.
However if you really don't like this behavior, you can turn it off, the parameter isignore_case,
here is an example:fromqpd_pandasimportrun_sql_on_pandasimportpandasaspddf=pd.DataFrame([[0,1],[2,3],[0,5]],columns=["a","b"])res=run_sql_on_pandas("select a, sum(b) as b, count(*) as c from df group by a",{"df":df},ignore_case=True)print(res)Things to clarifyQPD on Spark (Koalas)?No, that will not happen. QPD is using Spark SQLsyntax file.
Spark SQL is highly optimized. If we create a Koalas backend, correctness and consistency can
be guaranteed, but there will be no performance advantage. So for Spark, please use Spark SQL.
If you use Fugue SQL on Spark backend, it will also directly use Spark to run the SQL statements.
We don't see the value to make QPD run on Spark.Update History0.4.4: Remove triad version constraint0.4.3: Fix packing issue0.4.2: Refactor to use the latest triad, fix packaging issues0.4.1: Make Pandas 2 compatible0.4.0: Support arbitrary column name0.2.6: Update pandas indexer import0.2.5: Update antlr to 4.90.2.4: Fix a bug: set operations will alter the input dataframe to add columns0.2.3: Refactor and extract out PandasLikeUtils class0.2.2: Accept constant select withoutFROM,SELECT 1 AS a, 'b' AS b<= 0.2.1: Pandas, Dask, Ray SQL support