Package: grizzly-sql
GrizzlyGrizzly is a transpiler from a Python-API to SQL to move computations from the client into a database system.Grizzly implements its ownDataFramestructure that tracks operations, like projection, filter, joins, ...
Only when the result of the sequence of operations is needed, a SQL string is produced, resembling all those operations, and sent to a DBMS.
This way, you don't have to care about Out-of-Memory problems, un-optimized queries, and high CPU load.PublicationsWe presented the idea as well as key concepts at several conferences:Stefan Hagedorn:When sweet and cute isn't enough anymore: Solving scalability issues in Python Pandas with Grizzly.,CIDR 2020Stefan Hagedorn, Steffen Kläbe, Kai-Uwe Sattler:Putting Pandas in a Box,CIDR 2021Presentation on YoutubeSteffen Kläbe, Stefan Hagedorn:When Bears get Machine Support: Applying Machine Learning Models to Scalable DataFrames with Grizzly,BTW 2021Stefan Hagedorn, Steffen Kläbe, Kai-Uwe Sattler:Conquering a Panda’s weaker self - Fighting laziness with laziness,EDBT 2021, Demo PaperPresentation on YoutubeSteffen Kläbe, Robert DeSantis, Stefan Hagedorn, Kai-Uwe Sattler:Accelerating Python UDFs in Vectorized Query Execution,CIDR 2022Presentation on YoutubeInstallationGrizzly is available on PyPi:https://pypi.org/project/grizzly-sqlpip3install--usergrizzly-sqlDependenciesGrizzly usesPython 3SQLite3(currently for tests only)BeautifulTablefor pretty outputPyYAMLfor support of vendor-specific query templatesantlr4-python3-runtime 4.9.3for compiling Python UDFs to prozedual sqlGetting startedImportAs with any Python module, just import itimportgrizzlyConnectionConnect to your database using an appropriate connection string. In order to load the shipped test database containing events from theGDELTproject:importsqlite3con=sqlite3.connect("grizzly.db")Grizzly uses different classes for code generation and executing the produced query.
Currently, Grizzly includes a SQL code generator and execution wrapper for relational DBMS (more will follow).
In order to activate them, set:fromgrizzly.relationaldbexecutorimportRelationalExecutorfromgrizzly.sqlgeneratorimportSQLGeneratorgrizzly.use(RelationalExecutor(con,SQLGenerator("sqlite")))TheRelationalExecutorconstructor has a parameter for the code generator to use. By default this is agrizzly.sqlgenerator.SQLGenerator, but can be set to some own implementation.The parameter toSQLGeneratordefines the SQL dialect of the underlying database system. We store vendor-specific code in a configuration filegrizzly.yml. The dialect is only needed forlimitoperation which some SQL engines implement asLIMITwhereas others haveTOP. Also UDFs (see below) require system-specific code.Now, reference the table(s) you want to work with:df=grizzly.read_table("events")Here,dfis just a reference, it contains no data from your table.
To show its complete contents, use theshowmethod:df.show(pretty=True)This will print the table's content on the screen. Alternatively, you can convert the dataframe into a string usingstr(df).In order to collect the result of a query/program into a local list, usedf.collect(includeHeader=True)Filter & ProjectionOperations are similar to Pandas:df[df["globaleventid"]==470747760]# filterdf=df[["actor1name","actor2name"]]#projectionA column can also be referenced using the dot notation, e.g.df.actor1name.JoinsADataFramecan be joined with anotherDataFrame:df1=grizzly.read_table("t1")df2=grizzly.read_table("t2")joined=df1.join(df2,on=["actor1name","actor2name"],how="inner",comp='=')In theonparameter, you specify the join columns. The first one is for the left input (df1), the second one for the right input (df2).
Thehowparameter is used to select the join type:inner,left outer, etc. This value is directly placed into the generated query, and thus depends on
the dialect of the underlying DBMS. An additionalcompparameter lets you choose the comparison operator.You sometimes want to join on multiple columns with different comparisons. For this, in Grizzly you define the expression as if it was for filters:df1=grizzly.read_table("t1")df2=grizzly.read_table("t2")j=df1.join(df2,on=(df1.actor1name==df2.actor2name)|(df1["actor1countrycode"]<=df2["actor2countrycode"]),how="left outer")This results in the following SQL code:SELECT*FROM(SELECT*FROMt1_t0)_t1leftouterJOIN(SELECT*FROMt2_t2)_t3ON_t1.actor1name=_t3.actor2nameor_t1.actor1countrycode<=_t3.actor2countrycodeGrouping & AggregationYou can also group the data on multiple columns and compute an aggregate over the groups usingagg:fromgrizzly.aggregatesimportAggregateTypedf=grizzly.read_table("events")g=df.groupby(["year","actor1name"])a=g.agg(col="actor2name",aggType=AggregateTyoe.COUNT)Here,arepresents a DataFrame with three columns:year,monthyearand thecountvalue. In the above example,a.generateQuery()will giveSELECT_t0.year,_t0.actor1name,count(_t0.actor2name)FROMevents_t0GROUPBY_t0.year,_t0.actor1nameIf no aggregation function and projection is used, only the grouping columns are selected upon query generation.You can apply aggregation functions on non-groupedDataFrames of course. In this case the aggregates will be computed for the whole content. For example,g.count()immediately runs the following query and returns the scalar valueSELECTcount(*)FROM(SELECT_t1.year,_t1.actor1nameFROM(SELECT*FROMevents_t0)_t1GROUPBY_t1.year,_t1.actor1name)_t2Adf.count()(i.e. before the grouping) for the above piece of code will return the single scalar value with the number of records indf(22225).
The query executed for this is:SELECTcount(*)FROMeventsGrizzly supports predefined aggregations, defined in theAggregateTypeenum:MIN,MAX,MEAN,SUM,COUNT.
Other functions can be applied by passing the name of the functions as a string instead of theENUMvalue.User Defined Functions & Computed ColumnsGrizzly allows to apply almost any function defined in Python on your data. Currently, we support scala functions only.defmyfunc(a:int)->str:returna+"_grizzly"df=grizzly.read_table("events")# load tabledf=df[df.globaleventid==467268277]# filter itApply function with Python code on dbms (supported by PostgreSQL, Actian Vector and MonetDB)df["newid"]=df["globaleventid"].map(myfunc)# apply myfuncApply translated function with procedural SQL code (Oracle and PostgreSQL supported)df["newid"]=df["globaleventid"].map(myfunc,lang='sql',fallback=True)# apply myfuncThelangparameter defines whether the function is executed with Python code or the code is translated with the integratedudfcompilermodule to a procedural language. Thefallbackparameter allows to apply the function with Python code or locally to aPandas DataFrameif compilation errors occur.In the example above, the functionmyfuncis applied to all entries in theglobaleventidcolumn and the result is stored in a new columnnewid.This way new columns can be added to the result. The value of a computed column can be any expression.df["newcol"]=df.theyear+df.monthyearApply Machine Learning ModelsUsing the UDF mechanism described above, we enable users to easily apply their pre-trained models to their data inside the DB.For ONNX models, users only need to specify the path to the model file (must be availble for the database engine) as well as two conversion functions:first functions converts the tuple into the format expected by the modelthe second function converts the output of the model into a format the DB (and user) can handle.TheONNX model zooprovides a rich set of models with the according conversion functions.definput_to_model(a:str):...defmodel_to_output(a)->str:...df=grizzly.read_table('tab')# load table# apply model to every value in column 'col'# using provided input and output conversion functions# store model output in computed column 'classification'df['classification']=df['col'].apply_model("/path/to/model",input_to_model,model_to_output)# group by e.g. predicted classesdf=df.groupby(['classification']).count()df.show()SQLYou can inspect the produced query string (in this case SQL) withgenerateQuery():print(df.generateQuery())Supported operationsfilter/selectionprojectionjoingroup byaggregation functions: min, max, mean (avg), count, sumuser defined functionsapply TensorFlow, PyTorch, ONNX modelsLimitationsOur DataFrame implementation is not yet fully compatibile with Pandas, but we are working on it.Grizzly is under active development and things might change.There are certainly some bugs. Probably with complex queries.VisionGrizzly is a research project. We aim at bringing data-intensive operations back into the database system. Our plan is to extend Grizzly in the following ways - some of them are inspired by our other projects:Support for heterogeneous data sources:Combine data from different sources (relational DB, file, HDFS, NoSQL) in one program/query (i.e. Polystores, federated query processing)automatically import external data when neccessaryAdd spatial operationsStream processing operationsCode generationProcude native code from the Python API