Package: pydiverse-pipedag
pydiverse.pipedagA pipeline orchestration library executing tasks within one python session. It takes care of SQL table
(de)materialization, caching and cache invalidation. Blob storage is supported as well for example
for storing model files.This is an early stage version 0.x, however, it is already used in real projects. Please contacthttps://github.com/orgs/pydiverse/teams/code-ownersif you like to become an early adopter. We will
help you getting started and are curious about how it fits your needs.Usagepydiverse.pipedag can either be installed via pypi withpip install pydiverse-pipedagor via conda-forge
withconda install pydiverse-pipedag -c conda-forge.ExampleA flow can look like this (i.e. put this in a file namedrun_pipeline.py):importpandasaspdimportsqlalchemyassafrompydiverse.pipedagimportFlow,Stage,Table,materializefrompydiverse.pipedag.contextimportStageLockContext@materialize(lazy=True)deflazy_task_1():returnsa.select(sa.literal(1).label("x"),sa.literal(2).label("y"),)@materialize(lazy=True,input_type=sa.Table)deflazy_task_2(input1:sa.Table,input2:sa.Table):query=sa.select((input1.c.x*5).label("x5"),input2.c.a,).select_from(input1.outerjoin(input2,input2.c.x==input1.c.x))returnTable(query,name="task_2_out",primary_key=["a"])@materialize(lazy=True,input_type=sa.Table)deflazy_task_3(input1:sa.Table):returnsa.text(f"SELECT * FROM{input1.original.schema}.{input1.name}")@materialize(lazy=True,input_type=sa.Table)deflazy_task_4(input1:sa.Table):returnsa.text(f"SELECT * FROM{input1.original.schema}.{input1.name}")@materialize(nout=2,version="1.0.0")defeager_inputs():dfA=pd.DataFrame({"a":[0,1,2,4],"b":[9,8,7,6],})dfB=pd.DataFrame({"a":[2,1,0,1],"x":[1,1,2,2],})returnTable(dfA,"dfA"),Table(dfB,"dfB_%%")@materialize(version="1.0.0",input_type=pd.DataFrame)defeager_task(tbl1:pd.DataFrame,tbl2:pd.DataFrame):returntbl1.merge(tbl2,on="x")defmain():withFlow()asf:withStage("stage_1"):lazy_1=lazy_task_1()a,b=eager_inputs()withStage("stage_2"):lazy_2=lazy_task_2(lazy_1,b)lazy_3=lazy_task_3(lazy_2)eager=eager_task(lazy_1,b)withStage("stage_3"):lazy_4=lazy_task_4(lazy_2)_=lazy_3,lazy_4,eager# unused terminal output tables# Run flowresult=f.run()assertresult.successful# Run in a different way for testingwithStageLockContext():result=f.run()assertresult.successfulassertresult.get(lazy_1,as_type=pd.DataFrame)["x"][0]==1if__name__=="__main__":main()Create a file calledpipedag.yamlin the same directory:instances:__any__:network_interface:"127.0.0.1"auto_table:-"pandas.DataFrame"-"sqlalchemy.sql.expression.TextClause"-"sqlalchemy.sql.expression.Selectable"fail_fast:trueinstance_id:pipedag_defaulttable_store:class:"pydiverse.pipedag.backend.table.SQLTableStore"args:url:"postgresql://sa:Pydiverse23@127.0.0.1:6543/{instance_id}"create_database_if_not_exists:Trueprint_materialize:trueprint_sql:truelocal_table_cache:store_input:truestore_output:trueuse_stored_input_as_cache:trueclass:"pydiverse.pipedag.backend.table.cache.ParquetTableCache"args:base_path:"/tmp/pipedag/table_cache"blob_store:class:"pydiverse.pipedag.backend.blob.FileBlobStore"args:base_path:"/tmp/pipedag/blobs"lock_manager:class:"pydiverse.pipedag.backend.lock.DatabaseLockManager"orchestration:class:"pydiverse.pipedag.engine.SequentialEngine"If you don't have a postgres database at hand, you can start a postgres database, with the followingdocker-compose.yamlfile:version:"3.9"services:postgres:image:postgresenvironment:POSTGRES_USER:saPOSTGRES_PASSWORD:Pydiverse23ports:-"6543:5432"Rundocker-compose upin the directory of yourdocker-compose.yamland then execute
the flow script as follows with a shell likebashand a python environment that
includespydiverse-pipedag,pandas, andsqlalchemy:poetryrunpythonrun_pipeline.pyFinally, you may connect to your localhost postgres databasepipedag_defaultand
look at tables in schemasstage_1..stage_3.If you don't have a SQL UI at hand, you may usepsqlcommand line tool inside the docker container.
Check out theNAMEScolumn indocker psoutput. If the name of your postgres container isexample_postgres_1, then you can look at output tables like this:dockerexecexample_postgres_1psql--username=sa--dbname=pipedag_default-c'select * from stage_1.dfa;'Or more interactively:dockerexec-t-iexample_postgres_1bash
psql--username=sa--dbname=pipedag_default\dtstage_*.*select*fromstage_2.task_2_out;TroubleshootingInstalling mssql odbc driver for linuxInstalling with
instructionshereworked.
Butodbcinst -jrevealed that it installed the configuration in/etc/unixODBC/*. But conda installed pyodbc brings
its ownodbcinstexecutable and that shows odbc config files are expected in/etc/*. Symlinks were enough to fix the
problem. Trypython -c 'import pyodbc;print(pyodbc.drivers())'and see whether you get more than an empty list.
Furthermore, make sure you use 127.0.0.1 instead of localhost. It seems that /etc/hosts is ignored.