Package: glide
Glide: Easy ETLUpdate 2022-08-12Given my lack of bandwidth for this project and the presence of a number of great alternatives popping up in recent years, I currently recommend against using Glide in long term / production scenarios.Check outpypelnfor an alternative that is more
focused on flow & parallelization features that are useful in Glide.IntroductionGlide is an easy-to-use data pipelining tool inspired byConsecutionandApache Storm Topologies.Like those libraries,Glide is:A simple, reusable approach to building robust ETL pipelinesA system for wiring together processing nodes to form a directed acyclic graph (DAG)Glide also has:An expanding suite of built-in nodes and pipelines that extract, transform, and load data from/to any combination of:SQL databases (SQLite, DBAPI, and SQLAlchemy support)Local or remote files (CSVs, Excel, and raw file support)URLs (JSON endpoints, file downloads, APIs, etc.)HTML TablesEmailsExtensions forPandas,Dask,Celery,Redis Queueand moreA variety of node and DAG parallel/concurrent/distributed processing strategiesA simple decorator to generate a command line interface from a pipeline in ~one line of codeFlexible pipeline templatingGlide is nota task orchestration and/or dependency management tool like
Airflow. Use Glide to define your easily developed/contained/reusable/testable
data processing pipelines and then rely on a tool like Airflow to do what it's
good at, namely scheduling and complex task dependency management.Table of ContentsInstallationPrimerBasic ExamplesCSV Extract, Transform, and LoadSQL Extract and LoadSQL TransactionsURL ExtractionFlow Control ExamplesFiltersIterPushSplitPushSplitByNodeReduceJoinRoutersWindow ProcessingDate WindowsReturn ValuesParallelization & ConcurrencyParallel TransformationParallel Pipelines via ParaGliderParallel BranchingThread ReducersAsyncioUtility ExamplesTemplated Nodes and PipelinesData Integrity ChecksDebuggingProfiling PipelinesComplex PipelinesPlotting Pipeline DAGsCLI GenerationExtensionsPandasDaskCeleryRedis QueueSwifterDocsHow to ContributeInstallation⚠️Warning: This project is in an alpha state and is maintained on an as-needed basis. Please test carefully for production usage and report any issues.$pipinstallglidePrimerYou are encouraged to take a deeper look at thedocs, but the short of it is
the following:1. ANodeis a part of a pipeline which has arunmethod that typically
accepts data from upstream nodes, and pushes data to downstream nodes. For example:classMyNode(Node):defrun(self,data):# Some node-specific code hereself.push(data)2. AGlideris a pipeline ofNodeobjects wired together in a DAG. It
accepts input data in itsconsumemethod. For example:glider=Glider(MyExtractNode("extract")|MyTransformNode("transform")|MyLoadNode("load"))glider.consume(data)If a node'srunmethod has additional parameters, they are populated from
the node'scontext. More info on creating nodes and populating runtime context
can be foundhere.Basic ExamplesThe following examples serve to quickly illustrate some core features and
built-in nodes. There is much moreGlidecan do that is not shown
here. Everything below assumes you have used the following shortcut to import
all necessary node and pipeline classes:fromglideimport*Example: CSV Extract, Transform, and LoadApply a transformation to data from a CSV, use a function to lowercase all
strings, and load into an output CSV:deflower_rows(data):forrowindata:fork,vinrow.items():row[k]=v.lower()iftype(v)==strelsevreturndataglider=Glider(CSVExtract("extract")|Func("transform",func=lower_rows)|CSVLoad("load"))glider.consume(["/path/to/infile.csv"],extract=dict(chunksize=100),load=dict(outfile="/path/to/outfile.csv"),)Example: SQL Extract and LoadRead from one table, write to another:conn=get_my_sqlalchemy_conn()sql="select * from in_table limit 10"glider=Glider(SQLExtract("extract")|SQLLoad("load"),global_state=dict(conn=conn)# conn is automagically passed to any nodes that accept a "conn" argument)glider.consume([sql],load=dict(table="out_table"))Example: SQL TransactionsStart a transaction before writing to a database, rollback on failure:glider=Glider(SQLExtract("extract")|SQLTransaction("tx")|SQLLoad("load",rollback=True),global_state=dict(conn=conn))glider.consume(...)Example: URL ExtractionExtract data from each URL in the list of requests and load to a URL endpoint:glider=Glider(URLExtract("extract")|URLLoad("load"))reqs=["https://jsonplaceholder.typicode.com/todos/1","https://jsonplaceholder.typicode.com/todos/2",]glider.consume(reqs,extract=dict(data_type="json"),load=dict(url="https://jsonplaceholder.typicode.com/todos",data_param="json",headers={"Content-type":"application/json; charset=UTF-8"},),)Flow Control ExamplesExample: FiltersFilter the propagation of data based on the result of a function:defdata_check(node,data):# do some check on data, return True/False to control filteringreturnTrueglider=Glider(MyExtract("extract")|Filter("filter",func=data_check)|MyLoad("load"))Example: IterPushPush each row of an input iterable individually:glider=Glider(CSVExtract("extract",nrows=20)|IterPush("iter")|Print("load"))Example: SplitPushSplit an iterable before pushing:glider=Glider(SplitPush("push",split_count=2)|Print("print"))glider.consume([range(4)])Example: SplitByNodeSplit an iterable evenly among downstream nodes:glider=Glider(SplitByNode("push")|[Print("print1"),Print("print2")])glider.consume([range(4)])Example: ReduceCollect all upstream node data before pushing:glider=Glider(CSVExtract("extract")|Reduce("reduce")|Print("load"))glider.consume(["/path/to/infile1.csv","/path/to/infile2.csv"])This will read both input CSVs and push them in a single iterable to the
downstream nodes. You can also use theflattenoption ofReduceto
flatten the depth of the iterable before pushing (effectively a concat
operation).Example: JoinJoin data on one or more columns before pushing:glider=Glider(Reduce("reduce")|Join("join")|Print("load"))d1=<listofdictsorDataFrame>d2=<listofdictsorDataFrame>glider.consume([d1,d2],join=dict(on="common_key",how="inner"))Example: RoutersRoute data to a particular downstream node using a router function:defparity_router(row):ifint(row["mycolumn"])%2==0:return"even"return"odd"glider=Glider(CSVExtract("extract",nrows=20)|IterPush("iter")|[parity_zip_router,Print("even"),Print("odd")])glider.consume(...)This will push rows with evenmycolumnvalues to the "even"Printnode,
and rows with oddmycolumnvalues to the "odd"Printnode.Example: Window ProcessingPush a sliding window of the data:glider=Glider(CSVExtract("extract",nrows=5)|WindowPush("window",size=3)|MyWindowCalcNode("calc"))Example: Date WindowsGenerate a set of datetime windows and push them downstream:importdatetimetoday=datetime.date.today()glider=Glider(DateTimeWindowPush("windows")|PrettyPrint("print"))glider.consume(windows=dict(start_date=today-datetime.timedelta(days=3),end_date=today,num_windows=2))Or useDateWindowPushfor date objects. Note that the data arg toconsumecan be ignored because the top node (DateTimeWindowPush) is a subclass ofNoInputNodewhich takes no input data and generates data to push
on its own.Example: Return ValuesBy defaultconsumedoes not return any values and assumes you will be
outputting your results to one or more endpoints in your terminating
nodes (files, databases, etc.). TheReturnnode will collect the data
from its parent node(s) and set it as a return value forconsume.glider=Glider(CSVExtract("extract")|MyTransformer("transform")|Return("return"))data=glider.consume(...)Parallelization & ConcurrencyExample: Parallel TransformationCall a function in parallel processes on equal splits of data from a CSV:glider=Glider(CSVExtract("extract")|ProcessPoolSubmit("transform",push_type=PushTypes.Result)|CSVLoad("load"))glider.consume(["infile.csv"],transform=dict(func=lower_rows),load=dict(outfile="outfile.csv"),)We passedpush_type=PushTypes.Resultto forceProcessPoolSubmitto fetch
and combine the asynchronous results before pushing to the downstream
node. The default is to just pass the asynchronous task/futures objects
forward, so the following would be equivalent:glider=Glider(CSVExtract("extract")|ProcessPoolSubmit("transform")|FuturesReduce("reduce")|Flatten("flatten")|CSVLoad("load"))TheFuturesReducenode waits for the results from each futures object, and
thenFlattenwill combine each subresult back together into a single result
to be loaded in the finalCSVLoadnode.Example: Parallel Pipelines via ParaGliderCompletely parallelize a pipeline using aParaGlider(who said ETL isn't
fun?!?). Split processing of the inputs (two files in this case) over the
pool, with each process running the entire pipeline on part of the consumed
data:glider=ProcessPoolParaGlider(CSVExtract('extract')|Print('load'))glider.consume(["/path/to/infile1.csv","/path/to/infile2.csv"],extract=dict(nrows=50))Example: Parallel BranchingBranch into parallel execution in the middle of the DAG utilizing a parallel
push node:glider=Glider(CSVExtract("extract",nrows=60)|ProcessPoolPush("push",split=True)|[Print("load1"),Print("load2"),Print("load3")])glider.consume(["/path/to/infile.csv"])The above example will extract 60 rows from a CSV and then push equal slices
to the logging nodes in parallel processes. Usingsplit=False(default)
would have passed the entire 60 rows to each logging node in parallel
processes.Once you branch off into processes with a parallel push node there is no way
to reduce/join the pipeline back into the original process and resume
single-process operation. The entire remainder of the pipeline is executed in
each subprocess. However, that is possible with threads as shown in the next
example.Example: Thread Reducersglider=Glider(CSVExtract("extract",nrows=60)|ThreadPoolPush("push",split=True)|[Print("load1"),Print("load2"),Print("load3")]|ThreadReduce("reduce")|Print("loadall"))glider.consume(["/path/to/infile.csv"])The above code will split the data and push to the first 3 logging nodes in
multiple threads. TheThreadReducenode won't push until all of the previous
nodes have finished, and then the final logging node will print all of the
results.Example: AsyncioLimited, experimental support is also available for concurrency viaasyncioin Python >= 3.7:importasyncioasyncdefasync_sleep(data):# Dummy example. Await some real async work in here.awaitasyncio.sleep(0.5)returndataglider=Glider(CSVExtract("extract",nrows=5)|AsyncIOSubmit("transform",push_type=PushTypes.Result)|Print("load"))glider.consume(["/path/to/infile.csv"],transform=dict(func=async_sleep))The above example will split the input data into items to be processed on anasyncioevent loop and synchronously wait for the results before pushing.AsyncIOSubmitsupports specifying asplit_countas well as atimeoutwhen waiting for results. Alternatively, one can pushasynciofutures and
later reduce their results as follows:glider=Glider(CSVExtract("extract",nrows=5)|AsyncIOSubmit("transform",push_type=PushTypes.Async)|AsyncIOFuturesReduce("reduce",flatten=True)|Print("load"))Note that theasyncionodes will create and start an event loop for you if
necessary. It's also perfectly fine to manage the event loop on your own, in
which caseglidewill run tasks on the current thread's event loop.Utility ExamplesExample: Templated Nodes and PipelinesDrop replacement nodes into an existing pipeline. Any node can be replaced by
name:glider=Glider(PlaceholderNode("extract")|CSVLoad("load"))glider["extract"]=CSVExtract("extract")glider.consume(...)Or reuse an existing structure of nodes with aNodeTemplate:nodes=NodeTemplate(CSVExtract("extract")|CSVLoad("load"))glider=Glider(nodes())# Copy of nodes created with each callOr reuse an existing pipeline structure withGliderTemplate:template=GliderTemplate(CSVExtract("extract")|CSVLoad("load"))glider=template()# Copy of pipeline created with each callExample: Data Integrity ChecksYou can use theAssertFuncnode to assert that some condition of the data is
met:glider=Glider(CSVExtract("extract",chunksize=10,nrows=20)|AssertFunc("length_check",func=lambdanode,data:len(data)==10)|CSVLoad("load"))Thefunccallable must accept two parameters, a reference to the node object
and the data passed into that node. Any truthy value returned will pass the
assertion test.Similarly, you can do a sql-based check withAssertSQL, in this case simply
verifying the number of rows inserted:glider=Glider(SQLExtract("extract")|SQLLoad("load")|AssertSQL("sql_check"))sql="select * from in_table limit 10"assert_sql="select (select count(*) as x from out_table) == 10 as assert"glider.consume([sql],extract=dict(conn=in_conn),load=dict(conn=out_conn,table="out_table"),sql_check=dict(conn=out_conn,sql=assert_sql))This looks for a truthy value in theassertcolumn of the result to pass the
assertion. You can also use thedata_checkoption ofAssertSQLto instead
have it do a comparison to the result of some function of the data:glider=...sql="select * from in_table limit 10"assert_sql="select count(*) as assert from out_table"glider.consume([sql],extract=dict(conn=in_conn),load=dict(conn=out_conn,table="out_table",push_data=True),sql_check=dict(conn=out_conn,sql=assert_sql,data_check=lambdanode,data:len(data)))Note that we also addedpush_data=Trueto theSQLLoadnode to have it push
the data instead of a table name.Example: DebuggingTo enable debug logging for Glide change the log level of the "glide" logger:importlogginglogging.getLogger("glide").setLevel(logging.DEBUG)Glide will then print debug information about data passed through your
pipeline.You can also pass_log=Trueto the init method of any node to enable logging
of processed data:glider=Glider(SQLExtract("extract",_log=True)...)Additionaly, you can pass_debug=Trueto the init method of any node to
cause the node to drop into PDB right before callingrun, assuming you
aren't executing the pipeline in a subprocess:glider=Glider(SQLExtract("extract",_debug=True)...)Finally, there are a variety of print nodes you can place in your pipeline for
general logging or debugging, such asPrint,PrettyPrint,LenPrint,ReprPrint, andFormatPrint. See the node documentation for more info.Example: Profiling PipelinesInsert aProfilenode somewhere in your pipeline to get profiler information
for all downstream nodes:glider=Glider(Profile("profile")...)Example: Complex PipelinesIf the hierarchy of nodes you want to form is not achievable with the|operator, you can use theadd_downstreamNodemethod to form more complex
graphs. More info can be foundhere.Example: Plotting Pipeline DAGsIf you have theGraphvizpackage installed, you
can generate a plot of your pipelines by simply doing the following:glider=Glider(...)glider.plot("/path/to/filename.png")CLI GenerationWith Glide you can create parameterized command line scripts from any pipeline
with a simple decorator:glider=Glider(SQLLoad("extract")|SQLExtract("load"))@glider.cli()defmain(glide_data,node_contexts):glider.consume(glide_data,**node_contexts)if__name__=="__main__":main()The script arguments, their types, and whether they are required or not is all
inferred by inspecting therunarguments on the nodes of the pipeline and
prefixing the node name. Please see the full documentationherefor more details.ExtensionsTo install all extensions and dev dependencies:$pipinstallglide[complete]You can also just install Glide plus a specific extension:$pipinstallglide[dask]$pipinstallglide[celery]$pipinstallglide[rq]$pipinstallglide[swifter]To access installed extensions import from theglide.extensionssubmodules
as necessary. Review the documentation and tests for current extensions for
help getting started.PandasThe Pandas extension is actually supported by default with allglideinstalls.
Below is a simple example that extracts from a CSV, lowercases all strings,
and then loads to another CSV using Pandas under the hood:deflower(s):returns.lower()iftype(s)==strelsesglider=Glider(DataFrameCSVExtract("extract")|DataFrameApplyMap("transform",func=lower)|DataFrameCSVLoad("load",index=False,mode="a"))glider.consume(...)There are a variety of other helpful nodes built in, includingToDataFrame,FromDataFrame, nodes to read/write other datasources, and nodes to deal withrollingcalculations. There is also a genericDataFrameMethodnode that
passes through to any DataFrame method.See the extension docsherefor node/pipeline reference information. See the testsherefor some additional examples.Dask - ExperimentalSee the extension docsherefor node/pipeline reference information. See the testsherefor some additional examples.Celery - ExperimentalSee the extension docsherefor node/pipeline reference information. See the testsherefor some additional examples.Redis Queue - ExperimentalSee the extension docsherefor node/pipeline reference information. See the testsherefor some additional examples.Swifter - ExperimentalSee the extension docsherefor node/pipeline reference information. See the testsherefor some additional examples.DocumentationMore thorough documentation can be foundhere.
You can supplement your knowledge by perusing thetestsdirectory
or themodule reference.How to ContributeSee theCONTRIBUTINGguide.