Package: pydags
pydagspydagsis a Python package to facilitate the creation and running of
lightweight DAG-based workloads locally. Whereas technologies like Airflow,
Kubeflow, and Luigi are more heavyweight, enterprise-level workflow managers,pydagsis instead an extensible, simple, lightweight alternative tailored
for local development and execution. There are no dependencies on Docker,
Kubernetes, or any other such technologies, and it is use-case agnostic (unlike
Kubeflow/Luigi).TerminologyStageAStageis synonymous to a node in a DAG.PipelineAPipelineis synonymous to a DAG, and is comprised of 1 or moreStagesin
a DAG-like structure.Get StartedExternal Dependenciespydagsrequires multiple non-Python dependencies in order to function properly.
These include Redis (for internal operation) and GraphViz (for visualization). To install these, run the following command in terminal:sudoapt-getinstallredisgraphvizInstalling redis with this command on most *nix systems will result in the
Redis server starting automatically. You can verify this by running theredis-servercommand, which should result in anAddress already in usemessage, or similar.Python DependenciesThe requirements for the SDK are defined in therequirements.txtfile, and
can be installed with the following command:pipinstall-rrequirements.txtInstall pydagsCurrently, pydags needs to be installed from source. Simply run the following command:python3setup.pyinstallYou may requiresudopermissions to run this command. If you do not havesudopermissions on your system, then you can point to a different
installation directory using the--home /path/to/folder/flag and point it
to a folder you have write permissions to. Ensure that thebuild/libsdirectory  that gets created is in yourPYTHONPATHenvironment variable.Example UsageExpressing a DAGBelow is a simple example that aims to simply demonstrate how to specify DAGs
inpydags. In this case, each stage of the pipeline is a Python function
decorated with the@stagedecorator.frompydags.pipelineimportPipelinefrompydags.stageimportstage@stagedefstage_1():print('Running stage 1')@stagedefstage_2():print('Running stage 2')@stagedefstage_3():print('Running stage 3')@stagedefstage_4():print('Running stage 4')@stagedefstage_5():print('Running stage 5')@stagedefstage_6():print('Running stage 6')defbuild_pipeline():stage1=stage_1()stage2=stage_2()stage3=stage_3().after(stage2)stage4=stage_4().after(stage2)stage5=stage_5().after(stage1)stage6=stage_6().after(stage3).after(stage4).after(stage5)pipeline=Pipeline()pipeline.add_stages([stage1,stage2,stage3,stage4,stage5,stage6])returnpipelinepipeline=build_pipeline()pipeline.visualize()pipeline.start()Stages of the pipeline that can be run in parallel (in the above case, stages 1
and 2, and stages 3, 4, 5) will only be run in parallel is you set thenum_coresargument of the.start()method to a positive integer
(representing the number of cores to distribute computation across). For
example, if you want to parallelize the execution of such nodes that can be run
in parallel, then simply replacepipeline.start()withpipeline.start(num_cores=8)(to use 8 cores).A Simple ML PipelineBelow is an example of a simple ML pipeline consisting of 3 stages: 1) data
download, 2) preprocessing, 3) model training. All 3 stages are subclasses ofRedisStage, and thus inherit the functionality to read from and write to a
Redis server. The data is thus passed between the stages using Redis. You may
subclass DiskCacheStage, or implement your own cache/storage backend, if Redis
is not suited to your use case.Please note that you will need to install the additionalscikit-learndependency to run this example.importredisfromsklearn.datasetsimportload_winefromsklearn.ensembleimportRandomForestClassifierfromsklearn.model_selectionimporttrain_test_splitfromsklearn.preprocessingimportMinMaxScalerfrompydags.pipelineimportPipelinefrompydags.stageimportRedisStageclassDataIngestor(RedisStage):@staticmethoddefdownload_data():data=load_wine()features=data['data']targets=data['target']returnfeatures,targetsdefrun(self,*args,**kwargs):features,targets=self.download_data()self.write('features',features)self.write('targets',targets)classDataPreprocessor(RedisStage):@staticmethoddefnormalize(features):returnMinMaxScaler().fit_transform(features)@staticmethoddefsplit(features,targets):returntrain_test_split(features,targets,test_size=0.2)defrun(self,*args,**kwargs):features=self.read('features')targets=self.read('targets')xtr,xte,ytr,yte=self.split(features,targets)xtr=self.normalize(xtr)xte=self.normalize(xte)data={'xtr':xtr,'xte':xte,'ytr':ytr,'yte':yte}self.write('preprocessed_data',data)classModelTrainer(RedisStage):def__init__(self,*args,**kwargs):super(ModelTrainer,self).__init__(*args,**kwargs)self.model=Nonedeftrain_model(self,xtr,ytr):self.model=RandomForestClassifier().fit(xtr,ytr)deftest_model(self,xte,yte):acc=self.model.score(xte,yte)returnaccdefrun(self,*args,**kwargs):preprocessed_data=self.read('preprocessed_data')xtr=preprocessed_data['xtr']xte=preprocessed_data['xte']ytr=preprocessed_data['ytr']yte=preprocessed_data['yte']self.train_model(xtr,ytr)acc=self.test_model(xte,yte)print('Accuracy:',acc)defbuild_pipeline():redis_instance=redis.Redis(host='localhost',port=6379,db=0)data_ingestor=DataIngestor(redis_instance=redis_instance)data_preprocessor=DataPreprocessor(redis_instance=redis_instance).after(data_ingestor)model_trainer=ModelTrainer(redis_instance=redis_instance).after(data_preprocessor)pipeline=Pipeline()pipeline.add_stages([data_ingestor,data_preprocessor,model_trainer])returnpipelinep=build_pipeline()p.visualize()p.start()