Package: litepipeline
LitePipelineA lightweight, scalable, distributed pipeline system, based on Python3, tornado, venv-pack, pyinstaller.All code based on Python3, do not use Python2!It still under development, so, maybe have some bugs or not stable enough!You can use LitePipeline with LiteDFS, a distributed file system, based on Python3, tornado, inspired by HDFS.See LiteDFS athttps://github.com/fiefdx/LiteDFSSee more details athttps://github.com/fiefdx/LitePipelineInstall# install from pip$pip3install-Ulitepipeline# or install from source code$cd./litepipeline
$python3./setup.pyinstall# this will install 5 commands: liteconfig, litemanager, litenode, litepipeline, liteviewer# liteconfig: to generate manager's or node's configuration file# litemanager: to start LitePipeline manager# litenode: to start LitePipeline node# litepipeline: command line tool to communicate with LitePipeline cluster# liteviewer: the web UI service, for communicate with LitePipeline cluster with web browser.runRun Manager# create manager's data directory$mkdir./manager_data# generate manager's configuration file$cd./manager_data# this will generate a configuration.yml file under ./manager_data$litemanager-g./# run manager$litemanager-c./configuration.yml# test$curllocalhost:8000# return this message{"message":"LitePipeline manager service"}Run Node# create node's data directory$mkdir./node_data# generate node's configuration file$cd./node_data# this will generate a configuration.yml file under ./node_data$litenode-g./# run node# after start node, node will register to manager, and get a unique node id$litenode-c./configuration.yml# test$curllocalhost:8001# return this message{"message":"LitePipeline node service"}Run Viewer# create viewer's data directory$mkdir./viewer_data# generate viewer's configuration file$cd./viewer_data# this will generate a configuration.yml file and other scripts under ./viewer_data$liteviewer-g./# run viewer$liteviewer-c./configuration.yml# test# use web browser open: http://localhost:8088Use Web Browser Communicate With LitePipeline ClusterUse web browser open the liteviewer's host & port in it's configuration fileUse Command Line Communicate With LitePipeline Cluster$litepipelinelocalhost:8000clusterinfo# | node_id                              | http_host | http_port | action_slots | app_path                                                 | data_path1|32313239-e7ee-4f90-8c05-4e08fb48be70|127.0.0.1|8001|2|/usr/local/lib/python3.7/dist-packages/litepipeline/node|/home/pi/Develop/litenode/data# use -h/--help parameter to see help message$litepipeline--help
usage:litepipeline[-h][-WCOLUMN_WIDTH][-v]address{app,app_history,task,cluster,workspace,workflow,work,schedule,service}...

positionalarguments:addressmanageraddress,host:port{app,app_history,task,cluster,workspace,workflow,work,schedule,service}sub-commandhelpappoperatewithappAPIapp_historyoperatewithapp_historyAPItaskoperatewithtaskAPIclusteroperatewithclusterAPIworkspaceoperatewithworkspaceAPIworkflowoperatewithworkflowAPIworkoperatewithworkAPIscheduleoperatewithscheduleAPIserviceoperatewithserviceAPI

optionalarguments:-h,--helpshowthishelpmessageandexit-WCOLUMN_WIDTH,--column_widthCOLUMN_WIDTHcolumnmaxwidth-v,--versionshowprogram's version number and exit$ litepipeline localhost:8000 app --helpusage: litepipeline address app [-h]{create,delete,update,list,info,download} ...positional arguments:{create,delete,update,list,info,download}sub-command app helpcreate              create applicationdelete              delete applicationupdate              update applicationlist                list applicationsinfo                application'sinfodownloaddownloadapplication

optionalarguments:-h,--helpshowthishelpmessageandexit$litepipelinelocalhost:8000task--help
usage:litepipelineaddresstask[-h]{create,delete,list,info,rerun,recover,stop}...

positionalarguments:{create,delete,list,info,rerun,recover,stop}sub-commandtaskhelpcreatecreatetaskdeletedeletetasklistlisttasksinfotask's inforerun               rerun taskrecover             recover taskstop                stop taskoptional arguments:-h, --help            show this help message and exit$ litepipeline localhost:8000 cluster --helpusage: litepipeline address cluster [-h] {info} ...positional arguments:{info}      sub-command cluster helpinfo      cluster'sinfo

optionalarguments:-h,--helpshowthishelpmessageandexit