Package: nb-workflows
:rocket: NB Workflows:books: DescriptionNB Workflows empowers different data roles to put notebooks into production, simplifying the time required to do so. It enables people to go from a data exploration instance to an entirely project deployed in production, using the same notebooks files made by a data scientist, analyst or whatever role working with data in an iterative way.NB Workflows is a library and a service that allows you to run parametrized notebooks in a distributed way.A Notebook could be launched remotly on demand, or could be scheduled by intervals or using cron syntax.Internally it usesSanicas web server,papermillas notebook executor, anRQfor task distributions and coordination.:tada: Demo :tada::floppy_disk:Example project:telescope: PhilosophyNB Workflows it insn't a complete MLOps solution and it will never be.
We try hard to simply and expose the right APIs to the user for the part of scheduling notebooks with reproducibility in mind.We also try to give the user the same freedom that lego tiles can give, but we are opinated in some aspects: we understand the process of writing code for data science or/and data analytics, as a engineer problem to be solvedWith this point of view, then:Git is neccesary :wink:Docker is necessary for environment reproducibility.Although you can push not versioned code,  versioning is almost enforced, and is always a good practice in software developmentThe idea comes from aNetflix postwhich suggest using notebooks like an interface or a some kind of DSL to orchestrate different workloads like Spark and so on. But it also could be used to run entire process: like training a model, crawlings sites, performing etls, and so on.The benefits of this approach is that notebooks runned could be stored and inspected for good or for bad. If something fails, is easy to run in a classical way: cell by cell.The last point to clarify and it could challange the common sense or the way that we are used to use Jupyter's Notebooks, is that each notebook is more like a function definition with inputs and outputs, so a notebook potentially could be used for different purposes, hence the name ofworkflow, and indeed this idea is common in the data space. Then a workflow will be a notebook with params defined to be used anytime that a user wants, altering or not the parameters sent.:nut_and_bolt: FeaturesDefine a notebook like a function, and execute it on demand or scheduled itAutomatic Dockerfile generation. A project should share a unique environment but could use different versions of the same environmentExecution History, Notifications to Slack or Discord.InstallationServerDocker-composeThe project provides adocker-compose.yamlfile as en example.:construction: Note :construction:BecauseNB Workflowswill spawn docker instance for each workload, the installation inside docker containers could be tricky.
The most difficult part is the configuration of the worker that needs access to thedocker socket.ADockerfileis provided for customization of uid and guid witch should match with the local environment. A second alternative is expose the docker daemon through HTTP, if that is the case aDOCKER_HOSTenv could be used, seedocker client sdkgit clone https://github.com/nuxion/nb_workflows
cd nb_workflowsThe next step is intializing the database and creating a user (please review the script first):docker-compose postgres up -d 
./scripts/initdb_docker.shNow you can start everything else:docker-compose up -dWithout dockerpip install nb-workflows[server]==0.6.0first terminal:export NB_SERVER=True
nb manager db upgrade
nb manager users create
nb web --apps workflows,history,projects,events,runtimessecond terminal:nb rqworker -w 1 -q control,mch.defaultBefore all that, redis postgresql and thenginx in webdav modeshould be configuratedClientClient:pip install nb-workflows==0.6.0
nb startporject .:earth_americas: RoadmapSeeRoadmapdraft:post_office: Architecture:bookmark_tabs: References & inspirationsNotebook Innovation - NetflixTensorflow metastoreMaintainable and collaborative pipelines