Package: dsml4s8e
Dsml4s8eDsml4s8e is a Python library that extends Dagster to help manage the ML workflow around Jupyter notebooks development.Dsml4s8eaddresses issues:Building of pipelines fromstandalonenotebooksStandardizing a structure of ML/DS pipeline projects to easy share and continuous improve themManaging constantly evolving pipelinesDsml4s8e designed to support the following workflow:Define a project structure and a structure of the pipeline data catalog  for your pipeline by using classStorage Catalog ABCDevelopstandaloneJupyter notebooks corresponding specification requirementsDifine apipeline-- a sequence of notebooks and deloy the pipelien in vary environments(experimental/test/prod)Execute pipelines many times with different configurations in vary environments and on vary infrastructureA job is a set of notebooks arranged into a DAG.
Dsml4s8e simplify jobs definition based on a set of Jupyter notebooks.# dag.pyfromdsml4s8e.define_jobimportdefine_jobfromdagstermillimportlocal_output_notebook_io_managerfromdagsterimportjobfrompathlibimportPath@job(name='simple_pipeline',tags={"cdlc_stage":"dev"},resource_defs={"output_notebook_io_manager":local_output_notebook_io_manager,})defdagstermill_pipeline():module_path=Path(__file__)define_job(root_path=module_path.parent.parent,nbs_sequence=["data_load/nb_0.ipynb","data_load/nb_1.ipynb","data_load/nb_2.ipynb"])In this code block, we usedefine_jobto automate adagstermiljob definition:
As a result, a dagster pipeline will be built from standalone notebooks:Dsml4s8e proper for build a cloud agnostic DSML platform.You can play with a demo pipelineskeleton project.Installation of local dev container# Create a work directory to clone a git repository. The work directory will be mounted to the containermkdirwork#Step into the work directorycdwork# Clone repositorygitclonehttps://github.com/dsml4/dsml4s8e.git# Step into a derictory with a Dockerfilecddsml4s8e/images/dev# Build a imagedockerbuild-tdsml4s8e.# Go back into the work directory to a correct using pwd command inside the next docker run instructioncd../../../# Create and run a container staying in the work directory.dockerrun--rm--namemy_dag-p3000:3000-p8888:8888-v$(pwd):/home/jovyan/work-eDAGSTER_HOME=/home/jovyan/work/daghomedsml4s8ebashsetup_pipeline.shOpen JupyterLab in a browser:http://localhost:8888/labOpen Dagster in a browser:http://localhost:3000/The standalone notebook specificationA standalone notebook is a main building block in our pipelines development flow.To transform a standalone notebook to  Dagster Op(a pipeline step) we need to add4specific cells to the notebook. Next, we will discuss what concerns are addressed each of the cells and what library classes are responsible for each one.Cell 1: Op parameters defenitionIn a cell with the tagop_parametersdefines parameters which will be transformed and passed todefine_dagstermill_opas arguments on a job definition stage.
On the Dagster job definition stage this cell will be called and parametrs needed to define Op will be passed from the cell 'op' variabel to the functiondefine_dagstermill_opfrom the dagstermill library. Then, these parameters will be avalible in Dagster Launchpad to edit run configuration in the launge stage.A definition of a Dagster op in a standalone notebook in JupyterLab:Configure op in Dagster Launchpad:Cell 2: Op context initializationDefine variabel with 'contex' name in a cell wiht the tagparametrs.A methodget_contextpasses default values from config_schema in cellop_parameterstocontextvariable.context=op.get_context()Now, if a notebook is executed by Dagster as an Op then it is replaced the parameters cell with the injected-parameters cell. Thus, the variable context can be used in the notebook body when the notebook  is executed in standalone mode and when the notebook is executed as Dagster Op.To be clear, let's look at one of the notebooks executed by Dagster:Notice that the injected-parameters cell in your output notebook defines a variable called context.Cell 3: data catalog initializationA data catalog is a set of paths. A catalog path is a unique name of a data entity in pipelines namespaces. Catalog paths are used for linking ops into pipeline.op=NbOp(config_schema={"b":Field(int,description='this is the paramentr description',default_value=20)},ins={'simple_pipeline.data_load.nb_0.data0':'path_nb_0_data0','simple_pipeline.data_load.nb_1.data1':'path_nb_1_data1','simple_pipeline.data_load.nb_1.data3':'path_nb_1_data3',},outs=['data2'])In this code block, we use theinsargumet to link outputs produced by nb_0 and nb_1 with inputs of nb_2.
The keys if theinsdict are catalog paths. And the values are names of paths variables which are used in a notebook.
Catalog paths and paths variables names are generated automatically by theoutslist, the classNbOpis responsible for generating.A name of a path variable must be unique for each notebook namespace where the variable is used, thus a name of path variable could be shorter than the corresponding catalog path.The methodget_catalog()use 'DagsterStorageCatalog' derived fromStorageCatalogABCto create paths in a file system (storage) catalog. Creating of file system paths for each the data entity name inoutslist must be implemented in the methodget_outs_data_paths.classStorageCatalogABC(ABC):@abstractmethoddef__init__(self,runid):self.runid=runid@abstractmethoddefis_valid(self)->bool:...@abstractmethoddefget_outs_data_paths(self,catalog:DataCatalogPaths)->Dict[str,str]:"""A map of catalog paths to storage paths"""...A catalog structure in file system is fully customizable usingStorageCatalogABC.Cell 4: passing notebook outputs to next pipeline stepsIn the last cell we inform next steps of the pipeline where data produced by the current notebook is stored. Now we can link this notebook(step) with the next notebooks representing pipeline steps. To do this we need to declare this notebook outputs as inputs for the next netbooks.op.pass_outs_to_next_step()Format of output message returning bypass_outs_to_next_step()allow as to copy strings from the cell output and paste them toNbOpdeclarations in the cellsop_parametrsof the pipeline next steps notebooks.
Look at the illustration below: