Package: mltrack
mltrack - Track and organize your machine learning projects from the terminalmltrackis a terminal based tool to track and organize machine learning pipelines. It does so by:Expressing ML pipelines as adirected acyclic graph(DAG), andOrganizing your repository in an opinionated folder structure.It is designed to be extremely lightweight, using tools such as git, bash and make that are usually available by default in a standard unix distribution. It is written in pure python, offers a simple command line interface, and integrates easily with existing python codebases.What are other similar tools?There are many, many build / workflow automation tools and data pipelining schedulers available that let one do similar things asmltrack. Apache Airflow, Luigi and Snakemake / Nextflow are notable examples. Seehereandherefor more comprehensive lists.Most of these tools are fairly heavy-duty to install, configure and use, often requiring one to use a special domain specific language to specify the workflow. Many of them are meant to be used for big data processing in a distributed computing environment, or for compiling large codebases in C/C++.mltrackis optimized towards machine learning pipelines that are collaboratively built by a medium-sized team that is trying to experiment and iterate rapidly.The distinguishing feature ofmltrackis that it lets one specify the workflow DAG as a list of plain Python dictionaries. That means the workflow / pipeline itself can be easily shared in a team setting,as well as made available for automatic versioning, testing and documentation. The pipelines can be programatically generated, nested within each other, and reconfigured at runtime.mltrackis extremely lightweight, and doesn't require you to install special runtimes / servers etc. After adopting it, most projects will end up usingmltas a top-level command line interface that runs and documents the project's workflow.mltrackgenerates a folder structure for new projects in a similar way tocookiecutter.Benefits ofmltrackSpecify ML pipelines / workflows as plain Python code.Guaranteed reproducibility, testability and versioning for pipelines / workflows.Usemltas a CLI interface to your workflow. Easily define custom commands with help text to run and document sub-parts of your workflow.Collaborate painlessly: common folder structure and shared knowledge of themltinterface unifies project workflow and defines a common way of doing things.Save time with incremental processing:mltwill only run sub-parts of your workflow that need to be updated.Apply the full power and flexibility of Python - generate and reconfigure workflows / pipelines dynamically based on external data, build arbitrarily large and complex task graphs.Extremely lightweight: uses tools that are available by default in a standard Unix box such as git, bash, make and Python.No need to learn a special domain specific language / protocol or install heavy duty software or servers.Usage and installationmltrackis available on PyPI. The easiest way to install is with pip:pip install --upgrade mltrackTo start with,mlt helpwill print help text:~->mlt help


 mlt - track and organize machine learning workflows from the terminal


FORMAT -

 mlt [command]

mlt commands:
  create:                     creates a new mlt repository
  init:                       initializes existing repository for use with mlt
  help:                       prints help text

 No user defined commands available.

Cannot find mlt repo, or repository info in .mlt might be corrupted. Type 'mlt init' to initialize new mlt repository. Delete existing .mlt folder if one exists.Create a new project repository withmlt create:(base) ubuntu@ip-172-31-32-135:~/mldk_dev$ mlt create
What's your project's name? Yoda ML
Enter repository name [recommended repository name: yoda_ml, press enter to use yoda_ml ]:
Enter author name: Shankar
Enter license type (optional, press enter for no license) [MIT (1), GPL (2), Apache (3)]: 1
Enter AWS profile name (press enter to use 'default' ):
Enter bucket name: shankar-test-ml
repo_dir is : /home/ubuntu/mldk_dev/yoda_ml
Successfully initialized mlt project...A new project repository comes pre-loaded with a basic MNIST classification example.Let's change to the project directory and typemlt helpagain:~/mldk_dev/yoda_ml->mlt help


 mlt - track and organize machine learning workflows from the terminal


FORMAT -

 mlt [command]

mlt commands:
  create:                     creates a new mlt repository
  init:                       initializes existing repository for use with mlt
  help:                       prints help text

 Commands defined in mlt_dependencies.py:

  clean:                      Deletes compiled Python files
  sync_data_to_s3:            Upload data to S3 bucket (default-bucket)
  sync_data_from_s3:          Download data from S3 bucket (default-bucket)
  data:                       Run "mlt data" to download data, preprocess it and generate features
  train_eval_model:           train and evaluate model
  download:                   Downloads an updated version of data/external/mnist.pkl.gz
  rawpreprocess:              Run raw preprocessing
  featurize:                  Generates features from raw data
  learn_features:             learn parameters to prepare for feature generation
  train_model:                Trains and persists model
  evaluate_model:             Evaluate model loss in test setNote that the help text now shows user defined commands. These commands are defined by the user in the project'smlt_dependencies.pyfile.Let's evaluate the model by runningmlt evaluate_model. Before running this command, you can verify that the data directories are empty. In order to evaluate the model, we will first have to download data, preprocess it, generate features and train the model itself.These steps are defined as a dependency graph inmlt_dependencies.py, somltwill execute them in the right order:(base) ubuntu@ip-172-31-32-135:~/mldk_dev/yoda_ml$ mlt evaluate_model
python -m src.data.download_data
Downloaded http://deeplearning.net/data/mnist/mnist.pkl.gz to /home/ubuntu/mldk_dev/yoda_ml/data/external/mnist.pkl.gz
python -m src.data.make_dataset
train_set data: (50000, 784), train_set labels: (50000,)
valid_set data: (10000, 784), valid_set labels: (10000,)
test_set data: (10000, 784), test_set labels: (10000,)
saved raw train, valid and test splits in /home/ubuntu/mldk_dev/yoda_ml/data/raw
python -m src.features.learn_features
fit Standard Scaler and PCA with n_components_ = 329 components, original components = 784
python -m src.features.build_features
python -m src.models.train_model
/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:758: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
starting model training...
finished training model...
train error (log loss) : 0.23456826576340226
validation error (log loss): 0.28539054233742156
python -m src.models.eval_model
model evaluation completed:

 Test set score for model LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=200, multi_class='multinomial',
          n_jobs=None, penalty='l2', random_state=None, solver='lbfgs',
          tol=0.0001, verbose=0, warm_start=False) = 0.31476026852394734


(base) ubuntu@ip-172-31-32-135:~/mldk_dev/yoda_ml$You can see that the full pipeline is run in the output above, and verify that the data directories now have the downloaded, preprocessed and final data / model files.If you runmlt evaluate_modelagain, it will do nothing because the pipeline is up to date:(base) ubuntu@ip-172-31-32-135:~/mldk_dev/yoda_ml$ mlt evaluate_model
mlt: Nothing to be done for 'evaluate_model'.If you runmlt featurizenow,mltwill recognize that nothing needs to be run, because thefeaturizestep was a precursor toevaluate_model, and all files involved are up to date:(base) ubuntu@ip-172-31-32-135:~/mldk_dev/yoda_ml$ mlt featurize
mlt: Nothing to be done for 'featurize'.Now let's edit and savesrc/features/build_features.py(for purposes of this demonstration, you can simply make an inconsequential edit, such as adding new line or space to the end of the file).src/features/build_features.pyis used to learn and build features. Since we have updated it, thefeaturizestep as well as all downstream steps which depend on the feature generation will need to be rerun in order to stay updated.Let's now runmlt rawpreprocess. Since therawpreprocessstep is done prior tofeaturize, it is up to date, andmltwill not run anything:(base) ubuntu@ip-172-31-32-135:~/mldk_dev/yoda_ml$ mlt rawpreprocess
mlt: Nothing to be done for 'rawpreprocess'.Let's runmlt train_model. Since thetrain_modelstep is a successor tofeaturize,mltwill rerun all steps betweenfeaturizeandtrain_model:(base) ubuntu@ip-172-31-32-135:~/mldk_dev/yoda_ml$ mlt train_model
python -m src.features.build_features
python -m src.models.train_model
/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:758: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
starting model training...
finished training model...
train error (log loss) : 0.23456826576340226
validation error (log loss): 0.28539054233742156Let's runmlt train_eval_model. This step trains and evaluates the model. Since we just trained our new model,mltwill skip the training step and proceed straight to model evaluation:(base) ubuntu@ip-172-31-32-135:~/mldk_dev/yoda_ml$ mlt train_eval_model
python -m src.models.eval_model
model evaluation completed:

 Test set score for model LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=200, multi_class='multinomial',
          n_jobs=None, penalty='l2', random_state=None, solver='lbfgs',
          tol=0.0001, verbose=0, warm_start=False) = 0.31476026852394734The steps in the pipeline along with their dependencies are defined inmlt_dependencies.py. To learn how to writemlt_dependencies.pyfor your own project, let's first understand how to organize ML pipelines as a DAG.Core principles: organizing ML pipelines as a DAGThe core design philosophy behindmltrackis organize a machine learning pipeline as adirected acyclic graph(DAG).ML pipelines naturally lend themselves to a DAG representation: one normally starts with a process to gather data, followed by preprocessing it, generating features from it, followed by producing a model file or binary.Each step might depend on the output of previous steps.If you update an intermediate step, all subsequent steps that depend on its outputs will need to be rerun and updated.mltrackwill build the graph and keep track of which outputs are out of date, and only re-run the steps that are necessary - that way, you won't rerun your expensive 2-day training process which doesn't depend on the newest change to your pipeline.Advantages of using plain Python to describe pipelines as DAGsThe DAG representing ML pipelines inmltis a plain Python dictionary. Since it is Python code, it can be versioned, linted and shared like any other piece of code.But perhaps more importantly, it allows one to bring the full power and expressivity of Python to bear on the problem of describing ML pipelines. You can configure the pipeline at dynamically at runtime, or create sub-pipelines within a larger pipeline based on say, user input or external data.Using plain Python also makes the pipeline easy to read and understand between developers working on different parts of a pipeline in a team.In the sample pipeline above,download_data.pydownloads a data source from s3 andpreprocess_data.pygeneratesprocessed_datafrom it. In the next step,feature_1andfeature_2are generated, followed bymodel_initialandmodel_final.If we updatemodel_initial.py,mltrackwill remember to updatemodel_final- at the same time, it willnotrerun the process to generatefeature_1orfeature_2.Walkthrough: describing ML pipelines as a DAG withmlt_dependencies.pyLet us define a simple, 3 step pipeline and describe it as a DAG in the formatmltexpects.Let's create a new directory:~/mldk_dev->mkdir temp_ml
~/mldk_dev->cd temp_ml/
~/mldk_dev/temp_ml->ls
~/mldk_dev/temp_ml->In order to usemltin your project, you must first initialize it withmlt init:~/mldk_dev/temp_ml->mlt init
What's your project's name? temp ml
Successfully initialized mlt project. Define your project dependencies in mlt_dependencies.py to get started.
~/mldk_dev/temp_ml->Let's say that the first step in our pipeline is to generateraw_data_1.txtby running the python scriptgen_raw_1.py. Our Python script simply prints a list of numbers from 1 to 10:gen_raw_1.pynumbers = range(10)
print(",".join(str(num) for num in numbers))We generateraw_data_1.txtlike so:~/mldk_dev/temp_ml->python gen_raw_1.py
0,1,2,3,4,5,6,7,8,9
~/mldk_dev/temp_ml->python gen_raw_1.py > raw_data_1.txt
~/mldk_dev/temp_ml->cat raw_data_1.txt
0,1,2,3,4,5,6,7,8,9This "pipeline" is represented as:The output israw_data_1.txt, and the action to produce this output ispython gen_raw_1.py > raw_data_1.txt. This output only depends on the python scriptgen_raw_1.py.Here is themlt_dependencies.pyfile that encodes these relationships:mlt_dependencies.pymlt_dependency_graph = {
    'graph': [
        {
            'outputs': ['raw_data_1.txt'],
            'inputs': ['gen_raw_1.py'],
            'actions': ['python gen_raw_1.py > raw_data_1.txt'],
            'help': 'generate raw_data_1.txt'
        }
    ]
}As you can see, we added'help': 'generate raw_data_1.txt'. Themltdependency graph is specified as a list of dictionaries. Theinputsare the files that theoutputsdepend on.Adding ahelptag (as we did here) make theoutputsavailable as a user defined command. After you create themlt_dependencies.pyfile, trymlt help:~/mldk_dev/temp_ml->ls
gen_raw_1.py  mlt_dependencies.py  raw_data_1.txt
~/mldk_dev/temp_ml->mlt help


 mlt - track and organize machine learning workflows from the terminal


FORMAT -

 mlt [command]

mlt commands:
  create:                     creates a new mlt repository
  init:                       initializes existing repository for use with mlt
  help:                       prints help text

 Commands defined in mlt_dependencies.py:

  raw_data_1.txt:             generate raw_data_1.txt
~/mldk_dev/temp_ml->For purposes of demonstration, let's deleteraw_data_1.txtand generate it withmlt:~/mldk_dev/temp_ml->rm raw_data_1.txt
~/mldk_dev/temp_ml->ls
__pycache__  gen_raw_1.py  mlt_dependencies.py
~/mldk_dev/temp_ml->mlt raw_data_1.txt
python gen_raw_1.py > raw_data_1.txt
~/mldk_dev/temp_ml->ls
__pycache__  gen_raw_1.py  mlt_dependencies.py  raw_data_1.txt
~/mldk_dev/temp_ml->What happens if you runmlt raw_data_1.txtagain?~/mldk_dev/temp_ml->mlt raw_data_1.txt
mlt: 'raw_data_1.txt' is up to date.mltwill correctly identify that your files are up to date, and that nothing needs to be rerun.Let's add another step to our pipeline. We will write a second Python scriptgen_processed.pythat readsraw_data_1.txtand produces a fileprocessed.txtthat contains the sum of the numbers inraw_data_1.txt:gen_processed.pyfrom pathlib import Path
import os
fh = open('./raw_data_1.txt', 'r')
data = [int(item) for item in fh.readline().split(',')]
output_file = Path(os.getcwd()) / 'processed.txt'
output_file.write_text(str(sum(data))+"\n")~/mldk_dev/temp_ml->ls
__pycache__  gen_processed.py  gen_raw_1.py  mlt_dependencies.py  raw_data_1.txt
~/mldk_dev/temp_ml->python gen_processed.py
~/mldk_dev/temp_ml->cat processed.txt
45Our pipeline now looks like this:Let us add these steps tomlt_dependencies.py:mlt_dependencies.pymlt_dependency_graph = {
    'graph': [
        {
            'outputs': ['raw_data_1.txt'],
            'inputs': ['gen_raw_1.py'],
            'actions': ['python gen_raw_1.py > raw_data_1.txt'],
            'help': 'generate raw_data_1.txt'
        },

        {
            'outputs': ['processed.txt'],
            'inputs': ['gen_processed.py', 'raw_data_1.txt'],
            'actions': ['python gen_processed.py'],
            'help': 'generate processed.txt'
        }
    ]
}Our output fileprocessed.txtdepends on the previous output fileraw_data_1.txtand the source filegen_processed.py. Theactionskeypython gen_processed.pyencodes what you would type at the terminal to produce the output.~/mldk_dev/temp_ml->mlt help


 mlt - track and organize machine learning workflows from the terminal


FORMAT -

 mlt [command]

mlt commands:
  create:                     creates a new mlt repository
  init:                       initializes existing repository for use with mlt
  help:                       prints help text

 Commands defined in mlt_dependencies.py:

  raw_data_1.txt:             generate raw_data_1.txt
  processed.txt:              generate processed.txt
~/mldk_dev/temp_ml->Let's generate the output withmlt:~/mldk_dev/temp_ml->rm processed.txt
~/mldk_dev/temp_ml->mlt processed.txt
python gen_processed.py
~/mldk_dev/temp_ml->cat processed.txt
45
~/mldk_dev/temp_ml->What happens if we say, editgen_processed.py?gen_processed.pyfrom pathlib import Path
import os
fh = open('./raw_data_1.txt', 'r')
data = [int(item) for item in fh.readline().split(',')]
output_file = Path(os.getcwd()) / 'processed.txt'
output_file.write_text(str(sum(data))+"\n")
output_file.write_text("*******\n")   ## --> made changes to gen_processed.pymltwill recognize that the last step in the pipeline needs to be rerun, but there is no need to regenerateraw_data_1.txt:~/mldk_dev/temp_ml->ls
__pycache__  gen_processed.py  gen_raw_1.py  mlt_dependencies.py  processed.txt  raw_data_1.txt
~/mldk_dev/temp_ml->mlt processed.txt
python gen_processed.py
~/mldk_dev/temp_ml->cat processed.txt
*******
~/mldk_dev/temp_ml->Runningmltagain will not rerun anything:~/mldk_dev/temp_ml->mlt processed.txt
mlt: 'processed.txt' is up to date.