Package: ml-versioning-tools
Machine Learning Versioning Tools - MLV-toolsPublic repository for versioning machine learning data.InstallingMLV-tools can be installed fromPyPi:pip install ml-versioning-toolsIt is also possible to install it directly from sources:git clone https://github.com/peopledoc/ml-versioning-tools.git
cd ml-versioning-tools

    make develop
OR
    make package
    pip install ./package/*.whlTutorialA tutorial is available to showcase how to use the tools.
SeeMLV-tools tutorial.KeywordsStep metadata: in this document it refers to the first code cell when it
is used to declare metadata such as parameters, dvc inputs/outputs, etc.Working directory: the project's working directory. Files specified in the
user configuration are relative to this directory. The--working-directory(or-w) flag is used to specify the working directory.Toolsipynb_to_python: this command converts a givenJupyter Notebookto a
parameterized and executablePython3 script(see specific syntax in section below)ipynb_to_python -n [notebook_path] -o [python_script_path]gen_dvc: this command creates aDVC commandwhich call the script generated by ipynb_to_python.gen_dvc -i [python_script] --out-py-cmd [python_command] \
              --out-bash-cmd [dvc_command]export_pipeline: this command exports the pipeline corresponding to the given DVC meta file into a bash script.
Pipeline steps are called sequentially in a dependency order. Only for local steps.export_pipeline --dvc [DVC target meta file] -o [pipeline script]ipynb_to_dvc: this command converts a givenJupyter Notebookto a
parameterized and executablePython3 scriptand aDVC command. It is the combination
ofipynb_to_pythonandgen_dvc. It only works with a configuration file.ipynb_to_dvc -n [notebook_path]check_script_consistencyandcheck_all_scripts_consistency: those commands ensure consitency between a Jupyter
notebook and its generated python script. It is possible to use them as git hook or in the project continuous
integration. The consistency check ignores blank lines and comments.check_script_consistency -n [notebook_path] -s [script_path]

check_all_scripts_consistency -n [notebook_directory]
# Works only with a configuration file (provided or auto-detected)ConfigurationA configuration file can be provided, but it is not mandatory.  Its default location is[working_dir]/.mlvtools. Use the flag--conf-path(or-c) on the command line to
specify a specific configuration file path.The configuration file format is JSON{"path":{"python_script_root_dir":"[path_to_the_script_directory]","dvc_cmd_root_dir":"[path_to_the_dvc_cmd_directory]","dvc_metadata_root_dir":"[path_to_the_dvc_metadata_directory]"[optional]},"ignore_keys: ["keywords", "to", "ignore"],"dvc_var_python_cmd_path": "MLV_PY_CMD_PATH_CUSTOM","dvc_var_python_cmd_name": "MLV_PY_CMD_NAME_CUSTOM","docstring_conf": "./docstring_conf.yml"}All given paths must be relative to theworking directorypath_to_the_script_directory: is the directory wherePython 3script will be generated usingipynb_to_scriptcommand. ThePython 3script name is based on the notebook name.ipynb_to_script -n ./data/My\ Notebook.ipynb

  Generated script: `[path_to_the_script_directory]/my_notebook.py`path_to_the_dvc_cmd_directory: is the directory whereDVCcommands will be generated usinggen_dvccommand. Generated command names are based onPython 3script names.gen_dvc -i ./scripts/my_notebook.py

  Generated commands: `[path_to_the_python_cmd_directory]/my_notebook_dvc`path_to_the_dvc_metadata_directory: is the directory whereDVCmetadata files will be generated when executinggen_dvccommands. This value is optional, by defaultDVCmetadata files will be saved in theworking directory.
GeneratedDVCmetadata file names are based onPython 3script names../[path_to_the_python_cmd_directory]/my_notebook_dvc

  Generated files: `[path_to_the_dvc_metadata_directory]/my_notebook.dvc`ignore_keys: list of keywords use to discard a cell. Default value is['# No effect ].
(SeeDiscard cellsection)dvc_var_python_cmd_path,dvc_var_python_cmd_name,dvc_var_meta_filename: they allow to customize variable names which
can be used indvc-cmdDocstring parameter. They respectively correspond to the variables holding the python command
file path, the file name and the variable holding theDVCdefault meta file name. Default values are 'MLV_PY_CMD_PATH',
'MLV_PY_CMD_NAME' and 'MLV_DVC_META_FILENAME'. (See DVC Command/Complex cases section for usage)docstring_conf: the path to the docstring configuration used for Jinja templating (see DVC templating section).
This parameter is not mandatory.Jupyter Notebook syntaxTheStep metadatacell is used to declare script parameters andDVCoutputs and dependencies.
This can be done using basic Docstring syntax. This Docstring must be the first statement is this cell, only
comments can be writen above.Good practicesAvoid using relative paths in your Jupyter Notebook because they are relative to
the notebook location which is not the same when it will be converted to a script.Python Script ParametersParameters can be declared in theJupyter Notebookusing basic Docstring syntax.
This parameters description is used to generate configurable and executable python scripts.Parameters declaration inJupyter Notebook:Jupyter Notebook: process_files.ipynb#:param [type]? [param_name]: [description]?
"""
:param str input_file: the input file
:param output_file: the output_file
:param rate: the learning rate
:param int retry:
"""GeneratedPython3 script:[...]
def process_file(input_file: str, output_file, rate, retry:int):
    """
     ...
    """
[...]Script command line parameters:my_script.py -h

usage: my_cmd [-h] --input-file INPUT_FILE --output-file OUTPUT_FILE --rate
             RATE --retry RETRY

Command for script [script_name]

optional arguments:
  -h, --help            show this help message and exit
  --input-file INPUT_FILE
                        the input file
  --output-file OUTPUT_FILE
                        the output_file
  --rate RATE           the rate
  --retry RETRYAll declared arguments are required.DVC commandADVCcommand is a wrapper overdvc runcommand called on aPython 3script generated
withipynb_to_pythoncommand. It is a step of a pipeline.It is based on data declared innotebook metadata,
2 modes are available:
- describe only input/output for simple cases (recommended)
- describe full command for complex casesSimple casesSyntax:param str input_csv_file: Path to input file
:param str output_csv_file: Path to output file
[...]

[:dvc-[in|out][\s{related_param}]?:[\s{file_path}]?]*
[:dvc-extra: {python_other_param}]?

:dvc-in: ./data/filter.csv
:dvc-in input_csv_file: ./data/info.csv
:dvc-out: ./data/train_set.csv
:dvc-out output_csv_file: ./data/test_set.csv
:dvc-extra: --mode train --rate 12Provided{file_path}path can be absolute or relative to the working directory.The{related_param}is a parameter of the correspondingPython 3script,
it is filled in for the python script callThedvc-extraallows to declare parameters which are not dvc outputs or dependencies.
Those parameters are provided to the call of thePython 3command.pushd /working-directory

INPUT_CSV_FILE="./data/info.csv"
OUTPUT_CSV_FILE="./data/test_set.csv"

dvc run \
-d ./data/filter.csv\
-d $INPUT_CSV_FILE\
-o ./data/train_set.csv\
-o $OUTPUT_CSV_FILE\
gen_src/python_script.py --mode train --rate 12
        --input-csv-file $INPUT_CSV_FILE
        --output-csv-file $OUTPUT_CSV_FILEComplex casesSyntax:dvc-cmd: {dvc_command}

:dvc-cmd: dvc run -o ./out_train.csv -o ./out_test.csv
    "$MLV_PY_CMD_PATH -m train --out ./out_train.csv &&
     $MLV_PY_CMD_PATH -m test --out ./out_test.csv"This syntax allows to provide the full dvc command to generate. All paths can be absolute or relative to the working directory.
The variables $MLV_PY_CMD_PATH and $MLV_PY_CMD_NAME are available. They respectively contains the path and the name
of the corresponding python command.
The variable $MLV_DVC_META_FILENAME contains the default name of theDVCmeta file.pushd /working-directory
MLV_PY_CMD_PATH="gen_src/python_script.py"
MLV_PY_CMD_NAME="python_script.py"

dvc run -f $MLV_DVC_META_FILENAME -o ./out_train.csv \
    -o ./out_test.csv \
    "$MLV_PY_CMD_PATH -m train --out ./out_train.csv && \
    $MLV_PY_CMD_PATH -m test --out ./out_test.csv"
popdDVC templatingIt is possible to use Jinja2 template in DVC Docstring part. For example, it can be useful to declare all
steps dependencies, outputs and extra parameters.Example:# Docstring in Jupyter notebook
"""
[...]
:dvc-in: {{ conf.train_data_file_path }}
:dvc-out: {{ conf.model_file_path }}
:dvc-extra: --rate {{ conf.rate }}
"""

# Docstring configuration file (Yaml format): ./dc_conf.yml

train_data_file_path: ./data/trainset.csv
model_file_path: ./data/model.pkl
rate: 45

# DVC command generation
gen_dvc -i ./python_script.py --docstring-conf ./dc_conf.ymlTheDocstring configuration filecan be provided through the main configuration or using--docstring-confargument. This feature is only available forgen_dvccommand.Discard cellSome cells inJupyter Notebookare executed only to watch intermediate results.
In aPython 3script those are statements with no effect.
The comment# No effectallows to discard a whole cell content to avoid waste of
time running those statements. It is possible to customize the list of discard keywords, seeConfigurationsection.ContributingWe happily welcome contributions to MLV-tools. Please see ourcontributionguide for details.