How to build an integration between AutoML and MLFlow | by Bogdan Cojocar | Towards Data Science

In this article, we will go over the concept of AutoML, how it can improve your productivity and how to track it’s performance metrics.
We will use the following technologies:
Python 3
H2O AutoML
MLFlow
As the name suggests, it is the process to automate a machine learning pipeline. We can automate various stages of the process like data preparation, feature engineering, model selection, and hyperparameter selection. In this tutorial, we will focus mainly on model selection.
There are numerous benefits in using this approach such as:
cost reduction by increasing the productivity of the data scientist
reducing the amount of repetitive work and boring tasks that they have to do
the performance of the model selection can be used as a benchmark and to figure out what models would be best to investigate further by the data scientists
What is MLFlow?
MLFlow is an open-source platform used to monitor and save machine learning models after training. The great thing about it is that it can integrate with other frameworks such as H2O or Spark building a uniform and easy to use environment.
Now let’s dive into the steps to use AutoML in practice.
TL;DR: The code is stored in Github.
To get the environment ready we need to install the new dependencies found in the folder of the project:
pip install -r requirements.txt
Also we need to start the MLFlow server by typing in the terminal:
mlflow ui
We sound be able to see the MlFlow ui now by running in the browser http://127.0.0.1:5000:
Now we should be able to visualise all the experiments (training steps) we run. We can also start the Jupyter lab notebook which is the environment where we write our code:
jupyter lab
H2O is a distributed framework and in order to use it locally we need to start a server. This might take a minute or so, depending on your setup:
h2o.init()
MLFlow uses the concept of experiments to track the progress of a machine learning model.
try:    experiment = mlflow.create_experiment(experiment_name)except:    experiment = client.get_experiment_by_name(experiment_name)mlflow.set_experiment(experiment_name)
If the experiment is already created, we just reuse it.
We have reached the most interesting part, using auto ml to do the training. Multiple models are trained in parallel and validated (6 fold cross-validation) using an independent subset of the data. Every new training run will save some validation metrics and the model that has performed the best (leader).
with mlflow.start_run():    model = H2OAutoML(max_models=10, max_runtime_secs=300, seed=24, nfolds=6)    model.train(x=x_cols, y=y_cols, training_frame=train, validation_frame=valid)    mlflow.log_metric("rmse", model.leader.rmse())    mlflow.log_metric("log_loss", model.leader.logloss())    mlflow.log_metric("mean_per_class_error", model.leader.mean_per_class_error())    mlflow.h2o.log_model(model.leader, "model")        lb = model.leaderboard    lb = get_leaderboard(model, extra_columns='ALL')    print(lb.head(rows=lb.nrows))
We have decided to run 10 models. Using the get_leaderboard function we get to see what machine learning algorithms have run:
And the training is recorded in MLFlow, and we can compare it with future experiments:
The last step is to actually do the prediction with the newly elected leader. For the purpose of this tutorial we will just use the validation data, but in practice we need to use new data.
all_mlflow_runs = client.list_run_infos(experiment.experiment_id)if len(all_mlflow_runs) > 0:    run_info = all_mlflow_runs[-1]    model = mlflow.h2o.load_model("mlruns/{exp_id}/{run_id}/artifacts/model/".format(exp_id=experiment.experiment_id,run_id=run_info.run_uuid))    result = model.predict(valid)else:    raise Exception('Run the training first')
Again we use MLFlow to read the latest run in the experiment and load the GLM model. If we never run the training we can throw an exception to notify the user that he has to do that first.
We’ve seen how in a few simple steps we can have an automated process for model selection, tracking and loading/saving the best machine learning model for a given set of data. Although on competitions like Kaggle we can see that humans are still easily beating AutoML frameworks, we can still use them as another tool in our data science kit, to boost and improve our performance.