Putting ML in production II: logging and monitoring | by Javier Rodriguez Zaurin | Towards Data Science

In our previous post we showed how one could use the Apache Kafka’s Python API (Kafka-Python) to productionise an algorithm in real time. In this post we will focus more on the ML aspects, more specifically on how to log information during the (re)training process and monitor the results from the experiments. To that aim we will use MLflow along with Hyperopt or HyperparameterHunter.
A detailed description of both the scenario and the solution can be found in the post mentioned before.
In summary, we would like to run an algorithm in real time, and some immediate action needs to be taken based on the algorithm’s outputs (or predictions). In addition, after N interactions (or observations) the algorithm needs to be retrained without stopping the prediction service.
Our solution relies mostly on Kafka-Python distributing information among the different components of the process (see Figure 1 in our first post for more details):
A Service/App generates a message (JSON) with the required information for the algorithm (i.e. the features).The “Predictor” component receives the message, processes the information and runs the algorithm, sending the prediction back to the Service/App.After N number of processed messages (or observations) the Predictor sends a message to the “Trainer” component that starts a new training experiment. This new experiment will include the original dataset plus all the new observations collected. As we described in the first post, in the real world one would have to wait until it receives as many real outcomes (i.e. true labels or numerical results) as observations before retraining the algorithm.Once the algorithm has been retrained, the Trainer sends the corresponding message and the Predictor will load the new model without stopping the service.
A Service/App generates a message (JSON) with the required information for the algorithm (i.e. the features).
The “Predictor” component receives the message, processes the information and runs the algorithm, sending the prediction back to the Service/App.
After N number of processed messages (or observations) the Predictor sends a message to the “Trainer” component that starts a new training experiment. This new experiment will include the original dataset plus all the new observations collected. As we described in the first post, in the real world one would have to wait until it receives as many real outcomes (i.e. true labels or numerical results) as observations before retraining the algorithm.
Once the algorithm has been retrained, the Trainer sends the corresponding message and the Predictor will load the new model without stopping the service.
For the sake of brevity, we prefer to focus here on the processes rather than the algorithms themselves. However, let me at least give some direction in case you want to explore further.
The core algorithm we use is LightGBM. LightGBM has become my go-to algorithm for almost every project that involves classification or regression (it can also do ranking). There is an overwhelming amount of information online about the package, and one can learn how to use it in a variety of scenarios in their fantastic example section at the github repo. However, I always recommend reading the corresponding paper for a given algorithm. In this case, Guolin Ke et al 2017 did a fantastic job. The paper is very well written and, in general, quite accessible.
The optimisation packages used will be HyperparameterHunter (hereafter HH) and Hyperopt, both with Bayesian optimisation methods. HH uses Skopt as a backend, and its BayesianOptimization method is based on Gaussian Processes. On the other hand Hyperopt is, to my knowledge, the only Python package that implements the TPE (tree of Parzen estimators) algorithm. I have found some other libraries that use that algorithm, but are all dependent on Hyperopt (e.g. Optunity or Project Ray’s tune).
If you want to learn about Bayesian optimization methods I recommend doing the following. Read first the Bayesian Optimization section in the Skopt site. There you can find the description of the problem statement and the bayesian process (or loop), which I’d say is fairly common for bayesian optimisation approaches. Then go to the Hyperopt paper (Bergstra et al., 2011). Again, this is a “must-read” paper if you want to be familiar with Bayesian approaches. In particular, there you will learn about Gaussian Processes (GP) and TPE in the context of Sequential Model-based Global Optimization (SMBO) algorithms (Sections 2–4).
The remaining ML “ingredient” is MLflow, which will be used here to help tracking and monitoring the training process (although you will see that HH already does a good job saving all the important data).
Following a similar approach to the one used in our fist post, we will use the code as a guideline commenting on the most important parts. All the code in this section can be found within the trainmodule in our repo. We will start with Hyperopt and then move to HH, where we will illustrate what makes the later unique.
Hyperopt
The code in this section can be found within the in the script train_hyperopt_mlflow.py at the train module.
Remember, the goal is to minimise an objective function. Our Hyperopt objective function looks like this:
Where params could be, for example:
Let’s discuss the code within the function. The function must depend only on params. Within the function we use cross validation and we output the best metric, which is binary_logloss in this case. Note that we use LightGBM (imported as lgb) methods ( lgb.cv in line 14, Snippet 1) as opposed as the corresponding sklearn wrap up. This is because to my experience, LightGBM’s own methods are normally a bit faster. Also note that LightGBM does not implement metrics such as f1_score . Nonetheless, we have included a LightGBM f1 customised metric function in the train_hyperopt.py and train_hyperopt_mlfow.py scripts, just in case.
It is worth stopping for a second at line 22 in Snippet 1, where we record the number of boosting rounds used for that particular iteration. This is because when using Hyperopt (or Skopt or HH) the algorithm will optimise based on the input value of the parameters, one being num_boost_round. Within the objective function we do cross validation with early stopping to avoid overfitting. This means that the final number of boosting rounds might differ from the input value. This information will get "lost" within the optimisation process. To overcome this limitation we simply save the final num_boost_round to the dictionary early_stop_dict. However, it is not always clear that this is the best solution. For a full discussion on this and other issues regarding the optimisation of GBMs, please, have a look to this notebook in my github.
Finally, remember that we need to minimise the output value. Therefore, if the output is a score, the objective function must output its negative value, while if is an error ( rmse) or loss ( binary_logloss), the function must output the value itself.
The code that runs the optimisation process is simply:
Every set of parameter tried will be recorded in the trials object. Once the optimisation is done, we could code our own functionalities to record the results. Alternatively, we could use tools such as MLflow to help us with that task, and visually monitor the different experiments. As you can imagine, one could write a number of posts only on MLflow. Here, we will simply illustrate how we have used it to record the best performing parameters, model and metric, and monitor the algorithm’s performance.
MLflow
The MLflow block that tracks the results per experiment remains almost identical for both Hyperopt and HH, and is described in the snippet below.
Lines 1–8: one “annoying” behaviour we found in the newest MLflow version ( 0.8.2) is that the first time you instantiate the class MLflowClient() or create an experiment ( mlflow.create_experiment('test')) it will create two directories, mlruns/0 and mlruns/1 . The former is called Default and it will remain empty as you run experiments. Here we show this behaviour in an empty directory called test_mlflow :
infinito:test_mlflow javier$ lsinfinito:test_mlflow javier$ ipythonPython 3.6.5 (default, Apr 15 2018, 21:22:22)Type ‘copyright’, ‘credits’ or ‘license’ for more informationIPython 7.2.0 — An enhanced Interactive Python. Type ‘?’ for help.In [1]: import mlflowIn [2]: mlflow.__version__Out[2]: ‘0.8.2’In [3]: mlflow.create_experiment(‘test’)Out[3]: 1In [4]: ls mlruns/0/ 1/
Therefore, when you open the UI the first screen will be an empty screen with a experiment named Default . If you can live with that (I can’t), then there are easier ways to code lines 1–8 in the MLflow block, such as for example:
In the current set up (Snippet 4) , our first initialisation of the process ( python initialize.py) will be referred as Default and stored in directory mlruns/0 .
On the other hand, a more elegant way of defining the experiment_id per run would be to list the existing experiments and get the last elements’ id:
experiments = client.list_experiments()with mlflow.start_run(experiment_id=experiments[-1].experiment_id):
However, another “inconvenient” behaviour I found is that client.list_experiments() does not preserve the order. This is why we use the “slightly-less-elegant” solution n_experiments .
Line 9 in advance: we just run the experiment and record all, parameters, metrics and the model as an MLflow artifact.
At this stage it is worth mentioning that we are completely aware we are “under-using” MLflow. As well as tracking the results of the algorithms, MLflow can package and deploy projects. In other words, with MLflow one can manage almost the entire machine learning cycle. However, it is my impression that to do such a thing one needs to start a project with MLflow in mind. Moreover, it is not straightforward for me to see how one could package the entire project described in this and the previous posts using MLflow without adding unnecessary complexity. Nonetheless, I see lots of potential in this library and I clearly see myself using it in future projects.
Hyperparameter Hunter (HH)
After having a look to the code in snippets 1–4 one might wonder: “what if I want to record every hyper-optimisation run and keep track in a leaderboard?”. Well, there are two possibilities: i) one could simply move the MLflow block to the body of the objective function and adapt the code accordingly or ii) simply use HH.
As I write this post I see two drawbacks in using HH. In the first place, there is no need for you to code your own objective function. While this is mostly a positive aspect, it also means that it is less flexible. However, I have been using HH for some time now and unless you need to design a complex objective function (e.g. some unusual data manipulation within the objective, or internal update of parameters) HH will do the job. If you do need to design a highly customised objective function, you could always code one using sklearn ‘s syntax as pass it to the HH’s optimizer object as the model_initializer .
The second drawback, and perhaps more important, is not directly related to HH, but Skopt. HH is built on top of Skopt, which is notably slower that Hyperopt (see this notebook in my repo). However, I am aware that there are current efforts to add Hyperopt as an alternative backend (along with other upcoming features such as feature engineering, so stay tuned).
In conclusion, if you do not need to design a particularly complex objective function, and you can afford “Skopt-speed”, HH offers a number of functionalities that make it unique. To start with, HH records and organises all the experiments for you. Moreover, it learns as you run additional tests, as none of the past tests go to waste and. In other words:
“HyperparameterHunter is already aware of all that you’ve done, and that’s when HyperparameterHunter does something remarkable. It doesn’t start optimization from scratch like other libraries. It starts from all of the Experiments and previous optimization rounds you’ve already run through it.” Hunter McGushion.
Let’s have a look to the code. The following 3 snippets are all you need when using HH (see the documentation for more details). The full code in this section can be found in the script train_hyperparameterhunter_mlflow.py at the train module.
As you will see, the syntax very concise. We first set an Environment, which is simply a class to organise the parameters that allow experiments to be fairly compared.
We then run the experiment
Where model_init_params and model_extra_params are:
When looking carefully to Snippet 7, we can find a further difference between HH and Hyperopt, again purely related to the Skopt backend. You will see that while when using Hyperopt one can use quantile uniform distributions (hp.quniform(low, high, step)). There is no such option in Skopt. This means that for parameters like num_boost_rounds or num_leaves the search is less efficient. For example, a priori, one would not expect two experiments with 100 and 101 boosting rounds to yield different results. This is why when using Hyperopt we set num_boost_rounds to hp.quniform(50, 500, 10) , for example.
A potential way around is to use Skopt’s Categorical variables:
num_leaves=Categorical(np.arange(31, 256, 4)),
However, this is not the best solution given the fact that in reality, num_boost_roundsor num_leaves are not categorical variables, but are going to be treated at such. For example, by default, Skopt will build a one-hot encoded representation of the input space for categorical features. As categories have no intrinsic ordering, the distance between two points in that space is one if cat_a != cat_band zero otherwise. This behaviour is not what we want during the search process for parameters like num_leaves, since 32 leaves would be as distant from 31 as 255. Skopt offers the possibility of not transforming the space, which is still not ideal, but better:
num_leaves=Categorical(np.arange(31, 256, 4), transform=’identity’),
But for one reason or another this throws an error every time I have tried to use it. Nevertheless, we will use Integerand live with it while HH implements Hyperopt as a backend.
In this section we will run a minimal example to illustrate how we use HH and MLflow to track every detail of the training process. Here we will keep things simple, but one could use the Callbacks functionality in HH to seamlessly integrate the two packages.
The minimal example consists in processing 200 messages and retraining the algorithm every 50 processed messages. In every retraining experiment HH will run just 10 iteration. In the real world, there would be no limit for the incoming messages (i.e Kafka consumers are always listening), the retraining might happen after thousands of new observations have been processed or perhaps after some time step (i.e. every week) and HH should run for hundreds of iterations.
In addition, for the purpose of illustration, here the process is “over-logged” using our own logging methods (mainly pickle), HH and MLflow. In production, I would recommend using a customised solution combining MLflow and HH.
let’s have a look
Figure 1 shows a screen shot after processing the 200 messages and having retrained the model 4 times (once every 50 messages). In the upper-left terminal we run the Predictor (predictor.py ), the middle terminal runs the Trainer(trainer.py), where we can see the output from the last HH run, and the lower-terminal runs the App/Service(sample_app.py ), where we can see the requests received and the output prediction.
The reader might notice the alternation in the new model loaded in the upper-left terminal ( NEW MODEL RELOADED 0 -> NEW MODEL RELOADED 1 -> NEW MODEL RELOADED 0 -> NEW MODEL RELOADED 1 ). This is because when using our own logging methods we use a parameter called EXTRA_MODELS_TO_KEEP that sets how many past models we keep. It is currently set to one, and the current loading process points towards our output directory. This can be easily changed in the code to save M past models or to point towards the HH or MLflow corresponding output directories, where all the past best performing models are stored.
The upper-right terminal in Figure 1 starts the MLflow Tracking Server. A screen capture of the MLflow UI is shown below in Figure 2.
The figure shows the information that MLflow saved for the particular case of what we call “experiment_2” (i.e. retraining the models with 100 accumulated new observations/messages). For consistency with HH, we have saved every retraining process as a different experiment. If you’d prefer to save all retraining processes as one experiment simply go to the optimize method in the LGBOptimizer class at train_hyperparameter_hunter.py and change the reuse_experiment parameter to True.
With the current set up, there will be one subdirectory per experiment within the mlruns directory. For example, the structure of the mlruns/1 directory is:
As you can see, all the information one would need is there, very well organised. Let me insist, the current structure of our MLflow-code saves only the best performing set of parameters and model. As mentioned before, one could move the MLflow block in Snippet 4 to the objective function and adapt the code so that MLflow records every single iteration. Alternatively, one could use HH.
HH logs absolutely everything. Let’s have a look to the HyperparameterHunterAssets directory structure. A detail explanation of what is in each sub-directory can be found here. As one can see, plenty of information is saved about every single iteration, including the training datasets per experiment (5 datasets including the original/default one, plus 4 new datasets that include 50 additional observations per retraining cycle) and a leaderboard with the current winning iteration.
Hopefully, at this stage the reader has a good idea on how one could combine MLflow and HH to track and monitor an algorithm’s performance in production.
However, while all elements described here need to be considered when productionising ML, not all that needs to be considered is discussed here. Let me at least mention the missing pieces in the next section.
Unit Test
One aspect we have (intentionally) ignored in this and the previous post is unit-test. This is because unit-test normally depends on the application of your algorithm. For example, at your company, there might be some parts of the pipeline that have been widely tested in the past while some new pieces of code might require thorough unit-test. If I have the time I will include some code related to this task in the repo.
Concept Drift
Often ignored in production is “Concept Drift”. Concept Drift refers to the fact that the statistical properties of your dataset can change over time, and that can have a notable impact on the quality of your predictions. For example, let’s say your company has an app that targets mostly people under 30. One week you run an advertising campaign that broadens the age range of your users. Chances are that your existing models will not perform well for your new audience.
There are a number of options to detect and address Concept Drift. One could code a customised class to make sure that the distributions of the features in the training and testing datasets remains stable within certain limits. Alternatively, one could use libraries such as MLBox. MLBox fits within the new wave of automated ML libraries and comes with a series of nice functionalities, including an optimisation method that relies on, guess what...Hyperopt. Another functionality within the package is the Drift_thresholder() class. This class will automatically handle Concept Drift for you.
The MLBox implementations uses a classifier (a Random Forest by default) and tries to predict whether an observation belongs to the training or testing dataset. Then the estimated drift is measure as:
drift = (max(np.mean(S), 1-np.mean(S))-0.5)*2
Where S is a list containing the roc_auc_score corresponding to the n folds used during the process. If there is not Concept Drift, the algorithm should not be able to distinguish observations from the two datasets, the roc_auc_score per fold should be close to a random guess (0.5) and the drift between the datasets should be close to zero. Alternatively, if one or more of the features has changed over time then the algorithm will easily distinguish between the training and testing datasets, the roc_auc_score per fold would be close to 1 and in consequence, the drift will also be close to one. Note that if the datasets are highly imbalanced and/or very sparse, you might want to use a more suitable metric and algorithm (random forest is known to not perform well in very sparse datasets).
Overall, if you want to learn more about the package and the concept of Concept Drift you can find more information here. The source code for their DriftEstimator() is here. Again, I will include some code related to Concept Drift in our repo if I find the time.
Finally, so far we have used Kafka locally and we did not need to scale or maintain it. However, in the real world one would have to scale according to traffic and maintain the algorithm’s components. Our initial thought was to use Sagemaker for that task, and write a 3rd post. However, after a good, long dive into the tool (that you can find in the branch sagemakerin the repo) we found that using Sagemaker brought a lot of unnecessary complexity. In other words, using it turned out to be more complex than simply moving the code to the cloud, adapt it to use AWS tools (mostly EC2s and S3) and run it automatically using simple, customised scripts.
Let’s summarise the concepts and components one would have to consider when putting ML in production.
Well written code and properly structured project (big thanks to Jordi)Logging and monitoring the algorithm’s performance through timeUnit-testConcept DriftAlgorithm/Service scaling and maintenance.
Well written code and properly structured project (big thanks to Jordi)
Logging and monitoring the algorithm’s performance through time
Unit-test
Concept Drift
Algorithm/Service scaling and maintenance.
In this and our previous posts we have use Kafka-Python, MLflow and HyperparameterHunter or Hyperopt to illustrate points 1 and 2. Points 3, 4 and 5 will not be covered here. Regarding to Points 3 and 4 I believe that there is not much point in writing a further detailed explanation in this or another post (although if I have the time I will add some code to the repo). Regarding to Point 5, as we mentioned before, we think that simply moving the code and the structure described here to the cloud, and adapt it to the available tools there (e.g. EC2s, S3, ...), would be sufficient.
As always, comments/suggestions: jrzaurin@gmail.com
[1] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye and Tie-Yan Liu, LightGBM: A highly efficient gradient boosting decision tree, Advances in Neural Information Processing Systems, 3149–3157, 2017.
[2] Bergstra, James; Bardenet, Remi; Bengio, Yoshua; Kegl, Balazs (2011), “Algorithms for hyper-parameter optimization”, Advances in Neural Information Processing Systems.