Getting Started with Comet ML. An overview of the popular... | by Angelica Lo Duca | Towards Data Science

Comet ML is an experimentation platform, which permits testing Machine Learning projects, from the beginning up to the final monitoring. Many other similar platforms exist on the Web, including Neptune.ai, Guild.ai, Sacred, and so on.
Comet ML can be easily integrated with the most popular Machine Learning libraries, including scikit-learn, Pytorch, Tensorflow, Keras, and so on. Experiments can be written in Python, Javascript, Java, R, and REST APIs.
In this article, I focus on Python.
The article is organized as follows:
Overview of Comet ML
Working with Comet ML
Example of usage
Comet ML is an online platform, which permits tracking experiments. The main advantage of Comet ML is that I can easily build a reporting dashboard and a monitoring system.
Comet ML provides the following features:
compare experiments: I can easily build different experiments for the same project, and compare the results, in terms of metrics, system metrics, parameters, and so on;
monitor the model: I can monitor the model from the early stages up to production. This can be done through alerts and debugging strategies;
collaborate with other people: I can share my workspace project with other people;
build reports and panels: starting from the results of my experiment, I can easily build reports and dashboards;
keep my project public or make it public.
Once I enter the Comet ML Web site, I can create a free account. Then, I log in to the platform and I create a new project, by clicking the relative top right button. I fill the form with needed information:
When I click on the Create button, an empty dashboard appears.
I can add a new experiment, by clicking the button on the top right of the page (+Add). I select Experiment from the dropdown menu.
The platform generates a new API for the experiment, that can be used in my Python code:
# import comet_ml at the top of your filefrom comet_ml import Experiment# Create an experiment with your api keyexperiment = Experiment(    api_key="PUT HERE MY API KEY",    project_name="decisiontree",    workspace="PUT HERE MY ACCOUNT NAME",)
Now, I need to install the comet_ml Python package on my local computer:
pip3 install comet_ml 
And then I can start coding, as usual.
The Experiment class is the interface of the local code to Comet ML. It defines many methods, which are described in the Comet Official Documentation.
Here, I describe some common methods:
log_metric() and log_metrics() — register one or more evaluation metrics to the experiment, such as accuracy, precision, and recall;
log_confusion_matrix() —register a confusion matrix to the experiment;
log_figure() — register a figure (e.g. pyplot) to the experiment and save it as an SVG image;
display() — create an interactive environment in Jupyter, showing the Comet ML dashboard as the output of a cell;
end() — if the experiment runs in Jupyter, this method specifies that the experiment is concluded.
Once an experiment has been completed, Comet ML provides a dashboard, where I can see all the metrics, figures, parameters, and so on, that I have tracked in my code.
Evaluation metrics are automatically plotted. For example, if an experiment is run over multiple steps, I can easily draw graphs showing metrics versus step numbers.
In the left part of the Experiment Dashboard, many options can be selected, included but not limited to:
Panels — final dashboards, that can be used as reports for the final user;
Metrics — a table with all the metrics and their values;
System Metrics — a visual representation of all the system metrics, including Memory Usage, CPU Utilization.
As a practical use case of Comet ML, I exploit the Heart Attacks dataset, provided by Kaggle.
Firstly, I set up the experiment, by specifying the project name and the workspace (my account name).
from comet_ml import Experimentexperiment = Experiment(    api_key="PUT HERE MY API KEY",    project_name="decisiontree",    workspace="PUT HERE MY ACCOUNT NAME",)
Now I can work as usual. I load the dataset as a pandas dataframe.
import pandas as pddf = pd.read_csv('source/heart.csv')df.head()
I define feature columns.
cols = []for col in df.columns:    if col != 'output':        cols.append(col)
I define input and output variables:
X = df[cols]y = df['output']
I scale all the input features through a MinMaxScaler
from sklearn.preprocessing import MinMaxScalerscaler = MinMaxScaler()X_scaled = scaler.fit_transform(X)
I set up the dataset in training and test sets.
from sklearn.model_selection import train_test_splitrandom_state = 42X_train, X_test, y_train, y_test = train_test_split(X_scaled,y, random_state=random_state)
I exploit a DecisionTreeClassifier as a model. I fit it with a progressive number of samples included in the training set and I predict the output class for all the samples in the test set. I also build a classification report for each step.
I log the metrics returned by the classification_report() function through the Comet Experiment function, called log_metrics(). I also log the confusion matrix, through the experiment.log_confusion_matrix() function.
from sklearn.tree import DecisionTreeClassifierfrom sklearn.metrics import classification_reportimport numpy as npmodel = DecisionTreeClassifier(random_state=random_state, max_depth=2)min_samples = 5target_names = ["class 0", "class 1"]for step in range(min_samples, len(X_train)):    model.fit(X_train[0:step], y_train[0:step])    y_pred = model.predict(X_test)    pred_proba = np.max(model.predict_proba(X_test), axis=1)    report = classification_report(y_test, y_pred, target_names=target_names, output_dict=True)    for label, metric in report.items():        try:            experiment.log_metrics(metric, prefix=label, step=step)except:            experiment.log_metric(label, metric,step=i)    experiment.log_confusion_matrix(y_test.tolist(), y_pred.tolist())
I show the experiment results in Jupyter.
experiment.display(tab="confusion-matrices")
The following figure shows an excerpt of the produced dashboard:
I conclude the experiment.
experiment.end()
Congratulations! You have learned how to configure and run an experiment in Comet ML, an Experimentation Platform for Machine Learning.
Comet ML can be exploited to run more complex experiments. Stay tuned for more tips 😃
If you have come this far to read, for me it is already a lot for today. Thanks! You can read more about me in this article.