An entry-level project of testing various machine learning models in particle identification
By: Gakki Cheng
Tags: Data Scientist, Gridsearchcv, Physics, Data Science, Machine Learning

A little bit introduction about myself and this project: I am a university graduate major in mathematical physics and after graduation I joined a data science diploma program hopefully to start my professional career as a data scientist. This particle identification project was also my capstone project for that diploma program. During the job searching period, now I have more time to learn about data science and keep on refining my skills. I started to look back on this project I finished 2 months ago, and decided to write a story post as a milestone on the road of becoming a real data scientist :)

First, the source of the dataset is from the link as following: https://www.kaggle.com/lavanyashukla01/particle-identification

I was inspired by one of the blog posts (link as below): https://lavanya.ai/2019/05/31/searching-for-dark-matter/

In the post above, AdaBoost and Neural Network models were tested and already showed relatively good results. But for me, 2 months ago, as a beginner, I still wanted to see how other types of machine learning models performed on this enormous particle identification dataset. Although I knew that the some of the model’s results might be poor but I called my project ‘A Ninja Road of Modelling’ as an opportunity to learn to tune the models and build a basic machine learning pipeline.

So, in general, my project includes:

Basic machine learning models

Ensemble methods

Hyperparameters tuning

Basic machine learning pipeline and use of GridSearchCV

Now, let’s start from some basic models. As a beginner, there are two models in my mind to start from, Logistic Regression Model and Decision Tree Model. I didn’t choose Linear Regression Model because particle identification problem is a multi-class classification problem instead of a numerical regression problem like price prediction. Therefore, with 49 features in the original dataset, our goal is to classify these particles into 6 buckets: Electron, Ghost, Kaon, Muon, Pion and Proton.

Then let’s see, how the vanilla Logistic Regression Model and Decision Tree Model performed on this dataset.

For the Logistic Regression Model, below is my code.

from sklearn.linear_model import LogisticRegression # Instantiate and train a vanilla logistic regresssion model

logistic_regression_model = LogisticRegression()

logistic_regression_model.fit(X_train, y_train)



# Evaluate it

print(f'The train accuracy is: {logistic_regression_model.score(X_train,y_train):0.3f}')

print(f'The test accuracy is: {logistic_regression_model.score(X_test,y_test):0.3f}') The train accuracy is: 0.715

The test accuracy is: 0.720

For the Decision Tree Model, I set the max_depth = 10.

#Import the decision tree classifier

from sklearn import tree #fitting the DT

DT_model = tree.DecisionTreeClassifier(max_depth = 10)

DT_model.fit(X_train, y_train) #Getting the score

print(f"The classification train accuracy is: {DT_model.score(X_train, y_train)}")

print(f"The classification test accuracy is: {DT_model.score(X_test, y_test)}") The classification train accuracy is: 0.7447

The classification test accuracy is: 0.7035

Once we have our vanilla models tested, we can proceed to the hyper-parameter tuning step to pursue best performance (accuracy). The parameters I chose were simple:

C_Range in Logistic Regression

Max_Depth in Decision Tree

To make the process easier, we can import cross-validation to best evaluate our model accuracy. Code as below:

from sklearn.model_selection import cross_val_score #Store the results

cross_validation_scores = []



#Choose a hyperparameter to tune, C

C_range = np.array([.00000001,.0000001,.000001,.00001,.0001,.001,.1,\

1,10,100,1000,10000,100000,1000000,10000000,100000000,1000000000])



#Do some cross validation

for c in C_range:

LR_model = LogisticRegression(C=c,random_state=1)



# the cross validation score (mean of scores from all folds)

cv_score = np.mean(cross_val_score(LR_model, X_remainder, y_remainder, cv = 5))



cross_validation_scores.append(cv_score)



plt.figure()

plt.plot(C_range, cross_validation_scores,label="Cross Validation Score",marker='.')

plt.legend()

plt.xscale("log")

plt.xlabel('Regularization Parameter: C')

plt.ylabel('Cross Validation Score')

plt.grid()

plt.show();

Logistic Regression

#Store the results

cross_validation_scores_DT = []



#Choose the hyperparameter to tune, max_depth

max_depth = np.array([5,10,15,20,25,30,35,40,45,50])



#Do some cross validation

for m in max_depth:

DT_model = tree.DecisionTreeClassifier(max_depth = m)



# the cross validation score (mean of scores from all folds)

cv_score_DT = np.mean(cross_val_score(DT_model, X_remainder, y_remainder, cv = 5))



cross_validation_scores_DT.append(cv_score_DT)



plt.figure()

plt.plot(max_depth, cross_validation_scores_DT,label="Cross Validation Score",marker='.')

plt.legend()

plt.xscale("log")

plt.xlabel('Regularization Parameter: max_depth')

plt.ylabel('Cross Validation Score')

plt.grid()

plt.show();

Decision Tree

It’s not too hard to find the peak cross-validation score.

After above preparation steps, we already have a general feeling of how our models perform on the classification problem. Then the question remaining would be how to improve our classification accuracy based on our machine learning knowledge. Well, there are indeed several ways I tried in my capstone project.

There are several machine learning models like Random Forest Classifier, Bagging Classifier, AdaBoost Classifier and Gradient Boosting Classifier. By using them, we are actually using methods of bagging and boosting, namely, ensemble methods.

If you are interested in learning more about ensemble methods, I recommend the reading: https://towardsdatascience.com/basic-ensemble-learning-random-forest-adaboost-gradient-boosting-step-by-step-explained-95d49d1e2725

After understanding why we try ensemble methods, we can start to build our code as below:

from sklearn.ensemble import RandomForestClassifier # Instantiate and train a vanilla Random Forest model

my_random_forest = RandomForestClassifier()

my_random_forest.fit(X_train, y_train) #Evaluate the score

print(f"The classification train accuracy is: {my_random_forest.score(X_train, y_train)}")

print(f"The classification test accuracy is: {my_random_forest.score(X_test, y_test)}") The classification train accuracy is: 0.9999834656084656

The classification test accuracy is: 0.7423148148148148

Still, we start from default vanilla models and use cross-validation to tune the parameters. Code as following

cross_validation_scores_FR = []



#Choose the hyperparameter, n_estimators

n_estimators = np.array([10,20,30,40,50,60,70,80,90,100,120])



#Do some cross validation

for n in n_estimators:

my_random_forest = RandomForestClassifier(n_estimators=n, max_depth=10)



# the cross validation score (mean of scores from all folds)

cv_score_FR = np.mean(cross_val_score(my_random_forest, X_remainder, y_remainder, cv = 3))



cross_validation_scores_FR.append(cv_score_FR)



plt.figure()

plt.plot(n_estimators, cross_validation_scores_FR,label="Cross Validation Score",marker='.')

plt.legend()

plt.xscale("log")

plt.xlabel('Regularization Parameter: n_estimators')

plt.ylabel('Cross Validation Score')

plt.grid()

plt.show();

Random Forest

As we can see, the cross validation score may still increase after a threshold of 100 (n_estimators). But there would be a tradeoff once it’s higher than 100. The computation time would not be worth it for a small accuracy increment. Hence I will stop at 100 (n_estimators).

from sklearn.ensemble import AdaBoostClassifier



#Store the results

cross_validation_scores_AD = []



n_estimators = np.array([10, 20, 30, 40, 50, 60, 70, 80, 90, 100])



#Do some cross validation

for n in n_estimators:

model_ad = AdaBoostClassifier(base_estimator=DecisionTreeClassifier(max_depth=10),n_estimators=n)



# the cross validation score (mean of scores from all folds)

cv_score_AD = np.mean(cross_val_score(model_ad, X_remainder, y_remainder, cv = 3))



cross_validation_scores_AD.append(cv_score_AD)



plt.figure()

plt.plot(n_estimators, cross_validation_scores_AD,label="Cross Validation Score",marker='.')

plt.legend()

plt.xscale("log")

plt.xlabel('Regularization Parameter: n_estimators')

plt.ylabel('Cross Validation Score')

plt.grid()

plt.show();

AdaBoosting

Similarly for the AdaBoost Classifier, 90 (n_estimators) should be a reasonable value.

Check for the Gradient Boosting Classifier.

from sklearn.ensemble import GradientBoostingClassifier #Store the results

cross_validation_scores_GR = []



n_estimators = np.array([10, 20, 30, 40, 50, 60, 70, 80, 90, 100])



#Do some cross validation

for n in n_estimators:

model_GR = GradientBoostingClassifier(n_estimators=n)



# the cross validation score (mean of scores from all folds)

cv_score_GR = np.mean(cross_val_score(model_GR, X_remainder, y_remainder, cv = 2))



cross_validation_scores_GR.append(cv_score_GR)



plt.figure()

plt.plot(n_estimators, cross_validation_scores_GR,label="Cross Validation Score",marker='.')

plt.legend()

plt.xscale("log")

plt.xlabel('Regularization Parameter: n_estimators')

plt.ylabel('Cross Validation Score')

plt.grid()

plt.show();

Gradient Boosting

import xgboost as xgb #Store the results

cross_validation_scores_XG = []



n_estimators = np.array([20,30,40,50,60,80,90,100,120,140,160,180,200])



#Do some cross validation

for n in n_estimators:

xgboost = xgb.XGBClassifier(objective='multi:softmax', n_estimators=n, max_depth=6)



# the cross validation score (mean of scores from all folds)

cv_score_XG = np.mean(cross_val_score(xgboost, X_remainder, y_remainder, cv = 2))



cross_validation_scores_XG.append(cv_score_XG)



plt.figure()

plt.plot(n_estimators, cross_validation_scores_XG,label="Cross Validation Score",marker='.')

plt.legend()

plt.xscale("log")

plt.xlabel('Regularization Parameter: n_estimators')

plt.ylabel('Cross Validation Score')

plt.grid()

plt.show();

XGBoosting

For the last model, XGBoosting, n_estimators can be set to 100.

So far, we have seen all models used in my capstone project. We now have a sense of which models could give us relatively high accuracy predictions. But still, we need an evaluation metric to reveal the accuracy in a more visualized way. The method I learned from the blog post mentioned at the beginning is the ROC curves with AUC scores.

The plot function can be defined as below:

Reference: https://lavanya.ai/2019/05/31/searching-for-dark-matter/

# Defining a roc_curve graph function suitable for this project

def roc_curves(predictions, labels):

plt.figure(figsize=(9, 6))

u_labels = np.unique(labels)

for lab in u_labels:

y_true = labels == lab

y_pred = predictions[:, lab]

fpr, tpr, _ = roc_curve(y_true, y_pred)

auc = roc_auc_score(y_true, y_pred)

plt.plot(tpr, 1-fpr, linewidth=3, label=class_label_correspondence[lab] + ', AUC = ' + str(np.round(auc, 4)))

plt.xlabel('Signal efficiency (TPR)', size=15)

plt.ylabel("Background rejection (1 - FPR)", size=15)

plt.xticks(size=15)

plt.yticks(size=15)

plt.xlim(0., 1)

plt.ylim(0., 1)

plt.legend(loc='lower left', fontsize=15)

plt.title('One particle vs rest ROC curves', loc='right', size=15)

plt.grid(b=1)

To have a better understanding on how useful ROC curves & AUC scores can be, I recommend the reading: https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5

Below, I will show some of the valuable plots I collect from my models.

Random Forest

AdaBoost

Overall, Random Forest and AdaBoost models give us the best curves close to our expectation.

The final step I did in my capstone project is to build a model selection pipeline from four models: Random Forest, AdaBoost, Gradient Boosting, XGBoost. By using technique GridSearchCV, we are able to find a machine learning model which achieve the highest accuracy score on a given classification dataset. In the pipeline, we can do the hyper-parameter tuning simply by adding/deleting parameter’s available choices.

I recommend this reading about pipeline and GridSearchCV: https://towardsdatascience.com/hyper-parameter-tuning-and-model-selection-like-a-movie-star-a884b8ee8d68

Also, I will share my code here:

# Create a pipeline

pipe = Pipeline([("classifier", RandomForestClassifier())]) # Create dictionary with candidate learning algorithms and their hyperparameters

search_space = [

{"classifier": [RandomForestClassifier()],

"classifier__n_estimators": [100],

"classifier__max_depth":[10],

"classifier__min_samples_leaf":[1,2,5,10],

"classifier__max_leaf_nodes": [2,5,10]

},

{"classifier": [AdaBoostClassifier()],

"classifier__base_estimator": [DecisionTreeClassifier(max_depth = 10)],

"classifier__n_estimators": [100]

},

{"classifier": [GradientBoostingClassifier()],

"classifier__n_estimators": [100],

"classifier__max_depth": [10]

},

{"classifier": [xgb.XGBClassifier()],

"classifier__objective": ['multi:softmax'],

"classifier__n_estimators": [100],

"classifier__max_depth": [10]

}]





# create a gridsearch of the pipeline, the fit the best model

gridsearch = GridSearchCV(pipe, search_space, cv=3, verbose=1,n_jobs=-1) # Fit grid search

best_model = gridsearch.fit(X_train, y_train) # Find the best model out of the pipeline

print(best_model.best_estimator_)

The result can be various based on your input data and how you define your search_space. However, I do think that GridSearchCV and pipeline is convenient when we perform hyper-parameters tuning for the best possible model.

Finally, I will show my each model’s tuned parameters for any reference. I only worked on some very basic parameters, for more complex ones, I might update them in future if anyone is interested in.

Logistic Regression Model (C=0.1, Accuracy = 0.72, log_loss = 21.23)

Decision Tree Classifier (max_depth = 10, Accuracy = 0.70, log_loss = 12.05)

Random Forest Classifier (max_depth = 10, n_estimators = 100, Accuracy = 0.74, log_loss = 1.38)

Bagging Classifier (max_depth = 10, n_estimators = 60, Accuracy = 0.73, log_loss = 1.81)

AdaBoost Classifier (max_depth = 10, n_estimators = 90, Accuracy = 0.72, log_loss = 2.70)

GradientBoostingClassifier (n_estimators=100, Accuracy = 0.75, log_loss = 3.56)

XGBClassifier(objective = ‘multi:softmax’, n_estimators = 100, max_depth = 6, Accuracy = 0.74, log_loss = 1.71)

This is my very first post on Medium. Thank you very much for your reading!! I really hope this post can help anyone to have a better understanding on some basic machine learning model techniques.