A hands on an end to end data science project using Python | by Jonathan Leban | Towards Data Science

Machine learning, data science, prediction, you may have heard a lot of these words, but what exactly are they? In what situation is it useful?Born in the 1960s, Machine Learning is a branch of artificial intelligence that provides systems with the ability to learn models and make predictions such as predicting the price of a house based on its characteristics (location, area, number of rooms...) or predicting whether a tumor is benign or not. In the house problem, we want to predict a continuous value and this type of problem belongs to what we call the regression problem. In the case of the tumor, the result is a binary value or more generally a discrete value and this type of problem is called the classification problem. When tackling a data science problem, it is extremely important to know the type of outcome you are looking for to know whether you are tackling a regression or classification problem.
In this article, I write a guide to conducting an end-to-end machine learning project. As you can see in the graph below, which shows the different steps of a data science project problem, there are 6 steps. We will delve into each of them throughout the article.
To illustrate the different steps of a data science project, I will use a dataset which describes a sample of pulsar candidates collected during the High Time Resolution Universe Survey. Pulsars are a rare type of Neutron star that produce radio emission detectable here on Earth. They are of considerable scientific interest as probes of space-time, the inter-stellar medium, and states of matter . As pulsars rotate, their emission beam sweeps across the sky, and when this crosses our line of sight, produces a detectable pattern of broadband radio emission. As pulsars rotate rapidly, this pattern repeats periodically. Thus pulsar star involves looking for periodic radio signals with large radio telescopes. Each pulsar produces a slightly different emission pattern, which varies slightly with each rotation . Thus a potential signal detection known as a ‘candidate’, is averaged over many rotations of the pulsar, as determined by the length of an observation. In the absence of additional info, each candidate could potentially describe a real pulsar. However in practice, almost all detections are caused by radio frequency interference (RFI) and noise, making legitimate signals hard to find.
Here, using machine learning models is totally legitimate to predict whether a start is a pulsar star or not as the purely physics models have a lot of barriers to overcome. By using Machine Learning models we can achieve accurate results bending the complexity of the problem.Let’s see how we can do it!You can find the dataset here.
In this project, the idea is to predict if a star is a pulsar star or not. We can see clearly that the result of our models will be binary: 1 if the star is a pulsar star and 0 otherwise. Thus, we will implement classification models to answer this problem.
As we use Python, we will use the Pandas library to extract the data from the csv file. For the data visualization, the library seaborn is a very good one as a lot of tools for data science are already implemented. The matplotlib.pylab library is also useful to plot graphs.
import numpy as np import pandas as pd # for data visualizationimport seaborn as sns import matplotlib.pylab as plt
Then we can extract the data from the csv file and take a look at the different characteristics. One action you should always do is to check the number of elements in each characteristic. If the number of elements is different for some characteristics, you have to apply certain operations to fill the gap or if there are too many missing values, you have to abandon that characteristic.
pulsar_stars = pd.read_csv("pulsar_stars.csv") # read the data# to check if we have to clean the data print(pulsar_stars.apply(pd.Series.count))
We can see that they are 8 features which define a star and the target class is a binary value (1 if the candidate star is a pulsar star and 0 otherwise). All the different features have the same number of elements so no operations to fill the gap are necessary.
The next step is to split the data into the data that we will use to train the machine learning algorithm and the data we will use to test the performance of each of them.
# we set the seed for the random cursor random.seed(10)  # we split the data X = pulsar_stars.drop(['target_class'], axis = 1) y = pulsar_stars['target_class'] X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.8, random_state = 10) X_columns = X_train.columns
Next we can take a look on the correlation between the different features but also on the link between one feature and the target class in order to see the impact on the result.
#correlation plotcorr = pulsar_stars.corr()sns.heatmap(corr)
We notice that the excess kurtosis of the integrated profile is the more correlated feature with the target_class. Now, we will look into the distribution of each variable to see if we have to apply a transformation to make it more valuable.
#create numeric plotsnum = [f for f in X_train.columns if X_train.dtypes[f] != 'object']nd = pd.melt(X_train, value_vars = num)n1 = sns.FacetGrid (nd, col='variable', col_wrap=3, height = 5.5, sharex=False, sharey = False)n1 = n1.map(sns.distplot, 'value')
Most of the variables have a gaussian distribution (you may think the data, for most of the features, follows a right-skewed distribution but it is only because the number of data is not big enough).
Now that we have a better understanding of the data we can pre-process it in order to feed our machine learning algorithms.
Since different characteristics have completely different scales, we need to standardize our data. This step is not necessary when the characteristics have the same scale, except for some algorithms such as PCA .
They are different ways to normalize the data:
scaler = StandardScaler()X_train = scaler.fit_transform(X_train)X_test = scaler.transform(X_test)
In this part, we implement different machine learning models that deal with classification problems and define measures to evaluate their performance.For the machine learning models, I will use the decision tree algorithm, the logistic regression algorithm, the random forest algorithm and the K-Nearest Neighbors algorithm. I will use time, precision, recall, weighted f1 score and AUC score to evaluate the models. I will explain my choices and the meaning of each measure in more detail.
Some of these models have hyperparameters, i.e. parameters whose value is set before the start of the learning process. One method is to establish lists of values that the hyperparameters can take and to search by grid according to as precise a metric as possible in order to find the best model. Another method would be to use Bayesian optimization (a more efficient method).
# we start by DecisionTreeClassifierdc = DecisionTreeClassifier(max_depth = 4)dc.fit(X_train, y_train)params = {'max_depth' : [2,4,8]}dcgrid = GridSearchCV(estimator = dc, param_grid = params, cv = KFold(5, random_state = 10) , scoring = 'accuracy')dcgrid.fit(X_train, y_train)# Then by LogisticRegressionlg = LogisticRegression(C=0.001, solver='liblinear')lg.fit(X_train, y_train)params = {'C':[0.01,0.1,1,10]}lggrid = GridSearchCV(estimator = lg, param_grid = params, cv = KFold(5, random_state = 10), scoring = 'accuracy')lggrid.fit(X_train, y_train)# then by RandomForestClassifierrf = RandomForestClassifier(n_estimators = 10, max_depth = 10)rf.fit(X_train, y_train)params = {'n_estimators' : [10, 20, 50, 100], 'max_depth' : [10, 50]}rfgrid = GridSearchCV(estimator = rf, param_grid = params, cv = KFold(5, random_state = 10), scoring = 'accuracy')rfgrid.fit(X_train, y_train)# then KNeighborsClassifierkn = KNeighborsClassifier(n_neighbors = 10, p = 2)kn.fit(X_train, y_train)params = {'n_neighbors' : [2, 5, 10, 50], 'weights' : ['uniform', 'distance'], 'p' :[1,2]}kngrid = GridSearchCV(estimator = kn, param_grid = params, cv = KFold(5, random_state = 10), scoring = 'accuracy')kngrid.fit(X_train, y_train)
Then we find the best models and we can also see the time needed to train these models which can be a metric used to choose our final model.
# we define the best models dc_best = dcgrid.best_estimator_lg_best = lggrid.best_estimator_rf_best = rfgrid.best_estimator_kn_best = kngrid.best_estimator_# now we will launch each model and see the time and the performance of each start = time.time()dc = dc_bestdc_best.fit(X_train, y_train)end = time.time()print('time for Decision Tree Classifier = ', end - start, 's')performance_df['Decision_tree']['time(s)'] = end - start# Then by LogisticRegressionstart = time.time()lg = lg_bestlg_best.fit(X_train, y_train)end = time.time()print('time for Logisitic Regression= ', end - start, 's')performance_df['Logistic_regression']['time(s)'] = end - start# then by RandomForestClassifierstart = time.time()rf = rf_bestrf_best.fit(X_train, y_train)end = time.time()print('time for Random Forst Classifier = ', end - start, 's')performance_df['Random_forest']['time(s)'] = end - start# then KNeighborsClassifierstart = time.time()kn = kn_bestkn_best.fit(X_train, y_train)end = time.time()print('time for K_Neighbors Classifier = ', end - start,'s')performance_df['K-NNeighbors']['time(s)'] = end - start
Now we can also evaluate the models along another metric which is the recall. The recall, also known as sensitivity, measures the strength of the model to predict a positive outcome — the proportion of 1s correctly identifies.
# we will calculate the scores of each model y_predict_dc = dc_best.predict(X_test)accuracy = accuracy_score(y_test, y_predict_dc)recall = recall_score(y_test, y_predict_dc)performance_df['Decision_tree']['accuracy'] = accuracyperformance_df['Decision_tree']['recall'] = recally_predict_lg = lg_best.predict(X_test)accuracy = accuracy_score(y_test, y_predict_lg)recall = recall_score(y_test, y_predict_lg)performance_df['Logistic_regression']['accuracy'] = accuracyperformance_df['Logistic_regression']['recall'] = recally_predict_rf = dc_best.predict(X_test)accuracy = accuracy_score(y_test, y_predict_rf)recall = recall_score(y_test, y_predict_rf)performance_df['Random_forest']['accuracy'] = accuracyperformance_df['Random_forest']['recall'] = recally_predict_kn = dc_best.predict(X_test)accuracy = accuracy_score(y_test, y_predict_kn)recall = recall_score(y_test, y_predict_kn)performance_df['K-NNeighbors']['accuracy'] = accuracyperformance_df['K-NNeighbors']['recall'] = recall
Then, we can compute the confusion matrix which is an error matrix:
# generate confusion matrix for Decision Tree classifier conf_mat_dc = confusion_matrix(y_test, y_predict_dc) # put it into a dataframe for seaborn plot function conf_math_dc_df = pd.DataFrame(conf_mat_dc)  # Use a seaborn heatmap to plot confusion matrices # The dataframe is transposed to make Actual values on x-axis and predicted on y-axis # annot = True includes the numbers in each box # vmin and vmax just adjusts the color value fig, ax = plt.subplots(figsize = (7,7)) sns.heatmap(conf_math_dc_df.T, annot=True, annot_kws={"size": 15}, cmap="Oranges", vmin=0, vmax=800, fmt='.0f',              linewidths=1, linecolor="white", cbar=False, xticklabels=["no pulsar star","pulsar star"],              yticklabels=["no pulsar star","pulsar star"]) plt.ylabel("Predicted", fontsize=15) plt.xlabel("Actual", fontsize=15) ax.set_xticklabels(["no pulsar star","pulsar star"], fontsize=13) ax.set_yticklabels(["no pulsar star","pulsar star"], fontsize=13) plt.title("Confusion Matrix for 'Decision Tree' Classifier", fontsize=15)  plt.show()print("")print(classification_report(y_test, y_predict_dc))
After doing this for the other models, we calculate a final measure which is the AUC curve. The AUC — ROC curve is a performance measure for classification problems at different thresholds. ROC is a probability curve and AUC is the degree or measure of separability. It indicates how well the model is able to distinguish between classes. The higher the AUC, the better the model is able to predict that 0s are 0s and 1s are 1s.
#Plotting the ROC curve#Generating points to plot on ROC curve (logistic model)dc_best_prob = dc_best.predict_proba(X_test)fpr_logis, tpr_logis, thresholds_logis = roc_curve(y_test, dc_best_prob[:, 1])fig, ax = plt.subplots(figsize = (10,7))#plotting the "guessing" modelplt.plot([0, 1], [0, 1], 'k--')#plotting the logistic modelplt.plot(fpr_logis, tpr_logis)plt.fill_between(fpr_logis, tpr_logis, alpha=0.2, color='b')plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')AUC = roc_auc_score(y_test, dc_best_prob[:, 1])plt.title('Decision Tree Classifier ROC curve: AUC={0:0.3f}'.format(AUC))plt.show()performance_df['Decision_tree']['AUC'] = AUC
Finally we end up with the following table:
Depending on our problem, the metrics don’t have the same weight on our results. The AUC score is the most important metric in our model as predicting a star as a non super star and making a mistake is what we want to avoid. Thus the logistics regression seems to be the most efficient model.
Based on the model we have chosen, we can vet the importance of the different features on the results.
# thus the logistic regression is the best model : we will keep it # Now we want to see the importance of each feature on the result  dataframe_importance = pd.DataFrame()columns = X_train.columnsimportances = np.abs(lg_best.coef_[0])for i in range(len(columns)):    dataframe_importance[columns[i]] = [importances[i]]dataframe_importance.insert(0, '', 'Importance features')dataframe_importance.head(10)
Sometimes we may have time constraints or computing power limits and we need to make the model easier. One way to do this is to make a feature selection. We can do this by using a statistical method such as the chi-square test or by using a dimensionality reduction algorithm such as PCA. Then we re-evaluate the model and we can conclude whether or not the constraints we have in terms of time and precision are taken into account.
Do not hesitate to spend a lot of time defining the problem. Once you become accustomed to the implementation techniques of data science projects, you will realize that this is the easiest part of your project.Another part that is quite tricky is data extraction and data cleaning. In the project I presented to you in this article, the data was super clean, but this is almost never the case.Finally, I hope you enjoyed the article and that this is a practical science, the best way to get familiar with it is to practice it. Choose a topic that interests you and put into practice what you’ve learned!
You can find the whole project with some other implementation as neural networks on my github.
PS: I am currently a Master of Engineering Student at Berkeley, and if you want to discuss about the topic, feel free to reach me. Here is my email.