Machine Learning Project Planning
By: Benjamin Obi Tayo Ph.D.
Tags: Machine Learning, Analytics, Project Management, Data Science, Project Planning

Before building any machine learning model, it is important to sit down carefully and plan what you want your model to accomplish. Before delving into writing code, it is important that you understand the problem to be solved, the nature of the dataset, the type of model to build, how the model will be trained, tested, and evaluated.

This article will discuss 10 important questions to consider before building a machine learning model.

(1) What are the predictor variables?

In academic training programs, we often work with very simple datasets and the problem to be solved is well defined. For instance, a homework problem could provide you with some clean dataset and it might ask you to use certain set of features as your predictor features, and one feature as your target. The problem can go even further to tell you what kind of model to build. In a real life data science task, you don’t know what the predictors variables are. Depending on the organization you are working for, you may have to work with a team of engineers (industrial dataset), doctors (healthcare dataset), etc, in order to figure out what the target variable is. For example maybe an industrial system has sensors that generated data in real time, in this case, as a data scientist, you may not have technical knowledge about the system in question. So you have to work with engineers and technicians for them to guide you to decide what features are of interest and what are the predictor variables and the target variable. Depending on the type and complexity of the problem, there might be hundreds or thousand of features in your dataset. So you will have to decide which of these are important for model building.

(2) What is the target variable? Is my target variable discrete or continuous?

Your target variable is the feature that your model is going to be trained to predict. The target variable could be continuous (e.g. predicting the price of a house in a city based on features such as number of rooms, age, square footage, zip code, number of school districts,number of employers, etc.) or discrete (predicting iris species — Iris setosa, Iris virginica and Iris versicolor, based on features such as petal length, petal width, sepal length, sepal width). Again, in a real industrial data science projects, identifying what the target variable is would require the data scientist to work with other professionals that may have domain expertise. For example the problem could be to design a machine learning model to predict when an important device in the assembly line could switch off. In this case, there are two discrete states of the device, namely, the ON and OFF states. The device might have several sensors that generate data in real time such as temperature data, air pressure data, humidity data, etc. In this case, the data scientist would have to work with engineers to determine what features should be used for training the model.

(3) Should I use clustering, classification, regression analysis, or time series analysis?

If you target variable is continuous, then regression analysis is the appropriate model to use. If your target variable is discrete, then you have to build a classification model. In the case of unsupervised learning, there is no target variable. In this case, you can use a clustering algorithm for training your model.

If your problem involves forecasting a time-dependent variable, then you may consider using time series analysis.

(4) How do I handle missing values in my dataset?

Often, the removal of samples or dropping of entire feature columns is simply not feasible, because we might lose too much valuable data. In this case, we can use different interpolation techniques to estimate the missing values from the other training samples in our dataset. One of the most common interpolation techniques is mean imputation, where we simply replace the missing value by the mean value of the entire feature column. Other options for the imputing missing values are median or most frequent (mode), where the latter replaces the missing values by the most frequent values. This is useful for imputing categorical feature values. Another imputation technique that can also be used is median imputation.

(5) Should I use normalization or standardization when bringing variables to the same scale?

It is important to bring variables to the same scale before building your model. For example, assume we would like to build a model to predict credit worthiness as follows:

credit worthiness =A + B(income) + C(credit_score)

In this example, income could range for example from $30,000 to $200,000 while credit score ranges typically from 0 to 800. Building a model without scaling your variables will produce a model that is bias towards income.

There are two common approaches to bringing different features onto the same scale: normalization and standardization. Those terms are often used quite loosely in different fields, and the meaning has to be derived from the context. Most often, normalization refers to the rescaling of the features to a range of [0, 1], which is a special case of min-max scaling.

Using standardization, we center the feature columns at mean (mu) with standard deviation (sigma) so that the feature columns take the form of a normal distribution, which makes it easier to learn the weights. Furthermore, standardization maintains useful information about outliers and makes the algorithm less sensitive to them.

(6) Should I reduce the dimensionality of my feature space?

Complex datasets could contain hundreds or thousand of features. training a model on a dataset with so many features could be very complex, especially when the features are highly correlated. In such a case, it could be worthwhile to reduce the dimensionality of the system. This could be done using the covariance matrix plot (Feature Selection and Dimensionality Reduction Using Covariance Matrix Plot). Advanced methods for dimensionality reduction included PCA, LDA, or LASSO.

(7) How do I tune hyperparameters in my model?

A machine learning model is not a one-size fit all model. A model has so many hyperparameters (Model Parameters and Hyperparameters in Machine Learning — What is the difference?). A model has to be trained against different hyperparameters in order to obtain the model with optimal quality.

(8) How do I evaluate my model to detect biases in the dataset?

Some common metrics for model evaluation are are mean squared error (MSE), mean error (ME), mean absolute error (MAE), r-squared value (R2 score), accuracy (for classification problems), confusion matrix, positive predictive value (PPV), area under ROC (AUC) score, etc.

(9) Should I use ensemble methods where I train using different models then perform an ensemble average, e.g. using classifiers such as SVM, KNN, Logistic Regression, then average over 3 models?

For example if we would like like to build a model to determine if a customer is credit worthy or not, then we can use different classifiers such as logistic regression, KNN, SVM, decision tree, etc., then evaluate our model by calculating the R2 scores for each of these classifiers. We may choose the classifier with the optimal performance or calculate average R2 score across different classifiers.

(10) How do I select the final model?

To select final model, we may choose the method with the least generalization error, that is, the model with the optimal performance when model is used with unseen (or test) date.

In summary, before building any machine learning model, it is important to sit down carefully and plan what your project goals and objectives are. That way, you will ensure your model when completed, will be the best model ever, to the best of your abilities.