How to approach any ML problem
By: Akshay Paliwal
Tags: Data Science, Model Drift, Metrics And Analytics, Outlier Detection, Exploratory Data Analysis

1. Explain the dataset and the problem

The first step can be just to have a high-level understanding of the data. Try to find the answers to some simple questions.

What do these features mean?

How was the data collected and what was the purpose for collecting the data ( This will help you in understanding any sampling anomalies)?

Do we already know what we are trying to find out or do we need to lay down the problem ourselves? ( what kind of a problem we are trying to solve — supervised, unsupervised, prescriptive, or re-enforcement )

2. Descriptive analytics

Post that let us try to do basic descriptive analytics. See what all features are present in our data — categorical, numerical, or ordinal in nature.

This looks like a small step but it can give you a lot of intuition about the data. Let’s have a look at the example from the Titanic data set and get a sense of how the data looks like.

Just by doing this basic descriptive feature analysis, the author of the highest voted kernel for the titanic data set on Kaggle was able to gather the following inferences.

Total samples are 891 or 40% of the actual number of passengers on board the Titanic (2,224).

Around 38% of samples survived representative of the actual survival rate at 32%.

Most passengers (> 75%) did not travel with parents or children.

Nearly 30% of the passengers had siblings and/or spouse aboard.

Fares varied significantly with few passengers (<1%) paying as high as $512.

Similar inferences can be gathered from the categorical features.

Lastly, try to do some visualization of the data just to get the sense of it — see if the data is not imbalanced or some sort of regression is identifiable from the features be it target variable or any other, etc. I try to be as vague as possible at this step.

3. Checking for Missing values

Most of the models cannot deal with missing values even if they are able to it is almost always required to deal with them on our own, as we will be able to impute or remove the records based on a more logical approach than any ML model who just treats them as a number.

Before going ahead and dealing with the missing values, try to find out how many values are missing from your data set on a feature by feature basis. Summarize and collate your figures in a structured tabular format.

Most often we will try to do one of the following things in order to deal with missing values

Deleting the records

a. Dropping the feature — There can scenarios when you find the whole feature in your data set is quite useless when you have a lot of missing values in them, but again it depends on what that feature is — a procedure code in a claim for a hospital bill can have a lot of null values and it makes sense we could still make use of the data — we should not be deleting it but on the other hand income as a feature with more than 40% values missing is not quite useful.

As a general practice, I always try to keep the feature until and unless more than 40% of data is missing but again sensing what the data represents — but always try to fight for keeping it — dropping the same means we might be losing some valuable information.

b. Dropping a row — If for a row we do not have values for most of the column it generally is a good idea to drop off that row. Maybe do use the columns that have values present in them for outlier detection or initial descriptive analysis but don’t feed it to your model. Garbage in is garbage out. Imputation

Here we are trying to impute those values with various approaches and this can vary a lot from use case to use case. There is no specific rule for imputation of the values but these are some of the approaches that I have seen.

1. Mean, Median, Mode, or impute by 0/constant

2. Linear regression

3. K nearest neighbor

4. Multivariate Imputation by Chained Equation (MICE)

5. Imputation Using Deep Learning (Datawig)

There is a really good article on imputation you can go through- the author goes through the pros and cons of the approaches :

4. Outlier detection

So as the name suggests outliers are values that will skew your model in one direction or the other due to extreme values which truly do not depict the general scenario. We try to keep the model away from learning anything from these extreme values.

Below are some of the common approaches in dealing with outlier detection.

Z-score — Some visualization techniques — quartile analysis — KDE DB Scan Isolation Forest

The below article is quite accurate and spot-on with the approaches that I have seen in practice.

5. EDA — Feature engineering — One hot encoding — PCA

Even though the last two parts — dealing with missing values and outlier detection also come under EDA, I listed them down before this section to maintain the chronological order of when that needs to be done.

Now coming to the most time-consuming part of your whole ML model creation the exploratory data analysis and feature engineering.

At this stage we need to solve the below issues in our data:

Make the data as simple for our ML model as possible — for example, if we have age as a continuous number maybe create bins of 5 and label them in an ordinal fashion. Taking care of the categorical features — find out if the categorical feature is ordinal in nature in that case we can use label encoding. If it is not then probably we need to go ahead with One hot encoding followed by PCA or SVD on the one-hot encoded columns. Another note many times one hot encoding will explode your features and even PCA or SVD on top of that will not be able to reduce the column size — in such scenarios just one hot encode those values inside the feature which you are doing a one-hot encoding on which are above a certain threshold will be useful— may just take top 50 procedures which account for 95% of the values out of the 10,000 procedures. (If you do not know any of the techniques that I listed above I would recommend you read about them at this stage)

One important note — Always after one hot encoding do check the size of your training data and test data it should match. Feature engineering — The downside in step two with one-hot encoding and PCA on top of it is the explainability of the feature goes drastically down. Imagine creating a model to detect cancer cells and not being able to tell the physician on what basis the model is predicting would look quite bad.

on the other hand, feature engineering is quite explainable — maybe for your model, BMI is a good metric instead of just the height of the individual. Feature Scaling — For an ML model who just understands numbers and maths it makes sense for us to help provide some functional knowledge. In a scenario where we want to predict the price of the house, the number of bedrooms is as important a feature as the area in sq feet. Otherwise, the model will give more importance to one feature and might totally ignore the other.

There are some models which perform well despite not being fed a scaled data like Decision tree-based models but it is almost always wise to go ahead and scale your data. There are two popular ways of scaling the data :

Normalization typically means rescales the values into a range of [0,1]. Standardization typically means rescales data to have a mean of 0 and a standard deviation of 1 (unit variance).

Please go through this article to know when to use which approach :

5. Train Test split — Split the data into training and testing data a general practice can be 80 % of the data being training and 20% being test data.

6. Fitting the model and tuning the model’s parameters and hyperparameters

Now once we are satisfied with the types of features we have and done with all that we could have done in order to get our features in the correct format, comes the point when we need to try out different models and see which one performs the best.

By this point, we would already know what kind our problem at hand is — supervised, unsupervised, prescriptive, or re-enforcement.

At this point, it is good to have a general idea of which models generally perform better on what kind of data, example — A decision tree is able to handle non-linear data quite well similarly SVM is able to handle sparse data well.

Trying different models help but always start with a simple one and see how it performs. This will help in figuring out how much information is present in our data — if even a simple model is giving a fairly decent outcome my EDA process was successful. Moreover, a gradual increase in complexity will help you explain why certain models perform better than others and in which specific scenarios. Example — For solving an NLP problem start with a simple Naive Bayes classifier and if that does not perform well then move on to LSTM or CNN.

Hyperparameters Tuning

Now comes the important part of any model training — tuning the hyperparameters. If you are new to model training I would suggest taking your time to read about how the maths work behind each model and then go on to read about what all hyperparameters are present for that model. In my experience, I have even seen many times data scientists would straight up try the grid search approach for a model and are able to reach a fairly decent outcome but that would not be the correct approach(or the best model).

While grid search is a really effective approach there might be a scenario where if you knew the maths behind the model you might have never picked up that model even for consideration, for which you will end up tuning the hyperparameters which cannot handle the type of data that you are dealing with.

There are a lot of great articles on Medium on Hyperparameter tuning for Random forest or SVM among other ML models and on how the activation function works which more often than not is the key for how well a model’s hyperparameter performs.

Regularization

Regularization comes as part of model creation and training but it is an important concept and deserves a small section to itself. This topic comes straight out of the famous bias-variance tradeoff problem and regularization tries to solve this issue for us.

This technique discourages learning a more complex or flexible model, so as to avoid the risk of overfitting.

The below post will help in your understanding of the types of regularization

7. Model metrics and validation

Once the model is trained on the data we will try to compare the models based on certain criteria(this is where you put your test data to use), this is also the point when you actually measure how your model is doing. There are various methods and below are the most common ones :

Classification problems :

Confusion Matrix Type I Error Type II Error Accuracy Recall or True Positive Rate or Sensitivity Precision Specificity F1 Score ROC Curve- AUC Score PR Curve

Regression problems:

Mean Squared Error (MSE) Root Mean Squared Error (RMSE) Mean Absolute Error (MAE) Adjusted Root Mean Squared Error

Keep in mind that the metric might be different for you in different use cases. We might want to reduce False negatives in a certain scenario and letting go of accuracy might be acceptable.

8. Dealing with overfitting or underfitting

Overfitting is a modeling error that occurs when a function is too closely fit a limited set of data points

Underfitting is said to occur in our model when it cannot capture the underlying trend of the data.

Overfitting and Underfitting destroy the accuracy of our machine learning model.

The solution to this problem is provided at the model creation step, most often the below three steps are used in order to reduce overfitting and underfitting in a model.

Drop out layers in ANN Regularization parameters Cross-validation

9. Model Drift

Finally, I would like to point out an important concept which is many times omitted from the model creation workflow. Model drift occurs when overtime the model is not as accurate as it used to be, this can be due to various reasons — changes in the patterns that the model has never seen before or there are new kinds of scenarios which our model is too naive to understand the nuances of.

We can majorly segregate model drift into two types :

Data Drift and Concept drift. I would urge you to read this small article on the same here.



https://mlopshowto.com/what-is-model-drift-and-how-it-can-negatively-affect-your-machine-learning-investment-ce7f5b9b6a37

But finally, the solution for the same is to keep your model updated and expose it to more and more data as you are able to gather it. This way the model won’t be oblivious to the changes over time.