Build a Machine Learning Model to a Practical Use-Case From Scratch | by Yasas Sandeepa | Towards Data Science

Machine Learning is an enormous area. It is empowering technology for allowing us to develop software solutions much faster than before and currently the state-of-the-art solution for a wide range of problems. We can apply it to almost every domain. In this article, we are going to study in-depth how the process for developing a machine learning model to a practical use case.
In the article, we will be discussing :
Problem Definition and Data gathering
Data Preprocessing
Data Transformation
Feature Encoding
Scaling and Standardization
Feature Engineering
Dimension reduction with Principal Component Analysis
Regression
Accuracy measures and Evaluation techniques
Assumption:- I believe you have prior knowledge in python programming as well as basic libraries related to machine learning.
Use-Case:- Is there a relationship between humidity and temperature? What about between humidity and apparent temperature? Can you predict the apparent temperature given the humidity?
As the first thing, we have to understand the problem, the environment that problem lies in, and gather the domain knowledge regarding the scenario.
The problem mainly asks to predict the apparent temperature given the humidity. What is this apparent temperature?
Apparent temperature is the temperature equivalent perceived by humans, caused by the combined effects of air temperature, relative humidity and wind speed. -Wikipedia
This reveals one important factor. Not only humidity, but air temperature and wind speed are also affecting the apparent temperature. So for a given scenario, we need to find out what are the direct as well as indirect(hidden) facts that going to affect our problem.
A good example of this is the banking sector. Imagine you need to identify a customer whether he is eligible to receive a loan or not. (performer or non-performer) You cannot just predict it by only looking at the previous bank transactions. You need to analyze, what is the domain he working on, if he is a cooperate customer what are the other industries that producing profit to him (Although he fails in one domain, he may rise in another domain), whether he has any political supports, such as a broad area (indirect) you may need to cover in order to provide a good prediction. So keeping that in mind, let’s go to our problem.
Then we need a data set to tackle this problem. Kaggle website provides a massive collection of data resources for anyone and we easily can find a data set from there. Most importantly, on Kaggle you can learn from others in a variety of problem contexts. I will be using the Kaggle — Weather History Data set to analyze this problem.
www.kaggle.com
Note: All the coding was done in a Google Colab python notebook and you can find it under the resources section.
As a data scientist, you need to have a clear idea of your data set and the complexity of data. Let’s first visualize the data, just to get some insight.
The below code snippet will load the weather data CSV file into a pandas data frame and display the first 5 rows.
weatherDataframe = pd.read_csv(‘weatherHistory.csv’);weatherDataframe.head();
You can analyze the data set furthermore by weatherDataframe.info() method and then you will see there are 96453 entries with 12 columns.
We can identify our target column as the apparent temperature and the rest of the columns as the features.
Preprocessing is the most important part of machine learning. The success of our model highly depends on the quality of the data fed into the machine learning model. Real-world data is usually dirty. It contains duplicates missing values outliers, irrelevant features, non-standardized data..etc. So we need to clean the data set in a proper way. Let’s see step by step, how we going to achieve it.
First, we need to find any unique fields in the data set. If found any, we should drop them because they are not useful when recognizing patterns. If a data set contains columns with very few unique values, that will provide a good basis for data cleaning. Also, if the columns have so many unique values, such as ID number, email (unique for every data point), we should remove them.
# summarize the number of unique values in each columnprint(weatherDataframe.nunique())
As you can see in the results, the Loud cover column only has a single value (0). So we can drop that column entirely. Also, we need to analyze what are the unique values row-wise. To analyze that, we can calculate the number of unique values for each variable as a percentage of the total number of rows in the data set. This is a custom function made for that.
Formatted date giving 100% unique ratio to the total data frame. So we can drop that column. Also by looking at the data set we can conclude that the Daily summary and Summary columns have a great similarity. So we can remove one of them. So I’ll keep the Summary.
The next thing that, there may be duplicate rows. So we need to identify rows that contain duplicate data and delete them.
There is one important thing to remember. When we removing rows, this will cause the indexes are to be different. As an example, if you remove row 18 from the data frame, now the indexes will be ..16,17,19,20.. like this. This will ultimately create a difference between the actual row count and the last index. In order to avoid this, we need to reset indexes. Then it will correct the order.
There can be a number of missing values in the data set. Mainly, there are two things that we can do for missing values, it’s either drop or do imputation to replace them. Imputation is replacing missing values with mean or median values. But the problem with this imputation is, it can lead to causing a bias in the data set. So it is advised to manipulate the imputation carefully.
If the number of cases of missing values is extremely small, the best option is to drop them. (If the number of the cases is less than 5% of the sample, you are safer to drop them)
# Check for any missing valuesweatherDataframe.isnull().values.any()# This gives - True#Getting the summary of what are missing value columnsweatherDataframe.isnull().sum()
As you can see, there are 517 missing values in the Precip type column. So, let’s examine the overall probability to find whether we can drop those columns.
weatherDataframe['Precip Type'].isna().sum()/(len(weatherDataframe))*100
This returns the result as 0.536 and it’s a very low percentage compared to the total data set. So we can drop them. It will ensure no bias or variance is added or removed, and ultimately results in a robust and accurate model.
# make copy to avoid changing original datanew_weatherDf = weatherDataframe.copy()# removing missing valuesnew_weatherDf=new_weatherDf.dropna(axis=0)# Resetting Indexesnew_weatherDf=new_weatherDf.reset_index(drop=True)
An outlier is a data point (or can be a small number of data points) that is significantly stay away from the mainstream.
But if you are seeing many points are staying away from the mainstream, they are not considered as outliers and they could be some kind of cluster pattern or something related to an anomaly. In such cases, we need to treat them separately.
There are numerous methods to discover outliers.
Box Plots —glancing at the variability outside the upper and lower quartiles
Scatter Plots — using Cartesian coordinates of two data columns
Z- Score — using a mathematical function
I’ll use the Box Plot method for detecting them visually.
You can see there are some outliers exists in the data set. So we should examine them one by one to treat them separately. The best option is to remove them. You can apply mean and median values considering the data distribution. But there is a high risk of making the data set bias. So if you want to do that, handle it very precisely.
I’ll demonstrate removing outliers in one column. You can see the rest of the things in my python notebook. As you have seen in the plot, there is an outlier in the pressure column. It’s a pressure and it would not become zero under normal conditions. This may due to an error in data entry or a problem with the collected equipment. So we should remove them.
There is one thing we need to keep in mind before doing transformations. That is data Leakage. We need to keep a test data set in order to estimate our model. But if we do the below transformation steps to all the data in the data set and then split it out (as training and testing set), we have committed the sin of data leakage.
Don’t forget that testing data points represent real-world data. So the model should not see that data. If so, the consequence will be over-fitting your training data and having an overly optimistic evaluation of your model’s performance on unseen data.
In order to avoid that, we should split our data set into train and test sets now and do the transformation steps. This will ensure no peeking ahead. Otherwise, information from the test set will “leak” into your training data.
First, we need to define our features and the target (what we going to predict.) The use case tells us to predict the Apparent Temperature. So it will be our target. The rest of the columns we can take as features.
features_df= new_weatherDf.drop('Apparent Temperature (C)', 1)target = pd.DataFrame(new_weatherDf['Apparent Temperature (C)'], columns=["Apparent Temperature (C)"])
Now we can split them out as training and test tests to 80% — 20% ratio. random_state ensures that the splits that you generate are reproducible. It always splits in the same random order and will not change for every run.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(features_df, target, test_size = 0.2, random_state = 101)
Also, keep in mind to reset the indexes as it may lead to confusion in the latter parts.
X_train=X_train.reset_index(drop=True)X_test=X_test.reset_index(drop=True)y_train=y_train.reset_index(drop=True)y_test=y_test.reset_index(drop=True)
Okay. We are all set for transformations!
The data that you have cleaned previously, may not be in the right format or right scale and it will be difficult to understand for the model. Therefore we need to do data transformations.
Skewness is the asymmetry in a set of data that deviates from a normal distribution. It also can define as a distortion of the symmetrical bell curve. If the curve is shifted to the left (tail in the right), it is right-skewed and if the curve shifted to the right (tail in the left), it is left-skewed. We should apply proper transformations in order to bring it to a symmetrical shape. This will lead to increase the accuracy of our model.
As per the image shown above, we need to apply Log-transformation to the right-skewed data and Exponential-transformation to the left-skewed data in order to bring the data set to symmetrical.
But how we detect the skewness? For that, we have Q-Q plots (quantile-quantile plots) and histograms. Let’s analyze with our dataset.
import scipy.stats as stats# Temperature (C). - Trainingstats.probplot(X_train["Temperature (C)"], dist="norm", plot=plt);plt.show();X_train["Temperature (C)"].hist();
As you can see in the Q-Q plot, most of the data lie in the red line. Also, the histogram is not showing any skewness and it’s symmetric. So we don’t need to transform this column.
If you analyze the Humidity column, you will notice a left-skewed distribution in there. So we need to apply Exponential transformation to make it symmetrical.
# create columns variables to hold the columns that need transformationcolumns = ['Humidity']# create the function transformer object with exponential transformationexp_transformer = FunctionTransformer(lambda x:x**3, validate=True)# apply the transformation data_new = exp_transformer.transform(X_train[columns])df_new = pd.DataFrame(data_new, columns=columns)# replace new values with previous data frameX_train.Humidity=df_new['Humidity']
Now you will see a nice symmetric distribution. So likewise we can do the transformation to all the necessary columns. I’ll show one another important transformation. All the transformations are explained in my python notebook. Refer it for more clarifications.
If you analyze the wind speed column, you would see a right-skewed distribution. To bring it symmetrical we need to apply log transformation. But here is a special case that there are 0 value data points can see in the column. If we apply log transformation, that 0 value data points will be replaced with minus infinity. To avoid that, we can apply log(x+1) transformation.
# create columns variables to hold the columns that need transformationcolumns = ['Wind Speed (km/h)']# create the function transformer object with logarithm transformationlogarithm_transformer = FunctionTransformer(np.log1p, validate=True)# apply the transformation data_new = logarithm_transformer.transform(X_train[columns])df_new = pd.DataFrame(data_new, columns=columns)# replace new values with previous data frameX_train['Wind Speed (km/h)']=df_new['Wind Speed (km/h)']
There are categorical (text) and numeric data in our dataset. But most of the models only accept numeric data. So we need to convert those categorical data, or text data columns into numbers. To do this, we can use two encoders.
Label EncoderOne Hot Encoder (OHE)
Label Encoder
One Hot Encoder (OHE)
Note — Some algorithms can work with categorical data directly like Decision Trees. But most of them cannot operate on label data directly.
Label Encoder encodes the classes with a value between 0 and n-1 where n is the number of distinct classes(labels). If a class replicates, it will assign the same value as assigned earlier.
Let’s take a simple example for clarifying things. There is a column with having different countries. If we apply a label encoder to that, it will provide the below output.
It will assign a unique value from 0 to n classes. As you can see here, label encoding uses alphabetical ordering. Hence, Brazil has been encoded with 0, India with 1, and Italy with 2. If the same class appears in a different row, it will assign the same value as assigned earlier.
But the problem in the label encoding is, it can lead to find relationships between the encoded values. As an example, if the countries have marked 1,2,3,4.. categories, the model can create a pattern like the label 2 country is more powerful than the label 1 country and label 3 country is more powerful than the both 1 and 2 like that. But actually, there is no relation, of any kind, between the countries.
So by label encoding, it will confuse the model into thinking that a column has data with some kind of order or hierarchy. To overcome this problem, we use One hot encoding.
One Hot Encoding takes a column that has categorical data and then splits the column into multiple columns. The classes are replaced by binaries (1s and 0s), depending on which column has a particular class.
Look at the below example. First, it will create separate columns for every class, and then it will assign a binary value of 1 for the presented class and 0 for others as row-wise. E.g: The first row represents Sri Lanka. So value 1 only applied for the Sri Lankan column and the rest of the columns in the first row will get a value of 0.
This will not generate a pattern among classes because the result is binary rather than ordinal and that everything sits in an orthogonal vector space. But it can lead to a massive number of columns and ultimately lead to a curse of dimensionality. But if we apply a Dimensional reduction technique like PCA, it will resolve the problem. Also, It’s advised to use the OHE over Label Encoder.
In our data set, now we have 2 categorical columns, Precip Type and Summary. So we can apply encoding to both columns.
We are dealing with a training and a testing set. So we need to keep in mind one important thing. When you are training a model, you will use the training data set. For a trained model, we need to input the same scale of data we used in training. So how do we get that previous scale? Here comes the interesting part. To carter that issue, there are two methods in most libraries. That is fit() and transform()
You need to first apply fit() method on your training data set to only calculate the scale and keep it internally as an object. Then you can call transform() method to apply the transformation for both training and testing datasets. That will ensure the data is transformed on the same scale.
I applied one-hot encoding for the summary column. For the precip type column, we can apply categorical coding (similar to label encoding) as there are only two classes.
X_train['Precip Type']=X_train['Precip Type'].astype('category')X_train['Precip Type']=X_train['Precip Type'].cat.codes
Our final training data frame will look as follows.
We use Standardization to center the data (make it have zero mean and unit standard deviation). What actually doing by this is, data will subtract by the mean and then divide the result by the standard deviation. ( x′=(x−μ)/σ )
As we did previously, we need to apply standardization for both training and testing data. I hope you have remembered the important fact that I mentioned in the feature encoding section.
You have to use the exact same two parameters μ and σ (values) that you used for centering the training set.
So in sklearn’s StandardScaler provides two methods to achieve this. Hence, every sklearn’s transform’s fit() just calculates the parameters (e.g. μ and σ in this case) and saves them as an internal object's state. Afterward, you can call its transform() method to apply the transformation to any particular set of data.
Another very important thing is that we don’t do any kind of standardization to previously encoded categorical variables. So keep them aside before applying the standardization.
to_standardize_train = X_train[['Temperature (C)', 'Humidity','Wind Speed (km/h)','Visibility (km)','Pressure (millibars)']].copy()
Note: I purposefully left the Wind Bearing column as it shows a large variation in data points. (0–360 degrees) I will be discretizing that column as the next step after standardization.
Before we do the standardization, we can examine how the histograms look like to understand the x scales.
As you can see, they are in different x scales. Now let’s do the standardization.
# create the scaler objectscaler = StandardScaler()# Same as previous -  we only fit the training data to scalerscaler.fit(to_standardize_train)train_scaled = scaler.transform(to_standardize_train)test_scaled = scaler.transform(to_standardize_test)standardized_df_train = pd.DataFrame(train_scaled, columns = to_standardize_train.columns)standardized_df_test = pd.DataFrame(test_scaled, columns = to_standardize_test.columns)
Now you can see all the x axes have come to a standard scaler. You can apply the standardization for the target variable as well.
From Data Discretization, we can transform continuous values into the discrete form. This process can be used to reduce the large range of a data set into a small range that can smooth out the relationships between observations.
As an example, if we take age, it can be varied probably up to 1 to 100 years. It is a large range of data and the gaps between data points will also be large. To reduce that we can discretize that column by dividing it into meaningful categories or groups as follows :Under 12 (kids), between 12–18 (teens), between 18–55 (adults), and over 55 (old)
Therefore, discretization helps make the data easier to understand to the model. There are various methods to do the discretization. (Decision trees, Equal width, Equal-Frequency..etc) Here I am using the K-means discretization.
In the data set, we have a feature called Wind Bearing. If you remember, I kept that column aside previously because of this reason. It provides 0–360 degrees coverage about the wind direction. It’s a large range. We can discretize this into eight bins representing the actual wind directions. (North, North-East..etc)
The n_bins argument controls the number of bins that will be created. We can see the output using histograms.
So all the preprocessing and transformation steps are completed. Now we need to further improve it using feature engineering.
Actually we have already done some feature engineering processes in transformations. But here comes the big part. Here we have to manually decide what are the relations between the features and how to select the optimal feature set for training the model. We need to find the most significant features as irrelevant or partially relevant features can negatively impact our model performance.
If we take a practical example like the banking industry, there will be millions of features that need to be analyzed, and also the data points will be immense. Imagine how complex will be the data and how difficult be the data handling. It will lead data analysis tasks to become significantly harder as the dimensionality of the data increases. This phenomenon is called as the curse of dimensionality.
Also, it can lead to another problem that it may cause machine learning models to massively overfit and makes the model too complex. The model will be unable to generalize well on unseen data. Higher dimensions lead to high computation/training time as well as difficulty in visualizing. So it is essential to do a dimensional reduction.
Before doing the dimension reduction, we should keep in mind mainly 2 things.1. Original data should be able to approximately reconstructed.2. Distance between data points should be preserved.
We can use the Correlation Matrix with Heatmap to easily identify significant features. It shows how the features are related to each other or the target variable.
Correlation can be proportional or inversely proportional. When it is inversely proportional, it displays with a (-) sign. Let’s plot the Heatmap to identify which features are most related to our target variable and what are the features with high correlations.
correlation_mat = features_df.iloc[:,:6].corr()plt.figure(figsize=(10,8))sns.heatmap(correlation_mat, annot = True, cmap="RdYlGn")plt.title("Correlation matrix for features")plt.show()
As you can see in the heat-map, there are high correlations between:* temperature and humidity* temperature and visibility* humidity and visibility
Any 2 features (independent variables) are considered to be redundant if they are highly correlated. Normally it is recommended to remove such features as it will stabilize the model. But we cannot conclude that these features will not be worth enough. It should decide after training the model. Also dropping a variable is highly subjective and should always be done keeping the domain in mind.
Look at the last column. Apparent Temperature is highly correlated with Humidity followed by Visibility and Pressure. So we can state that they are the most significant features. We should keep these variables as that will cause the model to perform well.
Here we get some kind of contradiction. From figure 1 we identified some features that are highly correlated and they need to be removed as well as figure 2 implies the same set of features are highly correlated with the target variable and we should keep them. So we need to perform several experiments with the model before coming to a conclusion. But don’t worry! We can apply a way better method called PCA to reduce the dimensions.
PCA is a powerful dimensionality reduction algorithm that identifies patterns in the dataset, based on the correlations between the features. By looking at the variance_ratio explained by the PCA object, we can decide how many features (components) can be reduced without affecting the actual data.
PCA actually does is, it projects the data into dimensions in eigenvector space. Then it will identify the most important dimensions that preserve the actual information. Other dimensions will be dropped. So the important part here is, it does not remove the entire feature column but takes the most of information from it and projecting to other dimensions. Finally, it will come as an entirely new set of dimensions (columns).
If you are interested to know PCA in a detailed manner, I recommend a very good article written by Matt Brems and published in towardsdatascience.com
towardsdatascience.com
We will dig deeper into this technique in future articles. So let’s apply PCA to our dataset.
from sklearn.decomposition import PCApca = PCA()pca.fit(X_train)
First, we have to identify the number of dimensions that we can reduce up to. For that, we analyze the explained_variance_ratio_ in the PCA object that we fitted earlier.
pca.explained_variance_ratio_
Here we have a sorted array (vector) of the variance explained by each dimension. We have to keep the high variance dimensions and remove the rest of the dimensions. If the sum of high variance dimensions (n_components) is over 95%, it will be a good number.
By analyzing the vector, we can identify first 7 dimensions are preserving over 95% of the information in the data set. So from that we able to reduce the 33 dimensions into 7 dimensions with a loss of only 5% of the information.
Now we can apply PCA again by adding the no of components (dimensions) that needed to be remain.
pca = PCA(n_components=7)pca.fit(X_train)X_train_pca = pca.transform(X_train)X_test_pca = pca.transform(X_test)
All set! Let’s do the modeling to understand how well it performs on our dataset keeping those core concepts in mind.
Quick reminder- You can compare the model with or without PCA and check the accuracy values. If the accuracy in PCA is less than the original one, you can try out increasing the n_components one by one.
Before you apply any model, you need to understand the domain properly, the complexity of the domain, and whether the given dataset represents the actual complexity of your problem. Also if the problem is related to a business domain, you need to understand the business process, principles, and theories to some extent.
When modeling, there are numerous models available to use according to your problem. I’m using Multiple Linear Regression model as we need to predict a one response variable (target) given multiple explanatory variables (features). This model basically doing is fitting a linear equation to observed data. That line should give a minimum deviation to the original points.
Formally, the model for multiple linear regression, given n observations:
Let’s apply the model to our dataset.
from sklearn import linear_modellm = linear_model.LinearRegression()model2 = lm.fit(X_train_pca,y_train)
For the prediction, we already have prepared a test dataset. If you want to predict for entire new data, you need to apply all the preprocessing and transformation steps (transformation should be done according to previously fitted values) as well as the PCA in order to get the correct prediction.
predictions = lm2.predict(X_test_pca)y_hat_pca = pd.DataFrame(predictions, columns=["Predicted Temparature"])
You can see, our model predicts well for unseen data. Further, this can be visualized in a plot.
import matplotlib.pyplot as pltplt.figure(figsize=(20, 10))# Limiting the data set to 100 rows for more clearanceplt.plot(y_hat_pca[:100], label = "Pred")plt.plot(y_test[:100], label = "Actual")plt.title('Comparison of Prediction vs Actual - With PCA')plt.legend()plt.show()
As we discussed earlier, linear regression tries to fit a line that gives a minimum deviation to the original points. So there is an error between the actual and the predicted. By measuring this error (loss), we can check the accuracy of our model.
We measure this using loss functions. When the loss is minimal, we can decide that the model has very good accuracy. There are several loss functions available:
MSE — Mean Square Error
MAE — Mean Absolute Error
RMSE — Root Mean Square Error
When recording the efficiency of the model as an output, it’s recommended to use the RMSE over MSE as MSE is widely used for removing the error in the training stage. (better for model optimization.) MAE is not recommended for testing the accuracy as it does not gives us an idea of the direction of the error is.
Rather than these, there are various evaluation techniques as F1-Score, Area Under Curve, Confusion Matrix...etc to measure the accuracy of models. You can search more about these things to improve your knowledge. I intend to write a separate article on that as well.
Okay. Let’s cheak the MSE and RMSE for our model.
We use the R-squared value to evaluate the overall fit in the linear model. It’s between 0 and 1. Higher the values, the better the performance. Because it means that more variance is explained by the model.
#Percentage of explained variance of the predictionsscore_pca=lm2.score(X_test_pca,y_test)score_pca#This has given - 0.9902083367554705
Those are actually pretty good values. But before coming to a conclusion, let’s analyze the weight factors as well.
#W parameters of the modelprint(lm2.coef_)
The weights also can be used to evaluate the model. If the weights are giving higher values, that indicates your model is overfitted. If so, you need to revisit your preprocessing and transformation steps to analyze whether you have done something wrong.
Also, another thing that can cause overfitting is the model complexity. If you apply a polynomial or higher regression to a simple dataset, it may get overfitted. So you can try to simpler the model. Also, you can further study how to do weight regularizations. In our scenario, the weight factors are pretty much smaller and not overfitted.
You can further do K-fold cross-validation to get an overall accuracy score. I have done it in my notebook. You can refer that as well for additional testings.
So now we can conclude that our model has achieved over 99% of accuracy.
Complete Co-lab notebook. (With preloaded data set)
colab.research.google.com
Kaggle — Weather History Dataset
www.kaggle.com
We have covered a lot of important concepts through this article. I hope now you have a clear idea about how to train and test a model from scratch with understanding the core concepts.
Data Science is a very wide field and it is impossible to know everything! But there are tons of articles available to explore this amazing world. So learn the fundamental concepts well and use them to solve real problems alongside improving the knowledge.
I would like to extend special thanks to Dr. Subha Fernando (Senior Lecturer at University of Moratuwa) for inspiring me to write this article.
Thank you very much for sticking with me until the end! Hope this article helps you out with your journey in learning about ML. If there anything you need to clarify or mention, please drop by the comments.
Happy Learning! ❤️