Predicting Micronutrients using Neural Networks and Random Forest (Part 2)
By: Gilang Ilhami
Tags: Random Forest, Decision Tree, Python, Machine Learning, Scikit Learn

Predicting Micronutrients using Neural Networks and Random Forest (Part 2)

Photo by Julien R on Unsplash

Welcome back! Glad you can join me in part 2 of this series of “Predicting Micronutrients using Neural Networks and Random Forest.”

In the previous blog post, I mentioned that UNICEF has invited us to join their data science team to predict micronutrients composition in certain foods by using the power of machine learning.

Also, we went through the process data cleansing with a dataset from the US Department of Agriculture.

Now, we are going to one of the most exciting parts of almost every data science project, building the machine learning model.

As you can probably tell by the title itself, we are going to use Random Forest and Neural Networks to build the model. But for part 2, we are not advancing to neural networks just yet.

In fact, it is recommended when you are working on a data science or machine learning project to start using simple models and move your way to a more complex model.

Okay, our colleagues in the UNICEF data science team are waiting for us in the meeting room, so let’s get going and build that machine learning model.

Overview

This blog post contains the following:

Introduction

A brief introduction to machine learning

Multivariate Linear Regression: Linear Regression and Multivariate Regression

Random Forest: Decision Trees and Bootstrapping

A Brief Introduction to Machine Learning

In the first part of the series, I mentioned some sources that act as prerequisites in order for anybody to be able to have a better understanding of what I am talking about in this series.

One of them was about “Introduction to Machine Learning.” However, I decided that we should address a minor introduction just to keep the flow nice for this blog post.

But if you wish to gain a better understanding, I highly recommend that you read this blog post by Vihar Kurama.

What is Machine Learning?

We will have to be a little bit more critical about this question since there have been many attempts to define what is and what is not machine learning. Here is a couple of examples.

Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed. Arthur Samuel (1959)

In the 1950s, Arthur Samuel was able to create a program that lets the computer learns to play checkers by playing against itself. In the end, even Samul himself lost to his own program in checkers.

The second definition is probably the most popular and well-known among the machine learning community.

Well posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E. Tom Mitchell (1998)

Let’s use our current mission to predict micronutrients within certain foods as an example.

In this case, E is the records of the amount of nutrient composition within each food, T is the task of predicting the micronutrients based on other nutrients, and P is how far are the predicted values from the ground truth.

E: the records of the amount of nutrient composition within each food. T: the task of predicting the micronutrients based on other nutrients. P: how far are the predicted values from the ground truth.

In common practice, there are three different kinds of machine learning algorithms when it comes to tasks:

Supervised Learning . This is what we are actually doing when we have input variables and output variables for training.

. This is what we are actually doing when we have input variables and output variables for training. Unsupervised Learning . It is basically the same as supervised learning, but there are no output variables.

. It is basically the same as supervised learning, but there are no output variables. Reinforcement Learning. This type of machine learning handles a task where there are no input or output variables. Instead, a computer program will interact with a dynamic environment to achieve a certain goal through trials and errors.

When it comes to supervised learning, we actually trying to learn a function that maps the input variables to the output variables.

The form of this function is unknown and our job is to evaluate various machine learning algorithms and observe which one is better in approximating the underlying function.

Another way that we can think of a function is like a magical box that can never be opened. But the box can take input and make a prediction based on that input.

For example, if you put in an email from a suspiciously unknown email into that box, it can correctly tell you 100% the of the time if it is a spam or just one of your friend who’ve just create a new email.

Since we can’t see what is inside the box, then maybe we can create something similar to that box. And the form of this something doesn’t have to be a box. It can be a ball, prism, or any other shape.

(Let’s say we ended up with a ball)

The most important thing is that our ball can make an approximation that is as good as the prediction of the magical box. This means that it can predict 99% correctly instead of 100%.

Multivariate Linear Regression

As I mentioned before, when dealing with a data science or machine learning project, it is recommended that we start with a simpler model.

Here, we are going to cover Multivariate Linear Regression and implement it using Scikit-learn.

Linear Regression

Linear Regression is one of the most widely used machine learning algorithms for predictive modeling. It also commonly implemented in a supervised learning scenario.

It is called a linear model because it infers a linear relationship between our input and output variable. Basically, it gives us a linear equation that looks something like the image below

Note — input and output variables are usually referred to as independent and dependent variable respectively when we ware talking about Linear Regression. I’m going to use the terms interchangeably. This also goes for coefficients, weights, and parameters.

In the equation above we have one dependent variable, y, and one independent variable, x. Also, we have two betas that are the coefficients of the linear equation.

The Epsilon is the error term due to random sampling.

The first coefficient (beta null) is the intercept term, while the second one (beta one) is the weight that is assigned to the variable. The weight signifies the importance of the variable.

This means that if beta one is 2.5, then the unit of our variable (x one) will also increase by 2.5.

Note that the equation above has only one variable with one coefficient. When we have more than one, let’s say n variables, then we need to include n coefficients in the equation.

When we are building a predictive model using linear regression with multiple variables, it is usually more known as multivariable linear regression.

Based on the preprocessing step that we’ve done in the previous blog post, we have 43 input variables for training. That means we will have 44 coefficients (43 for each variable plus the intercept term).

The goal of the linear model is to find the parameters that best learn the data by minimizing the residual sum of squares (RSS).

RSS is the total of the squared differences between the true value, y, and the predicted value by the model, ŷ (pronounced y-hat), which indicates an estimation.

We are not going to cover the details of RSS in this blog post. For those who are interested in the detailed explanation and derivation, you can check it out here.

Okay, let’s hold here for a bit. Can you spot something’s missing from our discussion here? Something related to our proposed task in particular?

If you were thinking something like “but we have more then one output labels” or “why is there only one y in the linear equation?” then you deserve a promotion my friend (just kidding, maybe a candy would be enough?)

When we have to predict values for one output variable, it’s called univariate regression. In fact, this pretty common in most linear regression tutorials that you can find on the web.

But what to do when we have more than one output variables?

Multivariate Regression

In the case when we have to make predictions for multiple output variables, it is called a multivariate regression, which we will briefly need to cover first.

Remember the linear equation we talked about just a moment ago? We can actually further extend that formula to look like something like this

Basically, we now have more dependent variables, parameters, and error terms. Our goal is similar to univariate regression. Only this time, we have to find the parameters that minimize the RSS across all the output variables.

Also, it means that instead of having 44 parameters to estimate, now we have 264 parameters (44x6).

Wow, that’s a lot, right? Not really. This pretty common in most machine learning practice. Especially when we are talking about deep learning, where we can have parameters up to the thousands.

We’re not going to cover RSS in the context of the multivariate regression. However, if you want to know more in details about multivariate regression, please refer to this slide.

The slide can be too technical for those who are not familiar with statistics. But If you are interested, give it a go!

Next, we are going to use Scikit-learn to implement multivariate linear regression. Don’t forget to take a break if you need to.

It’s time to code!

Alright, we are finally going to start coding. Make sure that you have Scikit-learn installed. Also, we are going to do data visualization in this section, so make sure you also have Matlotlib and Seaborn.

Before we build the model, let’s use some visualization to get a better understanding of our data.

Let’s analyze the strength of the linear relationship between every two variables. This is also known as the correlation between every two variables.

We can build a correlation matrix using the corr attribute provided by pandas.

You will see something like this as the output.

Ah, look at all those numbers! I think we should make this in a way so that our team member can understand the correlation better.

Let’s use Seaborn to make visualization of the correlation matrix. There’s a snippet of code that we can use from here to quickly make a heatmap of the correlation matrix.

With some minor adjustments, we can run the following lines of code…

…and something like this below should appear.

Alright, that looks much better. Let’s take a look.

We can see that some of the nutrients are either positively correlated or show no correlation with other nutrients, while only a couple is negatively correlated.

If you want to know the difference between positive correlation, negative correlation, and no correlation, please take a look at here.

Although the correlation matrix displays visually about the linear relationship between every two variables, it does not explain if two variables cause each other to go up or down.

In fact, a lot of time you here will people who are involved heavily statistics (including machine learning) say something like “correlation does not cause causation” or “correlation does not imply causation”.

For that reason, it is a good idea to plot our data in order to get a better understanding of the relationship between each variable.

Let’s do just that with Matplotlib!

Looks like Carbohydrate and Iron shows a positive correlation. We can plot the data using the following line of code.

Hmm, looks like even though the correlation matrix suggests that carbohydrate and iron are positively correlated, they seem to lack colinearity.

This why it is important to plot the data before building the model.

Okay, getting back to our data, it seems all cluttered up on the bottom part of the plot. How about we try this again with Fiber and Iron.

Looks like it is also the same for fiber and iron. However, in this case, the data points are all cluttered up on the bottom left o the plot.

Another thing that we should pay attention to is the high variability of outliers shown on the plot.

Clearly, if our initial goal to predict the amount of iron based on the amount of carbohydrate or fiber, a linear model will definitely show poor results.

But we are dealing with multiple variables here, let’s move from 2D into 3D. Again, Matplotlib offers a great way to plot 3D plots.

Okay, now this plot shows definitely that linear model won’t work here. Our data is highly non-linear and using a linear regression won’t provide us with any good performance.

But for the sake of fun, let’s try anyway.

First of all, we need to split the input data and labels in a training and test set. The Scikit-learn’s train_test_split function makes this process convenient.

We will split 75% of the data to the training set and 15% to the test set.

It is highly suggested in most practice that we should an additional set called the validation set. But since our dataset is quite small, we don’t have to do that here.

Now, we need to standardize our data before coding the model. This is important because some of the nutrients use different units. For example, Vitamin A uses the UI unit, while Carbohydrate uses the gram.

We will be using Scikit-learn’s RobustScaler to standardize our data so that all of them are robust against the differences in units.

Alright, looks like we are ready to use the data to build the model.

I assume that most of the readers are already familiar with Scikit-learn’s LinearRegression class. It is commonly used in many online tutorials.

However, the class assumes that only one output variable is provided.

For multiple output variable, we need to add a class, which is not usually mentioned in most tutorials, called MultiOutputRegressor .

To put it simply, it is a way by Scikit-learn to extend regressor that does not natively support multi-output regression.

We’ll start by simply importing the classes.

Next, we can start fitting the model.

Normally, in most tutorials about linear regression, you will most likely see something like this.

We need to modify that just a bit to include the MultiOutputRegressor class.

Looking good so far. Next, we need to check our model has performed well during training. Scikit-learn’s score attribute should do the trick.

Our model has shown an accuracy of ~72%. What do we make of this? Well, it seems with just linear regression, we can get performance better than flipping a coin.

But let’s continue a little bit with a very important step in every machine learning practice, validating our training performance on the test set.

We do this to avoid overfitting during training. To put it simply, overfitting occurs when a model tries to predict a trend in data that is too noisy.

The black line is when the model was able to generalize well on the data. On the other hand, the blue line displays the line when the model has overfitted. (source: wikipedia.org)

So let’s check how the model performs with the test set. We can do the same as we did with the training set with some minor changes.

Looks like our model was able to escape overfitting to a certain extent (difference of around 7% between the training and test set accuracy).

You may now have a whisper in your head saying “so how do we avoid overfitting and underfitting?” Well, it has something to do with the bias-variance tradeoff.

If we go back to the plot, our data looks all cluttered up. Visually, this supposed to be a sign of high bias, which should result in underfitting. However, looks like our model has more chance to overfit instead.

While we did a good job to avoid overfitting and underfitting, it’s not good enough to show to our team in UNICEF.

Since the number of people, we can influence is large (I mean it is UNICEF), then we should increase the performance of our model.

Like I said before, the dataset itself is highly non-linear and consist of many outliers. Doing a linear regression with such dataset will surely not result in an optimal performance

As a fun exercise, you can try to plot the data again with the predicted value of the model. This should give a better visual understanding of our data(we will do more visualization in a later part of this series).

So basically, what we need to do is use a different model that can handle non-linear data, and robust against outliers.

Enter Random Forest!

Random Forest

If we try to find some of the most used machine learning model to increase performance in Kaggle, it will either be Random Forest or Gradient Boosting Trees.

In this blog post, however, we will be talking about the first one.

But before diving into the algorithm right away, we need to talk about decision trees and bootstrapping.

Decision Trees

Imagine walking in the park while observing your surroundings. For no obvious reason, you decided to look at a particular tree. It starts from the root and makes it ways to the top until the last leaf.

This the same thing with decision trees, but upside down. Decision trees algorithm is also known as CART, Classification and Regression Trees.

For now, we will focus our attention on Regression Trees since, of course, we are dealing with a multivariate regression task.

Unlike linear regression, decision trees algorithm does not concern with the assumption of linearity within our data. Also, it does not incorporate the need for any weights.

Instead, when using decision trees, we are attempting to use the satisfaction of the feature space to make predictions.

According to Chapter 8 of the Introduction to Statistical Learning book, there are two steps that we need to follow to build decision trees:

1. We divide the predictor space — that is, the set of possible values for X1, X2,…,Xp — into J distinct and non-overlapping regions, R1, R2,…,RJ .

2. For every observation that falls into the region Rj , we make the same prediction, which is simply the mean of the response values for the training observations in Rj .

Note that the author is using, Xj ,as the notation for the independent variables instead of xi. We are going to do the same thing from here on.

Now for those who are not familiar with the decision trees algorithm might find the above statement rather unclear.

But to put it simply, the first statement state that we need to choose a variable, and established regions on the observations (data points) based on those observations.

While the second one is stating that after we have our regions, we can make predictions by calculating the mean of the observations in each region.

There are two ways that we can do to understand this concept intuitively through visualization.

First, we can plot our data points and draw a line that separates two areas of it. Let’s use the fiber and carbohydrate to estimate the value of iron.

Here we see a line that splits the plot into two regions. If we look closer, the line starts the split at about the 41 marks in the axis that represents carbohydrate.

In terms of regression, decision trees algorithms make a prediction by averaging the data points in each region.

Then the algorithm will continue to split on a variable until the estimated value is just short from the ground truth.

Another way we can visualize decision trees is by using graphs.

Here we can see that the graph starts the split at the top. The boxes are called the nodes while the arrows that point/connect to another node are called the edges.

If you noticed, the first node makes a split on the 41.32 value, which is almost the same as the plot.

Unlike real-life trees, decision trees are usually visualized by having the branches going downwards.

The two nodes that branched out of the top node are called the internal nodes. In the picture above, there is the word value inside each internal node.

Those are the values are the predictions made by the algorithm by averaging the number of observations in each region.

However, both the plot and the graph above have only a depth of one. If we let the tree grow to a depth of two, then we will have something like this.

The plot and the graph with a depth of two.

On the plot, we can see that now that another line has made a split on the previous line. While the graph has more internal nodes which made a split the fiber.

For the graph, if we let the tree grows until it made the best split, it will have more internal nodes, which in the end might not be visually comprehensible anymore.

But the question is what is the best split? Well, the best split is the split that minimizes the error between the predicted value and the true value.

If we look closer at our graph, we can see that there’s the word mse in it.

This refers to mean squared error which is given by the following formula

This is similar to RSS that we’ve covered before with some slight difference.

As we see that the formula has the 1/N notation. This means that after summing all of the squared errors, we average them by the number of N observations.

MSE might look like RSS plus the averaging notation, but this discussion might worth checking out before making that conclusion.

Anyways, understanding the decision trees algorithm is a crucial step in order to work with the Random Forest algorithm.

However, this is just a brief explanation of what decision trees are capable of and recommend you to check out chapter 8 of the Introduction to Statistical Learning book.

The book gives a better high-level mathematical explanation for decision trees.

Before we continue, I might need to mention one more important aspects when using decision trees. It’s called the Gini Index.

In short, it is used to measure the probability of a certain variable being mistakenly classified when it is arbitrarily chosen. The Gini Index is based on the concept of entropy, which is the degree of uncertainty.

This concept is usually common in various blog posts that use decision tree related algorithm for classification tasks.

In a classification task, we want to measure the information gain in order to understand which features give the most information about a certain class.

The information is then used to explain the importance of each variable and use it understand which variable has the most effect on the prediction or most commonly understood as variable importance.

In terms of regression, and as we discussed before, we use mse find the best split.

We are probably going to discuss more variable importance in a featured blog post in this series.

Next, we will cover another essential aspect of Random Forest, Bootstrapping.

Bootstrapping

One of the challenges of working with machine learning projects is that gathering data can be very expensive and difficult.

Although various open datasets — like the one we are using — are available on the internet, some models require more data to enhance performance during training.

But gathering data can sometimes be expensive and difficult in practice. We can use a resampling method called Bootstrapping

In short, bootstrapping involves resampling data from our already available dataset randomly with replacement. “With replacement” basically means we can use the same observations twice in the new dataset.

We repeat the process for some amount of time until we have the desired amount of bootstrapped datasets. Each bootstrapped dataset contains the same number of observations of the original dataset.

Let’s use an example with our dataset and use the first five observations of the carbohydrate variable.

The first five observations are 0.06, 2.87, 00.0, 2.34, and 2.79.

Let’s denote the bootstrapped dataset as Zi. If we have 4 datasets, then each dataset will be named Z1, Z2, Z3, and Z4 respectively.

If you look at the image above, all of the bootstrapped datasets’ observations have repeating value in it. This is what “with replacement” means.

So what do we do with these bootstrapped datasets? Well, we use them to make machine learning models of course.

This might sound a little bit strange since we already have our original dataset to train on.

But it will all make sense once we move on the exciting part of this topic, coding the Random Forest!

Coding the Random Forest

Now that we have some intuition about decision trees and bootstrapping, we can move on to talking about Random Forest.

You see, there’s a reason why the algorithm is called Random Forest. Think about the two important aspects that we’ve just discussed.

The algorithm uses bootstrapping to create randomly subsets of data with subsets of features from the original dataset. Then decision trees algorithm are used to build a model on each subset.

Then each prediction of each model is averaged to create one final prediction.

This is the same for multivariates tasks. The only difference is that each decision trees output values for Vitamin A, iron, Zinc, Calcium, Vitamin D, and Folate.

Let’s get started! The same with Linear Regression, we need to import the RandomForestRegression class and the MultiOutputRegressor class.

Now we can fit the data into the model.

As we can see there are some parameters that were set here. n_estimators to 100 because the next version of Scikit-learn will have that values as the default value.

max_features is just make sure that we subset the square root of the number of features. This is one of the hyperparameters that we need to tune for our model. We will cover this in another blog post.

Next, we can try to check the accuracy of our model.

Wow looks like our model performs really well on the training data. I mean, ~95% accuracy? That sounds pretty good, right?

Well, not really. We still have to check out the test accuracy, because what we want is the model to perform well on data that we have not observed before.

Looks like the performance seems pretty good. Almost 20% better than linear regression

Unlike Linear Regression, our model overfits since the training accuracy is way larger than the test accuracy (around ~15% difference).

There are various ways that we can do to improve the model. However, let’s see if we can make the model using an algorithm that has gained high popularity in recent years.

In the next part of this series, we will build a Neural Network algorithm to train on our data.

What do you think will happen when we use Neural Network on our dataset? Will it be the same? better? or even worse?

Let me know what you think in the comments. Looking forward to the feedback and further discussions.

See you in part 3. Cheers!

An Indonesian student currently studying in Korea and has a deep interest in the world of AI. Recently moved to IBM as an AI Research Intern. Believe in continuous learning. Want to give back to the AI that has contributed so much in my pursuit of knowledge. When I’m not coding, I like to read up the latest trends in the space industry and play for my college’s American football team.