11 things I learned from the Machine Learning for Coders course at fast.ai
By: Jaime Durán
Tags: Deep Learning, Machine Learning, Fastai, Python, Data Science

1) The main idea behind ​​the course

We start with good news: recent studies proved that modern machine learning can be reduced to a couple of key techniques with a very broad applicability, being able to model the vast majority of datasets with good results:

Ensembles of decision trees (Random Forests and Gradient Boosting Machines). Mainly used with structured data (which can be found in any database at most companies). Multilayer neural networks, with learning based on techniques such as Stochastic Gradient Descent (SGD). This is also known as Deep Learning. Mainly used with unstructured data (audio, image, video, natural language, etc).

2) Reasons to use Random Forest

It’s a fast and parallelizable algorithm, since it’s based on decision trees.

Example of random forest prediction

It’s simple, with only a few parameters (maximum tree depth, minimum number of samples per leaf, and not much else).

In general it doesn’t overfit (and if it does, it’ll be easy to fix).

It assumes very little or nothing about the input; in practice it doesn’t matter if data follow a certain statistical distribution. This implies that bias for Random Forest is lower than for other algorithms; although a little bit higher than for a single decision tree.

for Random Forest is lower than for other algorithms; although a little bit higher than for a single decision tree. On the other hand, variance is higher than for others, but noticeably less than for a single decision tree.

is higher than for others, but noticeably less than for a single decision tree. It doesn’t need much feature engineering. Basically we’ll need to convert all input variables not being numerical.

It’s very flexible: almost everything can be modeled with them.

Although Random Forest is sometimes seen like a black box, the truth is that in practice it’s the opposite: it allows us to understand our data more easily than other traditional approaches.

Advantages over Gradient Boosting Machines nowadays: Random Forest scales better and it’s more complicated to screw up things.

3) When we should NOT use Random Forest

Random Forest works very well most of the time, but it’s a somewhat limited algorithm. Being based on decision trees implies not being able to extrapolate (a new input sample with features never seen in the previous training phase won’t fit to any leaf).

For example, we can’t predict what would happen if prices are increased by 20%, because it’s something never occured before. It couldn’t also estimate something simpler such as next year’s sales.

Comparison example

Moreover, Random Forest is only able to model a certain degree of complexity, having problems with unstructured data.

Neural networks on the other hand work very well predicting the future, or dealing with more complex problems such as voice recognition or handwritten text, where context is important.

4) How to reduce overfitting in our Random Forest model

We can use several techniques (which usually mean adjusting model parameters):

Subsampling: Random Forest uses different subsets of data to generate different decision trees. Each subset is created by replacing a number of random samples in the original dataset with repetitions of the rest of samples (each subset will have the same size as the original dataset). Subsampling creates each subset by using only a number of random samples, without repetitions within the same subset or in any other (so subsets will be smaller and completely different). We’ll train faster and the result could be improved by using more trees. This technique is very useful for large datasets. Reducing tree depth (this is also known as pruning). This way, each tree or estimator will be less accurate, but its stability will improve (averaging more samples in leaves implies less volatility) and won’t fit so much to our training data, generalizing better for future samples and obtaining better results. To optimize the model, we’ll adjust parameters such as the maximum tree depth or the minimum number of samples per leaf. Adjusting the number of features used in each algorithm iteration. In order to create new branches in a decision tree, Random Forest looks for the value of the feature that minimizes the error in the resulting branches, using only a random subset of features on each iteration, instead of using them all; meaning more variety for the trees, and a better final result due to the fact that partial results will be less correlated. The parameter we can adjust here is the number of random features to be taken into account for each iteration.

Apart from these techniques, the key to avoid overfitting will always be a proper model validation…

5) The need to have validation data

In any Machine Learning problem it should be mandatory to have a validation dataset apart from the training one, to avoid overfitting. If we don’t have it (which will happen to us in real world) we must split our original dataset in two new subsets: one for training the model and another one for its later evaluation.

By default, the validation subset is created with random samples, but for time series (where the dataset includes the date, and we want to predict something in the future) the validation subset should be created using the latest samples. The main idea is always choosing a subset of samples faithfully representing the data our model will receive afterwards (whether we face a real world problem or a Kaggle competition).

It may also be interesting to have a third dataset for test. Maybe our model results for the validation set are pretty good only because we tried a lot of different parameter values for optimization, and picked the best combination among hundreds. We want our model to generalize as good as possible, without over-adjusting to training or validation samples.

6) Feature importance for model interpretation

The most interesting tool when interpreting any model is possibly the feature importance, which tell us which variables have more influence in the final prediction. This technique gives us a very useful knowledge about the domain of the problem, which comes in handy when we aren’t experts in the field; and even is useful for the client, who could be surprised by the empirical importance of some features (both by excess and lack), showing them data science is useful :)

Example of feature importance graph

When we observe something we don’t know how to interpret (like the excessive importance of a feature) it’s convenient to consult the client to find meaning. It’s also interesting to analyze these features; perhaps we have missing values, finding something wrong at origin. We’d like to inform about the most important variables, so they could collect them better or try to get more detail in the future.

One of the problems we can detect here is data leakage. It’s possible that the values for a feature in our dataset weren’t actually available at the time the dependent variable was measured, because those are obtained afterwards (we shouldn’t have them). Or maybe they gave us a feature from which our target variable derives (for instance we get the exam score of some students to predict the final grade). We could detect this problem if the prediction totally depends on a feature. In real life this problem can lead us to a wrong analysis and bad conclusions.

Another issue we are dealing with is data collinearity: we can have a categorical feature only filled when another categorical one is of a certain type. Both will give us similar information. The underlying problem is that they are splitting the real importance this information could have if only one of the two features existed.

It may be interesting to remove unimportant or redundant variables to see how the behavior of our model is affected. Normally we won’t get a worse result without them; it could even improve. In Random Forest it would also imply fewer features to create new branches, so the model would be simpler and faster.

Be careful! the importance of a feature it’s also in the possibility of modifying it in order to change the result. If we don’t have that possibility, in the end we won’t be able to improve the current situation. We talk about this later.

7) Other interpretation techniques

We can help our model by applying one hot encoding (also known as dummy variables): when we have a categorical variable holding few different values, we can generate as many boolean variables as possible values (always removing the original). This way we’ll know exactly the importance of each category in the final result, apart from surely improving the efficiency of our model.

Example of one hot encoding

Another technique that could improve our model interpretability would be detecting similar variables. We’ll use dendrograms (we could also use K-means clustering) to get the most similar ones, which are those separated closer to the end (right margin in the below graph). We can remove one of each pair of similar variables, and check if there’s some effect on the model behavior. Normally the fact of discarding very similar variables will imply a simpler model with almost the same result.

Dendrogram example

One more technique we can use is partial dependence. It’s about getting a deeper knowledge by measuring how the target variable is related to each feature, without the influence of the rest. With partial dependence plots we can observe the importance of each relationship in a more isolated way. If the analyzed feature is categorical, we can group samples by class and see the trend for each of them. We could also apply a similar approach for numerical features by defining intervals.

Representation example for partial dependence

Finally, in the case of Random Forest we have the interpretation of decision trees. Like the previous technique, this is very important in real life, and justifies we are not using a black box. It measures the weight of each feature in the output result, numerically. We can even obtain the global influence, if we do the same for all samples and take the average. We use treeinterpreter to apply this technique. There are also libraries to represent it graphically:

Example of tree interpretation

8) Tricks to validate our model

In Random Forest we could avoid setting aside part of the original data for validation by using the Out-of-Bag score (which also tells us if there is overfitting). It could be a good idea if we have very few samples to train our model (small datasets), being a great alternative to cross validation. Just be careful because the OoB score measures how our model generalizes in a statistical sense, not in a practical way.

A good strategy (especially for time series) could be training and validating the model as usual (using different subsets), and once we have chosen the parameters giving us a good result, we can re-train the model using all the available samples. Our future predictions could be better because the model would have seen more data.

How to know in a Kaggle competition if our validation dataset is a representative sample of the test set used for the Leaderboard? There’s a small trick that involves creating and validating a few different models, and also submitting to Kaggle. Then we would plot a graph with the validation score on one axis, and the test score on the other. If the resulting points for each model form a straight line, our validation set will be a good representation of the test data; otherwise we should choose other samples.

Course snapshot

In real world you also have to be very careful choosing the test set, because we want it to be as close as possible to the data in production. You have to ask yourself: Which clients are going to use it? How long will it be since we finish the model until it’s deployed in production? How often can we update it?

If we choose the right test set (in real world) or validation set (Kaggle) we can detect every generalization problem of our model.

9) How to work with huge datasets

What can we do if we have a dataset so big it doesn’t fit in memory?

Yes… we could use more memory or several machines, but we could also try to optimize the process and use the same resources we already have.

To optimally load data into a pandas dataframe, we can explicitly specify the data type for each column, rather than letting pandas to infer that info. To do that, the read_csv() function has the dtype parameter, expecting a column-type pair dictionary. The key is telling pandas to use the smallest possible type for each numeric field; if the content can be fitted in an int8... why using int32? In addition to less memory allocation, operations will be faster too.

An option to know how to load the complete dataset with the appropriate types is loading part of it using unix commands such as head , tail or shuf . Example for a notebook:

! shuf -n 5 -o sample_training.csv train.csv

Another important problem when we work with huge datasets is training time. In Random Forest this time depends on the number of estimators (trees) and the number of samples used to generate each one, so we could decide to apply subsampling (which also reduces overfitting). In this case we won’t try to calculate the OoB score because we’d need the whole dataset and that would take a lot of time.

Another trick to reduce training time is converting the input dataframe into a numpy array, and passing that to the fit() function. Actually that conversion is the same Scikit-learn applies internally, so with this trick we could save a few seconds every time we try to train the model with different parameters.

We can always write %time in front of any command in our notebook, to see how much execution time it takes. If we want more detail we’ll use %prun , which tells us exactly where the time has been consumed. This is known as profiling, and it helps us to detect things like the prior one, in order to reduce our total execution time.

10) The goal of applying Machine Learning

The first thing we must always know before facing a Machine Learning problem is what we want to achieve (a specific goal of the company, or of society). Next, we want to know which variables we can control to achieve our goal, and what data we have (and can use). After these steps, we generate the models to predict or simulate the effect that our possible actions would have over a specific target. These 4 steps are part of the drivetrain approach for creation of data products.

A couple of examples using this approach:

Example 1: A recommender should make you to buy more books (optimization) instead of offering things you obviously like (prediction) and even probably you already know or have. The best behavior would be offering something that could change your behavior, and make you more likely to buy, increasing benefits.

Example 2: We have a model in our company predicting which customers are leaving next month. We’ll need another model simulating the effect of the actions our company can take, and apply the one that solves the problem with the highest success probability.

So on the one hand we’ll have all the levers that we can operate. On the other, the features that are important variables for the final result. The intersection of both will be what really matters: the levers that we can use to influence on the result; mixing prediction and interpretability. Thanks to the feature importance analysis we know what we should touch to modify the result, and with partial dependence plots we could simulate what would happen by modifying each variable.

11) The ethical part also matters (a lot)

We must be very careful with the data used to create our models, because sometimes the samples don’t take minorities into account. For example, a dataset with information related to people may not reflect the actual presence of different races in real world, but for sure the race of its creators is not affected. A biased dataset will imply results being also biased.

Following this line we can point several problems, for instance in facial recognition applied to criminal detection. If the detector makes more mistakes with black people for the mere fact of being trained with fewer samples from black race, this will be translated into more notifications to the police and more arrests among the black population, something that will increase the stats and enhance the initial bias.

Other well-known cases were the artificial intelligence that became racist only by reading, the camera filter that improved your appearance whitening your skin (see picture below), or the person labeled as a gorilla in a picture.

There are also problems with gender bias, for instance with online translators, which are often sexist (like the language actually). Another curious example: there could be more recommendations of technological meetups to men, for the simple fact they already attend more than women; resulting in a gender discrimination (again it could evolve to worse if we don’t stop it).

A more serious example is the automatic recommendation of news on social networks. As algorithms try to optimize the number of clicks on those recommended contents, the most displayed ones are often sensationalist or controversial news, even false; sometimes having disastrous consequences, as happened in Myanmar with Facebook .

Sometimes algorithms do things we don’t know how to interpret, but the fault of them being racist or sexist will be partly human, because we should be more careful. Other times the problem is the misuse of these tools which in the wrong hands can cause a lot of damage (we can cheat an autonomous vehicle, or a facial recognition system).

One more thing…

The Machine Learning for Coders course has a second half dedicated mainly to neural networks and gradient descent, where all the basic concepts are explained with a practical approach, and even a neural network is created from scratch using PyTorch. Of course I also found it very interesting and I learned things that could be part of the previous list; but taking into account that I’ll soon write a similar article for the Deep Learning for Coders course, we better stop now :)

What I recommend you, whether you want to start in the world of Machine Learning, or get a new point of view, don’t hesitate to follow this one. It uses a top-down approach, and you have all the used notebooks available to view and test the code shown in the videos (maybe you’re interested in a revised version with more comments). I already recommended this course in a previous article and personally I’m very satisfied with it.

Finally, I would like to thank Jeremy Howard for his great contribution to the democratization of Machine Learning, both for sharing online the courses he teaches at University of San Francisco, and for creating a library as simple and useful as fastai. Don’t miss his updates! Thanks also to Rachel Thomas and of course to all the people who contribute to the fastai code and docs.