How To Fit A Random Forest Classifier In Julia | by Emmett Boudreau | Towards Data Science

Within the past few years of very rapid development in data science technology, we have seen the dramatic escalation and adoption of a multitude of open-source tools. Among these are well-known tools like SkLearn and Tensorflow. With the recent early adoption of Julia, however, there are oppurtunities to dramatically improve our strategic approaches to solving machine-learning problems. The problem that effects the Julia language most is easily the lack of mature packages for various science or data related operations.
notebook
While this has been a long road that is certainly still being traveled, I think it is work taking a look at what Julia has to offer in its current state that hadn’t been there before. One recent addition I have been excited about is the addition of a Random Forest Classifier to Lathe. Using Lathe, DataFrames, and CSV, we can now easily explore, encode, and predict with various features!
Of course, in order to work with a machine-learning model, we are first going to need to take a look at getting some categorical data to train it with. For this example, I am going to be using some UCLA Data that has been published on European Cars. Since the data is in comma separated value (CSV) format, we are going to use the CSV.jl package. One fantastic attribute of Julia’s ecosystem is the level of segmentation — though it can be a burden. CSV.jl is a package exclusively developed to read CSV.jl and return a data frame just as JSON.jl is a package exclusively developed to read JSON formatted data. While we are at it, we are going to want to also import DataFrames.jl, which will allow us to actually visualize our data in a tabular spreadsheet.
using DataFramesusing CSV
Now I will read in our CSV file:
df = CSV.read("car data.csv")
In the very near future, this method of reading .CSV files will be deprecated, so it is probably best to use a sink argument:
using DataFrames; df = CSV.read("car data.csv", DataFrame)
The easiest way to get a great view of our DataFrame is to use the show() method. In order to visualize our features better, we can use the allcols key-word argument, which will take a Bool type:
show(df, allcols = true)
Given that the purpose here is demonstration, not much thought is going be given towards the features other than what will work for this application. In order to see unique categories, we can use the Set() type:
Set(df[!, :Transmission])
Given that Gini-Index based models like this classifier that work off of impurity often do better with higher categorical counts, I selected the fuel type to be my target. As for the feature in this application, it was quite a difficult decision to make. I decided to go with a continuous target, since it would be quite hard to predict off of one set of 0's and 1’s and the dispatch for DataFrame data still has yet to be implemented in Lathe (meaning we can only support one feature in this unstable version.)
The first thing I decided to do was extract these two features I wanted to work with into a new and smaller DataFrame, like so:
mldf = DataFrame(:Fuel => df[!, :Fuel_Type], :Kms => df[!, :Kms_Driven])
If you would like to learn more about the DataFrames.jl package, I have a recent and effective article that might do some work to help you in getting started!:
An important step of working with any model is preprocessing, and evaluation. This is where you might want to question whether or not a model could actually help to solve this problem. This can be done with a majority class baseline. A majority class baseline is a simple measurement of how effective of an approach it might be to guess the most common category for every scenario. Let’s start with evaluation. The first step in approaching this problem is probably going to be to get our data into simple one-dimensional arrays that can be put into some functions. I like to do this with symbols and DataFrame calls because it seems effective to me:
X = :KmsY = :Fuel
Next, we will split the data into both a train and a test set. This is so we have both data to load the model with and data to get a solid evaluation out of. In order to do this we will use the TrainTestSplit() function from Lathe.jl:
using Lathe.preprocess: TrainTestSplittrain, test = TrainTestSplit(mldf)
Now we have two DataFrames, train and test, which we can extract our one dimensional arrays from.
trainX = train[!, X]trainy = train[!, Y]testX = test[!, X]testy = test[!, Y]
Next, we’re going to get our majority class baseline:
using Lathe.models: majClassBaseline
Since this is Julia, we can always check input by using the ?() method:
As seen in the documentation, the function takes one parameter, y, and will return a Lathe model object. This object will then have the function predict(), which we can use to return a prediction. Let’s create that object by providing our trainy, and then predicting with our testX:
evaluation = majClassBaseline(trainy)yhat = evaluation.predict(testX)
As you can see, the most common fuel type among these cars is clearly Petrol. Let’s evaluate this basic prediction using the catacc function from Lathe.lstats:
using Lathe.lstats: catacccatacc(yhat, testy)
81 percent is quite a high accuracy level for this test, and commonly problems like this might not even require a model to be solved, but merely an assumption. That being said, we came here to fit a Random Forest Classifier, and that is what we are going to do! Another cool thing we can look at is the counts in our new baseline model, which might give us some insights into why that accuracy is so high:
evaluation.counts()
Before we go on with preprocessing, it might be wise to do a basic evaluation of our minimum viable product model. Let’s put together a classifier and we may continue onto tuning it afterwards.
The model we are going to we working with today is the RandomForestClassifier object from Lathe.models:
using Lathe.models: RandomForestClassifier
Like before, we can use the ?() method in order to view the in-code documentation on this particular object:
As seen in the documentation, the model takes both key-word arguments and positional parameters as input. Our first model is going to simply provide the X and Y as parameters.
model = RandomForestClassifier(trainX, trainy)
Now we can go ahead and evaluate it in the same way as we did with the majority class baseline:
yhat = model.predict(testX)
And evaluate it using catacc:
Oh dear!
As I expressed prior, this certainly isn’t an optimal application for this model, but there are some simple and interesting things we could attempt to boost this accuracy. First, given the low number of categories, we could be over-fitting with our max_depth at 6. To test this theory, I would first give it a bump up, then down. If the suspected theory is correct, then it is likely that bumping it up is going to result in a serious drop in accuracy:
model = RandomForestClassifier(trainX, trainy, n_trees = 100, max_depth = 11)yhat = model.predict(testX)
Just as well, bringing the parameter low results in the dramatic opposite:
model = RandomForestClassifier(trainX, trainy, n_trees = 100, max_depth = 2)yhat = model.predict(testX)
While a basic reaction might be to turn the depth down and work with the model in that way, I imagine that the model might do better if there is a more balanced solution on depth with more trees instead:
model = RandomForestClassifier(trainX, trainy, n_trees = 1000, max_depth = 5)yhat = model.predict(testX)
Well great! A little bit of tuning, and our model, we have raised the accuracy dramatically! For one last trick, we can try scaling our feature. In order to do so, we are going to use the StandardScalar from Lathe.preprocess:
scaler = StandardScaler(trainX)trainX = scaler.predict(trainX)
Next, we will run the same on our testX:
testX = scaler.predict(testX)
This is going to return an array of complex numbers. In order to fix their imaginary bounds, we can change the data back to real numbers buy using the real() method. Of course, this needs to be casted onto the entire array, so we can do that by changing the requirements of the data types inside the array to be real:
trainX = Array{Real}(trainX)testX = Array{Real}(testX)
Now pass that through the same code as before, and we can see if that registers a further accuracy boost:
model = RandomForestClassifier(trainX, trainy, n_trees = 1500, max_depth = 5)yhat = model.predict(testX)catacc(yhat, testy)
So we lost accuracy?
Allow me to explain. Whenever we normalized the data using the standard scalar, we created less depth to the problem by counting our observations as standard deviations (from the mean) rather than arbitrary numbers that apply to whatever domain the data is extracted from. That being said, our accuracy gain might be felt with a quick tuning of the depth:
model = RandomForestClassifier(trainX, trainy, n_trees = 1500, max_depth = 3)yhat = model.predict(testX)
After some hyper-parameter optimization, these are the results recorded:
model = RandomForestClassifier(trainX, trainy, n_trees = 10, max_depth = 2, min_node_records = 4)yhat = model.predict(testX)catacc(yhat, testy)
Categorical problems have been somewhat of a chore in the Julia ecosystem for a while now. Fortunately, thanks to the further development of many Julia packages, there are ways to get quite accurate categorical predictions quite effectively in Julia. While this feature might be lacking, and there certainly are some issues that need to be addressed, this early preview into Lathe 0.2.0’s future is exciting for the future of machine-learning in Julia, especially when it comes to types. Another related article you could be interested in is this one, where I test the speed of this new classifier against similar solutions in the industry!
https://medium.com/@emmettgb/performance-testing-python-and-sklearn-with-julia-and-lathe-f5516fa497d7
Thank you for reading, and happy learning!