Preprocessing BIG Data In Julia With Lathe | by Emmett Boudreau | Towards Data Science

When it comes to the wonderful and vast world of machine-learning, there are many popular options that many Data Scientists choose to work worth. Truthfully, in my personal outlook on the Data Science ecosystem right now, I think that many aspects of the field are outdated while others are charging forward at the speed of light. In a way this makes sense, because an exploding industry like this has not had the time to properly mature to its size, and thus we ultimately face some issues.
One significant issue that Data Science is facing is that of programming languages. To many entry-level developers, Data Science can be rather hard to enter as a result. Because there are so many options, and avid, sometimes blatantly religious, followers of certain languages who will likely construe certain opinions on why X or Y is better than the other. If you happen to be in that boat, I actually have an article that could answer this question, because the language you choose will very likely make a significant impact on the type of work you ultimately end up doing. You can check that article out here if you desire:
towardsdatascience.com
I digress... Ultimately I think that while it is very unclear and cannot be known what language will become the new Data Science poster-child, if the language even changes, I think that the problems with languages are often brought to light with Data Science work. Let us all recall that back in 2015, most Data Science job listings were for Java. That being said, Python has only been dominating the field for about 7 years now, meaning that the language has only recently started to target this audience. Many will debate that Python is the perfect language for Data Science, but truthfully there cannot be a perfect language for anything as there is always some sort of flaw.
source
While Python has the ultimate Data Science ecosystem, it is also an interpreted language, and is more akin to something like JavaScript than something like Scala or C++. This of course slows things down, and while most packages are written in C, and can actually run very quickly thanks to optimization and good code, they still are not without their problems. The problem with Python is that if you want to work with more complex, multi-dimensional, big-data, then you are simply going to run into a lot of unexpected problems. Needless to say, trying to squeeze every last drop of performance out of your computer because of the programming language you use is not a perfect experience.
Python also has an after-thought of parallel computing, meaning it was not necessarily designed with parallel computing in mind. Parallel computing is becoming more and more important and more and more widely-used every single day. The fact that Python can sometimes have trouble with it is certainly a hindrance to the language, especially when compounded with everything else.
Of course, Julia is not perfect either, as Julia has many flaws as well. While I think that its paradigm is incredibly great to write in and there is nowhere else that I would rather write my code, I also can see some issues with the language. Of course, the language is still relatively new, so this is definitely sensible. And the fact that it only recently climbed in popularity means that the ecosystem is still very immature. With an immature ecosystem, it is doubtful that much Julia will be adopted by the industry, primarily because they do not want to pay people to write packages that are already written in another language.
All of this out of the way, I really think that there is no perfect language at this time. I think Python is great, and I think Julia is great, but I think they both have flaws that detract from their greatness. That being said, I also think that Julia shows a lot of promise for the future, even if that future is being a side-kick to Python. Today I wanted to demonstrate the potential of this language, with a matured eco-system, and some more work on the core itself, I really think that Julia could do a lot of very exciting things in the Data Science domain, which of course makes me excited.
Notebook
Lathe is a machine-learning package for Julia that has been in broadly active development since 2019. One problem with Julia is that there is a big gap inside of the ecosystem for general-purpose machine-learning. Many of the packages target linear modeling, data processing, statistics, or deep-learning individually without consistent direction or types. This is problematic because we have all of these independently created tools that depend on each-other, yet somehow do not work together — which is just silly as Julia was made to easily be extendable using sub-typing. If you would like to learn more about this package, you may visit
lathe.ai
or the Github page,
github.com
I remember when I originally started in Julia in 2018, and for the life of me could not find a good way to train-test-split my data. I ended up having to write a function to do it, and I would use that same function each time I needed to train-test-split something. Lathe aims to set up some level of consistency, although it is object-oriented, the tools are made to be extended. If we think of Python, SkLearn set a standard for predicting with models that has been followed by other packages since.
Today I wanted to work with some huge data-sets, and truly put Lathe through its paces in order to both demonstrate how to actually use the module to some extent, and also demonstrate how capable the module actually is. To get started, let us generate an absolute TON of random data. I would like to do both a categorical and a continuous option with both encoders and scalers put into a pipeline. First lets import our dependencies:
using DataFramesusing Lathe.preprocess: TrainTestSplitusing Lathe.preprocess: StandardScaler, OrdinalEncoder, OneHotEncoderusing Lathe.models: LinearRegression, RandomForestClassifier, Router
For the continuous data, I pretty much just used randn() to generate some random numbers. As for number of observations, I chose 5,000,000. Millions of observations are essentially impossible to work with in most of the industry standard tools, even from a 1-dimensional perspective. I do not imagine Python would even create this array, though I could be wrong. I generated this data using the randn() function:
x = randn(5000000)y = randn(5000000)
Now categorical data is a bit more tricky, as Julia really does not have a random choice function like Python. You can draw samples iteratively, that is about it. I wrote this neat little function, making use of Lathe’s ordinal encoder object in order to create a lookup dictionary, reverse it, and then index it with integers in a given range:
function randchoice(opts::Array{String}, count = 5)mapping = OrdinalEncoder(opts).lookupinverse_dct = Dict([pair[2] => pair[1] for pair in Set(mapping)])[inverse_dct[Int64(rand(1:length(opts)))] for i in 1:count]end
Next I created some categorical options in the form of strings. I created 3,000,000 of these, because I thought I would challenge the model and put the CUDA implementation of this model to the test at the same time.
opts = ["A", "B", "C"]caty = randchoice(opts, 3000000)catx = randchoice(opts, 3000000)
Finally, let us condense our data into two dataframes:
df = DataFrame(:X => x, :Y => y)catdf = DataFrame(:CATX => catx, :caty => caty)
And we will be splitting each dataframe, I timed these using the time macro, and from here on out everything Lathe will have some time screenshots below it!
@time train, test = TrainTestSplit(df)
Then I made a trainX and trainy array:
tr = :Xte = :YtrainX = Array(train[!, tr])trainy = Array(train[!, te])testX = Array(test[!, tr])testy = Array(test[!, te])
Next, let’s create a standard scaler to scale our incoming X:
@time scaler = StandardScaler(trainX)
Then we will use its predict() function in order to get some data to train with:
@time scaled_tX = scaler.predict(trainX)
Finally, let us fit this to a basic linear regression model:
@time model = LinearRegression(scaled_tX, trainy)
Now I will make a pipeline with our scaler and model using the addition operator:
pipe = scaler + model@time yhat = pipe.predict(testX)
For our categorical model, I did the same exact splitting process as before:
@time train, test = TrainTestSplit(catdf)
Then got that data into 1-dimensional arrays:
tr = :CATXte = :catyctrainX = Array(train[!, tr])ctrainy = Array(train[!, te])ctestX = Array(test[!, tr])ctesty = Array(test[!, te])
After that, I fit an ordinal incoder:
@time encoder = OrdinalEncoder(ctrainX)
And once again used the predict() function to predict:
@time tx_enc = encoder.predict(ctrainX)
Finally, I finished off by fitting a Random Forest Classifier:
@time model = RandomForestClassifier(tx_enc, ctrainy, # cuda = true # < uncomment for cuda!)
Next, let us use the addition operator again to create another pipe for categorical input:
catpipe = encoder + model@time catpipe.predict(ctesty)
I think it is clear that Lathe is powerful and speedy for preprocessing data for most machine-learning scenarios. I think that being able to go through this big data so easily with these preprocessing tools definitely gives Lathe a significant amount of promise to stay in the industry, along with Julia. This could be the case, even if the systems were used only to feed Python-written machine-learning models. Of all of these tests, the only one that ever went over 10 seconds in compilation time was the random forest classifier, which I think still has a pretty radical time for dealing with this much data.
I tried to compare these times with CUDA times, but it would appear as though my CUDA is not working, which ruins the fun! If you happen to have a graphics card and CUDA, you can feel free to try it out in the notebook and let me know how it goes, that would mean the world to me! Thank you for reading, and I hope this article went to show just how powerful the Julia language can potentially be with the right steam behind it!