Machine-Learning In Julia is FINALLY Getting Better | by Emmett Boudreau | Towards Data Science

Whenever I first started using Julia, an issue that I, as well as many other defectors coming from languages with strong ecosystems like Python and R took, is specifically with data processing. Data processing seems to of been quite neglected in Julia, often with individual packages being used to solve a single problem, and some problems being missed entirely. Sometimes there would be functions already available, but they would be incredibly difficult to use, requiring you to assert types and convert data before passing it through any pre-processing algorithms. For this reason not only was getting a model trained difficult, but putting a model into a Pipeline and deploying it was entirely out of the question.
However, with the futuristic and exciting strides that Julia developers have made, pre-processing has evolved further than I ever would have originally thought possible. As a result, Julia has become a much more viable option for machine-learning in general. With the luxurious additions to Genie, and its respective documentation as well, deploying APIs has become a
cake walk.
Genie manages all of the virtual environments for you, and now that documentation has been released, and is constantly being improved by the amazing developer Essenciary (Adrian Salceanu), making Julia more usable as a web framework everyday.
This might come as a surprise, but among the greatest challenges that Julia currently faces is PyCall.jl. This is because as it stands, too many packages depend on it, and in turn depend on Python. While I fully understand, the Python ecosystem is far greater than Julia’s, Julia, in turn, loses all of the speed gained, and with PyCall can even be slower than using Python itself,
so why even use Julia?
Train-Test-Splitting is an essential tool that is entirely required for any stretch of machine-learning. In order to validate an algorithm, you must first have a train and a test set, and some even opt for a validation set as well. Referring to this thread,
discourse.julialang.org
on Julia discourse, I think it’s pretty obvious that Test-Train-Splitting for data-frames has been overlooked to some extent. MLDataUtils is likely the package that most have used for Train-Test-Split, and take my word for it:
It could certainly be better.
Fortunately, for a few months now, there has been better from the module Lathe.
using Lathe.preprocess: TrainTestSplittrain, test = TrainTestSplit(df)
And we can even adjust our split percentage, like so:
train, test = TrainTestSplit(df,0.5)
This will split the data in half for our test and train sets.
Something that I think is all too common inside of Julia modules is SkLearn formality. I don’t think that we should be embracing or enforcing this conformation from an entirely separate language based in an entirely separate paradigm. A great example of this is the feature scalars inside of Julia. Most if not all feature scalars conform to SkLearn’s writing style which has defined Python machine-learning modules since 2010. But the reason why SkLearn’s scalars behave this way is for one sole reason, and that is SkLearn’s wonderful Pipelines.
In Julia, we don’t have classes, and in turn we can’t have methods be children of an object. Instead, we have structs which can take a method to perform an action with the data that the struct holds. Additionally, the only pipelines that Julia has to offer are syntactical pipelines — which are fine, but certainly eliminate the point of the conformation to SkLearn, given that they don’t use the same methods. To put this in perspective, this is the formula for a Standard Scalar (Z-Score normalizer):
With such a simple formula, the only things calculate on fit are the sample mean and the standard deviation. With Julia’s speed, these calculations are of course done in microseconds, as they are similarly in Python. This is not in any stretch of the imagination a difficult calculation to make, being done in one concise for loop and two additional lines. Fortunately, Lathe strikes again with an easy method that takes any array:
using Lathe.preprocess: StandardScalarscaledX = StandardScalar(trainX)
Pretty simple, right?
In addition to the standard scalar, Lathe also comes with plenty of other scalars, and in the future there are plans to implement a Unit Length Scalar, as well. As of right now, listing off Lathe.preprocess with the ?() method will show us all of the scalars available:
Rescalar
Arbitrary Rescale
Mean Normalization
Categorical encoders are another region of the Julia ecosystem that has been extremely lacking. There is only one encoder available, and typically it is used through Flux, and it is called the one hot encoder. To give you an idea of how senselessly over complicated Flux’s one hot encoder is, you can take a look at the source here:
github.com
The code totals out to 130 lines in that file, a size that is nowhere near necessary, in my opinion. To use Lathe’s one hot encoder, we simply call on the method “ OneHotEncode”:
using Lathe.preprocess: OneHotEncodeencoded_df = OneHotEncode(df,:columnname)
And here is the code that computes that, done in just 7 lines total:
Pipelines are a very important feature to SkLearn and machine-learning as a whole. Julia is severely lacking a really good library for using pipelines. Most pipelines in Julia don’t take constructed models, and disallow for any sort of easy serialization reading and writing. With Lathe, however there are in fact pipelines which use the same prediction function as the rest of the Lathe models, and allows for an iterable set of steps as well as a model to predict with. If you’d like to read a full, though outdated tutorial on Lathe’s pipelines, you can check it out here:
towardsdatascience.com
Additionally, for valuable documentation, you can simply use the ?() method on Lathe.models.Pipeline.
I believe a lot of Julia modules leave a lot to be desired. Some don’t offer any data-frame compatibility whatsoever, while others make even more problems by senselessly over-complicating simple algorithms. The Julia language is extremely flexible, and that can be used to our advantage not only in the notebooks, but also in the modules. I think that the Julia ecosystem puts off a lot of new people coming to the language for this exact reason, and that is frustrating to me. The truth is, in terms of packages, right now Python easily has done it better. I think this can easily be changed, and with eyes into the future I am very excited for where the Julia ecosystem, including Lathe will be!