Pythonic Big Data Using Julia?. Can Python handle large heaps of data... | by Emmett Boudreau | Towards Data Science

So recently, I have been experimenting with the idea of using Julia as a back-end for Python. While this is certainly a great concept in theory, some things do appear to have been lost in translation. This is a discussion that I have been having with several Julia and Python scientists, as the road to using Julia from Python is one that is at the moment traveled on very little. I also touched on this concept in another article, which you can read here:
towardsdatascience.com
In this article, I discussed the potential that Julia could have to replace C as a Python back-end. While to some extent, I do think that this is a weird idea — as Julia is not too difficult and is quite similar to Python in syntax, there would also be a significant advantage to this. The reason why I think using a compatibility layer from Python with Julia could be useful is that Python has far more mature web-frameworks that are far easier to deploy, and Python has far more high-level APIs that have been created for it. Being able to combine these powerful tools with the speed of Julia to handle more data and larger algorithms would be a dream come true.
As part of the tech-stack for Python and Julia symbiosis, we will need a package called TopLoader. TopLoader is a Pythonic interface for managing Julia virtual environments and packages. You can add TopLoader with Pip like so:
sudo pip3 install TopLoader
You can find TopLoader on Github here:
github.com
To set up TopLoader, you’ll first need to enter the Python REPL and import TopLoader. This will install and update PyCall, and set up the Julia executable to be ran from Python. Of course, you will also need Julia to be properly installed for this to work. There is a known issue with the Apt package manager’s installation of Julia where TopLoader will not be able to find Julia.
To resolve this,
DON’T INSTALL JULIA THROUGH APT!
Here is some Julian advice straight from the mouth of Julia Computing: Install the Julia binaries straight from Julia Computing, or http://julialang.org . Using a package manager version of Julia will create the following issues:
You will likely have an outdated version of Julia. For example, I am on Fedora and Dnf only has Julia up to version 1.2, whereas the current version is 1.4. Without the latest version of Julia, you will lose access to new features, such as parallel computing with @spawn, meaning you won’t be able to run a thread on anything but a multi-line for loop.
Your Julia executable could potentially end up in the wrong place. Your package manager places files on your system in a particular way, and having an executable link to /opt is certainly more optimal than having to predict where each package manager is going to put your Julia binaries, and then having to export them in Bash via ~/.bashrc or by running the command prior to starting Julia each time.
When you finally do get to a point where you can import TopLoader, make sure it is done specifically through the REPL. The setup involved with interpreting across Julia and Python cannot take place anywhere else.
(Notebook)
lathe.ai
Lathe uses Julia’s syntactical expressions to emulate classes in a typical duck-typed language like Python and C++. This is great because it means that models get an initialization prior to being ran through a prediction method. This means that we can fit a model like so:
model = LogisticRegression(trainX, trainy)
And this expression will go ahead and calculate any values that are needed for the LogisticRegression model to predict. Then we can call the predict method, which is a child of our new model object, to get our ŷ.
ŷ = model.predict(testX)
To demonstrate using Lathe from Python, we will first have to create an environment with TopLoader, and then add Lathe to it:
This environment already exists, and already has Lathe in it. However, we will need to instead install the Unstable branch of Lathe so that we can modify and test it. While we can do this from Python, I would also like to demonstrate how we could do this from the Julia REPL, which in my opinion is a lot easier and makes a lot more sense. All we need to do is enter the Julia REPL in our current directory:
And then enter the Pkg REPL with ] and activate our environment with the activate command:
Now we can add or remove any packages we desire.
Now if we reload our environment in TopLoader and use the List() function, we get the new updated version of Lathe.
We can import Lathe into Python using the Using() method from our environment:
Whenever we create a model type, everything works exactly as expected.
And just as well, so does the predict() method:
In order to determine if we are actually getting a performance gain from using Julia as apposed to Python, we’ll need a baseline. To do this, I carried over the same Linear Regression function translated into Python. This is a rather basic machine-learning model, so fortunately Python will be able to complete a lot of our tests on its own.
def dot(x,y):    lst = []    for i,w in zip(x,y):        lst.append(i * w)    return(lst)def sq(x):    x = [c ** 2 for c in x]    return(x)class LinearRegression:    def __init__(self,x,y):        # a = ((∑y)(∑x^2)-(∑x)(∑xy)) / (n(∑x^2) - (∑x)^2)        # b = (x(∑xy) - (∑x)(∑y)) / n(∑x^2) - (∑x)^2        if len(x) != len(y):            pass        # Get our Summations:        Σx = sum(x)        Σy = sum(y)        # dot x and y        xy = dot(x,y)        # ∑dot x and y        Σxy = sum(xy)        # dotsquare x        x2 = sq(x)        # ∑ dotsquare x        Σx2 = sum(x2)        # n = sample size        n = len(x)        # Calculate a        self.a = (((Σy) * (Σx2)) - ((Σx * (Σxy)))) / ((n * (Σx2))-(Σx**2))        # Calculate b        self.b = ((n*(Σxy)) - (Σx * Σy)) / ((n * (Σx2)) - (Σx ** 2))        # The part that is super struct:    def predict(self,xt):        xt = [self.a + (self.b * i) for i in xt]        return(xt)
I also imported Pandas and read in two CSV files I have on my machine that I use for speed tests. The data-set has 10,000,000 observations in it, which might still be apt to perform, but will almost certainly give Python a great workout. To time these calculations, I will be using the IPython magic command timeit.
%timeit LinearRegression(df["X"],df["Y"]).predict(testdf["X"])
Our result came out to 8.78 seconds.
Compare this with Julia’s speed on this exact calculation, where the result came out to 0.362333 seconds
But what about using TopLoader from Python? Could we expect performance similar to Julia, Python, or somewhere in between? The performance could also take a significant hit due to the translation layer in a worst-case-scenario. Unfortunately, that seems to be precisely what happened with TopLoader coming in with a solid 1 minute and 10 seconds.
Awful!
I’m going to have to accept the null hypothesis on this one.
The short answer is no, I was correct in my estimation that the layers of compatibility caused too much slowdown. However, I think that this observation could insight the reality that with a better system, Julia from Python could be fantastic. The problem with TopLoader is that it needs to translate Pythonic data into Julian data, and then translate that data back into Julian data. Perhaps if there was a way to run a Julia instance in the background that could receive large heaps of data from Python more efficiently, there might be a way to get this working.
With the need for a better system clearly illustrated, perhaps I will start a new project to achieve just that. In the meantime, however, Julia is certainly of no assistance to Python in terms of handling data, and using compatibility layers seems to be slower than using Python itself. While the future of this concept might be bold, and could even mean faster Python packages with less effort, the future certainly isn’t now.
Regardless of TopLoader’s failings in this aspect, I still think it is a pretty cool package. Bridging Julia virtual environments with Python is certainly a cool idea, and I can only hope that eventually a way can be determined to bring the full capabilities of this concept into fruition.