Julia Speed Battle: Syntactual, Recursive, And Iterative Loops | by Emmett Boudreau | Towards Data Science

With all of the writing i have been doing for the machine learning module, Lathe, and the new algorithms i have been implementing into 0.0.4, my mind has been set on optimization to really set the bar for speedy machine learning. A couple weeks ago, while working on a time-sensitive project, I was trying to build an algorithm from an enormous data-set with Python, and the Jupyter kernel would crash just from trying to read it in. On the other hand, I thought to my Julia experience, and without even having to go to Scala I was well able to read in data-sets with over thirty-five million observations, and time the predictions out to only a few seconds at the most with Lathe. It was a difficult concept to grasp that rather than hitting the limitations of the hardware that I was running on, I was hitting the limitations of the language.
This got me thinking about low-level algorithms and their place in the speed of calculation in the form of looping. After a-lot of thought, I set my mind to testing out the validity and severity of the well-known computer science ideologies about processing and language specifically in Julia. We all know that Julia doesn’t follow anybody’s rules, and really is a unique language in that regard. I have an article here on what you should know about Julia before using it for ML you can checkout for more information.
In the early day computing-sphere, typically iteration was always considered to be the fastest type of loop for manipulating and individualizing indices in an array. With this in mind, a lot of older systems could transfer less stacks with recursion than modern computers, for obvious reasons (mb of memory vs gb of memory), however a lot of the legacy languages from the 80’s are still moreso optimized for iteration over syntactual and recursive loops. It should also be noted that recursion is frequently frowned upon, and regarded as the slowest way to do a quick task. As for syntactual expressions, typically the result is completely dependent on your language and its compiler.
On that particular note; I would like to introduce you to Julia’s (stock) compiler:
just-ahead-of-time (AOT)
And with AOT in mind we can assume that many other compilers and languages as a whole are probably not going to produce the same and possibly produce entirely different results. I wanted to point this out, because I was personally taken back by the results of this test, and it really interests me on how they were able to manipulate the language and compiler to such a degree to make Julia such a sweet language. I guess that’s MIT for you.
Our data-set is going to be the one I used in this project, for the sheer size of it. The cleaned shape of the data is 864,863 rows × 3 columns, which should be more than enough observations for a decent speed test.
I’m going to be using a Linear Regression function as a control to get the speed tests, and the only changes that will be made are the parts necessary for the loop (one line at most,) and the loop itself. The formula for Linear Regression is:
y^ = a+(b*x^) where
a = ((∑y)(∑x2)-(∑x)(∑xy)) / (n(∑x2) — (∑x)2)
Andb = (x(∑xy) — (∑x)(∑y)) / n(∑x2) — (∑x)2
So let’s write our iterative version of a simple linear regression function:
function pred_LinearRegression(x,y,xt)    # a = ((∑y)(∑x^2)-(∑x)(∑xy)) / (n(∑x^2) - (∑x)^2)    # b = (x(∑xy) - (∑x)(∑y)) / n(∑x^2) - (∑x)^2    # Get our Summatations:    Σx = sum(x)    Σy = sum(y)    # dot x and y    xy = x .* y    # ∑dot x and y    Σxy = sum(xy)    # dotsquare x    x2 = x .^ 2    # ∑ dotsquare x    Σx2 = sum(x2)    # n = sample size    n = length(x)    # Calculate a    a = (((Σy) * (Σx2)) - ((Σx * (Σxy)))) / ((n * (Σx2))-(Σx^2))    # Calculate b    b = ((n*(Σxy)) - (Σx * Σy)) / ((n * (Σx2)) - (Σx ^ 2))    # Empty array:    ypred = []    for i in xt        yp = a+(b*i)        append!(ypred,yp)    end    return(ypred)end
We will be using the timer from Julia to time our predictions. I figured this would be the ideal way as we can eliminate any middle-ware in that we can time the calculation of the values without any additional overhead timers.
@time pred_LinearRegression(trainX,trainy,testX)
And now we’ll do the same for a recursive loop:
function pred_LinearRegression(m,xt)    # a = ((∑y)(∑x^2)-(∑x)(∑xy)) / (n(∑x^2) - (∑x)^2)    # b = (x(∑xy) - (∑x)(∑y)) / n(∑x^2) - (∑x)^2    # Get our Summatations:    x = m.x    y = m.y    Σx = sum(x)    Σy = sum(y)    # dot x and y    xy = x .* y    # ∑dot x and y    Σxy = sum(xy)    # dotsquare x    x2 = x .^ 2    # ∑ dotsquare x    Σx2 = sum(x2)    # n = sample size    n = length(x)    # Calculate a    a = (((Σy) * (Σx2)) - ((Σx * (Σxy)))) / ((n * (Σx2))-(Σx^2))    # Calculate b    b = ((n*(Σxy)) - (Σx * Σy)) / ((n * (Σx2)) - (Σx ^ 2))    [i = a+(b*i) for i in xt]    return(xt)end
And the syntactual:
function pred_LinearRegression(m,xt)    # a = ((∑y)(∑x^2)-(∑x)(∑xy)) / (n(∑x^2) - (∑x)^2)    # b = (x(∑xy) - (∑x)(∑y)) / n(∑x^2) - (∑x)^2    # Get our Summatations:    x = m.x    y = m.y    Σx = sum(x)    Σy = sum(y)    # dot x and y    xy = x .* y    # ∑dot x and y    Σxy = sum(xy)    # dotsquare x    x2 = x .^ 2    # ∑ dotsquare x    Σx2 = sum(x2)    # n = sample size    n = length(x)    # Calculate a    a = (((Σy) * (Σx2)) - ((Σx * (Σxy)))) / ((n * (Σx2))-(Σx^2))    # Calculate b    b = ((n*(Σxy)) - (Σx * Σy)) / ((n * (Σx2)) - (Σx ^ 2))    f(X) = a.+b.*X    ypred = f(xt)    return(ypred)end
These results might just shock you:
Recursion with a mean of 0.11437939999999999 secondsSyntactual with a mean of 0.1685546 secondsIteration (for loop) with a mean of 0.4208722 seconds
Recursion with a mean of 0.11437939999999999 seconds
Syntactual with a mean of 0.1685546 seconds
Iteration (for loop) with a mean of 0.4208722 seconds
Iteration was last!?
More than anything, I suppose that this really illustrates that Julia is very well optimized to be a functional language. Recursion is a fantastic building block of functional programming, and is one of the basic algorithms to pioneer the concept.
With these results, I wanted to minimize any uncontrolled variables, so the first thing I targeted was memory, given that recursion is heavily reliant upon memory.
As seen on the left in the system monitor, we certainly weren’t at a shortage of processor speed or memory, with the resources well beneath over-used.
So the next thing I decided to do was broaden the spectrum, and see if a slower run-time had a significant effect on the results, and to do this I under-clocked my processing unit to 1824 MHz, and set the governer to only allow the use of two cores. This was easier said than done, because it took a decent while to get all of the calculations for the data (it was so slow.) I originally went for a single core, but that was too far for me feeling like I was in the 90's waiting twenty minutes to switch windows.
And this yielded significantly higher speed, but the trend completely continued, and this made no difference on the result...
Recursion with a speed of .33Syntactual with a speed of .39Iteration with a speed of 1.37
Recursion with a speed of .33
Syntactual with a speed of .39
Iteration with a speed of 1.37
That’s right, the difference was actually that significant, and with that in mind, I’ll certainly be targeting recursion whenever possible in Julia from now on! I’m not sure where this thirst for speed comes from, but I’m really impressed with Julia’s ability to work with such large data easily. I really do enjoy doing these tests because the results make me a much better programmer who can manipulate the language and the compiler more efficiently. Hopefully this was very informative for any other data scientists working in Julia as well, I know I will certainly be implementing these results into all of my functions from here on out.