How to embed your Julia code into Python to speed up performance | by Yuval Blau | Towards Data Science

This blog post is about working with Python and Julia together and aimed mainly for Pythonists Data scientists who want to quickly accelerate their run time performance and do it by another programming language that gains its popularity within universities, companies, and users.
By the end of this post, you will gain:
An environment of docker image that also contains Julia and Python
A straightforward comparison between Julia and Python in terms of speed.
Three different methods to embed Julia code into Python.
I hope you will find it informative!
To set the environment, I used this docker image containing Julia, Python, and Jupyter:
https://hub.docker.com/r/jupyter/datascience-notebook
A step by step guide for dockers can be found in the references section.
Before I demonstrate how to embed Julia code into Python to boost the performance, I want to convince you that there is a value to use Julia, and it can work much faster than Python when using for loops.So, let’s run a simple for loop in Julia and Python and compare their running times:
In Python:
def for_loop(iterations = 100):    a = 0   for i in range(iterations):        a = a+1    return a
It took about 1 second.
%%timeres = for_loop(iterations = 10_000_000)>>> CPU times: user 953 ms, sys: 0 ns, total: 953 ms>>> Wall time: 951 ms
And in Julia:
function for_loop(iterations = 100)        a = 0        for i in 1:iterations               a = a+1    end            return aend
It took 0.000001 seconds:
@time res = for_loop(10_000_000)>>> 0.000001 seconds (1 allocation: 16 bytes)
Quite a difference!
But it was just an extreme case; usually, the differences won’t be that significant.
So, after we are convinced that there are some potential here, let’s look at a real example from the NLP area in Machine Learning.We will pick a task that involves for loop to use Julia’s strength.We will use the Stemming task (the readers who are not familiar with this task can read about is here: https://en.wikipedia.org/wiki/Stemming).
We will be using the data from the Shakespeare-hamlet dataset on the NLTK package. It can be accessed like this:
from nltk.corpus import gutenbergdata = gutenberg.raw('shakespeare-hamlet.txt')data = data.replace("\n", " ")data = data.replace("  ", " ")
This is how the data looks:
"[The Tragedie of Hamlet by William Shakespeare 1599] Actus Primus. Scoena Prima. Enter Barnardo and Francisco two Centinels. Barnardo. Who's there? Fran. Nay answer me: Stand & vnfold your selfe Bar. Long liue the King Fran. Barnardo? Bar. He Fran. You come most carefully vpon your houre Bar. 'Tis now strook twelue, get thee to bed Francisco Fran. For this releefe much thankes: 'Tis bitter cold, And I am sicke at heart Barn. Haue you had quiet Guard? Fran. Not a Mouse stirring ..."
We will also use NLTK to do the Stemming part:
import nltkfrom nltk.stem import PorterStemmerporter = PorterStemmer()%%timestem_words = []nltk_tokens = nltk.word_tokenize(data)for token in nltk_tokens:        new_token = porter.stem(token)    stem_words.append(new_token)>>> CPU times: user 1.52 s, sys: 0 ns, total: 1.52 s>>> Wall time: 1.57 s
It took us 1.57 seconds.
Now for the interesting part.
There are few methods to embed your Julia code into Python, and we will cover three of them:
Run whole expressions of Julia
Run Julia with “magic” command
Run Julia with a script
First, we will import the relevant modules:
from julia.api import Juliajl = Julia(compiled_modules=False)from julia import Mainjl.using("TextAnalysis")
(We imported the relevant packages from Python and Julia).
Let’s run the code, and then understand what happened here:
%%timeMain.data = dataMain.token_data = jl.eval("tokens_data = TokenDocument(data) ; return(tokens_data)")stem_list = jl.eval("stem!(tokens_data);stem_tokens_data =  tokens(tokens_data) ; return(stem_tokens_data)")>>> CPU times: user 747 ms, sys: 112 μs, total: 747 ms>>> Wall time: 741 ms
So, in the first row, we store the data variable in the Main.data variable so Julia can read this variable.
Main is a module that we imported from Julia. We use it here to store the data variable in Julia’s format to use it later (when we call the TokenDocument(data)).Please note that if we didn’t write that line (Main.data = data), Julia can’t recognize any variable with the name ‘data’.Next, we tokenize the data in Julia and store it in Main.token_data variable. (Note: we should use the return command to store the variable).Then, we do the stemming on the tokenized data and store it in a regular python list.
So, all we need to do to run Julia’s expression in Python is to use the jl.eval function, concatenate Julia’s expressions with “;” between them, and finish with the return command.
It took us about 740 milliseconds — about half a time from Python.
%load_ext julia.magic
To use the magic command, we first need to load it.Please note that this method will work only in Jupyter Notebooks.%%time
tokens = %julia TokenDocument($data)%julia stem!($tokens)stem_list = %julia tokens($tokens)>>> CPU times: user 849 ms, sys: 3.39 ms, total: 852 ms>>> Wall time: 845 ms
So, after we load the julia.magic, we run those three lines of codes.
We should pay attention to two things:
put the % symbol before using syntax from Julia.
Use the $ sign before referring to a python variable.
And like before, we first tokenized the data, then stem it, and finely store it as a Python list.
In this method, we can write an entire independent Julia script and call it from python.
So, if we write a script named julia_stemming.jl, and store there this Julia function:
function stemming_document(document_string)        tokens_data = TokenDocument(document_string)    stem!(tokens_data)    stem_tokens_data =  tokens(tokens_data)        return stem_tokens_dataend
We can just call this function from our Python script/notebook:
%%timejl.eval('include("julia_stemming.jl")')Main.data = datastem_list = jl.eval("stemming_document(data)")>>> CPU times: user 602 ms, sys: 3.24 ms, total: 606 ms>>> Wall time: 602 ms
First, we include this script to make it usable, and then we call the function inside it.
We showed that for this specific dataset (the shakespeare-hamlet.txt), Julia is more than two times faster than Python. But what happens when we use larger datasets?
We executed our stemming function multiple times with varying dataset length (duplicating our original dataset), and plotted the results:
As you can see, the largest the dataset, the more significant the difference between the run time — which makes Julia very useful when dealing with large datasets and for loops.
We used this code to calculate the difference:
time_list_python = []time_list_julia = []dataset_length = []duplicates_list = [1,2,5,10]for t in duplicates_list:    data_repeat = ' '.join([data] * t)    dataset_length.append(len(data_repeat))        time_start_python = time.perf_counter()        stemming_python(data_repeat)            time_end_python = time.perf_counter()query_time_python = time_end_python - time_start_python    time_list_python.append(query_time_python)            time_start_julia = time.perf_counter()        Main.data_repeat = data_repeat    jl.eval("stemming_document(data_repeat)")            time_end_julia = time.perf_counter()query_time_julia = time_end_julia - time_start_julia    time_list_julia.append(query_time_julia)
And this code to plot:
import pandas as pdimport matplotlib.pyplot as pltdf = pd.DataFrame({"running_time_python" : time_list_python,                   "running_time_julia" :  time_list_julia,                   "dataset_length" : dataset_length})df.set_index("dataset_length", inplace=True)df.plot(figsize = (10,6), title = "Python vs Julia running time comparison")plt.ylabel('seconds', fontsize = 20)plt.xlabel('dataset_length', fontsize = 20)plt.show()
I hope I convinced you about the advantages of connecting Julia and Python.
From what I have seen, there are still multiple areas that pure Python is much better — like working with vectorized computing (such as NumPy), so we need to make sure that switching to Julia is worth the trouble.
Thank you for reading, and sure, I would be glad to hear your response.
https://julia.mit.edu/ (universities that use Julia)
https://juliacomputing.com/media/2017/09/19/julia-the-ai-language-for-next-150-years.html (companies that uses Julia)
https://juliacomputing.com/blog/2018/12/04/december-newsletter.html (GitHub stars)
https://hub.docker.com/r/jupyter/datascience-notebook (the image we used)
https://dagshub.com/blog/setting-up-data-science-workspace-with-docker/ (step by step guide for docker)
https://pyjulia.readthedocs.io/_/downloads/en/latest/pdf/ (PyJulia package documentation)
https://pyjulia.readthedocs.io/en/latest/usage.html#high-level-interface (PyJulia package documentation)
https://juliatext.github.io/TextAnalysis.jl/latest/documents/ (Text analysis package)