The Depressing Challenges Facing The Julia Programming Language In 2021 | by Emmett Boudreau | Towards Data Science

The Julia programming language has been quite an object of interest in the wonderful world of programming recently, and this is for good reason in my subjective opinion. There are so many things to love about Julia that simply are not existent in other languages. However, no programming language is without its issues, and being relatively new to the world of Data Science, Julia has a multitude of different challenges it needs to overcome in order to truly break through into this industry.
Before we dive into the problems with Julia, and certain issues that the language has right now, let us first touch on why this language is worth using and saving. Firstly, it has been coined before that all of the programming language paradigms have already been created and worked with, but Julia proved this not to be the case. Before we dive into Julia’s paradigm, let us first briefly remind ourselves what a paradigm is. A programming paradigm is just a system that is used for types and methods to interact with one another. I actually have an entire article on programming paradigms that goes a bit more into depth, and is almost a year old now you may read here:
towardsdatascience.com
Julia took the idea of a programming paradigm and centered it around the generic programming concept multiple dispatch. Multiple dispatch has its roots seeded in a small functional programming language from the 80’s called ML. Although the language itself certainly did not make a huge splash in the market, some of the genius ideas which were conceived in that code base have of course been completely considered in all sorts of modern multi-paradigm applications. And this brings us back to Julia, which has taken multiple dispatch and the parametric polymorphism generic programming concept to an entirely new level. Julia resides in its own new paradigm that was conceived with the language, the multiple dispatch paradigm.
Frankly, all of the fantastic ideas that the Julia developers came up with to use multiple dispatch, such as inner and outer constructors, and abstract-type hierarchies deserves its own article. However, given all of these interesting features, Julia is a very unique language to program in. For me subjectively, multiple dispatch feels like a very natural and intuitive way to program. Working with methods more like property of the types we are working with, with calls that can completely change what they do depending on the type passed through them, just feels like the most natural approach to programming one could have.
On top of just being an awesome language to write, Julia is fast — like really fast. Julia often trails just behind much lower-level languages like C and Rust while still remaining ridiculously simple to write. This is where the tired old motto will show its ugly head again, ideally for the last time...
“ Walks like Python, Runs like C.”
You can certainly understand why this motto was chosen. Julia code can look very similar to Python and be incredibly easy to use. Even better, the code runs nearly as fast as C in most cases, but why is this motto strange? Because Julia resides in an entirely different programming paradigm to Python, of course! Regardless, the point of this motto still makes sense, and can give one a good idea what Julia is about. Now that we understand why Julia is uncontested on many fronts, and what the language is all about, let us dive into some of the issues that the language is facing this year in its struggle to remain relevant in multiple scientific discipline.
Easily the biggest thing that affects new Julia users, and the Julia community as a whole is Julia’s ecosystem. While Julia does have the ability to call Python, for example, as well as has many different ported packages from that language and others, in many cases it is just odd to call on those languages from Julia. While PyCall and similar programming interfaces for other languages tend to be well-programmed and actually work really well with data-types, running the code in Python rather than Julia destroys the whole purpose of using Julia in the first place. Ultimately, the packages tend to go even slower than they would in Python, so why you would use Julia in this way is a very curious question.
If I were to name off some machine-learning packages I would actually recommend someone to use in Julia, I would honestly struggle to provide solid examples. Still, I will do my best to provide some prominent packages I like to use personally for each individual portion of Data Science. When it comes to machine learning, for neural networks I would definitely recommend the Flux library. The Flux library is actually quite robust, and has been in development for a while now.
Another great library you should certainly look into is Lathe, for which I am the creator. Lathe includes a few nice machine-learning models that might come in handy, with more coming soon. The biggest focus of the package with the release of the next long-term non-breaking version is preprocessing, however. In Julia, there really are not that great of preprocessing tools, encoders, scalers, and even train-test-split functions or objects are really hard to come by. That being said, Lathe includes all of those, along with a pipelining interface. Frankly, there is a lot of work to be done, particularly in the consistency and documentation department. However, this is set to change in a few weeks with the release of 0.2.0, which will be a significant step up in the abilities of the package. Other than Lathe and Flux, I would recommend Knet and GLM. Here are links to all of the packages I just mentioned. GLM is a library of generalized linear models, and Knet is a less-robust neural network framework.
github.com
fluxml.ai
github.com
github.com
I also have introductory articles for both Lathe and Flux that can certainly help to get you started, which I would highly recommend if you have interest in Julia, but are not sure where to start. The Flux tutorial will go over creating an image classifier with Flux, and the Lathe tutorial will take you through pre-processing, encoding, and fitting a random forest classifier before building, serializing, and even deploying with Ubuntu and NGINX — it is certainly a concise tutorial.
medium.com
towardsdatascience.com
Moving on on the ecosystem front, let us consider statistics. Julia has quite a hefty and mature statistics ecosystem that I actually think most developers will find no problem with. The only issue with this ecosystem is that it will be a little segmented, you can expect to add a lot of dependencies for a project that might require a lot of testing and so forth. Even further, these statistical packages are not necessarily targeted at Data Science, however, Lathe does happen to have a smaller statistics library with some great functions for working with data when it comes to Data Science-friendly statistics. These packages are StatsBase, Statistics, Distributions, Lathe, and many many more. Part of the problem is what I touched on, the packages are everywhere and often rely on each other. I think part of the reason this can actually be quite problematic is that Julia does have a thing called pre-compilation, where packages and their respective dependencies are compiled in advance — which can lead to some pretty horrible startup-to-execution times in the language when each package has twenty dependencies to pre-compile. This is actually a big complaint among Julia users, especially those who are new to the language.
I think the problem there is that Julia has an awesome typing and extension system. It is easy to extend the abilities and types from a certain module using a different module. And this is very often done across a broad array of packages. Personally, I have always been one for keeping dependencies low! And in Julia, with pre-compilation times being sometimes quite ridiculous.
Returning to the ecosystem conversation, one region I think Julia does have very well-covered is the data-visualization part of Data Science. With awesome packages like GadFly and VegaLite doing an amazing job of displaying data in multiple dimensions with stunning and sometimes interactive visualizations, Julia does truly have some awesome choices when it comes to plotting. There is of course also the Plots.jl package, which comes with GR and Matplotlib. However, I think although this is a go-to for newbies, and is easy to use, it is also a horrible choice. The pre-compilation times are just ridiculous. The visual performance is also terrible, as well as the time-to-first-plot. To top off the other two packages on this list, however, we also have Plot.ly coming into the field. Since this is a Data Science blog and Data Science publication, I doubt I need to go into much detail on what exactly Plot.ly is. Here is a link to all of the packages I mentioned that I actually love.
github.com
github.com
github.com
I also have an article that compares the visualization options of Julia, excluding Plot.ly, as it was not yet released when this article was published:
towardsdatascience.com
Finally, we will move onto likely my least favorite implementation — data management. For data management in Julia right now, there is one quintessential option, DataFrames.jl. DataFrames.jl is a rather mature, but not incredibly mature package for handling data in Julia. Part of the problem with this package is that it is, like many other packages we have touched on, very intertwined with the ecosystem. This means immediately pre-compile times are high, and DataFrames.jl also is developed rather slowly because even the biggest contributor is also working on many other packages, some of which are on the back-end of DataFrames.jl.
Needless to say, when it comes to a lot of dependencies for DataFrames.jl, DataFrames.jl cannot move forward without them. I also think that the API is a tid-bit tedious to use. Of course, this is my subjective opinion, I still love this package, but there are certainly some ways I would like to handle my data differently to how DataFrames.jl handles it. That in mind, I am actually creating my own DataFrames package, OddFrames.jl. While it is still in an early version, I would definitely check up on it, as I have been putting effort into it recently prior to releasing a new version of Lathe, which I plan to include support for OddFrames with. Here is a link to both DataFrames.jl and OddFrames.jl, for those who are interested. OddFrames.jl can be looked at as a Pandas-like API with some Julian approaches to certain aspects, and some unique features thrown in there as well. It mixes typical functional programming and object-oriented programming, along with some indexing tricks for some pretty interesting results!
github.com
github.com
Also, if you are interested in getting started in DataFrames.jl, I have another article that could potentially help you with that:
towardsdatascience.com
Another significant problem with Julia software that some might actually not expect is documentation. There are many Julia packages with absolutely no documentation at all. Of course, this makes Julian software extremely more difficult to use compared to more mature options with advanced documentation that is constantly maintained by several developers. Many of the Julian packages that are commonly used in the language for various domains, not just Data Science, are only maintained by a very small amount of people.
That being said, Julia does have a rather cool implementation when it comes to automated documentation. There is a package called Documenter.jl that will automatically create documentation web-pages for you, which needless to say is awesome! However, this also is somewhat part of the problem, because it is easy to simply call Documenter.jl with auto-doc and do nothing else — which creates a multitude of problems for packages because it is hard to find the documentation you might actually need to use. This is especially true when only automated documentation is used and there is no organization to the documentation web-pages. If you would like to learn more about Documenter.jl, you may take a look at an article I wrote about it specifically here (yes I have quite a catalog built up at this point that comes in handy.):
towardsdatascience.com
And wow, that is from over a year ago!
Another really cool thing about this documentation is that an automated bot is able to both generate it and serve pages. That being said, this exactly exists on the wonderful Julia hub. It is one of those things where Julia is so close, and has a lot of really cool ideas, but they are either in the works or not quite into fruition yet. Or in many cases — not taken advantage of nearly to their potential. I have a whole article where I talk about Julia hub and how awesome it is, as well that I really enjoyed writing — also I really love the artwork I did for it for some reason. Anyway, you can check it out here:
towardsdatascience.com
The last that I think Julia needs to overcome to break out as incredibly successful is really finding its position in the market. Julia Computing has expressed undoubtedly that their goal is not to replace Python, but instead to co-exist with Python. The question still remains, however, what exactly does this mean? As I have expressed in this article and tested in the past, Julia as a back-end for Python simply is not that feasible of an option. This is even more so the case when we consider that Python has a close relationship with C, which tends to run a lot faster as a back-end language — and frankly, while being more difficult than Python and Julia, is still not that hard of a language to write.
This raises the question,
“ Where exactly does Julia belong?”
As a personal lover of the programming language’s paradigm, and just so many aspects of the language as well, I would love to see it become a frequently used language in my domain, Data Science. Julia seemed to have gained steam in the past year or two, but also seems to have peaked a little bit late last year. We can see from the PYPL Index that Julia is about to succumb to Haskell after just falling less popular than Perl.
pypl.github.io
However, knowing the industry — these things do go up and down. That being said, I do think that this is a vital time for some ecosystem improvements to come to the language. The language simply needs better packages, with better documentation, and even more familiar with expressions. It is quite painful to come into this weird new world of Julia and having to use something like
select!(df[!, Not(:B)])
In order to drop a column from a DataFrame. I mean this is a very Julian approach to such a thing, but I think those coming from the world of
df.drop()
might find this kind of confusing thing agitating. I think Julia needs a clear sense of direction in this regard. If the goal is to become the new crown jewel of Data Science, then we must build an ecosystem to support such a thing. However, if the goal is to work alongside Python, then we need more sturdy APIs to do so. As of right now, without the purpose of developing new software for use in the future of Julia, the language is basically just a talking point that is capable of a few things.
That last paragraph might sound harsh, but I am just worried. I really love Julia, in case you have not noticed yet. I love how much it has taught me, I love how it is unique and has its own awesome paradigm that I find to be incredibly expressive. I love how it never feels like the language is in the way, and there is no other language I would ever write essentially anything in. Even when I write Python, I find myself sometimes writing Julia, or wishing I was currently writing Julia so I could do this or that to solve a problem I am dealing with.
I have been using Julia since 2017 now, that is about four years of experience in this language, and I must say — I have certainly fallen in love with it. However, the fact that I do not see a clear direction for the language, or its ecosystem, makes me genuinely upset. Julia almost feels like my baby, it really is my language. You know how you program all over the place, and you eventually find the one? I used to think that language for me was C++, but after getting into Julia, I really discovered what I love in a programming language, and Julia is certainly that.
If you are in any scientific discipline, or even a general purpose programmer, I highly recommend giving Julia a try. I believe that every programmer should strive to learn at least one programming language from each programming paradigm. That in mind, while Julia, like many other modern programming languages, is multi-paradigm and takes advantage of many generic programming concepts, I think it is a great option for learning functional, methodized programming concepts in an environment similar to Python’s. Thank you very much for reading my article! Please support the Julia ecosystem and language, because it is truly a fantastic — and still underrated thing! Have a great day!