Parallelization caveats in R #1: performance issues | by Imre Gera | Towards Data Science

About this work. I started this as an individual article to sum up parallel processing problems I came across while working on a research project. After seeing that the article had become fairly long, I decided to split it up into multiple parts, each covering a specific kind of pitfalls that can, in the best cases, lead to errors or, in the worst cases, to just plain bad performance or unseen consequences. As a result, you’ll [hopefully] see multiple articles covering these pitfalls with solutions that I’ve successfully applied to solve my issues. This article deals with the basic setup and possible performance issues.
Most of us have reached a point in our work where our code was simply not fast enough because it only utilized a single CPU core and took hours to finish. To solve this, we’ve turned to good ol’ multiprocessing, hoping it would remedy the slowness of our calculations. Then, at the end of the much faster, 30-minute-long run we see an error that simply doesn’t tell us what went wrong, maybe something like Error reading from connection 7(this is not the article that covers this specific error message, if you came from a search engine). In this series I’ll summarize my experience with parallelism in R and try to point out things that can go very wrong during parallel computations. I will not dwell deeply into how parallel processing works — though I will attempt to at least define the concepts I’m using — , the series is meant to be a collection of things to consider and look out for when deciding to switch into parallel mode.
In this first part, I’ll talk about why we would turn to parallel workflows (just to give us a bit of motivation), then I’ll show some techniques to achieve it in R and finally we’ll start combining them to unleash utter chaos on our computer and we’ll see why this or that should not be used together or why doing something a certain way can slow your calculations. Okay, maybe I was a bit too dramatic, but I’ve recently managed to freeze out a 32-core server by giving it a touch too many things to work on — unwillingly, of course.
Ultimately I decided to write these articles because I myself bumped into these problems while working on my code and there was no ready-made cookbook (at least I haven’t found one) that would point out the mistakes I made — some were arguably language or package-specific, too, but most of them were difficult to figure out — and give me some advice on how to work around them. While a larger portion of the series contains R-specific advices, some of the problems I’ll talk about are language-agnostic, meaning that if you use MATLAB or Python or Assembly (hah) to do similar work, you might just face the same issues as I had. Also note that I’m not a parallelization master (neither the R guru), so if you find something that is incorrect or requires a better explanation, please do not hesitate to tell me!
Have you tried executing a long for loop with some heavy maths inside? Like a series of matrix inversions, the results of which then get multiplied, multiple functions are called on them, possibly even some quadratic optimization problem is solved, etcetera? If you have and opened up Task Manager (or its equivalent on your favourite operating system) while your code was running, you’ve probably seen that the R process is not quite using your whole processing power, but rather just a small chunk of it. That CPU usage percentage could range from about 50% (on a 2 core CPU) to as little as 4–5% (on 24 [logical] cores). You might just ask the critical question here:
Wouldn’t my calculations run faster if R was using 100% of my CPU?
The answer is unfortunately just: probably. See, before we dive into making our code use more of our CPU, we have to understand a few things about how we’re calculating stuff, more importantly, whether our calculations’ results depend on each other. Generally speaking, if you have two tasks that depend on each other, those cannot be executed in parallel (what would the second one do without the first one’s results?). Otherwise, if your tasks don’t depend on each other, they can probably be run in parallel. You can also think of your tasks as the bodies of a for loop: if you need data from the previous iteration, that cannot be run in parallel.
To create a more grounded example, we’ll calculate the sums of the rows of a matrix. Yes, there is a function called rowSums to do this a lot faster (which you should use instead of the following implementation), but this example should be simple enough to help us understand the basics. Here’s one way to do this:
This code will calculate the sum of each row in the matrix and store it in the results vector (please try to avoid allocating vectors like this, because they will have to grow dynamically and that is costly, a bonus tip is to use results = numeric(nrow(bigMatrixWithLotsOfData)) instead — given that you know that the results will be numbers). As a result, we will calculate the sum of each row sequentially, one after the other. But the thing is, for whichever i we do not need any other iteration’s result! So, if our calculations do not depend on each other, we could theoretically run them in parallel, giving some work to our other CPU cores!
By the way, I’ll be mentioning for loops a lot because a popular parallel solution in R also uses them — let’s not get ahead of ourselves, though. Now there would be a lot more to discuss in terms of what contributes to errorless parallelism (thread safety etcetera), but this is not a general parallel computation lecture, it’s about what can go wrong when doing something parallel.
Okay, so what does it take to calculate the row sums in parallel? There’s a few ways you could go about it, but all solutions involve starting multiple, independent computations at once (like creating an R instance for each i) and when they all finished, just glue their results together. Now comes the part where I’ll show you a way to do this.
In R, a typical way to do parallel calculations is with the help of the foreach package (here’s some more information about how you can use it). It provides a convenient way to assign your tasks (jobs) to multiple processes, without the hassle of figuring out how and when to schedule something to a certain process. When these processes finish, the package also helps you concatenate the results into a single list or vector (or whatever, actually). In itself, it might not be enough, because you’ll need something that provides a so-called parallel backend (that handles the creation and destruction of processes, along with the communication between them). One such back-end is provided by the package doParallel. Without going too deep into how they work, here’s a code snippet that does the parallel row summing for us:
I inserted some longer comments to show what these instructions are for. The snippet should give you a quick overview of how the package can be used to run parallel computations. Generally, you’ll want to supply a list to the foreach construct in an item = theListToProcess form, then you can refer to the current element of the list with the item variable in your child processes — you can choose (almost) any name for it.
Unfortunately, our real-life calculations are rarely this easy to implement. Maybe we’re running complex simulations (calculating covariances on multiple matrices, solving linear programming problems and so on) and our results appear only after executing a handful of functions. Maybe you’ve created your code to run sequentially and now you want to make it run in parallel. Maybe you’ve made your code run in parallel but it’s actually slower than running it sequentially (yes, that can happen, we’ll see). So many things can go wrong, but if you pay attention to a few of them, you can write code that has a lower chance to cause headaches.
If you already have some code and are thinking about making it faster with parallel execution, it’s a good idea to stop and think about the steps it goes through to calculate the final results. Creating a simple flowchart can greatly help, too — take a look at the following quick sketch:
The image shows a process that assembles portfolios from some parts of a dataset (eg. creates a portfolio based on data between 2020–06–01 and 2020–06–05, then between 2020–06–08 2020–08–12 and so on) and aggregates the results of these operations. In the green region we can see that the processes assembling the portfolios do not depend on each other, only on the data that was acquired beforehand, so we can rearrange them to run in parallel. Generally, for loops are great place to start looking around: if you’re not passing data from one iteration to another (and you also do not modify data outside the loop), you should be fine making that loop run in parallel (although it’s easy to try to parallelize everything, see the Performance section).
If you have nested for loops and want to turn one of them into a parallel task, make sure you do the outermost loop that can still run independently (if it loops more than just a few times).
This can be a daunting task and there really is no universal recipe to do this: you’ll have to make the call to turn something into parallel or leave it as it is. Alternatively, you might have to refactor your code to make it easier to parallelize. Now — after this long-long introduction— let’s jump into what you actually came for:
What could possibly go wrong (part 1)?
In the remaining part of the article, we’ll take a look at some seemingly subtle issues that do not necessarily crash your R session or throw errors, but instead sneakily cripple your computation’s performance, possibly even bringing your computer into an unresponsive state. First, I’ll talk about the overhead of parallelization that slowly diminishes as the performance gains from the parallel execution grow. Then we’ll see why mixing multiple parallelization techniques is a bad idea and finally I’ll show you two edge cases that, if combined with some of the other problems, can cause even more issues.
It’s easy to think about parallelization as a golden hammer to all our performance issues. In fact,
why shouldn’t I just run everything in parallel?
Good question, but here’s why. It’s easy to quickly draw conclusions from the example above and say “well, I have a vector of length greater than 100, let’s throw it onto 32 processes to calculate something”. The problem is that creating processes is not free, nor is the communication between them. Each of them needs to start up (at least when not forking them) and they have their own memory space (a bare-bones R instance on Windows can eat up 30–50 MiB RAM, multiplied even by just 12 processes is already 480–600 MiB — and we haven’t even loaded packages and data on them!). If you need to access the entirety of a large data frame on all of them, that data needs to be loaded onto all of them — only move variables onto your processes that you truly need, though. If you are using packages with lots of dependencies, they will use approximately 12 times more the RAM — in our 12-process example. You’ll soon see why we need to load data onto all of our processes.
Since I was using a “SOCK” cluster on a Linux server with 32 cores (ie. I was not forking my processes), I had to wait until all 32 processes were properly initialized which caused a wait time of about 45 to 90 seconds: each of them had to load multiple packages and call some functions to be ready for processing. In such cases, it’s better to rethink whether that startup time (and increased memory usage) is truly worth the performance improvements brought by multiprocessing. For me it was absolutely worth it since I was solving hundreds of thousands quadratic programming problems: here the sum of the overhead and the parallel execution time was still much lower than the sequential runtime. For small tasks however, choosing a smaller number of processes to work with or not going parallel at all can be more performant.
In the next section we’ll see another way to improve speeds without having to resort to multiprocessing and how it can cause more trouble.
If you want to make maths-heavy calculations faster, it might be a good choice to switch over from base R to Microsoft R Open (MRO). It is a slightly modified version of R that relies on a different library (the Intel Math Kernel Library (MKL)) to do mathematical calculations. What’s important to know about Microsoft R Open is that (1) it’s compatible with base R and more importantly (2) it achieves quite a lot faster speeds in matrix multiplication and similar vectorized tasks by executing them in a multithreaded fashion, thanks to the Intel MKL — a performance comparison made by Microsoft can be seen here. Hang on. What, now?
Conceptually what is being done is very similar to what we did in the the row summing example above. Multithreaded workflows, however, create separate threads, not separate processes (as with multiprocessing). If you don’t mind, I wouldn’t go into the exact technical differences, but let’s not leave a black hole here, either. In multithreaded environments, the memory is usually shared between the computations (threads), while in multiprocessing environments each process has its own memory. If you remember the previous section, this is why we have to initialize each R process in our cluster: because they are separate processes (instances) with their own memory, each of them starts as an empty R instance with no extra packages loaded. A single process can have multiple threads, so there is also a hierarchy here, but like I said, this is not a parallel computing or operating systems masterclass.
On a high level, creating threads is usually cheaper than creating processes (CPU and memory-wise as well), but in the case of threads, you have to be very careful with the memory, as you might just overwrite something that another thread is working with. In the case of processes though, this is not a problem due to separate memory spaces, but you have to handle the communication between the processes (which is done by the parallel back-end and foreach). Each has their advantages and disadvantages, but it should suffice to know that you might just be using both of them without knowing and that can lead to massive slowdowns. Can we call this overparallelization?
Let’s count. You are using MRO (Microsoft R Open) because it is faster in sequential workflows, but you notice that one part of your code can be executed in parallel, so you throw a parallel foreach into the mix. You have a 4-core CPU (4 logical cores, no hyper-threading), so you spawn 4 R processes. What you are not aware of, though, is that you are creating 4 MRO processes, that were probably not informed of the parallel situation, so each of them decides to use 4 threads for their maths. You start your parallel calculations that do a lot of matrix inversion or multiplication only to suddenly realize that it’s. just. slow. How is this possible?
Well, the thing is that you’re actually using 4*4 = 16 threads! Four times the number of threads your CPU would be capable of simultaneously running. As a result, your CPU ends up switching back and forth between these threads, hugely slowing down both your matrix operations and your parallel execution. It might slow down to the point where a sequential execution would finish a lot quicker.
To avoid this problem, you should use MRO’s command (setMKLthreads) to limit the number of used threads to 1 on each of the child processes. I’ll give you a complete example that achieves this using the doParallel package. Make sure to run the following snippet before the actual foreach calculation!
Nice. Try running your calculations again and you might notice that they just became a lot quicker, as they did for me. Oh, by the way, clusterEvalQ simply executes the same code on all your child processes (documentation here, I’ll talk about it later on). It can be used to do some setup work on your processes like loading packages and adjusting settings.
The (not so) fun part in missing this point is that you will likely not even notice what’s happening in the background. You won’t get any errors, but at some point your calculations might seem to greatly slow down. You will probably be fine as long as your processes don’t hit some heavy multi-threaded part at the same time, but once they do, you’ll find your progress bar (if you are using one) stop and just stare at the monitor, thinking “meh, it can just not go any faster, it seems”. My simulations, too, seemed to stall and I thought it was because I was using matrices of size 500×500 — they ran fine with 50×50 matrices! Once I told MRO to stop using multiple threads on each of its processes, my parallel execution blazed through even with the big (500×500) matrices! I was running these on a 32-core server, so I actually spawned 32*32 threads, yummy.
Even if you don’t use MRO, you might still run into this issue. See, the data.table package for example is also multithreaded, but while the documentation says it can detect if it’s running on a forked process and switch back to single-threaded operation, I was a bit worried about the explicit mention of forking. Forking is the other method when creating clusters that clones processes (pass type = "FORK" to makeCluster on Linux) but it does not work on Windows, so I used “SOCK” clusters — I developed my code on Windows and ran it on a Linux server, the best thing to do. To extend our previous code, we can execute this instead:
This should be enough to get you started with multithreading and multiprocessing. Remember, you’ll need to make sure that every package you use on your child processes is running in single-threaded mode. Packages usually tell you if they use multiple threads like data.table does.
By the way, if you’re wondering about the performance of MRO on a single thread, it does not become useless when switched into single-threaded operation. It still uses Intel’s MKL, which can be significantly faster than base R’s BLAS/LAPACK libraries even on that one thread (see the benchmarks I referred to previously, here).
TLDR. You really don’t want to use these two types of parallelism together, watch out for multithreaded parts in a multiprocessing environment!
There’s another thing to mention regarding performance. See, I’ve had this issue where I practically froze a 32-core server. Turns out the error wasn’t simply me spawning too many threads, but that the child processes didn’t quit when I forcefully told the main one to stop (using everyone’s favourite key combination, CTRL + C). Since I was working on a Windows machine, I naively thought that things should work roughly the same way on the Linux machine with my child processes: shutting down the main process from the console with CTRL + C should stop the child processes as well — just not on the Linux machine, it seems.
These child processes were in the middle of an overparallelized situation: each thread was fighting for the CPU (1024 threads for 32 cores!), so their estimated finishing time was basically never. Now, since I didn’t realize this, I actually restarted the computation a few times (after continuously updating the code), so there were more child processes and even more threads running than I possibly imagined — this is why you should run at least top, if not htop to check your server’s performance from the command line.
To make sure your processes actually stop, there is something you can do before you start your calculations. R provides a function called on.exit (documentation here), a handy little utility that executes code when the function it is in exits, even when CTRL + C was issued. We can use it to try to shut our cluster down regardless of whether we’ve successfully finished executing the current function or not:
But that’s two stop commands, isn’t it? Well, yes, but I really wanted to make sure my cluster was properly shut down — it did in the end. I also wrapped the codes in a try block, just in case one of them errors out (eg. because the cluster was already stopped). Whether both of them are necessary, I could not verify, but while it looks ugly to issue to multiple stop commands, I have never had issues with it. Like I said, on a Windows machine you probably won’t have, either (if your main process quits, the children are also shut down), but it’s always a good idea to explicitly stop your clusters once you’ve finished working with them and handle the abnormal shutdowns as well.
While it might be equally easy to use clusterCall instead of clusterEvalQ that we’ve used earlier — to the point where they seem exchangeable — , this function has an interesting side effect that you might just not be aware of. To do my simulations, I generated a huge list of expressions (it took a few hundreds of MiB of memory) that I then fed to the parallel foreach loop — the cluster’s processes then evaluated these expressions, giving me the final results. The list of expressions was generated before I set up my cluster so that I didn’t start the cluster until I was sure I had no errors in the tasks. This generation process finished without errors (as seen by the large increase in memory usage of the base R process). Then to set up my cluster I switched from using a foreach parameter named .packages to using clusterCall() to both load the packages I needed and set the number of threads for MKL and data.table on the child processes.
This was all fine until I suddenly noticed that the child processes’ memory usage started rising from 40–50 MiB to about 6–900 MiB each (to about the same size on each child process, just to be precise), and it happened one after the other: the processes spawned, process #1 went up from 40 MiB to ~670 MiB under a few seconds, then process #2 did the same and so on. It cost me quite a bit of debugging to figure out what actually is going on — I went as far as to not execute anything on the child processes, just an empty clusterCall was there and the memory still raised to this absurdly high amount. If you’re wondering, this was the line that caused the issue:
clusterCall(cl = parallelCluster, function () { }) # do nothing
Weird, huh? How is this possible? Also, what’s even better is that this memory usage problem did not happen when I executed the very same lines of code from the console: memory usage stayed around 40 MiB. I could load packages on the cluster without raising the memory usage by almost a gigabyte when I issued the command from the console instead of inside a function.
You’re probably suspecting the issue here, given the context of this section, but I’ve never seen any indication of clusterCall copying all of the current function’s local variables onto the child processes first. It’s probably a convenience feature to be able to evaluate functions on them without having to explicitly copy variables first (with the help of clusterExport), but it was still quite the headache.
So, make sure you do not place too many (large) variables into the same function you’re calling clusterCall from! Instead you can try to create a separate function that calls clusterCall to set your cluster up, or make sure you call this function right at the start of your function, but after makeCluster was issued (I believe all local variables would copy over, so if you pass a huge data frame to your function as an argument, it might as well get copied to the child processes). There is a much simpler way, though, as you might not want to use clusterCall for doing setup work (loading packages for example).
An alternative that I recommend is using clusterEvalQ instead, especially for doing initialization. The latter does not accept a function as its second argument, but an expression instead (syntactically similar), and it does not result in the unwanted copying of local variables onto the cluster. If you want to, you can still manually (explicitly) move variables to your child processes using clusterExport. So use this instead:
clusterCall still has its uses, though! It’s especially handy if you want to use multiple local variables from your current function and don’t want to list all of them in a clusterExport call explicitly.
Fun fact. As per the documentation (and the source code), clusterEvalQ calls clusterCall behind the scenes. This seems strange at first, given the fact that clusterEvalQ does not result in copying variables, but I have an answer for that, if you’re still reading! Since clusterCall is wrapped inside clusterEvalQ, only clusterEvalQ‘s local variables would get copied over (of which there are none, except the expression itself), so no extra variables get copied over to the cluster. Here’s the source code of clusterEvalQ (the code is taken from the source of the package , I just reformatted it — click ClusterApply.R on the sidebar to view the relevant file):
clusterEvalQ <- function(cl = NULL, expr) {  clusterCall(cl, eval, substitute(expr), env=.GlobalEnv)}
With that, we’ve reached the end of the topic. While this article hasn’t touched on all parallelization topics (like OpenMP with Rcpp for multithreading), I hope it was informative enough to help you avoid some sneaky pitfalls. I also hope that the 20 minute read time didn’t scare you away — I’m quite susceptible to over-explaining myself. Next time I’ll try to cover some problems that do bring along error messages (so they are easier to detect), but those messages might not be very helpful and can be cumbersome to debug and find the root cause of them — like occasionally (ie. randomly) getting an Error reading from connection error.
If you have any suggestions or corrections, feel free to add them in the comments!
Here are the topics I’ve covered or touched in this article (for reference)
How to do multiprocessing in R with foreach and why you would want to use it
Multiprocessing has a cost (called overhead) that might outweigh the benefit of parallelism on smaller tasks
Combining multiprocessing with multithreaded work can unexpectedly cause huge slowdowns
Microsoft R Open can be a great alternative to speed up your calculations using multithreading, especially when not using multiprocessing. It can be a great alternative to the base R version
Pressing CTRL + C will not necessarily terminate your cluster’s processes, only your main process: make sure you handle unexpected exits as well
clusterCall can fill the memory of your child processes with data you wouldn’t want to load on them, and you might not be aware of this. Use clusterEvalQ instead if you don’t need your variables to become magically available on your processes.