Project: datamanagement
Issue DM-4714: Bad OpenBlas setting in miniconda/numpy causes very poor performance for running multiple processes
I have been running many processes of processCcdDecam.py on my new linux machine in Tucson (bambam). To my surprise, running 40 processes at once gets very poor performance (~70 sec per process) compared to running a single process (~16 sec). I expected some performance hit because of larger overheads but not a factor of 4! I ran it both on a spinning HDD and PCIe SSD but they both had the same problem. I also tried running it on multiple visits versus multiple chips for a single visit (all accessing the same MEF FITS file) but this made no difference. I tested it with various numbers of processes and found that the time per processes increases linearly with the number of processes running. [~jmatt] has been helping me track this down. We used some performance tools (htop, iotop, and perf) to figure out what was going on. It was clear that the issue was not a RAM or I/O problem. By watching htop while the 40 processes were running it became clear that once some of the processes hit "deblending" everything slowed down considerably and all cores were maxed out and showing lots of kernel traffic. I also ran processCcdDecam.py with deblending turned off and the performance was much more reasonable (~24 sec. per process). After more digging (with perftop), we found that there was a lot of swapping going on during the deblending step by "openblas". This is a package that numpy uses for speeding up certain computationally intensive tasks using multithreading (e.g. linear algebra). By default each openblas instance takes advantage of ALL cores on a machine. So all 40 processes were trying to use all available cores and most of the time was spent swapping between all of these threads. OpenBlas can be configured to use a more reasonable number of cores/threads, but the version that LSSTSW uses is installed by miniconda via a dependency of numpy and, as far as we could tell, it's not possible to configured NUM_THREADS for OpenBlas with miniconda. We ended up compiling our own version of OpenBlas with NUM_THREADS = 6 (the maximum threads that OpenBlas uses) and the performance was great, 24 sec. I'm not sure what the solution is for this but we probably don't want to go with miniconda for the default LSSTSW installation (uses currently done by bin/deploy). JMatt might have comments to add.
Story points: 4