Repository: scverse/scanpy

Issue:
regress_out n_jobs question 
<!--
⚠ If you need help using Scanpy, please ask in https://scanpy.discourse.group/ instead ⚠
If you want to know about design decisions and the like, please ask below:
-->
...

Hi,

I notice that even if I set n_jobs=1 in sc.pp.regress_out, all cpus are utilized. This also happens if I set n_jobs to other numbers. Basically there isn't a noticeable difference in cpu usage no matter what number of n_jobs I set. I'm using CentOS6.8 on a machine with 28 physical cores and hyper threading on (appears as 56 cores in the os).

Is this an intended behavior, or just my installation? I understand that some numpy functions are naturally multi-threaded because of the setup of BLAS libraries.

Thanks.


Discussion:
I have the same issue on Ubuntu 20.04.
AFAICT, I think the parallelization you're seeing will be due to the underlying calls in statsmodels. If you turn down the number of threads blas can use, do you see the same utilization?
> AFAICT, I think the parallelization you're seeing will be due to the underlying calls in statsmodels. If you turn down the number of threads blas can use, do you see the same utilization?

FYI more n_jobs seems to be slower for regress_out if I don't disable the BLAS multi threading:
```
sc.pp.regress_out(adata, ['percent_mito'], n_jobs=1)
```
```
regressing out ['percent_mito']
    sparse input is densified and may lead to high memory use
    finished (0:04:05)
```
```
sc.pp.regress_out(adata, ['percent_mito'], n_jobs=24)
```
```
regressing out ['percent_mito']
    sparse input is densified and may lead to high memory use
    finished (0:07:41)
```

I'm using scanpy 1.5.2.dev104+g8611dba1

Indeed after disabling BLAS multi threading sc.pp.regress_out will only use one core if setting n_jobs = 1. But it has to be disable by exporting these environmental variables before starting python ..., but I guess it is not a good thing because other scanpy functions may be affected?
```
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1
```
More interesting is that regress_out becomes lightning fast when n_jobs = 24 and with BLAS multi threading disabled:
```
sc.pp.regress_out(adata, ['percent_mito'], n_jobs=24)
```
```
regressing out ['percent_mito']
    sparse input is densified and may lead to high memory use
    finished (0:00:23)
```
> More interesting is that regress_out becomes lightning fast when n_jobs = 24 and with BLAS multi threading disabled:

Thats not too surprising to me. This must be significantly over scheduling your machine.

--------------------------------

This got me doing a little more digging into this, and it look's like there's actually a solution now! We can use [`threadpoolctl`](https://github.com/joblib/threadpoolctl) to dynamically manage the number of threads BLAS uses via the `threadpool_limits` context manager. 

I'm definitely interested in using this inside scanpy to manage the number of threads used here. Not quite sure yet what the right behaviour/ api is. Some options:

* Should all calls use 1 blas thread by default, so parallelization will only happen through the number of jobs?
* Do we only limit the number of threads if `n_jobs` is specified? 
* Do we try and be fancy, with something like `n_threads = n_cpus // n_jobs`?

*Minor update*

[If we use `joblib` (with the `loky` backend) instead of `multiprocessing`, the fancy solution will be used by default.](https://joblib.readthedocs.io/en/latest/parallel.html#avoiding-over-subscription-of-cpu-resources)

I think this is what the code would look like inside of `regress_out`:

```python
    from joblib import Parallel, delayed
    res = Parallel(n_jobs=n_jobs)(delayed(_regress_out_chunk)(task) for task in tasks)
```

Resolving patch:
diff --git a/scanpy/preprocessing/_simple.py b/scanpy/preprocessing/_simple.py
--- a/scanpy/preprocessing/_simple.py
+++ b/scanpy/preprocessing/_simple.py
@@ -662,15 +662,10 @@ def regress_out(
             regres = regressors
         tasks.append(tuple((data_chunk, regres, variable_is_categorical)))
 
-    if n_jobs > 1 and n_chunks > 1:
-        import multiprocessing
+    from joblib import Parallel, delayed
 
-        pool = multiprocessing.Pool(n_jobs)
-        res = pool.map_async(_regress_out_chunk, tasks).get(9999999)
-        pool.close()
-
-    else:
-        res = list(map(_regress_out_chunk, tasks))
+    # TODO: figure out how to test that this doesn't oversubscribe resources
+    res = Parallel(n_jobs=n_jobs)(delayed(_regress_out_chunk)(task) for task in tasks)
 
     # res is a list of vectors (each corresponding to a regressed gene column).
     # The transpose is needed to get the matrix in the shape needed


Test patch: