Repository: Parsl/parsl

Issue:
Improve thread affinity with OpenMP
**Is your feature request related to a problem? Please describe.**
I find that I need to add "OMP_NUM_THREADS" before calling an OpenMP executable when running multiple OpenMP applications on a single node to avoid them contending with each other. 

**Describe the solution you'd like**
It would be great for HTEx to set the environment variables that control OpenMP's number of threads per process and affinity automatically, rather than me setting them in my job.

**Describe alternatives you've considered**
- Placing more than one HTEx per node with an MPI launcher and using the MPI launcher's affinity mechanisms. 

**Additional context**
A primer on thread affinity and OpenMP: https://hpc-tutorials.llnl.gov/openmp/ProcessThreadAffinity.pdf



Discussion:
This is a problem i've seen more broadly with other stuff, where multiple layers all decide they are the decision maker for who should get cores or not.

I'm torn here, because there is some scope creep which has caused support load in other situations where some guy has decided that my code shouldn't work the same inside a program which launches my executable vs when I launch my executable.

A support example there would be "when I run my code via htex, it only uses one core; when I submit it with sbatch, it uses all the cores. parsl is broken."

At that point, we would add an option to set OMP_NUM_THREADS. And then we're back to "here's a config option which lets you... set an environment variable you could set yourself, but in a way that your sysadmins understand much less well", which is something I've pushed against in parsl.

> A support example there would be "when I run my code via htex, it only uses one core; when I submit it with sbatch, it uses all the cores. parsl is broken."

That's a great point, Ben. I can see that the more Parsl messes with thread affinity, the more opportunity of unseen behavior. 

I'd prefer to see the OMP thread affinity in the `worker_init` for this reason, but need something that is different for each worker. (BTW, should we make an Issue about documenting "worker init is for the manager but not the workers")

I'm testing out a "fix" or, at least, an proposal for how we could accomplish this, and will make sure to document what OMP environment variables Parsl is touching.

Resolving patch:
diff --git a/parsl/executors/high_throughput/process_worker_pool.py b/parsl/executors/high_throughput/process_worker_pool.py
--- a/parsl/executors/high_throughput/process_worker_pool.py
+++ b/parsl/executors/high_throughput/process_worker_pool.py
@@ -560,6 +560,13 @@ def worker(worker_id, pool_id, pool_size, task_queue, result_queue, worker_queue
         else:
             raise ValueError("Affinity strategy {} is not supported".format(cpu_affinity))
 
+        # Set the affinity for OpenMP
+        #  See: https://hpc-tutorials.llnl.gov/openmp/ProcessThreadAffinity.pdf
+        proc_list = ",".join(map(str, my_cores))
+        os.environ["OMP_NUM_THREADS"] = str(len(my_cores))
+        os.environ["GOMP_CPU_AFFINITY"] = proc_list  # Compatible with GCC OpenMP
+        os.environ["KMP_AFFINITY"] = f"explicit,proclist=[{proc_list}]"  # For Intel OpenMP
+
         # Set the affinity for this worker
         os.sched_setaffinity(0, my_cores)
         logger.info("Set worker CPU affinity to {}".format(my_cores))


Test patch: