Repository: horovod/horovod

Issue:
Unexpected performance decrease after hvd.init on CPU nodes
**Environment:**
1. Framework: TensorFlow
2. Framework version: 2.3.1
3. Horovod version: 0.21.0
4. MPI version: OpenMPI-4.0.3
5. CUDA version: 11.0.2
6. NCCL version: 2.7.8
7. Python version: 3.8.2
8. Spark / PySpark version: N/A
9. Ray version: N/A
10. OS and version: Red Hat Enterprise Linux Server release 7.9 (Maipo) 
11. GCC version: 9.3.0
12. CMake version: 3.16.4

**Bug report:**

I see an unexpected performance decrease when running Horovod (using a single MPI rank) compared to just native TensorFlow. Essentially, I have a slightly altered version of the `tensorflow2_synthetic_benchmark.py` (see [here](https://github.com/casparvl/software-layer/blob/tensorflow/tests/reframe/eessi-checks/applications/src/tensorflow2_synthetic_benchmark.py)) to which I've added an extra argument `--use-horovod`. With that argument, it runs the exact same example, but with native tensorflow (i.e. it skips the hvd.init(), skips the wrapping in the Horovod distributed gradient tape, etc).

Now, I performed 4 different runs all using a single rank. Run 1 & 2 were performed on a GPU node, run 3 & 4 on a CPU node.

1. `srun -n 1 python tensorflow2_synthetic_benchmark.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5`
2. `srun -n 1 python tensorflow2_synthetic_benchmark.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --use-horovod`
3. `srun -n 1 python tensorflow2_synthetic_benchmark.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda`
4. `srun -n 1 python tensorflow2_synthetic_benchmark.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --use-horovod`

And the the respective performance numbers are:

1.  51.7 img/s
2. 50.0 img/s
3. 28.7 img/s
4. 6.9 img/s

Clearly, I'm wondering about that last result. 

I've checked a horovod timeline, there's virtually nothing there (I see the broadcast operations, but as expected a broadcast with one worker only takes a few microseconds).

Finally, I nailed it down to the hvd.init(): if I repeat run 3, but only add the lines
```
import horovod.tensorflow as hvd
hvd.init()
```
That's when my performance drops (note: only importing the horovod.tensorflow module doesn't change the performance, it's really the `hvd.init()` that causes the performance drop).

Would you have any idea what is causing the performance difference between run 3 and 4, as defined above? I saw that `horovodrun` has an option to disable mpi-thread support and mentions this sometimes 'can slow down other components'. Is there an equivalent way of disabling mpi-thread support when launching with other mpi wrappers such as `mpirun` or `srun`? (I thought I remember an environment variable that could be set, but can't seem to find it... )


Discussion:
Can you try with gloo? I have observed some similar performance difference when using mpi and gloo for some heavy CPU-threading applications. 
I've tried with gloo, but it didn't make a difference. 

> I have observed some similar performance difference when using mpi and gloo for some heavy CPU-threading applications.

Indeed, this is a training that completely maxes out the 24 cores of my CPU.

Regardless, I do think I'm one step closer to what's going on: the performance hit seems to have something to do with thread scheduling. My assumption is that the `hvd.init()` spins up one (or more) extra threads, which put _some_ load on the processor (despite the fact that I make no further horovod calls).

I tested a few further scenarios (on a 24 core node, no hyperthreading). 

1. Pure TensorFlow run (no hvd.init()): `OMP_NUM_THREADS=24 srun -n 1 python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda` => 27.8 img/s
2. Same as (1), but added an hvd.init(): `OMP_NUM_THREADS=24 srun -n 1 python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --use-horovod` => 9.2 img/s
3. Same as (2), but added thread affinity setting: `OMP_NUM_THREADS=24 KMP_AFFINITY=verbose,compact srun -n 1 python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --use-horovod` => 23.2 img/s
4. Same as (2), but reduced number of threads to 23: `OMP_NUM_THREADS=23 srun -n 1 python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --use-horovod` => 30.5 img/s
5. Same as (4), but with thread affinity setting: `OMP_NUM_THREADS=23 KMP_AFFINITY=verbose,compact srun -n 1 python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --use-horovod` => 31.2 img/s
6. Same as (1), but 23 threads: `OMP_NUM_THREADS=23 KMP_AFFINITY=verbose,compact srun -n 1 python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda` => 30.8 img/s

(for `KMP_AFFINITY` settings and their effect on thread placement, see e.g. https://software.intel.com/content/www/us/en/develop/articles/process-and-thread-affinity-for-intel-xeon-phi-processors-x200.html).

What I conclude from these experiements is that going from 1 => 2, the hvd.init() effectively creates additional load on the CPU, and essentially we're oversubscribing the cores. This causes the OS to migrate threads all the time, which in turn causes the performance penalty.

Scenario 3 prevents at least the TensorFlow-related threads to be migrated, thus partially recovering the performance (it's still 15-20% lower than the scenario 1, probably because the horovod thread still has to be swapped in-and-out on one of the cores, i.e. we're essentially still oversubscribing).

Scenario 4,5,6 show nearly the same performance. Thus, in these scenarios, even if the hvd.init() causes one extra thread to put load on the CPU cores, the OS is clearly capable of scheduling the threads once, and (mostly) leave them alone (i.e. no further thread swapping) since there is no oversubscription anymore. (note that the fact that performance is a bit higher than for scenario (1) is probably simply that TensorFlow multithreading simply doesn't scale to this number of threads, but that's not relevant for this issue)

**Summary**
A call to `hvd.init()` seems to result in extra load on the CPU cores, causing threads to be migrated. The performance penalty can be mitigated either by thread pinning (to some degree) using e.g. `KMP_AFFINITY`, or by reducing the number of threads to the core count minus 1.

**Open questions**
What this doesn't explain is _why_ an `hvd.init()` causes extra CPU load - even if no further horovod calls are made Also, I've repeated the above tests with older versions (Horovod 0.19.4 and TensorFlow 2.2.0) and did NOT see the same performance impact for that setup. Has `hvd.init()` changed between 0.21.0 and 0.19.4 in a way that could explain this extra load?
Thanks for checking the version difference, that's interesting.
@tgaddair Are you aware any major changes from `0.19.4` to `0.21.0`?
Just double checked, since you're so particularly interested in that: running Scenario 2 from the above list, but with Horovod 0.19.4 and TensorFlow 2.2.0 I get 29 img/s. That's as expected, since it's pretty much the same as Scenario 1 (without any Horovod).
That's very interesting. I am not aware of any changes that would cause this, though there were some changes to the way thread affinity is handled.  I believe it should only have an effect when set via an environment variable, however. @romerojosh do you know if any of the thread affinity changes in the last couple of releases could have an effect on performance without tweaking the environment?
There shouldn't be any affinity changes, unless the `HOROVOD_THREAD_AFFINITY` env var is set. In fact, @casparvl , I think for CPU based workloads, you should look into option to pin the Horovod background thread to a core to avoid these performance issues. 
One change that might've increased the CPU load between these versions is the reduction in the default cycle time to 1ms from 5ms. @casparvl, if you run your experiment with the env var `HOROVOD_CYCLE_TIME=5` set, does the performance issue go away?
Thanks for the suggestions, I really appreciate your willingness to figure it out!

@romerojosh : I've ran two new experiments:

1. ```OMP_NUM_THREADS=24 HOROVOD_CYCLE_TIME=5 srun python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --use-horovod``` => 8.4 img/s
2. ```OMP_NUM_THREADS=24 HOROVOD_THREAD_AFFINITY=0 srun python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --init-hvd``` => 7.1 img/s

So... setting the cycle time back to 5ms doesn't seem to fix it. Also, the specific horovod thread affinity doesn't really change anything (though I'm not entirely sure I use it in the right way here?).

I also realized something: in my tests with the older versions, it's not just the horovod (and TF) versions that were different, also the underlying stack was different (e.g. `OpenMPI` was used in both cases, but version `4.0.3` in one case, and `3.1.1` in the other case). So, I did another test, to make the comparison more fair: I used the exact same software stack up to and including TensorFlow-2.3.1, and only installed various versions of Horovod on top of it. Running the same `OMP_NUM_THREADS=24 srun python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter 5 --num-warmup-batches 5 --no-cuda --use-horovod` I did before, I get

1. Horovod 0.19.4: 30.2 img/s
2. Horovod 0.19.5: 30.0 img/s
3. Horovod 0.20.0: 30.0 img/s
4. Horovod 0.20.1: 7.6 img/s
5. Horovod 0.20.2: 7.1 img/s
6. Horovod 0.20.3: 7.8 img/s
7. Horovod 0.21.0: 7.3 img/s

So, the change really seems to have been going from version `0.20.0` to `0.20.1`. Since this is a minor version change, maybe this makes it easier to track down what has changed...?
Thanks for running these tests @casparvl! The only PR from 0.20.0 to 0.20.1 that looks like it may have increased the cost of the Horovod cycle is #2215, as it adds a few additional locks and also the use of a `std::atomic_bool` in the main loop, though these performance numbers seem to suggest something much more expensive. If you are able, could you try running a copy of Horovod 0.20.1 with this PR reverted to see if it resolves the issue?

Out of curiousity, if you run `numactl --show` via your `srun` launch: `srun -n 1 numactl --show`, what is reported? Some MPI distributions, like OpenMPI, will automatically bind launches of 2 processes or less to single cores, rather than full sockets. 
> If you are able, could you try running a copy of Horovod 0.20.1 with this PR reverted to see if it resolves the issue?

Hm, I might need some pointers on how to do this. Previously I just created python virtual environments and ran a `pip install horovod==<someversion>` there. I guess in order to what you ask, I need to clone the repo, checkout the 0.20.1 branch, revert the PR (I'm sure the web can tell me how to do this part :)) and build a horovod wheel from the sources (this is where I might need some pointers :))?

> Out of curiousity, if you run numactl --show via your srun launch: srun -n 1 numactl --show, what is reported? Some MPI distributions, like OpenMPI, will automatically bind launches of 2 processes or less to single cores, rather than full sockets.

```
srun -n 1 numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpubind: 0 1
nodebind: 0 1
membind: 0 1
```
I've also checked with `htop` during a run, and all cores are being used in all of my prior experiments. So it's not that all threads end up on a single core.
Actually, it might be easier for you to just install at a commit right before #2215 landed. Try running this command to install: `pip install git+https://github.com/horovod/horovod.git@440629ec2bdab02aeca2f16bcad9ee6b6842b283`

Thanks for sharing the `numactl` output. I figured based on your earlier comments that this probably wasn't an issue, but I've seen bad bindings cause headaches for some in the past. 
With `pip install git+https://github.com/horovod/horovod.git@440629ec2bdab02aeca2f16bcad9ee6b6842b283` I got 30.7 img/s. Then, I tried `pip install git+https://github.com/horovod/horovod.git@a7de4ae47e8697e1b917b79732b63c2cfb4c2507` (this commit, to my understanding, is the first that includes [#2215](https://github.com/horovod/horovod/pull/2215) ). I got 6.7 img/s.

So your intuition that it could have been this PR seems to be correct! There's been quite a few changes in this PR, so it may be difficult to nail down further. But the fact that it happens _even_ when I'm only importing `hvd.tensorflow` might limit the amount of code that could actually cause this? I mean, unless I'm badly mistaken somehow, I'm not actually writing a timeline, so I expect much of the code in this PR isn't even executed...
The only changes to the `RunLoopOnce` function in `operations.cc` (this is what the Horovod background thread runs repeatedly once you call `hvd.init()`)  from that PR are shown here:
https://github.com/horovod/horovod/pull/2215/files#diff-f95a5a532ec19779ad45f2eafb72c67580231e0720cc7d2d2f1aee9fd09568baL585-L586

This PR made `state.mark_cycles_in_timeline` a `std::atomic_bool` so the line setting this value will be more expensive than setting a regular `bool`; with that said, I'm not sure why it was made a `std::atomic_bool` since only the main Horovod background thread ever modifies it as far as I can tell.

Beyond this, the `SynchronizeTimelineEnabled()` and `MarkCyclesInTimelinePending()` each lock a `std::recursive_mutex`. Not sure if locking `std::recursive_mutex` is significantly more expensive than locking a plain `std::mutex` but I would suspect that is a possibility. Not sure why a `std::recursive_mutex` is used here instead of `std::mutex` either.

@Vikas-kum Could you comment about the use of `std::atomic_bool` and `std::recursive_mutex` in #2215 mentioned above? Would reverting to `bool` and using a regular `std::mutex` work?
> The only changes to the `RunLoopOnce` function in `operations.cc` (this is what the Horovod background thread runs repeatedly once you call `hvd.init()`) from that PR are shown here:
> https://github.com/horovod/horovod/pull/2215/files#diff-f95a5a532ec19779ad45f2eafb72c67580231e0720cc7d2d2f1aee9fd09568baL585-L586
> 
> This PR made `state.mark_cycles_in_timeline` a `std::atomic_bool` so the line setting this value will be more expensive than setting a regular `bool`; with that said, I'm not sure why it was made a `std::atomic_bool` since only the main Horovod background thread ever modifies it as far as I can tell.
> 
> Beyond this, the `SynchronizeTimelineEnabled()` and `MarkCyclesInTimelinePending()` each lock a `std::recursive_mutex`. Not sure if locking `std::recursive_mutex` is significantly more expensive than locking a plain `std::mutex` but I would suspect that is a possibility. Not sure why a `std::recursive_mutex` is used here instead of `std::mutex` either.
> 
> @Vikas-kum Could you comment about the use of `std::atomic_bool` and `std::recursive_mutex` in #2215 mentioned above? Would reverting to `bool` and using a regular `std::mutex` work?

This seems interesting. It seems like regression is only during  hvd.init() ? I was wondering how much of regression can atomic_bool and recursive_mutex can cause and if it should be even visible. Going from 30 image to 7 image/sec doesn't sound realistic for this reason, but may be I am underestimating. But, irrespective of that I think that bool and std::mutex should be good. May be we can check the state of regression after making that change.
Just want to check if I understand the plan correctly: @Vikas-kum do you plan to make a feature branch where you make the replacements with `bool` and `std::mutex`? And then a) you run the regression tests to see if that indeed produces sane results and b) I run my example with the feature branch to see if it solves the performance issue?
Hi @casparvl, can you test this branch/commit and see if it resolves your performance issues (https://github.com/romerojosh/horovod/commit/320f69e89f13b84ba2c1c9b862edee45bb8acb5a)?
Ok, in a clean `virtualenv` I ran a `pip install git+https://github.com/romerojosh/horovod.git@320f69e89f13b84ba2c1c9b862edee45bb8acb5a`
and ran my previous test:
```
OMP_NUM_THREADS=24 srun python tensorflow2_synthetic_benchmark_native_TF.py --model ResNet50 --batch-size 32 --num-iters 5 --num-batches-per-iter5 --num-warmup-batches 5 --no-cuda --use-horovod
```
And... I got a disappointing 7.8 img/s :( 

I don't get it... seeing as how I nailed it down to the one PR, and you guys nailed down the part of the code within that PR that would actually be executed with an `hvd.init()`, I really expected this to solve it...

I see two potential causes for it still going wrong: either it was another part of the PR after all, or, since that PR, other changes have been made that cause the performance decrease to persist even after revering this (part) of the PR... (e.g. maybe the `std::recursive_mutex` was introduced elsewhere in the code as well in the meantime?)

How difficult would it be for me to take the current version of the code and revert the full PR? I'm guessing I should 
- fork
- create my own feature branch from the latest release
- run a `git-revert` on that particular commit
- push that to my branch
- run a `pip install` in the same way I installed your particular commit

That should undo all changes of that PR, right? I'm not sure how easy this is (I can imagine a git-revert may not be trivial if other changes where made to the same code since then), but I could try. If that solves the issue, it must have been another part of the PR that was to blame. If it doesn't solve the issue, maybe some other part of the code was indeed also changed since this original PR - with the same effect on the performance.
Thanks for testing @casparvl! I've taken a closer look and I see now what the issue is. It isn't related to any new of the new calls added to the cycle loop as I previously thought. 

 #2215 made a modification that initializes the timeline unconditionally here: https://github.com/horovod/horovod/pull/2215/files#diff-f95a5a532ec19779ad45f2eafb72c67580231e0720cc7d2d2f1aee9fd09568baL417-L421
What I did not realize is that this initialization spins up an additional thread for timeline writing here, which I believe is causing the performance degradation: https://github.com/horovod/horovod/blob/master/horovod/common/timeline.cc#L149-L150
Prior to this PR, the timeline writing (and related thread) would only be spawned if the timeline was enabled by the user. 

@Vikas-kum do you have a suggestion for what we can do here? Is it possible to only initialize the timeline on user request, rather than initializing it unconditionally and spawning this additional thread for all Horovod workloads, even those not using the timeline feature?
Hi all, I'm running into this issue as well. On a machine that typically runs ~150 images per second when training a PyTorch or TensorFlow model, I'm getting only 2 images per second with Horovod. Is there any update on this problem yet?

Resolving patch:
diff --git a/horovod/common/basics.py b/horovod/common/basics.py
--- a/horovod/common/basics.py
+++ b/horovod/common/basics.py
@@ -85,8 +85,10 @@ def start_timeline(self, file_path, mark_cycles=False):
         result = self.MPI_LIB_CTYPES.horovod_start_timeline(
             ctypes.c_char_p(file_path.encode('utf-8')),
             ctypes.c_bool(mark_cycles))
-        if not result:
+        if result == -1:
             raise ValueError('Horovod has not been initialized; use hvd.init().')
+        elif result == -2:
+            raise ValueError('Set HOROVOD_TIMELINE=DYNAMIC to enable dynamic timeline usage.');
 
     def stop_timeline(self):
         """Stops the active timeline recording and closes the file.
@@ -94,7 +96,7 @@ def stop_timeline(self):
         Raises a `ValueError` if Horovod is not initialized.
         """
         result = self.MPI_LIB_CTYPES.horovod_stop_timeline()
-        if not result:
+        if result == -1:
             raise ValueError('Horovod has not been initialized; use hvd.init().')
 
     def size(self):


Test patch:
diff --git a/test/parallel/test_torch.py b/test/parallel/test_torch.py
--- a/test/parallel/test_torch.py
+++ b/test/parallel/test_torch.py
@@ -47,6 +47,8 @@
                            torch.IntTensor, torch.LongTensor, torch.FloatTensor,
                            torch.DoubleTensor])
 
+# Set environment variable for dynamic timeline API test
+os.environ["HOROVOD_TIMELINE"] = "DYNAMIC"
 
 class TorchTests(unittest.TestCase):
     """
@@ -2296,7 +2298,7 @@ def test_horovod_sync_batch_norm(self):
             assert torch.allclose(hvd.allreduce(sync_bn.bias.grad, name='sync_bn.bias.grad'), bn.bias.grad, 1e-6)
             assert torch.allclose(hvd.allreduce(ts1.grad, name='ts1.grad'), ts2.grad, 1e-6)
 
-    @pytest.mark.skip(reason='https://github.com/horovod/horovod/issues/2496')
+    @pytest.mark.skipif(platform.system() == 'Darwin', reason='https://github.com/horovod/horovod/issues/2496')
     def test_timeline_api(self):
         hvd.init()