internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: horovod/horovod

Issue:
Please make "cmake" name configurable
**Is your feature request related to a problem? Please describe.**
This is not a huge problem because I can always put a script named "cmake" in my PATH to do whatever I want. But that is an ugly hack.

**Describe the solution you'd like**
Right now, when I do "envHOROVOD_WITH_GLOO=1 pip install horovod", the installer looks for "cmake" in its path. But on my system, the cmake binary is named "cmake3".

**Describe alternatives you've considered**
As mentioned, I can hack my PATH to put something named "cmake" in it. But, you know, yuck.

**Additional context**
I might be able to work up a pull request if you agree this would be a handy feature. Although I am not really a Python guy.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -533,7 +533,8 @@ def get_common_options(build_ext):
         # determining if system has cmake installed
         compile_with_gloo = os.environ.get('HOROVOD_WITH_GLOO')
         try:
-            subprocess.check_output(['cmake', '--version'])
+            cmake_bin = get_cmake_bin()
+            subprocess.check_output([cmake_bin, '--version'])
             have_cmake = True
         except Exception:
             if compile_with_gloo:
@@ -1304,8 +1305,12 @@ def build_torch_extension_v2(build_ext, global_options, torch_version):
         customize_compiler(build_ext.compiler)
 
 
+def get_cmake_bin():
+    return os.environ.get('HOROVOD_CMAKE', 'cmake')
+
+
 def build_cmake(build_ext, ext, prefix, additional_flags, options, plugin_ext=None):
-    cmake_bin = 'cmake'
+    cmake_bin = get_cmake_bin()
 
     # All statically linked libraries will be placed here
     lib_output_dir = os.path.abspath(os.path.join(build_ext.build_temp, 'lib', prefix))


Test patch:
Repository: horovod/horovod

Issue:
Fix non-x86 build failures caused by #1485
Add extra checks for x86 compiler flags and x86 AVX headers/functions


Discussion:
@Tixxx @tgaddair 
Hey @nvcastet, any idea why this is causing tests to break?

```
Unsupported data type.
[[Node: HorovodAllreduce_Cast_0 = HorovodAllreduce[T=DT_HALF, reduce_op=2, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Cast)]]
```

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -107,7 +107,7 @@ def get_supported_instruction_set_flags(flags_to_check):
             'gcc -march=native -E -v - </dev/null 2>&1 | grep cc1',
             shell=True, universal_newlines=True).strip()
         flags = shlex.split(flags_output)
-        supported = [x for x in flags_to_check if x or x.replace('-m', '+') in flags]
+        supported = [x for x in flags_to_check if x in flags or x.replace('-m', '+') in flags]
     except subprocess.CalledProcessError:
         # Fallback to no advanced instruction set flags if were not able to get flag information.
         pass


Test patch:
Repository: horovod/horovod

Issue:
Importing ABC directly from collections was deprecated and will be removed in Python 3.10. Use collections.abc instead.
**Bug report:**

```
test/test_torch.py
1114:                if not isinstance(p, collections.Iterable):
horovod/torch/__init__.py
526:        if isinstance(x, collections.Iterable):
```



Discussion:
I will raise a PR for this.

Resolving patch:
diff --git a/horovod/torch/__init__.py b/horovod/torch/__init__.py
--- a/horovod/torch/__init__.py
+++ b/horovod/torch/__init__.py
@@ -34,6 +34,11 @@
     check_extension('horovod.torch', 'HOROVOD_WITH_PYTORCH',
                     __file__, 'mpi_lib', '_mpi_lib')
 
+try:
+    from collections.abc import Iterable
+except ImportError:
+    from collections import Iterable
+
 
 from horovod.torch.compression import Compression
 from horovod.torch.mpi_ops import allreduce, allreduce_async, allreduce_, allreduce_async_
@@ -523,7 +528,7 @@ def broadcast_optimizer_state(optimizer, root_rank):
 
     # Returns the full type structure of the possibly nested objects for recursive casting back
     def _get_types(x):
-        if isinstance(x, collections.Iterable):
+        if isinstance(x, Iterable):
             return type(x), [_get_types(xi) for xi in x]
         else:
             return type(x)


Test patch:
diff --git a/test/test_torch.py b/test/test_torch.py
--- a/test/test_torch.py
+++ b/test/test_torch.py
@@ -20,7 +20,6 @@
 
 from distutils.version import LooseVersion
 
-import collections
 import inspect
 import itertools
 import os
@@ -38,6 +37,11 @@
 
 from common import mpi_env_rank_and_size
 
+try:
+    from collections.abc import Iterable
+except ImportError:
+    from collections import Iterable
+
 _v2_api = LooseVersion(torch.__version__) >= LooseVersion('1.0.0')
 _fp16_supported = _v2_api
 
@@ -1111,7 +1115,7 @@ def create_model(opt_class):
             }
             for k, p in p0.items():
                 p_actual = optimizer.param_groups[0][k]
-                if not isinstance(p, collections.Iterable):
+                if not isinstance(p, Iterable):
                     p_actual = [p_actual]
                     p = [p]
                 for i in range(len(p)):
Repository: horovod/horovod

Issue:
Hard Torchvision Dependency
Hi, 
we're trying to build a docker image for lightning, which also installs horovod as an extra dependency.

We noted that in https://github.com/horovod/horovod/blob/master/setup.py#L1578 you have a hard torch vision dependency.

We don't want a hard dependency on torch vision, since guys doing NLP or other stuff don't need it. 

So I just wanted to ask, whether you need a hard dependency on it and why?

Best,
Justus



Discussion:
is is used only for examples, see https://github.com/horovod/horovod/search?q=torchvision&unscoped_q=torchvision so could it be moved to optional dependencies?
Hey @justusschock, @Borda is correct that it's only used in examples, so we can definitely remove it.  I can make a PR for this today.

As a workaround for now, I think you should be able to change the requirement `horovod[pytorch]` to just `horovod` in `requirements-extra.txt` for now, since `torch` is guaranteed to already be installed (from `requirements-txt`) before you install `requirements-extra.txt`.

Let me know if that works for now.  If not, we can aim for a patch release `0.19.2` early next week.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1575,7 +1575,7 @@ def build_extensions(self):
 tensorflow_cpu_require_list = ['tensorflow-cpu']
 tensorflow_gpu_require_list = ['tensorflow-gpu']
 keras_require_list = ['keras>=2.0.8,!=2.0.9,!=2.1.0,!=2.1.1']
-pytorch_require_list = ['torch','torchvision']
+pytorch_require_list = ['torch']
 mxnet_require_list = ['mxnet>=1.4.1']
 spark_require_list = ['h5py>=2.9', 'numpy', 'petastorm==0.8.2', 'pyarrow>=0.15.0', 'pyspark>=2.3.2']  # Petastorm 0.7.7 is not compatible with pyarrow<0.15.0
 # all frameworks' dependencies


Test patch:
Repository: horovod/horovod

Issue:
Dependency links in setup.py causing issues with private registry 
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet) all
2. Framework version: n/a
3. Horovod version: 0.20.1+
4. MPI version: n/a
5. CUDA version: n/a
6. NCCL version:
7. Python version:
8. Spark / PySpark version:
9. OS and version:
10. GCC version:
11. CMake version:

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?

**Bug report:**
Please describe erroneous behavior you're observing and steps to reproduce it.

In horovod 0.20.x the setup.py was updated to add dependency links. Although it says it is removed from pip, it still causes issues in enterprise settings when a private registry is used as pip for some reason tries to connect to download.pytorch.org. This causes horovod versions 0.20.x to be uninstallable as pip just waits for a timeout on these for a long long time. Can we please remove these from the setup.py? 


Discussion:
Fwiw Im on pip version 20.3
You are talking about these lines?

https://github.com/horovod/horovod/blob/4099c2b7f34f709f0db1c09f06b2594d7b4b9615/setup.py#L179-L185

I will remove them then. Will try to document that elsewhere.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -179,13 +179,6 @@ def get_package_version():
           'dev': dev_require_list,
           'test': test_require_list,
       },
-      # not used by pip since 19.0: https://github.com/pypa/pip/issues/4187#issuecomment-415067034
-      # here for completeness as pip install needs some of these via -f for versions with '+cpu'
-      # for examples, see Dockerfile.test.cpu and Dockerfile.test.gpu
-      dependency_links=[
-          'https://download.pytorch.org/whl/torch_stable.html',
-          'https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html'
-      ],
       python_requires='>=3.6',
       zip_safe=False,
       entry_points={


Test patch:
Repository: horovod/horovod

Issue:
Wrong default for horovod.tensorflow.keras.allreduce(average...
In Horovod 0.21.1 the default for `average` in `allreduce` is still `True` leading to 

> ValueError: The op parameter supersedes average. Please provide only one of them.

when using `op=...` (only).

This is only in in `horovod.tensorflow.keras`, not in `horovod.tensorflow`

BTW: In TF2, is there any benefit of using `horovod.tensorflow.keras` over `horovod.tensorflow` when not disabling eager execution (which in my tests is pretty much unfeasible)


Discussion:
Hi Flamefire,

Thanks for reporting the bug.  I am here to help with a fix.  May I know the steps to repro this bug?  And if you can provide your running environment, that will be helpful as well.
As written the function `horovod.tensorflow.keras.allreduce` has a default value of `allreduce=True` while it should be `None`. To "reproduce" it is enough to look at the source code and compare e.g.:
https://github.com/horovod/horovod/blob/2f154afad39a0ef6d4f337a0da7e422d43961325/horovod/tensorflow/keras/__init__.py#L122
https://github.com/horovod/horovod/blob/2f154afad39a0ef6d4f337a0da7e422d43961325/horovod/tensorflow/__init__.py#L54

If you check the condition when the quoted error is shown it should be clear why `None` is correct here and why it fails when you pass any `op` to `horovod.tensorflow.keras.allreduce` : https://github.com/horovod/horovod/blob/994913a2abc5812042a5c3540ced5b3316d23b72/horovod/common/util.py#L230-L232
Thanks for the comment.   It makes sense.  I suppose in the "`allreduce=True`" you mean  "`average=True`", right?  The fix should be very straight forward by change it to `average=None`.  We will fix this soon.

Resolving patch:
diff --git a/horovod/tensorflow/keras/__init__.py b/horovod/tensorflow/keras/__init__.py
--- a/horovod/tensorflow/keras/__init__.py
+++ b/horovod/tensorflow/keras/__init__.py
@@ -119,7 +119,7 @@ def broadcast_global_variables(root_rank):
     return _impl.broadcast_global_variables(K, root_rank)
 
 
-def allreduce(value, name=None, average=True,
+def allreduce(value, name=None, average=None,
               prescale_factor=1.0,
               postscale_factor=1.0,
               op=None,


Test patch:
Repository: horovod/horovod

Issue:
bug in checking sample_weights
this line 
```if sample_weights: ```
in [here](https://github.com/horovod/horovod/blob/994913a2abc5812042a5c3540ced5b3316d23b72/horovod/spark/torch/remote.py#L268) should change to 
```if sample_weights is not None: ```

otherwise it complains about 
```RuntimeError: bool value of Tensor with more than one value is ambiguous.```


Discussion:

Resolving patch:
diff --git a/horovod/spark/torch/remote.py b/horovod/spark/torch/remote.py
--- a/horovod/spark/torch/remote.py
+++ b/horovod/spark/torch/remote.py
@@ -265,7 +265,7 @@ def prepare_batch(row):
                         if cuda_available:
                             inputs = [input.cuda() for input in inputs]
                             labels = [label.cuda() for label in labels]
-                            if sample_weights:
+                            if sample_weights is not None:
                                 sample_weights = sample_weights.cuda()
                         return inputs, labels, sample_weights
 


Test patch:
Repository: horovod/horovod

Issue:
Allow Horovod to be installed without native libraries
Now that Horovod supports running separate driver and worker processes (horovod.run, horovod.spark, horovod.ray), processes that run on the driver do not need native libraries to operate correctly. Being able to install Horovod without native libraries on driver nodes can speed up compile times and simplify some integration testing scenarios.

The proposed solution is to provide a build env variable `HOROVOD_SKIP_COMPILE=1` that will avoid calling `cmake`.


Discussion:

Resolving patch:
diff --git a/horovod/common/util.py b/horovod/common/util.py
--- a/horovod/common/util.py
+++ b/horovod/common/util.py
@@ -141,7 +141,7 @@ def mpi_built(verbose=False):
             ext_base_name, built_fn, 'built with MPI', verbose)
         if result is not None:
             return result
-    return False
+    return None
 
 
 @_cache
@@ -152,9 +152,7 @@ def gloo_built(verbose=False):
             ext_base_name, built_fn, 'built with Gloo', verbose)
         if result is not None:
             return result
-    raise RuntimeError('Failed to determine if Gloo support has been built. '
-                       'Run again with --verbose for more details.')
-
+    return None
 
 @_cache
 def nccl_built(verbose=False):
@@ -164,9 +162,7 @@ def nccl_built(verbose=False):
             ext_base_name, built_fn, 'built with NCCL', verbose)
         if result is not None:
             return result
-    raise RuntimeError('Failed to determine if NCCL support has been built. '
-                       'Run again with --verbose for more details.')
-
+    return None
 
 @_cache
 def ddl_built(verbose=False):
@@ -176,9 +172,7 @@ def ddl_built(verbose=False):
             ext_base_name, built_fn, 'built with DDL', verbose)
         if result is not None:
             return result
-    raise RuntimeError('Failed to determine if DDL support has been built. '
-                       'Run again with --verbose for more details.')
-
+    return None
 
 @_cache
 def ccl_built(verbose=False):
@@ -188,9 +182,7 @@ def ccl_built(verbose=False):
             ext_base_name, built_fn, 'built with CCL', verbose)
         if result is not None:
             return result
-    raise RuntimeError('Failed to determine if CCL support has been built. '
-                       'Run again with --verbose for more details.')
-
+    return None
 
 @contextmanager
 def env(**kwargs):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -59,6 +59,11 @@ def get_cmake_bin():
 
 class custom_build_ext(build_ext):
     def build_extensions(self):
+        if os.getenv('HOROVOD_SKIP_COMPILE') == '1':
+            # Skip building extensions using CMake
+            print("Horovod is being installed without native libraries")
+            return
+
         cmake_bin = get_cmake_bin()
 
         config = 'Debug' if self.debug else 'RelWithDebInfo'


Test patch:
Repository: horovod/horovod

Issue:
PyTorch sparse allreduce fails with torch nightly
Repro:

```
pip install --no-cache-dir --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
# install horovod
horovodrun -np 2 pytest -vs "test/parallel/test_torch.py::TorchTests::test_async_sparse_allreduce"
```

The test hangs, which has been causing test failures.

cc @chongxiaoc @romerojosh 

@chongxiaoc can you or @irasit take a look?


Discussion:
hmmm, only for nightly? I'm curious what's recent change from pytorch nightly.
Without fusion, it can pass:

(hvd-py3.7) chongxiaoc@chongxiaoc-C02DL7WPMD6R parallel % horovodrun --fusion-threshold-mb 0 -np 2 pytest -vs test_torch.py::TorchTests::test_async_sparse_allreduce


Resolving patch:
diff --git a/horovod/torch/mpi_ops.py b/horovod/torch/mpi_ops.py
--- a/horovod/torch/mpi_ops.py
+++ b/horovod/torch/mpi_ops.py
@@ -518,8 +518,11 @@ def sparse_allreduce_async(tensor, name, op):
     values_handle = allgather_async(t._values(), name=f'{name}.values')
 
     def handle():
-        indices = synchronize(indices_handle)
+        # We need to sync values handle firstly for torch nightly >= 10.0
+        # Issue: https://github.com/horovod/horovod/issues/2961
         values = synchronize(values_handle)
+        indices = synchronize(indices_handle)
+
         values = (values / size()) if op == Average else values
 
         if indices.dim() == 0 or values.dim() == 0:


Test patch:
diff --git a/test/parallel/test_torch.py b/test/parallel/test_torch.py
--- a/test/parallel/test_torch.py
+++ b/test/parallel/test_torch.py
@@ -2421,8 +2421,6 @@ def forward(self, x):
             loss.backward()
             opt.step()
 
-    @pytest.mark.skipif(LooseVersion(torch.__version__) > LooseVersion('1.10.0'),
-                        reason='https://github.com/horovod/horovod/issues/2961')
     def test_async_sparse_allreduce(self):
         """Test that allgather over indices and values is equivalent to allreduce."""
         hvd.init()
Repository: horovod/horovod

Issue:
【Elastic Horovod】torch op handles is not recollect right away as new elastic loop begins
Environment:

Framework: (TensorFlow, Keras, PyTorch, MXNet): Pytorch
Framework version: 1.6.0
Horovod version: 0.21.3
MPI version: 4.0.3
CUDA version: 10.2
NCCL version: 2.7.6
Python version: 3.6
Checklist:

Did you search issues to find if somebody asked this question before? Yes.
If your question is about hang, did you read this doc?
If your question is about docker, did you read this doc?
Did you check if you question is answered in the [troubleshooting guide] (https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)? Yes

**Bug report:**
horovod torch currently use a dict to hold references to horovod operation results and release such reference in synchronize function.

```python
def synchronize(handle):
    if handle not in _handle_map:
        return
    try:
        mpi_lib.horovod_torch_wait_and_clear(handle)
        output = _handle_map.pop(handle)[-1]
        return output
    except RuntimeError as e:
        raise HorovodInternalError(e)
```
this code, however, does not releases the handle when exception occurs.as a result, in elastic scenario, handle references may still be retained in the dict for some time in new training loop, and will not be released until the handle counter reaches the same point again. this may be a problem when memory/gpumem is heavily used, prone to oom.

I am going to submit a pr to fix this.


Discussion:

Resolving patch:
diff --git a/horovod/torch/mpi_ops.py b/horovod/torch/mpi_ops.py
--- a/horovod/torch/mpi_ops.py
+++ b/horovod/torch/mpi_ops.py
@@ -40,8 +40,8 @@
 _NULL = ""
 
 _basics = _HorovodBasics(__file__, 'mpi_lib_v2')
+
 # import basic methods
-init = _basics.init
 is_initialized = _basics.is_initialized
 start_timeline = _basics.start_timeline
 stop_timeline = _basics.stop_timeline
@@ -61,10 +61,16 @@
 ccl_built = _basics.ccl_built
 cuda_built = _basics.cuda_built
 rocm_built = _basics.rocm_built
+
 def shutdown(*args, **kwargs):
     mpi_lib.horovod_torch_reset()
     return _basics.shutdown(*args, **kwargs)
 
+def init(*args, **kwargs):
+    global _handle_map
+    _handle_map = {}
+    return _basics.init(*args, **kwargs)
+
 # import reduction op values
 Average = _basics.Average
 Sum = _basics.Sum
@@ -939,6 +945,7 @@ def synchronize(handle):
         output = _handle_map.pop(handle)[-1]
         return output
     except RuntimeError as e:
+        _handle_map.pop(handle, None)
         raise HorovodInternalError(e)
 
 


Test patch:
Repository: horovod/horovod

Issue:
Unable to extract storage_options from URL
Integration of fsspec in Horovod was a great step.

Fsspec support extracting storage_options from input url. In Horovod, the url format is restricted i.e the option of extracting the storage options from url is blocked.

It would be of great help if we could have mechanism to extract the storage_options from the input url similar to fsspec.

For above to work, we need to make changes to _get_fs_and_protocol(self) method in store file. The modified def will look like:
```
def _get_fs_and_protocol(self):
        protocol, path = split_protocol(self.prefix_path)
        cls = fsspec.get_filesystem_class(protocol)
        options = cls._get_kwargs_from_urls(self._dataset_url)
        update_storage_options(options, storage_options)
        fs = cls(**options)
        return fs, protocol
```


Discussion:

Resolving patch:
diff --git a/docs/mocks.py b/docs/mocks.py
--- a/docs/mocks.py
+++ b/docs/mocks.py
@@ -39,6 +39,7 @@ def _dummy():
 
     'fsspec',
     'fsspec.core',
+    'fsspec.utils',
 
     'pyarrow',
     'pyarrow.parquet',
diff --git a/horovod/spark/common/store.py b/horovod/spark/common/store.py
--- a/horovod/spark/common/store.py
+++ b/horovod/spark/common/store.py
@@ -29,6 +29,7 @@
 
 import fsspec
 from fsspec.core import split_protocol
+from fsspec.utils import update_storage_options
 
 from horovod.spark.common.util import is_databricks, host_hash
 
@@ -337,8 +338,12 @@ def fs(self):
 
     #@staticmethod
     def _get_fs_and_protocol(self):
+        storage_options = self.storage_options or {}
         protocol, path = split_protocol(self.prefix_path)
-        fs = fsspec.filesystem(protocol, **self.storage_options)
+        cls = fsspec.get_filesystem_class(protocol)
+        options = cls._get_kwargs_from_urls(self.prefix_path)
+        update_storage_options(options, storage_options)
+        fs = cls(**options)
         return fs, protocol
 
     @classmethod


Test patch: