Repository: horovod/horovod

Issue:
Cannot install horovod[spark] for Tensorflow 2.6
**Environment:**
1. Framework: TensorFlow
2. Framework version:2.6.2
3. Horovod version: 0.23
4. MPI version:4.1.1
5. CUDA version:N/A
6. NCCL version:N/A
7. Python version: 3.7
8. Spark / PySpark version: 2.4.5
9. Ray version:N/A
10. OS and version: RHEL 8.4
11. GCC version: 9.3.0
12. CMake version: 3.5.0

**Checklist:**
1. Did you search issues to find if somebody asked this question before?  Yes
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)? N/A
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)? N/A
4. Did you check if you question is answered in the [troubleshooting guide] (https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)? Yes

**Bug report:**
Please describe erroneous behavior you're observing and steps to reproduce it.

```
Installing collected packages: pyparsing, pycparser, pyzmq, pyyaml, pyarrow, psutil, packaging, future, fsspec, diskcache, dill, cloudpickle, cffi, petastorm, horovod, h5py
  Attempting uninstall: h5py
    Found existing installation: h5py 3.1.0
    Uninstalling h5py-3.1.0:
      Successfully uninstalled h5py-3.1.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow 2.6.2 requires h5py~=3.1.0, but you have h5py 2.10.0 which is incompatible.
```

**Reproduce Steps:**

1. `conda create -n horovod python=3.7`
2. `conda activate horovod`
3. `conda install pyspark=2.4.5 openmpi-mpicc cmake -c conda-forge`
4. `pip install tensorflow==2.6.2`
5. `HOROVOD_WITH_MPI=1 HOROVOD_WITH_TENSORFLOW=1 pip install horovod[spark]`



Discussion:
Which pip version do you use?
Hi, @EnricoMi . The pip version I used is 21.0.1. 
I think the `h5py==2.10.0` is needed for TF < 2.5. Can you try

    HOROVOD_WITH_MPI=1 HOROVOD_WITH_TENSORFLOW=1 pip install horovod[spark] h5py~=3.1.0
@tgaddair we are seeing the same error in our CI builds, but build does not fail:

https://github.com/horovod/horovod/runs/4199828356?check_suite_focus=true#step:10:7398
```
Successfully built horovod
Installing collected packages: pycparser, pyzmq, pyarrow, psutil, hiredis, diskcache, dill, cloudpickle, cffi, petastorm, horovod, h5py, aioredis
  changing mode of /usr/local/bin/plasma_store to 755
  changing mode of /usr/local/bin/petastorm-copy-dataset.py to 755
  changing mode of /usr/local/bin/petastorm-generate-metadata.py to 755
  changing mode of /usr/local/bin/petastorm-throughput.py to 755
  changing mode of /usr/local/bin/horovodrun to 755
  Attempting uninstall: h5py
    Found existing installation: h5py 3.1.0
    Uninstalling h5py-3.1.0:
      Removing file or directory /usr/local/lib/python3.7/dist-packages/h5py-3.1.0.dist-info/
      Removing file or directory /usr/local/lib/python3.7/dist-packages/h5py.libs/
      Removing file or directory /usr/local/lib/python3.7/dist-packages/h5py/
      Successfully uninstalled h5py-3.1.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow-cpu 2.6.0 requires h5py~=3.1.0, but you have h5py 2.10.0 which is incompatible.
Successfully installed aioredis-1.3.1 cffi-1.15.0 cloudpickle-2.0.0 dill-0.3.4 diskcache-5.2.1 h5py-2.10.0 hiredis-2.0.0 horovod-0.23.0 petastorm-0.11.3 psutil-5.8.0 pyarrow-6.0.0 pycparser-2.21 pyzmq-22.3.0
```

It successfully installs horovod and then, due to `horovod[spark]` applies the `h5py<3` constraint, which conflicts with TF's dependency. I think `h5py<3` is only needed for tf<2.5, at least that is what `Dockerfile.test.cpu` says:

    # Pin h5py only for tensorflow<2.5: https://github.com/h5py/h5py/issues/1732

We cannot articulate that dependency based on the installed tensorflow version. I think we should remove `h5py<3` entirely from the `spark_require_list` in `setup.py` as we have TF 2.5, 2.6 and 2.7.
This is the same issue I encountered when I tried to build a docker image using TensorFlow 2.5 with horovod[spark]. However, the `h5py<3` constraint broke the building process. I'm wondering if there would be workaround for this? Thanks in advance.

Resolving patch:
diff --git a/.github/gen-workflow-ci.py b/.github/gen-workflow-ci.py
--- a/.github/gen-workflow-ci.py
+++ b/.github/gen-workflow-ci.py
@@ -505,6 +505,7 @@ def build_and_test_macos(id: str, name: str, needs: List[str], attempts: int = 3
                 f'\n'
                 f'          python -m pip install -U pip\n'
                 f'          pip install tensorflow==${{TENSORFLOW}} keras==${{KERAS}}\n'
+                f'          if [[ ${{TENSORFLOW}} == 1.* ]] || [[ ${{TENSORFLOW}} == 2.[012345].* ]]; then pip install "h5py<3"; fi\n'
                 f'          pip install torch==${{PYTORCH}} pytorch_lightning==${{PYTORCH_LIGHTNING}} torchvision==${{TORCHVISION}}\n'
                 f'          pip install mxnet==${{MXNET}}\n'
                 f'          HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_MXNET=1 pip install --no-cache-dir .[test]\n'
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -124,8 +124,7 @@ def build_extensions(self):
 mxnet_require_list = ['mxnet>=1.4.1']
 pyspark_require_list = ['pyspark>=2.3.2;python_version<"3.8"',
                         'pyspark>=3.0.0;python_version>="3.8"']
-# Pin h5py: https://github.com/h5py/h5py/issues/1732
-spark_require_list = ['h5py<3', 'numpy', 'petastorm>=0.11.0', 'pyarrow>=0.15.0', 'fsspec']
+spark_require_list = ['numpy', 'petastorm>=0.11.0', 'pyarrow>=0.15.0', 'fsspec']
 # https://github.com/ray-project/ray/pull/17465
 ray_require_list = ['ray', 'aioredis<2']
 pytorch_spark_require_list = pytorch_require_list + \
@@ -154,8 +153,7 @@ def build_extensions(self):
 # torchvision 0.5.0 depends on torch==1.4.0
 
 # python packages required only to run tests
-# Pin h5py: https://github.com/h5py/h5py/issues/1732
-test_require_list = ['mock', 'pytest', 'pytest-forked', 'parameterized', 'h5py<3']
+test_require_list = ['mock', 'pytest', 'pytest-forked', 'parameterized']
 
 # Skip cffi if pytorch extension explicitly disabled
 if not os.environ.get('HOROVOD_WITHOUT_PYTORCH'):


Test patch:
diff --git a/Dockerfile.test.cpu b/Dockerfile.test.cpu
--- a/Dockerfile.test.cpu
+++ b/Dockerfile.test.cpu
@@ -140,7 +140,7 @@ RUN if [[ ${MPI_KIND} != "None" ]]; then \
 RUN if [[ ${TENSORFLOW_PACKAGE} != "tf-nightly" ]]; then \
         pip install --no-cache-dir ${TENSORFLOW_PACKAGE}; \
         if [[ ${KERAS_PACKAGE} != "None" ]]; then \
-            if [[ ${TENSORFLOW_PACKAGE} == tensorflow*==1.* ]] || [[ ${TENSORFLOW_PACKAGE} == tensorflow*==2.[01234].* ]]; then \
+            if [[ ${TENSORFLOW_PACKAGE} == tensorflow*==1.* ]] || [[ ${TENSORFLOW_PACKAGE} == tensorflow*==2.[012345].* ]]; then \
                 h5py="h5py<3"; \
             fi; \
             pip uninstall -y keras-nightly; \
@@ -186,7 +186,6 @@ COPY . /horovod
 # Install nightly packages here so they do not get cached
 
 # Install TensorFlow and Keras (nightly).
-# Do not pin h5py since tf>=2.5 requires h5py~=3.1
 # Pin scipy!=1.4.0: https://github.com/scipy/scipy/issues/11237
 RUN if [[ ${TENSORFLOW_PACKAGE} == "tf-nightly" ]]; then \
         pip install --no-cache-dir ${TENSORFLOW_PACKAGE}; \
diff --git a/Dockerfile.test.gpu b/Dockerfile.test.gpu
--- a/Dockerfile.test.gpu
+++ b/Dockerfile.test.gpu
@@ -108,7 +108,7 @@ RUN if [[ ${MPI_KIND} != "None" ]]; then \
 RUN if [[ ${TENSORFLOW_PACKAGE} != "tf-nightly-gpu" ]]; then \
         pip install --no-cache-dir ${TENSORFLOW_PACKAGE}; \
         if [[ ${KERAS_PACKAGE} != "None" ]]; then \
-            if [[ ${TENSORFLOW_PACKAGE} == tensorflow*==1.* ]] || [[ ${TENSORFLOW_PACKAGE} == tensorflow*==2.[01234].* ]]; then \
+            if [[ ${TENSORFLOW_PACKAGE} == tensorflow*==1.* ]] || [[ ${TENSORFLOW_PACKAGE} == tensorflow*==2.[012345].* ]]; then \
                 h5py="h5py<3"; \
             fi; \
             pip uninstall -y keras-nightly; \
@@ -155,7 +155,6 @@ COPY . /horovod
 # Install nightly packages here so they do not get cached
 
 # Install TensorFlow and Keras (nightly).
-# Do not pin h5py since tf>=2.5 requires h5py~=3.1
 # Pin scipy!=1.4.0: https://github.com/scipy/scipy/issues/11237
 RUN if [[ ${TENSORFLOW_PACKAGE} == "tf-nightly-gpu" ]]; then \
         pip install --no-cache-dir ${TENSORFLOW_PACKAGE}; \