Repository: horovod/horovod

Issue:
Horovod does not compile with old framework versions that setup.py accept
Building Horovod, the log says:

But actually building Horovod against PyTorch 1.2.0, it fails with:

```
    /tmp/pip-req-build-13fs79hx/horovod/torch/mpi_ops_v2.cc:61:12: error: ‘class at::Tensor’ has no member named ‘floor_divide_’
         tensor.floor_divide_(divisor);
                ^~~~~~~~~~~~~
```

Looks like the minimum PyTorch that Horovod compiles with is 1.5.0.


Discussion:

Resolving patch:
diff --git a/.github/gen-workflow-ci.py b/.github/gen-workflow-ci.py
--- a/.github/gen-workflow-ci.py
+++ b/.github/gen-workflow-ci.py
@@ -270,12 +270,14 @@ def build_and_test_images(id: str,
                               name: str,
                               needs: List[str],
                               images: List[str],
-                              parallel_images: str,
                               tests_per_image: Dict[str, Set[str]],
                               tests: Dict[str, Dict],
+                              parallel_images: int = None,
                               attempts: int = 3) -> str:
         if 'init-workflow' not in needs:
             needs.insert(0, 'init-workflow')
+        if parallel_images is None:
+          parallel_images = len(images)
         failure = "'failure'"
         return (f'  {id}:\n'
                 f'    name: "{name} (${{{{ matrix.image }}}})"\n'
@@ -286,13 +288,13 @@ def build_and_test_images(id: str,
                 f'    runs-on: ubuntu-latest\n'
                 f'\n'
                 f'    strategy:\n'
-                f'      max-parallel: {len([image for image in images if parallel_images in image])}\n'
+                f'      max-parallel: {parallel_images}\n'
                 f'      fail-fast: false\n'
                 f'      matrix:\n'
                 f'        include:\n' +
                 '\n'.join([f'          - image: {image}\n' +
                            f''.join([f'            {test}: true\n'
-                                     for test in sorted(list(tests_per_image[image]))]) +
+                                     for test in sorted(list(tests_per_image.get(image, [])))]) +
                            f'            build_timeout: {30 if "-cpu-" in image else 40}\n'
                            for image in sorted(images)
                            # oneccl does not compile on GitHub Workflows:
@@ -603,7 +605,7 @@ def publish_docker_images(needs: List[str], images: List[str]) -> str:
         return (f'  docker-config:\n'
                 f'    name: Configure docker build\n'
                 f'    needs: [{", ".join(needs)}]\n'
-                f"    # build-and-test-cpu, build-gpu and buildkite might have been skipped (! needs.init-workflow.outputs.run-builds-and-tests)\n"
+                f"    # build-and-test and buildkite might have been skipped (! needs.init-workflow.outputs.run-builds-and-tests)\n"
                 f'    # buildkite might have been skipped (workflow runs for a fork PR),\n'
                 f'    # we still want to build docker images (though we might not want to push them)\n'
                 f'    if: >\n'
@@ -795,16 +797,19 @@ def sync_files(needs: List[str]) -> str:
                 f'          fi\n')
 
     with open(path.joinpath('workflows', 'ci.yaml').absolute(), 'wt') as w:
+        mins = ['tfmin', 'torchmin', 'mxnetmin']
         heads = ['tfhead', 'torchhead', 'mxnethead']
-        release_images = [image for image in images if not all(head in image for head in heads)]
+        allmin_images = [image for image in images if all(min in image for min in mins)]
+        allhead_images = [image for image in images if all(head in image for head in heads)]
+        release_images = [image for image in images if image not in allhead_images + allmin_images]
         cpu_release_images = [image for image in release_images if '-cpu-' in image]
         gpu_release_images = [image for image in release_images if '-gpu-' in image or '-mixed-' in image]
-        allhead_images = [image for image in images if all(head in image for head in heads)]
         workflow = workflow_header() + jobs(
             init_workflow_job(),
-            # changing these names require changes in the workflow-conclusion step in ci-fork.yaml
-            build_and_test_images(id='build-and-test', name='Build and Test', needs=['init-workflow'], images=release_images, parallel_images='-cpu-', tests_per_image=tests_per_image, tests=tests),
-            build_and_test_images(id='build-and-test-heads', name='Build and Test heads', needs=['build-and-test'], images=allhead_images, parallel_images='', tests_per_image=tests_per_image, tests=tests),
+            # changing these names require changes in the workflow-conclusion step in ci-results.yaml
+            build_and_test_images(id='build-and-test', name='Build and Test', needs=['init-workflow'], images=release_images, parallel_images=len(cpu_release_images), tests_per_image=tests_per_image, tests=tests),
+            build_and_test_images(id='build-and-test-heads', name='Build and Test heads', needs=['build-and-test'], images=allhead_images, tests_per_image=tests_per_image, tests=tests),
+            build_and_test_images(id='build-mins', name='Build mins', needs=['build-and-test'], images=allmin_images, tests_per_image=tests_per_image, tests={}),
             build_and_test_macos(id='build-and-test-macos', name='Build and Test macOS', needs=['build-and-test']),
             trigger_buildkite_job(id='buildkite', name='Build and Test GPU (on Builtkite)', needs=['build-and-test'], mode='GPU NON HEADS'),
             trigger_buildkite_job(id='buildkite-heads', name='Build and Test GPU heads (on Builtkite)', needs=['build-and-test'], mode='GPU HEADS'),


Test patch:
diff --git a/docker-compose.test.yml b/docker-compose.test.yml
--- a/docker-compose.test.yml
+++ b/docker-compose.test.yml
@@ -97,6 +97,21 @@ services:
         TORCHVISION_PACKAGE: torchvision
         PYTORCH_LIGHTNING_PACKAGE: pytorch_lightning==1.3.8
         MXNET_PACKAGE: mxnet-nightly
+  # these are the lowest framework versions that Horovod compiles with, but they are not tested
+  test-cpu-gloo-py3_7-tfmin-kerasmin-torchmin-mxnetmin-pysparkmin:
+    extends: test-cpu-base
+    build:
+      args:
+        UBUNTU_VERSION: 18.04
+        PYTHON_VERSION: 3.7
+        TENSORFLOW_PACKAGE: tensorflow-cpu==1.15.0
+        KERAS_PACKAGE: keras==2.2.4
+        PYTORCH_PACKAGE: torch==1.5.0+cpu
+        PYTORCH_LIGHTNING_PACKAGE: pytorch-lightning==0.7.3
+        TORCHVISION_PACKAGE: torchvision==0.6.0+cpu
+        MXNET_PACKAGE: mxnet==1.4.1
+        PYSPARK_PACKAGE: pyspark==2.4.0
+        SPARK_PACKAGE: spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
 
   test-cpu-gloo-py3_7-tf2_8_0-keras2_8_0-torch1_11_0-mxnet1_9_0-pyspark2_4_8:
     extends: test-cpu-base
@@ -213,6 +228,23 @@ services:
         PYTORCH_LIGHTNING_PACKAGE: pytorch_lightning==1.3.8
         TORCHVISION_PACKAGE: torchvision
         MXNET_PACKAGE: mxnet-nightly-cu112
+  # these are the lowest framework versions that Horovod compiles with, but they are not tested
+  test-gpu-gloo-py3_7-tfmin-kerasmin-torchmin-mxnetmin-pysparkmin:
+    extends: test-gpu-base
+    build:
+      args:
+        CUDA_DOCKER_VERSION: 10.0-devel-ubuntu18.04
+        CUDNN_VERSION: 7.6.5.32-1+cuda10.1
+        NCCL_VERSION_OVERRIDE: 2.7.8-1+cuda10.1
+        PYTHON_VERSION: 3.7
+        TENSORFLOW_PACKAGE: tensorflow-gpu==1.15.0
+        KERAS_PACKAGE: keras==2.2.4
+        PYTORCH_PACKAGE: torch==1.5.0+cu101
+        PYTORCH_LIGHTNING_PACKAGE: pytorch-lightning==0.7.3
+        TORCHVISION_PACKAGE: torchvision==0.6.0+cu101
+        MXNET_PACKAGE: mxnet-cu100==1.4.1
+        PYSPARK_PACKAGE: pyspark==2.4.0
+        SPARK_PACKAGE: spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
 
   test-mixed-openmpi-gloo-py3_8-tf2_8_0-keras2_8_0-torch1_11_0-mxnet1_9_0-pyspark3_2_1:
     extends: test-gpu-base
diff --git a/test/single/data/expected_buildkite_gpu_non_heads_pipeline.yaml b/test/single/data/expected_buildkite_gpu_non_heads_pipeline.yaml
--- a/test/single/data/expected_buildkite_gpu_non_heads_pipeline.yaml
+++ b/test/single/data/expected_buildkite_gpu_non_heads_pipeline.yaml
@@ -67,6 +67,23 @@ steps:
     automatic: true
   agents:
     queue: cpu-v572
+- label: ':docker: Build test-gpu-gloo-py3_7-tfmin-kerasmin-torchmin-mxnetmin-pysparkmin'
+  env:
+    COMPOSE_HTTP_TIMEOUT: 300
+  plugins:
+  - docker-compose#v3.5.0:
+      build: test-gpu-gloo-py3_7-tfmin-kerasmin-torchmin-mxnetmin-pysparkmin
+      image-repository: 823773083436.dkr.ecr.us-east-1.amazonaws.com/buildkite
+      cache-from: test-gpu-gloo-py3_7-tfmin-kerasmin-torchmin-mxnetmin-pysparkmin:823773083436.dkr.ecr.us-east-1.amazonaws.com/buildkite:SLUG-test-gpu-gloo-py3_7-tfmin-kerasmin-torchmin-mxnetmin-pysparkmin-latest
+      config: docker-compose.test.yml
+      push-retries: 5
+  - ecr#v1.2.0:
+      login: true
+  timeout_in_minutes: 40
+  retry:
+    automatic: true
+  agents:
+    queue: cpu-v572
 - label: ':docker: Build test-mixed-openmpi-gloo-py3_8-tf2_8_0-keras2_8_0-torch1_11_0-mxnet1_9_0-pyspark3_2_1'
   env:
     COMPOSE_HTTP_TIMEOUT: 300