Repository: horovod/horovod

Issue:
Incremental build support
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet) TensorFlow
2. Framework version: TF 1.14.0
3. Horovod version: tip of master
4. MPI version:
5. CUDA version: 10.0
6. NCCL version: tip of master
7. Python version: 3.6.8
8. OS and version: Ubuntu 18.04
9. GCC version: 7.4.0

**Checklist:**
1. Did you search issues to find if somebody asked this question before? Yes.
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)? N/A
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)? N/A
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)? Yes.

**Your question:**

Is there any way to do an incremental build of horovod?  I can't figure out a way to build horovod from a local copy of the source code, except through `pip install .`, but that seems to build from scratch every time, regardless of changes to the source code.


Discussion:
Hey @rb-determined-ai, are you modifying C++ or Python files?  For C++, I'm not aware of any way to speed things up currently other than to exclude frameworks you're not using (e.g., `HOROVOD_WITHOUT_TENSORFLOW=1`.  That said, we have plans to migrate our build process to CMake sometime next year, which should enable incremental build support (though we haven't yet tested that).

For Python, things are a bit easier, you can just rsync or cp the files into the install path and things should work fine.  
Yeah, I'm referring to the C++ code.  I'll look forward to the CMake build system.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -16,6 +16,7 @@
 # ==============================================================================
 
 import os
+import shutil
 import subprocess
 import sys
 import textwrap
@@ -25,6 +26,7 @@
 
 from horovod import __version__
 
+_FRAMEWORK_METADATA_FILE = 'horovod/metadata.json'
 
 class CMakeExtension(Extension):
     def __init__(self, name, cmake_lists_dir='.', sources=[], **kwa):
@@ -52,6 +54,8 @@ def is_build_action():
     if sys.argv[1].startswith('install'):
         return True
 
+    if sys.argv[1].startswith('develop'):
+        return True
 
 def get_cmake_bin():
     return os.environ.get('HOROVOD_CMAKE', 'cmake')
@@ -66,7 +70,7 @@ def build_extensions(self):
 
         cmake_bin = get_cmake_bin()
 
-        config = 'Debug' if self.debug else 'RelWithDebInfo'
+        config = 'Debug' if self.debug or os.environ.get('HOROVOD_DEBUG') == "1" else 'RelWithDebInfo'
 
         ext_name = self.extensions[0].name
         build_dir = self.get_ext_fullpath(ext_name).replace(self.get_ext_filename(ext_name), '')
@@ -98,6 +102,13 @@ def build_extensions(self):
         except OSError as e:
             raise RuntimeError('CMake failed: {}'.format(str(e)))
 
+        if sys.argv[1].startswith('develop'):
+            # Copy over metadata.json file from build directory
+            shutil.copyfile(os.path.join(build_dir, _FRAMEWORK_METADATA_FILE),
+                            os.path.join(self.extensions[0].cmake_lists_dir, _FRAMEWORK_METADATA_FILE))
+            # Remove unfound frameworks, otherwise develop mode will fail the install
+            self.extensions = [x for x in self.extensions if os.path.exists(self.get_ext_fullpath(x.name))]
+
 
 # python packages required to use horovod in general
 require_list = ['cloudpickle', 'psutil', 'pyyaml', 'dataclasses;python_version<"3.7"']


Test patch: