Repository: horovod/horovod

Issue:
DistributedOptimizer cannot be used when underlying optimizer overwrites compute_gradients and apply_gradients
In our use case we have a TensorFlow optimizer that explicitly defines `compute_gradients` and `apply_gradients` functions. The current version of `DistributedOptimizer` cannot be used in our case because it does not overwrite `apply_gradients` to use` self._optimizer.apply_gradients` and it calls (lines 178-179 of `horovod/tensorflow/__init__.py`)
```python
gradients = super(DistributedOptimizer, self).compute_gradients(*args, **kwargs)
```
instead of `gradients = self._optimizer.compute_gradients(*args, **kwargs)`.

Is it possible to update `DistributedOptimizer` to work in our use case and similar situations when `compute_gradients` and `apply_gradients` are redefined? As far as I can see simply changing lines 178-179 of `horovod/tensorflow/__init__.py` to 
```python
gradients = self._optimizer.compute_gradients(*args, **kwargs)
```
and adding additional method:
```python
  def apply_gradients(self, grads_and_vars, global_step=None, name=None):
    """Calls this same method on the underlying optimizer."""
    return self._optimizer.apply_gradients(grads_and_vars, global_step, name)
```
should make it work.


Discussion:

Resolving patch:
diff --git a/horovod/__init__.py b/horovod/__init__.py
--- a/horovod/__init__.py
+++ b/horovod/__init__.py
@@ -1 +1 @@
-__version__ = '0.13.0'
+__version__ = '0.13.1'
diff --git a/horovod/tensorflow/__init__.py b/horovod/tensorflow/__init__.py
--- a/horovod/tensorflow/__init__.py
+++ b/horovod/tensorflow/__init__.py
@@ -175,8 +175,7 @@ def compute_gradients(self, *args, **kwargs):
         In DistributedOptimizer, compute_gradients() is overriden to also
         allreduce the gradients before returning them.
         """
-        gradients = (super(DistributedOptimizer, self)
-                     .compute_gradients(*args, **kwargs))
+        gradients = self._optimizer.compute_gradients(*args, **kwargs)
         if size() > 1:
             averaged_gradients = []
             with tf.name_scope(self._name + "_Allreduce"):
@@ -191,42 +190,18 @@ def compute_gradients(self, *args, **kwargs):
         else:
             return gradients
 
-    def _apply_dense(self, *args, **kwargs):
+    def apply_gradients(self, *args, **kwargs):
         """Calls this same method on the underlying optimizer."""
-        return self._optimizer._apply_dense(*args, **kwargs)
+        return self._optimizer.apply_gradients(*args, **kwargs)
 
-    def _resource_apply_dense(self, *args, **kwargs):
+    def get_slot(self, *args, **kwargs):
         """Calls this same method on the underlying optimizer."""
-        return self._optimizer._resource_apply_dense(*args, **kwargs)
+        return self._optimizer.get_slot(*args, **kwargs)
 
-    def _resource_apply_sparse_duplicate_indices(self, *args, **kwargs):
+    def get_slot_names(self, *args, **kwargs):
         """Calls this same method on the underlying optimizer."""
-        return self._optimizer._resource_apply_sparse_duplicate_indices(*args, **kwargs)
+        return self._optimizer.get_slot_names(*args, **kwargs)
 
-    def _resource_apply_sparse(self, *args, **kwargs):
+    def variables(self, *args, **kwargs):
         """Calls this same method on the underlying optimizer."""
-        return self._optimizer._resource_apply_sparse(*args, **kwargs)
-
-    def _apply_sparse_duplicate_indices(self, *args, **kwargs):
-        """Calls this same method on the underlying optimizer."""
-        return self._optimizer._apply_sparse_duplicate_indices(*args, **kwargs)
-
-    def _apply_sparse(self, *args, **kwargs):
-        """Calls this same method on the underlying optimizer."""
-        return self._optimizer._apply_sparse(*args, **kwargs)
-
-    def _prepare(self, *args, **kwargs):
-        """Calls this same method on the underlying optimizer."""
-        return self._optimizer._prepare(*args, **kwargs)
-
-    def _create_slots(self, *args, **kwargs):
-        """Calls this same method on the underlying optimizer."""
-        return self._optimizer._create_slots(*args, **kwargs)
-
-    def _valid_dtypes(self, *args, **kwargs):
-        """Calls this same method on the underlying optimizer."""
-        return self._optimizer._valid_dtypes(*args, **kwargs)
-
-    def _finish(self, *args, **kwargs):
-        """Calls this same method on the underlying optimizer."""
-        return self._optimizer._finish(*args, **kwargs)
+        return self._optimizer.variables(*args, **kwargs)


Test patch: