Repository: horovod/horovod

Issue:
AttributeError: 'NoneType' object has no attribute 'dtype'
Some models, like [RFCN](https://github.com/tensorflow/models/tree/master/research/object_detection) can return variables without gradients
```Traceback (most recent call last):
  File "train.py", line 249, in <module>
    tf.app.run()
  File "/home/csq/.cache/bazel/_bazel_csq/0a3580d8ecd2fafe9cc9970e974f5dc4/execroot/source/bazel-out/release_links/lib/python_env/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "train.py", line 229, in main
    worker_job_name, is_chief, FLAGS.train_dir, FLAGS.trace_every_n_steps)
  File "/home/csq/src/models/research/object_detection/trainer.py", line 271, in train
    for (gradient, var) in gradients]
  File "/home/csq/.cache/bazel/_bazel_csq/0a3580d8ecd2fafe9cc9970e974f5dc4/execroot/source/bazel-out/release_links/lib/python_env/horovod/tensorflow/__init__.py", line 76, in allreduce
    horovod_size = tf.cast(size(), tensor.dtype)
AttributeError: 'NoneType' object has no attribute 'dtype'
```
I just work around it by replacing compute_gradients() with
```
gradients = (super(hvd.DistributedOptimizer, training_optimizer)
             .compute_gradients(total_loss))
gradients = [(g, v) for g, v in gradients if g is not None]
if hvd.size() > 1:
  with tf.name_scope(training_optimizer._name + "_Allreduce"):
    grads_and_vars = [(hvd.allreduce(gradient, device_dense=training_optimizer._device_dense,
                                                    device_sparse=training_optimizer._device_sparse), var)
                      for (gradient, var) in gradients]
```
but think adding
```
gradients = [(g, v) for g, v in gradients if g is not None]
``` 
after this line: https://github.com/uber/horovod/blob/64317537436ae4225a004043acaa37a205a18dc3/horovod/tensorflow/__init__.py#L166

could fix it for others.


Discussion:
Thanks @crizCraig for reporting this issue!

I think the more transparent way could be to retain `(None, var)` in the list of computed gradients and just bypass all reduce phase for those gradients.  This seems more aligned with original optimizer interface.

Do you agree?

Thanks,
Alex
That sounds reasonable per https://github.com/tensorflow/tensorflow/blob/84e00bf64d89b8b39fe3b82fa305ee13b2a5613b/tensorflow/python/training/optimizer.py#L381-L383

The underlying optimizer removed the variables without a gradient in my case, so maybe that solution is more specific to me than I thought.

Resolving patch:
diff --git a/horovod/tensorflow/__init__.py b/horovod/tensorflow/__init__.py
--- a/horovod/tensorflow/__init__.py
+++ b/horovod/tensorflow/__init__.py
@@ -166,10 +166,16 @@ def compute_gradients(self, *args, **kwargs):
         gradients = (super(DistributedOptimizer, self)
                      .compute_gradients(*args, **kwargs))
         if size() > 1:
+            averaged_gradients = []
             with tf.name_scope(self._name + "_Allreduce"):
-                return [(allreduce(gradient, device_dense=self._device_dense,
-                                   device_sparse=self._device_sparse), var)
-                        for (gradient, var) in gradients]
+                for grad, var in gradients:
+                    if grad is not None:
+                        avg_grad = allreduce(grad, device_dense=self._device_dense,
+                                             device_sparse=self._device_sparse)
+                        averaged_gradients.append((avg_grad, var))
+                    else:
+                        averaged_gradients.append((None, var))
+            return averaged_gradients
         else:
             return gradients
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -315,7 +315,7 @@ def build_extensions(self):
 
 
 setup(name='horovod',
-      version='0.9.9',
+      version='0.9.10',
       packages=find_packages(),
       description='Distributed training framework for TensorFlow.',
       author='Uber Technologies, Inc.',


Test patch: