opened
DEKHTIARJonathan
username_0
For performance optimization I try to use prefetch to device, however I consistently obtain the same error

```python
# https://www.tensorflow.org/api_docs/python/tf/data/experimental/copy_to_device
ds = ds.apply(
        tf.data.experimental.copy_to_device(
            target_device='/GPU:%d' % hvd.local_rank(),
            source_device='/cpu:0'
        )
)
# ===============
# OR
# ===============

# https://www.tensorflow.org/api_docs/python/tf/data/experimental/prefetch_to_device 
ds = ds.apply(
       tf.data.experimental.prefetch_to_device(
          device,
          buffer_size=None
       )
)
```

Error

```python
ERROR:tensorflow:Model diverged with loss = NaN.

tensorflow.python.framework.errors_impl.UnknownError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
	 [[{{node DistributedLossScalingOptimizer_Allreduce/HorovodAllreduce_DistributedLossScalingOptimizer_Allreduce_Cast_481_0}}]]
	 [[{{node acc_top5_ref}}]]
```
Prefetch to GPU device 
issue
created
alsrgv
username_1
@username_0, do you pin GPU in your session config [like this](https://github.com/uber/horovod/blob/master/examples/tensorflow_mnist.py#L141)?  If so, you should use '/gpu:0' in prefetch configuration.
comment
closed
DEKHTIARJonathan
username_0
issue
created
DEKHTIARJonathan
username_0
Seems like the issue is on TF side. I will re-open if I have any evidence that this is caused by HVD ;)
Thanks for your answer
comment