Repository: horovod/horovod

Issue:
[Horovod on Ray]Failed on example of tensorflow2_mnist_ray.py
**Environment:**
1. Framework: (TensorFlow1.15.0)
3. Horovod version:0.20.0

**Your question:**
Please ask your question here.
Hi @tgaddair @richardliaw I tried the example of Horovod on ray on CPU nodes, but failed.
The ray cluster is ok.

Using TensorFlow backend.
[2m[36m(pid=262081)[0m Using TensorFlow backend.
[2m[36m(pid=262071)[0m Using TensorFlow backend.
Traceback (most recent call last):
  File "tensorflow2_mnist_ray.py", line 94, in <module>
    executor.run(train, kwargs=dict(num_epochs=1))
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/ray/runner.py", line 426, in run
    lambda w: fn(*args, **kwargs)) for worker in self.workers])
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/worker.py", line 1516, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RecursionError): [36mray::BaseHorovodWorker.execute()[39m (pid=262081, ip=10.0.0.131)
  File "python/ray/_raylet.pyx", line 414, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 431, in ray._raylet.execute_task
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 317, in deserialize_objects
    self._deserialize_object(data, metadata, object_id))
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 257, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 238, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 226, in _deserialize_pickle5_data
    obj = pickle.loads(in_band)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)

   .......
  

  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
RecursionError: maximum recursion depth exceeded while calling a Python object




Discussion:
Hey @csdingbin, the example is for TensorFlow 2 but looks like you're using TensorFlow 1.15.

If you want to use TensorFlow 1.15 with Horovod on Ray, it should be possible, but based on the error you're seeing, I would recommend import tensorflow inside the `train` function.  The script would also need to be refactored to look similar to [tensorflow_keras_mnist.py](https://github.com/horovod/horovod/blob/master/examples/tensorflow_keras_mnist.py).
But when I installed Tensorflow==2.1.0 and horovod==0.20.0.
import horovod.tensorflow.keras as hvd will appeared such error:
>>> import horovod.tensorflow.keras as hvd
2020-09-10 21:50:00.960249: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/home/xianyang/sw/miniconda3/lib/:/home/xianyang/sw/miniconda3/lib/
2020-09-10 21:50:00.960360: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/home/xianyang/sw/miniconda3/lib/:/home/xianyang/sw/miniconda3/lib/
2020-09-10 21:50:00.960377: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/tensorflow/__init__.py", line 26, in <module>
    from horovod.tensorflow import elastic
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/tensorflow/elastic.py", line 24, in <module>
    from horovod.tensorflow.functions import broadcast_object, broadcast_object_fn, broadcast_variables
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/tensorflow/functions.py", line 24, in <module>
    from horovod.tensorflow.mpi_ops import allgather, broadcast
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/tensorflow/mpi_ops.py", line 46, in <module>
    MPI_LIB = _load_library('mpi_lib' + get_ext_suffix())
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/tensorflow/mpi_ops.py", line 42, in _load_library
    library = load_library.load_op_library(filename)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow_core/python/framework/load_library.py", line 57, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: libtensorflow_framework.so.1: cannot open shared object file: No such file or directory

Resolving patch:
diff --git a/examples/tensorflow2_mnist_ray.py b/examples/tensorflow2_mnist_ray.py
--- a/examples/tensorflow2_mnist_ray.py
+++ b/examples/tensorflow2_mnist_ray.py
@@ -1,6 +1,5 @@
 import tensorflow as tf
 import horovod.tensorflow.keras as hvd
-import horovod.keras as hvd
 
 import ray
 from horovod.ray import RayExecutor


Test patch: