opened
yselivonchyk
username_0
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet) mxnet
2. Framework version: 1.4.0
3. Horovod version: 0.16.1
4. MPI version: 4.0.1
5. CUDA version: N/A
6. NCCL version: 
7. Python version: 3.7
8. OS and version: 10.13.4

**Bug report:**
Please describe errorneous behavior you're observing and steps to reproduce it.

I run the 2 lines from the readme page:
```
import mxnet as mx
import horovod.mxnet as hvd
```
and it fails on the second line asking for mxnet.so:
```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-8-7a9ad6e7861b> in <module>
      1 import mxnet as mx
----> 2 import horovod.mxnet as hvd
      3 from mxnet import autograd
      4 
      5 # Initialize Horovod

~/env/dev/lib/python3.7/site-packages/horovod/mxnet/__init__.py in <module>
     23                 __file__, 'mpi_lib')
     24 
---> 25 from horovod.mxnet.mpi_ops import allgather
     26 from horovod.mxnet.mpi_ops import allreduce, allreduce_
     27 from horovod.mxnet.mpi_ops import broadcast, broadcast_

~/env/dev/lib/python3.7/site-packages/horovod/mxnet/mpi_ops.py in <module>
     27 from horovod.common import get_ext_suffix
     28 from horovod.common import HorovodBasics as _HorovodBasics
---> 29 _basics = _HorovodBasics(__file__, 'mpi_lib')
     30 
     31 # import basic methods

~/env/dev/lib/python3.7/site-packages/horovod/common/__init__.py in __init__(self, pkg_path, *args)
     54     def __init__(self, pkg_path, *args):
     55         full_path = get_extension_full_path(pkg_path, *args)
---> 56         self.MPI_LIB_CTYPES = ctypes.CDLL(full_path, mode=ctypes.RTLD_GLOBAL)
     57 
     58     def init(self, comm=None):

/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    354 
    355         if handle is None:
--> 356             self._handle = _dlopen(self._name, mode)
    357         else:
    358             self._handle = handle

OSError: dlopen(/Users/yauheni/env/dev/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-darwin.so, 10): Library not loaded: lib/libmxnet.so
  Referenced from: /Users/yauheni/env/dev/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-darwin.so
  Reason: image not found
```
I can run mxnet code, no problem. What can be the issue here?
Run horovod on local machine without GPU
issue
created
tgaddair
username_1
Hey @username_0, can you share the command you used to install Horovod and the installation log output?  Thanks.
comment
created
yselivonchyk
username_0
```
pip install horovod
Collecting horovod
  Using cached https://files.pythonhosted.org/packages/89/70/327e1ce9bee0fb8a879b98f8265fb7a41ae6d04a3ee019b2bafba8b66333/horovod-0.16.1.tar.gz
Requirement already satisfied: cffi>=1.4.0 in /Users/user/env/dev/lib/python3.7/site-packages (from horovod) (1.12.3)
Requirement already satisfied: cloudpickle in /Users/user/env/dev/lib/python3.7/site-packages (from horovod) (1.0.0)
Requirement already satisfied: psutil in /Users/user/env/dev/lib/python3.7/site-packages (from horovod) (5.6.2)
Requirement already satisfied: six in /Users/user/env/dev/lib/python3.7/site-packages (from horovod) (1.12.0)
Requirement already satisfied: pycparser in /Users/user/env/dev/lib/python3.7/site-packages (from cffi>=1.4.0->horovod) (2.19)
Building wheels for collected packages: horovod
  Building wheel for horovod (setup.py) ... done
  Stored in directory: /Users/user/Library/Caches/pip/wheels/e1/96/14/e2ed67d3e5cc5a5c5d3bbb681b0cffaab08790035939772f8e
Successfully built horovod
Installing collected packages: horovod
Successfully installed horovod-0.16.1
```
comment
created
alsrgv
username_2
cc @username_3 @apeforest any suggestions?
comment
created
yuxihu
username_3
@username_0 I think the path the libmxnet.so is not set correctly in your local machine. Try the following command:
```
install_name_tool -change "lib/libmxnet.so" "/Users/yauheni/env/dev/lib/python3.7/site-packages/mxnet/libmxnet.so" /Users/yauheni/env/dev/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-darwin.so
```
comment
created
yselivonchyk
username_0
Even so mxnet works on my system, I don't have an libmxnet.so file not where you mentioned nor in the virtualenv that I am using. And even "find / -name libmxnet.so" wasn't able to get it.
comment
created
yuxihu
username_3
@username_0 I think you first need to figure out where your mxnet (libmxnet.so) is installed. You can uninstall mxnet and install mxnet. You can usually see the path it is installed.

You can use the following command to check the path to libmxnet.so:

```
/Users/yauheni/env/dev/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-darwin.so
```

In my local environment, it was `lib/libmxnet.so`.

Then you can use `install_name_tool` to change the path to where your libmxnet.so locates.

`install_name_tool -change <cur_path> <new_path> horovod_binary`

And I do not think this is a bug. It has something to do with your local environment.
comment
created
yselivonchyk
username_0
You are probably right.

I actually have mxnet.so at the right place according to command you provided:
`lib/libmxnet.so (compatibility version 0.0.0, current version 0.0.0)`

Tried to setup a new virtualenv - no improvement.
comment
created
yuxihu
username_3
@username_0 can you post the path to your libmxnet.so file? And the output of running the following command?
```
otool -L <path-to-your-horovod-library>
```
comment
closed
yselivonchyk
username_0
issue
created
yselivonchyk
username_0
Pointing to libmxnet within my virtualenv solved the issue:
`install_name_tool -change "lib/libmxnet.so" "/Users/yauheni/env/dev/lib/python3.7/site-packages/mxnet/libmxnet.so" /Users/yauheni/env/dev/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-darwin.so
`

Thank you.
comment