Repository: horovod/horovod

Issue:
SegFault when using horovod.mxnet.allgather
**Environment:**
1. Framework: MXNet
2. Framework version: 1.5.0
3. Horovod version: 0.18.1
4. MPI version: 4.0.1
5. CUDA version: 10.0
6. NCCL version: 2.3.7
7. Python version: 3.6.8
8. OS and version: Linux 4.14.138-89.102.amzn1.x86_64 (EC2 instance on AWS)
9. GCC version: 4.8.5

**Checklist:**
1. Did you search issues to find if somebody asked this question before? Yes
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)? N/A
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)? N/A
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)? N/A

**Bug report:**
Please describe errorneous behavior you're observing and steps to reproduce it.

The function `horovod.mxnet.allgather` will lead to `Segmentation fault: 11` when I use more than 1 worker. The minimal code block to reproduce the error is as below:
```
import horovod.mxnet as hvd
import mxnet as mx

hvd.init()

ctx = mx.gpu(hvd.local_rank())

v = mx.nd.ones((100, 50), ctx=ctx)

r = hvd.allgather(v)

print(r.shape)
print(r)
```

Running it with the command `horovodrun -np 2 -H localhost:2 python3 test_allgather.py`  will result in stack traces pasted below:

```
[1,1]<stderr>:
[1,1]<stderr>:Segmentation fault: 11
[1,1]<stderr>:
[1,0]<stderr>:
[1,0]<stderr>:Segmentation fault: 11
[1,0]<stderr>:
[1,1]<stderr>:Stack trace:
[1,1]<stderr>:  [bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e6b160) [0x7f5c59089160]
[1,1]<stderr>:  [bt] (1) /lib64/libc.so.6(+0x362f0) [0x7f5d3593b2f0]
[1,1]<stderr>:  [bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(void mxnet::CopyFromToImpl<mshadow::cpu, mshadow::gpu>(mxnet::NDArray const&, mxnet::NDArray const&, mxnet::RunContext, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&)+0x119) [0x7f5c589f7399]
[1,1]<stderr>:  [bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x27d9895) [0x7f5c589f7895]
[1,1]<stderr>:  [bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c1ce1) [0x7f5c587dfce1]
[1,1]<stderr>:  [bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c80c0) [0x7f5c587e60c0]
[1,1]<stderr>:  [bt] (6) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c8356) [0x7f5c587e6356]
[1,1]<stderr>:  [bt] (7) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c0444) [0x7f5c587de444]
[1,1]<stderr>:  [bt] (8) /usr/lib64/libstdc++.so.6(+0xbb890) [0x7f5d29649890]
[1,0]<stderr>:Stack trace:
[1,0]<stderr>:  [bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e6b160) [0x7fd67db0b160]
[1,0]<stderr>:  [bt] (1) /lib64/libc.so.6(+0x362f0) [0x7fd75a3bd2f0]
[1,0]<stderr>:  [bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(void mxnet::CopyFromToImpl<mshadow::cpu, mshadow::gpu>(mxnet::NDArray const&, mxnet::NDArray const&, mxnet::RunContext, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&)+0x119) [0x7fd67d479399]
[1,0]<stderr>:  [bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x27d9895) [0x7fd67d479895]
[1,0]<stderr>:  [bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c1ce1) [0x7fd67d261ce1]
[1,0]<stderr>:  [bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c80c0) [0x7fd67d2680c0]
[1,0]<stderr>:  [bt] (6) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c8356) [0x7fd67d268356]
[1,0]<stderr>:  [bt] (7) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c0444) [0x7fd67d260444]
[1,0]<stderr>:  [bt] (8) /usr/lib64/libstdc++.so.6(+0xbb890) [0x7fd74f242890]
[1,0]<stderr>:
[1,0]<stderr>:Segmentation fault: 11
[1,0]<stderr>:
[1,1]<stderr>:[ip-172-31-20-44:29671] *** Process received signal ***
[1,1]<stderr>:[ip-172-31-20-44:29671] Signal: Aborted (6)
[1,1]<stderr>:[ip-172-31-20-44:29671] Signal code:  (-6)
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 0] /lib64/libpthread.so.0(+0xf5e0)[0x7f5d363ea5e0]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 1] [1,1]<stderr>:/lib64/libc.so.6(gsignal+0x37)[0x7f5d3593b277]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 2] [1,1]<stderr>:/lib64/libc.so.6(abort+0x148)[0x7f5d3593c968]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 3] [1,1]<stderr>:/lib64/libc.so.6(+0x78d97)[0x7f5d3597dd97]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 4] [1,1]<stderr>:/lib64/libc.so.6(+0x814f9)[0x7f5d359864f9]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 5] [1,1]<stderr>:/usr/lib64/python3.6/dist-packages/horovod/mxnet/mpi_lib.cpython-36m-x86_64-linux-gnu.so(_ZN7horovod6common11NCCLContext8ShutDownEv+0xf6)[0x7f5c174dd346]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 6] [1,1]<stderr>:/usr/lib64/python3.6/dist-packages/horovod/mxnet/mpi_lib.cpython-36m-x86_64-linux-gnu.so(+0x53f38)[0x7f5c1748bf38]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 7] [1,1]<stderr>:/usr/lib64/libstdc++.so.6(+0xbb890)[0x7f5d29649890]
[1,1]<stderr>:[ip-172-31-20-44:29671] [ 8] /lib64/libpthread.so.0(+0x7de5)[0x7f5d363e2de5]
[1,1]<stderr>:[ip-172-31-20-44:29671] [1,1]<stderr>:[ 9] [1,1]<stderr>:/lib64/libc.so.6(clone+0x6d)[0x7f5d35a02f1d]
[1,1]<stderr>:[ip-172-31-20-44:29671] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 0 on node ip-172-31-20-44 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
```

mxnet allgather is borken. it crashed when using GPU and produces wrong results when using CPU
**Environment:**
1. Framework: MXNet
2. Framework version: 1.5.1
3. Horovod version: built from source
4. MPI version: 3.1.1
5. CUDA version: 10.2
6. NCCL version: 2.5.6
7. Python version: 3.6.9
8. OS and version: Linux Ubuntu 18.04.3
9. GCC version: 7.4.0

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)? Yes
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?

**Bug report:**
Following PR #1639, `allgather()` in mxnet is still borken. It produces incorrect results when using CPU and crashes when using GPUs


I installed horovod from source:
```bash
git clone --recurse-submodules -j8 https://github.com/horovod/horovod.git
cd horovod
export HOROVOD_GPU_ALLREDUCE=NCCL
export HOROVOD_NCCL_INCLUDE=/usr/include
export HOROVOD_NCCL_LIB=/usr/lib/x86_64-linux-gnu
export HOROVOD_NCCL_LINK=SHARED
export HOROVOD_WITHOUT_PYTORCH=1
export HOROVOD_WITHOUT_TENSORFLOW=1
export HOROVOD_WITH_MXNET=1
export HOROVOD_WITH_MPI=1
ln -s /usr/local/cuda/lib64/stubs/libcuda.so ./libcuda.so.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD
python setup.py install
```

Then ran the following sample code:

```python
# CUDA_VISIBLE_DEVICES=0,1 mpiexec -n 2 --allow-run-as-root python all_gather_bug.py
import mxnet as mx
from mxnet import ndarray
import horovod.mxnet as hvd
from mpi4py import MPI

use_gpu = False
use_mpi4py = False

if use_mpi4py:
    comm = MPI.COMM_WORLD
    hvd.init(comm=comm)
    local_rank = comm.Get_rank()
else:
    hvd.init()
    local_rank = hvd.local_rank()

ctx = mx.gpu(local_rank) if use_gpu else mx.cpu(local_rank)
mx.random.seed(local_rank)

a = ndarray.random.uniform(shape=[10, 100, 100], ctx=ctx)
print("(before) a.shape = {}".format(a.shape))
print("(before) a[0]={}".format(a[0]))

a = hvd.allgather(a)
print("(after) a.shape = {}".format(a.shape))
print("(after) a[0]={}".format(a[0]))

mx.nd.waitall()
hvd.shutdown()
```



When using CPU, it looks like rank 0 is overwritting other ranks (ie broadcast instead of allgather):
```
(before) a.shape = (10, 100, 100)
(before) a.shape = (10, 100, 100)
(before) a[0]=
[[5.4881352e-01 5.9284461e-01 7.1518934e-01 ... 6.0783064e-01
  4.1702199e-01 9.9718481e-01]
 [7.2032452e-01 9.3255734e-01 1.1438108e-04 ... 1.8508208e-01
  2.5926229e-02 9.3154085e-01]
 [5.4966247e-01 9.4773060e-01 4.3532240e-01 ... 8.3994907e-01
  2.9090473e-01 1.2132858e-01]
 ...
 [2.0300540e-01 3.4745708e-01 6.7227858e-01 ... 9.6727759e-01
  4.6623814e-01 1.4771296e-01]
 [5.4340494e-01 6.7115563e-01 2.7836940e-01 ... 1.1400783e-02
  5.1639861e-01 5.5254018e-01]
 [5.7066756e-01 9.5268434e-01 2.8474227e-02 ... 3.9755744e-01
  6.7598689e-01 4.6260124e-01]]
<NDArray 100x100 @cpu(0)>
(after) a.shape = (10, 100, 100)
(before) a[0]=
[[0.52383333 0.05501367 0.03996297 ... 0.57466674 0.86663705 0.89201903]
 [0.26314485 0.8478666  0.13140848 ... 0.47916418 0.26825976 0.62353116]
 [0.60529524 0.56331843 0.08485638 ... 0.32175666 0.5547923  0.8856785 ]
 ...
 [0.8128833  0.50981677 0.89495724 ... 0.37793323 0.36698005 0.35633445]
 [0.12747145 0.10071229 0.580568   ... 0.38354123 0.4238108  0.21727636]
 [0.54174274 0.33404106 0.6885549  ... 0.9920475  0.7385572  0.32574102]]
<NDArray 100x100 @cpu(1)>
(after) a.shape = (10, 100, 100)
(after) a[0]=
[[5.4881352e-01 5.9284461e-01 7.1518934e-01 ... 6.0783064e-01
  4.1702199e-01 9.9718481e-01]
 [7.2032452e-01 9.3255734e-01 1.1438108e-04 ... 1.8508208e-01
  2.5926229e-02 9.3154085e-01]
 [5.4966247e-01 9.4773060e-01 4.3532240e-01 ... 8.3994907e-01
  2.9090473e-01 1.2132858e-01]
 ...
 [2.0300540e-01 3.4745708e-01 6.7227858e-01 ... 9.6727759e-01
  4.6623814e-01 1.4771296e-01]
 [5.4340494e-01 6.7115563e-01 2.7836940e-01 ... 1.1400783e-02
  5.1639861e-01 5.5254018e-01]
 [5.7066756e-01 9.5268434e-01 2.8474227e-02 ... 3.9755744e-01
  6.7598689e-01 4.6260124e-01]]
<NDArray 100x100 @cpu(0)>
(after) a[0]=
[[5.4881352e-01 5.9284461e-01 7.1518934e-01 ... 6.0783064e-01
  4.1702199e-01 9.9718481e-01]
 [7.2032452e-01 9.3255734e-01 1.1438108e-04 ... 1.8508208e-01
  2.5926229e-02 9.3154085e-01]
 [5.4966247e-01 9.4773060e-01 4.3532240e-01 ... 8.3994907e-01
  2.9090473e-01 1.2132858e-01]
 ...
 [2.0300540e-01 3.4745708e-01 6.7227858e-01 ... 9.6727759e-01
  4.6623814e-01 1.4771296e-01]
 [5.4340494e-01 6.7115563e-01 2.7836940e-01 ... 1.1400783e-02
  5.1639861e-01 5.5254018e-01]
 [5.7066756e-01 9.5268434e-01 2.8474227e-02 ... 3.9755744e-01
  6.7598689e-01 4.6260124e-01]]
<NDArray 100x100 @cpu(1)>
```


When using GPUs the operation fails:
```
(before) a.shape = (10, 100, 100)
(before) a[0]=
[[0.6686509  0.17409194 0.3850025  ... 0.43011498 0.0661214  0.2502998 ]
 [0.7005292  0.19000232 0.6673837  ... 0.27718288 0.16084558 0.223108  ]
 [0.96042585 0.81086403 0.54152083 ... 0.5650488  0.5196334  0.6767488 ]
 ...
 [0.96879214 0.9387428  0.04036242 ... 0.13176239 0.3436321  0.47154343]
 [0.8069018  0.91234195 0.01141495 ... 0.35816687 0.57390726 0.68393874]
 [0.72049534 0.67948174 0.44702923 ... 0.87448525 0.63809574 0.7006303 ]]
<NDArray 100x100 @gpu(0)>
(after) a.shape = (10, 100, 100)
(before) a.shape = (10, 100, 100)
(before) a[0]=
[[0.7685592  0.10232276 0.8685353  ... 0.93769354 0.62144864 0.21535844]
 [0.85973674 0.3420865  0.6202223  ... 0.5464046  0.41442537 0.32170743]
 [0.11786121 0.23281038 0.95843846 ... 0.17739207 0.5901362  0.28355032]
 ...
 [0.70749086 0.61171615 0.37854642 ... 0.3485002  0.29636437 0.7359518 ]
 [0.4345038  0.90834665 0.5242443  ... 0.0793817  0.40161872 0.6579807 ]
 [0.8531954  0.18177992 0.7053579  ... 0.5257004  0.24457276 0.74836564]]
<NDArray 100x100 @gpu(1)>
(after) a.shape = (10, 100, 100)
Traceback (most recent call last):
  File "all_gather_bug.py", line 29, in <module>
    print("(after) a[0]={}".format(a[0]))
  File "/opt/mxnet/python/mxnet/ndarray/ndarray.py", line 194, in __repr__
    return '\n%s\n<%s %s @%s>' % (str(self.asnumpy()),
  File "/opt/mxnet/python/mxnet/ndarray/ndarray.py", line 1996, in asnumpy
    ctypes.c_size_t(data.size)))
  File "/opt/mxnet/python/mxnet/base.py", line 252, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: MPI_Allgatherv failed, see MPI output for details.
Traceback (most recent call last):
  File "all_gather_bug.py", line 29, in <module>
    print("(after) a[0]={}".format(a[0]))
  File "/opt/mxnet/python/mxnet/ndarray/ndarray.py", line 194, in __repr__
    return '\n%s\n<%s %s @%s>' % (str(self.asnumpy()),
  File "/opt/mxnet/python/mxnet/ndarray/ndarray.py", line 1996, in asnumpy
    ctypes.c_size_t(data.size)))
  File "/opt/mxnet/python/mxnet/base.py", line 252, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: MPI_Allgatherv failed, see MPI output for details.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[59002,1],1]
  Exit code:    1
```




Discussion:
Hey @yuxihu, could you take a look at this MXNet issue when you get a chance?
@yjxiong The allgather function is not well tested in mxnet-horovod. Currently the mxnet-horovod integration uses allreduce and broadcast functions. Any specific use case that you want to use allgather?
Many use cases in distributed training require Allgather, also AllScatter. For example, to gather output from all workers into each worker for model parallelism and testing. Really hoping to see this get fixed. 
There is a known issue with MXNet allgather. https://github.com/horovod/horovod/issues/1409

We are currently working to resolve it. Will keep you posted. Thanks! 
I believe I found the issue.
The operation receive buffer is defined to have the same size as the send buffer (instead of the sum of sizes from all ranks):
https://github.com/horovod/horovod/blob/master/horovod/mxnet/mpi_ops.py#L155
This obviously results in truncated results on CPU and illegal memory access in CUDA.

I'm working on a PR.


Thanks. That'll be really great. I would love to review when it's ready. 
@ahmadki Do you need any help with this PR? Thanks!
There is another report of this issue (https://github.com/horovod/horovod/issues/1519). It would be great if you could provide an ETA for this fix. Thanks!
Sorry for going  AWOL. Incorrect output tensor shape was only the symptom, and the problem is much deeper than I originally though. I got tied up with other things at work and I won't be able to take a second crack at a solution before next week. A summary of my findings so far:

The allgather output nd-tensor is defined [here](https://github.com/horovod/horovod/blob/master/horovod/mxnet/mpi_ops.py#L155), This by it self is not an issue since MPI wrappers *should* reshape and reallocate the output memory buffers as needed. 

However, because `hvd.allgather()` is async, the original output tensor is returned immediately without modifications. We would expect subsequent calls to the output tensor to be blocking until it is re-evaluated but this is not the case.

The output tensor buffer is shallow copied [here](https://github.com/horovod/horovod/blob/master/horovod/mxnet/mpi_ops.cc#L127) and changes made to it by [AllocateOutput](https://github.com/horovod/horovod/blob/master/horovod/common/ops/collective_operations.h#L91) are not reflected on the original tensor.

So a solution (probably) should include the creation of a null tensor using ctypes to keep memory ownership at the python level ? and removing the shallow copies.
Any updates here ? Got the same problem recently.

Resolving patch:
diff --git a/horovod/mxnet/mpi_ops.py b/horovod/mxnet/mpi_ops.py
--- a/horovod/mxnet/mpi_ops.py
+++ b/horovod/mxnet/mpi_ops.py
@@ -1,4 +1,5 @@
 # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# Modifications copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -148,7 +149,9 @@ def allgather(tensor, name=None, priority=0):
         first dimensions of the tensors in different Horovod processes.
     """
     assert(isinstance(tensor, mx.nd.NDArray))
-    output = mx.nd.zeros(shape=tensor.shape, ctx=tensor.context,
+    # Size of output is unknown, create output array that
+    # will be resized during Horovod operation
+    output = mx.nd.empty(shape=[1], ctx=tensor.context,
                          dtype=tensor.dtype)
     c_in = tensor.handle
     c_out = output.handle
@@ -158,6 +161,9 @@ def allgather(tensor, name=None, priority=0):
     else:
         check_call(MPI_MXNET_LIB_CTYPES.horovod_mxnet_allgather_async(
             c_in, c_out, name, ctypes.c_int(priority)))
+
+    # Need to block here so changes to output tensor are visible
+    output.wait_to_read()
     return output
 
 


Test patch:
diff --git a/test/test_mxnet.py b/test/test_mxnet.py
--- a/test/test_mxnet.py
+++ b/test/test_mxnet.py
@@ -498,6 +498,112 @@ def test_horovod_broadcast_deferred_init_parameters(self):
             assert same(tensor.asnumpy(), root_tensor.asnumpy()), \
                 'horovod did not broadcast deferred initialized parameter correctly'
 
+    def test_horovod_allgather(self):
+        """Test that the allgather correctly gathers 1D, 2D, 3D tensors."""
+        hvd.init()
+        rank = hvd.rank()
+        size = hvd.size()
+
+        dtypes = ['int32',   'int64',
+                  'float32', 'float64']
+        dims = [1, 2, 3]
+        ctx = self._current_context()
+        for dtype, dim in itertools.product(dtypes, dims):
+            tensor = mx.ndarray.ones(shape=[17] * dim, dtype=dtype, ctx=ctx) * rank
+            gathered = hvd.allgather(tensor)
+
+            assert list(gathered.shape) == [17 * size] + [17] * (dim - 1)
+
+            for i in range(size):
+                rank_tensor = gathered[i * 17:(i + 1) * 17]
+                assert list(rank_tensor.shape) == [17] * dim, \
+                    'hvd.allgather produces incorrect gathered shape'
+                assert rank_tensor.min() == i, 'hvd.allgather produces incorrect gathered tensor'
+                assert rank_tensor.max() == i, 'hvd.allgather produces incorrect gathered tensor'
+
+    def test_horovod_allgather_variable_size(self):
+        """Test that the allgather correctly gathers 1D, 2D, 3D tensors,
+        even if those tensors have different sizes along the first dim."""
+        hvd.init()
+        rank = hvd.rank()
+        size = hvd.size()
+
+        dtypes = ['int32',   'int64',
+                  'float32', 'float64']
+        dims = [1, 2, 3]
+        ctx = self._current_context()
+        for dtype, dim in itertools.product(dtypes, dims):
+            # Support tests up to MPI Size of 35
+            if size > 35:
+                break
+
+            tensor_sizes = [17, 32, 81, 12, 15, 23, 22] * 5
+            tensor_sizes = tensor_sizes[:size]
+
+            tensor = mx.ndarray.ones(
+                shape=[tensor_sizes[rank]] + [17] * (dim - 1), dtype=dtype, ctx=ctx) * rank
+
+            gathered = hvd.allgather(tensor)
+
+            expected_size = sum(tensor_sizes)
+            assert list(gathered.shape) == [expected_size] + [17] * (dim - 1)
+
+            for i in range(size):
+                rank_size = [tensor_sizes[i]] + [17] * (dim - 1)
+                rank_tensor = gathered[sum(
+                    tensor_sizes[:i]):sum(tensor_sizes[:i + 1])]
+                assert list(rank_tensor.shape) == rank_size
+                assert rank_tensor.min() == i
+                assert rank_tensor.max() == i
+
+    def test_horovod_allgather_error(self):
+        """Test that the allgather returns an error if any dimension besides
+        the first is different among the tensors being gathered."""
+        hvd.init()
+        rank = hvd.rank()
+        size = hvd.size()
+
+        # This test does not apply if there is only one worker.
+        if size == 1:
+            self.skipTest("Only one worker available")
+
+        ctx = self._current_context()
+
+        tensor_size = [17] * 3
+        tensor_size[1] = 10 * (rank + 1)
+        tensor = mx.ndarray.ones(shape=tensor_size, ctx=ctx)
+
+        try:
+            hvd.allgather(tensor)
+            assert False, 'hvd.allgather did not throw error'
+        except (MXNetError, RuntimeError):
+            pass
+
+    def test_horovod_allgather_type_error(self):
+        """Test that the allgather returns an error if the types being gathered
+        differ among the processes"""
+        hvd.init()
+        rank = hvd.rank()
+        size = hvd.size()
+
+        # This test does not apply if there is only one worker.
+        if size == 1:
+            self.skipTest("Only one worker available")
+
+        ctx = self._current_context()
+
+        tensor_size = [17] * 3
+        if rank % 2 == 0:
+            tensor = mx.ndarray.ones(shape=tensor_size, dtype="int32", ctx=ctx)
+        else:
+            tensor = mx.ndarray.ones(shape=tensor_size, dtype="float32", ctx=ctx)
+
+        try:
+            hvd.allgather(tensor)
+            assert False, 'hvd.allgather did not throw error'
+        except (MXNetError, RuntimeError):
+            pass
+
     @unittest.skipUnless(has_gpu, "no gpu detected")
     def test_gluon_trainer(self):
         """Test using horovod allreduce in MXNet Gluon trainer."""