MPI does not gracefully terminate

Archie_Nidhi: <p>So I have been following the code and tutorial on using pytorch to do distributed machine learning <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="noopener nofollow ugc">here</a>. I am able to run the code  (and it completes all the tasks) but my program does not terminate and I need to manually kill it using ctrl+C. The exact code is <a href="https://github.com/seba-1511/dist_tuto.pth/blob/gh-pages/train_dist.py" rel="noopener nofollow ugc">here</a></p>
<p>Right now after completing the task, it hangs after displaying the following warning messages</p>
<blockquote>
<pre><code>/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:406: UserWarning: For MPI backend, world_size (0) and rank (0) are ignored since they are assigned by the MPI runtime.
  "MPI runtime.".format(world_size, rank))
train_dist.py:72: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  return F.log_softmax(x)
/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:125: UserWarning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead
  warnings.warn("torch.distributed.reduce_op is deprecated, please use "
</code></pre>
</blockquote>
<p>I run the code using basic command</p>
<blockquote>
<pre><code> /anaconda3/bin/mpirun -np 3 --host node-0,node-1,node-2 python train_dist.py
</code></pre>
</blockquote>
<p>Do I need to add something in the code to exit gracefully?</p>

mrshenli: <aside class="quote no-group" data-username="Archie_Nidhi" data-post="1" data-topic="89655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/archie_nidhi/48/26847_2.png" class="avatar"> Archie_Nidhi:</div>
<blockquote>
<p>Do I need to add something in the code to exit gracefully?</p>
</blockquote>
</aside>
<p>This shouldn’t be necessary. Which MPI implementation are you using?</p>

mrshenli: <p>I tried to run <a href="https://github.com/seba-1511/dist_tuto.pth/blob/fb8293ae710279d580dc273da3702a4b21b1cf49/train_dist.py" rel="nofollow noopener">the code</a> with gloo backend to check if it is a MPI-only problem, but initially hits the following error:</p>
<pre><code class="lang-auto">...
    self._target(*self._args, **self._kwargs)
  File "test.py", line 132, in init_processes
    fn(rank, size)
  File "test.py", line 103, in run
    train_set, bsz = partition_dataset()
  ...
ValueError: batch_size should be a positive integer value, but got batch_size=64.0
</code></pre>
<p>After fixing that, hits the error below:</p>
<pre><code class="lang-auto">  File "test.py", line 132, in init_processes
    fn(rank, size)
  File "test.py", line 118, in run
    epoch_loss += loss.data[0]
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item&lt;T&gt;()` in C++ to convert a 0-dim tensor to a number
</code></pre>
<p>Can we have a min example that can reproduce the hang issue? Thanks!</p>

Archie_Nidhi: <p>I figured the issue. The mpi part was stuck because one of the processes were waiting for the other process to send some data.</p>

Archie_Nidhi: <p>Yeah, I resolved that issues by converting batch_size to integer and loss.data[0] to loss.data.item()</p>