`terminate called after throwing an instance of 'std::system_error'`

sid: <p>I am experimenting with some point-to-point distributed communication while training. This appeared after several epochs:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::system_error'
   what():  Invalid argument
</code></pre>
<p>Any thoughts on what might have caused this?</p>

albanD: <p>Could you give more context on what your are doing when this happens, and if possible a small code sample that reproduce the issue please?</p>

sid: <p>I’m experimenting with a synchronous version of gossip-based SGD described in this paper on the MNIST dataset: <a href="https://arxiv.org/pdf/1611.04581.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1611.04581.pdf</a></p>
<p>For this, I’m using point-to-point communication with the TCP backend, which involves frequent transfer of parameters between GPU &lt;=&gt; CPU. I hadn’t encountered this issue with my implementation of all-reduce based SGD, using <code>torch.distributed.all_reduce</code> either with TCP or with Gloo, and is why I suspect this is because of point-to-point communication (<code>isend</code>/<code>irecv</code>).</p>
<p>Regarding environment, I’m running this with Python 3.6 / CUDA-8 / Ubuntu 16.04 on a GCE <code>n1-standard-4</code> that has 4 cores / 15GB RAM / 32GB SSD / one K-80.</p>
<p>I’ll try reproducing this with more granular logging, and will provide a code sample once I can narrow it down.</p>
<p>Meanwhile, I’m curious if this has been encountered before and under what circumstances.</p>

sid: <p>Here’s a snippet showing what happen’s with “gossip”-based parameter aggregation:</p>
<pre><code class="lang-python">for param in model.parameters():
    # A container to hold async requests
    # ---- MARKER A ----
    requests = []

    # We've to exchange params with peers
    to_send = param.data
    if args.gpu:
        to_send = to_send.cpu()

    receive_buffers = [
        (peer, torch.zeros(param.data.shape))
        for peer in peers
    ]

    for peer, buffer in receive_buffers:
        requests.append(dist.isend(
            tensor=to_send,
            dst=peer
        ))
        requests.append(dist.irecv(
            tensor=buffer,
            src=peer
        ))

    # Wait for all the requests to complete
    for r in requests:
        r.wait()
    # ---- MARKER B ----

    # Then compute the average
    for _, buffer in receive_buffers:
        if args.gpu:
            buffer = buffer.cuda(device_id)
        param.data += buffer
    param.data /= len(receive_buffers) + 1

    # ---- MARKER C ----
</code></pre>
<p>This is called after every <code>x</code> updates to the parameters. Based on logs, the error seems to occur between <code>---- MARKER A ----</code> and <code>---- MARKER B ----</code> 
consistently
. (Edit: I just observed this happen between  <code>---- MARKER B ----</code> and <code>---- MARKER C ----</code>)</p>
<p>If you’re curious about the model architecture (this is for the MNIST dataset):</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.l1 = nn.Linear(28 * 28, 1024)
        self.l2 = nn.Linear(1024, 1024)
        self.l3 = nn.Linear(1024, 1024)
        self.out = nn.Linear(1024, 10)
        for name, param in self.named_parameters():
            if 'weight' in name:
                nn.init.kaiming_normal(param)

    def forward(self, x):
        # input
        x = x.view(-1, 28 * 28)
        # reshaped
        x = F.dropout(x, p=0.2, training=self.training)
        x = self.l1(x)
        x = F.relu(x)
        # l1
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.l2(x)
        x = F.relu(x)
        # l2
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.l3(x)
        x = F.relu(x)
        # l3
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.out(x)
        x = F.log_softmax(x, dim=1)
        # output
        return x
</code></pre>
<p>Please let me know if you’d like more information.</p>

sid: <p>I created a gist that should help reproduce this, it generally occurs within the 10th epoch.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/anonymous/a5b4282a46be7e2971baf690e8cde054" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/anonymous/a5b4282a46be7e2971baf690e8cde054" target="_blank" rel="nofollow noopener">https://gist.github.com/anonymous/a5b4282a46be7e2971baf690e8cde054</a></h4>
<h5>invalid_argument_error.py</h5>
<pre><code class="Python">import os
import torch
import torch.distributed as dist
from torch import multiprocessing, nn, randperm
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torch.utils.data.sampler import SubsetRandomSampler
from torch.autograd import Variable
from torch.optim import SGD
from torchvision import datasets, transforms</code></pre>
This file has been truncated. <a href="https://gist.github.com/anonymous/a5b4282a46be7e2971baf690e8cde054" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


sid: <p>The more I look at the logs, the more this seems related to CPU &lt;=&gt; GPU communication, and mostly incidental that it’s around the same places that distributed communication occurs.</p>

sid: <p>Curious if there are any thoughts on this, suggestions for alternatives to pursue, etc.</p>

albanD: <p>Hi,</p>
<p>I don’t have a full python 3.5 install here sorry.<br>
I guess to debug this, I would remove features one by one until it does not crash anymore. Possibly forcing some cuda synchronization before doing the distributed communications?</p>

jpcenteno: <p>Curious if you were ever able to figure this out. I’m getting your error and an additional one when using Nvidia’s APEX library for mixed precision: <code>'std::out_of_range' what(): vector::_M_range_check: __n (which is 671091489) &gt;= this-&gt;size() (which is 3)</code>. I’m training a 3d network for segmentation in pytorch and due to GPU memory constraints, I am placing all activation computations in GPU1 and loss computation in CPU. GPU0 takes care of the rest.</p>

ptrblck: <p>I’m not sure, what is causing this error, but I would recommend to try out native amp as described <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">here</a> without the need to build apex.</p>

sid: <p>I found no solution to my issue. The workaround was to checkpoint aggressively and resume from checkpoint.</p>

jpcenteno: <p><a class="mention" href="/u/sid">@sid</a>, yes, that is what I am doing as well… Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, the implementation of amp into pytorch works really well. It almost seems faster… I’m not sure if the error mentioned above will go away, but again, pytorch’s amp new feature is nice to have.</p>

ani0075: <p>This might be related.</p>
<p>While running main_moco.py in <a href="https://github.com/facebookresearch/moco" rel="nofollow noopener">https://github.com/facebookresearch/moco</a>, I keep getting this after a few epochs.</p>
<pre><code class="lang-python">terminate called after throwing an instance of 'std::system_error'
  what():  Transport endpoint is not connected
/home/aniruddha/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 32 leaked semaphores to clean up at shutdown
  len(cache))
Traceback (most recent call last):
  File "main_moco_cln.py", line 591, in &lt;module&gt;
    main()
  File "main_moco_cln.py", line 197, in main
    mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "/home/aniruddha/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/aniruddha/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 107, in join
    (error_index, name)
Exception: process 0 terminated with signal SIGABRT
/home/aniruddha/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 32 leaked semaphores to clean up at shutdown
  len(cache))
/home/aniruddha/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 32 leaked semaphores to clean up at shutdown
  len(cache))
(base) aniruddha:moco$ /home/aniruddha/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 32 leaked semaphores to clean u
p at shutdown
  len(cache))

</code></pre>
<p><a href="https://github.com/facebookresearch/moco/issues/25" rel="nofollow noopener">https://github.com/facebookresearch/moco/issues/25</a> says “Transport endpoint is not connected means your network is probably unreliable.”</p>
<p><a class="mention" href="/u/sid">@sid</a> <a class="mention" href="/u/jpcenteno">@jpcenteno</a> any thoughts? Is this because of instability in my network communication?</p>
<p>Using Python 3.7 and Pytorch 1.4.0</p>

collinmccarthy: <p>I’m getting this same error using different models in Detectron2 (DeepLab and now MaskFormer). Does anyone have a solution to this other than checkpointing and re-trying? Maybe a different communication backend or something?</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::system_error'
what():  Transport endpoint is not connected
Traceback (most recent call last):                                                                 
File "/train_net.py", line 264, in &lt;module&gt;                          
    launch(                                                                                        
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/detectron2/engine/launch.py", line 67, in launch                                                                           mp.spawn(                                                                                      
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn                                                                    
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')                   
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes                                                              while not context.join():                                                                      
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 130, in join                                                                     
    raise ProcessExitedException(                                                                
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGABRT
</code></pre>

veast: <p>hello Aniruddha, I met the same problem, have you resolved or made progress?</p>

Antonia_Calvi: <p>If you found out I’d like to hear too!</p>