Distributed training failed without errors

username: Feywell
cooked: <p>I just change</p>
<blockquote>
<p><a href="https://github.com/ZPdesu/SEAN" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ZPdesu/SEAN: SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020, Oral)</a> to distributed training.</p>
</blockquote>
<p>And I get a weird error.<br>
It is no error，when I use one gpu.<br>
But, It will stop without printing errors when I use multi gpus.<br>
It will stop after Every 13 epoches by two gpus.</p>
<p>What could be the cause of such a problem？</p>
<p>my environments:<br>
ubuntu: 16.04<br>
gpu：nvidia-2080ti<br>
cuda: 10.1 / 10.2<br>
pytorch: 1.6.0 / 1.7.0<br>
nccl: 2.4.8 / 2.7.6<br>
python:3.6 / 3.7</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/feywell">@Feywell</a></p>
<p>By “change <a href="https://github.com/ZPdesu/SEAN" rel="nofollow noopener">https://github.com/ZPdesu/SEAN</a> distributed training”, which distributed training API are you referring to (e.g., DistributedDataParallel, c10d, RPC)?</p>
<p>Could you please share the code that uses distributed APIs?</p>

username: Feywell
cooked: <p>I just use DistributedDataParallel like this:</p>
<pre><code class="lang-auto">    if opt.distributed:
        cudnn.benchmark = True
        opt.device = "cuda"

        torch.cuda.set_device(opt.local_rank)
        torch.distributed.init_process_group(backend="nccl",
                                             init_method="env://") 
 
        synchronize()
</code></pre>
<p>And model :</p>
<pre><code class="lang-auto">         if opt.distributed:
            self.pix2pix_model = torch.nn.parallel.DistributedDataParallel(self.pix2pix_model,
                                                              device_ids=[opt.local_rank],
                                                              output_device=opt.local_rank,
                                                              find_unused_parameters=True)
            self.pix2pix_model_on_one_gpu = self.pix2pix_model.module
</code></pre>

username: mrshenli
cooked: <p>The initialization looks correct to me.</p>
<blockquote>
<p><code>self.pix2pix_model_on_one_gpu = self.pix2pix_model.module</code></p>
</blockquote>
<p>Question: why retrieving the local model from DDP model?</p>
<blockquote>
<p>It will stop after Every 13 epoches by two gpus.</p>
</blockquote>
<p>You mean the program crashes without any error message? How did you launch the two DDP processes?</p>

username: Feywell
cooked: <p>This line just be used to save model:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/ZPdesu/SEAN/blob/04c7536ff3fecd2d1a09c9ae046a1144636033a5/trainers/pix2pix_trainer.py#L23" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/ZPdesu/SEAN/blob/04c7536ff3fecd2d1a09c9ae046a1144636033a5/trainers/pix2pix_trainer.py#L23" target="_blank" rel="nofollow noopener">ZPdesu/SEAN/blob/04c7536ff3fecd2d1a09c9ae046a1144636033a5/trainers/pix2pix_trainer.py#L23</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="13" style="counter-reset: li-counter 12 ;">
<li>updates the weights of the network while reporting losses</li>
<li>and the latest visuals to visualize the progress in training.</li>
<li>"""</li>
<li>
</li><li>def __init__(self, opt):</li>
<li>    self.opt = opt</li>
<li>    self.pix2pix_model = Pix2PixModel(opt)</li>
<li>    if len(opt.gpu_ids) &gt; 0:</li>
<li>        self.pix2pix_model = DataParallelWithCallback(self.pix2pix_model,</li>
<li>                                                      device_ids=opt.gpu_ids)</li>
<li class="selected">        self.pix2pix_model_on_one_gpu = self.pix2pix_model.module</li>
<li>    else:</li>
<li>        self.pix2pix_model_on_one_gpu = self.pix2pix_model</li>
<li>
</li><li>    self.generated = None</li>
<li>    if opt.isTrain:</li>
<li>        self.optimizer_G, self.optimizer_D = \</li>
<li>            self.pix2pix_model_on_one_gpu.create_optimizers(opt)</li>
<li>        self.old_lr = opt.lr</li>
<li>
</li><li>def run_generator_one_step(self, data):</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The program will crash in differenct epoches by different number’s gpus without error message.<br>
But It is ok in one gpu.<br>
I use pytorch launch function:<br>
<code>python -m torch.distributed.launch --nproc_per_node=$NGPUS train.py</code></p>