My multi-GPU training OAR job keeps being killed

f10w: <p>Hello,</p>
<p>I encountered a very strange problem that repeatedly happened.</p>
<p>I have a training started with the following command on a Linux server:</p>
<pre><code class="lang-auto">oarsub -l "host=1/gpuid=4,walltime=480:0:0" \
"/home/username/.env/py37/bin/python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --coco_path /data/coco --output_dir /home/username/code/output --resume /home/username/code/output/checkpoint.pth"
</code></pre>
<p>After a few hours, the training was killed. And this happened every time I restarted it. Our system admin could not figure out what was wrong.</p>
<p>The std error messages (content of <code>OAR.&lt;jobID&gt;.stderr</code>) are the following:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/username/.local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/username/.local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/username/.env/py37/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in &lt;module&gt;
    main()
  File "/home/username/.env/py37/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/username/.env/py37/bin/python', '-u', 'main.py', '--coco_path', '/data/coco', '--output_dir', '/home/username/code/output', '--resume', '/home/username/code/output/checkpoint.pth']' died with &lt;Signals.SIGKILL: 9&gt;.
</code></pre>
<p>In the std output file <code>OAR.&lt;jobID&gt;.stdout</code>, the last lines are the following:</p>
<blockquote>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
<hr>
</blockquote>
<p>This message is displayed only at the end of <code>OAR.&lt;jobID&gt;.stdout</code>, when the crash happened, so maybe it has something to do with the crash.</p>
<p>Could you please help? Thank you very much in advance!</p>

mrshenli: <aside class="quote no-group" data-username="f10w" data-post="1" data-topic="88726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/f10w/48/1352_2.png" class="avatar"> f10w:</div>
<blockquote>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
</blockquote>
</aside>
<p>This is printed immediately after you run <code>launch.py</code>. See the code below:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/0fc0a9308ab1e8f6b65d2565c45f54afb449d438/torch/distributed/launch.py#L215-L222">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/0fc0a9308ab1e8f6b65d2565c45f54afb449d438/torch/distributed/launch.py#L215-L222" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/0fc0a9308ab1e8f6b65d2565c45f54afb449d438/torch/distributed/launch.py#L215-L222" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/0fc0a9308ab1e8f6b65d2565c45f54afb449d438/torch/distributed/launch.py#L215-L222</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="215" style="counter-reset: li-counter 214 ;">
          <li>if 'OMP_NUM_THREADS' not in os.environ and args.nproc_per_node &gt; 1:</li>
          <li>    current_env["OMP_NUM_THREADS"] = str(1)</li>
          <li>    print("*****************************************\n"</li>
          <li>          "Setting OMP_NUM_THREADS environment variable for each process "</li>
          <li>          "to be {} in default, to avoid your system being overloaded, "</li>
          <li>          "please further tune the variable for optimal performance in "</li>
          <li>          "your application as needed. \n"</li>
          <li>          "*****************************************".format(current_env["OMP_NUM_THREADS"]))</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>During the few hours when the job is running, did the <code>DistributedDataParallel</code> training making progress as expected? You can, e.g., print some logs in every iteration to check this.</p>
<p>And since the command line contains a path to checkpoint, I assume the training job would write into that checkpoint file periodically? Did the job successfully generated any checkpoint?</p>

f10w: <p>Thanks for your reply. The training made progress (and saved checkpoints) for a few hours before being killed.</p>
<p>Here’s the beginning of the log file (<code>head -n 20 OAR.&lt;jobID&gt;.stdout</code>):</p>
<blockquote>
<p>| distributed init (rank 2): env://<br>
| distributed init (rank 3): env://<br>
| distributed init (rank 0): env://<br>
| distributed init (rank 1): env://<br>
git:<br>
sha: ae03a2d6e52a9ec1b67f85437d0a275c5abbe9ac, status: has uncommited changes, branch: master</p>
<p>Namespace(aux_loss=True, backbone=‘resnet50’, batch_size=2, bbox_loss_coef=5, clip_max_norm=0.1, coco_panoptic_path=None, coco_path=‘/data/coco’, dataset_file=‘coco’, dec_layers=6, device=‘cuda’, dice_loss_coef=1, dilation=False, dim_feedforward=2048, dist_backend=‘nccl’, dist_url=‘env://’, distributed=True, dropout=0.1, enc_layers=6, eos_coef=0.1, epochs=300, eval=False, frozen_weights=None, giou_loss_coef=2, gpu=0, hidden_dim=256, lr=0.0001, lr_backbone=1e-05, lr_drop=200, mask_loss_coef=1, masks=False, nheads=8, num_queries=100, num_workers=2, output_dir=‘/home/username/code/output’, position_embedding=‘sine’, pre_norm=False, rank=0, remove_difficult=False, resume=‘/home/username/code/output/checkpoint.pth’, seed=42, set_cost_bbox=5, set_cost_class=1, set_cost_giou=2, start_epoch=0, weight_decay=0.0001, world_size=4)<br>
number of params: 41302368<br>
loading annotations into memory…<br>
Done (t=22.53s)<br>
creating index…<br>
index created!<br>
loading annotations into memory…<br>
Done (t=0.75s)<br>
creating index…<br>
index created!<br>
Start training<br>
Epoch: [23]  [    0/14786]  eta: 7:42:07  lr: 0.000100  class_error: 22.68  loss: 10.4300 (10.4300)  loss_bbox: 0.3688 (0.3688)  loss_bbox_0: 0.3812 (0.3812)  loss_bbox_1: 0.4038 (0.4038)  loss_bbox_2: 0.3718 (0.3718)  loss_bbox_3: 0.3781 (0.3781)  loss_bbox_4: 0.3690 (0.3690)  loss_ce: 0.5279 (0.5279)  loss_ce_0: 0.6643 (0.6643)  loss_ce_1: 0.5894 (0.5894)  loss_ce_2: 0.5849 (0.5849)  loss_ce_3: 0.5311 (0.5311)  loss_ce_4: 0.5083 (0.5083)  loss_giou: 0.8055 (0.8055)  loss_giou_0: 0.8359 (0.8359)  loss_giou_1: 0.7730 (0.7730)  loss_giou_2: 0.7711 (0.7711)  loss_giou_3: 0.7646 (0.7646)  loss_giou_4: 0.8013 (0.8013)  cardinality_error_unscaled: 8.8750 (8.8750)  cardinality_error_0_unscaled: 13.2500 (13.2500)  cardinality_error_1_unscaled: 12.7500 (12.7500)  cardinality_error_2_unscaled: 8.1250 (8.1250)  cardinality_error_3_unscaled: 8.1250 (8.1250)  cardinality_error_4_unscaled: 8.6250 (8.6250)  class_error_unscaled: 22.6786 (22.6786)  loss_bbox_unscaled: 0.0738 (0.0738)  loss_bbox_0_unscaled: 0.0762 (0.0762)  loss_bbox_1_unscaled: 0.0808 (0.0808)  loss_bbox_2_unscaled: 0.0744 (0.0744)  loss_bbox_3_unscaled: 0.0756 (0.0756)  loss_bbox_4_unscaled: 0.0738 (0.0738)  loss_ce_unscaled: 0.5279 (0.5279)  loss_ce_0_unscaled: 0.6643 (0.6643)  loss_ce_1_unscaled: 0.5894 (0.5894)  loss_ce_2_unscaled: 0.5849 (0.5849)  loss_ce_3_unscaled: 0.5311 (0.5311)  loss_ce_4_unscaled: 0.5083 (0.5083)  loss_giou_unscaled: 0.4027 (0.4027)  loss_giou_0_unscaled: 0.4180 (0.4180)  loss_giou_1_unscaled: 0.3865 (0.3865)  loss_giou_2_unscaled: 0.3855 (0.3855)  loss_giou_3_unscaled: 0.3823 (0.3823)  loss_giou_4_unscaled: 0.4006 (0.4006)  time: 1.8753  data: 0.4317  max mem: 2509<br>
Epoch: [23]  [   10/14786]  eta: 2:39:48  lr: 0.000100  class_error: 30.30  loss: 11.0897 (10.6174)  loss_bbox: 0.3473 (0.3555)  loss_bbox_0: 0.3888 (0.3989)  loss_bbox_1: 0.3834 (0.3796)  loss_bbox_2: 0.3662 (0.3772)  loss_bbox_3: 0.3590 (0.3603)  loss_bbox_4: 0.3520 (0.3548)  loss_ce: 0.5279 (0.5271)  loss_ce_0: 0.6043 (0.6137)  loss_ce_1: 0.5870 (0.5653)  loss_ce_2: 0.5627 (0.5542)  loss_ce_3: 0.5400 (0.5402)  loss_ce_4: 0.5083 (0.5214)  loss_giou: 0.8325 (0.8231)  loss_giou_0: 0.9057 (0.8922)  loss_giou_1: 0.8793 (0.8482)  loss_giou_2: 0.8800 (0.8514)  loss_giou_3: 0.8392 (0.8296)  loss_giou_4: 0.8540 (0.8247)  cardinality_error_unscaled: 9.1250 (10.3182)  cardinality_error_0_unscaled: 13.2500 (13.6477)  cardinality_error_1_unscaled: 12.7500 (12.3068)  cardinality_error_2_unscaled: 9.6250 (10.9091)  cardinality_error_3_unscaled: 9.1250 (10.1705)  cardinality_error_4_unscaled: 9.1250 (10.0341)  class_error_unscaled: 22.6786 (23.9301)  loss_bbox_unscaled: 0.0695 (0.0711)  loss_bbox_0_unscaled: 0.0778 (0.0798)  loss_bbox_1_unscaled: 0.0767 (0.0759)  loss_bbox_2_unscaled: 0.0732 (0.0754)  loss_bbox_3_unscaled: 0.0718 (0.0721)  loss_bbox_4_unscaled: 0.0704 (0.0710)  loss_ce_unscaled: 0.5279 (0.5271)  loss_ce_0_unscaled: 0.6043 (0.6137)  loss_ce_1_unscaled: 0.5870 (0.5653)  loss_ce_2_unscaled: 0.5627 (0.5542)  loss_ce_3_unscaled: 0.5400 (0.5402)  loss_ce_4_unscaled: 0.5083 (0.5214)  loss_giou_unscaled: 0.4162 (0.4116)  loss_giou_0_unscaled: 0.4528 (0.4461)  loss_giou_1_unscaled: 0.4397 (0.4241)  loss_giou_2_unscaled: 0.4400 (0.4257)  loss_giou_3_unscaled: 0.4196 (0.4148)  loss_giou_4_unscaled: 0.4270 (0.4123)  time: 0.6489  data: 0.0493  max mem: 3574</p>
</blockquote>
<p>And here’s the end (<code>tail -n 4 OAR.&lt;jobID&gt;.stdout</code>):</p>
<blockquote>
<p>Epoch: [23]  [ 4730/14786]  eta: 1:07:17  lr: 0.000100  class_error: 40.61  loss: 9.2283 (10.3881)  loss_bbox: 0.3610 (0.3562)  loss_bbox_0: 0.4111 (0.4107)  loss_bbox_1: 0.3788 (0.3752)  loss_bbox_2: 0.3783 (0.3661)  loss_bbox_3: 0.3680 (0.3598)  loss_bbox_4: 0.3660 (0.3571)  loss_ce: 0.4747 (0.5366)  loss_ce_0: 0.5628 (0.6116)  loss_ce_1: 0.5367 (0.5860)  loss_ce_2: 0.5133 (0.5601)  loss_ce_3: 0.4722 (0.5455)  loss_ce_4: 0.4595 (0.5364)  loss_giou: 0.7070 (0.7790)  loss_giou_0: 0.7854 (0.8533)  loss_giou_1: 0.7170 (0.8021)  loss_giou_2: 0.7175 (0.7903)  loss_giou_3: 0.7327 (0.7818)  loss_giou_4: 0.7180 (0.7802)  cardinality_error_unscaled: 7.7500 (9.0408)  cardinality_error_0_unscaled: 10.8750 (11.5247)  cardinality_error_1_unscaled: 10.1250 (11.1548)  cardinality_error_2_unscaled: 8.3750 (9.9196)  cardinality_error_3_unscaled: 7.3750 (9.3645)  cardinality_error_4_unscaled: 7.7500 (9.0276)  class_error_unscaled: 30.4464 (32.1511)  loss_bbox_unscaled: 0.0722 (0.0712)  loss_bbox_0_unscaled: 0.0822 (0.0821)  loss_bbox_1_unscaled: 0.0758 (0.0750)  loss_bbox_2_unscaled: 0.0757 (0.0732)  loss_bbox_3_unscaled: 0.0736 (0.0720)  loss_bbox_4_unscaled: 0.0732 (0.0714)  loss_ce_unscaled: 0.4747 (0.5366)  loss_ce_0_unscaled: 0.5628 (0.6116)  loss_ce_1_unscaled: 0.5367 (0.5860)  loss_ce_2_unscaled: 0.5133 (0.5601)  loss_ce_3_unscaled: 0.4722 (0.5455)  loss_ce_4_unscaled: 0.4595 (0.5364)  loss_giou_unscaled: 0.3535 (0.3895)  loss_giou_0_unscaled: 0.3927 (0.4267)  loss_giou_1_unscaled: 0.3585 (0.4011)  loss_giou_2_unscaled: 0.3587 (0.3952)  loss_giou_3_unscaled: 0.3664 (0.3909)  loss_giou_4_unscaled: 0.3590 (0.3901)  time: 0.4095  data: 0.0113  max mem: 7106</p>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
<hr>
</blockquote>
<p>The OMP message is at the very end of the log file, so it doesn’t seem to be printed right after launch.</p>
<p>Please let me know if you need further information. Thanks!</p>

mrshenli: <p>hmm, this is weird. One possibility could be the print buffer from the main process wasn’t full at the beginning and then was only flushed on exit, leading to the message to be shown in the end. But I am not sure if this is case.</p>
<p>One reason for this behavior might be some DDP process hits OOM (or some other error) after a while and crashed, causing other DDP processes to hang. Did you try to do any try-except around DDP in <code>main.py</code>? If so, you might want to try <a href="https://pytorch.org/elastic" rel="nofollow noopener">https://pytorch.org/elastic</a>, as try-except in one process could lead to DDP communication de-sync/hang/timeout.</p>

f10w: <p>In the code, the <code>print</code> function is tweaked to print on the master process only (and do nothing on the others), so the content of <code>OAR.&lt;jobID&gt;.stdout</code> comes from the master. The thing I’m not sure about is the file <code>OAR.&lt;jobID&gt;.stderr</code>. If there is an OOM error, on <em>any</em> process, then the error message should be added to <code>OAR.&lt;jobID&gt;.stderr</code>, right? Because this has nothing to do with the <code>print</code> function I guess.</p>

Jelly123456: <p>Hi,</p>
<p>I faced the same problem as you. This error usually caused by the compiling process of various libraries. I created a brand-new environment using Anaconda and train again. This error disappeared.</p>
<p>I hope this information could help you.</p>

JL_lee: <p>Yes, So clever you are, I solve the problem according to your answer, thank you.</p>