DDP Error: torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers

wachhu: <p>Hi,<br>
Since last week I upgraded pytorch to 1.10.0, this error occurred. Here’s the log:</p>
<blockquote>
<p>Blockquote<br>
WARNING:torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15342 closing signal SIGHUP<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15343 closing signal SIGHUP<br>
Traceback (most recent call last):<br>
File “/home/anaconda3/envs/aa/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/anaconda3/envs/aa/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 193, in <br>
main()<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 189, in main<br>
launch(args)<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 174, in launch<br>
run(args)<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/run.py”, line 710, in run<br>
elastic_launch(<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 252, in launch_agent<br>
result = agent.run()<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py”, line 125, in wrapper<br>
result = f(*args, **kwargs)<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py”, line 709, in run<br>
result = self._invoke_run(role)<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py”, line 843, in _invoke_run<br>
time.sleep(monitor_interval)<br>
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py”, line 60, in _terminate_process_handler<br>
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)<br>
torch.distributed.elastic.multiprocessing.api.SignalException: Process 15265 got signal: 1</p>
</blockquote>

mrshenli: <p>Hey <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a>, is there any BC-breaking change in 1.10?</p>

Kiuk_Chung: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> No BC issue that I know of yet. Looks like in this case the worker (not the agent) died due to a SIGHUP. Could you share the command you are running, how you are running it (from a terminal, on a job scheduler, etc) and the script you are running?</p>

wachhu: <p><a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> I use shell script from a terminal, and the commad is just like :</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=6,7 python -m torch.distributed.launch --nproc_per_node 2 --master_port=26500  training.py --arg1 --arg2
</code></pre>
<p>This error always appeared after 10 epochs, .</p>

Kiuk_Chung: <p>If the job terminates with a SIGHUP mid-execution then there’s something else other than <code>torch.distributed.launch</code> that is causing the job to fail (typically <code>torch.distributed.lauch</code> issues happen on startup not mid-execution).</p>
<p>Since your trainers died with a signal (SIGHUP) which is typically sent when the terminal is closed, you’ll have to dig through the log (console) output to see what the actual exception was (or where the program was before it got killed). I recommend you register this faulthandler to get the trace information : <a href="https://docs.python.org/3/library/faulthandler.html" class="inline-onebox" rel="noopener nofollow ugc">faulthandler — Dump the Python traceback — Python 3.10.0 documentation</a></p>

dabs: <p>Did you find a solution? I am running into the same problem.</p>

zhihe_lu: <p>How do you submit job? I met the same problem when using nohup command (affected by terminal shutting down?). Now, I am trying to use screen command.</p>

Ziyu_Huang: <p>I am using 1.10 and I also face this problem…</p>

wachhu: <p>I just gave up using ddp <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"></p>

wachhu: <p>I also use the nohup command and I’m sure it’s not about terminal shutting down.</p>

pritamdamania87: <p><a class="mention" href="/u/wachhu">@wachhu</a> Do you have a simple repro for this issue? It would help us narrow down the issue further.</p>
<p><a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> Was wondering if there is a parent/controlling process involved here in our framework which might exit unexpectedly causing this issue?</p>

Kiuk_Chung: <p>Its hard to tell what the root cause was from the provided excerpt of the logs. I need the full logs. But from this line:</p>
<blockquote>
<p>WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15342 closing signal SIGHUP<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15343 closing signal SIGHUP</p>
</blockquote>
<p>what is probably happening is that the launcher process (the one that is running <code>torch.distributed.launch</code> got a <code>SIGHUP</code>. To make sure the worker PIDs are not orphaned, torchelastic will forward any signals the launcher process received down to the worker process (code here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/elastic/agent/server/api.py#L716" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/distributed/elastic/agent/server/api.py#L716</a>).</p>
<p>In this case PIDs 15342 and 15343 are the worker PIDs that get sent a SIGHUP because the launcher PID (the parent) got a SIGHUP.</p>
<p>If the launcher is wrapped with nohup:</p>
<pre><code class="lang-auto">$ nohup python -m torch.distributed.launch ...
</code></pre>
<p>Then theoretically the SIGHUP shoud’ve been ignored and not passed-through to the launcher PID, but from the logs its clear that’s not what happened. FWIW here are the signals that are passed-through to the worker PIDs from the elastic agent: SIGTERM, SIGKILL, SIGHUP, SIGQUIT (for unix) and SIGTERM, SIGKILL (for windows). See: <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/elastic/multiprocessing/api.py#L233-L237" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/distributed/elastic/multiprocessing/api.py#L233-L237</a></p>

Kiuk_Chung: <p>So I was able to repro this with nohup. Currently <code>torch.distributed.launch</code>, <code>torchrun</code>, <code>torch.distributed.run</code> will all not work with nohup since we register our own termination handler for SIGHUP, which overrides the ignore handler by nohup. If you need to close the terminal but keep the process running, then you can use tmux or screen.</p>

kings-rgb: <p>If you are using the server terminal to run the nohup ddp-program, <code>torch.distributed.launch</code>, <code>torchrun</code>, <code>torch.distributed.run</code>, you need to use the <code>exit</code> command to exit instead of closing the terminal directly. Nohup has a little bug when multi-process.</p>

Sunny_Sanyal: <p>I am also having the same issue. any solutions for this?</p>

yuchang: <p>hi,i met the same problem, do you have any solutions now?</p>

duo_ma: <p>This solution is effective. Thanks a lot.</p>

Ke_Jiang: <p>I have the same problem. any idea of this?</p>

nahidalam: <p>Anyone looking into this in future, this is how I was able to solve it with nohup</p>
<pre><code class="lang-auto">nohup your_command &gt; output.log 2&gt;&amp;1 &amp;
</code></pre>
<p>for example in this case</p>
<pre><code class="lang-auto">nohup python -m torch.distributed.launch --nproc_per_node 4 --master_port 9527 train.py ... &gt; output.log 2&gt;&amp;1 &amp;
</code></pre>
<p>This should output you something like</p>
<pre><code class="lang-auto">[1] 232423
</code></pre>
<p>Here <code>1</code> is jobid and the big number is PID. Now you have to use the jobid with the <code>disown</code> command</p>
<pre><code class="lang-auto">disown %1
</code></pre>
<p>This seems to be working for me so far</p>

drscotthawley: <p>As a regular user of <code>nohup</code> I was surprised to find this thread after receiving this error. I’ve never heard of another process, e.g. <code>torch</code>, taking over <code>nohup</code>’s no-hang-up functionality.</p>
<p><code>tmux</code> and <code>screen</code> were suggested above, but without any guidance for us re. how to use them in a similar way to <code>nohup</code>.   e.g., <code>tmux</code> creates a whole new “session” one has to manage somehow; this is not like <code>nohup</code>.</p>
<p>So, to whom it may concern, the following is the same bash function presented in original <code>nohup</code> form and new-improved <code>tmux</code> form – though I confess it was ChatGPT who provided the latter:</p>
<pre data-code-wrap="bash"><code class="lang-bash"># for launching jobs
#   usage: launchrun &lt;gpus&gt; &lt;logfile&gt; &lt;command and all args&gt;
function launchrun() {
    CUDA_VISIBLE_DEVICES="$1" nohup "${@:3}" &gt; "$2" 2&gt;&amp;1 &amp;
}
</code></pre>
<pre data-code-wrap="bash"><code class="lang-bash"># usage: launchrun &lt;gpus&gt; &lt;logfile&gt; &lt;command and all args&gt;
function launchrun() {
    gpus="$1"
    logfile="$2"
    shift 2  # Remove the first two arguments (gpus and logfile)

    # Generate a semi-unique session ID using date
    session_id="session_$(date +"%Y%m%d%H%M%S")"

    # Create a tmux session, run the command inside it, and detach
    tmux new-session -d -s "$session_id" "CUDA_VISIBLE_DEVICES=$gpus $* &gt; $logfile 2&gt;&amp;1"

    # Optionally, you can also rename the window (replace "mywindow" with your desired name)
    # tmux rename-window -t "$session_id:0" mywindow

    # Detach from the session
    tmux detach-client

    echo "Session ID: $session_id"
}
</code></pre>
<p>I’d be interested in receiving comments or improvements to the above. Thanks!</p>