Having "ChildFailedError"..?

ooodragon: <p>I was using my code without any problem until now.<br>
Tried running more than one process in a single machine.<br>
It causes error saying the port is already in use, and I solved by setting<br>
CUDA_VISIBLE_DEVICES=1,2 python3 -m torch.distributed.launch --nproc_per_node=2 --master_port=$RANDOM</p>
<p>As I’ve already mentioned, I had no problem… until now…</p>
<pre><code class="lang-auto">**********************************************************************             
               CHILD PROCESS FAILED WITH NO ERROR_FILE                                          
**********************************************************************                         
CHILD PROCESS FAILED WITH NO ERROR_FILE                                                                          
Child process 12190 (local_rank 0) FAILED (exitcode 1)                                          
Error msg: Process failed with exitcode 1                                                                  
Without writing an error file to &lt;N/A&gt;.                                                         
While this DOES NOT affect the correctness of your application,                                
no trace information about the error will be available for inspection.                                           
Consider decorating your top level entrypoint function with                                     
torch.distributed.elastic.multiprocessing.errors.record. Example:                 
                                                                                               
  from torch.distributed.elastic.multiprocessing.errors import record                          
                                                                                                                                                                                                                                                                            
  @record                                                                                                                                                                                                                                                                  
  def trainer_main(args):                                                                                                                                                                                                                                                   
     # do train                                                                                                                                                                                                                                                             
**********************************************************************                                                                                                                                                                                                      
  warnings.warn(_no_error_file_warning_msg(rank, failure))                                                                                                                                                                                                                  
Traceback (most recent call last):                                                                                                                                                                                                                                          
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main                                                                                                                                                                                                
    return _run_code(code, main_globals, None,                                                                                                                                                                                                                              
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code                                                                                                                                                                                                           
    exec(code, run_globals)                                                                                                                                                                                                                                                 
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 173, in &lt;module&gt;                                                                                                                                                                          
    main()                                                                                                                                                                                                                                                                  
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 169, in main                                                                                                                                                                              
    run(args)                                                                                                                                                                                                                                                               
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 621, in run                                                                                                                                                                                  
    elastic_launch(                                                                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in __call__                                                                                         
    return launch_agent(self._config, self._entrypoint, list(args))                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper           
    return f(*args, **kwargs)                                                                                                                
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(                              
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:                
***************************************                                                       
            train.py FAILED                                                                    
=======================================                                                                                                                                                                 
Root Cause:                                                                                                   
[0]:                                                                                                        
  time: 2021-08-16_10:44:53        
  rank: 0 (local_rank: 0)                                                                                     
  exitcode: 1 (pid: 12190)
  error_file: &lt;N/A&gt;
  msg: "Process failed with exitcode 1"
=======================================
Other Failures:
[1]:
  time: 2021-08-16_10:44:53
  rank: 1 (local_rank: 1)
  exitcode: 1 (pid: 12191)
  error_file: &lt;N/A&gt;
  msg: "Process failed with exitcode 1"
***************************************
</code></pre>
<p>I am running my code again with that “record” decorator on my train function as the error told me to. (but should I also add that to validation function…?)</p>
<p>Not sure if this will fix the error but I don’t see many threads having same error as mine…</p>

cbalioglu: <p>Hi,</p>
<p>Were you able to run your training code with more than one process before?</p>
<p>The <code>record</code> decorator should be applied to the <em>entrypoint</em> of your script (e.g. <code>def main(args)</code>). It is not meant for fixing things, but for dumping the Python stack trace of your failed subprocess(es) to the log output so you can have more information about the problem.</p>
<p>Right now there is no chance for us to suggest anything since your log output does not contain any meaningful information to root cause the issue.</p>
<p>Also besides the <code>record</code> decorator, you can also the new <code>torch.distributed.run</code> script in place of <code>torch.distributed.launch</code>, and set its <code>--log-dir</code>, <code>--redirects</code>, and <code>--tee</code> options to dump the stdout/stderr of your worker processes to a file. You can learn more about our new launcher script <a href="https://pytorch.org/docs/stable/elastic/run.html" rel="noopener nofollow ugc">here</a>.</p>

ooodragon: <p>Hi thx for your reply!<br>
Yes I was able to run multiple processes with using random ports without problem…<br>
I dont know what might be the error…</p>

cbalioglu: <p>Ok. I would still recommend giving <code>torch.distributed.run</code> a try and see what log output you get for worker processes.</p>

ooodragon: <p>Oh right<br>
Yes I will try that<br>
I did run processes again without that … strangly it works now… so  i have no idea what is wrong… <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

hasansalimkanmaz: <p>I have experienced the same issue tonight while training on a cluster in kubernetes. The exit code was 6 for me instead of 1. However, it is fixed again strangely after retrying this morning.</p>

Zhang_Kin: <p>I met this error today, is because I using the cluster which limits the RAM. When I asked for more RAM, it works well now. In Slurm Job submission System.</p>

mohit_dulani: <p>What exactly you did <a class="mention" href="/u/ooodragon">@ooodragon</a> to overcome this code ??<br>
Please share the code if possible !!<br>
<a class="mention" href="/u/cbalioglu">@cbalioglu</a> Please help</p>

pvtien96: <p>I got a similar error but with <code>exitcode: -11</code></p>
<pre><code class="lang-auto">[2023-08-05 22:18:08 internimage_t_1k_224](main.py 307): INFO Start training
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 33003) of binary: /home/van-tien.pham/anaconda3/bin/python
Traceback (most recent call last):
  File "/home/van-tien.pham/anaconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/van-tien.pham/anaconda3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/van-tien.pham/anaconda3/lib/python3.9/site-packages/torch/distributed/launch.py", line 196, in &lt;module&gt;
    main()
  File "/home/van-tien.pham/anaconda3/lib/python3.9/site-packages/torch/distributed/launch.py", line 192, in main
    launch(args)
  File "/home/van-tien.pham/anaconda3/lib/python3.9/site-packages/torch/distributed/launch.py", line 177, in launch
    run(args)
  File "/home/van-tien.pham/anaconda3/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/van-tien.pham/anaconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/van-tien.pham/anaconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
main.py FAILED
-------------------------------------------------------
Failures:
  &lt;NO_OTHER_FAILURES&gt;
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-08-05_22:18:15
  host      : siim-3.lis-lab.fr
  rank      : 0 (local_rank: 0)
  exitcode  : -11 (pid: 33003)
  error_file: &lt;N/A&gt;
  traceback : Signal 11 (SIGSEGV) received by PID 33003
=======================================================
</code></pre>

djaym7: <p>Same -11 error, any solution ?</p>

Wassim_Jaoui: <p>I had <code>exiterror=1</code><br>
I found out that I was running my code in an uncorrect environment, I had defined everything in anaconda before.<br>
<code>conda activate nameEnvironment</code><br>
to install GPU with pytorch:<br>
<code>conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia</code><br>
instead of <code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code><br>
it resolved the problem for me.<br>
Get the suitable installation command here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/get-started/locally/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">Start Locally</a></h3>

  <p>Start Locally</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>