Torch distributed data parallel child fail error, exit code 2

vsiddons: <p>I am attempting to run a program on a slurm cluster of 4 gpus. I am attempting to fine-tune LLaVa using QLoRA.  Here is my bash script:</p>
<pre><code class="lang-auto">#!/bin/bash

#SBATCH -J llava_fine_tuning
#SBATCH -p gpu
#SBATCH -o output.txt
#SBATCH -e errors.err
#SBATCH --mail-type=fail
#SBATCH --mail-user=myname@address
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=4
#SBATCH --time=00:20:00
#SBATCH --mem=159G
#SBATCH -A 000000

#Get IP of random node
#https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/slurm/sbatch_run.sh
nodes=( $( scontrol show hostnames $SLURM_JOB_NODELIST ) )
nodes_array=($nodes)
head_node=${nodes_array[0]}
head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address)

echo Node IP: $head_node_ip
export LOGLEVEL=INFO

#Load any modules that your program needs
module load python/gpu/3.11.5
pip install transformers
pip install bitsandbytes
pip install peft

open_port=$(python path/find_open_port.py) #uses socket to bind to port 0

echo "Using port: $open_port"

#Run your program
torchrun \
    --nnodes 1 \
    --nproc_per_node 4 \
    --rdzv_id $RANDOM \
    --rdzv_backend c10d \
    --rdzv_endpoint $head_node_ip:$open_port \
    /path/Main.py
</code></pre>
<p>I understand that this error can have many causes, so I have attempted to use the torch.distributed.elastic.multiprocessing.errors decorator to get a traceback, but the information is not helpful to discovering the root of the problem for me.<br>
The error begins before I even assign a huggingface model. Here is my Main.py file:</p>
<pre><code class="lang-auto">from TrainLLaVa import TrainLLaVa
from torch.distributed.elastic.multiprocessing.errors import record

@record
def main():
    llava_trainer = TrainLLaVa("llava-hf/llava-1.5-7b-hf", "llava-hf/llava-1.5-7b-hf", True, True)
    print(llava_trainer.train("llava-hf/llava-1.5-7b-hf_ft", 0.00005, 4, 6))

if __name__=="__main__":
   main()
</code></pre>
<p>Here is the errors.err file which includes the traceback:</p>
<pre><code class="lang-auto">W0729 10:24:16.323000 139737339278528 torch/distributed/run.py:779] 
W0729 10:24:16.323000 139737339278528 torch/distributed/run.py:779] *****************************************
W0729 10:24:16.323000 139737339278528 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0729 10:24:16.323000 139737339278528 torch/distributed/run.py:779] *****************************************
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189] Starting elastic_operator with launch configs:
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   entrypoint       : 10.253.0.223:47125
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   min_nodes        : 1
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   max_nodes        : 1
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   nproc_per_node   : 4
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   run_id           : 21189
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   rdzv_backend     : c10d
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   rdzv_endpoint    : 10.253.2.221
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   rdzv_configs     : {'timeout': 900}
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   max_restarts     : 0
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   monitor_interval : 0.1
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   log_dir          : /tmp/torchelastic_mv2lwx44
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189]   metrics_cfg      : {}
I0729 10:24:16.326000 139737339278528 torch/distributed/launcher/api.py:189] 
I0729 10:24:16.347000 139737339278528 torch/distributed/elastic/agent/server/api.py:825] [default] starting workers for entrypoint: python3.11
I0729 10:24:16.349000 139737339278528 torch/distributed/elastic/agent/server/api.py:646] [default] Rendezvous'ing worker group
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512] [default] Rendezvous complete for workers. Result:
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   restart_count=0
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   master_addr=x1000c0s6b0n0.hsn0.cm.bigred200.uits.iu.edu
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   master_port=60523
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   group_rank=0
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   group_world_size=1
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   local_ranks=[0, 1, 2, 3]
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   role_ranks=[0, 1, 2, 3]
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   global_ranks=[0, 1, 2, 3]
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   role_world_sizes=[4, 4, 4, 4]
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512]   global_world_sizes=[4, 4, 4, 4]
I0729 10:24:16.765000 139737339278528 torch/distributed/elastic/agent/server/api.py:512] 
I0729 10:24:16.767000 139737339278528 torch/distributed/elastic/agent/server/api.py:654] [default] Starting worker group
I0729 10:24:16.769000 139737339278528 torch/distributed/elastic/agent/server/local_elastic_agent.py:185] Environment variable 'TORCHELASTIC_ENABLE_FILE_TIMER' not found. Do not start FileTimerServer.
I0729 10:24:16.771000 139737339278528 torch/distributed/elastic/agent/server/local_elastic_agent.py:217] Environment variable 'TORCHELASTIC_HEALTH_CHECK_PORT' not found. Do not start health check.
/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/10.253.0.223:47125': [Errno 2] No such file or directory
/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/10.253.0.223:47125': [Errno 2] No such file or directory
/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/BigRed200/10.253.0.223:47125': [Errno 2] No such file or directory
/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/BigRed200/10.253.0.223:47125': [Errno 2] No such file or directory
E0729 10:24:16.877000 139737339278528 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 2) local_rank: 0 (pid: 110038) of binary: /N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11
I0729 10:24:16.928000 139737339278528 torch/distributed/elastic/multiprocessing/errors/__init__.py:361] ('local_rank %s FAILED with no error file. Decorate your entrypoint fn with @record for traceback info. See: https://pytorch.org/docs/stable/elastic/errors.html', 0)
Traceback (most recent call last):
  File "/N/soft/sles15/deeplearning/Python-3.11.5/bin/torchrun", line 8, in &lt;module&gt;
    sys.exit(main())
             ^^^^^^
  File "/[path]/.local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/[path]/.local/lib/python3.11/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/[path]/.local/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/[path]/.local/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[path]/.local/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
10.253.0.223:47125 FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-29_10:24:16
  host      : [address]
  rank      : 1 (local_rank: 1)
  exitcode  : 2 (pid: 110039)
  error_file: &lt;N/A&gt;
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-29_10:24:16
  host      : [address]
  rank      : 2 (local_rank: 2)
  exitcode  : 2 (pid: 110040)
  error_file: &lt;N/A&gt;
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-29_10:24:16
  host      : [address]
  rank      : 3 (local_rank: 3)
  exitcode  : 2 (pid: 110041)
  error_file: &lt;N/A&gt;
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-29_10:24:16
  host      : [address]
  rank      : 0 (local_rank: 0)
  exitcode  : 2 (pid: 110038)
  error_file: &lt;N/A&gt;
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
</code></pre>
<p>How can I get <span class="mention">@record</span> to work (if it is not working, which it doesn’t seem to be) and what could be the cause of this error? How can I fix it?</p>

H-Huang: <p>It looks like the error is coming from:</p>
<pre><code class="lang-auto">/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/10.253.0.223:47125': [Errno 2] No such file or directory
/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/10.253.0.223:47125': [Errno 2] No such file or directory
/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/BigRed200/10.253.0.223:47125': [Errno 2] No such file or directory
/N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11: can't open file '/[path]/BigRed200/10.253.0.223:47125': [Errno 2] No such file or directory
E0729 10:24:16.877000 139737339278528 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 2) local_rank: 0 (pid: 110038) of binary: /N/soft/sles15/deeplearning/Python-3.11.5/bin/python3.11
</code></pre>
<p>Are you using NFS? Is this accessible by the machine accessing it?</p>

vsiddons: <p>Thank you for responding.</p>
<p>The snippet you posted seems to be completely blank.</p>

vsiddons: <p>I typed the command <code>service nfs-kernel-server status</code> and my result was <code>service: no such service nfs-kernel-server</code>, so I am assuming that I do not have NFS.</p>
<p>I do not understand the meaning of your second question.</p>