Keep getting ChildFailedError Error in distributed setup

IdoAmit198: <p>Hi everyone, and thanks in advance for any help.</p>
<p>I’m running a slightly modified <a href="https://github.com/huggingface/transformers/blob/v4.24.0/examples/pytorch/language-modeling/run_clm.py" rel="noopener nofollow ugc">run_clm.py script</a> with vary number of A100 GPUs (4-8) on a single node, and keep getting the ChildFailedError right after the training/evaluation ends.<br>
I train/evaluate GPT2 (smallest model) on the <a href="https://huggingface.co/datasets/openwebtext" rel="noopener nofollow ugc">OpenWebText dataset</a>.</p>
<h3><a name="an-example-how-i-run-my-evaluation-code-from-a-shell-script-is-as-follow-1" class="anchor" href="#an-example-how-i-run-my-evaluation-code-from-a-shell-script-is-as-follow-1"></a>An example how I run my evaluation code from a shell script is as follow:</h3>
<blockquote>
<p>GPU=1,2,3,4,5<br>
export TORCH_CPP_LOG_LEVEL=INFO NCCL_DEBUG=INFO<br>
export CUDA_VISIBLE_DEVICES=$GPU</p>
<p>torchrun <br>
–standalone <br>
–nnodes=1 <br>
–nproc_per_node=${NUM_GPU} <br>
run_clm.py <br>
–model_name_or_path ${MODEL} <br>
–dataset_name ${DS_NAME} <br>
–preprocessing_num_workers 16 <br>
–logging_steps 5000 <br>
–save_steps ${SAVE_STEPS} <br>
–do_eval <br>
–per_device_eval_batch_size ${EVAL_BATCH} <br>
–seed ${RANDOM} <br>
–evaluation_strategy steps <br>
–logging_dir ${OUTPUT_DIR} <br>
–output_dir ${OUTPUT_DIR} <br>
–overwrite_output_dir <br>
–ddp_timeout 324000 \</p>
</blockquote>
<h3><a name="and-getting-the-following-error-2" class="anchor" href="#and-getting-the-following-error-2"></a>And getting the following error:</h3>
<blockquote>
<p>100%|██████████| 2209/2209 [39:03&lt;00:00,  1.06s/it]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4041 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4042 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4043 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4045 closing signal SIGTERM<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 3 (pid: 4044) of binary: /venv/bin/python3<br>
Traceback (most recent call last):<br>
File “/venv/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 345, in wrapper<br>
return f(*args, **kwargs)<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/run.py”, line 719, in main<br>
run(args)<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/run.py”, line 710, in run<br>
elastic_launch(<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 259, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:<br>
============================================================<br>
./code/gpt2/Model-Compression-Research-Package/examples/transformers/language-modeling/run_clm.py FAILED<br>
Failures:<br>
&lt;NO_OTHER_FAILURES&gt;<br>
Root Cause (first observed failure):<br>
[0]:<br>
time      : 2022-12-21_21:57:29<br>
host      : december-ds-2h4b6-5hpkj<br>
rank      : 3 (local_rank: 3)<br>
exitcode  : -9 (pid: 4044)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 9 (SIGKILL) received by PID 4044<br>
============================================================</p>
</blockquote>
<h3><a name="full-log-after-truncating-unnecessary-logs-of-tqdm-etc-3" class="anchor" href="#full-log-after-truncating-unnecessary-logs-of-tqdm-etc-3"></a>Full log (after truncating unnecessary logs of tqdm, etc):</h3>
<blockquote>
<p>12/21/2022 20:37:39 - WARNING - <strong>main</strong> - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/21/2022 20:37:39 - WARNING - <strong>main</strong> - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/21/2022 20:37:39 - WARNING - <strong>main</strong> - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/21/2022 20:37:39 - INFO - <strong>main</strong> - Training/evaluation parameters TrainingArguments(<br>
_n_gpu=1,<br>
adafactor=False,<br>
adam_beta1=0.9,<br>
adam_beta2=0.999,<br>
adam_epsilon=1e-08,<br>
auto_find_batch_size=False,<br>
bf16=False,<br>
bf16_full_eval=False,<br>
data_seed=None,<br>
dataloader_drop_last=False,<br>
dataloader_num_workers=0,<br>
dataloader_pin_memory=True,<br>
ddp_bucket_cap_mb=None,<br>
ddp_find_unused_parameters=None,<br>
ddp_timeout=324000,<br>
debug=<span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
deepspeed=None,<br>
disable_tqdm=False,<br>
do_eval=True,<br>
do_predict=False,<br>
do_train=False,<br>
eval_accumulation_steps=None,<br>
eval_delay=0,<br>
eval_steps=5000,<br>
evaluation_strategy=steps,<br>
fp16=False,<br>
fp16_backend=auto,<br>
fp16_full_eval=False,<br>
fp16_opt_level=O1,<br>
fsdp=<span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
fsdp_min_num_params=0,<br>
fsdp_transformer_layer_cls_to_wrap=None,<br>
full_determinism=False,<br>
gradient_accumulation_steps=1,<br>
gradient_checkpointing=False,<br>
greater_is_better=None,<br>
group_by_length=False,<br>
half_precision_backend=auto,<br>
hub_model_id=None,<br>
hub_private_repo=False,<br>
hub_strategy=every_save,<br>
hub_token=&lt;HUB_TOKEN&gt;,<br>
ignore_data_skip=False,<br>
include_inputs_for_metrics=False,<br>
jit_mode_eval=False,<br>
label_names=None,<br>
label_smoothing_factor=0.0,<br>
learning_rate=5e-05,<br>
length_column_name=length,<br>
load_best_model_at_end=False,<br>
local_rank=0,<br>
log_level=passive,<br>
log_level_replica=passive,<br>
log_on_each_node=True,<br>
logging_dir=.cache/results/GPT2_Compression/baseline_results/OpenWebText/test_saved_data_eval,<br>
logging_first_step=False,<br>
logging_nan_inf_filter=True,<br>
logging_steps=5000,<br>
logging_strategy=steps,<br>
lr_scheduler_type=linear,<br>
max_grad_norm=1.0,<br>
max_steps=-1,<br>
metric_for_best_model=None,<br>
mp_parameters=,<br>
no_cuda=False,<br>
num_train_epochs=3.0,<br>
optim=adamw_hf,<br>
optim_args=None,<br>
output_dir=.cache/results/GPT2_Compression/baseline_results/OpenWebText/test_saved_data_eval,<br>
overwrite_output_dir=True,<br>
past_index=-1,<br>
per_device_eval_batch_size=8,<br>
per_device_train_batch_size=8,<br>
prediction_loss_only=False,<br>
push_to_hub=False,<br>
push_to_hub_model_id=None,<br>
push_to_hub_organization=None,<br>
push_to_hub_token=&lt;PUSH_TO_HUB_TOKEN&gt;,<br>
ray_scope=last,<br>
remove_unused_columns=True,<br>
report_to=[‘wandb’],<br>
resume_from_checkpoint=None,<br>
run_name=openwebtext_inference,<br>
save_on_each_node=False,<br>
save_steps=1000,<br>
save_strategy=steps,<br>
save_total_limit=None,<br>
seed=6298,<br>
sharded_ddp=<span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
skip_memory_metrics=True,<br>
tf32=None,<br>
torchdynamo=None,<br>
tpu_metrics_debug=False,<br>
tpu_num_cores=None,<br>
use_ipex=False,<br>
use_legacy_prediction_loop=False,<br>
use_mps_device=False,<br>
warmup_ratio=0.0,<br>
warmup_steps=0,<br>
weight_decay=0.0,<br>
xpu_backend=None,<br>
)<br>
12/21/2022 20:37:39 - WARNING - <strong>main</strong> - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/21/2022 20:37:39 - WARNING - <strong>main</strong> - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False<br>
[INFO|configuration_utils.py:654] 2022-12-21 20:37:40,050 &gt;&gt; loading configuration file config.json from cache at /store/.cache/huggingface/hub/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/config.json<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-21 20:37:41,351 &gt;&gt; loading file vocab.json from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/vocab.json<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-21 20:37:41,351 &gt;&gt; loading file merges.txt from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/merges.txt<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-21 20:37:41,351 &gt;&gt; loading file tokenizer.json from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/tokenizer.json<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-21 20:37:41,351 &gt;&gt; loading file added_tokens.json from cache at None<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-21 20:37:41,351 &gt;&gt; loading file special_tokens_map.json from cache at None<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-21 20:37:41,351 &gt;&gt; loading file tokenizer_config.json from cache at None<br>
[INFO|configuration_utils.py:654] 2022-12-21 20:37:41,351 &gt;&gt; loading configuration file config.json from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/config.json<br>
[INFO|modeling_utils.py:2204] 2022-12-21 20:37:44,384 &gt;&gt; loading weights file pytorch_model.bin from cache at /store/.cache/huggingface/hub/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/pytorch_model.bin<br>
[INFO|modeling_utils.py:2708] 2022-12-21 20:37:50,669 &gt;&gt; All model checkpoint weights were used when initializing GPT2LMHeadModel.<br>
======================================================<br>
[INFO|modeling_utils.py:2716] 2022-12-21 20:37:50,669 &gt;&gt; All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.<br>
If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.<br>
december-ds-2h4b6-5hpkj:4041:4041 [0] NCCL INFO Bootstrap : Using eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4041:4041 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
======================================================<br>
december-ds-2h4b6-5hpkj:4041:4041 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
december-ds-2h4b6-5hpkj:4041:4041 [0] NCCL INFO NET/Socket : Using [0]eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4041:4041 [0] NCCL INFO Using network Socket<br>
NCCL version 2.10.3+cuda11.3<br>
december-ds-2h4b6-5hpkj:4042:4042 [1] NCCL INFO Bootstrap : Using eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4044:4044 [3] NCCL INFO Bootstrap : Using eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4043:4043 [2] NCCL INFO Bootstrap : Using eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4045:4045 [4] NCCL INFO Bootstrap : Using eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4042:4042 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
======================================================<br>
december-ds-2h4b6-5hpkj:4042:4042 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
december-ds-2h4b6-5hpkj:4044:4044 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
december-ds-2h4b6-5hpkj:4042:4042 [1] NCCL INFO NET/Socket : Using [0]eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4042:4042 [1] NCCL INFO Using network Socket<br>
december-ds-2h4b6-5hpkj:4043:4043 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
december-ds-2h4b6-5hpkj:4045:4045 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
======================================================<br>
december-ds-2h4b6-5hpkj:4044:4044 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
======================================================<br>
december-ds-2h4b6-5hpkj:4045:4045 [4] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
======================================================<br>
december-ds-2h4b6-5hpkj:4043:4043 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
december-ds-2h4b6-5hpkj:4043:4043 [2] NCCL INFO NET/Socket : Using [0]eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4043:4043 [2] NCCL INFO Using network Socket<br>
december-ds-2h4b6-5hpkj:4044:4044 [3] NCCL INFO NET/Socket : Using [0]eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4045:4045 [4] NCCL INFO NET/Socket : Using [0]eth0:10.42.92.136&lt;0&gt;<br>
december-ds-2h4b6-5hpkj:4044:4044 [3] NCCL INFO Using network Socket<br>
december-ds-2h4b6-5hpkj:4045:4045 [4] NCCL INFO Using network Socket<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Trees [0] 4/-1/-1-&gt;3-&gt;2 [1] 4/-1/-1-&gt;3-&gt;2<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Setting affinity for GPU 4 to ffff,fff00000,00ffffff,f0000000<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Trees [0] -1/-1/-1-&gt;4-&gt;3 [1] -1/-1/-1-&gt;4-&gt;3<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Channel 00/02 :    0   1   2   3   4<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Channel 01/02 :    0   1   2   3   4<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Setting affinity for GPU 5 to ffff,fff00000,00ffffff,f0000000<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Setting affinity for GPU 1 to 0fffff,ff000000,0fffffff<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0 [1] 2/-1/-1-&gt;1-&gt;0<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Setting affinity for GPU 2 to 0fffff,ff000000,0fffffff<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Trees [0] 3/-1/-1-&gt;2-&gt;1 [1] 3/-1/-1-&gt;2-&gt;1<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Setting affinity for GPU 3 to 0fffff,ff000000,0fffffff<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Channel 00 : 2[3e000] → 3[88000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Channel 00 : 4[89000] → 0[1b000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Channel 01 : 2[3e000] → 3[88000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Channel 01 : 4[89000] → 0[1b000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Channel 00 : 0[1b000] → 1[3d000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Channel 00 : 1[3d000] → 2[3e000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Channel 01 : 0[1b000] → 1[3d000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Channel 00 : 3[88000] → 4[89000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Channel 01 : 1[3d000] → 2[3e000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Channel 01 : 3[88000] → 4[89000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Connected all rings<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Connected all rings<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Connected all rings<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Connected all rings<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Channel 00 : 4[89000] → 3[88000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Connected all rings<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Channel 01 : 4[89000] → 3[88000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Channel 00 : 1[3d000] → 0[1b000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Channel 01 : 1[3d000] → 0[1b000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Channel 00 : 3[88000] → 2[3e000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Channel 01 : 3[88000] → 2[3e000] via direct shared memory<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Channel 00 : 2[3e000] → 1[3d000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO Connected all trees<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Channel 01 : 2[3e000] → 1[3d000] via P2P/IPC<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO Connected all trees<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO Connected all trees<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO Connected all trees<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO Connected all trees<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
december-ds-2h4b6-5hpkj:4042:4139 [1] NCCL INFO comm 0x7f97e8002fb0 rank 1 nranks 5 cudaDev 1 busId 3d000 - Init COMPLETE<br>
december-ds-2h4b6-5hpkj:4043:4141 [2] NCCL INFO comm 0x7f5d80002fb0 rank 2 nranks 5 cudaDev 2 busId 3e000 - Init COMPLETE<br>
december-ds-2h4b6-5hpkj:4044:4140 [3] NCCL INFO comm 0x7fee20002fb0 rank 3 nranks 5 cudaDev 3 busId 88000 - Init COMPLETE<br>
december-ds-2h4b6-5hpkj:4041:4138 [0] NCCL INFO comm 0x7f21a4002fb0 rank 0 nranks 5 cudaDev 0 busId 1b000 - Init COMPLETE<br>
december-ds-2h4b6-5hpkj:4045:4142 [4] NCCL INFO comm 0x7f7078002fb0 rank 4 nranks 5 cudaDev 4 busId 89000 - Init COMPLETE<br>
december-ds-2h4b6-5hpkj:4041:4041 [0] NCCL INFO Launch mode Parallel<br>
12/21/2022 20:56:29 - INFO - <strong>main</strong> - *** Evaluate ***<br>
[INFO|trainer.py:703] 2022-12-21 20:56:29,129 &gt;&gt; The following columns in the evaluation set don’t have a corresponding argument in <code>GPT2LMHeadModel.forward</code> and have been ignored: special_tokens_mask. If special_tokens_mask are not expected by <code>GPT2LMHeadModel.forward</code>,  you can safely ignore this message.<br>
[INFO|trainer.py:2944] 2022-12-21 20:56:29,133 &gt;&gt; ***** Running Evaluation *****<br>
[INFO|trainer.py:2946] 2022-12-21 20:56:29,133 &gt;&gt;   Num examples = 88340<br>
[INFO|trainer.py:2949] 2022-12-21 20:56:29,133 &gt;&gt;   Batch size = 8<br>
0%|          | 0/2209 [00:00&lt;?, ?it/s]<br>
0%|          | 2/2209 [00:00&lt;18:23,  2.00it/s]<br>
0%|          | 3/2209 [00:02&lt;26:56,  1.36it/s]<br>
0%|          | 4/2209 [00:03&lt;31:30,  1.17it/s]<br>
…<br>
…<br>
100%|██████████| 2209/2209 [39:03&lt;00:00,  1.06s/it]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4041 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4042 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4043 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4045 closing signal SIGTERM<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 3 (pid: 4044) of binary: /venv/bin/python3<br>
Traceback (most recent call last):<br>
File “/venv/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 345, in wrapper<br>
return f(*args, **kwargs)<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/run.py”, line 719, in main<br>
run(args)<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/run.py”, line 710, in run<br>
elastic_launch(<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 259, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:<br>
============================================================<br>
./code/gpt2/Model-Compression-Research-Package/examples/transformers/language-modeling/run_clm.py FAILED<br>
Failures:<br>
&lt;NO_OTHER_FAILURES&gt;<br>
Root Cause (first observed failure):<br>
[0]:<br>
time      : 2022-12-21_21:57:29<br>
host      : december-ds-2h4b6-5hpkj<br>
rank      : 3 (local_rank: 3)<br>
exitcode  : -9 (pid: 4044)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 9 (SIGKILL) received by PID 4044<br>
============================================================</p>
</blockquote>
<h3><a name="versions-4" class="anchor" href="#versions-4"></a>Versions:</h3>
<p>According to running  <code>pip freeze --local</code>:</p>
<p><strong>torch==1.10.2+cu113<br>
torchaudio==0.10.2+cu113<br>
torchvision==0.11.3+cu113<br>
nvidia-ml-py==11.495.46<br>
multiprocess==0.70.14<br>
packaging==21.3<br>
transformers==4.25.1<br>
huggingface-hub==0.11.1</strong><br>
absl-py==1.0.0<br>
accelerate==0.15.0<br>
aiohttp==3.8.3<br>
aiosignal==1.3.1<br>
async-timeout==4.0.2<br>
attrs==22.1.0<br>
blessed==1.19.1<br>
cachetools==5.0.0<br>
certifi==2021.10.8<br>
charset-normalizer==2.0.12<br>
click==8.1.3<br>
datasets==2.7.1<br>
deepspeed==0.6.0<br>
dill==0.3.6<br>
docker-pycreds==0.4.0<br>
evaluate==0.4.0<br>
fairscale==0.4.6<br>
filelock==3.8.2<br>
frozenlist==1.3.3<br>
fsspec==2022.11.0<br>
gitdb==4.0.10<br>
GitPython==3.1.29<br>
google-auth==2.6.0<br>
google-auth-oauthlib==0.4.6<br>
gpustat==1.0.0<br>
grpcio==1.44.0<br>
hjson==3.0.2<br>
idna==3.3<br>
importlib-metadata==4.11.2<br>
joblib==1.2.0<br>
Markdown==3.3.6<br>
model-compression-research @ file:///<br>
multidict==6.0.3<br>
ninja==1.10.2.3<br>
nltk==3.8<br>
numpy==1.22.3<br>
oauthlib==3.2.0<br>
pandas==1.5.2<br>
pathtools==0.1.2<br>
Pillow==9.0.1<br>
pkg_resources==0.0.0<br>
promise==2.3<br>
protobuf==3.19.4<br>
psutil==5.9.0<br>
py-cpuinfo==8.0.0<br>
pyarrow==10.0.1<br>
pyasn1==0.4.8<br>
pyasn1-modules==0.2.8<br>
pyparsing==3.0.7<br>
python-dateutil==2.8.2<br>
pytz==2022.7<br>
PyYAML==6.0<br>
regex==2022.10.31<br>
requests==2.27.1<br>
requests-oauthlib==1.3.1<br>
responses==0.18.0<br>
rsa==4.8<br>
scikit-learn==1.2.0<br>
scipy==1.9.3<br>
sentencepiece==0.1.97<br>
sentry-sdk==1.12.0<br>
setproctitle==1.3.2<br>
shortuuid==1.0.11<br>
six==1.16.0<br>
sklearn==0.0.post1<br>
smmap==5.0.0<br>
tensorboard==2.8.0<br>
tensorboard-data-server==0.6.1<br>
tensorboard-plugin-wit==1.8.1<br>
threadpoolctl==3.1.0<br>
tokenizers==0.13.2<br>
tqdm==4.63.0<br>
typing_extensions==4.1.1<br>
urllib3==1.26.13<br>
wandb==0.13.7<br>
wcwidth==0.2.5<br>
Werkzeug==2.0.3<br>
xxhash==3.1.0<br>
yarl==1.8.2<br>
zipp==3.7.0</p>
<h3><a name="notes-5" class="anchor" href="#notes-5"></a>Notes:</h3>
<ol>
<li>The error occurs in training and in evaluation.</li>
<li>In order to eliminate the option of timeout I deliberately fixed very high value for timeout.</li>
<li>I tried to run using torchrun and using torch.distributed.launch and faced the same issue.</li>
<li>The number of samples in my training/eval doesn’t affect and the issue remain.</li>
<li>I track my memory usage and OOM is not the case here (kinda wish it was).</li>
<li>The error occurs only in distributed setup. When not using distributed, or when using it with a single GPU, the problem doesn’t pop.</li>
</ol>
<p>Would really appreciate any help on the subject, since I’m really stuck on my research till I figure it out <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

IdoAmit198: <p>Hey everyone,<br>
anyone might know something regard it?</p>

IdoAmit198: <p>Hey guys,</p>
<p>Tried to upgrade my torch version to <code>1.12.1+cu113</code>.<br>
The error remains, but there are other details in the warnings which I still don’t understand and may be connected to my issue:</p>
<blockquote>
<p>[I socket.cpp:522] [c10d] The server socket has started to listen on [::]:29400.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [localhost]:29400 on [localhost]:51966.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [localhost]:29400 on [localhost]:51980.<br>
[I socket.cpp:522] [c10d] The server socket has started to listen on [::]:40069.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43714.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43722.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43734.<br>
[I ProcessGroupNCCL.cpp:587] [Rank 4] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_DESYNC_DEBUG: 0<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 324000000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
[I ProcessGroupNCCL.cpp:751] [Rank 4] NCCL watchdog thread started!<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43726.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43742.<br>
[I ProcessGroupNCCL.cpp:751] [Rank 3] NCCL watchdog thread started!<br>
[I ProcessGroupNCCL.cpp:587] [Rank 3] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_DESYNC_DEBUG: 0<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 324000000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43754.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43758.<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43770.<br>
[I ProcessGroupNCCL.cpp:587] [Rank 1] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_DESYNC_DEBUG: 0<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 324000000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
[I ProcessGroupNCCL.cpp:751] [Rank 1] NCCL watchdog thread started!<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43776.<br>
[I ProcessGroupNCCL.cpp:587] [Rank 2] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_DESYNC_DEBUG: 0<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 324000000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
[I ProcessGroupNCCL.cpp:751] [Rank 2] NCCL watchdog thread started!<br>
[I socket.cpp:725] [c10d] The client socket has connected to [::ffff:10.42.91.117]:40069 on [::ffff:10.42.91.117]:43778.<br>
[I ProcessGroupNCCL.cpp:587] [Rank 0] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_DESYNC_DEBUG: 0<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 324000000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
[I ProcessGroupNCCL.cpp:751] [Rank 0] NCCL watchdog thread started!<br>
12/24/2022 17:09:12 - WARNING - <strong>main</strong> - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/24/2022 17:09:12 - WARNING - <strong>main</strong> - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/24/2022 17:09:12 - WARNING - <strong>main</strong> - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/24/2022 17:09:12 - WARNING - <strong>main</strong> - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/24/2022 17:09:12 - WARNING - <strong>main</strong> - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False<br>
12/24/2022 17:09:12 - INFO - <strong>main</strong> - Training/evaluation parameters TrainingArguments(<br>
_n_gpu=1,<br>
adafactor=False,<br>
adam_beta1=0.9,<br>
adam_beta2=0.999,<br>
adam_epsilon=1e-08,<br>
auto_find_batch_size=False,<br>
bf16=False,<br>
bf16_full_eval=False,<br>
data_seed=None,<br>
dataloader_drop_last=False,<br>
dataloader_num_workers=0,<br>
dataloader_pin_memory=True,<br>
ddp_bucket_cap_mb=None,<br>
ddp_find_unused_parameters=None,<br>
ddp_timeout=324000,<br>
debug=<span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
deepspeed=None,<br>
disable_tqdm=False,<br>
do_eval=True,<br>
do_predict=False,<br>
do_train=False,<br>
eval_accumulation_steps=None,<br>
eval_delay=0,<br>
eval_steps=5000,<br>
evaluation_strategy=steps,<br>
fp16=False,<br>
fp16_backend=auto,<br>
fp16_full_eval=False,<br>
fp16_opt_level=O1,<br>
fsdp=<span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
fsdp_min_num_params=0,<br>
fsdp_transformer_layer_cls_to_wrap=None,<br>
full_determinism=False,<br>
gradient_accumulation_steps=1,<br>
gradient_checkpointing=False,<br>
greater_is_better=None,<br>
group_by_length=False,<br>
half_precision_backend=auto,<br>
hub_model_id=None,<br>
hub_private_repo=False,<br>
hub_strategy=every_save,<br>
hub_token=&lt;HUB_TOKEN&gt;,<br>
ignore_data_skip=False,<br>
include_inputs_for_metrics=False,<br>
jit_mode_eval=False,<br>
label_names=None,<br>
label_smoothing_factor=0.0,<br>
learning_rate=5e-05,<br>
length_column_name=length,<br>
load_best_model_at_end=False,<br>
local_rank=0,<br>
log_level=passive,<br>
log_level_replica=passive,<br>
log_on_each_node=True,<br>
logging_dir=.cache/results/GPT2_Compression/baseline_results/OpenWebText/gpt2/distributed_inference,<br>
logging_first_step=False,<br>
logging_nan_inf_filter=True,<br>
logging_steps=5000,<br>
logging_strategy=steps,<br>
lr_scheduler_type=linear,<br>
max_grad_norm=1.0,<br>
max_steps=-1,<br>
metric_for_best_model=None,<br>
mp_parameters=,<br>
no_cuda=False,<br>
num_train_epochs=3.0,<br>
optim=adamw_hf,<br>
optim_args=None,<br>
output_dir=.cache/results/GPT2_Compression/baseline_results/OpenWebText/gpt2/distributed_inference,<br>
overwrite_output_dir=True,<br>
past_index=-1,<br>
per_device_eval_batch_size=4,<br>
per_device_train_batch_size=8,<br>
prediction_loss_only=False,<br>
push_to_hub=False,<br>
push_to_hub_model_id=None,<br>
push_to_hub_organization=None,<br>
push_to_hub_token=&lt;PUSH_TO_HUB_TOKEN&gt;,<br>
ray_scope=last,<br>
remove_unused_columns=True,<br>
report_to=[‘wandb’],<br>
resume_from_checkpoint=None,<br>
run_name=openwebtext-distributed-inference,<br>
save_on_each_node=False,<br>
save_steps=1000,<br>
save_strategy=steps,<br>
save_total_limit=None,<br>
seed=2520,<br>
sharded_ddp=<span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
skip_memory_metrics=True,<br>
tf32=None,<br>
torchdynamo=None,<br>
tpu_metrics_debug=False,<br>
tpu_num_cores=None,<br>
use_ipex=False,<br>
use_legacy_prediction_loop=False,<br>
use_mps_device=False,<br>
warmup_ratio=0.0,<br>
warmup_steps=0,<br>
weight_decay=0.0,<br>
xpu_backend=None,<br>
)<br>
[INFO|configuration_utils.py:654] 2022-12-24 17:09:12,874 &gt;&gt; loading configuration file config.json from cache at /store/.cache/huggingface/hub/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/config.json<br>
[INFO|configuration_utils.py:706] 2022-12-24 17:09:12,875 &gt;&gt; Model config GPT2Config {<br>
“_name_or_path”: “gpt2”,<br>
“activation_function”: “gelu_new”,<br>
“architectures”: [<br>
“GPT2LMHeadModel”<br>
],<br>
“attn_pdrop”: 0.1,<br>
“bos_token_id”: 50256,<br>
“embd_pdrop”: 0.1,<br>
“eos_token_id”: 50256,<br>
“initializer_range”: 0.02,<br>
“layer_norm_epsilon”: 1e-05,<br>
“model_type”: “gpt2”,<br>
“n_ctx”: 1024,<br>
“n_embd”: 768,<br>
“n_head”: 12,<br>
“n_inner”: null,<br>
“n_layer”: 12,<br>
“n_positions”: 1024,<br>
“reorder_and_upcast_attn”: false,<br>
“resid_pdrop”: 0.1,<br>
“scale_attn_by_inverse_layer_idx”: false,<br>
“scale_attn_weights”: true,<br>
“summary_activation”: null,<br>
“summary_first_dropout”: 0.1,<br>
“summary_proj_to_labels”: true,<br>
“summary_type”: “cls_index”,<br>
“summary_use_proj”: true,<br>
“task_specific_params”: {<br>
“text-generation”: {<br>
“do_sample”: true,<br>
“max_length”: 50<br>
}<br>
},<br>
“transformers_version”: “4.25.1”,<br>
“use_cache”: true,<br>
“vocab_size”: 50257<br>
}<br>
[INFO|tokenization_auto.py:449] 2022-12-24 17:09:13,190 &gt;&gt; Could not locate the tokenizer configuration file, will try to use the model config instead.<br>
[INFO|configuration_utils.py:654] 2022-12-24 17:09:13,508 &gt;&gt; loading configuration file config.json from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/config.json<br>
[INFO|configuration_utils.py:706] 2022-12-24 17:09:13,509 &gt;&gt; Model config GPT2Config {<br>
“_name_or_path”: “gpt2”,<br>
“activation_function”: “gelu_new”,<br>
“architectures”: [<br>
“GPT2LMHeadModel”<br>
],<br>
“attn_pdrop”: 0.1,<br>
“bos_token_id”: 50256,<br>
“embd_pdrop”: 0.1,<br>
“eos_token_id”: 50256,<br>
“initializer_range”: 0.02,<br>
“layer_norm_epsilon”: 1e-05,<br>
“model_type”: “gpt2”,<br>
“n_ctx”: 1024,<br>
“n_embd”: 768,<br>
“n_head”: 12,<br>
“n_inner”: null,<br>
“n_layer”: 12,<br>
“n_positions”: 1024,<br>
“reorder_and_upcast_attn”: false,<br>
“resid_pdrop”: 0.1,<br>
“scale_attn_by_inverse_layer_idx”: false,<br>
“scale_attn_weights”: true,<br>
“summary_activation”: null,<br>
“summary_first_dropout”: 0.1,<br>
“summary_proj_to_labels”: true,<br>
“summary_type”: “cls_index”,<br>
“summary_use_proj”: true,<br>
“task_specific_params”: {<br>
“text-generation”: {<br>
“do_sample”: true,<br>
“max_length”: 50<br>
}<br>
},<br>
“transformers_version”: “4.25.1”,<br>
“use_cache”: true,<br>
“vocab_size”: 50257<br>
}<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-24 17:09:14,133 &gt;&gt; loading file vocab.json from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/vocab.json<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-24 17:09:14,133 &gt;&gt; loading file merges.txt from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/merges.txt<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-24 17:09:14,133 &gt;&gt; loading file tokenizer.json from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/tokenizer.json<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-24 17:09:14,133 &gt;&gt; loading file added_tokens.json from cache at None<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-24 17:09:14,133 &gt;&gt; loading file special_tokens_map.json from cache at None<br>
[INFO|tokenization_utils_base.py:1799] 2022-12-24 17:09:14,133 &gt;&gt; loading file tokenizer_config.json from cache at None<br>
[INFO|configuration_utils.py:654] 2022-12-24 17:09:14,133 &gt;&gt; loading configuration file config.json from cache at .cache/datasets/processed/openwebtext/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/config.json<br>
[INFO|configuration_utils.py:706] 2022-12-24 17:09:14,134 &gt;&gt; Model config GPT2Config {<br>
“_name_or_path”: “gpt2”,<br>
“activation_function”: “gelu_new”,<br>
“architectures”: [<br>
“GPT2LMHeadModel”<br>
],<br>
“attn_pdrop”: 0.1,<br>
“bos_token_id”: 50256,<br>
“embd_pdrop”: 0.1,<br>
“eos_token_id”: 50256,<br>
“initializer_range”: 0.02,<br>
“layer_norm_epsilon”: 1e-05,<br>
“model_type”: “gpt2”,<br>
“n_ctx”: 1024,<br>
“n_embd”: 768,<br>
“n_head”: 12,<br>
“n_inner”: null,<br>
“n_layer”: 12,<br>
“n_positions”: 1024,<br>
“reorder_and_upcast_attn”: false,<br>
“resid_pdrop”: 0.1,<br>
“scale_attn_by_inverse_layer_idx”: false,<br>
“scale_attn_weights”: true,<br>
“summary_activation”: null,<br>
“summary_first_dropout”: 0.1,<br>
“summary_proj_to_labels”: true,<br>
“summary_type”: “cls_index”,<br>
“summary_use_proj”: true,<br>
“task_specific_params”: {<br>
“text-generation”: {<br>
“do_sample”: true,<br>
“max_length”: 50<br>
}<br>
},<br>
“transformers_version”: “4.25.1”,<br>
“use_cache”: true,<br>
“vocab_size”: 50257<br>
}<br>
[INFO|modeling_utils.py:2204] 2022-12-24 17:09:17,271 &gt;&gt; loading weights file pytorch_model.bin from cache at /store/.cache/huggingface/hub/models–gpt2/snapshots/e7da7f221d5bf496a48136c0cd264e630fe9fcc8/pytorch_model.bin<br>
[I ProcessGroupNCCL.cpp:2012] Rank 4 using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.<br>
[I ProcessGroupNCCL.cpp:2012] Rank 3 using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.<br>
[I ProcessGroupNCCL.cpp:2012] Rank 2 using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.<br>
[INFO|modeling_utils.py:2708] 2022-12-24 17:09:19,025 &gt;&gt; All model checkpoint weights were used when initializing GPT2LMHeadModel.<br>
[INFO|modeling_utils.py:2716] 2022-12-24 17:09:19,026 &gt;&gt; All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.<br>
If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.<br>
/venv/lib/python3.8/site-packages/datasets/dataset_dict.py:1241: FutureWarning: ‘fs’ was is deprecated in favor of ‘storage_options’ in version 2.8.0 and will be removed in 3.0.0.<br>
You can remove this warning by passing ‘storage_options=fs.storage_options’ instead.<br>
warnings.warn(<br>
[I ProcessGroupNCCL.cpp:2012] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.<br>
[I ProcessGroupNCCL.cpp:2012] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.<br>
distributed-4dc7b-wvh75:282:282 [0] NCCL INFO Bootstrap : Using eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:282:282 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
distributed-4dc7b-wvh75:282:282 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
distributed-4dc7b-wvh75:282:282 [0] NCCL INFO NET/Socket : Using [0]eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:282:282 [0] NCCL INFO Using network Socket<br>
NCCL version 2.10.3+cuda11.3<br>
distributed-4dc7b-wvh75:286:286 [4] NCCL INFO Bootstrap : Using eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:284:284 [2] NCCL INFO Bootstrap : Using eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:285:285 [3] NCCL INFO Bootstrap : Using eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:283:283 [1] NCCL INFO Bootstrap : Using eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:286:286 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
distributed-4dc7b-wvh75:286:286 [4] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
distributed-4dc7b-wvh75:283:283 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
distributed-4dc7b-wvh75:286:286 [4] NCCL INFO NET/Socket : Using [0]eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:286:286 [4] NCCL INFO Using network Socket<br>
distributed-4dc7b-wvh75:283:283 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
distributed-4dc7b-wvh75:283:283 [1] NCCL INFO NET/Socket : Using [0]eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:283:283 [1] NCCL INFO Using network Socket<br>
distributed-4dc7b-wvh75:285:285 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
distributed-4dc7b-wvh75:285:285 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
distributed-4dc7b-wvh75:285:285 [3] NCCL INFO NET/Socket : Using [0]eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:285:285 [3] NCCL INFO Using network Socket<br>
distributed-4dc7b-wvh75:284:284 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
distributed-4dc7b-wvh75:284:284 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
distributed-4dc7b-wvh75:284:284 [2] NCCL INFO NET/Socket : Using [0]eth0:10.42.91.117&lt;0&gt;<br>
distributed-4dc7b-wvh75:284:284 [2] NCCL INFO Using network Socket<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Channel 00/02 :    0   1   2   3   4<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Channel 01/02 :    0   1   2   3   4<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Setting affinity for GPU 1 to 0fffff,ff000000,0fffffff<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0 [1] 2/-1/-1-&gt;1-&gt;0<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Trees [0] 3/-1/-1-&gt;2-&gt;1 [1] 3/-1/-1-&gt;2-&gt;1<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Setting affinity for GPU 3 to 0fffff,ff000000,0fffffff<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Setting affinity for GPU 2 to 0fffff,ff000000,0fffffff<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Trees [0] 4/-1/-1-&gt;3-&gt;2 [1] 4/-1/-1-&gt;3-&gt;2<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Trees [0] -1/-1/-1-&gt;4-&gt;3 [1] -1/-1/-1-&gt;4-&gt;3<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Setting affinity for GPU 4 to ffff,fff00000,00ffffff,f0000000<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Setting affinity for GPU 5 to ffff,fff00000,00ffffff,f0000000<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Channel 00 : 2[3e000] → 3[88000] via direct shared memory<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Channel 00 : 4[89000] → 0[1b000] via direct shared memory<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Channel 01 : 2[3e000] → 3[88000] via direct shared memory<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Channel 01 : 4[89000] → 0[1b000] via direct shared memory<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Channel 00 : 1[3d000] → 2[3e000] via P2P/IPC<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Channel 01 : 1[3d000] → 2[3e000] via P2P/IPC<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Channel 00 : 3[88000] → 4[89000] via P2P/IPC<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Channel 01 : 3[88000] → 4[89000] via P2P/IPC<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Channel 00 : 0[1b000] → 1[3d000] via direct shared memory<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Channel 01 : 0[1b000] → 1[3d000] via direct shared memory<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Connected all rings<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Connected all rings<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Channel 00 : 3[88000] → 2[3e000] via direct shared memory<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Connected all rings<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Connected all rings<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Channel 01 : 3[88000] → 2[3e000] via direct shared memory<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Channel 00 : 4[89000] → 3[88000] via P2P/IPC<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Connected all rings<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Channel 01 : 4[89000] → 3[88000] via P2P/IPC<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO Connected all trees<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Channel 00 : 1[3d000] → 0[1b000] via direct shared memory<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Channel 01 : 1[3d000] → 0[1b000] via direct shared memory<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Channel 00 : 2[3e000] → 1[3d000] via P2P/IPC<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Channel 01 : 2[3e000] → 1[3d000] via P2P/IPC<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO Connected all trees<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO Connected all trees<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO Connected all trees<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO Connected all trees<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO threadThresholds 8/8/64 | 40/8/64 | 8/8/512<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
distributed-4dc7b-wvh75:285:382 [3] NCCL INFO comm 0x7fc5c80030d0 rank 3 nranks 5 cudaDev 3 busId 88000 - Init COMPLETE<br>
distributed-4dc7b-wvh75:283:380 [1] NCCL INFO comm 0x7f75e80030d0 rank 1 nranks 5 cudaDev 1 busId 3d000 - Init COMPLETE<br>
distributed-4dc7b-wvh75:284:383 [2] NCCL INFO comm 0x7f6f700030d0 rank 2 nranks 5 cudaDev 2 busId 3e000 - Init COMPLETE<br>
distributed-4dc7b-wvh75:286:381 [4] NCCL INFO comm 0x7f150c0030d0 rank 4 nranks 5 cudaDev 4 busId 89000 - Init COMPLETE<br>
distributed-4dc7b-wvh75:282:379 [0] NCCL INFO comm 0x7f53ec0030d0 rank 0 nranks 5 cudaDev 0 busId 1b000 - Init COMPLETE<br>
[I ProcessGroupNCCL.cpp:1196] NCCL_DEBUG: INFO<br>
distributed-4dc7b-wvh75:282:282 [0] NCCL INFO Launch mode Parallel<br>
/venv/lib/python3.8/site-packages/datasets/dataset_dict.py:1241: FutureWarning: ‘fs’ was is deprecated in favor of ‘storage_options’ in version 2.8.0 and will be removed in 3.0.0.<br>
You can remove this warning by passing ‘storage_options=fs.storage_options’ instead.<br>
warnings.warn(<br>
/venv/lib/python3.8/site-packages/datasets/dataset_dict.py:1241: FutureWarning: ‘fs’ was is deprecated in favor of ‘storage_options’ in version 2.8.0 and will be removed in 3.0.0.<br>
You can remove this warning by passing ‘storage_options=fs.storage_options’ instead.<br>
warnings.warn(<br>
/venv/lib/python3.8/site-packages/datasets/dataset_dict.py:1241: FutureWarning: ‘fs’ was is deprecated in favor of ‘storage_options’ in version 2.8.0 and will be removed in 3.0.0.<br>
You can remove this warning by passing ‘storage_options=fs.storage_options’ instead.<br>
warnings.warn(<br>
/venv/lib/python3.8/site-packages/datasets/dataset_dict.py:1241: FutureWarning: ‘fs’ was is deprecated in favor of ‘storage_options’ in version 2.8.0 and will be removed in 3.0.0.<br>
You can remove this warning by passing ‘storage_options=fs.storage_options’ instead.<br>
warnings.warn(<br>
12/24/2022 17:12:10 - INFO - <strong>main</strong> - *** Evaluate ***<br>
[INFO|trainer.py:703] 2022-12-24 17:12:10,400 &gt;&gt; The following columns in the evaluation set don’t have a corresponding argument in <code>GPT2LMHeadModel.forward</code> and have been ignored: special_tokens_mask. If special_tokens_mask are not expected by <code>GPT2LMHeadModel.forward</code>,  you can safely ignore this message.<br>
[INFO|trainer.py:2944] 2022-12-24 17:12:10,405 &gt;&gt; ***** Running Evaluation *****<br>
[INFO|trainer.py:2946] 2022-12-24 17:12:10,405 &gt;&gt;   Num examples = 88340<br>
[INFO|trainer.py:2949] 2022-12-24 17:12:10,405 &gt;&gt;   Batch size = 4<br>
0%|          | 0/4417 [00:00&lt;?, ?it/s]<br>
0%|          | 2/4417 [00:00&lt;21:13,  3.47it/s]<br>
…<br>
…<br>
100%|█████████▉| 4416/4417 [43:27&lt;00:00,  1.69it/s]<br>
100%|██████████| 4417/4417 [43:28&lt;00:00,  1.69it/s]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 282 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 284 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 285 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 286 closing signal SIGTERM<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 1 (pid: 283) of binary: /venv/bin/python3<br>
Traceback (most recent call last):<br>
File “/venv/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 345, in wrapper<br>
return f(*args, **kwargs)<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/run.py”, line 761, in main<br>
run(args)<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/run.py”, line 752, in run<br>
elastic_launch(<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 245, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:<br>
============================================================<br>
./code/gpt2/Model-Compression-Research-Package/examples/transformers/language-modeling/run_clm.py FAILED<br>
Failures:<br>
&lt;NO_OTHER_FAILURES&gt;<br>
Root Cause (first observed failure):<br>
[0]:<br>
time      : 2022-12-24_17:58:51<br>
host      : distributed-4dc7b-wvh75<br>
rank      : 1 (local_rank: 1)<br>
exitcode  : -9 (pid: 283)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 9 (SIGKILL) received by PID 283<br>
============================================================</p>
</blockquote>
<p>Hope anyone can help me to locate the issue <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

mrshenli: <p>cc <a class="mention" href="/u/d4l3k">@d4l3k</a> for TorchElastic questions</p>
<p>Hey <a class="mention" href="/u/idoamit198">@IdoAmit198</a> , IIUC, the child failure indicates the training process crashed, and the SIGKILL was because TorchElastic detected a failure on peer process and then killed other training processes. It will be helpful to narrow down which part of the training code caused the original failure. Is it possible to add logs to figure out which line caused the failure?</p>