torch.distributed.DistBackendError

SirLPS: <p>Hello! I got this ERROR when I run the code from ‘<a href="https://github.com/tatsu-lab/stanford_alpaca" class="inline-onebox" rel="noopener nofollow ugc">GitHub - tatsu-lab/stanford_alpaca: Code and documentation to train Stanford's Alpaca models, and generate the data.</a>’.  Any suggestions here? Thanks a lot!</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’</p>
<p>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:115 NCCL WARN Cuda failure ‘named symbol not found’</p>
<h1><a name="h-2023-12-10-221651931-n193-016-214208729208819-1-enqueuecc130-nccl-warn-cuda-failure-named-symbol-not-found-2023-12-10-221651931-n193-016-214208729208819-1-nccl-info-initcc1285-1-2023-12-10-221651931-n193-016-214208729208819-1-nccl-info-groupcc64-1-async-thread-2023-12-10-221651932-n193-016-214208729208729-1-nccl-info-groupcc422-1-2023-12-10-221651932-n193-016-214208729208729-1-nccl-info-groupcc106-1-traceback-most-recent-call-last-file-opttigerstanford_alpacatrainpy-line-222-in-module-train-file-opttigerstanford_alpacatrainpy-line-186-in-train-model-transformersautomodelforcausallmfrom_pretrained-file-hometigerlocallibpython39site-packagestransformersmodelsautoauto_factorypy-line-471-in-from_pretrained-return-model_classfrom_pretrained-file-hometigerlocallibpython39site-packagestransformersmodeling_utilspy-line-2498-in-from_pretrained-model-clsconfig-model_args-model_kwargs-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-459-in-wrapper-fmodule-args-kwargs-file-hometigerlocallibpython39site-packagestransformersmodelsllamamodeling_llamapy-line-659-in-__init__-selfmodel-llamamodelconfig-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-459-in-wrapper-fmodule-args-kwargs-file-hometigerlocallibpython39site-packagestransformersmodelsllamamodeling_llamapy-line-463-in-__init__-selfembed_tokens-nnembeddingconfigvocab_size-confighidden_size-selfpadding_idx-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-466-in-wrapper-self_post_init_methodmodule-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-1000-in-_post_init_method-self_zero_init_paramparam-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-956-in-_zero_init_param-distbroadcastparam-0-selfget_dp_process_group-file-hometigerlocallibpython39site-packagesdeepspeedcommcommpy-line-117-in-log_wrapper-return-funcargs-kwargs-file-hometigerlocallibpython39site-packagesdeepspeedcommcommpy-line-224-in-broadcast-return-cdbbroadcasttensortensor-srcsrc-groupgroup-async_opasync_op-file-hometigerlocallibpython39site-packagesdeepspeedcommtorchpy-line-196-in-broadcast-return-torchdistributedbroadcasttensortensor-srcsrc-groupgroup-async_opasync_op-file-usrlocallibpython39dist-packagestorchdistributedc10d_loggerpy-line-47-in-wrapper-return-funcargs-kwargs-file-usrlocallibpython39dist-packagestorchdistributeddistributed_c10dpy-line-1906-in-broadcast-work-default_pgbroadcasttensor-opts-torchdistributeddistbackenderror-nccl-error-in-torchcsrcdistributedc10dprocessgroupncclcpp1367-unhandled-cuda-error-run-with-nccl_debuginfo-for-details-nccl-version-2181-ncclunhandledcudaerror-call-to-cuda-function-failed-last-error-cuda-failure-named-symbol-not-found-2023-12-10-221654324-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-1-local_rank-0-pid-208728-of-binary-usrbinpython3-traceback-most-recent-call-last-file-usrlocalbintorchrun-line-8-in-module-sysexitmain-file-usrlocallibpython39dist-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-346-in-wrapper-return-fargs-kwargs-file-usrlocallibpython39dist-packagestorchdistributedrunpy-line-806-in-main-runargs-file-usrlocallibpython39dist-packagestorchdistributedrunpy-line-797-in-run-elastic_launch-file-usrlocallibpython39dist-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-usrlocallibpython39dist-packagestorchdistributedlauncherapipy-line-264-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#h-2023-12-10-221651931-n193-016-214208729208819-1-enqueuecc130-nccl-warn-cuda-failure-named-symbol-not-found-2023-12-10-221651931-n193-016-214208729208819-1-nccl-info-initcc1285-1-2023-12-10-221651931-n193-016-214208729208819-1-nccl-info-groupcc64-1-async-thread-2023-12-10-221651932-n193-016-214208729208729-1-nccl-info-groupcc422-1-2023-12-10-221651932-n193-016-214208729208729-1-nccl-info-groupcc106-1-traceback-most-recent-call-last-file-opttigerstanford_alpacatrainpy-line-222-in-module-train-file-opttigerstanford_alpacatrainpy-line-186-in-train-model-transformersautomodelforcausallmfrom_pretrained-file-hometigerlocallibpython39site-packagestransformersmodelsautoauto_factorypy-line-471-in-from_pretrained-return-model_classfrom_pretrained-file-hometigerlocallibpython39site-packagestransformersmodeling_utilspy-line-2498-in-from_pretrained-model-clsconfig-model_args-model_kwargs-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-459-in-wrapper-fmodule-args-kwargs-file-hometigerlocallibpython39site-packagestransformersmodelsllamamodeling_llamapy-line-659-in-__init__-selfmodel-llamamodelconfig-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-459-in-wrapper-fmodule-args-kwargs-file-hometigerlocallibpython39site-packagestransformersmodelsllamamodeling_llamapy-line-463-in-__init__-selfembed_tokens-nnembeddingconfigvocab_size-confighidden_size-selfpadding_idx-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-466-in-wrapper-self_post_init_methodmodule-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-1000-in-_post_init_method-self_zero_init_paramparam-file-hometigerlocallibpython39site-packagesdeepspeedruntimezeropartition_parameterspy-line-956-in-_zero_init_param-distbroadcastparam-0-selfget_dp_process_group-file-hometigerlocallibpython39site-packagesdeepspeedcommcommpy-line-117-in-log_wrapper-return-funcargs-kwargs-file-hometigerlocallibpython39site-packagesdeepspeedcommcommpy-line-224-in-broadcast-return-cdbbroadcasttensortensor-srcsrc-groupgroup-async_opasync_op-file-hometigerlocallibpython39site-packagesdeepspeedcommtorchpy-line-196-in-broadcast-return-torchdistributedbroadcasttensortensor-srcsrc-groupgroup-async_opasync_op-file-usrlocallibpython39dist-packagestorchdistributedc10d_loggerpy-line-47-in-wrapper-return-funcargs-kwargs-file-usrlocallibpython39dist-packagestorchdistributeddistributed_c10dpy-line-1906-in-broadcast-work-default_pgbroadcasttensor-opts-torchdistributeddistbackenderror-nccl-error-in-torchcsrcdistributedc10dprocessgroupncclcpp1367-unhandled-cuda-error-run-with-nccl_debuginfo-for-details-nccl-version-2181-ncclunhandledcudaerror-call-to-cuda-function-failed-last-error-cuda-failure-named-symbol-not-found-2023-12-10-221654324-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-1-local_rank-0-pid-208728-of-binary-usrbinpython3-traceback-most-recent-call-last-file-usrlocalbintorchrun-line-8-in-module-sysexitmain-file-usrlocallibpython39dist-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-346-in-wrapper-return-fargs-kwargs-file-usrlocallibpython39dist-packagestorchdistributedrunpy-line-806-in-main-runargs-file-usrlocallibpython39dist-packagestorchdistributedrunpy-line-797-in-run-elastic_launch-file-usrlocallibpython39dist-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-usrlocallibpython39dist-packagestorchdistributedlauncherapipy-line-264-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] enqueue.cc:130 NCCL WARN Cuda failure ‘named symbol not found’<br>
2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] NCCL INFO init.cc:1285 → 1<br>
2023-12-10 22:16:51.931 n193-016-214:208729:208819 [1] NCCL INFO group.cc:64 → 1 [Async thread]<br>
2023-12-10 22:16:51.932 n193-016-214:208729:208729 [1] NCCL INFO group.cc:422 → 1<br>
2023-12-10 22:16:51.932 n193-016-214:208729:208729 [1] NCCL INFO group.cc:106 → 1<br>
Traceback (most recent call last):<br>
File “/opt/tiger/stanford_alpaca/train.py”, line 222, in <br>
train()<br>
File “/opt/tiger/stanford_alpaca/train.py”, line 186, in train<br>
model = transformers.AutoModelForCausalLM.from_pretrained(<br>
File “/home/tiger/.local/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py”, line 471, in from_pretrained<br>
return model_class.from_pretrained(<br>
File “/home/tiger/.local/lib/python3.9/site-packages/transformers/modeling_utils.py”, line 2498, in from_pretrained<br>
model = cls(config, *model_args, **model_kwargs)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py”, line 459, in wrapper<br>
f(module, *args, **kwargs)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py”, line 659, in <strong>init</strong><br>
self.model = LlamaModel(config)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py”, line 459, in wrapper<br>
f(module, *args, **kwargs)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py”, line 463, in <strong>init</strong><br>
self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py”, line 466, in wrapper<br>
self._post_init_method(module)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py”, line 1000, in _post_init_method<br>
self._zero_init_param(param)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/runtime/zero/partition_parameters.py”, line 956, in _zero_init_param<br>
dist.broadcast(param, 0, self.get_dp_process_group())<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/comm/comm.py”, line 117, in log_wrapper<br>
return func(*args, **kwargs)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/comm/comm.py”, line 224, in broadcast<br>
return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)<br>
File “/home/tiger/.local/lib/python3.9/site-packages/deepspeed/comm/torch.py”, line 196, in broadcast<br>
return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/c10d_logger.py”, line 47, in wrapper<br>
return func(*args, **kwargs)<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/distributed_c10d.py”, line 1906, in broadcast<br>
work = default_pg.broadcast([tensor], opts)<br>
torch.distributed.DistBackendError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1367, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1<br>
ncclUnhandledCudaError: Call to CUDA function failed.<br>
Last error:<br>
Cuda failure ‘named symbol not found’<br>
[2023-12-10 22:16:54,324] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 208728) of binary: /usr/bin/python3<br>
Traceback (most recent call last):<br>
File “/usr/local/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 346, in wrapper<br>
return f(*args, **kwargs)<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/run.py”, line 806, in main<br>
run(args)<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/run.py”, line 797, in run<br>
elastic_launch(<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/launcher/api.py”, line 134, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/launcher/api.py”, line 264, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2><a name="trainpy-failed-2" class="anchor" href="#trainpy-failed-2"></a>train.py FAILED</h2>
<h2><a name="failures-1-time-2023-12-10_221654-host-n193-016-214bytedorg-rank-1-local_rank-1-exitcode-1-pid-208729-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-3" class="anchor" href="#failures-1-time-2023-12-10_221654-host-n193-016-214bytedorg-rank-1-local_rank-1-exitcode-1-pid-208729-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-3"></a>Failures:<br>
[1]:<br>
time      : 2023-12-10_22:16:54<br>
host      : <a href="http://n193-016-214.byted.org" rel="noopener nofollow ugc">n193-016-214.byted.org</a><br>
rank      : 1 (local_rank: 1)<br>
exitcode  : 1 (pid: 208729)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.1 documentation</a></h2>
<h1><a name="root-cause-first-observed-failure-0-time-2023-12-10_221654-host-n193-016-214bytedorg-rank-0-local_rank-0-exitcode-1-pid-208728-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4" class="anchor" href="#root-cause-first-observed-failure-0-time-2023-12-10_221654-host-n193-016-214bytedorg-rank-0-local_rank-0-exitcode-1-pid-208728-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4"></a>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2023-12-10_22:16:54<br>
host      : <a href="http://n193-016-214.byted.org" rel="noopener nofollow ugc">n193-016-214.byted.org</a><br>
rank      : 0 (local_rank: 0)<br>
exitcode  : 1 (pid: 208728)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.1 documentation</a></h1>
<p>Sun Dec 10 22:19:05 2023<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 450.191.01   Driver Version: 450.191.01   CUDA Version: 12.2     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  A100-SXM-80GB       On   | 00000000:4A:00.0 Off |                    0 |<br>
| N/A   35C    P0    93W / 400W |      0MiB / 81252MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  A100-SXM-80GB       On   | 00000000:4E:00.0 Off |                    0 |<br>
| N/A   36C    P0    92W / 400W |      0MiB / 81252MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|  No running processes found                                                 |<br>
±----------------------------------------------------------------------------+</p>

ptrblck: <p>Your 450 NVIDIA driver is too old unless you are using forward compatibility in your setup.</p>