Apple MACOS M2. Silicon NCCL. Distributed. error - runtime

akramIOT: <p>PROBLEM:  NCCL. Distributed not. found error.</p>
<p>I do undersytand that Apple M2 Silicon does not. support. CUDA and. NCCL but how to fix this error.?</p>
<p>(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama % python3 test_torch.py<br>
tensor([[0.5552, 0.4753, 0.6758],<br>
[0.3080, 0.7625, 0.7667],<br>
[0.5621, 0.6176, 0.2445],<br>
[0.6803, 0.3974, 0.7331],<br>
[0.3485, 0.3801, 0.9699]])<br>
False<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %</p>
<p>Test Script:<br>
import torch<br>
x = torch.rand(5,3)<br>
print(x)</p>
<p>print(torch.cuda.is_available())<br>
<span class="hashtag-raw">#print</span>(torch.cuda.nccl.is_available(tensors=x))<br>
<span class="hashtag-raw">#print</span>(torch.cuda.nccl.is_available(torch.randn(1).cuda(</p>
<h1><a name="new_torch-akram_personalakrams-macbook-pro-llama-torchrun-nproc_per_node-1-example_text_completionpy-ckpt_dir-llama-2-7b-tokenizer_path-tokenizermodel-max_seq_len-128-max_batch_size-6-2024-03-05-233017309-torchdistributedelasticmultiprocessingredirects-warning-note-redirects-are-currently-not-supported-in-windows-or-macos-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy608-userwarning-attempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-warningswarnattempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-traceback-most-recent-call-last-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-69-in-module-firefiremain-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-141-in-fire-component_trace-_firecomponent-args-parsed_flag_args-context-name-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-475-in-_fire-component-remaining_args-_callandupdatetrace-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-691-in-_callandupdatetrace-component-fnvarargs-kwargs-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-32-in-main-generator-llamabuild-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamallamagenerationpy-line-85-in-build-torchdistributedinit_process_groupnccl-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedc10d_loggerpy-line-86-in-wrapper-func_return-funcargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1184-in-init_process_group-default_pg-_-_new_process_group_helper-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1302-in-_new_process_group_helper-raise-runtimeerrordistributed-package-doesnt-have-nccl-built-in-runtimeerror-distributed-package-doesnt-have-nccl-built-in-2024-03-05-233022330-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-1-local_rank-0-pid-48795-of-binary-optanaconda3envsnew_torchbinpython-traceback-most-recent-call-last-file-optanaconda3envsnew_torchbintorchrun-line-33-in-module-sysexitload_entry_pointtorch221-console_scripts-torchrun-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-347-in-wrapper-return-fargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-812-in-main-runargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-803-in-run-elastic_launch-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-135-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-268-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#new_torch-akram_personalakrams-macbook-pro-llama-torchrun-nproc_per_node-1-example_text_completionpy-ckpt_dir-llama-2-7b-tokenizer_path-tokenizermodel-max_seq_len-128-max_batch_size-6-2024-03-05-233017309-torchdistributedelasticmultiprocessingredirects-warning-note-redirects-are-currently-not-supported-in-windows-or-macos-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy608-userwarning-attempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-warningswarnattempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-traceback-most-recent-call-last-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-69-in-module-firefiremain-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-141-in-fire-component_trace-_firecomponent-args-parsed_flag_args-context-name-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-475-in-_fire-component-remaining_args-_callandupdatetrace-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-691-in-_callandupdatetrace-component-fnvarargs-kwargs-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-32-in-main-generator-llamabuild-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamallamagenerationpy-line-85-in-build-torchdistributedinit_process_groupnccl-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedc10d_loggerpy-line-86-in-wrapper-func_return-funcargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1184-in-init_process_group-default_pg-_-_new_process_group_helper-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1302-in-_new_process_group_helper-raise-runtimeerrordistributed-package-doesnt-have-nccl-built-in-runtimeerror-distributed-package-doesnt-have-nccl-built-in-2024-03-05-233022330-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-1-local_rank-0-pid-48795-of-binary-optanaconda3envsnew_torchbinpython-traceback-most-recent-call-last-file-optanaconda3envsnew_torchbintorchrun-line-33-in-module-sysexitload_entry_pointtorch221-console_scripts-torchrun-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-347-in-wrapper-return-fargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-812-in-main-runargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-803-in-run-elastic_launch-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-135-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-268-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama % torchrun --nproc_per_node 1 example_text_completion.py <br>
–ckpt_dir llama-2-7b/ <br>
–tokenizer_path tokenizer.model <br>
–max_seq_len 128 --max_batch_size 6<br>
[2024-03-05 23:30:17,309] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.<br>
/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:608: UserWarning: Attempted to get default timeout for nccl backend, but NCCL support is not compiled<br>
warnings.warn(“Attempted to get default timeout for nccl backend, but NCCL support is not compiled”)<br>
Traceback (most recent call last):<br>
File “/Users/akram_personal/AKRAM_CODE_FOLDER/AKRAM_LLM/LLAMA_MODELS/Meta_Proj/llama/example_text_completion.py”, line 69, in <br>
fire.Fire(main)<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/fire/core.py”, line 141, in Fire<br>
component_trace = _Fire(component, args, parsed_flag_args, context, name)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/fire/core.py”, line 475, in _Fire<br>
component, remaining_args = _CallAndUpdateTrace(<br>
^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/fire/core.py”, line 691, in _CallAndUpdateTrace<br>
component = fn(*varargs, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^<br>
File “/Users/akram_personal/AKRAM_CODE_FOLDER/AKRAM_LLM/LLAMA_MODELS/Meta_Proj/llama/example_text_completion.py”, line 32, in main<br>
generator = Llama.build(<br>
^^^^^^^^^^^^<br>
File “/Users/akram_personal/AKRAM_CODE_FOLDER/AKRAM_LLM/LLAMA_MODELS/Meta_Proj/llama/llama/generation.py”, line 85, in build<br>
torch.distributed.init_process_group(“nccl”)<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/c10d_logger.py”, line 86, in wrapper<br>
func_return = func(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py”, line 1184, in init_process_group<br>
default_pg, _ = _new_process_group_helper(<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py”, line 1302, in _new_process_group_helper<br>
raise RuntimeError(“Distributed package doesn’t have NCCL built in”)<br>
RuntimeError: Distributed package doesn’t have NCCL built in<br>
[2024-03-05 23:30:22,330] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 48795) of binary: /opt/anaconda3/envs/New_Torch/bin/python<br>
Traceback (most recent call last):<br>
File “/opt/anaconda3/envs/New_Torch/bin/torchrun”, line 33, in <br>
sys.exit(load_entry_point(‘torch==2.2.1’, ‘console_scripts’, ‘torchrun’)())<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 347, in wrapper<br>
return f(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/run.py”, line 812, in main<br>
run(args)<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/run.py”, line 803, in run<br>
elastic_launch(<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/launcher/api.py”, line 135, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/launcher/api.py”, line 268, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2><a name="example_text_completionpy-failed-2" class="anchor" href="#example_text_completionpy-failed-2"></a>example_text_completion.py FAILED</h2>
<h2><a name="failures-no_other_failures-3" class="anchor" href="#failures-no_other_failures-3"></a>Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</h2>
<h1><a name="root-cause-first-observed-failure-0-time-2024-03-05_233022-host-10000000000000000000000000000000ip6arpa-rank-0-local_rank-0-exitcode-1-pid-48795-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4" class="anchor" href="#root-cause-first-observed-failure-0-time-2024-03-05_233022-host-10000000000000000000000000000000ip6arpa-rank-0-local_rank-0-exitcode-1-pid-48795-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4"></a>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2024-03-05_23:30:22<br>
host      : 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa<br>
rank      : 0 (local_rank: 0)<br>
exitcode  : 1 (pid: 48795)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.2 documentation</a></h1>
<p>(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %</p>
<p>Reference:  <a href="https://github.com/aggiee/llama-v2-mps/discussions/2" class="inline-onebox" rel="noopener nofollow ugc">I am still getting the. NCCL Error on MACOS. M2 Chip even with the. workaround mentioned for. MPS as per this. Repo, any thoughts ? · aggiee/llama-v2-mps · Discussion #2 · GitHub</a></p>

ptrblck: <p>Don’t use any CUDA or NCCL calls on your setup which does not support them by removing the corresponding PyTorch operations. Alternatively, run your code on a Linux platform with a GPU and it should work.</p>

akramIOT: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> : how do i ensure that  no  CUDA and NCCL calls are there  as this  is  Basic  Vanilla code i have taken for MACOS as per  recommendation. Is there any command output i can check and  validate ?</p>

akramIOT: <h1><a name="ptrblck-new_torch-akram_personalakrams-macbook-pro-llama-new_torch-akram_personalakrams-macbook-pro-llama-torchrun-nproc_per_node-1-example_text_completionpy-ckpt_dir-llama-2-7b-tokenizer_path-tokenizermodel-max_seq_len-128-max_batch_size-6-2024-03-06-213516499-torchdistributedelasticmultiprocessingredirects-warning-note-redirects-are-currently-not-supported-in-windows-or-macos-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy608-userwarning-attempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-warningswarnattempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-traceback-most-recent-call-last-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-69-in-module-firefiremain-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-141-in-fire-component_trace-_firecomponent-args-parsed_flag_args-context-name-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-475-in-_fire-component-remaining_args-_callandupdatetrace-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-691-in-_callandupdatetrace-component-fnvarargs-kwargs-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-32-in-main-generator-llamabuild-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamallamagenerationpy-line-85-in-build-torchdistributedinit_process_groupnccl-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedc10d_loggerpy-line-86-in-wrapper-func_return-funcargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1184-in-init_process_group-default_pg-_-_new_process_group_helper-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1302-in-_new_process_group_helper-raise-runtimeerrordistributed-package-doesnt-have-nccl-built-in-runtimeerror-distributed-package-doesnt-have-nccl-built-in-2024-03-06-213521518-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-1-local_rank-0-pid-56823-of-binary-optanaconda3envsnew_torchbinpython-traceback-most-recent-call-last-file-optanaconda3envsnew_torchbintorchrun-line-33-in-module-sysexitload_entry_pointtorch221-console_scripts-torchrun-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-347-in-wrapper-return-fargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-812-in-main-runargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-803-in-run-elastic_launch-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-135-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-268-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#ptrblck-new_torch-akram_personalakrams-macbook-pro-llama-new_torch-akram_personalakrams-macbook-pro-llama-torchrun-nproc_per_node-1-example_text_completionpy-ckpt_dir-llama-2-7b-tokenizer_path-tokenizermodel-max_seq_len-128-max_batch_size-6-2024-03-06-213516499-torchdistributedelasticmultiprocessingredirects-warning-note-redirects-are-currently-not-supported-in-windows-or-macos-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy608-userwarning-attempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-warningswarnattempted-to-get-default-timeout-for-nccl-backend-but-nccl-support-is-not-compiled-traceback-most-recent-call-last-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-69-in-module-firefiremain-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-141-in-fire-component_trace-_firecomponent-args-parsed_flag_args-context-name-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-475-in-_fire-component-remaining_args-_callandupdatetrace-file-optanaconda3envsnew_torchlibpython311site-packagesfirecorepy-line-691-in-_callandupdatetrace-component-fnvarargs-kwargs-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamaexample_text_completionpy-line-32-in-main-generator-llamabuild-file-usersakram_personalakram_code_folderakram_llmllama_modelsmeta_projllamallamagenerationpy-line-85-in-build-torchdistributedinit_process_groupnccl-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedc10d_loggerpy-line-86-in-wrapper-func_return-funcargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1184-in-init_process_group-default_pg-_-_new_process_group_helper-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributeddistributed_c10dpy-line-1302-in-_new_process_group_helper-raise-runtimeerrordistributed-package-doesnt-have-nccl-built-in-runtimeerror-distributed-package-doesnt-have-nccl-built-in-2024-03-06-213521518-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-1-local_rank-0-pid-56823-of-binary-optanaconda3envsnew_torchbinpython-traceback-most-recent-call-last-file-optanaconda3envsnew_torchbintorchrun-line-33-in-module-sysexitload_entry_pointtorch221-console_scripts-torchrun-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-347-in-wrapper-return-fargs-kwargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-812-in-main-runargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedrunpy-line-803-in-run-elastic_launch-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-135-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-optanaconda3envsnew_torchlibpython311site-packagestorchdistributedlauncherapipy-line-268-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a><a class="mention" href="/u/ptrblck">@ptrblck</a> : (New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama % torchrun --nproc_per_node 1 example_text_completion.py <br>
–ckpt_dir llama-2-7b/ <br>
–tokenizer_path tokenizer.model <br>
–max_seq_len 128 --max_batch_size 6<br>
[2024-03-06 21:35:16,499] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.<br>
/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:608: UserWarning: Attempted to get default timeout for nccl backend, but NCCL support is not compiled<br>
warnings.warn(“Attempted to get default timeout for nccl backend, but NCCL support is not compiled”)<br>
Traceback (most recent call last):<br>
File “/Users/akram_personal/AKRAM_CODE_FOLDER/AKRAM_LLM/LLAMA_MODELS/Meta_Proj/llama/example_text_completion.py”, line 69, in <br>
fire.Fire(main)<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/fire/core.py”, line 141, in Fire<br>
component_trace = _Fire(component, args, parsed_flag_args, context, name)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/fire/core.py”, line 475, in _Fire<br>
component, remaining_args = _CallAndUpdateTrace(<br>
^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/fire/core.py”, line 691, in _CallAndUpdateTrace<br>
component = fn(*varargs, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^<br>
File “/Users/akram_personal/AKRAM_CODE_FOLDER/AKRAM_LLM/LLAMA_MODELS/Meta_Proj/llama/example_text_completion.py”, line 32, in main<br>
generator = Llama.build(<br>
^^^^^^^^^^^^<br>
File “/Users/akram_personal/AKRAM_CODE_FOLDER/AKRAM_LLM/LLAMA_MODELS/Meta_Proj/llama/llama/generation.py”, line 85, in build<br>
torch.distributed.init_process_group(“nccl”)<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/c10d_logger.py”, line 86, in wrapper<br>
func_return = func(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py”, line 1184, in init_process_group<br>
default_pg, _ = _new_process_group_helper(<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py”, line 1302, in _new_process_group_helper<br>
raise RuntimeError(“Distributed package doesn’t have NCCL built in”)<br>
RuntimeError: Distributed package doesn’t have NCCL built in<br>
[2024-03-06 21:35:21,518] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 56823) of binary: /opt/anaconda3/envs/New_Torch/bin/python<br>
Traceback (most recent call last):<br>
File “/opt/anaconda3/envs/New_Torch/bin/torchrun”, line 33, in <br>
sys.exit(load_entry_point(‘torch==2.2.1’, ‘console_scripts’, ‘torchrun’)())<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 347, in wrapper<br>
return f(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/run.py”, line 812, in main<br>
run(args)<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/run.py”, line 803, in run<br>
elastic_launch(<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/launcher/api.py”, line 135, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/distributed/launcher/api.py”, line 268, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2><a name="example_text_completionpy-failed-2" class="anchor" href="#example_text_completionpy-failed-2"></a>example_text_completion.py FAILED</h2>
<h2><a name="failures-no_other_failures-3" class="anchor" href="#failures-no_other_failures-3"></a>Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</h2>
<h1><a name="root-cause-first-observed-failure-0-time-2024-03-06_213521-host-10000000000000000000000000000000ip6arpa-rank-0-local_rank-0-exitcode-1-pid-56823-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4" class="anchor" href="#root-cause-first-observed-failure-0-time-2024-03-06_213521-host-10000000000000000000000000000000ip6arpa-rank-0-local_rank-0-exitcode-1-pid-56823-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4"></a>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2024-03-06_21:35:21<br>
host      : 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa<br>
rank      : 0 (local_rank: 0)<br>
exitcode  : 1 (pid: 56823)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.2 documentation</a></h1>
<p>(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama % pip list<br>
Package                      Version    Editable project location</p>
<hr>
<p>absl-py                      1.4.0<br>
accelerate                   0.27.2<br>
aiohttp                      3.9.3<br>
aiosignal                    1.2.0<br>
anyio                        3.5.0<br>
appdirs                      1.4.4<br>
appnope                      0.1.2<br>
argon2-cffi                  21.3.0<br>
argon2-cffi-bindings         21.2.0<br>
asgiref                      3.4.1<br>
asttokens                    2.0.5<br>
astunparse                   1.6.3<br>
async-lru                    2.0.4<br>
attrs                        21.2.0<br>
azure-core                   1.14.0<br>
azure-storage-blob           12.9.0<br>
Babel                        2.11.0<br>
backcall                     0.2.0<br>
beautifulsoup4               4.12.2<br>
bitsandbytes                 0.42.0<br>
black                        24.2.0<br>
bleach                       4.1.0<br>
blinker                      1.6.2<br>
Bottleneck                   1.3.7<br>
Brotli                       1.1.0<br>
cachetools                   4.2.2<br>
certifi                      2024.2.2<br>
cffi                         1.16.0<br>
chardet                      4.0.0<br>
charset-normalizer           2.0.4<br>
click                        8.1.7<br>
cloudpickle                  1.1.1<br>
colorama                     0.4.4<br>
coloredlogs                  15.0.1<br>
comm                         0.2.1<br>
contourpy                    1.2.0<br>
cryptography                 41.0.3<br>
cycler                       0.11.0<br>
datasets                     2.18.0<br>
debugpy                      1.6.7<br>
decorator                    5.1.0<br>
defusedxml                   0.7.1<br>
Deprecated                   1.2.13<br>
dill                         0.3.8<br>
entrypoints                  0.4<br>
etils                        1.7.0<br>
evaluate                     0.4.1<br>
executing                    0.8.3<br>
fairscale                    0.4.13<br>
fastjsonschema               2.16.2<br>
filelock                     3.13.1<br>
fire                         0.5.0<br>
Flask                        2.0.2<br>
flatbuffers                  23.5.26<br>
fonttools                    4.25.0<br>
frozenlist                   1.4.0<br>
fsspec                       2024.2.0<br>
future                       0.18.2<br>
gast                         0.4.0<br>
gmpy2                        2.1.2<br>
google-auth                  2.22.0<br>
google-auth-oauthlib         0.5.2<br>
google-pasta                 0.2.0<br>
grpcio                       1.48.2<br>
gunicorn                     20.1.0<br>
h11                          0.12.0<br>
h5py                         3.9.0<br>
huggingface-hub              0.21.3<br>
humanfriendly                10.0<br>
idna                         3.3<br>
importlib_resources          6.1.2<br>
inflate64                    1.0.0<br>
ipykernel                    6.28.0<br>
ipython                      8.20.0<br>
ipython-genutils             0.2.0<br>
ipywidgets                   7.6.5<br>
isodate                      0.6.0<br>
isort                        5.9.3<br>
itsdangerous                 2.0.1<br>
jax                          0.3.15<br>
jedi                         0.18.1<br>
Jinja2                       3.0.2<br>
jmespath                     0.10.0<br>
joblib                       1.1.0<br>
json5                        0.9.6<br>
jsonschema                   4.17.3<br>
jupyter                      1.0.0<br>
jupyter-client               7.1.2<br>
jupyter-console              6.6.3<br>
jupyter_core                 5.5.0<br>
jupyter-events               0.6.3<br>
jupyter-lsp                  2.2.0<br>
jupyter-server               1.13.5<br>
jupyter_server_terminals     0.4.4<br>
jupyterlab                   2.3.1<br>
jupyterlab-pygments          0.1.2<br>
jupyterlab-server            1.2.0<br>
jupyterlab-widgets           1.0.0<br>
keras                        2.15.0<br>
Keras-Preprocessing          1.1.2<br>
kiwisolver                   1.4.4<br>
lazy-object-proxy            1.6.0<br>
libclang                     16.0.6<br>
llama                        0.0.1      /Users/akram_personal/AKRAM_CODE_FOLDER/AKRAM_LLM/LLAMA_MODELS/Meta_Proj/llama<br>
llama-recipes                0.0.1<br>
loralib                      0.1.2<br>
lxml                         4.9.3<br>
Markdown                     3.4.1<br>
MarkupSafe                   2.1.3<br>
matplotlib                   3.8.0<br>
matplotlib-inline            0.1.6<br>
mccabe                       0.6.1<br>
mistune                      0.8.4<br>
ml-dtypes                    0.2.0<br>
mmh3                         3.0.0<br>
mpmath                       1.3.0<br>
msrest                       0.6.21<br>
multidict                    6.0.4<br>
multiprocess                 0.70.16<br>
multivolumefile              0.2.3<br>
munkres                      1.1.4<br>
mypy-extensions              1.0.0<br>
nbclassic                    1.0.0<br>
nbclient                     0.8.0<br>
nbconvert                    6.5.4<br>
nbformat                     5.1.3<br>
nest-asyncio                 1.5.1<br>
networkx                     3.1<br>
notebook                     6.5.4<br>
notebook_shim                0.2.3<br>
numexpr                      2.8.7<br>
numpy                        1.23.5<br>
oauthlib                     3.2.2<br>
opt-einsum                   3.3.0<br>
optimum                      1.17.1<br>
overrides                    7.4.0<br>
packaging                    23.2<br>
pandas                       2.0.3<br>
pandocfilters                1.5.0<br>
parso                        0.8.2<br>
pathspec                     0.12.1<br>
peft                         0.9.0<br>
pexpect                      4.8.0<br>
pickleshare                  0.7.5<br>
pillow                       10.2.0<br>
pip                          23.3.1<br>
platformdirs                 3.10.0<br>
ply                          3.11<br>
prometheus-client            0.14.1<br>
prompt-toolkit               3.0.43<br>
protobuf                     3.20.3<br>
psutil                       5.9.0<br>
ptyprocess                   0.7.0<br>
pure-eval                    0.2.2<br>
py7zr                        0.21.0<br>
pyarrow                      15.0.0<br>
pyarrow-hotfix               0.6<br>
pyasn1                       0.4.8<br>
pyasn1-modules               0.2.8<br>
pybcj                        1.0.2<br>
pycparser                    2.20<br>
pycryptodomex                3.20.0<br>
pydantic                     1.8.2<br>
Pygments                     2.10.0<br>
PyJWT                        2.4.0<br>
pyOpenSSL                    23.2.0<br>
pyparsing                    3.0.9<br>
pyppmd                       1.1.0<br>
PyQt5                        5.15.10<br>
PyQt5-sip                    12.13.0<br>
pyrsistent                   0.18.0<br>
PySocks                      1.7.1<br>
python-dateutil              2.8.2<br>
python-editor                1.0.4<br>
python-json-logger           2.0.7<br>
pytz                         2021.3<br>
PyYAML                       6.0<br>
pyzmq                        24.0.1<br>
pyzstd                       0.15.9<br>
qtconsole                    5.5.1<br>
QtPy                         2.4.1<br>
regex                        2023.12.25<br>
requests                     2.26.0<br>
requests-oauthlib            1.3.0<br>
requests-toolbelt            0.9.1<br>
responses                    0.18.0<br>
rfc3339-validator            0.1.4<br>
rfc3986-validator            0.1.1<br>
rpds-py                      0.10.6<br>
rsa                          4.7.2<br>
safetensors                  0.4.2<br>
scipy                        1.11.4<br>
seaborn                      0.11.2<br>
Send2Trash                   1.8.0<br>
sentencepiece                0.2.0<br>
setuptools                   68.2.2<br>
sip                          6.7.12<br>
six                          1.16.0<br>
sniffio                      1.2.0<br>
soupsieve                    2.5<br>
stack-data                   0.2.0<br>
sympy                        1.12<br>
tenacity                     8.0.1<br>
tensorboard                  2.15.2<br>
tensorboard-data-server      0.7.0<br>
tensorboard-plugin-wit       1.8.1<br>
tensorflow                   2.12.0<br>
tensorflow-estimator         2.15.0<br>
tensorflow-io-gcs-filesystem 0.36.0<br>
tensorflow-macos             2.15.0<br>
tensorflow-metal             1.1.0<br>
termcolor                    2.1.0<br>
terminado                    0.17.1<br>
testpath                     0.5.0<br>
texttable                    1.7.0<br>
tinycss2                     1.2.1<br>
tokenize-rt                  5.2.0<br>
tokenizers                   0.15.2<br>
toml                         0.10.2<br>
torch                        2.2.1<br>
torchaudio                   2.2.1<br>
tornado                      6.3.3<br>
tqdm                         4.62.3<br>
traitlets                    5.7.1<br>
transformers                 4.38.2<br>
typing_extensions            4.10.0<br>
tzdata                       2023.3<br>
urllib3                      1.26.18<br>
uvloop                       0.16.0<br>
wcwidth                      0.2.5<br>
webencodings                 0.5.1<br>
websocket-client             0.58.0<br>
Werkzeug                     2.0.2<br>
wheel                        0.35.1<br>
widgetsnbextension           3.5.2<br>
wrapt                        1.14.1<br>
xxhash                       3.4.1<br>
yarl                         1.9.3<br>
zipp                         3.6.0<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama % pip list | grep nccl<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama % pip list | grep cpu<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %<br>
(New_Torch) akram_personal@AKRAMs-MacBook-Pro llama %</p>

akramIOT: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> : Infact i have also validated that. Torch is not compiled with NCCL  support. appreciate all your thoughts to. resolve this on M2 MACOS. Chip.</p>
<p>/opt/anaconda3/envs/New_Torch/lib/python3.11/site-packages/torch/cuda/nccl.py:15: UserWarning: PyTorch is not compiled with NCCL support<br>
warnings.warn(“PyTorch is not compiled with NCCL support”)<br>
False</p>

ptrblck: <p>There is no solution to compile NCCL on MacOS, as it does not support NVIDIA GPUs. The right approach is to check the code you are running and either disable all NCCL calls (or replace these with another library supported on Mac) or to use a Linux workstation with NVIDIA GPUs as already mentioned.</p>