Torchrun breaks between 2.0.1 and 2.1.0

Winton_Davies: <p>I’m training a simple distilBERT binary classifier from a pretrained model. I have a simple trainer set up file, that runs solo, or with torchrun.</p>
<p>It works fine without torchrun no matter which pytorch I have. However, when I upgrade pytorch from 2.0.1 to 2.1, when running it through torchrun it breaks with the following (cryptic error). Has anyone seen this before, or have any idea how to prevent it from happening in 2.1?</p>
<p>NCCL WARN Invalid config blocking attribute value -2147483648<br>
Traceback (most recent call last):<br>
File “/home/user/repos/user_skills_from_title_ensemble/model-interfaces/training_pipelines/model/stages/train_edu_job_binary_model.py”, line 212, in <br>
model = train_edu_job_binary_model(training_data_input_path, model_output_path, num_epochs=num_epochs, num_steps=num_steps, learning_rate=learning_rate, resume_from_checkpoint=resume, train_percentage=train_percentage, train_test_split=train_test_split)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/user/repos/user_skills_from_title_ensemble/model-interfaces/training_pipelines/model/stages/train_edu_job_binary_model.py”, line 187, in train_edu_job_binary_model<br>
trainer.train()<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/transformers/trainer.py”, line 1645, in train<br>
return inner_training_loop(<br>
^^^^^^^^^^^^^^^^^^^^<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/transformers/trainer.py”, line 1756, in _inner_training_loop<br>
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/accelerate/accelerator.py”, line 1350, in prepare<br>
result = tuple(<br>
^^^^^^<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/accelerate/accelerator.py”, line 1351, in <br>
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/accelerate/accelerator.py”, line 1226, in _prepare_one<br>
return self.prepare_model(obj, device_placement=device_placement)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/accelerate/accelerator.py”, line 1477, in prepare_model<br>
model = torch.nn.parallel.DistributedDataParallel(<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/torch/nn/parallel/distributed.py”, line 795, in <strong>init</strong><br>
_verify_param_shape_across_processes(self.process_group, parameters)<br>
File “/home/user/.pyenv/versions/3.11.6/lib/python3.11/site-packages/torch/distributed/utils.py”, line 265, in _verify_param_shape_across_processes<br>
return dist._verify_params_across_processes(process_group, tensors, logger)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
torch.distributed.DistBackendError: NCCL error in: …/torch/csrc/distributed/c10d/NCCLUtils.hpp:219, invalid argument, NCCL version 2.14.3<br>
ncclInvalidArgument: Invalid value for an argument.<br>
Last error:<br>
Invalid config blocking attribute value -2147483648<br>
[2024-12-03 23:00:15,738] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 3273295) of binary: /home/user/.pyenv/versions/3.11.6/bin/python3.11</p>

ptrblck: <p>Do you see the same issue using the latest stable (2.5.1) or nightly release?</p>