Setting up DistributedDataParallel for single-node "multi-CPU" training

username: Thersites
cooked: <p>Hi Forum,</p>
<p>I am adapting model training for multiple processes on CPU (multiple cores) on one node with DistributedDataParallel.</p>
<p>When running the script with torchrun I receive the following error message:</p>
<blockquote>
<p>NOTE: Redirects are currently not supported in Windows or MacOs.<br>
master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.<br>
NOTE: Redirects are currently not supported in Windows or MacOs.</p>
<p>[W socket.cpp:601] [c10d] The IPv6 network addresses of<br>
(1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 61266) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).<br>
……</p>
<p>WARNING:torch.distributed.elastic.agent.server.api:Received 2 death signal, shutting down workers<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 37667 closing signal SIGINT<br>
File “/Users/ms/ms_transformer/ms_transformer/src/embedding/train_embedding.py”, line 145, in run_node<br>
init_process_group()<br>
File “/Users/ms/ms_transformer/ms_transformer/src/embedding/train_embedding.py”, line 128, in init_process_group<br>
dist.init_process_group(“gloo”,rank=1)<br>
File “/Users/ms/envs/pytorch/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py”, line 900, in init_process_group<br>
store, rank, world_size = next(rendezvous_iterator)<br>
File “/Users/ms/envs/pytorch/lib/python3.10/site-packages/torch/distributed/rendezvous.py”, line 245, in _env_rendezvous_handler<br>
store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout)<br>
File “/Users/ms/envs/pytorch/lib/python3.10/site-packages/torch/distributed/rendezvous.py”, line 176, in _create_c10d_store<br>
return TCPStore(<br>
RuntimeError: Interrupted system call</p>
</blockquote>
<p>(I terminate the script with Control-C</p>
<p>I invoke:</p>
<blockquote>
<p>torchrun --standalone --nnodes=1 --nproc_per_node=1 train.py</p>
</blockquote>
<p>The error appears regardless of what I choose for —nproc_per_node.</p>
<p>It appears node discovery does not work correctly? My impression is that it ought to be possible to distribute training on multiple processes. What might be the problem?<br>
PyTorch 2.0.1, M1 on OS X Ventura.</p>
<p>I set up the process group with:</p>
<blockquote>
<p>def init_process_group():<br>
local_rank = int(os.environ[“LOCAL_RANK”])<br>
dist.init_process_group(“gloo”, rank=local_rank)</p>
<p>def cleanup_process_group():<br>
dist.destroy_process_group()</p>
</blockquote>
<p>LOCAL_RANK rank is read correctly from env.</p>
<p>I wrap the model:</p>
<blockquote>
<p>model = DDP(model, find_unused_parameters=True)</p>
</blockquote>
<p>and leave ‘device_ids’ and ‘output_device’ as None, as pr the documentation.</p>
<p>Any input would be greatly appreciated,<br>
Thersites</p>

username: eqy
cooked: <p>I’m not familiar with training on the M1 CPU, but I’m curious why you would need DDP on a single-node for <em>CPU</em> training. My understanding is that typical numerical libraries are able to leverage multicore CPUs behind the scenes for operations such as matrix multiply and many pointwise operations. From the stack trace I’m guessing that you’re training some kind of transformer model which should be fairly matmul heavy. Have you checked whether training without DDP yields acceptable utilization of your system?</p>

username: Thersites
cooked: <p>Hi,<br>
I am training an embedding with tensors too large to fit on my GPU.<br>
PyTorch does indeed distribute work across processes on my machine, but not as efficiently as I would like, even though it can be tweaked.<br>
The aim is to scale up training, and so I am concerned with effective scaling. I don’t think multi-GPU training will be cost-effective for this application.</p>