Ddp: exclusive writing on disk w/ multiple nodes

username: sbelharbi
cooked: <p>– this is no longer a pytorch issue. but you can find some helpful tips for ddp pytorch. thanks</p>
<p>hi,<br>
i am using <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">ddp</a>.<br>
on a single machine (node=1) w/ many gpus, it is fine.<br>
but with many nodes w/many gpus, i find an issue with file writing.<br>
assuming i run job where each gpu handles a process.<br>
if:<br>
case 1: each gpu is located on a different node and never 2 gpus on the same node</p>
<p>case 2: each gpu is located on the same node (case 2), there is no issue.</p>
<p>let’s say we want to write a tmp file that every process will need later.<br>
in case 1, all of them need to write without worries about writing simultaneously in the same file.<br>
in case 2, only the master can write.</p>
<p>but the problem is when some gpus are located on the same node.<br>
in that case, many process cold attempt to write in the same file which is a problem.</p>
<p>questions:<br>
1- how to properly deal with the mixed case?<br>
2- is there a way to know if the process is a node master?</p>
<p>for second question, which could solve q1, one needs to check the local rank.</p>
<p>this is useful when writing files such as:</p>
<ul>
<li>copy training data</li>
<li>checkpoints.</li>
</ul>
<p>because processes in a node can not see disk of other nodes.<br>
in a node, only one process needs to write.<br>
a node master needs to be designed.</p>
<p>also, a master (global) can be designed as well to handle unique operations that need to be done only once such as logging.</p>
<p>thanks</p>

username: wayi
cooked: <p>If you need different processes on the same node to write to a combined single file, then you can first call <code>gather</code> or <code>gather_objects</code> to collect all the outputs, and then write them to the file. Or you can just let all the processes collectively write to a sharded file, where each shard is a single file output by a process.</p>
<p>If you only want the master to write, you can specify the condition <code>torch.distributed.get_rank() == 0</code>. This is usually good enough for checkpointing the model, since DDP guarantees the model on each process is the same.</p>

username: sbelharbi
cooked: <p>yes, for the synch part, that’s what i was doing (model is self synch using ddp, but for other objects i use torch.distrbuted.all_gather). so, this is fine.</p>
<p>for writing, i allow only node master to write.<br>
for the last point, this is fine on paper but it does not work… not because ddp or pytorch but because of <a href="https://slurm.schedmd.com/documentation.html" rel="noopener nofollow ugc">slurm</a>.</p>
<p>i ask every master node to write/bring/decompress some data from somewhere else to the node local disk. but at the end, they did all instruction, but processes dont find decompressed files. i wrote to the server it staff asking for some explanations. waiting…</p>
<p>so, i’ll face the same issue when writing checkpoints, or any file in each process… later when asked to reload the checkpoint, it will be missing.</p>
<p>thanks</p>

username: wayi
cooked: <p>I think the answer now is quite specific to the cluster and not quite relevant to PyTorch. At the high level, I think you need to write the output to a dedicated distributed/shared file system or directory instead of the local disk of each node. Better ask the cluster admins for the right output destination.</p>

username: sbelharbi
cooked: <ul>
<li>yes, pytorch is needed only to control which process is doing the writing</li>
<li>the rest is cluster dependent.</li>
<li>i/o in local disk is the fastest and the recommended way by admins. writing in network disk or other shared storage is slow and can be extremely slow. in multi-node jobs, they provided a way to dispatch the data across all nodes using <code>srun</code> in a job script but it is not clear how to do it from inside a python code. each process needs to do some i/o operations. it has to be done right in this case. i wrote to admins. no answer yet.</li>
</ul>
<p>so, yes, this question does not concern pytorch anymore. i will edit the question’s header to clarify.<br>
thanks.</p>