RuntimeError: flock: Function not implemented

Philip_Solomon_See: <p>Hi ,<br>
I code a distributed data parallel in pytorch (windows 10) , everything goes fine without error . When I load it in Linux , it give me and error:</p>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap<br>
fn(i, *args)<br>
File “/scratch/users/industry/ite/philipso/m/pixelsnail_25062021/main.py”, line 121, in train<br>
rank=args[0]   )<br>
File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 455, in init_process_group<br>
barrier()<br>
File “/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py”, line 1960, in barrier<br>
work = _default_pg.barrier()<br>
RuntimeError: flock: Function not implemented</p>
<p>This is how I code it</p>
<p>I used</p>
<p>for machine_rank in range(world_size):<br>
mp.spawn( train,args=(world_size,backend,c_param),nprocs=world_size,join=True)</p>
<p>then follow by:</p>
<p>def train(*args):<br>
if (args[3].training_type==1):<br>
torch.distributed.init_process_group(backend=args[2],init_method=r"file:///"+args[3].classifier+".log",world_size=args[1],rank=args[0] )</p>
<p>I suspect the problem is in join=True in mp.spawn</p>
<p>any help will be much appreciated</p>

pritamdamania87: <p>This might be a file system related issue (see <a href="https://github.com/wenet-e2e/wenet/issues/20" class="inline-onebox" rel="noopener nofollow ugc">Run code for training got some errors · Issue #20 · wenet-e2e/wenet · GitHub</a> for a similar issue). What kind of file system are you using for the file in <code>init_method</code>?</p>

Philip_Solomon_See: <p>The System in Linux is CentOS 6<br>
my laptop is Windows 10</p>
<p>Thank Bro for helping me <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

Philip_Solomon_See: <p>The init_method i am using is  <strong>Shared File System</strong></p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Philip_Solomon_See: <p>my share file is located at Scratch of file system Lustre <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pritamdamania87: <p>It could be possible that the shared file system doesn’t support the <code>flock</code> system call. If you are training on one node, can you use the local file system instead?</p>
<p>If you are training with multiple nodes, you can instead use TCP initialization described here: <a href="https://pytorch.org/docs/stable/distributed.html#initialization" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation</a>.</p>

pritamdamania87: <p>Also, looking at <a href="https://wiki.lustre.org/Mounting_a_Lustre_File_System_on_Client_Nodes" class="inline-onebox" rel="noopener nofollow ugc">Mounting a Lustre File System on Client Nodes - Lustre Wiki</a>, it seems like there are options to mount Lustre with support for <code>flock</code>, that might be another option.</p>

Philip_Solomon_See: <p>Hi thank<br>
I have solved all  problems except this “flock function not implemented”. Interestingly , the  shared file did created however, it did not manage to flock. The problem is with init_process_group. What can I replace with this?</p>

pritamdamania87: <p>As I mentioned above, you could use TCP initialization as follows:</p>
<pre><code class="lang-auto">import torch.distributed as dist

# Use address of one of the machines
dist.init_process_group(backend, init_method='tcp://10.1.1.20:23456',
                        rank=args.rank, world_size=4)
</code></pre>