Synchronization for sharing/updating shared model state dict across multi-process

wwiiiii: <p>Hello, while I’m doing a RL project, I need some way to share model state dict across multi-processes.</p>
<p>I found the <a href="https://github.com/dgriff777/rl_a3c_pytorch" rel="nofollow noopener">A3C implementation</a> which has this feature, but I have some questions about it.</p>
<ol>
<li>
<p>In the above repo, the author defined model instance on CPU shared memory, and then all the sub processes also share that. In each subprocess, it has its own model instance on GPU, and it loads the state from globally shared CPU model instance without any lock.<br>
So my question is, Is it guranteed that it’s safe to access parameters of the model on CPU shared memory simultaneously in multiple processes without any lock or synchronization?<br>
I’m pretty sure it’s fine since model parameters are treated as read-only variable here, but I just want to double check it.</p>
</li>
<li>
<p>The next question is my main interest. <a href="https://github.com/dgriff777/rl_a3c_pytorch/blob/master/train.py#L104" rel="nofollow noopener">At here</a>, the author simply copies the gradient of model in sub-process to shared model without any lock mechanism, and also optimization is done without synchronization. How come is it possible? Why isn’t everything messed up when multiple processes overwrite the gradient of shared model without any lock mechanism?</p>
</li>
</ol>
<p>If I missed out some points, please let me know <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

wwiiiii: <p>Okay, I found there’s already well documented answer at <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a>.<br>
And also with hogwild!-spirit, it’s fine to overwrite gradient from other process.</p>
<p>So the only thing I wonder now is that how the <a href="https://github.com/dgriff777/rl_a3c_pytorch/blob/master/shared_optim.py#L184" rel="nofollow noopener">updating shared model parameter</a> could be done without any lock mechanism. Are the <code>add_</code> or <code>addcdiv_</code> atomic operation?</p>

Yanzzzzz: <p>I have the same question, and I think it’s not atomic operatino.</p>

neoql: <p>I found a possible answer. <a class="mention" href="/u/yanzzzzz">@Yanzzzzz</a> <a class="mention" href="/u/wwiiiii">@wwiiiii</a></p>
<aside class="quote quote-modified" data-post="2" data-topic="10635">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dgriff/48/1361_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/grad-sharing-problem-in-a3c/10635/2">Grad sharing problem in a3c</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    yeah it really there just to make sure initializing goes smoothly. 
async updates without locks mean they updates from the parallel processes are done asynchronously which means they are run on their own clock and the different process threads can actually make updates at the same time which can lead to bad updates. The reason they do this even though there is risk of bad updates being made is that updates are done much faster which speeds up training as the process of acquiring and releasing lo…
  </blockquote>
</aside>

<p>And  I found another related answer</p>
<aside class="quote quote-modified" data-post="1" data-topic="72857">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/s_q/48/15892_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-update-with-share-memory-need-lock-protection/72857">Model update with "share_memory" need lock protection</a> <a class="badge-category__wrapper " href="/c/reinforcement-learning/6"><span data-category-id="6" data-drop-close="true" class="badge-category " title="A section to discuss RL implementations, research, problems"><span class="badge-category__name">reinforcement-learning</span></span></a>
  </div>
  <blockquote>
    TL;DR. 
Maybe a lock is needed for optimizer.step() while using multiprocessing. 

The example provided in [MULTIPROCESSING BEST PRACTICES] (<a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/multiprocessing.html</a>) is to update parameters without lock protection. 
import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        o…
  </blockquote>
</aside>