Model update with "share_memory" need lock protection

username: S_Q
cooked: <p>TL;DR.<br>
Maybe a lock is needed for <code>optimizer.step()</code> while using multiprocessing.</p>
<hr>
<p>The example provided in [MULTIPROCESSING BEST PRACTICES] (<a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a>) is to update parameters without lock protection.</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        optimizer.step()  # This will update the shared parameters

if __name__ == '__main__':
    num_processes = 4
    model = MyModel()
    # NOTE: this is required for the ``fork`` method to work
    model.share_memory()
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=train, args=(model,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p>And I checked some A3C code from github, and they do it in the similar way.</p>
<p>To double check whether it’s safe to do that, I created a test case:</p>
<ul>
<li>The global network is init with 0</li>
<li>one worker will add it 1 for 1000 times</li>
<li>another worker will less it 1 for 1000 times</li>
<li>if it is safe, the final network’s weight shall be 0</li>
<li>repeat 100 times to check whether it is safe</li>
</ul>
<p>Test results:</p>
<ul>
<li>With lock protection, it’s safe</li>
<li>Without lock protection (comment the lock, and unindent the <code>optimizer.step()</code>), from time to time, the weight can not come back to 0, which means: not safe</li>
</ul>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.multiprocessing as mp
import torch.optim as optim
from torch.autograd import Variable
import os


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc1.weight.data.fill_(0.0)
        self.fc1.bias.data.fill_(0.0)

    def forward(self):
        return 0

def worker1(net, grad, lock):
    optimizer = optim.SGD(net.parameters(), lr=1)
    for _ in range(1000):
        optimizer.zero_grad()
        for param in net.parameters():
            param._grad = Variable(torch.ones_like(param))
        with lock:
            optimizer.step()
        # print(f'in process {os.getpid()}, the grad of weight is: {net.fc1.weight.grad.item()}, weight is {net.fc1.weight.item()}')

def worker2(net, grad, lock):
    optimizer = optim.SGD(net.parameters(), lr=1)
    for _ in range(1000):
        optimizer.zero_grad()
        for param in net.parameters():
            param._grad = -Variable(torch.ones_like(param))
        with lock:
            optimizer.step()
        # print(f'in process {os.getpid()}, the grad of weight is: {net.fc1.weight.grad.item()}, weight is {net.fc1.weight.item()}')


if __name__ == '__main__':
    n = Net()
    n.share_memory()

    lock = mp.Lock()

    for idx in range(100):
        # print(idx)
        # print(f'in main {os.getpid()}, weight is {n.fc1.weight.item()}')

        p1 = mp.Process(target = worker1, args=(n, 1.0, lock))
        p2 = mp.Process(target = worker2, args=(n, 1.0, lock))

        p1.start()
        p2.start()

        p1.join()
        p2.join()

        # print(f'in main {os.getpid()}, weight is {n.fc1.weight.item()}')

        if n.fc1.weight.item() != 0:
            print(f'{idx} in main {os.getpid()}, weight is {n.fc1.weight.item()}')
            break
    
    print('finished')
</code></pre>
<p>My pytorch version is 1.4.0+cpu</p>
<p>However, as I find it works without lock in the official “Hogwild” example and other examples, maybe the safe issue is not that important…</p>

username: iffiX
cooked: <p>thread safety is not important, the hogwild essay basically spent pages trying to explain that the update is sparse and you can update in <strong>whatever</strong> way(sequence). It’s magic, so forget about the safety. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>

username: Brando_Miranda
cooked: <aside class="quote no-group" data-username="iffiX" data-post="2" data-topic="72857">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>you can update in <strong>whatever</strong> way(sequence)</p>
</blockquote>
</aside>
<p>Can you clarify what you mean by “it’s magic”?</p>

username: S_Q
cooked: <p>Hi, does “whatever way” include update with data race? <img src="https://discuss.pytorch.org/images/emoji/apple/ghost.png?v=9" title=":ghost:" class="emoji" alt=":ghost:"></p>

username: OrNot
cooked: <p>Any clue on this? I have the same question.</p>