Concurrency concerns on the example of parameter server using RPC

ZiyiZhu: <p>Dear <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>I have noticed that your team/colleague released a new tutorial on the parameter server using the RPC framework (<a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="nofollow noopener">rpc_param_server_tutorial</a>). I really appreciate the example with detailed and helpful explanations, and it seems to me that it can work with multiple trainers accessing to the same parameter server. I think the code below makes sure there is only one parameter server can be created by the trainers.</p>
<pre><code class="lang-auto"># The global parameter server instance.
param_server = None
# A lock to ensure we only have one parameter server.
global_lock = Lock()


def get_parameter_server(num_gpus=0):
    """
    Returns a singleton parameter server to all trainer processes
    """
    global param_server
    # Ensure that we get only one handle to the ParameterServer.
    with global_lock:
        if not param_server:
            # construct it once
            param_server = ParameterServer(num_gpus=num_gpus)
        return param_server

def run_parameter_server(rank, world_size):
    # The parameter server just acts as a host for the model and responds to
    # requests from trainers.
    # rpc.shutdown() will wait for all workers to complete by default, which
    # in this case means that the parameter server will wait for all trainers
    # to complete, and then exit.
    print("PS master initializing RPC")
    rpc.init_rpc(name="parameter_server", rank=rank, world_size=world_size)
    print("RPC initialized! Running parameter server...")
    rpc.shutdown()
    print("RPC shutdown on parameter server.")
</code></pre>
<p>However, when it comes to Distributed Autograd, forward, and back passes using the training loop below:</p>
<pre><code class="lang-auto">def run_training_loop(rank, num_gpus, train_loader, test_loader):
...
    for i, (data, target) in enumerate(train_loader):
        with dist_autograd.context() as cid:
            model_output = net(data)
            target = target.to(model_output.device)
            loss = F.nll_loss(model_output, target)
            if i % 5 == 0:
                print(f"Rank {rank} training batch {i} loss {loss.item()}")
            dist_autograd.backward(cid, [loss])
            # Ensure that dist autograd ran successfully and gradients were
            # returned.
            assert remote_method(
                ParameterServer.get_dist_gradients,
                net.param_server_rref,
                cid) != {}
            opt.step(cid)

     print("Training complete!")
     print("Getting accuracy....")
     get_accuracy(test_loader, net)
</code></pre>
<p>How would I make sure there are no concurrency issues? For example, if you have two trainers and have a situation where one trainer is doing the forward propagation and the other is doing the backward pass, how to make sure the two processes are not conflicting with each other?</p>
<p>Thanks,</p>

mrshenli: <p>Hey <a class="mention" href="/u/ziyizhu">@ZiyiZhu</a></p>
<p>That’s a great question! I have two comments on this concurrent param updating approach:</p>
<ol>
<li>Each trainer has its dedicated gradients on the parameter server. When using distributed autograd, the computed gradients are stored in the autograd context (instead of in <code>param.grad</code>), which can be identified by the unique <code>cid</code>. So there is no concern on race errors for gradient computation.</li>
<li>It is true that there can be multiple trainers updating the same parameter concurrently, and it is true that the parameter might change after a trainer computes the gradients and before it applies the gradients to the parameter. This idea is partially borrowed from <a href="https://people.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf" rel="nofollow noopener">the Hogwild! paper</a>. And the approach of using “not perfectly up-to-date gradients” can also be found in other projects (e.g., <a href="https://www.microsoft.com/en-us/research/uploads/prod/2019/08/fiddle_pipedream_sosp19.pdf" rel="nofollow noopener">PipeDream</a>).</li>
</ol>
<p>In general, this is a trade-off between model accuracy and training speed. And in practice, we saw several use cases where this helped to accelerate training a lot with little or none accuracy penalty.</p>
<p>cc <a class="mention" href="/u/rvarm1">@rvarm1</a></p>

ZiyiZhu: <p>Dear <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thank you very much for your explanations with suggested paper references. I will follow up if I still have more questions on the RPC when I finish reading the papers.</p>
<p>Best,<br>
Ziyi</p>

ZiyiZhu: <p>Dear <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>I have briefly gone through the <a href="https://www.microsoft.com/en-us/research/uploads/prod/2019/08/fiddle_pipedream_sosp19.pdf" rel="noopener nofollow ugc">PipeDream</a> paper. Now I understand better how this parameter RPC example can be implicitly pipelined which speeds up the training phase for model parallelism. However, I still have several questions below and hope you could help to answer:</p>
<ol>
<li>
<p>This <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="noopener nofollow ugc">rpc_parameter_example</a> seems to me not a strict parameter server strategy for data parallelism training. Multiple trainers can update the “parameter server” however that is done separately by the distributed  Autograd which means there is no averaging for the gradients of each trainer, IIUC? Is a typical parameter server strategy looking more like<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cf67ed17792fc2d71384f7ffb79ba49081877b7.png" alt="image" data-base62-sha1="fxVGcJBHs6XyhQqOGE8VRufc5JJ" width="445" height="428"><br>
from <a href="https://www.cs.cmu.edu/~muli/file/parameter_server_osdi14.pdf" rel="noopener nofollow ugc">Scaling Distributed</a> paper instead? If I want to do this parameter server strategy (averaging the gradients of each trainer ) for data-parallel , is it possible to do that with RPC?</p>
</li>
<li>
<p>I remember last time you mentioned that by default the Distributed Data Parallel (<a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">DDP</a>) uses ring allreduce for averaging the gradients. Is ring allreduce better than parameter server all the time? But can the DDP use parameter server strategy instead?</p>
</li>
<li>
<p>In the <a href="https://www.microsoft.com/en-us/research/uploads/prod/2019/08/fiddle_pipedream_sosp19.pdf" rel="noopener nofollow ugc">PipeDream</a> paper, the workload can be fine-grained and controlled as below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7baf3f04037b0332b273258765bdb758c05407c3.png" alt="image" data-base62-sha1="hEabU7UcpkiOkt24v1HU4Jizfl9" width="613" height="355"><br>
Would RPC be able to do the same? Or what would be the order/priority for the RPC to choose running which job (1,2,3,4 forward or 1,2,3,4 backward) on the GPU?</p>
</li>
</ol>
<p>Thank you very much!<br>
Best,</p>

mrshenli: <ol>
<li>I was referring to this part of the PipeDream paper:</li>
</ol>
<blockquote>
<p>Each backward pass in a stage results<br>
in weight updates; the next forward pass uses the latest version<br>
of weights available, and “stashes" a copy of these weights to use<br>
during the corresponding backward pass. Although the forward<br>
pass will not see updates from incomplete in-flight mini-batches,<br>
learning is still effective because model weights change relatively<br>
slowly and bounded staleness has been found effective in improving training speeds</p>
</blockquote>
<p>And the Hogwild paper linked above also mentioned sth with similar no-lock spirit:</p>
<blockquote>
<p>In this work, we propose a simple strategy for eliminating the overhead associated with locking:<br>
run SGD in parallel without locks, a strategy that we call Hogwild!. In Hogwild!, processors are<br>
allowed equal access to shared memory and are able to update individual components of memory<br>
at will. Such a lock-free scheme might appear doomed to fail as processors could overwrite each<br>
other’s progress. However, when the data access is sparse, meaning that individual SGD steps only<br>
modify a small part of the decision variable, we show that memory overwrites are rare and that<br>
they introduce barely any error into the computation when they do occur. We demonstrate both<br>
theoretically and experimentally a near linear speedup with the number of processors on commonly<br>
occurring sparse learning problems.</p>
</blockquote>
<p>In general, if lock is necessary (e.g., due to unacceptable accuracy drop), applications can do so by explicitly acquiring locks, but this will certainly have impact on training speed. So it is up to the application to decide how to play with the trade-off.</p>
<ol start="2">
<li></li>
</ol>
<blockquote>
<p>Is ring allreduce better than parameter server all the time?</p>
</blockquote>
<p>No, the merit of allreduce is that it can (actually depend on the loss function) achieve mathematical equivalence with local training. But whether synchronous training is better than asynchronous training is an open question. <code>DistributedDataParallel</code> in PyTorch is using allreduce, but there are also that types of DDP, e.g., <a href="https://openreview.net/forum?id=SkxJ8REYPH" rel="noopener nofollow ugc">SlowMo</a> is using gossip.</p>
<blockquote>
<p>But can the DDP use parameter server strategy instead?</p>
</blockquote>
<p>Yes, it certainly can. The holgwild paper linked above can be one example of using parameter-server-based data parallel on the “decision variable”, e.g., an embedding table.</p>
<ol start="3">
<li></li>
</ol>
<p>Currently the RPC package does not support priority-based communication/execution yet, but this might be doable from the application side. The callee can maintain priority queues to order incoming tasks and use an RPC argument to indicate the priority. The <a href="https://github.com/pytorch/pytorch/issues/36071" rel="noopener nofollow ugc">WIP async user function</a> will help to reduce the callee-side overhead for this use case.</p>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thank you very much again for the provided explanations and references. I will look into them and hopefully have efficient implementations of DDP and RPC in Pytorch.</p>
<p>Best,<br>
Ziyi</p>

ZiyiZhu: <p>Dear <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Following up on the <a href="https://people.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf" rel="noopener nofollow ugc">Hogwild!</a> paper mentioned previously. I also found out that Pytorch has the <a href="https://github.com/pytorch/examples/tree/master/mnist_hogwild" rel="noopener nofollow ugc">PytorchHogwild</a> example using multiprocessing techniques.</p>
<p>I would want to redo this and then extend it for multi-machine and distributed training, either using existing Pytorch DDP or custom design built upon <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">distributed communication APIs</a>.</p>
<p>The big picture is shown below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e7646b72ce1f5ee0d15c505e9aacf85525b49a.png" data-download-href="https://discuss.pytorch.org/uploads/default/c4e7646b72ce1f5ee0d15c505e9aacf85525b49a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e7646b72ce1f5ee0d15c505e9aacf85525b49a.png" alt="image" data-base62-sha1="s5Tl4nnNH4dSUctrEgh6rJwWJvk" width="663" height="500" data-dominant-color="EFEFF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">737×555 20.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Within the Machine#0 !Hogwild can be performed. Does this make sense to you?</p>
<p>However, when I am making a simple example to test the torch.multiprocessing first locally, it seems that multiprocessing cannot work. Please see the result below:</p>
<pre><code class="lang-auto">import os
import torch
from torch import nn
import torch.distributed as dist
import torch.multiprocessing as mp
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(torch.__version__)
</code></pre>
<pre><code class="lang-auto">&gt;&gt; 1.4.0
</code></pre>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__() 
        self.out = nn.Linear(in_features=5, out_features=1)

    def forward(self, t):
        # output
        t = self.out(t)
        return t
</code></pre>
<pre><code class="lang-auto">def addNet(model):
    for para in model.parameters():
        tmp = torch.ones_like(para.data)
        para.data = para.data + tmp
        print(para.data)
</code></pre>
<pre><code class="lang-auto">torch.manual_seed(101)
model = Net()
model.share_memory()
for para in model.parameters():
    print(para.data)
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor([[-0.2701, -0.0445, -0.3659,  0.3463, -0.1884]])
&gt;&gt;&gt; tensor([-0.4306])
</code></pre>
<pre><code class="lang-auto">num_processes = 2
processes = []
for rank in range(num_processes):
    p = mp.Process(target=addNet, args=(model,))
    p.start()
    processes.append(p)
for p in processes:
    p.join()
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor([[0.7299, 0.9555, 0.6341, 1.3463, 0.8116]])
&gt;&gt;&gt; tensor([[0.7299, 0.9555, 0.6341, 1.3463, 0.8116]])
&gt;&gt;&gt; tensor([0.5694])
&gt;&gt;&gt; tensor([0.5694])
</code></pre>
<pre><code class="lang-auto">for para in model.parameters():
    print(para.data)
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor([[-0.2701, -0.0445, -0.3659,  0.3463, -0.1884]])
&gt;&gt;&gt; tensor([-0.4306])
</code></pre>
<p>It seems to me that the children processes just made a copy of the NN in the parent process and the NN was not being shared among the parent and two children processes. Did I miss anything from <a href="https://github.com/pytorch/examples/tree/master/mnist_hogwild" rel="noopener nofollow ugc">PytorchHogwild</a>? Or it will be very much appreciated if you could share any thoughts.</p>
<p>Best,<br>
Ziyi</p>

mrshenli: <p>Hey <a class="mention" href="/u/ziyizhu">@ZiyiZhu</a> did you use the “spawn” mode? If you run the example as is, does it work?</p>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>It seems not. Even the addNet is not printing anything.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da5a5b93615e3d2e64a007a8f3028034ba2fffe9.png" data-download-href="https://discuss.pytorch.org/uploads/default/da5a5b93615e3d2e64a007a8f3028034ba2fffe9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5a5b93615e3d2e64a007a8f3028034ba2fffe9_2_690x332.png" alt="image" data-base62-sha1="v9DG6mAy31zsmyLU1v4D4MOJOsx" width="690" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5a5b93615e3d2e64a007a8f3028034ba2fffe9_2_690x332.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5a5b93615e3d2e64a007a8f3028034ba2fffe9_2_1035x498.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5a5b93615e3d2e64a007a8f3028034ba2fffe9_2_1380x664.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1446×696 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mrshenli: <p>I sometimes run into weird errors when using multiprocessing in notebook. Does it work if you directly launch the script from command line? Let me try that locally.</p>

mrshenli: <p>Oh I see, you need to use the inplace <code>add_</code>, sth like:</p>
<pre><code class="lang-python">import os
import torch
from torch import nn
import torch.distributed as dist
import torch.multiprocessing as mp
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(torch.__version__)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.out = nn.Linear(in_features=5, out_features=1)

    def forward(self, t):
        # output
        t = self.out(t)
        return t


def addNet(model):
    for para in model.parameters():
        tmp = torch.ones_like(para.data)
        with torch.no_grad():
            para.add_(tmp)
            print(para.data)

if __name__=="__main__":
    mp.set_start_method('spawn')

    torch.manual_seed(101)
    model = Net()
    model.share_memory()
    for para in model.parameters():
        print(para.data)

    num_processes = 2
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=addNet, args=(model, ))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()


    for para in model.parameters():
        print(para.data)
</code></pre>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thank you very much for running the code on your end. Yes, I tested and it can work by launching the script from command line. However, the Jupter notebook cannot work, which is interesting.</p>
<p>Going back to the first question, do you see any potential problems with this?</p>
<aside class="quote no-group quote-modified" data-username="ZiyiZhu" data-post="7" data-topic="79577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/ce7236/48.png" class="avatar"> ZiyiZhu:</div>
<blockquote>
<p>The big picture is shown below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e7646b72ce1f5ee0d15c505e9aacf85525b49a.png" data-download-href="https://discuss.pytorch.org/uploads/default/c4e7646b72ce1f5ee0d15c505e9aacf85525b49a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e7646b72ce1f5ee0d15c505e9aacf85525b49a.png" alt="image" data-base62-sha1="s5Tl4nnNH4dSUctrEgh6rJwWJvk" width="663" height="500" data-dominant-color="EFEFF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">737×555 20.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</blockquote>
</aside>
<p>Thank you!</p>
<p>Best,<br>
Ziyi</p>

mrshenli: <p>Hii <a class="mention" href="/u/ziyizhu">@ZiyiZhu</a>, I think the code would run, and it’s like using multiple NCCL allreduce in a HogWild manner, but I am not confident on the correctness on DDP in this use case. The allreduce is not an atomic operation, it needs to go through the ring (say we are using ring allreduce here) twice to collect and propagate the values. So, it is possible that, after allreduce, model parameters on different processes in the same group are no longer in sync. This breaks DDP’s assumption, and this gap could get larger over more iterations. You might need some extra code to re-sync DDP model parameters every n iterations.</p>

mrshenli: <p>I guess we can only find out by training some real models with this scheme. Async training is full of surprises.</p>

ZiyiZhu: <p>I see, and thank you very much for the insights. I would try to re-sync after some iterations or maybe use other schemes such as ps-worker instead.</p>
<p>Best,</p>