About distributed autograd and optimizer

iffiX: <p>What’s the best practice of using distributed autograd and optimizer?<br>
Is it designed to apply the master-slave paradigm, where a master distributes task to slaves and collect results, then performes gradient optimization.<br>
Or is it designed to apply a client-host paradigm, where clients actively push results (such as gradients) to the host and host performs post processing like reduction?</p>
<p>I have read your documentation and code in examples. Personally I think it is the first one, though I am not sure what your future plans are, I am currently laying more application layers on rpc and I would like to know your design ideas.</p>

iffiX: <p>The second question: since your implementation of rpc is using <a href="https://pytorch.org/docs/stable/rpc/distributed_autograd.html#fast-mode-algorithm" rel="nofollow noopener">the FAST &amp; SMART mode algorithm</a>, I have also read the detailed <a href="https://github.com/pytorch/pytorch/issues/23110" rel="nofollow noopener">PR</a> on your github. I think your workers are <strong>stateful</strong>, so what will happen if during an execution of the rpc context, some worker fail? like the below scenario in your rpc parameter server example:</p>
<pre><code class="lang-auto">       with dist_autograd.context() as cid:
            model_output = net(data)
            target = target.to(model_output.device)
            loss = F.nll_loss(model_output, target)
            if i % 5 == 0:
                print(f"Rank {rank} training batch {i} loss {loss.item()}")
            dist_autograd.backward(cid, [loss])
            # Ensure that dist autograd ran successfully and gradients were
            # returned.
            assert remote_method(
                ParameterServer.get_dist_gradients,
                net.param_server_rref,
                cid) != {}
            opt.step(cid)
</code></pre>
<p>what kind of exeptions will be thrown on the if the failure occurs in forward &amp; backward pass?</p>

mrshenli: <p>Hey <a class="mention" href="/u/iffix">@iffiX</a>, thanks for the question.</p>
<aside class="quote no-group" data-username="iffiX" data-post="1" data-topic="84047">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>Is it designed to apply the master-slave paradigm, where a master distributes task to slaves and collect results, then performes gradient optimization.<br>
Or is it designed to apply a client-host paradigm, where clients actively push results (such as gradients) to the host and host performs post processing like reduction?</p>
</blockquote>
</aside>
<p>They are designed in the way that all RPC workers are peers, such that each RPC worker has its own server running in background, and any RPC worker A can talk to any other RPC worker B by sending a function to run on B. In the foreseeable future, we are not going to change this design.</p>
<p>We do see many applications using RPC to build master-slave applications, where one worker serves as a master telling everyone else what to do. One example is <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-reinforcement-learning-using-rpc-and-rref" rel="noopener nofollow ugc">this tutorial</a> where the agent is serving as the master, and commanding all observers.</p>
<p>It is also possible to build server-client applications using the RPC API. Here is an example: <a href="https://github.com/pytorch/examples/pull/702/files" class="inline-onebox" rel="noopener nofollow ugc">[WIP] adding batch server/client by mrshenli · Pull Request #702 · pytorch/examples · GitHub</a></p>
<p>In general, we try to make the RPC API as flexible as possible, and hope it can support a wide range of applications. It’s up to the application developers to decide how to decompose the entire logic into multiple functions and how to stitch those functions together using RPC.</p>
<p>Regarding distributed autograd and distributed optimizer, the worker that creates the distributed autograd context serves as the driver for its backward pass and optimizer step. But there can be multiple drivers in a cluster. One example would be <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="noopener nofollow ugc">this tutorial</a>, where we launch one Parameter-Server process that passively sits there waiting for requests from trainers and we also launch multiple trainer processes with each trainer actively running a training loop. In this case, every trainer serves as a driver of its own backward pass and optimizer step.</p>

mrshenli: <aside class="quote no-group" data-username="iffiX" data-post="2" data-topic="84047">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>I think your workers are <strong>stateful</strong> , so what will happen if during an execution of the rpc context, some worker fail?</p>
</blockquote>
</aside>
<p>This is true, they are stateful, and states including RRef context, distributed autograd context and application states. In the current version, an RPC gang cannot survive any node crash. We are actively collaborating with <a href="https://pytorch.org/elastic" rel="noopener nofollow ugc">torchelastic</a> to provide fault-tolerance and elasticity support.</p>
<blockquote>
<p>what kind of exeptions will be thrown on the if the failure occurs in forward &amp; backward pass?</p>
</blockquote>
<p>If the error is recoverable, e.g., just an exception instead of hard node crash, RPC should throw the same exception (or a <code>RuntimeError</code>, this is not great, still working on improvements) on the caller. Below are some example tests:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/dist_autograd_test.py#L1204-L1210">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/dist_autograd_test.py#L1204-L1210" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/dist_autograd_test.py#L1204-L1210" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/dist_autograd_test.py#L1204-L1210</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1204" style="counter-reset: li-counter 1203 ;">
          <li>def test_backward_autograd_engine_error(self):</li>
          <li>    with dist_autograd.context() as context_id:</li>
          <li>        t1 = torch.rand((3, 3), requires_grad=True)</li>
          <li>        t2 = torch.rand((3, 3), requires_grad=True)</li>
          <li>        # Perform some ops before error simulation.</li>
          <li>        tmp = (t1 + t2) * (t1 + t2)</li>
          <li>        t3 = SimulateBackwardError.apply(tmp)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/rpc_test.py#L1284-L1290">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/rpc_test.py#L1284-L1290" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/rpc_test.py#L1284-L1290" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/c767d65cafc2615bcee9ec1bbc8d5dadc07ea0e8/torch/testing/_internal/distributed/rpc/rpc_test.py#L1284-L1290</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1284" style="counter-reset: li-counter 1283 ;">
          <li>@dist_init</li>
          <li>def test_py_raise_in_user_func(self):</li>
          <li>    n = self.rank + 1</li>
          <li>    dst_rank = n % self.world_size</li>
          <li>    fut = rpc.rpc_async(worker_name(dst_rank), raise_func)</li>
          <li>    with self.assertRaises(ValueError):</li>
          <li>        fut.wait()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


iffiX: <p>Thank you <strong>mrshenli</strong>, you are so helpful!</p>
<p><strong>About my question:</strong><br>
Well the first question is actually about the autograd based on rpc, and not rpc apis themselves, but on hindsight I think it is not really a good question, so lets set it aside. The second answer is sufficient. I think I cannot use the distributed autograd now as they are a little bit too brittle.</p>
<p><strong>About distributed programming:</strong><br>
Indeed, I have noticed your amazing work on <code>torchelastic</code>, I have thought about your rpc design, and find that it could be very hard for you to implement a mechanism which rejoins an rpc worker after their crash (whether a hard node crash or a soft crash like an exception), since you are blocking all processes on init and exchange their connection information, it would be overly complex to try and recover these information on crash.</p>
<p>Therefore, I instead worked out another solution to provide some robustness for raw rpc apis, since under some conditions the “map-reduce” (very similar so I will use this description, since workers will receive their work portion when they reach the start barrier) paradigm in torchelastic Rendezvous is not very handy, and rpc is just right.</p>
<p>My solution basically implements a stable distributed election process &amp; perfect failure detector based on rpc, and let each worker in the rpc group takes one/many role(s), when a worker failes (hard/soft), its role will be reassigned to other workers and this worker is marked as tainted  permanently. Users only needs to declare the number of total workers on start and  specify what a role have to do when they are: initialized, running, and stopped. Adds a little bit overhead on start &amp; worker failure, but near zero overhead during a normal rpc call.</p>
<p>My future plan is to integrate my framework with NNI tuner developed by the microsoft team, so I can have some taste of distributed RL training on kubernetes.</p>
<p>That’s all for now, thank you again! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

mrshenli: <aside class="quote no-group" data-username="iffiX" data-post="5" data-topic="84047">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>since you are blocking all processes on init and exchange their connection information, it would be overly complex to try and recover these information on crash.</p>
</blockquote>
</aside>
<p>The rendezvous in RPC init will be gone in future releases (not sure which release yet <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> ). We are thinking about using <code>c10d::Store</code> + listeners to support dynamic worker join/leave.</p>
<blockquote>
<p>My solution basically implements a stable distributed election process &amp; perfect failure detector based on rpc, and let each worker in the rpc group takes one/many role(s), when a worker failes (hard/soft), its role will be reassigned to other workers and this worker is marked as tainted permanently. Users only needs to declare the number of total workers on start and specify what a role have to do when they are: initialized, running, and stopped. Adds a little bit overhead on start &amp; worker failure, but near zero overhead during a normal rpc call.</p>
</blockquote>
<p>This is awesome! Are these in an open-source repo? It will be great if we could have the opportunity to learn more about it.  cc <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>

iffiX: <p>Dynamic worker join/leave is great! With that we can just discard failed workers and start new ones, looking forward to that.</p>
<p>I am still writing docs for my framework, and haven’t begin the unit test &amp; CI/CD process yet <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"> <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
<p>Currently my framework aims more at RL things, so tests for RL algortihms will come first, distributed part is mainly aimed at resource hungry algorithms such as DQN-APEX, DDPG-APEX, IMPALA etc., and multi agent scenes. I really hope to complete code, doc and test for major RL algorithms part <strong>before July</strong>, then completes distributed scenes <strong>before August / September</strong> so you will have to wait a l-i-t-t-le more. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"> <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
<p>They will be in a new github repo, and I will post updates as soon as possible.</p>

mrshenli: <p>Great! Looking forward to that!</p>