Simulating federated learning on single machine multiple gpus

username: Shayan_Talaei
cooked: <p>Hi,<br>
I’m trying to implement federated learning, with a server and 100 clients, on a machine with 8 GPUs. Each client has its private model and its own private dataset. In the parallel setup, each client is taking some local steps, and after some time server asks a group of clients to send their models to the server.<br>
However, I’m implementing a simulator and all these actions are happening consecutively, which means the server chooses a group of clients and makes each of them take their local steps one by one. This takes so long and I want to do this part in parallel. This means I want to make 8 clients take their steps simultaneously, each of which on a different GPU separately. With this, I hope to get an x8 speedup.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/shayan_talaei">@Shayan_Talaei</a>, are you asking for what are the recommended ways to parallelize your system?</p>
<blockquote>
<p>which means the server chooses a group of clients and makes each of them take their local steps one by one. This takes so long and I want to do this part in parallel.</p>
</blockquote>
<p>This looks like the CPU-launching overhead is the bottleneck and therefore not fully utilize the GPU capacity. You can verify if this is indeed the source of slowness using the profiler: <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<p>To parallelize it, you can try multi-threading or multi-processing. PyTorch does drop Python GIL when entering C++ side, so multi-threading should be able to help to some degree. This is also how <code>DataParallel</code> was implemented:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/d08157d5168015c3de0a6d182d08a77a38e5c207/torch/nn/parallel/parallel_apply.py#L23">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/d08157d5168015c3de0a6d182d08a77a38e5c207/torch/nn/parallel/parallel_apply.py#L23" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/d08157d5168015c3de0a6d182d08a77a38e5c207/torch/nn/parallel/parallel_apply.py#L23" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/d08157d5168015c3de0a6d182d08a77a38e5c207/torch/nn/parallel/parallel_apply.py#L23</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="13" style="counter-reset: li-counter 12 ;">
          <li>        for result in map(get_a_var, obj):</li>
          <li>            if isinstance(result, torch.Tensor):</li>
          <li>                return result</li>
          <li>    if isinstance(obj, dict):</li>
          <li>        for result in map(get_a_var, obj.items()):</li>
          <li>            if isinstance(result, torch.Tensor):</li>
          <li>                return result</li>
          <li>    return None</li>
          <li></li>
          <li></li>
          <li class="selected">def parallel_apply(modules, inputs, kwargs_tup=None, devices=None):</li>
          <li>    r"""Applies each `module` in :attr:`modules` in parallel on arguments</li>
          <li>    contained in :attr:`inputs` (positional) and :attr:`kwargs_tup` (keyword)</li>
          <li>    on each of :attr:`devices`.</li>
          <li></li>
          <li>    Args:</li>
          <li>        modules (Module): modules to be parallelized</li>
          <li>        inputs (tensor): inputs to the modules</li>
          <li>        devices (list of int or torch.device): CUDA devices</li>
          <li></li>
          <li>    :attr:`modules`, :attr:`inputs`, :attr:`kwargs_tup` (if given), and</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If you wanna completely get rid of GIL, then you will need multi-processing, a launcher to launch 8 processes, and tools to communicate across processes.</p>
<ol>
<li>Launcher: <a href="https://pytorch.org/docs/stable/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 2.1 documentation</a></li>
<li>Comm: <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed RPC Framework — PyTorch Tutorials 2.1.1+cu121 documentation</a></li>
</ol>

username: Shayan_Talaei
cooked: <p>Thank you so much <a class="mention" href="/u/mrshenli">@mrshenli</a>. I implemented it via multi-threading and it works faster now. At some points, all the GPUs have non-zero utilizations which means they are running in parallel. However, during the server averaging times all of them have zero utilization which is inevitable I guess.</p>

username: mrshenli
cooked: <blockquote>
<p>However, during the server averaging times all of them have zero utilization which is inevitable I guess.</p>
</blockquote>
<p>Yep, looks so to me with synchronous training. To keep the GPUs busy during that, it might need some asynchrony.</p>

username: hongquan
cooked: <p>Hi, I am encountering the same question now. How do you figure this out? Could you please share the code?</p>

username: Travis_Chen
cooked: <p>I am facing the same problems as you have. But I don’t know how to launch devices in multi-threading. Could you please share your snippet or code about the training of FL in multi-threading? Thank you very much</p>