MAML inner loop parallelization

username: zzoon91
cooked: <p>Hi,<br>
I want to parallelize the inner loop of MAML.<br>
Each inner loop of the MAML will produce individual loss along with individual gradient graphs,<br>
and after the iteration, I have to aggregate the losses followed by backpropagation.</p>
<p>My naive idea is replacing the loop to map.<br>
To do this, I guess I need to aggregate the loss from multiple threads.<br>
(e.g. torch.mean(torch.stack(list_of_loss_from_multiple_threads))</p>
<p>Is it possible to aggregate graphs from worker threads and then do the backprop at once?</p>
<p>Thanks</p>

username: rvarm1
cooked: <p>Hi, it’s pretty tough to give any concrete advice without first knowing what exactly you’re doing and what you’ve tried. Would you mind posting a snippet of code that indicates the inner loop that you’d like to parallelize, and any instructions needed to run the code? Thanks!</p>

username: Sudarshan_Babu
cooked: <p>I actually want to try this out, I am afraid that the gains in parallelizing the inner loop might be outweighed by communication overhead, were you able to see speed ups ?</p>

username: Sudarshan_Babu
cooked: <p>further, you dont have to aggregate the loss from the different threads, you could compute the gradients within each thread and then aggregate the gradients from different threads.</p>

username: zzoon91
cooked: <p>Sorry, I gave up parallelizing MAML. So I don’t have results that might help your concern. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="zzoon91" data-post="1" data-topic="62289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zzoon91/48/27230_2.png" class="avatar"> zzoon91:</div>
<blockquote>
<p>Is it possible to aggregate graphs from worker threads and then do the backprop at once?</p>
</blockquote>
</aside>
<p>Yes, this is possible, and this is how <code>DataParallel</code> is implemented. The <code>parallel_apply()</code> in the code below will launch multiple threads with each creating their own autograd graph.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/2de549518e7f0ce2820650b401cd21a9901c74a9/torch/nn/parallel/data_parallel.py#L147-L162">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/2de549518e7f0ce2820650b401cd21a9901c74a9/torch/nn/parallel/data_parallel.py#L147-L162" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/2de549518e7f0ce2820650b401cd21a9901c74a9/torch/nn/parallel/data_parallel.py#L147-L162" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/2de549518e7f0ce2820650b401cd21a9901c74a9/torch/nn/parallel/data_parallel.py#L147-L162</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="147" style="counter-reset: li-counter 146 ;">
          <li>def forward(self, *inputs, **kwargs):</li>
          <li>    if not self.device_ids:</li>
          <li>        return self.module(*inputs, **kwargs)</li>
          <li></li>
          <li>    for t in chain(self.module.parameters(), self.module.buffers()):</li>
          <li>        if t.device != self.src_device_obj:</li>
          <li>            raise RuntimeError("module must have its parameters and buffers "</li>
          <li>                               "on device {} (device_ids[0]) but found one of "</li>
          <li>                               "them on device: {}".format(self.src_device_obj, t.device))</li>
          <li></li>
          <li>    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)</li>
          <li>    if len(self.device_ids) == 1:</li>
          <li>        return self.module(*inputs[0], **kwargs[0])</li>
          <li>    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])</li>
          <li>    outputs = self.parallel_apply(replicas, inputs, kwargs)</li>
          <li>    return self.gather(outputs, self.output_device)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>