Model's parameters update during DDP training

Scott_Hoang: <p>I’m using DDP to train Neural Architecture Search networks which contained a controller and a model network. During training, my controller predictss a model’s architecture that maximize reward. the call looks like this.</p>
<pre><code class="lang-auto"># both model and controller are torch.nn.DistributedDataParallel
arch = controller.forward(conditions)
model.module.set_arch(arch) # modified model internal architecture.
output = model.forward(input)...
</code></pre>
<p>However, in DDP docs I noticed the following:</p>
<blockquote>
<p>… warning::<br>
You should never try to change your model’s parameters after wrapping<br>
up your model with DistributedDataParallel. In other words, when<br>
wrapping up your model with DistributedDataParallel, the constructor of<br>
DistributedDataParallel will register the additional gradient<br>
reduction functions on all the parameters of the model itself at the<br>
time of construction. If you change the model’s parameters after<br>
the DistributedDataParallel construction, this is not supported and<br>
unexpected behaviors can happen, since some parameters’ gradient<br>
reduction functions might not get called.</p>
</blockquote>
<p>So I’m just wondering what is the correct way to do this? or if NAS is not suitable with DDP.</p>

mrshenli: <blockquote>
<p><code>model.module.set_arch(arch) # modified model internal architecture.</code></p>
</blockquote>
<ol>
<li>By doing the above, are you removing parameters from the model or adding new parameters into the model? If yes, then it won’t work with DDP, as DDP creates communication buckets at construction time using the parameters returned by <code>model.parameters()</code> field. Hence, if the <code>model.parameters()</code> returns a different set of parameters, DDP won’t adapt to it.
<ul>
<li>To make it work, you can create a new DDP instance using the modified model whenever the model gets updated. But all DDP processes need to do the same at the same time using the same model.</li>
</ul>
</li>
<li>If it just changes the value of those parameters, it should be fine.</li>
</ol>

Scott_Hoang: <p>can you clarify the different between modifying and replacing?</p>
<pre><code class="lang-auto">def __init__(self):
    self._arch =  torch.variable(&lt;shape&gt; , required_grad=True)
def set_arch(self, arch):
    self._arch = arch # is this modifying or replacing? 

</code></pre>

mrshenli: <p>I believe this is replacing. You can use <code>self._arch.copy_(arch)</code> to override the value. See the code below.</p>
<pre><code class="lang-python">import torch

x = torch.zeros(2, 2)
y = torch.ones(2, 2)
print("x storage: ", x.data_ptr())
print("y storage: ", y.data_ptr())
x = y
print("x storage: ", x.data_ptr())
z = torch.zeros(2, 2) + 2
print("z storage: ", z.data_ptr())
x.copy_(z)
print("x storage: ", x.data_ptr()) 
print(x)
</code></pre>
<p>outputs are:</p>
<pre><code class="lang-auto">x storage:  94191491020800                                                                                                                                           y storage:  94191523992320                                                                                                                                           
x storage:  94191523992320                                                                                                                                        
z storage:  94191523994816                                                                                                                                     
x storage:  94191523992320                                                                                                                                    
tensor([[2., 2.],                                                                                                                                               
        [2., 2.]])   
</code></pre>

Scott_Hoang: <p>This might be it. if DDP wrapper kept a ptr to my arch settings, then it will not see the new value since it with a different pointer.<br>
So does that mean that DDP.module params is a stale copy of our model??</p>

mrshenli: <blockquote>
<p>So does that mean that DDP.module params is a stale copy of our model??</p>
</blockquote>
<p>I believe so. As DDP remembers the variables at construction time:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/09285070a70d146b158db1e1e44b2c031a5c70b0/torch/csrc/distributed/c10d/reducer.cpp#L32">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/09285070a70d146b158db1e1e44b2c031a5c70b0/torch/csrc/distributed/c10d/reducer.cpp#L32" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/09285070a70d146b158db1e1e44b2c031a5c70b0/torch/csrc/distributed/c10d/reducer.cpp#L32" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/09285070a70d146b158db1e1e44b2c031a5c70b0/torch/csrc/distributed/c10d/reducer.cpp#L32</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="22" style="counter-reset: li-counter 21 ;">
          <li>}</li>
          <li></li>
          <li>} // namespace</li>
          <li></li>
          <li>Reducer::Reducer(</li>
          <li>    std::vector&lt;std::vector&lt;torch::autograd::Variable&gt;&gt; replicas,</li>
          <li>    std::vector&lt;std::vector&lt;size_t&gt;&gt; bucket_indices,</li>
          <li>    std::shared_ptr&lt;c10d::ProcessGroup&gt; process_group,</li>
          <li>    std::vector&lt;std::vector&lt;bool&gt;&gt; expect_sparse_gradients,</li>
          <li>    int64_t bucket_bytes_cap)</li>
          <li class="selected">    : replicas_(std::move(replicas)),</li>
          <li>      process_group_(std::move(process_group)),</li>
          <li>      expect_sparse_gradients_(std::move(expect_sparse_gradients)),</li>
          <li>      expect_autograd_hooks_(false),</li>
          <li>      require_finalize_(false),</li>
          <li>      next_bucket_(0),</li>
          <li>      has_marked_unused_parameters_(false),</li>
          <li>      local_used_maps_reduced_(false),</li>
          <li>      backward_stats_base_(0),</li>
          <li>      has_rebuilt_bucket_(false),</li>
          <li>      bucket_bytes_cap_(bucket_bytes_cap) {</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And there might be more than that. DDP might not be able to read that value at all. Because DDP registers a backward hook on each parameter, and relying on that hook to notify DDP when and what to read. Those hooks are installed at DDP construction time as well. If you create a new variable and assign it to <code>self._arch</code>, that hook might be lost.</p>
<p>cc <a class="mention" href="/u/alband">@albanD</a> is the above statement on variable hook correct?</p>

albanD: <p>Hi,</p>
<p>Yes I think this note explicitly warns you against doing this. You should not change the Parameters.</p>
<p>As a side note, you should never call the <code>forward()</code> of your module directly but call <code>module(input)</code>.</p>

Scott_Hoang: <p>what if i modified my forward function such that</p>
<pre><code class="lang-auto">&gt; forward(input, arch):
&gt;      self._arch = arch
</code></pre>
<p>will this works?<br>
Also does DDP keeps DDP.module value up-to-date?</p>

mrshenli: <p>IIUC, that will still remove DDP autograd hooks on <code>self._arch</code>.</p>
<p>Question, do you need the backward pass to compute the gradients for <code>self._arch</code>? If not, you can explicitly setting <code>self._arch.requires_grad = False</code> before passing the model to DDP ctor to tell DDP to ignore <code>self._arch</code>. Then, the above assignment would work.</p>

Scott_Hoang: <p>Thank you. My model is now performing as expected <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>