What the DDP wrapper do before pass args into self.module?

username: siesta
cooked: <p>in the newest version of pytorch, i notice that the DDP wrapper recursively convert all tensors to cuda tensor when use multigpu training. this behavior is mostly expected.<br>
but recently, i need some moudels use eval mode in my model even in training (mainly for batchnorm), i just code:</p>
<pre><code class="lang-python"># model has been wrapped by DDP
model.eval()
model.training = True
</code></pre>
<p>since there’s only batchnorm that behaves different between training and testing, so the code above should work properly.</p>
<p>i manually set the flag <code>model.training</code> to True because some of my forward code depends on it. but every time step into training forward, this flag automatically become False.<br>
since the ddp source code is complex, i just wonder what the DDP do before passing args (also kwargs) to model? is there any flags or behavior not mentioned in docs?</p>
<p>thanks!</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/siesta">@siesta</a>, I don’t recall DDP implicitly set <code>.trianing</code> to <code>False</code><br>
in forward. The code below is what happens before calling forward on the original model.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/e5915a2216972592ab640dcc0635f0bc7ea8c061/torch/nn/parallel/distributed.py#L969-L1002">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/e5915a2216972592ab640dcc0635f0bc7ea8c061/torch/nn/parallel/distributed.py#L969-L1002" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/e5915a2216972592ab640dcc0635f0bc7ea8c061/torch/nn/parallel/distributed.py#L969-L1002" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/e5915a2216972592ab640dcc0635f0bc7ea8c061/torch/nn/parallel/distributed.py#L969-L1002</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="969" style="counter-reset: li-counter 968 ;">
            <li>def forward(self, *inputs, **kwargs):</li>
            <li>    with torch.autograd.profiler.record_function("DistributedDataParallel.forward"):</li>
            <li>        if torch.is_grad_enabled() and self.require_backward_grad_sync:</li>
            <li>            self.logger.set_runtime_stats_and_log()</li>
            <li>            self.num_iterations += 1</li>
            <li>            self.reducer.prepare_for_forward()</li>
            <li>
            </li>
<li>        # Notify the join context that this process has not joined, if</li>
            <li>        # needed</li>
            <li>        work = Join.notify_join_context(self)</li>
            <li>        if work:</li>
            <li>            self.reducer._set_forward_pass_work_handle(</li>
            <li>                work, self._divide_by_initial_world_size</li>
            <li>            )</li>
            <li>
            </li>
<li>        # Calling _rebuild_buckets before forward compuation,</li>
            <li>        # It may allocate new buckets before deallocating old buckets</li>
            <li>        # inside _rebuild_buckets. To save peak memory usage,</li>
            <li>        # call _rebuild_buckets before the peak memory usage increases</li>
            <li>        # during forward computation.</li>
        </ol>
      </code>
    </pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/e5915a2216972592ab640dcc0635f0bc7ea8c061/torch/nn/parallel/distributed.py#L969-L1002" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Is there a repro that we can dig into?</p>

username: siesta
cooked: <p>thanks for reply! i’m checking and rearrange my code. i’ll reply this thread when repo published and also check this problem can be reoccurred~</p>