Model and ddp wrapped model

username: valiantljk
cooked: <p>Hi, I’m using allennlp to do distributed bert training.<br>
In their code, model has some customized functions, e.g., <a href="https://github.com/allenai/allennlp/blob/cf113d705b9054d329c67cf9bb29cbc3f191015d/allennlp/models/model.py#L237" rel="noopener nofollow ugc">get_metrics</a>, and  <a href="https://github.com/allenai/allennlp/blob/cf113d705b9054d329c67cf9bb29cbc3f191015d/allennlp/models/model.py#L87" rel="noopener nofollow ugc">get_regularization_penalty</a>. After wrapping it with ddp, there is a comment says</p>
<pre><code class="lang-auto">        # Using `DistributedDataParallel`(ddp) brings in a quirk wrt AllenNLP's `Model` interface and its
        # usage. A `Model` object is wrapped by `ddp`, but assigning the wrapped model to `self.model`
        # will break the usages such as `Model.get_regularization_penalty`, `Model.get_metrics`, etc.
        #
        # Hence a reference to Pytorch's object is maintained in the case of distributed training and in the
        # normal case, reference to `Model` is retained. This reference is only used in
        # these places: `model.__call__`, `model.train` and `model.eval`.
</code></pre>
<aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/allenai/allennlp/blob/c5bff8ba0d835eb03931f10f4f427ffe936cf796/allennlp/training/gradient_descent_trainer.py#L302" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/allenai/allennlp/blob/c5bff8ba0d835eb03931f10f4f427ffe936cf796/allennlp/training/gradient_descent_trainer.py#L302" target="_blank" rel="noopener nofollow ugc">allenai/allennlp/blob/c5bff8ba0d835eb03931f10f4f427ffe936cf796/allennlp/training/gradient_descent_trainer.py#L302</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="292" style="counter-reset: li-counter 291 ;">
          <li>self._num_gradient_accumulation_steps = num_gradient_accumulation_steps</li>
          <li>
          </li>
<li># Enable automatic mixed precision training.</li>
          <li>self._scaler: Optional[amp.GradScaler] = None</li>
          <li>self._use_amp = use_amp</li>
          <li>if self._use_amp:</li>
          <li>    if self.cuda_device == torch.device("cpu"):</li>
          <li>        raise ValueError("Using AMP requires a cuda device")</li>
          <li>    self._scaler = amp.GradScaler()</li>
          <li>
          </li>
<li class="selected"># Using `DistributedDataParallel`(ddp) brings in a quirk wrt AllenNLP's `Model` interface and its</li>
          <li># usage. A `Model` object is wrapped by `ddp`, but assigning the wrapped model to `self.model`</li>
          <li># will break the usages such as `Model.get_regularization_penalty`, `Model.get_metrics`, etc.</li>
          <li>#</li>
          <li># Hence a reference to Pytorch's object is maintained in the case of distributed training and in the</li>
          <li># normal case, reference to `Model` is retained. This reference is only used in</li>
          <li># these places: `model.__call__`, `model.train` and `model.eval`.</li>
          <li>if self._distributed:</li>
          <li>    self._pytorch_model = DistributedDataParallel(</li>
          <li>        self.model,</li>
          <li>        device_ids=None if self.cuda_device == torch.device("cpu") else [self.cuda_device],</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>My question is what is the relationship between self.model and its wrapped version self._pytorch_model?</p>
<p>Do they share parameters and runtime state?</p>

username: tom
cooked: <aside class="quote no-group" data-username="valiantljk" data-post="1" data-topic="123023">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valiantljk/48/38045_2.png" class="avatar"> valiantljk:</div>
<blockquote>
<p>My question is what is the relationship between self.model and its wrapped version self._pytorch_model?</p>
</blockquote>
</aside>
<p>You have one object for each of the three classes</p>
<ol>
<li>m = ThePyTorchModel   (without DDP)</li>
<li>ddp_m = DistributedDataParallel(ThePyTorchModel)</li>
<li>anlp_m = Model(ThePyTorchModel) (AllenNLP’s model class)</li>
</ol>
<p>ddp_m and anlp_m wrap (i.e. contain a reference to) the (same) instance m as <code>.module</code> and <code>.model</code> usually.</p>
<p>Now AllenNLP doesn’t want to special case and write <code>.model.module if isinstance(.model, DDP) else .model</code> all the time, so it leaves <code>.model</code> to be the regular model <code>m</code> but stores <code>ddp_m</code>  as the PyTorch <code>._pytorch_model</code>.</p>
<p>So you should have <code>anlp_m._pytorch_model.module is anlp_m.model</code> return <code>True</code>, they are indeed the very same object.</p>
<aside class="quote no-group" data-username="valiantljk" data-post="1" data-topic="123023">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valiantljk/48/38045_2.png" class="avatar"> valiantljk:</div>
<blockquote>
<p>Do they share parameters and runtime state?</p>
</blockquote>
</aside>
<p>Yes in the above sense (that you have an additional hierarchy level when going through DDP).</p>
<p>Best regards</p>
<p>Thomas</p>

username: valiantljk
cooked: <p>Thanks <a class="mention" href="/u/tom">@tom</a>. Very clear now.<br>
A follow up question, when the ddp wrapped model is copied onto one cuda device, does the original model still hold the reference to their common ancestor?</p>

username: tom
cooked: <p>Yes, I think so. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ← Service smile to reach the minimum length.</p>