Does DataParallel applies own attribute changes to replicas?

username: FruitVinegar
cooked: <p>Hello.<br>
I have a question about DataParallel.</p>
<p>Assume I have a model, which have its own attribute ‘mode’. The model’s forward flow can be changed by setting this ‘mode’.<br>
For example:</p>
<pre><code class="lang-auto">class myModel(nn.Module):
    def __init__(self, mode='A'):
        # do essential initialization...
        self.mode = mode
        self.selectible_layers = nn.ModuleDict(
            {'A': moduleA(), 'B': moduleB(), ...}
        )

    def forward(self, x):
         # forwarding...
         x = self.selectible_layers[self.mode](x)
         # forwarding...
         return x
</code></pre>
<p>When I parallelize this model into multiple GPUs, then I would access this  ‘mode’ attribute by using <code>model.module</code> because nn.DataParallel don’t know about ‘mode’.</p>
<p>Then,</p>
<pre><code class="lang-auto">model = myModel(mode='A') # initialize with mode 'A'
model = nn.DataParallel(model, device_ids = [0, 1, 2, 3]) # distributed into multiple GPUs

for mode in ['A', 'B', 'C', 'D']:
    model.module.mode = mode # change mode
    
    # (?)
</code></pre>
<p>In (?) on above code, does nn.DataParallel guarantee that all model replicas in multiple GPUs have same ‘mode’ when changing it?<br>
I worried about that the ‘mode’ in replicas still have mode ‘A’, while original model(on host GPU; maybe 0)'s ‘mode’ changes.</p>
<p>+)<br>
I tried myself but I don’t know how to access each replica in multiple GPUs. How to access them?</p>

username: mrshenli
cooked: <p>The following method is called by <code>DataParallel</code> to create replicas. So the attributes in <code>__dict__</code> should be replicated as well. But as the flow is “DataParallel forward” → “replicate models” → “app model forward”, you need to make sure that the <code>mode</code> is set properly before calling DataParallel forward.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/modules/module.py#L1210-L1219">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/modules/module.py#L1210-L1219" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/modules/module.py#L1210-L1219" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/modules/module.py#L1210-L1219</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1210" style="counter-reset: li-counter 1209 ;">
          <li>def _replicate_for_data_parallel(self):</li>
          <li>    replica = self.__new__(type(self))</li>
          <li>    replica.__dict__ = self.__dict__.copy()</li>
          <li></li>
          <li>    # replicas do not have parameters themselves, the replicas reference the original</li>
          <li>    # module.</li>
          <li>    replica._parameters = OrderedDict()</li>
          <li>    replica._buffers = replica._buffers.copy()</li>
          <li>    replica._modules = replica._modules.copy()</li>
          <li>    replica._is_replica = True</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>I tried myself but I don’t know how to access each replica in multiple GPUs. How to access them?</p>
</blockquote>
<p>The replicas are created in every forward pass of <code>DataParallel</code> module. To access them and check, you can modify your model’s forward function and print out the mode value.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/parallel/data_parallel.py#L154-L156">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/parallel/data_parallel.py#L154-L156" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/parallel/data_parallel.py#L154-L156" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/parallel/data_parallel.py#L154-L156</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="154" style="counter-reset: li-counter 153 ;">
          <li>replicas = self.replicate(self.module, self.device_ids[:len(inputs)])</li>
          <li>outputs = self.parallel_apply(replicas, inputs, kwargs)</li>
          <li>return self.gather(outputs, self.output_device)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: cruzas2
cooked: <p>Hello there!</p>
<p>Is it also possible to access the replicas directly somehow?<br>
E.g., accessing the model.parameters() of each replica?</p>