How to access a class object when I use torch.nn.DataParallel()?

yamand16: <p>I want to train my model using PyTorch with multiple GPUs. I included the following line:</p>
<pre><code class="lang-auto">model = torch.nn.DataParallel(model, device_ids=opt.gpu_ids)
</code></pre>
<p>Then, I tried to access the optimizer that was defined in my model definition:</p>
<pre><code class="lang-auto">G_opt = model.module.optimizer_G
</code></pre>
<p>However, I got an error:</p>
<blockquote>
<p>AttributeError: ‘DataParallel’ object has no attribute optimizer_G</p>
</blockquote>
<p>I think it is related with the definition of optimizer in my model definition. It works when I use single GPU without <code>torch.nn.DataParallel</code>. But it does not work with multi GPUs even though I call with <code>module</code> and I could not find the solution.</p>
<p>Here is the model definition:</p>
<pre><code class="lang-auto">class MyModel(torch.nn.Module):
    ...
   self.optimizer_G = torch.optim.Adam(params, lr=opt.lr, betas=(opt.beta1, 0.999))   
</code></pre>
<p>I used Pix2PixHD implementation in <a href="https://github.com/NVIDIA/pix2pixHD" rel="noopener nofollow ugc">GitHub</a> if you want to see the full code.</p>
<p>Thank you, Best.</p>

Manuel_Alejandro_Dia: <p>I do not know if you have tried this already, but since the error code tells you that the object is of the <code>DataParallel</code> class, can you make sure that the module is the class you expect?</p>
<p>Something like:</p>
<pre><code class="lang-auto">print(type(model.module))
</code></pre>

yamand16: <p>Here is the output of the <code>print(type(model.module))</code></p>
<blockquote>
<p>&lt;class ‘torch.nn.parallel.data_parallel.DataParallel’&gt;</p>
</blockquote>
<p>Another point in the original code is that when there is no <code>torch.nn.DataParallel()</code> code to use multi GPUs, it still uses model.module to access to the class objects and it works. But when I include <code>torch.nn.DataParallel()</code>, it throws the error in my first post.</p>
<p>Edit: When I used <code>model.module.module.optimizer_G</code>, it worked.</p>

Manuel_Alejandro_Dia: <aside class="quote no-group" data-username="yamand16" data-post="3" data-topic="114628">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yamand16/48/23423_2.png" class="avatar"> yamand16:</div>
<blockquote>
<p>Edit: When I used <code>model.module.module.optimizer_G</code>, it worked.</p>
</blockquote>
</aside>
<p>This is what I suspected was going to be the answer.<br>
You could carefully go through all of your code (or at least the parts where the model gets used), because if this happens:</p>
<aside class="quote no-group quote-modified" data-username="yamand16" data-post="3" data-topic="114628">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yamand16/48/23423_2.png" class="avatar"> yamand16:</div>
<blockquote>
<p>Here is the output of the <code>print(type(model.module))</code></p>
<blockquote>
<p>&lt;class ‘torch.nn.parallel.data_parallel.DataParallel’&gt;</p>
</blockquote>
</blockquote>
</aside>
<p>It means that the model is being wrapped into the <code>DataParallel</code> class before, and I don’t know if wrapping it twice will cause any bugs or unexpected behavior.</p>
<p>Also, the recommended way is to use <code>DistributedDataParallel</code> whenever possible instead of <code>DataParallel</code>, since it creates one process per GPU. Here is a <a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" rel="noopener nofollow ugc">link</a> to a tutorial that I found very helpful for learning how to use <code>DistributedDataParallel</code>.<br>
I hope it helps!</p>