Access weights of a specific module in nn.Sequential()

mbp28: <p>Hi,</p>
<p>this should be a quick one, but I wasn’t able to figure it out myself.<br>
When I use a pre-defined module in PyTorch, I can typically access its weights fairly easily.<br>
However, how do I access them if I wrapped the module in nn.Sequential() first?<br>
Please see toy example below.</p>
<pre><code>class My_Model_1(nn.Module):
    def __init__(self,D_in,D_out):
        super(My_Model_1, self).__init__()
        self.layer = nn.Linear(D_in,D_out)
    def forward(self,x):
        out = self.layer(x)
        return out

class My_Model_2(nn.Module):
    def __init__(self,D_in,D_out):
        super(My_Model_2, self).__init__()
        self.layer = nn.Sequential(nn.Linear(D_in,D_out))
    def forward(self,x):
        out = self.layer(x)
        return out

model_1 = My_Model_1(10,10)
print(model_1.layer.weight)
model_2 = My_Model_2(10,10)
# How do I print the weights now?
# model_2.layer.0.weight doesn't work.
</code></pre>
<p>Many thanks.<br>
Any help much appreciated.</p>

smth: <pre><code class="lang-auto">model_2.layer[0].weight
</code></pre>

mbp28: <p>Hi! Many thanks, this is what I was looking for. Was trying the wrong braces.</p>

Vaijenath_Biradar: <p>When I am doing this, the error i am getting is "The model has no attribute ‘layer’ ".</p>

ptrblck: <p><a class="mention" href="/u/vaijenath_biradar">@Vaijenath_Biradar</a><br>
<code>layer</code> was defined in <code>__init__</code>:</p>
<pre><code>self.layer = nn.Sequential(nn.Linear(D_in,D_out))
</code></pre>
<p>You have to use the variable name defined in your model.</p>

amirbarghi: <p>Hi,</p>
<p>Is there any way in Pytorch to get access to the layers of a model and weights in each layer without typing the layer name. Something like model.layers in keras which is discussed in the following:<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/50151157/keras-how-to-get-layer-index-when-already-know-layer-name" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/9734424/nguy%e1%bb%85n-c%c3%b4ng-minh" target="_blank" rel="nofollow noopener">
    <img alt="Nguyễn C&amp;#244;ng Minh" src="https://graph.facebook.com/1554042384668580/picture?type=large" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/50151157/keras-how-to-get-layer-index-when-already-know-layer-name" target="_blank" rel="nofollow noopener">Keras: How to get layer index when already know layer name?</a>
</h4>

<div class="tags">
  <strong>python, keras</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/9734424/nguy%e1%bb%85n-c%c3%b4ng-minh" target="_blank" rel="nofollow noopener">
    Nguyễn C&amp;#244;ng Minh
  </a>
  on <a href="https://stackoverflow.com/questions/50151157/keras-how-to-get-layer-index-when-already-know-layer-name" target="_blank" rel="nofollow noopener">09:10AM - 03 May 18</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

Rick_Sanchez: <p>When I using this method with model.eval() I getting different values of weights for the same example.<br>
Is that mean that my model don’t work correctly ?</p>

saba: <p>Hi ptrblck</p>
<p>Happy to find you here.<br>
I am building 2 CNN layers with 3 FC layers and using drop out two times.<br>
My neural network is defined as follow: Do you see any thing wrong in that? I appreciate your feedback.</p>
<p>import torch<br>
import torchvision<br>
import torchvision.transforms as transforms<br>
from torch.utils.data import TensorDataset, DataLoader<br>
import torch.optim as optim<br>
import torch.nn as nn<br>
from torch.utils.data.dataset import random_split<br>
from torch.nn import functional as F<br>
import matplotlib.pyplot as plt<br>
from torch.autograd import Variable</p>
<p>class ConvNetRedo1(nn.Module):<br>
def <strong>init</strong>(self,numf1,numf2,fz1,fz2,nn2,nn3): # numf1( nnumber of filters first layer)numf2(nnumber of filters first layer)),fz1 kernel size(),fz2,nn2,nn3<br>
super(ConvNetRedo1, self).<strong>init</strong>()<br>
self.numf1=numf1<br>
self.numf2=numf2<br>
self.fz1=fz1<br>
self.fz2=fz2<br>
self.nn2=nn2<br>
self.nn3=nn3<br>
self.layer1 = nn.Sequential(nn.Conv3d(1, self.numf1, kernel_size=self.fz1, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))<br>
self.layer2 = nn.Sequential(nn.Conv3d(self.numf1,self.numf2, kernel_size=self.fz2, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))<br>
self.fc1 = nn.Linear(3072, self.nn2) #<span class="hashtag">#3027</span><br>
self.fc2 = nn.Linear( self.nn2, self.nn3) # FULLY CONNECTED LAYERS<br>
self.fc3 = nn.Linear( self.nn3, 1) # FULLY CONNECTED LAYERS<br>
self.relu = nn.ReLU()                          # Non-Linear ReLU Layer: max(0,x)<br>
self.sigmoid = nn.Sigmoid()<br>
self.drop_out1 = nn.Dropout(0.5)<br>
self.drop_out2 = nn.Dropout(0.5)<br>
self.Relu=nn.LeakyReLU(0.1, inplace=True)</p>
<pre><code>def forward(self, x):


    x=x.unsqueeze(1).float()
    out = self.layer1(x)
    
    out = self.layer2(out)
    
    out = out.view(out.size(0), -1)
    
    out = self.fc1(out)
    out=self.drop_out1(out)
    out=self.relu(out)

    out = self.fc2(out)
    out=self.drop_out2(out)
    out=self.relu(out)
    
    out = self.fc3(out)
    out = self.sigmoid(out)
    return out</code></pre>

ptrblck: <p>I’m not sure, if the number of features are correctly defined without knowing the input shape, but skimming through the model definition, I cannot find any obvious mistakes.<br>
Are you seeing any issues with the model?</p>

saba: <p>Hi<br>
Many thanks for your reply.<br>
The size of the features are good. I just want to know the way of designing, is  dropout used in the good step? Is it better to use after Relu?</p>

ptrblck: <p>It doesn’t matter if dropout is applied before or after the relu. I cannot see any obvious mistakes.</p>

saba: <p>Many thanks for your reply. Sorry I am transferring my model and data and labels to the GPU.I am not sure if I should transfer criterion and optimizer to the GPU or not?</p>
<p>I used them in this way</p>
<p>criterion = nn.BCELoss()</p>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=.03)</p>
<p>I appreciate your help</p>
<p>Do u know any books or links which is usable?</p>
<p>S</p>

ptrblck: <p>You could transfer the criterion to the GPU just to avoid possible issues, but it shouldn’t be necessary for <code>nn.BCELoss</code>.<br>
One minor advice: I would remove the last sigmoid in your model and use <code>nn.BCEWithLogitsLoss</code> instead, as it will be numerically more stable.</p>
<p>Check out “Deep Learning with PyTorch” by <a class="mention" href="/u/lantiga">@lantiga</a>, <a class="mention" href="/u/elistevens">@elistevens</a>, and <a class="mention" href="/u/tom">@tom</a>, which can be downloaded for free on the <a href="https://pytorch.org/deep-learning-with-pytorch">official website</a>.<br>
(It’s not the full book if I’m not mistaken, as it’s still work in progress <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> )</p>

saba: <p>Means finished the model to<br>
out = self.fc3(out)<br>
and use <code>nn.BCEWithLogitsLoss</code> . it has the in-built  sigmoid in it?</p>

saba: <pre><code class="lang-auto">def forward(self, x):


    x=x.unsqueeze(1).float()
    out = self.layer1(x)
    
    out = self.layer2(out)
    
    out = out.view(out.size(0), -1)
    
    out = self.fc1(out)
    out=self.drop_out1(out)
    out=self.relu(out)

    out = self.fc2(out)
    out=self.drop_out2(out)
    out=self.relu(out)
    
    out = self.fc3(out)

    return out

and 

criterion=nn.BCEWithLogitsLoss()    indeed the input of the this loss function should be the out put from linear layer ?</code></pre>

ptrblck: <p>Yes, that’s the correct usage.</p>

saba: <p>Dear ptrblck</p>
<p>I read now the book that you suggest, I am really confused in some cases during training I saw (optimizer.zero_grad()) is used before getting output and applying model and in this book ( lantiga)  it is after getting the output?!<br>
which one is correct?</p>
<p>t_p = model(t_un, *params)<br>
loss = loss_fn(t_p, t_c)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step( )</p>

ptrblck: <p>It depends on your coding style, but it should be called before <code>loss.backward()</code> in case you don’t want explicitly to accumulate the gradients (which is a valid use case, but not the usual work flow).<br>
I personally try to add it right at the beginning of the training loop, since I think I remember it better, but I still forget it from time to time. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

saba: <p>I really appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>you are really helping me</p>

saba: <p>Dear ptrblck</p>
<p>I used the thing that we discussed but the outputs are more than 1 <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> !!</p>