Parameters initialised by nn.Parameter not present in the model.parameters()

username: durv
cooked: <p>Hi,<br>
I have defined the weight parameters as follows but still these trainable parameters are not listed in the model.parameters().</p>
<pre><code> class Module(nn.Module):
    def __init__(input_dim, output_dim)
    #some variables

    def build(self):
       self.wt_dict = nn.ParameterDict()
       # self.wt_dict = {}
       for i in range(3):
          self.wt_dict["weight_%s"%i] = nn.Parameter(torch.FloatTensor(self.input_dim,self.output_dim))            
          for j in range(14):
             self.wt_dict["weight_left_%s%s"%(i,j)] = nn.Parameter(torch.FloatTensor(self.output_dim,1))
             self.wt_dict["weight_right_%s%s"%(i,j)] = nn.Parameter(torch.FloatTensor(self.output_dim,1))
</code></pre>
<p>I have tried it using the ParameterDict() and also with the standard dict, but the parameters are not reflected in the model parameters.<br>
Even used the self.register_parameter() to register the parameters but with no success.</p>
<p>Any help would be appreciated.<br>
Thanks!</p>

username: ptrblck
cooked: <p>Your code works fine after adding the <code>self.build()</code> call and the missing <code>super().__init__()</code>:</p>
<pre><code class="lang-python">class Module(nn.Module):
   def __init__(self):
       super().__init__()
       self.build()

   def build(self):
      self.wt_dict = nn.ParameterDict()
      for i in range(3):
         self.wt_dict["weight_%s"%i] = nn.Parameter(torch.randn(1, 1)) 
         for j in range(14):
            self.wt_dict["weight_left_%s%s"%(i,j)] = nn.Parameter(torch.randn(1, 1))
            self.wt_dict["weight_right_%s%s"%(i,j)] = nn.Parameter(torch.randn(1, 1))
    
model = Module()
print(dict(model.named_parameters()))
&gt; {'wt_dict.weight_0': Parameter containing:
  tensor([[-0.6057]], requires_grad=True), 'wt_dict.weight_left_00': Parameter containing:
  tensor([[1.0553]], requires_grad=True), 'wt_dict.weight_right_00': Parameter containing:
  ...
</code></pre>

username: durv
cooked: <p>However, when I call the “Module” from other custom class, these parameters do not appear in the list.</p>
<p>So if I want to call the function “Module2” from some other custom class</p>
<pre><code>  self.conv = Module1(input_dim, output_dim, support1, support2)
  

   class Module1(nn.Module):
       def __init__(self, input_dim, output_dim, support1, support2, iter):
           super(Module1, self).__init__()
              self.input_dim = input_dim
              self.output_dim = output_dim
              self.support1 = support1
              self.support2 = support2
              self.iter = iter
              self.build()

     def build(self):
        self.dict = {}
        for i in range(self.iter):
             self.dict["no_%s"%i] = Module2(input_dim=self.input_dim,
                                                    output_dim=self.output_dim,
                                                    support1=self.support1,
                                                    support2=self.support2,
                                                    )
class Module2(nn.Module):
    def __init__(self, self, input_dim, output_dim, support1, support2):
       super(Module2,self).__init__()
       self.build()

  def build(self):
     self.wt_dict = nn.ParameterDict()
     for i in range(3):
        self.wt_dict["weight_%s"%i] = nn.Parameter(torch.randn(output_dim, input_dim)) 
        for j in range(14):
           self.wt_dict["weight_left_%s%s"%(i,j)] = nn.Parameter(torch.randn(output_dim, 1))
           self.wt_dict["weight_right_%s%s"%(i,j)] = nn.Parameter(torch.randn(output_dim, 1))
</code></pre>
<p>Are there any things I need to keep in mind to have the parameters from the custom layers to be present in the model parameters when finally we call the model.??</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>In <code>Module1</code> you are assigning instances of <code>Module2</code> to a plain Python <code>dict</code>, which won’t register the parameters, so you would have to use <code>nn.ModuleDict</code> instead.</p>

username: durv
cooked: <p>Yes, Thank you it worked for the plain dict part.</p>
<p>So for assigning instances of Module2 in module1, and assigning Module1 instance from some other class (apart from a dictionary),  nn.ModuleList should be used.</p>
<p>Eg: -</p>
<pre><code>self.mod1 = nn.ModuleList(Module2(input_dim=self.input_dim,
                                                output_dim=self.output_dim,
                                                support1=self.support1,
                                                support2=self.support2,))

self.mod2 = nn.ModuleList(Module1(input_dim, output_dim, support1, support2))
</code></pre>
<p>Thanks.</p>

username: ptrblck
cooked: <p>Yes, plain Python “containers” such as <code>list</code>, <code>dict</code> etc. won’t (recursively) register the parameters and buffers, so you should use the PyTorch equivalents instead such as <code>nn.ParameterList</code>, <code>nn.ModuleList</code>, <code>nn.ModuleDict</code> etc.</p>

username: durv
cooked: <p>Thank you so much !!!</p>

username: durv
cooked: <p>If we are initialising a class instance of Module 1 in Module 2’s forward layer, then Module1’s parameters along with Layer 1 and Layer2’s don’t get registered.</p>
<pre><code>class Layer1(nn.Module):
  def __init__(self, a, n, dim):
    super(Layer1, self).__init__()
    self.a = a
    self.n = n
    self.dim = dim
    self.w_input = nn.Parameter(torch.FloatTensor(self.dim, self.n,1,1))
    if a:
      self.wt = nn.Parameter(torch.FloatTensor(2 * self.dim, n, 1))

class Layer2(nn.Module):
  def __init__(self, n, dim):
    super(Layer1, self).__init__()
    self.n = n
    self.dim = dim
    self.g = nn.Parameter(torch.ones((1,dim,1,n)))


class Module1(nn.Module):
  def __init__(self, a, n, dim):
    super(Module1, self).__init__()
    self.a = a
    self.n = n
    self.dim = dim
    self.layer1 = Layer1(a, dim, dim)
    self.layer2 = Layer2(n, dim)

def forward(self, input):
    x = self.layer1(input)
    x = self.layer2(x)
    return x


 class Module2(nn.Module):
    def __init__():
       super(Module2, self).__init__()
       self.a = a
       self.n=n
       self.dim = dim
       self.layer3 = Layer3(a,n)
       
    def forward(self, input):
       x = self.layer3(input)
       inp_size = list(x.size())
       _, _, _, ndim = inp_size[0],inp_size[1],inp_size[2],inp_size[3]
       k = 2*(ndim-self.dim)
       if(k&gt;1):
          layer_output = Module1(self.a, k, self.dim)
          output = layer_output(x)
       return output
</code></pre>
<p>The value of k (in Module2 forward layer) can only be determined from the values obtained in this forward layer, so we cannot define Module 1 instance in the init layer of Module 2.</p>
<p>So how do we register the parameters of the instance initialised in the forward layer?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You would have to assign the module to <code>self</code> in the forward.<br>
Also, make sure to register this module only once. Otherwise you would be overwriting the module and no training will be performed.<br>
Since you are now defining new models after the <code>__init__</code>, you should also take care of passing these new parameters to the optimizer after all modules were created in the <code>forward</code> method.</p>

username: durv
cooked: <p>That was very helpful, thank you!</p>