Torch model parameters by dictionary type

complexhhs: <p>Hi! guys!</p>
<p>When I coded fully connected layer model  into like below:<br>
(The number of fully connected layer is dependent on “net_info” variable)</p>
<blockquote>
<p>class Model(nn.Module):</p>
<blockquote>
<pre><code>def __init__(self,net_info):
   super(Model,self).__init__()
       self.net_info = net_info
       self.w,self.b = {},{}
     for idx,hidden_size in enumerate(self.net_info):
         if idx == 0:
             self.w[idx] = nn.Parameter(torch.rand(1,hidden_size),requires_grad=True)
            self.b[idx] = nn.Parameter(torch.rand(hidden_size),requires_grad=True)
        else:
           self.w[idx] = nn.Parameter(torch.rand(net_info[idx-1],hidden_size),requires_grad=True)
          self.b[idx] = nn.Parameter(torch.rand(hidden_size),requires_grad=True)
  self.w[idx+1] = nn.Parameter(torch.rand(hidden_size,1),requires_grad=True)
  self.b[idx+1] = nn.Parameter(torch.rand(1),requires_grad=True)
</code></pre>
<p>def forward(self,x):</p>
<blockquote>
<pre><code>  for idx,hidden_size in enumerate(self.net_info):
      x = F.elu(torch.matmul(x,self.w[idx])+self.b[idx])
   x = torch.matmul(x,self.w[idx+1])+self.b[idx+1]
</code></pre>
<p>return x</p>
</blockquote>
</blockquote>
</blockquote>
<p>The problem of above torch model is that does not create “model.parmeters()”</p>
<blockquote>
<p>net_info = [50,60,70,50]<br>
model = Model(net_info)<br>
for p in model.parameters():</p>
<blockquote>
<p>print p</p>
</blockquote>
</blockquote>
<p>Nothing happen, No trainable parameters! Do you have any idea?</p>

ptrblck: <p>Using a Python <code>dict</code> will not properly register the parameters so use <code>nn.ParameterDict</code> instead. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

hamedB: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Regarding this topic, kindly I have a follow up question.<br>
Suppose I define the parameters using paramDic like below, then during the backpropagation, Is the error properly propagate from nth parameters toward 1th parameters in the dictionary? or the gradient of the error w.r.t each parameter is computed independently?<br>
What I want to do is to define a tensor parameter for each layer, and during the loss.backward, the gradient of the error w.r.t. these parameters is propagated from the latest parameters toward the lower one.</p>
<p>Thanks</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
      def __init__(self):
        super(MyModule, self).__init__()
        self.params = nn.ParameterDict({
                '1': nn.Parameter(...),
                '2': nn.Parameter(...),
                 '3': nn.Parameter(...) ,
                     .
                     .
                     .
                 'n': nn.Parameters(...),
        })
</code></pre>
<pre><code class="lang-auto">def forward(self, x):
           for paramKey in self.params.keys() 
                 x = self.params[paramKey] * x
        return x
</code></pre>

ptrblck: <p>The gradient calculation depends on the actual computation so if your computation graph uses the parameters sequentially (as seems to be the case in your example) the gradients will be calculated for each parameter and <code>Autograd</code> will calculate the gradients in the backward pass using the multiplication of the parameters.<br>
Your example would thus work as if you were calling e.g. <code>nn.Linear</code> layers sequentially.<br>
Each internal <code>weight</code> (and <code>bias</code>) parameter will get the corresponding gradient and Autograd will use the chain rule in the backward pass.</p>