Two networks and gradients of parameters of one of those models is none

username: samin_hamidi
cooked: <p>I have trained an autoencoder and saved this trained mode. In another notebook I have uploaded this network:</p>
<blockquote>
<p>def load_checkpoint(filepath, model):<br>
checkpoint = torch.load(filepath)<br>
model.load_state_dict(checkpoint[‘state_dict’])<br>
for parameter in model.parameters():<br>
parameter.requires_grad = True<br>
model.train()<br>
return model</p>
</blockquote>
<p>Here I set</p>
<blockquote>
<p>parameter.requires_grad=True and also model.train()</p>
</blockquote>
<p>Then I define a new network called logits_net and this network is trained with the output images of the pre-trained encoder which has been loaded. Loss is calculated by policy gradient method and then I propagate this loss in both networks. this way:</p>
<blockquote>
<p>params = list(logits_net.parameters()) + list(model.parameters())<br>
opt = Adam(params, lr=lr)</p>
</blockquote>
<p>then</p>
<blockquote>
<p>opt.zero_grad()<br>
batch_loss = compute_loss(logp = torch.as_tensor(batch_logp, dtype=torch.float32),  weights = torch.as_tensor(batch_weights, dtype=torch.float32))<br>
batch_loss.backward()<br>
opt.step()</p>
</blockquote>
<p>After this I check the gradient of autoencoder parameters and they are all none. The other network’s parameters have gradients.</p>
<p>I am wondering why they do not get trained despite the fact that I have set</p>
<blockquote>
<p>parameter.requires_grad=True and also model.train()<br>
and also included the parameters into  the optimizer function.</p>
</blockquote>

username: ptrblck
cooked: <p>The loss computation looks a bit concerning, as it seems you might be recreating tensors and thus detaching them from the computation graph.<br>
What is the type of <code>batch_logp</code> and <code>batch_weights</code>? Is one of these tensors the output of both models (called sequentially)?<br>
Also, could you post the code of <code>compute_loss</code>? If you are detaching the tensors, the <code>batch_loss.backward()</code> call should raise an error, so I’m not sure where exactly the issue comes from.</p>

username: samin_hamidi
cooked: <p>I will add more code so you can take a look at them:<br>
this is my first network: an autoencoder:</p>
<blockquote>
<p>class DEC_AE(nn.Module):<br>
def <strong>init</strong>(self, num_classes, num_features):<br>
super(DEC_AE, self).<strong>init</strong>()<br>
self.dropout = nn.Dropout(p=0.1)<br>
self.fc1 = nn.Linear(28 * 28, 500)<br>
self.fc2 = nn.Linear(500, 500)<br>
self.fc3 = nn.Linear(500, 2000)<br>
self.fc4 = nn.Linear(2000, num_features)<br>
self.relu = nn.ReLU()<br>
self.fc_d1 = nn.Linear(500, 28 * 28)<br>
self.fc_d2 = nn.Linear(500, 500)<br>
self.fc_d3 = nn.Linear(2000, 500)<br>
self.fc_d4 = nn.Linear(num_features, 2000)<br>
self.alpha = 1.0</p>
</blockquote>
<pre><code>    self.pretrainMode = True
    #here we initialize all the weights
    for m in self.modules():
        if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
            torch.nn.init.xavier_uniform_(m.weight)

def setPretrain(self, mode):
    """To set training mode to pretrain or not,
    so that it can control to run only the Encoder or Encoder+Decoder"""
    self.pretrainMode = mode


def forward(self, x):#a batch of images
    x = x.view(-1, 1 * 28 * 28)
    x = self.dropout(x)
    x = self.fc1(x)
    x = self.relu(x)
    x = self.fc2(x)
    x = self.relu(x)
    x = self.fc3(x)
    x = self.relu(x)
    x = self.fc4(x)
    x_e = x 

    # if not in pre_train mode, we need encoder and t distribution output
    if self.pretrainMode is False:
        return x
        

    # encoder is done, followed by decoder
    x = self.fc_d4(x)
    x = self.relu(x)
    x = self.fc_d3(x)
    x = self.relu(x)
    x = self.fc_d2(x)
    x = self.relu(x)
    x = self.fc_d1(x)
    x_de = x.view(-1, 1, 28, 28)
    
    return x_e, x_de
</code></pre>
<p>loading the AE:</p>
<blockquote>
<p>def load_checkpoint(filepath, model):<br>
checkpoint = torch.load(filepath)<br>
<span class="hashtag-raw">#model</span> = checkpoint[‘model’]<br>
model.load_state_dict(checkpoint[‘state_dict’])<br>
for parameter in model.parameters():<br>
parameter.requires_grad = True</p>
</blockquote>
<blockquote>
<pre><code>model.train()
return model
</code></pre>
</blockquote>
<p>and this is the architecture od the second network (I am testing with both sequential and also a model inherited from nn.module ):</p>
<p>def mlp(sizes, activation=nn.Tanh, output_activation=nn.Identity):<br>
# Build a feedforward neural network. outputs are the logits<br>
layers = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for j in range(len(sizes)-1):<br>
act = activation if j &lt; len(sizes)-2 else output_activation<br>
layers += [nn.Linear(sizes[j], sizes[j+1]), act()]<br>
return nn.Sequential(*layers)</p>
<p>and loss is calculated this way:</p>
<blockquote>
<p>logits_net = mlp(sizes=[obs_dim]+hidden_sizes+[n_acts])</p>
</blockquote>
<pre><code class="lang-auto">

</code></pre>
<pre><code class="lang-auto">def get_policy(obs):
        logits = logits_net(obs)
        return Categorical(logits=logits)
def get_action(obs):
        return get_policy(obs).sample().item()
def Logp(obs, act):
        logp = get_policy(obs).log_prob(act)
        return logp
def compute_loss(logp, weights):
        return -(logp * weights).mean()
</code></pre>
<pre><code class="lang-auto">
 opt = MultipleOptimizer(SGD(model.parameters(), lr=1, momentum=0.9), Adam(logits_net.parameters(), lr=lr)

for i, data in enumerate(train_loader):
           
            
            x, label = data
            x = model(x.cuda())
            obs = x.data.cpu().numpy()
            batch_obs.append(obs.copy())
          
            #act in the environment
            act = get_action(torch.as_tensor(obs, dtype=torch.float32))
            
            #log probability
            logp = Logp(torch.as_tensor(obs, dtype=torch.float32),act = torch.as_tensor(act, dtype=torch.int32))
                
          
            rew = reward(obs, act+2)
           

            # save action, reward
            batch_acts.append(act)
            batch_weights.append(rew)#episode rewards
            batch_logp.append(logp)
            
        
        opt.zero_grad()
        batch_logp = torch.stack(batch_logp, dim=0)
       
        batch_loss = compute_loss(logp = torch.as_tensor(batch_logp, dtype=torch.float32), 
                                  weights = torch.as_tensor(batch_weights, dtype=torch.float32))
        batch_loss.backward()
        opt.step()</code></pre>