Why didn't loss.backward() update the network's grads?

Jingyang_Zhang: <p>Hi, recently I used the <em><strong>DDPG agent</strong></em> based on PyTorch to build a project.</p>
<p>When I was trying to read a batch of combinations of (state, action, reward, next_state) from the buffer and to update the weights of the 4 networks, it seemed loss.backward() function didn’t update the grads at all! There is just no data shown on the tensorboard page.</p>
<p>Here is the code for “update based on the buffer” part:</p>
<pre><code class="lang-auto">def learn(self, adj):
    if self.buffer.mem_cntr &lt; self.batch_size:
        return
    state_vectors, action_vectors, rewards, next_state_vectors = \
                               self.buffer.sample_buffer(self.batch_size)

    adj = T.tensor(adj, dtype = T.float32).to(self.actor.device)
    for i in range(self.batch_size):
        state_vector = T.tensor(state_vectors[i], dtype = T.float32).unsqueeze(0).to(self.actor.device)
        next_state_vector = T.tensor(next_state_vectors[i], dtype = T.float32).unsqueeze(0).to(self.actor.device)
        action_vector = T.tensor(action_vectors[i], dtype = T.float32).unsqueeze(0).to(self.actor.device)
        reward = T.tensor(rewards[i], dtype = T.float32).to(self.actor.device)

        next_target_action_vector = self.target_actor.forward(next_state_vector, adj)
        next_target_q = self.target_critic(next_state_vector, next_target_action_vector, adj)
        q = self.critic.forward(state_vector, action_vector, adj)

        target_q = reward + self.gamma * next_target_q
    
        self.critic.optimizer.zero_grad()
        critic_loss = F.mse_loss(q, target_q)
        critic_loss.backward()
        self.critic.optimizer.step()
            
        self.actor.optimizer.zero_grad()
        actor_loss = -self.critic.forward(state_vector, 
                                              self.actor.forward(state_vector, adj), 
                                              adj) / self.batch_size
        actor_loss.backward()
        self.actor.optimizer.step()

    self.soft_update()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/0502cb48b929ffb8175613ecddd7d036d60f0a88.png" data-download-href="https://discuss.pytorch.org/uploads/default/0502cb48b929ffb8175613ecddd7d036d60f0a88" title="Screenshot from 2020-09-16 16-04-20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0502cb48b929ffb8175613ecddd7d036d60f0a88_2_461x500.png" alt="Screenshot from 2020-09-16 16-04-20" data-base62-sha1="IknagjsSAyBN6d2uA5rIVGjUtW" width="461" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0502cb48b929ffb8175613ecddd7d036d60f0a88_2_461x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/5/0502cb48b929ffb8175613ecddd7d036d60f0a88.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/5/0502cb48b929ffb8175613ecddd7d036d60f0a88.png 2x" data-dominant-color="F7E3DC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-09-16 16-04-20</span><span class="informations">685×742 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
How to modify the codes to make the actor-network work correctly?</p>

liuruiqi1107: <p>That’s not how batch samples works. You need to sample a batch of samples simultaneously and update the actor/critic network by these samples. The loss of the actor is the mean of critic loss on these samples, not just decided by the batch_size. Hope I made my point, good luck!</p>

Jingyang_Zhang: <p>Hi Liu, thank you for your reply! I know your point but for some reason, I have to replace some layers in the original actor and critic network, the networks consist of GCN(Graph Convolution Network) layers and linear layer, instead of mere Linear layers now. For GCN layers, I have two inputs: state_vector, which is a 1XN tensor, and an adjacency matrix, which is an NXN tensor. Inside the layer, the following computation is performed:</p>
<pre><code class="lang-auto">[weight matrix]     X      [input]        X         [adjacency matrix]
(out_dim X in_dim)  X    (in_dim X N)     X               (N X N)
</code></pre>
<p>and the output is a (out_dim X N) tensor, then this output is passed to the next GCN layer<br>
As you can see, the first input(state_vector) is an 1 X N tensor, but the batch tensor is a (batch_size X N) tensor, which makes it impossible to pass it to the GCN layer, this is the reason I have to use a for loop to extract a single tuple of (state, action, reward, next_state) to make sure the dimensions match.<br>
If I don’t update the weights in each loop, does that count as “simultaneously”?</p>
<p>To be honest, I am not sure if I have written the correct code to perform loss.backward() and optimizer.step()</p>
<p>Can I recode in this way?</p>
<pre><code class="lang-auto">self.critic.optimizer.zero_grad()
self.actor.optimizer_zero_grad()

# for loop to extract tuples and perform loss.backward()

self.critic.optimizer.step()
self.actor.optimizer.step()
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji only-emoji" alt=":frowning:"></p>

liuruiqi1107: <p>Hi, you are welcome. You don’t need to worry about the dimension, because PyTorch will consider the first dimension of the input to the network as <code>BATCH</code> dimension (as long as your network is inherited from <code>nn.Module</code>). For example, the input size of your network is <code>1xD</code>, and you feed <code>BxD</code> arrays to your network, that will definitely work well, PyTorch will consider <code>B</code> as the <code>BATCH</code> dimension. And you don’t need to recode, the original code is ok. Just fix the batch update part. Hope that helps, Good luck!</p>

Jingyang_Zhang: <p>Hi Liu,</p>
<p>I did as you say, but it did show dimension mismatch:</p>
<pre><code class="lang-auto">
  File "/home/yubi/practice/GCN_RL_Circuit_Project/circuit_rl_experiment/stage5_gcn_ddpg/main_circuit_gcn_ddpg.py", line 85, in &lt;module&gt;
    agent.learn(normalized_adj)

  File "/home/yubi/practice/GCN_RL_Circuit_Project/circuit_rl_experiment/stage5_gcn_ddpg/gcn_ddpg_agent.py", line 221, in learn
    target_actions = self.target_actor.forward(states_, adj)

  File "/home/yubi/practice/GCN_RL_Circuit_Project/circuit_rl_experiment/stage5_gcn_ddpg/gcn_ddpg_agent.py", line 61, in forward
    x = self.gc1(x, adj)

  File "/home/yubi/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)

  File "/home/yubi/practice/GCN_RL_Circuit_Project/circuit_rl_experiment/stage5_gcn_ddpg/gcn_layers.py", line 37, in forward
    support = torch.mm(self.weight, input)

RuntimeError: size mismatch, m1: [18 x 1], m2: [32 x 9] at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THC/generic/THCTensorMathBlas.cu:290
</code></pre>
<p>and this is my GCN layer code:</p>
<pre><code class="lang-auto">import math

import torch

from torch.nn.parameter import Parameter
from torch.nn.modules.module import Module


class GraphConvolution(Module):
    """
    Simple GCN layer, similar to https://arxiv.org/abs/1609.02907
    """

    def __init__(self, in_features, out_features, bias=True):
        super(GraphConvolution, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        # self.weight = Parameter(torch.FloatTensor(in_features, out_features))
        self.weight = Parameter(torch.FloatTensor(out_features, in_features))
        if bias:
            # self.bias = Parameter(torch.FloatTensor(out_features))
            self.bias = Parameter(torch.FloatTensor(out_features, 1))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        stdv = 1. / math.sqrt(self.out_features)
        self.weight.data.uniform_(-stdv, stdv)
        if self.bias is not None:
            self.bias.data.uniform_(-stdv, stdv)

    def forward(self, input, adj):
        #support = torch.mm(input, self.weight)
        #output = torch.spmm(adj, support)
        
        support = torch.mm(self.weight, input)
        output = torch.mm(support, torch.transpose(adj, 0, 1))
        if self.bias is not None:
            return output + self.bias
        else:
            return output

    def __repr__(self):
        return self.__class__.__name__ + ' (' \
               + str(self.in_features) + ' -&gt; ' \
               + str(self.out_features) + ')'
</code></pre>