Gradients not being calculated as expected

username: Felipe_Ramirez_Lastr
cooked: <p>In the following example, the gradient values for critic.bias.grad and critic.weight.grad are zero, but the loss is -0.5. I set up the custom loss to make sure there was nothing funny happening with mse_loss but using mse_loss gives the same result. Why is this the case? I would expect bias grad to be 1 because:<br>
Error = ((target_output - output)^2)/2, so dE/dY would be 2(out[1]-1-out[1])^2)*(-1)/2 = 1</p>
<p>Any help that guides me into why this happens would be very useful to be able to implement custom layers successfully in pytorch.</p>
<p>import torch.nn as nn<br>
import torch</p>
<p>class CartPoleNN(nn.Module):<br>
def <strong>init</strong>(self, input_size):<br>
super(CartPoleNN, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(input_size, 20)  # Fully connected layer 1<br>
self.fc1.weight.data.uniform_(-0.4, 0.4)<br>
self.fc1.bias.data.uniform_(-0.4, 0.4)</p>
<pre><code>    # Actor network
    self.actor = nn.Sequential(
        nn.Linear(20, 2),
        nn.Softmax(dim=-1)  # Apply softmax to get action probabilities
    )

    for layer in self.actor:
        if isinstance(layer, nn.Linear):
            layer.weight.data.uniform_(-0.4, 0.4)
            layer.bias.data.uniform_(-0.4, 0.4)

    # Critic network
    self.critic = nn.Sequential(
        nn.Linear(20, 1),
    )

    for layer in self.critic:
        if isinstance(layer, nn.Linear):
            layer.weight.data.uniform_(-0.4, 0.4)
            layer.bias.data.uniform_(-0.4, 0.4)

def forward(self, x):
    x = self.fc1(x)  # Pass through the first fully connected layer
    action_probs = self.actor(x)
    critic_value = self.critic(x)
    return action_probs, critic_value
</code></pre>
<p>class CustomLoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CustomLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, inputs, targets):
    loss = -0.5 * (targets - inputs)**2
    return loss.sum()
</code></pre>
<p>torch.manual_seed(0)<br>
torch.set_grad_enabled(True)<br>
model = CartPoleNN(4)<br>
Inputs = torch.tensor([0.0132, -0.2175, -0.0469, 0.2295])<br>
out = model(Inputs)<br>
Values = out[1]<br>
AdvantagePY = out[1]<br>
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)<br>
Loss = CustomLoss()<br>
CriticLoss = Loss(out[1], out[1]-1)<br>
model.critic.zero_grad()  # Zero the gradients<br>
CriticLoss.backward()<br>
print(model.critic[0].bias.grad)</p>

username: soulitzer
cooked: <p>The loss here is always <code>-0.5 * (out[1] -1  - out[1])**2 = -0.5</code><br>
Since 0.5 is constant the derivative and the derivative wrt a constant to anything is 0, the derivative of loss wrt to any of the parameters is 0.</p>

username: Felipe_Ramirez_Lastr
cooked: <p>Hi soulitzer:<br>
Thanks for your kind reply. As you point out, the derivative will be zero in this case because it’s a constant. In my original example I had (taget_output - expected_output) = 1 but target_output was not = out[1] -1. To simplify the post and recreate the value I used this example, which turns the loss into a constant and doesn’t help show my issue at all, so I apologize for wasting your time.<br>
It turns out that the issue is with scaling the error before passing it to the loss.<br>
When I have Advantage = (Reward - Values) (with both being vectors) I’m trying to use<br>
Advantage = Advantage/Advantage.abs().max() to normalize gradient values at [-1,1], this is what’s producing the unexpected gradient values for me. I’m trying to scale the gradients based on the value in advantage = (target_output - critic_output), and with a manual implementation I did elsewhere I’m able to scale the gradients like this, but it seems pytorch does not allow this probably due to the way autograd takes the loss. I will look into it further and maybe create a new post (this time with an acceptable non-constant working example).<br>
Appreciate your help, and again sorry for the time waste.</p>