No gradient calculated for custom loss function

Yihong_Li: <p>Hi, I designed my custom loss function all using pytorch operations, but I cannot get the gradient when I call the backward( ) function. Can anybody help me to fix this? Many thanks.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af97cafe6482472165e467bf512da0b547247511.png" data-download-href="https://discuss.pytorch.org/uploads/default/af97cafe6482472165e467bf512da0b547247511" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af97cafe6482472165e467bf512da0b547247511_2_690x333.png" alt="image" data-base62-sha1="p3mLSnFExIbHUZ27tu3HxcfrkL7" width="690" height="333" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af97cafe6482472165e467bf512da0b547247511_2_690x333.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af97cafe6482472165e467bf512da0b547247511_2_1035x499.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af97cafe6482472165e467bf512da0b547247511_2_1380x666.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1658×802 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The gradient is zero when I used my custome loss function (log_rmse). If I switch to the standard nn.MSELoss function, I can do backprop. But I can’t figure out what’s wrong with my custom loss function.</p>

ptrblck: <p>Could you post the custom loss function by wrapping it into three backticks ```, please, as it would make debugging easier?</p>

Yihong_Li: <p>Okay, here is my loss function:</p>
<pre><code class="lang-auto">def log_rmse(y, y_hat):
    clipped_preds = torch.clamp(y_hat, 1, float('inf'))
    log_y_hat = torch.sum((torch.log(clipped_preds) - torch.log(y)) ** 2)
    rmse = torch.sqrt(log_y_hat / y.shape[0])
    return rmse
</code></pre>
<pre><code class="lang-auto">def train_network(net, train_data, train_label, val_data, val_label, iteration, lr, batch_size):
    train_ls, val_ls = [], []
    optimizer = torch.optim.Adam(net.parameters(), lr=lr)
    dataset = data.TensorDataset(train_data, train_label)
    mse = nn.MSELoss()
    for i in range(iteration):
        data_iter = data.DataLoader(dataset, batch_size=batch_size, shuffle=True)
        for X, y in data_iter:
            optimizer.zero_grad()
            loss = log_rmse(y, net(X))
            loss.backward()
            optimizer.step()
        epoch_loss = log_rmse(train_label, net(train_data))
        train_ls.append(epoch_loss)
        if val_data is not None and val_label is not None:
            val_ls.append(log_rmse(val_label, net(val_data)))
    return train_ls, val_ls
</code></pre>

ptrblck: <p>Thanks for the update!<br>
Your method works for me and calculates the gradients for the outputs which are not clipped as expected:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10)
data = torch.randn(1, 10)
target = torch.rand(1, 10)

output = model(data)
print(output)
# tensor([[ 0.9157, -0.3162, -0.6357,  0.5272, -0.9480,  1.1473,  0.3734,  0.1775,
#           0.4148, -0.9100]], grad_fn=&lt;AddmmBackward0&gt;)
loss = log_rmse(target, output)
print(loss)
# tensor(2.3577, grad_fn=&lt;SqrtBackward0&gt;)

loss.backward()
print(model.weight.grad)
# tensor([[-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.2285,  0.1268, -0.1046,  0.0679,  0.2792,  0.0948, -0.3489,  0.1557,
#          -0.0147,  0.0082],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000],
#         [-0.0000,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,  0.0000,
#          -0.0000,  0.0000]])
</code></pre>

Yihong_Li: <p>Thank you so much for the feedback. I think I got zero gradient because when I called the torch.clamp() function during forward prop, because my initial weight is very small, most values are clamped to one and lost gradient. When I initialized larger weight in my neural network, the back prop could work.</p>