Custom loss function has no attribute backward

flyingpig: <p>I am just starting to learn Pytorch, and trying to build a custom loss function</p>
<pre><code class="lang-auto">class distance_loss(torch.nn.Module):
  def __init__(self, in_features = 10, out_features = 1):
    super(distance_loss, self).__init__()
    self.in_features = in_features
    self.out_features = out_features
    self.center = torch.nn.Parameter(torch.randn(in_features))
    self.center.requires_grad = True

  def forward(self, input):
    return torch.norm(input - self.center, 2)
</code></pre>
<p>Then I want to use it in some model:</p>
<pre><code class="lang-auto">model = SomeModel()

loss = distance_loss()

scores = model(torch.rand(1,3,32,32))

optimizer = torch.optim.SGD(model.parameters(), lr = 0.001 )

optimizer.zero_grad()

loss.backward()

optimizer.step()
</code></pre>
<p>But this returns error: “distance_loss” object has no attribute ‘backward’. It is my understanding that backward method is automatically implemented in nn.module. What goes wrong here? Also do I somehow need to append the learnable parameters of distance_loss() to model.parameters() for optimizer to know?</p>

bapi: <p><code>.backward()</code> is done once you have a computation graph created; where is your computation? You can see a small use case here</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.tensor([[1., -1.], [1., 1.]], requires_grad=True)
&gt;&gt;&gt; out = x.pow(2).sum()
&gt;&gt;&gt; out.backward()
&gt;&gt;&gt; x.grad
tensor([[ 2.0000, -2.0000],
        [ 2.0000,  2.0000]])
</code></pre>
<p>on the page <a href="https://pytorch.org/docs/stable/tensors.html" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html</a> .</p>

flyingpig: <p>Sorry, I missed one line in the second part of the code and now I have edited it.</p>

albanD: <p>Hi,</p>
<p>I think you have two things called loss here that are confusing.<br>
The <code>loss</code> in your code is actually the loss module you defined. You need to call it with the input to get the loss value from your batch. Then you will be able to call <code>.backward()</code> on that.</p>

flyingpig: <p>Hi. Thank you for answering!</p>
<p>I changed it to something like this:</p>
<pre><code class="lang-auto">model = SomeModel()

x = torch.rand(1,3,32,32)

scores = model(x)

loss_func = distance_loss(10)

loss = loss_func.forward(scores)

loss.backward()
</code></pre>
<p>but x.grad now prints an empty list</p>

bapi: <aside class="quote no-group" data-username="flyingpig" data-post="5" data-topic="90425">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flyingpig/48/27083_2.png" class="avatar"> flyingpig:</div>
<blockquote>
<pre><code class="lang-auto">x = torch.rand(1,3,32,32)
</code></pre>
</blockquote>
</aside>
<p>Replace <code>x = torch.rand(1,3,32,32)</code> with <code>x = torch.rand(1,3,32,32, requires_grad=True)</code>.</p>

flyingpig: <p>Thank you!! Is there a way that I can integrate the gradient of the loss into the optimizer so that it will perform gradient descent on the learnable parameters of my cusom defined loss function together with the parameters from model?</p>

bapi: <p>Putting loss inside the model definition? For example, see <a href="https://discuss.pytorch.org/t/trainable-variable-in-loss-function-and-matrix-multiplication/495/6" class="inline-onebox">Trainable Variable in Loss Function and Matrix Multiplication</a></p>