Backward() for costume loss function

username: netaglazer
cooked: <p>hi,<br>
i created a loss function, and it returns a float<br>
i get the error ‘float’ object has no attribute ‘backward’<br>
which is makes sense…</p>
<p>if i return tensor(some_float)<br>
i get the error:<br>
element 0 of tensors does not require grad and does not have a grad_fn<br>
how can i handle it?</p>

username: G.M
cooked: <p>U have to return a FloatTensor that is the result of some operation of ur loss function, so ur loss function won’t break the gradients graph. To be more specific, can u provide an example code of ur loss function?</p>

username: netaglazer
cooked: <p>the function:</p>
<blockquote>
<p>def loss_function(p, y):<br>
losses = 0<br>
grad_loss = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in range(len(p)):<br>
temp_p = torch.tensor(p[i])<br>
temp_y = torch.tensor(y[i])<br>
k = len(temp_y)<br>
loss1 = -tensot_mul(vec_minus(one_Vector(k),y[i]), list(np.log(vec_minus(one_Vector(k),p[i]))))<br>
l = y[i]*p[i].log()<br>
prod = vec_minus(vec_minus(one_Vector(k),y[i]), - l)<br>
loss2 = loss1 + sum(softmax(vec_minus(one_Vector(k),prod)))<br>
losses = losses + loss2<br>
returnfloat(losses/(len(p)))</p>
</blockquote>
<p>p - prediction<br>
y - target</p>

username: G.M
cooked: <p>As I said, I think in this code u r returning a Python float object, and that doesn’t have a <code>.backward()</code> method.<br>
I’m unclear about what the <code>tensor_mul</code>, <code>vec_minus</code> are, but the operations u apply to the outputs should all the Pytorch operations, involving some numpy operations would probably break the graph(I’m not sure).<br>
If u return a <code>FloatTensor</code> like u described. The tensor would be a “Leaf variable”(I’m not sure what exactly it should be called), and these kind of variable does not have gradients because u created them out of a <code>float</code>, which holds no gradients information.</p>

username: netaglazer
cooked: <p>im sorry, it didnt work…</p>
<p>i changed the function, and i also tried to add Variable()</p>
<p>def loss_function(p, y, b):<br>
losses = 0<br>
for i in range(b):<br>
temp_p = torch.tensor(p[i])<br>
temp_y = torch.tensor(y[i])<br>
k = len(temp_y)<br>
loss1 = -(torch.ones(k).cuda()-temp_y)@((torch.ones(k).cuda()-temp_p).log())<br>
l = temp_y*(temp_p.log())<br>
prod = (torch.ones(k).cuda()-temp_y) - l<br>
loss2 = loss1 + torch.max(torch.ones(k).cuda()-prod)<br>
losses = loss1 + loss2<br>
f = (losses/(b))<br>
return Variable(torch.FloatTensor([f]))</p>
<p>i still get:</p>
<p>element 0 of tensors does not require grad and does not have a grad_fn</p>

username: G.M
cooked: <p>what is the type of “p” and “y”?</p>

username: netaglazer
cooked: <p>p,y - tensor<br>
b - int (batch size)</p>

username: G.M
cooked: <p>can u reformat the code? I’m having trouble distinguishing what is inside the loop.</p>

username: tux
cooked: <p>Are you sure you have set <code>tensor.requires_grad()=True</code> ?</p>

username: netaglazer
cooked: <blockquote>
<p>def loss_function(p, y, b):<br>
—losses = 0<br>
—for i in range(b):<br>
------temp_p = torch.tensor(p[i])<br>
------temp_y = torch.tensor(y[i])<br>
------k = len(temp_y)<br>
------loss1 = -(torch.ones(k).cuda()- temp_y)@((torch.ones(k).cuda()-temp_p).log())<br>
------l = temp_y*(temp_p.log())<br>
------prod = (torch.ones(k).cuda()-temp_y) - l<br>
------loss2 = loss1 + torch.max(torch.ones(k).cuda()-<br>
prod)<br>
------losses = loss1 + loss2<br>
—f = (losses/(b))<br>
—return Variable(torch.FloatTensor([f]))</p>
</blockquote>

username: netaglazer
cooked: <p>Where should i define it?</p>

username: tux
cooked: <aside class="quote no-group" data-username="netaglazer" data-post="6" data-topic="67994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f1d935/48.png" class="avatar"> netaglazer:</div>
<blockquote>
<p>temp_p = torch.tensor(p[i])<br>
temp_y = torch.tensor(y[i])</p>
</blockquote>
</aside>
<p>I would say there :</p>
<pre><code class="lang-auto">temp_p = torch.tensor(p[i], requires_grad=True)
temp_y = torch.tensor(y[i], requires_grad=True)
</code></pre>
<p>By default, the requires_grad are set to False. You need to activate them yourself. More on this here : <a href="https://pytorch.org/docs/stable/notes/autograd.html" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 2.1 documentation</a></p>

username: tux
cooked: <p>May be you should do it even before … At the beginning of your function I would try:</p>
<pre><code class="lang-auto">p = torch.tensor(p, requires_grad=True)
y = torch.tensor(y, requires_grad=True)
</code></pre>

username: netaglazer
cooked: <p>i get<br>
Only Tensors of floating point dtype can require gradients</p>

username: G.M
cooked: <ol>
<li>
<code>Variable</code> is now deprecated.</li>
<li>When applying the tensor to some linear transformation with scalers, u can just use Python value. For example <code>a: tc.Tensor = 1 - b</code>
</li>
<li>Using torch.tensor(OLD_TENSOR) will not pass the gradients of the OLD_TENSOR to the new tensor.</li>
</ol>
<p>Note: u should read more about pytorch docs and the autograd mechanism.</p>

username: netaglazer
cooked: <p>ok<br>
tnx for your note, i will do it</p>
<p>right now, i have this</p>
<pre><code class="lang-auto">def loss_function(p, y, b):
    losses = 0
    for i in range(b):
        k = len(y[i])
        loss1 = -(torch.ones(k).cuda()-y[i])@((torch.ones(k).cuda()-p[i]).log())
        l = y[i]*(p[i].log())
        prod = (torch.ones(k).cuda()-y[i]) - l
        loss2 = loss1 + torch.max(torch.ones(k).cuda()-prod)
        losses = loss1 + loss2
    return losses/(b)
</code></pre>
<p>so im not doing this:  torch.tensor(OLD_TENSOR) anymore</p>

username: G.M
cooked: <p>Just to note: U can change <code>torch.ones(k).cuda() - TEN</code> to <code>1 - TEN</code>. Also, ur loss function looks like cross entropy loss to me, and Pytorch has a implementation for that.</p>

username: netaglazer
cooked: <p>it is not cross entropy,<br>
it is a loss for a multilabel problem, that i want to minimize when the model correct in one lable of all the labels of an object in my data</p>