One-hot preds: element 0 of tensors does not require grad and does not have a grad_fn

username: Amos_Brazzoli
cooked: <p>Hi there,</p>
<p>I’ve coded a model which takes a tensor <code>(batch_size,8,26)</code> input of one-hot rows and yields a tensor <code>(batch_size, 8, 189)</code>.<br>
My labels are also tensors <code>(batch_size, 8, 189)</code> of one-hot rows, so i wanted to make  <code>preds=model(batch)</code> into one-hot rows trasformin the max of the row in one and others in zeros.<br>
So I can determine if the prediction is correct, not just the drop in loss, which I can get now.</p>
<p>I tried with this:</p>
<pre><code class="lang-auto">    def MaxT1(self, batch_size, batch):
        #self.mphonsl,self.phons = 8, 189, because inputs from a fc layer
        batch=batch.view(batch_size,self.mphonsl,self.phons) 
        view=batch
        batch=(view==batch.max(dim=2, keepdim=True)[0]).view_as(batch)
        return batch
</code></pre>
<p>and calling it in the <code>.forward</code> of the model as:</p>
<pre><code class="lang-auto">        t=self.MaxT1(batch, batch_size)
</code></pre>
<p>However when calling <code>loss.backward()</code> I get</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-18-f2b8b1fbff2f&gt; in &lt;module&gt;
     22         loss=loss(labels.float(),preds.float())
     23         optimizer.zero_grad() # otherwise it keeps all gradients
---&gt; 24         loss.backward()
     25         optimizer.step()
     26         tb.add_scalar('Loss', loss, epoch)

~\.conda\envs\Pytorch\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    148                 products. Defaults to ``False``.
    149         
--&gt; 150         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    151 
    152     def register_hook(self, hook):

~\.conda\envs\Pytorch\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 99         allow_unreachable=True)  # allow_unreachable flag
    100 
    101 

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Any idea about how can i do it differently or solve this?</p>
<p>Thanks in advance</p>

username: KFrank
cooked: <p>Hi Amos!</p>
<aside class="quote no-group quote-modified" data-username="Amos_Brazzoli" data-post="1" data-topic="58709" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amos_brazzoli/48/16701_2.png" class="avatar"> Amos_Brazzoli:</div>
<blockquote>
<p>So I can determine if the prediction is correct, not just the drop in loss, which I can get now.</p>
<p>Any idea about how can i do it differently or solve this?</p>
</blockquote>
</aside>
<p>You say “So I can determine if the prediction is correct.”  I would<br>
call this the <em>accuracy</em> – the fraction of predictions that are correct.</p>
<p>Then you say “not just the drop in loss, which I can get now.”  From<br>
this I assume that (when you don’t try to compute the accuracy)<br>
you can compute your loss function, train your network to make<br>
the loss smaller, and the loss does indeed go down.</p>
<p>In the standard approach, you train to reduce a loss function, and<br>
then compute the accuracy independent of the training process.</p>
<p>(You typically compute the accuracy – or other statistics not directly<br>
involved in the training process – in a <code>with torch.no_grad():</code><br>
block.)</p>
<p>The idea is that you want your loss function:</p>
<ol>
<li>
<p>to measure not just whether your prediction is correct, but also<br>
how far off it is;</p>
</li>
<li>
<p>and to be nicely differentiable so that you can get gradients that<br>
are useful for training (back-propagation).</p>
</li>
</ol>
<p>Based on this, the accuracy generally doesn’t make a good loss<br>
function, so you don’t want to use it for training.</p>
<p>If I understand correctly what you are trying to do, this standard<br>
framework applies to your case.  So you should train using your<br>
loss function (as I am assuming that you have been able to do<br>
successfully), and place your accuracy calculation in a<br>
<code>with torch.no_grad():</code> block.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Amos_Brazzoli
cooked: <p>Thanks KFrank,</p>
<p>Maybe I wasn’t that clear.<br>
I’m trainin on loss with <code>MSELoss</code> between the last layer’s output and labels.<br>
I want to apply after the last layer something  like the <code>MaxT1</code> function I made in order to keep the max value of each row.</p>
<p>Something like:</p>
<p>Reshaped output from the last FCL</p>
<pre><code class="lang-auto">t1=tensor([[3.6600, 2.0000, 4.0900],
        [5.4400, 6.0000, 5.9000]])
</code></pre>
<p>Label</p>
<pre><code class="lang-auto">l1=tensor([[0, 0, 1],
        [0, 0, 1]])
</code></pre>
<p>I’d like to get from the first tensor</p>
<pre><code class="lang-auto">t2=tensor([[0, 0, 1],
        [0, 1, 0]])
</code></pre>
<p>and then do <code>MSELoss(l1,t2)</code> instead of <code>MSELoss(l1,t1)</code> as I’m doing now, which way higher. Also buy doing this it would be easier to check how many <code>preds</code> are correct.</p>

username: KFrank
cooked: <p>Hello Amos!</p>
<aside class="quote no-group" data-username="Amos_Brazzoli" data-post="3" data-topic="58709" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amos_brazzoli/48/16701_2.png" class="avatar"> Amos_Brazzoli:</div>
<blockquote>
<p>Thanks KFrank,</p>
<p>Maybe I wasn’t that clear.<br>
I’m trainin on loss with <code>MSELoss</code> between the last layer’s output and labels.<br>
I want to apply after the last layer something  like the <code>MaxT1</code> function I made in order to keep the max value of each row.</p>
</blockquote>
</aside>
<p>Have you tried putting the stuff that isn’t part of your actual<br>
training – the stuff “like the <code>MaxT1</code> function” – inside of a<br>
<code>with torch.no_grad():</code> block?  You want to keep<br>
calculations that don’t contribute to the loss you are training<br>
on out of the gradients used for training so they don’t pollute<br>
things or cause errors.  That is the purpose of <code>torch.no_grad()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Amos_Brazzoli
cooked: <p>Thanks K. Frank,</p>
<p>I managed to find a wordaround using <code>nn.Softmax(dim=2)</code> instead of my own function.<br>
They yield similar enough results</p>