Getting RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

bing: <p>Hi,<br>
I am doing binary image classification and using <code>BCEWithLogitLoss</code>.<br>
Initally, I was getting <code>RuntimeError: result type Float can’t be cast to the desired output type Long</code><br>
So after searching, I converted the <code>pred</code> and <code>target</code> to float but now I am getting <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn </code><br>
I really don’t have an idea where I am doing wrong -</p>
<p>My training loop looks like the below -</p>
<pre><code class="lang-auto">model.train()
for batch_idx, (data, target) in enumerate(loader['train']):
            # move to GPU
      if torch.cuda.is_available():
          data, target = data.to('cuda', non_blocking=True), target.to('cuda', non_blocking = True) # noqa
      optimizer.zero_grad()
      output = model(data)
      pred = torch.argmax(output, dim=1)
      loss = criterion(pred.float(), target.float()). # Conversion of pred and target to float
      loss.backward()
      optimizer.step()
      train_loss += ((1 / (batch_idx + 1)) * ((loss.data) - train_loss))
</code></pre>
<p>My transfer learning &amp; loss function loading code snippet looks like below -</p>
<pre><code class="lang-auto">criterion_transfer = nn.BCEWithLogitsLoss()
model = timm.create_model('convnext_tiny_in22k', pretrained=True,num_classes=2) # noqa
optimizer = torch.optim.SGD(model_transfer.parameters(), lr=config.lr)
optimizer.zero_grad()
model_transfer.to(device)
criterion_transfer.to(device)
</code></pre>

AlphaBetaGamma96: <aside class="quote no-group" data-username="bing" data-post="1" data-topic="158642">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<pre><code class="lang-auto">      pred = torch.argmax(output, dim=1)
</code></pre>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/bing">@bing</a>, you can’t differentiate <code>torch.argmax</code> with respect to <code>output</code> (as <code>torch.argmax</code> has no <code>grad_fn</code>) so you need to find another way to convert your <code>output</code> tensor to a prediction with an operation that has a <code>grad_fn</code>. A minimal example below to show that <code>torch.argmax</code> has no <code>grad_fn</code>.</p>
<pre><code class="lang-auto">import torch
x=torch.randn(10,4,requires_grad=True)
output = torch.argmax(x, dim=1)
print(output.grad_fn) #returns None
</code></pre>
<p>You might just be able to remove the <code>torch.argmax</code> call as your Loss seems to expect the raw logits  and replace the loss calculation as,</p>
<pre><code class="lang-auto">      loss = criterion(output.float(), target.float()). # Conversion of pred and target to float
</code></pre>
<p>More info in this post <a href="https://discuss.pytorch.org/t/nn-crossentropyloss-optimizer-not-updating-weights/158620/3">here</a> (about logits with a different loss function you might find useful)</p>

bing: <p>I did tried calculating the <code>loss function</code> as you suggested earlier but I was getting<br>
<code>ValueError: Target size (torch.Size([1])) must be the same as input size (torch.Size([1, 2]))</code><br>
So I pivoted to calculating the argmax<br>
Below is the shape of my output and target</p>
<pre><code class="lang-auto">Shape of output: torch.Size([1, 2])
Shape of target: torch.Size([1])
</code></pre>

AlphaBetaGamma96: <p>So, I had a quick read through the docs for <code>BCEWithLogitsLoss</code> (docs <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">here</a>). These shapes represent the <code>[batch, num_classes]</code> respectively (and obviously should have the same size).</p>
<ol>
<li>So check your <code>target</code> Tensor has the right shape</li>
<li>Or perhaps you need to reduce <code>output</code> to match the shape of <code>target</code> (as that’s what <code>torch.argmax</code> was effectively doing)</li>
</ol>

bing: <p>I am initially trying to run the network on 1 sample only, so the target is supposed to be of shape 1, I tried unsqueezing also but it didn’t work out.<br>
Yes, the output is supposed to be a single value but I really don’t know if not to use argmax then how to do it.<br>
Below are my Target and output values -</p>
<pre><code class="lang-auto">Target -  tensor([0], device='cuda:0')
Output -  tensor([[0.3863, 0.1197]], device='cuda:0', grad_fn=&lt;GatherBackward&gt;)
</code></pre>

AlphaBetaGamma96: <p>If your target is a scalar for a single sample, it should have a shape of <code>[1,1]</code> because the shape is defined as <code>[num_samples, size_of_one_sample]</code> which corresponds to <code>[1,1]</code>.</p>
<p>If you’re trying to get the position of the max value of <code>output</code>, surely you should be using <code>torch.max</code> instead of <code>torch.argmax</code> (as <code>torch.max</code> has a <code>grad_fn</code>)?</p>
<aside class="quote no-group" data-username="bing" data-post="1" data-topic="158642">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<pre><code class="lang-auto">      pred = torch.argmax(output, dim=1)
</code></pre>
</blockquote>
</aside>
<p>So,</p>
<pre><code class="lang-auto">pred = torch.max(output, dim=1, keepdim=True)[0] #need the [0] to return values, [1] is indices
</code></pre>
<p>and make sure to have <code>keepdim=True</code> (so your shape is correct!), and this approach has a <code>grad_fn</code>,</p>
<pre><code class="lang-auto">import torch
x=torch.randn(10,4,requires_grad=True)
output = torch.max(x, dim=1, keepdim=True)[0]
print(output.grad_fn) #prints MaxBackward0 
</code></pre>

bing: <p>Thanks, It worked now.</p>