Correct usage of torch::nll_loss

username: markl
cooked: <p>This toy program crashes when trying to call <code>torch::nll_loss</code>. It’s supposed to simulate a pixel-wise classification loss.</p>
<p>I was hoping that the python documentation <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss</a> (the subsection on shape) would work for the c++ function.</p>
<p>Here is the toy program</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main(int argc, char** argv) {
  std::cout &lt;&lt; "NLL Example" &lt;&lt; std::endl;

  constexpr int kBatchSize = 1;
  at::Tensor net_output =
      torch::log_softmax(torch::randn({kBatchSize, 2, 300, 300}), /*dim=*/1);
  std::cout &lt;&lt; "Log softmax succeeded" &lt;&lt; std::endl;

  at::Tensor ground_truth = torch::ones({kBatchSize, 300, 300});
  std::cout &lt;&lt; "Ground truth created" &lt;&lt; std::endl;

  at::Tensor loss = torch::nll_loss(net_output, ground_truth);
  std::cout &lt;&lt; "Done" &lt;&lt; std::endl;
} 
</code></pre>
<p>The program crashes silently with the output</p>
<pre><code class="lang-auto">NLL Example
Log softmax succeeded
Ground truth created
</code></pre>

username: ebarsoum
cooked: <aside class="quote no-group" data-username="markl" data-post="1" data-topic="58843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/markl/48/16511_2.png" class="avatar"> markl:</div>
<blockquote>
<p>nll_loss</p>
</blockquote>
</aside>
<p>Can you attach a debugger and share the stacktrace? Here an example usage from <a href="https://github.com/pytorch/pytorch/blob/master/test/cpp/api/integration.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/test/cpp/api/integration.cpp</a></p>
<pre><code class="lang-auto">  for (size_t epoch = 0; epoch &lt; number_of_epochs; epoch++) {
    for (torch::data::Example&lt;&gt; batch : *data_loader) {
      auto data = batch.data.to(device), targets = batch.target.to(device);
      torch::Tensor prediction = forward_op(std::move(data));
      torch::Tensor loss = torch::nll_loss(prediction, std::move(targets));
      AT_ASSERT(!torch::isnan(loss).any().item&lt;int64_t&gt;());
      optimizer.zero_grad();
      loss.backward();
      optimizer.step();
    }
  }
</code></pre>

username: markl
cooked: <p>Here is a capture from WinDbg<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/931109aad2d14cac58fb2c12a600fc548f09a5c8.png" data-download-href="https://discuss.pytorch.org/uploads/default/931109aad2d14cac58fb2c12a600fc548f09a5c8" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/931109aad2d14cac58fb2c12a600fc548f09a5c8.png" alt="Capture" data-base62-sha1="kZ0GRIF8KFv2oxdqvq5IAkwzJI4" width="638" height="500" data-dominant-color="E7E8E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1324×1036 57.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: markl
cooked: <p>I caught the exception and printed <code>e.what()</code></p>
<pre><code class="lang-auto">NLL Example
Log softmax succeeded
Ground truth created
Exception occured
Expected object of scalar type Long but got scalar type Float for argument #2 'target' (checked_tensor_unwrap at C:\w\1\s\windows\pytorch\aten\src\ATen/Utils.h:80)
(no backtrace available)
</code></pre>

username: markl
cooked: <p>Based on the exception, I have updated the code to call <code>.to(at::kLong)</code> on the ground truth tensor. I now get different error (below). I think this means I need to look for a multi-target version of the nll function.</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main(int argc, char** argv) {
  std::cout &lt;&lt; "NLL Example" &lt;&lt; std::endl;

  constexpr int kBatchSize = 1;
  at::Tensor net_output =
      torch::log_softmax(torch::randn({kBatchSize, 2, 300, 300}), /*dim=*/1);
  std::cout &lt;&lt; "Log softmax succeeded" &lt;&lt; std::endl;

  at::Tensor ground_truth = torch::ones({kBatchSize, 300, 300}).to(at::kLong);
  std::cout &lt;&lt; "Ground truth created" &lt;&lt; std::endl;

  try {
    at::Tensor loss = torch::nll_loss(net_output, ground_truth);
  } catch (const std::exception&amp; e) {
    std::cout &lt;&lt; "Exception occured\n" &lt;&lt; e.what() &lt;&lt; std::endl;
    exit(-1);
  }
  std::cout &lt;&lt; "Done" &lt;&lt; std::endl;
}
</code></pre>
<p>The output is now</p>
<pre><code class="lang-auto">NLL Example
Log softmax succeeded
Ground truth created
Exception occured
multi-target not supported at C:\w\1\s\windows\pytorch\aten\src\THNN/generic/ClassNLLCriterion.c:22
</code></pre>