How to get multi-target NLL in C++? (multi-target not supported at ClassNLLCriterion.c)

markl: <p>In the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="nofollow noopener">python API</a>, the <code>NLLLoss</code> is allowed to take a target shape (N, d1, …, dk). However, in the c++ api, the <code>torch::nll_loss</code> will crash with an exception <code>multi-target not supported at C:\w\1\s\windows\pytorch\aten\src\THNN/generic/ClassNLLCriterion.c:22</code></p>
<p>Is there a multi-target NLL for c++?</p>
<p>Here is my code</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main(int argc, char** argv) {
  std::cout &lt;&lt; "NLL Example" &lt;&lt; std::endl;

  constexpr int kBatchSize = 1;
  at::Tensor net_output =
      torch::log_softmax(torch::randn({kBatchSize, 2, 300, 300}), /*dim=*/1);
  std::cout &lt;&lt; "Log softmax succeeded" &lt;&lt; std::endl;

  at::Tensor ground_truth = torch::ones({kBatchSize, 300, 300}).to(at::kLong);
  std::cout &lt;&lt; "Ground truth created" &lt;&lt; std::endl;

  try {
    at::Tensor loss = torch::nll_loss(net_output, ground_truth);
  } catch (const std::exception&amp; e) {
    std::cout &lt;&lt; "Exception occured\n" &lt;&lt; e.what() &lt;&lt; std::endl;
    exit(-1);
  }
  std::cout &lt;&lt; "Done" &lt;&lt; std::endl;
}

</code></pre>
<p>With output</p>
<pre><code class="lang-auto">NLL Example
Log softmax succeeded
Ground truth created
Exception occured
multi-target not supported at C:\w\1\s\windows\pytorch\aten\src\THNN/generic/ClassNLLCriterion.c:22```</code></pre>

ptrblck: <p>This error is usually thrown, if your target still contains the class dimension:</p>
<pre><code class="lang-python">criterion = nn.NLLLoss()

N, nb_classes = 2, 3
output = torch.randn(N, nb_classes, requires_grad=True)
target = torch.randint(0, nb_classes, (N,))

loss = criterion(output, target) # works

target = torch.randint(0, nb_classes, (N, nb_classes))
loss = criterion(output, target) # fails with same error
</code></pre>
<p>For a multi-label classification (a single sample can contain more than a single valid class), you should use <code>nn.BCEWithLogitsLoss</code> (or the C++ equivalent).</p>

markl: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. It looks like you are using the form of the target with dimension (N,). However, I need the form where the target has dimension (N, d1, … dk). More explicitly (N, 300, 300). I am trying to train a semantic segmentation task where each pixel in the image has its own label.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b46739bd459e0a49b840487d966367221f3a273a.png" data-download-href="https://discuss.pytorch.org/uploads/default/b46739bd459e0a49b840487d966367221f3a273a" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b46739bd459e0a49b840487d966367221f3a273a_2_690x271.png" alt="Capture" data-base62-sha1="pJV6tgWSzLFSDq2fcymoNTSuGjE" width="690" height="271" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b46739bd459e0a49b840487d966367221f3a273a_2_690x271.png, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b46739bd459e0a49b840487d966367221f3a273a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b46739bd459e0a49b840487d966367221f3a273a.png 2x" data-dominant-color="F6F4F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">855×337 27 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m not sure the <code>BCEWithLogitsLoss</code> will work for me, since my target dimension is not the same as my input dimension.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8942745845ae6b3e82f5df8837c8e4a72dbf0f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8942745845ae6b3e82f5df8837c8e4a72dbf0f0" title="Capture3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8942745845ae6b3e82f5df8837c8e4a72dbf0f0_2_690x256.png" alt="Capture3" data-base62-sha1="sCoUv17yeqz4gnOo2Nh0HQGMSE8" width="690" height="256" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8942745845ae6b3e82f5df8837c8e4a72dbf0f0_2_690x256.png, https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8942745845ae6b3e82f5df8837c8e4a72dbf0f0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8942745845ae6b3e82f5df8837c8e4a72dbf0f0.png 2x" data-dominant-color="F8F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture3</span><span class="informations">839×312 16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I assume the semantic segmentation is a multi-class segmentation, i.e. each pixel belongs to exactly one class.<br>
If so, this should be possible using these shapes</p>
<pre><code class="lang-python">N, nb_classes, H, W = 2, 3, 4, 4
output = torch.randn(N, nb_classes, H, W, requires_grad=True)
target = torch.randint(0, nb_classes, (N, H, W))

loss = criterion(output, target)
</code></pre>
<p>Note that the <code>nb_classes</code> dimension is missing in the target as described in the docs, so make sure to pass these shapes to your criterion.</p>
<p><code>nn.BCEWithLogitsLoss</code> is most likely not, what you are looking for.</p>

markl: <p>I understand what you are saying, but I can’t see where my code sample is broken. My input is a tensor of dimension (1, 2, 300, 300) (1 batch, 2 classes, 300x300 image). My target (ground_truth) is a tensor (1, 300, 300) filled with 1.</p>

ptrblck: <p>This seems to be indeed the right shape and I remembered I’ve seen this issue before!<br>
Could you check, if <code>nll_loss2d</code> is defined and if so use it instead of <code>nll_loss</code> (I’m currently not on my machine to check it)?</p>

markl: <p>There is! <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1a5759854ee24eb9a93e7afd6c2f23aae4.html#exhale-function-namespaceat-1a5759854ee24eb9a93e7afd6c2f23aae4" rel="nofollow noopener">https://pytorch.org/cppdocs/api/function_namespaceat_1a5759854ee24eb9a93e7afd6c2f23aae4.html#exhale-function-namespaceat-1a5759854ee24eb9a93e7afd6c2f23aae4</a></p>
<p>Unfortunately there is no documentation besides the function declaration.</p>
<p>Hopefully it will work as a drop-in replacement in my code.</p>
<p>Edit: It worked! Well at least it succeeded without crashing. I’m now manually checking the output to see if it does what I think it should do (pixel-wise NLL loss, and then taking the mean)</p>

ptrblck: <p>Awesome!<br>
Yeah, we are working on the parity of the C++ API, so these things might be missing currently.<br>
Fortunately, we have the discussion board so feel free to ask, if you get stuck somewhere <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

markl: <p>For completeness, here is a version that does not crash and looks like it does the right thing</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main(int argc, char** argv) {
  std::cout &lt;&lt; "NLL Example" &lt;&lt; std::endl;

  constexpr int kBatchSize = 1;
  constexpr int kImageDim = 2;
  at::Tensor net_output = torch::log_softmax(
      torch::randn({kBatchSize, 2, kImageDim, kImageDim}), /*dim=*/1);
  std::cout &lt;&lt; "Log softmax succeeded" &lt;&lt; std::endl;

  at::Tensor ground_truth =
      torch::ones({kBatchSize, kImageDim, kImageDim}).to(at::kLong);
  std::cout &lt;&lt; "Ground truth (ones) created" &lt;&lt; std::endl;

  float actual_loss = 0;
  try {
    at::Tensor loss = torch::nll_loss2d(net_output, ground_truth);
    actual_loss = loss.item&lt;float&gt;();
  } catch (const std::exception&amp; e) {
    std::cout &lt;&lt; "Exception occured\n" &lt;&lt; e.what() &lt;&lt; std::endl;
    exit(-1);
  }

  std::cout &lt;&lt; "Expected loss " &lt;&lt; -net_output[0][1].mean().item&lt;float&gt;()
            &lt;&lt; std::endl;
  std::cout &lt;&lt; "Done with actual loss " &lt;&lt; actual_loss &lt;&lt; std::endl;
}

</code></pre>
<p>It outputs</p>
<pre><code class="lang-auto">NLL Example
Log softmax succeeded
Ground truth (ones) created
Expected loss 0.828792
Done with actual loss 0.828792</code></pre>

botelho: <p>Hi there,</p>
<p>I have the same semantic segmentation problem that <a class="mention" href="/u/markl">@markl</a> has, but my images are three-dimensional. Therefore, I would need a loss function that takes target tensors of shape (N, d1, d2, d3). There doesn’t seem to be a torch::nll_loss3d() in C++ Pytorch, as far as I can tell. What is your recommendation in this case? Do I just have to wait until libTorch catches up with python? If so, any idea when that will happen?</p>
<p>Thanks in advance!</p>

yf225: <p>Is there a <code>torch.nll_loss3d()</code> function currently in PyTorch Python API?</p>

botelho: <p>No, but Python’s <code>torch.nn.NLLLoss()</code> accepts tensors of arbitrary ranks – see <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss</a></p>

yf225: <p>In C++ frontend, <code>torch::nn::NLLLoss()</code> should be able to do the same <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

botelho: <p>Thanks for your answer! Is that supported after some given version? Because I’m using 1.2.0 and NLLLoss only accepts rank-2 tensors, as described by <a class="mention" href="/u/markl">@markl</a> up above in this thread. If you try a higher rank tensor, you get the <code>multi-target not supported</code> error.</p>

yf225: <p>Yes it is supported since the newly released version 1.4.0 <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

botelho: <p>Cool, thank you! I’ll try it out.</p>

botelho: <p>Hi, this still doesn’t seem to work even after 1.7. From the code in aten/src/ATen/native/LossNLL.cpp:</p>
<pre><code class="lang-auto">  TORCH_CHECK(
      input.dim() &gt; 0 &amp;&amp; input.dim() &lt;= 2, "input tensor should be 1D or 2D");
  TORCH_CHECK(
      target.dim() == 1,
      "1D target tensor expected, multi-target not supported");
</code></pre>
<p>This works in python but not in libtorch. Was this feature just forgotten as part of the parity work, or am I missing something?</p>
<pre><code class="lang-auto"></code></pre>

ptrblck: <p>It works for me in <code>1.7.1</code> using:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  auto input = torch::randn({3, 10, 24, 24});
  auto target = torch::randint(0, 10, {3, 24, 24}, torch::kLong);
  torch::nn::NLLLoss criterion;
  auto loss = criterion-&gt;forward(input, target);
  std::cout &lt;&lt; loss &lt;&lt; std::endl;
}
</code></pre>
<p>Also, note that there are different implementations, such as <a href="https://github.com/pytorch/pytorch/blob/573f4aa3529371e9d30a0884d56eac0e6d3ddd71/aten/src/ATen/native/LossNLL2d.cpp"><code>LossNLL2d.cpp</code></a>, which might accept multiple dimensions (I haven’t checked, which function is called in my example though).</p>

botelho: <p>Hi, thanks for the response. Yes, your code works for me too. What doesn’t work is to use the free-function form of the NLL loss:</p>
<pre><code class="lang-auto">auto input = torch::randn({3, 10, 24, 24});
auto target = torch::randint(0, 10, {3, 24, 24}, torch::kLong);
auto loss = torch::nll_loss(input, target);
</code></pre>
<p>Is there any design reason for this to behave differently from <code>NLLLoss</code>? If not, I suggest it either gets brought up-to-par with the class, or removed altogether, to avoid confusion. Thanks again!</p>