Alternating runtime errors: RuntimeError: expected scalar type Long but found Float and RuntimeError: Expected floating point type for target with class probabilities, got Long

Santiago_Amaya: <p>Hi, I am currently training a NN to classify inputs into 199 categories using pytorch cross_entropy loss. If I try using the probabilities for each class as the targets and cast the targets to float:</p>
<pre><code class="lang-auto">def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
    _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
    loss = F.cross_entropy(receiver_output.squeeze(), _labels.float(), reduction='none',label_smoothing=0.1)
    return loss, {}
</code></pre>
<p>I get the output for one epoch:</p>
<pre><code class="lang-auto">{"loss": 5.204253673553467, "baseline": 5.205080986022949, "sender_entropy": 0.001109135104343295, "receiver_entropy": 0.0, "mode": "train", "epoch": 1}
</code></pre>
<p>and then the following error:</p>
<pre><code class="lang-auto">&lt;ipython-input-91-936be60121ba&gt; in cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input)
      5 def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
      6     _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
----&gt; 7     loss = F.cross_entropy(receiver_output.squeeze(), _labels.float(), reduction='none',label_smoothing=0.1)
      8     return loss, {}

~\Anaconda3\envs\egg36\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: expected scalar type Long but found Float
</code></pre>
<p>If, on the other hand, I cast the target to long:</p>
<pre><code class="lang-auto">def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
    _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
    loss = F.cross_entropy(receiver_output.squeeze(), _labels.long(), reduction='none',label_smoothing=0.1)
    return loss, {}
</code></pre>
<p>I inmediately get the following error:</p>
<pre><code class="lang-auto">&lt;ipython-input-94-e319bc27aeaa&gt; in cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input)
      5 def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
      6     _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
----&gt; 7     loss = F.cross_entropy(receiver_output.squeeze(), _labels.long(), reduction='none',label_smoothing=0.1)
      8     return loss, {}

~\Anaconda3\envs\egg36\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: Expected floating point type for target with class probabilities, got Long
</code></pre>
<p>As you can see, in both cases the error is being triggered by the loss function I defined (in line 7), and internally in line 2846 of the torch.nn.functional. And it is asking for contradictory target formats.</p>
<p>If I change the loss function to work with index targets instead of probabilities:</p>
<pre><code class="lang-auto">def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
    #_labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
    loss = F.cross_entropy(receiver_output.squeeze(), _labels.long(), reduction='none',label_smoothing=0.1)
    return loss, {}
</code></pre>
<p>I also get the first batch output, but after that I get this error (in the same line as the others):</p>
<pre><code class="lang-auto">~\Anaconda3\envs\egg36\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: Expected target size [32, 199], got [32]
</code></pre>
<p>Which acording to the documentation should not be a problem for index targets:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4e19bdf64edec8d418fa5aa75b297799696f831.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4e19bdf64edec8d418fa5aa75b297799696f831" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4e19bdf64edec8d418fa5aa75b297799696f831.png" alt="image" data-base62-sha1="nwBFxgvi4TScG27ATZEFmjtrZnj" width="690" height="221" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">726×233 21.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am completely at a loss of what is happening here, I tried the same exact loss with <code>reduction='mean'</code> and it worked fine, so maybe the issue is there. Still I need this part to have <code>reduction='none' </code> so I have to fix it.</p>

ptrblck: <p>Could you check which PyTorch version you are using?<br>
If I’m not mistaken <code>nn.CrossEntropyLoss</code> accepts probabilities as a <code>FloatTensor</code> for the targets since <code>1.10.0</code>.</p>

Santiago_Amaya: <p>Hi, I am using version <code>1.10.1+cpu</code>, but as you can see in the first block of code, I use <code>_labels.float()</code> as targets, It runs for one epoch and then it outputs the error in the third block of code.</p>

ptrblck: <p>If the code runs for some iterations and raises an error later, check the shape as well as <code>dtype</code> of all used tensors as one of them seems to have changed.</p>