Tensor shapes for loss_functions: crossEntropyLoss and SmoothL1Loss

username: MikeTensor
cooked: <p>For the training of my project I use two models, which means I have to outputs.<br>
They use the following loss_fn:</p>
<p>ModelA: <code>nn.CrossEntropyLoss()</code><br>
ModelB: <code>nn.SmoothL1Loss(reduction='mean')</code></p>
<p>I am a bit confused about their needed tensor_shapes.<br>
Just wanted to reassure:<br>
loss_A_fn will get:<br>
predictedTensor: [BATCH, VALUE] like [8,1]<br>
labelTensor: [VALUE, VALUE, …] like [8]</p>
<p><strong>1) Is that right?</strong></p>
<p>I think this points into this direction: <a href="https://discuss.pytorch.org/t/looking-for-a-cross-entropy-loss-that-accepts-two-tensors-of-the-same-shape/61302" class="inline-onebox">Looking for a cross entropy loss that accepts two tensors of the same shape</a></p>
<p>But my loss_B_fn will need on both [8,1] and [8,1] (pred, label).<br>
<strong>2) Is that right?</strong></p>
<p>If yes:<br>
Since my labels are all <code>hot</code>(?) Tensors [8] I reshape it with:<br>
<code>distance_label_dl = distance_label_dl.reshape(distance_label_dl.shape[0], 1)</code><br>
<strong>3) Is that okay, or shall I rather use another (better) way?</strong></p>
<p><strong>4) Do I have to do this within my <code>train_loop</code> or shall I do that within the <code>CustomDataset</code>?</strong></p>

username: ptrblck
cooked: <p>I think the main question is what your use case is and what your targets represent.<br>
<code>nn.CrossEntropyLoss</code> is used for multi-class classification/segmentation use cases and expects class labels by default. However, in newer PyTorch releases you can also pass “soft” targets to it representing probabilities in <code>[0, 1]</code> in the same shape as the model output.<br>
<code>nn.SmoothL1Loss</code> might be used for regression use cases, but I doubt it’s a good loss function for a classification use case.<br>
You could of course one-hot encode your target (i.e. all classes are set to zero while the one active class is set to one), which might satisfy the shape requirements of both loss functions, but might not make much sense for the actual model training.</p>

username: MikeTensor
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Okay. Maybe the numbers are a bit confusing.<br>
Let’s say the batch_size is: 6.<br>
The Class_categories are: 9 [0, 1, 2, 3, 4, 5, 6, 7, 8]. → (0: Car, 1: Cyclist, 2: Person, …).<br>
regeression_labels: anything between 0–99.</p>
<p>So regarding to my questions</p>
<p>ModelA: <code>nn.CrossEntropyLoss()</code> → Is used for <strong>Classification</strong>.<br>
ModelB: <code>nn.SmoothL1Loss(reduction='mean')</code> → Is used for <strong>regression</strong>.</p>
<p>So what I am doing right now:<br>
Classification;  <code>nn.CrossEntropyLoss()</code>: <code>yc_predicted</code> is tensor [batch number, class_categories]<br>
<code>label_tensor</code> (shape): [6] So there is no direct batch_number included anymore.<br>
Values look like: [0, 0, 3, 7, 4, 8]</p>
<p>So here it’s ok to feed the <code>nn.CrossEntropyLoss()</code> with ([6,1], [6]). →  (predicted_tensor, target_tensor).<br>
<strong>1) Is that right?</strong></p>
<p>But in contrast the <code>loss_B</code> for the regression (with <code>nn.SmoothL1Loss(reduction='mean')</code> needs other shapes:<br>
loss_smoothL1([6,1] , [6,1]) → (predicted_tensor, target_tensor).<br>
<strong>2) Is that right?</strong></p>
<p>If yes:<br>
Since my labels are all Tensors with only one single-dim I reshape it with:<br>
<code>distance_label_dl = distance_label_dl.reshape(distance_label_dl.shape[0], 1)</code></p>
<p>So from a tensor which first looks like shape [6]: <code>[3, 83, 32, 57, 67, 22]</code> I make a tensor with shape [6, 1]:</p>
<pre><code class="lang-auto">[[3], 
[83], 
[32], 
[57], 
[67], 
[22]]
</code></pre>
<p><code>distance_label_dl = distance_label_dl.reshape(distance_label_dl.shape[0], 1)</code><br>
<strong>3) Is that okay, or shall I rather use another (better) way?</strong></p>
<p><strong>4) Do I have to do this <code>reshpae</code> within my <code>train_loop</code> or shall I do that within the <code>CustomDataset</code>?</strong></p>

username: ptrblck
cooked: <p>Yes to 1. and 2., which is also explained in the docs for both loss functions.</p>
<ol start="3">
<li>
<p>Yes, <code>reshape</code> can be used to add an additional dimension, but you could also use the more explicit <code>.unsqueeze(1)</code> operation.</p>
</li>
<li>
<p>I would add these ops into the <code>Dataset.__getitem__</code> and make sure all tensors are already in the expected shape inside the training loop.</p>
</li>
</ol>