Loss calculation in multi-class classifier Neural Network

username: Anshu_Garg
cooked: <p>Hi All,</p>
<p>I am new to Pytorch and ML development. I have a basic doubt.<br>
I am implementing fully connected Neural Network and using pytorch dataset and torch.utils.data.DataLoader on MNIST dataset for handwritten digit classification.<br>
Suppose the batch_size passed to torch.utils.data.DataLoader is 5, after each batch run, I will get model output which has dimension (5,10) (let’s call it predicted_output). 5 - represent batch size and 10 represent the number of classes.<br>
We calculate the loss for each batch.<br>
Data Loader will provide us the labels list which size will be (5,1) (let’s call it original_labels)</p>
<p>So, doubt is - when we use any loss function (let’s say nn.MSELoss) then we pass (predicted_output and original_labels).<br>
But ‘original_labels’ - contains actual labels (for example [5, 8, 9, 3, 0] (column vector)  etc)<br>
and ‘predicted_output’  contains a probability for each class. So, for one sample input, we have a row of size (1,10), where for each class there is a probability associated.</p>
<p>So, ideally we should convert ‘original_labels’ list to (5,10) size and then pass to loss function. For each row probability should be 1 for the actual label and 0 for other labels (for example: if label/class is digit 7: then row should be [0,0,0,0,0,0,0,1,0,0]).</p>
<p>Or does this work is internally done by nn.MSELoss function?<br>
Should I convert original_labels array to (5,10) format before sending to loss function (for example, change [5 8 9 3 0] (shape is 5,1) —&gt;  to format [[0,0,0,0,0,1,0,0,0,0], [0,0,0,0,0,0,0,0,1,0], [0,0,0,0,0,0,0,0,0,1], [0,0,0,1,0,0,0,0,0,0], [1,0,0,0,0,0,0,0,0,0]] ) ?</p>

username: KFrank
cooked: <p>Hi Anshu!</p>
<p>The short answer is use <code>CrossEntropyLoss</code>.</p>
<aside class="quote no-group" data-username="Anshu_Garg" data-post="1" data-topic="99754" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anshu_garg/48/29960_2.png" class="avatar"> Anshu_Garg:</div>
<blockquote>
<p>I will get model output which has dimension (5,10)</p>
</blockquote>
</aside>
<p>This is fine.  You want the <code>input</code> to <code>CrossEntropyLoss</code> (the output<br>
of your model) to have shape <code>[nBatch = 5, nClass = 10]</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Data Loader will provide us the labels list which size will be (5,1).</p>
</blockquote>
</aside>
<p>I don’t know what <code>DataLoader</code> provides, but you want the <code>target</code><br>
for <code>CrossEntropyLoss</code> (your labels) to have shape <code>[nBatch = 5]</code>.</p>
<p>If you get labels with shape <code>[nBatch, 1]</code>, <code>squeeze()</code> it to get rid of<br>
the singleton dimension.</p>
<aside class="quote no-group">
<blockquote>
<p>But ‘original_labels’ - contains actual labels (for example 5, 8, 9, 3, 0  etc)</p>
</blockquote>
</aside>
<p>This is what you want.  <code>CrossEntropyLoss</code> expects a <code>target</code> that is a<br>
<code>LongTensor</code> of integer class labels that range from <code>0</code> to <code>nClass - 1</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Or does this work is internally done by nn.MSELoss function?</p>
</blockquote>
</aside>
<p>Just to be clear, <code>MSELoss</code> is not a very good loss function for<br>
classification.  Use <code>CrossEntropyLoss</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Anshu_Garg
cooked: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="99754">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>CrossEntropyLoss</p>
</blockquote>
</aside>
<p>Thank you for your reply. I am talking about torch.utils.data.DataLoader.</p>