Multi-label regression with target tensor as A) normalized class counts or just B) class counts?

username: stark
cooked: <p>I want to perform the multi-label regression. The idea is to just  <em>detect the presence of class objects</em>  in an input image. Each 224x224 sized image contains multiple objects from a total of six different classes. The label is a normalized count for each class i.e.  <code>class_i_instances_per_image / sum(all_instances_per_image) = 3/12 = 0.25</code>  where i ∈ [0, 5]</p>
<pre><code class="lang-auto">target = tensor([[0.0588, 0.8235, 0.0882, 0.0000, 0.0000, 0.0294]], device='cuda:0', dtype=torch.float64)
</code></pre>
<p>Output from the last FC layer:</p>
<pre><code class="lang-auto">tensor([[-0.0649,  0.0636, -0.0299, -0.0651, -0.8145, -0.3030]], device='cuda:0', grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<p>Prediction from the  <code>F.softmax()</code> :</p>
<pre><code class="lang-auto">output = tensor([[0.1841, 0.2093, 0.1906, 0.1840, 0.0870, 0.1451]], device='cuda:0', grad_fn=&lt;SoftmaxBackward&gt;)
</code></pre>
<p>Computation of loss using the above tensors results into:</p>
<pre><code class="lang-auto">criterion = torch.nn.CrossEntropyLoss().cuda()
loss = criterion(torch.nn.functional.softmax(output), target)
</code></pre>
<blockquote>
<p><strong>RuntimeError:</strong>  Expected object of scalar type <strong>Long</strong> but got scalar type <strong>Double</strong> for argument <span class="hashtag-raw">#2</span> ‘target’ in call to _thnn_nll_loss_forward</p>
</blockquote>
<p>The suitable loss functions for regression problems are  <code>mean square error (MSE)</code>  or  <code>mean absolute error (MAE)</code> . But using  <code>F.mse_loss()</code>  takes much time and returns  <code>inf</code>  values.</p>
<p>I will highly appreciate your comments/suggestions to make use of these soft-labels.</p>

username: KFrank
cooked: <p>Hi Bran!</p>
<aside class="quote no-group" data-username="stark" data-post="1" data-topic="94054" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>The idea is to just  <em>detect the presence of class objects</em>  in an input image.</p>
</blockquote>
</aside>
<p>Could you clarify your use case?</p>
<p>Let’s say you have classes A through F.  Let me focus on class C.<br>
Do you wish your network to tell you just whether you have any<br>
instances of class C in an image?  That is “no” means no instances<br>
of class C, while “yes” means you have at least one instance of C,<br>
but you don’'t care how many (as long as it’s not zero).</p>
<p>Or do you want your network to give you a quantitative measure<br>
of how many instances of class C appear in your image?  That is,<br>
do you want a different network output if you have four instances<br>
of C than if you have one?</p>
<p>Best.</p>
<p>K. Frank</p>

username: stark
cooked: <p>I would like my network to provide me the quantitative measure. It could be the probability for each present class OR the estimated class count.</p>

username: KFrank
cooked: <p>Hi Bran!</p>
<aside class="quote no-group" data-username="stark" data-post="3" data-topic="94054" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>I would like my network to provide me the quantitative measure. It could be the probability for each present class OR the estimated class count.</p>
</blockquote>
</aside>
<p>My intuition would be to use <code>MSELoss</code> on the class counts.</p>
<p>To me this seems to make more sense than the class “probabilities”<br>
(normalized class counts).  Let’s say a sample image has one<br>
instance of class B and one of class C, but your network only<br>
detects class B.  Your actual normalized class count is 1/2 for<br>
class B (and class C, as well), but your predicted value is 1.</p>
<p>So your network got class B right – one instance – but is being<br>
penalized because the normalized class count is wrong.  It seems<br>
to me that you want to give your network “full credit” for correctly<br>
detecting exactly one instance of B (while penalizing it, of course,<br>
for missing the instance of C).</p>
<p>Mean-squared-error makes sense because if, let’s say, you have<br>
4 instances of C, detecting 3 or 5 is wrong, but not as wrong as<br>
detecting 2 or 6.  (Mean-absolute-error would also work this way;<br>
I don’t have intuition about which would be better.)</p>
<p>Note, if you use normalized class counts you are throwing away<br>
information that might be helpful in training your network.  An image<br>
with three instances of B and three of C has the same normalized<br>
class counts as an image with one of each.  But rewarding your<br>
network for correctly detecting all of the B and C instances would<br>
seem to be making better use of your training data.</p>
<p>One comment on cross-entropy:  If you want to go the probability<br>
route, it is true that cross-entropy is a measure of how much two<br>
probability distributions deviate from one another.  But pytorch’s<br>
built-in <code>CrossEntropyLoss</code> won’t work in this case because it<br>
only handles the special case where the targets are integer class<br>
labels – that is, where the target probability distribution is all 0.0’s<br>
except for one 1.0.  If you want to use cross-entropy for this case<br>
you would have to write your own “soft-label” cross-entropy loss<br>
function.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: stark
cooked: <p>Thanks for the valuable explanation!</p>
<p>I gave <code>MSELoss</code> a try using the <strong>actual class counts</strong> as below:</p>
<pre><code class="lang-auto">criterion = torch.nn.MSELoss()
loss = criterion(output, target)
</code></pre>
<p>where</p>
<pre><code class="lang-auto">output = tensor([[-0.0272, -0.1516, -0.1283,  0.0954,  0.1588, -0.2674]], device='cuda:0', grad_fn=&lt;AddmmBackward&gt;)
target = tensor([[3., 0., 0., 0., 0., 1.]], device='cuda:0')
loss = tensor(1.8074, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;)
</code></pre>
<p>The returned loss value after a complete epoch is <code>nan</code> probably because there was an order of magnitude difference in the <code>output</code> and <code>target</code> tensors. The output was obtained from a pre-trained VGG19 with the <code>relu</code> activations in <code>linear</code> layers (i.e. classifier part). What options do I have to get this network working?</p>

username: KFrank
cooked: <p>Hi Bran!</p>
<aside class="quote no-group quote-modified" data-username="stark" data-post="5" data-topic="94054" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>I gave <code>MSELoss</code> a try using the <strong>actual class counts</strong> as below:</p>
<pre><code class="lang-auto">criterion = torch.nn.BCELoss().cuda()
</code></pre>
<pre><code class="lang-auto">loss = tensor(nan, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyBackward&gt;)
</code></pre>
</blockquote>
</aside>
<p>The immediate problem is that you are actually using <code>BCELoss</code><br>
(binary cross-entropy), rather than <code>MSELoss</code> (mean-squared-error)<br>
as your loss function.</p>
<p><code>BCELoss</code> is not conceptually appropriate for your use class-count<br>
use case.  Also, technically, it requires its target values to be<br>
probabilities – numbers between 0.0 and 1.0 – for its computations<br>
to make any sense, and your class-count targets are often greater<br>
than 1.</p>
<p>Best.</p>
<p>K. Frank</p>

username: stark
cooked: <p>Sorry for the mistake! I’ve edited my recent reply and included another example here.</p>
<p><code>loss = torch.nn.MSELoss(output, target)</code></p>
<p>The above statement returns a high loss value. For instance the loss is <code>169.3941</code> for the following input tensors:<br>
<code>output</code> = [13.7210,  1.6992, -0.1286, -0.9545, -0.9148,  2.3547], and <code>target</code> = [ 0.,  0.,  0.,  0., 14.,  1.])</p>
<p>How may I get a non-negative integer output from the network?</p>

username: KFrank
cooked: <p>Hi Bran!</p>
<p>Something fishy is going on here …</p>
<aside class="quote no-group" data-username="stark" data-post="7" data-topic="94054" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p><code>loss = torch.nn.MSELoss(output, target)</code></p>
</blockquote>
</aside>
<p>This incorrectly constructs an instance of the <code>MSELoss</code> class, but<br>
doesn’t actually calculate the value of the loss.</p>
<p>You want <code>MSELoss() (output, target)</code><br>
(or <code>mse_loss (output, target)</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>For instance the loss is <code>169.3941</code> for the following input tensors:<br>
<code>output</code> = [13.7210,  1.6992, -0.1286, -0.9545, -0.9148,  2.3547], and <code>target</code> = [ 0.,  0.,  0.,  0., 14.,  1.])</p>
</blockquote>
</aside>
<p>The value you quote for the loss, <code>169.3941</code>, seems to be off by 100.0.<br>
I get <code>69.3945</code>.</p>
<p>Here is a pytorch version 0.3.0 script that illustrates these two issues:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

output = torch.autograd.Variable (torch.FloatTensor ([13.7210, 1.6992, -0.1286, -0.9545, -0.9148, 2.3547]))
target = torch.autograd.Variable (torch.FloatTensor ([ 0., 0., 0., 0., 14., 1.]))
torch.nn.MSELoss (output, target)               # construct a bogus instance of the class MSELoss
torch.nn.MSELoss() (output, target)             # construct a legitimate instance of the class MSELoss and apply it to (output, target)
torch.nn.functional.mse_loss (output, target)   # apply the function mse_loss() to (output, target)
((output - target)**2).mean()                   # compute mean-squared-error directly, just to check
169.3941 - ((output - target)**2).mean()        # your result is off by 100
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; output = torch.autograd.Variable (torch.FloatTensor ([13.7210, 1.6992, -0.1286, -0.9545, -0.9148, 2.3547]))
&gt;&gt;&gt; target = torch.autograd.Variable (torch.FloatTensor ([ 0., 0., 0., 0., 14., 1.]))
&gt;&gt;&gt; torch.nn.MSELoss (output, target)               # construct a bogus instance of the class MSELoss
MSELoss(
)
&gt;&gt;&gt; torch.nn.MSELoss() (output, target)             # construct a legitimate instance of the class MSELoss and apply it to (output, target)
Variable containing:
 69.3945
[torch.FloatTensor of size 1]

&gt;&gt;&gt; torch.nn.functional.mse_loss (output, target)   # apply the function mse_loss() to (output, target)
Variable containing:
 69.3945
[torch.FloatTensor of size 1]

&gt;&gt;&gt; ((output - target)**2).mean()                   # compute mean-squared-error directly, just to check
Variable containing:
 69.3945
[torch.FloatTensor of size 1]

&gt;&gt;&gt; 169.3941 - ((output - target)**2).mean()        # your result is off by 100
Variable containing:
 99.9996
[torch.FloatTensor of size 1]
</code></pre>
<p>Good luck.</p>
<p>K. Frank</p>

username: stark
cooked: <p>Thanks for your reply. I’m creating an instance of the <code>MSELoss</code> class and calling it later (as I mentioned in the original post)</p>
<aside class="quote no-group" data-username="stark" data-post="1" data-topic="94054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<pre><code class="lang-auto">criterion = torch.nn.CrossEntropyLoss().cuda()
loss = criterion(torch.nn.functional.softmax(output), target)
</code></pre>
</blockquote>
</aside>
<p>but I simply mentioned it as <code>loss = torch.nn.MSELoss(output, target)</code> ← this staement will cause this error: <code>RuntimeError: bool value of Tensor with more than one value is ambiguous</code>.</p>
<hr>
<p>I noticed the value returned by <code>MSELoss</code> function was not an average of individual losses of examples in the batch. For instance, considering the batch size = 2, the loss was different from the average of the two losses when batch size was 1 while inputs/ targets were kept the same. <strong>Which of these should be given more importance?</strong></p>
<p>a) loss calculated per <strong>image</strong> <em>(for batch size =1)</em><br>
b) loss calculated per <strong>batch</strong> <em>(for batch size &gt; 1)</em><br>
c) mean loss per <strong>epoch</strong> <em>(irrespective of the batch size)</em></p>
<hr>

username: KFrank
cooked: <p>Hi Bran!</p>
<aside class="quote no-group" data-username="stark" data-post="9" data-topic="94054" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>I noticed the value returned by <code>MSELoss</code> function was not an average of individual losses of examples in the batch. For instance, considering the batch size = 2, the loss was different from the average of the two losses when batch size was 1 while inputs/ targets were kept the same.</p>
</blockquote>
</aside>
<p>Could you post a small, self-contained, runnable script, together with<br>
its complete output, that illustrates this issue?  Don’t include a model<br>
or anything – just hard wire the <code>output</code> and <code>target</code> tensors that you<br>
pass to your loss function.</p>
<p>Best.</p>
<p>K. Frank</p>

username: stark
cooked: <p>Thanks, Frank! <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
<p>While preparing an example for you, I found some possible reasons for having different losses.</p>
<hr>
<p>Brief context: I removed the <code>nn.BatchNorm2d()</code> layers to fine-tune a pretrained VGG19 but kept the n<code>n.Dropout(p=0.5)</code> layers <strong>(is this strategy fine for fine-tuning?)</strong>.</p>
<hr>
<p>At first, I thought the presence of these dropout layers following the <code>nn.Linear</code> layers altered the output every time hence I got different losses. So, I have to perform another experiment just to verify this. Following is what was done next</p>
<ol start="0">
<li>̶R̶e̶m̶o̶v̶e̶d̶ ̶B̶a̶t̶c̶h̶N̶o̶r̶m̶(̶)̶ ̶l̶a̶y̶e̶r̶s̶</li>
<li>Removed all <code>nn.Dropout()</code> layers</li>
<li>Set a constant seed value for torch/Cuda random number generator</li>
<li>Initialized the weight/bias of all three linear layers <em>(in the classifier part)</em></li>
</ol>
<p>To my surprise, the network still generated different outputs for the same inputs for batch size = 1, 2, and 4. The results are tabularized below. <strong>Can you please share the needed wisdom to understand this?</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58aefcf16ca3d5009a99cf708e0f88a1207ef3fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/58aefcf16ca3d5009a99cf708e0f88a1207ef3fc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58aefcf16ca3d5009a99cf708e0f88a1207ef3fc.png" alt="image" data-base62-sha1="cEwVvDQ7cgCJqor08kBBX3PWOAY" width="517" height="248" data-dominant-color="DEE1DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">691×332 13.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The green color indicates the same output for the relevant inputs.</p>
<p>Now, coming back to an older question.</p>
<aside class="quote no-group" data-username="stark" data-post="9" data-topic="94054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p><strong>Which of these should be given more importance?</strong></p>
<p>a) loss calculated per <strong>image</strong> <em>(for batch size =1)</em><br>
b) loss calculated per <strong>batch</strong> <em>(for batch size &gt; 1)</em><br>
c) mean loss per <strong>epoch</strong> <em>(irrespective of the batch size)</em></p>
</blockquote>
</aside>