Binary segmentation BCEWithLogitsLoss pos_weight?

username: roryw10
cooked: <p>I’m segmenting foreground vs background using unet and there are many more 0s than 1s due to this. It looks like    i should be using BCEWithLogitsLoss as my loss function?</p>
<p>It looks like this function takes as an argument the proportion of class imbalance - “pos_weight (Tensor, optional) – a weight of positive examples. Must be a vector with length equal to the number of classes.”</p>
<p>Does that mean i need to know across all my images how many pixels are 1 v 0 <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> If so is there any best practice on how to do this?</p>
<p>I do have some images that are pure “background” without any 1s in as part of the dataset, so an alternative is i could use pos_weight as the number of positive vs negative at a global scale?</p>
<p>Thanks</p>

username: KFrank
cooked: <p>Hi Rory!</p>
<aside class="quote no-group" data-username="roryw10" data-post="1" data-topic="71667" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3bc359/48.png" class="avatar"> roryw10:</div>
<blockquote>
<p>Does that mean i need to know across all my images how many pixels are 1 v 0 <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"> If so is there any best practice on how to do this?</p>
</blockquote>
</aside>
<p>There is nothing magic about the <em>exact</em> choice of <code>pos_weight</code>.  It<br>
provides a rough reweighting of positive samples vs. negative.<br>
Nothing goes wrong if you choose one reasonable value instead<br>
of another.</p>
<p>I don’t think that there is any benefit in trying to choose <code>pos_weight</code><br>
on a per-batch basis – just get an approximate estimate of the ratio<br>
of “1” pixels to “0” pixels in your entire training set.  There would be<br>
nothing wrong with counting the exact numbers of "1"s and "0"s in<br>
your training set, but if you want to save time, you could just count<br>
the pixels in a representative sample of your images.  You could<br>
even count a representative sample of pixels within an image (in<br>
your representative sample of images).  You just need to count<br>
enough (representative) pixels that you get a reasonable statistical<br>
estimate of the number of "1"s vs. "0"s.</p>
<p>The reasonable and simple choice for <code>pos_weight</code> is then<br>
<code>number_of_0s / number_of_1s</code>.  (The idea is that if your training<br>
set consists of almost all "0"s, then your classifier will do a very<br>
good job by <em>always</em> predicting “0” – clearly not what you want.<br>
The above choice of <code>pos_weight</code> means that in aggregate your<br>
“0”-pixels and “1”-pixels will now contribute to your loss function<br>
– and hence your training – with approximately equal weight.)</p>
<p>(A side note on a potentially confusing bit of terminology.  The “number<br>
of classes” for the length of the <code>pos_weight</code> vector in your use case<br>
is <code>1</code>.  The one class is, let’s call it, “foreground”, and a pixel can be<br>
either the binary choice “yes foreground” or “no foreground”.  Your<br>
network will provide <em>one</em> logit output per pixel.  <em>If</em> we were<br>
speaking in terms of a multi-class classifier (using, for example,<br>
<code>CrossEntropyLoss</code>) we would say you had <em>two</em> classes,<br>
“yes foreground” and “no foreground.”  The reason we don’t<br>
stamp out this confusing terminology for <code>BCEWithLogitsLoss</code><br>
is that you can use <code>BCEWithLogitsLoss</code> for a <em>multi-label,<br>
multi-class</em> classification problem where you can have <code>n</code> classes,<br>
any number of which can be “yes” or “no” at the same time.  This<br>
is then a set of <code>n</code> binary classification problems, but we (smartly)<br>
train a single classifier network that performs the <code>n</code> binary<br>
classifications at the same time, (usefully) sharing features and<br>
training, etc.  In this case we say we have <code>n</code> classes, not <code>2^n</code><br>
classes – that is, we don’t count the number of combinations of<br>
"yes"s and "no"s as the number of classes.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: roryw10
cooked: <p>Thanks <a class="mention" href="/u/kfrank">@KFrank</a> that is a very clear and concise reply and makes things very easy to comprehend.</p>
<p>The second part in terms of the labels also sheds a bit of light on another problem i’m having with the shape of the input based around different loss functions so will give me a solid place to start trying to unravel that too!</p>

username: roryw10
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> ok - so the second point you made about the number of classes really helped me clear something up i was struggling with as i had wrongly put in a [‘class1’, ‘class 2’] rather than just ‘class1’ because of the background issue you mentioned.</p>
<p>I tried using pytorchs BCEWithLogitsLoss out the box however i was getting an error “expecting target data type as long but BCElosswithlogistic loss function expecting as float datatype”</p>
<p>To get around this, i tried to define my own loss function with a long data type, however i’m back at square one in terms of calculating pos_weight (though from a much more informed perspective which is great!).</p>
<p>I understand you said there isnt a magical right way to do this, but is there even any best practice / automated way to get this? I’ve got a folder with X images in and if each image has N pixels - some are 1, the only way I could see to do this is to turn each image into an array and then iterate over all the images in the folder to see how many 0s and how many 1s?</p>

username: KFrank
cooked: <p>Hello Rory!</p>
<aside class="quote no-group" data-username="roryw10" data-post="4" data-topic="71667" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3bc359/48.png" class="avatar"> roryw10:</div>
<blockquote>
<p>I tried using pytorchs BCEWithLogitsLoss out the box however i was getting an error “expecting target data type as long but BCElosswithlogistic loss function expecting as float datatype”</p>
</blockquote>
</aside>
<p>The error message you quote seems somewhat garbled.  However,<br>
I think your issue is this:  <code>BCEWithLogitsLoss</code> expects a target of<br>
type float, rather than a long.  So convert your target tensor from<br>
long to float.</p>
<p>(You can think of integer targets as being class labels: <code>0</code> means “no,”<br>
and <code>1</code> means “yes.”  But you can also understand them as probabilities:<br>
<code>0.0</code> mean 0% probability of being “yes” (thus, “no”), and <code>1.0</code> means<br>
100% probability of being “yes” (thus “yes”).  Unlike something like<br>
<code>CrossEntropyLoss</code> that requires its targets to be integer class labels,<br>
<code>BCEWithLogitsLoss</code> <em>does</em> accept non-integer probabilities for its<br>
targets.  So you can have <code>0.15</code> (probably “no”) or <code>0.90</code> (probably<br>
“yes”).  This is a good thing, but in order to support this use case,<br>
<code>BCEWithLogitsLoss</code> requires floats for its targets, even if you are<br>
only passing it <code>0.0</code>s and <code>1.0</code>s.)</p>
<aside class="quote no-group">
<blockquote>
<p>the only way I could see to do this is to turn each image into an array and then iterate over all the images in the folder to see how many 0s and how many 1s?</p>
</blockquote>
</aside>
<p>Well, yes, at some point you have to count the 0s and 1s, at least for a<br>
representative sample of your data.</p>
<p>If I were doing it, I would recognize that my pytorch code is somewhere<br>
already converting those images into target tensors that consist of 0s<br>
and 1s.  Then <code>target.mean()</code> will give you the fraction of 1s in that<br>
target tensor.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: roryw10
cooked: <p>hmmm ok,</p>
<p>well i think i’m running in to another headache which links to the long vs float issue.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0b680005080f8b0126b96de3a5c4806d1584ea8.png" data-download-href="https://discuss.pytorch.org/uploads/default/f0b680005080f8b0126b96de3a5c4806d1584ea8" title="Screenshot 2020-03-08 at 20.40.27"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0b680005080f8b0126b96de3a5c4806d1584ea8_2_550x500.png" alt="Screenshot 2020-03-08 at 20.40.27" data-base62-sha1="ylrB2gKpmJpWC8CDWiTPIdqEXDa" width="550" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0b680005080f8b0126b96de3a5c4806d1584ea8_2_550x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0b680005080f8b0126b96de3a5c4806d1584ea8_2_825x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0b680005080f8b0126b96de3a5c4806d1584ea8_2_1100x1000.png 2x" data-dominant-color="E4EBF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-03-08 at 20.40.27</span><span class="informations">1530×1390 174 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So it looks like again this is a case of after i open the masks i need to convert these tensors into floats</p>

username: Patrice
cooked: <p>Hello Mr. Frank,</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="71667">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If I were doing it, I would recognize that my pytorch code is somewhere<br>
already converting those images into target tensors that consist of 0s<br>
and 1s. Then <code>target.mean()</code> will give you the fraction of 1s in that<br>
target tensor.</p>
</blockquote>
</aside>
<p>I have applied target.mean() as you recommended, to determine the fraction of 1s in the target tensor and it gave me very small values (0.2128, 0.099, 0.0421) for different tensors (images). I have used 0.2128 as the pos_weight. But, I could not notice any changes during the training process. Please, could you provide with some clarification?</p>

username: KFrank
cooked: <p>Hi Patrice!</p>
<aside class="quote no-group" data-username="Patrice" data-post="7" data-topic="71667" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e274bd/48.png" class="avatar"> Patrice:</div>
<blockquote>
<p>I have applied target.mean() as you recommended, to determine the fraction of 1s in the target tensor and it gave me very small values (0.2128, 0.099, 0.0421) for different tensors (images). I have used 0.2128 as the pos_weight.</p>
</blockquote>
</aside>
<p>You’re doing it backwards.  You have fewer 1s than 0s, but by using<br>
<code>0.2128</code> for <code>pos_weight</code> you are further reducing the weight of the 1s<br>
pixels in your loss function.</p>
<p>Let’s say that averaged over multiple images your fraction of 1s is<br>
about <code>0.10</code>.  Then, approximately, you would want to use a value<br>
of <code>pos_weight</code> of <code>1.0 / 0.10 = 10.0</code> to <em>increase</em> the weight of<br>
the 1s.</p>
<p>More precisely (but we don’t need to be precise about the value of<br>
<code>pos_weight</code>), you would use <code>pos_weight = (1.0 - 0.10) / 0.10</code>.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Patrice
cooked: <p>Hello Mr. Frank!</p>
<p>Thank you very much for your continuous help. I have conducted some experiments passing a value to pos_weight as follows.</p>
<pre><code class="lang-auto">pos_weight = (1.0 - 0.2128) / 0.2128 = 3.69
</code></pre>
<p>However, the loss instead got higher. Please, what could be the reason?<br>
From my understanding, target.mean() returns the fraction of 1s in the label. Given that my class of interest (foreground) is 1s, why is that the fraction of 1s cannot be directly passed to pos_weight?</p>
<p>Please, could anyone provide me with some clarification?</p>