How to decide weight parameters of nn.Cross_entropy() correctly

bing: <p>I am solving a multi-class image segmentation problem.<br>
Classes are imbalanced.<br>
I am trying to calibrate weight parameter of nn.Cross_entropy() using the below weight -</p>
<pre><code class="lang-auto">weights = torch.tensor([0.90,0.40,0.40,0.20])
criterion = torch.nn.CrossEntropyLoss(weights)
</code></pre>
<p>Above number , I used were arbitrary.<br>
On calculating number of different classes in a single image, following it the result -<br>
<code>(tensor([0, 1, 2, 3]), tensor([305903,  12655,  10342,   1060]))</code></p>
<p>Generally, the above values will vary for each image but class 0 will remain in abundance(it is background), class 1 and class 2 in moderation( desired segmented classes), class 3 will be very less( it is for unknown cases.<br>
How to choose weights values for it?</p>

KFrank: <p>Hi Mr. Bing!</p>
<aside class="quote no-group quote-modified" data-username="bing" data-post="1" data-topic="68996" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<p>Classes are imbalanced.</p>
<pre><code class="lang-auto">weights = torch.tensor([0.90,0.40,0.40,0.20])
criterion = torch.nn.CrossEntropyLoss(weights)
</code></pre>
<p>On calculating number of different classes in a single image, following it the result -<br>
<code>(tensor([0, 1, 2, 3]), tensor([305903,  12655,  10342,   1060]))</code><br>
…<br>
How to choose weights values for it?</p>
</blockquote>
</aside>
<p>There is no single right answer for choosing weights to reweight an<br>
unbalanced data set.  But a good starting point is:<br>
<code>weight_for_class = 1 / number_of_samples_in_class</code>.</p>
<p>The idea is that you might want each class to be making about the<br>
same overall contribution to the loss function – so, more samples<br>
in a class means you use a lower weight.</p>
<p>So, you might start with:</p>
<pre data-code-wrap="python"><code class="lang-python">counts = torch.FloatTensor ([305903, 12655, 10342, 1060])
weights = 1.0 / counts
weights_normalized = (1.0 / counts) / (1.0 / counts).sum()
</code></pre>
<p>(I’m using the old pytorch version 0.3.0.)</p>
<p>You don’t need this, but if you want your weights to be normalized (sum<br>
to one), the last line illustrates this.</p>
<p>This yields:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; counts

 3.0590e+05
 1.2655e+04
 1.0342e+04
 1.0600e+03
[torch.FloatTensor of size 4]

&gt;&gt;&gt; weights

1.00000e-04 *
  0.0327
  0.7902
  0.9669
  9.4340
[torch.FloatTensor of size 4]

&gt;&gt;&gt; weights_normalized

 0.0029
 0.0704
 0.0862
 0.8405
[torch.FloatTensor of size 4]
</code></pre>
<p>Good luck!</p>
<p>K. Frank</p>

moreshud: <p>Hi K. Frank,</p>
<p>Thanks for your explanation. I would like to validate my approach of getting the <code>counts</code> in the sample code above and some of my concerns for volumetric CT data.</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(3)

image = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
labelA = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
labelB = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
</code></pre>
<p>Generating the background label from the annotated label A &amp; B i.e 0 in pixel value where either A or B is 1 and 1 elsewhere</p>
<pre><code class="lang-auto">background = torch.zeros((labelB.shape), dtype=torch.long)

for label in [labelA, labelB]:
    background |= background.long()

background = background^1
outputs = f"Image: {image},\n background: {background}, \n labelA: {labelA}, \n labelB: {labelB}"
print(outputs)
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; outputs
Image: tensor([[0.0043, 0.1056, 0.2858],
        [0.0270, 0.4716, 0.0601],
        [0.7719, 0.7437, 0.5944]]),
 background: tensor([[0, 1, 0],
        [0, 0, 1],
        [0, 0, 0]]), 
 labelA: tensor([[1., 0., 1.],
        [1., 1., 0.],
        [1., 1., 0.]]), 
 labelB: tensor([[0., 0., 0.],
        [1., 1., 0.],
        [0., 0., 1.]])
</code></pre>
<pre><code class="lang-auto">counts = torch.tensor([torch.sum(background), torch.sum(labelA), torch.sum(labelB)])
weights = 1.0 / counts
weights_normalized = (1.0 / counts) / (1.0 / counts).sum()
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; counts
tensor([4., 3., 4.])
&gt;&gt;&gt; weights
tensor([0.2500, 0.3333, 0.2500])
&gt;&gt;&gt;weights_normalized
tensor([0.3000, 0.4000, 0.3000])
</code></pre>
<p>These weights can now be passed into the loss function. My concerns are:</p>
<ul>
<li>I believe my approach of creating the background is not well optimized, is there an inbuilt PyTorch function that can be used or another way around it?</li>
<li>Is it computationally efficient to create the background label class (for each corresponding image and label A/B) in the data loader or separately create it out the data loader, save it (.npy), and then reference it just like the label class A &amp; B?</li>
</ul>
<p>Thanks for your enlightenment</p>