How to make use of class weights + data augmentation together?

stark: <p>Does <a href="https://pytorch.org/vision/stable/transforms.html" rel="noopener nofollow ugc">torchvision.transforms</a> or <a href="https://albumentations.ai/" rel="noopener nofollow ugc">albumentations</a> provide some means to make use of the <code>class weights</code> calculated for the training data?</p>
<p>There are several arguments against using <code>class weights</code> in loss functions and performing data augmentation at the same time. Mostly the same reason is stated that</p>
<blockquote>
<p>the augmentation process changes the original class distribution</p>
</blockquote>
<p>This made me look for ways to force the augmentation process to consider minority class samples more often than the majority class samples.</p>
<p>What if <code>class weights</code> are computed on the fly for an augmented batch of training examples (as opposed to the traditional way of calculating for the whole training data)? This way both techniques may work together.</p>
<p>Your experiences with the data augmentation to counter the class imbalance problem will be highly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

Manuel_Alejandro_Dia: <p>If you are using <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc"><code>CrossEntropyLoss</code></a>, my idea is that you could change the <code>weight</code> parameter on the fly for your batch. Something along the lines of:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropy()

....
for input, target in dataset:
    cel_weights = calculate_weights(target)
    criterion.weights = cel_weights
    ...
</code></pre>
<p>If you are not using CEL, maybe check if your criterion has the weight parameter, so you could change it.</p>

stark: <p>Thanks, <a class="mention" href="/u/manuel_alejandro_dia">@Manuel_Alejandro_Dia</a> for your reply!</p>
<p>Let me rephrase the questions:</p>
<ol>
<li>Is it a general rule that the class weights should be <strong>pre-calculated</strong> for the <em>entire training data</em> rather than calculating for <em>each batch</em> <strong>on the fly</strong>?</li>
<li>Is the use of <em>“class weights”</em> in loss calculation and <em>“data augmentation”</em> <strong>mutually exclusive</strong>?</li>
<li>Should one use weights calculated from each input batch of <strong>A)</strong> augmented images or <strong>B)</strong> images before augmenting them?</li>
</ol>

Manuel_Alejandro_Dia: <aside class="quote no-group" data-username="stark" data-post="3" data-topic="114940">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<ol>
<li>Is it a general rule that the class weights should be <strong>pre-calculated</strong> for the <em>entire training data</em> rather than calculating for <em>each batch</em> <strong>on the fly</strong>?</li>
</ol>
</blockquote>
</aside>
<p>From my experience, yes. Here you just do the processing of the weights in the dataset once, instead of doing it on each batch.</p>
<aside class="quote no-group" data-username="stark" data-post="3" data-topic="114940">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<ol start="2">
<li>Is the use of <em>“class weights”</em> in loss calculation and <em>“data augmentation”</em> <strong>mutually exclusive</strong>?</li>
</ol>
</blockquote>
</aside>
<p>That is a good question. I would assume that depending on your data augmentation you would have to decide if you recalculate or not your weights. For example, the usual data augmentation I do in images is a horizontal flip and re-scaling, and I would argue that by doing these I am not modifying the <strong>ratio</strong> of each by such an amount that would I would need to recalculate the class weights.</p>
<aside class="quote no-group" data-username="stark" data-post="3" data-topic="114940">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<ol start="3">
<li>Should one use weights calculated from each input batch of <strong>A)</strong> augmented images or <strong>B)</strong> images before augmenting them?</li>
</ol>
</blockquote>
</aside>
<p>If you think that the data augmentation changes somehow the ratios between classes by a considerable amount, then I would suggest you to do one of these things:</p>
<ul>
<li>Update the weights on the fly, but be aware that some classes might not be present when you do the weight recalculation.</li>
<li>Apply the augmentation to your entire dataset and get the weights from there. But since some augmentations are based on random values, I would suggest you run this a few times to get an average of the class weights after augmentation.</li>
</ul>
<p>Thank you for the questions!<br>
I hope this helps!</p>