Unbalanced Data, Overfitting occurs

username: skyunyoo
cooked: <p>I am currently working on binary segmentation.</p>
<p>When using unbalance data, overfitting occurs.<br>
Below is a loss curve.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f934cdf848c9055373f4793d8c22a4231eae977.png" alt="다운로드" data-base62-sha1="ku7RNmARimV26IaILihOzx51t8H" width="392" height="278"><br>
I am using BCEWithLogitsLoss, and pos_weight is applied by calculating neg/positive.</p>
<p><strong>The distribution of data has a ratio [data without segments: data with segments] = about 20:1</strong></p>
<p>The learning rate varied from 1e-3 to 1e-7, and weight_decay applied 1e-4 and dropout.</p>
<p>I tried applying the various methods found on the pytorch forum, but the result is not improved.<br>
I think it’s because the data is too unbalanced,<br>
How can I solve it?</p>
<p>Thanks for reading!</p>

username: ptrblck
cooked: <p>Did you check the confusion matrix for the original loss function and the one using <code>pos_weight</code> and did something change?<br>
I would recommend to use unreasonably large or small values for the weighting and make sure that your training achieves the desired effect. You could also try to use <code>WeightedRandomSampler</code> for over-/undersampling.</p>

username: skyunyoo
cooked: <p>Thank you for the reply!</p>
<p>I am applying pos_weight by counting the number of negative and positive numbers as shown below.</p>
<p>As a difference between before and after applying pos_weight, I was confirmed whether training loss (+dice) was reduced.</p>
<pre><code class="lang-auto">        data_loader = torch.utils.data.DataLoader(traindataset, batch_size = batch_size, shuffle=True, num_workers=0, pin_memory=False)
        
        pos = 0.0
        neg = 0.0
        for b, (x,y) in enumerate(data_loader):
            for p in range(len(y)):
                if y[p][0].max()==1:
                    pos+=1
                else:
                    neg+=1
        weights = neg/pos
        pos_weight = torch.tensor(weights)



        for batch_idx , (data,target) in enumerate(data_loader):
            inputs,target = data.to(device),target.to(device) 
         ...
         criterion = nn.BCEWithLogitsLoss(reduction='mean', pos_weight=pos_weight)
</code></pre>
<p>Should I set the value of pos_weight randomly without calculating like this?</p>
<p>I also read about the sampler from the link.<br>
<a href="https://discuss.pytorch.org/t/how-to-prevent-overfitting/1902/6">About WeightedRandomSampler</a></p>
<p>The confusing part of this sampler is when there is llabel(value have 1) and no label(value only 0) (i.e. binary problem),<br>
<code>class_sample_count = []</code> What value should I put in this part?</p>
<p><code>class_sample_count = [positive_count]</code> ? or<br>
<code>class_sample_count = [negative_count,positive_count]</code>?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="skyunyoo" data-post="3" data-topic="84298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>Should I set the value of pos_weight randomly without calculating like this?</p>
</blockquote>
</aside>
<p>Yes, for the same of debugging try out different values and check the confusion matrix to verify that the predictions really change.</p>
<aside class="quote no-group" data-username="skyunyoo" data-post="3" data-topic="84298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>The confusing part of this sampler is when there is llabel(value have 1) and no label(value only 0) (i.e. binary problem),</p>
</blockquote>
</aside>
<p>You should use both classes to calculate the class count and assign the computed weight to each sample.</p>

username: skyunyoo
cooked: <ol>
<li>I tried the following debugging in the same setting.</li>
</ol>
<p>1). pos_weight &lt;0; 0.2 scale<br>
2). pos_weight = None<br>
3). pos_weight &gt;= 1; scale 10 1 to 100</p>
<p>As a result,<br>
In 1) confirmed that the loss value decreased similarly for both train and valid, but it was confirmed that the confusion matrix and dice values did not improve each epoch.</p>
<p>In 2), I saw a lot of changes in both train loss and valid loss.</p>
<p>In 3), similar to the graph in the question, train loss decreased well, but valid loss changed significantly.</p>
<p>I think the 3) case is appropriate, but I wonder if there are other factors that reduce the fluctuation of valid?</p>
<ol start="2">
<li>I configured the code as below to use <code>WeightedRandomSampler</code>, but I am not sure how to load the target because I am using <code>concat dataset</code>.</li>
</ol>
<pre><code class="lang-auto">        train_datasets = []
        for i in range(5):
            secure_random = random.SystemRandom()
            random_patient = secure_random.choice(patient_index)
            train_datasets.append(trainDataset(random_patient,"data_path",augmentation=True))
            patient_index.remove(random_patient)
        traindataset = torch.utils.data.ConcatDataset(train_datasets)
        
        class_sample_count = [neg,pos]  # I already calculated negative/positive count
        weights = 1/torch.tensor(class_sample_count, dtype=torch.float)
        samples_weights = [weights[t] for t in target]
        sampler = torch.utils.data.sampler.WeightedRandomSampler(weights=samples_weights, num_samples=len(samples_weights, replacement=True)
        data_loader = torch.utils.data.DataLoader ....
</code></pre>
<p>Is there a way to load the target in this code <code>samples_weights = [weights[t] for t in target]</code>?</p>
<p>My custom dataset (=<code>trainDataset</code>) follows the structure below.</p>
<pre><code class="lang-auto">class trainDataset(torch.utils.data.Dataset):
    def __init__(self, i, data_path, augmentation=True):
        self.data_path = data_path
        self.data = np.load(data_path)
        self.target = np.load(target_path).astype(np.uint8)
        self.data = self.data-self.data.mean()
        self.data = self.data/self.data.std()
        self.augmentation = augmentation
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        x, y = self.transform(x, y)
            
        return x, y
    
    def transform(self, data, target):
        data, target = data_augmentation(data, target, self.augmentation)
        return data, target
    
    def __len__(self):
        return len(self.data)
</code></pre>
<ol start="3">
<li>In my think, if I use <code>WeightedRandomSampler</code>, use of <code>pos_weight</code> doesn’t seem to make sense. Is this correct?</li>
</ol>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Did your model predictions change from the negative to the positive class or vice verse when you’ve changed the <code>pos_weight</code>?</p>
<aside class="quote no-group" data-username="skyunyoo" data-post="5" data-topic="84298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>pos_weight &lt;0; 0.2 scale</p>
</blockquote>
</aside>
<p>I don’t think a negative <code>pos_weight</code> would make sense, would it?</p>
<aside class="quote no-group" data-username="skyunyoo" data-post="5" data-topic="84298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>Is there a way to load the target in this code <code>samples_weights = [weights[t] for t in target]</code> ?</p>
</blockquote>
</aside>
<p>If you are creating the targets lazily in each iteration, you could create the target tensor once before the training via:</p>
<pre data-code-wrap="python"><code class="lang-python">targets = []
for _, target in dataset:
    targets.append(target)
targets = torch.stack(targets)
</code></pre>
<p>and use it to create the <code>WeightedRandomSampler</code>.</p>
<aside class="quote no-group" data-username="skyunyoo" data-post="5" data-topic="84298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>In my think, if I use <code>WeightedRandomSampler</code> , use of <code>pos_weight</code> doesn’t seem to make sense. Is this correct?</p>
</blockquote>
</aside>
<p>Might be the case, but I would recommend to run experiments and verify that each step is working as expected.</p>

username: skyunyoo
cooked: <p>Oh, I wrote the wrong thing about 1). Correct with <code>0&lt;pos_weight &lt;1</code>.</p>
<p>The model’s prediction changes from 0. to &lt;1 unless the value of pos_weight is less than 1.</p>
<p>Here is an example of learning when pos_weight is greater than 1.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec786f34b4064a5b8148d1b49bb92ab53ec9f89b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec786f34b4064a5b8148d1b49bb92ab53ec9f89b" title="0608-1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec786f34b4064a5b8148d1b49bb92ab53ec9f89b.png" alt="0608-1" data-base62-sha1="xJUIaYGXXHaSChdnyBoheNYOn0L" width="252" height="500" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">0608-1</span><span class="informations">349×691 2.53 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6c28ee89f17f030c238a379a782c2aad8041a11.png" alt="0608-2" data-base62-sha1="q4LJNEa2xXPD208dhknDpsxIv6N" width="229" height="320"><br>
As you can see from the above valid loss value, there was a change in the value that increased and decreased at a certain moment.</p>
<p>In addition, when Icreate targets using the method you informed, I get the following error.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7de12e683468acb26b123ccdb84c3d118eb2147.png" alt="캡처" data-base62-sha1="nX1wLKwPGYyoh6M1yugsdR02J4b" width="450" height="134"></p>
<p>Is it wrong to create targets and weight each sample with <code>samples_weights = [weights[t] for t in targets]</code>?</p>
<p>Thanks!</p>