How to go about Unbalanced data in a binary classification in Pytorch?

Shisho_Sama: <p>Hello everyone.<br>
I’m dealing with a binary classification problem that has imbalanced data. basically the dataset is about identifying <strong>live</strong> vs <strong>impersonate</strong> . basically its a real vs fake classification.<br>
There are already many questions regarding this in the forum, but I’m kind of lost at the moment and I’m not sure if I’m doing it wrong or the issue stems from somewehre/something else.</p>
<p>Here is how the training set distribution looks like :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c109cfbdcdc942dbce4cd77e64fc274714708c08.png" data-download-href="https://discuss.pytorch.org/uploads/default/c109cfbdcdc942dbce4cd77e64fc274714708c08" title="Figure_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c109cfbdcdc942dbce4cd77e64fc274714708c08_2_690x367.png" alt="Figure_1" data-base62-sha1="rxHaIj6oOArpU38SC3avIswMkKk" width="690" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c109cfbdcdc942dbce4cd77e64fc274714708c08_2_690x367.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c109cfbdcdc942dbce4cd77e64fc274714708c08_2_1035x550.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c109cfbdcdc942dbce4cd77e64fc274714708c08_2_1380x734.png 2x" data-dominant-color="F2E9E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_1</span><span class="informations">1500×798 20.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
There are 505 samples for real and 1558 samples for fake class. totaling to 2063 samples in the whole dataset.<br>
Now there are two ways to tackle this issue as far as I know, oversampling and weighted loss.<br>
I went for the oversampling solution and tried to use <strong>WeightedRandomSampler</strong> .<br>
This is how I implemented the sampler:</p>
<pre data-code-wrap="python"><code class="lang-python">dt_train = FASDataset(root=os.path.join(cdir, 'CASIA_faceAntisp', 'train_release_imgs'), transformations=trans_train)

num_classes = 2
class_sample_counts = [dt_train.real_sample_count, dt_train.fake_sample_count]
# compute weight for all the samples in the dataset
# samples_weights contain the probability for each example in dataset to be sampled  
# shouldnt we do highestclass/class_c for all classes? that is do :
# [dt_train.fake_sample_count/dt_train.real_sample_count, dt_train.fake_sample_count/dt_train.fake_sample_count]
class_weights = 1./torch.Tensor(class_sample_counts)
# get list of all labels 
train_targets = dt_train.get_labels(True)
# then get the weight for each target!
train_samples_weight = [class_weights[class_id] for class_id in train_targets]
train_sampler = WeightedRandomSampler(train_samples_weight, len(train_samples_weight), replacement=True)
dl_train = torch.utils.data.DataLoader(dt_train, sampler=train_sampler, batch_size=32, pin_memory=True, shuffle=(train_sampler==None))

</code></pre>
<p>and the dataset itself looks like this :</p>
<pre data-code-wrap="python"><code class="lang-python">class FASDataset(torch.utils.data.Dataset):
    def __init__(self, root, transformations=tf.ToTensor()):
        """            
        """
        super().__init__()
        # read the files for training and testsets
        self.root= root
        self.transforms = transformations
        self.img_list = []
        self.real_sample_count = 0
        self.fake_sample_count = 0
        self.num_classes = 2
        self.classes = {0:'fake', 1:'live'}

        for dir_path, dirnames, filenames in os.walk(root):
            for filename in filenames:
                img_path = os.path.join(dir_path, filename)
                label = self._get_label(img_path)
                self.img_list.append((img_path, label))
                self.real_sample_count += int(label)

        self.fake_sample_count = len(self.img_list) - self.real_sample_count

    def plot_data_distribution(self):
        plt.figure(figsize=(15,8))
        sns.barplot(data=pd.DataFrame.from_dict([self.class_distributions]).melt(), 
                    x="variable", y="value", hue="variable").\
                    set_title('Fake/Real Class Distribution')
        plt.show()

    def get_labels(self, return_as_ints=False):
        integer = lambda x : int(x) if return_as_ints else x
        return [integer(label) for _,label in self.img_list]

    def _get_label(self, image_full_path):
        filename = os.path.split(image_full_path)[-1]
        # only the files starting with 1_, 2_ and HR_1 are real/positive samples, the rest are negative samples
        # for example, 1_img_0.jpg, 1_img_1.jpg, ... 2_img_0.jpg, ... HR_1_img.jpg are positive examples
        # and 3_img_0.jpg ..., HR_2_img_0.jpg, etc are negative examples
        return float(re.match(r"((^1_)|(^2_)|(^HR_1))", filename) != None)

    def __getitem__(self, index):
        (image_file, label) = self.img_list[index]
        img = Image.open(image_file)
        if self.transforms:
            img = self.transforms(img)
        return img, label

    def __len__(self):
        return len(self.img_list)
</code></pre>
<p>and my forward looks like this :</p>
<pre data-code-wrap="python"><code class="lang-python">class MyNet(nn.Module) : 
    def __init__(self)
        super().__init__()
        self.features = ...
        self.classifier = nn.Linear(32, 1)

    def forward(self, input_batch):
        output = self.features(input_batch)
        output = F.max_pool2d(output, kernel_size=output.size()[2:])
        
        output = output.view(-1, 32)
        output = self.classifier(output)
        # we use BCEwithlogits for more numerically stable training, so we dont use sigmoid here
        # output = torch.sigmoid(output)
        return output

    def predict(self, input_batch):
        out_raw = self.forward(input_batch)
        out_sig = torch.sigmoid(out_raw)
        preds = torch.round(out_sig)
        return preds
</code></pre>
<p>and the training :</p>
<pre data-code-wrap="python"><code class="lang-python">def train_val(model, dataloader, optimizer, is_training, device, interval):
    batch_cnt = len(dataloader)
    status = 'Training' if is_training else 'validation'
    # we use BCEWithLogits which uses sigmoid internally
    criterion = torch.nn.BCEWithLogitsLoss()
    total_loss = 0.0
    accuracy = 0.0
    with torch.set_grad_enabled(is_training):
        model.train() if is_training else model.eval()
        for i, (imgs, labels) in enumerate(dataloader):
            imgs = imgs.to(device)
            preds = model(imgs)
            loss = criterion(preds.cpu().view(*labels.shape), labels.cpu())
            total_loss += loss.item()
            if is_training:
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
            accuracy += binary_acc(preds.cpu(), labels.cpu())

    accuracy = accuracy/batch_cnt
    total_loss = total_loss/batch_cnt
    print(f'[{status}] acc: {accuracy:.2f} loss: {total_loss:6f}')
    return accuracy, total_loss
</code></pre>
<p>So this is the basic building block that I’m using and as you can see they are pretty normal.<br>
Now I have some questions concerning all of this :<br>
First of all, am I doing it correctly ? is the way I’m creating the <strong>WeightedRandomSampler</strong> alright?<br>
For example :<br>
1.Shouldnt we for example divide all the sample counts from the highest class sample count? I mean doing</p>
<pre data-code-wrap="python"><code class="lang-python">class_weights = [dt_train.fake_sample_count/dt_train.real_sample_count, dt_train.fake_sample_count/dt_train.fake_sample_count]
</code></pre>
<p>instead of:</p>
<pre><code class="lang-auto">class_weights = 1./torch.Tensor(class_sample_counts)
</code></pre>
<p>and<br>
2. Shouldn’t we use the <strong>batch_size</strong> for the <strong>WeightedRandomSampler</strong> instead of the the actual <strong>len(train_samples_weight)</strong> ?</p>
<p>Becasue if I use the later, the network achives a very high acc (100% train and 96% validation, and as you can guess, it just predicts the fake class!) if I use batch_size, the accuracies drop but ultimately, it will reaach 99% train and 93% validation whcih this time, it predicts all as live class!!</p>
<p>what am I doing wrong ? what am I missing here?<br>
If I want to add the weights to the <strong>BCEWithLogitsLoss</strong> what should I be doing?<br>
As there are two arguemnts: <strong>weight</strong> and <strong>pos_weight</strong> and its not clear from the documentation which one to use and how.<br>
For example, how is the network going to know, which class is positive and which one is negative (or in other words, which one is underrepresented and which one isnt) ?<br>
3.is the same weight that is used with WerightedRandomSampler, can be used here with the loss?</p>
<p>Thanks a lot in advance.</p>

Shisho_Sama: <p>OK, looks like, the WeightedRandomSampler is correctly set up as the distribution of labels in each batch is considerably more balanced than before :<br>
<strong>Before using the WeightedRandomSampler:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58e14b4069eed8b34a927e795c3de5b0af5e4b70.png" data-download-href="https://discuss.pytorch.org/uploads/default/58e14b4069eed8b34a927e795c3de5b0af5e4b70" title="BeforeWeightedRandomSampler"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58e14b4069eed8b34a927e795c3de5b0af5e4b70_2_690x364.png" alt="BeforeWeightedRandomSampler" data-base62-sha1="cGgHRuECfEuTskKDATryljyxmFO" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58e14b4069eed8b34a927e795c3de5b0af5e4b70_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58e14b4069eed8b34a927e795c3de5b0af5e4b70_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58e14b4069eed8b34a927e795c3de5b0af5e4b70_2_1380x728.png 2x" data-dominant-color="C3ADE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">BeforeWeightedRandomSampler</span><span class="informations">1430×756 12.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<strong>After using WeightedRandomSampler:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cebd94b7c7851ef5f96e7b8234886e5bd6b4023.png" data-download-href="https://discuss.pytorch.org/uploads/default/5cebd94b7c7851ef5f96e7b8234886e5bd6b4023" title="AfterWeightedRandomSampler"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cebd94b7c7851ef5f96e7b8234886e5bd6b4023_2_690x364.png" alt="AfterWeightedRandomSampler" data-base62-sha1="dg1ek33q8F6iugs5Qo5zrkfDwd5" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cebd94b7c7851ef5f96e7b8234886e5bd6b4023_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cebd94b7c7851ef5f96e7b8234886e5bd6b4023_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cebd94b7c7851ef5f96e7b8234886e5bd6b4023_2_1380x728.png 2x" data-dominant-color="BE8BCB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">AfterWeightedRandomSampler</span><span class="informations">1430×756 12.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The whole training set:<br>
Before:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf19b34bd85c1d5a182feb63bb7a62e31b719cea.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf19b34bd85c1d5a182feb63bb7a62e31b719cea" title="BeforeWeightedRandomSampler_all"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf19b34bd85c1d5a182feb63bb7a62e31b719cea_2_690x364.png" alt="BeforeWeightedRandomSampler_all" data-base62-sha1="ty5TKsTkBKQwoQH1qQHIgCAWV9E" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf19b34bd85c1d5a182feb63bb7a62e31b719cea_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf19b34bd85c1d5a182feb63bb7a62e31b719cea_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf19b34bd85c1d5a182feb63bb7a62e31b719cea_2_1380x728.png 2x" data-dominant-color="C1AFEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">BeforeWeightedRandomSampler_all</span><span class="informations">1430×756 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
After :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/3126b4bb57a2e96d2b54839c39030ebc2048a097.png" data-download-href="https://discuss.pytorch.org/uploads/default/3126b4bb57a2e96d2b54839c39030ebc2048a097" title="AfterWeightedRandomSampler_all"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3126b4bb57a2e96d2b54839c39030ebc2048a097_2_690x364.png" alt="AfterWeightedRandomSampler_all" data-base62-sha1="70Ok2x0G6uHq5Okscpt94lx72IL" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3126b4bb57a2e96d2b54839c39030ebc2048a097_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3126b4bb57a2e96d2b54839c39030ebc2048a097_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/3126b4bb57a2e96d2b54839c39030ebc2048a097_2_1380x728.png 2x" data-dominant-color="C78EC5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">AfterWeightedRandomSampler_all</span><span class="informations">1430×756 15.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I guess I got my answer to the first question.<br>
For the second question concerning the class_weight I noticed something strange:</p>
<pre data-code-wrap="python"><code class="lang-python">class_weights = 1./torch.Tensor(class_sample_counts)
</code></pre>
<p>results in this :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c4a452a4d82be2d96e4f09a4ec6493e41f4c318.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c4a452a4d82be2d96e4f09a4ec6493e41f4c318" title="class_weights_1_over"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c4a452a4d82be2d96e4f09a4ec6493e41f4c318_2_690x364.png" alt="class_weights_1_over" data-base62-sha1="hJwkqqsQgDgGb8TyPMQ2HzMFMhy" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c4a452a4d82be2d96e4f09a4ec6493e41f4c318_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c4a452a4d82be2d96e4f09a4ec6493e41f4c318_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c4a452a4d82be2d96e4f09a4ec6493e41f4c318_2_1380x728.png 2x" data-dominant-color="B5AFF7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">class_weights_1_over</span><span class="informations">1430×756 16.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
but when I calculated the class_weights like this :</p>
<pre data-code-wrap="python"><code class="lang-python"># get list of all labels 
train_targets = dt_train.get_labels(True)
labels, counts = np.unique(train_targets, return_counts=True)
class_weights = [sum(counts)/c for c in counts]
</code></pre>
<p>This is the result :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b7137e110d939a94685bfee5412f17219a8df53.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b7137e110d939a94685bfee5412f17219a8df53" title="class_weights_using_numpy_unique_on_labels"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b7137e110d939a94685bfee5412f17219a8df53_2_690x364.png" alt="class_weights_using_numpy_unique_on_labels" data-base62-sha1="3ULsz6bUSX1wmsPrOKm5ap0UZGj" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b7137e110d939a94685bfee5412f17219a8df53_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b7137e110d939a94685bfee5412f17219a8df53_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b7137e110d939a94685bfee5412f17219a8df53_2_1380x728.png 2x" data-dominant-color="C489C2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">class_weights_using_numpy_unique_on_labels</span><span class="informations">1430×756 16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>No wonder I was getting skewed results when using WeightedRandomSampler.</p>
<p>And for the second question : the batch_size vs the length of weighted_labels, the answer is, the length of weighted labels is necessary as it provides you with the whole data you have. if you use batch_size, you’ll only be using the batch_size number of data and thats done in one go! so the length of weighted_labels (or the number of labels) should be sent as the right number of samples ( hence the name <code>num_samples</code>)!</p>
<p>also here is the snippet of code I wrote for the batch distribution visualization that you see here. it might come in handy for someone out there specially if they are not well versed with plotting like me.<br>
Pardon me for my lack of experience with matplotlib and plotting in general! :</p>
<pre data-code-wrap="python"><code class="lang-python">zeros = []
ones = []
for img, label in dl_train:
    lbl = list(label.detach().numpy())
    res, cnts = np.unique(lbl, return_counts=True)
    if len(res) == 2:
        zeros.append(cnts[0])
        ones.append(cnts[1])
    else:
        if res == 0:
            zeros.append(cnts[0])
            ones.append(0)
        else:
            zeros.append(0)
            ones.append(cnts[0])

num_samples = len(zeros)
plt.figure(figsize=(15, 8))
plt.bar(np.arange(len(ones[:num_samples])), zeros[:num_samples], color='b', label='fake', width=0.35)
plt.bar(np.arange(len(ones[:num_samples]))+0.35, ones[:num_samples], color='r', label='real', width=0.35)
plt.xlabel('fake-real')
plt.ylabel('counts')
plt.title('distribution')
plt.legend()
plt.tight_layout()
plt.show()
</code></pre>
<p>also concerning the weights for losses see <a href="https://discuss.pytorch.org/t/dealing-with-imbalanced-datasets-in-pytorch/22596/2">this</a></p>

yogaaltariz: <p>hello, i have a question. how about the result after using this sampler method? thanks</p>

Shisho_Sama: <p>Do you mean if it was beneficial to the training? Yes, it resulted in a much higher accuracy and a much better model if my memory serves me right.</p>

yogaaltariz: <p>thanks for the quick response</p>