How to balance mini-batches during each epoch

username: twister9458
cooked: <p>Hello All;</p>
<p>I have a very unbalanced dataset, which I tried to balance using the following code (Class Dataset then my code):</p>
<pre><code class="lang-auto">class myDataset(Dataset):
    def __init__(self, csv_file, root_dir, target, length, transform=None):
        self.annotations = pd.read_csv(csv_file).iloc[:length,:]
        self.root_dir = root_dir
        self.transform = transform
        self.target = target
        self.length = length

    def __len__(self):
        return len(self.annotations)
    
    def __alltargets__(self):
        return self.annotations.loc[:,self.target]

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.loc[index, 'image_id'])
        image = Image.open(img_path)
        image = np.array(image)

        if self.transform:
            image = self.transform(image=image)["image"]

        image = np.transpose(image, (2, 0, 1)).astype(np.float32)
        image = torch.tensor(image)# device=torch.device('cuda:0'))

        y_label = torch.tensor(int(self.annotations.loc[index, str(self.target)]))# device=torch.device('cuda:0'))

        return image, y_label
</code></pre>
<p>And then my code:</p>
<pre><code class="lang-auto">aug = al.Compose([
    al.RandomResizedCrop(H, W, p=0.2),
    al.Resize(H, W),
    al.Transpose(p=0.2),
    al.HorizontalFlip(p=0.5),
    al.VerticalFlip(p=0.2),
    al.augmentations.Normalize(max_pixel_value=255.0, 
                               always_apply=True, 
                               p=1.0)
])

dataset = myDataset(csv_file=LABEL_PATH,
                    root_dir=IMAGE_PATH,
                    target='gender',
                    length=LENGTH,
                    transform=aug)

l = dataset.__len__()
y = dataset.__alltargets__()

train_idx, valid_idx = train_test_split(np.arange(l), test_size=0.2, shuffle=True, stratify=y)

train_sampler = torch.utils.data.SubsetRandomSampler(train_idx)
test_sampler = torch.utils.data.SubsetRandomSampler(valid_idx)


train_loader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=False, pin_memory=True, num_workers=4, sampler=train_sampler)
test_loader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=False, pin_memory=True, num_workers=4, sampler=test_sampler)
</code></pre>
<p>My Question please: My actual code is just splitting evenly the classes among train and test datasets. How can I make the mini-batches balanced also ?</p>
<p>Thank you very much,<br>
Habib</p>

username: ptrblck
cooked: <p>You could use a <code>WeightedRandomSampler</code> as described in <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this post</a> with an example.<br>
If you want to split the dataset in a stratified way, you could use e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"><code>sklearn.model_selection.train_test_split</code></a> with the <code>stratify</code> option for the indices of the dataset (and the targets as the inputs for <code>stratify</code>) and use these indices in <code>Subset</code>s.</p>

username: twister9458
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your response.</p>
<p>In my code above, I’ve already used the <code>stratify</code> option of the method <code>train_test_split</code>. So my Train and Test are both well balanced.</p>
<p>After that, I used SubsetRandomSampler based on these well balanced indexes.</p>
<p>I’m struggling on how to add WeightedRandomSampler on top of SubsetRandomSampler. Thank you for your help.<br>
As you can see, all my data are initially in the <code>dataset</code> placeholder, which I have to evenly split for overfit monitoring, my issue is that I don’t know how to do WeightedRandomSampler on Train dataset only.</p>
<p>Thank you very much,<br>
Habib</p>

username: ptrblck
cooked: <p>You could create the <code>weights</code> using the code snippet from my previous post and replace the <code>SubsetRandomSampler</code> with the <code>WeightedRandomSampler</code> using the <code>train_idx</code>.<br>
Since you are already using a subset via the <code>train_idx</code>, note that you would also only need to calculate the weights for these indices and use it in the <code>WeightedRandomSampler</code>.</p>

username: Animesh_Kumar_Paul
cooked: <p>I am Unable to get a Balanced batch from WeightedRandomSampler.</p>
<p>The below code is providing the mini-batch with 60% of bigger class and 40% of the smaller class, not 50-50%</p>
<pre><code class="lang-auto">from torch.utils.data import WeightedRandomSampler, DataLoader, TensorDataset
class_counts=[sum(Y==0),sum(Y==1)]
weights=[1/float(class_counts[i]) for i in range(len(class_counts))]


samples_weights_train = np.array([weights[int(t)] for t in y_train])
sampler_train = WeightedRandomSampler(weights=samples_weights_train, num_samples=len(y_train)*2, replacement=True)
</code></pre>
<p>Could you please help?</p>

username: sidra_Aleem
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I have a similar situation where I have created stratified train and validation sets, and my dataset has an equal number of samples for each class in both sets. However, at the mini-batch level, the samples are not balanced. I followed your suggested approach of using class weights, and since my dataset is balanced, both classes have the same weight, i.e., 0.004. When I checked the sample distribution at the batch level, it still wasn’t balanced.</p>
<pre><code class="lang-auto">class_sample_count = torch.tensor([labels.count(i) for i in set(labels)])

    # Inverse of class counts
    weights = 1.0 / class_sample_count[labels]
</code></pre>
<pre><code class="lang-auto">epoch  batch class_0 class_1
0	0	3	5
0	1	2	6
0	2	3	5
0	3	4	4
0	4	3	5
0	5	5	3
0	6	6	2
0	7	6	2
0	8	4	4
</code></pre>

username: ptrblck
cooked: <p>The sampling process is still random so small differences in the class distribution are expected.</p>