WeightedRandomSampler not sampling balanced batches

manelpk: <p>Hello there!</p>
<p>I’ve been trying to get a WeightedRandomSampler to work but somehow allways end up not getting the expected output… Following some posts here on the PyTorch forums this is what I have so far:</p>
<p>The dataset I am working with contains both the training and validation sets and is highly unbalanced: 0/1 -&gt; 7153/1532<br>
First, to separete the data into two sets I run the following code:</p>
<pre><code class="lang-auto">validation_split = .1
dataset_size = len(labels)
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))
np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]

training_images = np.array(images_path)[train_indices]
training_labels = np.array(labels)[train_indices]
validation_images = np.array(images_path)[val_indices]
validation_labels = np.array(labels)[val_indices]
</code></pre>
<p>Here, <strong>labels</strong> is a list of all the labels present in the dataset and <strong>images_path</strong> a list containing the path to each image in the same order as the labels.<br>
Then I create the two distinct datasets:</p>
<pre><code class="lang-auto">data_transform = transforms.Compose([RandomCrop(size=IMG_SIZE-20, padding=(10,10)),
                                    Resize(IMG_SIZE),
                                    RandomRotation(20),
                                    ToTensor(),
                                    Normalize()])
training_dataset = CustomDataset(images_path=training_images, labels=training_labels, transform=data_transform)
validation_dataset = CustomDataset(images_path=validation_images, labels=validation_labels)
</code></pre>
<p>CustomDataset is definde as follows:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
  def __init__(self, images_path, labels, transform=None):
    self.labels = labels
    self.images_path = images_path
    self.transform = transform

  def __len__(self):
    return len(self.labels)

  def __getitem__(self, idx):
    if torch.is_tensor(idx):
      idx = idx.tolist()
    img = Image.open(images_path[idx])
    label = labels[idx]
    if label == 1 and self.transform:
      sample = {'image': img, 'label': label}
      sample = self.transform(sample)
    elif label == 0:
      img = torch.from_numpy(np.array(img))
      img = (img - 127.5) / 127.5
      sample = {'image': img, 'label': torch.tensor(label, dtype=torch.long)}
    return sample
</code></pre>
<p>Note how the transforms are only applied to the underepresented class.</p>
<p>And then my idea was to use a DataLoader with a WeightedRandomSampler to get balanced batches on each iteration during training. This is how I have tried to achieve this:</p>
<pre><code class="lang-auto">balance = torch.FloatTensor(balance)
weights = balance / len(labels) 
samples_weight = torch.tensor([weights[t] for t in training_labels])
weighted_sampler = WeightedRandomSampler(samples_weight, len(samples_weight))

train_loader = DataLoader(training_dataset, batch_size=BATCH_SIZE, num_workers=0, sampler=weighted_sampler)
val_loader = DataLoader(validation_dataset, batch_size=BATCH_SIZE, num_workers=0)
</code></pre>
<p>Weights contains: tensor([0.1764, 0.8236])</p>
<p>When I run the following code:</p>
<pre><code class="lang-auto">total_0 = 0
total_1 = 0
for batch in train_loader:
  total_0 += (batch['label'] == 0).sum()
  total_1 += (batch['label'] == 1).sum()
print(total_0, total_1)
</code></pre>
<p>This is the output I get:<br>
tensor(6576) tensor(1241)<br>
Shouldn’t the two be balanced? Each batch contains a distribution similar to: tensor(108) tensor(20)<br>
I have tried to reverse the weights so that it returns tensor([0.8236, 0.1764]) (just to see what would happen) and the output is exactly the same. Is the sampler not applying any weights at all?<br>
This is my first attempt at learning about PyTorch so I might be missing something important (and hopefully easy to fix), most of this concepts are new to me. Any ideas on what to test or change will be extremly appreciated!</p>
<p>Thank you for your time! <img src="https://discuss.pytorch.org/images/emoji/apple/evergreen_tree.png?v=9" title=":evergreen_tree:" class="emoji" alt=":evergreen_tree:"></p>

Mughees: <p>Iam unable to understand the problem correctly, but I can suggest few directions.</p>
<ol>
<li>
<p>Use Splitting with stratification for imbalanced problems.(scikit has a function). It creates the train/test split with equal ratios among classes. see <a href="https://discuss.pytorch.org/t/issues-with-torch-utils-data-random-split/22298/31">this</a></p>
</li>
<li>
<p>In Dataloader, drops_last is also the parameter which you need to focus on because it drops the samples which doesnot fit in the batch size.</p>
</li>
</ol>
<p>Thanks.</p>

manelpk: <p>Hello!</p>
<p>Thank you for your answer!<br>
I tried to follow your first direction but from what I understand this approach would help me in balancing examples <strong>between</strong> the train and test sets.<br>
What I am trying to achieve is to obtain a well balanced train set, without focusing on the test one at all.</p>
<p>My goal would be to sample balanced batches from the training dataset (carrying out data augmentation only on the minority class since this examples would theoretically get picked more).<br>
Per epoch the idea would be to see all minority examples multiple times (with different transforms applied) and only once the majority examples. Does that make sense?</p>
<p>For the second direction you proposed, i don’t see how it would help in this situation. Could you expand on the topic please?</p>
<p>Again, thank you for your response!</p>

manelpk: <p>Found the solution! Here it is, in case someone finds it useful in the future.</p>
<p>I was splitting the data into train and test sets incorreclty (due to a wrong intepretation on how numpy arrays work), so I decided to change the approach by using list comprehensions:</p>
<pre><code class="lang-auto">training_images = [images_path[i] for i in train_indices]
training_labels = [labels[i] for i in train_indices]
validation_images = [images_path[i] for i in val_indices]
validation_labels = [labels[i] for i in val_indices]
</code></pre>
<p>And now I am getting the expected result from the WeightedRandomSampler, balanced batches on every iteration! <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=9" title=":tada:" class="emoji" alt=":tada:"></p>