Using WeightedRandomSampler with ConcatDataset

borisgiba: <p>Hi <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I have a rather small image dataset and want want to augment my training images.<br>
However I want the training-dataloader to use unaugmented images as well as augmented images.<br>
For that I am using the ConcatDataset-class. I also want to use WeightedRandomSampler, because some classes have more images than others, the amounts of images per class being:  [602,536,1088,751].</p>
<pre><code class="lang-auto">datasetBasic = datasets.ImageFolder('path', transform=transformsBasic)
datasetAugmented = datasets.ImageFolder('path', transform=transformsAugment)
concatDataset=torch.utils.data.ConcatDataset((datasetBasic,datasetAugmented))

dataloaders_dict = {"train": torch.utils.data.DataLoader(concatDataset, batch_size=batch_size, sampler=sampler, num_workers=0, pin_memory=True),
                     "val": torch.utils.data.DataLoader(datasetBasic, batch_size=batch_size, shuffle=True, num_workers=0,pin_memory=True)}
</code></pre>
<p>However I do not know how I should use WeightedRandomSampler together with ConcatDataset.<br>
Is there any suggestion to how I can solve this issue?</p>
<p>This post ist very similar to: <a href="https://discuss.pytorch.org/t/sampling-from-a-concatenated-dataset/30688/3" class="inline-onebox">Sampling from a concatenated dataset</a> , but there were no more replies on that one so I am asking again ^^’.</p>
<p>PS: Great forums, the people seem to be really active here ^^</p>

ptrblck: <p>For weighted sampling you would have to create a weight for each sample.<br>
If you don’t have the target tensors already computed, you could iterate your dataset and store the target tensors.<br>
Here is a small example, which should match your use case:</p>
<pre><code class="lang-python"># Create dummy data with class imbalance 99 to 1
numDataPoints = 1000
data_dim = 5
bs = 100
data = torch.randn(numDataPoints, data_dim)
target = torch.cat((torch.zeros(int(numDataPoints * 0.99), dtype=torch.long),
                    torch.ones(int(numDataPoints * 0.01), dtype=torch.long)))

print('target train 0/1: {}/{}'.format(
    (target == 0).sum(), (target == 1).sum()))

# Create ConcatDataset
dataset = torch.utils.data.TensorDataset(data, target)
train_dataset = ConcatDataset((dataset, dataset))

# Get all targets
targets = []
for _, target in train_dataset:
    targets.append(target)
targets = torch.stack(targets)

# Compute samples weight (each sample should get its own weight)
class_sample_count = torch.tensor(
    [(targets == t).sum() for t in torch.unique(targets, sorted=True)])
weight = 1. / class_sample_count.float()
samples_weight = torch.tensor([weight[t] for t in targets])

# Create sampler, dataset, loader
sampler = WeightedRandomSampler(samples_weight, len(samples_weight))

train_loader = DataLoader(
    train_dataset, batch_size=bs, num_workers=1, sampler=sampler)

# Iterate DataLoader and check class balance for each batch
for i, (x, y) in enumerate(train_loader):
    print("batch index {}, 0/1: {}/{}".format(
        i, (y == 0).sum(), (y == 1).sum()))
</code></pre>
<p>In the first part I’m creating a dummy imbalanced dataset.<br>
You should of course just skip this step and use your original <code>concatDataset</code>.</p>
<p>After storing all targets, the <code>class_sample_count</code> and the corresponding <code>samples_weight</code> tensor is created, which is used to create the <code>WeightedRandomSampler</code>.<br>
As you can see in the last loop, each batch should be balanced using the sampler.</p>
<p>Let me know, if that would work for you.</p>

borisgiba: <p>First up thank you for the quick and detailed response!</p>
<p>I have tried executing the code at <span class="hashtag">#Get</span> all targets but have run into the following error:</p>
<pre><code class="lang-auto">TypeError: expected Tensor as element 0 in argument 0, but got int
</code></pre>
<p>which appears at the following line:</p>
<pre><code class="lang-auto">targets = torch.stack(targets)
</code></pre>
<p>When looking at targets, I noticed in my case it is full of ints, but torch.stack rather expects Tensors, if I understand it correctly?<br>
Is this problematic or should I just use different functions to compute the samples weight?</p>

ptrblck: <p>In that case you could use <code>torch.tensor</code> instead.<br>
Let me know, if that works.</p>

borisgiba: <p>Yes, that worked!<br>
Thanks again for the help.</p>