WeightedRandomSampler for one hot encoded targets

ananthk: <p>Hi,<br>
I am working on an imbalanced dataset of around 5000 images with 10 classes. I have to consider each of these classes as individual binary classification tasks during the forward pass for my project. Therefore I have got weights for each task, i.e a numpy array of dimensions (10,2). May I know how to implement the weight sampler here, so that it becomes balanced for each task?</p>
<p>Thanks a lot in advance.</p>

ptrblck: <p>I’m not sure how the samples should be drawn to balance them. The <code>WeightedRandomSampler</code> uses a weight for each sample and applies it to draw the data. Would it be possible to provide a weight for each sample? If not, could you explain how each batch should be created?</p>

ananthk: <p>I am basically working on a model that changes the binary classification task every batch during training. So I want to balance the samples for that particular task every forward pass. I have collected the weights over the whole dataset for each task and stored them in a (10,2) array (10 tasks). For example [[1  2.43] [1  3.5]…]]. Could you tell me about a way to go about this?</p>
<p>About the data:<br>
I get a sample of dimensions (64,3,224,224) where 64 is the batch size, and the labels array of size (64,10) from the dataloader. I extract columns accordingly to get the labels for a particular task.</p>

ptrblck: <p>Based on the description it seems that each batch would have specific requirements, since each forward pass would expect to use a special task.<br>
If that’s the case, I think the cleanest way would be to create a custom <code>sampler</code> and precalculate the indices based on your task logic (or pass the precalculated indices to a <code>SequentialSampler</code>).</p>

ananthk: <p>Got it. The aim is to generate augmented images to reduce the imbalance present in the dataset for the respective tasks. Probably I could create dataloaders for each task. If that is possible, is there a way to cleanly call the dataloaders in the batch loop during training?</p>

ptrblck: <p>Yes, you could create the iterators via:</p>
<pre><code class="lang-python">iter1 = iter(loader1)
iter2 = iter(loader2)
...
</code></pre>
<p>and get the batches via: <code>batch = next(iter1)</code> etc.<br>
Note that you would have to catch the exception once the iterator is done and would have to recreate the iterators afterwards.</p>

ananthk: <p>I see! Thanks a lot for the help!</p>

ananthk: <p>I have one more doubt. Is there a way to augment only the extra images got from oversampling using the <code>WeightedRandomSampler</code>?</p>

ptrblck: <p>If you are using two different <code>DataLoader</code>s (one with the standard sampler, the other with the <code>WeightedRandomSampler</code>), you could also pass different transformations to the datasets and thus make sure the augmentations are only applied to one of them.</p>

ananthk: <p>Yes, but wouldn’t the one with the standard sampler, be imbalanced again? Is there a way to extract just the oversampled images?</p>

ptrblck: <p>Sorry, I misunderstood the question. If I understand it now correctly, you would like to only augment data samples, which were re-drawn while the first occurrence of the sample should not be transformed?</p>
<p>If that’s the case, I think the cleanest approach would be to write a custom <code>sampler</code>, reuse the <code>WeightedSampler</code> logic and add another flag to indicate if the current sample index should be transformed or not.<br>
You could theoretically also track the indices in the <code>Dataset</code>, but it would break if you are using multiple workers (which is the common case).</p>

ananthk: <p>Yes that’s what I meant.</p>
<p>I see! Regarding the tracking of indices, could you tell me how to figure out if the index points to the re-drawn image, or are they generally appended at the end?</p>

ptrblck: <p>The indices are drawn from <code>torch.multinomial</code> and are thus not appended at the end as seen <a href="https://github.com/pytorch/pytorch/blob/eb9516eaa425f89d8d6729b376dd1d45f1f615f4/torch/utils/data/sampler.py#L186-L188">here</a>. You could check for duplicates in these indices and create the flags accordingly.</p>

ananthk: <p>Got it. Thank you so much! I think that extracting the re-drawn images would be a useful feature in this sampler in the future.</p>

ananthk: <pre><code class="lang-auto">weight=torch.Tensor(np.array([0.0018,0.01]))
rand_tensor = torch.multinomial(weight, 64, replacement=True)
print(rand_tensor)
</code></pre>
<p>Output : tensor([1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,<br>
1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,<br>
1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])<br>
These are the indices returned by the multinomial function.</p>
<p>I have a doubt regarding the working of the sampler. May I know how this can be used to find duplicate images? Sorry if the question is a basic one, I am new to writing code for samplers in PyTorch.</p>

ptrblck: <p>The <code>weight</code> tensor should contain a weight value for each sample.<br>
In your case you are using only 2 samples and draw them 64 times, which explains the result.</p>