Weighted Random Sampling with unique samples for each mini-batch

Neofytos: <p>Hey,</p>
<p>I’ve been using WeightedRandomSampler but due to the stochastic nature of the process a mini-batch can oftentimes contain the same instance twice. Is there a way to guarantee that all instances are unique within a mini-batch while maintaining the other properties of the sampler? Note that since I am trying to oversample one of the two classes, I am using sampling with replacement. Thank you!</p>
<p>Neo</p>

ptrblck: <p>I think you could write a custom sampler, which could use the <code>WeightedRandomSampler</code> <a href="https://github.com/pytorch/pytorch/blob/aeb3e933515672a61cb21498fdbdef80a46ae0ee/torch/utils/data/sampler.py#L186-L187">implementation</a> as the base and create indices with non-duplicates using the batch size.<br>
What kind of issues are you seeing with these duplicates?</p>

Neofytos: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the reply.</p>
<p>The issue is technical as I am using a stateful dataset where each instance has information stored and processed at each batch. Hence when the same instance is found twice in a mini-batch I have all sorts of errors that arise that I’d rather solve via the sampler rather than modifying my current implementation.</p>
<p>I’ve implemented a sampler that seems to work for now but is probably not that optimal. Any thoughts on it?</p>
<pre><code class="lang-auto">from torch.utils.data import WeightedRandomSampler
from typing import Sequence
import torch

class CustomWeightedSampler(WeightedRandomSampler):
    def __init__(self, weights: Sequence[float], num_samples: int, bs: int,
                 generator=None) -&gt; None:
        if not isinstance(num_samples, int) or isinstance(num_samples, bool) or \
                num_samples &lt;= 0:
            raise ValueError("num_samples should be a positive integer "
                             "value, but got num_samples={}".format(num_samples))
        if not isinstance(bs, int) or isinstance(bs, bool) or \
                num_samples &lt;= 0:
            raise ValueError("bs should be a positive integer "
                             "value, but got bs={}".format(num_samples))
        if bs &gt; num_samples:
            raise ValueError("bs should be smaller than num_samples "
                             "but got bs={} and num_samples={}".format(bs, num_samples))
        self.weights = weights
        self.num_samples = num_samples
        self.bs = bs
        self.generator=generator


    def __iter__(self):
        rand_tensor = torch.multinomial(self.weights, self.bs, False, generator=self.generator)
        for _ in range((self.num_samples - self.bs) // self.bs):
            rand_tensor = torch.cat([rand_tensor,
                                      torch.multinomial(self.weights, self.bs, False, generator=self.generator)])
        if self.num_samples % self.bs != 0:
            rand_tensor = torch.cat([rand_tensor,
                                      torch.multinomial(self.weights,  self.num_samples % self.bs, False, generator=self.generator)])
        return iter(rand_tensor.tolist())

    def __len__(self):
        return super(CustomWeightedSampler, self).__len__()
</code></pre>

ptrblck: <p>I don’t see where you are checking for duplicated indices in the code and think that:</p>
<pre><code class="lang-python">rand_tensor = torch.multinomial(self.weights, self.bs, False, generator=self.generator)
for _ in range((self.num_samples - self.bs) // self.bs):
    rand_tensor = torch.cat([rand_tensor,
                             torch.multinomial(self.weights, self.bs, False, generator=self.generator)])
</code></pre>
<p>could sample repeated values.<br>
While <code>torch.multinominal</code> wouldn’t use replacement, you are still passind the same weights, and thus the sampled values might already be in <code>rand_tensor</code>, no?</p>

Neofytos: <p>Hey, sorry for my late reply.</p>
<p>Correct me if I am wrong but I assumed that "_<em>iter</em>_()"returns the order of indexes that will be sampled during dataloading. In other words, given a list [0, 1, 2, 3] and a batch size of 2, the indexes of the two mini–batches would be [0, 1] and [2, 3].</p>
<p>Given that the above is correct, to solve my initial issue, I simply sampled without replacement each mini–batch independently. In theory this would translate to sampling with replacement over multiple mini–batches while maintaining unique instance within each mini–batch. Let me know if I am missing something!</p>

ptrblck: <p>Yes, <code>__iter__</code> will return an <code>iterator</code>, which will return the indices sampled from <code>rand_tensor</code>.<br>
And yes, you are right. <code>rand_tensor</code> uses the batch size as the “stride” and doesn’t use replacement, so you should be fine. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>