How to use WeightedSampler along with a custom Sampler?

username: Achu_Chandran
cooked: <p>I am a using a custom sampler class similar to the one described here : <a href="https://discuss.pytorch.org/t/how-upload-sequence-of-image-on-video-classification/24865" class="inline-onebox">How upload sequence of image on video-classification</a> suggested by <a class="mention" href="/u/ptrblck">@ptrblck</a>  (<a href="https://discuss.pytorch.org/u/ptrblck">https://discuss.pytorch.org/u/ptrblck</a>)  to stack up images sequentially from the datasets before feeding into the network.</p>
<p>Now I also wanted to use WeightedSampler to balance the classes.</p>
<p>Any ideas to use a WeightedSampler along with a custom sampler class ?</p>
<p>Thanks in advance !!  <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: tom
cooked: <p>I’d probably take <a class="mention" href="/u/ptrblck">@ptrblck</a>’s sampler and</p>
<ul>
<li>compute the class weights in <code>__init__</code>,</li>
<li>have a tensor that contains the class weight for each index,</li>
<li>use torch.multinomial with the weights instead of randperm.</li>
</ul>
<p>For the latter, you can compare <a href="https://github.com/pytorch/pytorch/blob/e2bcdc7b697de757991664d81735d797e70be59b/torch/utils/data/sampler.py#L145" rel="nofollow noopener">RandomSampler</a></p>
<pre><code class="lang-auto">    def __iter__(self):
        return (self.indices[i] for i in torch.randperm(len(self.indices), generator=self.generator))
</code></pre>
<p>to <a href="https://github.com/pytorch/pytorch/blob/e2bcdc7b697de757991664d81735d797e70be59b/torch/utils/data/sampler.py#L187" rel="nofollow noopener">WeightedRandomSampler</a></p>
<pre><code class="lang-auto">    def __iter__(self):
        rand_tensor = torch.multinomial(self.weights, self.num_samples, self.replacement, generator=self.generator)
        return iter(rand_tensor.tolist())
</code></pre>
<p>(except you want the indices her, too).</p>
<p>Note that you likely want replacement=True.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Achu_Chandran
cooked: <p>Hello <a class="mention" href="/u/tom">@tom</a>,  thanks for your response. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>To compute the weights, I need the count for each class. But whereas in my case the sampler is used to create the datasets too. So, only after creating the datasets i can get access to the  sample count.<br>
( I used sampler to get the indices of images and stack up images sequentially to create samples, then create datasets and load it using the DataLoader)</p>
<p>Do you have any suggestion for this? I can attach the code if required.</p>
<p>Thanks again</p>

username: tom
cooked: <p>I don’t think there is anything wrong per se with looping over the dataset to get the class distribution. That said, if it takes a long time and you expect to run your training often, the typical thing is to make it a preprocessing step (just like e.g. the famous ImageNet mean and std for normalization have been part of preprocessing before people just kept them hardcoded).<br>
Creating a dataloader isn’t that expensive (the expensive stuff is only done when iterating them and re-done every epoch), so there isn’t anything wrong with having one that is used to collect statistics and then creating a new one with the weighted sampler.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Achu_Chandran
cooked: <p>Thanks. If I understood correctly, you mean I can have one sampler for preparing datasets and a weighted sampler for iterating them ?</p>

username: tom
cooked: <p>Yes, you can use the same Dataset but wrap it in different Dataloaders (with different RandomSamplers).</p>

username: Achu_Chandran
cooked: <p>Thank you, <a class="mention" href="/u/tom">@tom</a>.</p>
<p>This helps a lot. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>