Random Subset from DataLoader, Unique Random Numbers

Kale-ab_Tessera: <p>I want to generate a random subset from DataLoader.</p>
<p>I have the following implementation:</p>
<pre><code class="lang-auto">train_indices = torch.LongTensor(args.train_set_size).random_(0, totalNumInTrainSet)
train_loader = torch.utils.data.DataLoader(
            datasets.MNIST('../data', train=True, download=True,
                           transform=transforms.Compose([
                               transforms.ToTensor(),
                               transforms.Normalize((0.1307,), (0.3081,),
                                                    )
                           ])),
            batch_size=args.batch_size, sampler=SubsetRandomSampler(train_indices), ** kwargs)
</code></pre>
<p>This works, however generating the train indices results in duplicates, I want all random numbers generated to be unique. How would I do this efficiently ?</p>

MariosOreo: <p>How about just pass <code>shuffle</code> to the dataloader.</p>
<p>If you are playing with Samplers, you also could pass RandomSampler with <code>replacement=False</code> to the dataloader, it is the same to the <code>shuffle</code> method.</p>

Kale-ab_Tessera: <p>If I use the <code>RandomSampler</code> with <code>replacement=False</code>, I can’t specify the number of elements.<br>
Message:<br>
<code>With replacement=False, num_samples should not be specified, since a random permute will be performed.</code></p>
<p>Will the <code>RandomSampler</code> with <code>replacement=True</code> and <code>num_samples=n</code>, always return a sample with n unique elements?</p>

MariosOreo: <p>Because <code>replacement=True</code>, so it will return the same sample probably.</p>
<p>With <code>replacement=False</code>, RandomSampler will go through the whole dataset.</p>
<p>If you want to specify the number of samples. You could have a try on <code>SubsetRandomSampler</code>.</p>

Kale-ab_Tessera: <p>Yep, that is what I have done above. I used <code>SubsetRandomSampler</code> and generated random indexes to match the size I wanted. I wanted to know if there was an efficient way to generate <em>unique</em> random indexes to pass to SubsetRandomSampler .</p>

Mazhar_Shaikh: <p>You could try this</p>
<pre><code class="lang-auto">import numpy as np
train_indices = torch.from_numpy(np.random.choice(totalNumInTrainSet, size=(args.train_set_size,), replace=False)) 
</code></pre>

MariosOreo: <p>Sorry for forgetting that.</p>
<p>I think you could have a try on <a href="https://pytorch.org/docs/stable/torch.html#torch.randperm" rel="nofollow noopener">torch.randperm()</a> with fixed torch.manual_seed() and <a href="https://pytorch.org/docs/stable/torch.html#torch.multinomial" rel="nofollow noopener">torch.multinomial()</a> with sampling probability.</p>
<pre><code class="lang-auto">torch.manual_seed(1)
random_indices = torch.randperm(num_samples)
# it will return random permutation from 0 to num_samples-1
</code></pre>

Kale-ab_Tessera: <p>I will use numpy until this or equivalent is merged in -&gt; <a href="https://github.com/pytorch/pytorch/pull/18624" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/18624</a> .</p>