Train on a fraction of the data set

Brando_Miranda: <p>I want to train with SGD on say 10% of cifar10. However, I want that 10% to be the same fraction of cifar10 and not change once training has started. How does one do this?</p>
<p>The only way I’ve thought of doing this was loop through all the data of cifar10 and once I’ve extracted the first 10% of the data, save it in a numpy file and then use that for training later and load that with a different dataloader. Would that work? What do people think?</p>

arturml: <p>You can use <code>torch.utils.data.sampler.SubsetRandomSampler</code>. Something like:</p>
<pre><code class="lang-python">cifar_dataset = torchvision.datasets.CIFAR10(root='./data', transform=transform)
subset_indices = # select your indices here
subset_loader = torch.utils.data.DataLoader(cifar_dataset, batch_size=32, shuffle=True, sampler=SubsetRandomSampler(subset_indices), ...)

</code></pre>

Brando_Miranda: <p>Ah I think that should work? But I find it confusing that the description says that:</p>
<blockquote>
<p>Samples elements randomly from a given list of indices, without replacement.</p>
</blockquote>
<p>but at the same time it receives a list of indices. Do you know what’s the deal with that?</p>

Brando_Miranda: <p>btw do you know if the answer u provided can aid in solving:</p>
<aside class="quote quote-modified" data-post="1" data-topic="16742">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-one-create-a-data-set-in-pytorch-and-save-it-into-a-file-to-later-be-used/16742">How does one create a data set in pytorch and save it into a file to later be used?</a> 
  </div>
  <blockquote>
    I want to extract data from cifar10 in a specific order according to some criterion f(Image,label) (for the sake of an example lets say f(Image,label) simply computes the sum of all the pixels in Image). Then it I want to generate 1 file for the train set and 1 file for the test set that I can later load in a dataloader to use for training a neural net. 
How do I do this? My current idea was simply to loop through the data with data loader with shuffle off and remember the indices of the images …
  </blockquote>
</aside>

<p>thats really the problem I am trying to solve.</p>

Brando_Miranda: <p>I think I know how. Just sort the indices according to the criteron I have, save those indices and recover them from a file whenever I need them. Then use ur data sampler!</p>

arturml: <p>SubsetRandomSampler restricts the DataLoader to the list of indices you pass. Otherwise everything is the same: the dataloader generates batches from these indices without replacement and in random order. This is useful for example when you want to split your dataset in train/validation, you can split the indices and build two data_loaders restricted two each subset.</p>
<p>And I believe you can do what you want with <code>SequentialSampler</code> (assuming it its important not to shuffle). Generate the train/test indices using your rule and build two data loaders.</p>

Brando_Miranda: <p>The thing is that during training I do want the batches to be produced Randomly, but I don’t want my subset to be produced randomly, I want it to be created according to the criterion I developed. How does that change things? Should I use <code>SequentialSampler</code> or <code>RandomSampler</code>? This second sampler through me off a bit.</p>

arturml: <p>Maybe something like this:</p>
<pre><code class="lang-python">cifar_dataset = torchvision.datasets.CIFAR10(root='./data', transform=transform)
train_indices = # select train indices according to your rule
test_indices = # select test indices according to your rule
train_loader = torch.utils.data.DataLoader(cifar_dataset, batch_size=32, shuffle=True, sampler=SubsetRandomSampler(train_indices))
test_loader = torch.utils.data.DataLoader(cifar_dataset, batch_size=32, shuffle=True, sampler=SubsetRandomSampler(test_indices))
</code></pre>
<p>If your rule can be computed fast, you don’t even have to bother saving it to a file. Btw, CIFAR10 has a standard train/test division and the snippet above splits the train set further. Do you want to ignore the original split or just using the train set is fine?</p>

Brando_Miranda: <p>oh no I just realized my criterion is sort of expensive to compute I requires changing the labels according to my criterion…I think Im going to have to create and save a new data set so that I change the labels according to this criterion only once…</p>

Brando_Miranda: <p>actually artur, how do you get any indices?</p>

OrielBanne: <p>Hi Artur -</p>
<p>I tried your solution with another dataset (my data). however, the sampler did not cut the data into two parts according to the indices . total # of elements I had for inference training and validation was 300 altogether. train indices included 260  and test indices included 40. after construction of the dataloader - I check: len(test_loader.dataset.tensors[0])<br>
and I get 300 …<br>
I must be doing something wrong<br>
one difference I had - I could not use shuffle = True, because then I get: ValueError: sampler option is mutually exclusive with shuffle</p>
<p>here are the two lines:<br>
train_loader = torch.utils.data.DataLoader(dataset, batch_size=BATCH_SIZE, sampler=SubsetRandomSampler(train_idx))<br>
test_loader = torch.utils.data.DataLoader(dataset, batch_size=BATCH_SIZE, sampler=SubsetRandomSampler(test_idx))</p>
<p>can you see a reason why this should not work?</p>