Selecting data from dataset by indices

username: pytorchuser
cooked: <p>I’m loading the MNIST dataset using torchvision like this:</p>
<p>train_loader = torch.utils.data.DataLoader(<br>
torchvision.datasets.MNIST(’/home/achennault/research/mnist/’, train=True, download=True,<br>
transform=torchvision.transforms.Compose([<br>
torchvision.transforms.ToTensor(),<br>
torchvision.transforms.Normalize(<br>
(0.1307,), (0.3081,))<br>
])),<br>
batch_size=batch_size_train, shuffle=True, drop_last = True)<br>
test_loader = torch.utils.data.DataLoader(<br>
torchvision.datasets.MNIST(’/home/achennault/research/mnist/’, train=False, download=True,<br>
transform=torchvision.transforms.Compose([<br>
torchvision.transforms.ToTensor(),<br>
torchvision.transforms.Normalize(<br>
(0.1307,), (0.3081,))<br>
])),<br>
batch_size=batch_size_test, shuffle=True, drop_last = True)</p>
<p>I have a list of indices of train_loader.dataset containing elements I want to keep. I want to selecct only these elements of the DataSet, then use the restricted DataSet in the DataLoader. Is there any way to do this?</p>

username: ptrblck
cooked: <p>You could wrap your <code>Dataset</code> into a <code>Subset</code> and pass the indices to only sample from these.</p>

username: pytorchuser
cooked: <p>Thanks for your reply. I’ve created a Subset object using the relevant indices and the original dataset. However, I’m not sure how to use this in my original DataLoader. Do I simply overwrite the original dataset?</p>
<p>i.e. train_loader.dataset = mysubset</p>
<p>or is there something more complicated that I must do?</p>

username: ptrblck
cooked: <p>You can simply pass the <code>Subset</code> as the new dataset to the <code>DataLoader</code>:</p>
<pre><code class="lang-python">my_subset = Subset(dataset, indices)
loader = DataLoader(my_subset)
</code></pre>

username: Steven_ari
cooked: <p>This works! Note we have to specify batch_size if we want multiple indices</p>
<pre><code class="lang-auto">my_subset = Subset(dataset, indices)
loader = DataLoader(my_subset, batch_size=indices.size)
</code></pre>

username: Dadatata-JZ
cooked: <p>I am confused about the Subset() for torch.dataset. I have a list of indices and a pytorch dataset (e.g. cifar). When I used the indices to get a subset from the dataset, the new subset.dataset still keeps the same length as the original dataset, even though when it is loaded into a dataloader, the length becomes correct.</p>
<p>I would like to find out a solution to check the length of a subset, and how to iterate the subset.</p>
<p>Thank you in advance</p>

username: ptrblck
cooked: <p>The underlying <code>.dataset</code> will not be changed and will keep its original size.<br>
You can check the length of the <code>Subset</code> via <code>len(subset)</code> and iterate it with <code>for data in subset</code>.</p>

username: bohare
cooked: <p>May I ask how did you create the Subset object?</p>

username: bohare
cooked: <p>Nevermind, it’s <code>torch.utils.data.Subset()</code> .</p>

username: sri9s
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I have a dataset defined, and I want to define a sampler that samples data points of batch size n such that the n indices are to be given by the user. How to achieve this?</p>

username: ptrblck
cooked: <p>I don’t fully understand your use case. Do you want the user to pass the batch indices in each iteration such that no sampling is used anymore or should these samples be somehow predefined by the user?</p>