How to get a part of datasets?

sjt524: <p>Sorry for my poor English.<br>
This is my code:<br>
trainset = datasets.MNIST(‘data’, train=True, download=False, transform=transform)<br>
trainloader = torch.utils.data.DataLoader(trainset,batch_size=32, shuffle=True)</p>
<p>Now I want to choose a part of train sets(like 3000 images and labels) from shuffled datasets every epoch. I want to know how to shuffle the datasets and then choose index from 0 to 2999. Please help me</p>

ptrblck: <p>You could manually shuffle the indices using:</p>
<pre><code class="lang-python">indices = torch.randperm(len(train_dataset))[:3000]
</code></pre>
<p>and pass these indices to a <code>RandomSubsetSampler</code>, which can then be passed to the <code>DataLoader</code>.</p>

sjt524: <p>Thanks for your help</p>

sjt524: <p>Sorry to distrub you.<br>
There are 10 classes in MNIST datasets.Like 0,1,2,3… Now I want to choose 100 samples in each class and convert them to ‘trainset’ then use the new trainset like follow.<br>
<em>trainloader = torch.utils.data.DataLoader(trainset,batch_size=batch_size, shuffle=True)</em><br>
Can you help me?</p>

ptrblck: <p>If you are fine with approx. 100 samples, which were randomly drawn, this code should work:</p>
<pre><code class="lang-python"># Setup
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5,), (0.5,))])

dataset = torchvision.datasets.MNIST('./data/', train=True, transform=transform)

# Split the indices in a stratified way
indices = np.arange(len(dataset))
train_indices, test_indices = train_test_split(indices, train_size=100*10, stratify=dataset.targets)

# Warp into Subsets and DataLoaders
train_dataset = Subset(dataset, train_indices)
test_dataset = Subset(dataset, test_indices)

train_loader = DataLoader(train_dataset, shuffle=True, num_workers=2, batch_size=10)
test_loader = DataLoader(train_dataset, shuffle=False, num_workers=2, batch_size=10)


# Validation
train_targets = []
for _, target in train_loader:
    train_targets.append(target)
train_targets = torch.cat(train_targets)

print(train_targets.unique(return_counts=True))
&gt; (tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), tensor([ 99, 112,  99, 102,  97,  90,  99, 105,  98,  99]))
</code></pre>
<p>Otherwise you could probably use a loop for each class to get 100 random corresponding class indices and could then use the same <code>Subset</code> approach.</p>

sjt524: <p>Thanks, sorry to occupy your precious time.</p>

randbit: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I’d like to access the classes of the dataloader, but currently have to switch between <code>loader.dataset.classes</code> and <code>loader.dataset.dataset.classes</code>, depending on whether I used <code>Subset</code> or not. Is there a way around this?</p>

ptrblck: <p>I don’t know what the best approach would be besides checking, if the <code>loader.dataset</code> is an instance of <code>Subset</code>.</p>

randbit: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks. Currently, I’ve written the following utility function:</p>
<pre><code class="lang-auto">def get_classes(dataset):
    while hasattr(dataset, 'dataset'):
        dataset = dataset.dataset
    return dataset.classes
</code></pre>
<p>From an API standpoint, might it not be better for Subset to inherit <code>classes</code>, <code>targets</code> (properly filtered according to the indices), and perhaps other relevant attributes from its parent?</p>

ptrblck: <p>That’s not necessarily that easy, if your original <code>Datasets</code> lazily loads the data.<br>
Currently <code>Subset</code> only uses the passed indices to forward them to the underlying <code>Dataset</code>.<br>
This works fine since <code>Subset</code> has no knowledge about the <code>Dataset</code> and just acts as a “filter” for the indices.<br>
If you want to forward some dataset internals such as <code>.classes</code>, <code>.targets</code> etc., <code>Subset</code> would need to know what kind of <code>Dataset</code> you are using and which attributes to expose.<br>
Your underlying dataset can of course be a custom <code>Dataset</code>, which doesn’t provide these attributes as they might be unknown during initialization.</p>