Load mnist how to get the labels?

blackbirdbarber: <p>Strange but I would like to load mnist labels using <code>torchvision.datasets.MNIST</code>.  I loaded images like this:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('/data/mnist', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (0.1307,), (0.3081,))
                             ])),
  batch_size=16, shuffle=False)
</code></pre>
<p>If I print the dataset I get this but where are the labels?</p>
<blockquote>
<p>Dataset MNIST<br>
Number of datapoints: 60000<br>
Split: train<br>
Root Location: /data/mnist<br>
Transforms (if any): Compose(<br>
ToTensor()<br>
Normalize(mean=(0.1307,), std=(0.3081,))<br>
)<br>
Target Transforms (if any): None</p>
</blockquote>
<p>What do you suggest? The origin is <a href="http://yann.lecun.com/exdb/mnist/" class="inline-onebox" rel="noopener nofollow ugc">MNIST handwritten digit database, Yann LeCun, Corinna Cortes and Chris Burges</a></p>

ptrblck: <p>You can print the labels using <code>dataset.targets</code>.</p>

blackbirdbarber: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="48410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>dataset.targets</p>
</blockquote>
</aside>
<p>Happens to be that easy.</p>
<p>If I would try to work on this <code>train=True</code> dataset to split it into validation part 60000=50000+10000, would that be easy or I should use <code>train=False</code> to load another dataset (test dataset) so that should be my validation.</p>

ptrblck: <p>I would suggest to wrap your training <code>Dataset</code> into <code>Subset</code> and pass the training and validation indices, while <code>train=False</code> would create the test dataset.</p>

blackbirdbarber: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="48410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>Dataset</code> into <code>Subset</code></p>
</blockquote>
</aside>
<p>OK so just indexing will decide on train and validation, and test set is test set. I may not use that for my blitz experiments. Good.</p>

blackbirdbarber: <p>I am using the subsets now, but their datasets point to the original set:</p>
<pre><code class="lang-auto">test_ds = torch.utils.data.Subset(train_loader.dataset, (0, 50000-1))
valid_ds = torch.utils.data.Subset(train_loader.dataset, (50000, 60000-1))
print(test_ds.indices) #(0, 49999)
print(valid_ds.indices) #(50000, 59999)
print(test_ds.dataset.targets[0]) #tensor(5)
print(valid_ds.dataset.targets[0]) #tensor(5)

#SequentialSampler ???
</code></pre>
<p>Should I cast them to Set or use a sampler… no idea.</p>

ptrblck: <p><code>Subset</code> is just a thin wrapper using the passed indices to index the underlying <code>Dataset</code> without actually manipulating it.<br>
Have a look at <a href="https://github.com/pytorch/pytorch/blob/d4119f8fcbde15569b8f2993973439f2961ea22a/torch/utils/data/dataset.py#L107" rel="nofollow noopener">this line of code</a>.</p>
<p>You’ll get the right samples, if you index the <code>Subset</code> directly:</p>
<pre><code class="lang-python">x, y = test_ds[0]
</code></pre>

blackbirdbarber: <p>Helpful:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models
import torchvision

train_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('/data/mnist', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                (0.1307,), (0.3081,))
                             ])),
  batch_size=16, shuffle=False)
test_ds  = torch.utils.data.Subset(train_loader.dataset, range(0, 50000-1))
valid_ds = torch.utils.data.Subset(train_loader.dataset, range(50000, 60000-1))
for i in range(0, 10):
    ty = test_ds[i][1]
    vy = valid_ds[i][1]
    print(ty, vy)
</code></pre>
<p>out:</p>
<pre><code class="lang-auto">5 3
0 8
4 6
1 9
9 6
2 4
1 5
3 3
1 8
4 4

</code></pre>

monster: <p>I have a subset of more then 30000 images and I am trying this code but it is taking forever.</p>
<pre><code class="lang-auto">labels=[]
for _,data in train_set:
    labels.append(data['labels'])
print(labels)
</code></pre>
<p>Is there any faster way of doing this?</p>

ptrblck: <p>If each target is lazily created, you would have to iterate the <code>Dataset</code> or <code>DataLoader</code> at least once.<br>
Depending how <code>train_set</code> is defined, the targets might have been loaded completely in its <code>__init__</code>, so you could check it. Also, a <code>DataLoader</code> should speed up the loop, if you increase the batch size and use multiple workers.</p>
<p>How long does this loop take?</p>

monster: <p>It was taking more then 30 minutes,if I run loop for only 100 times it is also taking lots of time. I think this is because of tensor on each index.</p>

ptrblck: <p>I’m not sure, if you are using <code>MNIST</code> as in the original question, but this code takes ~1 second on my laptop:</p>
<pre><code class="lang-python">dataset = datasets.MNIST(root=PATH)

labels = []
for _, label in dataset:
    labels.append(label)
</code></pre>

monster: <p>I am loading different data from multiple folders with images and annotations(20000 images)</p>
<pre><code class="lang-auto">(tensor_data,['boxes':[....],'labels':'a'])=dataset[0]
</code></pre>