Where can I find the simplest vision dataset?

Vinayaka_Hegde: <p>Hi, I want to find a very simple dataset of let’s say 10 images per class and few very classes. I would need small dataset since I’m modifying pytorch source code. And adding log statements. So, when there are <code>num_workers</code> &gt; 0, then it would be very difficult to trace what’s happening internally.</p>
<p>The simplest dataset I know is lenet / mnist. But, this itself is very big for me and would need a much smaller one.<br>
I know that I could always prune the existing dataset manually. But, that;s the last option that I want to have.</p>
<p>Thanks a lot !!!</p>

ptrblck: <p>Writing a custom <code>Dataset</code> might be a good idea as it would give you the flexibility to manipulate it as you wish:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, nb_images=10):
        data = torch.ones(3, 32, 32).expand(nb_images, -1, -1, -1)
        self.data = data * torch.arange(nb_images)[:, None, None, None].expand_as(data)
        self.target = torch.arange(nb_images)
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        return x, y
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
for x, y in dataset:
    print(x)
    print(y)
</code></pre>
<p>or using <code>Subset</code> might also work, but would “hide” the dataset behind the wrapper.</p>

Vinayaka_Hegde: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I’m not able to follow the operations in the first and second line of the <code>__init__</code> function.<br>
In the first line, we create a tensor of <code>3*32*32</code> and expand it to <code>10</code> images. So, the shape of <code>data</code> at this point is <code>10*3*32*32</code></p>
<p>In the second line, we are constructing a new <code>10*1</code> tensor using <code>arange</code> and the expanding it to the same dimesnion as that of <code>data</code> which would be <code>10*3*32*32</code>.</p>
<p>And this new tensor of shape <code>10*3*32*32</code> is multiplied with <code>data</code> of shape <code>10*3*32*32</code> ?</p>

ptrblck: <p>Yes, your explanation is right and I’ve used the code to create <code>nb_images</code> each containing a different value.<br>
I’ll reduce the <code>nb_images</code> as well as the spatial size as it would otherwise be a long output in this post but here is the result:</p>
<pre><code class="lang-python">nb_images = 5
data = torch.ones(3, 5, 5).expand(nb_images, -1, -1, -1)
data = data * torch.arange(nb_images)[:, None, None, None].expand_as(data)
print(data)

tensor([[[[0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.]],

         [[0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.]],

         [[0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.]]],


        [[[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]],

         [[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]]],


        [[[2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.]],

         [[2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.]],

         [[2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.],
          [2., 2., 2., 2., 2.]]],


        [[[3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.]],

         [[3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.]],

         [[3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.],
          [3., 3., 3., 3., 3.]]],


        [[[4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.]],

         [[4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.]],

         [[4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.],
          [4., 4., 4., 4., 4.]]]])
</code></pre>
<p>You can of course use any random input data if this fits your use case better.</p>