Why do we need Subsets at all?

blackbirdbarber: <p>I am using this code and I have single question:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, Dataset, TensorDataset
from torch.optim import *
import torchvision


dl = DataLoader(
  torchvision.datasets.MNIST('/data/mnist', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                (0.5,), (0.5,))
                             ])),
  batch_size=16, shuffle=False)

train_ds  = torch.utils.data.Subset(dl.dataset, range(0, 50000-1))
valid_ds = torch.utils.data.Subset(dl.dataset, range(50000, 60000-1))

# this won't work 
# train_ds = TensorDataset(train_ds, valid_ds)
# AttributeError: 'Subset' object has no attribute 'size'

tensor = dl.dataset.data
tr = tensor.reshape(tensor.size(0), -1)  # reshaped
targets = dl.dataset.targets

x_train = tr[0:50000-1]
y_train = targets[0:50000-1]
x_valid = tr[50000:60000-1]
y_valid = targets[50000:60000-1]

bs=64

x_train = x_train.to(dtype=torch.float32)
y_train = y_train.to(dtype=torch.long)

x_valid = x_valid.to(dtype=torch.float32)
y_valid = y_valid.to(dtype=torch.long)

train_ds = TensorDataset(x_train, y_train)
train_dl = DataLoader(train_ds, batch_size=bs, drop_last=False, shuffle=True)

valid_ds = TensorDataset(x_valid, y_valid)
valid_dl = DataLoader(valid_ds, batch_size=bs * 2)

loaders={}
loaders['train'] = train_dl
loaders['valid'] = valid_dl


class M(nn.Module):
    'custom module'
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(784, 10)

    def forward(self, xb):
        return self.lin(xb)

model = M()

criterion = nn.CrossEntropyLoss()

bs=64
epochs = 2
lr = 0.0001
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
# optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9)

for epoch in range(0,epochs):
    train_loss = 0
    valid_loss = 0
    print(f"Epoch {epoch}")

    model.train()
    for i, (data,target) in enumerate(loaders['train']):
                
        optimizer.zero_grad()
        output = model(data)                                    
        loss = criterion(output, target)        
        loss.backward()            
        optimizer.step()
        
        if (i%100==0):
            print(f"Batch {i}, loss {loss}")
</code></pre>
<h1>Question:</h1>
<p>When I used <code>torch.utils.data.Subset</code> it gave me the “Subset object has no attribute size” error message, so had no idea how to use subsets to feed dataloaders with <code>TensorDataset</code>. To me at this level of understanding, <code>Subset</code> looks very impractical and it even creates confusion. Any comments why do we need them?</p>

ptrblck: <p>Which line of code threw this error?<br>
Could you explain your confusion a bit?<br>
<code>Subset</code> is a convenient way of using certain indices to specify subsets of your <code>Dataset</code>.<br>
While in your example you can easily split the loaded data manually, this won’t be possible in a lot of cases, where you would have to lazily load the data.</p>

blackbirdbarber: <blockquote>
<p>Which line of code threw this error?</p>
</blockquote>
<p>To me the <code>Subsets</code> do not work well with TensorDataset, since Subsets are not the tensors. This line will fail:</p>
<pre><code class="lang-auto">train_ds = TensorDataset(train_ds, valid_ds)
</code></pre>
<blockquote>
<p>Could you explain your confusion a bit?</p>
</blockquote>
<p>I was not so sure why do we need Subsets. Since for MNIST dataset (60000 images) I cannot use them to make <code>TensorDataset</code>.</p>
<p>I am not sure if I can use something other than <code>TensorDataset</code>, but the idea was to<br>
to move tensors on CUDA.</p>
<p>I guess it is faster to move data and targets on CUDA togeter, but I am not sure, maybe just data on CUDA is OK, and targes on CPU may be still be fast.</p>
<p>So based on making data and targets togeter I see <code>TensorDataset</code> as a good choice` for any unsupervised architecture, having data and targets.</p>
<blockquote>
<p>Subset is a convenient way of using certain indices to specify subsets of your Dataset. While in your example you can easily split the loaded data manually, this won’t be possible in a lot of cases, where you would have to lazily load the data</p>
</blockquote>
<p>This is maybe it, I haven’t experimented with lazy loading data.</p>

ptrblck: <p>You don’t need to wrap the subsets again in <code>TensorDatasets</code>.<br>
In fact, you would create your <code>Dataset</code> first (e.g. as a <code>TensorDataset</code> or any other) and wrap it into a <code>Subset</code> to get the samples for the corresponding indices. This subset can then be wrapped by a <code>DataLoader</code>.</p>
<p>Note that <code>TensorDataset</code> is used for tensors already loaded into memory.<br>
If you are dealing with a large dataset, this might not be possible, so you could use another dataset, e.g. <code>torchvision.datasets.ImageFolder</code> or a custom <code>Dataset</code> implementation for lazy loading.</p>
<aside class="quote no-group" data-username="blackbirdbarber" data-post="3" data-topic="49391">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>I am not sure if I can use something other than <code>TensorDataset</code> , but the idea was to<br>
to move tensors on CUDA.</p>
</blockquote>
</aside>
<p>The usual way would be to load the samples into the RAM and push each batch onto the GPU in the training loop.<br>
If your model is small and you are not working with a lot of data samples, you could of course push the whole dataset onto the GPU beforehand, but note that you might be wasting precious GPU memory.</p>

blackbirdbarber: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="49391">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This subset can then be wrapped by a <code>DataLoader</code> .</p>
</blockquote>
</aside>
<p>You suggest I put a <code>Subset</code> to <code>Dataloader</code> but I just used <code>Dataloader</code> to provide data for the <code>Subset</code>.</p>
<p>I have few problems, with that, I never saw lazy loading, and I am not sure what is this exactly. Searching on PyTorch website gave me 0 results.</p>
<pre><code class="lang-auto">dl = DataLoader(
  torchvision.datasets.MNIST('/data/mnist', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),                               
                               torchvision.transforms.Normalize(
                                (0.5,), (0.5,))
                             ])), shuffle=False)

train_ds  = torch.utils.data.Subset(dl.dataset, range(0, 50000-1))
valid_ds = torch.utils.data.Subset(dl.dataset, range(50000, 60000-1))


tensor = dl.dataset.data
tensor = tensor.to(dtype=torch.float32)
tr = tensor.reshape(tensor.size(0), -1) 
tr = tr/128 # tr = tr/255
targets = dl.dataset.targets
targets = targets.to(dtype=torch.long)
</code></pre>
<p>It would be cool if I could use <code>DataLoader</code> (first line of code) to also scale my tensors by 128 or 256 (divide by 128 or 256) for faster learning and set the float <code>dytpe</code> so once I have <code>Subsets</code> I am perfect to use <code>DataLoader</code> again, but this time I guess two <code>DataLoader</code>, one for training and other for validation.</p>

blackbirdbarber: <p>I found the idea of lazy loading is similar term like load on demand, or load single item (image) on demand to be more precise.</p>
<p>Load in here means load it from non GPU memory to GPU, which is what I was most interested.</p>
<p>This further means, writing a custom dataset and overwriting <code>__getitem__</code> as described in <a href="https://discuss.pytorch.org/t/making-iterable-objects-using-torch-utils-data-dataloader/16681">here</a>.</p>
<p>If we are writing a custom <code>Dataset</code> why do we need a <code>Subset</code> which is also a custom <code>Dataset</code>?</p>
<p>If we will load a single item, and we will do that for every item in the batch, wouldn’t that be slower, comparing to loading a batch from once (<code>DataLoader</code>).</p>
<p>Which leads to the quest when is the best time to load to GPU, in <code>DataSet</code> or in <code>DataLoader</code>?</p>

blackbirdbarber: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, can you share any example from Internet, where the lazy loading technique is used,  and also some example where Subsets are used to load MNIST.</p>
<p>I recall I searched for hours but somehow I ended short.</p>

ptrblck: <p>You should find plenty of custom <code>Datasets</code> in this discussion board.<br>
Here is an example:</p>
<pre><code class="lang-python">class MyLazyDataset(Dataset):
    def __init__(self, image_paths, targets, transform=None):
        self.image_paths = image_paths
        self.targets = targets
        self.transform = transform
        
    def __getitem__(self, index):
        # Load actual image here
        x = Image.open(self.image_paths[index])
        if self.transform:
            x = self.transform(x)
        y = self.targets[index]
        
        return x, y
    
    def __len__(self):
        return len(self.image_paths)
</code></pre>
<p>As you can see, you are just passing the image paths without loading all images beforehand.<br>
In the <code>__getitem__</code> method, the actual sample is loaded and transformed.<br>
In my example I would have to compute the targets before and pass them to <code>MyLazyDataset</code>.<br>
However, you could of course create the target in <code>__getitem__</code> e.g. by using the current image path and check for a class name.</p>
<p>If you are using <code>ImageFolder</code>, the images will be lazily loaded by default.</p>
<p>Here is a small example of the usage of <code>Subset</code>:</p>
<pre><code class="lang-python">dataset = datasets.MNIST(
    root='./data',
    download=False,
    transform=transforms.ToTensor()
)


train_dataset = Subset(dataset, indices=range(50000))
val_dataset = Subset(dataset, indices=range(50000, 60000))
</code></pre>
<p>You just have to pass your <code>Dataset</code> and the indices you would like to use in the subset.</p>