Custom dataset based on CIFAR10

Sayyed_Ali_Mousavi: <p>Hello everyone.<br>
I want to create a dataset based on CIFAR10. Then create a dataloader and train my model on it.<br>
I have a function that gives some noises to the images of CIFAR10, say:</p>
<pre><code class="lang-auto">def create_noise(model, image):
      .....
      return noisy_image
</code></pre>
<p>What is the best way to create this dataset and dataloader of noisy images?<br>
Things I did:</p>
<ol>
<li>
<p>I tried to append the new data in a list, But the problem with this method is that this list becomes very large and a memory error may appear.</p>
</li>
<li>
<p>I use custom dataset approach as follows:</p>
</li>
</ol>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, model):
        transform = transforms.ToTensor()
        self.dst_train = datasets.CIFAR10('data', train=True, download=True, transform=transform)
        self.model = model


    def __getitem__(self, idx):
        image = create_noise(self.model, self.dst_train[idx][0].unsqueeze(0))
        image = image.squeeze(0)
        label = self.dst_train[idx][1]
        return image, label
    def __len__(self):
        return len(self.dst_train)
</code></pre>
<p>This method works. But the problem is about ‘Time’. Although the <code>create_noise</code> function can receive batch data, here it has to take the images one by one, which takes a lot of time.<br>
Can we pass a batch of data in <code>__getitem__</code> . If the answer is yes, how? And in this case, how is the dataloader made for the training?<br>
Finally, I don’t have enough experience with PyTorch and I may not have chosen the right method.<br>
I appreciate any help.</p>

ptrblck: <p>A custom <code>Dataset</code> should certainly work and depending on the <code>create_noise</code> method you could directly add the noise to the data as seen in <a href="https://discuss.pytorch.org/t/how-to-add-noise-to-inputs-as-a-function-of-input/54839/2">this post</a> or sample it in each iteration.<br>
Alternatively, you could also write a custom transformation as seen in <a href="https://discuss.pytorch.org/t/how-to-add-noise-to-mnist-dataset-when-using-pytorch/59745/2">this post</a>, which might be a better approach.</p>
<p>However, based on your description I understand that <code>create_noise</code> might be expensive and you want to avoid calling it for each sample and would thus prefer to call it for the entire batch.<br>
In this case you could use the <code>BatchSampler</code> and pass the indices for the entire batch to <code>__getitem__</code> as seen in <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">this post</a>. This would also mean that the specified <code>batch_size</code> in the <code>DataLoader</code> is not representing the actual batch size any more and the number of samples in each batch would be defined by <code>loader.batch_size * sampler.batch_size</code>. In my example I’ve defined it only in the <code>sampler</code> and kept it default in the <code>DataLoader</code>.</p>

Sayyed_Ali_Mousavi: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. With your help, I created the following code:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, model):
        transform = transforms.ToTensor()
        self.dst_train = datasets.CIFAR10('data', train=True, download=True, transform=transform)
        self.model = model

    def __getitem__(self, idx):
        image = create_noise(self.model, self.dst_train[idx][0])
        image = image.data
        label = self.dst_train[idx][1]
        return image, label
    def __len__(self):
        return len(self.dst_train)

</code></pre>
<p>and</p>
<pre><code class="lang-auto">net = net.to(device)
noisy_dataset = MyDataset(net)
sampler = torch.utils.data.sampler.BatchSampler(
torch.utils.data.sampler.RandomSampler(noisy_dataset),
batch_size=64, drop_last=False)

train_loader = DataLoader(noisy_dataset, sampler=sampler)
</code></pre>
<p>But in the training model with this <code>train_loader</code>, I get the following error:</p>
<blockquote>
<p>TypeError: list indices must be integers or slices, not list</p>
</blockquote>
<p>Can you help me with this?</p>

thecho7: <p>It seems <code>self.dst_train</code> is not a tensor but a list.<br>
From the example which is <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned,</p><aside class="quote quote-modified" data-post="4" data-topic="78788">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">How to use BatchSampler with __getitem__ dataset</a> 
  </div>
  <blockquote>
    Maybe this code snippet might be helpful: 
class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(100).view(100, 1).float()
        
    def __getitem__(self, index):
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()        
sampler = torch.utils.data.sampler.BatchSampler(
    torch.utils.data.sampler.RandomSampler(dataset),
    batch_size=10,
    drop_last=False)

loader = DataLoader(
    da…
  </blockquote>
</aside>

<p>The dataset is a tensor has size of (100, 1) but your <code>self.dst_train</code> is a list.<br>
List type cannot be indexed with list where a tensor can do.</p>
<p>Try convert <code>self.dst_train</code> to a tensor.<br>
Should work</p>