How to create a train-val split in a custom image datasets using Subset and SubsetRandomSampler?

username: theairbend3r
cooked: <p>I want to create a train+val set from my original trainset. The directory is split into train and test. I load the original train set and want to split it into train and val sets so I can evaulate validation loss during training using the <strong>train_loader</strong> and <strong>val_loader</strong>.</p>
<p>I’ve gone through other answers on this forum and the following is what I’ve come up with. There’s not a lot of documentation about this which explains things clearly.</p>
<p><code>image_transforms</code> simply converts the images ToTensor.</p>
<pre><code class="lang-auto">hotdog_dataset = datasets.ImageFolder(root = root_dir + "train",
                                      transform = image_transforms["train"]
                                     )

hotdog_dataset


######### OUTPUT ############

Dataset ImageFolder
    Number of datapoints: 498
    Root location: ../../../data/computer_vision/image_classification/hot-dog-not-hot-dog/train
    StandardTransform
Transform: Compose(
               Resize(size=(224, 224), interpolation=PIL.Image.BILINEAR)
               ToTensor()
           )
</code></pre>
<p>Now, I want to create a train+val  datasets out of this original train dataset. So, I get the length of this dataset as <code>hotdog_dataset_size</code> and create a list called <code>hotdog_dataset_indices</code> which I use as indices. I then obtain the <code>val_split_index</code> to create two lists which contains the train and validation indices called - <code>train_idx</code> and <code>val_idx</code>. I then pass this to SubsetRandomSampler to get <code>train_sampler</code> and <code>test_sampler</code>.</p>
<pre><code class="lang-auto">VAL_SPLIT_RATIO=0.2

hotdog_dataset_size = len(hotdog_dataset)
hotdog_dataset_indices = list(range(hotdog_dataset_size))

val_split_index = int(np.floor(VAL_SPLIT_RATIO * hotdog_dataset_size))

train_idx, val_idx = hotdog_dataset_indices[val_split_index:], hotdog_dataset_indices[:val_split_index]

train_sampler = SubsetRandomSampler(train_idx)
val_sampler = SubsetRandomSampler(val_idx)

print(train_sampler)


##### OUTPUT #####

&lt;torch.utils.data.sampler.SubsetRandomSampler at 0x7fec33c72d68&gt;
</code></pre>
<p><strong>Q1. What is <code>train_sampler</code> and <code>val_sampler</code>? What does the output of <code>SubsetRandomSampler</code> contain? How do we use it?</strong></p>
<p>I then use <code>Subset</code> on the original dataset and pass <code>train_sampler</code> and <code>test_sampler</code>.</p>
<pre><code class="lang-auto">hotdog_dataset_train = Subset(dataset=hotdog_dataset, indices=train_sampler)
hotdog_dataset_val = Subset(dataset=hotdog_dataset, indices=val_sampler)

print(hotdog_dataset_train)

### OUTPUT ####
&lt;torch.utils.data.dataset.Subset at 0x7fec2a6720f0&gt;
</code></pre>
<p><strong>Q2. Did I use <code>Subset</code> and <code>SubsetRandomSampler</code> above correctly? Or should I have directly passed the <code>train_sampler</code> to a dataloader using the <code>sampler</code> argument?</strong></p>
<p>After this I created dataloaders for train and val.</p>
<pre><code class="lang-auto">train_loader = DataLoader(dataset=hotdog_dataset_train, shuffle=False, batch_size=8)
val_loader = DataLoader(dataset=hotdog_dataset_val, shuffle=False, batch_size=8)
</code></pre>
<p><strong>Q3. Do we use <code>shuffle=True</code> here?</strong><br>
<strong>Q4. Can we use different <code>batch_size</code> for train and val?</strong></p>
<p>When I try to get a single batch from the train loader, I get an error.</p>
<pre><code class="lang-auto">single_batch = next(iter(train_loader))


#### OUTPUT ####
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-19-f4d37beb80cc&gt; in &lt;module&gt;
----&gt; 1 single_batch = next(iter(train_loader))

~/miniconda3/envs/toothless/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    343 
    344     def __next__(self):
--&gt; 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

~/miniconda3/envs/toothless/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--&gt; 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

~/miniconda3/envs/toothless/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/miniconda3/envs/toothless/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in &lt;listcomp&gt;(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/miniconda3/envs/toothless/lib/python3.6/site-packages/torch/utils/data/dataset.py in __getitem__(self, idx)
    255 
    256     def __getitem__(self, idx):
--&gt; 257         return self.dataset[self.indices[idx]]
    258 
    259     def __len__(self):

TypeError: 'SubsetRandomSampler' object does not support indexing
</code></pre>
<p><strong>The overarching question is, what are the different ways to create a train-val split for image datasets and which of these methods is the recommended way of doing things(especially for the case outlined above).</strong></p>
<p>It would be great if you could show it using code based on the above case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Blueberry
cooked: <p>when you use <em>SubsetRandomSampler</em> it by defaults suffles the dataset so you can’t use shuffle while using it.</p>
<pre><code class="lang-auto">train_loader = DataLoader(dataset=hotdog_dataset_train,  batch_size=8)
val_loader = DataLoader(dataset=hotdog_dataset_val, batch_size=8)
</code></pre>
<p>Should work.<br>
Or you can directly use following commands and remove previous declarations of <em>hotdog_dataset_train</em> and <em>hotdog_dataset_val</em></p>
<pre><code class="lang-auto">train_loader = DataLoader(dataset=hotdog_dataset,  batch_size=8, sampler = train_sampler)
val_loader = DataLoader(dataset=hotdog_dataset, batch_size=8, sampler = val_sampler)
</code></pre>

username: MJN
cooked: <p>the train_loader and val_loader length are same as train_data length. But the train_sampler and val_sampler length are different is that possible?</p>
<pre><code class="lang-auto">    TRAIN_DATA_PATH = "./Dataset/Train"
    TEST_DATA_PATH = "./Dataset/Test"
    VALID_DATA_PATH = "./Dataset/Valid"
    BATCH_SIZE = 32

    TRANSFORM_IMG = transforms.Compose([transforms.Resize((224,224)),
    transforms.ToTensor(),transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225] )
    ])

    train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, 
  transform=TRANSFORM_IMG)
    
    train_loader_all = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)

  
    validation_split = .3
    shuffle_dataset = True
    random_seed= 42

    # Creating data indices for training and validation splits:  
    dataset_size = len(train_loader_all.dataset)
    indices = list(range(dataset_size))
    split = int(np.floor(validation_split * dataset_size))
    if shuffle_dataset :
       np.random.seed(random_seed)
       np.random.shuffle(indices)
    train_indices, val_indices = indices[split:], indices[:split]

    # Creating PT data samplers and loaders:
    train_sampler = SubsetRandomSampler(train_indices)
    valid_sampler = SubsetRandomSampler(val_indices)
    print("leng:",len(train_sampler))
    print("leng:",len(valid_sampler))

    train_loader = torch.utils.data.DataLoader(train_data, batch_size=BATCH_SIZE,sampler=train_sampler)
    val_loader = torch.utils.data.DataLoader(train_data, batch_size=BATCH_SIZE,sampler=valid_sampler)
    
    print("val loader:",len(val_loader.dataset))
    print("train loader",len(train_loader.dataset))
</code></pre>
<p>this is the output</p>
<blockquote>
<p>leng: 25970<br>
leng: 11130<br>
val loader: 37100<br>
train loader 37100</p>
</blockquote>