What is the difference between creating a validation set using random_split as opposed to SubsetRandomSampler?

username: theairbend3r
cooked: <p>Hi all!</p>
<p>I was wondering what the difference between using <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split" rel="nofollow noopener">random_split</a> and <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.SubsetRandomSampler" rel="nofollow noopener">SubsetRandomSampler</a> is.</p>
<p>Do we use SubsetRandomSampler together with Subset?</p>
<p>What are the pros and cons of using <code>SubsetRandomSampler</code> and <code>random_split</code> methods for creating a train-val split (or any split for that matter)for image datasets?</p>

username: ptrblck
cooked: <p><code>random_split</code> returns two <code>Datasets</code> with non-overlapping indices, which were drawn randomly based on the passed lengths, while <code>SubsetRandomSampler</code> accepts the indices directly.<br>
Both can be used for different use cases.</p>
<p>E.g. if you want to run a training epoch with certain samples only (e.g. from a subset of classes or “hard to learn” samples), you could calculate these indices somehow and use a <code>SubsetRandomSampler</code> for this training epoch.</p>
<p><code>random_split</code> on the other hand splits the dataset randomly, so that you won’t have that much control over it.</p>

username: theairbend3r
cooked: <p>Ah. So, with <code>random_split</code>, I wouldn’t be able, for instance, do a stratified split based on output labels in case of imbalanced data? But I would able to do the same with <code>SubsetRandomSampler/ WeightedRandomSampler</code>?</p>

username: ptrblck
cooked: <p>Yes, you could create balanced batches using <code>WeightedRandomSampler</code> or over/undersample specific classes with <code>SubsetRandomSampler</code>, while this wouldn’t be possible by just using <code>random_split</code>.<br>
However, you could of course use a <code>random_split</code> in combination with the samplers. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: theairbend3r
cooked: <p>After going through a lot of blogs and answers on the discuss page, I think I finally got the hang of it.</p>
<p>Let me know If I got the following implementations of  train + val set creation using <code>random_split</code> and <code>SubsetRandomSampler</code>  right.</p>
<p>And finally, discuss the challenge of doing this with <code>WeightedRandomSampler</code>.</p>
<p>The data is read using <code>ImageFolder</code>. Task is binary image classification with 498 images in the dataset which are equally distributed among both classes (249 images each).</p>
<pre><code class="lang-auto">img_dataset = ImageFolder(..., transforms=t)
</code></pre>
<h3><a name="h-1-subsetrandomsampler-1" class="anchor" href="#h-1-subsetrandomsampler-1"></a>1. <code>SubsetRandomSampler</code></h3>
<pre><code class="lang-auto">dataset_size = len(img_dataset)
dataset_indices = list(range(dataset_size))

np.random.shuffle(dataset_indices)

val_split_index = int(np.floor(0.2 * dataset_size))

train_idx, val_idx = dataset_indices[val_split_index:], dataset_indices[:val_split_index]

train_sampler = SubsetRandomSampler(train_idx)
val_sampler = SubsetRandomSampler(val_idx)


train_loader = DataLoader(dataset=img_dataset, shuffle=False, batch_size=8, sampler=train_sampler)
validation_loader = DataLoader(dataset=img_dataset, shuffle=False, batch_size=1, sampler=val_sampler)
</code></pre>
<h3><a name="h-2-random_split-2" class="anchor" href="#h-2-random_split-2"></a>2. <code>random_split</code></h3>
<p>Here, out of the 498 total images, 400 get randomly assigned to train and the rest 98 to validation.</p>
<pre><code class="lang-auto">dataset_train, dataset_valid = random_split(img_dataset, (400, 98))

train_loader = DataLoader(dataset=dataset_train, shuffle=True, batch_size=8)
val_loader = DataLoader(dataset=dataset_valid, shuffle=False, batch_size=1)
</code></pre>
<h3><a name="h-3-weightedrandomsampler-3" class="anchor" href="#h-3-weightedrandomsampler-3"></a>3. <code>WeightedRandomSampler</code></h3>
<blockquote>
<p>if someone stumbled here searching for <code>WeightedRandomSampler</code>, check <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answer <a href="https://discuss.pytorch.org/t/some-problems-with-weightedrandomsampler/23242/40">here</a> for a good explanation of what is happening below.</p>
</blockquote>
<p>Now, how does <code>WeightedRandomSampler</code> fit in creating train+val set? Because unlike <code>SubsetRandomSampler</code> or <code>random_split()</code>,  we’re not splitting for train and val here. We’re simply ensuring that each batch gets equal number of classes during training.</p>
<p>So, my guess is we need to use <code>WeightedRandomSampler</code> <strong>after</strong> <code>random_split()</code> or <code>SubsetRandomSampler</code>. But this wouldn’t ensure that train and val have similar ratio between classes.</p>
<pre><code class="lang-auto">target_list = []

for _, t in imgdataset:
    target_list.append(t)
    
target_list = torch.tensor(target_list)
target_list = target_list[torch.randperm(len(target_list))]

# get_class_distribution() is a function that takes in a dataset and 
# returns a dictionary with class count. In this case, the 
# get_class_distribution(img_dataset)  returns the following - 
# {'class_0': 249, 'class_0': 249}
class_count = [i for i in get_class_distribution(img_dataset).values()]
class_weights = 1./torch.tensor(class_count, dtype=torch.float) 

class_weights_all = class_weights[target_list]

weighted_sampler = WeightedRandomSampler(
    weights=class_weights_all,
    num_samples=len(class_weights_all),
    replacement=True
)
</code></pre>
<p>So, how do we ensure that our <strong>train and val sets</strong> as well as <strong>batches in train_loader and val_loader</strong> have an equal distribution of output classes?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="theairbend3r" data-post="5" data-topic="72462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/b4bc9f/48.png" class="avatar"> theairbend3r:</div>
<blockquote>
<p>So, how do we ensure that our <strong>train and val sets</strong> as well as <strong>batches in train_loader and val_loader</strong> have an equal distribution of output classes?</p>
</blockquote>
</aside>
<p>I would use a stratified split using e.g. <code>sklearn.model_selection.train_test_split</code>, calculate the weights separately for each split, and use a <code>WeightedRandomSampler</code> for each subset to get balanced batches.</p>

username: theairbend3r
cooked: <p>Thank you so much for your patience in answering! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Ah. Is there a native PyTorch function that does that or do I have to use sklearn only?</p>
<p>Also, are the above implementations correct?</p>

username: ptrblck
cooked: <p>I’m not aware of a <code>stratified</code> split in PyTorch and would just use the <code>sklearn</code> implementation.</p>
<p>The implementations look correct, but I haven’t tested them. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Odejide_Meshach
cooked: <p>please can you show us how the code will look like?</p>