How to split dataset into test and validation sets

Sanmitra: <p>I have a dataset in which the different images are classified into different folders.<br>
I want to split the data to test, train, valid sets.</p>
<p>Please help</p>

ranklord: <p>You can use built in function <code>torch.utils.data.random_split(dataset, lengths)</code>.</p>
<p>Check docs here: <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split</a><br>
Check source here: <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#random_split" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#random_split</a></p>
<p>Also, you can see this nice <a href="https://stackoverflow.com/a/50544887/4440387" rel="nofollow noopener">example</a>.</p>

Ashima_Garg: <pre><code class="lang-auto">import torch
from torchvision.datasets import MNIST
transform = transforms.Compose([transforms.ToTensor(), 
                                        transforms.Normalize((0.5,), (0.5,))])
dataset = MNIST(root = './data', train = train, transform = transform, download=True)
train_set, val_set = torch.utils.data.random_split(dataset, [50000, 10000])
</code></pre>
<p>The function of random_split to split the dataset is not working. The size of train_set and val_set returned are both 60000 which is equal to the initial dataset size.</p>
<p>A similar error is also reported on stack overflow:<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/57789645/torch-utils-data-random-split-is-not-splitting-the-data" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/9625519/user9625519" target="_blank" rel="nofollow noopener">
    <img alt="user9625519" src="https://www.gravatar.com/avatar/b081342eb633703ad7be65384ca09295?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/57789645/torch-utils-data-random-split-is-not-splitting-the-data" target="_blank" rel="nofollow noopener">torch.utils.data.random_split() is not splitting the data</a>
</h4>

<div class="tags">
  <strong>deep-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/9625519/user9625519" target="_blank" rel="nofollow noopener">
    user9625519
  </a>
  on <a href="https://stackoverflow.com/questions/57789645/torch-utils-data-random-split-is-not-splitting-the-data" target="_blank" rel="nofollow noopener">01:37PM - 04 Sep 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Please look into the issue.</p>
<p>Thanks.</p>

ptrblck: <p>Double post, answered <a href="https://discuss.pytorch.org/t/issues-with-torch-utils-data-random-split/22298/12">here</a>.</p>

msminhas93: <p><strong>You can use the following code for creating the train val split</strong>. You can specify the val_split float value (between 0.0 to 1.0) in the train_val_dataset function. You can modify the function and also create a train test val split if you want by splitting the indices of list(range(len(dataset))) in three subsets. <strong>Just remember to shuffle the list before splitting else you won’t get all the classes in the three splits</strong> since these indices would be used by the <strong>Subset</strong> class to <strong>sample from the original dataset.</strong></p>
<pre><code class="lang-auto">import torch
from torchvision.datasets import ImageFolder
from torch.utils.data import Subset
from sklearn.model_selection import train_test_split
from torchvision.transforms import Compose, ToTensor, Resize
from torch.utils.data import DataLoader

def train_val_dataset(dataset, val_split=0.25):
    train_idx, val_idx = train_test_split(list(range(len(dataset))), test_size=val_split)
    datasets = {}
    datasets['train'] = Subset(dataset, train_idx)
    datasets['val'] = Subset(dataset, val_idx)
    return datasets

dataset = ImageFolder('C:\Datasets\lcms-dataset', transform=Compose([Resize((224,224)),ToTensor()]))
print(len(dataset))
datasets = train_val_dataset(dataset)
print(len(datasets['train']))
print(len(datasets['val']))
# The original dataset is available in the Subset class
print(datasets['train'].dataset)

dataloaders = {x:DataLoader(datasets[x],32, shuffle=True, num_workers=4) for x in ['train','val']}
x,y = next(iter(dataloaders['train']))
print(x.shape, y.shape)
</code></pre>
<pre><code class="lang-auto">50080
37560
12520
Dataset ImageFolder
    Number of datapoints: 50080
    Root location: C:\Datasets\some-dataset
    StandardTransform
Transform: Compose(
               Resize(size=(224, 224), interpolation=PIL.Image.BILINEAR)
               ToTensor()
           )
torch.Size([32, 3, 224, 224]) torch.Size([32])

</code></pre>

20am847: <p>Could you explain what ‘32’ (<br>
<code>dataloaders = {x:DataLoader(datasets[x],32, shuffle=True, num_workers=4) for x in ['train','val']}</code> ) means?</p>

Amin_Jun: <p>I believe it would be the <strong>batch_size</strong> for the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">DataLoader</a>.</p>

msminhas93: <p>Yes as Amin_Jun pointed out it is the batch size of the datalodaers.</p>

20am847: <p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> !!!</p>

Arun_Sagar: <p>If we add augmentation to images using transform , this will augment validation set too ?<br>
If yes , how can i avoid that ?</p>

ptrblck: <p>You are usually creating separate training and validation <code>Dataset</code>s and can thus pass the desired transformations to them. The <a href="https://github.com/pytorch/examples/blob/2639cf050493df9d3cbf065d45e6025733add0f4/imagenet/main.py#L207-L233">ImageNet example</a> shows this usage.</p>

Kriti_Ohri: <p>I have a dataset that i have divided into train and validate datasets, how can we apply train and validate transform to the two splits?</p>

ptrblck: <p>You can pass different transformations to each dataset, as they will be applied in the corresponding <code>__getitem__</code> method.</p>

Kriti_Ohri: <p>This is my code,</p>
<h1>
<a name="creating-a-custom-class-for-reading-my-dataset-1" class="anchor" href="#creating-a-custom-class-for-reading-my-dataset-1"></a>creating a custom class for reading my dataset</h1>
<p>class DiabeticRetinopathy(Dataset):</p>
<p>def <strong>init</strong>(self, csv_file, root_dir, total=None,transform = None):</p>
<pre><code>self.annotations = pd.read_csv(csv_file)

self.root_dir = root_dir

self.transform = transform
</code></pre>
<p>def <strong>len</strong>(self):</p>
<pre><code>return len(self.annotations)  
</code></pre>
<p>def <strong>getitem</strong>(self, index):</p>
<pre><code>#img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])

img_path = os.path.join(self.root_dir, self.annotations.iloc[index].image +".jpg")

image = Image.open(img_path)

y_label = torch.tensor(int(self.annotations.iloc[index,1]))  

if self.transform:

  image = self.transform(image)

  return(image, y_label)
</code></pre>
<p>full_dataset = DiabeticRetinopathy(</p>
<pre><code>    csv_file="/content/DR/B.%20Disease%20Grading/B. Disease Grading/2. Groundtruths/a. IDRiD_Disease Grading_Training Labels.csv",

    root_dir="/content/DR/B.%20Disease%20Grading/B. Disease Grading/1. Original Images/a. Training Set", transform=  transforms.ToTensor())
</code></pre>
<p>**# Data augmentation for train and validate **</p>
<p>train_transform = transforms.Compose([</p>
<pre><code>                         transforms.Resize((860,860)),

                         transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0, hue=0.1),

                         transforms.RandomCrop(828,828),

                         transforms.GaussianBlur(11, sigma=(0.1, 2.0)),

                         transforms.RandomRotation(degrees=90),

                         transforms.RandomHorizontalFlip(p=0.5),

                         transforms.RandomVerticalFlip(p=0.5),

                         transforms.ToTensor(), #converting the dimension from (height,weight,channel) to (channel,height,weight) convention of PyTorch

                  transforms.Normalize([0.4346, 0.2110, 0.0705],[0.3102, 0.1663, 0.0852]) # Normalize by 3 means 3 StD's of the image net, 3 channels
</code></pre>
<p>])</p>
<p>validate_transform = transforms.Compose([</p>
<pre><code>                         transforms.Resize((828,828)),

                         transforms.ToTensor(),

                         transforms.Normalize([0.4346, 0.2110, 0.0705],[0.3102, 0.1663, 0.0852]) # Normalize the  3 channels
</code></pre>
<p>])</p>
<p><strong><span class="hashtag">#Splitting</span> the dataset into train and validate</strong></p>
<p>from torchvision import transforms, utils, datasets</p>
<p>from torch.utils.data import Dataset, DataLoader, random_split, SubsetRandomSampler, WeightedRandomSampler</p>
<p>train_sampler = SubsetRandomSampler(list(range(373)))</p>
<p>valid_sampler = SubsetRandomSampler(list(range(40)))</p>
<h1>
<a name="from-here-i-am-not-able-to-understand-how-to-apply-transformations-to-the-splits-train-and-validate-and-get-the-train-and-validate-loaders-ready-2" class="anchor" href="#from-here-i-am-not-able-to-understand-how-to-apply-transformations-to-the-splits-train-and-validate-and-get-the-train-and-validate-loaders-ready-2"></a>from here I am not able to understand how to apply transformations to the splits (train and validate) and get the train and validate loaders ready.</h1>
<p>Please help.</p>

ptrblck: <p>Something like this should work:</p>
<pre><code class="lang-python">train_dataset = DiabeticRetinopathy(csv_file, root_dir, transform=train_transform)
val_dataset = DiabeticRetinopathy(csv_file, root_dir, transform=validate_transform)
train_loader = DataLoader(train_dataset, sampler=train_sampler)
val_loader = DataLoader(val_dataset, sampler=valid_sampler)
</code></pre>

Kriti_Ohri: <p>Thank you soo much, I will try.</p>

MoNsj: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I have a dataset containing images. I am doing Federated learning using pytorch and pysyft. In pysyft, we basically create different workers so that the data can be trained on them in a decentralized manner. I have been using the train_test_split and subset to create two training indices which will be sent to the two defined workers for training. I would like to increase my training indices to 10 in order to train my 10 workers on separate data. So the question is how to make a dataset divided into 10 train indices?<br>
Looking forward to your help</p>

Kriti_Ohri: <p>Hello sir,<br>
Iam a beginnner in pytorch. I have a dataset of images that I want to split into train and validate datasets. I realized that the dataset is highly imbalanced containing 134 (mages) → label 0, 20(images)-&gt; label 1,136 (images)-&gt;label 2, 74(images)-&gt;lable 3 and 49(images)-&gt;label 4. So after splitting that dataset into train and validate, I would want to take the training dataset and balance it, and after that I want to apply transformations to train and validate dataset separately and get the train and validate loader ready. How can I go about it and the right approach? What I have done is…</p>
<p><strong># Custom dataset</strong><br>
class DiabeticRetinopathy(Dataset):<br>
def <strong>init</strong>(self, csv_file, root_dir, total=None,transform = None):<br>
self.annotations = pd.read_csv(csv_file)<br>
self.root_dir = root_dir<br>
self.transform = transform<br>
def <strong>len</strong>(self):<br>
return len(self.annotations)<br>
def <strong>getitem</strong>(self, index):<br>
<span class="hashtag">#img_path</span> = os.path.join(self.root_dir, self.annotations.iloc[index, 0])<br>
img_path = os.path.join(self.root_dir, self.annotations.iloc[index].image +".jpg")<br>
image = Image.open(img_path)<br>
y_label = torch.tensor(int(self.annotations.iloc[index,1]))<br>
if self.transform:<br>
image = self.transform(image)<br>
return(image, y_label)</p>
<p><strong># transform for the entire dataset</strong><br>
normalize = transforms.Normalize(mean=[x / 255.0 for x in [0.4346, 0.2110, 0.0705]],<br>
std=[x / 255.0 for x in [0.3102, 0.1663, 0.0852]])</p>
<p>my_transforms = transforms.Compose([<br>
transforms.ToTensor(),<br>
normalize<br>
])</p>
<p>full_dataset = DiabeticRetinopathy(<br>
csv_file="/content/B.%20Disease%20Grading/B. Disease Grading/2. Groundtruths/a. IDRiD_Disease Grading_Training Labels.csv",<br>
root_dir="/content/B.%20Disease%20Grading/B. Disease Grading/1. Original Images/a. Training Set", transform=my_transforms)</p>
<p><strong><span class="hashtag">#Splitting</span> the dataset into train and validate</strong><br>
train_set, val_set=torch.utils.data.random_split(full_dataset, [373,40 ])</p>
<p><strong># code for balancing</strong></p>
<h1>
<a name="get-all-targets-1" class="anchor" href="#get-all-targets-1"></a>Get all targets</h1>
<p>targets = []<br>
for _, target in train_dataset:<br>
targets.append(target)<br>
targets = torch.stack(targets)</p>
<h1>
<a name="compute-samples-weight-each-sample-should-get-its-own-weight-2" class="anchor" href="#compute-samples-weight-each-sample-should-get-its-own-weight-2"></a>Compute samples weight (each sample should get its own weight)</h1>
<p>class_sample_count = torch.tensor(<br>
[(targets == t).sum() for t in torch.unique(targets, sorted=True)])<br>
weight = 1. / class_sample_count.float()<br>
samples_weight = torch.tensor([weight[t] for t in targets])</p>
<h1>
<a name="create-sampler-dataset-loader-3" class="anchor" href="#create-sampler-dataset-loader-3"></a>Create sampler, dataset, loader</h1>
<p>sampler = WeightedRandomSampler(samples_weight, len(samples_weight))</p>
<p>train_loader = DataLoader(<br>
train_dataset, batch_size=batch_size, num_workers=1, sampler=sampler)</p>
<h1>
<a name="iterate-dataloader-and-check-class-balance-for-each-batch-4" class="anchor" href="#iterate-dataloader-and-check-class-balance-for-each-batch-4"></a>Iterate DataLoader and check class balance for each batch</h1>
<p>for i, (x, y) in enumerate(train_loader):<br>
print(“batch index {}, 0/1: {}/{}”.format(<br>
i, (y == 0).sum(), (y == 1).sum(), (y == 2).sum(), (y == 3).sum(), (y == 4).sum()))<br>
<strong><span class="hashtag">#In</span> the above code, how will I make my train dataset</strong></p>
<h1>
<a name="after-this-i-want-to-apply-the-train_tranform-to-the-balanced-training-dataset-and-create-train-and-validate-dataloaders-5" class="anchor" href="#after-this-i-want-to-apply-the-train_tranform-to-the-balanced-training-dataset-and-create-train-and-validate-dataloaders-5"></a>after this I want to apply the train_tranform to the balanced training dataset and create train and validate dataloaders.</h1>
<p>train_transform = transforms.Compose([</p>
<pre><code>                         transforms.Resize((2500,2500)),
                         transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0, hue=0.1),
                         transforms.RandomCrop(1490,1490),
                         transforms.GaussianBlur(11, sigma=(0.1, 2.0)),
                         transforms.RandomRotation(degrees=90),
                         transforms.RandomHorizontalFlip(p=0.5),
                         transforms.RandomVerticalFlip(p=0.5),
                         transforms.ToTensor(), #converting the dimension from (height,weight,channel) to (channel,height,weight) convention of PyTorch
                  transforms.Normalize([0.4346, 0.2110, 0.0705],[0.3102, 0.1663, 0.0852])
</code></pre>
<p>])</p>

senitent_signal: <p>Was looking for this</p>

Varun_Khare: <p>Can you write the code if one needs 80:10:10 split for train, val and test ?</p>