How to split test and train data keeping equal proportions of each class?

somnath: <p>Suppose I have a dataset with the following classes:</p>
<p>Class A: 3000 items<br>
Class B: 1000 items<br>
Class C: 2000 items</p>
<p>I want to split this dataset in two parts so that there are 25% data in test set. However, how can I do this so that equal percentage of each class is present in the test set? These items should be randomly selected. For e.g., the test data should be like the following:</p>
<p>Class A: 750 items<br>
Class B: 250 items<br>
Class C: 500 items</p>

Pfaeff: <p>Make a list for each class, take 25% at random from each list, combine the lists and shuffle.</p>

somnath: <p>Is there any PyTorch code for the same? Or do we need to do it using some other library.</p>
<p>I am asking for PyTorch code for performance reasons in case of large datasets.</p>

Pfaeff: <p>You could keep lists of pairs of filenames and labels and prepare batches asynchronously in a background worker thread. That should be efficient even when training with millions of images. You might wanna take a look at the  <code>torch.utils.data.dataset.Dataset</code> and <code>torch.utils.data.sampler.Sampler</code> classes that you can use in conjunction with the <code>torch.utils.data.DataLoader</code>.</p>

ptrblck: <p>If you load the dataset completely before passing it to the <code>Dataset</code> and <code>DataLoader</code> classes, you could use <a href="http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">scikit-learn’s train_test_split</a> with the <code>stratified</code> option.</p>

somnath: <p>In that case, will it be possible to use something like num_workers while loading?</p>

ptrblck: <p>This would split the dataset before using any of the PyTorch classes.<br>
You would get different splits and create different <code>Dataset</code> classes:</p>
<pre><code class="lang-python">
X = np.random.randn(1000, 2)
y = np.random.randint(0, 10, size=1000)

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.1, stratify=y)
np.unique(y_train, return_counts=True)
np.unique(y_val, return_counts=True)

train_dataset = Dataset(X_train, y_train, ...)
train_loader = DataLoader(train_dataset, ...)
</code></pre>
<p>The <code>DataLoader</code> is thus completely unaffected by this and you can use <code>num_workers</code> as you wish.<br>
If you need lazy loading, <a class="mention" href="/u/pfaeff">@Pfaeff</a> had a good approach. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

somnath: <p>Understood, thanks a lot.</p>

sampa: <p>I also have a same question. How to solve this issue?</p>

ptrblck: <p>Is the mentioned <code>train_test_split</code> not working for your use case?</p>

sampa: <p>I have image data. I want to split the image data such a way that class proportion is maintained.</p>

ptrblck: <p>Are you able to get all the targets without loading the actual data?<br>
If so, you could use <code>train_test_split</code> passing the indices and targets, and use the index splits for your <code>Subsets</code>.</p>

sampa: <p>The problem is Solved. Thanks</p>

saba: <p>Dear Ptrblck,</p>
<p>I create the splited data for validation and training, Now I want to pass it to the data loader. I am confused by knowing how to make my data to pass to the data loader indeed I don’t know how to make DatasetTrain and  DatasetValid I applied this:</p>
<pre><code class="lang-auto">    [train_D, valid_D,train_L,valid_L]= train_test_split(WholeData.numpy(),WholeTargetArray, test_size=0.2,train_size=0.8, shuffle=True, stratify=WholeTargetArray)
    DatasetTrain=Dataset(train_D,train_L)???????
    DatasetValid=Dataset(valid_D,valid_L)?????
trainloader=torch.utils.data.DataLoader(DatasetTrain, batch_size=32,shuffle=True,drop_last=True, num_workers=0)
 validationloader=torch.utils.data.DataLoader(DatasetValid, batch_size=6, drop_last=True,num_workers=0)</code></pre>

saba: <p>It is my Final attempt:</p>
<pre><code class="lang-auto">    [train_D, valid_D,train_L,valid_L]= train_test_split(WholeData.numpy(),WholeTargetArray, test_size=0.2,train_size=0.8, shuffle=True, stratify=WholeTargetArray)
   
    DatasetTrain = TensorDataset(torch.from_numpy(train_D),torch.from_numpy(train_L))
        
    DatasetValid=TensorDataset(torch.from_numpy(valid_D),torch.from_numpy(valid_L))

    trainloader=torch.utils.data.DataLoader(DatasetTrain,batch_size=32,shuffle=True,drop_last=True, num_workers=0)

    validationloader=torch.utils.data.DataLoader(DatasetValid, batch_size=6, drop_last=True,num_workers=0)
</code></pre>
<p>I think it is correct. What do you think?</p>

ptrblck: <p>This approach looks correct.</p>

saba: <p>This code works very good for me. and my data is balanced <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

saba: <p>Hi Ptrblck,</p>
<p>Sorry, for finding the best number of epoch, I use the list of validation loss from all  epochs. and then convert the list to the numpy array to find the minimum loss and index and get the optimum epoch there.<br>
with the CPU it works good but when I run the code with GPU it give me error, the code is <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">val_lossesArray=np.asarray(val_losses)
   vvv2=torch.from_numpy(val_lossesArray)
   result = np.where(val_lossesArray == np.amin(val_lossesArray))
   vv1=result[-1]
   EpochFinal=vv1[0]
   print("best epoch",Epoc) ```


and the error is ( result = np.where(val_lossesArray == np.amin(val_lossesArray))
TypeError: eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
* (Tensor other)
     didn't match because some of the arguments have invalid types: (!numpy.ndarray!)
* (Number other)
     didn't match because some of the arguments have invalid types: (!numpy.ndarray!)
)</code></pre>

saba: <p>I tried this one too but does not work</p>
<pre><code class="lang-auto">    val_lossestensor=torch.from_numpy(val_lossesArray)
    VV2,Index=torch.min(val_lossestensor)
</code></pre>
<pre><code>VV2,Index=torch.min(val_lossestensor)
</code></pre>
<p>File “/apps/pytorch/1.2.0-py36-cuda90/lib/python3.6/site-packages/torch/tensor.py”, line 384, in <strong>iter</strong><br>
raise TypeError(‘iteration over a 0-d tensor’)</p>

nicaushaz: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I was wondering is there any other way to do this now? A torch approach, instead of reading a dataframe doing a train test split and then creating 3 dataloaders and 3 datasets for train/val/split?</p>
<p>Thank you in advance.</p>