Torch.utils.data.dataset.random_split

username: berat
cooked: <p>Hi,</p>
<p>torch.utils.data.dataset.random_split returns a <code>Subset</code> object which has no <code>transforms</code> attribute. How can I split a <code>Dataset</code> object and return another <code>Dataset</code> object with the same <code>transforms</code> attribute?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I’m not sure if that would be easily achievable without writing an own derived class.<br>
You could split your <code>Dataset</code> as the first step and pass the <code>Subset</code> to your main <code>Dataset</code> with all transformations.<br>
Would that work for you?</p>

username: berat
cooked: <p>How can I pass <code>Subset</code> to <code>Dataset</code>?</p>

username: ptrblck
cooked: <p>Here is a small example:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, subset, transform=None):
        self.subset = subset
        self.transform = transform
        
    def __getitem__(self, index):
        x, y = self.subset[index]
        if self.transform:
            x = self.transform(x)
        return x, y
        
    def __len__(self):
        return len(self.subset)

    
init_dataset = TensorDataset(
    torch.randn(100, 3, 24, 24),
    torch.randint(0, 10, (100,))
)

lengths = [int(len(init_dataset)*0.8), int(len(init_dataset)*0.2)]
subsetA, subsetB = random_split(init_dataset, lengths)
datasetA = MyDataset(
    subsetA, transform=transforms.Normalize((0., 0., 0.), (0.5, 0.5, 0.5))
)
datasetB = MyDataset(
    subsetB, transform=transforms.Normalize((0., 0., 0.), (0.5, 0.5, 0.5))
)
</code></pre>
<p>Let me know, if that would work for you.</p>

username: berat
cooked: <p>Thank you ptrblck. I will try this.<br>
But I would like ask whether is it a good idea to manually split my dataset and then create two separate <code>Dataset</code> objects from two different image folders? Is this a good practice?</p>

username: ptrblck
cooked: <p>Would you like to create separate folders for both splits?<br>
In that case you should manually split the indices and move/copy the files to the corresponding folders.<br>
<code>random_split</code> returns splits from a single <code>Dataset</code>.</p>
<p>It’s usually a good idea to split the data into different folders. However, in that case you won’t need <code>random_split</code>, but just two separate <code>Datasets</code>.</p>

username: saba
cooked: <p>Hi Ptrbclk</p>
<p>Sorry I have aquestion , I passed the balanced data 4000 positive and 4000 negative as DatasetTrain to the random split train_len for 70 % and valid_len for 30 %</p>
<pre><code>TrainData1, ValidationData1 = random_split(DatasetTrain,[train_len, valid_len])
</code></pre>
<p>After that TrainData1 and ValidationData1 can be unbalanced?<br>
I think since it is randomly they can be unbalanced.</p>

username: saba
cooked: <p>Indeed, I need after splitting(70% and 30% ) have<br>
training set and validation set again in the balanced mode</p>

username: ptrblck
cooked: <p>The more samples you use the lower the likelihood of creating an imbalance is.<br>
However, if you need strictly the same distribution, I would recommend to create the training and testing indices with <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">sklearn.model_selection.train_test_split</a> and provide the <code>stratify</code> argument.</p>

username: saba
cooked: <p>Sorry u mean that my DatasetTrain which is 4000 positive and 4000 negative if used this function it wil give me 70% balanced training and 30% balanced validation?</p>
<p>train_test_split( DatasetTrain,test_size=.3,train_size=.7, stratify??)</p>
<p>I am using:````<br>
DatasetTrain=CMBDataClassifier(root_dirTrain,root_dirTest,split=‘train’,transforms=transform,debug=False,CounterIteration=Iteration,SubID=0,TPID=0)</p>
<pre><code>train_len = int(0.7*len(DatasetTrain))
valid_len = len(DatasetTrain) - train_len
</code></pre>
<p>TrainData1, ValidationData1 = random_split(DatasetTrain,[train_len, valid_len])</p>
<pre><code>trainloader=torch.utils.data.DataLoader(TrainData1, batch_size=32,shuffle=True,drop_last=True, num_workers=0)

validationloader=torch.utils.data.DataLoader(ValidationData1, batch_size=6, drop_last=True,num_workers=0)```
</code></pre>
<p>my problem is maybe TrainData1 and  ValidationData1 will be unbalanced in case of positive and negative class.</p>

username: ptrblck
cooked: <p>If you want to make sure, both splits are balanced, you could get the target tensor and create the split indices using <code>train_test_split</code> and pass the target array to <code>stratify</code>.<br>
The returned train and test indices can then be used in <code>Subset</code> to create the datasets.</p>

username: saba
cooked: <p>sorry how I can get the target tensors I am using this class to load data</p>
<pre><code class="lang-auto">import scipy
import scipy.integrate as integrate
import numpy as np
import os
from scipy import io
import matplotlib.pyplot as plt
from os.path import dirname, join as pjoin
import scipy.io as sio
import matplotlib.pyplot as plt
from torch.autograd import Variable
from os.path import dirname, join as pjoin
import scipy.io as sio

ss=os.path.isdir(root_dirDurringTraining11)
if ss==False:
   os.mkdir(root_dirDurringTraining11)
   
class CMBDataClassifier():

   def __init__(self, root_dirTrain,root_dirTest,split,transforms,debug,CounterIteration,SubID,TPID):
       self.patches, self.labels = None, None
       if split=='train':
           patch_path = os.path.join(root_dirTrain,"Patches"+str(CounterIteration)+".mat")
           label_path = os.path.join(root_dirTrain,"Labels"+str(CounterIteration)+".mat")
           self.patches = scipy.io.loadmat(patch_path)['TrainPatchFinal_4']
           self.labels = scipy.io.loadmat(label_path)['TargetTrainFinal_4']

       else:
           patch_path = os.path.join(root_dirTest,"Patches"+".mat")
           label_path = os.path.join(root_dirTest,"Labels"+".mat")
           self.patches = scipy.io.loadmat(patch_path)['PatchTestFinal']
           
           self.labels = scipy.io.loadmat(label_path)['TargetTestFinal']

       self.split = split
       self.transforms = transforms
       self.debug = debug
       
   def __getitem__(self, index):

       patchF = self.patches[:, :, : , index]

       labelF = self.labels[index].astype(np.float32)
   
       if self.transforms is not None:
           patchF = self.transforms(patchF)

       return patchF,labelF


   def __len__(self):
       return self.patches.shape[-1]````</code></pre>

username: ptrblck
cooked: <p>You could iterate the dataset once and store all targets in e.g. a list or alternatively just load the targets using your current logic outside of the <code>Dataset</code>.</p>

username: saba
cooked: <p>Iu se that but give me nothing no labels</p>
<pre><code class="lang-auto">
    def __init__(self, root_dirTrain,root_dirTest,split,transforms,debug,CounterIteration,SubID,TPID):
        self.patches, self.labels = None, None
        if split=='train':


            label_path = os.path.join(root_dirTrain,"Labels"+str(CounterIteration)+".mat")

            self.labels = scipy.io.loadmat(label_path)['TargetTrainFinal_4']

        else:
            patch_path = os.path.join(root_dirTest,"Patches"+".mat")
            label_path = os.path.join(root_dirTest,"Labels"+".mat")
            self.patches = scipy.io.loadmat(patch_path)['PatchTestFinal']

            self.labels = scipy.io.loadmat(label_path)['TargetTestFinal']

        self.split = split
        self.transforms = transforms
        self.debug = debug
        
    def __getitem__(self, index):
        
 
        labelF = self.labels[index].astype(np.float32)

#        
        if self.transforms is not None:
            patchF = self.transforms(patchF)

        return labelF


    def __len__(self):
        return self.labels.shape[-1]
</code></pre>