Remove indices from Dataset

username: abbab
cooked: <p>I have a 2 datasets <code>supervised_data</code>  and <code>validation_data</code> which I used in a previous training</p>
<p>I want to exclude Indices of the <code>validation_data</code> from the <code>supervised_data</code></p>
<p>I tried <code>torch.utils.data.Subset(supervised_data, validation_data.indices)</code> but this selects only the validation indices that exist in the supervised_data</p>
<p>How can I get a subset of the <code>supervised_data</code> that doesn’t exist in the <code>validation_data</code>?</p>

username: ptrblck
cooked: <p>Could you explain how these datasets were created?<br>
Both datasets will use their own indices in the range <code>[0, len(dataset)-1]</code>.<br>
If both datasets are also using the same samples in the same order internally and assuming thatr <code>supervised_data</code> contains more samples than <code>validation_data</code>, then you could use a <code>Subset</code> with <code>indices = torch.arange(len(validation_data), len(supervised_data))</code>.</p>
<p>However, if the aforementioned conditions are not met, you might need to create a mapping between the samples of both datasets or, probably better, split them during their creation in a clean way.</p>

username: abbab
cooked: <p>Thank you for your response</p>
<p>This is the code for the dataset</p>
<pre><code class="lang-auto">class SLD_Labeled(Dataset):

    def __init__(self):
        self.root = "/home/ubuntu/workdir/data/real_stuff/"
        self.image_dir = os.listdir(self.root+'images/')
        self.bands = ['B02', 'B03', 'B04']
        
        
    def __len__(self):
        return len(self.image_dir)

    def __getitem__(self, index):
        
        subf = os.path.join(self.root, f'images/'+self.image_dir[index])
        multiband = []

        b = Image.open(subf).convert('RGB')
        label_file = os.path.join(self.root, f'labels/{self.image_dir[index]}').replace('.jpg', '.tif')
        
        label = np.array(Image.open(label_file).convert('P'))
        if ':' in label_file:
            #print('original image')
            pass
        else:
            #print('In generated image')
            label= np.where(label&lt;128, 0, label)
            label= np.where(label&gt;127, 1, label)
        bgr_images = np.array(b)

        
        data_tensor = torch.from_numpy(np.transpose(bgr_images,(2, 1, 0))).float()
        label_tensor = torch.from_numpy(label).long()
       

        return data_tensor, label_tensor,str(self.image_dir[index])
</code></pre>
<p>The same dataset was used for Both the <code>validation_data</code> and <code>supervised_data</code><br>
I am trying to remove the <code>validation_data</code> indices that exist in the supervised data so I can have a subset that doesn’t contain the <code>validation_data</code></p>

username: ptrblck
cooked: <p>I don’t see any indices used in the <code>Dataset</code> definition and thus assume that you’ve created the <code>supervised_data</code> and <code>validation_data</code> manually before somehow.<br>
If so, I think the easiest approach would be to split the indices before creating the <code>Subset</code>s as seen here:</p>
<pre><code class="lang-python">nb_samples = 1000 # set to your value
indices = np.arange(nb_samples)
train_idx, val_idx = train_test_split(indices, train_size=0.8)

train_dataset = Subset(dataset, train_idx)
val_dataset = Subset(dataset, val_idx)
</code></pre>