How do I split a custom dataset into training and test datasets with SubsetRandomSampler?

Kla: <p>Hi everyone, I’m trying to create the train, test and evaluate datasets with SubsetRandomSampler but I’m unfortunately not understanding how it actually works. Can anyone please help me?</p>
<p>csv_file=‘AudioData/iden_split2.csv’<br>
root_dir=‘AudioData/wav/’</p>
<p>class SpeakerRecognitionDataset(Dataset):<br>
def <strong>init</strong>(self, csv_file, root_dir, transform=None):<br>
“”"<br>
Args:<br>
csv_file (string): Path to the csv file with annotations.<br>
root_dir (string): Directory with all the sounds.<br>
transform (callable, optional): Optional transform to be applied<br>
on a sample.<br>
“”"<br>
# Read the Data of every speaker<br>
self.speaker_sound = pd.read_csv(csv_file, sep=’ ')<br>
self.root_dir = root_dir<br>
self.transform = transform</p>
<pre><code>def __len__(self):
    # Number of all sounds
    return len(self.speaker_sound)

def __getitem__(self, idx):
    #if torch.is_tensor(idx):
    #    idx = idx.tolist()
    sound_file_name = os.path.join(self.root_dir, self.speaker_sound.iloc[idx, 1])
    if os.path.isfile(sound_file_name):
        waveform, sample_rate = torchaudio.load(sound_file_name)
        if self.transform:
            waveform = self.transform(waveform)

    return waveform
</code></pre>
<p>speaker = SpeakerRecognitionDataset(csv_file,<br>
root_dir)</p>
<p>dataloader = DataLoader(speaker, batch_size=10,<br>
shuffle=True, num_workers=1, pin_memory=True)</p>

ptrblck: <p>It seems you have already created the custom <code>Dataset</code> to load all data.<br>
Now you could create the indices for all samples e.g. using <code>torch.arange(len(dataset))</code>. You could then split these indices into training, validation, as well as test indices, and pass these indices to <code>SubsetRandomSampler</code>. These samplers can then be passed to <code>DataLoader</code>s to create the training, validation, and test loaders.</p>

Kla: <p>Hi, I guess it would work if I would randomly choose my Audio files. But in the photo the ones with number 1 (pink color) should go to train, number 2 val and number 3 to test. Can you please suggest me how can I create the indices in this case? Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b84b09f715b95c145ed51fbcaf73cc7e74c79ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b84b09f715b95c145ed51fbcaf73cc7e74c79ef" title="Screenshot from 2020-05-22 15-29-47"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b84b09f715b95c145ed51fbcaf73cc7e74c79ef_2_690x387.png" alt="Screenshot from 2020-05-22 15-29-47" data-base62-sha1="1DTxq5x4fbu61TY5EVMXcuUJSpN" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b84b09f715b95c145ed51fbcaf73cc7e74c79ef_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b84b09f715b95c145ed51fbcaf73cc7e74c79ef_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b84b09f715b95c145ed51fbcaf73cc7e74c79ef.png 2x" data-dominant-color="E8E8E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-22 15-29-47</span><span class="informations">1366×768 193 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Since your <code>csv</code> file already has the split indices, you could load it e.g. with <code>pandas</code> and create three separate lists of file paths (one list corresponding to each split).<br>
In that case you could directly create three separate <code>Dataset</code>s using these file paths lists.</p>