Transforms according to type of data

shwe87: <p>Hi all,</p>
<p>I divide the training, validation and testing in my custom dataset as follows:</p>
<pre><code class="lang-auto">total_dataset = speechdataset.SpanishSpeechDataSet(csv_files=csv_files, root_dir=root_dir)
train_size = int(0.8 * len(total_dataset))
val_test_size = len(total_dataset) - train_size
train_dataset, val_test_dataset = torch.utils.data.random_split(total_dataset, [train_size, val_test_size])
valid_size = int(0.9 * len(val_test_dataset))
test_size = len(val_test_dataset) - valid_size
val_dataset, test_dataset = torch.utils.data.random_split(val_test_dataset, [valid_size, test_size])
</code></pre>
<p>In this case, I would like to inject noise to the training dataset only, how can I do that? Because I can’t pass the type of data=“train” or “valid” like normally done…</p>
<p>Thanks.<br>
BR,<br>
Shweta.</p>

ptrblck: <p>You could add the noise in the training loop (outside of the <code>Dataset</code>).<br>
Alternatively you could create a <code>Dataset</code> instance for each split (and add the noise to the training dataset), create the split indices, and wrap the datasets together with the corresponding split indices in <code>Subset</code>s.</p>

shwe87: <blockquote>
<p>Alternatively you could create a  <code>Dataset</code>  instance for each split (and add the noise to the training dataset), create the split indices, and wrap the datasets together with the corresponding split indices in  <code>Subset</code> s.</p>
</blockquote>
<p>This is what I am looking for but I don’t know how to do this… <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Here is a small dummy example:</p>
<pre><code class="lang-python">from sklearn.model_selection import train_test_split

dataset_train = MyDataset(noise=...)
dataset_val = MyDataset()
dataset_test = MyDataset()

idx = np.arange(len(dataset_train))
train_idx, val_idx = train_test_split(idx, train_size=0.8)
val_idx, test_idx = train_test_split(val_idx, train_size=0.5)

dataset_train = Subset(dataset_train, train_idx)
dataset_val = Subset(dataset_val, val_idx)
dataset_test = Subset(dataset_test, test_idx)
</code></pre>

shwe87: <p>I am really sorry, I don’t get it.<br>
So, I get the whole dataset and then split it.</p>
<pre><code class="lang-auto">total_dataset = speechdataset.SpanishSpeechDataSet(csv_files=csv_files, root_dir=root_dir)
</code></pre>
<p>Are you suggesting to create 3 different datasets before the split? Like lets say for train_set I take x rows, for valid y rows and for test z rows.</p>
<p>But then i f I do so, I don’t understand why would I need to do the split again?</p>

ptrblck: <p>No, you should create the “same” dataset three times (for the training dataset, you should add the noise argument, if available).<br>
Each dataset is then passed to <code>Subset</code> with the corresponding indices.</p>
<p>If you are lazily loading the data, you won’t see any performance penalties using this approach.</p>

shwe87: <p>Ok, thank you.</p>
<p>I didn’t understand this approach. Now, I understand it better. Thanks you.</p>