Multiple Sequential Datasets

noahw31: <p>Hello everybody. I want to train a Network and my data is of following nature:<br>
I have multiple measurement results stored in np.arrays, these measurements are indpenedent from each other and should not be mixed up, however they should all be used for training. The data is sequential, meaning we have to maintain the temporal dependencies (no shuffling).</p>
<p>How i want to train the network:<br>
Concatenate the datasets (i am using ConcatDataset for this but i am not sure if the data is mixed here somehow).<br>
Create a dataloader, that contains all these datasets and takes batches from each of them without mixing the data. For each batch training step i want the order of the datasets to be random, meaning in the first training step, dataset3 may be first and dataset1 may be last, and a different order in the next step.</p>
<p>I have this code to create my dataloader with a list of datasets</p>
<pre><code class="lang-auto">    train_loader = DataLoader(ConcatDataset(train_datasets), batch_size=batch_size, shuffle=False)
</code></pre>
<p>Would be happy to receive some recommendations on how this should be solved!<br>
Thanks in advance</p>

ptrblck: <aside class="quote no-group" data-username="noahw31" data-post="1" data-topic="179599">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/noahw31/48/58737_2.png" class="avatar"> noahw31:</div>
<blockquote>
<p>Create a dataloader, that contains all these datasets and takes batches from each of them without mixing the data.</p>
</blockquote>
</aside>
<p>This might not be trivial since the <code>DataLoader</code> will just index the <code>ConcatDataset</code> in order as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">dataset1 = TensorDataset(torch.arange(10).float())
dataset2 = TensorDataset(torch.arange(1, 11).float() * 10)
dataset3 = TensorDataset(torch.arange(1, 11).float() * 100)

dataset = torch.utils.data.ConcatDataset((dataset1, dataset2, dataset3))
loader = DataLoader(dataset, shuffle=False, batch_size=8)

for data in loader:
    print(data)
    
# [tensor([0., 1., 2., 3., 4., 5., 6., 7.])]
# [tensor([ 8.,  9., 10., 20., 30., 40., 50., 60.])]
# [tensor([ 70.,  80.,  90., 100., 100., 200., 300., 400.])]
# [tensor([ 500.,  600.,  700.,  800.,  900., 1000.])]
</code></pre>
<p>If the batch size doesn’t perfectly match the length of the subset you will end up with mixed samples.</p>
<aside class="quote no-group" data-username="noahw31" data-post="1" data-topic="179599">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/noahw31/48/58737_2.png" class="avatar"> noahw31:</div>
<blockquote>
<p>For each batch training step i want the order of the datasets to be random, meaning in the first training step, dataset3 may be first and dataset1 may be last, and a different order in the next step.</p>
</blockquote>
</aside>
<p>It might be easier to just create separate datasets and <code>DataLoader</code>s, select them randomly, and iterate them sequentially which would also avoid mixing samples from different subsets.</p>

noahw31: <p>Thank you for the valuable answer, it helps a lot as i was not aware of how exactly the dataloader gets the data from the ddataset. Now that i know this, i agree with your proposal to create separate datasets and dataloaders. I am using pytorch lightning to train (<code>Trainer</code> class) to which i can pass the argument e.g., <code>test_dataloaders</code>. Do you have an idea what would be the best way to pass multiple loaders? From what i have seen it always is recommended to use the <code>ConcatDataset</code>.</p>
<p>Thanks again for taking the time to reply!</p>

ptrblck: <p>I’m not familiar enough with Lightning but it seems <code>test_dataloaders</code> would accept multiple <code>DataLoader</code>s. I’m also unsure why you cannot mix samples from different subsets, but in case it’s a hard requirement you would need to check how Lightning handles multiple loaders in its <code>Trainer</code> class and in particular if it would mix samples.</p>

noahw31: <p>Thanks, i will check it out. I cannot mix the datasets because i am modeling a system in the time domain with memory dependencies, and the datasets are from different measurement setups so the data from the first dataset should not be in the same batch as the second dataset because it is uncorrelated, but the goal is to learn those correlations. Thanks</p>