Splitting up sequential batches into randomly shuffled train-test subsets

asafbenj: <p>Hi,<br>
I have a sequential data set with shape (n_time_points, n_features). Obviously, I want my batches to contain consecutive timepoint sequences, but I also want to <strong>randomly</strong> split them up into training, validation and test batches. Going through the tutorials and forums, I came across some of the relevant classes (<code>Dataset</code>, <code>Sebset</code>, <code>DataLoader</code>, <code>SequentialSampler</code>, <code>BatchSampler</code>), but I can’t quite figure out how to combine them to achieve what I want. The methods I’ve found seemed to either shuffle the timepoints within a sequence or create subsets with consecutive batches. The <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">seq-to-seq tutorial</a> actually implements the data loading functions from scratch, which doesn’t seem like a great idea for me, especially if at some point I want to parallelize on multiple GPU, etc. For simplicity, I’m first trying a <code>batch_size</code> of 1, i.e. I just want to sample 1 <code>n_bptt</code>-length sequence at a time, but at some point I might want to use a larger one.<br>
(As you might’ve guessed, I’m pretty new to PyTorch. Any help would be highly appreciated.)</p>

ptrblck: <p>How would you like to create the time sequence from the data and how is the data currently stored?<br>
E.g. if your original data is in the shape <code>[n_time_points, features]</code> and you would like to create overlapping windows, you could slice it in the <code>__getitem__</code> of your custom <code>Dataset</code>.<br>
Here is a simple example code:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, data, window_size):
        self.data = data
        self.window_size = window_size
        
    def __getitem__(self, index):
        x = self.data[index:index+self.window_size]
        return x
    
    def __len__(self):
        return len(self.data) - self.window_size + 1

data = torch.arange(100).view(100, 1).float()
dataset = MyDataset(data, window_size=10)
loader = DataLoader(dataset, batch_size=2, shuffle=False)

for batch in loader:
    print(batch)
</code></pre>
<p>You can also set <code>shuffle=True</code> to shuffle the passed indices.</p>

asafbenj: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="106466">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data, window_size):
        self.data = data
        self.window_size = window_size
        
    def __getitem__(self, index):
        x = self.data[index:index+self.window_size]
        return x
    
    def __len__(self):
        return len(self.data) - self.window_size + 1

data = torch.arange(100).view(100, 1).float()
dataset = MyDataset(data, window_size=10)
loader = DataLoader(dataset, batch_size=2, shuffle=False)

for batch in loader:
    print(batch)
</code></pre>
</blockquote>
</aside>
<p>Thanks a lot for the super fast reply! IMHO, your answers on this forum are the single best learning resource for PyTorch beginners out there… <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
<p>I was hoping for a solution that loads consecutive seq’s within batch, while randomly splitting batches to train-test subsets, but I guess this may over-complicate things and is not strictly necessary. Instead, I’ve tried to adapt your suggestion for the common case of non-overlapping sequences (e.g. as in the PyTorch tutorial on seq-to-seq with Transformers mention above):</p>
<pre><code class="lang-auto">class NoOverlapDataset(Dataset):
    def __init__(self, data, window_size):
        self.data = data
        self.window_size = window_size

    def __getitem__(self, index):
        seq_i = index * self.window_size
        x = self.data[seq_i : seq_i + self.window_size]
        return x

    def __len__(self):
        return len(self.data) // self.window_size
</code></pre>
<p>Dose that make sense/look correct?</p>