Is there a data.Datasets way to use a sliding window over time series data?

HenryPowell: <p>Up until now I’ve always dealt with splitting time series data into inputs of a specific length by running a sliding window of a particular size over each datapoint and saving each of the windows to a seperate directory to train a model on.</p>
<p>This is very time and memory consuming and means that it’s pretty long winded to try different sized inputs into my models.</p>
<p>I’m wondering if there’s a way I could do this just by using transforms with a data.Datasets object. I’ve managed to do it by taking random samples of a particular size from each datapoint but obviously this means that I end up with significantly less data than by my normal means as I’m only taking a small number of windowed samples from each datapoint rather than lots of windows by using the sliding window method.</p>
<p>For a number of cases the dataset is too large to fit into RAM so it’s not possible to load it all and just slide a window over the whole dataset and get the output using <strong>getitem</strong></p>
<p>Cheers.</p>

JuanFMontesinos: <p>Save everything as numpy memory maps and then load specific parts of the array.<br>
You can just generate a file with the information about each sample and specific stamps to load.</p>

HenryPowell: <p>Thanks I’ll look into this. I can see how this might work by constructing a single np.memmap file for the whole data set and then randomly sample from this very large single array but can you say a bit more about how this would work with saving each time series as a memmap?</p>
<p>Cheers.</p>

JuanFMontesinos: <p>Hi,<br>
It varies a lil depending on some variables.<br>
The main idea is that mmap allow you to set an instance of a numpy array without reading its content.<br>
The content is read once you slice the array. This is obviously super efficient to read data from large arrays/tensors.</p>
<p>There are some variations you can do depending on two things, how many samples you have and how large they are.</p>
<p>The most efficient one is to create a large array by stacking all your samples (you may need to pad if they are different length).<br>
Then you instantiate this mmap in the init function and slice it (read it) in the getitem.</p>
<p>If you have hundreds of files you can save them independently, instantiate the mmap in the init and read them in the getittem.<br>
If you ahve millions of files, you can save them independently but you would probably need to instantiate the mmap and read in getitem.</p>
<p>This is really up to you.</p>
<p>All you need to do is to make an algorithm which predifines which samples and segments you are going to read, ensuring you use your data as much as possible.<br>
From this you can infer the length of the dataset and code the getitem.</p>

HenryPowell: <p>Thanks a lot for your suggestion. I actually managed to solve the issue by creating a list of tuples of (data_file_path, desired_index, class_label) and then just returned tuple((torch.load(data_file)[desired_index: desired_index + window_size], label)) and used this to generate batches.</p>
<p>For anyone interested this is the class:</p>
<pre><code class="lang-auto">class TimeSeriesDataSet(data.Dataset):

    def __init__(self, tensor_dir, transform, window_size, stride):

        self.tensor_directory = tensor_dir
        self.transform = transform
        self.files = os.listdir(tensor_dir)
        self.window_size = window_size
        self.stride = stride
        self.data_tuples = []

        for f in self.files:
            file = os.path.join(tensor_dir, f)
            data, label = torch.load(file)

            # pad with zeros with tensor is not of right length
            if data.size(0) % self.window_size != 0:
                zeros = torch.zeros(abs(self.window_size - (data.size(0) % self.window_size)), data.size(1)).double()
                data = torch.cat((data, zeros), axis=0)

            idxs = [i for i in range(0, data.size(0) - self.window_size, self.stride)]

            if len(idxs) == 0:
                continue

            for j in idxs:
                data_tuple = (file, j, label)
                self.data_tuples.append(data_tuple)

        shuffle(self.data_tuples)

    def __len__(self):
        return len(self.data_tuples)

    def __getitem__(self, idx):

        if torch.is_tensor(idx):
            idx = idx.tolist()

        sample_tuple = self.data_tuples[idx]
        sample, _ = torch.load(sample_tuple[0])
        label = sample_tuple[2]
        sample = sample[sample_tuple[1]: sample_tuple[1] + self.window_size]

        if self.transform:
            sample = self.transform(sample)

        return {'sample': sample, 'label': label}
</code></pre>

JuanFMontesinos: <p>Well note that (it’s more or less what i was suggesting).<br>
If you use that with mmap you can just load the desired slice and it will work faster <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

HenryPowell: <p>Yes I didn’t mean to suggest that it was completely different from your suggestion. It was deinfitly very helpful. At the moment the speed doesn’t seem to be an issue but I’ll definitly look into using memmaps in the future.</p>