Dataloader Indexing

username: Zhuangzhuang_Yu
cooked: <p>Hi everyone! I am trying to use a sliding window fashion to training my neural network. The sliding window slides inside one sequence, and multiple sequences are present in the dataset. When designing my dataset class, I use the np.vstack to concatenate different sequence together so as to have a sliding window to go through all data.</p>
<pre><code class="lang-auto">    def load_data(self, files):
        lookuptable = np.array([])
        for fp_inlist in files:
            h5_file = h5py.File(fp_inlist, 'r')
            print('Start iterations among files')

            # Walk through all groups, extracting datasets
            for gname, group in h5_file.items():
                # Count how many datasets inside one group
                # num_datasets = len(group)
                print('Start iterations among groups')

                for dname, ds in group.items(): 
                    print('Start iterations among datasets')
                    signal = np.array(ds[0:60000, :])
                    # Mapping data into image scale(0-255)
                    data = ((signal)/np.max(signal)*255).astype(np.uint8)
#                     figure = plt.figure()
#                     plt.imshow(data.squeeze(), cmap="gray")
#                     plt.show()
                    print(data.shape)
                    if lookuptable.size == 0:
                        lookuptable = data
                    else:
                        lookuptable = np.vstack((lookuptable, data))
</code></pre>
<p>However, this can also provide a problem when the window slides to an end of one sequence, it takes the spliced window from two sequence. As it is illustrated in the image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0ad0f18e6269ffca70ca2c56b1e34acc19f6918b.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ad0f18e6269ffca70ca2c56b1e34acc19f6918b" title="Picture1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0ad0f18e6269ffca70ca2c56b1e34acc19f6918b.png" alt="Picture1" data-base62-sha1="1xGqVhKxTi6k67eXUGNqOjTQUjx" width="690" height="245" data-dominant-color="8B6F74"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Picture1</span><span class="informations">2021×720 6.49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">    def __getitem__(self, idx):   # Each time this func implemented by dataloader, rest lengths should be checked

        # x = self.data[index:index+self.window]
        # return x
        # lookuptable = self.load_data(self.files)
#         for idx in range(1, self.length - self.window):        

#       Mapping to [0,1] range and convert from numpy to torch.tensor
        data = torch.from_numpy(self.lookuptable[10*idx : self.window+(10*idx), :]/255)
        data = data.unsqueeze_(0)
        if self.transform:
            data = self.transform(data.unsqueeze_(0))
        
        return data, label
</code></pre>
<p>The stride of the sliding is 10, goes through the whole concatenated sequence.<br>
My question is, is there anyway to have the index picked by the dataloader can skip these spliced segments such that it only slides inside one sequence and jump to the next(without preparing data file of all sliding windows, which is really tedious and consuming lot of space).</p>
<p>Thanks in advance</p>