How to load files multiple .mat files each having multiple sequences

teenaxta: <p>Hello</p>
<p>so i have multiple files in my directory that begin with either P or C. I am trying to train a RNN to predict values of C given a sequence of P<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/7992199fb5145d959e531ef2f85767f071c33b35.png" data-download-href="https://discuss.pytorch.org/uploads/default/7992199fb5145d959e531ef2f85767f071c33b35" title="Screenshot 2023-02-17 at 11.23.42 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/7992199fb5145d959e531ef2f85767f071c33b35_2_274x499.png" alt="Screenshot 2023-02-17 at 11.23.42 PM" data-base62-sha1="hlsN6bbhb4bJzIQi3MQ9zLryqMJ" width="274" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/7992199fb5145d959e531ef2f85767f071c33b35_2_274x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/7992199fb5145d959e531ef2f85767f071c33b35_2_411x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/9/7992199fb5145d959e531ef2f85767f071c33b35.png 2x" data-dominant-color="2A2C2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-02-17 at 11.23.42 PM</span><span class="informations">512×932 34.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now each file has a signal. I will break the signal into smaller part each with dimension (sequence length, 1) as there is only feature. Ideally my output dimension should be something like <em>(num_batches, batch_size, seq_length, features)</em>. However as i have multiple files, i get something like <em>(num_files,num_batches, batch_size, seq_length, features)</em></p>
<p>Here’s my code</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, PATH, seq_length):
        self.seq_length=seq_length
        self.c_paths=[]
        self.p_paths=[]
        for i in os.scandir(PATH):
                    name=i.name
                    if name.split('.')[-1] == 'mat':
                        file_name = name.split('.')[0]
                        if 'C' in file_name:
                            self.c_paths.append(i.path)
                        if 'P' in file_name:
                            self.p_paths.append(i.path)
                            
    def __getitem__(self, index):
        p_noise = sio.loadmat(self.p_paths[index])['P_noise']
        cm = sio.loadmat(self.c_paths[index])['Cm']

        inputs=[]
        outputs=[]
        start=0
        for j in range (len(p_noise) - self.seq_length):
            stop = start + self.seq_length
            
            input = p_noise[start:stop]
            output = cm[stop-1]
            start += 1
            inputs.append(input)
            outputs.append(output)

        inputs = torch.from_numpy(np.array(inputs).reshape((-1, self.seq_length,1)))

        outputs= torch.from_numpy(np.array(outputs).reshape((-1, 1)))

        self.x=inputs
        self.y=outputs

        return self.x, self.y
    
    def __len__(self):
        return len(self.c_paths)
</code></pre>
<pre><code class="lang-auto">PATH='Dataset'
dataset=MyDataset(PATH, seq_length=400)

dataloader = DataLoader(dataset=dataset, batch_size=2, shuffle=False)
datatiter=iter(dataloader)
data=datatiter.next()

x,y=data
x.shape, y.shape
</code></pre>
<p>Here’s the output:<br>
<code>(torch.Size([2, 99600, 400, 1]), torch.Size([2, 99600, 1]))</code></p>