Custom dataset for hyperspectral data (.mat file)

Riko: <p>Hi! I am trying to make custom dataset for HSI (HxWxD) and groundtruth (HxW) data stored in .mat file.<br>
Which i would like to use in:<br>
train_loader = torch.utils.data.DataLoader()<br>
I found class like:<br>
from torch.utils.data.dataset import Dataset<br>
class MyCustomDataset(Dataset):<br>
def <strong>init</strong>(self, …):<br>
# stuff<br>
def <strong>getitem</strong>(self, index):<br>
# stuff<br>
return (img, label)<br>
def <strong>len</strong>(self):<br>
return count<br>
To make further 3D CNN mdel for classification.<br>
Could you give me some help or examples for this problem of mine?<br>
Or is this even necessary for making custom data class for cnn?<br>
Thank you in advance!</p>

ptrblck: <p>You could pass the paths to your <code>Dataset</code>'s <code>__init__</code> function and lazily load each sample in <code>__getitem__</code>.<br>
Since your data is stored in <code>.mat</code> files, you could try to use <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html" rel="nofollow noopener">scipy.io.loadmat</a> to load it.</p>

Riko: <p>Ty for reply. I came up with something like this:</p>
<pre><code> 
from torch.utils.data.dataset import Dataset
import scipy.io as io
 class MyDataset(Dataset):
   def __init__(self, mat_path, gt_path):      
       data = io.loadmat(mat_path)['pavia']
       data = numpy.uint8(data)
       self.images = torch.from_numpy(data)
       data_gt = io.loadmat(gt_path)
       self.target = torch.from_numpy(data_gt['pavia_gt'])            
 
  def __getitem__(self, index):
       x = self.images[index]
       y = self.target[index]
       return x, y
     
   def __len__(self):
       return len(self.images)

mat_path = ('./data/Pavia.mat')
gt_path = ('./data/Pavia_gt.mat')
custom_dataset = MyDataset(mat_path, gt_path)
train_loader = torch.utils.data.DataLoader(dataset=custom_dataset,
                                           batch_size=64, 
                                           shuffle=True)
</code></pre>
<p>Now i have no idea how to use this in <code>torch.utils.data</code><br>
Or should i make a Train/Test Set split first?</p>

ptrblck: <p>You could perform the splitting beforehand.<br>
Based on your code, it seems that using a <code>torch.utils.data.Subset</code> would be easier, as you would only have to provide the corresponding indices.</p>
<aside class="quote no-group" data-username="Riko" data-post="3" data-topic="48173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/riko/48/13179_2.png" class="avatar"> Riko:</div>
<blockquote>
<p>Now i have no idea how to use this in <code>torch.utils.data</code></p>
</blockquote>
</aside>
<p>You can just iterate the <code>DataLoader</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">for data, target in train_loader:
    data = data.to(device)
    target =  target.to(device)
    optimizer.zero_grad()
    output = model(data)
    ...
</code></pre>
<p>It will automatically create the batches, shuffle your <code>Dataset</code>, and use multiprocessing to load the data (if <code>num_workers &gt; 0</code>).</p>