Create a custom dataloader for 3-d segmented images in .mat

Sherlock_Wan: <p>Hi! I am a beginner in PyTorch. I have a folder containing thousands of 3-d segmented images (H x W x D x C) in .mat (Matlab files). I have searched for similar topics on PyTorch forum (e.g., <a href="https://discuss.pytorch.org/t/creating-a-custom-dataloader-to-load-mat-files/90548">this link</a>), but my dataloader remains not workable. Specifically, elapsed time is too long when I call ‘real_batch = next(iter(dataloader))’. My dataloader is written as below. Can anyone offer any idea? Your time is highly appreciated. Many thanks!</p>
<pre><code class="lang-auto">import torch
import torch.utils.data
import scipy.io as spio

# Create the dataset
class customDataset(torch.utils.data.Dataset):
    '''
    Custom dataset for .mat
    '''
    def __init__(self, image_folder):
        self.image_folder = os.path.abspath(image_folder)
        self.image_list = os.listdir(self.image_folder)
    def __getitem__(self, index):
        image_path = self.image_list[index]
        image = spio.loadmat(os.path.join(self.image_folder, image_path))['data']
        return image
    def __len__(self):
        return len(self.image_list)

dataroot = r"C:\dataset"
batch_size = 16
workers = 2
dataset = customDataset(os.path.join(dataroot, 'images'))
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                        shuffle=True, num_workers=workers)
real_batch = next(iter(dataloader))  # Wrong: this line does not work
</code></pre>

ptrblck: <p>The code looks generally alright.<br>
How long does it take to load a batch and where are you storing the data (HDD, SSD, M.2, etc.)?</p>

Sherlock_Wan: <aside class="quote no-group" data-username="Sherlock_Wan" data-post="1" data-topic="147007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sherlock_wan/48/45577_2.png" class="avatar"> Sherlock_Wan:</div>
<blockquote>
<p>ched for similar topics on PyTorch forum (e.g., <a href="https://discuss.pytorch.org/t/creating-a-custom-dataloader-to-load-mat-files/90548">this link</a>), but my dataloader remains not workable. Specifically, elapsed time is too long when I call ‘real_batch = next(iter(dataloader))’. My dataloader is written as below. Can anyone offer any id</p>
</blockquote>
</aside>
<p>Hi, ptrblck. Thanks a lot for your reply! I know you always offer helpful ideas.<br>
When calling ‘real_batch = next(iter(dataloader))’, I waited for about 10 min and then stopped the program <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20">. For comparison, I ran the PyTorch DCGAN tutorial, and the dataloader for loading the Celeba dataset (about 202K images) worked quite fast, say around 1 sec.<br>
In addition, the dataset is stored in my One-Drive folder.<br>
Thank you!</p>

ptrblck: <aside class="quote no-group" data-username="Sherlock_Wan" data-post="3" data-topic="147007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sherlock_wan/48/45577_2.png" class="avatar"> Sherlock_Wan:</div>
<blockquote>
<p>the dataset is stored in my One-Drive folder.</p>
</blockquote>
</aside>
<p>I assume you have a local copy of the data or are you lazily downloading each sample?<br>
In any case, how long does <code>spio.loadmat</code> take for a single sample and for all <code>16</code> samples?</p>

Sherlock_Wan: <p>Thanks for your kind help, ptrblck. Unfortunately, it does not change the situation after I move the dataset to  a local folder. For finding the bug, I did another test: I create a new dataset containing only 50 images and load data from this dataset. Surprisingly, an error pops up in a few seconds (as attached below). It seems that the dataloader cannot get data, but why? I have googled the error, but have not idea yet. Could you offer any idea for this error?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
Empty                                     Traceback (most recent call last)
File ~\anaconda3\envs\pythonProject\lib\site-packages\torch\utils\data\dataloader.py:761, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
    760 try:
--&gt; 761     data = self._data_queue.get(timeout=timeout)
    762     return (True, data)

File ~\anaconda3\envs\pythonProject\lib\multiprocessing\queues.py:108, in Queue.get(self, block, timeout)
    107     if not self._poll(timeout):
--&gt; 108         raise Empty
    109 elif not self._poll():

Empty: 

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
Input In [5], in &lt;cell line: 1&gt;()
----&gt; 1 real_batch = next(iter(dataloader))

File ~\anaconda3\envs\pythonProject\lib\site-packages\torch\utils\data\dataloader.py:345, in _BaseDataLoaderIter.__next__(self)
    344 def __next__(self):
--&gt; 345     data = self._next_data()
    346     self._num_yielded += 1
    347     if self._dataset_kind == _DatasetKind.Iterable and \
    348             self._IterableDataset_len_called is not None and \
    349             self._num_yielded &gt; self._IterableDataset_len_called:

File ~\anaconda3\envs\pythonProject\lib\site-packages\torch\utils\data\dataloader.py:841, in _MultiProcessingDataLoaderIter._next_data(self)
    838     return self._process_data(data)
    840 assert not self._shutdown and self._tasks_outstanding &gt; 0
--&gt; 841 idx, data = self._get_data()
    842 self._tasks_outstanding -= 1
    844 if self._dataset_kind == _DatasetKind.Iterable:
    845     # Check for _IterableDatasetStopIteration

File ~\anaconda3\envs\pythonProject\lib\site-packages\torch\utils\data\dataloader.py:808, in _MultiProcessingDataLoaderIter._get_data(self)
    804     # In this case, `self._data_queue` is a `queue.Queue`,. But we don't
    805     # need to call `.task_done()` because we don't use `.join()`.
    806 else:
    807     while True:
--&gt; 808         success, data = self._try_get_data()
    809         if success:
    810             return data

File ~\anaconda3\envs\pythonProject\lib\site-packages\torch\utils\data\dataloader.py:774, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
    772 if len(failed_workers) &gt; 0:
    773     pids_str = ', '.join(str(w.pid) for w in failed_workers)
--&gt; 774     raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
    775 if isinstance(e, queue.Empty):
    776     return (False, None)

RuntimeError: DataLoader worker (pid(s) 8268, 18776) exited unexpectedly
</code></pre>

ptrblck: <p>Could you set <code>num_workers=0</code> and rerun the test, which could hopefully give a better error message, please?</p>

Sherlock_Wan: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="147007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>the test, which could hopefully give a better er</p>
</blockquote>
</aside>
<p>ptrblck, your solution are really amazing! It works and the dataset is successfully loaded in a few seconds. Thank you so much! This weird error has been bothering me for days. Btw, could you tell me why ‘num_workers = 0’ can help?</p>

ptrblck: <p>I don’t know why multiple workers fail to load the data, but would guess that you are running into issues with multiprocessing in <code>loadmat</code> or HDF5. Based on the docs HDF5 seems to be used internally:</p>
<blockquote>
<p>You will need an HDF5 Python library to read MATLAB 7.3 format mat files.</p>
</blockquote>

Sherlock_Wan: <p>Got it. Thank you so much for your kind help! Have a nice day!</p>

pulok: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, I am new to PyTorch. Can you please have a look at the error? It would be really helpful if I could get some insights on how to solve this issue. Thanks in advance.</p>
<p>I have a custom .mat dataset. The data looks like this:</p>
<p>print(x_train.shape, x_train, y_train.shape, y_train)</p>
<p>((30508, 256),<br>
array([[ 0.42129787-0.16991965j,  0.36276836-0.26392504j,<br>
0.26979125-0.33810524j, …, -0.02828086-0.17962576j,<br>
-0.13922529-0.11955454j, -0.18388582+0.02993972j],<br>
[-0.16232005+0.22424035j, -0.01326727+0.28548271j,<br>
0.16299008+0.25424894j, …, -0.05188078+0.1124093j ,<br>
0.03477429+0.12717485j,  0.10394195+0.0970679j ],<br>
[-0.14510996-0.06126717j, -0.16250012+0.0387307j ,<br>
-0.09767821+0.1080583j , …,  0.3880517 -0.02151187j,<br>
0.28764753-0.23641774j,  0.09376167-0.34648794j],<br>
…,<br>
[-0.15492494-0.05131445j, -0.13771235+0.04141434j,<br>
-0.08409637+0.10275555j, …,  0.11677223+0.18459846j,<br>
0.20143423+0.08588961j,  0.19610319-0.09028822j],<br>
[-0.09103348-0.05533604j, -0.13293657+0.00201364j,<br>
-0.05136824+0.10219111j, …,  0.15189247+0.15919279j,<br>
0.16143821+0.11385279j,  0.16244496+0.00631943j],<br>
[-0.20020891-0.08525992j, -0.14457089+0.05950821j,<br>
-0.0999863 +0.18009997j, …,  0.05141494-0.10477183j,<br>
0.00139747-0.14153811j, -0.07764385-0.10034507j]]),<br>
(30508, 2048),<br>
array([[1.85357980e-03, 1.74808095e-03, 1.40078473e-03, …,<br>
4.68181052e-05, 4.92985346e-05, 5.30191787e-05],<br>
[2.31476919e-04, 2.12018706e-04, 1.54907589e-04, …,<br>
1.33759553e-04, 1.29119840e-04, 1.18369283e-04],<br>
[2.46307219e-04, 2.45744874e-04, 2.40683767e-04, …,<br>
1.82749537e-04, 1.44496678e-04, 1.15648177e-04],<br>
…,<br>
[1.50155607e-03, 1.23499299e-03, 6.21390513e-04, …,<br>
4.37290822e-05, 4.56725970e-05, 4.64284083e-05],<br>
[3.46533912e-04, 3.50528539e-04, 3.49529882e-04, …,<br>
7.40785739e-05, 3.70392870e-05, 5.15122217e-05],<br>
[7.22670712e-03, 6.78763140e-03, 5.49826145e-03, …,<br>
1.67431347e-04, 1.65770322e-04, 1.62448272e-04]]))<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a26871b618728ca64a726a9a12009dbebae5d871.png" data-download-href="https://discuss.pytorch.org/uploads/default/a26871b618728ca64a726a9a12009dbebae5d871" title="Screenshot from 2022-08-01 18-43-04"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a26871b618728ca64a726a9a12009dbebae5d871_2_640x500.png" alt="Screenshot from 2022-08-01 18-43-04" data-base62-sha1="naJ7ux95xNZ1I7p80M3DroLNSVz" width="640" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a26871b618728ca64a726a9a12009dbebae5d871_2_640x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a26871b618728ca64a726a9a12009dbebae5d871_2_960x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a26871b618728ca64a726a9a12009dbebae5d871.png 2x" data-dominant-color="F3F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-08-01 18-43-04</span><span class="informations">1073×837 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>In your <code>__len__</code> function you are returning <code>(self.m)</code> directly instead of its <code>len</code>.</p>