Cannot Load Data Lazily Using Dataloader

username: tired_grad_student
cooked: <p>Hello,</p>
<p>I have been trying to load data one line at a time from two excel files into a dataloader since my files are larger than the RAM I have available. One file contains the data and the other one contains the labels and both are excel files.</p>
<p>I tried following examples such as the ones below, but to no avail. Can anyone please point out what I am doing wrong? Both files are in the “current working data” root folder.</p>
<p>Examples:</p>
<p><a href="http://pyroomacoustics.readthedocs.io/en/pypi-release/pyroomacoustics.room.html" class="onebox" target="_blank" rel="noopener nofollow ugc">pyroomacoustics.readthedocs.io/en/pypi-release/pyroomacoustics.room.html</a></p>
<p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>
<p>I get the following error:</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 1146, in _try_get_data<br>
raise RuntimeError(f’DataLoader worker (pid(s) {pids_str}) exited unexpectedly’) from e</p>
<p>RuntimeError: DataLoader worker (pid(s) 113228, 108680, 109552, 119108, 117720, 116756, 114436, 68888) exited unexpectedly</p>
<p>My code is as follows:</p>
<p>class MyDataset(torch.utils.data.Dataset):</p>
<pre><code>def __init__(self):
    self.data_files = os.listdir('current working data')
 

def __getindex__(self, idx):
   
    return torch.load(self.data_files[idx])

def __len__(self):
    return len(self.data_files)
</code></pre>
<p>training_data = MyDataset()<br>
train_loader = DataLoader(training_data, batch_size = batch_size, shuffle = False, num_workers=8)</p>
<p>Any help is greatly appreciated.</p>
<p>Thank you very much.</p>

username: ptrblck
cooked: <p>Could you use <code>num_workers=0</code> and rerun your code to see if a better error message might be raised?</p>

username: tired_grad_student
cooked: <p>Hi Patrick. Thank you for your feedback. When I set the workers to zero and rerun the code, I get the following error:</p>
<p>NotImplementedError: Subclasses of Dataset should implement <strong>getitem</strong>.</p>

username: ptrblck
cooked: <p>Ah great! Replace the <code>__getindex__</code> function name with <code>__getitem__</code>.</p>

username: tired_grad_student
cooked: <p>Hi Patrick,</p>
<p>Thank you. I replaced “getindex” with “getitem” and now it says the following:</p>
<p>FileNotFoundError: [Errno 2] No such file or directory: ‘train_data.xlsx’</p>
<p>The issue appears to stem from the line in “getitem” that tries to load the data. The odd thing is that this file does indeed exist and is titled as shown above. Not quite sure what to do from here. Thank you again for all your help! I appreciate it very much!</p>
<p>The full traceback is this:</p>
<p>Traceback (most recent call last):</p>
<p>File “C:\Users\johnt\Documents\SDSU\Thesis\Room Acoustics\PyRoom Acoustics\CNN and Data Generator\CNN Model\CNN_Model_3_NGrid_2592_Lazy_data_loader.py”, line 190, in <br>
MUSIC_data, labels = next(dataiter)</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 631, in <strong>next</strong><br>
data = self._next_data()</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 675, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py”, line 51, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py”, line 51, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]</p>
<p>File “C:\Users\johnt\Documents\SDSU\Thesis\Room Acoustics\PyRoom Acoustics\CNN and Data Generator\CNN Model\CNN_Model_3_NGrid_2592_Lazy_data_loader.py”, line 171, in <strong>getitem</strong><br>
return torch.load(self.data_files[idx]) # orignal line</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\serialization.py”, line 998, in load<br>
with _open_file_like(f, ‘rb’) as opened_file:</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\serialization.py”, line 445, in _open_file_like<br>
return _open_file(name_or_buffer, mode)</p>
<p>File “C:\Users\johnt\anaconda3\lib\site-packages\torch\serialization.py”, line 426, in <strong>init</strong><br>
super().<strong>init</strong>(open(name, mode))</p>
<p>FileNotFoundError: [Errno 2] No such file or directory: ‘train_data.xlsx’</p>

username: ptrblck
cooked: <p>Depending on the passed path to <code>os.listdir</code> you might need to recreate the full path again while trying to loading the data.<br>
E.g. if you’ve used <code>self.data_files = os.listdir("/home/myusername/data")</code>, you would need to load the data later using the same path: <code>torch.load("/home/myusername/data/" + self.data_files[idx])</code>.</p>
<p>With that being said, it seems you are trying to load a Microsoft Excel spreadsheet, which will fail as PyTorch won’t be able to open it.</p>