Error that I haven't understand and solve

PytorchLearning: <p>Hi all,</p>
<p>I get this error when I have write:</p>
<pre><code> batch_size = 3
 train_loader = DataLoader(dataset     = train_dataset
                     ,batch_size  = batch_size
                     ,shuffle     = False
                     ,num_workers = 2)
 for original_img, noisy_img in train_loader: 
            originalImage = original_img
</code></pre>
<p>Can someone please explain to me what that error means?<br>
The following error is:</p>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 for original_img, noisy_img in train_loader:<br>
2         originalImage = original_img<br>
…<br>
…</p>
<p>RuntimeError: DataLoader worker (pid(s) 24484, 1632) exited unexpectedly</p>

JuanFMontesinos: <p>Try to debug setting num_workers=0. It just means that you had an exception in some dataloader but you don’t know which one</p>

PytorchLearning: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>First thanks a lot for your answer.</p>
<p>I have tried  num_workers=0 but it gave me another error. I think I have an error in my code but I don’t known where???</p>
<p>Here is the complete code:</p>
<pre><code> import os
 import glob

 import torch
 import torch.cuda as cuda
 import torch.nn as nn
 from torch.nn import functional as F
 from torch.utils import data

 import matplotlib.pyplot as plt
 import matplotlib.image as matpltimg

 import torchvision
 from torchvision import datasets
 from torchvision import transforms

 import numpy as np

 from torch.cuda import amp

 datadir  = '/DataSets/SimplifyData'

  traindir_OI = datadir + '/imagesForTrain/OriginalImages/'
  traindir_NI = datadir + '/imagesForTrain/NoiseImages/'

  list1 = os.listdir(traindir_OI) 
  number_files1 = len(list1)
  print(number_files1)

  list2 = os.listdir(traindir_NI) 
  number_files2 = len(list2)
  print(number_files2)

 class Dataset(data.Dataset):
       def __init__(self, path_input_1, path_input_2):
                self.filenames_OI = []
                self.filenames_NI = []
                self.path_input_1 = path_input_1
                self.path_input_2 = path_input_2

                filenames_OI = glob.glob(osp.join(path_input_1, '*.jpg'))
                filenames_NI = glob.glob(osp.join(path_input_2, '*.jpg'))
                print(len(filenames_OI)) 
                print(len(filenames_NI))
    
                for fn_OI, fn_NI in zip(filenames_OI, filenames_NI):
                           self.filenames_OI.append(fn_OI)
                           self.filenames_NI.append(fn_NI)

    def __len__(self):
           return len(self.path_input_1) 

    def __getitem__(self, index):
          x = Image.open(self.filenames_OI[index])
          y = Image.open(self.filenames_NI[index])
         return X, y

train_dataset = Dataset(path_input_1 = traindir_OI
                                    ,path_input_2 = traindir_NI)
batch_size = 3

train_loader = DataLoader(dataset     = train_dataset
                     ,batch_size  = batch_size
                     ,shuffle     = False
                     ,num_workers = 0)

for original_img, noisy_img in train_loader:           
        originalImage = original_img
        noisyImage = noisy_img
</code></pre>
<p>When I write num_workers = 1 I get this error:</p>
<hr>
<p>Empty                                     Traceback (most recent call last)<br>
~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in _try_get_data(self, timeout)<br>
769         try:<br>
–&gt; 770             data = self._data_queue.get(timeout=timeout)<br>
771             return (True, data)</p>
<p>~.conda\envs\pytorch2\lib\multiprocessing\queues.py in get(self, block, timeout)<br>
104                     if not self._poll(timeout):<br>
–&gt; 105                         raise Empty<br>
106                 elif not self._poll():</p>
<p>Empty:</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
----&gt; 1 for original_img, noisy_img in train_loader:<br>
2         originalImage = original_img</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in <strong>next</strong>(self)<br>
352<br>
353     def <strong>next</strong>(self):<br>
–&gt; 354         data = self._next_data()<br>
355         self._num_yielded += 1<br>
356         if self._dataset_kind == _DatasetKind.Iterable and \</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)<br>
963<br>
964             assert not self._shutdown and self._tasks_outstanding &gt; 0<br>
–&gt; 965             idx, data = self._get_data()<br>
966             self._tasks_outstanding -= 1<br>
967</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in _get_data(self)<br>
930         else:<br>
931             while True:<br>
–&gt; 932                 success, data = self._try_get_data()<br>
933                 if success:<br>
934                     return data</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in _try_get_data(self, timeout)<br>
781             if len(failed_workers) &gt; 0:<br>
782                 pids_str = ', '.join(str(w.pid) for w in failed_workers)<br>
–&gt; 783                 raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str))<br>
784             if isinstance(e, queue.Empty):<br>
785                 return (False, None)</p>
<p>RuntimeError: DataLoader worker (pid(s) 16460) exited unexpectedly</p>
<p>and when I write num_workers = 0 I get this error:</p>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 for original_img, noisy_img in train_loader:<br>
2         originalImage = original_img</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in <strong>next</strong>(self)<br>
352<br>
353     def <strong>next</strong>(self):<br>
–&gt; 354         data = self._next_data()<br>
355         self._num_yielded += 1<br>
356         if self._dataset_kind == _DatasetKind.Iterable and \</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)<br>
392     def _next_data(self):<br>
393         index = self._next_index()  # may raise StopIteration<br>
–&gt; 394         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
395         if self._pin_memory:<br>
396             data = _utils.pin_memory.pin_memory(data)</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\fetch.py in fetch(self, possibly_batched_index)<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]<br>
—&gt; 47         return self.collate_fn(data)</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\collate.py in default_collate(batch)<br>
82             raise RuntimeError(‘each element in list of batch should be of equal size’)<br>
83         transposed = zip(*batch)<br>
—&gt; 84         return [default_collate(samples) for samples in transposed]<br>
85<br>
86     raise TypeError(default_collate_err_msg_format.format(elem_type))</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\collate.py in (.0)<br>
82             raise RuntimeError(‘each element in list of batch should be of equal size’)<br>
83         transposed = zip(*batch)<br>
—&gt; 84         return [default_collate(samples) for samples in transposed]<br>
85<br>
86     raise TypeError(default_collate_err_msg_format.format(elem_type))</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\collate.py in default_collate(batch)<br>
84         return [default_collate(samples) for samples in transposed]<br>
85<br>
—&gt; 86     raise TypeError(default_collate_err_msg_format.format(elem_type))</p>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.JpegImagePlugin.JpegImageFile’&gt;</p>
<p>Do you have any idea on how to solve this problem?</p>

JuanFMontesinos: <pre><code class="lang-auto">TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.JpegImagePlugin.JpegImageFile’&gt;
</code></pre>
<p>This roughly means that you are loading images but you are not converting them to pytorch tensors.<br>
I would tell you to use torchvision reader or imageio. Makes no sense to use PIL if you are not doing any preprocess</p>

PytorchLearning: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>Thanks a lot for your answer.</p>
<p>Can you please show me with an example or change directly in the code, just for learning and improve my very modest level.</p>

JuanFMontesinos: <aside class="quote no-group" data-username="PytorchLearning" data-post="3" data-topic="88676">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/d2c977/48.png" class="avatar"> PytorchLearning:</div>
<blockquote>
<pre><code class="lang-auto">    def __getitem__(self, index):
          x = Image.open(self.filenames_OI[index])
          y = Image.open(self.filenames_NI[index])
         return X, y
</code></pre>
</blockquote>
</aside>
<p>Jus replace the Image.open by any other like</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
          x = imageio.imread(self.filenames_OI[index])
          y = imageio.imread(self.filenames_NI[index])
         return X, y
</code></pre>
<p>There are many libraries to open images.</p>

PytorchLearning: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>Thanks a lot for this example.</p>
<p>I have change with imageio.imread() but I get the same error:</p>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 for original_img, noisy_img in train_loader:<br>
2         originalImage = original_img</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in <strong>next</strong>(self)<br>
352<br>
353     def <strong>next</strong>(self):<br>
–&gt; 354         data = self._next_data()<br>
355         self._num_yielded += 1<br>
356         if self._dataset_kind == _DatasetKind.Iterable and \</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)<br>
392     def _next_data(self):<br>
393         index = self._next_index()  # may raise StopIteration<br>
–&gt; 394         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
395         if self._pin_memory:<br>
396             data = _utils.pin_memory.pin_memory(data)</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\fetch.py in fetch(self, possibly_batched_index)<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]<br>
—&gt; 47         return self.collate_fn(data)</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\collate.py in default_collate(batch)<br>
82             raise RuntimeError(‘each element in list of batch should be of equal size’)<br>
83         transposed = zip(*batch)<br>
—&gt; 84         return [default_collate(samples) for samples in transposed]<br>
85<br>
86     raise TypeError(default_collate_err_msg_format.format(elem_type))</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\collate.py in (.0)<br>
82             raise RuntimeError(‘each element in list of batch should be of equal size’)<br>
83         transposed = zip(*batch)<br>
—&gt; 84         return [default_collate(samples) for samples in transposed]<br>
85<br>
86     raise TypeError(default_collate_err_msg_format.format(elem_type))</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\collate.py in default_collate(batch)<br>
84         return [default_collate(samples) for samples in transposed]<br>
85<br>
—&gt; 86     raise TypeError(default_collate_err_msg_format.format(elem_type))</p>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘imageio.core.util.Array’&gt;</p>

JuanFMontesinos: <p>hmm soz you still have to convert that into tensors</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
          x = imageio.imread(self.filenames_OI[index])
          y = imageio.imread(self.filenames_NI[index])
         return torch.from_numpy(X), torch.from_numpy(y)
</code></pre>

PytorchLearning: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>Thanks for your help. I had introduced this conversion; it remove the error but I get another error, which is :</p>
<hr>
<p>IndexError                                Traceback (most recent call last)<br>
 in <br>
----&gt; 1 for original_img, noisy_img in train_loader:<br>
2         originalImage = original_img<br>
3</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in <strong>next</strong>(self)<br>
352<br>
353     def <strong>next</strong>(self):<br>
–&gt; 354         data = self._next_data()<br>
355         self._num_yielded += 1<br>
356         if self._dataset_kind == _DatasetKind.Iterable and \</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)<br>
392     def _next_data(self):<br>
393         index = self._next_index()  # may raise StopIteration<br>
–&gt; 394         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
395         if self._pin_memory:<br>
396             data = _utils.pin_memory.pin_memory(data)</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\fetch.py in fetch(self, possibly_batched_index)<br>
42     def fetch(self, possibly_batched_index):<br>
43         if self.auto_collation:<br>
—&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]</p>
<p>~.conda\envs\pytorch2\lib\site-packages\torch\utils\data_utils\fetch.py in (.0)<br>
42     def fetch(self, possibly_batched_index):<br>
43         if self.auto_collation:<br>
—&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]</p>
<p> in <strong>getitem</strong>(self, index)<br>
24         <span class="hashtag">#noisyIMG</span> = Image.open(self.filenames_NI[index])<br>
25<br>
—&gt; 26         originalIMG = imageio.imread(self.filenames_OI[index])<br>
27         noisyIMG = imageio.imread(self.filenames_NI[index])<br>
28</p>
<p>IndexError: list index out of range</p>

JuanFMontesinos: <p>that means that the len of dataset is larger than the real one.</p>
<pre><code class="lang-auto">return len(self.path_input_1)
</code></pre>

PytorchLearning: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>Thanks for your help, I have done a print to check, as follow:</p>
<p>‘’’<br>
class Dataset(data.Dataset):<br>
def <strong>init</strong>(self, path_input_1, path_input_2):</p>
<pre><code>    self.filenames_OI = []
    self.filenames_NI = []
    
    self.path_input_1 = path_input_1
    self.path_input_2 = path_input_2        
    print(len(self.path_input_1))
    
    filenames_OI = glob.glob(osp.join(self.path_input_1, '*.jpg'))
    filenames_NI = glob.glob(osp.join(self.path_input_2, '*.jpg'))
    print(len(filenames_OI)) 
    print(len(filenames_NI))
    
    for fn_OI, fn_NI in zip(filenames_OI, filenames_NI):
        self.filenames_OI.append(fn_OI)
        self.filenames_NI.append(fn_NI)

def __len__(self):
    return len(self.path_input_1) 

def __getitem__(self, index):
    #originalIMG = Image.open(self.filenames_OI[index])
    #noisyIMG = Image.open(self.filenames_NI[index])
    
    originalIMG = imageio.imread(self.filenames_OI[index])       
    noisyIMG = imageio.imread(self.filenames_NI[index])
    
    return torch.from_numpy(originalIMG), torch.from_numpy(noisyIMG)
</code></pre>
<p>‘’’</p>
<p>I get as result<br>
53<br>
6<br>
6</p>
<p>I’ve been trying to figure out where this 53 is coming from, and I found it calculates the length of the string (’/DataSets/SimplifyData’’/imagesForTrain/OriginalImages/’) of the code:</p>
<pre><code class="lang-auto"> datadir  = '/DataSets/SimplifyData'
  traindir_OI = datadir + '/imagesForTrain/OriginalImages/'
</code></pre>
<p>Do you known why and how to solve it.</p>

JuanFMontesinos: <p>It’s an inputs so there is no way I know what have you coded there <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
If it’s a string thats probably the length of the path (number of characters in the string) (Since the rest of the code is working)</p>

PytorchLearning: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>It’s the first time I’ve used this class, with:</p>
<pre><code>    def __init__
    def __len__(self):
    def __getitem__(self, index):
</code></pre>
<p>I have solved this problem by writing</p>
<pre><code>def __len__(self):
       list = os.listdir(self.path_input_1)
       return len(list) 
</code></pre>
<p>Now I get no error. Thanks a lot for your precious help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Amnesie: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="12" data-topic="88676">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>If it’s a string thats probably the length of the path (number of characters in the string) (Since the rest of the code is working)</p>
</blockquote>
</aside>
<p>This help me out of “time-out trouble”. Thanks a lot!!!<br>
I also used API from <code>Pil</code> to open the image but I transfered it to torch but the programm will still be crushed due to time-out.  After seeing your comment I tried to use <code>imageio</code> to read the images and it worked!</p>
<p>Can you explain a little what is causing this bug? It seems to me that it is just a different way of opening the image, nothing with computation. Maybe some issue with Pillow library?</p>

JuanFMontesinos: <p>Sorry this post is 2 years old, could you give me some context?</p>

pkb: <p>I got an error while loading Wine dataset using DataLoader. The problem I was facing was “Empty” like listed above. When I removed num_workers=2 parameter completely, it started working. Even when value was set to 1, it was still throwing the error.</p>