Questions about Dataloader and Dataset

chenyuntc: <p>I run my dataloader like this:</p>
<pre><code class="lang-python">dataset= ImageFolder('/home/x/data/pre/train/',
                      transform=transforms.Compose([transforms.Scale(opt.image_size),
                                              transforms.RandomCrop(opt.image_size) ,
                                              transforms.ToTensor(),
                                              transforms.Normalize([0.5]*3,[0.5]*3)
                                             ]))
 
dataloader=t.utils.data.DataLoader(dataset,opt.batch_size,True,num_workers=opt.workers)
</code></pre>
<p>but some images are corrupted, and it raised an error(an PIL error in <code>default_loader: Image.open(path).convert('RGB')</code></p>
<p>my temporary fix is modifying <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L65" rel="nofollow noopener">/torchvision/datasets/folder.py Line 65</a> to</p>
<pre><code class="lang-python">        try:
            img = self.loader(os.path.join(self.root, path))
        except Exception as e:
            index = index - 1 if index &gt; 0 else index + 1 
            return self.__getitem__(index)
</code></pre>
<p>Is there a better way to solve this, i.e. modifying the code of dataloader to load another Image when exception or  write a new loader.<br>
or does it work if I simply return None when caught an exception?</p>

smth: <p>you can look at the default_collate function if it handles <code>None</code> or not.<br>
Or you can give a custom collate function to your <code>DataLoader</code> which will handle <code>None</code>:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L32" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L32" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L32</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="22" style="counter-reset: li-counter 21 ;">
<li>_use_shared_memory = False</li>
<li>"""Whether to use shared memory in default_collate"""</li>
<li>
</li>
<li>
</li>
<li>class ExceptionWrapper(object):</li>
<li>"Wraps an exception plus traceback to communicate across threads"</li>
<li>
</li>
<li>def __init__(self, exc_info):</li>
<li>    self.exc_type = exc_info[0]</li>
<li>    self.exc_msg = "".join(traceback.format_exception(*exc_info))</li>
<li class="selected">
</li>
<li>
</li>
<li>def _worker_loop(dataset, index_queue, data_queue, collate_fn, seed, init_fn, worker_id):</li>
<li>global _use_shared_memory</li>
<li>_use_shared_memory = True</li>
<li>
</li>
<li># Intialize C side signal handlers for SIGBUS and SIGSEGV. Python signal</li>
<li># module's handlers are executed after Python returns from C low-level</li>
<li># handlers, likely when the same fatal signal happened again already.</li>
<li># https://docs.python.org/3/library/signal.html Sec. 18.8.1.1</li>
<li>_set_worker_signal_handlers()</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L61-L84" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L61-L84" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L61-L84</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="61" style="counter-reset: li-counter 60 ;">
<li>
</li>
<li>
</li>
<li>def _worker_manager_loop(in_queue, out_queue, done_event, pin_memory, device_id):</li>
<li>if pin_memory:</li>
<li>    torch.cuda.set_device(device_id)</li>
<li>
</li>
<li>while True:</li>
<li>    try:</li>
<li>        r = in_queue.get()</li>
<li>    except Exception:</li>
<li>        if done_event.is_set():</li>
<li>            return</li>
<li>        raise</li>
<li>    if r is None:</li>
<li>        break</li>
<li>    if isinstance(r[1], ExceptionWrapper):</li>
<li>        out_queue.put(r)</li>
<li>        continue</li>
<li>    idx, batch = r</li>
<li>    try:</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L61-L84" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


apaszke: <p>The DataLoader doesn’t support recovery from errors in datasets because it would be too complex to add and keep the guarantee that it will <strong>always return the batches in the exact same order</strong>, as the sampler generated. It’s not even clear what’s the behaviour that user expects (some people might want to be notified about the error, others don’t).</p>

chenyuntc: <p>thanks <a class="mention" href="/u/smth">@smth</a>  <a class="mention" href="/u/apaszke">@apaszke</a>, that really makes me have deeper comprehension of dataloader.</p>
<p>At first I try:</p>
<pre><code class="lang-python">def my_loader(path):
    try: 
        return Image.open(path).convert('RGB')
    except Exception as e:
        print e
def my_collate(batch):
    "Puts each data field into a tensor with outer dimension batch size"
    batch = filter (lambda x:x is not None, batch)
    return default_collate(batch)

dataset= ImageFolder('/home/x/train/',
            transform=transforms.Compose([transforms.ToTensor()]),
            loader = my_loader)
dataloader=t.utils.data.DataLoader(dataset,4,True,collate_fn=my_collate)
</code></pre>
<p>it raise exception, because <code>transforms</code> in <code>dataset</code> can’t handle None</p>
<p>so then I try this：</p>
<pre><code class="lang-auto">def my_collate(batch):
    batch = filter (lambda x:x is not None, batch)
    return default_collate(batch)
class MyImageFolder(ImageFolder):
    __init__ = ImageFolder.__init__
    def __getitem__(self, index):
        try: 
            return super(MyImageFolder, self).__getitem__(index)
        except Exception as e:
            print e

dataset= MyImageFolder('/home/x/train/', transform = transforms.Compose([transforms.ToTensor()..]) )
dataloader=t.utils.data.DataLoader(dataset, 4, True, collate_fn=my_collate)
</code></pre>
<p>not so pythonic, but it works.<br>
and I think the best way maybe just cleaning the data.</p>

PermanentPon: <p>To solve this particular problem with corrupted images you can just add two lines before your code:</p>
<pre><code class="lang-auto">from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True
</code></pre>
<p>as suggested <a href="https://stackoverflow.com/a/23575424/1719231" rel="nofollow noopener">here</a></p>

Sun_ShiJie: <p>Hi, Chen. Does the batch size decrease when use this way to filter invalid data.</p>

msamogh: <p>I wrote a PyTorch library that allows you to do exactly this - drop samples, handle bad items, use Transforms as Filters (and more). <strong>Check out <a href="https://github.com/msamogh/nonechucks" rel="nofollow noopener">nonechucks</a>.</strong></p>
<p>Using nonechucks, your code would look something like this:</p>
<pre><code class="lang-python">dataset= ImageFolder('/home/x/data/pre/train/',
                      transform=transforms.Compose([transforms.Scale(opt.image_size),
                                              transforms.RandomCrop(opt.image_size) ,
                                              transforms.ToTensor(),
                                              transforms.Normalize([0.5]*3,[0.5]*3)
                                             ]))

import nonechucks as nc
dataset = nc.SafeDataset(dataset)
dataloader = nc.SafeDataLoader(dataset,opt.batch_size,True,num_workers=opt.workers)

# You can now use `dataloader` as though it was a regular DataLoader without
# having to worry about the bad samples!
</code></pre>
<p>Feel free to check out the documentation on the <a href="https://github.com/msamogh/nonechucks" rel="nofollow noopener">Github page</a>!</p>

nimning: <aside class="quote no-group" data-username="chenyuntc" data-post="4" data-topic="806">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/48/299_2.png" class="avatar"> chenyuntc:</div>
<blockquote>
<p>default_collate</p>
</blockquote>
</aside>
<p>How you get the internal function ‘default_collate’? It is within the ‘dataloder.py’.</p>

tom: <p>It’s imported under that module, just not directly under <code>.data</code>:</p>
<pre><code class="lang-python">import torch
import torch.utils.data
print(torch.utils.data.dataloader.default_collate)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

zanaa: <p>Note: for python 3, replace</p>
<p><code>batch = filter(lambda x : x is not None, batch)</code></p>
<p>with</p>
<p><code>batch = list(filter(lambda x : x is not None, batch))</code></p>

Shivam_Chandhok: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/ae8628a2aab8a0db0a7ea5eb256957acf707e0a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae8628a2aab8a0db0a7ea5eb256957acf707e0a8" title="36%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae8628a2aab8a0db0a7ea5eb256957acf707e0a8_2_690x431.png" alt="36%20PM" data-base62-sha1="oTUvQl50BcN0PjCF6nN6M5B02AU" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae8628a2aab8a0db0a7ea5eb256957acf707e0a8_2_690x431.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae8628a2aab8a0db0a7ea5eb256957acf707e0a8_2_1035x646.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae8628a2aab8a0db0a7ea5eb256957acf707e0a8_2_1380x862.png 2x" data-dominant-color="F1F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">36%20PM</span><span class="informations">2880×1800 510 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/36f43ad452a09ae4ca32b0a2d3ae30c751ad7480.png" data-download-href="https://discuss.pytorch.org/uploads/default/36f43ad452a09ae4ca32b0a2d3ae30c751ad7480" title="46%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/36f43ad452a09ae4ca32b0a2d3ae30c751ad7480_2_690x431.png" alt="46%20PM" data-base62-sha1="7Q92GBHTX8DQIyPfacnNgGzQsWQ" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/36f43ad452a09ae4ca32b0a2d3ae30c751ad7480_2_690x431.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/36f43ad452a09ae4ca32b0a2d3ae30c751ad7480_2_1035x646.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/36f43ad452a09ae4ca32b0a2d3ae30c751ad7480_2_1380x862.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">46%20PM</span><span class="informations">2880×1800 539 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ead78cdaf9cad77c8566254e352f1ff79180fcc4.png" data-download-href="https://discuss.pytorch.org/uploads/default/ead78cdaf9cad77c8566254e352f1ff79180fcc4" title="54%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ead78cdaf9cad77c8566254e352f1ff79180fcc4_2_690x431.png" alt="54%20PM" data-base62-sha1="xvvxCzL8IIoFNJ330nnSAsAblcw" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ead78cdaf9cad77c8566254e352f1ff79180fcc4_2_690x431.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ead78cdaf9cad77c8566254e352f1ff79180fcc4_2_1035x646.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ead78cdaf9cad77c8566254e352f1ff79180fcc4_2_1380x862.png 2x" data-dominant-color="F3F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">54%20PM</span><span class="informations">2880×1800 645 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am getting an error:</p>
<p>samples = collate_fn([dataset[i] for i in batch_indices])<br>
TypeError: ‘DataLoader’ object does not support indexing</p>
<p>What am I doing wrong.</p>

weiwei: <p>hi, i have similar problem about share memory, could you help me to solve the problem ? thanks very much, the topic link is :</p><aside class="quote" data-post="1" data-topic="47342">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/f9ae1b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/can-we-disable-share-memory-usage-in-pytorch/47342">Can we disable share memory usage in pytorch?</a> 
  </div>
  <blockquote>
    since i am not able to adjust the share memory usage in the remote server, can we disable share memory usage in pytorch. the same experiment run with tensorflow without shm size problem, so i just want to find a solution for this problem.
  </blockquote>
</aside>


Martin36: <p>This does not work if the batch size is 1, since <code>default_collate</code> will be called with an empty list in that case. The same thing could happen if the batch size is larger and all the data samples in the batch size are corrupt (highly unlikely though).</p>
<p>Is there a way to get this to work when the batch size is only 1?</p>

Keyv_Krmn: <p>I faced the same issue. my hot fix was to return a datapoint drawn randomly from the dataset.<br>
so basically, catch the error and return a valid batch with a single datapoint.<br>
It can happen in other cases where the batch_size &gt; 1,but last batch has only 1 datapoint.</p>

xu555: <p>from torch.utils.data.dataloader import default_collate</p>

Syzygianinfern0: <p>Did you find any fix?</p>
<p>Edit: I found a fix <a href="https://stackoverflow.com/a/67583699/8878627" rel="noopener nofollow ugc">here</a>. It has many other relevant answers as well.</p>

Anirudh257: <p>I also found a fix inspired by <a href="https://stackoverflow.com/questions/57815001/pytorch-collate-fn-reject-sample-and-yield-another/69578320#69578320" rel="noopener nofollow ugc">here</a>.</p>
<pre><code class="lang-auto">def collate_fn_replace_corrupted(batch, dataset):
    if batch[0] is None:
        batch = [dataset[randint(0, len(dataset) - 1)]]
        return collate_fn_replace_corrupted(batch, dataset)

    return torch.utils.data.dataloader.default_collate(batch)

</code></pre>
<pre><code class="lang-auto">  collate_fn = functools.partial(collate_fn_replace_corrupted, dataset = dataset)
</code></pre>