OSError: cannot identify image file <_io.BufferedReader name='Goa/train/Chapora_fort/fortress-tower-chapora-fort-background-600w-1452260633.jpg'>

username: kabilan
cooked: <p>OSError: cannot identify image file &lt;_io.BufferedReader name=‘Goa/train/Chapora_fort/fortress-tower-chapora-fort-background-600w-1452260633.jpg’&gt;</p>
<p>I build a custom dataset everything went good. While training I encountered an os error. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8abe0c93f6585fe56527b604f057a315603cdf94.png" data-download-href="https://discuss.pytorch.org/uploads/default/8abe0c93f6585fe56527b604f057a315603cdf94" title="Screenshot from 2020-01-08 06-34-47"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8abe0c93f6585fe56527b604f057a315603cdf94_2_690x387.png" alt="Screenshot from 2020-01-08 06-34-47" data-base62-sha1="jNn4iqJPSVINSb2S9EBRWU1PU56" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8abe0c93f6585fe56527b604f057a315603cdf94_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8abe0c93f6585fe56527b604f057a315603cdf94_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8abe0c93f6585fe56527b604f057a315603cdf94.png 2x" data-dominant-color="E6E6E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-01-08 06-34-47</span><span class="informations">1366×768 206 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you check the printed path and try to load this image manually in another script?<br>
Also, are you passing file paths or file pointers to <code>Image.open</code>?</p>

username: kabilan
cooked: <p>yeah I try to open the image it showed me error, other images works. there are many such images in the dataset<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/6413bfa7857805e7e6ead247c4a0590b5a2335f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/6413bfa7857805e7e6ead247c4a0590b5a2335f3" title="Screenshot from 2020-01-08 17-56-26"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6413bfa7857805e7e6ead247c4a0590b5a2335f3_2_690x387.png" alt="Screenshot from 2020-01-08 17-56-26" data-base62-sha1="ehk3CecBgWVoQf93AUaWTl15jUf" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6413bfa7857805e7e6ead247c4a0590b5a2335f3_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6413bfa7857805e7e6ead247c4a0590b5a2335f3_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/4/6413bfa7857805e7e6ead247c4a0590b5a2335f3.png 2x" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-01-08 17-56-26</span><span class="informations">1366×768 188 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Does the size of the file look correct or might it have been corrupted during e.g. the download?<br>
I’m not sure if there is a better workaround than to remove these files from your dataset.</p>

username: midhunharikumar
cooked: <p>You could consider skipping the file in the data loader and using a custom collate function to remove the missing values.</p>
<pre><code class="lang-auto">def my_collate(batch):
    "Puts each data field into a tensor with outer dimension batch size"
    batch = list(filter(lambda x:x is not None, batch))
    return default_collate(batch)
train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=args.batch_size, shuffle=True,
        num_workers=args.num_workers, pin_memory=True, sampler=train_sampler,collate_fn=my_collate)
</code></pre>
<p>If you have a lot of missing images this might cause your batch size to fluctuate too much. In which case preprocessing the images offline and training with images that are in-tact will resolve your issue.</p>

username: kabilan
cooked: <p>I found that PIL is not opening webp images. It asked me to delete those files. Is there any way to delete those files using python code</p>

username: amin_asadi
cooked: <p>Hello<br>
Have a nice time<br>
I have a problem similar to yours.<br>
Did you get the result?</p>