Dataloader with zipfile failed

username: wgting
cooked: <p>I am trying to load data from a zip file by  <strong>Python zipfile</strong> library. However, it seems that it cannot compatible with the <strong>torch’s Dataloader</strong> class.</p>
<pre><code class="lang-auto">import numpy as np
import cv2
import io
from torch.utils.data import DataLoader, Dataset
from torchvision.transforms import ToTensor
import zipfile

class ZipDataset(Dataset):
    def __init__(self, root_path, cache_into_memory=False):
        if cache_into_memory:
            f = open(root_path, 'rb')
            self.zip_content = f.read()
            f.close()
            self.zip_file = zipfile.ZipFile(io.BytesIO(self.zip_content), 'r')
        else:
            self.zip_file = zipfile.ZipFile(root_path, 'r')
        self.name_list = list(filter(lambda x: x[-4:] == '.jpg', self.zip_file.namelist()))
        self.to_tensor = ToTensor()

    def __getitem__(self, key):
        buf = self.zip_file.read(name=self.name_list[key])
        img = self.to_tensor(cv2.imdecode(np.fromstring(buf, dtype=np.uint8), cv2.IMREAD_COLOR))
        return img

    def __len__(self):
        return len(self.name_list)

if __name__ == '__main__':
    dataset = ZipDataset('COCO.zip', cache_into_memory=False)
    dataloader = DataLoader(dataset, batch_size=2, num_workers=2)
    for batch_idx, sample in enumerate(dataloader):
        print(batch_idx, sample.size())
</code></pre>
<p>When <strong>num_workers=0</strong> or <strong>num_workers=1</strong>, everything works well. But if the <strong>num_workers</strong> is larger than 1, the program will raise a strange error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test_zip_file.py", line 31, in &lt;module&gt;
    for batch_idx, sample in enumerate(dataloader):
  File "/home/admin/anaconda3/envs/pytorch1_0/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 637, in __next__
    return self._process_next_batch(batch)
  File "/home/admin/anaconda3/envs/pytorch1_0/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
zipfile.BadZipFile: Traceback (most recent call last):
  File "/home/admin/anaconda3/envs/pytorch1_0/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/admin/anaconda3/envs/pytorch1_0/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in &lt;listcomp&gt;
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "test_zip_file.py", line 21, in __getitem__
    buf = self.zip_file.read(name=self.name_list[key])
  File "/home/admin/anaconda3/envs/pytorch1_0/lib/python3.6/zipfile.py", line 1337, in read
    with self.open(name, "r", pwd) as fp:
  File "/home/admin/anaconda3/envs/pytorch1_0/lib/python3.6/zipfile.py", line 1419, in open
    % (zinfo.orig_filename, fname))
zipfile.BadZipFile: File name in directory '000000000009.1.jpg' and header b'\x00(\xa2\x8a\x00(\xa2\x8a\x00(\xa2\x8a\x00(\xa2\x8a\x00(\xa2\x8a\x00(\xa2\x8a\x00(\xa2\x8a\x00(\... '  differ.
</code></pre>
<p>It looks like that zipfile cannot be read in the multiprocessing manner. But interestingly, if the we set <strong>cache_into_memory=True</strong> (which means that the total zip file will be read into memory), the program will work fine.</p>
<p>This code has been tested in <em>Windows 10 / Ubuntu 16.04</em>, <em>torch 0.41 &amp; 1.0.0</em>. All of them have the same results.</p>

username: mathematics
cooked: <p>similar like error is happening to me when using dataloader for Celeba but not for mnist.</p>
<p>Error is</p>
<pre><code class="lang-auto">-----------------------------------------------------------------------
BadZipFile                                Traceback (most recent call last)
&lt;ipython-input-21-51d4dd0d2430&gt; in &lt;module&gt;()
      4                   transform=transforms.Compose([
      5       transforms.ToTensor(),
----&gt; 6       transforms.Normalize((1, -1, 0.5), (0.5, -0.5, 1))
      7       ])
      8                   )

3 frames
/usr/lib/python3.6/zipfile.py in _RealGetContents(self)
   1196             raise BadZipFile("File is not a zip file")
   1197         if not endrec:
-&gt; 1198             raise BadZipFile("File is not a zip file")
   1199         if self.debug &gt; 1:
   1200             print(endrec)

BadZipFile: File is not a zip file
</code></pre>
<p>where only i was just downloading through <code>DataLoader</code><br>
as</p>
<p><code>dataset = torch.utils.data.DataLoader(     datasets.CelebA('data/',                     download=True,                    transform=transforms.Compose([       transforms.ToTensor(),       transforms.Normalize((1, -1, 0.5), (0.5, -0.5, 1))       ])                   ) )</code><br>
I have torch version of <strong>‘1.5.0+cu101’</strong></p>
<p>Even for ImageNet it is now not publicly accessible ,why</p>
<blockquote>
<p>RuntimeError: The dataset is no longer publicly accessible. You need to download the archives externally and place them in the root directory.</p>
</blockquote>
<p>Appreciate for helping <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>You could try to download the files again to a different folder to check, if maybe the download failed.</p>
<p>The authors of the ImageNet dataset removed a publicly available download link, so that you would have to register again in order to download it (or use another mirror you can find).</p>

username: mathematics
cooked: <p>oh , Apologies for the slow reply, Downloading in different folder does work.</p>

username: haowxu
cooked: <p>I’d like to know if someone knows the solution of the original question. I am facing a similar question now. My data loader using <code>zipfile</code> fails when the <code>num_workers</code> is set greater than 1, but the exception I got is CRC-32 error. I guess this is related to the multi-threading of reading from a single <code>ZipFile</code> instance.</p>
<p>Thanks!</p>

username: superbock
cooked: <p>I had a similar issue with <code>zipfile</code> giving random errors. It seems to be related to having the handle opened and saved during instantiation – which does not work with multiprocessing. The solution was to open the handle in <code>__getitem__()</code> and cache it.</p>
<p>HTH</p>

username: Rohit_Choudhary
cooked: <p>Yes. I had the same issue. Everything works fine while reading using single thread, but using dataloader with multi-thread raises exception <code>zipfile.BadZipFile: Bad CRC-32 for file 'train2017/000000431472.jpg'</code></p>
<p>train2017 is the COCO dataset.</p>

username: Rohit_Choudhary
cooked: <p>Isn’t it the case that caching will need more RAM ? I have 18GB data, but colab have only 12GB RAM. Did some have some solution for this case ?</p>
<p>zipfile working fine on single thread, but with dataloader and multi-threading throws error.</p>

username: Rohit_Choudhary
cooked: <p>This is not working for me.</p>

username: haowxu
cooked: <p>So I just guess <code>zipfile</code> does not support multi-threading. Need to find a workaround.</p>

username: aaaa
cooked: <p>I had the same problem. Have you solved it yet</p>

username: usryokousha
cooked: <p>You should be able to fix the num_workers issue by getting around the python GIL (global interpreter lock).  This is done via a lock as shown below.<br>
python```<br>
from threading import Lock<br>
from torch.utils.data import Dataset<br>
import zipfile<br>
import io<br>
import PIL.Image as Image</p>
<p>class ImageDataset(Dataset):<br>
def <strong>init</strong>(self, zip_path, transform):<br>
zipobj= zipfile.ZipFile(zip_path, ‘r’)<br>
self.nameslist = zipobj.nameslist()<br>
self.lock = Lock()</p>
<p>def <strong>len</strong>(self):<br>
return len(self.nameslist)</p>
<p>def <strong>getitem</strong>(self, idx):<br>
with self.lock:<br>
img = Image.open(io.BytesIO(self.zipobj.read(self.nameslist[idx])))<br>
return img</p>
<pre><code class="lang-auto"></code></pre>