Dataloader stucks

username: kaiyuyue
cooked: <p>Hi, developers:</p>
<p>I have the large training dataset which is packed in a <code>zip</code> file.<br>
In <code>train.py</code>, I load it once and then pass it into dataloader, here is the code:</p>
<pre><code class="lang-python">import zipfile
# load zip dataset
zf = zipfile.ZipFile(zip_path)
# read the images of zip via dataloader
train_loader = torch.utils.data.DataLoader(
                   DataSet(zf, transform),
                   batch_size = args.batch_size,
                   shuffle = True,
                   num_workers = args.workers,
                   pin_memory=True))
</code></pre>
<p>If the <code>num_workers</code> is set into the num larger than 1, the dataloader will stuck into the procedure that read images in zip file, it never stop to read images. How to fix this? Thanks.</p>

username: rasbt
cooked: <p>When I understand correctly, this happens with regular use not e.g., as child processes upon e.g., quitting the run (was thinking that it might be potentially related to the discussion here <a href="https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775/14">PyTorch doesn’t free GPU’s memory of it gets aborted due to out-of-memory error</a> and a bug in Python’s multiprocessing)</p>
<p>How does your “data loading / training” loop look like?<br>
Maybe you could run sth like</p>
<pre><code class="lang-python">for epoch in range(num_epochs):
    for batch_idx, (features, targets) in enumerate(train_loader):
</code></pre>
<p>and print the <code>batch_idx</code> – I would be curious what numbers you get if num_workers &gt; 1, i.e. getting a condition batch_idx &gt; num_training_examples / batch_size</p>

username: SimonW
cooked: <p>Reading same file from multiple process can be troublesome. You should look into if <code>zipfile</code> is multiprocessing safe.</p>

username: kaiyuyue
cooked: <p>This is the reason. That <code>zipfile</code> does not support multiple process to load it.</p>

username: victorhcm
cooked: <p>I had the same issue when opening a <code>tarfile</code>. A quick fix is to open a <code>zipfile</code> handle once at the start of <code>__getitem__</code>.</p>
<pre><code class="lang-python">class MyDataSet(Dataset):
   def __init__(self, filename):
        self.zip_handle = None
        self.fname = filename

    def __getitem__(self, x):
        if self.zip_handle is None:
            self.zip_handle = zipfile.ZipFile(self.fname, 'r')
        # do stuff
</code></pre>

username: robotcator123
cooked: <p>Hi, how do you solve this problem? I found that when I using multi-thread to read the zipfile, it seems fine. But in I create a dataset class in you style and pass it to dataloader, it raise that:</p>
<p>File “/opt/conda/envs/pytorch-py3.6/lib/python3.6/zipfile.py”, line 925, in _read1<br>
data = self._decompressor.decompress(data, n)<br>
zlib.error: Error -3 while decompressing data: invalid distance too far back</p>

username: kaiyuyue
cooked: <p>I think this is independent to my original problem and highly related to the issue <a href="https://bugs.python.org/issue27164" rel="nofollow noopener">here</a>. Try Python-3.5.</p>

username: robotcator123
cooked: <p>Do you mean there is something wrong with zip lib?<br>
I  using a global zipfile object to resolve this problem because I found that dataloader using multiprocessing.Process. I am not sure this way is correct.</p>

username: karmus89
cooked: <p>Hi,</p>
<p>I’ve noticed that sometimes the closing of the opened resource helps. Using <a class="mention" href="/u/victorhcm">@victorhcm</a>’s example I’m using the following kind of setup:</p>
<pre><code class="lang-auto">class MyDataSet(Dataset):
  def __init__(self, filename):
       self.zip_handle = None
       self.fname = filename

   def __getitem__(self, x):
       with zipfile.ZipFile(self.fname, 'r') as zip_handle:
           # do stuff while file is open
       # file is now closed
</code></pre>
<p>To add, my go-to files are either CSV’s or CSV-derived local SQLite databases. Especially the latter are very sensitive to leaving the connections open, i.e.</p>
<pre><code class="lang-auto">    ...
    def __getitem__(self, x):
        with self.db:
            #do stuff and close after
    ....
</code></pre>
<p>Petteri</p>

username: YixuanWei
cooked: <p>I think if you create a zip handler directly in <strong>init</strong> of dataset classes, instead of transfer the params from outside, it will also works</p>