IndexError when running dataloader for multiple workers

username: ruchitpatel
cooked: <p>I have a Dataset class as follows. I want to generate batches of size 64 using DataLoader of torch.</p>
<pre><code class="lang-auto">from torch.utils import data
import random
import numpy as np
import torch

class Dataset(data.Dataset):
    def __init__(self, qrel, query_vecs, offsets, fname):
        self.qrel = qrel
        self.qrel_list = list(qrel)
        self.query_vecs = query_vecs
        self.offsets = offsets
        self.f = open(fname, 'rt')
        self.keys = offsets.keys()
        self.skipped = 0
        
    def __len__(self):
        return len(self.qrel)
    
    def __getitem__(self, index):
        key = self.qrel_list[index]
        query = self.query_vecs[key].reshape(1, 300)

        posid = self.qrel[key][0]
        offset = self.offsets[posid]
        self.f.seek(offset)
        pos = np.fromstring(self.f.readline().split('\t')[1], sep=" ").reshape(1, 300)        
        
        neg_keys = random.sample(self.keys, 4)
        negs = []
        for neg in neg_keys:
            self.f.seek(self.offsets[neg])
            negs.append(np.fromstring(self.f.readline().split('\t')[1], sep=" "))
            
        query = torch.tensor(query).float()
        pos = torch.tensor(pos).float()
        negs = torch.tensor(np.array(negs).reshape(4, 300)).float()
        
        return (query, pos, negs)
</code></pre>
<p>I initialise this Dataset as follows.</p>
<pre><code class="lang-auto">qrel = {}
with gzip.open("msmarco-doctrain-qrels.tsv.gz", 'rt', encoding='utf8') as f:
    tsvreader = csv.reader(f, delimiter="\t")
    for [topicid, _, docid, rel] in tsvreader:
        assert rel == "1"
        if topicid in qrel:
            qrel[topicid].append(docid)
        else:
            qrel[topicid] = [docid]
            
with open('query_vecs.pickle', 'rb') as f:
    query_vecs = pickle.load(f)
    
with open('doc_vec_offsets.p', 'rb') as f:
    offsets = pickle.load(f)
    
fname = 'doc_vecs.tsv'
params = {'batch_size': 16,
          'shuffle': False,
          'num_workers': 8,
          'drop_last': True}

ds = Dataset(qrel, query_vecs, offsets, fname)
dl = data.DataLoader(ds, **params)
</code></pre>
<p>If I run following script, it works fine for 1 worker, but if I set <code>num_workers=8</code>, in DataLoader, I get IndexError. What is the reason for this error?</p>
<pre><code class="lang-auto">for batch in dl:
    ...
    train_model(batch)
</code></pre>
<p>When running it with 8 workers, i get this error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "/home/ruchit/Desktop/NLP/project/playground.py", line 49, in &lt;module&gt;
    for s in dl:

  File "/home/ruchit/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)

  File "/home/ruchit/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)

IndexError: Traceback (most recent call last):
  File "/home/ruchit/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ruchit/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in &lt;listcomp&gt;
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ruchit/Desktop/NLP/project/my_dataset.py", line 26, in __getitem__
    pos = np.fromstring(f.readline().split('\t')[1], sep=" ").reshape(1, 300)
IndexError: list index out of range
</code></pre>
<p>Can someone help me here?</p>

username: dhpollack
cooked: <p>I think you should refactor your code to read the docs file in the <code>__init__</code> function and then grab from that variable instead of reading the file in the <code>__getitem__</code> method.  For one, with negative sampling, you are doing 4x the number of reads because you are re-reading data then.  Also, you can check to make sure that every line has a tab character, which doesn’t seem to be the case here because the list created by <code>self.f.readline().split</code> is not a length of two.  It could be that the file gets locked during read so when you have too many workers reading the same file then it tries to read the file, but it’s locked and returns nothing.</p>

username: ruchitpatel
cooked: <p>Thank you for this excellent piece of advice. I’ll try your suggestions. However, about tab characters, I don’t think this might be the issue because this works fine with one worker. But I’ll still test for the tab characters. I’ll get back with the update soon. Thank you so much again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ruchitpatel
cooked: <p><a class="mention" href="/u/dhpollack">@dhpollack</a>, Is there a way in which I can open a separate files for each workers? After all each worker reads from a same file, and I do not have any issue with opening 8 instances of the same file.</p>

username: ruchitpatel
cooked: <p>So I tested for tab characters, every line of file has a tab character. I also opened 5 instances of same file in order to read different doc from different instance. But still I’m getting the same error.</p>

username: dhpollack
cooked: <p>I don’t think you can ensure each dataset opens a different file. Especially since your negative sample seems to choose a random file / line in your entire dataset. This you need access to the entire set for random sampling.</p>