KeyError when enumerating over dataloader

washizzle: <p>When enumerating over dataloaders I get the following error:</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 218, in <br>
main()<br>
File “train.py”, line 109, in main<br>
train_valid(model, optimizer, scheduler, epoch, data_loaders, data_size, t)<br>
File “train.py”, line 128, in train_valid<br>
for batch_idx, batch_sample in enumerate(dataloaders[phase]):<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 819, in <strong>next</strong><br>
return self._process_data(data)<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 846, in _process_data<br>
data.reraise()<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/_utils.py”, line 369, in reraise<br>
raise self.exc_type(msg)<br>
KeyError: Caught KeyError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py”, line 178, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch<br>
return self.collate_fn(data)<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py”, line 75, in default_collate<br>
return {key: default_collate([d[key] for d in batch]) for key in elem}<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py”, line 75, in <br>
return {key: default_collate([d[key] for d in batch]) for key in elem}<br>
File “/home/mhouben/miniconda3/envs/pytorch12/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py”, line 75, in <br>
return {key: default_collate([d[key] for d in batch]) for key in elem}<br>
KeyError: ‘anc_img’</p>
<p>I am running out of ideas on how to figure out what causes this error. Any thoughts on what could cause this? Or how I should approach this error?</p>

ptrblck: <p>Could you post the definition of your <code>Dataset</code>?<br>
Also, you could try to iterate the dataset directly without using a <code>DataLoader</code> for debugging purposes.</p>

washizzle: <p>My dataset consists of about 500k .png images, each with dimension 250x250x3 (CASIA Webface, which I aligned using <a href="https://github.com/davidsandberg/facenet/tree/master/src/align" rel="nofollow noopener">https://github.com/davidsandberg/facenet/tree/master/src/align</a>). Each epoch it takes 20k triplets of images from this dataset, 10k for training and 10k for validation.</p>
<p>I will try to iterate over the dataset directly and post a response once I am done.</p>

washizzle: <p>I still have not found a solution for my error. I tried to leave out the DataLoader but it seems like an awful lot of work to remake how the DataLoader fills these batches, or is there another way than making my own dataloaderesque class and implementing my own <strong>iter()</strong> implementation to fill the batches?</p>
<p>Perhaps it helps if I post my github repo: <a href="https://github.com/washizzle/facenet_pytorch" rel="nofollow noopener">https://github.com/washizzle/facenet_pytorch</a></p>
<p>I call:<br>
python train.py --train_format=.png --valid_format=.png --start-epoch=159 --pure_validation --load_pth_from=./log/20190814-221208/ --train-root-dir=/lustre2/0/wsdarts/datasets/vggface2_train/ --valid-root-dir=/lustre2/0/wsdarts/datasets/CASIA_aligned/ --train-csv-name=~/facenet_pytorch/datasets/train_vggface2.csv --valid-csv-name=~/facenet_pytorch/datasets/CASIA.csvpython train.py --train_format=.png --valid_format=.png --start-epoch=159 --pure_validation --load_pth_from=./log/20190814-221208/ --train-root-dir=/lustre2/0/wsdarts/datasets/vggface2_train/ --valid-root-dir=/lustre2/0/wsdarts/datasets/CASIA_aligned/ --train-csv-name=~/facenet_pytorch/datasets/train_vggface2.csv --valid-csv-name=~/facenet_pytorch/datasets/CASIA.csv</p>
<p>and it goes wrong on line 141 in train.py</p>

ptrblck: <p>I would just create a look using your <code>Dataset</code> without wrapping it into a <code>DataLoader</code>:</p>
<pre><code class="lang-python">dataset = ...
for idx, (data, image) in enumerate(dataset):
    print(idx)
</code></pre>
<p>This loop should raise an exception once the <code>KeyError</code> is raised. Using the last printed <code>idx</code> you could narrow down, which samples creates this issue.</p>

washizzle: <p>Doing this results in the following error instead, for every image:<br>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 64 3 7, but got 3-dimensional input of size [3, 182, 182]<br>
I guess because it is missing the batch dimension.</p>

ptrblck: <p>You should use this loop to isolate the issue regarding loading a specific sample and the raised <code>KeyError</code>.<br>
Once you’ve found the issue, you should use the <code>DataLoader</code> again.</p>

Upgrade_Yourself: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="54210">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>for idx, (data, image) in enumerate(dataset): print(idx)</p>
</blockquote>
</aside>
<p>Thank you for the tip</p>

sciarrilli: <p>this helped me figured out my IndexError. I added sampler to my validation set on accident and doing</p>
<pre><code class="lang-python">next(iter(dataloaders['val']))
</code></pre>
<p>did the trick!</p>

shivamsaboo17: <p>Hi, I am getting same error. Here is the traceback. <strong>Also I am able to iterate the full dataset without any exceptions as suggested by <a class="mention" href="/u/ptrblck">@ptrblck</a>  above.</strong></p>
<pre><code class="lang-python">Traceback (most recent call last):
  File "iBatchLearn.py", line 150, in &lt;module&gt;
    acc_table, task_names = run(args)
  File "iBatchLearn.py", line 78, in run
    agent.learn_batch(train_loader, val_loader)
  File "/home/shivam/ssaboo2020/Continual-Learning-Benchmark/agents/exp_replay.py", line 39, in learn_batch
    super(Naive_Rehearsal, self).learn_batch(new_train_loader, val_loader)
  File "/home/shivam/ssaboo2020/Continual-Learning-Benchmark/agents/default.py", line 222, in learn_batch
    self.validation(val_loader)
  File "/home/shivam/ssaboo2020/Continual-Learning-Benchmark/agents/default.py", line 115, in validation
    for i, (input, target, task) in enumerate(dataloader):
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 838, in _next_data
    return self._process_data(data)
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/shivam/ssaboo2020/Continual-Learning-Benchmark/dataloaders/wrapper.py", line 51, in __getitem__
    img,target = self.dataset[index]
  File "/home/shivam/ssaboo2020/Continual-Learning-Benchmark/dataloaders/wrapper.py", line 81, in __getitem__
    img,target = self.dataset[self.indices[index]]
  File "/home/shivam/ssaboo2020/Continual-Learning-Benchmark/dataloaders/wrapper.py", line 33, in __getitem__
    img,target = self.dataset[index]
  File "/home/shivam/ssaboo2020/Continual-Learning-Benchmark/dataloaders/custom_datasets.py", line 22, in __getitem__
    img = Image.open(self.df['PATH'].iloc[index])
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/pandas/core/indexing.py", line 1500, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/pandas/core/indexing.py", line 2230, in _getitem_axis
    self._validate_integer(key, axis)
  File "/home/shivam/ssaboo2020/incremental/lib/python3.5/site-packages/pandas/core/indexing.py", line 2139, in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds
</code></pre>

ptrblck: <p>That’s strange, as an indexing error should be also raised without the <code>DataLoader</code>.<br>
Anyway, could you print the passed index to your <code>__getitem__</code> and use <code>num_workers=0</code>, to see which index is failing?</p>

shivamsaboo17: <p>I did that and it fails on <code>6021-th</code> index. Here is my simple custom dataset. The length of the dataframe is 6134. I tried removing the csv entry at 6021th index and trying again but the dataset fails at the same index again.</p>
<pre><code class="lang-python">class FlowerBirds(Dataset):

    def __init__(self, reverse=True, mode='train', transform=None):
        ROOT = 'data_stuf/'
        if reverse:
            self.df = pd.read_csv(ROOT+mode+'_'+'bf.csv')
        else:
            self.df = pd.read_csv(ROOT+mode+'_'+'fb.csv')
        self.tfm = transform
        #print(len(self.df), 'DF LEN')
        #self.normalize = transforms.Normalize(mean=[0.491, 0.482, 0.447], std=[0.247, 0.243, 0.262]) 

    def __getitem__(self, index):
        #print(index, "INDEX")
        img = Image.open(self.df['PATH'].iloc[index])
        label = self.df['LABEL'].iloc[index]
        img = self.tfm(img)
        if img.shape[0] == 1:
            img = img.repeat(3, 1, 1)
        #img = self.normalize(img)
        label = torch.LongTensor(np.array([label])).item()
        return img, label

    def __len__(self):
        return len(self.df)

</code></pre>

ptrblck: <p>Do you get a proper error message, when you call your <code>Dataset</code> directly with this index via:</p>
<pre><code class="lang-python">dataset[6021]
</code></pre>

shivamsaboo17: <p>I get proper output tensors rather. However I am testing this dataset outside seperately and in my actual code it is wrapped by another dataset. But still, the <strong>final traceback is upto this dataset</strong> (<strong>and the index at which it shows error is valid when I test this dataset separately</strong>) hence I am unable to understand the problem.</p>

shivamsaboo17: <p>The issue was with the way I was splitting my data into tasks. Fixed it. The error was infact not related to dataset or datalaoder.</p>

kalilamali: <p>Thank you for your comment! It really help me.<br>
I got the same error and it was because I was splitting my data and forgetting to reset the indexes after splitting.</p>

Jim_Xu: <p>Hello, I meet the same problem, could you tell me how to reset the indexes?? Thanks a lot!!</p>

Jim_Xu: <p>Hello, Karen said the indexes should be reset, but I do not know how to reset  it, could you tell me how to reset the indexe??</p>
<p>Thanks a lot!!</p>

ptrblck: <p>I’m not sure what <a class="mention" href="/u/kalilamali">@kalilamali</a> means exactly by resetting the indexes after splitting.<br>
Maybe the original indices (from the complete dataset) were used to index the <code>Subsets</code>, which would yield an <code>IndexError</code>.</p>

kalilamali: <p>Hi. The indexes I forgot to reset were in my df.<br>
I was splitting my data like this:</p>
<pre><code class="lang-auto">if folds == 0:
        # Train and validation
        train, val = train_test_split(train_val, test_size=0.2, random_state=seed, shuffle=True)
        train, val = train.reset_index(drop=True), val.reset_index(drop=True)
</code></pre>
<p>And then I created this custom dataloader to retrieve the index.</p>
<pre><code class="lang-auto"># Custom pytorch dataloader for this dataset
    class Derm(Dataset):
        """
        Read a pandas dataframe with
        images paths and labels
        """
        def __init__(self, df, transform=None):
            self.df = df
            self.transform = transform

        def __len__(self):
            return len(self.df)

        def __getitem__(self, index):
            # Load image data and get label
            try:
                X = Image.open(self.df['filenames'][index]).convert('RGB')
                #y = torch.tensor(self.df.iloc[index,2:])
                y = torch.tensor(self.df['label_code'][index])
            except IOError as err:
                pass

            if self.transform:
                X = self.transform(X)
            # Sanity check
            print('id:', self.df['id'][index], 'label', y)
            return index, X, y
</code></pre>
<p>And then on the train/eval loop:</p>
<pre><code class="lang-auto">for index, inputs, labels in dataloader:
...
</code></pre>