KeyError in Dataloader

praggya: <pre><code class="lang-auto">Traceback (most recent call last):
  File "EuroSAT_train.py", line 119, in &lt;module&gt;
    main()
  File "EuroSAT_train.py", line 76, in main
    for step, (x_spt, y_spt, x_qry, y_qry) in enumerate(db):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/opt/conda/lib/python3.7/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/voyager-volume/MAML-Pytorch/EuroSAT.py", line 156, in __getitem__
    for sublist in self.support_x_batch[index] for item in sublist]).astype(np.int32)
  File "/voyager-volume/MAML-Pytorch/EuroSAT.py", line 156, in &lt;listcomp&gt;
    for sublist in self.support_x_batch[index] for item in sublist]).astype(np.int32)
KeyError: '5594'
</code></pre>
<p>The code</p>
<pre><code class="lang-auto">euro = EuroSAT('/voyager-volume/MAML-Pytorch/EuroSAT/',mode='train', n_way=args.n_way, k_shot=args.k_spt,
                        k_query=args.k_qry,
                        batchsz=10000, resize=args.imgsz)
    def get_subset(indices, start, end):
        return indices[start:start+end]
    indices = torch.randperm(len(euro))
    train_indices = get_subset(indices, 0, len(euro))
    train_sampler = SubsetRandomSampler(train_indices)
    euro_test = EuroSAT('/voyager-volume/MAML-Pytorch/EuroSAT/', mode='test', n_way=args.n_way, k_shot=args.k_spt,
                             k_query=args.k_qry,
                             batchsz=100, resize=args.imgsz)

    for epoch in range(args.epoch//10000):
        # fetch meta_batchsz num of episode each time
        db = DataLoader(euro, args.task_num, sampler=train_sampler, shuffle=False, num_workers=1, pin_memory=True)

        for step, (x_spt, y_spt, x_qry, y_qry) in enumerate(db):


            x_spt, y_spt, x_qry, y_qry = x_spt.to(device), y_spt.to(device), x_qry.to(device), y_qry.to(device)

            accs = maml(x_spt, y_spt, x_qry, y_qry)

            if step % 30 == 0:
                print('step:', step, '\ttraining acc:', accs)

            if step % 500 == 0:  # evaluation
                db_test = DataLoader(euro_test, 1, shuffle=True, num_workers=1, pin_memory=True)
                accs_all_test = []

                for x_spt, y_spt, x_qry, y_qry in db_test:
                    x_spt, y_spt, x_qry, y_qry = x_spt.squeeze(0).to(device), y_spt.squeeze(0).to(device), \
                                                 x_qry.squeeze(0).to(device), y_qry.squeeze(0).to(device)

                    accs = maml.finetunning(x_spt, y_spt, x_qry, y_qry)
                    accs_all_test.append(accs)

                # [b, update_step+1]
                accs = np.array(accs_all_test).mean(axis=0).astype(np.float16)
                print('Test acc:', accs)

</code></pre>

praggya: <p>Can anyone please tell me why am I facing this issue.Obliged…</p>

ptrblck: <p>Based on the stacktrace the <code>Dataset</code> fails to load:</p>
<pre><code class="lang-python">for sublist in self.support_x_batch[index] for item in sublist]).astype(np.int32)
</code></pre>
<p>for an <code>index</code> of <code>5594</code>.<br>
Make sure you are defining the length of the <code>Dataset</code> properly and that <code>5594</code> is indeed a valid index.</p>

Vidsha_Rupani: <p>While the code seems logically structured, there are several potential issues that could lead to KeyError in Dataloader or unintended behavior:</p>
<ol>
<li><strong>Division in <code>range(args.epoch//10000)</code></strong>: If <code>args.epoch</code> is less than 10000, this will result in a range of 0, meaning the outer loop won’t execute. Ensure that <code>args.epoch</code> is correctly set and divisible by 10000 to get the desired number of epochs.</li>
<li><strong>DataLoader Initialization Inside Loop</strong>: The <code>DataLoader</code> for the test set (<code>db_test</code>) is initialized inside the loop for every 500 steps. This can be inefficient. If the test dataset (<code>euro_test</code>) does not change, you should initialize <code>db_test</code> outside of the loop.</li>
<li><strong>Potential Memory Issue with <code>pin_memory=True</code></strong>: Setting <code>pin_memory=True</code> can be beneficial as it enables faster data transfer to CUDA-enabled GPUs, but it also consumes more memory. Ensure you have sufficient memory resources, especially if working with large datasets.</li>
<li><strong>Batch Size in Test DataLoader</strong>: The batch size for <code>db_test</code> is set to 1. This is common for evaluation, but if your <code>maml.finetuning</code> method expects a different batch size, this could cause issues.</li>
<li><strong>Squeezing Tensor Dimensions</strong>: The use of <code>squeeze(0)</code> assumes that the first dimension of your tensors is 1 (common in single-batch scenarios). Be sure this assumption holds, as <code>squeeze(0)</code> will remove this dimension, which might cause dimensionality issues in your model.</li>
<li><strong>Error Handling in Model Forward Pass</strong>: The lines <code>accs = maml(x_spt, y_spt, x_qry, y_qry)</code> and <code>accs = maml.finetunning(x_spt, y_spt, x_qry, y_qry)</code> do not include any error handling. Any issues within the model’s forward pass or the finetuning method will cause the script to fail.</li>
</ol>

smth: <p><a class="mention" href="/u/vidsha_rupani">@Vidsha_Rupani</a> when you are using ChatGPT or other tools to post replies, please specify that clearly at the beginning of your comment <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>