Getting key error when training object detection model

username: Bisma_Akram
cooked: <h2>Hi! I am getting the error while training my object detection model.<br>
Epoch 1/1<br>
Mini-batch: 1/1457 Loss: 0.1182<br>
Mini-batch: 101/1457 Loss: 0.1295<br>
Mini-batch: 201/1457 Loss: 0.1242<br>
Mini-batch: 301/1457 Loss: 0.1200<br>
Mini-batch: 401/1457 Loss: 0.1188<br>
Mini-batch: 501/1457 Loss: 0.1164</h2>
<p>KeyError                                  Traceback (most recent call last)<br>
 in ()<br>
8     print("Epoch %i/%i " % (epoch + 1, num_epochs) )<br>
9     average_loss = 0<br>
—&gt; 10     for batch_id, (images, targets) in enumerate(train_data_loader):<br>
11         # Prepare the batch data<br>
12         images, targets = move_batch_to_device(images, targets)</p>
<p>3 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)<br>
393             # (<a href="https://bugs.python.org/issue2651" rel="nofollow noopener">https://bugs.python.org/issue2651</a>), so we work around it.<br>
394             msg = KeyErrorMessage(msg)<br>
–&gt; 395         raise self.exc_type(msg)</p>
<p>KeyError: Caught KeyError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py”, line 185, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “”, line 43, in <strong>getitem</strong><br>
damage_labels.append(damage_dict[label])<br>
KeyError: ‘D0w0’</p>
<p>here is my damage dataset class<br>
from torch.utils.data import Dataset, DataLoader</p>
<h1>Inherit from pytorch Dataset for convenience</h1>
<p>class DamageDataset(Dataset):</p>
<pre><code>def __init__(self, dataframe):

    super().__init__()

    self.filename = dataframe['filename'].unique()

    self.df = dataframe

def __len__(self) -&gt; int:

    return len(self.filename)



def __getitem__(self, index: int):

    filename = self.filename[index]

    image = read_image_from_train_folder(filename).astype(np.float32)

    # Scale to [0,1] range expected by the pre-trained model

    image /= 255.0

    # Convert the shape from [h,w,c] to [c,h,w] as expected by pytorch

    image = torch.from_numpy(image).permute(2,0,1)

    

    records = self.df[self.df['filename'] == filename]

    

    boxes = records[['xmin', 'ymin', 'xmax', 'ymax']].values

    classes= records['class'].values

    damage_labels=[]

    damage_dict={

        'D00': 1,

        'D01': 2, 

        'D10': 3,

        'D11': 4,

        'D20': 5,

        'D40': 6,

        'D43': 7,

        'D44': 8,

        'D50': 9

        

    }

    for label in classes:

      damage_labels.append(damage_dict[label])

   

    boxes = torch.as_tensor(boxes, dtype=torch.float32)

    n_boxes = boxes.shape[0]

    

    # there is only one foreground class, WHEAT

    labels = torch.as_tensor(damage_labels, dtype=torch.float32)

            

    target = {}

    target['boxes'] = boxes

    target['labels'] = labels

    

    return image, target
</code></pre>
<p>and here is my tarining code:<br>
num_epochs = 1</p>
<h1>Prepare the model for training</h1>
<p>model = model.to(device)</p>
<p>model.train()</p>
<p>for epoch in range(num_epochs):</p>
<pre><code>print("Epoch %i/%i " % (epoch + 1, num_epochs) )

average_loss = 0

for batch_id, (images, targets) in enumerate(train_data_loader):

    # Prepare the batch data

    images, targets = move_batch_to_device(images, targets)

    # Calculate losses

    loss_dict = model(images, targets)

    batch_loss = sum(loss for loss in loss_dict.values()) / len(loss_dict)

    

    # Refresh accumulated optimiser state and minimise losses

    optimizer.zero_grad()

    batch_loss.backward()

    optimizer.step()

    

    # Record stats

    loss_value = batch_loss.item()

    average_loss = average_loss + (loss_value - average_loss) / (batch_id + 1)

    print("Mini-batch: %i/%i Loss: %.4f" % ( batch_id + 1, len(train_data_loader), average_loss), end='\r')

    if batch_id % 100 == 0:

        print("Mini-batch: %i/%i Loss: %.4f" % ( batch_id + 1, len(train_data_loader), average_loss))
</code></pre>
<p>I would really appreciate some help into it, thank you.</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/bisma_akram">@Bisma_Akram</a></p>
<aside class="quote no-group" data-username="Bisma_Akram" data-post="1" data-topic="93903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bisma_akram/48/18760_2.png" class="avatar"> Bisma_Akram:</div>
<blockquote>
<p>damage_labels.append(damage_dict[label])<br>
KeyError: ‘D0w0</p>
</blockquote>
</aside>
<p>This happens inside your dataloader Because you don’t have a specific key for that.</p>
<aside class="quote no-group" data-username="Bisma_Akram" data-post="1" data-topic="93903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bisma_akram/48/18760_2.png" class="avatar"> Bisma_Akram:</div>
<blockquote>
<pre><code class="lang-auto">damage_dict={

        'D00': 1,

        'D01': 2, 

        'D10': 3,

        'D11': 4,

        'D20': 5,

        'D40': 6,

        'D43': 7,

        'D44': 8,

        'D50': 9

</code></pre>
</blockquote>
</aside>
<p>The key generating error is this</p>
<aside class="quote no-group" data-username="Bisma_Akram" data-post="1" data-topic="93903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bisma_akram/48/18760_2.png" class="avatar"> Bisma_Akram:</div>
<blockquote>
<p>KeyError: ‘D0w0’</p>
</blockquote>
</aside>

username: Bisma_Akram
cooked: <p>hey, but that isnt a class in my dataset, do you think there is some error in the csv file of the training dataset that contain labels and bounding boxes,how do you think I can find out which index this class exist at so that I can delete it, I am iterating through my dataframe but cant find a class that contains D0w0.</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/bisma_akram">@Bisma_Akram</a></p>
<aside class="quote no-group" data-username="Bisma_Akram" data-post="1" data-topic="93903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bisma_akram/48/18760_2.png" class="avatar"> Bisma_Akram:</div>
<blockquote>
<pre><code class="lang-auto"> for label in classes:

      damage_labels.append(damage_dict[label])
</code></pre>
</blockquote>
</aside>
<p>when you iterate over classes, your classes list contains the anomly either check this column or remove it from your dataset. It’s better if you use exception handling here.</p>

username: Bisma_Akram
cooked: <p>great! I found out and removed it, could you please help me with the other question I posted too. thank you</p>