Detectron2 test loader, error

a_friend: <p>I am following the <a href="https://colab.research.google.com/drive/1QtTW9-ukX2HKZGvt0QvVGqjuqEykoZKI#scrollTo=B9ErX0ZvYYHI" rel="noopener nofollow ugc">Detectron/ Detic tutorial</a> in order to auto crop a large training dataset from the <a href="https://www.kaggle.com/c/happy-whale-and-dolphin/data" rel="noopener nofollow ugc">Kaggle Whale and dolphin competition</a>.</p>
<p>Can anyone help out with the index error I am getting? It seems like Detectron tries to index a list with the dict key.</p>
<p>The config <code>cfg</code>, vocabulary etc. are the same as in the tutorial.</p>
<p>Here is  a snippet of my code, unfortunately not replicable without the actual image etc, hopefully informative enough to make sense.</p>
<pre><code class="lang-auto">from detectron2.modeling import build_model
from detectron2.data import build_detection_test_loader
import torch

def my_dataset_generator():
    data_list = []
    data_dict = {}
    for index, observation in df.iterrows():
        data_dict["file_name"] = observation['file_path']
        data_dict['image_id'] = observation['image']
        data_list.append(data_dict)
        data_dict = {}
        
    return data_list

model = build_model(cfg)  # returns a torch.nn.Module
test_loader = build_detection_test_loader(cfg, 'my_whales2')
</code></pre>
<p>which returns</p>
<pre><code class="lang-auto">[02/28 12:46:55 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333, sample_style='choice')]
[02/28 12:46:55 d2.data.common]: Serializing 51033 elements to byte tensors and concatenating them all ...
[02/28 12:46:55 d2.data.common]: Serialized dataset takes 6.42 MiB
</code></pre>
<p>Printing a sample of the test loader look like this:</p>
<pre><code class="lang-auto">a = next(iter(test_loader))
a
</code></pre>
<pre><code class="lang-auto">[{'file_name': '/home/jupyter/happy-whale-and-dolphin/train_images/00021adfb725ed.jpg',
  'image_id': '00021adfb725ed.jpg',
  'width': 804,
  'height': 671,
  'image': tensor([[[ 74,  74,  73,  ...,  71,  72,  73],
           [ 76,  76,  77,  ...,  63,  64,  65],
           [ 80,  81,  83,  ...,  60,  60,  61],
           ...,
           [ 44,  42,  43,  ...,  50,  51,  51],
           [ 43,  41,  42,  ...,  49,  50,  50],
           [ 43,  41,  41,  ...,  50,  50,  50]],
  
          [[120, 120, 119,  ..., 113, 114, 115],
           [122, 122, 123,  ..., 104, 104, 105],
           [126, 127, 130,  ...,  99,  99, 100],
           ...,
           [ 45,  43,  43,  ...,  48,  49,  50],
           [ 46,  44,  43,  ...,  49,  50,  50],
           [ 45,  43,  43,  ...,  50,  50,  50]],
  
          [[170, 170, 169,  ..., 163, 164, 165],
           [172, 172, 173,  ..., 155, 155, 156],
           [175, 177, 178,  ..., 149, 149, 150],
           ...,
           [ 54,  53,  53,  ...,  58,  59,  60],
           [ 55,  53,  53,  ...,  57,  58,  58],
           [ 57,  55,  55,  ...,  58,  58,  58]]], dtype=torch.uint8)}]
</code></pre>
<p>I try to do inference on that data loader</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
    outputs = model(test_loader)
</code></pre>
<p>which renders this stack trace:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_14737/2278559883.py in &lt;module&gt;
      1 model.eval()
      2 with torch.no_grad():
----&gt; 3     outputs = model(test_loader)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

~/Detic/detic/modeling/meta_arch/custom_rcnn.py in forward(self, batched_inputs)
    113         """
    114         if not self.training:
--&gt; 115             return self.inference(batched_inputs)
    116 
    117         images = self.preprocess_image(batched_inputs)

~/Detic/detic/modeling/meta_arch/custom_rcnn.py in inference(self, batched_inputs, detected_instances, do_postprocess)
     94         assert detected_instances is None
     95 
---&gt; 96         images = self.preprocess_image(batched_inputs)
     97         features = self.backbone(images.tensor)
     98         proposals, _ = self.proposal_generator(images, features, None)

/opt/conda/lib/python3.7/site-packages/detectron2/modeling/meta_arch/rcnn.py in preprocess_image(self, batched_inputs)
    222         Normalize, pad and batch the input images.
    223         """
--&gt; 224         images = [x["image"].to(self.device) for x in batched_inputs]
    225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)

/opt/conda/lib/python3.7/site-packages/detectron2/modeling/meta_arch/rcnn.py in &lt;listcomp&gt;(.0)
    222         Normalize, pad and batch the input images.
    223         """
--&gt; 224         images = [x["image"].to(self.device) for x in batched_inputs]
    225         images = [(x - self.pixel_mean) / self.pixel_std for x in images]
    226         images = ImageList.from_tensors(images, self.backbone.size_divisibility)

TypeError: list indices must be integers or slices, not str
</code></pre>

a_friend: <p>Here is the function that throws</p>
<pre><code class="lang-auto">    def preprocess_image(self, batched_inputs: List[Dict[str, torch.Tensor]]):
        """
        Normalize, pad and batch the input images.
        """
        images = [x["image"].to(self.device) for x in batched_inputs]
        images = [(x - self.pixel_mean) / self.pixel_std for x in images]
        images = ImageList.from_tensors(images, self.backbone.size_divisibility)
        return images
</code></pre>
<p>Did I set up, or call, my data loader improperly? The stack trace would make sense if each element of the <code>batched_inputs</code> is a dict wrapped in a list, right?</p>

nivek: <p>Can you try checking the type of the elements within <code>batched_inputs</code> in the function <code>preprocess_image</code>?</p>
<p>The error seems to be saying that <code>x</code> in this line</p>
<pre><code class="lang-python">images = [x["image"].to(self.device) for x in batched_inputs]
</code></pre>
<p>is a <code>list</code> rather than a <code>dict</code> that you are expecting.</p>

a_friend: <p><a class="mention" href="/u/nivek">@nivek</a> I agree. The output of</p>
<pre><code class="lang-auto">data_iter = iter(test_loader)
for x in data_iter:
    print(type(x))
</code></pre>
<p>is</p>
<pre><code class="lang-auto">&lt;class 'list'&gt;
&lt;class 'list'&gt;
&lt;class 'list'&gt;
(...)
&lt;class 'list'&gt;
&lt;class 'list'&gt;
</code></pre>
<p>From the Detectron docs regarding models and data loaders:</p>
<blockquote>
<p>The output of the default <a href="https://detectron2.readthedocs.io/en/latest/modules/data.html#detectron2.data.DatasetMapper" rel="noopener nofollow ugc">DatasetMapper</a> is a dict that follows the above format. After the data loader performs batching, it becomes <code>list[dict]</code> which the builtin models support.</p>
</blockquote>
<p>My data loader generates <code>list[list[dict]]</code> it seems, not sure why.</p>

nivek: <aside class="quote no-group" data-username="a_friend" data-post="1" data-topic="145190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/9de053/48.png" class="avatar"> a_friend:</div>
<blockquote>
<p><code>my_dataset_generator</code></p>
</blockquote>
</aside>
<p>I don’t know how <code>my_dataset_generator</code> is being used since its usage is excluded from your code snippet.</p>
<p>My best guess is that the function <code>my_dataset_generator</code> is returning <code>list[dict]</code>. Subsequently, <code>build_detection_test_loader</code> returns a DataLoader with a <code>batch_size&gt;1</code>. So when batching happens in the DataLoader, you will get <code>list[list[dict]]]</code>.</p>

a_friend: <p>Sorry, the <code>my_dataset_generator()</code> function is called by</p>
<pre><code class="lang-auto">from detectron2.data import DatasetCatalog
DatasetCatalog.register("my_whales2", my_dataset_generator)
</code></pre>
<p>I forgot to include that line in my original post. From the <a href="https://detectron2.readthedocs.io/en/latest/modules/data.html#detectron2.data.catalog.DatasetCatalog" rel="noopener nofollow ugc">Detectron docs</a>:</p>
<blockquote>
<p><strong>func</strong> (<em>callable</em> ) – a callable which takes no arguments and returns a list of dicts. It must return the same results if called multiple times</p>
</blockquote>
<p>Regarding the data loader, it <a href="https://detectron2.readthedocs.io/en/latest/modules/data.html#detectron2.data.build_detection_test_loader" rel="noopener nofollow ugc">accepts</a> a</p>
<blockquote>
<p><strong>dataset</strong> – a list of dataset dicts, or a pytorch dataset (either map-style or iterable)…</p>
</blockquote>
<p>We agree that the data loader returns the wrong format, but I fail to understand what I am doing wrong according to the documentation.</p>
<p>If I should pass the data dicts in another format to the data loader, what would that be?</p>

nivek: <p>The DataLoader processes each batch with the <code>collate_fn</code>, you can read more <a href="https://pytorch.org/docs/stable/data.html#working-with-collate-fn" rel="noopener nofollow ugc">about it here</a>.</p>
<p>By default, the <a href="https://github.com/pytorch/pytorch/blob/5749be4678a3b240b5b5d22a228a43141e2cd32c/torch/utils/data/_utils/collate.py#L84-L180" rel="noopener nofollow ugc"><code>default_collate</code></a> function is used. It is hard to debug why <code>default_collate</code> is not working as your case without look at the exact samples in your data.</p>
<p>I think the easiest way is for you to pass in a <a href="https://github.com/facebookresearch/detectron2/blob/main/detectron2/data/build.py#L476" rel="noopener nofollow ugc">custom <code>collate_fn</code> into <code>build_detection_test_loader</code></a>, your custom function should take a batch of samples and collate them (map <code>list[list[dict]]</code> to <code>list[dict]</code>). That should solve the issue for you.</p>
<p>Alternatively, you can try to figure out why <code>default_collate</code> is not working for your inputs (I think <a href="https://github.com/pytorch/pytorch/blob/5749be4678a3b240b5b5d22a228a43141e2cd32c/torch/utils/data/_utils/collate.py#L163-L180" rel="noopener nofollow ugc">this part is the most relevant for your input</a>), and modify your inputs accordingly before passing them into the DataLoader.</p>

a_friend: <p>I have made some progress, thanks to your clues. It turned out that if I copy-pasted <code>default_collate</code> into my notebook, I got a useful traceback about missing <code>torch.stack</code> dimension match. This seemed to caused by only the short edge of the images being resized.</p>
<p>After calling a custom DataLoader, resizing all edges of the images, the procedure runs longer, but produces <a href="https://discuss.pytorch.org/t/detic-valueerror-from-swintransformer-py/145966">another error</a>. Puzzling…</p>