Collate Issue with Fast RCNN (tries to transform dictionary in GeneralizedRCNN)

username: maddyschiappa
cooked: <p>I am trying to create a custom dataset to pass into torch.utils.data.DataLoader for training: <a href="https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn</a></p>
<p>The instructions indicate they want a list of images and a list of  (list[Dict[Tensor]]) for the targets with ‘boxes’ and ‘labels’. Now I create this in my dataset <strong>getitem</strong> which returns the image as a tensor and a dictionary with keys “boxes” and “labels” and a list of 4 floats for “boxes” and an int for “labels”.</p>
<p>When I try training I run into the following issues:</p>
<pre><code class="lang-auto">  File "/home/mschiappa/Desktop/VisualRelationshipsDetection/run_exp.py", line 118, in run_model
    outputs = model(inputs, targets)
  File "/home/mschiappa/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/lib/python3/dist-packages/torchvision/models/detection/generalized_rcnn.py", line 47, in forward
    images, targets = self.transform(images, targets)
  File "/home/mschiappa/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/lib/python3/dist-packages/torchvision/models/detection/transform.py", line 36, in forward
    target = targets[i] if targets is not None else targets
KeyError: 0
</code></pre>
<p>and when I print what the targets look like:</p>
<pre><code class="lang-auto">{'boxes': tensor([[0.6547, 1.0000, 0.4703, 0.4938],
        [0.7802, 0.9403, 0.0472, 0.0796],
        [0.9897, 0.9823, 0.0000, 0.4879],
        [1.0000, 0.4943, 0.8626, 0.3522],
        [1.0000, 1.0000, 0.0000, 0.0000],
        [1.0000, 0.2750, 0.8687, 0.0000],
        [0.9956, 0.7544, 0.8100, 0.5774],
        [1.0000, 0.9531, 0.0000, 0.1137]]), 'labels': tensor([227, 445, 432, 307, 390, 281, 419, 333])}
</code></pre>
<p>When I look into the GeneralizedRCNN code which is inherited by the fastrcnn, it tries to transform the targets: <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/generalized_rcnn.py#L64" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/detection/generalized_rcnn.py#L64</a><br>
Which shouldn’t work since it is a dictionary, and it appears to be where it is failing.</p>
<p>I am unsure how to fix this without modifying the PyTorch fastrcnn code.</p>

username: maddyschiappa
cooked: <p>Currently making progress with the fix:</p>
<pre><code class="lang-auto"># images, targets = self.transform(images, targets)
images, _ = self.transform(images)

</code></pre>
<p>to:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/generalized_rcnn.py#L64" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/generalized_rcnn.py#L64" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/torchvision/models/detection/generalized_rcnn.py#L64</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="54" style="counter-reset: li-counter 53 ;">
<li>
</li>
<li>"""</li>
<li>if self.training and targets is None:</li>
<li>    raise ValueError("In training mode, targets should be passed")</li>
<li>original_image_sizes = torch.jit.annotate(List[Tuple[int, int]], [])</li>
<li>for img in images:</li>
<li>    val = img.shape[-2:]</li>
<li>    assert len(val) == 2</li>
<li>    original_image_sizes.append((val[0], val[1]))</li>
<li>
</li>
<li class="selected">images, targets = self.transform(images, targets)</li>
<li>features = self.backbone(images.tensors)</li>
<li>if isinstance(features, torch.Tensor):</li>
<li>    features = OrderedDict([(0, features)])</li>
<li>proposals, proposal_losses = self.rpn(images, features, targets)</li>
<li>detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)</li>
<li>detections = self.transform.postprocess(detections, images.image_sizes, original_image_sizes)</li>
<li>
</li>
<li>losses = {}</li>
<li>losses.update(detector_losses)</li>
<li>losses.update(proposal_losses)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: ptrblck
cooked: <p>Could you try to pass the target as a <code>list of dicts</code>, where each <code>dict</code> in the <code>list</code> corresponds to the boxes and labels for the element in the data list.</p>
<p>EDIT: This code should work:</p>
<pre><code class="lang-python">model = models.detection.fasterrcnn_resnet50_fpn()

x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
y = [{'boxes': torch.tensor([[0.6547, 1.0000, 0.4703, 0.4938],
        [0.7802, 0.9403, 0.0472, 0.0796],
        [0.9897, 0.9823, 0.0000, 0.4879],
        [1.0000, 0.4943, 0.8626, 0.3522],
        [1.0000, 1.0000, 0.0000, 0.0000],
        [1.0000, 0.2750, 0.8687, 0.0000],
        [0.9956, 0.7544, 0.8100, 0.5774],
        [1.0000, 0.9531, 0.0000, 0.1137]]),
     'labels': torch.tensor([227, 445, 432, 307, 390, 281, 419, 333])},
    {'boxes': torch.tensor([[0.6547, 1.0000, 0.4703, 0.4938],
        [0.7802, 0.9403, 0.0472, 0.0796],
        [0.9897, 0.9823, 0.0000, 0.4879],
        [1.0000, 0.4943, 0.8626, 0.3522],
        [1.0000, 1.0000, 0.0000, 0.0000],
        [1.0000, 0.2750, 0.8687, 0.0000],
        [0.9956, 0.7544, 0.8100, 0.5774],
        [1.0000, 0.9531, 0.0000, 0.1137]]),
     'labels': torch.tensor([227, 445, 432, 307, 390, 281, 419, 333])}
]

output = model(x, y)
</code></pre>

username: maddyschiappa
cooked: <p>Now getting this error:</p>
<pre><code class="lang-auto">ImportError: /usr/lib/python3/dist-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c107Warning4warnENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
</code></pre>

username: maddyschiappa
cooked: <p>This leads to the same error as if I change the model:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/mschiappa/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/lib/python3/dist-packages/torchvision/models/detection/generalized_rcnn.py", line 52, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/home/mschiappa/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/lib/python3/dist-packages/torchvision/models/detection/rpn.py", line 411, in forward
    boxes, scores = self.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level)
  File "/usr/lib/python3/dist-packages/torchvision/models/detection/rpn.py", line 336, in filter_proposals
    keep = box_ops.batched_nms(boxes, scores, lvl, self.nms_thresh)
  File "/usr/lib/python3/dist-packages/torchvision/ops/boxes.py", line 72, in batched_nms
    keep = nms(boxes_for_nms, scores, iou_threshold)
  File "/usr/lib/python3/dist-packages/torchvision/ops/boxes.py", line 32, in nms
    _C = _lazy_import()
  File "/usr/lib/python3/dist-packages/torchvision/extension.py", line 12, in _lazy_import
    from torchvision import _C as C
ImportError: /usr/lib/python3/dist-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c107Warning4warnENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
</code></pre>
<p>also that is what I want to do but when I use the dataset class and make my own custom <strong>getitem</strong> and a I return the image and a dictionary, for some reason it collates them. How do I stop this so it returns the same format as you provided when using the DataLoader class to iterate through?</p>

username: ptrblck
cooked: <p>Could you post the modification to your model, so that we have a baseline we could debug against, please?</p>

username: maddyschiappa
cooked: <p>It won’t work because later code still requires it in that format. The summary is that I need to find out why the DataLoader keeps collating my batch into one big dictionary where the keys have a list of the batches values instead of a list of dictionaries, one per sample.</p>
<p>My getitem is returning <code>image, mydict</code> but when the DataLoader gets a batch it becomes [image1, image2], {key1: [target1, target2], key2: [target1, target2]}</p>

username: Tudor_Marian_Surdoiu
cooked: <p>Try something like this:</p>
<pre><code class="lang-auto">torch.utils.data.DataLoader(
     dataset, batch_size=8, shuffle=True, num_workers=1,
     collate_fn=collate_fn
)
</code></pre>
<p>where collate_fn is:</p>
<pre><code class="lang-auto">def collate_fn(batch):
    return list(zip(*batch))
</code></pre>