Torchvision - Faster RCNN - Empty Training Images

sebo470: <p>Hi,</p>
<p>I am using the Torchvision Faster RCNN model for Object detection.<br>
I am trying to train the model with my own dataset and followed the tutorial on: <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>Now I am face a problem how to deal training images without an object to detect.<br>
At the moment I am using</p>
<pre><code>__getitem__():
    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    labels = torch.as_tensor(labels, dtype=torch.int64)
    labels+=1
    image_id = torch.tensor([idx])
    if num_objs==0:
        area = torch.tensor([0]) #also tried torch.tensor([]) 
    else:
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["image_id"] = image_id
    target["area"] = area
    target["iscrowd"] = iscrowd
    if self.transforms is not None:
        img, target = self.transforms(img, target)

    return img, target
</code></pre>
<p>in combination with the engine in <code>references/detection/engine.py</code>.<br>
Now if, a empty image is selected I get the following error message:<br>
Traceback (most recent call last):</p>
<p>File “/home/user/Detection_Experiments/Model_Base.py”, line 155, in <br>
model, optimizer, loss, lr_scheduler=main()<br>
File “/home/user/Detection_Experiments/Model_Base.py”, line 146, in main<br>
loss=train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)<br>
File “/home/user/Detection_Experiments/references/detection/engine.py”, line 30, in train_one_epoch<br>
loss_dict = model(images, targets)<br>
File “/home/user/.virtualenvs/test_p3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/user/.virtualenvs/test_p3/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/models/detection/generalized_rcnn.py”, line 47, in forward<br>
File “/home/user/.virtualenvs/test_p3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/user/.virtualenvs/test_p3/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/models/detection/transform.py”, line 40, in forward<br>
File “/home/user/.virtualenvs/test_p3/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/models/detection/transform.py”, line 74, in resize<br>
File “/home/user/.virtualenvs/test_p3/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/models/detection/transform.py”, line 135, in resize_boxes<br>
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)</p>
<p>Can anyone tell my how my tensors should look I want to use an empty image, using the standart Torchvision Faster RCNN?</p>
<p>Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

2e1a8551ef7fe67ff3f0: <p>I am also trying to implement the mask RCNN following this tutorial. May i ask, where did you download the references/detection/engine.py? i just can’t find it anywhere.</p>

sebo470: <p>Hi,</p>
<p>its in the torchvision github repo:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/vision/tree/15e24bdfe0ba04d728afcff005ff2e01f1da9b33/references" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/vision/tree/15e24bdfe0ba04d728afcff005ff2e01f1da9b33/references" target="_blank" rel="nofollow noopener">pytorch/vision</a></h3>

<p>Datasets, Transforms and Models specific to Computer Vision - pytorch/vision</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


nkaiser: <p>Having exactly the same question…did you find a solution yet? Does it even make sense to train on images with no object at all?</p>

Data-drone: <p>Does anyone have a working training script with the torchvision faster rcnn implementation?<br>
I am trying to train from scratch with coco but I keep getting issues with tensor sizes in the roi code.</p>

FrankZLuffy: <p>原链接在TorchVision Object Detection Finetuning Tutorial的 <a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb" rel="nofollow noopener">Colab Version</a>.中有，是在https://github.com/pytorch/vision/tree/v0.3.0/references/detection这里。<br>
不过，cocoapi更新后，有一点不兼容，会出现楼主的类似报错。<br>
我把改好的放在下面的百度网盘里了<br>
链接：<a href="https://pan.baidu.com/s/1DCw_ApFSVlcxqoCILLc2tg" rel="nofollow noopener">https://pan.baidu.com/s/1DCw_ApFSVlcxqoCILLc2tg</a><br>
提取码：jihx</p>
<p>可以试试这个。</p>

ptrblck: <p>Translated using Google Translate:</p>
<blockquote>
<p>The original link is available in the Colab Version of the TorchVision Object Detection Finetuning Tutorial at <a href="https://github.com/pytorch/vision/tree/v0.3.0/references/detection">https://github.com/pytorch/vision/tree/v0.3.0/references/detection</a>.<br>
However, after the cocoapi update, there is a bit of incompatibility, there will be a similar error from the landlord.<br>
I put the change in the Baidu network disk below.<br>
Link: <a href="https://pan.baidu.com/s/1DCw_ApFSVlcxqoCILLc2tg">https://pan.baidu.com/s/1DCw_ApFSVlcxqoCILLc2tg</a><br>
Extraction code: jihx<br>
You can try this.</p>
</blockquote>
<p><a class="mention" href="/u/frankzluffy">@FrankZLuffy</a> I would like to ask you to translate your posts before posting <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

attila: <p>I’m having the same issue. For now I’m dynamically removing images from my batches that do not contain any objects using a custom collate function but I guess it would be beneficial to include these images too in the training loop.</p>
<p>Any help would be appreciated.</p>

Seba: <p>It seems that it is not possible to use image without annotation for the training: <a href="https://github.com/pytorch/vision/issues/1128#issuecomment-512230648" rel="nofollow noopener">https://github.com/pytorch/vision/issues/1128</a><br>
I’ve tried to pass boxes of size 0 but then ‘loss_rpn_box_reg’ becomes inf.</p>

tengerye: <p>That’s because the size 0 box has width of 0, which leads to inf for torch.log(gt_widths / ex_widths).</p>
<p>I have the same question as how to deal with images without objects. Is there any standard process? Shall we 1. filter out images without desired objects; or 2. is there a smart way for that?</p>

TriEightz: <p>I used a hacky way by always including a bounding box with label “0” for background since the Pytorch Tutorial mentioned it requires a background class and a person class. Hope this helps</p>

JF7: <p>Hi <a class="mention" href="/u/trieightz">@TriEightz</a>,</p>
<p>Could you elaborate what values you added to your bouding box with label “0” for a background image ?</p>
<p>I’ve tried setting an all zero tensor for instance [0, 0, 0, 0] as a bounding box for images with no objects and have set the label as “0” but recieved the same error as <a class="mention" href="/u/seba">@Seba</a>.<br>
As tengerye pointed out, option 1 is currently being used even in detectron2.</p>
<p>Any explanation that you or anyone else could provide on how to handle negative images (i.e. images with only background and no object) would be really appreciated.</p>

TriEightz: <p><a class="mention" href="/u/jf7">@JF7</a> I set the values to be (0,1,2,3) for label “0”. so the box actually does have a size, but so far it seems to work fine as it represents the background class and doesn’t affect training.</p>

JF7: <p>Hi <a class="mention" href="/u/trieightz">@TriEightz</a>,</p>
<p>Thanks for the explanation. Much appreciated.</p>
<p>Did you happen to notice any improvement in overall performance of the model when you add label “0” only images to the training set?</p>

lgvaz: <p>What’s the current solution to this? Is the following proposed solution still the best?</p>
<aside class="quote no-group" data-username="TriEightz" data-post="13" data-topic="46935">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/57b2e6/48.png" class="avatar"> TriEightz:</div>
<blockquote>
<p>I set the values to be (0,1,2,3) for label “0”.</p>
</blockquote>
</aside>

jahandar.jahani: <p>On PyTorch 2, I did the following:</p>
<pre><code class="lang-auto">target["boxes"] = torch.as_tensor(np.array(np.zeros((0, 4)), dtype=float))
target["labels"] = torch.as_tensor(np.array([], dtype=int), dtype=torch.int64)
</code></pre>
<p>make sure after the <code>transform(img, target)</code> the <code>shape</code> and <code>dtype</code> of both arrays are right.</p>
<pre><code class="lang-auto">target["boxes"].shape = torch.Size([0, 4])
target["labels"].shape = torch.Size([0])
target["boxes"].dtype = torch.float64
target["labels"].dtype = torch.int64</code></pre>