Can I feed a model with some "background-only" images?

Michael-J98: <p>It just appears in my mind that if I’m working on an object detection task, and there are no objects in some of the training images, can this will work? If can, how does it work?</p>

KFrank: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Michael-J98" data-post="1" data-topic="76279" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael-j98/48/20536_2.png" class="avatar"> Michael-J98:</div>
<blockquote>
<p>there are no objects in some of the training images, can this will work?</p>
</blockquote>
</aside>
<p>Yes, but how best to do this will depend on the overall structure of<br>
your task.</p>
<p>First, to me, “object detection” means finding one or more objects in<br>
an image (and reporting their locations, e.g., their bounding boxes).<br>
This is often done by sliding a series “detection” windows across the<br>
image and then running the sub-image in such a window through an<br>
object classifier.  I won’t say more about such full-scale object detectors<br>
(used in this sense) other than to note that they are built out of smaller<br>
pieces.</p>
<p>If you have a three-class classification problem – e.g., this image<br>
contains either a “person” or a “dog” or a "“shrubbery” – then, in<br>
order to be able to accept “background-only” images, you would<br>
add a fourth class, “background,” and train your classifier in the<br>
usual way with all four classes.</p>
<p>If you have multi-label, three-class classification problem – e.g., this<br>
image does or does not contain a “person” and also does or does<br>
not contain a “dog” and also does or does not contain a “shrubbery”<br>
(that is, your image can be “true” for any number of your three classes,<br>
including for none of them or all of them) – then you don’t have to do<br>
anything additional to account for training images containing no<br>
objects (other than to make sure that they are labelled correctly).</p>
<p>Best.</p>
<p>K. Frank</p>

Michael-J98: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="76279">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If you have multi-label, three-class classification problem – e.g., this<br>
image does or does not contain a “person” and also does or does<br>
not contain a “dog” and also does or does not contain a “shrubbery”<br>
(that is, your image can be “true” for any number of your three classes,<br>
including for none of them or all of them) – then you don’t have to do<br>
anything additional to account for training images containing no<br>
objects (other than to make sure that they are labelled correctly)</p>
</blockquote>
</aside>
<p>Thanks a lot for your delicate reply~~ One more question about the implement detail, that should I annotate the “background-only” images? Or in other words, how does the “ground-truth” look like? An empty bbox list for object detection task? An “all-zeros” mask for segmentation task？Or whatsoever?</p>

KFrank: <p>Hello Michael!</p>
<aside class="quote no-group" data-username="Michael-J98" data-post="3" data-topic="76279" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael-j98/48/20536_2.png" class="avatar"> Michael-J98:</div>
<blockquote>
<p>how does the “ground-truth” look like? An empty bbox list for object detection task? An “all-zeros” mask for segmentation task？Or whatsoever?</p>
</blockquote>
</aside>
<p>It really depends on the specific problem you are trying to solve.</p>
<p>Are you working on a single-image multi-class classifier?  Are<br>
you working on image segmentation?  We can make suggestions<br>
about such implementation details if you give us some concrete<br>
details about the specific model you are trying to implement.</p>
<p>Best.</p>
<p>K. Frank</p>

Michael-J98: <p>Well, okay, that’s my bad. (Actually, as I said at the very beginning, it just popped in my mind. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:">)<br>
We can suppose some scenarios:</p>
<ol>
<li>I’m trying to locate all cats and dogs in one image only with bboxes using faster-rcnn.</li>
<li>I’m trying to segment all cars and persons in one image with masks using deeplabv3.</li>
<li>I"m working on an instance segmentation using mrcnn.<br>
Really grateful to your patience. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:">
</li>
</ol>

Alieladi: <p>Hello Michael, It is possible to do that.<br>
For your example 1:<br>
In your dataset <strong>getitem</strong> method, you could set for images with background only:</p>
<pre><code class="lang-auto">box_count = 0
bboxes = np.array([]).reshape(-1, 4)
</code></pre>
<p>And convert them like any other image:</p>
<pre><code class="lang-auto">bboxes = torch.as_tensor(bboxes, dtype=torch.float32)
labels = torch.ones((box_count,), dtype=torch.int64)
</code></pre>

Michael-J98: <p>Okay~ thanks, I’ll try it.</p>