Faster RCNN Evaluation Doesn't Perform Well

David_Valensi: <p>Hi there,<br>
I’m fine-tuning Faster R-CNN on my custom dataset using the official PyTorch tutorial about fine-tuning object detection models. For an unknown reason the model succeeds in learning how to detect the objects of my dataset but the mean average precision is always 0. I know that the model succeeds in doing so because I checked the outputs of model during evaluation and I saw that the predicted bounding boxes match quite well with the ground truth bounding boxes. My conclusion is that the calculation of the evaluation metrics doesn’t work and I’d like to know why.</p>
<p>Firstly, here’s what I see in the console while training and evaluating for two epochs:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/7543e41512e6a40f38aec6d0389a40358326cd18.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7543e41512e6a40f38aec6d0389a40358326cd18" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/7543e41512e6a40f38aec6d0389a40358326cd18_2_690x251.jpeg" alt="image" data-base62-sha1="gJnjP1lLUuKg9QAHfziKKHRqRUk" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/7543e41512e6a40f38aec6d0389a40358326cd18_2_690x251.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/5/7543e41512e6a40f38aec6d0389a40358326cd18.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/5/7543e41512e6a40f38aec6d0389a40358326cd18.jpeg 2x" data-dominant-color="E6E6E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">975×355 262 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d13d0bf8de7a92aab42b10418a5b369aaa718ae4.png" data-download-href="https://discuss.pytorch.org/uploads/default/d13d0bf8de7a92aab42b10418a5b369aaa718ae4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d13d0bf8de7a92aab42b10418a5b369aaa718ae4_2_690x101.png" alt="image" data-base62-sha1="tR0AaEEHpf0kv8ZihWCMfiiG6Lq" width="690" height="101" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d13d0bf8de7a92aab42b10418a5b369aaa718ae4_2_690x101.png, https://discuss.pytorch.org/uploads/default/original/3X/d/1/d13d0bf8de7a92aab42b10418a5b369aaa718ae4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/1/d13d0bf8de7a92aab42b10418a5b369aaa718ae4.png 2x" data-dominant-color="E7E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">975×144 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78df6257f1113beab28d3b3ec2f28d1767eb8d12.png" data-download-href="https://discuss.pytorch.org/uploads/default/78df6257f1113beab28d3b3ec2f28d1767eb8d12" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78df6257f1113beab28d3b3ec2f28d1767eb8d12_2_690x259.png" alt="image" data-base62-sha1="hfhTtBgKAZLgPz6GyomQNG66CSS" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78df6257f1113beab28d3b3ec2f28d1767eb8d12_2_690x259.png, https://discuss.pytorch.org/uploads/default/original/3X/7/8/78df6257f1113beab28d3b3ec2f28d1767eb8d12.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/8/78df6257f1113beab28d3b3ec2f28d1767eb8d12.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">975×366 154 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/234f526d54165e447c3b6ff4f76f37124134965e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/234f526d54165e447c3b6ff4f76f37124134965e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234f526d54165e447c3b6ff4f76f37124134965e_2_690x239.jpeg" alt="image" data-base62-sha1="52mENrVIKP0enTsY5CdH08yc9Jk" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234f526d54165e447c3b6ff4f76f37124134965e_2_690x239.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/3/234f526d54165e447c3b6ff4f76f37124134965e.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/234f526d54165e447c3b6ff4f76f37124134965e.jpeg 2x" data-dominant-color="E5E6E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">975×338 250 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b416aabb0a95eca02d7ec41eb56904099078099f.png" data-download-href="https://discuss.pytorch.org/uploads/default/b416aabb0a95eca02d7ec41eb56904099078099f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b416aabb0a95eca02d7ec41eb56904099078099f_2_690x111.png" alt="image" data-base62-sha1="pH8vw0cxbKDlDJQtYhDuXOB7iUL" width="690" height="111" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b416aabb0a95eca02d7ec41eb56904099078099f_2_690x111.png, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b416aabb0a95eca02d7ec41eb56904099078099f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b416aabb0a95eca02d7ec41eb56904099078099f.png 2x" data-dominant-color="E7E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">975×157 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88735d0bb96b0265b32c948b9e2f6f6cd519daf2.png" data-download-href="https://discuss.pytorch.org/uploads/default/88735d0bb96b0265b32c948b9e2f6f6cd519daf2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88735d0bb96b0265b32c948b9e2f6f6cd519daf2_2_690x266.png" alt="image" data-base62-sha1="jt66gs7Df4tEfzVqZ8mdeNcZWCu" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88735d0bb96b0265b32c948b9e2f6f6cd519daf2_2_690x266.png, https://discuss.pytorch.org/uploads/default/original/3X/8/8/88735d0bb96b0265b32c948b9e2f6f6cd519daf2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/88735d0bb96b0265b32c948b9e2f6f6cd519daf2.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">975×377 159 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Secondly, here are some hyperparameters of my training:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/6350ce61717101fcb9079e4a89c29e78bb23547d.png" data-download-href="https://discuss.pytorch.org/uploads/default/6350ce61717101fcb9079e4a89c29e78bb23547d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/6350ce61717101fcb9079e4a89c29e78bb23547d_2_690x430.png" alt="image" data-base62-sha1="eaAoyeYKerhys5EAHEDaSMH2IDX" width="690" height="430" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/6350ce61717101fcb9079e4a89c29e78bb23547d_2_690x430.png, https://discuss.pytorch.org/uploads/default/original/3X/6/3/6350ce61717101fcb9079e4a89c29e78bb23547d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/3/6350ce61717101fcb9079e4a89c29e78bb23547d.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">875×546 47.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here’s the evaluation function that I use:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32c5c31e371eb89190d5aa5f81b3338209d84525.png" data-download-href="https://discuss.pytorch.org/uploads/default/32c5c31e371eb89190d5aa5f81b3338209d84525" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/32c5c31e371eb89190d5aa5f81b3338209d84525_2_616x500.png" alt="image" data-base62-sha1="7f9zM8rPCzHPDyjuUsQPxeXaxKt" width="616" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/32c5c31e371eb89190d5aa5f81b3338209d84525_2_616x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/32c5c31e371eb89190d5aa5f81b3338209d84525_2_924x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/2/32c5c31e371eb89190d5aa5f81b3338209d84525.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">951×771 69.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c20787bb9e6822006da90da31211d6d37d5382c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/c20787bb9e6822006da90da31211d6d37d5382c6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c20787bb9e6822006da90da31211d6d37d5382c6_2_690x290.png" alt="image" data-base62-sha1="rGsLhFj7Pk33kmYuANsU8wytZHM" width="690" height="290" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c20787bb9e6822006da90da31211d6d37d5382c6_2_690x290.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c20787bb9e6822006da90da31211d6d37d5382c6_2_1035x435.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c20787bb9e6822006da90da31211d6d37d5382c6.png 2x" data-dominant-color="F9F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1169×493 51.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here are the ground truth labels in the evaluation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba2c59c763107063b6685cb30a3d8356ab8254a2.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba2c59c763107063b6685cb30a3d8356ab8254a2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba2c59c763107063b6685cb30a3d8356ab8254a2_2_690x256.png" alt="image" data-base62-sha1="qyXPucqsXmEv0GxXRVYRQkjG6Yy" width="690" height="256" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba2c59c763107063b6685cb30a3d8356ab8254a2_2_690x256.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba2c59c763107063b6685cb30a3d8356ab8254a2_2_1035x384.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba2c59c763107063b6685cb30a3d8356ab8254a2.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1193×443 82.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could anyone guide me in finding the reason for the which the evaluation metrics are always zero while the model succeeds in finding the bounding boxes?<br>
In addition, does anyone know if it makes sense that the box regression loss barely improves during the training?</p>
<p>David.</p>

David_Valensi: <p>Just for all of you to know, I fixed the problem. The problem was that I assigned all the samples a label of 0 while they should have been 1. The predictions were with a label of 1, which is the only non-background label in my dataset. When the average precision was calculated the IoU was always 0 and that’s why the average precision was also 0.</p>

Skylar_White: <p>hi, i just wanna ask, can use CocoEvaluator on GPU? or just only on CPU?.</p>
<p>and also can u share full code, thanks</p>