Keypoint RCNN visibility flag for keypoints

username: mb1996
cooked: <p>Hello All,</p>
<p>This is only my first day posting on this forum, though I have visited many times over the past 3+ years of using Pytorch, and benefited from the questions and answers here. I apologize for the long post below.</p>
<p>I am fine-tuning Keypoint RCNN on my custom dataset and, using resources I have found in various places, am successfully training and obtaining bounding box and keypoint predictions on multiple classes together with very good results. Due to my use case, I am in need of employing the keypoint visibility flag which the model is meant to support. According to the model’s github page, this is supposed to be the same flag convention as the COCO dataset (v=0 ‘not in image’; v=1 ‘occluded’; v=2 ‘visible’) but it turns out that Keypoint RCNN does not strictly follow this convention in practice. Based on my results and those of other users I have encountered on different forums and sites, Keypoint RCNN always predicts a flag value of v=1 for all keypoints, no matter the training flag v&gt;0 (even v=0), and predicts coordinates for them as well. From what I can gather the model is supposed to exclude from loss calculation any keypoints with flag v=0, which it may be doing. But still goes on to predict v=1 and nonzero coordinates for them later. I have tried training such keypoints using coordinates (0.0,0.0) as found in COCO but with no success.</p>
<p>My hope in writing here is to get some help in tracking down the source code where Keypoint RCNN is converting all flags to v=1 and handling flags so that I might be able to modify this behavior, as the model does not seem to learn the flag values presently. In my use case, what I want is for Keypoint RCNN to successfully predict the right flag (e.g. v=0) so that I can use it later on, or at least predict a coordinate of (0.0,0.0) (or some other fixed value) for keypoints with v=0. The need is to be able to distinguish between visible and occluded keypoints. Even just two learned flags that work as expected (v=0 and v=1) would be very useful to have. Any suggestions or guidance would be great. I wonder if it is reasonable to make such a request on the github page, hoping that the feature can be added to the model? Thanks for taking the time to reply.   <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Sincerely, mb1996</p>

username: mb1996
cooked: <p>Dear Pytorch Community,</p>
<p>For anyone interested in an update on this topic, please see the below link for a github conversation I started with the Pytorch developers regarding the keypoint visibility flag behavior of Keypoint RCNN. Some additional helpful information can be found there. Hopefully, further dialogue will continue, and maybe even with progress on a solution.</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/vision/issues/5872">
  <header class="source">

      <a href="https://github.com/pytorch/vision/issues/5872" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/vision</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/vision/issues/5872" target="_blank" rel="noopener nofollow ugc">Keypoint RCNN visibility flag for keypoints</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-04-24" data-time="21:44:35" data-timezone="UTC">09:44PM - 24 Apr 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/mbadal1996" target="_blank" rel="noopener nofollow ugc">
          <img alt="mbadal1996" src="https://avatars.githubusercontent.com/u/87787985?v=4" class="onebox-avatar-inline" width="20" height="20">
          mbadal1996
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          question
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          topic: object detection
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🚀 The feature

Hello All,

This is only my first day posting a request her<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">e so I apologize for any errors on my part. Also, sorry for the long post below.

The purpose of this post is to request an improvement/correction for the visibility flag behavior of Keypoint RCNN. Based on my results and those of other users I have encountered on different forums and sites, Keypoint RCNN always predicts a flag value of v=1 for all keypoints, no matter the training flag value for v&gt;0 (even v=0), and predicts coordinates for them as well. In other words, the model does not appear to actually learn the flag value. My understanding is that the flag should be learned and is supposed to follow the COCO convention (v=0 ‘not in image’; v=1 ‘occluded’; v=2 ‘visible’) but does not do so.






### Motivation, pitch

Given the usefulness of the visibility flags, being able to accurately predict them and use the information during inference to mark occluded vs. visible keypoints would be an important addition to the model capability. My understanding is that this is already supposed to be the case, but for some reason the documentation as well as the model behavior on this are lacking. I have found the performance of Keypoint RCNN overall to be very good and I have successfully fine-tuned it on my custom (multiclass) dataset with very good success in predicting the class, bbox, and keypoints. It would be very helpful to be able to distinguish between keypoints using visibility flag.  

### Alternatives

_No response_

### Additional context

My hope in writing here is to request and encourage updating of the model to address the issue/addition suggested. If not, then if I could please get some help in tracking down the source code where Keypoint RCNN is converting all flags to v=1 and handling/training flags so that I might be able to modify this behavior, as the model does not seem to learn the flag values presently. In my use case, what I want is for Keypoint RCNN to successfully predict the right flag (e.g. v=0) so that I can use it later on, or at least predict a coordinate of (0.0,0.0) (or some other fixed value) for keypoints with v=0. The need is to be able to distinguish between visible and occluded keypoints. Even just two learned flags that work as expected (v=0 and v=1) would be very useful to have. Any suggestions or guidance would be great. Thanks for taking the time to reply.

cc @datumbox @YosuaMichael</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Sincerely, mb1996</p>