Keypoints detection program

Sylvain_Ard: <p>Hello,<br>
I am looking for two keypoints detection programs :<br>
1 : one with one point per class<br>
2 : the other with any number of points per class<br>
Both programs must accept big images (1500x 1000 pixels) without resizing before to enter in the neural network.<br>
Thank you<br>
Best regards</p>

Sylvain_Ard: <p>noon has an answer ?</p>

ATAboukhadra: <p>Hi,</p>
<p>You can use the Keypoint RCNN from torchvision: <a href="https://pytorch.org/vision/stable/generated/torchvision.models.detection.keypointrcnn_resnet50_fpn.html" rel="noopener nofollow ugc">Keypoint RCNN</a></p>
<p>You can specify the number of keypoints that you want and in case you have multiple values for the number of keypoints you can choose the maximum possible number and when preparing the data you can append zeros in case if you have a lower number of keypoints than the maximum .</p>
<p>You can also refer to this <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">tutorial</a> to understand more about how to deal with Mask RCNN architecture and how to prepare a dataset for it including bounding boxes as they are necessary to train this model.</p>

Sylvain_Ard: <p>Thank you very much ! I will try</p>

Sylvain_Ard: <p>I saw your tutorial but it can only be one point per class and per polygon, so it answers half of the question, for example I want to detect the teeth of a leaf, so there are a variable number of points of the same class per polygon(=leaf)<br>
thank you</p>

Sylvain_Ard: <p>you don’t know a program to do this ?</p>

ATAboukhadra: <p>Hi,</p>
<p>Sorry for not replying earlier as I didn’t receive a notification.</p>
<blockquote>
<p>one point per class and per polygon</p>
</blockquote>
<p>What do you mean one point per class? You can specify whatever number of keypoints you want to estimate within each bounding box -in your case ‘a leaf’- when you call the RCNN constructor. Of course it’s the case as long as you have training data that has the same nature as the testing samples.</p>
<blockquote>
<p>there are a variable number of points of the same class per polygon(=leaf)</p>
</blockquote>
<p>As I have suggested, you can assume that every leaf has the maximum possible number of teeth and whenever it’s not the case i.e. the leaf has lower number than the maximum, you can just assume that they are there but not visible and duplicate the keypoints until you reach the maximum. At least, this approach is working fine for me.</p>

Sylvain_Ard: <p>no in COCO keypoints format there is only one keypoint per keypoint name (= class) and per polygon, if I imagine keypoints names “teeth1”, “teeth2”, etc it wouldn’t work as teeth1 and teeth2 for two leaves have no correlation</p>

ATAboukhadra: <p>Ok it’s a bit difficult for me to understand your point. Could you please describe the dataset that you have for training?<br>
In my understanding, you have a set of RGB images that contain leaves and each leaf has a number of keypoints -teeth- that varies for each leaf e.g. 20, 100, 150, 500 and so on, and each keypoint has an x,y values that correspond to the pixel location on the image.<br>
If you don’t have such dataset, then you can’t train the Keypoint RCNN model to do this task afaik.</p>

Sylvain_Ard: <p>yes I have a such dataset but what do you think I should put in keypoints_names?</p>

Sylvain_Ard: <p>What I want to say is that I have only one keypoint_name : teeth (one class) and several keypoints and if I put teeth1 teeth2 and so on in keypoints names it can’t works</p>

ATAboukhadra: <p>Why do you need names? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"><br>
Keypoint RCNN doesn’t care about names of keypoints as long as you formulate the data correctly. Please refer to this tutorial for more information. <a href="https://debuggercafe.com/human-pose-detection-using-pytorch-keypoint-rcnn/" rel="noopener nofollow ugc">https://debuggercafe.com/human-pose-detection-using-pytorch-keypoint-rcnn/</a></p>

Sylvain_Ard: <p>it does ! for example :<br>
This model has been pre-trained on the <strong><a href="https://cocodataset.org/#keypoints-2020" rel="noopener nofollow ugc">COCO Keypoint dataset</a></strong>. It outputs the keypoints for 17 human parts and body joints. They are: <em><strong>‘nose’, ‘left_eye’, ‘right_eye’, ‘left_ear’, ‘right_ear’, ‘left_shoulder’, ‘right_shoulder’, ‘left_elbow’, ‘right_elbow’, ‘left_wrist’, ‘right_wrist’, ‘left_hip’, ‘right_hip’, ‘left_knee’, ‘right_knee’, ‘left_ankle’, ‘right_ankle’</strong></em>.</p>
<p>for example left_eye is a keypoint_name</p>

ATAboukhadra: <p>This is just to identify what every keypoint represent for visualization purpose. It also helps to create edges between them. I train this model to identify 778 keypoint for the hand not knowing which is which. No need for names!</p>

Sylvain_Ard: <p>but in this case, how it recognizes which point is of what name if it does not differency them ?</p>

Sylvain_Ard: <p>I will test, how many iterations is good for training please ?</p>

ATAboukhadra: <blockquote>
<p>but in this case, how it recognizes which point is of what name if it does not differency them ?</p>
</blockquote>
<p>Depending on the order of points while training</p>
<blockquote>
<p>how many iterations is good for training please ?</p>
</blockquote>
<p>According to my experience, very few number of epochs should be sufficient. 1 epoch can be enough, more than 5 could result in overfitting.</p>

maog77: <p>Hi, did you succeed in your aim? how did you create the json annotation file?</p>

Sylvain_Ard: <p>Hi,<br>
I succeeded with one keypoint per class with mmpose but not with several keypoints per class.<br>
I used for labeling my own program : <a href="https://www.sylvain-ard.fr/Programmes/LabelingProgram%20Setup.exe" rel="noopener nofollow ugc">https://www.sylvain-ard.fr/Programmes/LabelingProgram%20Setup.exe</a></p>

maog77: <p>Thank you,<br>
Do you mean one keypoint per class in the train dataset?<br>
When you make the prediction do you find only one or all of them?</p>