Add ViTPose
Author: NielsRogge
Pull request: yes
Labels: name: New model

name: Vision

name: run-slow

# What does this PR do?

This PR adds ViTPose as introduced in [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://arxiv.org/abs/2204.12484).

Here's a demo notebook - note that the API might change: 
 - https://colab.research.google.com/drive/15_3gjcC0wtKSH85k76zewt81eUJIEWWA?usp=sharing.
 - https://colab.research.google.com/drive/1e8fcby5rhKZWcr9LSN8mNbQ0TU4Dxxpo?usp=sharing (supervision visualization)

To do:

- [x] get rid of cv2 dependency (?)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30530). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@NielsRogge Hi Niels! Does this current PR works properly? (I want to do some test for this)

Thanks for working on this @NielsRogge! 

I can see there's a few bits unfinished e.g. tests. Is there a particular bit of code you'd like me to look at for a maintainers perspective? 

For the issue description, there's a related model request is here: #24915

@NielsRogge I'm unsubscribing atm, so that I don't get notifications on every new push. You just need to ping me again with my username when it's ready for review and I'll get notified 

It would be great to have a first round of review as the PR is in a ready state. @amyeroberts 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@amyeroberts this implementation requires preprocessing functions such as `cv2.warpAffine` and `cv2.GaussianBlur` which do not have a non-cv2 equivalent implementation. Shoud we proceed with adding a `requires_cv2` dependency checker?

> which do not have a non-cv2 equivalent implementation

An affine transformation is just a linear one, so it should be easy for us to add a numpy equivalent. OpenCV have an example on how to add in numpy: https://theailearner.com/tag/cv2-warpaffine/

For gaussian blur, I think there's a scipy guassian filter we could use. 

Another equivalent is just adding the [fast image processor equivalent](https://github.com/huggingface/transformers/pull/28847#issuecomment-1924674183) for the model and using torchvision. 

We should try to avoid adding another vision library dependency as much as possible 

[Inference_with_ViTPose_for_human_pose_estimation_ipynb.ipynb.zip](https://github.com/user-attachments/files/16478046/Inference_with_ViTPose_for_human_pose_estimation_ipynb.ipynb.zip)

@amyeroberts Hi amy this is the overall descriptions easy for you to look in major change (Skipped explaining minor changes like typo.)
1. https://github.com/huggingface/transformers/pull/30530/commits/1b439e20be2c07527ea89a54e4e0a749facad6be 
- Made the function to accept more variable
3. https://github.com/huggingface/transformers/pull/30530/commits/bbd534c53a9afc335875d5a142372a3728303d7d 
- Add more test, originally the `prepare_img` was just for the cat image which is not appropriate for this model (Since it is human pose estimation). Changed to human image in COCO. 
- Also made the output return of `post_process` result as tensor.

IMO, we can some more testing function if it is needed.

cc. @NielsRogge For additional review request

@amyeroberts @NielsRogge Ping for the reminder

@NielsRogge Requesting for second review (Need some answer for 2-3 questions)

Looking forward to this
@SangbumChoi thanks for doing this
@amyeroberts you're a great code reviewer 

@amyeroberts @NielsRogge Fortunately, every comments from both review was clear enough me to understand and modify to appropriate way. (Also fixed ViT -> Vit) Asking for second final review :)



current CI fail is unrelated for this PR

@amyeroberts Hi amy thanks again for the detail review. I have addressed all the suggestions except for removing `MoE`. We can remove the MoE logic but it is designed in the original code. Do we really need to remove it? Otherwise it is good to go.

cc. @qubvel @NielsRogge For to request PR review instead of amy

@SangbumChoi sorry for the delay, I will review this week!

@qubvel @NielsRogge Requesting for the final review :)
I left some comment you should look on.
1. https://github.com/huggingface/transformers/pull/30530#discussion_r1788991621
2. https://github.com/huggingface/transformers/pull/30530#discussion_r1807240536

@qubvel Hi pavel I finished suggestion without this comment https://github.com/huggingface/transformers/pull/30530#discussion_r1822397840.

Which I suggest the idea if we remove the `show_keypoint_weight` option which is not in supervision library. Pure code itself will be a lot easier. I prefer to give user advanced version of documentation, but I can erase that argument if it seems to complex. WDYT?

@qubvel Very long journey :) Can you review and see the changes of supervision itself?

@qubvel Soft reminder, (cc. @NielsRogge )

Hey @SangbumChoi we were on a company wide offsite for a week, @qubvel should answer soon! 🤗 

@qubvel All addressed! 

@xenova Hi I think it is deprecated notebook from niels. (Overall pipeline is same) Can you see the docs? If you run the docs then it will occur `edge` related problem which will be solved when we re-upload official weights.


> I suggest the following format for each box:
> 
> ```python
> {
>     "keypoints": np.ndarray of shape (m, 2),
>     "scores": np.ndarray of shape (m,),
>     "labels": np.ndarray of shape (m,),  (class ids)
>     "bbox": np.ndarray of shape (4,)
> }
> ```

100% agree - much better imo :)

Also, @qubvel, do you think we should introduce a `AutoModelForPoseEstimation` or `AutoModelForKeypointDetection` class for this?

@qubvel @xenova I agree with the schema but I think I have seen some comment from amy that the output should be tensor. numpy is okay too?

 In addition labels are equivalent to config.keypoint so I think it could be removed?

Additionally, I think it would make sense to include a `threshold` parameter to filter out low-scoring predictions (like [here](https://github.com/huggingface/transformers/blob/ce1d328e3b73cf6d1d993fc0d487b7dc8a14d7ee/src/transformers/models/owlv2/image_processing_owlv2.py#L470)). Another reason for this is that otherwise, `labels` would be meaningless (it would always return all labels, e.g., 0-16)

> do you think we should introduce a AutoModelForPoseEstimation or AutoModelForKeypointDetection class for this?

@xenova as soon as we have just one model for pose estimation I don't think it is necessary, what do you think?

>  I agree with the schema but I think I have seen some comment from amy that the output should be tensor. numpy is okay too?

@SangbumChoi my bad, indeed we usually return torch tensors, so no need to change this! Thanks for pointing this 🤗 

> In addition labels are equivalent to config.keypoint so I think it could be removed?

@SangbumChoi I think it's better to keep labels in the output too, it would be more obvious for users then. Also, I think one can consider threshold filtering based on confidence, labels would be useful for such cases to understand which keypoints are kept.

> Additionally, I think it would make sense to include a threshold parameter to filter out low-scoring predictions (like [here](https://github.com/huggingface/transformers/blob/ce1d328e3b73cf6d1d993fc0d487b7dc8a14d7ee/src/transformers/models/owlv2/image_processing_owlv2.py#L470)). Another reason for this is that otherwise, labels would be meaningless (it would always return all labels, e.g., 0-16)

well this is top-down keypoint estimation so it should return all prediction. Traditionally in general pose estimation algorithm, there is an additional argument called `hide` usually used if the corresponding point is behind or occluded by object but I don't see in this model.

> Also, I think one can consider threshold filtering based on confidence, labels would be useful for such cases to understand which keypoints are kept.

I will add threshold but default value as 0. WDYT?

> @xenova as soon as we have just one model for pose estimation I don't think it is necessary, what do you think?

That's fair 👍 I mainly asked to add better ONNX export support in Optimum; see PR [here](https://github.com/huggingface/optimum/pull/2098/files#diff-e796447e084c9988c5dcb85a562008bdac1d61f7dd9a7c531bca22306a36a2f9R215). Once we have more similar models, it would make more sense to add support then! :)

The export works as is, luckily. Here is the ONNX checkpoint (& transformers.js usage): https://huggingface.co/onnx-community/vitpose-base-simple

> I will add threshold but default value as 0. WDYT?

Sounds good. Just double check something: Can the scores be < 0? I've seen scores > 1. So maybe set to `None`?

@qubvel Can you re-upload the weight file to official repo (where it belongs to) based on current conversion script? (If there is error then I have to fix it). I want to make sure the docs has no issue after the conversion which includes configuration for visualize.

@qubvel All set :)

Hi @SangbumChoi, thanks for updating the output format and corresponding code!

Some slow tests fail in CI, can you please have a look?
```
FAILED tests/models/vitpose/test_modeling_vitpose.py::VitPoseModelIntegrationTest::test_batched_inference - RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
FAILED tests/models/vitpose/test_modeling_vitpose.py::VitPoseModelIntegrationTest::test_inference_pose_estimation - RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
```
----

> @qubvel Can you re-upload the weight file to official repo (where it belongs to) based on current conversion script? (If there is error then I have to fix it). I want to make sure the docs has no issue after the conversion which includes configuration for visualize.

Not sure I got it. It is better to upload all weights to your personal account first, to be able to create and modify model cards. Then we can transfer them to the organization account (https://huggingface.co/ViTAETransformer ?) and update the code references.

Anyway I tried to convert models locally: `vitpose-base` and `vitpose-base-simple` go smoothly, for `vitpose+-base` got the error of mismatched weights

```
Unexpected key(s) in state_dict: "associate_heads.0.deconv1.weight", "associate_heads.0.batchnorm1.weight", "associate_heads.0.batchnorm1.bias", "associate_heads.0.batchnorm1.running_mean", "associate_heads.0.batchnorm1.running_var", "associate_heads.0.batchnorm1.num_batches_tracked", "associate_heads.0.deconv2.weight", "associate_heads.0.batchnorm2.weight", "associate_heads.0.batchnorm2.bias", 
...
```

Also, I suggest to use `vitpose_plus_base` for model checkpoint instead of `+` sign in name



@qubvel Sorry for the inconvinience, the reason of failing was due to processing output + output cpu/gpu format problem.

 In addition conversion of other two model was getting unnecessary weight kind of `auxiliary` weight. https://github.com/huggingface/transformers/pull/30530/commits/78fe1b9c52d659facefdb9e06518ba45e9298ca4
 
 Test seems irrelevant and https://huggingface.co/danelcsb/vitpose-plus-base I will start writing readme

Thanks for fixing, one more test failed for multi-gpu setup (seems relevant)

```
FAILED tests/models/vitpose_backbone/test_modeling_vitpose_backbone.py::VitPoseBackboneModelTest::test_model_parallelism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
```

Added colab notebook to the first message with supervision visualization

@qubvel Hi Pavel, before I fix the last CI I have one fundamental question of 

>FAILED tests/models/vitpose_backbone/test_modeling_vitpose_backbone.py::VitPoseBackboneModelTest::test_model_parallelism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

I think the reason of failing this model_parallelism is due to

> second residual connection
layer_output = layer_output + hidden_states
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

While we can use the statement `layer_output = layer_output + hidden_states.to(layer_output.dtype)` to address immediate type compatibility issues, this approach doesn't fully resolve potential device mismatch problems that could arise in a model parallel architecture. The size and configuration of model parallelism can vary, which means device discrepancies could occur at different points in the model.
This solution appears to be a temporary fix to pass Continuous Integration (CI) tests, rather than a fundamental resolution to the underlying architectural challenge. It doesn't comprehensively address the potential issues that could emerge in various model parallel scenarios. Did I understand correctly? we can just skip this part but I just want to make sure is there anything that I misunderstand?

Hi @SangbumChoi, I suppose `_no_spit_modules` is designed to avoid this situation. As I see it's defined for the backbone, but might not be properly handled when the backbone is a part of the model.

@qubvel Now it is ready to review :)
https://huggingface.co/danelcsb/vitpose-base-simple
currently updating REDME.md here

@ArthurZucker please review whenever you have time!

The notebook to run: https://colab.research.google.com/drive/1e8fcby5rhKZWcr9LSN8mNbQ0TU4Dxxpo?usp=sharing

Checkpoints:
https://huggingface.co/danelcsb/vitpose-base-simple
https://huggingface.co/danelcsb/vitpose-base
https://huggingface.co/danelcsb/vitpose-base-coco-aic-mpii
https://huggingface.co/danelcsb/vitpose-plus-base

@ArthurZucker Reminder ping :)

@ArthurZucker Thanks for the review, I will change the thumbs-up reaction. For the question of backbone part, I think there is no necessary reason for dividing into two independent architecture (This work was done by Niels at the first and he designed this part). 

However, I assume that this is because of the intention of the author and Niels. This paper is aimed for human pose estimation based on very strong common backbone and just switching last MLP-like (e.g. head) layer for architecture efficiency to different task and datasets. For instance you can see that many model share same backbone architecture. So I think that this is why Niels build it this way. (@NielsRogge If there is a specific reason for dividing backbone architecture?)

https://github.com/facebookresearch/mae
https://github.com/ViTAE-Transformer/ViTPose?tab=readme-ov-file

It's mainly because ViTPose itself is a framework which could leverage various different backbones, so I made it compatible with the `AutoBackbone` API (similar to frameworks like DETR, DPT and Mask2Former). See here for example usage: https://huggingface.co/docs/transformers/en/model_doc/dpt#usage-tips

@qubvel @NielsRogge Hi can you check the changes and also the checkpoints? Always thanks for the support.

Hi @SangbumChoi! Thanks for the fixes, I will push the final fixes and merge the PR!

Checkpoints can be found here:
https://huggingface.co/usyd-community