All vitpose model were brokentransformers/models/vitpose_
Author: lucasjinreal
Pull request: no
Labels: name: bug

### System Info

transformers/models/vitpose_backbone/modeling_vitpose_backbone.py", line 304, in forward
    raise ValueError(transformers/models/vitpose_backbone/modeling_vitpose_backbone.py", line 304, in forward
    raise ValueError(
ValueError: dataset_index must be provided when using multiple experts (num_experts=6). Please provide dataset_index to the forward pass.
ValueError: dataset_index must be provided when using multiple experts (num_experts=6). Please provide dataset_index to the forward pass.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

transformers/models/vitpose_backbone/modeling_vitpose_backbone.py", line 304, in forward
    raise ValueError(
ValueError: dataset_index must be provided when using multiple experts (num_experts=6). Please provide dataset_index to the forward pass.

### Expected behavior

transformers/models/vitpose_backbone/modeling_vitpose_backbone.py", line 304, in forward
    raise ValueError(
ValueError: dataset_index must be provided when using multiple experts (num_experts=6). Please provide dataset_index to the forward pass.

Comments:
Can you share a script to reproduce this error?

Since you're using an MoE checkpoint with `num_experts=6`, you can specify a default expert index and pass that value for the entire batch when calling the model. For example:

```
from transformers import AutoImageProcessor, VitPoseForPoseEstimation
import torch
from PIL import Image
import requests

processor = AutoImageProcessor.from_pretrained("usyd-community/vitpose-plus-small")
model = VitPoseForPoseEstimation.from_pretrained("usyd-community/vitpose-plus-small")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
boxes = [[[412.8, 157.61, 53.05, 138.01], [384.43, 172.21, 15.12, 35.74]]]
inputs = processor(image, boxes=boxes, return_tensors="pt")
expert_index = 0
inputs['dataset_index'] = torch.full((inputs['pixel_values'].shape[0],), expert_index, dtype=torch.long)

with torch.no_grad():
  outputs = model(**inputs)
```

Hi, does this work?

```
 # ViTPose++ has 6 different MoE expert heads 
        # (COCO validation 0, AiC 1, MPII 2, AP-10K 3, APT-36K 4, COCO-WholeBody 5) 
        # which supports 6 different datasets.
        # dataset_index = torch.tensor([0]).to(self.device)
        dataset_index = torch.tensor([5]).to(self.device)

        with torch.no_grad():
            outputs = self.pose_model(**inputs, dataset_index=dataset_index)
```

Hi @lucasjinreal @junjunjd @i3hz, I have reproduced the issue and implemented a fix. The PR makes `dataset_index` optional with a default value of 0 (COCO expert). Waiting on your review!

Hi @rukundob451, I'm not sure this is actually a bug! We don't want to default to the COCO expert. @lucasjinreal's solution of actually passing the dataset index seems correct.

@Rocketknight1 I have noted this, thank you.

Hi, actually i tried only coco expert works, wholebody result seems not right.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Hi, actually i tried only coco expert works, wholebody result seems not right.

Hi @lucasjinreal, so what is the way forward?

Hi - would like to add on that I also am having problems getting the wholebody result to work, here is my code, which is mostly the sample code provided, with the additional image annotation code underneath:
```
import torch
import requests
import numpy as np

from PIL import Image

from transformers import (
    AutoProcessor,
    RTDetrForObjectDetection,
    VitPoseForPoseEstimation,
)
import torch
import requests
import numpy as np

from PIL import Image

from transformers import (
    AutoProcessor,
    RTDetrForObjectDetection,
    VitPoseForPoseEstimation,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

url = "http://images.cocodataset.org/val2017/000000000139.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# ------------------------------------------------------------------------
# Stage 1. Detect humans on the image
# ------------------------------------------------------------------------

# You can choose detector by your choice
person_image_processor = AutoProcessor.from_pretrained("PekingU/rtdetr_r50vd_coco_o365")
person_model = RTDetrForObjectDetection.from_pretrained("PekingU/rtdetr_r50vd_coco_o365", device_map=device)

inputs = person_image_processor(images=image, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = person_model(**inputs)

results = person_image_processor.post_process_object_detection(
    outputs, target_sizes=torch.tensor([(image.height, image.width)])
)
result = results[0]  # take first image results

# Human label refers 0 index in COCO dataset
person_boxes = result["boxes"][result["labels"] == 0]
person_boxes = person_boxes.cpu().numpy()

# Convert boxes from VOC (x1, y1, x2, y2) to COCO (x1, y1, w, h) format
person_boxes[:, 2] = person_boxes[:, 2] - person_boxes[:, 0]
person_boxes[:, 3] = person_boxes[:, 3] - person_boxes[:, 1]

# ------------------------------------------------------------------------
# Stage 2. Detect keypoints for each person found
# ------------------------------------------------------------------------

image_processor = AutoProcessor.from_pretrained("usyd-community/vitpose-plus-base")
model = VitPoseForPoseEstimation.from_pretrained("usyd-community/vitpose-plus-base", device_map=device)

inputs = image_processor(image, boxes=[person_boxes], return_tensors="pt").to(device)

# This is MOE architecture, we should specify dataset indexes for each image in range 0..5
inputs["dataset_index"] = torch.tensor([5]* len(person_boxes), device=device)

with torch.no_grad():
    outputs = model(**inputs)

pose_results = image_processor.post_process_pose_estimation(outputs, boxes=[person_boxes])
image_pose_result = pose_results[0]  # results for first image

for i, person_pose in enumerate(image_pose_result):
    print(f"Person #{i}")
    for keypoint, label, score in zip(
        person_pose["keypoints"], person_pose["labels"], person_pose["scores"]
    ):
        keypoint_name = model.config.id2label[label.item()]
        x, y = keypoint
        print(f" - {keypoint_name}: x={x.item():.2f}, y={y.item():.2f}, score={score.item():.2f}")


import supervision as sv

# After running inference and getting pose_results...
xy = torch.stack([pose_result['keypoints'] for pose_result in image_pose_result]).cpu().numpy()
scores = torch.stack([pose_result['scores'] for pose_result in image_pose_result]).cpu().numpy()

key_points = sv.KeyPoints(xy=xy, confidence=scores)

edge_annotator = sv.EdgeAnnotator(color=sv.Color.GREEN, thickness=1)
vertex_annotator = sv.VertexAnnotator(color=sv.Color.RED, radius=2)

annotated_frame = edge_annotator.annotate(scene=image.copy(), key_points=key_points)
annotated_frame = vertex_annotator.annotate(scene=annotated_frame, key_points=key_points)

annotated_frame.show()  
```

When the dataset_index is 0, I get the correct result:

<img width="942" height="629" alt="Image" src="https://github.com/user-attachments/assets/61662f09-a479-4743-a794-ea73091e0136" />

Dataset_index of 5 however (which I believe is Wholebody?), looks like this:

<img width="637" height="424" alt="Image" src="https://github.com/user-attachments/assets/7b977eb0-8e2d-4d1e-89e8-185a17aa2a3e" />

Have tried it on different images where the person is more clear as well and the results are the same, index 0 works well, index 5 does not

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.