Fine tuning SAM, 'The input_points must be a 3D tensor. Of shape `batch_size`, `nb_boxes`, `4`.', ' got torch.Size([2, 4]).'
Author: leemorton
Pull request: no

Hi,

I am trying to fine tune SAM on custom images and masks but am struggling and am hoping someone can point me in the right direction to resolving it.

I have been referencing this code:
https://github.com/bnsreenu/python_for_microscopists/blob/master/331_fine_tune_SAM_mito.ipynb
which is based on this:
https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine_tune_SAM_(segment_anything)_on_a_custom_dataset.ipynb

I cannot get the training to work as I get this message at the forward pass step:
`'The input_points must be a 3D tensor. Of shape `batch_size`, `nb_boxes`, `4`.', ' got torch.Size([2, 4]).'`

I think the input_boxes is wrong somehow?

![image](https://github.com/user-attachments/assets/6033863d-65b6-43db-99ea-1cab9be4dd5e)

The images I am using are colour PNG images rather than the tiff images in the reference code and are showing with 3 channels here....
![image](https://github.com/user-attachments/assets/1f9527e7-c4f2-46c6-9a12-1bd678d920c0)

My SamDataset code is:

```python
class SAMDataset(Dataset):
  """
  This class is used to create a dataset that serves input images and masks.
  It takes a dataset and a processor as input and overrides the __len__ and __getitem__ methods of the Dataset class.
  """
  def __init__(self, dataset, processor):
    self.dataset = dataset
    self.processor = processor

  def __len__(self):
    return len(self.dataset)

  def __getitem__(self, idx):
    item = self.dataset[idx]
    image = item["image"]
    ground_truth_mask = np.array(item["label"])
    
    # get bounding box prompt
    # prompt = get_bounding_box(ground_truth_mask)
    prompt = item["bounding_box"]

    # prepare image and prompt for the model
    inputs = self.processor(image, input_boxes=[[prompt]], return_tensors="pt")

    # remove batch dimension which the processor adds by default
    inputs = {k:v.squeeze(0) for k,v in inputs.items()}

    # add ground truth segmentation
    inputs["ground_truth_mask"] = ground_truth_mask

    return inputs
```

and this is where I run into trouble...
![image](https://github.com/user-attachments/assets/1991b162-ec9d-431b-b398-8bee4a666ce5)


Comments:
cc @nielsrogge for the tutorial!

I am having the same issue :(

Edit: Actually, I was able to resolve my issue. I suspect the OP also is having the same problem. My issue is related to a custom bounding_box function I wrote. It was returning multiple boxes, but the individual boxes were not in the required structure [[x,y,z,w]].

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> I am having the same issue :(
> 
> Edit: Actually, I was able to resolve my issue. I suspect the OP also is having the same problem. My issue is related to a custom bounding_box function I wrote. It was returning multiple boxes, but the individual boxes were not in the required structure [[x,y,z,w]].

Glad you got sorted, I am still struggling. Here is my bounding box function and I moved it earlier during dataset dict creation.
Is there anything obviously wrong here @FerRomeroGalvan that you can see?

```python
def get_bounding_box(ground_truth_map):
  # get bounding box from mask
  y_indices, x_indices = np.where(ground_truth_map > 0)
  x_min, x_max = np.min(x_indices), np.max(x_indices)
  y_min, y_max = np.min(y_indices), np.max(y_indices)
  # add perturbation to bounding box coordinates
  H, W = ground_truth_map.shape
  x_min = max(0, x_min - np.random.randint(0, 20))
  x_max = min(W, x_max + np.random.randint(0, 20))
  y_min = max(0, y_min - np.random.randint(0, 20))
  y_max = min(H, y_max + np.random.randint(0, 20))
  bbox = [x_min, y_min, x_max, y_max]

  return bbox

# Convert the NumPy arrays to Pillow images and store them in a dictionary
dataset_dict = {
    "image": [Image.fromarray((img * 255).astype('uint8')) for img in filtered_images],
    "label": [Image.fromarray(mask) for mask in filtered_masks],
    "bounding_box": [get_bounding_box(mask) for mask in filtered_masks]
}
```