Fix assertion and value errors in sam-vit-h convertion script
Author: jadechoghari
Pull request: yes

# What does this PR do?

When converting a `sam-vit-h` model, you encounter the following two errors:

1. An assertion error.
2. A `ValueError`: "Input boxes must be a list of list of list of floating points."

```python
raise ValueError("Input boxes must be a list of list of list of floating points.")
inputs = processor(
    images=np.array(raw_image), 
    input_points=[input_points], 
    input_labels=input_labels, 
    return_tensors="pt"
).to(device)
```

This PR fixes these two errors by:
- Updating the assertion to the correct values.
- Passing the correctly formatted `input_boxes` to the processor (by updating the format of `input_boxes`).

## How to reproduce the error (before the fix)?

Download the checkpoint file from [here](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth) or any checkpoint using `sam-vit-h`:

   ```bash
   !wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth -O sam_vit_h_4b8939.pth
   ```

Copy and paste the SAM conversion file and try running it:

   ```bash
   !python convert.py --model_name sam_vit_h_4b8939 --checkpoint_path sam_vit_h_4b8939.pth --pytorch_dump_folder_path OUTPUT_PATH
   ```

will ping @amyeroberts & @NielsRogge  for a quick check :)

Comments:
@jadechoghari For reference for anyone coming to this PR - could you share the full traceback of the errors encountered? 

First, try running this without modifying anything in the convert.py file.
```bash
!python convert.py --model_name sam_vit_h_4b8939 --checkpoint_path sam_vit_h_4b8939.pth --pytorch_dump_folder_path OUTPUT_PATH
```

You might encounter an assertion error:
```
2024-08-21 11:37:53.286514: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-08-21 11:37:53.314219: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-08-21 11:37:53.321949: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-08-21 11:37:53.340726: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-21 11:37:54.909160: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
sam_vit_h_4b8939.pth: 100% 2.56G/2.56G [00:36<00:00, 71.0MB/s]
Traceback (most recent call last):
  File "/content/convert.py", line 251, in <module>
    convert_sam_checkpoint(args.model_name, checkpoint_path, args.pytorch_dump_folder_path, args.push_to_hub)
  File "/content/convert.py", line 183, in convert_sam_checkpoint
    assert scores[-1].item() == 0.9712603092193604
AssertionError
```

When you update the assertions with the correct values (depending on the model), you may encounter the following value errors:
```
2024-08-21 12:03:44.144539: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-08-21 12:03:44.164503: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-08-21 12:03:44.170492: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-08-21 12:03:44.185939: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-21 12:03:45.422789: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
is score 1 equal:  False
score 1 0.9833240509033203
Traceback (most recent call last):
  File "/content/convert.py", line 257, in <module>
    convert_sam_checkpoint(args.model_name, checkpoint_path, args.pytorch_dump_folder_path, args.push_to_hub)
  File "/content/convert.py", line 189, in convert_sam_checkpoint
    inputs = processor(images=np.array(raw_image), input_boxes=input_boxes, return_tensors="pt").to(device)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/sam/processing_sam.py", line 85, in __call__
    input_points, input_labels, input_boxes = self._check_and_preprocess_points(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/sam/processing_sam.py", line 254, in _check_and_preprocess_points
    raise ValueError("Input boxes must be a list of list of list of floating points.")
ValueError: Input boxes must be a list of list of list of floating points.
```

It was also an issue that has been mentioned here: -> https://github.com/huggingface/notebooks/issues/403


all done @ArthurZucker !

cc @amyeroberts all done!

@jadechoghari Great! Last thing is to make sure all tests are passing - could you address the current quality check failures?

Oh @amyeroberts , the quality issue is that since we removed the `assert`, there is no longer a need to run the model (unused scores). Therefore, the fix I applied is not necessary if we don't run and test the model during conversion. 

Is it better to run the model just for testing, or should we remove the entire section?

```python
with torch.no_grad():
    output = hf_model(**inputs)
    scores = output.iou_scores.squeeze()
```

@jadechoghari Yes, in that case the forward pass logic should just be removed