Repository: ultralytics/ultralytics

Issue:
Problem with the bounding box when a single instance is detected.
### Search before asking

- [X] I have searched the YOLOv8 [issues](https://github.com/ultralytics/ultralytics/issues) and [discussions](https://github.com/ultralytics/ultralytics/discussions) and found no similar questions.


### Question

I am creating a home security system using a security camera that I bought (Tapo C200). Basically, I want to track my car and the people who pass near the car and make sure that they don't stay for a long time there. 

Since I only want to track cars and people, the base models should work ok for my application. However, I am obtaining a wrong bounding box when only one instance is detected. If I have more than one instance in the detection from any class, then the bounding box is ok.

**Only one instance of a car detected:**
<img width="1070" alt="Screenshot 2023-10-19 at 12 00 01 PM" src="https://github.com/ultralytics/ultralytics/assets/132937575/ce1e9a33-9532-49b7-b03a-95bc2c785fd8">


**An instance from a car and a person**
<img width="1025" alt="Screenshot 2023-10-19 at 11 59 51 AM" src="https://github.com/ultralytics/ultralytics/assets/132937575/fec7f559-06d8-4571-aab5-e4459ae490c1">

I am using the _yolov8m_ here on a M1 MacBook Air. However, I have tried all the versions and the behavior is exactly the same. Additionally, the configuration of the predictions is as follows:

`    results = model.predict(
        frame,
        device="mps",
        conf=0.35,
        classes=[0, 2],
        verbose=True,
        iou=0.8,
    )`

Strictly speaking, this isn't a problem for my application since my use case only makes sense when there is a car and a person in the image, and in that case, everything works ok.

However, I'd like to understand why is this likely happening.

Thank you.



### Additional

_No response_


Discussion:
👋 Hello @carlos-osorio-alcalde, thank you for your interest in YOLOv8 🚀! We recommend a visit to the [YOLOv8 Docs](https://docs.ultralytics.com) for new users where you can find many [Python](https://docs.ultralytics.com/usage/python/) and [CLI](https://docs.ultralytics.com/usage/cli/) usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a [minimum reproducible example](https://docs.ultralytics.com/help/minimum_reproducible_example/) to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our [Tips for Best Training Results](https://docs.ultralytics.com/yolov5/tutorials/tips_for_best_training_results/).

Join the vibrant [Ultralytics Discord](https://ultralytics.com/discord) 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

## Install

Pip install the `ultralytics` package including all [requirements](https://github.com/ultralytics/ultralytics/blob/main/requirements.txt) in a [**Python>=3.8**](https://www.python.org/) environment with [**PyTorch>=1.8**](https://pytorch.org/get-started/locally/).

```bash
pip install ultralytics
```

## Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including [CUDA](https://developer.nvidia.com/cuda)/[CUDNN](https://developer.nvidia.com/cudnn), [Python](https://www.python.org/) and [PyTorch](https://pytorch.org/) preinstalled):

- **Notebooks** with free GPU: <a href="https://console.paperspace.com/github/ultralytics/ultralytics"><img src="https://assets.paperspace.io/img/gradient-badge.svg" alt="Run on Gradient"/></a> <a href="https://colab.research.google.com/github/ultralytics/ultralytics/blob/main/examples/tutorial.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> <a href="https://www.kaggle.com/ultralytics/yolov8"><img src="https://kaggle.com/static/images/open-in-kaggle.svg" alt="Open In Kaggle"></a>
- **Google Cloud** Deep Learning VM. See [GCP Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/google_cloud_quickstart_tutorial/)
- **Amazon** Deep Learning AMI. See [AWS Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/aws_quickstart_tutorial/)
- **Docker Image**. See [Docker Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/docker_image_quickstart_tutorial/) <a href="https://hub.docker.com/r/ultralytics/ultralytics"><img src="https://img.shields.io/docker/pulls/ultralytics/ultralytics?logo=docker" alt="Docker Pulls"></a>

## Status

<a href="https://github.com/ultralytics/ultralytics/actions/workflows/ci.yaml?query=event%3Aschedule"><img src="https://github.com/ultralytics/ultralytics/actions/workflows/ci.yaml/badge.svg" alt="Ultralytics CI"></a>

If this badge is green, all [Ultralytics CI](https://github.com/ultralytics/ultralytics/actions/workflows/ci.yaml?query=event%3Aschedule) tests are currently passing. CI tests verify correct operation of all YOLOv8 [Modes](https://docs.ultralytics.com/modes/) and [Tasks](https://docs.ultralytics.com/tasks/) on macOS, Windows, and Ubuntu every 24 hours and on every commit.
@carlos-osorio-alcalde hi, thank you for detailed explanation along with the images. From your description, it seems like the bounding box is getting skewed or inaccurate when only a single instance (e.g., a car) is present in the image, but works fine when there are multiple instances (e.g., a car and a person).

This is unusual behavior as the model is trained to detect multiple classes independently, and the number of instances detected shouldn't affect the bounding box accuracy of other instances.

One possible reason could be the interplay between confidence score and Intersection over Union (IOU) during non-maximum suppression process, which might be affecting the bounding box accuracy with only a single instance. The "conf" parameter you used in the model.predict() function is a threshold value for object confidence score, and the "iou" parameter is a threshold value for non-maximum suppression. They could be tuned to improve detection results.

The other possibility could be scale variations. If the size of the car in the image changes significantly when people are also present, it might induce errors in bounding box.

Despite these potential causes, without an in-depth look into the specific instances and debugging, it's difficult to confirm the exact source of the problem. An examination of the score and bounding box coordinates could be insightful.

Given your application requirement, this may not be a pressing issue. However, your observation is valuable to us and will assist us in continuously improving our models. We appreciate you drawing our attention to this.
I have some updates regarding this...

The problem, somehow, seems to be related to the part of `device='mps'`. Indeed, I made a couple of experiments with `yolov8l`, and here are the results:

**Original image**
![picanto_blanco](https://github.com/ultralytics/ultralytics/assets/132937575/8ded9e56-89e9-46e5-9226-379d6eb4e954)

**Results with `device='mps'`**
![image_2023-10-20_100451632](https://github.com/ultralytics/ultralytics/assets/132937575/f72ada76-6cac-4756-b3c1-60b4396f9b6c)

```
model = YOLO('yolov8l.pt')
image_path = 'picanto_blanco.png'
image = cv2.imread(image_path)
results = model.predict(image_path, device='mps')
```

**Results without `device='mps'`**
![image_2023-10-20_100922900](https://github.com/ultralytics/ultralytics/assets/132937575/82a9ceb3-bf9b-4cfb-84e3-acb52c567037)
```
model = YOLO('yolov8l.pt')
image_path = 'picanto_blanco.png'
image = cv2.imread(image_path)
results = model.predict(image_path)
```

The problem also occurs for different classes, not only for the car class. It seems that other people are experiencing something similar, as stated here #1090.

Thank you for the amazing work on this!
I am experiencing the same issue. Code:

```
from ultralytics import YOLO

model = YOLO('yolov8n.pt')

results = model(source=1, show=True, conf=0.7, save=False, device="mps")

```
If we look at the purple line I placed, you can see as I move, there is one side of the bounding box that is fixed, so the box gets smaller, then has almost width=0, then it goes past the purple line. So basically, one side of the bounding box is fixed (and shouldn't).
<img width="887" alt="Screenshot 2023-10-20 at 20 29 57" src="https://github.com/ultralytics/ultralytics/assets/50542132/1b5df153-b2a1-421a-bc0e-595e9e5892d3">

<img width="889" alt="Screenshot 2023-10-20 at 20 29 47" src="https://github.com/ultralytics/ultralytics/assets/50542132/34647b36-0263-4eeb-ba0b-866bb2e0257e">

<img width="890" alt="Screenshot 2023-10-20 at 20 29 37" src="https://github.com/ultralytics/ultralytics/assets/50542132/ecb3aca6-68a9-4eb2-b1e7-c845c02999ba">

<img width="888" alt="Screenshot 2023-10-20 at 20 29 29" src="https://github.com/ultralytics/ultralytics/assets/50542132/d624d125-d395-4213-bb4a-3a606d9645e6">

Note the side of the bounding box that moves does track my shoulder correctly.
```
from ultralytics import YOLO

model = YOLO('yolov8n.pt')

results = model(source=1, show=True, conf=0.7, save=False, device="cpu")
```

This works well.


In case these are useful, here are some of the result.bboxes obtained (not with the code above, but from a lower-quality camera). It is the transition from 1 cell phone to 1 person 1 cell phone to 1 cell phone:

```
0: 480x640 1 cell phone, 35.6ms
Speed: 2.5ms preprocess, 35.6ms inference, 6.9ms postprocess per image at shape (1, 3, 480, 640)
ultralytics.engine.results.Boxes object with attributes:

cls: tensor([67.], device='mps:0')
conf: tensor([0.5080], device='mps:0')
data: tensor([[479.7716, 283.4486, 479.9041, 479.7716,   0.5080,  67.0000]], device='mps:0')
id: None
is_track: False
orig_shape: (480, 640)
shape: torch.Size([1, 6])
xywh: tensor([[4.7984e+02, 3.8161e+02, 1.3251e-01, 1.9632e+02]], device='mps:0')
xywhn: tensor([[7.4975e-01, 7.9502e-01, 2.0704e-04, 4.0901e-01]], device='mps:0')
xyxy: tensor([[479.7716, 283.4486, 479.9041, 479.7716]], device='mps:0')
xyxyn: tensor([[0.7496, 0.5905, 0.7499, 0.9995]], device='mps:0')

0: 480x640 1 person, 1 cell phone, 33.6ms
Speed: 2.3ms preprocess, 33.6ms inference, 4.3ms postprocess per image at shape (1, 3, 480, 640)
ultralytics.engine.results.Boxes object with attributes:

cls: tensor([67.,  0.], device='mps:0')
conf: tensor([0.4209, 0.3009], device='mps:0')
data: tensor([[2.8820e+02, 2.8735e+02, 4.8008e+02, 4.7972e+02, 4.2091e-01, 6.7000e+01],
        [8.0700e+00, 7.6148e+01, 3.6669e+02, 4.7953e+02, 3.0086e-01, 0.0000e+00]], device='mps:0')
id: None
is_track: False
orig_shape: (480, 640)
shape: torch.Size([2, 6])
xywh: tensor([[384.1398, 383.5364, 191.8800, 192.3723],
        [187.3815, 277.8402, 358.6229, 403.3853]], device='mps:0')
xywhn: tensor([[0.6002, 0.7990, 0.2998, 0.4008],
        [0.2928, 0.5788, 0.5603, 0.8404]], device='mps:0')
xyxy: tensor([[288.1998, 287.3503, 480.0798, 479.7226],
        [  8.0700,  76.1476, 366.6929, 479.5329]], device='mps:0')
xyxyn: tensor([[0.4503, 0.5986, 0.7501, 0.9994],
        [0.0126, 0.1586, 0.5730, 0.9990]], device='mps:0')

0: 480x640 1 cell phone, 33.7ms
Speed: 2.1ms preprocess, 33.7ms inference, 4.3ms postprocess per image at shape (1, 3, 480, 640)
ultralytics.engine.results.Boxes object with attributes:

cls: tensor([67.], device='mps:0')
conf: tensor([0.7324], device='mps:0')
data: tensor([[479.7606, 288.8503, 484.3975, 479.7606,   0.7324,  67.0000]], device='mps:0')
id: None
is_track: False
orig_shape: (480, 640)
shape: torch.Size([1, 6])
xywh: tensor([[482.0790, 384.3055,   4.6369, 190.9103]], device='mps:0')
xywhn: tensor([[0.7532, 0.8006, 0.0072, 0.3977]], device='mps:0')
xyxy: tensor([[479.7606, 288.8503, 484.3975, 479.7606]], device='mps:0')
xyxyn: tensor([[0.7496, 0.6018, 0.7569, 0.9995]], device='mps:0')
```

Looking at it, if this corresponds to the initial cell phone (xyxy): `[479.7716, 283.4486, 479.9041, 479.7716]` and this to the final cell phone: `[479.7606, 288.8503, 484.3975, 479.7606]` they seem pretty close. If we check the 2 object values: 
`[288.1998, 287.3503, 480.0798, 479.7226]` could be the cell phone, seeing 287.3503, 480.0798, 479.7226 do align with the 2nd, 3rd and 4th values. But the first value does not (479 with single detection && 288 with multiple detection) even though it's presumably the same object...
@carlos-osorio-alcalde thank you for sharing your observations, they're quite thorough and provide insights into the problem you're facing.

As per your observations with the 'mps' device mode, it seems there is an issue with the bounding box coordinates which is causing one side of the box to be fixed during detection. The fact it's functioning as expected in 'cpu' mode indicates it isn't related to the detection process per se but likely an issue specific to the processing with 'mps'.

The box coordinates you've posted reveal inconsistencies when the model transitions between single and multiple detection situations. As you pointed out, even for the same object, the first value of the coordinates seems to differ significantly. This suggests that there might be some sort of anchoring issue happening in 'mps' mode causing the fixed bounding box side.

Again, a possible explanation could be the interplay of confidence scores and Intersection over Union (IoU) thresholds during the non-max suppression process. However, the exact root cause can't be confirmed without an in-depth investigation. 

It would be really helpful to further investigate these unusual behaviors under 'mps' mode. This feedback could also lead to improvements in detection accuracy and reliability.

We really appreciate you bringing this to our attention. Your continued support helps us improve our models and the overall user experience.
I saw similar on my Mac. Downgrading to `torch-2.0.1` and `torchvision-0.15.2` (from `torch-2.1.0` and `torchvision 0.16.0`) fixed the issue for me. Not being sure what the problem was, I wasn't sure how to report it.
> I saw similar on my Mac. Downgrading to `torch-2.0.1` and `torchvision-0.15.2` (from `torch-2.1.0` and `torchvision 0.16.0`) fixed the issue for me. Not being sure what the problem was, I wasn't sure how to report it.

Interesting, that worked for me too. Thank you!
@bog thank you for sharing your experience! It's very helpful to know that downgrading to those versions of `torch` and `torchvision` resolved the issue for you. This suggests that the problem might be associated with recent updates to these libraries. We would make sure this insight is taken into account in future updates. We actively welcome such input, so please don't hesitate to share any problem or resolution you encounter. Your input is always valuable.
Thank you! It works well after downgrading to `torch-2.0.1` and `torchvision-0.15.2`
@Any-Winter-4079 i'm glad to hear that downgrading to `torch-2.0.1` and `torchvision-0.15.2` has resolved your issue! It seems that certain functionalities behave differently with updates in the newer versions of these libraries and can cause some unexpected behaviors. We highly appreciate your feedback as it helps us maintain compatibility and better address such issues in the future. Should you have any more insights or face any other problems, please feel free to share them with us. Happy coding!
I have this problem too. It never manifests itself in 8.0.176 but began when I upgraded to 8.0.200. I also upgraded to 8.0.205 and problem still there. (I have many examples.) The bounding box is based on rectangle but rectangles returned have wrong values (always on the xmin from my observation). I am also using mps, torchvision=0.16.0 and torch=2.1.0.

update - still issue after downgrading to 8.0.176 on this computer so it's likely not from ultralytics code. The other computer working fine is running ultralytics=8.0.176, torch=2.0.1 and torchvision=0.15.2.

update - reverted to  torch=2.0.1 and torchvision=0.15.2 on "this" computer  and issue is resolved
@stevemac00 thank you for sharing your experience in such a detailed manner. Your troubleshooting steps are certainly helpful for us to better understand the issue at hand. Based on what you reported, it seems the issue is tied to the particular versions of the torch and torchvision libraries. When an older version of the torchvision and torch libraries is used, as is the case with your other computer, the issue seems to be resolved.

It's important to note that while libraries and frameworks are designed to be backward compatible, sometimes changes in the newer versions may affect some functionalities. This is likely the case here, considering that the problem doesn't appear when reverting back to older versions.

We appreciate your patience and effort in troubleshooting this issue and sharing the solution. Your observations are particularly insightful and will certainly assist other users who may encounter similar problems. Thanks again for raising this issue, and your contributions to this community are highly valued.

Yes, this problem is ocurring. There are some bugs with detections.
With mps device is changing coordinates. See 0, and 2 column. The numbers are different. First image is with round applied
<img width="438" alt="Screenshot 2023-11-29 at 19 36 32" src="https://github.com/ultralytics/ultralytics/assets/22727137/ff00ed23-73c4-4b1c-8019-251ca68efb80">
<img width="1155" alt="Screenshot 2023-11-29 at 19 38 24" src="https://github.com/ultralytics/ultralytics/assets/22727137/a83e7796-aae6-446b-873f-0b46d566ea82">

I'm investigating:
in non_max_suppression the tensors are equals
<img width="833" alt="image" src="https://github.com/ultralytics/ultralytics/assets/22727137/f3f57c8e-292f-42b2-a3da-1101d8ee6e7d">


Resolving patch:
diff --git a/ultralytics/utils/ops.py b/ultralytics/utils/ops.py
--- a/ultralytics/utils/ops.py
+++ b/ultralytics/utils/ops.py
@@ -179,10 +179,6 @@ def non_max_suppression(
     if isinstance(prediction, (list, tuple)):  # YOLOv8 model in validation model, output = (inference_out, loss_out)
         prediction = prediction[0]  # select only inference output
 
-    device = prediction.device
-    mps = 'mps' in device.type  # Apple MPS
-    if mps:  # MPS not fully supported yet, convert tensors to CPU before NMS
-        prediction = prediction.cpu()
     bs = prediction.shape[0]  # batch size
     nc = nc or (prediction.shape[1] - 4)  # number of classes
     nm = prediction.shape[1] - nc - 4
@@ -256,8 +252,6 @@ def non_max_suppression(
         #         i = i[iou.sum(1) > 1]  # require redundancy
 
         output[xi] = x[i]
-        if mps:
-            output[xi] = output[xi].to(device)
         if (time.time() - t) > time_limit:
             LOGGER.warning(f'WARNING ⚠️ NMS time limit {time_limit:.3f}s exceeded')
             break  # time limit exceeded
@@ -273,14 +267,22 @@ def clip_boxes(boxes, shape):
       boxes (torch.Tensor): the bounding boxes to clip
       shape (tuple): the shape of the image
     """
+    device = boxes.device
+    mps = 'mps' in device.type  # Apple MPS
+    if mps:
+        boxes = boxes.cpu()
     if isinstance(boxes, torch.Tensor):  # faster individually
         boxes[..., 0].clamp_(0, shape[1])  # x1
         boxes[..., 1].clamp_(0, shape[0])  # y1
         boxes[..., 2].clamp_(0, shape[1])  # x2
         boxes[..., 3].clamp_(0, shape[0])  # y2
+        if mps:
+            boxes = boxes.to(device)
     else:  # np.array (faster grouped)
         boxes[..., [0, 2]] = boxes[..., [0, 2]].clip(0, shape[1])  # x1, x2
         boxes[..., [1, 3]] = boxes[..., [1, 3]].clip(0, shape[0])  # y1, y2
+        if mps:
+            boxes = boxes.to(device)
 
 
 def clip_coords(coords, shape):


Test patch: