Repository: pytorch/vision

Issue:
NMS returns different results on CPU & CUDA
## 🐛 Bug

Hi, I noticed that results of `torchvision.ops.nms` in CPU and CUDA have different value

## To Reproduce

Steps to reproduce the behavior:

1. `docker run --runtime=nvidia -it pytorch/pytorch:1.5-cuda10.1-cudnn7-devel bash`
2. run this script :
  ```
import torch
import torchvision
import random

random.seed(0)
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True

boxes_for_nms = torch.tensor([[0.3764, 0.0905, 0.6533, 0.4487],[0.3744, 0.0899, 0.6535, 0.4513],[0.3753, 0.0916, 0.6532, 0.4512]])
scores = torch.tensor([1., 1., 1.])
iou_threshold = 0.2

cpu_keep = torchvision.ops.nms(boxes_for_nms, scores, iou_threshold)
gpu_keep = torchvision.ops.nms(boxes_for_nms.to('cuda'), scores.to('cuda'), iou_threshold)

print(torch.__version__, torchvision.__version__)
print('cpu keep', cpu_keep)
print('gpu keep', gpu_keep)
print('cpu==gpu', int(cpu_keep)==int(gpu_keep))
  ```
3. output
```
1.5.0 0.6.0a0+82fd1c8
cpu keep tensor([0])
gpu keep tensor([2], device='cuda:0')
cpu==gpu False
```

## Expected behavior

`cpu==gpu True`

## Environment

```
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1060 with Max-Q Design
Nvidia driver version: 418.56
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.5.0
[pip] torchvision==0.6.0a0+82fd1c8
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.1.243             h6bb024c_0  
[conda] mkl                       2020.0                      166  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] numpy                     1.18.1           py37h4f9e942_0  
[conda] numpy-base                1.18.1           py37hde5b4d6_1  
[conda] pytorch                   1.5.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.6.0                py37_cu101    pytorch
```

## Additional context

testing on pytorch 1.4.0 and torchvision 0.5.0 also yields different result, 



Discussion:
Hi,

Thanks for opening the issue!

The problem here is that all the values of the scores are the same. In this case, the resulting indices of sorting the scores is arbitrary in PyTorch, and not guaranteed to be the same between CPU and GPU.
TorchVision uses PyTorch sort internally for NMS, and as such we don't guarantee that the results will be the same for CPU and GPU.

It would be great if we could improve the documentation of NMS in torchvision to explain this behavior. Could you send a PR improving the documentation?

Resolving patch:
diff --git a/torchvision/ops/boxes.py b/torchvision/ops/boxes.py
--- a/torchvision/ops/boxes.py
+++ b/torchvision/ops/boxes.py
@@ -14,6 +14,11 @@ def nms(boxes, scores, iou_threshold):
     IoU greater than iou_threshold with another (higher scoring)
     box.
 
+    If multiple boxes have the exact same score and satisfy the IoU 
+    criterion with respect to a reference box, the selected box is 
+    not guaranteed to be the same between CPU and GPU. This is similar 
+    to the behavior of argsort in PyTorch when repeated values are present.
+
     Parameters
     ----------
     boxes : Tensor[N, 4])


Test patch: