Issue 1:
Issue description

Hello all, I just found out that to do index_copy with duplicated elements in the index tensor will result in different results on CPU and GPU.

To give a little context, I am trying to perform tf.segment_max with pytorch.

The elements in a tensor are associated to segments, and I would like to get all the max value in all the segments.

Then I found out the api index_copy and test its behavior on cpu.

I realized that it will place the last segment value associated with the last segment index into the output tensor.

Therefore, I just need to sort the input tensor and make the segment Index tensor reorder as the sorted input tensor.

The maximum in every segment shall be the last one to copy, and the other values in the segments will be ignored. Please refer to following function _seg_max.

Running on CPU, the code just run smoothly and perfectly worked as the tf.segment_max.

However, on GPU, I found out that it will not keep the last value a the final output value.

(The torch.sort works correctly.)

I found that if there is no duplicated index in index tensor, the index_copy works just fine also on GPU.

TL;DR

index_copy will copy the last value if it meets duplicated indices on CPU. On GPU, it behaves differently.

Code example

import torch
import torch.nn as nn
import random

def _seg_max(data, seg, n_seg):
'Perform segment max'
_, sorted_id = torch.sort(data)
sorted_seg = seg[sorted_id].detach()
sorted_data = data[sorted_id].detach()
seg_max = torch.zeros((n_seg,), device=data.device).index_copy(0, sorted_seg, sorted_data).detach()
return seg_max

def random_seg(n_ele, n_seg):
'Create a random segment idx list'
seg_list = [random.randint(0, n_seg - 1) for _ in range(n_ele - n_seg)] + list(range(n_seg))
random.shuffle(seg_list)
return seg_list

def run(n_ele, n_seg, device):
print('Perform `reduce max` on {}-dim vector into {} segments on {}'.format(n_ele, n_seg, device))
with torch.no_grad():
for _ in range(10000):
data = torch.FloatTensor(n_ele).to(device)
nn.init.uniform_(data)
segment_ids = torch.LongTensor(random_seg(n_ele, n_seg)).to(device)
seg_max = _seg_max(data, segment_ids, n_seg)
a = seg_max.max().item()
b = data.max().item()
assert b == a, "{} != {}".format(a, b)
print(' -- Passed.')

gpu = torch.device('cuda')
cpu = torch.device('cpu')

run(n_ele=40, n_seg=40, device=cpu)
run(n_ele=40, n_seg=10, device=cpu)
run(n_ele=40, n_seg=40, device=gpu)
run(n_ele=40, n_seg=10, device=gpu)

Output:

Perform `segment max` on 40-dim vector into 40 segments on cpu
-- Passed.
Perform `segment max` on 40-dim vector into 10 segments on cpu
-- Passed.
Perform `segment max` on 40-dim vector into 40 segments on cuda
-- Passed.
Perform `segment max` on 40-dim vector into 10 segments on cuda
Traceback (most recent call last):
File "dbg.py", line 37, in <module>
run(n_ele=40, n_seg=10, device=gpu)
File "dbg.py", line 28, in run
assert b == a, "{} != {}".format(a, b)
AssertionError: 0.7791520357131958 != 0.8709555864334106

System Info

PyTorch version: 0.4.1

Is debug build: No

CUDA used to build PyTorch: 9.0.176

OS: Debian GNU/Linux 8.10 (jessie)

GCC version: (Debian 4.9.2-10+deb8u1) 4.9.2

CMake version: version 3.0.2

Python version: 3.6

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration: GPU 0: GeForce GTX TITAN X

Nvidia driver version: 387.26

Versions of relevant libraries:

[pip3] msgpack-numpy (0.4.1)

[pip3] numpy (1.14.5)

[pip3] numpydoc (0.7.0)

[pip3] torch (0.4.1)

[pip3] torchfile (0.1.0)

[pip3] torchtext (0.2.3)

[pip3] torchvision (0.2.0)

[conda] cuda80 1.0 0 soumith

[conda] pytorch 0.4.1 py36ha74772b_0

[conda] torchfile 0.1.0

[conda] torchtext 0.2.3

[conda] torchvision 0.2.0

[conda] torchvision 0.2.1 py36_1 pytorch

Issue 2:
🐛 Bug

In inherited torch.Tensor ambiguous type promotions appear (nightly release).

To Reproduce

class MyTensor(torch.Tensor):
...
ten = MyTensor(10)
assert isinstance(ten, MyTensor)
assert isinstance(ten + 1, MyTensor), "I expect this to fail"

Expected behavior

class MyTensor(torch.Tensor):
...
ten = MyTensor(10)
assert isinstance(ten, MyTensor)
assert not isinstance(ten + 1, MyTensor), "all good"

Or a way to properly subclass from torch.Tensor avoiding such issues

Environment

PyTorch version: 1.8.0.dev20201011+cpu
Is debug build: True
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 10.2.0
Clang version: 10.0.1
CMake version: version 3.18.4

Python version: 3.6 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.0.130
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A

Additional context

I use subclassing in geoopt and maintain the project

cc @nairbv @mruberry