Issue 1:
🐛 Bug

When I place a torch.arange() tensor on a GPU I get the following error:

RuntimeError: parallel_for failed: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device

This occurs on a system where I have a Tesla K80 and two Tesla K40c cards (also a GeForce 710 which is just for display purposes). It seems to be only related to the K40c cards and not the K80, but occurs regardless of what GPU I try to place the tensor on, if either of the K40c cards are visible via CUDA_VISIBLE_DEVICES.

To Reproduce

Steps to reproduce the behavior:

In a fresh conda environment, install the latest stable version of PyTorch (1.3.1) then a minimal working example is:

import torch
torch.arange(0, 10, dtype=torch.int32, device=torch.device("cuda"))

To be clear which GPUs are causing the problem, I first set CUDA_DEVICE_ORDER=PCI_BUS_ID so that I can be sure which GPUs are visible. If I set only either of the K80 dies to be visible, then this error doesn't happen, but setting either K40c visible (and also the GeForce 710, of course) does cause this error to happen. I suspect this is related to the lower compute capability of the K40c devices.

Expected behavior

The documentation for v1.3.1 states here that CUDA devices with compute capability >= 3.0 should be supported. I can't find anywhere else where there is a specification of which compute capabilities are supported from the pre-compiled binaries installed through conda, so I wouldn't expect this to be an intended deprecation, unless I missed it somewhere.

Environment

PyTorch version: 1.3.1

Is debug build: No

CUDA used to build PyTorch: 10.1.243

OS: Manjaro Linux

GCC version: (GCC) 9.2.0

CMake version: Could not collect

Python version: 3.6

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration:

GPU 0: GeForce GT 710

GPU 1: Tesla K80

GPU 2: Tesla K80

GPU 3: Tesla K40c

GPU 4: Tesla K40c

Nvidia driver version: 418.113

cuDNN version: Could not collect

Versions of relevant libraries:

[pip] numpy==1.17.4

[pip] torch==1.3.1

[pip] torchvision==0.4.2

[conda] blas 1.0 mkl

[conda] mkl 2019.4 243

[conda] mkl-service 2.3.0 py36he904b0f_0

[conda] mkl_fft 1.0.15 py36ha843d7b_0

[conda] mkl_random 1.1.0 py36hd6b4f25_0

[conda] pytorch 1.3.1 py3.6_cuda10.1.243_cudnn7.6.3_0 pytorch

[conda] torchvision 0.4.2 py36_cu101 pytorch

Edit: Forgot to add that this does not occur if I install PyTorch 1.3.0 instead.

cc @ezyang @gchanan @zou3519 @ngimel

Issue 2:
🐛 Bug

I am using a Tesla K40m, installed pytorch 1.3 with conda, using CUDA 10.1

To Reproduce

Steps to reproduce the behavior:

Have a box with a Tesla K40m

conda install pytorch cudatoolkit -c pytorch

show cuda is available

python -c 'import torch; print(torch.cuda.is_available());'
>>> True

Instantiate a model and call .forward()

Traceback (most recent call last):
File "./baselines/get_results.py", line 395, in <module>
main(args)
File "./baselines/get_results.py", line 325, in main
log_info = eval_main(eval_args)
File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/baselines/eval_task.py", line 165, in main
log_info = trainer.test(0, evaluate=True)
File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_trainers.py", line 110, in test
evaluate=evaluate)
File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_trainers.py", line 220, in iteration
model_output = self.model.forward(input_data, input_lengths)
File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_models.py", line 49, in forward
self.hidden = self.init_hidden(batch_size, device=device)
File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_models.py", line 40, in init_hidden
return (torch.randn(1, batch_size, self.hidden_dim, device=device),
RuntimeError: CUDA error: no kernel image is available for execution on the device

First tried downgrading to cudatoolkit=10.0, that exhibited same issue.

The code will run fine if you repeat steps above but instead conda install pytorch=1.2 cudatoolkit=10.0 -c pytorch.

Expected behavior

If no longer supporting a specific GPU, please bomb out upon load with useful error message.

Environment

Unfort ran your script after I 'fixed' so pytorch version will be 1.2 here - issue encountered with version 1.3.

Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Scientific Linux release 7.6 (Nitrogen)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
CMake version: version 2.8.12.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla K40m
Nvidia driver version: 430.50
cuDNN version: /usr/lib64/libcudnn.so.6.5.18

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] numpydoc==0.8.0
[conda] blas 1.0 mkl
[conda] mkl 2019.4 243
[conda] mkl-service 2.3.0 py37he904b0f_0
[conda] mkl_fft 1.0.15 py37ha843d7b_0
[conda] mkl_random 1.1.0 py37hd6b4f25_0
[conda] pytorch 1.2.0 py3.7_cuda10.0.130_cudnn7.6.2_0 pytorch
[conda] torchvision 0.4.0 py37_cu100 pytorch

cc @ezyang @gchanan @zou3519 @jerryzh168 @ngimel