Issue 1:
Issue description

ResNet-50 (resolution of 128, batch size 8) crashes on PyTorch 0.4.1, CUDA 9.2 with fp16. It works with CUDA 9.0 and fp32.

Code example

https://github.com/ddkang/fai-imagenet/tree/imagenet/imagenet_nv

Starting program: /home/daniel_d_kang/anaconda3/envs/dawnbench/bin/python main.py -a resnet50 --lr 0.40 --epochs 45 --small -j 8 --fp16 -b 8 --loss-scale 512 /mnt/disks/dawnbench/imagenet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
~~epoch hours top1Accuracy

[New Thread 0x7fffaf723700 (LWP 12826)]
[New Thread 0x7fffaef22700 (LWP 12827)]
0
[New Thread 0x7fff44a3f700 (LWP 13143)]
[New Thread 0x7fff426fe700 (LWP 13155)]
[New Thread 0x7fff4173f700 (LWP 13156)]
[New Thread 0x7fff40f3e700 (LWP 13157)]
[New Thread 0x7fff4073d700 (LWP 13158)]
[New Thread 0x7fff3ff3c700 (LWP 13159)]
[New Thread 0x7fff3f73b700 (LWP 13162)]
[New Thread 0x7fff3e77f700 (LWP 13164)]
[New Thread 0x7fff3df7e700 (LWP 13165)]
zero_grad
tensor(3648., device='cuda:0', dtype=torch.float16, grad_fn=<MulBackward>)
torch.Size([])
torch.cuda.HalfTensor
[New Thread 0x7fff3d77d700 (LWP 13216)]
[New Thread 0x7fff3cf7c700 (LWP 13217)]

Thread 14 "python" received signal SIGFPE, Arithmetic exception.
[Switching to Thread 0x7fff3cf7c700 (LWP 13217)]
0x00007fffc3151984 in cudnn::gemm::conv2d(cudnnContext*, void const*, cudnnTensor4dStruct*, void const*, cudnnFilter4dStruct*, void const*, cudnnConvolutionStruct*, cudnnConvWorkingStruct const*, void*, unsigned long, void const*, cudnnTensor4dStruct*, void*, cudnn::gemm::Conv2dType_t, cudnn::gemm::Conv2dConfig&, bool, void const*, cudnnActivationStruct*, void*) () from /home/daniel_d_kang/anaconda3/envs/dawnbench/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so

System Info

Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.2.148

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 396.44
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

Versions of relevant libraries:
[pip] numpy (1.14.3)
[pip] numpydoc (0.8.0)
[pip] torch (0.4.1)
[pip] torchvision (0.2.1)
[conda] cuda92 1.0 0 pytorch
[conda] pytorch 0.4.1 py36_cuda9.2.148_cudnn7.1.4_1 [cuda92] pytorch
[conda] torchvision 0.2.1 py36_1 pytorch

Issue 2:
Issue description

The following code produces a Floating point exception (core dumped) on Volta series with fp16:

import torch
from torch.nn import Conv2d
conv=torch.nn.Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False).cuda().half()
x=torch.rand(2,256,196,196, requires_grad=True).cuda().half()
y=conv(x)
loss=y.min()
loss.backward()

By changing the convolution parameters by +1/-1 the code works.

I suspect this to be a cudnn bug.

System Info

Collecting environment information...
PyTorch version: 0.4.0a0+3749c58
Is debug build: No
CUDA used to build PyTorch: 9.2.88

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.2.88
GPU models and configuration:
GPU 0: TITAN V
GPU 1: TITAN V
GPU 2: TITAN V
GPU 3: TITAN V

Nvidia driver version: 396.26
cuDNN version: Probably one of the following:
/usr/local/cuda-9.2/cuda/lib64/libcudnn.so.7.1.4
/usr/local/cuda-9.2/cuda/lib64/libcudnn_static.a
/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn.so.7.1.4
/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn_static.a

Versions of relevant libraries:
[pip3] numpy (1.14.5)
[pip3] torch (0.4.0a0+3749c58)
[pip3] torchvision (0.2.1)
[conda] Could not collect

Reproducible also on V100.