Too many resources requested for launch when use gradcheck

lshiwjx: <p>When I test my own c extension operator using gradcheck, I find it is correct only when use double.<br>
But when I test it with double, it show too many resources.<br>
Here is my code:<br>
import torch<br>
from deform3d_double.deform_conv3d_functions import ConvOffset3dFunction<br>
from torch.autograd import Variable<br>
import os<br>
from torch.autograd import gradcheck</p>
<pre><code>os.environ['CUDA_VISIBLE_DEVICES'] = '3'
batchsize = 1
c_in = 1
c_out = 1
inpu = 3
kernel = 1
stri = 1
pad = 0
out = int((inpu + 2 * pad - kernel) / stri + 1)
channel_per_group = 1
g_off = c_in // channel_per_group
c_off = g_off * kernel * kernel * kernel * 3

conv_offset3d = ConvOffset3dFunction((stri, stri, stri), (pad, pad, pad), channel_per_group)

inputs = Variable(torch.rand(batchsize, c_in, inpu, inpu, inpu).double().cuda(), requires_grad=True)
offsets = Variable(torch.rand(batchsize, c_off, out, out, out).double().cuda(), requires_grad=True)
weight = Variable(torch.rand(c_out, c_in, kernel, kernel, kernel).double().cuda(), requires_grad=True)

print(gradcheck(conv_offset3d, (inputs, offsets, weight)))
</code></pre>
<p>The error:</p>
<blockquote>
<p>THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generic/THCTensorMath.cu line=35 error=7 : too many resources requested for launch<br>
Traceback (most recent call last):<br>
File “/home/lshi/Project/Pytorch/deform_conv3d_pytorch_op/deform3d_double/test_gradient.py”, line 28, in <br>
print(gradcheck(conv_offset3d, (inputs, offsets, weight)))<br>
File “/home/lshi/Application/Anaconda/envs/pytorch36/lib/python3.6/site-packages/torch/autograd/gradcheck.py”, line 164, in gradcheck<br>
analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(<em>as_tuple(inputs), o)<br>
File “/home/lshi/Application/Anaconda/envs/pytorch36/lib/python3.6/site-packages/torch/autograd/gradcheck.py”, line 108, in get_analytical_jacobian<br>
output.backward(grad_output, create_graph=True)<br>
File “/home/lshi/Application/Anaconda/envs/pytorch36/lib/python3.6/site-packages/torch/autograd/variable.py”, line 156, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)<br>
File “/home/lshi/Application/Anaconda/envs/pytorch36/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 98, in backward<br>
variables, grad_variables, retain_graph)<br>
File “/home/lshi/Project/Pytorch/deform_conv3d_pytorch_op/deform3d_double/deform_conv3d_functions/deform_conv3d_function.py”, line 48, in backward<br>
grad_weight = weight.new(weight.size()).zero</em>()<br>
RuntimeError: cuda runtime error (7) : too many resources requested for launch at /opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generic/THCTensorMath.cu:35</p>
</blockquote>
<p>But when i test float version, I can get the result.</p>

sytrus-pytorch: <p>I have also encountered a similar issue just now with gradcheck on custom cuda extension. Moreover, by only switching an activation function from tanh to relu the error is suddenly gone … Seems to work without gradcheck, even with double Tensors.</p>

sytrus-pytorch: <p>I did some searches and, following this comment</p>
<p><a href="https://github.com/pytorch/pytorch/issues/7680#issuecomment-390729076" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/7680#issuecomment-390729076</a></p>
<p>I added <code>__launch_bounds__(1024)</code> before each custom kernel functions (those with <code>__global__</code>).</p>
<p>This solves the issue for my case.</p>