Pause autograd for a module

username: DDong
cooked: <p>Is there a way to make autograd “take a break” for a particular module? Implementing like is done <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">here</a> receives a whole bunch of negative infinities as grad_output.</p>

username: ptrblck
cooked: <p>What do you mean by “take a break”? Does a certain module get Inf gradients and you would like to somehow ignore it?</p>

username: DDong
cooked: <p>I’m just not sure why I’m getting Inf gradients. I suspect what might be happening is since I’m using non-differentiable functions inside my custom autograd function, autograd is turning requires_grad off. I don’t entirely understand how custom autograd functions work, though, so that may be a misunderstanding. What I mean by “take a break” is for autograd to not worry about the details of what’s going on inside my custom function; I want it to just trust me and those computations. That is, in the module following it, I want it to just send over values for me to use as grad_output, and for the module preceding it, I want it to just receive the grad_input I send over and assume those are right.<br>
TL;DR, if autograd doesn’t do anything for custom functions then it’s probably a problem on my side; I’m just trying to figure out what’s going on.</p>

username: ptrblck
cooked: <p>Autograd won’t interfere inside your custom forward and backward functions, as it’s disabled by default there.<br>
If you are getting Inf gradients, you might want to debug your code using <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection" rel="nofollow noopener">Anomaly Detection</a>.</p>

username: DDong
cooked: <p>I have actually already enabled Anomaly Detection. I receive the following errors:<br>
<code><br>
sys:1: RuntimeWarning: Traceback of forward call that caused the error:<br>
File “demo.py”, line 341, in <br>
outputs = vgg16(inputs)<br>
File “/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “demo.py”, line 264, in forward<br>
x = self.features(x)<br>
File “/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 92, in forward<br>
input = module(input)<br>
File “/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py”, line 338, in forward<br>
self.padding, self.dilation, self.groups)<br>
</code><br>
<code><br>
Traceback (most recent call last):<br>
File “demo.py”, line 345, in <br>
loss.backward()<br>
File “/opt/anaconda3/lib/python3.7/site-packages/torch/tensor.py”, line 107, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/opt/anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 93, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: Function ‘CudnnConvolutionBackward’ returned nan values in its 0th output.<br>
</code></p>

username: ptrblck
cooked: <p>If your first operation is a convolution, I would check the input for NaNs and Infs using <code>torch.isnan</code> and <code>torch.isinf</code>.</p>

username: DDong
cooked: <p>I don’t understand. My current structure is of the form:<br>
<em>Lots of ordinary modules from torch.nn</em>-&gt;Conv2d-&gt;ReLU-&gt;ConvMask(my custom module)-&gt;ReLU-&gt;Conv2d-&gt;<em>More ordinary modules from torch.nn</em><br>
Shouldn’t the backpropagated gradients until the ReLU following the ConvMask already be cleaned of NaNs and Infs, since those are just ordinary modules?</p>