RuntimeError: derivative for aten::heaviside is not implemented

username: daqianbao
cooked: <p>Hi Folks,</p>
<p>I was trying to implement a piecewise smooth function with torch.heaviside(). The pytorch send me an error message saying heaviside does not have a derivative.</p>
<p>I know that mathematically the derivative at the discontinuity of the heaviside function should be considered as infinity but is there any version of Pytorch that would produce something that works similarly?</p>
<p>Thanks,</p>
<p>DB</p>

username: KFrank
cooked: <p>Hi Daqian!</p>
<aside class="quote no-group" data-username="daqianbao" data-post="1" data-topic="164707" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daqianbao/48/45800_2.png" class="avatar"> daqianbao:</div>
<blockquote>
<p>I was trying to implement a piecewise smooth function with torch.heaviside(). The pytorch send me an error message saying heaviside does not have a derivative.</p>
</blockquote>
</aside>
<p>You are correct that the derivative for <code>heaviside()</code> is not implemented.*</p>
<p>However, if I understand your use case correctly, you do not need to<br>
backpropagate through <code>heaviside()</code> itself, rather, just through the pieces<br>
of your piecewise function.</p>
<p>There are a number of ways to do this, but you can do it with <code>heaviside()</code><br>
if you protect the call to <code>heaviside()</code> with a <code>torch.no_grad()</code> block.</p>
<p>Here is a script that illustrates backpropagating through a piecewise<br>
function built using <code>heaviside()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

def  square_cube (x):   # a piecewise-smooth function
    with torch.no_grad():
        heavy = torch.heaviside (x, torch.tensor ([0.5]))
    return  (1.0 - heavy) * x**2  +  heavy * x**3

t = torch.arange (-3.0, 3.1, 1.0, requires_grad = True)
print (t)
sc = square_cube (t)
sc.sum().backward()
print (t.grad)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-auto">1.12.0
tensor([-3., -2., -1.,  0.,  1.,  2.,  3.], requires_grad=True)
tensor([-6., -4., -2.,  0.,  3., 12., 27.])
</code></pre>
<p>*)  <code>heaviside()</code> is differentiable, with derivative zero everywhere except<br>
for its discontinuity.  So it’s arguable that pytorch should implement its<br>
derivative.  However, with zero derivative, backpropagation <em>through</em><br>
<code>heaviside()</code> isn’t really useful for anything, so not implementing its<br>
derivative isn’t an unreasonable choice.  It does break your use case,<br>
but the <code>torch.no_grad()</code> work-around seems adequate.</p>
<p>Best.</p>
<p>K. Frank</p>

username: daqianbao
cooked: <p>Hi Frank,</p>
<p>I appreciate your insight!</p>
<p>It is my fault that I did not specify what I wanted. So I was actually trying to model a piecewise smooth function and take a derivative of the function I am modeling. I think maybe approximating the torch.heaviside() with a very steep torch.sigmoid() is a better option for me since I need the derivative (and even the second derivative) of the function I am modeling. In this case, I realized that even if it is implemented, the delta function will still cause problems because of my need of its second derivative.</p>
<p>Still, I appreciate your insight!</p>
<p>Best,</p>
<p>DB</p>