Gradient of ReLu at 0

username: gebrahimi
cooked: <p>What is the gradient of relu(x) = max(0, x) with respect to x when x = 0 in pytorch?</p>

username: albanD
cooked: <p>Hi,</p>
<p>For relu at <code>0</code>, we return a gradient of <code>0</code>.</p>

username: phunc20
cooked: <p>Hello <a class="mention" href="/u/alband">@albanD</a>, I am interested in this question in 2023 and failed to find the source code on ReLU’s derivative at 0 in a reasonably short time. Could you help me point out where to read in the source code, please?</p>
<p>If possible, also where to read when this article <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/autograd.html</a> mentions about the derivative behaviours when the function is (locally) convex/concave.</p>

username: albanD
cooked: <p>Hey!</p>
<p>Sure it is indeed a bit tricky to track down:</p>
<ul>
<li>You can find most of the formulas here: <a href="https://github.com/pytorch/pytorch/blob/341c4227a87b62c5e9b8fa919cb1a50baddd87cb/tools/autograd/derivatives.yaml#L2020-L2022">https://github.com/pytorch/pytorch/blob/341c4227a87b62c5e9b8fa919cb1a50baddd87cb/tools/autograd/derivatives.yaml#L2020-L2022</a> in particular it is using the same thing as threshold for this one.</li>
<li>This one is a native_function that you can find defined here: <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native_functions.yaml#L5937">https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native_functions.yaml#L5937</a></li>
<li>Grepping for the CPU inplementation, you cna find it at <a href="https://github.com/pytorch/pytorch/blob/341c4227a87b62c5e9b8fa919cb1a50baddd87cb/aten/src/ATen/native/Activation.cpp#L678-L680">https://github.com/pytorch/pytorch/blob/341c4227a87b62c5e9b8fa919cb1a50baddd87cb/aten/src/ATen/native/Activation.cpp#L678-L680</a></li>
<li>So it is basically calling threshold again with 0, 0 as arguments.</li>
</ul>
<p>For the doc, do you mean this paragraph <a href="https://pytorch.org/docs/stable/notes/autograd.html#gradients-for-non-differentiable-functions" class="inline-onebox">Autograd mechanics — PyTorch 2.1 documentation</a> ?<br>
Note that relu, while it is nicely convex, it is not continuously differentiable. So you are in a pretty bad case.</p>

username: phunc20
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>First, thanks for helping point out the $ReLU’(0)$ source code. I hadn’t realized that I should have looked into these files. So, as far as I could understand, is PyTorch simply defining $ReLU’(x)$ as</p>
<ul>
<li>0 when x &lt;= 0;</li>
<li>dx/dx = 1 otherwise?</li>
</ul>
<p>As for the doc, yes. I am referring to the Autograd mechanics link. In particular, in the <a href="https://pytorch.org/docs/stable/notes/autograd.html#gradients-for-non-differentiable-functions" rel="noopener nofollow ugc">“Gradients for non-differentiable functions” Section</a>, it mentions the cases where the function in question is convex/concave. I had grepped the keyword “convex” but failed to find results in the source code. After reading the yaml files you guided me to, I believe that the mentioned convexity principle is just a guidance for both developers and users of PyTorch and that, to implement each new function, PyTorch developers must do it case-by-case. In other words, no code examination of convexity to automatically follow the principle</p>
<blockquote>
<p>If the function is convex (at least locally), use the sub-gradient of minimum norm (it is the steepest descent direction).</p>
</blockquote>
<p>Am I right?</p>
<p>Ps. One small detail also confuses me: “the sub-gradient of minimum norm” would be the flattest slope in the case of real-valued function of one real variable, which usually is not the steepest descent.</p>

username: albanD
cooked: <blockquote>
<p>no code examination of convexity to automatically follow the principle</p>
</blockquote>
<p>That is correct, these rules are implemented by hand in the derivatives formula that we have. Nothing automatic like that is done for backward mode AD.</p>
<blockquote>
<p>One small detail also confuses me: “the sub-gradient of minimum norm” would be the flattest slope in the case of real-valued function of one real variable, which usually is not the steepest descent.</p>
</blockquote>
<p>It is the flattest slope but it still is the direction of steepest descent. Note that this is all about the direction of descent.</p>

username: phunc20
cooked: <p>Ok <a class="mention" href="/u/alband">@albanD</a>, I see. Thanks for having shared your precious opinions and time.</p>