Confusion with ReLU as a non-linearity

bolt25: <p>I’m not sure if this is the right place to ask this question but I’m going to ask this anyways because people on this forum are really helpful and intelligent!</p>
<p>I was reading about non-linearity functions and given that ReLU or its family (leaky ReLU, PReLU, etc) all are discontinuous functions, yet they work really really well with gradient based optimization algorithms.<br>
How does this work? Shouldn’t the non-linearity create a problem while calculating gradients?</p>

Nikronic: <p>Hi,</p>
<p>The definition of <a href="https://en.wikipedia.org/wiki/Continuous_function" rel="noopener nofollow ugc">continutiy</a>: The function  <em>f</em>  is  <em>continuous at some point</em>   <em>c</em>  of its domain if the <a href="https://en.wikipedia.org/wiki/Limit_of_a_function" rel="noopener nofollow ugc">limit</a> of  <em>f</em> ( <em>x</em> ), as  <em>x</em>  approaches  <em>c</em>  through the domain of  <em>f</em> , exists and is equal to  <em>f</em> ( <em>c</em> ).</p>
<p><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/4ddaac6516cb3ef4538eeaca443e287c8551ac9e" alt="im _{xo c}{f(x)}=f(c)." width="15" height="3"></p>
<p>In detail this means three conditions: first,  <em>f</em>  has to be defined at  <em>c</em>  (guaranteed by the requirement that  <em>c</em>  is in the domain of  <em>f</em> ). Second, the limit on the left hand side of that equation has to exist. Third, the value of this limit must equal  <em>f</em> ( <em>c</em> ).</p>
<p>So, ReLU etc all are continuous.</p>
<p>Here is a great <a href="https://sebastianraschka.com/faq/docs/relu-derivative.html#:~:text=To%20address%20this%20question%2C%20let,that%20the%20function%20is%20continuous." rel="noopener nofollow ugc">post</a> by <a class="mention" href="/u/rasbt">@rasbt</a> .</p>
<p>In the end, I think you have confused non-linearity and discontinuity.</p>
<aside class="quote no-group" data-username="bolt25" data-post="1" data-topic="85600">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bolt25/48/7273_2.png" class="avatar"> bolt25:</div>
<blockquote>
<p>Shouldn’t the non-linearity create a problem while calculating gradients?</p>
</blockquote>
</aside>
<p>Actually, all we are looking for is to have non-linear activation functions to break linearity between each W.x+b calculations in different layers.</p>
<p>Bests</p>

bolt25: <p>Yeah sorry! I meant discontinuity there… I dont know why i wrote non-linearity there <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

Nikronic: <p>It’s ok it happens to me all the time. But still remember that ReLU is continuous but has not derivative at x=0 based on the post I referenced.</p>

bolt25: <p>Yeah! So just because x=0 does not usually exist that commonly in deep learning context, ReLU has turned out to be a better choice. Right?</p>

Nikronic: <p>Actually I do not think non-existence of countable points makes an activation better or affect performance as actually the functions can be implemented using look up tables too.</p>