Doubtful ReLU use

vainaijr: <p>when training a neural network, if we pass our input through ReLU, then it makes negative values zero, but this does not lead to a normal or symmetric distribution, for example,</p>
<pre><code class="lang-auto">import torch, seaborn
input = torch.randn(1, 3, 28, 28)
rlu = nn.ReLU()
seaborn.distplot(input.view(-1).detach().numpy())
</code></pre>
<p>gives<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05e08734e1dedf0a5d16b2c88036cb99529ff277.png" alt="image" data-base62-sha1="PZr4EZURdKd6a2pCmaozIzlu99" width="378" height="248"><br>
after applying ReLU,</p>
<pre><code class="lang-auto">seaborn.distplot(rlu(input).view(-1).detach().numpy())
</code></pre>
<p>this distribution becomes,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/922bc72a8132d2b07c91519a72741f4366165292.png" alt="image" data-base62-sha1="kR5vhGFOHMbWMidPDtSQz39JD1M" width="362" height="248"></p>
<p>do we not want to preserve symmetry of this distribution during training?</p>
<p>same thing happens when we use,</p>
<pre><code class="lang-auto">g = nn.CELU()
seaborn.distplot(g(((input))).view(-1))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce2ca626b5252880406472737cc2c67376d45130.png" alt="image" data-base62-sha1="tpU1b5AFG1nm92M7JkTTp1WnHFe" width="372" height="248"></p>
<pre><code class="lang-auto">h = nn.ELU()
seaborn.distplot(h(((input))).view(-1))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce2ca626b5252880406472737cc2c67376d45130.png" alt="image" data-base62-sha1="tpU1b5AFG1nm92M7JkTTp1WnHFe" width="372" height="248"></p>
<pre><code class="lang-auto">i = nn.SELU()
seaborn.distplot(i(((input))).view(-1))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55b52204f92469103d0b500b795700dcbbf07d01.png" alt="image" data-base62-sha1="cecEWCBUV6gRmVshJwctm4hErCh" width="378" height="248"></p>

KFrank: <p>Hi Ravi!</p>
<aside class="quote no-group quote-modified" data-username="vainaijr" data-post="1" data-topic="64884" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vainaijr/48/49167_2.png" class="avatar"> vainaijr:</div>
<blockquote>
<p>when training a neural network, if we pass our input through ReLU, then it makes negative values zero, but this does not lead to a normal or symmetric distribution, …</p>
</blockquote>
</aside>
<p>Well, the short answer is that <code>ReLU</code> (and similar activations) work<br>
well for various neural-network models, so people keep using them.</p>
<p>But there is some intuition and mathematical reasons behind some<br>
of this.</p>
<p>Let me start from the opposite direction:  Why <em>would</em> you want normal<br>
or symmetric distributions?</p>
<p>While one can imagine problems where the inputs to the neural network<br>
are approximately normally distributed, there are many real-world<br>
problems where this is not the case.  And even for problems where the<br>
inputs are normally distributed, why <em>would</em> you want the final outputs<br>
or outputs of intermediate layers to be normal?</p>
<p>Now for some concrete reasons:</p>
<p>Non-linear activations are essential to how neural networks work.<br>
(Two linear layers in a row without a non-linear activation in between is<br>
just a single linear layer with some potentially redundant parameters.)</p>
<p>I’m sure some mathematician will prove me wrong, but if you apply a<br>
non-linear function to a normal (or gaussian) distribution, you cannot<br>
get a normal (or gaussian) distribution back.  Neural networks won’t<br>
work without non-linear activations, so they have to have non-normal<br>
distributions.</p>
<p>There are, of course, symmetric non-linear activations – <code>tanh()</code><br>
is symmetric, and <code>sigmoid()</code> is symmetric with shift.  But, again,<br>
empirically non-symmetric activations often work better.</p>
<p>The intuition I’ve heard about <code>ReLU</code> in particular is that when the<br>
training drives the (inputs to) <code>ReLU</code> into the negative range (that<br>
gets mapped to zero), the <code>ReLU</code> “turns off” and your network<br>
becomes more sparse (mathematically speaking, but not in terms<br>
of storage or computation).  And the sparseness supposedly can<br>
help the network to “generalize” better.</p>
<p>Best.</p>
<p>K. Frank</p>

vainaijr: <p>Yes, I think it is used to have sparse representations.<br>
I think goal is to simulate how neurons in our brain work to accomplish image classification, that is we have bunch of neurons, but most of them stay inactive, and only a few of them fire at a time, that is why I see dropout also being used.</p>