Calculate_gain('tanh')

jpeg729: <p>I understand the results returned by <code>calculate_gain</code> for linear, relu, leaky_relu and sigmoid.</p>
<p>Can anyone tell me why <code>calculate_gain('tanh')</code> returns 5/3 ?</p>

jpeg729: <p><a href="https://arxiv.org/abs/1704.08863" rel="nofollow noopener">On weight initialization in deep neural networks</a> provides mathematical justification for using gain 1 with Tanh activation, and gain 3.6 with Sigmoid activation.</p>

tom: <p>So I know nothing about these the paper you cite. And I haven’t looked at the papers for He’s and Glorot’s initialization, but here is <em>Thomas Theory of Weight Initialization for the Theory Adverse™</em>:</p>
<ul>
<li>Why do we care about initialization? We want “stability” of activation distributions in deep networks. If no “signal” arrives in layer 100, we are screwed.</li>
<li>How do we measure “signal”? Let’s just take standard deviation.</li>
<li>What is then a good gain? One where the standard deviation “converges” to a reasonable positive value (or stays in some region).</li>
<li>Let’s not theorize, let’s just try:</li>
</ul>
<pre><code class="lang-auto">with torch.no_grad():
    a = torch.randn(1000,1000)
    b = a
    for i in range(10):
        l = torch.nn.Linear(1000,1000, bias=False)
        torch.nn.init.xavier_normal_(l.weight, torch.nn.init.calculate_gain('tanh'))
        b = l(b).tanh()
    print (f"in: {a.std().item():.4f}, out: {b.std().item():.4f}")
</code></pre>
<ul>
<li>You will get something like <code>in: 0.9998, out: 0.6518</code>.</li>
<li>You will get the more or less same thing (in particular 0.65x) when you take 20 or 100 layers instead of just 10. Stability!</li>
<li>You will also get the same (output, not input) if you multiply a by 0.5 before feeding it in.</li>
<li>It doesn’t work quite as nicely when you use relu for nonlinearity and gain.</li>
<li>It will not work as well if you use 1 as gain for tanh.</li>
</ul>
<p>The gain of 1 for tanh sounds like it is motivated by the derivative of 1 at 0. If that is the derivation, you might run into trouble with non-small variance.<br>
I’m not sure I’ve seen many deep networks with sigmoid activations.</p>
<p>I seem to remember watching A. Kaparthy explain this in some CS231n lecture (with histograms of the activations).</p>
<p>Best regards</p>
<p>Thomas</p>

jpeg729: <p>Thank you for the data-driven perspective. I hadn’t thought of that approach.</p>
<p>I modified your snippet to show the mean absolute value of the gradient too.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
import sys

a = torch.randn(1000,1000, requires_grad=True)
b = a
print (f"in: {a.std().item():.4f}")
for i in range(100):
    l = torch.nn.Linear(1000,1000, bias=False)
    torch.nn.init.xavier_normal_(l.weight, float(sys.argv[2]))
    b = getattr(F, sys.argv[1])(l(b))
    if i % 10 == 0:
        print (f"out: {b.std().item():.4f}", end=" ")
        a.grad = None
        b.sum().backward(retain_graph=True)
        print (f"grad: {a.grad.abs().mean().item():.4f}")
</code></pre>
<p>A few results</p>
<ul>
<li>
<p><code>sigmoid</code> seems stable with any gain &gt; 0, but the gradients vanish pretty fast.<br>
The more layers you have the higher the gain you will need.</p>
</li>
<li>
<p><code>tanh</code> seems stable with pretty much any gain &gt; 1<br>
With gain 5/3 the output stabilises at ~.65, but the gradients start to explode after around 10 layers<br>
Gain 1.1 works much better, giving output std stable around 0.30 and grads that are much more stable though they do grow slowly</p>
</li>
<li>
<p><code>softsign</code> with gain 1 has slowly vanishing output and gradients<br>
Gain &gt; 1 reduces the vanishing, but higher values eventually cause the gradients to explode<br>
The higher the gains the faster the gradients explode as you add layers</p>
</li>
<li>
<p><code>relu</code> seems to be inherently less stable than the others, but it works OK with gain ~= sqrt(2)</p>
</li>
<li>
<p><code>selu</code> only works with gain 1 and gives output std ~= 1.00, but the grad slowly explodes after 10-20 layers</p>
</li>
</ul>

tom: <p>So for these two:</p>
<aside class="quote no-group" data-username="jpeg729" data-post="4" data-topic="20854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<ul>
<li><code>tanh</code> seems stable with pretty much any gain &gt; 1<br>
With gain 5/3 the output stabilises at ~.65, but the gradients start to explode after around 10 layers<br>
Gain 1.1 works much better, giving output std stable around 0.30 and grads that are much more stable though they do grow slowly</li>
</ul>
</blockquote>
</aside>
<p>Then that might work. My impression was that the “usual” way to counter exploding gradients was clipping. (More prominently in RNNs where tanh still is very common, too.)</p>
<aside class="quote no-group" data-username="jpeg729" data-post="4" data-topic="20854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<ul>
<li><code>selu</code> only works with gain 1 and gives output std ~= 1.00, but the grad slowly explodes after 10-20 layers</li>
</ul>
</blockquote>
</aside>
<p>I thought <a href="https://arxiv.org/abs/1706.02515" rel="noopener nofollow ugc">Klambauer et al, Self-Normalizing Neural Networks</a> had the elaborate insights for this including gradients.</p>
<p>Best regards</p>
<p>Thomas</p>

jpeg729: <p>Regarding selu, the authors do have many elaborate insights that I don’t really understand. However they don’t use any models deeper than ~32 layers, and in my experiments the gradient doesn’t grow that much over 32 layers.</p>
<p>Either our experiments are somehow flawed, or we are misreading the paper. Now, I am not certain whether the paper claims that selus are not prone to exploding gradients, or whether they remain trainable regardless of any exploding gradients.</p>
<p><a href="https://arxiv.org/abs/1712.05577" rel="nofollow noopener">Philipp et al. The exploding gradient problem demystified</a> find that models tend to either be prone to exploding gradients, or suffer from a collapsing domain, both of which hinder training. They suggest using either skip connections or orthogonal initialisation.</p>
<p>I would suggest using layerwise learning rates or an automated approach to adjusting the learning rate such as hyper-gradient descent to better cope with the differences in gradient size at different layers.</p>

tom: <p>Selu seems to stabilize well with a gain of 0.75:</p>
<pre><code class="lang-auto">in: 1.0013
out: 0.7974 grad: 0.6252
out: 0.3138 grad: 0.2894
out: 0.2418 grad: 0.2414
out: 0.2097 grad: 0.2160
out: 0.2118 grad: 0.1993
out: 0.1939 grad: 0.1985
out: 0.1909 grad: 0.2259
out: 0.1849 grad: 0.2329
out: 0.2048 grad: 0.1998
out: 0.2060 grad: 0.2007
</code></pre>
<p>But I’m not sure why 0.75 is magic here or whether the scaling of the Glorot initialization achieves invariance under the layer size.</p>
<p>My takeaway was that there is a fixed point and I’m reasonably happy to try to find it myself. But don’t take that as advice, I am just a random clueless guy on the internet.</p>
<p>Best regards</p>
<p>Thomas</p>

mrTsjolder: <aside class="quote no-group" data-username="tom" data-post="5" data-topic="20854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>I thought <a href="https://arxiv.org/abs/1706.02515" rel="noopener nofollow ugc">Klambauer et al, Self-Normalizing Neural Networks </a> had the elaborate insights for this including gradients.</p>
</blockquote>
</aside>
<p>I understood it as: even after updating the weights, they will stay in the domain of the contraction mapping so that the fixed point does not change.</p>

tom: <p>Just a quick update: Thanks to Ayrton San Joaquin the SELU gain we found here is now in PyTorch 1.8: <a href="https://github.com/pytorch/pytorch/pull/50664" class="inline-onebox" rel="noopener nofollow ugc">Add SELU Activation to calculate_gain by ajsanjoaquin · Pull Request #50664 · pytorch/pytorch · GitHub</a></p>

David_Refaeli: <p>Hi Tom,<br>
Was this the reason 5/3 was chosen? I saw that Kai Arulkumaran was the one who committed the changes. Was it empirically driven or based on some paper?<br>
Thanks,<br>
David</p>

tom: <p>I don’t actually remember, maybe <a class="mention" href="/u/kaixhin">@Kaixhin</a> himself does?</p>

Kaixhin: <p>So the foundation for PyTorch’s weight initialisation module was based on <a href="https://github.com/Kaixhin/nninit" rel="noopener nofollow ugc">nninit</a>, a library I wrote for Torch7. nninit was itself a port of the weight initialisation module in <a href="https://github.com/Lasagne/Lasagne" rel="noopener nofollow ugc">Lasagne</a>, which is a high-level library for Theano. So I don’t know the true origin of that value, and if the empirical process suggests something else probably best to use that (and upstream to PyTorch).</p>

tom: <p>Thanks for this account!<br>
There also are these pointers from Soumith on Torch7’s default init (which PyTorch also has), but I’m was not entirely sure whether the gain calculation is actually related to calculate_gain.</p>
<aside class="onebox twitterstatus" data-onebox-src="https://twitter.com/soumithchintala/status/1108026537230503936">
  <header class="source">

      <a href="https://twitter.com/soumithchintala/status/1108026537230503936" target="_blank" rel="noopener nofollow ugc">twitter.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://pbs.twimg.com/profile_images/959995586689691648/DAFep10r_400x400.jpg" class="thumbnail onebox-avatar" width="400" height="400">
<h4><a href="https://twitter.com/soumithchintala/status/1108026537230503936" target="_blank" rel="noopener nofollow ugc">Soumith Chintala</a></h4>
<div class="twitter-screen-name"><a href="https://twitter.com/soumithchintala/status/1108026537230503936" target="_blank" rel="noopener nofollow ugc">@soumithchintala</a></div>

<div class="tweet">
  <span class="tweet-description"><a href="https://twitter.com/jeremyphoward" target="_blank" rel="noopener nofollow ugc">@jeremyphoward</a> historical accident that was never published, but was always in Torch code for ~15 years, because it was empirically deemed a good bug :) <a href="https://t.co/EiNMGeR1HG" target="_blank" rel="noopener nofollow ugc">https://t.co/EiNMGeR1HG</a><div class="aspect-image-full-size" style="--aspect-ratio:363/500;"><img class="tweet-image" src="https://pbs.twimg.com/media/D2CAFnAVAAA9MCb.jpg" width="362" height="500"></div></span>
</div>

<div class="date">
  <a href="https://twitter.com/soumithchintala/status/1108026537230503936" class="timestamp" target="_blank" rel="noopener nofollow ugc">3:24 PM - 19 Mar 2019</a>

    <span class="like">
      <svg viewbox="0 0 512 512" width="14px" height="16px" aria-hidden="true">
        <path d="M462.3 62.6C407.5 15.9 326 24.3 275.7 76.2L256 96.5l-19.7-20.3C186.1 24.3 104.5 15.9 49.7 62.6c-62.8 53.6-66.1 149.8-9.9 207.9l193.5 199.8c12.5 12.9 32.8 12.9 45.3 0l193.5-199.8c56.3-58.1 53-154.3-9.8-207.9z"></path>
      </svg>
      54
    </span>

    <span class="retweet">
      <svg viewbox="0 0 640 512" width="14px" height="16px" aria-hidden="true">
        <path d="M629.657 343.598L528.971 444.284c-9.373 9.372-24.568 9.372-33.941 0L394.343 343.598c-9.373-9.373-9.373-24.569 0-33.941l10.823-10.823c9.562-9.562 25.133-9.34 34.419.492L480 342.118V160H292.451a24.005 24.005 0 0 1-16.971-7.029l-16-16C244.361 121.851 255.069 96 276.451 96H520c13.255 0 24 10.745 24 24v222.118l40.416-42.792c9.285-9.831 24.856-10.054 34.419-.492l10.823 10.823c9.372 9.372 9.372 24.569-.001 33.941zm-265.138 15.431A23.999 23.999 0 0 0 347.548 352H160V169.881l40.416 42.792c9.286 9.831 24.856 10.054 34.419.491l10.822-10.822c9.373-9.373 9.373-24.569 0-33.941L144.971 67.716c-9.373-9.373-24.569-9.373-33.941 0L10.343 168.402c-9.373 9.373-9.373 24.569 0 33.941l10.822 10.822c9.562 9.562 25.133 9.34 34.419-.491L96 169.881V392c0 13.255 10.745 24 24 24h243.549c21.382 0 32.09-25.851 16.971-40.971l-16.001-16z"></path>
      </svg>
      3
    </span>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


David_Refaeli: <p>Ok, thanks for the answers. I’m going to take it as empirically driven then.</p>