Sigmoid vs tanh vs relu

username: Anuj_Chopra
cooked: <p>I was doing some experiment with GRU. This is the default code of GRU.</p>
<pre><code class="lang-auto">def initialize_gru(self,input_size,hidden_size):
    self.input_reset = nn.Linear(input_size, hidden_size)
    self.input_update = nn.Linear(input_size, hidden_size)
    self.input_new_gate = nn.Linear(input_size, hidden_size)

    self.hid_reset = nn.Linear(hidden_size, hidden_size)
    self.hid_update = nn.Linear(hidden_size, hidden_size)
    self.hid_new_gate = nn.Linear(hidden_size, hidden_size)


def gru_forward(self, inp, last_hid):
    r_t = torch.sigmoid(self.input_reset(inp)+ self.hid_reset(last_hid))
    z_t = torch.sigmoid(self.input_update(inp)+ self.hid_update(last_hid))
    n_t = torch.tanh(self.input_new_gate(inp) + r_t*self.hid_new_gate(last_hid))
    h_t = (1 - z_t) *n_t + z_t*last_hid
    return h_t
</code></pre>
<p>In the function “gru_forward” there are 2 sigmoids and 1 tanh ( sigmoid, sigmoid, tanh in order ). I was experimenting with these functions and found that if i replace the sigmoids with tanh at both places (all 3 tanh) then the network doesn’t learn (loss becomes nan) . Same happens if I replace sigmoids with relu (relu, relu, tanh ). if I put sigmoids at all three places, again the network doesn’t learn. But if I replace tanh by relu (sigmoid , sigmoid, relu), learning is even faster than default.<br>
My question is how to know that which of these are going to work? Can I get any intuition?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Deep Learning black magic? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
I don’t know the answer but does the paper that introduces the GRU model offer more details?</p>

username: Anuj_Chopra
cooked: <p>I read that paper (<a href="https://arxiv.org/pdf/1406.1078.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1406.1078.pdf</a>). I can’t see any explanation for the activation functions. <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

username: KFrank
cooked: <p>Hi Anuj!</p>
<aside class="quote no-group quote-modified" data-username="Anuj_Chopra" data-post="1" data-topic="99620" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anuj_chopra/48/9425_2.png" class="avatar"> Anuj_Chopra:</div>
<blockquote>
<p>In the function “gru_forward” there are 2 sigmoids and 1 tanh<br>
…<br>
if i replace the sigmoids with tanh at both places (all 3 tanh) then the network doesn’t learn (loss becomes nan).</p>
</blockquote>
</aside>
<p>This is more of a side comment than a direct answer:</p>
<p>Note that pytorch’s <code>sigmoid()</code> is the logistic function, and that is<br>
a rescaled and shifted version of <code>tanh()</code>.  Given that the weights<br>
in <code>Linear</code> layers do scaling and their biases do shifts, you would<br>
expect the two versions of your network to train to points where<br>
<code>sigmoid()</code> and <code>tanh()</code> act essentially equivalently.</p>
<p>I would speculate that your network (together with its training data)<br>
is close to being unstable, so that making the seemingly irrelevant<br>
change from <code>sigmoid()</code> to <code>tanh()</code> is enough to kick it into an<br>
unstable regime.</p>
<p>What happens if you use  a plain-vanilla <code>SGD</code> optimizer and/or lower<br>
your learning rate?</p>
<p><code>relu()</code> is different in character from <code>sigmoid()</code> / <code>tanh()</code>, but, even<br>
so, I wouldn’t expect a mixture of <code>relu()</code> and <code>tanh()</code> to break your<br>
training unless your network were already close to being unstable.</p>
<p>Best.</p>
<p>K. Frank</p>

username: googlebot
cooked: <p>sigmoid <em>gates</em> enforce convex/conic combinations (for RNNs, for values from two consecutive timesteps), with tanh() it is basically not a gate anymore (but a source of oscillations)</p>
<p>as for relu, it is similarly non-gated rnn design.</p>

username: KFrank
cooked: <p>Hello Alex!</p>
<aside class="quote no-group quote-modified" data-username="googlebot" data-post="5" data-topic="99620" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ba8739/48.png" class="avatar"> googlebot:</div>
<blockquote>
<p>sigmoid <em>gates</em> enforce convex/conic combinations<br>
…<br>
with tanh() it is basically not a gate anymore (but a source of oscillations)</p>
</blockquote>
</aside>
<p>Yes, I agree.  Your comment explains what is going on and the choice<br>
of <code>sigmiod()</code>.</p>
<p>In particular, with <code>sigmoid()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">z_t = torch.sigmoid(self.input_update(inp)+ self.hid_update(last_hid))
</code></pre>
<p><code>z_t</code> is forced to be between 0 and 1, so that:</p>
<pre data-code-wrap="python"><code class="lang-python">h_t = (1 - z_t) *n_t + z_t*last_hid
</code></pre>
<p>is, as you say, a convex combination of <code>n_t</code> and <code>last_hid</code>.</p>
<p>With <code>tanh()</code>, <code>z_t</code> can become negative, so that (assuming that <code>h_t</code><br>
becomes the new <code>last_hid</code>) <code>last_hid</code> can oscillate in sign.</p>
<aside class="quote no-group">
<blockquote>
<p>as for relu, it is similarly non-gated rnn design.</p>
</blockquote>
</aside>
<p>With <code>relu()</code>, <code>z_t</code> can become larger than 1 so that <code>last_hid</code> can<br>
grow exponentially (and also, <code>(1 - z_t)</code> can become negative).</p>
<p>Thanks for emphasizing the iterative structure of this use case.</p>
<p>Best.</p>
<p>K. Frank</p>