Why sigmoid is class not tanh

granth_jain: <p>Hi,</p>
<p>I am trying to understand why sigmoid is a class in pytorch and not tanh.<br>
Also can we directly use torch.tanh as an activation function or there is some better implementation of tanh as activation function in pytorch.</p>

albanD: <p>Hi,</p>
<p>There is no reason AFAIK.<br>
Both the class and function version use the exact same implementation and the two exist just to make writing NN nicer.<br>
If you write a custom forward function for your Module, you can use the function version. If you use constructs like Sequential, you will have to use the class version.</p>

KFrank: <p>Hello Granth -</p>
<aside class="quote no-group" data-username="granth_jain" data-post="1" data-topic="103287" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/granth_jain/48/27523_2.png" class="avatar"> granth_jain:</div>
<blockquote>
<p>I am trying to understand why sigmoid is a class in pytorch and not tanh.</p>
</blockquote>
</aside>
<p>I’m not quite sure what you’re asking, but just to be clear, pytorch does<br>
have both a class and a function version of <code>Tanh</code>:</p>
<p><a href="https://pytorch.org/docs/1.7.0/generated/torch.nn.Tanh.html#torch.nn.Tanh" rel="noopener nofollow ugc">torch.nn.Tanh</a></p>
<p><a href="https://pytorch.org/docs/1.7.0/nn.functional.html#torch.nn.functional.tanh" rel="noopener nofollow ugc">torch.nn.functional.tanh</a></p>
<p>(Pytorch also has both a class and function version of <code>Sigmoid</code>.)</p>
<p>As an aside, note that pytorch’s <code>sigmoid()</code> is the so-called <em>logistic<br>
function,</em> and that is essentially the same function as <code>tanh()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

granth_jain: <p>Hi,</p>
<p>Are both torch.tanh and NN.tanh same?<br>
Which is better to be used as an activation function</p>

KFrank: <p>Hello Granth -</p>
<aside class="quote no-group" data-username="granth_jain" data-post="4" data-topic="103287" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/granth_jain/48/27523_2.png" class="avatar"> granth_jain:</div>
<blockquote>
<p>Are both torch.tanh and NN.tanh same?</p>
</blockquote>
</aside>
<p>I haven’t checked the code, but I am quite certain that what Alban<br>
said is correct:  Not only do:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.tanh (some_tensor)
torch.nn.functional.tanh (some_tensor)
torch.nn.Tanh() (some_tensor)
</code></pre>
<p>all perform the same computation (and return the same result), but<br>
they all ultimately resolve to the same implementation to do so.</p>
<aside class="quote no-group">
<blockquote>
<p>Which is better to be used as an activation function</p>
</blockquote>
</aside>
<p>The first two versions in my above code are functions.  I believe that<br>
the second is being deprecated in favor of the first (for reasons that<br>
I don’t understand).</p>
<p>The third version is a class whose instances are <em>function objects.</em></p>
<p>(Note, there is no <code>torch.nn.tanh</code>.  There is a <code>torch.nn.Tanh</code> (a<br>
class) and a <code>torch.nn.functional.tanh</code> (a function) (and the<br>
function <code>torch.tanh</code>).)</p>
<p>If you like using functions, use <code>torch.tanh (some_tensor)</code>,<br>
although you can instantiate the function-object version on the fly<br>
and then call it: <code>torch.nn.Tanh() (some_tensor)</code>.  (The on-the-fly<br>
function-object approach has the de minimus inefficiency of<br>
instantiating a new object every time the <code>Tanh()()</code> call is made.)</p>
<p>If you want or need to have an instance of a function object, for<br>
example in order to include it as a “layer” in a <code>Sequential</code>, then<br>
you will need to use <code>torch.nn.Tanh</code>.</p>
<p>But again, in terms of the function that gets computed (rather than<br>
the packaging), they are all the same.</p>
<p>Best.</p>
<p>K. Frank</p>