Potentially incorrect re-implementation of RNN in PyTorch tutorial

MrRobot: <h2><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h2>
<p>I am going over the this <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="noopener nofollow ugc">PyTorch tutorial</a> on PyTorch official website, which tries to classify names into their origins without using <code>torch.nn.RNN</code> but uses basic building block <code>torch.nn.Linear</code> instead (as could be seen <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html#creating-the-network" rel="noopener nofollow ugc">here</a>.</p>
<pre data-code-wrap="python"><code class="lang-python">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

</code></pre>
<p>As much as I find this implementation quite insightful, I found its formulation inconsistent with formal definitions found in Goodfellow’s <a href="http://www.deeplearningbook.org/contents/rnn.html" rel="noopener nofollow ugc">deep learning book</a> (Page 374).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3def41a4e56e9d4850f1b0dd8a82b1f8afcada2.png" alt="image" data-base62-sha1="ueiqa8wbu7l3UPwpFlmbQSuwLGq" width="196" height="90">.<br>
Specifically,</p>
<ul>
<li>There is no tanh() applied to updated hidden state.</li>
<li>Output state is from combined vector instead of updated hidden state.</li>
</ul>
<p>However, according to this tutorial, the result <strong>does</strong> seem to be good. I am not sure if there is formal name for this architecture.</p>
<p>Could someone help me? Thank you in advance.</p>

Minseok: <blockquote>
<p>There is no tanh() applied to updated hidden state.</p>
</blockquote>
<p>If you use nn.RNN(), it uses tanh() as a default activation function even if it is not specified by the user.</p>
<blockquote>
<p>Output state is from combined vector instead of updated hidden state.</p>
</blockquote>
<p>I guess the tutorial is incorrect in terms of its implementation where the output and the new hidden state are different because they are derived from independent weights(i2h, i2o).  As you can see in the code snippet below, the result of hidden state and the last sequence output should be the same</p>
<pre><code class="lang-auto">import torch.nn as nn
rnn = nn.RNN(10, 10, batch_first=True)

inp = torch.randn((1,3,10))

result = rnn(inp)

print(result[0][0])
print(result[1])
</code></pre>
<p>which has the same result of</p>
<pre><code class="lang-auto">tensor([[[-0.0253,  0.5289, -0.2582, -0.5125, -0.4577,  0.5456, -0.4042,
          -0.6476, -0.4210, -0.5473]]], grad_fn=&lt;StackBackward&gt;)
</code></pre>