RNN implementation

cemicel: <p>Hi<br>
can some one explain me the difference between:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc11fe591bcc840c1035978242a071b3ae1f60b6.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc11fe591bcc840c1035978242a071b3ae1f60b6" title="Screenshot from 2020-11-29 17-22-50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc11fe591bcc840c1035978242a071b3ae1f60b6_2_690x206.png" alt="Screenshot from 2020-11-29 17-22-50" data-base62-sha1="t7hXcTLQSWj0HoOi0yoOqhYkZGm" width="690" height="206" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc11fe591bcc840c1035978242a071b3ae1f60b6_2_690x206.png, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc11fe591bcc840c1035978242a071b3ae1f60b6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc11fe591bcc840c1035978242a071b3ae1f60b6.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-11-29 17-22-50</span><span class="informations">1000×300 9.53 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>On the left side Pytorch official implementation.<br>
On the right side I took formulas for Wikipedia and A.Karpathy article.</p>
<p>Both of the formulas claims that they are Elman implementations. But in Wikipedia and  A.Karpathy article they have 3 set of weights W_i, W_h, W_y when in Pytorch implementation only 2 set of weights.</p>
<p>PS:<br>
This answer does not explain anything.</p><aside class="quote" data-post="2" data-topic="30575">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minseok/48/9774_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/rnn-implementation-incorrect-in-pytorch-tutorials/30575/2">RNN implementation incorrect in PyTorch Tutorials?</a> 
  </div>
  <blockquote>
    Agree. When you check out simple RNN code result, it makes us wonder why the code in the RNN tutorial is different from the definition of RNN. 
import torch.nn as nn
rnn = nn.RNN(10, 10, batch_first=True)

inp = torch.randn((1,3,10))

result = rnn(inp)

print(result[0].shape)
print(result[1].shape)
print(result[0][0])
print(result[1])

Same result of : 
tensor([[[-0.0253,  0.5289, -0.2582, -0.5125, -0.4577,  0.5456, -0.4042, 
-0.6476, -0.4210, -0.5473]]], grad_fn=)
  </blockquote>
</aside>

<p>Sources:</p>
<ul>
<li><a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html" class="inline-onebox" rel="noopener nofollow ugc">RNN — PyTorch 2.1 documentation</a></li>
<li><a href="https://en.wikipedia.org/wiki/Recurrent_neural_network" class="inline-onebox" rel="noopener nofollow ugc">Recurrent neural network - Wikipedia</a></li>
<li><a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" class="inline-onebox" rel="noopener nofollow ugc">The Unreasonable Effectiveness of Recurrent Neural Networks</a></li>
<li></li>
</ul>

steve_moody: <p>They are indeed the same.</p>
<p>The third weight is for the output vector. When RNNs have both outputs and hidden states that are output, there will be a third weight matrix for the y output. That third weight matrix is W_y. The PyTorch doc is specifically for the hidden activation function, which only has two weights in the equation.</p>
<p>On the left, PyTorch docs specify the tanh activation function, while the right equation has a generic activation function.</p>
<p>The right-hand side equation also has only one bias.</p>
<p>Hope this helps.</p>