Building vanilla RNN, GRU and LSTM from PyTorch blocks

username: MrRobot
cooked: <h1><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h1>
<p>I am trying to understand how RNN, GRU and LSTM work. I saw a LSTM implementation by just using Numpy <a href="https://github.com/nicodjimenez/lstm" rel="noopener nofollow ugc">here</a>. But it is too time-consuming to go over all those details since I currently just want to understand the algorithmic workflow of those modules (not how we calculate gradient, etc). So I would like to build those modules from PyTorch blocks like <code>nn.Linear()</code>.</p>
<p>This is a LSTM diagram, which looks simple.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f352993837f41785eff6d38d81fe3a336b370fc9.png" data-download-href="https://discuss.pytorch.org/uploads/default/f352993837f41785eff6d38d81fe3a336b370fc9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f352993837f41785eff6d38d81fe3a336b370fc9_2_690x222.png" alt="image" data-base62-sha1="yIwZuykfXPpylWzc4MNlo8uX7FT" width="690" height="222" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f352993837f41785eff6d38d81fe3a336b370fc9_2_690x222.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f352993837f41785eff6d38d81fe3a336b370fc9_2_1035x333.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f352993837f41785eff6d38d81fe3a336b370fc9.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1062×342 20 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, as easy as it might sound, I am not sure</p>
<ul>
<li>How should I carry the cell state and hidden state along different time step.</li>
<li>Whether the models constructed this way could backprop correctly.</li>
<li>Is there any standardized dataset I could use to check my implementation is correct.</li>
</ul>
<p>Could anyone help me? Thank you in advance!</p>

username: tom
cooked: <aside class="quote no-group" data-username="MrRobot" data-post="1" data-topic="60154">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b38774/48.png" class="avatar"> MrRobot:</div>
<blockquote>
<ul>
<li>How should I carry the cell state and hidden state along different time step.</li>
</ul>
</blockquote>
</aside>
<p>Looking at the PyTorch API for inspiration, you can keep it in a local variable and return it at the end.</p>
<blockquote>
<ul>
<li>Whether the models constructed this way could backprop correctly.</li>
</ul>
</blockquote>
<p>Yes.</p>
<blockquote>
<ul>
<li>Is there any standardized dataset I could use to check my implementation is correct.</li>
</ul>
</blockquote>
<p>The usual way is to feed random data through your module and then compare it to a reference implementation (e.g. PyTorch’s or the Numpy one you saw if you prefer).</p>
<p>Best regards</p>
<p>Thomas</p>