Understanding GRU/LSTM layers parameter

username: timdnewman
cooked: <p>Hi,</p>
<p>I think I understand but I wanted to check how the layers parameter works in LSTM and GRU. Am I correct in thinking that the layers parameter is effectively a short-hand repetition i.e.</p>
<p>for a univariate time series, and GRUs with a hidden state size 10</p>
<pre><code class="lang-auto">self.gru = nn.GRU(input_size=1,hidden_size=10,num_layers=2)
</code></pre>
<p>Is equivalent to:</p>
<pre><code class="lang-auto">self.gru1 = nn.GRU(input_size=1,hidden_size=10)
self.gru2 = nn.GRU(input_size=10,hidden_size=10)
</code></pre>
<p>I think that is what the documentation is saying.</p>
<p>Thanks,</p>
<p>Tim</p>

username: timdnewman
cooked: <p>For clarity the hidden size of 10 was picked at random, I think LSTM is similar but I only wrote up GRU as the example.</p>

username: timdnewman
cooked: <p>I’ll even take a yes or no answer so I know I should keep looking <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: timdnewman
cooked: <p>If anyone comes looking at this - someone just asked a similar question and got an answer they are the same:</p>
<aside class="quote quote-modified" data-post="1" data-topic="165056">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laro/48/47125_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-there-any-benefit-to-stack-multiple-gru-one-after-the-other/165056">Is there any benefit to stack multiple GRU one after the other?</a> 
  </div>
  <blockquote>
    I’m working on time series problems. 
On the web I saw 2 type of models: 

Models which used one GRU with multiple layers.


nn.GRU(num_layers=4,…) 


Models which used multiple GRU's (one after the other) with less (or equal) number of layers.


nn.GRU(num_layers=2, ...)
nn.GRU(num_layers=2, ...)


I understand that using one GRU or multiple GRU change the number of wights to calculates and to optimize, 
but I didn’t found any explanation for the benefits between the 2 options above: 

What is …
  </blockquote>
</aside>


username: mvalente
cooked: <p><a class="mention" href="/u/timdnewman">@timdnewman</a> Run the following code on a jupyter to see what nn.GRU() does :<br>
You’ll see that the prints have the same values.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

gru = nn.GRU(input_size=1,hidden_size=1,num_layers=2, bias=False)
gru_0 = nn.GRU(input_size=1,hidden_size=1,num_layers=1, bias=False)
gru_1 = nn.GRU(input_size=1,hidden_size=1,num_layers=1, bias=False)

#save the parameters of the gru. 
params = list(gru.named_parameters())

# Assign the weights to the other grus.
gru_0.weight_ih_l0 = params[0][1]
gru_0.weight_hh_l0 = params[1][1]
gru_1.weight_ih_l0 = params[2][1]
gru_1.weight_hh_l0 = params[3][1]


input_tensor = torch.tensor([[3.]])
_, hs = gru(input_tensor)
print(hs)

out0, h0 = gru_0(input_tensor)
out1, h1 = gru_1(out0)
print(out0, out1)
</code></pre>

username: timdnewman
cooked: <aside class="quote no-group" data-username="mvalente" data-post="5" data-topic="164389">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mvalente/48/39582_2.png" class="avatar"> mvalente:</div>
<blockquote>
<p>g code on a jupyter to see what nn.GRU() does :<br>
You’ll see that the prints have the same values.</p>
<pre><code class="lang-auto">import torch
</code></pre>
</blockquote>
</aside>
<p>Thanks! Although you’re about 3 hours too late <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I have marked you as the solution as that is definitive.</p>
<p>Do you know how to recommend an edit to the documentation?  Something like this would clarify it a lot, or at least I think so.</p>

username: mvalente
cooked: <p><a class="mention" href="/u/timdnewman">@timdnewman</a>. Thanks, a GRU check was in order for me. Regarding the docs, I have no idea, but if you find out do let me know haha.</p>

username: timdnewman
cooked: <p>Here is the method - but it is a sufficiently long-winded proces that I’ll probably just people search for this <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"></p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#writing-documentation">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#writing-documentation" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#writing-documentation" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/CONTRIBUTING.md#writing-documentation</a></h4>


      <pre><code class="lang-md"># Table of Contents

&lt;!-- toc --&gt;

- [Contributing to PyTorch](#contributing-to-pytorch)
- [Developing PyTorch](#developing-pytorch)
  - [Prerequisites](#prerequisites)
  - [Instructions](#instructions)
  - [Tips and Debugging](#tips-and-debugging)
- [Nightly Checkout &amp; Pull](#nightly-checkout--pull)
- [Codebase structure](#codebase-structure)
- [Unit testing](#unit-testing)
  - [Python Unit Testing](#python-unit-testing)
  - [Better local unit tests with `pytest`](#better-local-unit-tests-with-pytest)
  - [Local linting](#local-linting)
    - [Running `mypy`](#running-mypy)
  - [C++ Unit Testing](#c-unit-testing)
  - [Run Specific CI Jobs](#run-specific-ci-jobs)
- [Merging your Change](#merging-your-change)
- [Writing documentation](#writing-documentation)
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#writing-documentation" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>