Is there any benefit to stack multiple GRU one after the other?

username: laro
cooked: <p>I’m working on time series problems.</p>
<p>On the web I saw 2 type of models:</p>
<ul>
<li>Models which used one <code>GRU</code> with multiple layers.</li>
</ul>
<blockquote>
<p>nn.GRU(num_layers=4,…)</p>
</blockquote>
<ul>
<li>Models which used multiple <code>GRU's</code> (one after the other) with less (or equal) number of layers.</li>
</ul>
<blockquote>
<pre><code>nn.GRU(num_layers=2, ...)
nn.GRU(num_layers=2, ...)
</code></pre>
</blockquote>
<p>I understand that using one <code>GRU</code> or multiple <code>GRU</code> change the number of wights to calculates and to optimize,</p>
<p>but I didn’t found any explanation for the benefits between the 2 options above:</p>
<ol>
<li>What is the benefit (or when we want to use) multiple <code>GRU's</code> or one <code>GRU</code>) ?</li>
<li>Is there a benefit for useing one <code>GRU</code> with high number of layers vs multiple <code>GRU's</code> with less layers ?</li>
</ol>

username: J_Johnson
cooked: <p>The two above are practically identical, only difference being semantics. GRU structure for each layer is</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c7077162224fa24deb346ecaade25acd2103ead.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2c7077162224fa24deb346ecaade25acd2103ead" title="IMG_20221103_141520"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c7077162224fa24deb346ecaade25acd2103ead_2_690x153.jpeg" alt="IMG_20221103_141520" data-base62-sha1="6l7XMxlhENHRLbSC23mfdRYWhXf" width="690" height="153" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c7077162224fa24deb346ecaade25acd2103ead_2_690x153.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c7077162224fa24deb346ecaade25acd2103ead_2_1035x229.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c7077162224fa24deb346ecaade25acd2103ead_2_1380x306.jpeg 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20221103_141520</span><span class="informations">1687×376 91.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.GRU.html</a></p>
<p>You can think of the above as one building block, with additional GRU layers just repeating the same gated algorithm. So whether you want to give each a different label, or put them all under one label, is a matter of preference. The math works out the same.</p>

username: J_Johnson
cooked: <p>Additional thoughts:</p>
<p>I could see there being a benefit to splitting it up for very large models, as it’s much easier to assign each “layer” to different GPUs/TPUs during initialization and training. If you put them all in one nn.GRU unit, it’s going to take a lot more code and effort to assign each layer to separate GPUs.</p>

username: laro
cooked: <p>Thanks.<br>
If we use 2 <code>GRU</code> the <code>H0</code> of the second <code>GRU</code> may be initialised with zeros (or different values) (instead of getting the last hidden state of the first <code>GRU</code>),<br>
while using 1 <code>GRU</code> with multiple layers, the hidden state passed to the next layer, across all layers.</p>
<p>Is that something that can change the results dramatically ?</p>

username: J_Johnson
cooked: <p>Right, it would change the result. You would need to keep track of 2 hidden states. One for each separate unit. Notice how the hidden state contains the number of layers in the dims:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ece33cf83acd3f0a82b974401e4dc2e7059d46fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/ece33cf83acd3f0a82b974401e4dc2e7059d46fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ece33cf83acd3f0a82b974401e4dc2e7059d46fb.png" alt="image" data-base62-sha1="xNBxoa9FNiJsBB5hOlybUb1aMm7" width="690" height="76" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">768×85 9.16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If you set the hidden state to zeros for the second layer, it would no longer be the equivalent of using multiple layers in one GRU. It would be the same as if you used one GRU and set half of the hidden state to zeros.</p>