Adding a new data to to RNN to one of the intermediate layer

chirag_jain: <ul>
<li>
<p>Current model includes: Word embeddings as an input and the X_train_dataset as output. These variables are connected with two hidden layers, one of which is a recurrent layer. With the following structure: Embeddings ↔ recurrent ↔ hidden ↔ X_train</p>
</li>
<li>
<p>Need add an additional input which is identical to the current output(static) to be utilized as input layer. This layer will connect to both embeddings and X_train in the following manner.</p>
</li>
</ul>
<ul>
<li>Embeddings  ↔ hidden ↔ X_train_new AND
<ul>
<li>X_train ↔ hidden ↔ recurrent ↔ X_train_new</li>
</ul>
</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9f766a21d25c5380e6dfce7f4d9c98102a12663.png" data-download-href="https://discuss.pytorch.org/uploads/default/b9f766a21d25c5380e6dfce7f4d9c98102a12663" title="pytorch example"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9f766a21d25c5380e6dfce7f4d9c98102a12663_2_689x323.png" alt="pytorch example" data-base62-sha1="qx8nVWiqgvmJFVuro9JxKBaOHWX" width="689" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9f766a21d25c5380e6dfce7f4d9c98102a12663_2_689x323.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9f766a21d25c5380e6dfce7f4d9c98102a12663_2_1033x484.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9f766a21d25c5380e6dfce7f4d9c98102a12663.png 2x" data-dominant-color="6E233D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch example</span><span class="informations">1212×568 29.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

chirag_jain: <p>I have the code written for part highlighted in blue, need to add the red part.</p>
<p>class BiDir(torch.nn.Module):<br>
def <strong>init</strong>(self, weights, emb_dim, hid_dim, rnn_num_layers=2):<br>
super().<strong>init</strong>()<br>
<span class="hashtag">#Embedding</span> layers using glove as the pretrained weights<br>
self.embedding = nn.Embedding.from_pretrained(weights)<br>
<span class="hashtag">#Bidirectional</span> GRU module for forward pass with 2 hidden layers<br>
self.rnn = torch.nn.GRU(emb_dim, hid_dim, bidirectional=True, num_layers=rnn_num_layers)<br>
self.l1 = torch.nn.Linear(hid_dim * 2 * rnn_num_layers, 256)<br>
self.l2 = torch.nn.Linear(256, 2)</p>
<pre><code>def forward(self, samples):
    
    #Forward Pass        
    embedded = self.embedding(samples)
    _, last_hidden = self.rnn(embedded)
    hidden_list = [last_hidden[i, :, :] for i in range(last_hidden.shape[0])]
    #Calculating the loss
    encoded = torch.cat(hidden_list, dim=1)
    #RELU and Sigmoid Activation Function
    encoded = torch.nn.functional.relu(self.l1(encoded))
    encoded = torch.sigmoid(torch.FloatTensor(self.l2(encoded)))

    return encoded</code></pre>

ptrblck: <p>Double post or related to <a href="https://discuss.pytorch.org/t/bidirectional-nn-with-two-inputs-and-one-output/76252">this post</a>.<br>
Are you working on the same problem and where are you stuck implementing the red part?</p>

AlexSwiderski: <p>The OP is actually trying to help me out with this problem.  However, the post is actually <strong>backwards</strong> ( i think).  The objective is to create a neural network as pictured in the original image.  I was able to code the portion of the network that is outlined in red; however, I would like to add the portion in blue.  My main problem is that I am so new to pytorch that I don’t know how to actually do it.  Hence, me recruiting help, and finding it here ;).</p>
<p>To the best of my ability I created a model that utilizes word embeddings for an input and number sequences as output.  The number sequences are coded in a manner in which they represent the word that corresponds to the embedding.  The input and output are connected by two hidden layers, one of which is recurrent.</p>
<p>The language utilized in the image that reflects the current network I have created follows this:<br>
Embeddings &lt;-&gt; recurrent &lt;-&gt; hidden &lt;-&gt; arpabet_out</p>
<p>What my goal is now to add the remaining structure of the intended network which would look like this:</p>
<p>Embeddings &lt;-&gt; hidden &lt;-&gt; Arpabet_in AND<br>
arpaet_in &lt;-&gt; hidden &lt;-&gt; recurrent &lt;-&gt; arpabet_out</p>
<p>Please let me know if I can provide any more information.  Should it be easier to look at all the code; I have this on my original post in which you linked.</p>

googlebot: <p>It is hard to understand what is it that you’re having problem with, as it is easy to combine two tensors, e.g.: 1) sum them 2)torch.cat 3)trainable combination, simplest looks like input1.matmul(weight1) + input2.matmul(weight2)</p>
<p>Maybe you mean multi-stage training, in this case you can use Tensor.detach() on output from pre-trained part of the network to “freeze” it. You can then combine normal and detached tensors anyway you need to.</p>

AlexSwiderski: <p>Hm, Conceptually it seems like the multi-stage training might be the way to go.  This is my first every pytorch model, and I just picked up python in January so what is “easy” for some, is likely dependent on experience <img src="https://discuss.pytorch.org/images/emoji/apple/crazy_face.png?v=9" title=":crazy_face:" class="emoji" alt=":crazy_face:">.</p>
<p>I trained this network in a different NN language, but am really pushing to get it to work so that I can capitalize on the large dataset GLoVE provides.  Do you have any recommendations for an example model that uses multi-stage training?</p>
<pre><code class="lang-auto">class BiDir(torch.nn.Module):
def **init** (self, weights, emb_dim, hid_dim, rnn_num_layers=2):
super(). **init** ()
[/quote]
</code></pre>

googlebot: <aside class="quote no-group" data-username="AlexSwiderski" data-post="6" data-topic="76946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexswiderski/48/22870_2.png" class="avatar"> AlexSwiderski:</div>
<blockquote>
<p>Do you have any recommendations for an example model that uses multi-stage training?</p>
</blockquote>
</aside>
<p>Sorry, no tutorial-like examples come to mind, as such models are usually big with a lot of model-specific details. But it seems this is a common question in this forum (when people ask about “freezing layers”, “combining models” and such, variants of staged training are implied), see <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088">this topic</a> for example.</p>
<p>Simplest implementation would look like (if I understood your intent correctly):</p>
<pre><code class="lang-auto">output = red_network(input)
if stage2:
  output = combine_tensors(output.detach(), blue_network(input))
</code></pre>
<p>filtering optimizer params or changing their requires_grad to False achieves similar effect.</p>
<p>but note that in this form blue_network may be learning red_network errors (residuals), I don’t understand how you want to change your network loss with blue_network addition… If you use output = blue_network(input, red_network(input)), that’s residual network, if your blue_output and red_output must be independently useful, you want “model combining” instead.</p>