RNN many-to-One query

sgaur: <p>Hi I am pretty much new too pytorch and try to do sentimental analysis.</p>
<p><strong>My Input data is</strong><br>
array([[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  5,  8]<br>
[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  2,  1],<br>
[ 0,  0,  0,  0,  0,  0,  9,  5,  1,  6,  7],<br>
[ 0,  0,  0,  0,  9,  5,  1,  6,  7, 16, 17],<br>
[ 0,  0,  0,  0,  0,  0,  2,  1,  3, 10,  4],<br>
[ 2,  1,  3, 10,  4, 18, 19, 20, 21, 22, 23],<br>
[11,  1,  6,  7, 24, 12,  5,  1, 13, 25, 26],<br>
[ 2,  2,  2,  2,  8,  3,  3,  3,  4,  4,  4],<br>
[ 0,  0,  0,  0,  0,  0,  0, 14,  1, 27, 14],<br>
[ 0, 11,  1, 15,  2, 28, 12, 13, 15,  3,  4]]</p>
<p><strong>Output Data</strong><br>
<strong>tensor([1., 0., 1., 1., 0., 0., 1., 0., 0., 1.]) where 1 is pos and 0 is neg review.</strong></p>
<p><strong>My Model</strong></p>
<blockquote>
<p>class RNN(nn.Module):</p>
<p>def <strong>init</strong>(self,n_vocab,n_embed,hidden_size):<br>
super().<strong>init</strong>()<br>
self.hidden_size = hidden_size<br>
self.embedding = nn.Embedding(n_vocab+1,n_embed)<br>
self.rnn = nn.RNN(n_embed, hidden_size, num_layers = 1, batch_first = True)</p>
<p>def forward(self,x):<br>
x = x<br>
x = self.embedding(x) # batch-size x seq_length x embedding_dimension<br>
x,_ =self.rnn(x)   <span class="hashtag-raw">#batch-size</span> x seq_length x hidden_size<br>
x = torch.sigmoid(x[:,:,-1][:,-1])<br>
return x</p>
</blockquote>
<blockquote>
<p>n_vocab = len(char_to_int) ## 28<br>
n_embed = 100<br>
hidden_size = 8<br>
model = RNN(n_vocab,n_embed, hidden_size)</p>
</blockquote>
<p>I just Need to know that the indexing I did at last is correct or not (<strong>x = torch.sigmoid(x[:,:,-1][:,-1])</strong>).</p>
<p>Sorry for such a Naive question.</p>

G.M: <p>R u trying to take the last hidden states of the RNN? If that’s ur goal, u should use <code>x[ :, -1, :]</code> or simply just use the hidden states returned by rnn(the “<code>_</code>” in ur code).</p>

sgaur: <p>Hi</p>
<p>Thanks for replying.!!</p>
<p>Using slicing x[:,-1,:] I got below matrix.</p>
<blockquote>
<p>tensor([[-0.7032,  0.9809, -0.0366, -0.8463,  0.8746, -0.9249,  0.9830,  0.9964],<br>
[ 0.5889,  0.9578,  0.9545,  0.9815, -0.9984,  0.9589,  0.8505,  0.9543],<br>
[-0.3179,  0.9790,  0.9406,  0.9769, -0.6671, -0.7646,  0.9281, -0.9901],<br>
[ 0.5905,  0.8615,  0.7148, -0.7225,  0.2396,  0.4143, -0.1313, -0.9152],<br>
[-0.9196, -0.9925, -0.8548, -0.9785,  0.3584,  0.9860, -0.4431, -0.9985],<br>
[-0.9984,  0.4017, -0.9045,  0.4047, -0.6658, -0.4878, -0.9790,  0.7657],<br>
[ 0.8931,  0.6347, -0.7855, -0.9594, -0.9998,  0.8930,  0.3539, -0.0567],<br>
[-0.9782, -0.9661, -0.8166, -0.9830, -0.6294,  0.9168,  0.4004, -0.9968],<br>
[-0.8964, -0.8088,  0.9799,  0.8978, -0.2576,  0.8573, -0.7513, -0.9112],<br>
[-0.9389, -0.9517, -0.6023, -0.9468,  0.3026,  0.9553, -0.1915, -0.9959]],<br>
grad_fn=)</p>
</blockquote>
<p>Don’t we have to take the last values of all rows (that’s what my slicing do) ?? or we need to apply Linear transformation ?? What if I don’t want to apply linear transformation.</p>
<p>Basically my question is we have 8 hidden units out for each input features which hidden output we need to select.</p>
<p>If I need to tell below one is +ve or -ve review which value we need to consider.<br>
(-0.7032, 0.9809, -0.0366, -0.8463, 0.8746, -0.9249, 0.9830, 0.9964).</p>

G.M: <p>I’m quite unsure about what is your question. Usually, we take the last hidden states(output) and apply a linear transformation on it to change to the output shape. Basically, you want to change the shape of last step [B, H] to [B, O], where H is the hidden  size of lstm and O is the number of output class.</p>

sgaur: <p>Thanks I think that answer my question.I just wondering what if we don’t want to apply Linear transformation.</p>
<p>thanks for answering my question.</p>

sgaur: <p>hi <a class="mention" href="/u/g.m">@G.M</a><br>
One more question why do we need to use below function.?</p>
<blockquote>
<p>def init_hidden(self):<br>
return torch.zeros(1, self.hidden_size)</p>
</blockquote>
<p>what is the difference if we use it or don’t use it.?</p>

G.M: <p>This is for initializing the hidden and cell states for lstm. Usually, we initialize them to zeros. By default, pytorch also initialize those states to zeros if you don’t pass them, so you actually don’t need the function.</p>

vdw: <p>Well, if you don’t initialize the hidden state after each batch, you should at least <code>detach()</code> it after each batch. Otherwise the backprop graph continuously grows batch after batch.</p>

G.M: <p>Do you mean to detach the output? Can you explain a bit further about why the graph would grow continuously? Thanks</p>

sgaur: <p>So Basically after every epoch(Number of Iteration) it will reinitialize the hidden state back to zero if I use __init__hidden  function.?</p>
<p>If I don’t use __init__hidden function every new epoch will have hidden state of previous epoch.?</p>

vdw: <p>This <a href="https://discuss.pytorch.org/t/solved-why-we-need-to-detach-variable-which-contains-hidden-representation/1426">previous post</a> explains it quite well.</p>

vdw: <p>You usually call <code>init_hidden()</code> or <code>detach()</code> after each batch.</p>
<p>Yes, without <code>init_hidden()</code>, you given the last hidden state of the previous batch as first hidden state for the current batch. However, <code>detach()</code> ensures that the hidden state is a constant and the loss is not backpropagated to the previous batch(es); see the forum post I linked to above.</p>

G.M: <p>Hi, in the post you mentioned, I found that Pytorch lstm initializes hidden states to zeros if I don’t provide them as arguments, so that means I don’t necessarily have to initialize them after each batch. In <a class="mention" href="/u/sgaur">@sgaur</a>’s case, he/she is not providing the hidden states as arguments so I guess there’s no need to initialize/detach them each batch.</p>

vdw: <p>If PyTorch handles the hidden staten like this in that case, then it should work. I just got used to do it manually all the time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>