Example of Many-to-One LSTM

username: FuriouslyCurious
cooked: <p>Hi everyone,</p>
<p>Is there an example of <strong>Many-to-One</strong> LSTM in PyTorch?  I am trying to feed a long vector and get a single label out.  An LSTM or GRU example will really help me out.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/6415da0424dd66f2f5b134709b92baa59e604c55.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/6415da0424dd66f2f5b134709b92baa59e604c55" title="image.jpg"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/6415da0424dd66f2f5b134709b92baa59e604c55_2_690x215.jpg" width="690" height="215" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/6415da0424dd66f2f5b134709b92baa59e604c55_2_690x215.jpg, https://discuss.pytorch.org/uploads/default/optimized/1X/6415da0424dd66f2f5b134709b92baa59e604c55_2_1035x322.jpg 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/6415da0424dd66f2f5b134709b92baa59e604c55.jpg 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image.jpg</span><span class="informations">1329×416 85.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My problem looks kind of like this:<br>
Input = Series of 5 vectors, output = single class label prediction:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/1X/57c29c72b8ede2601f11c49b3c9016105476e0ea.jpg" width="589" height="401"></p>
<p>Thanks!</p>

username: Tudor_Berariu
cooked: <p><a href="http://pytorch.org/docs/nn.html#recurrent-layers" rel="nofollow noopener">Recurrent modules</a> from <code>torch.nn</code> will get an input sequence and output a sequence of the same length. Just take the last element from that output sequence.</p>
<p>Here is a small working example with a 2-layer LSTM neural network:</p>
<pre><code>import torch
import torch.nn as nn
from torch.autograd import Variable

time_steps = 10
batch_size = 3
in_size = 5
classes_no = 7

model = nn.LSTM(in_size, classes_no, 2)
input_seq = Variable(torch.randn(time_steps, batch_size, in_size))
output_seq, _ = model(input_seq)
last_output = output_seq[-1]

loss = nn.CrossEntropyLoss()
target = Variable(torch.LongTensor(batch_size).random_(0, classes_no-1))
err = loss(last_output, target)
err.backward()</code></pre>

username: osm3000
cooked: <p>I am still confused about the recurrent recurrent modules output. Its shape is <code>output (seq_len, batch, hidden_size * num_directions)</code>:</p>
<ul>
<li>Why the output shape has a <code>hidden_size</code> as a factor?</li>
<li>What is the <code>num_directions</code>?</li>
<li>What if I want the output sequence to have a specific size (number of features)?</li>
</ul>

username: tom
cooked: <p>Hello <a class="mention" href="/u/osm3000">@osm3000</a>,</p>
<aside class="quote no-group">
<blockquote>
<p>Why the output shape has a hidden_size as a factor?</p>
</blockquote>
</aside>
<p>The output per time step has dimension hidden_size per direction (it’s the hidden layer). In LSTM the output is “modulated” cell state.</p>
<aside class="quote no-group">
<blockquote>
<p>What is the num_directions?</p>
</blockquote>
</aside>
<p>This is 1 for “usual” LSTM and 2 for bidirectional ones.</p>
<aside class="quote no-group">
<blockquote>
<p>What if I want the output sequence to have a specific size (number of features)?</p>
</blockquote>
</aside>
<p>Commonly, you would then use hidden_size as the target size and use the last (time direction) output per batch item, i.e.</p>
<pre><code class="lang-auto">x = output[-1] # this is batch * hidden_size for unidirectional LSTM</code></pre>
<p>You can then use x as input into whatever layer you want to have above the LSTM.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Tudor_Berariu
cooked: <ul>
<li>
<p><code>hidden_size</code> represents the output size of the last recurrent layer. I guess it’s called <code>hidden_size</code> as the output of the last recurrent layer is usually further transformed (as in the Elman model referenced in the docs). Also, if there are several layers in the RNN module, all the hidden ones will have the same number of features: <code>hidden_size</code>.</p>
</li>
<li>
<p><code>num_directions</code> is either 1 or 2 depending on the boolean argument <code>bidirectional</code>. If the sequence is processed in both directions you’ll get two values for each time step.</p>
</li>
<li>
<p>If you need a fixed number of output features you either set <code>hidden_size</code> to that value, or you add an output layer that maps from <code>hidden_size</code> to your output space.</p>
</li>
</ul>

username: parth126
cooked: <aside class="quote no-group" data-username="Tudor_Berariu" data-post="2" data-topic="1728">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tudor_berariu/48/419_2.png" class="avatar"> Tudor_Berariu:</div>
<blockquote>
<p>Recurrent modules from torch.nn will get an input sequence and output a sequence of the same length. Just take the last element from that output sequence.</p>
</blockquote>
</aside>
<p>I am confused about when to give entire sequence as input to a RNN (specifcally LSTM) and when to give input stepwise. The way you suggested in this example seems more intuitive but the example given in <a href="http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 2.1.1+cu121 documentation</a> takes only one word at a time.</p>
<p>While at first it seems that the machine translation example does so, to get output at each time stamp (i.e. encoded version after giving each word), but wouldn’t the output sequence record all the time stamps anyway?</p>
<p>I tried a small example by giving the entire input sequence at a time (like in the example you mentioned), and giving one word at a time (like in the MT example). The output at each time stamps are different in both cases. Shouldn’t they ideally be similar.</p>

username: herleeyandi
cooked: <p><a class="mention" href="/u/furiouslycurious">@FuriouslyCurious</a> I also have same problem and same case with you. Did you solve that? I am using this to solve the problem <a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/recurrent_neural_network/main.py" rel="nofollow noopener">Code</a> . I just change the sequence_length and input_size . However I don’t know whether is it true like your problem or not.<br>
<a class="mention" href="/u/tudor_berariu">@Tudor_Berariu</a> What is the number_of_layer in LSTM?, How to make it becomes stacked LSTM?, or does the layer means we stacked the LSTM?</p>

username: Swift2046
cooked: <p>Because I need practice with LSTMs, here’s my go at solving the problem – 5 samples of 5 rows x 5 columns; one label per sample … Any critique of silly mistakes or bad practices appreciated greatly:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

EPOCHS = 500
IN_SIZE = 5
NUM_SAMPLES = 5

def generate_data(rows, columns, samples):
	X = []
	y = []
	transformations = {
		'11': lambda x, y: x + y,
		'15': lambda x, y: x - y,
		'10': lambda x, y: x * y,
		'30': lambda x, y: x / y,
		'2': lambda x, y: x + y,
		}
	for j in range(samples):
		data_set = []
		for i in range(columns):
			data = []
			for val, fn in transformations.items():
				data.append(int(fn(int(val), i+j+1)))
			data_set.append(data)
		X.append(data_set)
		y.append([j+1])
	return X, y


class RNN(nn.Module):
	def __init__(self):
		super(RNN, self).__init__()

		self.rnn = nn.LSTM(
			input_size=5,
			hidden_size=NUM_SAMPLES+1,
			num_layers=2,
			batch_first=True,
		)

	def forward(self, x):
		out, (h_n, h_c) = self.rnn(x, None)
		return out[:, -1, :]	# Return output at last time-step


X, y = generate_data(IN_SIZE, 5, NUM_SAMPLES)
X = torch.FloatTensor(X)
y = torch.LongTensor(y)

rnn = RNN()
optimizer = torch.optim.Adam(rnn.parameters(), lr=0.001)
loss_func = nn.CrossEntropyLoss()     


for j in range(EPOCHS):
	for i, item in enumerate(X):
		item = item.unsqueeze(0)
		output = rnn(item)
		loss = loss_func(output, y[i])
		optimizer.zero_grad()
		loss.backward()
		optimizer.step()

	if j % 5 == 0:
		print('Loss: ', np.average(loss.detach()))

print('Testing:\n========')
for i, item in enumerate(X):
	print(y[i])
	outp = rnn(item.unsqueeze(0))
	print(np.argmax(outp.detach()))

</code></pre>

username: Shisho_Sama
cooked: <p>The Pytorch RNN implementation confused me very badly! but I guess I finally figured it out.<br>
First of all, lets see how a simple RNN does its job. The formula for a vanilla RNN is as follows:</p>
<p>$$ h_t = tanh(W_1 . X_t +W_2 . h_{t-1}) $$</p>
<p>This as we all know is the formula to calculate the new hidden state, for the output part!, however a second step is needed, which is as follows :</p>
<p>$$ o_t = softmax(W_3 \cdot h_t ) $$<br>
W_3 has the shape of (output_size, hidden_size) which after being multiplied by hidden_state (h_t witch shape of (hiddensize, 1) will result in the output vector of shape (outputsize, 1).</p>
<p>Pytorch, doesnt calculate the output by default, so it is up to the user to write it down! RNNs in Pytorch return two results one usually called output and theother hiddenstate, what it really returns as output is actually all the hidden_states for all the timesteps (first result) and the final hidden_state as the second result. This is also the case for LSTM!<br>
This is also indicated in the <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="nofollow noopener">documentation</a> as it reads :</p>
<pre><code class="lang-auto"># the first value returned by LSTM is all of the hidden states throughout
# the sequence. the second is just the most recent hidden state
# (compare the last slice of "out" with "hidden" below, they are the same)
# The reason for this is that:
# "out" will give you access to all hidden states in the sequence
# "hidden" will allow you to continue the sequence and backpropagate,
# by passing it as an argument  to the lstm at a later time
</code></pre>
<p>So how do we do it in Pytorch then?<br>
So basically in order to have the output of your choice you should have a <code>nn.Linear</code> module with the specified output size as its <code>out_features</code> and hiddensize as its <code>in_features</code> .<br>
So basically lets say, we want a havea many to many RNN configuration.<br>
Our input dim is 26, our output is 26 as there are 26 letters in the alphabet for the sake of our example, (both one hot encoded), and our hidden_size is lets say 100. we would then write :</p>
<pre><code class="lang-auto">class ou_rnn(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_size, num_layer):
        super().__init__()

        self.rnn = nn.RNN(input_size = input_dim, hidden_size = hidden_size, num_layers = num_layer)
        # you specify your output size in the linear model. 
        self.fc = nn.Linear(hidden_size, output_dim)
  
   def forward(self, x, h):
        out,h = self.rnn(x,h)
        out = out.contiguous().view(-1, hidden_size)
        out = self.fc(out)
        return out,h
</code></pre>
<p>and in the training loop, the only thing that needs to be taken care of is the labels (they should be reshaped as well to have the shape <code>label.view(batch_size * sequence_length)</code> when used in <code>CrossEntropyLoss</code><br>
Thats it!<br>
As you can see, you can easily have any kind of RNN(or LSTM) configuration. many to many, or many to one, or what ever!</p>
<p>IMHO, the source for all of these issues is the misleading naming that is being used in Pytorch. instead of calling all the hidden_states as outputs, simply refer to them as all_hidden_states!</p>
<p>Hope this is useful.<br>
in case there is something that I missed, please correct me.<br>
I also found <a href="https://discuss.pytorch.org/t/rnn-for-many-to-many-classification-task/15457/2">this</a> to be very helpful</p>