Images as LSTM Input

username: anandaltekar
cooked: <p>Hi,</p>
<p>I want to feed in 18 images of size (3,128,128) into an lstm of 17 layers. I’m a bit confused about what my input should be. Docs mention that the input should be of shape(seq_len, batch_size, input_size), When I draw my 1st batch using a data loader I get a tensor of size (18,3,128,128) Does this mean that my LSTM input is: seq_len =18, batch_size=1, input size =3<em>128</em>128 ? Will this flatten the image to a 3<em>128</em>128 vector? Or do I have to reshape it manually?<br>
I also want to implement Teacher Forcing so I will be modifying the RNN class.<br>
What should forward look like? Here’s what I’m trying but I can’t figure out how to write it.</p>
<pre><code>class trialLSTM(nn.Module):
def __init__(self, seq_len, input_size, hidden_size, batch_size, num_layers):
    super(trialLSTM, self).__init__()

    self.seq_len = seq_len
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.batch_size = batch_size
    self.lstm = nn.LSTM(seq_len, batch_size, input_size)

def init_hidden(self):
    # initialize the hidden state and the cell state to zeros
    hidden = torch.zeros(self.batch_size, self.hidden_size)
    cell = torch.zeros(self.batch_size, self.hidden_size)
    if gpu:
        hidden = hidden.cuda()
        cell = cell.cuda()
    return hidden, cell

def forward(self, x, (h_0, c_0)):
    # Incoming x is (18,3,128,128)
    #do i need to reshape it to (1, 3, 128, 128)?like so:
    # for i in range(0, 18):
    #    x[i] = x[i].reshape(1, 3, 128, 128)
    # if yes, do I reshape it here or in the training loop?
    output = torch.empty(seq_len-1, seq_len-1, seq_len-1)
    for t in range(seq_len+1):
        if t==0:
            hidden, cell = self.lstm(x[0], (h_0,c_0))
        else:
            hidden, cell = self.lstm(x[t], (h_1,c_1)) 
</code></pre>
<p>TL;DR<br>
I want to pass data of shape (18,3,128,128) (these are 18 images of shape (3,128,128) at a time in LSTM of 17 layers.<br>
at time 0  input = (data[0], (h_0,c_0)) and output = (h_1,c_1)<br>
at time 1 input = (data[1], (h_1, c_1)) and output = (h_2_c_2)<br>
…<br>
at time 16 input = (data[15], (h_15, c_15)) and output = (h_16, c_16)</p>
<p>so in my training loop:</p>
<pre><code>model = trialLSTM(seq_len=18, input_size=(3*128*128), hidden_size=(3*128*128),
batch_size=1, num_layers=17)
def train(epoch):
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        target = data[1,:]
        if gpu:
              data, target = data.cuda(), target.cuda()
        output = model(data, (h_0, c_0)
        # Here I should get a tensor of 18 hidden states of shape (3,128,128) each right?
        loss = nn.BCELoss(output, target)
        loss.backward()
        optimizer.step()
</code></pre>
<p>What am I doing wrong? What should I be doing? Please help!</p>

username: Pisit_Bee
cooked: <aside class="quote no-group" data-username="anandaltekar" data-post="1" data-topic="61970">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anandaltekar/48/18141_2.png" class="avatar"> anandaltekar:</div>
<blockquote>
<p>Or do I have to reshape it manually?<br>
I also want to implement Teacher Forcing so I will be modifying the RNN class</p>
</blockquote>
</aside>
<p>You can’t pass input image size of (3 , 128 , 128) to LSTM. You should reshape to (batch,seq,feature). For example input image size of (3<em>128</em>128) → (1,128,3 * 128) or (1,3,128 * 128) . I think you need the CNN to extract feature before pass into LSTM.</p>

username: anandaltekar
cooked: <p>Thanks for you reply. I did try it by reshaping it to (1, 3, 128, 128) and that worked. However I was stacking up LSTM cells instead of a layered LSTM.<br>
Yes using CNN for feature extraction will work but I want to avoid it for a specific reason. I deliberately want to pass an image to the LSTM.</p>

username: Thabang_Lukhetho
cooked: <p>Did it work? I mean did you get a good accuracy?</p>