RNN LSTM image classifier

username: nicolasaraya77
cooked: <p>Hi,</p>
<p>I have array of rgb images with shape is (100, 3, 256, 256 )  (batch size,C,W,H) with shape of labels like (100, 7) (Batch size, labels parameters). labels are hot vector. but I am getting error like “Expected input batch_size (300) to match target batch_size (100), when it tries to calculate the cross entropy loss. would you please tell me what should i change?</p>
<pre><code class="lang-auto">class RNNGRO(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(RNNGRO, self).__init__()
        self.num_layers = num_layers
        self.hidden_dim = hidden_dim
        #self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True)
        57# -&gt; x needs to be: (batch_size, seq, input_size)
        # or:
        #self.gru = nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)
        
        
    def forward(self, x):
        # Set initial hidden states (and cell states for LSTM)
        #print("input value:",x.shape)
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).to(device) 
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).to(device) 
        #print("init value h0:",h0.shape)
        #print("init value c0:",c0.shape)
        
        # Forward propagate RNN
        #out, _ = self.rnn(x, h0)  
        # or:
        out, _ = self.lstm(x, (h0,c0))  
        #print("value ltsm exit:",out.shape)
        # out: tensor of shape (batch_size, seq_length, hidden_size)
        # out: (n, 256, 256)
        
        # Decode the hidden state of the last time step
        out = out[:, -1, :]
      
        #print("out for decode:",out.shape)
        # out: (n, 256)
         
        out = self.fc(out)
        
        #print("out for linear:",out.shape)
        
        # out: (n, 7) parameters
        return out

</code></pre>
<p>parameters to LSTM</p>
<pre><code class="lang-auto">BATCH_SIZE = 300
INPUT_SIZE = 256
SEQUENCE_LENGHT = 256
HIDDEN_SIZE = 100
NUM_NEURONS = 4
CHANNELS = 3
NUM_CLASSES = 7
LEARNING_RATE = 0.001
NUM_EPOCHS = 100 
# Model instance
model = RNNGRO(INPUT_SIZE, HIDDEN_SIZE,
                NUM_NEURONS, NUM_CLASSES).to(device)

#criterion 
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=LEARNING_RATE,momentum=0.9)

</code></pre>
<p>Training</p>
<pre><code class="lang-auto">n_total_steps = len(train_loader)
since = time.time()  
for epoch in range(NUM_EPOCHS):
    for i, (images, labels) in enumerate(train_loader):
        # origin shape: [N, 4, 256, 256] //aqui tengo el problema
        # resized: [N, 256, 256]
        #print("image moment", images.shape)
        images = images.reshape(-1,SEQUENCE_LENGHT,INPUT_SIZE).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images).to(device)
        print(outputs.shape)
        print(labels.shape)
        print(labels.float().view(-1,7).shape)
        loss = criterion(outputs,  labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print(
                f'Epoch [{epoch+1}/{NUM_EPOCHS}], Step [{i+1}/{n_total_steps}], Loss: {loss.item():.4f}')

</code></pre>
<p>The print outputs.shape:  torch.Size([900, 7])<br>
The print labels.shape: torch.Size([300, 7])</p>

username: ptrblck
cooked: <p>Could you check the shape of the <code>images</code> tensor in the <code>DataLoader</code> loop before applying the <code>reshape</code> operation?<br>
Based on the comments I guess it would be <code>[N=300, 4, 256, 256]</code> and after applying the <code>reshape</code> it would be:</p>
<pre><code class="lang-python">images = images.reshape(-1,SEQUENCE_LENGHT,INPUT_SIZE)
[batch_size=4*300=1200, SEQUENCE_LENGTH=256, INPUT_SIZE=256]
</code></pre>
<p>In that case you are flattening the channels into the batch dimension, which is usually wrong (you are increasing the number of samples in the current batch), which would later raise the shape mismatch error.</p>

username: nicolasaraya77
cooked: <p>exactly, raises a shape mismatch error. what would be the best way to work with this tensor in the lstm?</p>

username: ptrblck
cooked: <p>You should make sure that the reshaping doesn’t change the number of samples, but reshapes the <code>images</code> tensor in such a way, which would e.g. increase the sequence length or the number of features.<br>
Usually the important size during reshaping operations would be the batch size, which should stay the same. The shapes or other dimensions depend on your use case.</p>

username: nicolasaraya77
cooked: <p>thank you very much, what I did to solve it, it consisted of a cnnEncoder that gets a vector [Batch size, parameters] and that I use in my lstm. now I’m only looking to concatenate 2 tensors ([batch size, parameters] , 1 is the output cnn and the second is original sequence )to the lstm network</p>