Batch dimension and Batch fist unstable behaviour

skumyol: <p>Hi all,</p>
<p>I’ve been suffering of batch size and batch_first parameters since a while and I realized that there is a caveat. When the batch size is 1 I have  my features in the batch like this torch.Size([1, 1827]) but if i increase the batch size I get input as torch.Size([1223, 4]) which is not correct because each sentence is not 4 tokens long. Futhermore, I get error<br>
`&gt;input[0].size()</p>
<blockquote>
<p>torch.Size([4])`</p>
</blockquote>
<p>When I do torch.transpose(input, 0,1).size() it seems better suit to my model torch.Size([4, 1223])</p>
<p>I have a standard dataloader object returns the items as below:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        # this should return one sample from the dataset
        features = self.x[idx]
        target = self.y[idx]
        lenght = self.lengths[idx]
        return (features, target, lenght)
</code></pre>
<p>My model is as below:</p>
<pre><code class="lang-auto">class SimpleLSTM(nn.Module):
    """
    The RNN model 
    """

    def __init__(self, vocab_size: int, embedding_dim: int, hidden_dim: int, num_classes: int, num_layers: int, dropout: float = 0.5, seq_len: int = 2024, pretrained_dim=None, fixed_pad=False):
        """
        Initialize the model by setting up the layers.
        """
        super(SimpleLSTM, self).__init__()
        DEVICE = get_device()
        self.pick_lstm_output = "last_layer"
        self.output_size = num_classes
        self.n_layers = num_layers
        self.hidden_dim = hidden_dim
        self.input_dim = embedding_dim
        # embedding and LSTM layers
        self.embedding = nn.Embedding(vocab_size, embedding_dim).to(DEVICE)
        # self.embedding.weight = nn.Parameter(torch.tensor(embedding_matrix, dtype=torch.float32))
        # self.embedding.weight.requires_grad = False
        self.lstm = nn.LSTM(self.input_dim, hidden_dim, num_layers,
                            dropout=dropout, batch_first=True).to(DEVICE)
        
        # linear and sigmoid layer
        input_dim = hidden_dim
        if fixed_pad:
            input_dim = seq_len * hidden_dim
        self.fc = nn.Linear(input_dim, num_classes).to(DEVICE)

        self.init_weights()

    def forward(self, batch):
        """
        Perform a forward pass of our model on some input and hidden state.
        """
        input, _, lengths = batch
        # embeddings and lstm_out
        embeds = self.embedding(input)
        packed_input = pack_padded_sequence(embeds, lengths=lengths.to('cpu'), batch_first=True, enforce_sorted=False)
        packed_out, (h_n, c_n) = self.lstm(packed_input)
        lstm_output, output_lengths = pad_packed_sequence(packed_out)
        # stack up lstm outputs
        # lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
        # we pick the last output of the sequence as the output of the lstm layer for the classification task
        
        
        # Option 1 take the last output of lstm
        if self.pick_lstm_output == "last_layer":
            output = h_n[-1,:,:]  
        # Option 2 take the mean of the lstm output
        elif self.pick_lstm_output == "mean":
            output = lstm_output.mean(dim=1)
        # Option 3 take the max of the lstm output
        elif self.pick_lstm_output == "max":
            output = lstm_output.max(dim=1)[0]
        else:
            output = lstm_output.contiguous().view(input.shape[1], -1)
        out = self.fc(output)

        return out
</code></pre>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue as I don’t know what the input is supposed to be?</p>

skumyol: <p>Sorry, it’s very difficult to provide an example but I’ll try:</p>
<p>input: batches of 1d tensors which are coming from the dataloader. Each tensor is a tokenized/transformed text sequence.<br>
batch_first = True<br>
problem: When batch size 1 the input is dimension of Batch_SizexSequence_len<br>
When the batch size is greater than 1 the input dimension is Sequence_lenxBatch_Size</p>
<p>I think the default behavour of the batch first here is reversed. That’s why I made a modification to my code as follows</p>
<pre><code class="lang-auto">swapped = input.swapaxes(0,1)
embeds = self.embedding(swapped)
</code></pre>
<p>Then the swapped version can fit to the rest of the code when the batch size is greater than 1 but this is a temporal solution because it’ll not work when the batch size is 1.</p>

vdw: <aside class="quote no-group" data-username="skumyol" data-post="1" data-topic="193294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/skumyol/48/66018_2.png" class="avatar"> skumyol:</div>
<blockquote>
<p>When the batch size is 1 I have my features in the batch like this torch.Size([1, 1827]) but if i increase the batch size I get input as torch.Size([1223, 4])</p>
</blockquote>
</aside>
<p>What do you mean the <strong>input</strong> is of shape <code>(1223, 4)</code>? The input of what? If you mean the input for the LSTM, the your data gets mangled before it’s given to the LSTM.</p>