Linear layer with unknown dimension

username: Hari_Krishnan
cooked: <p>Hello everyone,</p>
<p>I have a tensor of size (batch_size, seq_length, embed_dim) and after passing through a linear layer I would like to get an input of  (batch_size, latent_dim). The part of the code is shown below</p>
<pre><code class="lang-auto">class TransformerEncoder(nn.Module):
    
    def __init__(self, head, embedding_size, pos_embed, latent_dim = 512):
        super(TransformerEncoder, self).__init__()
        self.head = head
        self.embedding_size = embedding_size
        self.pos_embed = pos_embed 
        self.self_attention = SelfAttention(head, embedding_size)
        self.fc1 = nn.Linear(embedding_size, latent_dim)
        self.fc2 = nn.Linear(embedding_size, latent_dim)
        self.dropout = nn.Dropout(0.1)
        
    def forward(self, x):
        batch_size, max_words = x.size()[:2]
        x = self.self_attention(x) 
        x = self.dropout(x)
        pos_x = self.pos_embed.get_postional_embeddings()
        x = (x + pos_x)
        x = x.view(batch_size, -1)
        mean, logvar = self.fc1(x), self.fc2(x)
        return mean, logvar
</code></pre>
<p>the variable <code>x</code> is of 3 dimensions, I don’t know the value of batch_size and seq_length until the forward propagation starts. So there will be a size mismatch in the operation <code>self.fc1(x)</code> and in <code>self.fc2(x)</code>. One solution I can think of is to assign a large value to seq_length and pad all the inputs to that length. But this would slow down the learning process. Is there anything else that can be done?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p><code>nn.Linear</code> accepts a variable number of dimensions as explained in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear">docs</a>:</p>
<blockquote>
<p>Input: (N, *, H_in) where ∗ means any number of additional dimensions and H_in = in_features</p>
</blockquote>
<p>So you should be able to pass the input in the shape <code>[batch_size, *any_number_of_dimensions, in_features]</code> to the linear layers.<br>
By using this approach the linear layer will be applied to each input in the <code>*</code> dimension(s).</p>

username: Hari_Krishnan
cooked: <p>But the batch_size value will be known at the time of initializing <code>self.fc1</code> and <code>self.fc2</code>. Only information i have will be the embedding length and the latent dimension length.</p>

username: ptrblck
cooked: <p>You don’t need to know the batch size during initialization of the layer, since if will take any batch size, which doesn’t let your system (or GPU) run out of memory.</p>

username: Hari_Krishnan
cooked: <p>I didn’t mean to give importance to batch_size in my previous reply. The issue is that the input to <code>self.fc1</code> is supposed to be <code>(batch_size, seq_length * embed_dim)</code> and output would be <code>(batch_size, latent_dim)</code>. From what I understood from your answer, my input will be <code>(batch_size, seq_length, embed_dim) </code> shaped tensor and output will be a <code>(batch_size, seq_length, latent_dim)</code> shaped tensor, which is different from what I wanted.</p>

username: ptrblck
cooked: <p>Thanks for clarifying the shapes.<br>
In that case you could try to apply padding to the input and use the max. expected size.</p>