Intergrating word embeddings make the output shape wrong

Hungreeee: <p>Hi everyone,<br>
As the title says, when I use word embeddings, my model output becomes <code>[batch_size, sequence_length, output_size]</code> instead of <code>[batch_size, output_size]</code>. I wonder if this behaviour is expected from <code>nn.Embedding</code>? How can I obtain the output I desire [batch_size, output_size]?</p>
<pre><code class="lang-auto">class FeedFoward(nn.Module):
  def __init__(self, output_size, embedding_dim, hidden_size):
    super(FeedFoward, self).__init__()
    self.embedding = nn.Embedding.from_pretrained(embedding_matrix)
    self.linear_relu_stack = nn.Sequential(
      nn.Linear(in_features=embedding_dim, out_features=hidden_size),
      nn.ReLU(),
      nn.Linear(in_features=hidden_size, out_features=hidden_size),
      nn.ReLU(),
      nn.Linear(in_features=hidden_size, out_features=output_size),
    ) 

  def forward(self, input):
    emb = self.embedding(input)
    out = self.linear_relu_stack(emb)
    return out

model = FeedFoward(OUTPUT_SIZE, EMBEDDINGS_DIM, HIDDEN_SIZE).to(device)
print(model)
</code></pre>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#torch.nn.Embedding">docs</a>:</p>
<blockquote>
<p>Input: <code>(*)</code>, <code>IntTensor</code> or <code>LongTensor</code> of arbitrary shape containing the indices to extract<br>
Output: <code>(*,H)</code>, where <code>*</code> is the input shape and <code>H=embedding_dim</code></p>
</blockquote>
<p>Passing an input in the shape of <code>[batch_size, sequence_length]</code> will create an output in the shape <code>[batch_size, sequence_length, embedding_dim]</code> since each “word” will map to a vector in the embedding.</p>

Hungreeee: <p>I understand that. Would there be a common technique to fixing this available in pytorch nn (averaging layer or something) to obtain the dimension I want?</p>

ptrblck: <p>If you want to get rid of the <code>sequence_length</code> you could just call any reduction on this dimension (e.g. <code>.sum(1), .mean(1), .max(1)</code> etc. of you could use e.g. a linear layer to map the (static) <code>sequence_length</code> to a single value. If depends on your use case and especially if the <code>sequence_length</code> is static or not.</p>

vdw: <p>I would recommend <code>mean()</code> to normalize w.r.t. the sequence lengths, in case they are very different.</p>

Hungreeee: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I am truly sorry for having to bother you again. I have figured out a way to reduce dimensionality using the methods you gave me. However, I discovered that I have an option to keep the word embeddings as they are for each word. But if I do this, the output’s dimension will be all wrong for a Feedfoward network (as mentioned). How should I approach this?<br>
PS: Yes my sequences length are all the same.</p>

ptrblck: <p>Don’t be sorry for asking questions.</p>
<aside class="quote no-group" data-username="Hungreeee" data-post="6" data-topic="187431">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hungreeee/48/63600_2.png" class="avatar"> Hungreeee:</div>
<blockquote>
<p>I discovered that I have an option to keep the word embeddings as they are for each word. But if I do this, the output’s dimension will be all wrong for a Feedfoward network (as mentioned). How should I approach this?</p>
</blockquote>
</aside>
<p>Not necessarily. I understand you want to keep the temporal dimension instead of reducing it. In this case note that e.g. <code>nn.Conv1d</code> layers would accept inputs in <code>[batch_size, channels, seq_len]</code> (unsure if conv layers would be a good fit for your model) and also e.g. <code>nn.Linear</code> layers accept any input in <code>[batch_size, *, in_features]</code> where <code>*</code> denotes additional dimensions. The linear layer will then be applied on each timestep separately.<br>
Would any of this work? I’m also sure <a class="mention" href="/u/vdw">@vdw</a> might have good ideas how these inputs are usually handled.</p>

vdw: <p>Hm, difficult to reply here :). What exactly is your input data and your task? It’s a bit odd that all your sentences have the same length.</p>
<p>It seems you’re trying to train a text classifier using a basic Feed Forward Network. This architecture does not allow to capture the order of words, so your sentences will be treated as a Bag-of-Words (BoW). This works alright for, say, classifying news articles into “politics”, “sports”, “entertainment”, etc. In this case aggregating the (pre-trained) word embeddings using, e.g., <code>mean()</code>, is the way to go. A Feed Forward network just does not support sequential data that way.</p>
<p>The obvious alternative would be to consider more advanced architectures such as Recurrent Neural Networks (RNN) or Concolutional Neural Networks (CNN) as <a class="mention" href="/u/ptrblck">@ptrblck</a> already hinted at.</p>

Hungreeee: <p>Thanks for the suggestion! I am currently working building multiple models for this topic so I just got to stick to this Feedforward one first.</p>