Categorical Embeddings -- can I only have 1 categorical column per embedding layer?

krishnab: <p>Hey Folks, I was just trying to understand the Pytorch <code>Embedding</code> layers. I am creating an time-series prediction model using an LSTM, but I also have some categorical information that I want to include in the model. I have some time series variables, such as the year, month, week number, and day, as well as some spatial variables including US State and county number.</p>
<p>Now, I was wondering if I have to create separate embedding layers for each categorical column, or can I just create a single embedding to cover all of the categorical columns? More specifically, do I need to create a separate embedding column for the year, then a separate one fr the month, and then one for the week, and day. In this case, then I would need to concatenate all of those layers to pass them to a fully connected layer or something.</p>
<p>Or can I just keep the (year, month, week number, and day) as the matrix that I input into the embedding layer? In other words, does the pytorch implementation of <code>Embedding</code> layers handle having these multiple columns as represented by a single output embedding matrix?</p>
<p>Hopefully my question is clear, but please let me know if I need to clarify anything. I just wanted to understand how to best use these embeddings for categorical features. Thanks.</p>

Kurtis_Petitboeuf: <p>Following, i have a similar problem.</p>

ptrblck: <aside class="quote no-group" data-username="krishnab" data-post="1" data-topic="104681">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/b487fb/48.png" class="avatar"> krishnab:</div>
<blockquote>
<p>Or can I just keep the (year, month, week number, and day) as the matrix that I input into the embedding layer? In other words, does the pytorch implementation of <code>Embedding</code> layers handle having these multiple columns as represented by a single output embedding matrix?</p>
</blockquote>
</aside>
<p>You could feed the data as a single tensor to the <code>nn.Embedding</code> layer.<br>
However, I would use separate embeddings, as your input data would have completely different ranges.<br>
<code>nn.Embedding</code> layers accept inputs containing values in <code>[0, nb_embeddings-1]</code>.<br>
If your <code>year</code> data contains values in e.g. <code>[1988, 2020]</code>, you would waste a lot of embedding vectors, as they are never used.<br>
However, if you normalize the data (subtract the min. value), you would have overlapping indices for all data attributes, i.e. the <code>year</code>, <code>week</code>, <code>day</code> etc. would all index the same embeddings.</p>

krishnab: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks so much for the response. Yes, this make sense, I was thinking along these lines as well. I suppose if I want to combine some categorical variables, I could Cartesian product the columns and then generate embedding values for that cartesian product. But I can see how keeping the embeddings separated makes sense most of the time. Thanks again.</p>

ptrblck: <p>That’s another interesting idea. Let us know, which approach worked better in case you are trying out both. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

sushmit_roy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  or <a class="mention" href="/u/krishnab">@krishnab</a>  Can you share how can we use separate embeddings?</p>

captain: <p>Is it finally implemented? Can you send the code git address?</p>