Predict a categorical variable and then embed it (one-hot?)

turian: <p>What is the best way to predict a categorical variable, and then embed it, as input to another net?</p>
<p>My instances are tabular, a mix of categorical and continuous variables. I currently have a siamese net (net1) that uses the instances as input. The categorical variables are integer indices used before an nn.Embedding layer. Once net1 is trained, it is fixed.</p>
<p>Now, I want net0 to <em>predict</em> (output) these tabular instances and feed them as input to the above net, and backprop the loss through the entire network (net0 =&gt; net1), even tho net1 will be fixed and not modified.</p>
<ol>
<li>What seems sensible to me is net0 predicts the one-hot (softmax) representation of the category, and then figure out to change net1 so that nn.Embedding uses a one-hot, not integer index, as the input. However, I don’t believe nn.Embedding can accept a one-hot input (<a href="https://discuss.pytorch.org/t/how-to-make-nn-embedding-support-one-hot-encoded-vector-input/50688" class="inline-onebox">How to make nn.Embedding support one-hot encoded vector input?</a>)</li>
<li>net0 predicts the one-hot (softmax) representation of the category, takes the argmax to get the integer index, and net1 looks up the nn.Embedding using the index. However, I believe that argmax is non-differentiable, which means I cannot pass losses back from net1 to net0 <a href="https://discuss.pytorch.org/t/differentiable-argmax/33020" class="inline-onebox">Differentiable argmax</a>
</li>
<li>I could have net0 directly predict the embedding of net1, and in postprogressing use kNN to find the nearest category. I don’t love this because a) it requires the net1 embeddings to be fixed and b) it seems a little inelegant.</li>
</ol>
<p>What is my best option here to predict a categorical variable and then embed it?</p>

turian: <p>One approach that is a little cheesy, and perhaps a little slower, but will be very straightforward to implement and understand is:</p>
<ol start="4">
<li>Convert the categorical variable to a one-hot encoding. Instead of an nn.Embedding layer, have a 1xE (E=embedding size) weight matrix for each dimension in the one-hot. Sum all the 1xE embeddings for the category and maybe squash them?</li>
</ol>
<p>In this case, if it is truly one-hot, then only one embedding weight matrix is active. If the input is a probability distribution over different categories, their embeddings are mixed together.</p>

tom: <p>Yeah. 4. In other words, you take the probability vector for the categories and feed it into a Linear layer that has the embedding as weight.</p>

turian: <p><a class="mention" href="/u/tom">@tom</a> what is the best way to code this, if I have many categorical variables?</p>
<p>If I split the categoricals into an array of one-hot tensors, then it is a little easier to construct the net. However, I am concerned this would not be performant.</p>
<p>If I have all the values together, then I might need to have the dataset provide a list of ranges to slice. It’s a little trickier, but is that much faster?</p>

turian: <p>Also, is it correct that this will be faster if the categorical one-hot inputs are sparse tensors? I have 10-50 categorical variables, each has about 10 categories, and about 50 more continuous variables. I suspect that with such a low number of categories, that dense is faster.</p>

tom: <p>So if I understand this correctly, you have several variables and each of these has a moderate number of categories.</p>
<ul>
<li>If they’re the same categories (i.e. you have the same embedding), it is more efficient to batch them.</li>
<li>If you have distinct embeddings, combining them would create a block-diagonal matrix with many zeros. That usually isn’t as efficient to process unless you use block-sparse variables, so I’d avoid that.</li>
</ul>
<p>If, on the other hand, you have so many categories that using a linear layer is impractical, it is common to do negative sampling: If you first nn’s output is a vector <code>probs</code>, you can grab the <code>K_pos</code> largest indices (<code>i_pos = topk(probs, K_pos, dim=-1).indices</code> or somesuch) and <code>K_neg</code> random indices (<code>i_neg = randint((batch, K_neg)); i_all = torch.cat((i_pos, i_neg), dim=-1)</code>). Then you take <code>probs[:, i_all] @ emb.weights.t()[i_all] + emb.bias()</code> as embeddings.<br>
This is not similar to negative sampling in word2vec (I recommend Richard Socher’s/Chris Manning’s lectures on them if you want a very detailed account), except that there the vectors are leaves and the network following them is shallow.</p>
<p>Best regards</p>
<p>Thomas</p>

turian: <p><a class="mention" href="/u/tom">@tom</a> Thank you for the detailed feedback. I am aware of the negative sampling technique, it was originally used by Collobert + Weston (2008) in their work that was the first fast neural embedding method. (I re-implemented their embedding method and published work on , I changed my username in case you are curious about my work on word embeddings.)</p>
<p>I am not aware of how this would work in the context of a siamese network. Regardless, I have few categories (roughly 5-30 per category, median 10) per embedding. I have about 10 categorical variables. So this is not my issue.</p>
<p>You are right that this is slow.</p>
<p>I did the most naive thing which was decompose the categoricals into dense vectors, each with their own embedding layer with dimension 8:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51cb7fd3394b56a0abb270fa5a662fa0f5395dd7.png" data-download-href="https://discuss.pytorch.org/uploads/default/51cb7fd3394b56a0abb270fa5a662fa0f5395dd7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51cb7fd3394b56a0abb270fa5a662fa0f5395dd7_2_690x447.png" alt="image" data-base62-sha1="bFAFyWKbkGdOfCRZ27buSEkCY1p" width="690" height="447" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51cb7fd3394b56a0abb270fa5a662fa0f5395dd7_2_690x447.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51cb7fd3394b56a0abb270fa5a662fa0f5395dd7_2_1035x670.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51cb7fd3394b56a0abb270fa5a662fa0f5395dd7_2_1380x894.png 2x" data-dominant-color="F1F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1574×1020 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>
This was about 10x slower with 1000 examples, as I would expect. But with 200K examples, it seems like waaaaay slower, like orders of magnitude, which I did not expect at all. Do you have any idea why? I am currently training on CPU while prototyping, FYI.
</p>
<p>[edit: there were some floats in my input categories which exploded the number of classes]</p>
<p>I do understand that rewriting this network to take sparse inputs for the categories would be faster. I want to avoid this because:<br>
If I switch to sparse inputs, then later to pass the output of a softmax into this siamese network for prediction, or moreover to train a joint network with softmax feeding the siamese, I need dense inputs.</p>
<p>p.s. greetings from Berlin</p>

tom: <p>Oh, right. You’re in the first case, so negative sampling is not applicable, and you already know all about it.<br>
Would a large batch matmul help? You would have the difficulty amending the shorter vectors, but I think it might work.</p>
<p>Best regards</p>
<p>Thomas</p>

turian: <p>Are you suggesting that I transform all the categoricals into a one-hot that is of fixed length?<br>
Thank you for the advice by the way. It has been really helpful</p>