Extract feature vector/latent factors from Embedding layer in Pytorch

username: Jona
cooked: <p>Hi,</p>
<p>I have some conceptual questions:</p>
<p>I trained a NN with the following arquitecture</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79e40c7bc0db00ab79a658150a21ac6e54d5ab3e.png" data-download-href="https://discuss.pytorch.org/uploads/default/79e40c7bc0db00ab79a658150a21ac6e54d5ab3e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79e40c7bc0db00ab79a658150a21ac6e54d5ab3e.png" alt="image" data-base62-sha1="hoimHsWcYYDY2cNeTbkz3wRuUA6" width="689" height="159" data-dominant-color="E3E7E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1299×301 6.76 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol>
<li>
<p>¿How can I extract the feature vectors from my embeddings layers?. If I’m not wrong, these feature vectors are the weights of the layers</p>
</li>
<li>
<p>¿Can I use the feature vectors to feed them into others models, like Randon Forest?, ¿Do I need a separate trained NN just for the embeddings, so that the can be used in others models?</p>
</li>
<li>
<p>If I were to train an NN just for the embbedings, ¿wich would be the target variable?. According to above arquitecture, the target needs also the continius features to be predicted.</p>
</li>
<li>
<p>¿What am I supposed to do in the case, let’s say, that the test/validation set contains a level for some categorical value, wich it’s not present in the training set? In that case, the trained embedding layers won’t have a feature vector for that absent level in test/validation</p>
</li>
</ol>
<p>Thanks, and best regards</p>

username: ptrblck
cooked: <p>Some partial answers:</p>
<ol>
<li>
<p>Depending what you mean by “feature vectors”, you might either access the <code>.weight</code> attribute or use forward hooks to get the activation output of this layer as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
</li>
<li>
<p>Yes, you could store the features and train another model with it. Based on this use case, you would refer to the output activations as features. Otherwise, if you are referring to the <code>.weight</code> parameter, you would only have a single sample and training wouldn’t make sense.</p>
</li>
<li>
<p>The embeddings would be trained as part of the complete model for your current use case, e.g. a classification. I’m not sure how you would like to train only the embeddings.</p>
</li>
<li>
<p>I don’t know what the best approach would be for missing tokens.</p>
</li>
</ol>

username: Jona
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<ol start="2">
<li>Yes, I want to extract the <code>weights</code>  of the <code>embeddings</code> layers (wich essentialy have captured semantic relationships between the labels o <code>levels</code> of a catagorical feature during the training of my NN) and treat them as feature for a Random Forest model … Obviously, I would use the same originals datasets. The change comes in the  form of using the <code>embeddings</code> instead of some other encoder technique (like LabelEncoder or OneHot, wich don’t provide meanindull relationships between the categorical features and its <code>levels</code>). I’ll also use, of course the numerical continius features.</li>
</ol>
<p>In a nutshell, my new dataset for a Random Forest model would be <code>embeddings</code> + continius features (<code>embeddings</code> replacing categorical features)</p>
<p>I hope I’ve explained myself well</p>
<p>Best regards</p>

username: ptrblck
cooked: <p>Thanks for the explanation.<br>
If I understand the use case correctly, you would like to use the pretrained embeddings to create the continuous output features for categorical inputs and train a new classifier with it?<br>
If so, then this would be supported and you could either store the output activations for all inputs directly using forward hooks or just use the <code>nn.Embedding</code> layer in isolation, feed the input to it, transform the output to numpy arrays, and train your new classifier with it.</p>

username: Jona
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> for your feedback …</p>
<blockquote>
<p>If I understand the use case correctly, you would like to use the pretrained embeddings to create the continuous output features for categorical inputs and train a new classifier with it?</p>
</blockquote>
<p>YES</p>
<blockquote>
<p>If so, then this would be supported and you could either store the output activations for all inputs directly using forward hooks or just use the  <code>nn.Embedding</code>  layer in isolation, feed the input to it, transform the output to numpy arrays, and train your new classifier with it.</p>
</blockquote>
<p>I don’t undertand very well … sorry</p>
<p>This the context<br>
<a href="https://github.com/jonmunm/nbs/blob/master/Kaggle/UCI%20Rep%20-%20Adults/UCI%20Rep%20-%20Adults.ipynb" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/jonmunm/nbs/blob/master/Kaggle/UCI%20Rep%20-%20Adults/UCI%20Rep%20-%20Adults.ipynb</a></p>
<p>I trained a model using a fully conected NN with embeddings. The target es the column <code>income</code>. The model performs well (you can see the metrics).</p>
<p>But now, I want to use XGBoost and give it a try. However, considering the originals dataset’s, <code>income</code> is predicted using the categorical and continius features. So, I don’t want to use OneHot o similars, because I would be losing the semantic between the categorical features, and that’s why I want to feed the embeddings vectors into XGBoost, with the others continius features obviously. So, my questions are:</p>
<ol>
<li>Can I just take the <code>weights</code> from my <code>nn.Embedding</code> layers from my model <code>UciAdultsClassifier </code> and feed them into XGBoost? As I said, with also the others continius features. I’m just replacing the form of the categorical features</li>
<li>Do I need to train a separate fully connected NN with just the categorical variables using <code>nn.Embedding</code> layers and predict <code>income</code>? is this correct or the right way of using <code>embeddings</code> vectors in others models than NN’s? Is this approach correct, that is, train a new NN with just <code>nn.Embedding</code> layers to predict <code>income</code> (and not consider the continius features. Danger !!), and then use those <code>embedding</code> vectors or <code>weights</code> to use them in XGBoost?</li>
</ol>
<p>I hope I’ve explained myself well</p>
<p>Thanks very much, best regards</p>

username: Andreas_Georgiou
cooked: <p>Your nn.Embedding contains one vector for each category. So of course you would have to repeat it as necessary to fill your dataset. I am just making that clear</p>

username: Jona
cooked: <p>Yes I know,</p>
<p>Specifically, one <code>embedding</code> vector for every level in each categorical feature. And, for every ocurrence of level (or label, value… whatever) <code>Never-married</code> of the categorical feature <code>marital-status</code> should be replaced by the <code>embedding</code> vector of <code>Never-married</code></p>
<p>Thanks for making it clearer <a class="mention" href="/u/andreas_georgiou">@Andreas_Georgiou</a></p>