How does the nn.Embedding module relate intuitively to the idea of an embedding in general?

username: Wesley_Neill
cooked: <p>So, I’m having a hard time understanding <code>nn.Embedding</code>. Specifically, I can’t connect the dots between what I understand about embeddings as a concept and what this specific implementation is doing.</p>
<p>My understanding of an embedding is that it is a smaller dimension representation of some larger dimension data point. So it maps data in <code>N-d</code> to a <code>M-d</code> latent/embedding space such that <code>M &lt; N</code>.</p>
<p>As I understand it, this mapping is achieved through the learning process, as in an auto-encoder. The encoder learns the optimal embedding so that the decoder can reconstruct the original input.</p>
<p>So my question is, how does this relate to <code>nn.Embedding</code> module:</p>
<blockquote>
<p><em>A simple lookup table that stores embeddings of a fixed dictionary and size.<br>
This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.</em></p>
</blockquote>
<p>does this layer “learn” a lower dimensional representation of a larger input space? Or is it something else entirely?</p>

username: francois-rozet
cooked: <p>In the general case, an embedding is, indeed, a mapping from a <code>N</code>-d space to a <code>M</code>-d space. The simplest form of embedding is therefore a matrix <code>A</code> of shape <code>(M,N)</code> (but there are also non linear embeddings), such that the transformation is <code>y = A * x</code>.</p>
<p>However, in the field of Neural Machine Translation (NMT), the <strong>words/tokens</strong> are not represented by dense vectors like <code>x = [0.556, 0.2465, 0.015, ...]</code>, but rather by one-hot encodings, like <code>x = [0, 0, 0, 1, 0, 0, 0, ...]</code>. Hence, they are uniquely identified by the <strong>index</strong> of the <code>1</code> in <code>x</code>.</p>
<p>Therefore, instead of doing a product <code>A * x</code>, you can <strong>lookup</strong> the row in <code>A</code> corresponding to the index, like <code>y = A[x]</code>, which is more efficient.</p>
<p>That is exactly what <code>nn.Embedding</code> implements: it is a matrix which is used as a <strong>lookup table</strong> for the <strong>word indices</strong>.</p>

username: Wesley_Neill
cooked: <p>Okay, that makes <em>a little</em> sense to me. Let me see if I can put what you said in my own words, and you tell me if I understood correctly:</p>
<p>In the linear sense, an embedding is just PCA…</p>
<p>Then there are two cases… Let’s say a sequence of numerical data (like stock prices) and a sequence of characters in a sentence.</p>
<p>In the first case <code>y = Ax</code> is easy. But with the string of characters you have to one-hot-encode each character, so then you get a sequence of sparse vectors. This isn’t so easy to transform to a lower dimension because it’s not a simple column vector. So <code>nn.Embedding</code> is the solution to this problem?</p>
<p>So in my personal use case, I want to do anomaly detection through reconstruction error. I’m trying to train an autoencoder to reconstruct timeseries data. This, I think, falls into the first category where the sequence is a dense vector.</p>
<p>How does that affect my use of <code>nn.Embedding</code>?</p>

username: francois-rozet
cooked: <p>That is pretty much it! Although the one-hot-encoding is more likely to be “word-level” than “character-level”, but I might be wrong.</p>
<p>Anyway, in the case of stock prices, you indeed have a sequence of dense vectors already (I guess). You cannot represent each vector by an index and, therefore, you need a <code>y = Ax</code> transformation, as you said.</p>
<p>But <code>nn.Embedding</code> doesn’t do that; <code>nn.Linear</code> does it! So just go ahead and use that <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Wesley_Neill
cooked: <p>Ah okay! To be clear, let’s say I have daily samples:</p>
<pre><code class="lang-auto">['hi temp', 'inches rain', 'low temp']
</code></pre>
<p>Originally, I just made an autoencoder using <code>nn.Linear</code> along with ReLU activations, and reconstructed each day’s sample (all three variables). The anomaly detection was pretty poor with this approach.</p>
<p>So instead, I’m trying to think about it as three timeseries/sequences basically… I’m trying to learn how to reconstruct sequences using LSTMs instead, following this article: <a href="https://blog.floydhub.com/attention-mechanism/#bahdanau-att-step1" rel="noopener nofollow ugc">https://blog.floydhub.com/attention-mechanism/#bahdanau-att-step1</a></p>
<p>So in this case, I could use the <code>nn.Embedding</code> to as a way to have a dictionary of 3 separate embeddings, one for each of the 3 time-series?</p>
<p>Thank you for your help!</p>

username: francois-rozet
cooked: <p>I’m not sure to understand what you mean by “reconstructing sequences”. Do you mean like detecting AND correcting the anomalies ? Anyway, I’m not really qualified for that, sry <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>But, in the blog post you referenced, the author uses <code>nn.Embedding</code> because he works with words (language translation).</p>
<p>You case is more similar to this tutorial<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://curiousily.com/favicon-32x32.png" class="site-icon" width="16" height="16">
      <a href="https://curiousily.com/posts/time-series-anomaly-detection-using-lstm-autoencoder-with-pytorch-in-python/" target="_blank" rel="noopener nofollow ugc">Curiousily</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:117/61;"><img src="https://curiousily.com/static/2c204d86a4a8e69bf25fa17189766b6e/73f08/social-image.png" class="thumbnail" width="117" height="61"></div>

<h3><a href="https://curiousily.com/posts/time-series-anomaly-detection-using-lstm-autoencoder-with-pytorch-in-python/" target="_blank" rel="noopener nofollow ugc">Time Series Anomaly Detection using LSTM Autoencoders with PyTorch in Python</a></h3>

<p>Find abnormal heartbeats in patients ECG data using an LSTM Autoencoder with PyTorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Btw, you will note that this one doesn’t use <code>nn.Embedding</code> (or <code>nn.Linear</code>) but rather feeds the vector sequences directly into the LSTM(s).</p>

username: Wesley_Neill
cooked: <p>This is a perfect example for what I’m trying to do. Thank you!</p>