What is the exactly implementation of torch.embedding?

username: jasonyoung
cooked: <p>I am currently working on class <code>Embedding()</code> in PyTorch and I looked at its implementation.</p>
<p>In the <code>forward()</code> method, it calls the <code>F.embedding()</code> method: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/sparse.py#L124" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/sparse.py#L124</a>.</p>
<p>Then I found that <code>F.embedding()</code> method finally calls the <code>torch.embedding()</code> method: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L1814" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L1814</a>.</p>
<p>However, I could not find where is the implementation of the method <code>torch.embedding()</code>.</p>
<p>Could someone help me, please?<br>
Thanks!</p>

username: ptrblck
cooked: <p>It should eventually call into <a href="https://github.com/pytorch/pytorch/blob/5b03ff0a09d43d721067e39da10aa23edc6997cd/aten/src/ATen/native/Embedding.cpp#L14-L29">this method</a> for the forward pass.</p>

username: jasonyoung
cooked: <p>Thank you! Your answer is very helpful to me! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

username: Thomas_Ahle
cooked: <p>I’m trying to understand the code here.<br>
Can you help me understand why Embedding needs to use <code>index_select</code>, rather than simple <code>[...]</code>?<br>
I also wonder why it has to have its own backwards method? Can’t autograd figure out the indexing?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Thomas_Ahle" data-post="4" data-topic="87166">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thomas_ahle/48/35801_2.png" class="avatar"> Thomas_Ahle:</div>
<blockquote>
<p>rather than simple <code>[...]</code></p>
</blockquote>
</aside>
<p>This would be Python syntax and won’t work in C++, see <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html">Libtorch Indexing</a>.</p>
<aside class="quote no-group" data-username="Thomas_Ahle" data-post="4" data-topic="87166">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thomas_ahle/48/35801_2.png" class="avatar"> Thomas_Ahle:</div>
<blockquote>
<p>I also wonder why it has to have its own backwards method? Can’t autograd figure out the indexing?</p>
</blockquote>
</aside>
<p>The backwards methods might have been added for specific checks (such as sparsity) and are hooked into Autograd.</p>

username: Thomas_Ahle
cooked: <p>Thank you, this makes a lot of sense.</p>
<p>I was wondering about <code>index_select</code> in case it would allow me to get a sparse gradient if I used it instead of […] in python. But this doesn’t seem to work?</p>
<p>I think you are saying that autograd <em>is</em> able to make sparse gradients itself, but that the Embedding class added a custom backwards method here because they wanted extra checks?</p>
<p>Is there an indexing function that provides sparse gradients? Or should I implement indexing by means of a sparse matrix multiplication in order to achieve this?</p>