Normalizing Embeddings

Mehran: <p>I’m trying to manually normalize my embeddings with their L2-norms instead of using pytorch max_norm (as max_norm seems to have some bugs). I’m following <a href="https://discuss.pytorch.org/t/how-to-normalize-embedding-vectors/1209">this link</a> and below is my code:</p>
<pre><code>emb = torch.nn.Embedding(4, 2)
norms = torch.norm(emb.weight, p=2, dim=1).detach()
emb.weight = emb.weight.div(norms.expand_as(emb.weight))
</code></pre>
<p>But I’m getting the following error:</p>
<pre><code>Traceback (most recent call last):
File "&lt;stdin&gt;", line 1, in &lt;module&gt;
File "/usr/local/lib/python2.7/site-packages/torch/autograd/variable.py", line 725, in expand_as
return Expand.apply(self, (tensor.size(),))
File "/usr/local/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward
result = i.expand(*new_size)
RuntimeError: The expanded size of the tensor (2) must match the existing size (4) at non-singleton dimension 1. at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:308
</code></pre>
<p>When I look at the size of norms, it’s (4L,)<br>
Any idea where I’m going wrong? Thanks!</p>

chenyuntc: <aside class="quote no-group quote-modified" data-username="Mehran" data-post="1" data-topic="7696">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mehran/48/1708_2.png" class="avatar"> Mehran:</div>
<blockquote>
<p>emb.weight = emb.weight.div(norms.expand_as(norms))</p>
</blockquote>
</aside>
<p>shouldn’t it be?</p>
<pre data-code-wrap="Python"><code class="lang-Python">emb.weight = emb.weight.div(norms.expand_as(emb.weight))
</code></pre>

Mehran: <p>Yes, that was a typo. I edited the code, and also edited the error. Any idea why I’m getting that error?</p>

chenyuntc: <pre><code class="lang-auto">emb = torch.nn.Embedding(4, 2)
norms = torch.norm(emb.weight, p=2, dim=1).data
emb.weight.data = emb.weight.data.div(norms.view(4,1).expand_as(emb.weight))
</code></pre>

Mehran: <p>Thanks Chen. It worked.<br>
I realized that you have removed the “detach()” from the second line. Is it because “norms” is not a “Variable” anymore?</p>

chenyuntc: <p>Yes, you’re right.<br>
I prefer <code>variable.data</code>.</p>

jdhao: <p>What is the use of <code>detach()</code>, I do not think it is necessary here. I think we can also use</p>
<pre><code class="lang-auto">emb = torch.nn.Embedding(4,2)
norm = emb.weight.norm(p=2, dim=1, keepdim=True)
emb.weight = emb.weight.div(norm.expand_as(emb.weight))
</code></pre>
<p>Is there any problems with the above snippet?</p>

Mehran: <p>My understanding is that if we don’t detach it, then norm will be a variable and PyTorch will aim at optimizing its values in the backward phase when calculating the gradients. But we don’t really want to optimize the norm values.</p>

jdhao: <p>The type of <code>norm</code> is torch <code>Variable</code>. PyTorch will only calculate the the gradient of loss w.r.t to the leaf node. Since norm is not a leaf node, I do think it will be updated when we do <code>optimizer.step()</code>. Only <code>emb.weight</code> will be updated since it is of type <code>torch.nn.Parameter</code> and it is the learnable parameter of the module.</p>

falmasri: <p><a class="mention" href="/u/jdhao">@jdhao</a> I was seriously looking over the web for this kind of answer, Sorry of taking this discussion again up. Could you please explain more why norm is not a leaf node in the gradient computation ?</p>

Arvind_Vepa: <p>So this doesn’t precisely replicate the max-norm functionality because it’s not checking if the norm of the vector is less than max-norm, right (in which case, this function would increase the norm of those vectors)?</p>