How to normalize embedding vectors?

maplewizard: <p>Hi, I am using a network to embed some entity into vector space. As the length of the vector decrease during the training. I want to normalize it’s length to 1 in the end of each step. Is there any tool that I can use to normalize the embedding vectors?</p>

apaszke: <p>I think the best thing you can do is to save the embedded indices, and normalize their rows manually after the update (just index_select them, compute row-wise norm, divice, index_copy back into weights). We only support automatic max norm clipping.</p>

samarth-robo: <p>If you want to normalize a vector as a part of a model, this should do it:</p>
<p>assume q is the tensor to be L2 normalized, along dim 1</p>
<blockquote>
<p>qn = torch.norm(q, p=2, dim=1).detach()<br>
q = q.div(qn.expand_as(q))</p>
</blockquote>
<p>Note the <code>detach()</code>, that is essential for the gradients to work correctly. I’m assuming you want the norm to be treated as a constant while dividing the Tensor with it.</p>

asdass: <p>Can I use it to normalise the embedding after each update in the training ?</p>

samarth-robo: <p>Yes it could be used to normalize an embedding. I suggest not using the <code>detach()</code> though. I found that it degrades performance.</p>

asdass: <p>I see, thanks.  But I need to do the constraints on the norms of embeddings( not bigger than 1 ). Do you have any better suggestions to do it?</p>

jdhao: <p>why do we have to use <code>detach()</code>? Also in the new PyTorch version, you have to use <code>keepdim=True</code> in the <code>norm()</code> method.  A simple implementation of L2 normalization:</p>
<pre><code class="lang-auto"># suppose x is a Variable of size [4, 16], 4 is batch_size, 16 is feature dimension
x = Variable(torch.rand(4, 16), requires_grad=True)
norm = x.norm(p=2, dim=1, keepdim=True)
x_normalized = x.div(norm.expand_as(x))
</code></pre>

moi90: <p>If you use <code>keepdim=True</code>, you don’t even need <code>expand_as(x)</code>. The following works for me:</p>
<pre><code class="lang-auto">norm = x.norm(p=2, dim=1, keepdim=True)
x_normalized = x.div(norm)
</code></pre>

jdhao: <p>Now PyTorch have a <a href="http://pytorch.org/docs/master/nn.html#torch.nn.functional.normalize">normalize function</a>, so it is easy to do L2 normalization for features. Suppose <code>x</code> is feature vector of size <code>N*D</code> (<code>N</code> is batch size and <code>D</code> is feature dimension), we can simply use the following</p>
<pre><code class="lang-auto">import torch.nn.functional as F
x = F.normalize(x, p=2, dim=1)
</code></pre>

Liang: <aside class="quote no-group" data-username="jdhao" data-post="9" data-topic="1209">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jdhao/48/2558_2.png" class="avatar"> jdhao:</div>
<blockquote>
<p>= F.normalize(x, p=2,</p>
</blockquote>
</aside>
<p>Yes, it failed quickly with <code>detach()</code></p>

sssohrab: <p>What if the variable I am trying to normalize is in fact a <code>Parameter </code> from <code>nn.parameter</code> rather than a  <code>Variable </code> ?</p>
<p>Then in this case I get the following error for any of the above options I try:</p>
<p><strong>TypeError: cannot assign ‘torch.autograd.variable.Variable’ as parameter ‘W’ (torch.nn.Parameter or None expected)</strong></p>
<pre><code class="lang-auto">class myUnit(nn.Module):
    def __init__(self,myParameter):
        super(myUnit, self).__init__()
        self.myParameter = Parameter(myParameter,requires_grad=True)
    def forward(self,input):
        """
        Whatever operation. Just an example:
        """
        self.myParameter = F.normalize(self.myParameter,p=2,dim=1)
        output = self.myParameter * input - 1
        return output</code></pre>
<p>Can I still keep using <code>Parameter </code>  in my customized network and be able to also normalize it?</p>

Guohai93: <p>May be you can try:</p>
<pre><code class="lang-auto">self.myParameter.weight.data = F.normalize(self.myParameter.weight.data, p=2, dim=1)
</code></pre>

Ritwick_Chaudhry: <p><a class="mention" href="/u/sssohrab">@sssohrab</a> I’m facing the exact same issue. Can you please let me how exactly you got through this? Also, did you want the gradients to pass through the norm step?</p>

f10w: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please help answer this question from <a class="mention" href="/u/sssohrab">@sssohrab</a>? I am facing the same issue (“RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss.”)<br>
Thank you so much in advance for your help!</p>

ptrblck: <p>Your issue seems to be unrelated to the linked post, as your error points towards an error running a DDP model.</p>

f10w: <p>Sorry I copied the wrong error message. The above message was for the following code:</p>
<pre><code class="lang-python">self.embeddings = torch.nn.Parameter(F.normalize(self.embeddings, p=2, dim=-1))
</code></pre>
<p>What I would like to do is to constraint the weights of my model to always lie on a sphere. And I finally found the solution:</p>
<pre><code class="lang-python">self.embeddings.data.copy_(torch.nn.Parameter(F.normalize(self.embeddings.data, p=2, dim=-1)))
</code></pre>
<p>Hope that this will be useful for future readers.</p>