Generating vector on an n-sphere

denvercoder9: <p>Hi:)</p>
<p>Is there an efficient way to create an unit vector on an n-sphere from n-1 parameters? That is I want to implement these equations (taken from <a href="https://en.wikipedia.org/wiki/N-sphere" rel="noopener nofollow ugc">wikipedia.org/wiki/N-sphere</a>):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/beb0c8b584bde50173960df8590e522cdc3be3ce.png" alt="nsphere" data-base62-sha1="rcVtB7Z1UqA01gcgiNmi4JMuxoa" width="380" height="180"><br>
with r=1 s.t. I can use the result  with autograd w.r.t. phi. The fact that I want to use autograd might be a constraint since one has to pay attention if vector elements are overwritten…</p>
<p>Currently I am using <code>softmax</code> on n parameters which is not equivalent to the <em>unit vector on n-sphere</em> approach since there is one additional degree of freedom. The problem is that my NN occasionally generates extremely small and large values for all n parameters which is not penalize in my loss function due to the normalization of the <code>softmax</code> function. (The magnitude of the vector of the n parameters does not play a role in the loss function.)</p>
<p>Any suggestions?</p>
<p>Thanks for your help!</p>

KFrank: <p>Hi Denver!</p>
<aside class="quote no-group" data-username="denvercoder9" data-post="1" data-topic="124352" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/denvercoder9/48/30544_2.png" class="avatar"> denvercoder9:</div>
<blockquote>
<p>Is there an efficient way to create an unit vector on an n-sphere from n-1 parameters?</p>
</blockquote>
</aside>
<p>First a brief word on mathematical terminology:  A <em>n-sphere</em> is<br>
described by n (independent) parameters and is often thought<br>
of as being embedded in (n+1)-space.</p>
<p>So a 2-sphere is the (two-dimensional) surface of a three-dimensional<br>
ball (embedded in 3-space).  A 1-sphere is just the ordinary circle.</p>
<aside class="quote no-group">
<blockquote>
<p>That is I want to implement these equations</p>
</blockquote>
</aside>
<p>You most likely <em>don’t</em> want to implement these equations because<br>
there are many locations where they are singular.  (Also, the different<br>
parameters are rather different in character.)</p>
<p>For most purposes you are much better off using n+1 slightly<br>
redundant parameters to describe an n-sphere, namely the n+1<br>
coordinates of a point in (n+1)-space that is constrained to be a<br>
distance of 1 away from the origin.  (This constraint describes /<br>
eliminates the redundancy.)</p>
<aside class="quote no-group">
<blockquote>
<p>Currently I am using <code>softmax</code> on n parameters which is not equivalent to the <em>unit vector on n-sphere</em> approach since there is one additional degree of freedom.</p>
</blockquote>
</aside>
<p>Using <code>softmax</code> here is likely to be sub-optimal, because, among<br>
other reasons, the “geometry” of <code>softmax</code> doesn’t really match well<br>
with the geometry of a sphere.</p>
<p>If you want the output of your model to be an n-sphere, you should<br>
have your model output n+1 unbounded real values (e.g., the<br>
n+1 outputs of a final <code>Linear</code> layer), and then normalize that<br>
(n+1)-dimensional vector to have unit norm.  (That extra degree of<br>
freedom that is normalized away isn’t really a problem.  Networks,<br>
in general, contain lots of redundancy.)</p>
<aside class="quote no-group">
<blockquote>
<p>The problem is that my NN occasionally generates extremely small and large values for all n parameters which is not penalize in my loss function due to the normalization of the <code>softmax</code> function. (The magnitude of the vector of the n parameters does not play a role in the loss function.)</p>
</blockquote>
</aside>
<p>I suggest directly normalizing your (n+1)-dimensional output vector,<br>
rather than passing it through <code>softmax</code>, but the general concept is<br>
fully analogous.</p>
<p>As you say, because the norm of your (pre-normalization) vector<br>
doesn’t enter into your loss function, there’s nothing that keeps it<br>
from running off to infinity (or zero).  But it’s easy to stabilize.  Just<br>
add a penalty like</p>
<pre data-code-wrap="python"><code class="lang-python">stabilization_loss = (1.0 - output_norm**2)**2
</code></pre>
<p>to your total loss.</p>
<p>Note, you don’t care whether your <code>stabilization_loss</code> forces your<br>
output vector to have a norm that is exactly (or very close to) 1 – you<br>
just want to keep the norm from running off to infinity and becoming<br>
singular.</p>
<p>Best.</p>
<p>K. Frank</p>

denvercoder9: <p>These are all valid points and thank you for your suggestion. Much appreciated!</p>