Are "parametrizations" registered by autograd?

andrecavalcante: <p>The tool torch.nn.utils.parametrize allows to register parametrizations for the weights of a layer. For instance, it allows us to factor the weights of layer into triangular matrices so that the layer is symmetric. Does autograd compute gradients for the parametrized factors? That is, do  optimization steps keep the layer symmetric?</p>

albanD: <p>Hey!</p>
<p>the gradients will indeed “flow” all the way to the new Parameters (before the parametrization is applied). And the optimization is then applied pre-parametrization (since these are the new parameters).<br>
This means that your parametrization will re-run after the update and thus the property provided by it will be preserved indeed.</p>

andrecavalcante: <p>But in that way, the optimization step by itself does not seem to preserve the restriction imposed by the parametrization. The restriction will be imposed by applying the parametrization again after optimization. In this case, autograd does not calculate the gradient for the factors of the parametrization.  <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi André!</p>
<aside class="quote no-group" data-username="andrecavalcante" data-post="3" data-topic="204301" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrecavalcante/48/43057_2.png" class="avatar"> andrecavalcante:</div>
<blockquote>
<p>But in that way, the optimization step by itself does not seem to preserve the restriction imposed by the parametrization. The restriction will be imposed by applying the parametrization again after optimization.</p>
</blockquote>
</aside>
<p>This is correct.   But I do believe that pytorch’s scheme for optimizing<br>
<code>model.parametrizations.weight.original</code>, that is, your model’s<br>
“original” unconstrained <code>Parameter</code>, works correctly.*</p>
<aside class="quote no-group">
<blockquote>
<p>In this case, autograd does not calculate the gradient for the factors of the parametrization.</p>
</blockquote>
</aside>
<p>Yes, this is also correct.</p>
<aside class="quote no-group" data-username="andrecavalcante" data-post="1" data-topic="204301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrecavalcante/48/43057_2.png" class="avatar"> andrecavalcante:</div>
<blockquote>
<p>That is, do optimization steps keep the layer symmetric?</p>
</blockquote>
</aside>
<p>No, not in the sense that an actual layer weight stored in memory is kept<br>
symmetric (or otherwise respects the constraint that your parametrization<br>
imposes).</p>
<p>But what would it take to do this?</p>
<p>Consider a parameter that is a single two-dimensional point, <code>[x, y]</code>,<br>
that is constrained to have unit length.  That is to say, the parameter<br>
is a single point on a circle of radius one.</p>
<p>(You can consider this to be a baby version of a matrix that is constrained<br>
to be orthonormal for which similar, but more nuanced issues arise.)</p>
<p>When autograd computes the gradient of some loss function with respect<br>
to <code>[x, y]</code>, there is no reason for that gradient to lie tangent to the unit<br>
circle to which <code>[x, y]</code> is constrained.  You could add a feature so that<br>
when you write your parametrization you also write code to project the<br>
gradient to be tangent to the constraint (and autograd could automatically<br>
use such code), but for many reasonable constraints, this could be difficult<br>
to do and seems unlikely that this could be automated (other than purely<br>
numerically) for general constraints.</p>
<p>Even if you were to project the gradient to be tangent to the constraint,<br>
a <em>finite</em> optimization step (which could include momentum or weight<br>
decay) will almost certainly move you off the constraint surface.</p>
<p>All in all, the only practical approach for general constraints seems to<br>
be to let the gradient do its thing, let the finite optimization step do its<br>
thing, have your “original” parameter not satisfy the constraint, but<br>
then derive a tensor from the “original” parameter that does satisfy the<br>
constraint and that is then used in the forward-pass computation.</p>
<p>*)  It is possible for the “original” unconstrained parameter, for example<br>
in the unit-circle case, to drift off to infinity or down to zero over the<br>
course of many optimization steps, potentially leading to <code>inf</code>s or <code>nan</code>s.<br>
In such a case, it would be logically consistent to reimpose the constraint<br>
on the “original” parameter from time to time.</p>
<p>Best.</p>
<p>K. Frank</p>