Proper way to assign tensor to a Parameter

username: grudloff
cooked: <p>A parameter can be set to an arbitrary tensor by use of the <code>.data</code>. I have read that this is discouraged, what would be the proper way of doing this?</p>
<p>This issue arose for me in the context of using a reparametrization, when one is used, one can assign the parameter directly without the use of <code>.data</code>. Furthermore, using <code>.data</code> doesn’t modify the parameter.</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn.utils.parametrize import register_parametrization

class Identity(nn.Module):
    def forward(self, X):
        return X
    def right_inverse(self, Z):
        return Z

linear = nn.Linear(1,2) 
# this raises an error
linear.weight = torch.ones_like(linear.weight)
# this works
register_parametrization(linear, 'weight', Identity())
linear.weight = torch.ones_like(linear.weight)
# this does nothing
register_parametrization(linear, 'weight', Identity())
linear.weight.data = torch.ones_like(linear.weight)

</code></pre>
<p>Then, if I have a model definition, it must be dependent on whether it is reparametrized or not. Is there a “proper” way of assigning an arbitrary tensor to a parameter that works in these two cases?</p>

username: KFrank
cooked: <p>Hi Gabriel!</p>
<aside class="quote no-group" data-username="grudloff" data-post="1" data-topic="143696" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/grudloff/48/46485_2.png" class="avatar"> grudloff:</div>
<blockquote>
<p>what would be the proper way of doing this?</p>
</blockquote>
</aside>
<p>You may assign another <code>Parameter</code> to the <code>Linear</code>’s <code>weight</code> property<br>
(although I am not sure that I understand your use case).</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; linear = torch.nn.Linear (1, 2)
&gt;&gt;&gt; linear.weight
Parameter containing:
tensor([[-0.9916],
        [ 0.0040]], requires_grad=True)
&gt;&gt;&gt; linear.weight = torch.nn.Parameter (torch.ones_like (linear.weight))
&gt;&gt;&gt; linear.weight
Parameter containing:
tensor([[1.],
        [1.]], requires_grad=True)
&gt;&gt;&gt; linear (torch.tensor ([0.123])).sum().backward()
&gt;&gt;&gt; linear.weight.grad
tensor([[0.1230],
        [0.1230]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: thatgeeman
cooked: <p>Yup, not sure of the use actual use case but there’s a similar thread <a href="https://discuss.pytorch.org/t/how-to-manually-set-the-weights-in-a-two-layer-linear-model/45902">here</a>.</p>

username: grudloff
cooked: <p>Thank you both for your replies!</p>
<p>My usecase is in the context of a parameter that represents a physical value of a simulation, so essentially it’s a modeling problem. I can change the values of the simulation and I can either retrain that parameter or, in this case, I want to set it to focus on something else.</p>
<p><a class="mention" href="/u/kfrank">@KFrank</a> If I use the Parameter wrapper on the assignation it won’t work if it has a parametrization. This means:</p>
<pre><code class="lang-auto">linear = nn.Linear(1,2)
w = ...
# This works
linear.weight = nn.Parameter(w)
register_parametrization(linear, 'weight', Identity())
# This doesn't works
linear.weight = nn.Parameter(w)
</code></pre>
<p>The second assignation throws <code>KeyError: "attribute 'weight' already exists"</code></p>
<p><a class="mention" href="/u/thatgeeman">@thatgeeman</a> on the thread you link, <a class="mention" href="/u/ptrblck">@ptrblck</a> emphasizes the use of <code>no_grad</code> (I think) for not messing up the grad calculation, but he doesn’t address what is a proper assignation. What I mean is that, for example, wrapping <code>layer.weight = w</code> in <code>no_grad</code> won’t work if there is no parametrization.</p>
<p>On a sidenote, related to the use of <code>no_grad</code>. If a tensor is assigned to a parameter, how could this mess up grad calculations if it isn’t inside a <code>with no_grad</code>?</p>

username: thatgeeman
cooked: <aside class="quote no-group" data-username="grudloff" data-post="4" data-topic="143696">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/grudloff/48/46485_2.png" class="avatar"> grudloff:</div>
<blockquote>
<p>Parameter wrapper on the assignation it won’t work if it has a parametrization</p>
</blockquote>
</aside>
<p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html" rel="noopener nofollow ugc">docs</a>:</p>
<blockquote>
<p>The first time that a module registers a parametrization, this function will add an attribute <code>parametrizations</code></p>
</blockquote>
<blockquote>
<p>parametrizing <code>weight</code> , <code>layer.weight</code> is turned into a Python property. This property computes <code>parametrization(weight)</code> every time we request <code>layer.weight</code></p>
</blockquote>
<blockquote>
<p>This <code>parametrizations</code> attribute is an <code>nn.ModuleDict</code></p>
</blockquote>
<blockquote>
<p>The list of parametrizations on the tensor <code>weight</code> will be accessible under <code>module.parametrizations.weight</code></p>
</blockquote>
<p>After parametrization, directly modifying <code>linear.weight = w</code> (<code>w</code> being an arbitrary tensor) should work. But re-registering a new parametrization with the same name <code>weight</code> may be erroneous. Maybe you want to remove the existing parametrization (see docs) before?</p>
<aside class="quote no-group" data-username="grudloff" data-post="4" data-topic="143696">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/grudloff/48/46485_2.png" class="avatar"> grudloff:</div>
<blockquote>
<p>If a tensor is assigned to a parameter, how could this mess up grad calculations if it isn’t inside a <code>with no_grad</code>?</p>
</blockquote>
</aside>
<p>In his answer, the usage of <code>no_grad</code> contextmanger is demonstrated where the weights are in addition being manipulated I believe.</p>

username: grudloff
cooked: <aside class="quote no-group" data-username="thatgeeman" data-post="5" data-topic="143696" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thatgeeman/48/44760_2.png" class="avatar"> thatgeeman:</div>
<blockquote>
<p>After parametrization, directly modifying <code>linear.weight = w</code> (<code>w</code> being an arbitrary tensor) should work. But re-registering a new parametrization with the same name <code>weight</code> may be erroneous. Maybe you want to remove the existing parametrization (see docs) before?</p>
</blockquote>
</aside>
<p>Yes, you are right! I guess I should reformulate my question. How can I have an assignation of a parameter that works both for a vanilla parameter and for a parametrized parameter? Because right now I don’t see a way that works in both cases.</p>
<p>Edit: I could add an if-else clause, but this would make the code cumbersome. Something like this:</p>
<pre><code class="lang-auto">    if linear.parametrizations:
        linear.weight = tensor(w)
    else:
        model.linear = nn.Parameter(tensor(w))
</code></pre>