How to "unregister" a Parameter from a module

username: milad
cooked: <p>Is it possible to <strong>un</strong>register a <code>Parameter</code> from an instance of a <code>nn.Module</code>? Let’s say I want to go through all <code>Conv2d</code> layers of a network and replace all <code>weight</code> parameters with my own custom nn.module? I can’t simply re-assign the <code>weight</code> attribute with my own module as I get:</p>
<pre><code class="lang-auto">TypeError: cannot assign 'CustomWeight' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>
<p>(I don’t want to define a custom <code>Conv2d</code> layer because I don’t want users  of my application to change code on their side)</p>

username: ptrblck
cooked: <p>Based on the error message it looks like you are trying to assign plain <code>tensors</code> as the parameters. Could you try to warp them in <code>nn.Parameter</code> and wrap the assignment in a <code>with torch.no_grad():</code> guard?</p>

username: milad
cooked: <p>That doesn’t seem to work:</p>
<pre><code class="lang-auto">TypeError: _make_subclass(): argument 'data' (position 2) must be Tensor, not CustomWeight
</code></pre>
<p>I think the error is fair. I’m basically trying to replace a <code>Parameter</code> with <code>nn.Module</code></p>

username: ptrblck
cooked: <p>Yeah, that won’t work.<br>
Could you share the code for <code>CustomWeight</code>?<br>
If that’s not possible, could you explain, how you wanted to assign this module to the conv parameters?<br>
I assume <code>CustomWeight</code> is not just storing the parameters, but also applies some operations. Is that correct?</p>

username: milad
cooked: <p>So imagine at some point in the training I want to multiply my weights by a mask before they’re used in the forward method (I want to freeze the weights and learn the mask instead).</p>
<p>I can of course inherit all the layers I’m interested in and add this mask and override the forward method but I was thinking if I could dynamically remove the <code>weight</code> parameter in the modules and monkey-patch them with something like below:</p>
<pre><code class="lang-auto">class CustomWeight(nn.Module):
    def __init__(self, weight):
        self.weight_mask = nn.Parameter(torch.ones_like(weight))
        
        self.weight = weight.clone()
        self.weight.requires_grad = False

    def forward(self, x):
        return self.weight * self.weight_mask
</code></pre>
<p>Does this make sense?</p>

username: vmirly1
cooked: <p>Can you remove that from the parameters passed to the optimizer? That way, this specific parameter will not be updated anymore.</p>

username: tom
cooked: <p>It does make sense, but what you want is currently not supported in PyTorch.<br>
There is <a href="https://github.com/pytorch/pytorch/issues/7313" rel="nofollow noopener">issue #7313 that discusses this</a>.<br>
I once implemented a little technical exploration (linked from there).<br>
There still is some latent interest in doing this eventually, but sadly It would seem that it’s not a priority.</p>
<p>Apropos spectral_norm in the title of that issue: There you have a similar goal and can see that the current approach is - let’s say - somewhat elaborate.</p>
<p>Best regards</p>
<p>Thomas</p>

username: eplu
cooked: <aside class="quote no-group" data-username="vmirly1" data-post="6" data-topic="36424" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vmirly1/48/3309_2.png" class="avatar"> vmirly1:</div>
<blockquote>
<p>Can you remove that from the parameters passed to the optimizer? That way, this specific parameter will not be updated anymore.</p>
</blockquote>
</aside>
<p>Is there some reason this doesn’t solve the problem (at least on a case-by-case basis)?</p>