Exclude parameters from model

LIKAN_BLK: <p>I have a pretty powerful model (inherited from nn.Module), that I want to reuse. The problem is that I want to replace some model Parameters with Tensors to make gradient flow through them. I need it because in my implementation these properties of the model will be outputs of another model. So I need a way to unregister Parameters with the corresponding names from the model. How I can do it.</p>

ptrblck: <p>I assume  you don’t want to retrain these parameters?<br>
If so, you could simply freeze them by setting their <code>.requires_grad</code> attribute to <code>False</code>.<br>
Alternatively, you could register them as buffers via <code>self.register_buffer(name, tensor)</code>.</p>

LIKAN_BLK: <p>I whant to train variables, used for constraction of this Parameters. So gradient should flow through them. But call to nn.Parameter(…) breaks the computational graph. So I have to replace class property, defined as Parameter with class property with the same name, defined as Tensor.</p>

ptrblck: <p>Could you post a code snippet which shows your use case, please?</p>

LIKAN_BLK: <p>I have</p>
<pre><code class="lang-auto">class VeryUsefullLegacyParent(nn.Module):
  def __init__(self,n):
    self.b = nn.Parameter(torch.zeros(n))
</code></pre>
<p>I want to create</p>
<pre><code class="lang-auto">class MyChild(VeryUsefullLegacyParent):
  def __init__(self, a: torch.Tensor):
    super(MyChild, self).__init__(len(a))
    self.b = nn.functional.sigmoid(a)
</code></pre>
<p>My aim to get a gradient w.r.t <code>a</code> (and deeper). But I can’t replace <code>b</code> in such straighforward way, beacause it already registered as a model parameter. I want to use the same name for this variable,  because I have a lot of functions working with <code>VeryUsefullLegacyParent</code> that access this variable.</p>

ptrblck: <aside class="quote no-group" data-username="LIKAN_BLK" data-post="3" data-topic="77238">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/ed8c4c/48.png" class="avatar"> LIKAN_BLK:</div>
<blockquote>
<p>I whant to train variables, used for constraction of this Parameters. So gradient should flow through them.</p>
</blockquote>
</aside>
<p>Does it mean you would like to get the gradient for <code>a</code> and update it?</p>
<aside class="quote no-group" data-username="LIKAN_BLK" data-post="5" data-topic="77238">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/ed8c4c/48.png" class="avatar"> LIKAN_BLK:</div>
<blockquote>
<p>My aim to get a gradient w.r.t  <code>a</code>  (and deeper). But I can’t replace <code>b</code> in such straighforward way, beacause it already registered as a model parameter.</p>
</blockquote>
</aside>
<p>You could get the gradients for <code>a</code>, if it’s a parameter.<br>
Why do you want to replace <code>self.b</code>, if it’s also a parameter, which is trainable?</p>

LIKAN_BLK: <ol>
<li>If I construct <code>b=nn.Parameter(a)</code>, I wouldn’t get the gradient w.r.t <code>a</code>, because of call to <code>nn.Parameter</code> breaks the computational graph and <code>b</code> will be a leaf of the graph. (Maybe I’m mistaking, but my experiments showed that it works like this)</li>
<li>In <code>MyChild</code> class<code> b</code> should be the function of a. So <code>b</code> shouldn’t be updated via optimizer. But gradient should flow throw <code>b</code>, because I will update <code>a</code>
</li>
</ol>

ptrblck: <ol>
<li>
<p>Yes, this behavior should be correct.</p>
</li>
<li>
<p>It seems you might be thinking about a static graph definition, was was used e.g. in Theano. In PyTorch you don’t define <code>b</code> as a “method” on <code>a</code>, but instead calculate it dynamically in the forward pass.</p>
</li>
</ol>