Is it possible to have trainable module parameters in between static layer weights?

username: kaimatzu
cooked: <p>Let’s say I have a module:</p>
<pre><code class="lang-auto">class mod(nn.Module):
    def __init__(self, in_features, out_features):
        super(mod, self).__init__()
        self.scalar = nn.Parameter(torch.tensor([1.1]), requires_grad=True)
        self.linear1 = nn.Linear(in_features, 10, bias=False, requires_grad=False)
        self.linear2 = nn.Linear(10, out_features, bias=False, requires_grad=True)
    def forward(self, input):
        x = input * scalar
        x = self.linear1(x) 
        x = torch.relu(x)
        x = self.linear2(x) 
        x = torch.relu(x)
        return x
</code></pre>
<p>Would autograd be able to adjust any arbitrary parameter of the entire forward pass even though there are layers in between that are not learnable? And what if I use some learnable non-linearity like a threshold in between static layers?</p>
<pre><code class="lang-auto">class threshold(nn.Module):
    def __init__(self):
        super(threshold, self).__init__()
        self.t = nn.Parameter(torch.tensor([0.2]), requires_grad=True)
    
    def forward(self, input):
        return torch.where(x &gt; self.t, x, torch.tensor(0.0))

class mod(nn.Module):
    def __init__(self, in_features, out_features):
        super(mod, self).__init__()
        self.scalar = nn.Parameter(torch.tensor([1.1]), requires_grad=True)
        self.linear1 = nn.Linear(in_features, 10, bias=False, requires_grad=False)
        self.linear2 = nn.Linear(10, out_features, bias=False, requires_grad=False)
        self.threshold1 = threshold()
        self.threshold2 = threshold()
    def forward(self, input):
        x = input * scalar
        x = self.linear1(x) 
        x = self.threshold1(x)
        x = self.linear2(x) 
        x = self.threshold2(x)
        return x
</code></pre>
<p>Can the gradient still be propagated through here? This is for a hypothetical scenario in which I need to finetune some arbitrary set of learnable module parameters without affecting a network’s main linear layer weights.</p>

username: KFrank
cooked: <p>Hi Karl!</p>
<aside class="quote no-group" data-username="kaimatzu" data-post="1" data-topic="215373" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaimatzu/48/72298_2.png" class="avatar"> kaimatzu:</div>
<blockquote>
<p>Would autograd be able to adjust any arbitrary parameter of the entire forward pass even though there are layers in between that are not learnable?</p>
</blockquote>
</aside>
<p>Yes.  As long as your non-learnable layers are differentiable, autograd will backpropagate<br>
properly through your interspersed learnable layers, populating their <code>.grad</code> properties<br>
correctly.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        self.linear1 = nn.Linear(in_features, 10, bias=False, requires_grad=False)
</code></pre>
</blockquote>
</aside>
<p>Note, as written, this won’t work, because <code>Linear</code> doesn’t have a <code>requires_grad</code><br>
constructor argument.  For a non-learnable <code>Linear,</code> you would probably want to use<br>
the functional version, <a href="https://pytorch.org/docs/2.5/generated/torch.nn.functional.linear.html" rel="noopener nofollow ugc">torch.nn.functional.linear().</a></p>
<p>Best.</p>
<p>K. Frank</p>