A layer whose weight is the sum of other two weights, but gradient is None

username: wangtj20
cooked: <p>I want to create a linear layer, whose weight is the weighted sum of other two linear layers. The weighted factor is denoted as <em>alpha</em>, i.e.</p>
<p><code>outputs = [alpha*W1+(1-alpha)*W2] * inputs</code></p>
<p>I want the gradient of loss wrt W1, W2 and also alpha, but after loss.backward(), both gradients are None. It seems that W1, W2 or alpha are all not used in the computational graph. Is there a way to incorporate these into the graph?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

fc1 = nn.Linear(4, 6, bias=False)
fc2 = nn.Linear(4, 6, bias=False)
fc_sum = nn.Linear(4, 6, bias=False)
alpha = torch.tensor(0.1, requires_grad=True)
fc_sum.weight = nn.Parameter(fc1.weight.data * alpha + fc2.weight.data * (1 - alpha))

inputs = torch.rand([1, 4])
outputs = fc_sum(inputs)
loss = torch.sum(outputs ** 2)
loss.backward()

print(alpha.grad)
print(fc1.weight.grad)
</code></pre>

username: KFrank
cooked: <p>Hi Wang!</p>
<aside class="quote no-group" data-username="wangtj20" data-post="1" data-topic="160657" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/838e76/48.png" class="avatar"> wangtj20:</div>
<blockquote>
<p><code>outputs = [alpha*W1+(1-alpha)*W2] * inputs</code></p>
<p>I want the gradient of loss wrt W1, W2 and also alpha</p>
</blockquote>
</aside>
<p>The simplest approach would simply be:</p>
<pre data-code-wrap="python"><code class="lang-python">outputs = alpha * fc1 (inputs) + (1 - alpha) * fc2 (inputs)
</code></pre>
<p>Calling <code>loss.backward()</code> will properly populate <code>grad</code> for <code>fc1.weight</code>,<br>
<code>fc2.weight</code>, and <code>alpha</code>.</p>
<p>There is no need to package the weighted combination of <code>fc1</code> and <code>fc2</code><br>
as a separate <code>Linear</code>.</p>
<p>(If the simple approach doesn’t work for you, could you explain your use<br>
case in greater detail?)</p>
<p>Best.</p>
<p>K. Frank</p>

username: wangtj20
cooked: <p>Thanks Frank.</p>
<p>However, the example I provide is only the simplest form. What I really want to do is to parametrize every weight in a deep neural network (conv weight, linear weight, bn weight etc) with the parameter alpha. Also, the number of Wi may be greater than 2, and the interpolation method could be much more complicated than linear combination. For example, rather than</p>
<pre><code class="lang-auto">W=alpha*W1+(1-alpha)*W2
</code></pre>
<p>I would like</p>
<pre><code class="lang-auto">W=f1(alpha)*W1+f2(alpha)*W2+...+fk(alpha)*Wk
</code></pre>
<p>W can be weight not only from linear layers, but also <strong>from conv layers</strong> and so on. I wonder if it’s still possible to explicitly write out the formula in this situation. Thus I would like to know whether we can deal with the layer weight directly.</p>

username: wangtj20
cooked: <p>I think I have come up with a solution, with the help of <strong>nn.functional</strong></p>
<pre><code class="lang-auto">class LinearSum(nn.Module):
    def __init__(self):
        super(LinearSum, self).__init__()
        self.weight1 = nn.Parameter(torch.randn(6, 4))
        self.weight2 = nn.Parameter(torch.randn(6, 4))

    def forward(self, x, a):
        weight = self.weight1 * a + self.weight2 * (1 - a)
        return F.linear(x, weight, bias=None)


fc_sum = LinearSum()
alpha = torch.tensor(0.1, requires_grad=True)
inputs = torch.rand([1, 4])
outputs = fc_sum(inputs, alpha)
loss = torch.sum(outputs ** 2)
loss.backward()

print(alpha.grad)
print(fc_sum.weight1.grad)
</code></pre>
<p>Any other implementation or suggestion is welcomed!</p>