Initialize nn.Linear with specific weights

username: Diego999
cooked: <p>Hi everyone,</p>
<p>Basically, I have a matrix computed from another program that I would like to use in my network, and update these weights.</p>
<pre><code class="lang-auto">In [1]: import torch
In [2]: import torch.nn as nn
In [4]: linear_trans = nn.Linear(3,2)
In [5]: my_weights = torch.tensor([[1,2],[3,4],[5,6]])
In [6]: linear_trans.weight
Out[6]: 
Parameter containing:
tensor([[ 0.4409, -0.2647,  0.3328],
        [ 0.1049, -0.1887, -0.2617]], requires_grad=True)

In [7]: linear_trans.weight.data = my_weights
In [8]: linear_trans.weight
Out[8]: 
Parameter containing:
tensor([[1, 2],
        [3, 4],
        [5, 6]], requires_grad=True)
</code></pre>
<p>Is this enough for :</p>
<ol>
<li>Use these weights as parameters for nn.Linear (without changing the values of the bias vector)</li>
<li>Update these with .backwards automatically</li>
</ol>
<p>In addition:</p>
<ol start="3">
<li>
<p>if I would like to move them to GPU, is it enough to do model.to(device) ? (if my model uses the nn.Linear with the weights loaded)</p>
</li>
<li>
<p>If I want to freeze the weights, is using linear_trans.weight.data.requires_grad = False enough ?</p>
</li>
</ol>
<p>Thank you very much for your help, I just want to be sure as I’ve never done this kind of weight initialization</p>

username: albanD
cooked: <p>You should not use <code>.data</code> anymore but use the <code>with torch.no_grad():</code> context manager with the most recent versions of pytorch.<br>
See how the <code>nn.init</code> module work for example <a href="https://github.com/pytorch/pytorch/blob/a132a7d9ce4fef1d5057c66e136935a9b9379bbf/torch/nn/init.py#L50-L64">here</a>.</p>
<p>And yes to all your questions otherwise, it will work exactly that way.<br>
Note that setting <code>requires_grad = False</code> will make it so that no gradients are computed (or kept at 0). This does not necerraly mean that the weights won’t be updated as for example, Adam will change the weights even for a gradient of 0 because of the momentum terms.</p>

username: Diego999
cooked: <p>Thank you for your answers and your note.</p>
<p>So if I understand correctly, this should be the way:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):

    def __init__(self, weights):
        super(MyModule, self).__init__()

        self.linear = nn.Linear(weights.shape[1], weights.shape[0])
        with torch.no_grad():
            self.linear.weight = torch.tensor(weights) # nn.Parameter(...) ?
</code></pre>
<p>Is this enough to update the weights with gradients later or should I enforce with nn.Parameter ?</p>

username: albanD
cooked: <p>You can change the original tensor inplace to avoid such problems with which type you should create by doing:</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.linear.weight.copy_(your_new_weights)
</code></pre>

username: Diego999
cooked: <p>Ok great, thank you very much !</p>

username: Andy_Zhao
cooked: <p>Hi,</p>
<pre><code class="lang-auto">            with torch.no_grad():
                w = torch.Tensor(weights).reshape(self.weight.shape)
                self.weight.copy_(w)
</code></pre>
<p>I have tried the code above, the weights are properly assigned to new values.<br>
However, the weights just won’t update after loss.backward() if I manually assign them to new values. The weights become the fixed value that I assigned. (The weights are updated correctly if not manually assigned)</p>
<p>Could you please help me with this problem？<br>
Your help is highly appreciated!</p>

username: ptrblck
cooked: <p>Could you post a code snippet to reproduce this issue, as the posed code example seems to work:</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self, weights):
        super(MyModule, self).__init__()

        self.linear = nn.Linear(weights.shape[1], weights.shape[0])
        with torch.no_grad():
            self.linear.weight.copy_(weights)
        
    def forward(self, x):
        x = self.linear(x)
        return x

weights = torch.randn(10, 10)
model = MyModule(weights)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

out = model(torch.randn(1, 10))
out.mean().backward()

for name, param in model.named_parameters():
    print(name, param.grad)

w0 = model.linear.weight.clone()
optimizer.step()
w1 = model.linear.weight.clone()
print(w1 - w0)
</code></pre>