How to change the weights of a pytorch model?

Deeply: <p>I need to change the weights at specific layers of ResNet-152 during training.<br>
I think there has been a similar question sometime earlier, but I cannot find it!</p>

ptrblck: <p>Would you like to change the weights manually?<br>
If so, you could wrap the code in a <code>torch.no_grad()</code> guard:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.fc.weight[0, 0] = 1.
</code></pre>
<p>to prevent Autograd from tracking these changes.</p>

Deeply: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
In fact, I am trying an attention mechanism and I want the weights that I update, according to attention, to take effect during training. Thus, from my understanding, <code>torch.no_grad()</code> should only be used during testing/validation. Here’s the code I have in mind (for a simple demonstration, I am replacing the attention weights here with some random values):</p>
<pre><code>    model.train(); total_loss = 0
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device) 
        optimizer.zero_grad()
        attention_matrix = torch.rand(model.fc.weight.size())
        model.fc.weight = model.fc.weight * attention_matrix
        output = model(data)           
        loss = criterion(output, target)
        total_loss += loss.item()
        loss.backward()
        optimizer.step()
</code></pre>
<p>During testing, however, I think we need to use <code>torch.no_grad()</code> , as follows:</p>
<pre><code>    model.eval()  
    with torch.no_grad():
        for data, target  in test_loader:
            data, target = data.to(device), target.to(device)
            model.fc.weight = model.fc.weight * attention_matrix
            output = model(data)
            loss = criterion(output, target)
            total_loss += loss.item()
</code></pre>
<p>I also need to do the same for <strong>the convolution layers</strong>, not yet sure how this could be done!</p>

ptrblck: <p>I’m not sure if you should manipulate the weights directly using the attention weights.<br>
In the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder" rel="nofollow noopener">Seq2Seq tutorial</a> the attention weights are multiplied with the encoder outputs to calculate new activations.<br>
Would that work for you, too?</p>

Deeply: <p>I agree with you that it is a bit critical to do this by manipulating the weights directly using attention, and the network thus might not converge. But, I intend to do this using simulated annealing, hopefully it is gonna work; I will not know unless I try it.</p>
<p>I still do not know how to change the parameters of the convolution layers.</p>

Deeply: <p>To make it work, I needed to cast the output of the multiplication, as follows:</p>
<p><code>model.fc.weight = torch.nn.parameter.Parameter( model.fc.weight * attention_matrix )</code></p>

sudonto: <p><a class="mention" href="/u/deeply">@Deeply</a> Just to make sure, so you were able to manipulate the weights manually by using the above code? No harm to model’s performance?<br>
I am asking because I want to use your approach but for conv layer. Thanks.</p>

Deeply: <p>Yes. It worked.<br>
As far as I remember, the model was very tolerant to the introduced noise and was able to adapt and recover very quickly. After being hit by the first noise multiply, the MSE (or whatever metric used) was reduced a bit, but not drastically. Then, it started to recover and tolerate the noise. I was trying to generate different models at the mid-epoch training journey. Then, by having a noise generator, I aimed at building a high-level ensemble by combining all the models.</p>
<p><strong>Training Configuration:</strong><br>
I don’t remember the exact configuration; but I can give an example about what I had in mine: suppose my training goes up to 100 epochs, the model learns normally and after epoch 50, it is hit with this noise at every 10 epochs {50, 60, 70, 80, 90} and it stops at epoch 100.  Something like that.</p>
<p><strong>No harm to model’s performance?</strong><br>
The Model performance depends on what you are trying to achieve, and how you are manipulating the weights.</p>

sudonto: <p><a class="mention" href="/u/deeply">@Deeply</a> Thanks for your confirmation.</p>

akvilonBrown: <p>Greetings,<br>
I apologize for reviving this topic, it’s so close to my needs.<br>
I want to change weights according to meta-information supplied with input images and I need intentionally to track these changes with <strong>Autograd</strong>.<br>
I wonder if <em>not using</em>  <strong>torch.no_grad()</strong>  is enough for it, so if I don’t use anything can I be sure that the results will be backpropagated in the usual way, and the manual alteration is compatible with “normal” gradient flow (when weights are within <strong>nn</strong> module).</p>

Deeply: <p>I’m not sure I understand what you are trying to achieve.<br>
Yet, IMHO, the best way to make sure that <code>autograd</code> does what it is supposed to, is to run the code in debug mode, track changes and see what has been updated via BackProbagation. I usually follow this approach to be confident about what is happening. You can run this debugging-test using a simply toy-example that mimics what you are trying to do.</p>

Abhishek_Agrawal: <p>This doesn’t seem to be the right way as in my case when I updated weights this way, my gradients became non-zero even after using the optimizer.zero_grad().</p>
<p>For calculating features with updated weight, I used torch.nn.functional as we have conv layer already initialized in <strong>init</strong> keeping new weights in a separate variable.</p>

Deeply: <p>In my case, I wanted the weights to be updated after hitting them with noise. Yet, you can use <code>torch.no_grad()</code> to prevent the gradient from updating your weights. You can also use detach() method, which constructs a new view on a tensor which is <strong>declared not to need gradients</strong> , i.e., it is to be excluded from further tracking of operations, and therefore the subgraph involving this view will not be recorded.</p>