How to freeze weights and update random vector?

username: vishalthengane
cooked: <p>Hi,</p>
<p>I have a trained model weights, and I want to replace those weights with <code>weights + some_random vector</code>, and update only the random vector not weights of a model (freeze those weights).</p>
<p>pseudo code;</p>
<pre><code class="lang-auto"># some_random_tensor with requires_grad==True
for layer in model_layers:
    layer.weight = nn.Parameter(layer.weight.data + some_random_tensor)
</code></pre>
<p>So now during training, I only want to update <code>some_random_tensor</code> not model weights. How I can approach this problem?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>On approach would be to freeze all parameters in the original layer and create <code>some_random_tensor</code> as a new <code>nn.Parameter</code>.<br>
The original module and the new parameter could be initialized in a custom <code>nn.Module</code> and in its <code>forward</code> method you could use the functional API (e.g. via <code>F.linear</code>) and apply your operation.</p>

username: vishalthengane
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>can you help me with the code snippet on custom <code>nn.Module</code>?</p>
<p>Below I try to use <code>nn.Linear</code> but the weights are still updating.</p>
<pre><code class="lang-auto"># eign_vectors == it is a tensor contains eign value for each layer.
for i, layer in enumerate(layers):
    alpha = torch.nn.Linear(shape[0], shape[1], bias=False)
    addn_to_weights = torch.matmul(alpha.weight, eign_vectors[i])
    model.layers[i].weight = model.layers[i].weight = torch.nn.Parameter(
                    torch.add(model.layers[i].weight.clone.detach(), addn_to_weights)
                )

# here i don't want to update weights, Only update the alpha value in a next training process.
</code></pre>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Here is a code snippet of my idea:</p>
<pre><code class="lang-python"># Setup
lin = nn.Linear(5, 5, bias=False)
weight = nn.Parameter(torch.randn(5, 5))
optimizer = torch.optim.SGD([weight], lr=1.)

# Freeze lin.weight
lin.weight.requires_grad = False

# Forward pass
x = torch.randn(1, 5)
out = F.linear(x, lin.weight + weight)

# Backward pass
out.mean().backward()

# Check grads
print(lin.weight.grad)
&gt; None

print(weight.grad)
&gt; tensor([[-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
          [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
          [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
          [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
          [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443]])

# Step
optimizer.step()
</code></pre>
<p>Let me know, if this would work for you.</p>

username: vishalthengane
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I modified the code for my use case but the <code>alpha</code> variable in the following code not have a gradient, what am I missing?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional  as F

# Setup
lin = nn.Linear(5, 5, bias=False)
alpha = nn.Parameter(torch.randn(5, 2))
eign_vect = torch.rand(2, 5)
weight = nn.Parameter(torch.matmul(alpha, eign_vect))
optimizer = torch.optim.SGD([weight, alpha], lr=1.)

# Freeze lin.weight
lin.weight.requires_grad = False

# Forward pass
x = torch.randn(1, 5)
out = F.linear(x, lin.weight + weight)
# Backward pass
out.mean().backward()

# Check grads
print(lin.weight.grad)
# &gt; None
print(alpha.grad)
# &gt; None
print(weight.grad)
# &gt; tensor([[-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
#           [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
#           [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
#           [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443],
#           [-0.0892, -0.3502, -0.0987, -0.3553, -0.2443]])

# Step
optimizer.step()
</code></pre>
<p>And I only want to update the <code>alpha</code> variable (not<code> weight</code>), How I can do it?</p>

username: ptrblck
cooked: <p><code>alpha</code> is not used in the computation graph, but instead is used to create the <code>weight</code> parameter.<br>
If you want to update <code>alpha</code> and keep <code>weight</code> constant, create <code>weight</code> as a tensor and use <code>alpha</code> during the forward pass.</p>

username: vishalthengane
cooked: <p>How do I use alpha in the forward pass, I am a bit confused.</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">lin = nn.Linear(5, 5, bias=False)
alpha = nn.Parameter(torch.randn(5, 2))
eign_vect = torch.rand(2, 5)
#weight = nn.Parameter(torch.matmul(alpha, eign_vect))
optimizer = torch.optim.SGD([alpha], lr=1.)

# Freeze lin.weight
lin.weight.requires_grad = False

# Forward pass
x = torch.randn(1, 5)
weight = torch.matmul(alpha, eign_vect)
out = F.linear(x, lin.weight + weight)

# Backward pass
out.mean().backward()

# Check grads
print(lin.weight.grad)
print(alpha.grad)
print(weight.grad)
</code></pre>
<p>This approach would calculate the gradients for <code>alpha</code> and would recompute <code>weight</code> using the new and updated <code>alpha</code> value.</p>

username: vishalthengane
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>your implemetation is working, but when I put <code>F.linear()</code> inside the for loop like below;</p>
<pre><code class="lang-auto">def my_model(x):
    for i, old_param in enumerate(model.parameters()):
        x = F.linear(x, old_param + new_weight[i])

    return x
</code></pre>
<p>I am getting an error;</p>
<pre><code class="lang-auto">
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>When I do <code>loss.backward(retain_graph=True)</code> the following error is commig in:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 399]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>Here is my full implementation:</p>
<pre><code class="lang-auto">
alphas = []
new_weights = []
for i, param in enumerate(model.parameters()):
    eign_shape = params_dicts[f"layer_{i}"]["eign_vector"].shape
    layer_shape = param.shape
    alpha = nn.Parameter(torch.randn(layer_shape[0], eign_shape[0]))
    alphas.append(alpha)
    new_weights.append(torch.matmul(alpha, params_dicts[f"layer_{i}"]["eign_vector"]))

def new_model(x): # x represents batch images.
    
    for i, old_param in enumerate(model.parameters()):
        x = func.linear(x, torch.add(old_param, new_weights[i]))
    
    return x


for epoch in range(num_epochs):
    for batch, (images, labels) in enumerate(train_loader):
        # print(batch)
        images, labels = images.to(device), labels.to(device)

        optimizer.zero_grad()
        with torch.set_grad_enabled(True):
            outputs = new_model(images)
            loss = criterion(outputs, labels)
        
            if epoch == (num_epochs-1) and batch == (train_loader):
                loss.backward()
            else:
                loss.backward(retain_graph=True)
            optimizer.step()

</code></pre>

username: ptrblck
cooked: <p>Could you post an executable code snippet, please?<br>
Currently most of the objects are undefined so that I could only speculate what to use for them.</p>

username: vishalthengane
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for the reply. I solved it by moving this line  <code>torch.matmul(alpha, params_dicts[f"layer_{i}"]["eign_vector"])</code>  inside the for loop. so the update code is;</p>
<pre><code class="lang-auto">def new_model(x): # x represents batch images.
    # print(x.shape)
    for i, old_param in enumerate(model.parameters()):
        weight = torch.matmul(alphas[i], params_dicts[f"layer_{i}"]["eign_vector"])
        x = func.linear(x, old_param + weight)
    
    return x
</code></pre>