Multiplication matrix weights update issue

username: algeriapy
cooked: <p>I am trying to define a matrix to be multiplied with a vector where the weights of this matrix will be learned through the training, the issue that the weights of this matrix still unchanged  during the training here my code:</p>
<pre><code class="lang-auto">class MyNetwork(nn.Module):
    def __init__(self, device):
        super(MyNetwork, self).__init__()
        self.model = torchvision.models.densenet161(pretrained=True)              
        self.model.classifier = nn.Sequential(
                nn.Dropout(0.5),
                nn.Linear(2208, 256), 
                nn.ReLU(),
                nn.Dropout(0.5)
        )
        self.fc1 = nn.Linear(256, 23)
        self.fc2 = nn.Linear(23, 10)
        
        self.w = nn.Parameter(torch.randn(10,23).to(device), requires_grad = True)


    def forward(self, input):
        output1 = self.fc1(F.relu(self.model(input1))
        feaco1 = self.fc2(F.relu(self.drop(output1)))
        feaco2 = torch.nn.functional.one_hot(torch.argmax(feaco1, dim=1), num_classes=10).float()        
        shap1 = feaco2.matmul(self.w)
        output1 = torch.add(shap1, output1)        
 
       return output1, shap1
----------------------------------------------------------------------------
device = torch.device("cuda:2") 
model = MyNetwork(device).to(device) 
optimizer = optim.Adam(model.parameters(), lr = lr)

optimizer.zero_grad()
loss.backward()
optimizer.step() 

</code></pre>
<p>Did I miss anything here</p>

username: eqy
cooked: <p>Can you share how you are computing the loss (e.g., what loss function is being used and what the inputs are) here?</p>

username: algeriapy
cooked: <p>Hi thanks for replying<br>
Actually my Network is different from I posted, which is Siamese Architecture and the loss function have 4 parts<br>
Actually the loss was working correctly before adding the multiplication matrix, where its weights are not updated during training.</p>
<p><code>loss =  loss1 + loss2 + loss3+ 0.3*loss4</code></p>
<p>loss2 is calculated from the shap1 output using focal loss function</p>

username: eqy
cooked: <p>Is there a reason you cannot use another linear layer (<code>self.matmul = nn.Linear(10, 23)</code>) rather than defining the parameters manually?</p>

username: algeriapy
cooked: <p>Actually, I tried to replace it with conv layer like this:</p>
<blockquote>
<p>self.w = nn.Conv1d(in_channels=10, out_channels=23, kernel_size=1, stride = 1)</p>
</blockquote>
<p>But the weights still unchanged</p>

username: eqy
cooked: <p>Can you post what the actual forward function looks like here?</p>

username: eqy
cooked: <aside class="quote no-group quote-post-not-found" data-username="algeriapy" data-post="7" data-topic="125376">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/48.png" class="avatar"> algeriapy:</div>
<blockquote>
<p><code>shap1 = feaco2.matmul(self.w)</code></p>
</blockquote>
</aside>
<p>I still see <code>shap1 = feaco2.matmul(self.w)</code> rather than a layer being used. What happens when you replace the <code>feaco2.matmul(self.w)</code> with something like <code>self.fc3(feaco2)</code>?</p>
<p>However, if you are referring to the rest of the model weights not changing, I think this is because  <code>torch.nn.functional.one_hot(torch.argmax(feaco1, dim=1), num_classes=10).float()</code> is not differentiable. You might want to consider replacing this with a softmax instead which is differentiable.</p>

username: algeriapy
cooked: <p>Hi, thanks for your reply<br>
I got the same result when I used:<br>
w as a conv layer<br>
<code>self.w = nn.Conv1d(in_channels=10, out_channels=23, kernel_size=1, stride = 1)</code></p>
<p>I will use the out features instead of the one hot vector and update you<br>
Best regards</p>

username: eqy
cooked: <p>I couldn’t reproduce the issue on my setup; note that you cannot use <code>ll</code> to compare the weights after each iteration as the underlying storage does not change and each of the new elements of the list actually point to the same memory. The changes to the weights also seem to be very small depending on the learning rate.<br>
Code I used:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision

ll = list()

class SiameseNetwork(nn.Module):
    def __init__(self, device):
        super(SiameseNetwork, self).__init__()
        self.model = torchvision.models.densenet161(pretrained=True)
        self.model.classifier = nn.Linear(2208, 23)
        #self.name = './ModelsCh/Densent161_focal_best.pt'
        #self.model.load_state_dict(torch.load(self.name))
        self.model.classifier = nn.Sequential(
                nn.Dropout(0.5),
                nn.Linear(2208, 256),
                nn.ReLU(),
                nn.Dropout(0.5)
        )
        self.fc1 = nn.Linear(256, 23)
        self.fc2 = nn.Linear(46, 10)

        self.ptsigmoid = nn.Sigmoid()
        self.drop = nn.Dropout(0.5)
        self.w = nn.Linear(10, 23)


    def forward_once(self, x):
        output = self.model(x)
        output = self.fc1(F.relu(output))
        if self.training:
           outpu2 =  1
        else:
           outpu2 = self.fc2(F.relu(torch.cat((output, output), dim=1)))
           output = torch.add(self.w(outpu2), output)
        return output, outpu2

    def forward(self, input1, input2):
        output1, out1 = self.forward_once(input1)
        output2, out2 = self.forward_once(input2)
        feaco =  torch.cat((output1, output2), dim=1)
        feaco1 = self.fc2(F.relu(self.drop(feaco)))

        shap1 = self.w(feaco1)
        ll.append(self.w.weight.data)
        print(torch.sum(self.w.weight))
        output1 = torch.add(shap1, output1)
        output2 =  torch.add(shap1, output2)
        return output1, output2, feaco1

net = SiameseNetwork('cuda')
net = net.cuda()
in1 = torch.randn(1, 3, 224, 224, device='cuda')
in2 = torch.randn(1, 3, 224, 224, device='cuda')
target = torch.tensor([3], device='cuda', dtype=torch.long)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), 1e-1,
                momentum=0.9,
                weight_decay=1e-4)
for i in range(0, 10):
    _, out, _ = net(in1, in2)
    loss = criterion(out, target)
    optimizer.zero_grad()
    loss.backward()
    print(loss.item())
    optimizer.step()

print([data.storage().data_ptr() for data in ll])
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor(1.6240, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
3.283719062805176
tensor(1.6240, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
1.6470584869384766
tensor(1.6240, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
tensor(1.6239, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
tensor(1.6239, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
tensor(1.6238, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
tensor(1.6237, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
tensor(1.6236, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
tensor(1.6235, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
tensor(1.6234, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
0.0
[139980355652608, 139980355652608, 139980355652608, 139980355652608, 139980355652608, 139980355652608, 139980355652608, 139980355652608, 139980355652608, 139980355652608]
</code></pre>

username: algeriapy
cooked: <p>Thanks</p>
<p>The weights are updating</p>
<p>Best Regards</p>