Why my custom loss function doesn't work for my training process?

Hoodythree: <p>Say I have <code>C</code> classes, and I want to get a <code>C*C</code> similarity matrix in which each entry is the cosine similarity between <code>class_i</code> and <code>class_j</code>. I write the below code to compute the similar loss based on the weights of last but one fc layer. Below is the part of the code for simplicity：</p>
<pre><code class="lang-auto">cos = nn.CosineSimilarity(dim=1, eps=1e-6)
for batch_idx, (data, target) in enumerate(self.data_loader):
    # C*M
    weights = self.model.module.model.fc[-1].weight
    num_iter = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter).cuda()
    for j in range(num_iter):
           weights_i = weights[j, :].expand_as(weights)
           sim = cos(weights, weights_i)
           sim = torch.unsqueeze(sim, 0)
           sim_mat = torch.cat((sim_mat, sim), 0)
    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))

    max_val = torch.max(sim_mat, dim=1).values

    loss_sim = (torch.sum(max_val) / num_classes**2)
    ...
    loss_sim.backward()  
</code></pre>
<p>Why my loss_sim doesn’t work for my training process? It seems that the loss_sim didn’t backward properly to affect the original model weights.<br>
I know we can wrap the loss function into a class like:</p>
<pre><code class="lang-auto">class Loss_sim(nn.Module):
    def __init__():

    def forward(self, weights):
</code></pre>
<p>Should I write like this? Is any problem here because of the copy of the original weights?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

KFrank: <p>Howdy Hoody!</p>
<p>Perhaps the short answer is that your <code>loss_sim</code> is bounded below<br>
by zero, and that might not be what you want.</p>
<aside class="quote no-group" data-username="Hoodythree" data-post="1" data-topic="99194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hoodythree/48/24384_2.png" class="avatar"> Hoodythree:</div>
<blockquote>
<p>I write the below code to compute the similar loss based on the weights of last but one fc layer.</p>
</blockquote>
</aside>
<p>I’m not sure what you are trying to do here, but I <em>think</em> you<br>
want a loss term that pushes the rows of <code>weights</code> to be dissimilar<br>
from one another.</p>
<aside class="quote no-group">
<blockquote>
<p>It seems that the loss_sim didn’t backward properly to affect the original model weights.</p>
</blockquote>
</aside>
<p>Your <code>loss_sim</code> will backpropagate.  Whether it will do so “properly”<br>
depends on what you are expecting.</p>
<p>These lines:</p>
<pre data-code-wrap="python"><code class="lang-python">    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
    max_val = torch.max(sim_mat, dim=1).values
</code></pre>
<p>place zeros on the diagonal of <code>sim_mat</code> so that <code>max_val</code> can<br>
never be negative.</p>
<p>The problem is that the rows of <code>weights</code> can all have negative<br>
cosine similarity with one another, at which point <code>loss_sim</code> becomes<br>
zero, has zero gradient, and no longer contributes to the training.</p>
<p>After quoting your code, I show a script that runs your version<br>
of <code>loss_sim</code> packaged as a function, <code>cc_sim</code>, and compares it<br>
with two possibly improved versions.  <code>cc_simA</code> returns the mean<br>
of the cosine similarities, while <code>cc_simB</code> removes the floor<br>
of zero on <code>loss_sim</code> so that it can become negative and fall<br>
to its most negative similarity (maximum dissimilarity).</p>
<p>This script shows that your version does backpropagate, but does<br>
get stuck at zero, and that the improved versions don’t get stuck<br>
at zero.</p>
<p>It also give an example of a tensor, <code>t</code>, whose rows all have<br>
negative cosine similarity with one another.</p>
<p>Your code:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">cos = nn.CosineSimilarity(dim=1, eps=1e-6)
for batch_idx, (data, target) in enumerate(self.data_loader):
    # C*M
    weights = self.model.module.model.fc[-1].weight
    num_iter = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter).cuda()
    for j in range(num_iter):
           weights_i = weights[j, :].expand_as(weights)
           sim = cos(weights, weights_i)
           sim = torch.unsqueeze(sim, 0)
           sim_mat = torch.cat((sim_mat, sim), 0)
    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))

    max_val = torch.max(sim_mat, dim=1).values

    loss_sim = (torch.sum(max_val) / num_classes**2)
    ...
    loss_sim.backward()  
</code></pre>
</blockquote>
</aside>
<p>The script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.random.manual_seed (2020)

def cc_sim (weights):
    cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
    num_iter = num_classes = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter)
    for j in range(num_iter):
        weights_i = weights[j, :].expand_as(weights)
        sim = cos(weights, weights_i)
        sim = torch.unsqueeze(sim, 0)
        sim_mat = torch.cat((sim_mat, sim), 0)
    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
    max_val = torch.max(sim_mat, dim=1).values
    loss_sim = (torch.sum(max_val) / num_classes**2)
    return loss_sim

def cc_simA (weights):
    cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
    num_iter = num_classes = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter)
    for j in range(num_iter):
        weights_i = weights[j, :].expand_as(weights)
        sim = cos(weights, weights_i)
        sim = torch.unsqueeze(sim, 0)
        sim_mat = torch.cat((sim_mat, sim), 0)
    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
    loss_sim = torch.mean (sim_mat)
    return loss_sim

def cc_simB (weights):
    cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
    num_iter = num_classes = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter)
    for j in range(num_iter):
        weights_i = weights[j, :].expand_as(weights)
        sim = cos(weights, weights_i)
        sim = torch.unsqueeze(sim, 0)
        sim_mat = torch.cat((sim_mat, sim), 0)
    sim_mat = sim_mat - torch.diag (float ('inf') * torch.ones (3))
    max_val = torch.max(sim_mat, dim=1).values
    loss_sim = (torch.sum(max_val) / num_classes**2)
    return loss_sim

t = torch.tensor ([[1.0, 0.0], [-0.5, 1.0], [-0.5, -1.0]])
t.requires_grad = True
print ('cc_sim:')
loss = cc_sim (t)
loss.backward()
print ('loss =', loss)
print ('t = ...\n', t)
print ('t.grad = ...\n', t.grad)
with torch.no_grad():
    _ = t.grad.zero_()

print ('cc_simA:')
lossA = cc_simA (t)
lossA.backward()
print ('lossA =', lossA)
print ('t = ...\n', t)
print ('t.grad = ...\n', t.grad)
with torch.no_grad():
    _ = t.grad.zero_()

print ('cc_simB:')
lossB = cc_simB (t)
lossB.backward()
print ('lossB =', lossB)
print ('t = ...\n', t)
print ('t.grad = ...\n', t.grad)

nDim = 2
w = torch.randn ((3, nDim))
wA = w.clone()
wB = w.clone()
w.requires_grad = True
wA.requires_grad = True
wB.requires_grad = True
print ('w = ...\n', w)
print ('wA = ...\n', wA)
print ('wB = ...\n', wB)

lr = 5.0

print ('cc_sim:')
for  i in range (10):
    loss = cc_sim (w)
    print ('loss =', loss)
    if  i != 0:
        _ = w.grad.zero_()
    loss.backward()
    with torch.no_grad():
        _ = w.sub_ (lr * w.grad)

print ('w = ...\n', w)
print ('w.grad = ...\n', w.grad)

print ('cc_simA:')
for  i in range (10):
    lossA = cc_simA (wA)
    print ('lossA =', lossA)
    if  i != 0:
        _ = wA.grad.zero_()
    lossA.backward()
    with torch.no_grad():
        _ = wA.sub_ (lr * wA.grad)

print ('wA = ...\n', wA)
print ('wA.grad = ...\n', wA.grad)


print ('cc_simB:')
for  i in range (10):
    lossB = cc_simB (wB)
    print ('lossB =', lossB)
    if  i != 0:
        _ = wB.grad.zero_()
    lossB.backward()
    with torch.no_grad():
        _ = wB.sub_ (lr * wB.grad)

print ('wB = ...\n', wB)
print ('wB.grad = ...\n', wB.grad)

</code></pre>
<p>The output of the script:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.6.0'
&gt;&gt;&gt; 
&gt;&gt;&gt; torch.random.manual_seed (2020)
&lt;torch._C.Generator object at 0x7f635efa4930&gt;
&gt;&gt;&gt; 
&gt;&gt;&gt; def cc_sim (weights):
...     cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
...     num_iter = num_classes = weights.size(0)
...     # similarity matrix
...     sim_mat = torch.empty(0, num_iter)
...     for j in range(num_iter):
...         weights_i = weights[j, :].expand_as(weights)
...         sim = cos(weights, weights_i)
...         sim = torch.unsqueeze(sim, 0)
...         sim_mat = torch.cat((sim_mat, sim), 0)
...     sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
...     max_val = torch.max(sim_mat, dim=1).values
...     loss_sim = (torch.sum(max_val) / num_classes**2)
...     return loss_sim
... 
&gt;&gt;&gt; def cc_simA (weights):
...     cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
...     num_iter = num_classes = weights.size(0)
...     # similarity matrix
...     sim_mat = torch.empty(0, num_iter)
...     for j in range(num_iter):
...         weights_i = weights[j, :].expand_as(weights)
...         sim = cos(weights, weights_i)
...         sim = torch.unsqueeze(sim, 0)
...         sim_mat = torch.cat((sim_mat, sim), 0)
...     sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
...     loss_sim = torch.mean (sim_mat)
...     return loss_sim
... 
&gt;&gt;&gt; def cc_simB (weights):
...     cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
...     num_iter = num_classes = weights.size(0)
...     # similarity matrix
...     sim_mat = torch.empty(0, num_iter)
...     for j in range(num_iter):
...         weights_i = weights[j, :].expand_as(weights)
...         sim = cos(weights, weights_i)
...         sim = torch.unsqueeze(sim, 0)
...         sim_mat = torch.cat((sim_mat, sim), 0)
...     sim_mat = sim_mat - torch.diag (float ('inf') * torch.ones (3))
...     max_val = torch.max(sim_mat, dim=1).values
...     loss_sim = (torch.sum(max_val) / num_classes**2)
...     return loss_sim
... 
&gt;&gt;&gt; t = torch.tensor ([[1.0, 0.0], [-0.5, 1.0], [-0.5, -1.0]])
&gt;&gt;&gt; t.requires_grad = True
&gt;&gt;&gt; print ('cc_sim:')
cc_sim:
&gt;&gt;&gt; loss = cc_sim (t)
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; print ('loss =', loss)
loss = tensor(0., grad_fn=&lt;DivBackward0&gt;)
&gt;&gt;&gt; print ('t = ...\n', t)
t = ...
 tensor([[ 1.0000,  0.0000],
        [-0.5000,  1.0000],
        [-0.5000, -1.0000]], requires_grad=True)
&gt;&gt;&gt; print ('t.grad = ...\n', t.grad)
t.grad = ...
 tensor([[0., 0.],
        [0., 0.],
        [0., 0.]])
&gt;&gt;&gt; with torch.no_grad():
...     _ = t.grad.zero_()
... 
&gt;&gt;&gt; print ('cc_simA:')
cc_simA:
&gt;&gt;&gt; lossA = cc_simA (t)
&gt;&gt;&gt; lossA.backward()
&gt;&gt;&gt; print ('lossA =', lossA)
lossA = tensor(-0.3321, grad_fn=&lt;MeanBackward0&gt;)
&gt;&gt;&gt; print ('t = ...\n', t)
t = ...
 tensor([[ 1.0000,  0.0000],
        [-0.5000,  1.0000],
        [-0.5000, -1.0000]], requires_grad=True)
&gt;&gt;&gt; print ('t.grad = ...\n', t.grad)
t.grad = ...
 tensor([[ 0.0000,  0.0000],
        [ 0.0168,  0.0084],
        [ 0.0168, -0.0084]])
&gt;&gt;&gt; with torch.no_grad():
...     _ = t.grad.zero_()
... 
&gt;&gt;&gt; print ('cc_simB:')
cc_simB:
&gt;&gt;&gt; lossB = cc_simB (t)
&gt;&gt;&gt; lossB.backward()
&gt;&gt;&gt; print ('lossB =', lossB)
lossB = tensor(-0.1491, grad_fn=&lt;DivBackward0&gt;)
&gt;&gt;&gt; print ('t = ...\n', t)
t = ...
 tensor([[ 1.0000,  0.0000],
        [-0.5000,  1.0000],
        [-0.5000, -1.0000]], requires_grad=True)
&gt;&gt;&gt; print ('t.grad = ...\n', t.grad)
t.grad = ...
 tensor([[ 0.0000,  0.0994],
        [ 0.1590,  0.0795],
        [ 0.0795, -0.0398]])
&gt;&gt;&gt; 
&gt;&gt;&gt; nDim = 2
&gt;&gt;&gt; w = torch.randn ((3, nDim))
&gt;&gt;&gt; wA = w.clone()
&gt;&gt;&gt; wB = w.clone()
&gt;&gt;&gt; w.requires_grad = True
&gt;&gt;&gt; wA.requires_grad = True
&gt;&gt;&gt; wB.requires_grad = True
&gt;&gt;&gt; print ('w = ...\n', w)
w = ...
 tensor([[ 1.2372, -0.9604],
        [ 1.5415, -0.4079],
        [ 0.8806,  0.0529]], requires_grad=True)
&gt;&gt;&gt; print ('wA = ...\n', wA)
wA = ...
 tensor([[ 1.2372, -0.9604],
        [ 1.5415, -0.4079],
        [ 0.8806,  0.0529]], requires_grad=True)
&gt;&gt;&gt; print ('wB = ...\n', wB)
wB = ...
 tensor([[ 1.2372, -0.9604],
        [ 1.5415, -0.4079],
        [ 0.8806,  0.0529]], requires_grad=True)
&gt;&gt;&gt; 
&gt;&gt;&gt; lr = 5.0
&gt;&gt;&gt; 
&gt;&gt;&gt; print ('cc_sim:')
cc_sim:
&gt;&gt;&gt; for  i in range (10):
...     loss = cc_sim (w)
...     print ('loss =', loss)
...     if  i != 0:
...         _ = w.grad.zero_()
...     loss.backward()
...     with torch.no_grad():
...         _ = w.sub_ (lr * w.grad)
... 
loss = tensor(0.3133, grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0.2794, grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0.2203, grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0.1282, grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0.0270, grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0.0421, grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0., grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0., grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0., grad_fn=&lt;DivBackward0&gt;)
loss = tensor(0., grad_fn=&lt;DivBackward0&gt;)
&gt;&gt;&gt; print ('w = ...\n', w)
w = ...
 tensor([[-0.5426, -1.7768],
        [ 1.8619, -0.0235],
        [-1.0342,  1.1212]], requires_grad=True)
&gt;&gt;&gt; print ('w.grad = ...\n', w.grad)
w.grad = ...
 tensor([[0., 0.],
        [0., 0.],
        [0., 0.]])
&gt;&gt;&gt; 
&gt;&gt;&gt; print ('cc_simA:')
cc_simA:
&gt;&gt;&gt; for  i in range (10):
...     lossA = cc_simA (wA)
...     print ('lossA =', lossA)
...     if  i != 0:
...         _ = wA.grad.zero_()
...     lossA.backward()
...     with torch.no_grad():
...         _ = wA.sub_ (lr * wA.grad)
... 
lossA = tensor(0.5826, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(0.1014, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.2646, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.3099, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.3240, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.3299, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.3322, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.3330, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.3332, grad_fn=&lt;MeanBackward0&gt;)
lossA = tensor(-0.3333, grad_fn=&lt;MeanBackward0&gt;)
&gt;&gt;&gt; print ('wA = ...\n', wA)
wA = ...
 tensor([[-1.0928, -1.7762],
        [ 1.6100, -0.0603],
        [-0.9511,  1.8146]], requires_grad=True)
&gt;&gt;&gt; print ('wA.grad = ...\n', wA.grad)
wA.grad = ...
 tensor([[ 1.7566e-03, -1.0688e-03],
        [-3.6173e-05, -8.9793e-04],
        [ 1.2300e-03,  6.3939e-04]])
&gt;&gt;&gt; 
&gt;&gt;&gt; 
&gt;&gt;&gt; print ('cc_simB:')
cc_simB:
&gt;&gt;&gt; for  i in range (10):
...     lossB = cc_simB (wB)
...     print ('lossB =', lossB)
...     if  i != 0:
...         _ = wB.grad.zero_()
...     lossB.backward()
...     with torch.no_grad():
...         _ = wB.sub_ (lr * wB.grad)
... 
lossB = tensor(0.3133, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(0.2794, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(0.2203, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(0.1282, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(0.0040, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(-0.1192, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(-0.0680, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(-0.1348, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(-0.1079, grad_fn=&lt;DivBackward0&gt;)
lossB = tensor(-0.1362, grad_fn=&lt;DivBackward0&gt;)
&gt;&gt;&gt; print ('wB = ...\n', wB)
wB = ...
 tensor([[-1.0508, -1.8293],
        [ 1.7102,  0.5224],
        [-1.1758,  1.3381]], requires_grad=True)
&gt;&gt;&gt; print ('wB.grad = ...\n', wB.grad)
wB.grad = ...
 tensor([[ 0.0950, -0.0278],
        [ 0.0077, -0.0571],
        [ 0.0489,  0.0305]])
&gt;&gt;&gt; 
</code></pre>
<p>Good luck.</p>
<p>K. Frank</p>

Hoodythree: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="99194">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>torch.diag (float ('inf') * torch.ones (3))</code></p>
</blockquote>
</aside>
<p>Thank you for your timely help. Basically, I want to penalize the most similar pairs of each class. That’s why I want to get a similar loss which serves as a regularizing term. My total loss is like <code>L_total = L_CE + L_sim</code>. My questions as follows:</p>
<ul>
<li>I want to know if this term can help to update the parameters of my network in the training process cause I did an assignment for model weights here <code>weights = self.model.module.model.fc[-1].weight</code>.</li>
<li>If I want to get a weighed similar loss, is the below code based on yours correct?</li>
</ul>
<pre><code class="lang-auto">import torch
torch.__version__

torch.random.manual_seed (2020)
def cc_sim (weights, w_s):
    '''
    weights: weights from NN model
    w_s: weights for every entry in similarity matrix, w_s.size = (num_classes, num_classes)
    '''
    cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
    num_iter = num_classes = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter)
    for j in range(num_iter):
        weights_i = weights[j, :].expand_as(weights)
        sim = cos(weights, weights_i)
        sim = torch.unsqueeze(sim, 0)
        sim_mat = torch.cat((sim_mat, sim), 0)
    sim_mat = sim_mat * w_s
    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
    max_val = torch.max(sim_mat, dim=1).values
    loss_sim = (torch.sum(max_val) / num_classes**2)
    return loss_sim

def cc_simA (weights, w_s):
    '''
    weights: weights from NN model
    w_s: weights for every entry in similarity matrix, w_s.size = (num_classes, num_classes)
    '''
    cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
    num_iter = num_classes = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter)
    for j in range(num_iter):
        weights_i = weights[j, :].expand_as(weights)
        sim = cos(weights, weights_i)
        sim = torch.unsqueeze(sim, 0)
        sim_mat = torch.cat((sim_mat, sim), 0)
    # element-wise multiply 
    sim_mat = sim_mat * w_s

    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
    loss_sim = torch.mean (sim_mat)
    return loss_sim

def cc_simB (weights, w_s):
    '''
    weights: weights from NN model
    w_s: weights for every entry in similarity matrix, w_s.size = (num_classes, num_classes)
    '''
    cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
    num_iter = num_classes = weights.size(0)
    # similarity matrix
    sim_mat = torch.empty(0, num_iter)
    for j in range(num_iter):
        weights_i = weights[j, :].expand_as(weights)
        sim = cos(weights, weights_i)
        sim = torch.unsqueeze(sim, 0)
        sim_mat = torch.cat((sim_mat, sim), 0)
    # element-wise multiply 
    sim_mat = sim_mat * w_s
    sim_mat = sim_mat - torch.diag (float ('inf') * torch.ones (num_classes))
    max_val = torch.max(sim_mat, dim=1).values
    loss_sim = (torch.sum(max_val) / num_classes**2)
    return loss_sim

if __name__ == '__main__':
    num_classes = nDim = 10
    weights = torch.randn((num_classes, nDim), requires_grad=True)
    w_s = torch.FloatTensor(num_classes, num_classes).uniform_(0, 1) 
    loss = cc_simB(weights, w_s)
    loss.backward()
    print('loss item:', loss)
    print('grad of weights:\n')
    print(weights.grad)

</code></pre>
<p>Thanks again!</p>

KFrank: <p>Howdy Hoody!</p>
<aside class="quote no-group" data-username="Hoodythree" data-post="3" data-topic="99194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hoodythree/48/24384_2.png" class="avatar"> Hoodythree:</div>
<blockquote>
<ul>
<li>I want to know if this term can help to update the parameters of my network in the training process cause I did an assignment for model weights here <code>weights = self.model.module.model.fc[-1].weight</code>.</li>
</ul>
</blockquote>
</aside>
<p>If you’re asking whether the assignment:</p>
<pre data-code-wrap="python"><code class="lang-python">weights = self.model.module.model.fc[-1].weight
</code></pre>
<p>will prevent gradients from flowing back through the assignment<br>
and break backpropagation, the answer is no.</p>
<p>In python, “variables” are <em>references.</em></p>
<p><code>self.model.module.model.fc[-1].weight</code> refers to a tensor<br>
in memory somewhere.  The above assignment creates a new<br>
reference that refers to the same tensor in memory.  No new<br>
tensor is created, nor is any data copied from one place to<br>
another.  Performing tensor operations on <code>weights</code> is essentially<br>
identical to performing tensor operations on<br>
<code>self.model.module.model.fc[-1].weight</code>, so backpropagation<br>
will work identically.</p>
<p>To emphasize this point:</p>
<pre data-code-wrap="python"><code class="lang-python">wtmp1 = self.model.module.model.fc[-1].weight
wtmp2 = wtmp1
wtmp3 = wtmp2
weights = wtmp3
</code></pre>
<p>would be almost equivalent to your assignment, with the only<br>
difference being the three extra (and unnecessary) temporary<br>
references that will be cleaned up when your script exits (or<br>
when an enclosing code block goes out of scope).</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>If I want to get a weighed similar loss, is the below code based on yours correct?</li>
</ul>
<pre><code class="lang-auto">...
    sim_mat = sim_mat * w_s
    sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))
    max_val = torch.max(sim_mat, dim=1).values
</code></pre>
</blockquote>
</aside>
<p>This will do what I believe you want.  Each element of <code>sim_mat</code><br>
will be multiplied by the corresponding element of <code>w_s</code>.  This<br>
will potentially change the result of <code>torch.max()</code>, depending on<br>
the specific values involved.</p>
<p>Note, if you set up <code>w_s</code> so that it has zeros along its diagonal, the<br>
multiplication will zero out the diagonal of <code>sim_mat</code> so you can forgo<br>
<code>sim_mat = sim_mat - torch.diag(torch.diag(sim_mat))</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

Hoodythree: <p>Thanks for your clear explanation.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fd175f82e66dfd5273ca30d2f765846314f2391.png" data-download-href="https://discuss.pytorch.org/uploads/default/1fd175f82e66dfd5273ca30d2f765846314f2391" title="training"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fd175f82e66dfd5273ca30d2f765846314f2391_2_423x375.png" alt="training" data-base62-sha1="4xtzisenJZcnyw6H1p1AESGVIZ3" width="423" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fd175f82e66dfd5273ca30d2f765846314f2391_2_423x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fd175f82e66dfd5273ca30d2f765846314f2391_2_634x562.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fd175f82e66dfd5273ca30d2f765846314f2391.png 2x" data-dominant-color="DBF0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training</span><span class="informations">748×663 231 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I still have an extra question here:</p>
<ul>
<li>Why my training and validation accuracy curves remained almost the same after I added the sim_loss term like above <code>cc_simB </code>? (Green is with <code>sim_loss</code>) It’s really wired.</li>
</ul>