Loss is changing but the weights are not

username: jpj
cooked: <pre><code class="lang-auto">#custom initiliazation of weight
self.hidden_layer_1.weight = torch.nn.Parameter(torch.from_numpy(weights))
a = self.hidden_layer_1.weight
num_epochs=10
for epoch in range(num_epochs):
            print('Epoch {}'.format(epoch))
            torch.autograd.set_detect_anomaly(True)
            epoch_loss_train = []
            epoch_loss_validation = []
            for i, (x,y) in enumerate(zip(feature_trainloader,label_trainloader), 0):
                optimizer.zero_grad()   
                output = self.forward(x)
                target = y
                loss = self.CrossEntropyLoss(output, target)
                epoch_loss_train.append(loss.item())
                y_true_train = torch.cat((y_true_train, target))
                _, pred_class = torch.max(output, dim=1)
                y_pred_train = torch.cat((y_pred_train, pred_class))
                loss.backward()
                optimizer.step() 
b = self.hidden_layer_1.weight
print(torch.equal(a, b)) #True
</code></pre>
<p>I don’t understand how weights are the same after 10 epochs even though the loss is changing</p>

username: ariG23498
cooked: <p>Could you share a colab notebook which can reproduce the issue?<br>
A minimal implementation would work.</p>

username: eqy
cooked: <p>As a sanity check, can you actually save (e.g., with something like (.detatch().clone()) (or print) the values before and after to compare them? I have a suspicion that <code>a</code> and <code>b</code> are the same reference so they are both updated as the model is being trained.</p>

username: KFrank
cooked: <p>Hi Jpj!</p>
<aside class="quote no-group" data-username="jpj" data-post="1" data-topic="120936" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ed655f/48.png" class="avatar"> jpj:</div>
<blockquote>
<pre><code class="lang-auto">#custom initiliazation of weight
self.hidden_layer_1.weight = torch.nn.Parameter(torch.from_numpy(weights))
</code></pre>
</blockquote>
</aside>
<p>At this point, you (presumably) have already created your optimizer, so<br>
your optimizer contains a collection of <code>Parameter</code>s that it will be updating.</p>
<p>Because you have overwritten <code>weight</code> with a new <code>Parameter</code> (rather<br>
than having overwritten <code>weight.data</code> with a new tensor) – and done so<br>
after creating your optimizer – your optimizer does not contain the new<br>
<code>Parameter</code> that is used (via the forward pass) to calculate the loss.<br>
(The <code>Parameter</code> that is modified by the optimizer-- or would be if it had<br>
a non-trivial gradient – is the old <code>Parameter</code> that you have, in a sense, “hidden.”)</p>
<p>So <code>self.hidden_layer_1.weight</code> doesn’t change, because it is not in<br>
the optimizer’s list of <code>Parameter</code>s to update.<br>
[/quote]</p>
<aside class="quote no-group">
<blockquote>
<p>I don’t understand how weights are the same after 10 epochs even though the loss is changing</p>
</blockquote>
</aside>
<p>Why the loss is changing even though <code>self.hidden_layer_1.weight</code><br>
is not:  I will assume that <code>self.hidden_layer_1</code> is a <code>Linear</code>.  A <code>Linear</code><br>
has both a <code>weight</code> and a <code>bias</code>.  You haven’t overwritten <code>bias</code>, so it is<br>
still updated by your optimizer, and still affects the <code>loss</code> you calculate.</p>
<p>Best.</p>
<p>K. Frank</p>

username: eqy
cooked: <p>Note that even after fixing the optimizer issue, the weights should not be compared using references.<br>
For example, this code snippet will report the “saved” weights being the same because both references refer to the same underlying tensor. Copying the weights before training will produce the expected results:</p>
<pre><code class="lang-auto">import torch

a = torch.nn.Linear(128,1)
b = torch.nn.Sigmoid()
optimizer = torch.optim.SGD(a.parameters(), 1e-3)
criterion = torch.nn.BCELoss()
saved = a.weight
detached = a.weight.detach().clone()
for i in range(0, 512):
    inp = torch.randn((1, 128))
    label = float(torch.sum(inp[64:]) &gt; 0)
    label = torch.tensor([[label]])
    output = b(a(inp))
    loss = criterion(output, label)
    print(loss)
    loss.backward()
    optimizer.grad_ = None
    optimizer.step()
print(torch.allclose(saved, a.weight))
print(saved == a.weight)
print(torch.allclose(detached, a.weight))
print(detached == a.weight)
</code></pre>

username: jpj
cooked: <p>Thanks for the reply.</p>
<p>I do create my optimizer before this statement, you are correct.<br>
So if I understand correctly, <code>self.hidden_layer_1.weight.data = my_custom_weights</code> should fix it?</p>

username: KFrank
cooked: <p>Hi Jpj!</p>
<aside class="quote no-group" data-username="jpj" data-post="6" data-topic="120936" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ed655f/48.png" class="avatar"> jpj:</div>
<blockquote>
<p><code>self.hidden_layer_1.weight.data = my_custom_weights</code> should fix it?</p>
</blockquote>
</aside>
<p>I <em>think</em> that will fix it.</p>
<p>My preferred idiom (not that I know what I’m doing) would be:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    self.hidden_layer_1.weight.data.copy_(my_custom_weights) 
</code></pre>
<p>Best.</p>
<p>K. Frank</p>