Network weights wont update

nn_beginner: <p>Hi,</p>
<p>I hope you can help. I am designing a controller for a system, where the controller NN is trained using an imported NN I made, which is the state space representation of the system.</p>
<p>But it seems the weights are not updating on each epoch, I have attached the code below:</p>
<p>This is my defined controller:<br>
<code><br>
class Net(nn.Module):</code></p><code>
<pre><code>def __init__(self, n1, n2, n3):
    super().__init__()
    self.fc1 = nn.Linear((10 * 2), n1)
    self.fc2 = nn.Linear(n1, n2)
    self.fc3 = nn.Linear(n2, n3)
    self.fc4 = nn.Linear(n3, 1)

def forward(self, x):
    x = F.gelu(self.fc1(x))
    x = F.gelu(self.fc2(x))
    x = F.gelu(self.fc3(x))
    x = F.gelu(self.fc4(x))
    return x
</code></pre>
</code>
<p>below is my training loop:</p>
<code>
</code><p><code>LR = 0.001<br>
w_decay = 0.005<br>
optimizer = optim.SGD(dpc.parameters(), lr=LR, weight_decay=w_decay)<br>
EPOCHS = 50<br>
batch = 2<br>
print(‘Training network…’)<br>
dpc.train(mode=True)<br>
for epoch in range(EPOCHS):<br>
for dd in range(0, len(x_train) - batch):<br>
X = x_train[dd:dd + batch]<br>
optimizer.zero_grad()<br>
output = dpc(X)<br>
input_ssm = X.detach().numpy()<br>
input_ssm = input_ssm.tolist()<br>
input_ssm[0].append(output.detach().numpy().tolist()[0][0])<br>
input_ssm[1].append(output.detach().numpy().tolist()[1][0])<br>
input_ssm = np.array(input_ssm)<br>
input_ssm = torch.Tensor(input_ssm)<br>
y = ss_model(input_ssm)<br>
y = (y*(max_a[1][0] - min_a[1][0])) + min_a[1][0]<br>
desired = np.array([[0], [0]])<br>
desired = torch.Tensor(desired)<br>
loss = (F.mse_loss(y, desired))<br>
combined_loss += loss<br>
loss.backward()<br>
optimizer.step()<br>
</code></p>
<p>Many thanks.</p>

KFrank: <p>Hi Finn!</p>
<aside class="quote no-group" data-username="nn_beginner" data-post="1" data-topic="146238" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/c4cdca/48.png" class="avatar"> nn_beginner:</div>
<blockquote>
<code>
        input_ssm = X.detach().numpy()
</code>
</blockquote>
</aside>
<p>Pytorch’s autograd facility – which automatically calculates the gradients<br>
that are then used to update your network weights when you call<br>
<code>optimizer.step()</code> – only tracks computations performed with pytorch<br>
tensor operations.</p>
<p>So as soon as you call <code>numpy()</code> (and <code>detach()</code>, for that matter) you<br>
have “broken the computation graph” and pytorch can no longer<br>
backpropagate through those calculations.</p>
<p>Rewrite the manipulations that you are performing with numpy using<br>
pytorch tensor operations, and autograd should work and your weights<br>
should update.</p>
<p>Best.</p>
<p>K. Frank</p>

nn_beginner: <p>Thank you for your response, I have updated the training loop as such:</p>
<code>
<p>for epoch in range(EPOCHS):<br>
for dd in range(0, len(x_train) - batch):<br>
X = x_train[dd:dd + batch]<br>
optimizer.zero_grad()<br>
output = dpc(X)<br>
input_ssm = torch.empty((1,21))<br>
for t in range(0,20):<br>
input_ssm[0][t] = X[0][t]<br>
input_ssm[0][20] = output[0][0]<br>
y = ss_model(input_ssm)<br>
y = (y*(max_a[1][0] - min_a[1][0])) + min_a[1][0]<br>
desired = torch.Tensor([[0]])<br>
loss = (F.mse_loss(y, desired))<br>
combined_loss += loss<br>
optimizer.step()</p>
</code>
<p>Yet the issue still persists, and the weights are not updating, am I doing this incorrectly?</p>
<p>Many thanks,<br>
Finn</p>

KFrank: <p>Hi Finn!</p>
<aside class="quote no-group quote-post-not-found" data-username="nn_beginner" data-post="4" data-topic="146238" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/c4cdca/48.png" class="avatar"> nn_beginner:</div>
<blockquote>
<code>
<pre><code>    loss = (F.mse_loss(y, desired))
    combined_loss += loss
    optimizer.step()
</code></pre>
</code>
</blockquote>
</aside>
<p>First, let me note that this second version you posted no longer has<br>
<code>loss.backward()</code> in it.  If that’s not just a typo, it would prevent your<br>
weights from being updated.</p>
<p>Second, please post a simplified, fully-self-contained, runnable script<br>
that illustrates your issue, together with its output.  For example, it looks<br>
like what you posted has two models in it.  Can you reproduce your issue<br>
with only one model?</p>
<p>I would also suggest that you try a single forward and backward pass<br>
(rather than the loops over epochs).  Does a single pass create <code>grad</code>s<br>
for your model weights (e.g., <code>my_model.fc1.weight.grad</code>)?  Are these<br>
gradients non-zero?   Does <code>optimizer.step()</code> then modify the weights?</p>
<p>Best.</p>
<p>K. Frank</p>