Optimizer not updating the weights/parameters

Vinayak_Vijay1: <p>I am using ADAM with LBFGS. The loss doesn’t change with each epoch when I try to use optimizer.step() with the closure function. If I use only ADAM with optimizer.step(), the loss function converges (albeit slowly which is why i decided to use LBFGS). Can you tell me where is my code wrong?</p>
<pre><code class="lang-auto">optimizer1 = torch.optim.Adam(net.parameters(),lr = 0.0001)
optimizer2 = torch.optim.LBFGS(net.parameters(),lr=0.001)
## Training
iterations = 10
loss_array = np.zeros((iterations))

for epoch in range(iterations):
  def closure():
    optimizer1.zero_grad() # to make the gradients zero
    optimizer2.zero_grad() # to make the gradients zero

    
    # # Data driven/boundary loss
    # pt_x_bc1 = Variable(torch.from_numpy(x_bc1).float(), requires_grad=False).to(device)
    # pt_x_bc2 = Variable(torch.from_numpy(x_bc2).float(), requires_grad=False).to(device)
    # pt_u_bc = Variable(torch.from_numpy(u_bc).float(), requires_grad=False).to(device)
    # net_bc_out1 = net(pt_x_bc1) 
    # net_bc_out2 = net(pt_x_bc2) 
    # mse_u1 = mse_cost_function(net_bc_out1, pt_u_bc)
    # mse_u2 = mse_cost_function(net_bc_out2, pt_u_bc)
    # mse_u = mse_u1 + mse_u2

    ## Physics informed loss
    all_zeros = np.zeros((500,1))
    pt_x_collocation = Variable(torch.from_numpy(x_collocation).float(), requires_grad=True).to(device)
    pt_all_zeros = Variable(torch.from_numpy(all_zeros).float(), requires_grad=False).to(device)
    f_out = f(pt_x_collocation, net) # output of f(x,t)
    mse_pinn = mse_cost_function(f_out, pt_all_zeros)
  
    ## Training data loss
    u_train = net(pt_x_collocation)
    pt_u_true = Variable(torch.from_numpy(u_true).float(), requires_grad=False).to(device)
    mse_training = mse_cost_function(u_train, pt_u_true)

    # Combining the loss functions
    loss = mse_pinn + mse_training
    loss_array[epoch] = loss
    loss.backward() 
    return loss

  if epoch&lt;5000:
    optimizer1.step(closure)
  else: 
    optimizer2.step(closure)

  with torch.autograd.no_grad():
    print(epoch,"Traning Loss:",loss.data)
    
</code></pre>
<p>This is the output:</p>
<p>0 Traning Loss: tensor(0.4883)<br>
1 Traning Loss: tensor(0.4883)<br>
2 Traning Loss: tensor(0.4883)<br>
3 Traning Loss: tensor(0.4883)<br>
4 Traning Loss: tensor(0.4883)<br>
5 Traning Loss: tensor(0.4883)<br>
6 Traning Loss: tensor(0.4883)<br>
7 Traning Loss: tensor(0.4883)<br>
8 Traning Loss: tensor(0.4883)<br>
9 Traning Loss: tensor(0.4883)</p>
<p>Thanks</p>

ptrblck: <p>You are detaching the output from the computation graph by rewrapping it into a deprecated <code>Variable</code> here:</p>
<pre data-code-wrap="python"><code class="lang-python">pt_u_true = Variable(torch.from_numpy(u_true).float(), requires_grad=False).to(device)
</code></pre>
<p>However, based on this code snippet it seems you are also using numpy arrays, which won’t be attached to the computation graph by Autograd in the first place.<br>
If you need to use other libraries such as numpy, you would need to write custom <code>autograd.Function</code>s and implement the <code>backward</code> method manually.</p>
<aside class="quote no-group" data-username="Vinayak_Vijay1" data-post="1" data-topic="147726">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vinayak_vijay1/48/48067_2.png" class="avatar"> Vinayak_Vijay1:</div>
<blockquote>
<p>If I use only ADAM with optimizer.step(), the loss function converges</p>
</blockquote>
</aside>
<p>That doesn’t seem right, as no training should happen.</p>

Vinayak_Vijay1: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="147726" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>That doesn’t seem right, as no training should happen.</p>
</blockquote>
</aside>
<p>The training is taking place when i use ADAM and just perform optimizer.step(). Here’s the code:</p>
<pre><code class="lang-auto">## Training
iterations = 10
optimizer1 = torch.optim.Adam(net.parameters(),lr = 0.01)
loss_array = np.zeros((iterations))

for epoch in range(iterations):
  optimizer1.zero_grad() # to make the gradients zero

  ## Physics informed loss
  all_zeros = np.zeros((500,1))
  pt_x_collocation = Variable(torch.from_numpy(x_collocation).float(), requires_grad=True).to(device)
  pt_all_zeros = Variable(torch.from_numpy(all_zeros).float(), requires_grad=False).to(device)
  f_out = f(pt_x_collocation, net) # output of f(x,t)
  mse_pinn = mse_cost_function(f_out, pt_all_zeros)

  ## Training data loss
  u_train = net(pt_x_collocation)
  pt_u_true = Variable(torch.from_numpy(u_true).float(), requires_grad=False).to(device)
  mse_training = mse_cost_function(u_train, pt_u_true)

  # Combining the loss functions
  loss = mse_pinn + mse_training
  loss_array[epoch] = loss
  loss.backward() 

  optimizer1.step()
    
  with torch.autograd.no_grad():
    print(epoch,"Traning Loss:",loss.data)
    
</code></pre>
<p>The output for just 10 epochs is:<br>
0 Traning Loss: tensor(0.7643)<br>
1 Traning Loss: tensor(0.7007)<br>
2 Traning Loss: tensor(0.6511)<br>
3 Traning Loss: tensor(0.6144)<br>
4 Traning Loss: tensor(0.5886)<br>
5 Traning Loss: tensor(0.5705)<br>
6 Traning Loss: tensor(0.5555)<br>
7 Traning Loss: tensor(0.5401)<br>
8 Traning Loss: tensor(0.5223)<br>
9 Traning Loss: tensor(0.5022)</p>

ptrblck: <p>Could you check the gradients of the parameters after the backward call?<br>
I would assume they are all set to zero since the <code>optimizer.zero_grad()</code> would be resetting them.<br>
I don’t see where the computation graph should be coming from in your code snippet based on my previous statements or a) detaching the graph explicitly and b) using numpy.</p>

Vinayak_Vijay1: <p>Hi, I have printed the gradients of first hidden layer to show that they are indeed being calculated when i use optimizer.step(). Here’s the code snippet and below that is the output for the first few epochs:</p>
<pre><code class="lang-auto">## Training
iterations = 10
optimizer1 = torch.optim.Adam(net.parameters(),lr = 0.01)
loss_array = np.zeros((iterations))

for epoch in range(iterations):
  optimizer1.zero_grad() # to make the gradients zero

  ## Physics informed loss
  all_zeros = np.zeros((500,1))
  pt_x_collocation = Variable(torch.from_numpy(x_collocation).float(), requires_grad=True).to(device)
  pt_all_zeros = Variable(torch.from_numpy(all_zeros).float(), requires_grad=False).to(device)
  f_out = f(pt_x_collocation, net) # output of f(x,t)
  mse_pinn = mse_cost_function(f_out, pt_all_zeros)

  ## Training data loss
  u_train = net(pt_x_collocation)
  pt_u_true = Variable(torch.from_numpy(u_true).float(), requires_grad=False).to(device)
  mse_training = mse_cost_function(u_train, pt_u_true)

  # Combining the loss functions
  loss = mse_pinn + mse_training
  loss_array[epoch] = loss
  loss.backward() 

  # for name, param in net.named_parameters():
  #   if param.requires_grad:
  #       print(name, param.grad)
  print('Hidden layer 1 weights gradient:',net.hidden_layer1.weight.grad)
  optimizer1.step()
    
  with torch.autograd.no_grad():
    print(epoch,"Traning Loss:",loss.data)
    
</code></pre>
<p>The output is:<br>
Hidden layer 1 weights gradient: tensor([[ 4.5100e-07],<br>
[ 8.3037e-08],<br>
[-3.1868e-09],<br>
[-1.2420e-08],<br>
[ 1.9485e-07],<br>
[ 1.8525e-08],<br>
[-3.7920e-07],<br>
[-3.4814e-07],<br>
[-1.6464e-07],<br>
[ 1.1097e-07]])<br>
0 Traning Loss: tensor(0.1058)<br>
Hidden layer 1 weights gradient: tensor([[-5.2541e-07],<br>
[ 1.9976e-06],<br>
[ 1.1666e-07],<br>
[ 1.9102e-07],<br>
[-7.6507e-07],<br>
[-5.7566e-08],<br>
[ 2.1010e-06],<br>
[ 5.4820e-07],<br>
[ 7.9201e-07],<br>
[ 8.8915e-07]])<br>
1 Traning Loss: tensor(0.1038)<br>
Hidden layer 1 weights gradient: tensor([[-4.4913e-08],<br>
[ 4.0377e-07],<br>
[ 5.9259e-08],<br>
[ 6.0829e-08],<br>
[-2.1468e-07],<br>
[-1.2919e-08],<br>
[ 3.1558e-07],<br>
[ 7.9497e-08],<br>
[ 2.5290e-07],<br>
[ 3.2178e-07]])<br>
2 Traning Loss: tensor(0.0976)<br>
Hidden layer 1 weights gradient: tensor([[ 4.2063e-07],<br>
[-1.0188e-06],<br>
[-4.8912e-08],<br>
[-4.5130e-08],<br>
[ 2.5593e-07],<br>
[-5.5157e-09],<br>
[-1.3609e-06],<br>
[-3.6893e-07],<br>
[-2.9356e-07],<br>
[-2.5360e-07]])<br>
3 Traning Loss: tensor(0.0938)</p>

ptrblck: <p>Thanks for the update!<br>
I missed this part of the code:</p>
<pre><code class="lang-python">f_out = f(pt_x_collocation, net) # output of f(x,t)
mse_pinn = mse_cost_function(f_out, pt_all_zeros)
...
loss = mse_pinn + mse_training
loss.backward()
</code></pre>
<p>which is not detaching the computation graph and will thus create the gradients.<br>
<code>mse_training</code> will still be detached and not influence the gradient calculation.</p>
<p>EDIT: I also misread the second part and don’t know why you are setting the <code>requires_grad</code> attribute of the targets to <code>True</code>.<br>
In any case that’s not necessary if you don’t need to update the targets and remove the <code>Variable</code> usage.<br>
Once this is done, check the gradients again, verify that they are calculated, check the parameters before and after the <code>step()</code> operation and you should see the update.</p>