Backward() does not work

Stephane: <p>Dear all,</p>
<p>I have the following code which is used to train a neural function. u(x,y) is my function and I am trying to get it by minimizing the loss (-u’‘(x,y)-f)^2 where u’'(x,y) stands for Laplacian and f as well.<br>
Everything is running fine but I have an issue with the backward() function which does not work here. The loss remains at the exact same level meaning that the neural network does not work.<br>
Can someone help?</p>
<p>Thanks in advance</p>
<p>import torch<br>
import torch.nn as nn<br>
import numpy as np</p>
<p>learning_rate = 0.01<br>
num_epochs = 15</p>
<p>N = 15</p>
<p>U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])<br>
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))</p>
<p>x = np.arange(0, 1.1, 1/(N+1))[1:N+1]<br>
p = np.array([])<br>
for i in range(N):<br>
p = np.append(p, x)<br>
q = np.sort(p)<br>
z = np.column_stack((p, q))<br>
input = torch.tensor(z).to(dtype = torch.float32).requires_grad_(True)</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))</p>
<pre><code>def forward(self, input):
    x = input[0]
    y = input[1]
    input = self.fc(input)
    return input * x * (1-x) * y * (1-y)
</code></pre>
<p>model = Net()<br>
gradient = torch.optim.Adam(model.parameters(), lr = learning_rate)</p>
<p>def loss(x, U, f):<br>
nablaU = torch.tensor([0]).to(dtype = torch.float32).requires_grad_(True)<br>
output = model.forward(x)<br>
Uneuronal_d = torch.autograd.grad(output, x, grad_outputs = torch.ones_like(output), allow_unused=True, retain_graph = True, create_graph=True)<br>
nabla_x = torch.tensor([0])<br>
for grad in Uneuronal_d:<br>
grads_inner = torch.autograd.grad(grad, x, grad_outputs = torch.ones_like(x), create_graph=True, retain_graph=True, allow_unused=True)[0]<br>
nabla_x = torch.tensor([torch.sum(nabla_x + grads_inner)])<br>
nablaU = torch.cat((nablaU , torch.tensor([nabla_x])), dim=0)<br>
return(torch.mean(nablaU[1:]))</p>
<p>for i in range(num_epochs):<br>
gradient.zero_grad()<br>
l = loss(input, U, f)<br>
l.backward()<br>
print(l)<br>
gradient.step()</p>

ptrblck: <p>Your code is hard to read as it’s not formatted properly.<br>
However, it seems you are detaching tensors implicitly by recreating a tensor in:</p>
<pre><code class="lang-python">nabla_x = torch.tensor([torch.sum(nabla_x + grads_inner)])
nablaU = torch.cat((nablaU , torch.tensor([nabla_x])), dim=0)
return(torch.mean(nablaU[1:]))
</code></pre>
<p><code>nablaU</code> is now a new tensor, without any Autograd history.<br>
Also, <code>nablaU</code> is a newly created leaf tensor also without any Autograd history:</p>
<pre><code class="lang-python">nablaU = torch.tensor([0]).to(dtype = torch.float32).requires_grad_(True)
</code></pre>

Stephane: <p>Hi,<br>
Thanks,<br>
Taking into account your comments, it seems to work but I am worried about the loss level.<br>
It has decreased but the level remains high.<br>
It is probably linked to the fact that we are using the second deirvative in the loss. Anyway, I need to recheck it and will revert if need be.<br>
Thank you again for your reactivity</p>

ptrblck: <p>No, I don’t think your approach works as intended.<br>
The <code>backward</code> call will not raise an error, since <code>nablaU</code> is differentiable, but since it’s a newly created tensor in the <code>forward</code> method it will never be updated.<br>
Also, since you are detaching <code>nabla_x</code>, the previously used parameters also shouldn’t get any gradients.<br>
Run a single iteration as:</p>
<pre><code class="lang-python">output = model(input)
loss = criterion(output, target)
loss.backward()
</code></pre>
<p>and check the <code>.grad</code> attributes of the model’s parameters via:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    print("param {}, grad {}".format(name, param.grad))
</code></pre>
<p>which should show <code>None</code> (unless I’m missing how the output is still attached to the computation graph).<br>
Do not call <code>optimizer.zero_grad()</code> before the first <code>backward</code> call as it will fill each <code>.grad</code> attribute with zeros (in the default setup).</p>

Stephane: <p>Thank you, seems that it is better but I still have an issue. The neural network seems to forget to take the second dimension y into account. See the 2 graphs after the codes : the 1st one stands for the nnfunction and the second one is the real function I wish to imitate (hope that you can see the graphs).<br>
Any clue why it still does not work properly ?<br>
Thank you very much for your support.</p>
<p>import numpy as np<br>
import matplotlib.pyplot as plt<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import numpy as np</p>
<p>learning_rate = 0.01<br>
num_epochs = 200</p>
<p>N = 200<br>
A = []</p>
<p>U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])<br>
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))</p>
<p>x = np.linspace(0, 1, N)<br>
y = np.linspace(0, 1, N)</p>
<p>z = []<br>
for i in range(N):<br>
z.append([x[i],y[i]])<br>
input = torch.tensor(z).to(dtype = torch.float32).requires_grad_(True)</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))</p>
<pre><code>    self.l1 = nn.Linear(2,128,)
    self.l2 = nn.Linear(128,10)
    self.l3 = nn.Linear(10,1)

def forward(self,input):
    x = input[0]
    y = input[1]
    out = F.tanh(self.l1(input))
    out = F.tanh(self.l2(out))
    out = self.l3(out)
    return out * x * (1-x) * y * (1-y)
</code></pre>
<p>model = Net()<br>
gradient = torch.optim.Adam(model.parameters(), lr = learning_rate)</p>
<p>def loss(input, U, A, f):<br>
nablaU = None</p>
<pre><code>for i in input:
    output = model(i)
    #print("output", output)
    Uneuronal_dx = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_dx", Uneuronal_dx)
    Uneuronal_d2x = torch.autograd.grad(Uneuronal_dx[0], i, create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_d2x", Uneuronal_d2x)
    
    Uneuronal_dy = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_dy", Uneuronal_dy)
    Uneuronal_d2y = torch.autograd.grad(Uneuronal_dy[1], i, create_graph=True, retain_graph=True)[0]
    #print("Uneuronal_d2y", Uneuronal_d2y)

    v = torch.unsqueeze(torch.pow(torch.sub(torch.mul(torch.add(Uneuronal_d2x[0], Uneuronal_d2y[1]), -1),f(i, U)), 2), dim=-1) 
    #print("v", v)
    if nablaU is None:
        nablaU = v
        #print("nablaU", nablaU)
    else:
        nablaU = torch.cat( (nablaU ,v), dim=0)
        #print("nablaU", nablaU)
        
  
return(torch.mean(nablaU))
</code></pre>
<p>for i in range(num_epochs):<br>
gradient.zero_grad()<br>
l = loss(input, U, A, f)<br>
l.backward()<br>
print(“loss”, l)<br>
gradient.step()</p>
<p>def u_plot(x, y):<br>
return np.sin(np.pi * x) * np.sin(2 * np.pi * y)</p>
<p>input1 = input[:,0].detach().numpy()<br>
input2 = input[:,1].detach().numpy()</p>
<p>output = []<br>
for i in input:<br>
output.append([model.forward(i).detach().numpy()])</p>
<p>output = np.array(output).reshape(N, 1)</p>
<p>fig = plt.figure()<br>
plt.subplots(figsize=(15, 8))<br>
ax = plt.axes(projection=‘3d’)<br>
X, Y = np.meshgrid(input1, input2)<br>
ax.plot_surface(Y, X, output, rstride=1, cstride=1, edgecolor=‘none’)<br>
ax.set_title(‘Fontion $u(x,y)$’);<br>
ax.set_xlabel(‘y’)<br>
ax.set_ylabel(‘x’)<br>
ax.set_zlabel(‘Network’)</p>
<p>fig = plt.figure()<br>
plt.subplots(figsize=(15, 8))<br>
ax = plt.axes(projection=‘3d’)<br>
X, Y = np.meshgrid(input1, input2)<br>
ax.plot_surface(Y, X, u_plot(X, Y), rstride=1, cstride=1, edgecolor=‘none’)<br>
ax.set_title(‘Fontion $u(x,y)$’);<br>
ax.set_xlabel(‘y’)<br>
ax.set_ylabel(‘x’)<br>
ax.set_zlabel(‘u(x, y)’);</p>

Stephane: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/76e203d3778eb8e6ac6c3a5819fc3dbce1079365" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365_2_690x358.jpeg" alt="image" data-base62-sha1="gXGzJiwy5WzRxtOmKb5IyIprFVX" width="690" height="358" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365_2_690x358.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365.jpeg 2x" data-dominant-color="C6D3DC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">816×424 67.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Stephane: <p>And here is the real function<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/400c4833b4a13134d40c802a593c3f9ffd273886" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886_2_690x357.jpeg" alt="image" data-base62-sha1="98AS4WZGDtbKd7g7SvcgpXACK7I" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886_2_690x357.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886.jpeg 2x" data-dominant-color="CAD5DD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">833×432 65.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Stephane: <p>Since my last question is more related to the code (and Backward now works), does it make sense that I post a new discussion?</p>

ptrblck: <p>Yes, I would suggest to create a new issue regarding details of your model and training procedure.<br>
You’ve also never followed up on my questions and suggestions from my previous post so I don’t know if gradients are calculated at all.</p>

Stephane: <p>OK good to know.<br>
Will do it immediately.<br>
Besides, I have followed up on your suggestions and can tell you that thanks to you, it works!<br>
The code I put here is the code I had  issues with the Backward() function.<br>
Now, I am quite near to the initial aim but have still a small other issue <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
Thank you very much!</p>