Issue with creating a mathematical function through neural network

Stephane: <p>Dear all,</p>
<p>I need to imitate a mathematical function (u(x,y)) by a neural network.<br>
The loss I need to minimize is (-u’‘(x,y)-f(x,y))^2 where u’’ is the second derivative of u and f is a function corresponding to this second derivative. In fact, I am comparing the Laplacian of both functions.<br>
In other word, I am trying to first determine u’’ for a given f(x,y), and after, using autograd pytorch function, I try to estimate u.</p>
<p>The following code is working fine but the second dimension (y) is not taken into account. See the 2 graphs after the codes : the 1st one stands for the nnfunction and the second one is the real function I wish to imitate.<br>
Any clue why it still does not work properly ?</p>
<p>import numpy as np<br>
import matplotlib.pyplot as plt<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import numpy as np<br>
learning_rate = 0.01<br>
num_epochs = 200<br>
N = 200<br>
A = []<br>
U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])<br>
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))<br>
x = np.linspace(0, 1, N)<br>
y = np.linspace(0, 1, N)<br>
z = []<br>
for i in range(N):<br>
z.append([x[i],y[i]])<br>
input = torch.tensor(z).to(dtype = torch.float32).requires_grad_(True)<br>
class Net(nn.Module):<br>
def init(self):<br>
super(Net, self).init()<br>
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))<br>
self.l1 = nn.Linear(2,128,)<br>
self.l2 = nn.Linear(128,10)<br>
self.l3 = nn.Linear(10,1)</p>
<p>def forward(self,input):<br>
x = input[0]<br>
y = input[1]<br>
out = F.tanh(self.l1(input))<br>
out = F.tanh(self.l2(out))<br>
out = self.l3(out)<br>
return out * x * (1-x) * y * (1-y)<br>
model = Net()<br>
gradient = torch.optim.Adam(model.parameters(), lr = learning_rate)<br>
def loss(input, U, A, f):<br>
nablaU = None<br>
for i in input:<br>
output = model(i)<br>
<span class="hashtag">#print</span>(“output”, output)<br>
Uneuronal_dx = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]<br>
<span class="hashtag">#print</span>(“Uneuronal_dx”, Uneuronal_dx)<br>
Uneuronal_d2x = torch.autograd.grad(Uneuronal_dx[0], i, create_graph=True, retain_graph=True)[0]<br>
<span class="hashtag">#print</span>(“Uneuronal_d2x”, Uneuronal_d2x)</p>
<pre><code>Uneuronal_dy = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
#print("Uneuronal_dy", Uneuronal_dy)
Uneuronal_d2y = torch.autograd.grad(Uneuronal_dy[1], i, create_graph=True, retain_graph=True)[0]
#print("Uneuronal_d2y", Uneuronal_d2y)

v = torch.unsqueeze(torch.pow(torch.sub(torch.mul(torch.add(Uneuronal_d2x[0], Uneuronal_d2y[1]), -1),f(i, U)), 2), dim=-1) 
#print("v", v)
if nablaU is None:
    nablaU = v
    #print("nablaU", nablaU)
else:
    nablaU = torch.cat( (nablaU ,v), dim=0)
    #print("nablaU", nablaU)
</code></pre>
<p>return(torch.mean(nablaU))<br>
for i in range(num_epochs):<br>
gradient.zero_grad()<br>
l = loss(input, U, A, f)<br>
l.backward()<br>
print(“loss”, l)<br>
gradient.step()<br>
def u_plot(x, y):<br>
return np.sin(np.pi * x) * np.sin(2 * np.pi * y)<br>
input1 = input[:,0].detach().numpy()<br>
input2 = input[:,1].detach().numpy()<br>
output = []<br>
for i in input:<br>
output.append([model.forward(i).detach().numpy()])<br>
output = np.array(output).reshape(N, 1)<br>
fig = plt.figure()<br>
plt.subplots(figsize=(15, 8))<br>
ax = plt.axes(projection=‘3d’)<br>
X, Y = np.meshgrid(input1, input2)<br>
ax.plot_surface(Y, X, output, rstride=1, cstride=1, edgecolor=‘none’)<br>
ax.set_title(‘Fontion $u(x,y)$’);<br>
ax.set_xlabel(‘y’)<br>
ax.set_ylabel(‘x’)<br>
ax.set_zlabel(‘Network’)<br>
fig = plt.figure()<br>
plt.subplots(figsize=(15, 8))<br>
ax = plt.axes(projection=‘3d’)<br>
X, Y = np.meshgrid(input1, input2)<br>
ax.plot_surface(Y, X, u_plot(X, Y), rstride=1, cstride=1, edgecolor=‘none’)<br>
ax.set_title(‘Fontion $u(x,y)$’);<br>
ax.set_xlabel(‘y’)<br>
ax.set_ylabel(‘x’)<br>
ax.set_zlabel(‘u(x, y)’);</p>

Stephane: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/76e203d3778eb8e6ac6c3a5819fc3dbce1079365" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365_2_690x358.jpeg" alt="image" data-base62-sha1="gXGzJiwy5WzRxtOmKb5IyIprFVX" width="690" height="358" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365_2_690x358.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e203d3778eb8e6ac6c3a5819fc3dbce1079365.jpeg 2x" data-dominant-color="C6D3DC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">816×424 67.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Stephane: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/400c4833b4a13134d40c802a593c3f9ffd273886" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886_2_690x357.jpeg" alt="image" data-base62-sha1="98AS4WZGDtbKd7g7SvcgpXACK7I" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886_2_690x357.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/0/400c4833b4a13134d40c802a593c3f9ffd273886.jpeg 2x" data-dominant-color="CAD5DD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">833×432 65.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Stephane: <p>Idea is after to generalize this into dim 4, 5…<br>
Can someone help?</p>

Stephane: <p>Looking more into details, I was confirmed by colleagues that there were some bugs in Pytorch since there is no error at all. Can someone from Pytorch just confirm this?</p>

ptrblck: <aside class="quote no-group" data-username="Stephane" data-post="5" data-topic="168918">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dfb087/48.png" class="avatar"> Stephane:</div>
<blockquote>
<p>I was confirmed by colleagues that there were some bugs in Pytorch since there is no error at all. Can someone from Pytorch just confirm this?</p>
</blockquote>
</aside>
<p>I don’t think anyone would confirm speculative claims about unknown errors, so could you add more details please and explain what exactly is broken?</p>
<p>Also, you have still not followed up on my previous request to check if you are detaching the computation graph as described in <a href="https://discuss.pytorch.org/t/backward-does-not-work/168732/4">this post</a>.</p>

ConvolutionalAtom: <p>Also, please put formatting for python code correctly. It makes it way more readable for others.</p>

Stephane: <p>I have followed up your previous request. I checked but the backward function seems to work properly. For a reason which seems to have nothing to do with your comments, the network seems to ignore y. That is our my only worry</p>

Stephane: <p><strong>OK, I think the code is more easy to read like this :</strong></p>
<p>import numpy as np<br>
import matplotlib.pyplot as plt<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import numpy as np</p>
<p>learning_rate = 0.01<br>
num_epochs = 200<br>
N = 200<br>
A = []<br>
U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])<br>
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))<br>
x = np.linspace(0, 1, N)<br>
y = np.linspace(0, 1, N)<br>
z = []<br>
for i in range(N):<br>
z.append([x[i],y[i]])<br>
input = torch.tensor(z).to(dtype = torch.float32).requires_grad_(True)</p>
<p>class Net(nn.Module):<br>
def init(self):<br>
super(Net, self).init()<br>
self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))<br>
self.l1 = nn.Linear(2,128,)<br>
self.l2 = nn.Linear(128,10)<br>
self.l3 = nn.Linear(10,1)</p>
<p>def forward(self,input):<br>
x = input[0]<br>
y = input[1]<br>
out = F.tanh(self.l1(input))<br>
out = F.tanh(self.l2(out))<br>
out = self.l3(out)<br>
return out * x * (1-x) * y * (1-y)</p>
<p>model = Net()<br>
gradient = torch.optim.Adam(model.parameters(), lr = learning_rate)<br>
def loss(input, U, A, f):<br>
nablaU = None<br>
for i in input:<br>
output = model(i)</p>
<p>Uneuronal_dx = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]<br>
Uneuronal_d2x = torch.autograd.grad(Uneuronal_dx[0], i, create_graph=True, retain_graph=True)[0]</p>
<p>Uneuronal_dy = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]<br>
Uneuronal_d2y = torch.autograd.grad(Uneuronal_dy[1], i, create_graph=True, retain_graph=True)[0]</p>
<p>v = torch.unsqueeze(torch.pow(torch.sub(torch.mul(torch.add(Uneuronal_d2x[0], Uneuronal_d2y[1]), -1),f(i, U)), 2), dim=-1)<br>
if nablaU is None:<br>
nablaU = v<br>
else:<br>
nablaU = torch.cat( (nablaU ,v), dim=0)<br>
return(torch.mean(nablaU))</p>
<p>for i in range(num_epochs):<br>
gradient.zero_grad()<br>
l = loss(input, U, A, f)<br>
l.backward()<br>
print(“loss”, l)<br>
gradient.step()<br>
def u_plot(x, y):<br>
return np.sin(np.pi * x) * np.sin(2 * np.pi * y)<br>
input1 = input[:,0].detach().numpy()<br>
input2 = input[:,1].detach().numpy()<br>
output = []<br>
for i in input:<br>
output.append([model.forward(i).detach().numpy()])<br>
output = np.array(output).reshape(N, 1)</p>
<p>fig = plt.figure()<br>
plt.subplots(figsize=(15, 8))<br>
ax = plt.axes(projection=‘3d’)<br>
X, Y = np.meshgrid(input1, input2)<br>
ax.plot_surface(Y, X, output, rstride=1, cstride=1, edgecolor=‘none’)<br>
ax.set_title(‘Fontion $u(x,y)$’);<br>
ax.set_xlabel(‘y’)<br>
ax.set_ylabel(‘x’)<br>
ax.set_zlabel(‘Network’)</p>
<p>fig = plt.figure()<br>
plt.subplots(figsize=(15, 8))<br>
ax = plt.axes(projection=‘3d’)<br>
X, Y = np.meshgrid(input1, input2)<br>
ax.plot_surface(Y, X, u_plot(X, Y), rstride=1, cstride=1, edgecolor=‘none’)<br>
ax.set_title(‘Fontion $u(x,y)$’);<br>
ax.set_xlabel(‘y’)<br>
ax.set_ylabel(‘x’)<br>
ax.set_zlabel(‘u(x, y)’);</p>

ConvolutionalAtom: <p>What I meant is to use the symbol ```</p>
<p>You can put the entire code between two of these symbols, like this <code>```  Your python code ```</code>.<br>
For instance, in the format you just used, nested structure of function definitions and loops is not clear.<br>
If you use the symbol ```, it will respect the original format of python code.<br>
If you change the original post according to this, it will be the best</p>

Stephane: <pre><code class="lang-auto">import numpy as np
import matplotlib.pyplot as plt
import torch 
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

learning_rate = 0.01
num_epochs = 200

N = 200
A = []

U = lambda x: torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1])
f = lambda x, u: (5 * torch.pi**2* torch.sin(torch.pi * x[0]) * torch.sin(2 * torch.pi * x[1]))

x = np.linspace(0, 1, N)
y = np.linspace(0, 1, N)

z = []
for i in range(N):
    z.append([x[i],y[i]])
input = torch.tensor(z).to(dtype = torch.float32).requires_grad_(True)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Sequential(nn.Linear(2, 64), nn.Tanh(), nn.Linear(64, 1, bias = False))

        self.l1 = nn.Linear(2,128,)
        self.l2 = nn.Linear(128,10)
        self.l3 = nn.Linear(10,1)
    
    def forward(self,input):
        x = input[0]
        y = input[1]
        out = F.tanh(self.l1(input))
        out = F.tanh(self.l2(out))
        out = self.l3(out)
        return out * x * (1-x) * y * (1-y)

model = Net()
gradient = torch.optim.Adam(model.parameters(), lr = learning_rate)

def loss(input, U, A, f):
    nablaU = None

    for i in input:
        output = model(i)
        
        Uneuronal_dx = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
        Uneuronal_d2x = torch.autograd.grad(Uneuronal_dx[0], i, create_graph=True, retain_graph=True)[0]

        Uneuronal_dy = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
        Uneuronal_d2y = torch.autograd.grad(Uneuronal_dy[1], i, create_graph=True, retain_graph=True)[0]
        
        v = torch.unsqueeze(torch.pow(torch.sub(torch.mul(torch.add(Uneuronal_d2x[0], Uneuronal_d2y[1]), -1),f(i, U)), 2), dim=-1) 
        
        if nablaU is None:
            nablaU = v
        else:
            nablaU = torch.cat( (nablaU ,v), dim=0)

    return(torch.mean(nablaU))

for i in range(num_epochs):
    gradient.zero_grad()
    l = loss(input, U, A, f)
    l.backward()
    print("loss", l)
    gradient.step()
    
def u_plot(x, y):
    return np.sin(np.pi * x) * np.sin(2 * np.pi * y)

input1 = input[:,0].detach().numpy()
input2 = input[:,1].detach().numpy()

output = []
for i in input:
  output.append([model.forward(i).detach().numpy()])

output = np.array(output).reshape(N, 1)

fig = plt.figure()
plt.subplots(figsize=(15, 8))
ax = plt.axes(projection='3d')
X, Y = np.meshgrid(input1, input2)
ax.plot_surface(Y, X, output, rstride=1, cstride=1, edgecolor='none')
ax.set_title('Fontion $u(x,y)$');
ax.set_xlabel('y')
ax.set_ylabel('x')
ax.set_zlabel('Network');

fig = plt.figure()
plt.subplots(figsize=(15, 8))
ax = plt.axes(projection='3d')
X, Y = np.meshgrid(input1, input2)
ax.plot_surface(Y, X, u_plot(X, Y), rstride=1, cstride=1, edgecolor='none')
ax.set_title('Fontion $u(x,y)$');
ax.set_xlabel('y')
ax.set_ylabel('x')
ax.set_zlabel('u(x, y)');

</code></pre>

Stephane: <p>So I have just sent a readable version.<br>
Again, aim is to train a neural network minimimizing (-u’'-f)^2 which is the loss function. Then, the use of the autograd functions shall help find the real u.<br>
In dim 3, we are comparing laplacian.</p>
<p>According to me, the error is to find in</p>
<pre><code class="lang-auto">      Uneuronal_dx = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
        Uneuronal_d2x = torch.autograd.grad(Uneuronal_dx[0], i, create_graph=True, retain_graph=True)[0]

        Uneuronal_dy = torch.autograd.grad(output, i, grad_outputs = torch.ones_like(output), create_graph=True, retain_graph=True)[0]
        Uneuronal_d2y = torch.autograd.grad(Uneuronal_dy[1], i, create_graph=True, retain_graph=True)[0]
        
        v = torch.unsqueeze(torch.pow(torch.sub(torch.mul(torch.add(Uneuronal_d2x[0], Uneuronal_d2y[1]), -1),f(i, U)), 2), dim=-1) 
 
</code></pre>
<p>Thank you for your help !</p>