Approximating sine function using neural network

MrRobot: <h1><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h1>
<p>I am trying to build a function approximator using PyTorch. But my neural network does not seem to learn anything.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73759c04862e9ad6184d39abd4a81c77b9d1939b.png" alt="image" data-base62-sha1="gtoSWrVyEzfv11nc4Y339NY6wgb" width="384" height="248"></p>
<p>The full executable code is as follows. I am not sure what mistakes I have made. Could someone help me? Thank you in advance.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import numpy as np

from torch import nn, optim
from torch.nn import functional as F
from torch.utils.data import TensorDataset, DataLoader

from sklearn.model_selection import train_test_split

LR = 1e-6
MAX_EPOCH = 10
BATCH_SIZE = 512

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

class SineApproximator(nn.Module):
    def __init__(self):
        super(SineApproximator, self).__init__()
        self.regressor = nn.Sequential(nn.Linear(1, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1))
    def forward(self, x):
        output = self.regressor(x)
        return x

X = np.random.rand(10**5) * 2 * np.pi
y = np.sin(X)

X_train, X_val, y_train, y_val = map(torch.tensor, train_test_split(X, y, test_size=0.2))
train_dataloader = DataLoader(TensorDataset(X_train.unsqueeze(1), y_train.unsqueeze(1)), batch_size=BATCH_SIZE,
                              pin_memory=True, shuffle=True)
val_dataloader = DataLoader(TensorDataset(X_val.unsqueeze(1), y_val.unsqueeze(1)), batch_size=BATCH_SIZE,
                            pin_memory=True, shuffle=True)

model = SineApproximator().to(device)
optimizer = optim.Adam(model.parameters(), lr=LR)
criterion = nn.MSELoss(reduction="mean")

# training loop
train_loss_list = list()
val_loss_list = list()
for epoch in range(MAX_EPOCH):
    print("epoch %d / %d" % (epoch+1, MAX_EPOCH))
    model.train()
    # training loop
    temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)

        optimizer.zero_grad()

        score = model(X_train)
        loss = criterion(input=score, target=y_train)
        loss.requires_grad = True
        loss.backward()

        optimizer.step()

        temp_loss_list.append(loss.detach().cpu().numpy())
    
    temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)

        score = model(X_train)
        loss = criterion(input=score, target=y_train)

        temp_loss_list.append(loss.detach().cpu().numpy())
    
    train_loss_list.append(np.average(temp_loss_list))

    # validation
    model.eval()
    
    temp_loss_list = list()
    for X_val, y_val in val_dataloader:
        X_val = X_val.type(torch.float32).to(device)
        y_val = y_val.type(torch.float32).to(device)

        score = model(X_val)
        loss = criterion(input=score, target=y_val)

        temp_loss_list.append(loss.detach().cpu().numpy())
    
    val_loss_list.append(np.average(temp_loss_list))

    print("\ttrain loss: %.5f" % train_loss_list[-1])
    print("\tval loss: %.5f" % val_loss_list[-1])
</code></pre>

MrRobot: <p>This is a dumb mistake. I wrongly return <code>x</code> instead of <code>output</code> in the forward function. The resulting model could successfully approximate the sine function. As could be seen below, the prediction could perfectly match the sine curve in validation data.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d4ffd427ee4a0bbe9ba3001ce7939456ab583fa.png" alt="image" data-base62-sha1="ka6HnivpaBlLTIQ4iHDYlEn1hRU" width="380" height="248"></p>

desmond13: <p>Hi <a class="mention" href="/u/mrrobot">@MrRobot</a>, I changed the <code>x</code> to <code>output</code> but I get the following error:</p>
<pre><code class="lang-auto">epoch 1 / 10
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-12-d02b18943b31&gt; in &lt;module&gt;
     40         score = model(X_train)
     41         loss = criterion(input=score, target=y_train)
---&gt; 42         loss.requires_grad = True
     43         loss.backward()
     44 

RuntimeError: you can only change requires_grad flags of leaf variables.
</code></pre>
<p>Could you maybe post the working code please?</p>

ptrblck: <p><code>loss</code> should already require gradients. Could you check its <code>loss.grad_fn</code> and make sure you see a valid function and not <code>None</code>?<br>
If you are getting a <code>None</code> output, it seems that the computation graph was detached at some point and we would need to check your code.</p>

desmond13: <p>Hi, I tried, if I do:<br>
<code>print("loss.grad_fn: ",loss.grad_fn)</code><br>
I get this:<br>
<code>loss.grad_fn:  &lt;MseLossBackward object at 0x7f9152ca7358&gt;</code></p>
<p>My code is the same as the one above in which I changed <code>return x</code> with <code>return output</code></p>
<p>However, the code is below for convenience<br>
I arrive to see only</p>
<pre><code class="lang-auto">Debug 5
loss.grad_fn:  &lt;MseLossBackward object at 0x7f9152905b00&gt;
</code></pre>
<pre><code class="lang-auto"># In[15]:


import torch
import numpy as np

from torch import nn, optim
from torch.nn import functional as F
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split

import matplotlib
import matplotlib.pyplot as plt
import numpy as np


# In[16]:


LR = 1e-6
MAX_EPOCH = 10
BATCH_SIZE = 512

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")


# In[17]:


class SineApproximator(nn.Module):
    def __init__(self):
        super(SineApproximator, self).__init__()
        self.regressor = nn.Sequential(nn.Linear(1, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1))
    def forward(self, x):
        output = self.regressor(x)
        return output

X = np.random.rand(10**5) * 2 * np.pi
y = np.sin(X)

print(X)
print(np.size(X))
print(y)
print(np.size(y))


# In[18]:



# # Data for plotting
# # t = np.arange(0.0, 2.0, 0.01)
# # s = 1 + np.sin(2 * np.pi * t)

# matplotlib.rcParams['agg.path.chunksize'] = 10000

# fig, ax = plt.subplots()
# ax.plot(X, y)

# # ax.set(xlabel='X', ylabel='y',
# #        title='Test plot')
# # ax.grid()

# # fig.savefig("test.png")
# plt.show()


# In[19]:


class SineApproximator(nn.Module):
    def __init__(self):
        super(SineApproximator, self).__init__()
        self.regressor = nn.Sequential(nn.Linear(1, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1))
    def forward(self, x):
        output = self.regressor(x)
        return output

X = np.random.rand(10**5) * 2 * np.pi
y = np.sin(X)

# Data for plotting
# t = np.arange(0.0, 2.0, 0.01)
# s = 1 + np.sin(2 * np.pi * t)

fig, ax = plt.subplots()
ax.plot(X, y)

ax.set(xlabel='X', ylabel='y',
       title='Test plot')
ax.grid()

# fig.savefig("test.png")
plt.show()

X_train, X_val, y_train, y_val = map(torch.tensor, train_test_split(X, y, test_size=0.2))
train_dataloader = DataLoader(TensorDataset(X_train.unsqueeze(1), y_train.unsqueeze(1)), batch_size=BATCH_SIZE,
                              pin_memory=True, shuffle=True)
val_dataloader = DataLoader(TensorDataset(X_val.unsqueeze(1), y_val.unsqueeze(1)), batch_size=BATCH_SIZE,
                            pin_memory=True, shuffle=True)

model = SineApproximator().to(device)
optimizer = optim.Adam(model.parameters(), lr=LR)
criterion = nn.MSELoss(reduction="mean")

# training loop
train_loss_list = list()
val_loss_list = list()
print("Debug 1")
for epoch in range(MAX_EPOCH):
    print("epoch %d / %d" % (epoch+1, MAX_EPOCH))
    model.train()
    # training loop
    temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)
        print("Debug 2")
        optimizer.zero_grad()
        print("Debug 3")
        score = model(X_train)
        print("Debug 4")
        loss = criterion(input=score, target=y_train)
        print("Debug 5")
        print("loss.grad_fn: ",loss.grad_fn)
        loss.requires_grad = True
        print("Debug 6")
        loss.backward()
        
        optimizer.step()

        temp_loss_list.append(loss.detach().cpu().numpy())
    
    temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)

        score = model(X_train)
        loss = criterion(input=score, target=y_train)

        temp_loss_list.append(loss.detach().cpu().numpy())
    
    train_loss_list.append(np.average(temp_loss_list))

    # validation
    model.eval()
    
    temp_loss_list = list()
    for X_val, y_val in val_dataloader:
        X_val = X_val.type(torch.float32).to(device)
        y_val = y_val.type(torch.float32).to(device)

        score = model(X_val)
        loss = criterion(input=score, target=y_val)

        temp_loss_list.append(loss.detach().cpu().numpy())
    
    val_loss_list.append(np.average(temp_loss_list))

    print("\ttrain loss: %.5f" % train_loss_list[-1])
    print("\tval loss: %.5f" % val_loss_list[-1])
</code></pre>
<p>Thanks for your help!</p>

ptrblck: <p>Since you have already a valid <code>grad_fn</code>, you wouldn’t need to use <code>loss.requires_grad = True</code> again.<br>
Do you see any issue, when you’re not calling it?</p>

desmond13: <p>Thanks, it works now.</p>
<p>Now I have the following problems (easy to solve I think). I am trying to understand better the code (below for convenience) and I do not understand what is the difference between the following block:</p>
<pre><code class="lang-auto">    temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)
        optimizer.zero_grad()
        score = model(X_train)
        loss = criterion(input=score, target=y_train)
        loss.backward()
        optimizer.step()
        temp_loss_list.append(loss.detach().cpu().numpy())
</code></pre>
<p>and this one</p>
<pre><code class="lang-auto">  temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)
        score = model(X_train)
        loss = criterion(input=score, target=y_train)
        temp_loss_list.append(loss.detach().cpu().numpy())
</code></pre>
<p>they are both present in the main <code>for loop</code> of my code below and in the second one the following lines are not present:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>It seems that we are doing twice the same thing but I think I am missing something important.</p>
<p>Another problem is that I am trying to plot the fit found from the neural network but I don’t seem to have good results, as you can see in this picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e6ef872950ac741800c39d1e7e8b98d95b53f7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e6ef872950ac741800c39d1e7e8b98d95b53f7f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e6ef872950ac741800c39d1e7e8b98d95b53f7f_2_690x435.png" alt="image" data-base62-sha1="fKWh2TfLhslB0EbX0j3x8EExYiX" width="690" height="435" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e6ef872950ac741800c39d1e7e8b98d95b53f7f_2_690x435.png, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e6ef872950ac741800c39d1e7e8b98d95b53f7f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e6ef872950ac741800c39d1e7e8b98d95b53f7f.png 2x" data-dominant-color="DBE9D8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">749×473 42.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import torch
import numpy as np

from torch import nn, optim
from torch.nn import functional as F
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split

import matplotlib
import matplotlib.pyplot as plt
import numpy as np


# In[2]:


LR = 1e-6
MAX_EPOCH = 10
BATCH_SIZE = 512

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")


# In[3]:



# # Data for plotting
# # t = np.arange(0.0, 2.0, 0.01)
# # s = 1 + np.sin(2 * np.pi * t)

# matplotlib.rcParams['agg.path.chunksize'] = 10000

# fig, ax = plt.subplots()
# ax.plot(X, y)

# # ax.set(xlabel='X', ylabel='y',
# #        title='Test plot')
# # ax.grid()

# # fig.savefig("test.png")
# plt.show()


# In[7]:


class SineApproximator(nn.Module):
    def __init__(self):
        super(SineApproximator, self).__init__()
        self.regressor = nn.Sequential(nn.Linear(1, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1024),
                                       nn.ReLU(inplace=True),
                                       nn.Linear(1024, 1))
    def forward(self, x):
        output = self.regressor(x)
        return output

X = np.random.rand(10**5) * 2 * np.pi
y = np.sin(X)

X_train, X_val, y_train, y_val = map(torch.tensor, train_test_split(X, y, test_size=0.2))
train_dataloader = DataLoader(TensorDataset(X_train.unsqueeze(1), y_train.unsqueeze(1)), batch_size=BATCH_SIZE,
                              pin_memory=True, shuffle=True)
val_dataloader = DataLoader(TensorDataset(X_val.unsqueeze(1), y_val.unsqueeze(1)), batch_size=BATCH_SIZE,
                            pin_memory=True, shuffle=True)

model = SineApproximator().to(device)
optimizer = optim.Adam(model.parameters(), lr=LR)
criterion = nn.MSELoss(reduction="mean")

# training loop
train_loss_list = list()
val_loss_list = list()

my_images = []
fig, ax = plt.subplots(figsize=(12,7))

for epoch in range(MAX_EPOCH):
    print("epoch %d / %d" % (epoch+1, MAX_EPOCH))
    model.train()
    ############
    # TRAINING loop
    temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)
        optimizer.zero_grad()
        score = model(X_train)
        loss = criterion(input=score, target=y_train)
        loss.backward()
        optimizer.step()
        temp_loss_list.append(loss.detach().cpu().numpy())
        
    ############
    # PLOTTING 1
    # plot and show learning process
    print("Plotting 1...")
    plt.cla()
    ax.set_title('Regression Analysis', fontsize=35)
    ax.set_xlabel('Independent variable', fontsize=24)
    ax.set_ylabel('Dependent variable', fontsize=24)
    ax.scatter(X_train.data.numpy(), y_train.data.numpy(), color = "orange")
    ax.plot(X_train.data.numpy(), score.data.numpy(), 'g-', lw=3)
    # Used to return the plot as an image array 
    # (https://ndres.me/post/matplotlib-animated-gifs-easily/)
    fig.canvas.draw()       # draw the canvas, cache the renderer
    image = np.frombuffer(fig.canvas.tostring_rgb(), dtype='uint8')
    image  = image.reshape(fig.canvas.get_width_height()[::-1] + (3,))
    my_images.append(image)
    print("...Plotting 1 done!")
    ############
        
    temp_loss_list = list()
    for X_train, y_train in train_dataloader:
        X_train = X_train.type(torch.float32).to(device)
        y_train = y_train.type(torch.float32).to(device)
        score = model(X_train)
        loss = criterion(input=score, target=y_train)
        temp_loss_list.append(loss.detach().cpu().numpy())
    
    train_loss_list.append(np.average(temp_loss_list))
   
    
    # validation
    model.eval()
    
    temp_loss_list = list()
    for X_val, y_val in val_dataloader:
        X_val = X_val.type(torch.float32).to(device)
        y_val = y_val.type(torch.float32).to(device)

        score = model(X_val)
        loss = criterion(input=score, target=y_val)

        temp_loss_list.append(loss.detach().cpu().numpy())
    
    val_loss_list.append(np.average(temp_loss_list))

    print("\ttrain loss: %.5f" % train_loss_list[-1])
    print("\tval loss: %.5f" % val_loss_list[-1])



# save images as a gif 
print("Plotting...")
# If you are in a jupyter notebook this will not show you a gif but just the last image (the gif is saved in the file specified below)
imageio.mimsave('./curve_1.gif', my_images, fps=10)
</code></pre>
<p>Thanks for the help.</p>

ptrblck: <p>The first code block trains the model, while the second one only appends the loss to <code>temp_loss_list</code>.<br>
I’m not sure why you would need the second code block, since you are also using a validation loop (which is correct).</p>
<p>Regarding the weird plot output: I would suggest to check the shape of the array you would like to plot, as it seems as <code>matplotlib</code> might interpret the input array differently than what you expect.<br>
In particular the shapes of <code>X_train</code> and <code>score</code> would be interesting to see.</p>
<p>PS: You shouldn’t use the <code>.data</code> attribute, as it might yield unwanted side effects. It should be fine in your code snippet, but I would generally advice to skip the <code>.data</code> usage completely. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

desmond13: <p>Thanks, I will look into this.</p>
<p>What do you suggest instead of the <code>.data</code> attribute?</p>

ptrblck: <p>You should use the tensor directly and wrap it into a <code>with torch.no_grad()</code> block, if you don’t want Autograd to track this operation.</p>

desmond13: <p>I changed my code to</p>
<pre><code class="lang-auto">    with torch.no_grad():
        ax.scatter(X_train.detach().numpy(), y_train.detach().numpy(), color = "orange")
        ax.plot(X_train.detach().numpy(), score.detach().numpy(), 'g-', lw=3)
</code></pre>
<p>I hope that this is what you meant.</p>

Neelesh: <p>I find your problem interesting so I wrote my own solution for this, and it’s working fine. Check it out here:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/llNeeleshll/PyTorch_Projects/blob/main/perceptron_pytorch_sine_function.ipynb">
  <header class="source">

      <a href="https://github.com/llNeeleshll/PyTorch_Projects/blob/main/perceptron_pytorch_sine_function.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/llNeeleshll/PyTorch_Projects/blob/main/perceptron_pytorch_sine_function.ipynb" target="_blank" rel="noopener nofollow ugc">llNeeleshll/PyTorch_Projects/blob/main/perceptron_pytorch_sine_function.ipynb</a></h4>


    <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import numpy as np\r\n",
    "import torch\r\n",
    "import torch.nn as nn\r\n",
    "from torch.nn.modules.container import Sequential\r\n",
    "import matplotlib.pyplot as plt\r\n",
    "from sklearn.model_selection import train_test_split\r\n",
    "from torch.autograd import Variable"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
</code></pre>


  This file has been truncated. <a href="https://github.com/llNeeleshll/PyTorch_Projects/blob/main/perceptron_pytorch_sine_function.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>