Increasing performance predicting curve chunk

fvadell: <p>Hi there !<br>
Im still pretty new to neural nets and pytorch and actually this is my first post here. I wanted to create a simple neural net that could train on a chunk of a parametric curve and then predict a missing part (see image below).</p>
<p>I created my Dataset object in the following way:</p>
<pre><code class="lang-auto">X = torch.arange(-np.pi, np.pi, .001)
Y = torch.stack((torch.cos(X+1)-2, torch.sin(X+1)+2))
</code></pre>
<pre><code class="lang-auto">class DS(Dataset):
    def __init__(self, x, y):
        self.x = x
        self.y = y
        
    def __getitem__(self, index):
        return self.x[index], self.y[index]
    
    def __len__(self):
        return len(self.x)
</code></pre>
<pre><code class="lang-auto">test_offset = 2120
test_size = 1500

test = [X[test_offset:test_offset + test_size], Y.t()[test_offset:test_offset + test_size]]
train = [torch.cat((X[0:test_offset], X[test_offset + test_size:])), torch.cat((Y.t()[0:test_offset], Y.t()[test_offset + test_size:]))]

test_ds = DS(X[test_offset:test_offset + test_size], Y.t()[test_offset:test_offset + test_size])
train_ds = DS(torch.cat((X[0:test_offset], X[test_offset + test_size:])), torch.cat((Y.t()[0:test_offset], Y.t()[test_offset + test_size:])))
</code></pre>
<p>And a straighforward nn as:</p>
<pre><code class="lang-auto">class fede_net(nn.Module):
    def __init__(self, leaky_slope = 0.1):
        super(fede_net, self).__init__()
        self.l1 = nn.Linear(1,100)
        self.l2 = nn.Linear(100,100)
        self.l3 = nn.Linear(100,100)
        self.l4 = nn.Linear(100,50)
        self.l5 = nn.Linear(50,2)
        self.slope = leaky_slope
        
    def forward(self, x):
        x = self.l1(x)
        x = F.leaky_relu(x , negative_slope= self.slope)
        x = self.l2(x)
        x = F.leaky_relu(x , negative_slope=self.slope)
        x = self.l3(x)
        x = F.leaky_relu(x , negative_slope=self.slope)
        x = self.l4(x)
        x = F.leaky_relu(x , negative_slope=self.slope)
        x = self.l5(x)
        return x
</code></pre>
<p>Finally i train a net and print the predictions</p>
<pre><code class="lang-auto">second_lr = False
epochs = 20
for i in range(epochs):
    for x_sample, y_sample in train_dl:
        pred = net(x_sample.reshape(-1,1))
        loss = loss_fn(pred, y_sample)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        
    if (second_lr==False)&amp;(loss &lt; 1e-5).item():
        second_lr=True
        print("Changing learning rate at epoch %d" %i)
        optimizer = optim.Adam(net.parameters(), lr = 0.0001)

    if i % 4 == 0:
        print(str(i) + "/" + str(epochs), end = "\r")
        print(loss)
        pred_tot = net(test[0].reshape(-1,1))
        plt.plot(train[1][:,0],train[1][:,1])
        plt.plot(test[1][:,0],test[1][:,1], linestyle = '--')
        plt.plot(pred_tot.detach()[:,0],pred_tot.detach()[:,1], alpha = i/epochs)
    
    print(str(i) + "/" + str(epochs), end = "\r")
plt.show()
</code></pre>
<p>The network actually works and predicts the missing part of the curve as you can see in the image<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f704354a769a7e624eaadd0e19713a834a683d84.png" alt="image" data-base62-sha1="zfcWYfCflfoctZhQXJk6kiRibEE" width="283" height="186"></p>
<p>The thing is i tried changing the size of the net both vertically and horizontally but could not get much better results. Actually i think that the size im using right now should be an overkill for this problem but no. I also tried changing the activation function to a tanh but didnt see any improvement either.</p>
<p>Any ideas on what should i try? Any feedback would be great !<br>
Thanks in advance !! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

googlebot: <p>how do you expect for network to draw an arc absent from the training set, without knowledge about geometry?</p>
<p>what you’re seeing is like nearest neighbour inference (from 1d inputs!), the nature of that curvature is different</p>

fvadell: <p>Is there any way i could try in order to improve this?</p>

googlebot: <p>sinusoid activation extrapolates perfectly:</p>
<pre><code class="lang-python">class fede_net(nn.Module):
	def __init__(self, leaky_slope = 0.1):
		super(fede_net, self).__init__()
	
		self.h1 = nn.Linear(1,2)
		self.h2 = nn.Linear(2,2)
		
	def forward(self, x):
		x = self.h1(x)
		x = torch.sin(x)
		x = self.h2(x)
		return x

train_dl = DataLoader(train_ds, 64)
optimizer = SGD(net.parameters(), 0.1)
scheduler = ExponentialLR(optimizer, 0.9)
</code></pre>
<p>but this is just an extrapolation exploiting geometry knowledge, you can’t build a [memorizing] approximating function for a region without training points</p>

fvadell: <p>This actually works perfectly. Now i get that its actually fitting 2d-sinusoid function both to the x and y coordinates and since these two are actually sinusoid functions it fits just right.<br>
Thanks for the answer !</p>