Failing to learn a quadratic function with pytorch

hgfedcba: <p>Hi<br>
I am new to pytorch and when I tried to use pytorch in my project I noticed that somehow it always predicts straight lines. I tried to isolate the problem and I completly failed to approximate a normal quadratic function with it. I very confused where I go wrong…</p>
<pre><code class="lang-auto">    import torch
    from torch import nn
    from torch.autograd import Variable
    import numpy as np
    import pandas as pd
    import matplotlib.pyplot as plt
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import accuracy_score
    import torch.optim as optim


    def f_x(x):
        return x * x / 64 - 5 * x / 4 + 25


    # Building dataset
    def build_dataset():
        # Given f(x), is_f_x defines whether the function is satisfied
        x_values = np.ones((21, 1))
        for i in range(0, 21):
            x_values[i] = i + 30  # True
        return x_values


    x_values = build_dataset()

    # Building nn
    # net = nn.Sequential(nn.Linear(1, 100), nn.ReLU(), nn.Linear(100, 100), nn.ReLU(), nn.Linear(100, 1))
    net = nn.Sequential(nn.Linear(1, 1000), nn.ReLU(), nn.Linear(1000, 1000), nn.ReLU(), nn.Linear(1000, 1000), nn.ReLU(), nn.Linear(1000, 1))

    # parameters
    optimizer = optim.Adam(net.parameters(), lr=0.00001)
    epochs = 200


    def out(k):
        # folder_name = "Testrun1"
        # working_directory = pathlib.Path().absolute()
        # output_location = working_directory / f'{folder_name}'

        a = 30
        b = 50

        # TODO: copy graph so i only use a copy when it was still open

        import matplotlib.backends.backend_pdf as pdfp
        from pylab import plot, show, grid, xlabel, ylabel
        import matplotlib.pyplot as plt
        # pdf = pdfp.PdfPages("graph" + str(k) + ".pdf")

        t = np.linspace(a, b, 20)
        x = np.zeros(t.shape[0])
        c_fig = plt.figure()

        for j in range(len(t)):
            h = torch.tensor(np.ones(1) * t[j], dtype=torch.float32)
            x[j] = net(h)
        plt.ylim([0, 1])
        plot(t, x, linewidth=4)
        xlabel('x', fontsize=16)
        ylabel('net(x)', fontsize=16)
        grid(True)
        show()
        # pdf.savefig(c_fig)

        # pdf.close()
        plt.close(c_fig)


    def train():
        net.train()
        losses = []
        for epoch in range(1, epochs):
            x_train = Variable(torch.from_numpy(x_values)).float()
            y_train = f_x(x_train)
            y_pred = net(x_train)
            loss = torch.sum(torch.abs(y_pred - y_train))
            print("epoch #", epoch)
            print(loss.item())
            losses.append(loss.item())
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        return losses


    print("training start....")
    losses = train()
    plt.plot(range(1, epochs), losses)
    plt.xlabel("epoch")
    plt.ylabel("loss train")
    plt.ylim([0, 100])
    plt.show()

    out(epochs)
</code></pre>

googlebot: <p>RELU is not sophisticated enough here - it is difficult to find good thresholds from univariate input, so optimization rather finds inferior local minimum using linear region of RELU activations.<br>
Also note that adding x^2 to network inputs explicitly considerably simplifies the task.</p>

hgfedcba: <p>Hi<br>
You were definitly correct that RELU was not an optimal choice. I used this code</p>
<p>activation_function = nn.SELU()<br>
net = nn.Sequential(nn.Linear(1, 1000), activation_function, nn.Linear(1000, 1000), activation_function, nn.Linear(1000, 1000), activation_function, nn.Linear(1000, 1))</p>
<p>and plugged in all 20 non-linear activation functions from here <a href="https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity</a></p>
<p>there were multiple where it was possible to see a curve but out of the 20 exactly one reached an somewhatdecent precision and that was SELU. However even the best activation function retained a loss of &gt;= 1 regardless of the number of iterations, which I consider unacceptable given the task to approximate a quadratic function with 2 hidden layer of 1000 neurons.</p>
<p>Do you have another idea that might help? I tried different optimizers as well, but Adam + Rprop are the two that work best, so I can’t make an improvement there…</p>
<p><span class="mention">@your</span> note: You mean I should have 2 input values, one is x and the other is x*x? I don’t get why this is better but I do believe you.</p>

googlebot: <p>That function is just not easy to learn, starting from noise. It also doesn’t help that inputs and outputs (target bias=25) are far from zero.</p>
<ol>
<li>amount of training may be too low (lr_rate and number of epochs). lr_scheduler may also be needed to fine-tune curves</li>
<li>you’re not using quadratic loss (MSE). this means that y_pred derivative doesn’t depend on distance from target, just on its sign</li>
<li>excessive network size just complicates training</li>
</ol>
<p>I actually ran you code, with:</p>
<pre><code class="lang-auto">net = nn.Sequential(nn.Linear(1, 50), nn.Tanh(), nn.Linear(50,50), nn.Tanh(), nn.Linear(50, 1))
optimizer = optim.Adam(net.parameters(), lr=0.01)
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, 0.99)
epochs = 1000
loss = ((y_pred-y_train)**2).sum()
</code></pre>
<p>gives kinda reasonable curve. But with your big network (3layers x 1000, tanh), I decreased lr to 0.0001 (0.01 gave linear solutions) and 1000 epochs was not enough.</p>
<p>So, in general, it is about tuning hyperparameters, all smooth activation functions should in principle work.</p>

hgfedcba: <p>I am very happy with this solution.</p>
<p>As far as I can tell the most important problems were wrong activation/loss functions. After these changes many nets seem to work, but your small net does the trick fast and very accurate. I am also very happy you showed an example of the lr_scheduler. I saw them a few days ago but forgot about them again.</p>
<p>However I do not understand your point about target bias. What exactly is target bias and how do you come up with the exact number of 25?</p>
<p>Edit: I just noticed very strange behaviour. I ran my code with all your chnges a few times and it always looks like the loss reaches a plateau at around 1 and stays there for a long time, then suddenly it gets much better. The length of this plateau varies between 200 and 800 epochs in my tests. Do you have an idea why this is? Especially when it stayed really constant at loss 1 for 800 epochs I surely would have stopped the net if it wasn’t so fast.</p>

googlebot: <aside class="quote no-group" data-username="hgfedcba" data-post="5" data-topic="89437">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hgfedcba/48/30372_2.png" class="avatar"> hgfedcba:</div>
<blockquote>
<p>However I do not understand your point about target bias. What exactly is target bias and how do you come up with the exact number of 25?</p>
</blockquote>
</aside>
<p>Your final linear layer does computation of form dot(h,w)+b, with w and b initialized to zero-centered noise, it is very far from true function f_x(x)=P(x)+25, so this offset may be modeled through h coming from earlier layers. But h is obtained via non-linear transform, so approximations suffer somewhat.</p>
<aside class="quote no-group" data-username="hgfedcba" data-post="5" data-topic="89437">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hgfedcba/48/30372_2.png" class="avatar"> hgfedcba:</div>
<blockquote>
<p>I ran my code with all your chnges a few times and it always looks like the loss reaches a plateau at around 1 and stays there for a long time, then suddenly it gets much better. The length of this plateau varies between 200 and 800 epochs in my tests. Do you have an idea why this is? Especially when it stayed really constant at loss 1 for 800 epochs I surely would have stopped the net if it wasn’t so fast.</p>
</blockquote>
</aside>
<p>Not sure, it seems that curve fitting starts late, network has to shift things to a good region first. lr scheduling may also play a role.<br>
Note that tanh is actually problematic, it has tiny gradients outside -5…5, and you’re using non-normalized inputs. So another possible reason is that some “neurons” are effectively disabled for a long time.</p>

winckelman: <p>All the suggestions proposed here can be effective. However, they do not address what I believe to be the root problem. <strong>You’re inputs aren’t normalized.</strong></p>
<p>The default initialization of network weights in pytorch expects data to be normalized, meaning roughly between -1 and 1. However, your inputs are between 30 and 50. If you just shift your x data by subtracting 40 (do not subtract 40 from your y data) in a preprocessing step, it trains fine when given enough iterations (you may need more than 200).</p>
<p>Ideally, outputs should also be normalized. However, I tried subtracting 40 from the data and that is sufficient.</p>
<p>It is common practice to pre-process data by subtracting the mean. Dividing by the standard deviation is, also, common.</p>
<pre><code class="lang-auto">import torch
from torch import nn
import matplotlib.pyplot as plt
import torch.optim as optim
from tqdm import trange

#
# ~~~ Make up data
torch.manual_seed(2025)
f_x = lambda x: x**2/64 - 5*x/4 + 25
x_train = torch.linspace(30,50,21).reshape(-1,1)
y_train = f_x(x_train)
pre_processed_x_train = x_train-40   # pre-process by subtracting the mean

#
# ~~~ Set up model
net = nn.Sequential(nn.Linear(1, 1000), nn.ReLU(), nn.Linear(1000, 1000), nn.ReLU(), nn.Linear(1000, 1000), nn.ReLU(), nn.Linear(1000, 1))

#
# ~~~ Train
optimizer = optim.Adam(net.parameters(), lr=0.005 )
epochs = 400
for _ in trange(epochs):
    y_pred = net(pre_processed_x_train)
    loss = torch.sum(torch.abs(y_pred - y_train))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

#
# ~~~ Plot the results
plt.scatter( x_train, y_train, color="green", label="Training Data" )
plt.plot( x_train, net(pre_processed_x_train).detach(), label="Fitted Model" )
plt.show()
</code></pre>

winckelman: <aside class="quote no-group" data-username="winckelman" data-post="7" data-topic="89437">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/ec9cab/48.png" class="avatar"> winckelman:</div>
<blockquote>
<p><code>pre_processed_x_train = x_train-40   </code></p>
</blockquote>
</aside>
<p>To be clear, if you comment out the <code>-40</code>, i.e., replace this line with <code>pre_processed_x_train = x_train#-40</code>, then the model fails to train even when given more iterations.</p>