Finding Polynomial Function Questions

username: Martijn
cooked: <p>I’m trying to create a model that is able to learn any lower polynomial function. (Can this even be done?) So far I’ve got code that is able to learn the y=6x^2 + 2x - 4 quadratic equation.</p>
<p>I noticed that by providing [x, x^2] instead of [x] as the input the model performs much better. However, when I use [x, x^2, x^3] as the input the model doesn’t work anymore (huge test set loss). Why does providing x^3 to the model result in such a poor result?</p>
<p>Below is the code that I’m using. I’m quite new to Pytorch so general tips/critique is also appriciated. Thanks!</p>
<pre><code class="lang-auto">import torch
from torch import Tensor
from torch.nn import Linear, MSELoss, functional as F
from torch.autograd import Variable
import numpy as np

def our_function(x):
    # calculate the y value using the function 6x^2 + 2x - 4
    return 6 * x * x + 2 * x - 4

def data_generator(data_size=1000):
    x = np.random.randint(-1000, 1000, size=data_size)
    y = our_function(x)

    # Adding x^2 enables the model to find quadratic function much better.
    inputs = np.column_stack([x, x ** 2])
    # inputs = np.column_stack([x, x ** 2, x ** 3])
    labels = y.reshape(-1, 1)  # without the reshape we get wrong results

    return inputs, labels


class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = Linear(2, 50)
        # self.fc1 = Linear(3, 50)
        self.fc2 = Linear(50, 50)
        self.fc3 = Linear(50, 1)

        self.criterion = MSELoss()
        self.optimizer = torch.optim.Adam(self.parameters(), lr=0.01)

    def forward(self, x):
        # Shoud i add a dropout?
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def train_model(self, epochs=1000, data_size=1000):
        x_train, y_train = data_generator(data_size)

        for epoch in range(epochs):
            y_pred = model(Variable(Tensor(x_train)))
            y_train = Variable(Tensor(y_train))
            loss = self.criterion(y_pred, y_train)
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()

            print(f"Epoch: {epoch} Loss: {loss / data_size:,.10f}")

model = Net()
model.train_model()
</code></pre>

username: googlebot
cooked: <p>Generally, a neural net fits a piecewise function of x, and there are no x*x terms in a network. Thus, adding x^2 input is beneficial.</p>
<aside class="quote no-group" data-username="Martijn" data-post="1" data-topic="104428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/dbc845/48.png" class="avatar"> Martijn:</div>
<blockquote>
<p>Why does providing x^3 to the model result in such a poor result?</p>
</blockquote>
</aside>
<p>Because cubic polynomial root finding is more difficult that quadratic, indeed it becomes a nonconvex problem, I believe.</p>
<p>If you mean a problem with quadratic target equation, perhaps it is just a numerical issue, as your x^3 column range is -1e9…1e9, while it should have zero effect after training.</p>

username: KFrank
cooked: <p>Hi Martijn!</p>
<aside class="quote no-group" data-username="Martijn" data-post="1" data-topic="104428" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/dbc845/48.png" class="avatar"> Martijn:</div>
<blockquote>
<p>I’m trying to create a model that is able to learn any lower polynomial function. (Can this even be done?)</p>
</blockquote>
</aside>
<p>Yes*!</p>
<p>*)  For some definition of “lower polynomial.”</p>
<aside class="quote no-group">
<blockquote>
<p>So far I’ve got code that is able to learn the y=6x^2 + 2x - 4 quadratic equation.</p>
<p>I noticed that by providing [x, x^2] instead of <span class="chcklst-box checked fa fa-check-square-o fa-fw"></span> as the input the model performs much better.</p>
</blockquote>
</aside>
<p>As Alex noted, by providing <code>x**2</code> as input, you are giving your<br>
model very valuable raw material with which to construct your<br>
quadratic polynomial.</p>
<p>In fact, including <code>x**2</code> lets you train an almost trivial model that<br>
consists of a single <code>Linear</code> layer (and no activation functions):</p>
<pre data-code-wrap="python"><code class="lang-python">class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = Linear(2, 1)

    def forward(self, x):
        x = self.fc1(x)
        return x
</code></pre>
<p>The <code>Linear (2, 1)</code> layer (with bias), is, in fact, exactly the general<br>
quadratic polynomial when fed with <code>[x, x**2]</code> as its input, so this<br>
model will train to essentially machine precision, and the <code>weight</code><br>
and <code>bias</code> of the <code>Linear</code> will train to match the coefficients of your<br>
<code>our_function()</code> quadratic polynomial.</p>
<aside class="quote no-group">
<blockquote>
<p>However, when I use [x, x^2, x^3] as the input the model doesn’t work anymore (huge test set loss). Why does providing x^3 to the model result in such a poor result?</p>
</blockquote>
</aside>
<p>First, adding <code>x**3</code> just confuses the issue, making the training less<br>
nimble.  But more importantly, as Alex noted, <code>x**3</code> is, in your case,<br>
very large.</p>
<p>Given the range of your sample data, <code>x</code> is of order one thousand,<br>
and <code>x**2</code> is of order one million, while <code>x**3</code> is of order one billion.<br>
As training starts, the network has no idea what the three inputs<br>
“mean” and has no sense of their relative scales.  The <code>x**3</code> term<br>
has the largest effect on the output initially.  So the network has to<br>
spend many training iterations learning to ignore <code>x**3</code> (or to process<br>
it into some approximation to a quadratic), which wastes valuable<br>
time, and potentially sends the training off on some wild goose chase<br>
from which it never recovers.</p>
<aside class="quote no-group">
<blockquote>
<p>Below is the code that I’m using. I’m quite new to Pytorch so general tips/critique is also appriciated.</p>
<pre><code class="lang-auto">        self.criterion = MSELoss()
...
            print(f"Epoch: {epoch} Loss: {loss / data_size:,.10f}")
</code></pre>
</blockquote>
</aside>
<p>A minor comment:  <code>MSELoss</code> (using its defaults) takes the mean<br>
of the loss over the batch, so <code>loss / data_size</code> is incorrectly<br>
dividing by the batch size a second time.</p>
<p>In terms of understanding your loss, <code>x</code> is of order one thousand,<br>
so your quadratic (that is, <code>y_train</code>) is of order one million.  If your<br>
errors (that is, <code>y_pred - y_train</code>) were comparable to <code>y_train</code><br>
(that is, your relative errors were of order 1), then your errors and<br>
squared errors would be of order one million and one trillion,<br>
respectively.  So your loss (<code>MSELoss</code> means “mean-squared-error”)<br>
will be of order one trillion (10**12) as training starts.  Just be aware<br>
that even if your training works well and your loss falls <em>a lot,</em> it will<br>
still be numerically large.</p>
<p>You should probably train on <code>MSELoss</code>, but maybe also print out<br>
<code>sqrt (MSELoss)</code> – the so-call root-mean-squared (rms) error.<br>
Even this will be large, starting out on the order of one million.  So<br>
I would suggest also tracking the <em>relative</em> rms error, something like<br>
<code>sqrt (MSELoss / (y_train**2).mean())</code>.  This gives you a sense of<br>
how well your predictions are doing relative to the scale of the values<br>
your are trying to predict.</p>
<aside class="quote no-group" data-username="googlebot" data-post="2" data-topic="104428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ba8739/48.png" class="avatar"> googlebot:</div>
<blockquote>
<p>Generally, a neural net fits a piecewise function of x, and there are no x*x terms in a network.</p>
</blockquote>
</aside>
<p>I’d like to expand on this a little.  It is true that <code>Linear</code> layers introduce<br>
no quadratic terms.  Also the <code>ReLU</code> activation function is piecewise<br>
linear.  But many non-linear activation functions do have quadratic (and<br>
higher-order) terms in their expansions, so they do introduce quadratic<br>
terms into the overall function computed by the network.</p>
<p>(For example, pytorch’s <code>ELU</code> (“exponential-linear unit”) has a regime<br>
where the quadratic term dominates.)</p>
<aside class="quote no-group">
<blockquote>
<p>Thus, adding x^2 input is beneficial.</p>
</blockquote>
</aside>
<p>My quibble about activation functions introducing quadratic terms<br>
notwithstanding, explicitly adding <code>x**2</code> is, of course, beneficial in a<br>
case like yours because the network doesn’t have to reproduce it by<br>
digging it out of non-linearities in the activation functions.</p>
<p>Best.</p>
<p>K. Frank</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="104428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I’d like to expand on this a little. It is true that <code>Linear</code> layers introduce<br>
no quadratic terms. Also the <code>ReLU</code> activation function is piecewise<br>
linear. But many non-linear activation functions do have quadratic (and<br>
higher-order) terms in their expansions, so they do introduce quadratic<br>
terms into the overall function computed by the network.</p>
<p>(For example, pytorch’s <code>ELU</code> (“exponential-linear unit”) has a regime<br>
where the quadratic term dominates.)</p>
</blockquote>
</aside>
<p>Yeah, but the thing is, usual networks won’t generate quadratic and linear “features” at once, so polynomials are approximated with something like additive models.</p>

username: Martijn
cooked: <p>Thanks for the replies <a class="mention" href="/u/kfrank">@KFrank</a> <a class="mention" href="/u/googlebot">@googlebot</a>!</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="104428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>In fact, including <code>x**2</code> lets you train an almost trivial model that<br>
consists of a single <code>Linear</code> layer (and no activation functions):</p>
</blockquote>
</aside>
<p>Indeed a  simple Linear module with one layer does a much better job at finding the polynomial function than my original model. The beauty is in simplicity apparently.</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="104428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>So the network has to spend many training iterations learning to ignore <code>x**3</code> (or to process it into some approximation to a quadratic), which wastes valuable</p>
<p>time, and potentially sends the training off on some wild goose chase<br>
from which it never recovers.</p>
</blockquote>
</aside>
<p>As both of you mentioned this seems to be cause of the problem I ran into in my original post. I’m still a bit surprised the network doesn’t figure out it simply needs to ignore x**3 though.</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="104428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>A minor comment: <code>MSELoss</code> (using its defaults) takes the mean<br>
of the loss over the batch, so <code>loss / data_size</code> is incorrectly<br>
dividing by the batch size a second time.</p>
</blockquote>
</aside>
<p>Oops! Thanks for pointing that out, some leftover code from a previous version.</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="104428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I’d like to expand on this a little. It is true that <code>Linear</code> layers introduce<br>
no quadratic terms. Also the <code>ReLU</code> activation function is piecewise<br>
linear. But many non-linear activation functions do have quadratic (and<br>
higher-order) terms in their expansions, so they do introduce quadratic<br>
terms into the overall function computed by the network.</p>
</blockquote>
</aside>
<p>So in general, if one suspects that the data contains a certain relation/function, would you add said function to the input instead of letting the Neural Network try to find it?</p>

username: KFrank
cooked: <p>Hi Martijn!</p>
<aside class="quote no-group" data-username="Martijn" data-post="5" data-topic="104428" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/dbc845/48.png" class="avatar"> Martijn:</div>
<blockquote>
<p>So in general, if one suspects that the data contains a certain relation/function, would you add said function to the input instead of letting the Neural Network try to find it?</p>
</blockquote>
</aside>
<p>I’d go further:  If I suspected that a dataset contained a certain<br>
relation, I would try to model that relation directly, and not use<br>
a neural network.</p>
<p>Things like performing linear regressions and fitting polynomials<br>
with neural networks are nice toy problems to explore how neural<br>
networks and gradient-descent optimization work, but if I were<br>
doing it for real, I would avoid the complication of a neural network,<br>
and just perform the regression or fit the polynomial.</p>
<p>Best.</p>
<p>K. Frank</p>