Applying gradient descent to a function using Pytorch

smu226: <p>Hello! I have 10000 tuples of numbers (x1,x2,y) generated from the equation: <code>y = np.cos(0.583*x1)+np.exp(0.112*x2)</code>. I want to use a NN like approach in pytorch to find the 2 parameters i.e. 0.583 and 0.112 using SGD. Here is my code:</p>
<pre><code class="lang-auto">class NN_test(nn.Module):
    def __init__(self):
        super().__init__()
        self.a = torch.nn.Parameter(torch.tensor(0.7))
        self.b = torch.nn.Parameter(torch.tensor(0.02))

    def forward(self, x):
        y = torch.cos(self.a*x[:,0])+torch.exp(self.b*x[:,1])
        return y

model = NN_test().cuda()

lrs = 1e-4
optimizer = optim.SGD(model.parameters(), lr = lrs)
loss = nn.MSELoss()

epochs = 30
for epoch in range(epochs):
    model.train()
    for i, dtt in enumerate(my_dataloader):
        optimizer.zero_grad()
        
        inp = dtt[0].float().cuda()
        output = dtt[1].float().cuda()

        ls = loss(model(inp),output)

        ls.backward()
        optimizer.step()
    if epoch%1==0:
        print("Epoch: " + str(epoch), "Loss Training: " + str(ls.data.cpu().numpy()))
</code></pre>
<p>where x contains the 2 numbers x1 and x2. In theory it should work easily, but the loss doesn’t go down. What am I doing wrong? Thank you!</p>

KFrank: <p>Hello Silviu</p>
<aside class="quote no-group quote-modified" data-username="smu226" data-post="1" data-topic="64912" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smu226/48/9063_2.png" class="avatar"> smu226:</div>
<blockquote>
<p>I have 10000 tuples of numbers (x1,x2,y) generated from the equation: <code>y = np.cos(0.583*x1)+np.exp(0.112*x2)</code>. I want to use a NN like approach in pytorch to find the 2 parameters i.e. 0.583 and 0.112 using SGD.</p>
<p>…</p>
<p>In theory it should work easily, but the loss doesn’t go down. What am I doing wrong?</p>
</blockquote>
</aside>
<p>I think you are trying to solve a problem that is hard to solve with<br>
gradient descent.</p>
<p>I don’t see any obvious errors in your code.  (I looked at it briefly,<br>
but not in detail.)  So I don’t think that you’re doing anything “wrong.”</p>
<p>Because you add your <code>x1</code> and <code>x2</code> terms together, your problem<br>
decouples into to solving for the two parameters independently.<br>
So let us look at just the <code>cos()</code> piece.</p>
<p>The oscillatory nature of <code>cos()</code> means that your loss function will<br>
likely have several local minima in which the gradient-descent<br>
minimization algorithm can get “stuck.”  Whether this happens will<br>
depend on the range and distribution of the <code>x1</code> you use (which you<br>
didn’t tell us).</p>
<p>To illustrate this I run two examples:  First I sample 10,000 values of<br>
<code>x</code> from a gaussian distribution with a mean of zero and a standard<br>
deviation of 5.0, and then again with a standard deviation of 500.0,<br>
and calculate <code>y = np.cos (0.583 * x)</code>.</p>
<p>I then try to fit <code>0.583</code> with trial values of <code>alpha</code> running from 0.0 to 2.0<br>
by calculating <code>y_alpha = np.cos (alpha * x)</code>, and calculating the<br>
mean-squared-error of the <code>y_alpha</code> with respect to the “ground-truth”<br>
<code>y</code>.</p>
<p>Here is a plot of the mean-squared-error vs <code>alpha</code> for both values<br>
of the standard deviation:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/7522879cedd5967982af65a6dc2c71fe3836328c.png" alt="cosine_mse" data-base62-sha1="gIdQj397zBxzPP0r4s2y0Rt6OEk" width="640" height="480"></p>
<p>As you can see in both cases the loss function takes on a global<br>
minimum of 0.0 for <code>alpha = 0.583</code>.  So far, so good.  But for<br>
stdev = 500, the global minimum is hiding in a tiny range of <code>alpha</code><br>
surrounded by lots of shallow local minima.  Unless you get lucky<br>
and start very close to 0.583 (and for stdev = 500, your starting<br>
value of 0.7 is <em>not</em> close), nothing about your gradient will tell you<br>
that a deep global minimum exists, and your gradient-descent<br>
algorithm won’t converge to the global minimum.</p>
<p>On the other hand (for the ensemble of values of <code>x</code> that I’ve generated<br>
for this example) for stdev = 5, your starting value of 0.7 is well within<br>
the basin of the global minimum, and gradient descent should converge<br>
nicely to the global minimum (for adequately small values of the learning<br>
rate).  However, if you were to start out above about 1.25 (or jump up<br>
above 1.25 because of an overly-large learning rate), you could get<br>
stuck in the local minimum around <code>alpha = 1.5</code>) (or one of the local<br>
minima that occur for still larger values of <code>alpha</code>).</p>
<p>(The batch size you use in your training loop – which you didn’t tell<br>
us –  will also affect how “noisy” your <code>loss.backward()</code> gradients<br>
are and how much your optimizer jumps around, but this doesn’t<br>
change the essence of what is going on.)</p>
<p>Note, <code>exp()</code> doesn’t have the oscillatory behavior that is the root<br>
cause of the local minima.</p>
<p>So try doing a one-dimensional problem where you just have<br>
<code>exp (0.112 * x)</code>, and when you get that working, try a<br>
two-dimensional problem where you replace <code>cos()</code> with <code>exp()</code>,<br>
i.e., <code>y = exp (0.583 * x1) + exp (0.112 * x2)</code>.  That should<br>
be an “easy” minimization problem, so please follow up with further<br>
questions if you can’t get that to work.</p>
<p>Good luck.</p>
<p>K. Frank</p>

smu226: <p>Thank you a lot for your reply! I understand what you mean, however I am still a bit confused. Shouldn’t gradient descent take care of this (my batch size was 512), and make sure I reach the global minima after a few iterations? Mainly, my confusion is this: in a normal NN (say for image recognition) one is able to fit a very complex and highly non linear function with (say) 64x64 inputs and several outputs, millions of parameters, a lot of local minima and without much effort (and less training examples) and get very good parameters. In my case, the function is a lot simpler, already written in the desired mathematical form and having just 2 parameters and yet the parameters are sometimes even double than their desired value. I am not sure I understand why does it perform so badly here, compared to a significantly more complex NN.</p>
<p>I also tried using np.sqrt(0.583<em>x1)+np.exp(0.112</em>x2) instead, so no oscillations, and still not getting convergence, using the ranges:</p>
<pre><code class="lang-auto">x1 = np.random.rand(10000)*10
x2 = np.random.rand(10000)*2-1
</code></pre>

crowsonkb: <p>I implemented this from scratch based on your description and provided partial code samples and it works for me: the loss value decreases steadily over time. Hope it helps:</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/crowsonkb/fc08028bc4907820626bea4f34bbe63b" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/crowsonkb/fc08028bc4907820626bea4f34bbe63b" target="_blank" rel="nofollow noopener">https://gist.github.com/crowsonkb/fc08028bc4907820626bea4f34bbe63b</a></h4>
<h5>ptf.py</h5>
<pre><code class="Python">"""https://discuss.pytorch.org/t/applying-gradient-descent-to-a-function-using-pytorch/64912"""

import torch
from torch import nn, optim
from torch.utils import data


class NNTest(nn.Module):
    def __init__(self):
        super().__init__()</code></pre>
This file has been truncated. <a href="https://gist.github.com/crowsonkb/fc08028bc4907820626bea4f34bbe63b" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


KFrank: <p>Hello Silviu -</p>
<aside class="quote no-group" data-username="smu226" data-post="3" data-topic="64912" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smu226/48/9063_2.png" class="avatar"> smu226:</div>
<blockquote>
<p>Shouldn’t gradient descent take care of this (my batch size was 512), and make sure I reach the global minima after a few iterations?</p>
</blockquote>
</aside>
<p>No.  In general gradient descent will drive you to the nearest local<br>
minimum, after which you will stay there.  (Putting in big jumps by<br>
hand, using large step sizes (large learning rate), or the randomness<br>
from using a batch (instead of averaging the gradient over the whole<br>
training set) can – but won’t necessarily – take you out of a local<br>
minimum.)</p>
<aside class="quote no-group">
<blockquote>
<p>Mainly, my confusion is this: in a normal NN (say for image recognition) one is able to fit a very complex and highly non linear function with (say) 64x64 inputs and several outputs, millions of parameters, a lot of local minima and without much effort (and less training examples) and get very good parameters. In my case, the function is a lot simpler, already written in the desired mathematical form and having just 2 parameters and yet the parameters are sometimes even double than their desired value. I am not sure I understand why does it perform so badly here, compared to a significantly more complex NN.</p>
</blockquote>
</aside>
<p>Let me give you two bits of intuition.  These things might not be true,<br>
but it’s how I think about it.</p>
<p>First, when you train a neural network, you aren’t trying to find the<br>
one, true (with respect to the training set) global minimum.  You’re<br>
looking for one of potentially many similar good / good enough / very<br>
good local minima.  (In fact, in many real-world cases, if you try to<br>
drive your model parameters to “too good” a minimum (global or not)<br>
you encounter <em>over-fitting</em> where your performance on an out-of-sample<br>
test set starts getting worse, even as your performance on you training<br>
set keeps getting better.)</p>
<p>Second, in higher-dimensional parameter spaces there is “more<br>
room” (more directions) to move around, so you can escape from<br>
what would have been a local minimum in a lower-dimensional<br>
parameter space by moving “around” the barrier that would have<br>
kept you in the local minimum by moving in one of the other directions.</p>
<p>(Having made such a bold assertion, I must confess that I find it very<br>
hard to develop intuition about the structure of the loss-function surface<br>
in these very-high-dimensional parameter spaces.)</p>
<aside class="quote no-group">
<blockquote>
<p>I also tried using np.sqrt(0.583<em>x1)+np.exp(0.112</em>x2) instead, so no oscillations, and still not getting convergence, using the ranges:</p>
<pre><code class="lang-auto">x1 = np.random.rand(10000)*10
x2 = np.random.rand(10000)*2-1
</code></pre>
</blockquote>
</aside>
<p>That should work fine.  Take a look at the code that Katherine posted,<br>
and see why hers works and yours doesn’t.</p>
<p>If you can’t get your version working, feel free to post a complete,<br>
runnable script (preferably one-dimensional, just to make things<br>
simpler) that illustrates the problems you are still having.</p>
<p>Good luck.</p>
<p>K. Frank</p>

KFrank: <p>Hi Katherine!</p>
<aside class="quote no-group" data-username="crowsonkb" data-post="4" data-topic="64912" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/crowsonkb/48/11236_2.png" class="avatar"> crowsonkb:</div>
<blockquote>
<p>I implemented this from scratch based on your description and provided partial code samples and it works for me: the loss value decreases steadily over time. Hope it helps:</p>
</blockquote>
</aside>
<p>Thanks for this.  Could you tell us the values you obtained for <code>a</code> and<br>
<code>b</code> after training?</p>
<p>One comment:  From your github code:</p>
<pre data-code-wrap="python"><code class="lang-python">x1 = torch.rand(10000) * 10
...
y = torch.cos(0.583 * x1) + torch.exp(0.112 * x2)
</code></pre>
<p><code>x1</code> will range over <code>[0, 10)</code> and the argument of <code>cos()</code> in <code>x1</code>’s<br>
contribution to <code>y</code> will range over <code>[0, 5.83)</code>.  That is less than one<br>
full period of <code>cos()</code> (<code>[0, 2 * pi = 6.28)</code>), so you won’t see multiple<br>
oscillations, you won’t see a forest of local minima, and your starting<br>
point for <code>a</code> of <code>0.7</code> should be well within the basin of attraction of the<br>
global minimum.</p>
<p>(The gaussian distribution that I used for the plot in my earlier post<br>
and your uniform distribution are not fully comparable, but the standard<br>
deviation of your uniform <code>[0, 10)</code> distribution is 2.89, not too different<br>
from my standard deviation of 5, so the scale of the global minimum’s<br>
basin will be broadly similar in the two cases.)</p>
<p>Best.</p>
<p>K. Frank</p>

crowsonkb: <pre><code class="lang-auto">Epoch: 49, Loss: 0.0027335386257618666
Final a: 0.5868821144104004, b: 0.02589535526931286
</code></pre>

crowsonkb: <aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="64912">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Thanks for this. Could you tell us the values you obtained for <code>a</code> and<br>
<code>b</code> after training?</p>
</blockquote>
</aside>
<p>Even better—if I use the Adam optimizer with its default learning rate of 1e-3, I get full convergence within 30 epochs:</p>
<pre><code class="lang-auto">Epoch: 29, Loss: 4.0959913412408344e-12
Final a: 0.5829995274543762, b: 0.11200009286403656
</code></pre>
<p>It looks like it really benefits from having per-parameter adaptive learning rates.</p>

KFrank: <p>Hi Katherine!</p>
<aside class="quote no-group" data-username="crowsonkb" data-post="9" data-topic="64912" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/crowsonkb/48/11236_2.png" class="avatar"> crowsonkb:</div>
<blockquote>
<p>Even better—if I use the Adam optimizer with its default learning rate of 1e-3, I get full convergence within 30 epochs:</p>
<pre><code class="lang-auto">Epoch: 29, Loss: 4.0959913412408344e-12
Final a: 0.5829995274543762, b: 0.11200009286403656
</code></pre>
</blockquote>
</aside>
<p>Thanks.  Very nice.</p>
<aside class="quote no-group">
<blockquote>
<p>It looks like it really benefits from having per-parameter adaptive learning rates.</p>
</blockquote>
</aside>
<p>Especially in this case where the optimizations for the <code>a</code> and <code>b</code><br>
parameters separate entirely.</p>
<p>Best.</p>
<p>K. Frank</p>