Loss is increasing

username: elbek
cooked: <p>I have this simple nonlinerar function that I want to fit data, but loss function keeps increasing after each iteration. Any idea what I am doing wrong?</p>
<pre><code class="lang-auto">import torch
# w2=2, w1=3

b = 5

x_data = []
y_data = []

for i in range(20):
    i = float(i+1)
    x_data.append(i)
    # y = w2*x^2 + w1*x
    y_data.append(i * i * 2 + i * 3)

w1 = torch.tensor([5.0], requires_grad=True)
w2 = torch.tensor([10.0], requires_grad=True)

# our model forward pass
def forward(x):
    return x * x * w2 + x * w1

# Loss function
def loss(y_pred, y_val):
    return (y_pred - y_val) ** 2

# Before training
print("Prediction (before training)",  4, forward(4).item())

# Training loop
for epoch in range(100):
    for x_val, y_val in zip(x_data, y_data):
        y_pred = forward(x_val) # 1) Forward pass
        l = loss(y_pred, y_val) # 2) Compute loss
        l.backward() # 3) Back propagation to update weights
        # print("\tgrad: ", x_val, y_val, w.grad.item())
        w1.data = w1.data - 0.01 * w1.grad.item()
        w2.data = w2.data - 0.01 * w2.grad.item()

        # Manually zero the gradients after updating weights
        w1.grad.data.zero_()
        w2.grad.data.zero_()

        print(l.item())

# After training
print("Prediction (after training)",  4, forward(4).item())
</code></pre>
<p>result:<br>
625.0<br>
3969.0<br>
4106.24658203125<br>
10951.541015625<br>
494030.0<br>
146011072.0<br>
176458432512.0<br>
690122123116544.0<br>
7.44382236290397e+18<br>
1.967258245916323e+23<br>
1.1615622546788777e+28<br>
1.4223510151775662e+33</p>

username: vainaijr
cooked: <p>I think .data is deprecated, so without it, neural network will work.<br>
so,</p>
<pre><code class="lang-auto">w1.data = w1.data - 0.01 * w1.grad.item()
w1.grad.data.zero_()
</code></pre>
<p>becomes</p>
<pre><code class="lang-auto">w1 = w1 - 0.01 * w1.grad.item()
w1.grad.zero_()
</code></pre>
<p>also, I think we should use in-place, otherwise a new w1 would be created, and its gradient is None, so zeroing it out is an invalid operation, for example,</p>
<pre><code class="lang-auto">w1 = w1 - 0.01 * w1.grad.item()
w1.grad.zero_()
</code></pre>
<p>will give error,</p>
<pre><code class="lang-auto">AttributeError: 'NoneType' object has no attribute 'zero_'
</code></pre>
<p>as we created new w1, for which grad is None.</p>
<p>so, we use in-place,</p>
<pre><code class="lang-auto">w1 -= 0.01 * w1.grad.item()
w1.grad.zero_()
</code></pre>
<p>this way, new w1 will not be created.<br>
also, if we modify w1 like this, then it will give error,</p>
<pre><code class="lang-auto">RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>
<p>so, we use</p>
<pre><code class="lang-auto">with torch.no_grad():
  w1 -= 0.01 * w1.grad*item()
  w1.grad.zero_()
</code></pre>
<p>this will ensure that these computations have required_grad=False, even when for w1 requires_grad is set as True</p>
<p>also, if you use square in your loss, then it gives big values, as listed in your post, so if we use only difference, and use absolute value of it, then it will start working, for example,</p>
<pre><code class="lang-auto"># Loss function
def loss(y_pred, y_val):
    return (y_pred - y_val).abs()
</code></pre>
<p>when I make these changes, and train for 1000 epochs, then I get</p>
<pre><code class="lang-auto">Prediction (after training) 4 43.99794006347656
</code></pre>

username: elbek
cooked: <p>thank you for answering, but I don’t see it converging:</p>
<p>here is the updated code:</p>
<pre><code class="lang-auto"># Loss function
def loss(y_pred, y_val):
    return (y_pred - y_val).abs()

# Training loop
for epoch in range(1000):
    for x_val, y_val in zip(x_data, y_data):
        y_pred = forward(x_val) # 1) Forward pass
        l = loss(y_pred, y_val) # 2) Compute loss
        l.backward() # 3) Back propagation to update weights
        # print("\tgrad: ", x_val, y_val, w.grad.item())
        with torch.no_grad():
            w1 -= 0.01 * w1.grad.item()
            w1.grad.zero_()

            w2 -= 0.01 * w2.grad.item()
            w2.grad.zero_()

        print(l.item())
</code></pre>
<p>my last loss values too high and not converging:</p>
<pre><code class="lang-auto">27.027191162109375
319.6322021484375
58.488525390625
529.6104125976562
127.1507568359375
814.5077514648438
248.1339111328125
1190.4041748046875
Prediction (after training) 4 113.91207122802734
</code></pre>
<p>What am I doing wrong?</p>

username: vainaijr
cooked: <p>I also changed learning rate to 0.0001.</p>

username: elbek
cooked: <p>got it, it is converging now, also I had to use 50K epoch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>thank you!</p>