Why is my weight in the linear layer not changing?

Kaiwen_Xie: <pre><code class="lang-auto">import torch
from torch import nn
import matplotlib.pyplot as plt

weight = 0.3
bias = 0.9

X = torch.arange(0, 200, 1.0).unsqueeze(dim=1)
y = X * weight + 0.9

X.shape, y.shape

train_split = int(0.8 * len(X))
X_train = X[:train_split]
y_train = y[:train_split]
X_test = X[train_split:]
y_test = y[train_split:]

len(X_train), len(y_train),len(X_test),len(y_test)

device = "cuda" if torch.cuda.is_available() else "cpu"
device

# X_train_cpu = X_train
# y_train_cpu = y_train
# X_test_cpu = X_test
# y_test_cpu = y_test

X_train = X_train.to(device)
y_train = y_train.to(device)
X_test = X_test.to(device)
y_test = y_test.to(device)

X_train[:10], y_train[:10], X_test[:10], y_test[:10]

class LinearRegressionModel(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear_layer = nn.Linear(in_features = 1, out_features = 1)

  def forward(self, x:torch.Tensor) -&gt; torch.Tensor :
    return self.linear_layer(x)

def plot_prediction(train_data = X_train.cpu(),
                    train_labels = y_train.cpu(),
                    test_data = X_test.cpu(),
                    test_labels = y_test.cpu(),
                    predictions = None):

  """
  plots training data, test data and comapres predictions
  """

  plt.figure(figsize=(10, 7))
  plt.scatter(train_data, train_labels, c="b", s = 4, label = "Training data")
  plt.scatter(test_data, test_labels, c="g", s = 4, label = "Test data")

  if predictions is not None:
    plt.scatter(test_data, predictions, c="r", s = 4, label = "predictions")

  plt.legend(prop={"size" : 14})

plot_prediction()

torch.manual_seed(42)

linear_model = LinearRegressionModel()
linear_model.to(device)
next(linear_model.parameters()).device

linear_model.state_dict()

list(linear_model.parameters())

loss_fn = nn.L1Loss()
optimizer = torch.optim.SGD(params = linear_model.parameters(), lr = 0.01)
optimizer

epochs = 300

for epoch in range(epochs):
  linear_model.train()

  y_preds = linear_model(X_train)

  loss = loss_fn(y_preds, y_train)

  optimizer.zero_grad()

  loss.backward()

  optimizer.step()

  linear_model.eval()

  with torch.inference_mode():
    test_preds = linear_model(X_test)
    test_loss = loss_fn(test_preds, y_test)

  if epoch % 20 == 0:
    print(f"Epoch: {epoch}; Loss: {loss}; Test loss: {test_loss}")

linear_model.state_dict()

with torch.inference_mode():
  y_preds = linear_model(X_test)
y_preds

# y_preds_cpu = y_preds.cpu()
# y_preds_cpu.get_device()

plot_prediction(predictions = y_preds.cpu())
</code></pre>
<p>I am new to PyTorch and i tried to build my first training loop, but could anyone tell me why is it that no matter how much i change my learning rate value in the optimizer. My bias changes (by checking the state_dict) but the weight will always remain the same.<br>
Another peculiar thing is that, the test_loss will also decrease very slowly across the epochs, and the learning rate value will sort of affect the value of the test_loss (and not in the diverging way). Is there some kind of bug here i missed out on?</p>

KFrank: <p>Hi Kaiwen!</p>
<aside class="quote no-group quote-modified" data-username="Kaiwen_Xie" data-post="1" data-topic="198913" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaiwen_xie/48/61509_2.png" class="avatar"> Kaiwen_Xie:</div>
<blockquote>
<pre><code class="lang-auto">class LinearRegressionModel(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear_layer = nn.Linear(in_features = 1, out_features = 1)
...
loss_fn = nn.L1Loss()
optimizer = torch.optim.SGD(params = linear_model.parameters(), lr = 0.01)
</code></pre>
<p>the weight will always remain the same.</p>
</blockquote>
</aside>
<p>Your weight does not remain the same – it jumps back an forth between two<br>
values that don’t change.</p>
<p>Because you are using <code>L1Loss</code> (together with the fact that your model is<br>
very simple), your optimum – the parameter value for which your loss is<br>
at its minimum – is at the bottom of  V-shaped loss function.  Then because<br>
you are using a rather large learning rate – which is the step size used by<br>
the optimizer – you simply jump back and forth across this V, so after <em>two</em><br>
steps, you’ve come back to where you were before.  (This is for the <code>weight</code>;<br>
the “scale” of the V that the <code>bias</code> sees, relative to the learning rate, is<br>
different, so <code>bias</code> does not start out jumping back and forth across the V<br>
and the <code>bias</code> value does make progress as you train.)</p>
<p>First try lowering your learning rate to <code>lr = 0.0001</code>.  Then try changing your<br>
loss function to <code>loss_fn = nn.MSELoss()</code> – this changes the shape of the<br>
loss-function well from a V to a parabola.  Finally try adding momentum to<br>
your <code>SGD</code> optimizer, say, <code>momentum = 0.95</code>.</p>
<p>Play around with the learning rate and momentum with both loss functions<br>
to get a feel for how these things can behave.  You might also experiment<br>
with the <code>Adam</code> optimizer.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Kaiwen_Xie: <p>ohhh i see, thanks for the clarification! I always thought that 0.01 learning rate is actually quite low already. But may I ask, what is the benefit of using a parabola shaped loss function to a V shape loss function? (in my intuition, im guessing its because the gradient on the parabola drops as it gets closer to the minimum so the optimizer takes smaller steps as it approaches the minimum)</p>
<p>And also, what does momentum usually do and how do we usually set its value? This is quite a new term for me</p>

KFrank: <p>Hi Kaiwen!</p>
<aside class="quote no-group" data-username="Kaiwen_Xie" data-post="3" data-topic="198913" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaiwen_xie/48/61509_2.png" class="avatar"> Kaiwen_Xie:</div>
<blockquote>
<p>in my intuition, im guessing its because the gradient on the parabola drops as it gets closer to the minimum so the optimizer takes smaller steps as it approaches the minimum</p>
</blockquote>
</aside>
<p>I’d say that this is correct (although intuition can be a tricky business with<br>
neural networks …).</p>
<p>Note, however, that this can help or hurt, depending on the details.</p>
<p>For the parabola (<code>MSELoss</code>) (in comparison with the V (<code>L1Loss</code>) and for<br>
a given learning rate):</p>
<p>If you’re quite far away from the minimum, the gradient can be too large<br>
and training will diverge.  If you’re far from the minimum, but not too far,<br>
the gradient will be large, so you take big steps toward the minimum and<br>
train faster.  As you get close to the minimum, the gradient gets smaller,<br>
you take smaller steps, and training slows down, perhaps unhelpfully.<br>
But as you get even closer to the minimum to where training for the V<br>
starts jumping back and forth across the V, stopping progress, even though<br>
the parabola training slows down because of small gradients, it does keep<br>
making progress, even if slowly.</p>
<aside class="quote no-group">
<blockquote>
<p>what does momentum usually do</p>
</blockquote>
</aside>
<p>Momentum keeps a running average of previously computed gradients and<br>
takes a step opposite the direction (that is, downhill) of that average gradient,<br>
rather than just using the current gradient.  This can help smooth out things<br>
like jumping back and forth across a valley with steep sides and help you<br>
follow the floor of the valley more gently downhill.  Also, it will help to average<br>
away noise, for example batch-to-batch noise caused by a small batch size.</p>
<aside class="quote no-group">
<blockquote>
<p>how do we usually set its value?</p>
</blockquote>
</aside>
<p>As with most everything with neural networks, you would set your momentum<br>
value empirically.  (That’s just a fancy way of saying try out a bunch of values<br>
and use the one that works best for your problem.)</p>
<p>Best.</p>
<p>K. Frank</p>

Kaiwen_Xie: <p>ah thanks for the explanation. Appreciate it!</p>