How does PyTorchs SGD work?

setsailforfail: <p>I have recently learned, that SGD does normally not converge, if the learning rate is fixed.<br>
I wondered, how PyTorch handles the learning rate then when using</p>
<p><code>optimizer = optim.SGD()</code></p>
<p>as the optimizer, since it requires no parameters that seem to affect the learning rate. I wondered if momentum is actually enough to guarantee convergence? Or does the learning rate gets adjusted behind the scenes?</p>

KFrank: <p>Hi Florian!</p>
<aside class="quote no-group" data-username="setsailforfail" data-post="1" data-topic="171504" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/setsailforfail/48/30110_2.png" class="avatar"> setsailforfail:</div>
<blockquote>
<p>I have recently learned, that SGD does normally not converge, if the learning rate is fixed.</p>
</blockquote>
</aside>
<p><code>SGD</code> converges just fine with a fixed learning rate.  Here is a simple example<br>
script with <code>SGD</code> (with a fixed learning rate and no momentum):</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

x = torch.ones (1, requires_grad = True)
t = torch.tensor ([16.0])
opt = torch.optim.SGD ([x], lr = 0.01)
loss_fn = torch.nn.MSELoss()

for  i in range (20):
    opt.zero_grad()
    p = x**2   # "model" is x**2
    loss = loss_fn (p, t)
    loss.backward()
    opt.step()
    print (x, loss)

</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.13.0
tensor([1.6000], requires_grad=True) tensor(225., grad_fn=&lt;MseLossBackward0&gt;)
tensor([2.4602], requires_grad=True) tensor(180.6336, grad_fn=&lt;MseLossBackward0&gt;)
tensor([3.4391], requires_grad=True) tensor(98.9550, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0131], requires_grad=True) tensor(17.4123, grad_fn=&lt;MseLossBackward0&gt;)
tensor([3.9963], requires_grad=True) tensor(0.0110, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0010], requires_grad=True) tensor(0.0009, grad_fn=&lt;MseLossBackward0&gt;)
tensor([3.9997], requires_grad=True) tensor(6.9633e-05, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0001], requires_grad=True) tensor(5.4771e-06, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0000], requires_grad=True) tensor(4.3050e-07, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0000], requires_grad=True) tensor(3.3528e-08, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0000], requires_grad=True) tensor(2.4593e-09, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0000], requires_grad=True) tensor(1.7826e-10, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.0000], requires_grad=True) tensor(1.4552e-11, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.], requires_grad=True) tensor(3.6380e-12, grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.], requires_grad=True) tensor(0., grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.], requires_grad=True) tensor(0., grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.], requires_grad=True) tensor(0., grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.], requires_grad=True) tensor(0., grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.], requires_grad=True) tensor(0., grad_fn=&lt;MseLossBackward0&gt;)
tensor([4.], requires_grad=True) tensor(0., grad_fn=&lt;MseLossBackward0&gt;)
</code></pre>
<p>It is true that complex models can converge (very) slowly and that training<br>
can sometimes be improved by varying the learning rate (e.g., “cycling” it)<br>
or by using momentum or fancier optimizers.</p>
<p>It is also possible (but, in my experience, unlikely) for training of a model<br>
to converge to a <em>true local minimum</em>, rather than to the “correct” solution<br>
of a lower local minimum or the actual global minimum.  Increasing the<br>
learning rate <em>a lot</em> so that a <em>large</em> optimization step jumps out of the local<br>
minimum can get training to progress further (but that’s kind of like whacking<br>
your starter motor with a sledgehammer to get your car started …).</p>
<p>Best.</p>
<p>K. Frank</p>

setsailforfail: <p>Hi Frank,</p>
<p>Thanks for your extensive response and explanation. However, your example is not really convincing to me. How does it differ from gradient descent? I am not understanding where the stochastic part comes into play, since you are not really dealing with different samples SGD could choose from?</p>
<p>Correct me if I am wrong though!</p>

tataganesh: <p>The “stochastic” part in SGD comes from computing the gradient for mini-batches of the dataset, since Gradient Descent involves calculating the gradient for the full dataset.<br>
This forum post might be helpful.</p><aside class="quote quote-modified" data-post="1" data-topic="9714">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/7bcc69/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-the-sgd-in-pytorch-a-real-sgd/9714">Is the SGD in Pytorch a real SGD?</a> 
  </div>
  <blockquote>
    After looking up the code of Pytorch’s SGD: 
<a href="https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63" rel="noopener nofollow ugc">Pytorch’s SGD</a> 
it seems (excuse me in advance if my assertion is wrong) that the SGD is not a real SGD. Indeed, the way gradients are accumulated and more especially the order wherein they are accumulated is up to the user. Thereupon where is the randomness or stochastic character of this gradient descent? 
According to Wikipedia: 
<a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="noopener nofollow ugc">SGD</a> 
the examples must be randomly shuffled in the training set after being used to compute the gradient. However here th…
  </blockquote>
</aside>