Is the SGD in Pytorch a real SGD?

username: Wmog
cooked: <p>After looking up the code of Pytorch’s SGD:<br>
<a href="https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63" rel="nofollow noopener">Pytorch’s SGD</a><br>
it seems (excuse me in advance if my assertion is wrong) that the SGD is not a real SGD. Indeed, the way gradients are accumulated and more especially the order wherein they are accumulated is up to the user. Thereupon where is the <em>randomness</em> or <em>stochastic</em> character of this gradient descent?</p>
<p>According to Wikipedia:<br>
<a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="nofollow noopener">SGD</a><br>
the examples must be randomly shuffled in the training set after being used to compute the gradient. However here this stage is carried out by the user when he accumulated the gradient with the <code>backward</code>.</p>

username: BartolomeD
cooked: <p>I think the stochastic character of SGD is that it approximates the true loss surface of the <em>entire</em> dataset by optimizing parameters based on <em>samples</em> of data (i.e. batches).</p>

username: Wmog
cooked: <p>Yes but this approximation and the making of samples (and so the stochastic character) are up to the user, are not they? If the answer is yes then its more a “GD” than a “SGD” no?</p>

username: BartolomeD
cooked: <p>In the end it is up to the user, I agree. If one would pass all data through the model and updating the weights with SGD afterwards, it would be normal Gradient Descent, I would say.</p>

username: Wmog
cooked: <p>Thank you for your answer. Ok so if i understood your answer it is more a user case-specific Gradient Descent than an SGD. I think that the name of this class is then somewhat confusing.</p>

username: albanD
cooked: <p>Hi,<br>
The thing is that by this argument, all optimizer names are confusing.<br>
The optimizers provided in pytorch actually implement the update steps corresponding to the algorithm that is used for their name.</p>

username: Wmog
cooked: <p>Ok perfect, that was exactly what I thought. Actually, they should be named “Stepper”. For example with SGD that will be “SGDStepper”. That seems more clear.</p>

username: K_Frank
cooked: <p>Hello Wmog (and others) -</p>
<p>I had the exact same question / confusion, so I’m glad I found this<br>
thread.  (Sorry to revive an oldish thread …)</p>
<aside class="quote no-group" data-username="Wmog" data-post="7" data-topic="9714" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/7bcc69/48.png" class="avatar"> Wmog:</div>
<blockquote>
<p>Ok perfect, that was exactly what I thought. Actually, they should be named “Stepper”. For example with SGD that will be “SGDStepper”. That seems more clear.</p>
</blockquote>
</aside>
<p>I agree with your analysis.  torch.optim.SGD doesn’t have any<br>
randomness (stochasticity) in it, so it does seem misleading to<br>
call it stochastic gradient descent.  Calling it gradient descent<br>
(or GD) would be more accurate.  Or, as you note, perhaps calling<br>
it “gradient descent optimization stepper” would emphasize that<br>
it only performs one step at a time.  (Maybe folks thought that<br>
leaving “stochastic” out of the name would confuse people who<br>
just wanted to use the “standard” SGD optimization algorithm.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: ghlai9665
cooked: <p>Yeah - newcomer to PyTorch here and I find the SGD name really confusing too. I understand SGD as gradient descent with a batch size of 1, but in reality the batch size is determined by the user. So I agree that it would be much less confusing if it was named just GD because that’s what it is.</p>

username: Aditya_Ranganath
cooked: <p>Shouldn’t it just be called a “batch” gradient descent ?<br>
(Apologies if I am wrong) If we compute the gradient using<br>
$\theta_{k+1} = \theta_{k} + \eta \sum_{i=0}^m lossfn(output, prediction)$ (assuming no momentum)<br>
where $m \in (0,N)$, are we not explicitly calculating the entire (averaged) gradient for that batch ?</p>

username: artificial-cerebrum
cooked: <p>In the <a href="https://www.cs.toronto.edu/~hinton/absps/imagenet.pdf" rel="noopener nofollow ugc">AlexNet paper</a> they say:</p>
<blockquote>
<p>We trained our models using <strong>stochastic gradient descent</strong> with a <strong>batch size of 128</strong> examples.</p>
</blockquote>

username: Chao_Kong
cooked: <p>I had exactly the same question too. And I suppose it should be called <code>GD</code> but neither <code>SGD</code> nor <code>BGD</code>(batch gradient descent).</p>
<p>What the <code>nn.optim.SGD</code> optimizer really does is to perform <strong>a step of gradient descent algorithm</strong> once <code>optimzer.step()</code> is called.</p>
<ul>
<li>The <em>stochastic</em> or <em>random</em> part is done in the <code>DataLoader</code>. We’ve done <code>shuffle</code> in the <code>DataLoader</code> and then the GD algorithm is an <code>SGD</code> algorithm.</li>
<li>The <em>batch</em> part is done both in the <code>DataLoader</code> and the loss function. We’ve assigned a <code>batch_size</code> to the <code>DataLoader</code> and well tell the loss function, take L1_loss as an example, that we want a <em>mean loss</em> or a <em>sum loss</em> for the whole batch.</li>
</ul>
<p>So, the SGD optimizer has nothing to do with neither the random part nor the batch part. It is only just a GD algorithm or a GD stepper.</p>