How does pytorch SGD implementation performs sample selection for optimization?

nile649: <p>Hi, I am not able to find or understand how does pytorch implementation of SGD is able to randomly select samples for optimization?</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/optim/sgd.py</a></h4>


      <pre><code class="lang-py">import torch
from torch import Tensor
from .optimizer import Optimizer, required, _use_grad_for_differentiable
from typing import List, Optional

__all__ = ['SGD', 'sgd']

class SGD(Optimizer):
    r"""Implements stochastic gradient descent (optionally with momentum).

    .. math::
       \begin{aligned}
            &amp;\rule{110mm}{0.4pt}                                                                 \\
            &amp;\textbf{input}      : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta)
                \text{ (objective)}, \: \lambda \text{ (weight decay)},                          \\
            &amp;\hspace{13mm} \:\mu \text{ (momentum)}, \:\tau \text{ (dampening)},
            \:\textit{ nesterov,}\:\textit{ maximize}                                     \\[-1.ex]
            &amp;\rule{110mm}{0.4pt}                                                                 \\
            &amp;\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do}                         \\
            &amp;\hspace{5mm}g_t           \leftarrow   \nabla_{\theta} f_t (\theta_{t-1})           \\
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I would really appreciate if anyone is able to point me to the line or function which does the random selection or correct me.<br>
Thanks</p>

nile649: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for tagging you. It is just you and few are only familiar with autograd.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/nile649">@nile649</a>,</p>
<p>The <code>SGD</code> optimizer doesn’t pick any samples at all, the ‘stochastic’ part of the algorthim comes from the data you use in the forward pass.</p>
<p>If you were to use mini-batches, that would create the stochasticity in gradient descent, likewise if you were to use all of your batch, it’d be full-batch gradient descent. It’s just a naming convention for it to be called “SGD”.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> wil know more precise details, so correct me if I’m wrong!</p>

ptrblck: <p>Yes, <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> is right and the optimizer does not create the samples but is responsible for the actual parameter updates. The naming (especially the S in SGD) might be a bit confusing.</p>