SGD and batch_size in data.DataLoader()

shirui-japina: <p>When we use SGD algorithm to update the parameters in the model, we use only one sample to feed the model each time, it it right?<br>
But in PyTorch, even when we use SGD, we have to set <code>batch_size</code> and its value can <strong>not</strong> be 1.<br>
What puzzles me is, when use SGD and <code>batch_size</code> is <strong>not</strong> 1, what exactly is the algorithm of the program? Is it still SGD?</p>

mariosasko: <aside class="quote no-group" data-username="shirui-japina" data-post="1" data-topic="92912">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/48/15285_2.png" class="avatar"> shirui-japina:</div>
<blockquote>
<p>we have to set <code>batch_size</code> and its value can <strong>not</strong> be 1</p>
</blockquote>
</aside>
<p>This statement is not true, e.g. 1 is a default <code>batch_size</code> of <code>torch.utils.data.DataLoader</code>. So to get “true” SGD, you are free to use 1 as <code>batch_size</code>.</p>
<p>When <code>batch_size</code> is greater than 1, the algorithm is called “mini-batch” gradient descent, and when <code>batch_size</code> is equal to <code>len(dataset)</code> we are talking about “batch” gradient descent.</p>

shirui-japina: <aside class="quote no-group" data-username="mariosasko" data-post="2" data-topic="92912">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mariosasko/48/27520_2.png" class="avatar"> mariosasko:</div>
<blockquote>
<p>This statement is not true, e.g. 1 is a default <code>batch_size</code> of <code>torch.utils.data.DataLoader</code> . So to get “true” SGD, you are free to use 1 as <code>batch_size</code> .</p>
</blockquote>
</aside>
<p>I see. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="mariosasko" data-post="2" data-topic="92912">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mariosasko/48/27520_2.png" class="avatar"> mariosasko:</div>
<blockquote>
<p>When <code>batch_size</code> is greater than 1, the algorithm is called “mini-batch” gradient descent, and when <code>batch_size</code> is equal to <code>len(dataset)</code> we are talking about “batch” gradient descent.</p>
</blockquote>
</aside>
<p>Thanks for the detailed explanation. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>