Dataloader shuffling best practice

username: ArthurV
cooked: <p>Hello,</p>
<p>I am fairly new to deep learning algorithms and a question comes to mind regarding how to properly shuffle my training data. Looking at the tutorial <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class" rel="noopener nofollow ugc">data_loading_tutorial.html#dataset-class</a> which describes how to create our own dataset, the <code>__get_item__()</code> method seems to be the key point of this class. My question is this:</p>
<p>Should the call to <code>__get_item__()</code> for a given <code>idx</code> must return the same tensor every time? or in other words: should the batch 0 be the same across epochs? Is the <code>shuffle=True</code> parameter in the dataloader sufficient to ensure a complete random distribution of the data? Does it depend on the optimizer or something else (I use Adam’s optimizer in a classification context)? In fact, the tutorial doesn’t seem to rearrange the contents of the batch at each time.</p>
<p>I am sorry if this is not the right place to post my question, please show me where it would be most appropriate to post my question.</p>
<p>Thank you,</p>

username: zetyquickly
cooked: <p>Hello <a class="mention" href="/u/arthurv">@ArthurV</a></p>
<p>It is the place where you can find the answers.</p>
<blockquote>
<p>Is the <code>shuffle=True</code> parameter in the dataloader sufficient to ensure a complete random distribution of the data?</p>
</blockquote>
<p>This parameter facilitates the permutation of dataset’s indices. That means each epoch you’ll draw random <code>batch_size</code> samples of data, each iteration.</p>
<blockquote>
<p>Should the call to <code>__get_item__()</code> for a given <code>idx</code> must return the same tensor every time?</p>
</blockquote>
<p>The <code>__getitem__</code> method is expected to return the data that available in dataset. You are definitely can introduce some randomization to it, i.e. each time you call <code>ds[idx]</code> it is able to return different data. For example if you do randomly parameterized augmentations (transformations) then it’ll give you different tensors each time you call <code>ds[idx]</code>. But in my opinion there’s no particular need to return <code>ds[10]</code> when you ask for <code>ds[139]</code>, while <code>shuffle</code> parameter in <code>Dataloader</code> exists.</p>
<blockquote>
<p>Does it depend on the optimizer or something else (I use Adam’s optimizer in a classification context)?</p>
</blockquote>
<p>Adam is a <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam" rel="noopener nofollow ugc">variation</a> of gradient descent. In PyTorch <code>SGD</code> perform an <a href="https://discuss.pytorch.org/t/how-sgd-works-in-pytorch/8060/2">operation</a> that is known as mini-batch gradient descent. So, when you do <code>shuffle</code> you introduce randomness (making it stochastic) that helps the loss to converge</p>
<blockquote>
<p>In fact, the tutorial doesn’t seem to rearrange the contents of the batch at each time.</p>
</blockquote>
<p>Unfortunately, I haven’t tried the tutorial but as long as <code>Dataloader</code> contains <code>shuffle=True</code> this loader will output each epoch different samples in a batch, i.e. if you inspect first batch of each epoch it probably will have different set of objects from dataset</p>
<p>Hope it helps!</p>

username: ArthurV
cooked: <p>Thank you for your quick answer. I note that it is not necessary to re-shuffle my data-set at each epoch as long as the dataloader has it’s <code>shuffle</code> parameter activated.</p>