Dataloader and BatchSize

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>I want to make a stateful lstm.<br>
I set the Dataloader with batch size of 10000 but when I am going to initialize the hidden and cell stat it says that the batch size should be 5000.<br>
Here it is my DataLoader<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4d3f52e772356d2049e6c97f311c0946d396a2e.png" data-download-href="https://discuss.pytorch.org/uploads/default/f4d3f52e772356d2049e6c97f311c0946d396a2e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4d3f52e772356d2049e6c97f311c0946d396a2e_2_690x46.png" alt="image" data-base62-sha1="yVQCrRZqX8GkKhJeUq5Tz1kPReK" width="690" height="46" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4d3f52e772356d2049e6c97f311c0946d396a2e_2_690x46.png, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4d3f52e772356d2049e6c97f311c0946d396a2e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4d3f52e772356d2049e6c97f311c0946d396a2e.png 2x" data-dominant-color="E6E9EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">699×47 12.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To initialize the states I only do it before a new epoch starts, like in the photo below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/8864a3b75683add28ac1771dbd9b16c117bae6fc.png" alt="image" data-base62-sha1="jsAypZQxluktQk3GBg9DEAnSQ5S" width="576" height="191"></p>
<p>I need help to figure out the dataloader batch size problem since in the initialization of the states I need to put the batch size!</p>
<p>Thanks for the help!<br>
Regards<br>
André</p>

username: Andrei_Cristea
cooked: <p>Hello Andre, do you have a particular reason why you want such a gigantic batch size? Most training pipelines I’ve seen have a batch size between 8 and 128. You may have a very good reason for setting it so high, but just wanted to explicitly check that, since it seems very different from what other practitioners tend to do.</p>

username: Andre_Amaral_IST
cooked: <p>Hey,<br>
I have about 89 million of data in my dataset so I thought that batch size would be a good choice. If I am wrong explain to me, please?</p>
<p>Moreover, I wanted to do a stateful LSTM, where should i put the initialization of the cell and hidden states?<br>
Thanks in advance,<br>
Regards<br>
André</p>

username: Andrei_Cristea
cooked: <p>Hi, I’d recommend you start with a batch size that’s close to what others have used, maybe on the high end of that (say, 64 or 128). In practice, training with too large a batch size is said to not generalize well to new data (overfitting). I think there can also be floating point issues. <a href="https://wandb.ai/ayush-thakur/dl-question-bank/reports/What-s-the-Optimal-Batch-Size-to-Train-a-Neural-Network---VmlldzoyMDkyNDU" rel="noopener nofollow ugc">Here’s an experiment</a> where they show that, at least in their setup, large batch sizes do poorly.</p>
<p>Regarding the stateful LSTM, I’m not sure if there’s a direct way to do that in PyTorch, but there are other threads <a href="https://discuss.pytorch.org/t/confusion-regarding-pytorch-lstms-compared-to-keras-stateful-lstm/44502/5">such as this one</a> that discuss how to accomplish the same thing.</p>

username: Andre_Amaral_IST
cooked: <p>From the study you pointed out, it is true that large batch sizes perform poorly, however with the amount of data I have with a batch o 128 it will take very much time to train.</p>

username: Andrei_Cristea
cooked: <p>You can have your DataLoader pass something less than the entire dataset per epoch, and see if you see good performance regardless. In other words, your model might train well even without seeing the full dataset each epoch.</p>