Documentation/tutorial data length mistake

username: KamWithK
cooked: <p>Hey guys,<br>
I just started using PyTorch and have been going through a lot of tutorials/documentation recently.<br>
I think I’ve found an error in a few places where you calculate stats like accuracy and loss.<br>
To calculate most stats you divide some running metric by the data set’s size. There are around 3 ways I’ve found online to accomplish this (all from my experimentation so far have yielded incorrect results):</p>
<ul>
<li>len(data_loader) - when using a batch size greater than 1, this seems to return the batch size (not what we’re looking for)</li>
<li>len(data_loader.dataset) - this works in some cases, however once you use a data sampler to create a dataloader which uses a smaller subset of your data this breaks down (as it doesn’t return the actual currently utilized data’s size)</li>
<li>len(data_loader) * batch_size - this results in a slightly off number as PyTorch adjusts the batch size when the length of the dataset you’re working with doesn’t divide perfectly with your batch size (i.e. mod isn’t 0)</li>
</ul>
<p>What I’ve found though was that using len(data_loader.sampler) WILL return precisely what you’re looking for (length of the training/validation data by itself). Just wanted to point this out as I’ve spent a huge amount of time looking for why my loss/accuracy were always incredably small.<br>
I’m willing to show my code if needed.</p>
<p>Official tutorials/documentation:<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a> - first case<br>
<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a> - second case</p>

username: ptrblck
cooked: <p>Thanks for pointing this out.<br>
If I’m not mistaken, both linked tutorials use the second approach (which would work, since no custom <code>sampler</code> is used).</p>
<p>I’m not sure, if your suggestions would work in all use cases, e.g. using a <a href="https://pytorch.org/docs/stable/data.html#disable-automatic-batching">batch_sampler</a>.<br>
However, I haven’t thought about all edge cases. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: KamWithK
cooked: <p>Not entirely sure about all cases, but I would have thought using a custom dataset/sampler would be quite common.<br>
By the way the first link uses the first approach (just written in a slightly more generic form), I’ve copy pasted the relavent line here: dataset_sizes = {x: len(image_datasets[x]) for x in [‘train’, ‘val’]}</p>
<p>Is it possible to add a note or something on the tutorials/documentation to point out exceptions where the code won’t work? I feel like it would be extremely useful, seeing that this is an elusive bug to find,which can easily lead to misinterpreted stats.</p>

username: ptrblck
cooked: <p>Personally, I wouldn’t want to add the note into the mentioned tutorials, as they are working fine in the current state. In my opinion there are a lot of edge cases, where just copy pasting the tutorial might not work out of the box and I don’t see a custom <code>sampler</code> as the usual use case.</p>
<p>That being said, I think your suggestion would fit into the FAQ section.<br>
Feel free to create an issue on GitHub and explain your proposal there, please. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KamWithK
cooked: <p>Ok, thanks. Will do soon <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>