Add automatic tuning flags to utils.data.dataloader

Sean_O_Bannon: <p>Hi! I’d like to highlight <a href="https://github.com/pytorch/pytorch/issues/23048" rel="nofollow noopener">a feature request made on the GitHub repo</a> for automatic tuning of batch_size and num_workers, and start some discussion around this topic.</p>
<p>Much like tensorflow has introduced a<a href="https://www.tensorflow.org/beta/guide/data_performance" rel="nofollow noopener">tf.data.experimental.AUTOTUNE</a> flag to automatically tune these parameters, I think this feature would be very relevant for PyTorch users as well.</p>
<p>I have a couple questions for the community to start gathering building concensus -</p>
<ul>
<li>Have you previously thought about this autotuning flag?</li>
<li>If you have thought about it before, what was the blocker to implementing it?</li>
<li>If this feature was introduced, would you use it?</li>
<li>What parameters do you use for batch_size and num_workers right now, and how do you set them?</li>
</ul>

ptrblck: <p>It would be interesting to know, how many users are choosing the batch size based on the computation performance vs. the methodological perspective (e.g. bad training behavior for tiny or huge batch sizes).<br>
E.g. if hypothetically a very small batch size would yield the best speedup, wouldn’t it also make some architectural changes necessary (replacing batchnorm layers for groupnorm etc.)?</p>

Sean_O_Bannon: <p>I think a flag like this might be most useful for inference jobs, where we care exclusively about performance, without regard for training behavior. You’re right - adjusting batch size for users automatically will have side effects for training; but we can avoid this issue by at least narrowing the scope to inference.</p>
<p>Additionally, tuning num_workers would improve performance for both training and inference, especially for large scale (GB/TB) inference jobs.</p>
<p>Also, experimentally, it seems that large batch sizes tend to yield the best speedups, up until data can no longer fit in memory.</p>

andravin: <p>batch_size: use the largest batch size that fits in GPU memory. Scale learning rate and warmup according to the method of <a href="https://arxiv.org/abs/1706.02677" rel="nofollow noopener">“Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.”</a></p>
<p>num_workers: how do you tune a Winnebago to get a good lap time at Le Mans? Step 1: replace it with a Ferrari. Step 2: tune the Ferrari. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Sean_O_Bannon: <p>Hey Andrew ! Your suggestion on batch_size makes a lot of sense.</p>
<p>For num_workers - are you suggesting a TF/other DL library solution or a different consumer of data within PyTorch?</p>

andravin: <p><a class="mention" href="/u/sean_o_bannon">@Sean_O_Bannon</a> I think it makes sense to have an optimized implementation of image folder and transforms for pytorch. The current API is nice, but the implementation is inefficient, and it slows down the entire system when it needs to feed 8xV100s and a small network (eg mobilenet_v2).</p>

Elegant_Lin: <p>Hi, I am also focusing on this research. Do you mean that I can follow this research by using <code>DistributedParralleDataset</code>?</p>
<p>I am very puzzled about this.</p>
<p>Thanks!</p>

ptrblck: <p>For such systems, you could try out <a href="https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/plugins/pytorch_tutorials.html" rel="nofollow noopener">DALI</a>.</p>