Tutorial on custom dataloaders (NOT datasets)?

username: Mahmoud_Abdelkhalek
cooked: <p>Hi,</p>
<p>I really appreciate <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#iterating-through-the-dataset" rel="noopener nofollow ugc">this</a> tutorial on custom datasets. However, the <code>torch.utils.data.DataLoader</code> class is only briefly mentioned in it:</p>
<blockquote>
<p>However, we are losing a lot of features by using a simple <code>for</code> loop to iterate over the data. In particular, we are missing out on:</p>
<ul>
<li>Batching the data</li>
<li>Shuffling the data</li>
<li>Load the data in parallel using <code>multiprocessing</code> workers.</li>
</ul>
</blockquote>
<blockquote>
<p><code>torch.utils.data.DataLoader</code> is an iterator which provides all these features. Parameters used below should be clear. One parameter of interest is <code>collate_fn</code> . You can specify how exactly the samples need to be batched using <code>collate_fn</code> . However, default collate should work fine for most use cases.</p>
</blockquote>
<p>Could we possibly get a tutorial on custom dataloaders using the <code>torch.utils.data.DataLoader</code> class? More specifically, how to interface with its <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">parameters</a>, especially the <code>num_workers</code> and <code>collate_fn</code> parameters. Also, an explanation on how to inherit from the abstract base class and a template for the <code>collate_fn</code> would be nice too.</p>
<p>I am aware of this <a href="https://github.com/pytorch/tutorials/issues/78" rel="noopener nofollow ugc">issue</a> and this <a href="https://github.com/pytorch/tutorials/issues/735" rel="noopener nofollow ugc">issue</a> but neither have led to a tutorial.</p>
<p>Again, I really appreciate the effort that goes into the tutorials that are currently available, but I feel that a tutorial on custom dataloaders would answer a lot of questions.</p>

username: ptrblck
cooked: <p>That makes sense and might be beneficial for advanced use cases.<br>
Would you be interested in creating such a tutorial, which is digging into the <code>DataLoader</code> internals? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Mahmoud_Abdelkhalek
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<p>Thanks for the response.</p>
<p>I would love to make a tutorial, but I am not sure where to start exactly. Any suggestions? Also, are there any prerequisite concepts that I need to know about?</p>

username: ptrblck
cooked: <p>I would suggest to create a feature request <a href="https://github.com/pytorch/tutorials/issues">here</a> with some information from this topic, such as what is currently not explained well and your suggestions how to improve it. After a discussion you should be good to go. If you need some help or guidance, feel free to ping me in the created issue.</p>

username: Mahmoud_Abdelkhalek
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I created the issue <a href="https://github.com/pytorch/tutorials/issues/1010" rel="nofollow noopener">here</a> but have not yet received a reply. Any suggestions on what to do next?</p>

username: ptrblck
cooked: <p>Thanks! I started the discussion in the issue.</p>