Create a custom dataloader

username: Flock1
cooked: <p>Hi,</p>
<p>I am working with a custom <code>dataloader</code> and I wanted to know if it is necessary to create a class with <code>__init__</code>, <code>__getitem__</code>? Because I tried to create my own method that returns batches of data. Since I am using <code>return</code> instead of <code>yield</code>, I only get one batch of data instead of all the data.</p>
<p>However, if I use <code>yield</code>, I get the following error:</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-35-30905f729d78&gt; in &lt;module&gt;
----&gt; 1 train_loss = training(model, train_dl, param.epochs)

&lt;ipython-input-34-100bda3ef538&gt; in training(model, train_loader, Epochs)
      3     for epoch in range(Epochs):
      4         running_loss = 0.0
----&gt; 5         for data in train_loader:
      6             img = data
      7             img = img.to(device, dtype=torch.float)

~/anaconda3/envs/work/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __iter__(self)
    277             return _SingleProcessDataLoaderIter(self)
    278         else:
--&gt; 279             return _MultiProcessingDataLoaderIter(self)
    280 
    281     @property

~/anaconda3/envs/work/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __init__(self, loader)
    744         # prime the prefetch loop
    745         for _ in range(2 * self._num_workers):
--&gt; 746             self._try_put_index()
    747 
    748     def _try_get_data(self, timeout=_utils.MP_STATUS_CHECK_INTERVAL):

~/anaconda3/envs/work/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _try_put_index(self)
    859         assert self._tasks_outstanding &lt; 2 * self._num_workers
    860         try:
--&gt; 861             index = self._next_index()
    862         except StopIteration:
    863             return

~/anaconda3/envs/work/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _next_index(self)
    337 
    338     def _next_index(self):
--&gt; 339         return next(self._sampler_iter)  # may raise StopIteration
    340 
    341     def _next_data(self):

~/anaconda3/envs/work/lib/python3.8/site-packages/torch/utils/data/sampler.py in __iter__(self)
    198     def __iter__(self):
    199         batch = []
--&gt; 200         for idx in self.sampler:
    201             batch.append(idx)
    202             if len(batch) == self.batch_size:

~/anaconda3/envs/work/lib/python3.8/site-packages/torch/utils/data/sampler.py in __iter__(self)
     60 
     61     def __iter__(self):
---&gt; 62         return iter(range(len(self.data_source)))
     63 
     64     def __len__(self):

TypeError: object of type 'generator' has no len()
</code></pre>
<p>So is it really necessary to use the prescribed method or is there some way I can use <code>yield</code>?</p>

username: ptrblck
cooked: <p>Could you describe your use case and why you need to create a custom <code>DataLoader</code>?<br>
Usually you would create a custom <code>Dataset</code> (as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>) and, if necessary, write a custom <code>collate_fn</code> for the <code>DataLoader</code>, but wouldn’t need to manipulate the <code>DataLoader</code> directly.</p>

username: Flock1
cooked: <p>Hi,</p>
<p>Thank you for the reply and my sincere apologies for the late response. That’s kinda why I want to create a custom dataloader because I am not working with a generic dataset and getting the dataset into a generic one will take a lot of time.</p>
<p>I think I was looking for this only, <code>collate_fn</code>. Probably this is why I am getting this error.</p>