Infinite DataLoader

AlexisW: <p>Is there a good way to have an infinite dataloader? That said, is there a class that will provide automatically looping for method like <code>data_loader.get_next()</code>? And how to maintain full iterations?</p>

Nabarun_Goswami: <p>Well one quick and dirty hack would be for your CustomDataset to return a very high number (e.g. np.iinfo(np.int64).max) in its <code>__len__</code> .</p>
<p>As for get_next(), you can get the iterator from the dataloader and call next on that:</p>
<p><code>next(dataloader.__iter__())</code></p>

AlexisW: <p>Thanks! However, in this way, will the data sampling strategy be affected? That said, if I have 10 batches for the data, and I do next for 20 times, will that be 2 full iteration?</p>

AlexisW: <p>Any thoughts?<img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=5" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

justusschock: <p>Alternatively you should be able to just catch the StopIteration exception and reinitialize the generator (maybe with some previous shuffling)</p>

AlexisW: <p>Could you provide a basic example on how to do so? Thanks!!</p>

AlexisW: <p>Also it seems that every time I called <code>next(dataloader.__iter__())</code>, I will get the same data.</p>

justusschock: <pre><code class="lang-auto">next(data loader. __iter__()) 
</code></pre>
<p>Creates a new iterator starting at the beginning of the dataset every time you call it.</p>
<pre><code class="lang-auto"># create dataloader-iterator
data_iter = iter(data_loader) 

# iterate over dataset
# alternatively you could use while(True) 
for i in range(NUM_ITERS_YOU_WANT)
    try:
        data = next(data_iter) 
    except StopIteration:
        # StopIteration is thrown if dataset ends
        # reinitialize data loader 
        data_iter = iter(data_loader)
        data = next(data_iter) 
</code></pre>
<p>Could do the trick (I did not test the code, just typed it here)</p>

AlexisW: <p>Oh yeah that’s the one. I also just figured it out! Thanks!</p>

Nabarun_Goswami: <p>Yes, that will be 2 full iterations. And you can even disregard the shuffling of the data indices by the dataloader and follow your own stragety in the next function. For examle irrespective of ehat index the dataloader sends, you can calculate your own index based on your dataset, that way your strategy would not be affected even if you put a huge nunber in len. It’s all subjective how you want to design your dataset.</p>

vaisakh: <p>This should do the trick.</p>
<pre><code class="lang-python">def loopy(dl):
    while True:
        for x in iter(dl): yield x
</code></pre>
<p>Then, everywhere you used to use <code>iter(dataloader)</code>, use <code>loopy(dataloader)</code>.</p>
<p>Beware though, that samplers passed to the DataLoader can raise <code>StopIteration()</code> exceptions.</p>

RylanSchaeffer: <p><a class="mention" href="/u/justusschock">@justusschock</a> how can I make the data loader shuffle the data every time its iterator is reinitialized?</p>
<p><a class="mention" href="/u/alexisw">@AlexisW</a> maybe you know?</p>

SimonW: <p>Official non-hacky support for this is happening in <a href="https://github.com/pytorch/pytorch/pull/19228" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/19228</a></p>

devashishkprasad: <p>As <a class="mention" href="/u/simonw">@SimonW</a> mentioned, the release of PyTorch 1.2 brought with it a new dataset class: torch.utils.data.IterableDataset.</p>
<p>Here you can read the official documentation related to it <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">IterableDataset</a></p>

Hariharan_J: <pre><code class="lang-auto">from torch.utils.data import DataLoader, Dataset, Sampler
import random

class listDataset(Dataset):
    def __init__(self):
        self.varList = [1,2,3,4]
    def __len__(self):
        return len(self.varList)
    def __getitem__(self, idx) :
        return self.varList[idx]

class customSampler(Sampler) :
    def __init__(self, dataset, shuffle):
        assert len(dataset) &gt; 0
        self.dataset = dataset
        self.shuffle = shuffle
    
    def __iter__(self):
        order = list(range((len(self.dataset))))
        idx = 0
        while True:
            yield order[idx]
            idx += 1
            if idx == len(order):
                if self.shuffle:
                    random.shuffle(order)
                idx = 0

if __name__ == "__main__":
    dset = listDataset()
    sampler = customSampler(dset, shuffle=True)
    loader = iter(DataLoader(dataset=dset, sampler=sampler, batch_size=6, num_workers=2))
    for x in range(10):
        i = next(loader)
        print(i)
    
</code></pre>

JH_S: <p>The dataloader without <code>StopIteration</code> can be made like,</p>
<pre><code class="lang-python">import torch
from torch.utils.data import DataLoader

class InfiniteDataLoader:
    def __init__(self, data_loader):
        self.data_loader = data_loader
        self.data_iter = iter(data_loader)

    def __iter__(self):
        return self

    def __next__(self):
        try:
            data = next(self.data_iter)
        except StopIteration:
            self.data_iter = iter(self.data_loader)  # Reset the data loader
            data = next(self.data_iter)
        return data

if __name__ == "__main__":
    # Replace this with your actual data loader creation
    data_loader = DataLoader(dataset=your_dataset, batch_size=your_batch_size, shuffle=True)

    infinite_loader = InfiniteDataLoader(data_loader)

    # You can now use the infinite_loader in a for loop or any other iteration
    for data in infinite_loader:
        # Your data processing code here
</code></pre>
<p>If you are using <code>DistributedSampler</code>, you may need the following</p>
<pre><code class="lang-python">class InfiniteDataLoader:
    def __init__(self, data_loader):
        self.data_loader = data_loader
        self.data_iter = iter(data_loader)
        if not hasattr(self.data_loader.sampler, 'epoch'): 
            self.data_loader.sampler.epoch = 0

    def __iter__(self):
        return self

    def __next__(self):
        try:
            data = next(self.data_iter)
        except StopIteration:
            self.data_loader.sampler.epoch += 1
            self.data_iter = iter(self.data_loader)  # Reset the data loader
            data = next(self.data_iter)
        return data
</code></pre>