Why does my custom dataset class loop forever?

Brando_Miranda: <p>I wrote my own custom dataset class but when I try to iterate through its data one by one I get an infinite loop.</p>
<p>I went to the extreme and have the <code>__len__</code> method always return <code>0</code> and that didn’t stop it from continually looping through my dataset. How do I stop it? Why does:</p>
<pre><code class="lang-auto">for i, data in enumerate(dataset):
    print(i)
    print(data)

</code></pre>
<p>why does this keep calling the <code>next</code> method? How does it stop?</p>
<hr>
<p>In the hope of having a reproducible error I coded this:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self):
        self.bob = [0,1,2]

    def __len__(self):
        return 0

    def __getitem__(self, idx):
        print(f'idx = {idx}')
        return self.bob[idx]
</code></pre>
<p>now it does stop even though <code>__len__</code> is zero…</p>

Prerna_Dhareshwar: <p>Hi,<br>
I tried this-</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self):
        self.bob = [0,1,2]
    def __len__(self):
        return len(self.bob)
    def __getitem__(self, idx):
        print(f'idx = {idx}')
        return self.bob[idx]

dataset = DataLoader(MyDataset(), batch_size = 1, shuffle = True, num_workers = 0)

for i, data in enumerate(dataset):
    print(i)
    print(data)
</code></pre>
<p>And got the following output that did not loop infinitely-</p>
<pre><code class="lang-auto">idx = 1
0
tensor([1])
idx = 0
1
tensor([0])
idx = 2
2
tensor([2])
</code></pre>
<p>Is your DataLoader() correct, because your custom Dataset looks correct.</p>

Brando_Miranda: <p>I think I figured it out. I think its that the stopping condition of looping through the dataset has to be using the <code>len</code> function. Though for some reason it DOES work on the strange example I cooked up without looping forever…so it might be a weird edge case. I expect once I wrap it with the dataloader class everything should work fine (I hope).</p>

colesbury: <p>Make sure your <code>__getitem__</code> raises an <code>IndexError</code> for illegal indexes. If your <code>__getitem__</code> function never raises an exception then it will loop forever. The for loop doesn’t make use of <code>__len__</code>.</p>
<p>The example you posted will raise an <code>IndexError</code> at the correct time because <code>__getitem__</code> with <code>3</code> calls <code>self.bob[3]</code> which raises the error.</p>
<p><a href="https://docs.python.org/3/reference/datamodel.html#object.__getitem__" rel="nofollow noopener">https://docs.python.org/3/reference/datamodel.html</a></p>

Brando_Miranda: <p>I ran it in python’s unit test with no errors. You sure it does raise an error in yours with that simple/trivial example?</p>

colesbury: <blockquote>
<p>You sure it does raise an error in yours with that simple/trivial example?</p>
</blockquote>
<p>Yes. Notice that it prints <code>idx = 3</code>. The evaluation of <code>self.bob[3]</code> raises an IndexError. The Python interpreter catches this error and stops the for loop.</p>
<p>This doesn’t happen if you wrap it in a DataLoader like in <a class="mention" href="/u/prerna_dhareshwar">@Prerna_Dhareshwar</a>’s example because DataLoader uses <code>__len__</code>.</p>
<p>Here is the full runnable example:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset

class MyDataset(Dataset):
    def __init__(self):
        self.bob = [0,1,2]

    def __len__(self):
        print('len')
        return 0

    def __getitem__(self, idx):
        print(f'idx = {idx}')
        return self.bob[idx]


dataset = MyDataset()
for i, data in enumerate(dataset):
    print(i)
    print(data)
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-auto">idx = 0
0
0
idx = 1
1
1
idx = 2
2
2
idx = 3
</code></pre>