Next(iter(dataloader)) does NOT stopp; return same files again

MikeTensor: <p>Hello!</p>
<p>When using<br>
<code>next(iter(train_dataloader))</code></p>
<p>I have two issues:</p>
<ol>
<li>The dataloader doesn’t stop.<br>
TEST_SET has length of 101. I can iterate 1000 times and more …<br>
Shouldn’t there be a <code>stopIteration</code>?</li>
</ol>
<p>– My CustomDataset contains a <code>__len__</code> which (if I print it out within the <code>__getitem__</code> seems to work).</p>
<ol start="2">
<li>Output is redundant<br>
I saved the DataLoader-Output to a CSV-File<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91be70b88ce18679bd8a45dc2cfcbeb226679a6d.png" data-download-href="https://discuss.pytorch.org/uploads/default/91be70b88ce18679bd8a45dc2cfcbeb226679a6d" title="Bildschirmfoto vom 2023-03-01 02-43-13"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91be70b88ce18679bd8a45dc2cfcbeb226679a6d.png" alt="Bildschirmfoto vom 2023-03-01 02-43-13" data-base62-sha1="kNjftSZ5EbDmvVLKd90twxJ75i5" width="217" height="500" data-dominant-color="DAD4D5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Bildschirmfoto vom 2023-03-01 02-43-13</span><span class="informations">365×839 28.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ol>
<p>Quite a lot of Images are more than double.<br>
0000003.png is not the only file.</p>
<p>This shouldn’t be the case, right?<br>
What could be the reason therefore?</p>
<p>I am using the <a href="https://pytorch.org/vision/main/_modules/torchvision/datasets/kitti.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.datasets.kitti — Torchvision main documentation</a> Dataset; just a bit adjusted to my own needings.</p>
<p>################</p>
<pre><code class="lang-auto">train_dataloader = DataLoader(dataset=train_data,
                              collate_fn= None,
                              batch_size=1, # how many samples per batch?
                              num_workers=1, # how many subprocesses to use for data loading? (higher = more)
                              pin_memory=True,
                              shuffle=True) # shuffle the data?
</code></pre>
<p>This is how I have tested my <code>DataLoader</code>:</p>
<pre><code class="lang-auto">image_list = []
for count in range(0, 1001):
    a = next(iter(train_dataloader))
    image_list.append(a)
    if count % 100 == 0:
        print(f"{count} of 1000")

print(f"Last index is: {count}")
liste = pd.DataFrame(image_list)
liste.to_csv('myDataTest/image_list.csv')
</code></pre>

nivek: <p>Hi,</p>
<p>You are repeatedly creating a new iterator of <code>DataLoader</code> in <code> a = next(iter(train_dataloader))</code>. Every time a new iterator of <code>DataLoader</code> is created, it can start reading from the beginning.</p>
<p>Instead, you should create it once (unless you actually want to read from the beginning multiple times).</p>
<p>This should work for your use case:</p>
<pre><code class="lang-python">for a in train_dataloader:
     image_list.append(a)
</code></pre>

MikeTensor: <p>Thank you very much <a class="mention" href="/u/nivek">@nivek</a><br>
Works! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Btw. Is there a way to get the length and the actual index out?<br>
(So that I can realize my</p>
<pre><code class="lang-auto"> if INDEX % 10 == 0:
        print(f"{INDEX} of {length}")
</code></pre>

nivek: <p>You can use <code>enumerate</code> as such:</p>
<pre><code class="lang-python">for i, a in enumerate(train_dataloader):
    image_list.append(a)
    if i % 100 == 0:
        print(f"{i} of {len(train_dataloader)}")
</code></pre>