Custom dataset delivers always idx 0

wilhelm: <p>I’m experiencing a very strange behavior, which I wouldn’t expect.</p>
<p>I created this minimal example here to show the problem.<br>
First I created a <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" rel="noopener nofollow ugc">Dataset</a>:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset
class SquadDataset(Dataset):

    def __init__(self, encodings):
        self.encodings = encodings

    def __len__(self):
        return len(self.encodings)

    def __getitem__(self, idx):
        print(idx)
        return self.encodings[idx]
</code></pre>
<p>and then running the following minimal cript:</p>
<pre><code class="lang-auto">lst = [1, 3, 4, 5, 6, 7, 8, 9, 10]

d = SquadDataset(encodings = lst)

for _ in range(10):

    e = next(iter(d))
</code></pre>
<p>I get only a long list of zeros.</p>
<p>Shouldn’t <em>idx</em> be a random number?<br>
Shouldn’t the method <em><strong>getitem</strong></em> call <em><strong>call</strong></em> before generating the number? Using a debugger I realized, that this is not the case (it does not call <em><strong>call</strong></em> but <em><strong>getitem</strong></em> directly)</p>

ptrblck: <p>You are recreating the <code>iterator</code> instead of calling <code>next</code> on it. This will work:</p>
<pre data-code-wrap="python"><code class="lang-python">d = SquadDataset(encodings = lst)
iter_d = iter(d)
for _ in range(10):
    e = next(iter_d)
</code></pre>
<p>Output:</p>
<pre data-code-wrap="python"><code class="lang-python">0
1
2
3
4
5
6
7
8
9
Traceback (most recent call last):

  Cell In[78], line 2
    e = next(iter_d)

StopIteration
</code></pre>

wilhelm: <p>Thanks,</p>
<p>but the problem was that I didn’t wrap the <em>Dataset</em> in the <em>DataLoader</em> class.</p>
<p>This worked and solved the problem:</p>
<pre><code class="lang-auto">data = DataLoader(SquadDataset, batch_size = 8)

...
iter_d = iter(data)
for _ in range(10):
    e = next(iter_d)
....
</code></pre>

ptrblck: <p>No, your approach won’t solve the actual issue of recreating the <code>iterator</code> as shown in my example and you are still returning the <em>same</em> samples defined by the <code>batch_size</code>, i.e. you are missing the last samples.<br>
If you reduce the <code>batch_size</code> to <code>1</code>, you are seeing the same issue:</p>
<pre data-code-wrap="python"><code class="lang-python">data = DataLoader(SquadDataset(encodings = lst), batch_size = 1)
for _ in range(10):
    e = next(iter(data))
# 0
# 0
# 0
# 0
# 0
# 0
# 0
# 0
# 0
# 0  
</code></pre>

wilhelm: <p>I tried your solution.</p>
<p>It works.<br>
But even my proposed solution works…</p>
<p>I’m going to recreate the file from scratch and then I report it to you.</p>

ptrblck: <p>Your solution is using a <code>DataLoader</code> would only work if <code>batch_size &gt;= num_samples</code>, i.e. if you are returning <em>all</em> samples in a single <code>next(iter(loader))</code> call. In your example you are never receiving the last <code>list</code> entry (<code>10</code>) and you can use my code snippet to see that recreating the <code>iterator</code> inside the <code>loop</code> is causing the issue.</p>

wilhelm: <p>I’m back here because I tried your way, but it does not deliver, what it is expected.<br>
I mean:</p>
<ol>
<li>The method <em><strong>len</strong></em> is never called. Which is not the case using the code I posted above</li>
<li><em>idx</em> increase incrementally from 0 to the last element of the range. But <em>idx</em> should be a random number</li>
</ol>
<p>Can you point me to the right direction?</p>

ptrblck: <ol>
<li>
<p>The <code>__len__</code> attribute is used to initialize the <code>sampler</code> used in a <code>DataLoader</code>. I’m currently unsure if you are trolling or just show no interest in understanding why recreating the <code>iterator</code> inside a nested loop will not sample from the entire dataset. I’ve already posted code snippets which are executable and which you can copy/paste to run. With that being said, feel free to stick to your approach. For other users, please don’t use it as you approach will miss samples and make sure to create the <code>iterator</code> outside the sampling loop.</p>
</li>
<li>
<p>Use <code>shuffle=True</code> in a <code>DataLoader</code>.</p>
</li>
</ol>