Why Dataset is not converting to list as expected?

Versus: <p>I’m trying to do <code>list(dataset)</code> but it returns more than expected. In some setups it runs out of bounds. Sample code:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset

class NumbersDataset(Dataset):
    def __init__(self):
        self.samples = list(range(1, 1001))

    def __len__(self):
        return len(self.samples)-500

    def __getitem__(self, idx):
        return self.samples[idx]


dataset = NumbersDataset()

len(dataset)
&gt; 500

len(list(dataset))
&gt; 1000
</code></pre>
<p>I would expect <code>list(dataset)</code> to return 500, but somehow this is ignored.<br>
The same problem happens when you iterate over the dataset with for loop.<br>
I mean how does it even know that there are 1000 elements? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Any ideas?</p>

Altiki: <p>You see this because len() and __ len __() are not the same things, as explained here:</p>
<p><a href="https://kushaldas.in/posts/len-function-in-python.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kushaldas.in/posts/len-function-in-python.html</a></p>
<p>If you do:</p>
<pre><code class="lang-auto">print(dataset.__len__())
</code></pre>
<p>you will get 500 as you expect.</p>

Versus: <p>There’s no problem with len. See</p>
<pre><code class="lang-python">print(dataset.__len__(), len(dataset))
&gt; 500 500

# But...
len(list(dataset))
&gt; 1000
</code></pre>
<p>Both <code>len</code> return 500. But len(list(dataset)) gives 1000…</p>

user_123454321: <p>I think the following example might help in understanding</p>
<pre><code class="lang-auto">&gt;&gt;&gt; class X:
...     def __init__(self):
...             self.arr = [1, 2, 3]
...     def __getitem__(self, idx):
...             print('getting: ', idx)
...             return self.arr[idx]
...
&gt;&gt;&gt; x = X()
&gt;&gt;&gt; list(x)
getting:  0
getting:  1
getting:  2
getting:  3
[1, 2, 3]
</code></pre>
<p>I guess the list function results in calling the <strong>getitem</strong> function for each element.<br>
Since, the dataset class is pretty much an abstract class in which one has to implement the __getitem__function. The <strong>len</strong> function allows one to have a custom size for the dataset separate from the actual iteratable object size.</p>

Versus: <p>Indeed it calls <code>__getitem___</code>. But <code>list()</code> uses <code>__getitem__</code> with more indexes than expected.<br>
In my first example the <code>__len__</code> returns 500. I’d expect <code>list()</code> to call <code>__getitem__</code> 500 times. But it calls it 1000 times.</p>

user_123454321: <p>I think it calls <strong>getitem</strong> for as many elements in the iterable regardless of what <strong>len</strong> returns.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; class X:
...     def __init__(self):
...             self.arr = [1, 2, 3, 4, 5]
...     def __getitem__(self, idx):
...             print('getting: ', idx)
...             return self.arr[idx]
...     def __len__(self):
...             return 1
...
&gt;&gt;&gt; x = X()
&gt;&gt;&gt; list(x)
getting:  0
getting:  1
getting:  2
getting:  3
getting:  4
getting:  5
[1, 2, 3, 4, 5]
</code></pre>

Versus: <p>Correct. And that is exactly why I am confused. How it even knows that it should call it 5 times (or in fact even 6!).</p>
<p><code>__getitem__</code> doesn’t tell how many items are there. <code>list()</code> doesn’t know that <code>self.arr</code> is the iterable it needs to iterate over. I thought that it should check <code>len(dataset)</code> first and then iterate over the length of the dataset. This sounds logical to me.</p>
<p>And I fail to understand how it iterates over more items…</p>

Versus: <p>Btw <a class="mention" href="/u/user_123454321">@user_123454321</a><br>
why it prints <code>getting:  5</code> and doesn’t raise a <code>IndexError</code> because there is no item with index 5?</p>

1chimaruGin: <aside class="quote no-group" data-username="Versus" data-post="1" data-topic="109284">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/versus/48/33590_2.png" class="avatar"> Versus:</div>
<blockquote>
<pre><code class="lang-auto">def __getitem__(self, idx):
        return self.samples[idx]
</code></pre>
</blockquote>
</aside>
<p>Because the samples list has 1000 elements. When you call list(dataset), it would contain 1000 elements.</p>

user_123454321: <p>Ah, interesting. I think it goes until it encounters IndexError.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; class X:
...     def __init__(self):
...             self.arr = [1, 2, 3, 4, 5]
...             self.arr2 = [1, 2, 3]
...     def __getitem__(self, idx):
...             print('getting: ', idx)
...             return self.arr[idx], self.arr2[idx]
...     def __len__(self):
...             return 1
...
&gt;&gt;&gt; x = X()
&gt;&gt;&gt; list(x)
getting:  0
getting:  1
getting:  2
getting:  3
[(1, 1), (2, 2), (3, 3)]
</code></pre>

Versus: <p><a class="mention" href="/u/1chimarugin">@1chimaruGin</a> let’s have another example:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset

class NumbersDataset(Dataset):
    def __init__(self):
        self.samples1 = list(range(1, 1001))
        self.samples2 = list(range(1, 11))

    def __len__(self):
        return len(self.samples1)-500

    def __getitem__(self, idx):
        return self.samples1[idx]
      
    def __iter__(self):
      return iter(self.samples1)


dataset = NumbersDataset()

len(dataset), len(list(dataset))
&gt;&gt;&gt; 500, 1000
</code></pre>
<p>Ho does <code>list()</code> know that it should iterate over <code>.samples1</code> and not <code>.samples2</code> which has only 10 items?</p>

user_123454321: <p>Probably because samples2 are not accessed in getitem ?</p>

Versus: <p><a class="mention" href="/u/user_123454321">@user_123454321</a> I thought so too. But it doesn’t stop at Index Error:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset

class NumbersDataset(Dataset):
    def __init__(self):
        self.samples = [1,2,3,4,5]

    def __len__(self):
        return 3

    def __getitem__(self, idx):
        print(f"Trying {idx}")
        if idx&gt;3: raise IndexError
        return self.samples[idx]



dataset = NumbersDataset()

len(dataset), len(list(dataset))
&gt;&gt;&gt; (3,5)

list(dataset)
&gt;&gt;&gt; [1, 2, 3, 4, 5]
</code></pre>

1chimaruGin: <aside class="quote no-group" data-username="Versus" data-post="11" data-topic="109284">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/versus/48/33590_2.png" class="avatar"> Versus:</div>
<blockquote>
<pre><code class="lang-auto">class NumbersDataset(Dataset):
    def __init__(self):
        self.samples1 = list(range(1, 1001))
        self.samples2 = list(range(1, 11))

    def __len__(self):
        return len(self.samples1)-500

    def __getitem__(self, idx):
        return self.samples1[idx]
      
    def __iter__(self):
      return iter(self.samples1)
</code></pre>
</blockquote>
</aside>
<p>In python, <code>__getitem__</code> in class make it iterable.</p>
<p><a href="https://stackoverflow.com/questions/926574/why-does-defining-getitem-on-a-class-make-it-iterable-in-python" rel="noopener nofollow ugc">stackoverflow Q&amp;A</a> about getitem.</p>

user_123454321: <p>I am not sure why you added <strong>iter</strong>, but it does stop at IndexError without it</p>
<pre><code class="lang-auto">&gt;&gt;&gt; class NumbersDataset:
...     def __init__(self):
...         self.samples = [1,2,3,4,5,6,7,9,10]
...     def __len__(self):
...         return 3
...     def __getitem__(self, idx):
...         print("Trying", idx)
...         if idx&gt;3:
...                 print("Raising Index error")
...                 raise IndexError
...         return self.samples[idx]
...
&gt;&gt;&gt; dataset = NumbersDataset()
&gt;&gt;&gt;
&gt;&gt;&gt; len(dataset), len(list(dataset))
Trying 0
Trying 1
Trying 2
Trying 3
Trying 4
Raising Index error
(3, 4)
</code></pre>
<p>I think <strong>iter</strong> bypasses the <strong>getitem</strong></p>

Versus: <p>Thanks to <a class="mention" href="/u/user_123454321">@user_123454321</a> we have an answer.<br>
Indeed, <code>__len__</code> doesn’t influence how an iterable is converted to a list or iterated over in for loops. In fact, you have to implement and <code>IndexError</code> which signals the end of the iteration. This is also described in <a href="https://docs.python.org/3/reference/datamodel.html#object.__getitem__" rel="noopener nofollow ugc">the <code>__getitem__</code> docs</a></p>
<p>So, the correct implementation is as follows:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset

class NumbersDataset(Dataset):
  def __init__(self):
    self.samples = [1,2,3,4,5]

  def __len__(self):
    return 3

  def __getitem__(self, idx):
    if idx&gt;2: 
      raise IndexError
    return self.samples[idx]


dataset = NumbersDataset()

len(dataset), len(list(dataset))
&gt;&gt;&gt; 3, 3

list(dataset)
&gt;&gt;&gt; [1, 2, 3]
</code></pre>

user_123454321: <p>Actually, I have been using this discrepancy in the actual length of the iterable vs what is returned in <strong>len</strong> for randomly selecting equal number (say 300) of  augmented data points  for an epoch disregarding the actual number of images using something like</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
  def __init__(self):
      self.samples = list(range(100))
  def __getitem__(self, index):
      act_index = random.randint(0, len(self.images))
      return self.samples[act_index]
  def __len__(self):
      return 300
</code></pre>
<p>This way I get to always have 300 points from the dataloader.</p>

Versus: <p>This may work under certain circumstances. But I think the for loops and <code>list(dataset)</code> will be infinite and there is no IndexError, hence, no end to the iteration.</p>
<p>PS. Assuming the implementation is like this<br>
<code>act_index = random.randint(0, len(self.images)-1)</code><br>
Minus 1 is missing.</p>

user_123454321: <p>Yeah, it would go infinite, but I don’t call list on it. And considering the number of images, I think I would crash RAM in the original (non-infinite) case.</p>

Versus: <p>In fact, for this use case I’d recommend using Sampler. It is meant to define the item picking strategy. So, it is more in line with Pytorch code.<br>
<a href="https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler</a></p>