Dataloader returns multiple index

username: BarCodeReader
cooked: <p>Here is a reproducible code:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset
from torch.utils.data.dataloader import DataLoader

class test(Dataset):
    def __init__(self):
        super(test, self).__init__()
        
    def __getitem__(self, idx):
        print(idx)
        return idx

    def __len__(self):
        return 100

test = test()
loader = DataLoader(dataset=test,
                    batch_size=2,
                    num_workers=1)

for data in loader:
    print(data)
    break
</code></pre>
<p>and it will print:</p>
<pre><code class="lang-auto">0
1
2
3
4
5
tensor([0, 1])
</code></pre>
<p>but why the index get printed multiple times? and if you change the batch_size to 1, it will give:</p>
<pre><code class="lang-auto">0
1
2
tensor([0])
</code></pre>
<p>torch==1.5.1<br>
torchvision==0.6.0a0+35d732a</p>

username: BarCodeReader
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry to bother you but if you got some spare time, maybe take a look? Really appreciate</p>

username: ptrblck
cooked: <p>I’m not sure, if I understand the question correctly, but I think you are concerned about the <code>print(idx)</code> line and are wondering why it’s printing so many indices before yielding the first batch?</p>
<p>If you are using multiprocessing, the workers will load multiple batches in the background and add them to a queue.<br>
Since the batch creation in your code snippet is really simple, the background worker is able to fetch 3 batches before the <code>DataLoader</code> loop returns the first <code>data</code> batch.</p>
<p>Setting <code>num_workers=0</code> will only print the <code>0</code> and <code>1</code> indices before returning the batch.</p>

username: BarCodeReader
cooked: <p>yes your understanding of the question is correct.</p>
<blockquote>
<p>Since the batch creation in your code snippet is really simple, the background worker is able to fetch 3 batches before the  <code>DataLoader</code>  loop returns the first  <code>data</code>  batch.</p>
</blockquote>
<p>acually I found this problem is because I have a dataloader with data augmentation and I want to check which image got loaded, then I try to print out the index. Thus I am not sure this is because the dataloader is simple, thus it can fetch 3 items and print out.</p>
<p>so what I worry is, if the batch size is large, say 128, when we load a batch, will it become slower just because we keep generating the index? because I created a notebook to test this and it seems when batch size is large, the index will get printed so many times.</p>
<p>but I just dont know if this kind of “slow” is because we have large batch, or becuase the loader generate index many times thus introduce some extra loading time.</p>
<p>Thanks for your reply.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="BarCodeReader" data-post="4" data-topic="94569">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/barcodereader/48/18717_2.png" class="avatar"> BarCodeReader:</div>
<blockquote>
<p>when we load a batch, will it become slower just because we keep generating the index? because I created a notebook to test this and it seems when batch size is large, the index will get printed so many times.</p>
</blockquote>
</aside>
<p>No, that shouldn’t be the case as the index generation should be really cheap in comparison to the data loading and processing. The batch is calculated by calling <code>__getitem__</code> with <code>batch_size</code> indices.<br>
E.g. for a batch size of 128 the <code>DataLoader</code> would call <code>Dataset.__getitem__</code> 128 times and thus you would see the index output 128 times.</p>
<aside class="quote no-group" data-username="BarCodeReader" data-post="4" data-topic="94569">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/barcodereader/48/18717_2.png" class="avatar"> BarCodeReader:</div>
<blockquote>
<p>but I just dont know if this kind of “slow” is because we have large batch, or becuase the loader generate index many times thus introduce some extra loading time.</p>
</blockquote>
</aside>
<p>Most likely you see the slowdown due to the data loading and processing.<br>
Note that multiple workers would potentially hide this latency in the background after the first iteration started.</p>

username: BarCodeReader
cooked: <p>Thanks for the explaination!</p>
<p>seems i need to set <code>num_workers=0</code>  in order to get the index correctly printed.</p>