Creating an iterator for `DataLoader` takes too much time

username: Kore_ana
cooked: <p>I have noticed that when the <code>enumerate(loader)</code> is called, considerable time is required, as the code snippet below and the time points show :</p>
<p>code snippiet :</p>
<pre><code class="lang-auto">  start_time = time.time()
    scaler = torch.cuda.amp.GradScaler()
    for epoch in range(start_epoch, args.epochs):
        print(f"rank {args.rank} set epoch start : {time.time() - start_time}")
        sampler.set_epoch(epoch)
        print(f"rank {args.rank} set epoch end : {time.time()- start_time}")
        for step, ((y1, y2), _) in enumerate(loader, start=epoch * len(loader)):
            print(f"rank {args.rank} step start : {time.time()- start_time}")
            y1 = y1.cuda(gpu, non_blocking=True)
            y2 = y2.cuda(gpu, non_blocking=True)

</code></pre>
<p>The time taken was :</p>
<pre><code class="lang-auto">rank 0 set epoch start : 4.7206878662109375e-05
rank 0 set epoch end : 8.702278137207031e-05
rank 0 step start : 72.13791012763977
{"epoch": 0, "step": 0, "lr_weights": 0.0, "lr_biases": 0.0, "loss": 13827.8291015625, "time": 82}
rank 0 step start : 82.55937671661377
{"epoch": 0, "step": 1, "lr_weights": 3.765060240963856e-06, "lr_biases": 9.036144578313253e-08, "loss": 12604.126953125, "time": 83}
rank 0 step start : 83.55196452140808
</code></pre>
<p>As the timepoints above shows, it takes about 70 seconds for the iterator to be made, while only 1 second for each iteration (i.e. step) to be taken.</p>
<p>This delay in the iterator creating amounts to 30% of the total computation time be wasted while not utilizing the GPU.</p>
<p>My questions are :</p>
<ol>
<li>Is this normal behavior?</li>
<li>What could be the cause of this?</li>
<li>Is there way to remedy this?</li>
</ol>
<p>p.s. One potentially pertinent information is that the batch size is 32, with dataloader <code>num_workers = 8</code>.</p>
<p>thank you in advance for any of your insights and replies <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>At the beginning of the <code>DataLoader</code> loop, all workers will spawn, initialize the <code>Dataset</code>s, and start loading their batches. Depending how expensive the <code>Dataset.__init__</code> method is, this could already take some time (especially if you are loading and processing data). Afterwards, each worker will load <code>batch_size</code> samples by calling <code>batch_size</code> times into <code>Dataset.__getitem__</code> and create the batch via the <code>collate_fn</code>.<br>
You should check where the bottleneck in your code is and which part takes the majority of the time.<br>
E.g. data loading from a mounted network drive would most likely be the bottleneck and you should consider copying to the node. Without knowing more about your use case I can only speculate and would recommend to profile the code.</p>

username: Kore_ana
cooked: <p>Thank you for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>My dataset code loads the whole dataset onto RAM during <code>__init__</code> then applies transformations in <code>__getitem__</code> part. Since the <code>__getitem__</code> part only gets run when the dataloader loop is actually being run (as oppose to creating the dataloader iterator), I believe that the cause of my problem is the <code>__init__</code> part of the dataset, which only loads and does basic normalization. I’ll try to move the normalization part of the code to the <code>__getitem__</code> to reduce time. Along with this, are there other ways to solve this?(for ex :  is there a way of spawning the workers and doing <code>dataset.__init__</code> before the actual dataloader iterator is being made to reduce time?)</p>
<p>I have also have some questions  :</p>
<ol>
<li>I thought that since the dataset was created before the dataloader iterator is made, there would be no time spent concerning the dataset during the creation of the dataloader iterator. Is there a way to create multiple same datasets in advance (before creating the dataloader iterator, or while the dataloader iteration is being run (i.e. preparing for the next epoch (new dataloader iterator)) so that the time can be reduced?</li>
<li>I thought that when I set workers as (for example) 4, with batch size 32, each worker loads 8 images so that in total 32 images can be made during <code>collate_fn</code>. However, your response seems to imply that each worker makes their own (whole) batch, then supply them to the GPU for training. If so, am I right in thinking that keeping on increasing the num_workers will at some point have no effect on decreeing data loading time since at some point since the speed of batch loading will be higher than the actual computation time of GPU for each batch?</li>
</ol>