Data Loader First Batch from each epoch is slow

BadTimeManagement: <p>Can someone explain why every first batch from each is slow?<br>
For example, if I run with following setting on cifar dataset</p>
<pre><code class="lang-auto">	trainloader = DataLoader(training_set, batch_size = 3, 
			shuffle = True, num_workers = 3)
</code></pre>
<p>The output I get was</p>
<pre><code class="lang-auto">Epoch : 1
----------------------------
Response Time : 0.10336828231811523 , worker : 1
Response Time : 0.03924226760864258 , worker : 1
Response Time : 0.052294015884399414 , worker : 1
Response Time : 0.04296731948852539 , worker : 1
Response Time : 0.04889869689941406 , worker : 1
Response Time : 0.05538439750671387 , worker : 1
Response Time : 0.11520791053771973 , worker : 0
Response Time : 0.054024457931518555 , worker : 0
Response Time : 0.04679131507873535 , worker : 0
Response Time : 0.05366873741149902 , worker : 0
Response Time : 0.046289920806884766 , worker : 0
Response Time : 0.052154541015625 , worker : 0
Response Time : 0.11386680603027344 , worker : 2
Response Time : 0.05026078224182129 , worker : 2
Response Time : 0.042822837829589844 , worker : 2
Response Time : 0.053331613540649414 , worker : 2
Response Time : 0.0486750602722168 , worker : 2
Response Time : 0.059424638748168945 , worker : 2
Train: Time taken to load batch 1 is 0.41985082626342773
Train: Time taken to load batch 2 is 0.0005145072937011719
Train: Time taken to load batch 3 is 0.0005793571472167969
Epoch : 1 , Total Time Taken : 0.42101407051086426
Epoch : 2
----------------------------
Response Time : 0.07382702827453613 , worker : 2
Response Time : 0.0440211296081543 , worker : 2
Response Time : 0.038907766342163086 , worker : 2
Response Time : 0.04715609550476074 , worker : 2
Response Time : 0.039115190505981445 , worker : 2
Response Time : 0.0476069450378418 , worker : 2
Response Time : 0.07493948936462402 , worker : 0
Response Time : 0.03332066535949707 , worker : 0
Response Time : 0.05902433395385742 , worker : 0
Response Time : 0.06337189674377441 , worker : 0
Response Time : 0.03606534004211426 , worker : 0
Response Time : 0.032224416732788086 , worker : 0
Response Time : 0.10121726989746094 , worker : 1
Response Time : 0.03180074691772461 , worker : 1
Response Time : 0.0492551326751709 , worker : 1
Response Time : 0.04465985298156738 , worker : 1
Response Time : 0.03739476203918457 , worker : 1
Response Time : 0.044376373291015625 , worker : 1
Train: Time taken to load batch 1 is 0.3801271915435791
Train: Time taken to load batch 2 is 0.009565353393554688
Train: Time taken to load batch 3 is 0.0005276203155517578
Epoch : 2 , Total Time Taken : 0.3902895450592041
Epoch : 3
----------------------------
Response Time : 0.053179264068603516 , worker : 2
Response Time : 0.0395512580871582 , worker : 2
Response Time : 0.033089399337768555 , worker : 2
Response Time : 0.04987502098083496 , worker : 2
Response Time : 0.0494382381439209 , worker : 2
Response Time : 0.09168267250061035 , worker : 2
Response Time : 0.06848335266113281 , worker : 0
Response Time : 0.06932640075683594 , worker : 0
Response Time : 0.09342813491821289 , worker : 0
Response Time : 0.07962489128112793 , worker : 0
Response Time : 0.0555117130279541 , worker : 0
Response Time : 0.03752279281616211 , worker : 0
Response Time : 0.05451393127441406 , worker : 1
Response Time : 0.07509756088256836 , worker : 1
Response Time : 0.10676288604736328 , worker : 1
Response Time : 0.08155059814453125 , worker : 1
Response Time : 0.05485701560974121 , worker : 1
Response Time : 0.036406517028808594 , worker : 1
Train: Time taken to load batch 1 is 0.44634461402893066
Train: Time taken to load batch 2 is 0.0030670166015625
Train: Time taken to load batch 3 is 0.0004851818084716797
Epoch : 3 , Total Time Taken : 0.44997358322143555
</code></pre>
<p>From the output, shouldn’t each worker loads three samples? Why do each worker load 6 samples?<br>
I’m very confused. Any help is appreciated</p>

ptrblck: <p>The <code>DataLoader</code> iterator is recreated in each epoch and with it all workers, which will all start the prefetching from the beginning.<br>
<a href="https://discuss.pytorch.org/t/enumerate-dataloader-slow/87778/4">Here</a> is a potential workaround for this behavior.</p>

BadTimeManagement: <p>Thanks it does seem a bit faster now. Unrelated to the topic, hope could i ask a different question. I tried passing a list of indices to the dataset however the  <code>__getitem__</code>  in the cifar10/coco dataset <code> returns a Type Error </code>list indices must be integers or slices, not list<code> when running the code. The list of indices is from </code>BatchSampler. Any reason why?</p>

ptrblck: <p>I think <code>torchvision.datasets.CIFAR10</code> is not able to accept a list of indices out of the box, as <a href="https://github.com/pytorch/vision/blob/27278ec8887a511bd7d6f1202d50b0da7537fc3d/torchvision/datasets/cifar.py#L113-L125">this code</a> would try to apply the <code>Image.fromarray</code> method as well as the transformations on a list of images and should yield an error.<br>
You could extend this class as a custom <code>CIFAR10</code> dataset and apply these methods in a loop for each loaded sample.</p>

BadTimeManagement: <p>So something like this would be fine? Would it make the data loader to be slower than normal?</p>
<pre><code class="lang-auto">
def __init(self,data):
        # DO STUFF
 		batch_list = [[],[]]
		c_batches = []

def __getitem__(self,index):
        for data in index:
			result = imagenet_data[data]
			batch_list[0].append(result[0])
			batch_list[1].append(result[1])
		c_batches.append(batch_list)

        return c_batches #&lt;------ This contain a batch

</code></pre>

ptrblck: <p>I don’t think you would see a significant slowdown, as currently each worker is creating a complete batch anyway.<br>
However, a quick profiling would be interesting to see nevertheless.</p>