MultiGPU Dataloader numpy to gpu and tensor to gpu different on CPU usage

poodarchu: <p>Hi all,</p>
<p>I encounter a strange problem:</p>
<p>Previously, I define my collat_batch function to collect preprocessed data in numpy format, and set pin_memory = Ture; In the training loop, I iter a batch, then move the batch from numpy to gpu device; This way couldn’t make full use of cpu so that even GPU is waiting for data, cpu isn’t running fully.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7c594ccad2256058173f197084212c9fd5211a40.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c594ccad2256058173f197084212c9fd5211a40" title="30%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7c594ccad2256058173f197084212c9fd5211a40_2_690x281.png" alt="30%20PM" data-base62-sha1="hK2wTM2ifDEQWpasjzJPu2NgzCM" width="690" height="281" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7c594ccad2256058173f197084212c9fd5211a40_2_690x281.png, https://discuss.pytorch.org/uploads/default/original/2X/7/7c594ccad2256058173f197084212c9fd5211a40.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/7c594ccad2256058173f197084212c9fd5211a40.png 2x" data-dominant-color="34322E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">30%20PM</span><span class="informations">958×391 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then I change the above process, I define a collate_batch_torch function to collect preprocessed data in numpy format then convert it to torch.tensor, set pin_memory = True; In the training loop, I iter a batch, then move the batch from torch.tensor to device; This can make cpu running fully, but the step time is much slower; so it seems that this modification make cpu the bottelneck.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9c0f9cfbbd62ab71610dfd4ed314ccba838beee8.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c0f9cfbbd62ab71610dfd4ed314ccba838beee8" title="39%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c0f9cfbbd62ab71610dfd4ed314ccba838beee8_2_453x500.png" alt="39%20PM" data-base62-sha1="mgzW46sPOXbBExFpl0NC50QzxB6" width="453" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c0f9cfbbd62ab71610dfd4ed314ccba838beee8_2_453x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c0f9cfbbd62ab71610dfd4ed314ccba838beee8_2_679x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c0f9cfbbd62ab71610dfd4ed314ccba838beee8_2_906x1000.png 2x" data-dominant-color="3F3D34"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">39%20PM</span><span class="informations">958×1056 383 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I test it using one gpu or two gpus, the accelerate rate is linear. for example, using one gpu, time cost per iteration is 1s, two gpu is also 1s, but when I use 8 gpus, time could be 8s.</p>
<p>What’s wrong?</p>

pietern: <p>The bottom image shows a TON of system time= and high system load. At 8 processes you may be spawning a large multiple of data workers that end up overloading your machine. Make sure to tune this to the available cores in your system.</p>