The bottleneck on loading a sequence of images to compose the input tensor

username: maudzung
cooked: <p>Hi everyone,</p>
<p>I’m implementing <a href="https://arxiv.org/pdf/2004.09927.pdf" rel="noopener nofollow ugc">the TTNet paper</a>. The input tensor of the model consists of <strong><em>N=9</em></strong> sequential images, so it has a size of (batch_size, 27, H, W).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb44754e1262d94b7fde700c39643d228395a286.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fb44754e1262d94b7fde700c39643d228395a286" title="input_tensor.PNG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb44754e1262d94b7fde700c39643d228395a286_2_690x465.jpeg" alt="input_tensor.PNG" data-base62-sha1="zQOw04wXi9rdkp6exLis57p4lL0" width="690" height="465" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb44754e1262d94b7fde700c39643d228395a286_2_690x465.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb44754e1262d94b7fde700c39643d228395a286.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb44754e1262d94b7fde700c39643d228395a286.jpeg 2x" data-dominant-color="8C8E8C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">input_tensor.PNG</span><span class="informations">722×487 79.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To load an image sequence, I implement the code as below (also <a href="https://github.com/maudzung/TTNet-Realtime-for-Table-Tennis-Pytorch/blob/611835acfd5de2373e24d7044d55cf2ac23586d4/src/data_process/ttnet_dataset.py#L46" rel="noopener nofollow ugc">here</a>)</p>
<pre><code class="lang-auto">origin_imgs = []
for img_path_idx, img_path in enumerate(img_path_list):
    origin_imgs.append(load_raw_img(img_path))
origin_imgs = np.dstack(origin_imgs)  # (1080, 1920, 27)
</code></pre>
<p>I have trained the model with 4 GPUs using <strong>torch.nn.parallel.DistributedDataParallel()</strong>, batch_size=32, n_workers = 8.</p>
<p>However, the loading data phase is a bottleneck. This leads to a very slow training process (around 1 hour/epoch).</p>
<p>In the case of <strong>N=1</strong>, the input tensor has a shape of (batch_size, 3, H, W), training 1 epoch just takes me around 4 minutes.</p>
<p>Please suggest me solutions to boost up the loading data phase.<br>
Thank you.</p>

username: tom
cooked: <p>You do have your data on SSD, right?</p>
<p>Is it the loading itself or augmentation? I usually try to move all augmentation to the GPU except gathering statistics which I do beforehand and store in a tensor.</p>
<p>For the loading:</p>
<ul>
<li>What exact image loading backend are you using?</li>
<li>For PIL, be sure to have the  SIMD version: <a href="https://github.com/uploadcare/pillow-simd" rel="nofollow noopener">https://github.com/uploadcare/pillow-simd</a>
</li>
<li>An alternative backend can be set in TorchVision: <a href="https://pytorch.org/docs/master/torchvision/index.html#torchvision.set_image_backend" rel="nofollow noopener">https://pytorch.org/docs/master/torchvision/index.html#torchvision.set_image_backend</a>
</li>
<li>You could also use OpenCV or so instead of TorchVision.</li>
</ul>
<p>Finally, you ask about the bottleneck of where your training spends its time. This is important and leads to reducing the time per epoch.<br>
You probably are aware, but it is worth reminding ourselves every now and then that other things (most prominently the learning rate) impact the overall runtime to a given target metric by changing the number of epochs needed. (And I say this as someone who  does quite a bit of client work of “benchmark-and-optimize-time-per-epoch”.)</p>
<p>Best regards</p>
<p>Thomas</p>

username: maudzung
cooked: <p>Yes. I load my data from an SSD disk. I’m using OpenCV to load images.</p>
<p>I’d like provide an example of the loading time:</p>
<ul>
<li>
<strong>Loading the sequence of 9 images</strong> <em>(resolution: 1080x1920x3)</em> — time: 0.8175349235534668 second</li>
<li>
<strong>Augmentation</strong> — time: 0.05149197578430176 second</li>
</ul>
<p>That’s why I think the loading raw images phase is the bottleneck.</p>

username: tom
cooked: <p>Right. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<ul>
<li>Is TorchVision+pillow-simd or the other backend faster or slower?</li>
<li>Is it IO or CPU bound?</li>
</ul>
<p>You could try if you can save time by writing tensors of 9 images to disk.<br>
However, in my experience, this can make the disk access a problem. One thing to do is to keep the images as <em>byte</em> tensors (if they have been 0…255) to cut the disk use.<br>
(Also, you don’t measure the just the first batch, right? It is known to be slower.)</p>
<p>Of course, you can also look at at the hardware (for IO: M.2 is significantly faster than SATA for SSDs is one obvious thing, for CPU, more cores, I always want to get one of these fancy CPUs…).</p>
<p>Best regards</p>
<p>Thomas</p>

username: maudzung
cooked: <p>I have solved the problem.<br>
My solutions:</p>
<ul>
<li>Using the <a href="https://github.com/lilohuang/PyTurboJPEG" rel="nofollow noopener">TurboJPEG</a> library to load images instead of OpenCV</li>
<li>Before stacking images to compose an input tensor, we should resize the images to a smaller size.<br>
Thank you so much!</li>
</ul>