Bottleneck on "decode", "resize", and "read"

igorkf: <p>Hello.</p>
<p>I’m training a regression task (output values between 0 and 100), and the inputs are images from plants. I’m using <code>resnet18</code> here from <code>torchvision</code>.</p>
<p>I realized the GPU is going in maximum to ~40% utilization, but the problem is that it usually stays at 0%.<br>
So I thought there was a bottleneck in the data loading/preprocessing steps, and used <code>python3 -m torch.utils.bottleneck src/train.py</code> to check.</p>
<p>Here are the results:</p>
<pre><code class="lang-auto">   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   121039  570.056    0.005  570.056    0.005 {method 'decode' of 'ImagingDecoder' objects}
    26068  180.834    0.007  180.834    0.007 {method 'resize' of 'ImagingCore' objects}
  1102194  104.741    0.000  104.741    0.000 {method 'read' of '_io.BufferedReader' objects}
    26068   14.408    0.001   14.408    0.001 {built-in method PIL._imaging.new}
    28838    7.336    0.000    7.336    0.000 {method 'to' of 'torch._C._TensorBase' objects}
     8160    5.703    0.001    5.703    0.001 {built-in method torch.conv2d}
    26071    4.836    0.000    4.836    0.000 {built-in method io.open}
    66454    4.492    0.000    4.492    0.000 {method 'item' of 'torch._C._TensorBase' objects}
      816    4.220    0.005    4.220    0.005 {built-in method torch.stack}
    26068    2.800    0.000    2.800    0.000 {method 'contiguous' of 'torch._C._TensorBase' objects}
    26068    2.656    0.000    2.656    0.000 {method 'close' of '_io.BufferedReader' objects}
      326    2.234    0.007    2.234    0.007 {method 'run_backward' of 'torch._C._EngineBase' objects}
    26068    2.161    0.000    2.161    0.000 {method 'div' of 'torch._C._TensorBase' objects}
    52136    1.247    0.000  590.483    0.011 /home/igorf/.conda/envs/my-env/lib/python3.8/site-packages/PIL/ImageFile.py:155(load)
    52136    0.999    0.000    4.623    0.000 /home/igorf/.conda/envs/my-env/lib/python3.8/site-packages/pandas/core/internals/managers.py:1027(fast_xs)
</code></pre>
<p>The “cumtime” shows high values for <code>decode</code>, <code>resize</code>, and <code>read</code>, but I don’t know where this <code>decode</code> is in my code. For <code>resize</code> I suppose it comes from <code>torchvision's Resize()</code> and for <code>read</code> it must be from <code>PIL</code>.</p>
<p>I would like to understand how I can make my model run faster from this output.</p>
<p>My dataset is as follows:</p>
<pre><code class="lang-auto">from PIL import Image

from torch.utils.data import Dataset
from torchvision import transforms


class CGIARDataset(Dataset):
    def __init__(self, df, transform=None):
        self.df = df
        self.transform = transform

    def __len__(self):
        return len(self.df)
    
    def __getitem__(self, idx):
        y = self.df.iloc[idx]['extent']
        img = Image.open(self.df.iloc[idx]['filename'])
        x = transforms.ToTensor()(img)
        if self.transform is not None:
            x = self.transform(x)
        return x, y
</code></pre>
<p>As you can see, I’m reading the image using <code>PIL</code>, and converting to tensor using <code>ToTensor</code> from <code>torchvision</code>.</p>
<p>The resize step is in my transform object:</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Resize(IMG_SIZE, antialias=True)
])
</code></pre>
<p>Could anyone give me some tips on this?<br>
For example, should I change my image reading from <code>PIL</code> to another library?<br>
Or where does the <code>decode</code> come from?</p>
<p>Thanks!</p>

ptrblck: <p>If multiple workers in the <code>DataLoader</code> don’t help you could replace <code>PIL</code> with <code>PIL-SIMD</code> which could accelerate the decoding and transformation steps.</p>

igorkf: <p>Thanks for your reply!</p>
<p>I installed <code>PIL-SIMD</code> as recommended <a href="https://github.com/uploadcare/pillow-simd#installation" rel="noopener nofollow ugc">here</a> but I got the following error when running my training script:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "src/train_coral.py", line 9, in &lt;module&gt;
    from torchvision import transforms
  File "/home/igorf/.conda/envs/my-env/lib/python3.8/site-packages/torchvision/__init__.py", line 6, in &lt;module&gt;
    from torchvision import datasets, io, models, ops, transforms, utils
  File "/home/igorf/.conda/envs/my-env/lib/python3.8/site-packages/torchvision/datasets/__init__.py", line 1, in &lt;module&gt;
    from ._optical_flow import FlyingChairs, FlyingThings3D, HD1K, KittiFlow, Sintel
  File "/home/igorf/.conda/envs/my-env/lib/python3.8/site-packages/torchvision/datasets/_optical_flow.py", line 10, in &lt;module&gt;
    from PIL import Image
  File "/home/igorf/.conda/envs/my-env/lib/python3.8/site-packages/PIL/Image.py", line 89, in &lt;module&gt;
    from . import _imaging as core
ImportError: /home/igorf/.conda/envs/my-env/lib/python3.8/site-packages/PIL/_imaging.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_CheckBuffer
</code></pre>
<p>Do you know what is the problem?</p>
<p>I’m using an HPC cluster (CentOS 7).</p>

ptrblck: <p>No, unfortunately I haven’t seen this error before and don’t know what might be causing it. Note that I’m using <code>PIL-SIMD</code> myself and didn’t run into any issues.</p>

igorkf: <p>Did you install with the “AVX2-enabled version” as stated <a href="https://github.com/uploadcare/pillow-simd#installation" rel="noopener nofollow ugc">here</a>?</p>

ptrblck: <p>Yes, I pass <code>CC="cc -mavx"</code> to the build command.</p>

igorkf: <p>I also passed the flag to install the AVX-2-enable version, but didn’t work.</p>
<p>To give you some updates:<br>
I managed to solve the problem by saving resized images to the disk using <code>Image.open(path).resize(IMG_SIZE)</code> from <code>PIL</code>.</p>
<p>Now the training script is reading directly from a folder with resized images and it’s way faster: before it was ~4min per epoch, and now it’s roughly 1min, which helped a lot to experiment with new ideas.<br>
Moreover, the GPU utilization is good now, so I think the resizing step was struggling with my training.</p>
<p>Thanks for your help and I also hope this helps future readers.</p>