Faster Transforms (Precompiled)

username: aauker
cooked: <p>Hi all,</p>
<p>I spent some time tracking down the biggest bottleneck in the training phase, which turned out to be the transforms on the input images.  500-3000 tiles need to be interactively transformed using the below Composition, which takes 5-20 seconds.  I tried a variety of python tricks to speed things up (pre-allocating lists, generators, chunking), to no avail.  I already use multiple workers.  The issue seems to be each individual transforms takes some time, and it really adds up:</p>
<pre><code class="lang-auto">  self.img_finalize = transforms.Compose([
            transforms.ToPILImage(), # 1 s.
            transforms.RandomRotation(25), # 2 s.
            transforms.RandomResizedCrop(1024, scale=(0.5, 1.5), ratio=(.8,1.2), interpolation=2), # 0.5 s.
            transforms.Resize(resolution), # 0.5 s.
            transforms.RandomHorizontalFlip(p=0.5), # 0.5 s.
            transforms.RandomVerticalFlip(p=0.5), # 0.5 s.
            transforms.ColorJitter(brightness=0.2, contrast=0.1, saturation=0.05, hue=0.02), # 3 s.
            transforms.ToTensor(), # 2 s.
            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), # 3 s.
        ])
</code></pre>
<p>Two things:</p>
<ol>
<li>I notice with the dataloader, every thread that was started needs to finish before any data is pushed to the GPU.  If I add a print statement <code>DONE</code> at the end of the <code>__getitem__()</code> method and before the forward pass <code>FOWARD</code>, there is significant delay from the first DONE and first <code>FORWARD</code>, in fact there needs to be 16 (=n_workers) <code>DONE</code>'s</li>
</ol>
<p>This <em>doesn’t</em> happen if n_workers=None.  I suspect some form of lock is implemented?  I am working in the unusual situation where my batchsize is 1, but each batch is huge (it’s a MIL paradigm…). So it would be useful to process a batch asap once it’s <code>__getitem__()</code> method is done.  This makes sense to me is batch_size &gt; 1, as you need to be sure all batches are down before collating.  But it’d be great if it could be relaxed if batch_size =1.</p>
<p>E.g. I see output like this, for example, and I don’t think this is the latency from sending to GPU (I checked that too, it’s much faster than the individual methods).  Now it might be latency from sending from forked process to main?</p>
<pre><code class="lang-auto">DONE!
DONE!
DONE!
DONE!
DONE!
DONE!
DONE!
DONE!
DONE!
FOWARD
</code></pre>
<ol start="2">
<li>Is there any use in pre-compliling the transform?  If there would be significant performance gains (and it’s possible), it’s something I definintely want to explore.  Any advice or helpful links on this?  I have some experience with exposing C++ to python.</li>
</ol>
<p>Thanks!<br>
Andy</p>

username: JuanFMontesinos
cooked: <p>Hi there.<br>
Each worker will process a whole batch. You need to use plenty of CPUs in order to make it to be efficient.<br>
Obviously pre-computing preprocessing leads to overfit unless you  make the system to do so in parallel to the training.</p>
<p>Btw try this library:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/uploadcare/pillow-simd" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/978311?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/uploadcare/pillow-simd" target="_blank" rel="nofollow noopener">uploadcare/pillow-simd</a></h3>

<p>The friendly PIL fork. Contribute to uploadcare/pillow-simd development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
and let me know if you got a good improvement.</p>
<p>Lastly, the “proper” way of knowing if there is a bottleneck is checking gpu utilization. If it’s always &gt;95% you are ok. If you have peaks then you should check what can you do.</p>

username: aauker
cooked: <p>Thanks for the reply.</p>
<p>I know that each worker processes the data defined in the getitem() method- the issue was two fold, the sheer time and the locking method of the dataloader.</p>
<p>I also know the bottleneck is this step because I profiled the usage using cProfile.  Nearly all the compute time is spent in dataloader methods.  Running the script with predefined tensors (putting the full weight on GPU side) is about 20 seconds per epoch, while with the transforms is about 4 minutes.</p>
<p>I guess the dataloader already implements PIL methods, can this pillow-simd be simply swapped?</p>
<p>Andy</p>

username: JuanFMontesinos
cooked: <p>The problem is there is no much you can do about it. Have you check it’s not an IO problem?<br>
PILLOW-SIMD is drop-in, totally compatible with PIL.</p>
<p>BTW, after seeing this line <code>transforms.RandomResizedCrop(1024</code> I realized you are working with high resolution images. That’s probably the reason why it takes ages.<br>
Another thing you can consider is to implement your own transforms for tensors. This way you can preprocess normalization.<br>
You can also crop and resize 1st, this way rotation will take less time</p>

username: aauker
cooked: <p>Yes, it is huge – the input size to the network alone is 2 GB <img src="https://discuss.pytorch.org/images/emoji/apple/grimacing.png?v=9" title=":grimacing:" class="emoji" alt=":grimacing:"></p>
<p>Everything is fine technically.  I was mostly curious if a pure C++ implementation would actually be faster/possible.</p>

username: JuanFMontesinos
cooked: <p>I don’t know so much about PIL underlying code but I assume it’s not very simple.<br>
If you have enough hard disk I would recommend you to use numpy memory map to load the data. If you save normalized data you would save plenty of time, namely: 1s ToPIL, 2s to tensor, 3s normalize.<br>
And probably 1 second on fliping.</p>
<p>Drawback: to implement rotation and colorjitter</p>

username: colesbury
cooked: <aside class="quote no-group" data-username="aauker" data-post="5" data-topic="68527">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aauker/48/17401_2.png" class="avatar"> aauker:</div>
<blockquote>
<p>Everything is fine technically. I was mostly curious if a pure C++ implementation would actually be faster/possible.</p>
</blockquote>
</aside>
<p>The transforms are all implemented in C under the hood. The PyTorch vision transform functions are just wrappers around the PIL (pillow) library and the PIL operations are implemented in C. It’s unlikely (but possible) that the overhead of the Python wrapper pieces are the bottleneck.</p>
<p>As <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> wrote, pillow-simd is faster than pillow. The <a href="https://github.com/pytorch/accimage">accimage</a> library is probably even faster, but only supports a few transforms.</p>
<p>If you’re trying to optimize data loading, you should try to simplify your program as much as possible and measure independent pieces. You’ve already ruled out the GPU part of the network. So temporarily get rid of the DataLoader and multiprocessing which are complicated and:</p>
<ol>
<li>Measure file loading times</li>
<li>Measure the transform times on a single tile</li>
<li>Measure the transform time on a composite (?) image</li>
</ol>
<p>If you’re on Linux, use <a href="http://www.brendangregg.com/perf.html"><code>perf top</code></a> to figure out if the bulk of the time is spent in Python or in the native C code.</p>
<p>From what you wrote, it seems likely that the transforms are just slow and your images are huge. There’s a good chance that some of your data augmentations are not very helpful for your task – try getting rid of some. For example, my experience with ColorJitter is that it had no effect for training ResNet classifiers on ImageNet. I’m also confused by the RandomResizedCrop followed by the Resize – that seems redundant?</p>

username: aauker
cooked: <p><a class="mention" href="/u/colesbury">@colesbury</a> and <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>Thanks for all your input.  I was worth the ask, but it seems clear that for now I’m stuck with how things are.  I will look into pillow-simd though, and accimage.  The unfortunate part is likely that a for loop is required, as no batch transformations exist (yet).</p>
<p>In case you’re curious, the transformations are meant to recreate real and possible changes in the input data.  ColorJitter is useful as these are stained tissue images, so this replicates different laboratory/scanner settings.  RandomResizedCrop is used to effectively change focal properties and Resize is need to standardized the input resolution.</p>
<p>Thanks for all the input, my questions/hopes are answered!</p>

username: achaiah
cooked: <p>I know this isn’t precisely what you asked about but to make your training phase faster you can always prepare your data ahead of time. So, for example, instead of running transforms for every iteration, just run them once on your entire dataset offline and then train with the resulting set. Yes, you will lose the randomness that you get with every single iteration, but in my experience it doesn’t improve things by a huge margin if you are smart about how you generate your training set offline.</p>
<p>You can even do a bit better than that and allow for some randomness by running a few fast transforms (resizes, flips etc) during each iteration on your otherwise offline-generated training set. That’s the approach I’m using and it seems to be a good middle ground.</p>

username: gabrieldernbach
cooked: <p>The pytorch transforms can consume pillow and tensor objects. Does this imply I should stay with pillow objects for as long as possible, as I can then benefit from the pillow-simd?</p>