Opinion about Batch Preprocessing vs Real Preprocessing for images

username: AlexandreBrown
cooked: <p>Hello,<br>
I’d like your opinion on some approaches for applying preprocessing to images for deep learning (eg: semantic segmentation).</p>
<h1>
<a name="important-note-1" class="anchor" href="#important-note-1"></a>Important note :</h1>
<ul>
<li>Take for account that this would be performed in a pipeline therefore every new training would re-apply this (consider the disk to be emptied between training for instance). Also take for granted that we have access to cpus and gpus.</li>
</ul>
<h1>
<a name="approach-1-batch-transforms-real-time-data-augmentation-2" class="anchor" href="#approach-1-batch-transforms-real-time-data-augmentation-2"></a>Approach 1 : Batch Transforms, Real time data augmentation</h1>
<p>This approach would essentially apply the transforms like resize, rescale (/255) and toTensor in a batch and save the transformed images on disk for training.<br>
Then during training, only the data augmentation would be left to be applied (for the training set only).</p>
<p>TLDR: Transforms (images) =&gt; onDisk =&gt; Training + Data aug on the fly</p>
<h1>
<a name="approach-2-batch-transforms-batch-data-augmentation-3" class="anchor" href="#approach-2-batch-transforms-batch-data-augmentation-3"></a>Approach 2 : Batch Transforms, Batch data augmentation</h1>
<p>This approach would apply the transforms like resize, rescale (/255),toTensor and the data augmentation for the training set then save it on disk for training.<br>
Then during training, we’d use a dataset with transforms=None meaning no Transforms are applied, it would simply read the already processed images.</p>
<p>TLDR : Transforms(images) =&gt; Data Aug(images) =&gt; onDisk =&gt; Training</p>
<h1>
<a name="approach-3-batch-data-augmentation-real-time-training-transforms-4" class="anchor" href="#approach-3-batch-data-augmentation-real-time-training-transforms-4"></a>Approach 3 : Batch Data augmentation, Real time Training + Transforms</h1>
<p>Same principle but for data aug (for training set)</p>
<p>Data augmentation (images) =&gt; onDisk =&gt; Training + Transforms on the fly</p>
<h1>
<a name="approach-4-real-time-transforms-real-time-data-augmentation-5" class="anchor" href="#approach-4-real-time-transforms-real-time-data-augmentation-5"></a>Approach 4 : Real time transforms, real time data augmentation</h1>
<p>Here nothing is saved to disk besides the raw train/valid/test images.</p>
<p>Training + Transforms + Data augmentation on the fly.</p>
<p>Important things to keep in mind :</p>
<ul>
<li>Keep in mind that we could decide to perform preprocessing on the cpu or gpu if we use any batch approach, this might or might not be advantageous.</li>
<li>We also might want to consider if having them on disk is useful to save it for reproducibility and lineage (in case we want to go back from a model in production for instance and see what it was trained on).</li>
<li>We also must keep in mind that if we perform the data aug on the fly then it is re-applied after each epoch therefore this might or might not improve model accuracy.</li>
</ul>
<p>So what do you guys think ?</p>

username: ptrblck
cooked: <ol>
<li>
<p>This sounds like a valid approach, as the data augmentation is applied on each sample during training and could potentially save some preprocessing time. In any case I would profile the image decoding (e.g. JPEG decoding) vs. loading the raw binary data (speed and file size). E.g. loading a <code>1200x1600</code> JPEG encoded file with a size of <code>~310kB</code> results in a tensor of <code>~23MB</code>, since raw pixels are stored if you are not resizing it.</p>
</li>
<li>
<p>This approach wouldn’t use any data augmentation and I would then claim it’s invalid. Even if you are applying it once, this could be seen as creating a “new” dataset (with flipped, cropped etc. images) which won’t be randomly transformed during training.</p>
</li>
<li>
<p>Same as 1. with the difference to store the augmented images on disk, which I believe shouldn’t yield any advantage.</p>
</li>
<li>
<p>Standard approach, so valid.</p>
</li>
</ol>