Save transformed/resized images after dataloader?

username: alx
cooked: <p>Hey guys, I have a big dataset composed of huge images that I’m passing throw a resizing and transformation process.</p>
<p>I would like to save a copy of the images once they pass through the dataloader in order to have a lighter version of the dataset. I haven’t been able to find much on google. Can anyone guide me through this?</p>

username: ptrblck
cooked: <p>You could save each sample using <code>torch.save</code>, if you would like to save the tensors directly.<br>
Note, that random data augmentation methods are applied with random parameters on the fly in your <code>Dataset</code>. If you store these augmented data samples, the transformations will be static now after reloading.<br>
Could you explain your use case a bit and why you would like to store these samples?</p>

username: alx
cooked: <p>Yes, so what I’m trying to do is train a resenet on a kaggle dataset. Unfortunately I don’t have the gpu’s for it so I have to rely on google colab.</p>
<p>As you may know, google colab resets every 12 hours and, in my case, one epoch takes around 1.5 hours.</p>
<p>The setup (downloading and unzipping the kaggle dataset) takes around 3 hours. So at the end of the 12 hours I’m barely able to get to 8 epochs. What I did was backup the weights for every epoch so that way I can reload them once it resets</p>
<p>Since the kaggle pictures are amazingly large, I’m trying to save a copy post transformations in order to be able to load them faster at the next reset.</p>
<p>Hope it makes sense</p>

username: ptrblck
cooked: <p>That would work using my approach, but as I said, you would stick to these “static” transformations/augmentations.<br>
I’m not really that familiar with Colab, but wouldn’t it be possible to download and unzipping the dataset once and store it in your Google Drive?</p>

username: alx
cooked: <p>I see what you mean.</p>
<p>The issue is that the dataset is 80GB which would blow up Google Drive.</p>
<p>Is there a way to save the resized/cropped images to then pass them through the random transforms?</p>
<p>Here’s what I’m using:</p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.6000, 0.3946, 0.6041], [0.2124, 0.2335, 0.2360])
    ]),

</code></pre>

username: ptrblck
cooked: <p>Your transformation does not include any random transforms, so it should be alright.<br>
If your approach is working, you could also add the random augmentation later when reloading the data.</p>
<p>PS: In which foramt is the data stored at the moment? If Kaggle is using some image format with compression (e.g. JPG), your current approach might take more memory than the original dataset.</p>

username: alx
cooked: <p>Great!</p>
<p>Right now the images are in .jpeg.</p>
<p>What do you suggest?</p>
<p>PS: is <code>torch.save</code> still the solution?</p>

username: ptrblck
cooked: <p>I’m not sure, since you said 80GB will blow up your Google drive.<br>
Where would you like to store the tensors now, if they take more memory than their compressed jpeg equivalent?</p>

username: alx
cooked: <p>On Google Drive. I think once they’re resized + cropped + normalized it will take less than 5GB which is fine.</p>
<p>Right now the images are 2000x3000</p>

username: ptrblck
cooked: <p>Oh yeah, I forgot about the spatial size.<br>
In that case, I would give it a go <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: alx
cooked: <p>Great! Not sure where in the code I should add torch.save.</p>
<p>What should it look like?</p>

username: ptrblck
cooked: <p>You could create a loop using the <code>Dataset</code> (or <code>DataLoader</code>) and just use <code>torch.save</code> inside.<br>
I’m not sure, how to access your Google Drive as a file path, but you would most likely know it already.</p>
<pre><code class="lang-python">dataset = MyDataset(..., transform=transform)

for idx, data, target in enumerate(dataset):
    torch.save(data, 'data_drive_path{}'.format(idx))
    torch.save(target, ...
</code></pre>
<p>This will run only once (and take some time of course).<br>
After executing this loop, you would write a new <code>Dataset</code> to load these tensors instead of the jpeg images for the training case.</p>

username: alx
cooked: <p>Thanks, I’m not sure I have the same setup. Here’s how my code looks:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/asmitapoddar/uncertainty-estimation-DR/blob/master/train.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/asmitapoddar/uncertainty-estimation-DR/blob/master/train.py" target="_blank" rel="nofollow noopener">asmitapoddar/uncertainty-estimation-DR/blob/master/train.py</a></h4>
<pre><code class="lang-py">from __future__ import print_function
from __future__ import division
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
from PIL import ImageFile
import copy
print("PyTorch Version: ",torch.__version__)
print("Torchvision Version: ",torchvision.__version__)

# Top level data directory. Here we assume the format of the directory conforms to the ImageFolder structure
data_dir = "/home/asmita/DRimages"

# Models to choose from [resnet, alexnet, vgg, squeezenet, densenet, inception]
</code></pre>

  This file has been truncated. <a href="https://github.com/asmitapoddar/uncertainty-estimation-DR/blob/master/train.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: ptrblck
cooked: <p>This approach should work:</p>
<ul>
<li>create a new script using your current <code>ImageFolder</code> approach and write a single loop over the complete training and validation dataset to store each elements in your drive (in the corresponding train/val folder)</li>
<li>Write a custom <code>Dataset</code> to load the tensors directly instead of the images via <code>ImageFolder</code>
</li>
<li>use the custom <code>Dataset</code> in your <code>train.py</code>
</li>
</ul>

username: raymongL
cooked: <p>I need to save the transformed images for better efficiency too. I tried saving the transformed tensors by <code>torch.save</code>, but a [3, 224, 224] image tensor takes about 100M memory? That’s irrational, why?</p>
<p>I am handling an image dataset with 100k images. I have tried saving the transformed image tensors into .jpg and .png images, they all look good. Then i use <code>Image.open(imageFile).convert('RGB')</code> and <code>transforms.ToTensor()</code> to read the saved images, but i can’t get the correct tensor. It’s different from the original transoformed image tensor, how did that happen and how can i get the correct tensor?</p>