How to create image crops in dataloader while training instead of saving offline beforehand?

Vivek_Reddy: <p>I have several (2160,2560) images that I am splitting into (64,64) crops to run through my network.  This means each image will generate 1350 crops.<br>
What would be the most efficient way to obtain these crops to run through my network?<br>
Right now, I’m thinking I could generate and save all the crops before training.  I would create several folders that each contain 1350 crops from the image and have the folder name as the name of the original image.  Then, I could just load them all in or use the ImageFolder dataloader.  However, this takes up a lot of time before I can start training.<br>
How can I load the image into memory and create crops on the go without saving them beforehand?<br>
Thank you!</p>

arya47: <p>If you’re looking for random crops in the image, you can use random crop transforms</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://albumentations.ai/docs/api_reference/augmentations/crops/transforms/">
  <header class="source">
      <img src="https://albumentations.ai/docs/api_reference/augmentations/crops/transforms/../../../../images/logo.png" class="site-icon" width="190" height="182">

      <a href="https://albumentations.ai/docs/api_reference/augmentations/crops/transforms/" target="_blank" rel="noopener nofollow ugc">albumentations.ai</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/309;"><img src="https://albumentations.ai/docs/images/albumentations_card.png" class="thumbnail" width="690" height="309"></div>

<h3><a href="https://albumentations.ai/docs/api_reference/augmentations/crops/transforms/" target="_blank" rel="noopener nofollow ugc">Albumentations Documentation - Crop transforms (augmentations.crops.transforms)</a></h3>

  <p>Albumentations: fast and flexible image augmentations</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Vivek_Reddy: <p>I would actually like to take every possible non-overlapping crop of the image.</p>

arya47: <p>What sample size are you working with?</p>

Vivek_Reddy: <p>Do you mean how many images?    I have several hundred images, each of which will need to be split into 1350 sized crops.  There will probably be roughly 300,000 tiles total.</p>

arya47: <p>I would suggest the following approach where you save cropped images</p>
<p>eg. for classfication in two classes</p>
<ol>
<li>Convert all images to numpy array → save [cropped_image_pixelarray, class]</li>
<li>Create a dataset class for loading from saved array</li>
</ol>
<p>You can use similar code for getting cropped images:</p>
<p>Loop over all images, open then as cv2 images and use this function (i have not verified if the function works, but something similar should suffice)</p>
<pre><code class="lang-auto">def process_data(image, label, df):
    '''
    image: cv2 image
    label: numpy array label eg. [0, 1] or [1, 0]
    df: main df
    '''
    image_x = 2160
    image_y = 2560
    crop = 64
    for i in range(math.floor(image_x/crop)): # loop over width 
        for y in range(math.floor(image_y/crop)):  # loop over height
            new_image = image[i*crop : (i+1)*crop, y*crop : (y+1)*crop]
            df.append([np.array(new_image), label]) 

np.save('data.npy', df)
</code></pre>
<p>The Dataset class would look like:</p>
<pre><code class="lang-auto">class Data(Dataset):
    def __init__(self, data_path, transform=None):
        self.data = np.load(data_path, allow_pickle=True)
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, x):
        # we saved data as [image_data, label]
        label = torch.from_numpy(data[x][1])
        image = torch.from_numpy(data[x][0])
        if self.transform is not None:
            image = self.transform(image)
        return image, label
</code></pre>
<p>Hope this helps<br>
thanks</p>

Vivek_Reddy: <p>Thank you for your suggestion.  This involves saving data offline before training begins right?  Is there a way to begin training before the crops are done saving?  Or alternatively,  is there a way to just load the (2160,2560), store its corresponding 1350 crops in memory, and run it that way without saving it beforehand?  This is to save time.</p>

Billy-ZTB: <p>You mean cropping the images in dataloader but not saving those cropped images before training? I wrote a function to implement this idea, you can check if it fits your demand:<br>
def read_images(step, crop_size):<br>
assert step in [‘train’, ‘valid’, ‘test’]<br>
features, labels = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
if step == ‘trai’:<br>
img_path = train_images<br>
mask_path = train_labels<br>
for file in os.listdir(img_path):<br>
feature = Image.open(os.path.join(img_path, file))  # (H,W,C) 6000<em>6000</em>3<br>
label = Image.open(os.path.join(mask_path, file))<br>
width, height = feature.size<br>
stride = 256<br>
for i in range(0, width, stride):<br>
for j in range(0, height, stride):<br>
box = (i, j, i + crop_size, j + crop_size)<br>
cropped_feature = feature.crop(box)<br>
cropped_label = label.crop(box)<br>
features.append(to_tensor(cropped_feature))<br>
labels.append(to_tensor(cropped_label))</p>
<pre><code>elif step == 'valid':
    img_path = valid_images
    mask_path = valid_labels
    for file in os.listdir(img_path):
        feature = Image.open(os.path.join(img_path, file))
        label = Image.open(os.path.join(mask_path, file))
        width, height = feature.size
        stride = 250
        for i in range(0, width, stride):
            for j in range(0, height, stride):
                box = (i, j, i + crop_size, j + crop_size)
                cropped_feature = feature.crop(box)
                cropped_label = label.crop(box)
                features.append(to_tensor(cropped_feature))
                labels.append(to_tensor(cropped_label))
else:
    img_path = test_images
    for file in os.listdir(img_path):
        feature = Image.open(os.path.join(img_path, file))
        width, height = feature.size
        stride = 250
        for i in range(0, width, stride):
            for j in range(0, height, stride):
                box = (i, j, i + crop_size, j + crop_size)
                cropped_feature = feature.crop(box)
                features.append(to_tensor(cropped_feature))
return features, labels
</code></pre>
<p>I used this method to train my model, but I found it occupying a lot of memory. It is not efficient. I have noticed that it has been more than 2 years since you asked this question, please let me know if you have solved it in a proper way, I am wondering how to do it efficiently, thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>