How to have dataloader access a larger image in memory to take small crops from instead of reloading that big image everytime?

Vivek_Reddy: <p>Hi.  When defining how to retrieve my image crops in the <strong>get_item</strong> function of the dataset, I currently have the generator load an image from a list of images and coordinates.  It then loads that image and crops out the coordinates.  Many of these images are the same images being loaded again each retrieval just to take crops from different coordinates.  Is there a way to store the big image in memory so the generator can just look there if it’s available rather than loading it?</p>
<p>batch_df = self.final_cell_df.loc[self.df_index[index * self.batch_size:(index + 1) * self.batch_size]]</p>
<pre><code>    output = []
    for ind, row in (batch_df).iterrows():
        marker_slice = io.imread(row['%s_Path' % self.marker]) #300,300

        marker_crop = marker_slice[row.y - int(self.crop_size[0] / 2):row.y + int(self.crop_size[0] / 2),
                      row.x - int(self.crop_size[1] / 2):row.x + int(self.crop_size[1] / 2)]

        label = np.array(row.Label)
        output.append((marker_crop, label))
</code></pre>
<p>Thank you!</p>

KFrank: <p>Hi Vivek!</p>
<aside class="quote no-group" data-username="Vivek_Reddy" data-post="1" data-topic="147460" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vivek_reddy/48/28909_2.png" class="avatar"> Vivek_Reddy:</div>
<blockquote>
<p>Many of these images are the same images being loaded again each retrieval just to take crops from different coordinates.  Is there a way to store the big image in memory so the generator can just look there if it’s available rather than loading it?</p>
</blockquote>
</aside>
<p>There could be a lot of ways of doing this.</p>
<p>You don’t say how big your “big images” are, how many you have,<br>
whether your coordinates are known in advance or generated on the<br>
fly, or whether you’re running your processing on the gpu or cpu.</p>
<p>Based on the details, you might consider the following approaches:</p>
<p>Plan A:  If you know your cropping coordinates in advance, you could<br>
simply  preprocess you big images and store all of the cropped sub-images<br>
on disk.  (Disk space for this is likely to be your cheapest resource.)</p>
<p>Then just implement the standard <code>Dataset</code> / <code>DataLoader</code> approach<br>
and have the <code>Dataset</code> read the pre-cropped images off disk as needed.</p>
<p>Plan B:  If you know that your <code>DataLoader</code> (or equivalent logic) will be<br>
accessing the sub-images from a given big image all in a row (that is,<br>
something like sub-images 1, 2, 3 from big image A followed by sub-images<br>
1, 2, 3, 4, 5 from big image B, and so on), you can simply cache the big<br>
images.</p>
<p>That is, your <code>DataLoader</code> (or equivalent logic) requests, for example,<br>
big image B, sub-image 3.  Your code checks whether big image B has<br>
already been read in and is still cached in memory,  If not, it reads in<br>
image B into memory (deleting other big images from the cache if<br>
necessary for space reasons).  Now image B is in the cache if it wasn’t<br>
already, so your <code>Dataset</code> (or equivalent logic) crops out sub-image 3<br>
and returns it.  As long as your <code>DataLoader</code> next requests sub-images<br>
4 and 5 from big image B, big image B will still be in the cache and will<br>
not need to be read in again.  Of course, this approach won’t let you use<br>
<code>shuffle = True</code> with your <code>DataLoader</code> – something that you would<br>
typically want to do.</p>
<p>Plan C:  Consider pre-loading (or lazy-loading) your big images into cpu<br>
regular ram memory if you have enough space to fit them all.  Depending<br>
on your hardware, you may well have a lot more cpu memory than gpu<br>
memory.  You still have to crop out your sub-images and move them to<br>
the gpu (or vice versa) which will be slower than if the big image were<br>
already in the gpu, but moving data from the cpu to the gpu is still a lot<br>
cheaper than reading it off the disk.</p>
<p>Plan D:  Write a custom <code>Sampler</code>.  This approach probably gives you the<br>
most control consistent with reusing the same big image multiple times<br>
before reading in a new one.</p>
<p>Pytorch’s data architecture imagines the following division of labor:  The<br>
<code>Dataset</code> knows how to retrieve images (usually from disk) by index or<br>
map key, possibly transforming them along the way.  The <code>DataLoader</code><br>
knows how to iterate over a <code>Dataset</code>, possibly collating the samples<br>
into batches.  The <code>Sampler</code> is a kind of glue between the two that lets<br>
you use more elaborate logic when sampling from the <code>Dataset</code>.  (For<br>
example, <code>WeightedRandomSampler</code> lets you pick different samples with<br>
different probabilities such as when you want to compensate for a data<br>
imbalance.)</p>
<p>Your custom <code>Sampler</code> would know about big images and sub-images.<br>
It could still shuffle the order in which it selects big images and still<br>
shuffle the sub-images within a big image, but when it reads in a big<br>
image, it would then sample all of that big image’s sub-images before<br>
moving on to the next big image, getting the most benefit out of the<br>
expensive read operation.</p>
<p>Good luck.</p>
<p>K. Frank</p>