How to feed patches of a large resolution image for classification

Droid786: <p>I am trying to do a image classification problem. I want to keep the original image size and just feed the patches of an images and then classify their category.<br>
I am quite unclear on few points-<br>
1)How to extract the patches, I have already resized the image from different aspect ratio to  2k,2k image?<br>
2)what will be the optimised way to train it, to extract it at the run time or just feed the patches made beforehand?<br>
3)How the inference will pan out in such case, I want to build a interface which accepts the complete image and not just a patch?</p>

ptrblck: <p>Could you explain the general use case a bit more?<br>
What would the final prediction be? Would you apply some kind of voting using the predictions for all patches of the current image or would each patch get a prediction and you wouldn’t care about the original image?</p>
<p>Based on the use case, there might be different approaches.<br>
E.g. you could use <code>torchvision.transforms.FiveCrop</code> to create 5 patches at predefined positions as a preprocessing step or you could use <code>unfold</code> to create the patches manually.</p>

Droid786: <p>Thanks for replying,</p>
<p>Yes, I am trying to find the certain textures of an image and depending on the texture then classifying it in 3 cases.<br>
I want to try both approaches i.e voting based approach and also each patches getting a prediction(as I have very small dataset) and then test which one is better?</p>

ptrblck: <p>I don’t think that one of these approaches is better, as you could achieve the same output.<br>
<code>unfold</code> would give you more flexibility, but if you want to use the 4 corner patches and a center one, <code>FiveCrop</code> would be easier than writing it manually.</p>

Droid786: <p>Is there a faster way to achieve that.<br>
I am using <code>FiveCrop</code>for size <code>224</code> but it is taking 35 minutes for 1 epoch.<br>
Also, how the testing will pan out in this case. Right now, I am doing the <code>RandomCrop</code> of the same <code>224</code> size?</p>

ptrblck: <aside class="quote no-group" data-username="Droid786" data-post="5" data-topic="86936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9de053/48.png" class="avatar"> Droid786:</div>
<blockquote>
<p>Is there a faster way to achieve that.<br>
I am using <code>FiveCrop</code> for size <code>224</code> but it is taking 35 minutes for 1 epoch.</p>
</blockquote>
</aside>
<p>Is <code>FiveCrop</code> slowing down the code and if so how long is an epoch taking without this transformation, i.e. for a single input only?</p>
<aside class="quote no-group" data-username="Droid786" data-post="5" data-topic="86936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9de053/48.png" class="avatar"> Droid786:</div>
<blockquote>
<p>Also, how the testing will pan out in this case. Right now, I am doing the <code>RandomCrop</code> of the same <code>224</code> size?</p>
</blockquote>
</aside>
<p>I’m not familiar with your use case and thus asked, if you could describe the general use case a bit.</p>
<p>E.g. if the image should get a single prediction, you could use the majority prediction of all 5 patches.<br>
If all predictions are different, you could try to use the prediction with the highest logit.</p>

Droid786: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="86936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Is <code>FiveCrop</code> slowing down the code and if so how long is an epoch taking without this transformation, i.e. for a single input only?</p>
</blockquote>
</aside>
<p>Yes, if I am using <code>RandomCrop</code> for size 224 then its takes 5 minutes for 1 epochs while 35 minutes is the output for other.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="86936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m not familiar with your use case and thus asked, if you could describe the general use case a bit.</p>
<p>E.g. if the image should get a single prediction, you could use the majority prediction of all 5 patches.<br>
If all predictions are different, you could try to use the prediction with the highest logit.</p>
</blockquote>
</aside>
<p>Okay, Let me explain,  So I am solving a multi-class image classification problem. I have 3 classes and I am using Pytorch’s <code>ImageFolder</code>  data loader so the labels are <code>0,1,2</code>.<br>
My image size are (2k,2k), I can’t downsample the dataset because each distinct class image differs because of the texture of the image and downsampling making all the images look same.<br>
So first, I resorted to <code>RandomCrop</code>, validation loss is skyrocketing high using the <code>RandomCrop</code>.<br>
Then I moved to <code>FiveCrop</code> which is moving the validation loss in the range of training loss.<br>
I am using the implementation of <code>FiveCrop</code> from the Pytorch documentation -<br>
Below -</p>
<pre><code class="lang-auto">transform = Compose([
&gt;&gt;&gt;    FiveCrop(size), # this is a list of PIL Images
&gt;&gt;&gt;    Lambda(lambda crops: torch.stack([ToTensor()(crop) for crop in crops])) # returns a 4D tensor
&gt;&gt;&gt; ])
&gt;&gt;&gt; #In your test loop you can do the following:
&gt;&gt;&gt; input, target = batch # input is a 5d tensor, target is 2d
&gt;&gt;&gt; bs, ncrops, c, h, w = input.size()
&gt;&gt;&gt; result = model(input.view(-1, c, h, w)) # fuse batch size and ncrops
&gt;&gt;&gt; result_avg = result.view(bs, ncrops, -1).mean(1) # avg over crops
</code></pre>
<p>Personally, I want the inference to watch every part of image and then give the prediction but I don’t know how it will pan out in this case as image size is very large.<br>
Please, let me know if I explained my use-case alright or not.(I’ll describe again)</p>

ptrblck: <aside class="quote no-group" data-username="Droid786" data-post="7" data-topic="86936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9de053/48.png" class="avatar"> Droid786:</div>
<blockquote>
<p>Yes, if I am using <code>RandomCrop</code> for size 224 then its takes 5 minutes for 1 epochs while 35 minutes is the output for other.</p>
</blockquote>
</aside>
<p>Could you profile the data loading as done in the <a href="https://github.com/pytorch/examples/blob/59caa16a29476f290d2e4d38162a89372e5ccb15/imagenet/main.py#L280">ImageNet example</a> to narrow down which part of the training gets the massive slowdown?</p>

Droid786: <p>I was using <code>num_workers = 1</code>. That was the bottleneck, it is fixed now.</p>

Droid786: <p>How does prediction works in <code>FiveCrop</code>. Is it majority of the class obtained? I have 3 classes, isn’t there a chance of tie if the majority of the class is the prediction or each prediction of each crop is considered an output?</p>

ptrblck: <p><code>FiveCrop</code> doesn’t automatically change the prediction so you would have to implement a voting logic.</p>
<p>You could use e.g. a majority voting (and in a tie situation pick the prediction(s) with the highest logic), average all output logits and get a single prediction, or probably even train another classifier on top of your predictions and treat your current model as an ensemble model.</p>