Large Input Size to UNet model without Overlap Patching

luxifeo: <p>Hi, I am trying to make a GAN model that takes an image and output an image. The model converged successfully and I can perform inference. However, my input resolution is so big (6000x8000) and I got not enough memory error. Resizing the image to lower resolution gives not good result. I tried to cut the images into smaller overlapping patches to inference, the result at each patch looked better. But when I merged the patches into the final output, there is a noticeable difference at the border and overlapping of the patches, so the image looked not whole. At present, my only solution is to feed the whole image.</p>
<p>I would like to ask if there is any method to inference on large input size with memory constraints, maybe saving immediate results to other storage like RAM, swap or hard disk. I can accept that the processing time will be much higher.</p>
<p>Any help appreciated.</p>

KFrank: <p>Hi Duc Hoa!</p>
<p>You refer to “UNet model” in your title, so I will reply in that context.</p>
<aside class="quote no-group" data-username="luxifeo" data-post="1" data-topic="201902" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luxifeo/48/69678_2.png" class="avatar"> luxifeo:</div>
<blockquote>
<p>But when I merged the patches into the final output, there is a noticeable difference at the border and overlapping of the patches</p>
</blockquote>
</aside>
<p>In the <a href="https://arxiv.org/abs/1505.04597" rel="noopener nofollow ugc">original U-Net paper</a> the authors have been very careful to design the<br>
U-Net architecture so that their “overlap-tile strategy” gives the exact same<br>
result (up to round-off error) as passing the entire image through the U-Net<br>
all at once.</p>
<p>I would recommend going through that paper carefully to understand how and<br>
why this works and then modify your U-Net, if necessary, to work the same way.</p>
<p>U-Net was specifically designed this way so that it can be used with arbitrarily<br>
large images, so this is the right solution to your problem.</p>
<aside class="quote no-group">
<blockquote>
<p>maybe saving immediate results to other storage like RAM</p>
</blockquote>
</aside>
<p>Yes.  A U-Net saves intermediate “images” as you go down the left side of<br>
the U so that it can recombine them with the corresponding “images” as you<br>
come back up the right side of the U.</p>
<p>You can modify your U-Net code so that you copy each tensor being saved<br>
on the right side of the U to cpu ram (assuming that you have enough of that):</p>
<pre data-code-wrap="python"><code class="lang-python">x_cpu = x_gpu.cpu()
x_gpu = None   # to free x_gpu's gpu memory
</code></pre>
<p>and then copy it back to the gpu when you need it again on the left side of<br>
the U.  (Of course, you still need to be able to fit whatever large “image” you<br>
are actively working on in your gpu.)</p>
<p>Best.</p>
<p>K. Frank</p>

luxifeo: <p>Hi Frank,<br>
Thank you so much for your detailed response. I will modify as your suggestion and review the training and data augmentation process.</p>