Make patches from images with their masks and reconstruct those images

NoobCoder: <p>Hello,</p>
<p>I have large images with their masks. I want to extract overlapping patches, then feed them to the network and at the end reconstruct the masks to calculate the loss based on the whole image.</p>
<p>Because of that I cannot load the whole data in one dataloader since I will lose which patch belongs to which image.</p>
<p>Ideally, I want to make a folder for each image, put the patches and masks to that folder, then feed the batches of that folder to the model, then reconstruct the images and calculate the loss.</p>
<p>I would appreciate it if someone can help me.</p>

ptrblck: <p>To create the patches you could use <code>nn.Unfold</code> or call <code>unfold</code> directly on the tensor.<br>
The image could be reconstructed using <code>nn.Fold</code>. Note that overlapping areas will be accumulated so you might need to normalize these areas separately.</p>
<p>Once you have the patches you could store them in separate folders and create a custom <code>Dataset</code> to load the patches from each folder using the passed index. Assuming you would like to load all patches into a batch, you should use <code>batch_size=1</code> in your <code>DataLoader</code> and create the complete batch in your <code>Dataset.__getitem__</code> method.</p>
<p>Alternatively, you could also load each image completely and create the patches in the <code>__getitem__</code>. This would avoid the previous step of storing the patches separately.</p>

NoobCoder: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your answer.<br>
Before you answer my question, I have written a dataloader that l load all the data (images) create batches from that (not a tensor, just a simple RGB image), then for each batch, I will save the image name and the number of batches, so at the end, I can know which patch belongs to which image and what is the number. I do not know if it is a efficient way or not.</p>
<p>For your solution, when we have folders, you mean in <code>Dataset</code>, in <code>__init__</code>, I load all the folders which have batches?<br>
Can you elaborate more? you mean I should have <code>n</code> dataloader which <code>n</code> is the number of images? and another thing is that for example for each image, I will create 1000 batches, then how can I put these 1000 patches into a batch_size of 32 and do not lose which patch belongs to which image?</p>
<p>Again, thanks for your time.</p>

ptrblck: <p>I’m not sure if I misunderstand the explanation, but could you explain this a bit more?</p>
<aside class="quote no-group" data-username="NoobCoder" data-post="3" data-topic="82400">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9bcb4/48.png" class="avatar"> NoobCoder:</div>
<blockquote>
<p>I have written a dataloader that l load all the data (images) create batches from that (not a tensor, just a simple RGB image), then for each batch, I will save the image name and the number of batches, so at the end, I can know which patch belongs to which image and what is the number.</p>
</blockquote>
</aside>
<p>How are you determining the number of batches and how do you store it with the image name?</p>
<aside class="quote no-group" data-username="NoobCoder" data-post="3" data-topic="82400">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9bcb4/48.png" class="avatar"> NoobCoder:</div>
<blockquote>
<p>and another thing is that for example for each image, I will create 1000 batches, then how can I put these 1000 patches into a batch_size of 32 and do not lose which patch belongs to which image?</p>
</blockquote>
</aside>
<p>Would you like to create batches with patches from mixed images or should a batch only contain patches from the same image?<br>
In the latter case: should these patches from the same image be shuffled or do you want a specific order?</p>

NoobCoder: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>For this question, I have written a python code that given an image, stride, and filter size, I will extract overlap patches from each image. Now in my custom <code>Dataset</code>, in <code>__init__</code>, I will load the image folder. Instead of returning the path of images, I will return a list of NumPy array that has the patches. Also, I will return the name of the image that each patch is extracted from, and the id of patches. In below, I mention code to understand a bit more.</p>
<blockquote>
<p>Question: How are you determining the number of batches and how do you store it with the image name?</p>
</blockquote>
<blockquote>
<p>Answer: CustomDataset:<br>
<strong>init</strong>(path_to_image, patch_size, stride):<br>
for each image in <code>path_to_image </code>:</p>
<ol>
<li>Run the extract_patches.py</li>
<li>Add this patches(numpy arrays) to a list</li>
<li>Add the <code>Name of the Image</code></li>
<li>Add <code>Patch ID</code></li>
</ol>
<p>at the end, return the lists. Then in <code>__getitem__</code>, I just return the <code>list[index]</code>.</p>
</blockquote>
<p>The Second Question:</p>
<blockquote>
<p>Question: Would you like to create batches with patches from mixed images or should a batch only contain patches from the same image? In the latter case: should these patches from the same image be shuffled or do you want a specific order?</p>
</blockquote>
<blockquote>
<p>Answer: No, there is no need to be the same image, but I want to know which patch belongs to which image and be able to reconstruct the whole image.</p>
</blockquote>
<p>I have another question too. let’s assume for each image, I have two labels {Mask, Edge} which are images too. The output of the model is the mask and based on the mask I am going to extract the patches. I want to write a custom <code>Loss Function</code>. Based on other discussions, as long as I am working with <code>Variables</code>, I can write the function.<br>
But in my case, because I have to calculate the edge based on mask (which needs some filtering and …) I have to access the <code>.data()</code> of <code>Variable</code>, so I am a bit confused about how to do that. Can you help with this part too? The loss should be in this format:</p>
<blockquote>
<p>CustomLoss(Model_Output, TruthMask, TruthEdge):</p>
<ol>
<li>extract data by Model_Output.data().cpu()</li>
<li>run edge_extraction.py on the previous data</li>
<li>calculate the difference between truth and predicted {edge, mask}</li>
</ol>
</blockquote>
<p>Thanks in advance.</p>

ptrblck: <aside class="quote no-group" data-username="NoobCoder" data-post="5" data-topic="82400">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9bcb4/48.png" class="avatar"> NoobCoder:</div>
<blockquote>
<p>No, there is no need to be the same image, but I want to know which patch belongs to which image and be able to reconstruct the whole image.</p>
</blockquote>
</aside>
<p>In that case you could just return the image name or a specific id for each patch and the corresponding mask in the <code>__getitem__</code> method.<br>
Since each model output will then have the corresponding image name, you should be able to sort these outputs according to the original image name.</p>
<aside class="quote no-group" data-username="NoobCoder" data-post="5" data-topic="82400">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9bcb4/48.png" class="avatar"> NoobCoder:</div>
<blockquote>
<p>Based on other discussions, as long as I am working with <code>Variables</code> , I can write the function.</p>
</blockquote>
</aside>
<p><code>Variables</code> are deprecated since PyTorch <code>0.4</code> so you should use tensors in newer versions.</p>
<aside class="quote no-group" data-username="NoobCoder" data-post="5" data-topic="82400">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9bcb4/48.png" class="avatar"> NoobCoder:</div>
<blockquote>
<p>But in my case, because I have to calculate the edge based on mask (which needs some filtering and …) I have to access the <code>.data()</code> of <code>Variable</code> , so I am a bit confused about how to do that.</p>
</blockquote>
</aside>
<p>You shouldn’t access the <code>.data</code> attribute, as it may yield unwanted side effects. If you need the post-process the output with an edge detection algorithm, you could either implement it in PyTorch directly or use another library and implement the <code>backward</code> method via a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>