Looking for an efficient way to load data and pass it to the model for UNet3D

username: Nour_Hacker
cooked: <p>Hi,</p>
<p>I am looking for an efficient way to load data and pass it to my UNet3D model during the training process.</p>
<p>Since I can’t pass whole volumes to my network, I am cropping the volumes and passing those crops to my model.</p>
<p>Btw, I am using SimpleITK to load my volumes and annotations (.nrrd files).</p>
<p>I have tried 2 different approaches:</p>
<ul>
<li>
<p>Preprocessing data and creating crops prior to the training process. This is very efficient in terms of GPU usage but it takes so much disk space to save those crops. My current dataset is not that big, but if I use a bigger dataset then this approach will become unfeasible.</p>
</li>
<li>
<p>Generating metadata about where to crop the volume and mask. Basically, prior to training, I go through my dataset and generate metadata that looks like this:</p>
</li>
</ul>
<pre><code class="lang-auto">                        crop_info = {
                            "image_path": imgpath,
                            "annot_path": annotpath,
                            "z_start": z_start,
                            "z_end": z_end,
                            "y_start": y_start,
                            "y_end": y_end,
                            "x_start": x_start,
                            "x_end": x_end,
                            "pad": pad_width 
                        }
</code></pre>
<p>This second approach is very efficient from disk memory perspective because my metadata does not take that much space at all. During the training, I am iterating over a list of metadata like the one above, then I read the volume image and annotation image and transform them and<br>
finally crop them and pass the crop to my model. Because there is so much I/O operations going on, when I monitor my GPU usage, I see that it goes in spikes mode. Meaning that it goes to 100% usage then 0% then 100 again then 0 and continues like this. So it’s very inefficient from GPU usage perspective.<br>
I tried to find a way where I load the volume and annotation files once and iterate through all the possible crops before going to the next image and annotation but I couldn’t find a proper way to do this.</p>
<p>What is the most efficient approach to both save on disk space and make the most out of the GPU?</p>
<p>Thanks.</p>

username: Tahamustapha_Nehdi
cooked: <p>Hi Nour <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20"><br>
For Unet3D and in term of performance and data efficiency you can try this techniques :</p>
<ul>
<li>patching  : help you split your volumes in small blocks for example (1024,1024,1024) to small (64,64,64) be careful with patching you should be large enough to capture relevant structures in the data but small enough to fit within the memory constraints.</li>
<li>After training, predictions on the patches need to be combined to reconstruct the output for the entire volume.</li>
<li>Try using mixed precision training. This technique will perform the forward pass of your model in 16-bit precision, but with scaling, you won’t lose performance. However, be careful with potential issues like null losses</li>
<li>data transfer from CPU to GPU can make training slow try to set <code>pin_memory=True</code> when using a GPU. This speeds up data transfer from CPU to GPU.</li>
<li>play with your batch size</li>
<li>try to use new pytorch utility torch.complie() which seed up your training</li>
</ul>

username: Soumya_Kundu
cooked: <aside class="onebox githubblob" data-onebox-src="https://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb">
  <header class="source">

      <a href="https://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb" target="_blank" rel="noopener nofollow ugc">Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License.\n",
    "\n",
    "# Fast training with MONAI features"
   ]
  },
</code></pre>



  This file has been truncated. <a href="https://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This is exactly what you are looking for!</p>

username: Nour_Hacker
cooked: <p>I am already doing this <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"> The issue I’m facing is a bit different then what you’re trying to solve here.</p>

username: Nour_Hacker
cooked: <p>Thanks! I’ll take a look at this.</p>