Is there a way to handle memory appropriately?

skyunyoo: <p>I’m dealing with MRI data, and I converted these files into numpy files and saved them.<br>
my data consisted of input : [3600, 512, 512] (N,H,W) -1.8G and mask : [3600,8,512,512] (N,classes,H,W)-28.125G.<br>
I’m using U-Net for segmentation and GPU is ‘NVIDIA GeForce RTX 2080Ti’ with 18G ram.</p>
<p>Data has been loaded as follows:</p>
<pre><code class="lang-auto">class trainDataset(torch.utils.data.Dataset):
    def __init__(self, data, target, transform=None):
        self.data = data.astype(np.float32)
        self.data = normalize(data)
        self.target = target.astype(np.float32)
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        
        if self.transform:
            x = self.transform(x)
            
        return x, y
    
    def __len__(self):
        return len(self.data)
</code></pre>
<pre><code class="lang-auto">numpy_data = np.load(image_path+'MRtrain.npy')
numpy_target = np.load(mask_path+'RStrain.npy')

traindataset = trainDataset(numpy_data, numpy_target ,transform = transform)         

trainloader = torch.utils.data.DataLoader(traindataset, batch_size = batch_size, shuffle=True, num_workers=0, pin_memory=False)

</code></pre>
<p>and train loop as follow :</p>
<pre><code class="lang-auto">def fit(epoch,model,data_loader,phase='train',volatile=False):
    if phase == 'train':
        exp_lr_scheduler.step()
        model.train()
    if phase == 'valid':
        model.eval()
    running_loss = 0.0
    for batch_idx , (data,target) in enumerate(data_loader):
        inputs,target = data.cpu(),target.cpu()
        if is_cuda:
            inputs,target = data.cuda(),target.cuda()
        inputs , target = Variable(inputs),Variable(target)
        if phase == 'train':
            optimizer.zero_grad()
            
        output = model(inputs)
        pred = torch.sigmoid(output)
        loss = dice(pred,target)

        running_loss += loss.data.item()
        
        if phase == 'train':
            loss.backward()
            optimizer.step()
    
    loss = running_loss/len(data_loader.dataset)
    
    print('{} Dice_Loss: {:.4f}'.format(
                phase, loss))
    return loss
</code></pre>
<p>Training to 515x515 size will cause cuda memory error.<br>
I think it’s inefficient to deal with numpy file, but is there a good way?</p>

ptrblck: <p>I assume you are running out or memory or are you really seeing a <code>MemoryError</code>?<br>
In the former case, could you try to lower the number of kernels in your UNet and check the memory usage?<br>
The OOM shouldn’t be related to loading the data as numpy arrays.</p>
<p>PS: based on the target shape, I assume you are dealing with a multi-label classification, i.e. each pixel might correspond to multiple classes?</p>

skyunyoo: <p>I got error massage as follow<br>
and memory usage is about 97%(14.8G),<br>
GPU memory usage is about 6.8G :</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-10-f1bcb8a65b63&gt; in &lt;module&gt;
     18     print('-' * 10)
     19     epoch_loss = fit(epoch,model,trainloader,phase='train')
---&gt; 20     val_epoch_loss = fit(epoch,model,validloader,phase='valid')
     21     train_losses.append(epoch_loss)
     22     val_losses.append(val_epoch_loss)

&lt;ipython-input-8-cb42f6cac88b&gt; in fit(epoch, model, data_loader, phase, volatile)
     14             optimizer.zero_grad()
     15 
---&gt; 16         output = model(inputs)
     17         pred = torch.sigmoid(output)
     18         loss = dice(pred,target)

C:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

&lt;ipython-input-5-ef5c656e99e9&gt; in forward(self, x)
     45 
     46         x = self.dconv_up3(x)
---&gt; 47         x = self.upsample(x)
     48         x = torch.cat([x, conv2], dim=1)
     49 

C:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

C:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\upsampling.py in forward(self, input)
    129     @weak_script_method
    130     def forward(self, input):
--&gt; 131         return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners)
    132 
    133     def extra_repr(self):

C:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py in interpolate(input, size, scale_factor, mode, align_corners)
   2561         raise NotImplementedError("Got 4D input, but linear mode needs 3D input")
   2562     elif input.dim() == 4 and mode == 'bilinear':
-&gt; 2563         return torch._C._nn.upsample_bilinear2d(input, _output_size(2), align_corners)
   2564     elif input.dim() == 4 and mode == 'trilinear':
   2565         raise NotImplementedError("Got 4D input, but trilinear mode needs 5D input")

RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 11.00 GiB total capacity; 8.19 GiB already allocated; 174.59 MiB free; 96.81 MiB cached)
</code></pre>
<p>as you’ve advised, lowering the number of kernels in my UNet work! Thanks!</p>
<p>I tried to do a multi-organ classification, but now I want to do one organ classification.<br>
my numpy file shapes are  image : [3600,512,512] , mask : [3600,512,512]<br>
input tensor is  image : [4,512,512] , mask : [4,512,512] divied by batch<br>
but output tensor is [4,1,512,512].<br>
I think class should be 2 because of background+organ, but I don’t know how to include background in mask class.<br>
Do you have a good idea?</p>

ptrblck: <p>Good to hear it’s working now.</p>
<p>Is the input shape defined as <code>[batch_size, height, width]</code> or <code>[channels, height, width]</code>?<br>
In either case, you should pass an input as <code>[batch_size, channels, height, width]</code> to conv layers.<br>
If you are dealing with single channel images, just unsqueeze dim1:</p>
<pre><code class="lang-python">input = input.unsqueeze(1)
</code></pre>
<p>For <code>nn.CrossEntropyLoss</code> (and <code>nn.NLLLoss</code>), the target should contain class indices in the shape <code>[batch_size, height, width]</code>.<br>
I assume your target already contains the background class as some class index (e.g. class0)?<br>
If so, just transform all other classes to the background class:</p>
<pre><code class="lang-python">background_class_index = 0
desired_class_index = 1
target[target!=desired_class_index] = background_class_index
</code></pre>

skyunyoo: <p>Thanks for your Reply!</p>
<p>Unfortunately my target doesn’t contain the background class as some class index.<br>
All of my targets consist of a pixel value 0 (background) or other(desire target).</p>
<p>ex. one of 3x3 size target array consists of<br>
[0 1 0<br>
0 1 1<br>
0 0 1] &lt;- stack 180 slice and batch_size 4 : target shape : [4,3,3] ; [batch_size, height, width]</p>
<p>I this case, If I apply the transform method that you suggested without having to process a pixel with a value of 0, will it have two classes?</p>

ptrblck: <aside class="quote no-group" data-username="skyunyoo" data-post="5" data-topic="50554">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>All of my targets consist of a pixel value 0 (background) or other(desire target).</p>
</blockquote>
</aside>
<p>That would mean, that the background class is the class index 0.</p>
<aside class="quote no-group" data-username="skyunyoo" data-post="5" data-topic="50554">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>I this case, If I apply the transform method that you suggested without having to process a pixel with a value of 0, will it have two classes?</p>
</blockquote>
</aside>
<p>In my example I created a target containing only two valid classes:</p>
<ul>
<li>background with class index 0</li>
<li>not-background with class index 1</li>
</ul>
<p>If your non-background class uses another index than 1, you should also convert it to 1 for a binary classification use case.</p>

skyunyoo: <p>Class index has been well solved as your reply.<br>
But I had another problem…</p>
<pre><code class="lang-auto">for x in [numpy_data, numpy_target]:
    print(x.min(), x.max())
</code></pre>
<p>When I loaded numpy files having shape (480,512,512),<br>
I got for input min: 0, max: 499<br>
target min: 0, max: 3</p>
<p>I use Normalize function as follow :</p>
<pre><code class="lang-auto">def normalize(img):   
    arr = img.copy().astype(np.float32)
    M = np.float32(np.max(img))
    if M != 0:
        arr *= 1./M
    return arr
</code></pre>
<p>And i use Dataset as follow:</p>
<pre><code class="lang-auto">class trainDataset(torch.utils.data.Dataset):
    def __init__(self, data, target, transform=None):
        self.data = data.astype(np.float32)
        self.data = normalize(data)

        self.target = target.astype(np.float32)
        self.target = normalize(target)

        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        
        if self.transform:
            x = self.transform(x)

        return x, y
    
    def __len__(self):
        return len(self.data)

transform = transforms.Compose([transforms.ToPILImage(mode=None),
                                                       transforms.Resize(512),
                                                       transforms.ToTensor()
                                                       ])

traindataset = trainDataset(train_numpy_data, train_numpy_target ,transform = transform)
validdataset = trainDataset(valid_numpy_data, valid_numpy_target ,transform = transform)                

trainloader = torch.utils.data.DataLoader(traindataset, batch_size = batch_size, shuffle=True, num_workers=0, pin_memory=False)
validloader = torch.utils.data.DataLoader(validdataset, batch_size = batch_size, shuffle=True, num_workers=0, pin_memory=False)
</code></pre>
<pre><code class="lang-auto">inputs, masks = next(iter(trainloader))
print(inputs.shape, masks.shape)
print(inputs.min(), inputs.max())
print(masks.min(), masks.max())
</code></pre>
<p>I got :<br>
torch.Size([4, 1, 512, 512]) torch.Size([4, 512, 512])<br>
tensor(0.) tensor(0.7675)<br>
tensor(0.) tensor(0.)</p>
<p>I don’t know why min-max value doesn’t get 0-1…</p>
<p>It’s annoying, but i’d really appreciate it if you help me once more.<br>
Thank you!</p>

ptrblck: <p>Using your code, I get normalized value in the range <code>[0, 1]</code>:</p>
<pre><code class="lang-python">train_numpy_data = np.random.randint(0, 500, (480, 512, 512))
train_numpy_target = np.random.randint(0, 4, (480, 512, 512))
    
traindataset = trainDataset(train_numpy_data, train_numpy_target ,transform = transform)
trainloader = torch.utils.data.DataLoader(traindataset, batch_size = 4, shuffle=True, num_workers=0, pin_memory=False)

inputs, masks = next(iter(trainloader))
print(inputs.shape, masks.shape)
&gt; torch.Size([4, 1, 512, 512]) torch.Size([4, 512, 512])
print(inputs.min(), inputs.max())
&gt; tensor(0.) tensor(1.)
print(masks.min(), masks.max())
&gt; tensor(0.) tensor(1.)
</code></pre>
<p>However, you are also normalizing the masks, which would be wrong for a classification use case.<br>
If you would like to get rid of unwanted classes, you should follow the code snippet given before:</p>
<pre><code class="lang-python">background_class_index = 0
desired_class_index = 3
train_numpy_target[train_numpy_target!=desired_class_index] = background_class_index
train_numpy_target[train_numpy_target==desired_class_index] = 1
</code></pre>

skyunyoo: <p>oh, problem is numpy type.<br>
The type was int64, but it changed to int32, so it was nomalized as you suggested.<br>
Thanks you!!<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>