CPU RAM gets full in the first epoch, while GPU's one is not utilised!

username: raghadalghonaim
cooked: <p>I’m using Colab Pro, along with Pytorch.</p>
<p>The GPU allocated for me is: Tesla P100-PCIE-16GB</p>
<p>However, CPU RAM filled in the first epoch, and it doesn’t even reach the validation step, my dataset is a set of images of size 128*128. Training set: 122k. valid set: 21k</p>
<p>Here’s my code:</p>
<pre><code class="lang-auto">def train(model, train_loader, validation_loader):

    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    model = model.to(device)

    loss_train_log = []
    loss_val_log = []
    epoch_val_log = []

    print('START TRAINING...')
    for epoch in range(1, num_epochs + 1):
        # Training
        model.train()
        epoch_loss = 0
        for batch_idx, batch_samples in enumerate(train_loader):
            img = batch_samples[0].view(batch_samples[0].size(0),
                                        batch_samples[0].size(1),
                                        batch_samples[0].size(2),
                                        batch_samples[0].size(3)).to(device)
            labels = batch_samples[1].to(device)

            optimizer.zero_grad()

            prd = model(img)

            loss = F.mse_loss(prd.view(-1).float(), labels.view(-1).float())

            loss.backward()
            optimizer.step()

            epoch_loss += loss.detach().item() * len(batch_samples[0]) / len(train_loader.sampler)
            
            del loss
            del prd

        loss_train_log.append(epoch_loss)

        del epoch_loss

        print('+ TRAINING \tEpoch: {} \tLoss: {:.6f}'.format(epoch, loss_train_log[-1]))

        # Validation
        if epoch == 1 or epoch % val_interval == 0:
            print("Validation")
            model.eval()
            loss_val = 0


            with torch.no_grad():
                for data_sample in validation_loader:
                    img = data_sample[0].to(device)
                    labels = data_sample[1].to(device)
                    img = img.view(img.size(0),img.size(1),img.size(2),img.size(3))
                    
                    prd = model(img)

                    prd_flat = prd.view(-1)
                    true = labels.view(-1)

                    loss_val += F.mse_loss(prd_flat.float(), true.float()).item() * len(data_sample[0]) / len(validation_loader.sampler)
                    del prd
                loss_val_log.append(loss_val)
                epoch_val_log.append(epoch)
                del loss_val


            print('-------------------------------------------------------------')
            print('+ VALIDATE \tEpoch: {} \tLoss: {:.5f}'.format(epoch, loss_val_log[-1]))
            print('-------------------------------------------------------------')


    print('\nFinished TRAINING.')


train_dataset=dataset(X_train,y_train, transform=transform) # dataset is a class I defined.
valid_dataset=dataset(X_valid,y_valid, transform=transform)

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
validation_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=batch_size, shuffle=True)

model = Model()
train(model, train_loader, validation_loader)

</code></pre>
<p>Could someone help me with this, please?</p>
<p>Here is my dataset class along with my transform:</p>
<pre><code class="lang-auto">
transform = T.Compose([
                T.ToPILImage(),
                T.ToTensor(),
                T.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5)),
            ])   

class dataset(Dataset):
    def __init__(self, data, label, transform=None):
        self.data=data
        self.label=label
        self.transform=transform
    def __getitem__(self, index):
        if self.transform:
            self.data[index]=self.transform(self.data[index])
        return self.data[index],self.label[index]
    def __len__(self):
        return len(self.data)

</code></pre>

username: jmandivarapu1
cooked: <p>can you print your <code>device</code> and <code>data</code> inside the training. Just to. confirm weather the data itself is getting transferred to the GPU. When you print the data t will show tensor ending with cuda device id</p>

username: raghadalghonaim
cooked: <p>Thanks for your response.</p>
<p>I figured out the problem, I was using list to store the data instead of NumPy Array.</p>
<p>I documented everything I tried in this post, if someone is interested:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="16" height="16">
      <a href="https://medium.com/@raghadalghonaim/memory-leakage-with-pytorch-23f15203faa4" target="_blank" title="08:35PM - 03 April 2020" rel="nofollow noopener">Medium – 3 Apr 20</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://medium.com/@raghadalghonaim/memory-leakage-with-pytorch-23f15203faa4" target="_blank" rel="nofollow noopener">Memory Leakage with PyTorch</a></h3>

<p>If you’re reading this post, then most probably you’re facing this problem. RAM is full, in the very beginning of the training, my data is…</p>

  <p><span class="label1">Reading time: 3 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>