Ram usage increases linearly

Nilesh_Pandey1: <p>Hi, the below code increases the memory usage linearly, and at certain point I am not able to train the model. Surprisingly it is the first time I am facing problem with the following code?</p>
<p>doubts:</p>
<ol>
<li>Vector images, Vector image is the only new data that is involved in the following code, commenting line which loads vector images makes the code run normally.</li>
</ol>
<p>I really have no idea, any hint or suggestion would be highly appreciated.<br>
Thank you,<br>
Nilesh Pandey</p>
<pre><code class="lang-auto">class ImgAugTransform:
    def __init__(self):
        sometimes = lambda aug: iaa.Sometimes(0.5, aug)

        self.aug = iaa.Sequential([
        iaa.Affine(
            translate_percent={"x":0.2, "y": 0.1},
            rotate=40,
            mode='symmetric'
        )
    ])
    def __call__(self, img, img1, img2,img3):
        img = np.array(img)
        img1 = np.array(img1)
        img2 = np.array(img2)
        img3 = np.array(img3)

        return self.aug.augment_image(img), self.aug.augment_image(img1), self.aug.augment_image(img2),self.aug.augment_image(img3)

</code></pre>
<pre><code class="lang-auto">class Dataset(data.Dataset):
    """Dataset for XXX.
    """
    def __init__(self,height):
        super(Dataset, self).__init__()
        # base setting

        
        self.files = []
        self.vector = []
        with open("/home/XXX/train.txt", 'r') as f:
            for line in f.readlines():
                im_name, v_name = line.strip().split()
                self.files.append(im_name)
                self.vector.append(v_name)
                
        
        self.masks = os.listdir("/home/XXX/")
        
        self.range = np.arange(len(self.masks))
        self.rotate = ImgAugTransform()
    def name(self):
        return "XXX"
    
    
    def transformData(self, src, mask, target,ref_lr):

        if random.random() &gt; 0.5:
            src, mask, target,ref_lr = self.rotate(src,mask, target,ref_lr)
           
        # Transform to tensor

        src = TF.to_tensor(src)
        mask = TF.to_tensor(mask)
        target = TF.to_tensor(target)
        ref_lr= TF.to_tensor(ref_lr)
        
        src = TF.normalize(src,(0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        mask = TF.normalize(mask, (0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        target = TF.normalize(target,(0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        ref_lr = TF.normalize(ref_lr,(0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        return src, mask, target,ref_lr

    
    def __getitem__(self, index):
        file = self.files[index]
        mask = self.masks[random.choice(self.range)]
        vector = self.vector[index]
        # person image 
        targ = rescale_intensity(plt.imread(osp.join('/homeXXX/', file))/255)#targ = rescale_intensity(plt.imread(osp.join('/homeXXX', file))/255)
        vec = rescale_intensity(plt.imread(osp.join('/home/XXX', vector))/255)
        mask = rescale_intensity(plt.imread(osp.join('/home/XXX', 'maskA', mask))/255)

        targ = resize(targ,(256,256))
        vec = resize(vec,(256,256))
        mask = resize(mask,(256,256))
        
        ms2 = mask*1
        ms2 = np.expand_dims(ms2,axis=2)
        ms2 = np.repeat(ms2,repeats=3,axis=2)
        
        src =targ*(1-ms2)+ms2

  
        src = Image.fromarray(np.uint8(src*255))
        mask = Image.fromarray(np.uint8(ms2*255))
        target = Image.fromarray(np.uint8(targ*255))
        vec = Image.fromarray(np.uint8(vec*255))
        source,mask,target,ref = self.transformData(src, mask, target, vec)
        return source,mask,target,ref
</code></pre>

nile649: <p>I am trying to run all my previous codes and project, and basically all are causing increase in RAM consumption linearly.<br>
The change in hardware setup is low storage space, currently I am on a storage space less than &lt;40GB, as far as I have known  it doesn’t make sense, but anyone think it could be related?</p>

ptrblck: <p>If I understand your issue correctly, commenting this line:</p>
<pre><code class="lang-python">vec = rescale_intensity(plt.imread(osp.join('/home/XXX', vector))/255)
</code></pre>
<p>yields normal behavior, while keeping it increases the memory?<br>
How did you define <code>rescale_indensity</code>?</p>
<p><a class="mention" href="/u/nile649">@nile649</a><br>
Your projects were running fine and now after changing the storage you are noticing an increasing memory usage? Did you change something else (PyTorch version etc.)?</p>

nile649: <p>actually, I just cross checked with previous projects, and it is happening with them as well. I doubt it is the code, but something else. I did look in previous post about  assigning different variables, not using list, and as such, but it seems it is a different problem.</p>
<p>All the research projects over the last year use the same dataloader, and it is the first time I am having memory issue even with the weak old projects.</p>

ptrblck: <p>Check, if you are accidentally storing the computation graph, e.g. by accumulating the loss without detaching or storing it in a list.<br>
Could you try to create a minimal code snippet to reproduce this issue?</p>

Diego: <p>Not sure if related, but I am having the same Issue with the <a href="https://github.com/facebookresearch/maskrcnn-benchmark/issues/1047" rel="nofollow noopener">mask-rcnn Pytorch version</a></p>

nile649: <p>You’re right, I did debugging step by step and the following line is the culprit, I am directly treating loss term as scalar while calculating average loss over the epoch.</p>
<pre><code class="lang-auto">avg_loss_g = (avg_loss_g+loss_G)/(i+1) 
</code></pre>

nile649: <p>If the problem is  not with variables in dataloader, then it is with the variables in main function.</p>

Diego: <p>We are talking about CPU Memory, right?</p>

Nilesh_Pandey1: <p>Yes, CPU memory.</p><aside class="quote" data-post="3" data-topic="6636">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mlagunas/48/1358_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-increases-ram-usage-every-iteration/6636/3">DataLoader increases RAM usage every iteration</a> 
  </div>
  <blockquote>
    Is there any reason why you load all of your images during init ? 

That was the error (and it makes lots of sense <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/sweat.png?v=12" title="sweat" alt="sweat" class="emoji">) . 
Now, I store the image paths during __init__ and load the images during __getitem__, as you mentioned. It works perfectly. 
Thanks,
  </blockquote>
</aside>

<aside class="quote" data-post="4" data-topic="21038">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/fbc32d/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/very-consitent-memory-leak/21038/4">Very consitent memory leak</a> 
  </div>
  <blockquote>
    I tried some more things: 

disabled eval mode
removed a non standard regularization step
replaced the non standard activation function with an out of the box ELU

In all cases I see the same consistent memory leak of ~108k pages of memory per epoch. 
Any additional ideas on how to analyze this would be welcome. 
Alberto
  </blockquote>
</aside>

<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/13843">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/13843" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/13843" target="_blank" rel="noopener nofollow ugc">Possible CPU-side memory leak even when fitting on GPU</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-11-12" data-time="17:07:05" data-timezone="UTC">05:07PM - 12 Nov 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-11-13" data-time="17:13:28" data-timezone="UTC">05:13PM - 13 Nov 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/trias702" target="_blank" rel="noopener nofollow ugc">
          <img alt="trias702" src="https://avatars.githubusercontent.com/u/25867060?v=4" class="onebox-avatar-inline" width="20" height="20">
          trias702
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          todo
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug

A possible CPU-side memory leak even when fitting on the GPU using P<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">yTorch 0.4.1.

## To Reproduce

I am quite new to PyTorch, having used TF/Keras extensively in the past, but am now trying to use PyTorch as a replacement. I decided to start small with a seq2seq Skip-Thought model, cobbled together using the PyTorch NLP tutorials. Everything seems to work fine when I run small scale tests, however, when I use the code to run a large scale fit on 3000 separate paragraphs (each paragraph having a variable number of sentences) I notice that my system RAM usage slowly goes up as the script runs, until eventually it hits 100% and the box becomes unresponsive and has to be force rebooted. The Linux box has 64 GB of RAM, and when the script starts, usage is 4.7% but this climbs steadily over time to 100%, at which point the box becomes unresponsive.

Since I'm new to PyTorch, I'm not sure if perhaps I'm doing something blatantly wrong in my code which would account for this behaviour?

This is my core fitting logic, which runs as a script on linux:

```
device = 'cuda'
model = SkipThought(len(text_dictionary.token2id), 128, 256, nn.NLLLoss()).to(device)

// corpus_orig is a list of lists where each list-element is a text paragraph with multiple sentences
n_iters=len(corpus_orig)

start = time.time()
print_loss_total = 0

optimizer = optim.SGD(model.parameters(), lr=0.01)

for xi in range(1, n_iters + 1):
    x = corpus_orig[xi - 1]
    sents = [x for x in map(str.strip, x.split('. ')) if len(x) &gt; 0]
    for i in range(1, len(sents) - 1):
        input_tensor, prev_tensor, next_tensor = tensorsFromPair( (sents[i], sents[i - 1], sents[i + 1]) )

        optimizer.zero_grad()
        
        loss, prev_output, next_output = model(input_tensor, prev_tensor, next_tensor, use_teacher_forcing=True)
        
        loss.backward(retain_graph=True)    // pytorch says I need retain_graph=True, otherwise I get an error here
        optimizer.step()
        
        print_loss_total += loss.item()

    print_loss_avg = print_loss_total
    print_loss_total = 0
    print('%s (%d %d%%) %.4f' % (timeSince(start, xi / n_iters),
                         xi, xi / n_iters * 100, print_loss_avg), flush=True)
```

Here are the helper functions which create the tensors passed to the model by converting all text words into indices from a predefined gensim dictionary:

```
def indexesFromSentence(sentence):
    return text_dictionary.doc2idx(sentence.split())

def tensorFromSentence(sentence):
    indexes = indexesFromSentence(sentence)
    indexes.append(EOS_token)
    return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1)

def tensorsFromPair(pair):
    input_tensor = tensorFromSentence(pair[0])
    prev_tensor = tensorFromSentence(pair[1])
    next_tensor = tensorFromSentence(pair[2])
    return (input_tensor, prev_tensor, next_tensor)
```

And, finally, here is the SkipThought model itself, with all its helper classes (apologies for the wall of code):

```
class EncoderRNN(nn.Module):
    def __init__(self, vocab_size, embedding_size, gru_size):
        super(EncoderRNN, self).__init__()
        self.vocab_size = vocab_size
        self.embedding_size = embedding_size
        self.gru_size = gru_size

        self.embedding = nn.Embedding(vocab_size, embedding_size)
        self.gru = nn.GRU(embedding_size, gru_size)

    def forward(self, sentence, hidden):
        embeddings = self.embedding(sentence)
        embeddings = F.tanh(embeddings)
        output, hidden = self.gru(embeddings, hidden)
        
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, 1, self.gru_size, device=device)
    
    def getEmbedding(self, sentence):
        return F.tanh(self.embedding(sentence))

class LocalAttention(nn.Module):
    def __init__(self, dim):
        super(LocalAttention, self).__init__()
        self.W = nn.Linear(dim, dim, bias=False)

    def score(self, decoder_hidden, encoder_out):
        encoder_out = self.W(encoder_out)
        encoder_out = encoder_out.permute(1, 0, 2)
        return encoder_out @ decoder_hidden.permute(1, 2, 0)

    def forward(self, decoder_hidden, encoder_out):
        energies = self.score(decoder_hidden, encoder_out)
        mask = F.softmax(energies, dim=1)
        context = encoder_out.permute(1, 2, 0) @ mask
        context = context.permute(2, 0, 1)
        mask = mask.permute(2, 0, 1)
        return context, mask


class SkipThought(nn.Module):
    def __init__(self, vocab_size, embedding_size, gru_size, criterion):
        super(SkipThought, self).__init__()
        self.vocab_size = vocab_size
        self.embedding_size = embedding_size
        self.gru_size = gru_size
        self.criterion = criterion
        
        self.encoder = EncoderRNN(vocab_size, embedding_size, gru_size)
        self.prev_gru = nn.GRU(embedding_size + gru_size, gru_size)
        self.next_gru = nn.GRU(embedding_size + gru_size, gru_size)
        self.attention = LocalAttention(gru_size)
        self.worder = nn.Linear(gru_size * 2, vocab_size)
        self.softmax = nn.LogSoftmax(dim=1)
        
        self.encoder_hidden = self.encoder.initHidden()
    
    def forward(self, input_tensor, prev_tensor, next_tensor, use_teacher_forcing=True):        
        encoder_hidden = self.encoder_hidden
        
        prev_length = prev_tensor.size(0)
        next_length = next_tensor.size(0)
    
        loss = 0
        
        encoder_output, encoder_hidden = self.encoder(input_tensor, encoder_hidden)
    
        prev_input = torch.tensor([[SOS_token]], device=device)
        next_input = torch.tensor([[SOS_token]], device=device)
    
        prev_hidden = encoder_hidden
        next_hidden = encoder_hidden
        
        self.encoder_hidden = encoder_hidden
    
        prev_output = []
        next_output = []
    
        if use_teacher_forcing:
            # Teacher forcing: Feed the target as the next input
            for di in range(prev_length):
                embedded = self.encoder.getEmbedding(prev_input)
                context, _ = self.attention(prev_hidden, encoder_output)
                decoder_output, prev_hidden = self.prev_gru(torch.cat([embedded, context], dim=2), prev_hidden)
                decoder_output = self.softmax(self.worder(torch.cat([decoder_output, context], dim=2)[0]))
                loss += self.criterion(decoder_output, prev_tensor[di])
                prev_output.append( decoder_output.topk(1)[1].squeeze().detach().item() )
                prev_input = prev_tensor[di].unsqueeze(0)  # Teacher forcing
            
            for di in range(next_length):
                embedded = self.encoder.getEmbedding(next_input)
                context, _ = self.attention(next_hidden, encoder_output)
                decoder_output, next_hidden = self.next_gru(torch.cat([embedded, context], dim=2), next_hidden)
                decoder_output = self.softmax(self.worder(torch.cat([decoder_output, context], dim=2)[0]))
                loss += self.criterion(decoder_output, next_tensor[di])
                next_output.append( decoder_output.topk(1)[1].squeeze().detach().item() )
                next_input = next_tensor[di].unsqueeze(0)  # Teacher forcing
    
        else:
            # Without teacher forcing: use its own predictions as the next input
            for di in range(prev_length):
                embedded = self.encoder.getEmbedding(prev_input)
                context, _ = self.attention(prev_hidden, encoder_output)
                decoder_output, prev_hidden = self.prev_gru(torch.cat([embedded, context], dim=2), prev_hidden)
                decoder_output = self.softmax(self.worder(torch.cat([decoder_output, context], dim=2)[0]))
                topv, topi = decoder_output.topk(1)
                prev_input = topi.squeeze().detach()  # detach from history as input
                
                loss += self.criterion(decoder_output, prev_tensor[di])
                prev_output.append( prev_input.item() )
                if prev_input.item() == EOS_token:
                    break
                prev_input = prev_input.unsqueeze(0).unsqueeze(0)
            
            for di in range(next_length):
                embedded = self.encoder.getEmbedding(next_input)
                context, _ = self.attention(next_hidden, encoder_output)
                decoder_output, next_hidden = self.next_gru(torch.cat([embedded, context], dim=2), next_hidden)
                decoder_output = self.softmax(self.worder(torch.cat([decoder_output, context], dim=2)[0]))
                topv, topi = decoder_output.topk(1)
                next_input = topi.squeeze().detach()  # detach from history as input
    
                loss += self.criterion(decoder_output, next_tensor[di])
                next_output.append( next_input.item() )
                if next_input.item() == EOS_token:
                    break
                next_input = next_input.unsqueeze(0).unsqueeze(0)
        
        return loss, prev_output, next_output
```

## Expected behavior

Memory usage in Linux should not linearly increase as the fitting script runs, especially not to the point at which the box dies.

## Environment

 - PyTorch Version (e.g., 1.0): 0.4.1
 - OS (e.g., Linux): Linux (Ubuntu 16.04)
 - How you installed PyTorch (`conda`, `pip`, source): pip install torch
 - Build command you used (if compiling from source): N/A
 - Python version: 3.6.6
 - CUDA/cuDNN version: CUDA 9.0.176 / CuDNN 7.4.1.5
 - GPU models and configuration: Tesla V100
 - Any other relevant information:</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/16144">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/16144" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/16144" target="_blank" rel="noopener nofollow ugc">[Bug] CPU memory keeps increasing when using gloo backend</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-01-18" data-time="07:37:01" data-timezone="UTC">07:37AM - 18 Jan 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-02-03" data-time="19:54:09" data-timezone="UTC">07:54PM - 03 Feb 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/stgzr" target="_blank" rel="noopener nofollow ugc">
          <img alt="stgzr" src="https://avatars.githubusercontent.com/u/6370811?v=4" class="onebox-avatar-inline" width="20" height="20">
          stgzr
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: distributed
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          cherry-picked
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug
I found CPU memory keeps increasing when using `dist.all_gather` in `g<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">loo` backend.
Update: it seems that `gather` and `all_reduce` have the same issue.


## To Reproduce
``` python
""" test.py """
import os
import sys
import gc
import logging
import time
import torch
import torch.distributed as dist
from torch.distributed import get_rank, get_world_size

from mpi4py import MPI
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
print (rank, size)
os.environ['CUDA_VISIBLE_DEVICES'] = str(rank)
os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = '23333'
os.environ['WORLD_SIZE'] = str(size)
os.environ['RANK'] = str(rank) 
dist.init_process_group(backend="gloo")

print ("initialize gloo successfully [rank {}] pid({})".format(rank, os.getpid()))

torch.backends.cudnn.benchmark = True

if __name__ == "__main__":
    batch_size = 16
    feat_dim = 2048
    pid = os.getpid()
    prev_mem = 0
    tensor_list = [torch.FloatTensor(batch_size, feat_dim) for _ in range(get_world_size())]
    values = torch.FloatTensor(batch_size, feat_dim)
    for idx in range(10000):
        if idx % 20 == 0 and get_rank() == 0:
            cur_mem = (int(open('/proc/%s/statm' % pid, 'r').read().split()[1]) + 0.0) / 256
            add_mem = cur_mem - prev_mem
            prev_mem = cur_mem
            print ("train iterations: %s, added mem: %s M" % (idx, add_mem))
        dist.all_gather(tensor_list=tensor_list, tensor=values)
```
Steps to reproduce the behavior:
``` bash
mpirun -n 4 python test.py
```
Result:
&gt;...
train iterations: 2340, added mem: 10.0546875 M
train iterations: 2360, added mem: 10.0546875 M
train iterations: 2380, added mem: 9.796875 M
train iterations: 2400, added mem: 10.0546875 M
train iterations: 2420, added mem: 10.0546875 M
train iterations: 2440, added mem: 10.0546875 M
train iterations: 2460, added mem: 10.0546875 M
train iterations: 2480, added mem: 10.0546875 M
train iterations: 2500, added mem: 9.796875 M
train iterations: 2520, added mem: 10.0546875 M
train iterations: 2540, added mem: 10.0546875 M
...



## Expected behavior
CPU memory keeps increasing (`added mem` has positive value)


## Environment
 - PyTorch Version: 1.0.0
 - OS: CentOS 7.2
 - GCC version: (GCC) 4.8.5 20150623
 - CMake version: version 3.12.2
 - How you installed PyTorch: `conda` and source
 - Build command you used (if compiling from source): `python setup.py install`
 - Python version: 2.7.15
 - CUDA/cuDNN version: CUDA 9.0 / cuDNN 7.2
 - GPU models and configuration:
     GPU 0: Tesla P100-PCIE-16GB
     GPU 1: Tesla P100-PCIE-16GB 
     GPU 2: Tesla P100-PCIE-16GB
     GPU 3: Tesla P100-PCIE-16GB
     GPU 4: Tesla P100-PCIE-16GB
     GPU 5: Tesla P100-PCIE-16GB
     GPU 6: Tesla P100-PCIE-16GB
     GPU 7: Tesla P100-PCIE-16GB

## Additional context
It works well when I use PyTorch v0.4.0.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This with the <a class="mention" href="/u/ptrblck">@ptrblck</a> response made me understand what are the reasons for potential memory leak in pytorch. I will suggest read thru the links I have posted.</p>