Tracking down a suspected memory leak

tom: Hello, thank you for pytorch! Would you have a hint how to approach ever increasing memory use? Thank you! Thomas P.S.: I have adapted the script to set a few defaults differently and use a different dataset, but did not do much with the actual calculation (and I’d be happy to provide my changes if they are of interest). I am using a git checkout of github pytorch master from today. <SCODE>  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/__init__.py", line 387
    return array_type(*itr)
  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/__init__.py", line 160
    int_array(tensor.size()), int_array(tensor.stride())))
  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/__init__.py", line 398
    descriptor.set(tensor)
  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/rnn.py", line 242
    fn.cy_desc = cudnn.descriptor(cx) if cx is not None else None
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py", line 269
    cudnn.rnn.forward(self, input, hx, weight, output, hy)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/function.py", line 224
    result = self.forward_extended(*nested_tensors)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/function.py", line 202
    flat_output = super(NestedIOFunction, self)._do_forward(*flat_input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py", line 327
    return func(input, *fargs, **fkwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/rnn.py", line 91
    output, hidden = func(input, self.all_weights, hx)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "deepspeech.pytorch/model.py", line 48
    x, _ = self.rnn(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 64
    input = module(input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "deepspeech.pytorch/model.py", line 94
    x = self.rnns(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 59
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "train_de.py", line 139
    out = model(inputs)<ECODE>

tom: It seems the memory-consumption increases if bidirectional is passed to the nn.LSTM instantiation. I have not found out why, though.

AjayTalati: I’ve seen this too (memory-consumption increases if bidirectional is passed) - but I can’t remember (no pun intended) how I fixed it? There is a way to get it to work though, as I’ve been using bidirectional’s for the last few weeks!

tom: Thanks. If you have gotten nn.LSTM it to work, I can stop trying to stare down nn._functions.rnn . Should you remember what you did to make it later, I’d appreciate a hint.

AjayTalati: I hope this helps?

apaszke: 

tom: 

XingxingZhang: Oh, here is my torch verison <SCODE>>>> import torch
>>> torch.__version__
'0.1.10+ac9245a'
<ECODE>

tom: Best regards Thomas

apaszke: Ugh, can you please get us a smallest possible snippet that still causes the memory to blow up? That’s the only way we can find and fix these things

XingxingZhang: I have added “gc.collect()” in my code and let it run for one or two days to see if I still have the problem.

AjayTalati: If anyone is concerned about bi-directional LSTM and memory leak issues in pytorch - maybe a good place to start coding a new project is from the already working tutorial?

tom: Ah, thanks. I was going to post a movie sentiment net, but that has the drawback of having the embedding layer. <SCODE># helper function to get rss size, see stat(5) under statm. This is in pages (4k on my linux)
def memory_usage():
    return int(open('/proc/self/statm').read().split()[1])
<ECODE> I saw similar effects with the sentiment analysis one, but I’m not sure I’ll keep saying that unidirectional LSTM always work…

apaszke: 

tom: So here is a small thing hitting 100 MB extra memory consumption after 12-18 “epochs” with bidirectional LSTM and not without. I don’t think it is the contiguous call + summation as it also happens when I double the input dimension of the 2nd-4th LSTM layer instead.

XingxingZhang: unfortunately, “gc.collect()” didn’t fix the leak in my program. It takes less RAM in the 3rd epoch, but it is still around 45G. I will look into it again.

tom: 

donglixp: 

tom: 

donglixp: hope this would help