CUDA 10.2 Out of memory

Tuong_Lam: <p>I run a model in GTX 1080 Ti with cuda 10.2 and pytorch 1.5. When I synthesize audio output, I use “with torch.no_grad(), torch.backends.cudnn.deterministic = False, torch.backends.cudnn.benchmark = False, torch.cuda.set_device(0), torch.cuda.empty_cache(), os.system(“sudo rm -rf ~/.nv”)” but GPU memory is still increased. Each time it increase about 10 MiB until out of memory.<br>
Can you help me to solve this problem? Thanks very much.</p>

ptrblck: <p>If I understand the issue correctly, your memory usage is increasing in each iteration.<br>
This might happen if you are storing tensors, which are still attached to the computation graph, in e.g. a <code>list</code>.<br>
Often you would like to append the loss to a list in order to calculate the mean for the epoch.<br>
Since the <code>loss</code> tensor is attached to the computation graph, you would also store the complete graph in each iteration, which might eventually yield the OOM issue.</p>
<p>To detach the tensor properly, you could use:</p>
<pre><code class="lang-python">losses.append(loss.cpu().detach().item())
</code></pre>
<p>The memory increase should not come from the options you’ve posted.</p>

Tuong_Lam: <p>The problem is in synthesizing, not training so I think it does not involve to loss.</p>

ptrblck: <p>Could you post your code, so that we can have a look?<br>
None of the mentioned settings should change the memory behavior and of course shouldn’t create / avoid a memory leak.</p>

Tuong_Lam: <p>Here is my code to synthesize audio from mel spectrogram<br>
pad_fn = torch.nn.ReplicationPad1d(self.config_gan[“generator_params”].get(“aux_context_window”, 0)).to(torch.device(“cuda”))<br>
<span class="hashtag">#Generative</span><br>
with torch.no_grad():<br>
c = self.scaler.transform(mels[0])<br>
x = ()<br>
z = torch.randn(1, 1, len(c ) * self.config_gan[“hop_size”]).to(torch.device(“cuda”))<br>
x += (z,)<br>
b = torch.from_numpy(c ).unsqueeze(0).transpose(2, 1).to(torch.device(“cuda”))<br>
c = pad_fn(b)<br>
x += (c,)<br>
y = self.model_gan(*x).view(-1).cpu().numpy()<br>
y = y[:len(y)-3000]<br>
del z<br>
torch.cuda.empty_cache()<br>
del b<br>
torch.cuda.empty_cache()<br>
del c<br>
torch.cuda.empty_cache()<br>
out = io.BytesIO()<br>
audio.save_wav(y, out, sr=hparams.sample_rate)<br>
del y<br>
torch.cuda.empty_cache()<br>
del pad_fn<br>
torch.cuda.empty_cache()<br>
gc.collect()<br>
with open(‘fmemory.txt’, ‘a’) as f:<br>
f.writelines(‘del all ’ + str(torch.cuda.memory_allocated()/1024<strong>2)+"\n") <span class="hashtag">#5</span>.27490234375<br>
f.writelines(str(torch.cuda.memory_cached()/1024</strong>2) +’\n’) <span class="hashtag">#6</span>.0<br>
self.model_gan.remove_weight_norm()<br>
return out.getvalue()</p>

ptrblck: <p>Are you able to reproduce the memory increase using random data? If so, could you post the input data shape as well as all other shapes, which would be necessary to reproduce this issue?</p>

Tuong_Lam: <p>Here is the memory that I noted when synthesizing a text with 3 sentences.<br>
before inference: allocated 5.10791015625<br>
cached 6.0<br>
inference: allocated 225.40185546875<br>
cached 970.0<br>
after inference allocated 6.48193359375<br>
cached 26.0<br>
del all allocated 6.48193359375<br>
cached 26.0<br>
before inference: allocated 5.10791015625<br>
cached 6.0<br>
inference: allocated 95.818359375<br>
cached 398.0<br>
after inference allocated 5.6748046875<br>
cached 6.0<br>
del all allocated 5.6748046875<br>
cached 6.0<br>
before inference: allocated 5.10791015625<br>
cached 6.0<br>
inference: allocated 31.6435546875<br>
cached 122.0<br>
after inference allocated 5.27490234375<br>
cached 6.0<br>
del all allocated 5.27490234375<br>
cached 6.0</p>

ptrblck: <p>The memory doesn’t seem to grow in each iteration or am I missing something?<br>
Depending on the size of your input the current iteration might need more memory than the previous one, but the memory footprint seems to go down as expected after deleting the tensors.</p>

Tuong_Lam: <p>Yes, the memory does not change in allocate and cache. But GPU memory usage is still increase until out of memory.</p>

ptrblck: <p>How can the GPU yield an OOM, if the allocated and cached memory is reduced in each iteration?<br>
Are you hitting an “extremely large” input sample, which might be too big for your device?</p>

Tuong_Lam: <p>Maybe it is a bug of Pytorch 1.5, I change to Pytorch 1.0.1 and it works fine, not leaking GPU RAM as in Pytoch 1.5.</p>

ptrblck: <p>I’m still unsure, how to interpret this statement:</p>
<aside class="quote no-group" data-username="Tuong_Lam" data-post="9" data-topic="85587" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tuong_lam/48/25619_2.png" class="avatar"> Tuong_Lam:</div>
<blockquote>
<p>Yes, the memory does not change in allocate and cache. But GPU memory usage is still increase until out of memory.</p>
</blockquote>
</aside>
<p>How does the memory increase, if the allocated and cached memory is not changed?<br>
Are you seeing the increase only via e.g. <code>nvidia-smi</code>?</p>

Tuong_Lam: <p>Yes, I see the increase in nvidia-smi.</p>

ptrblck: <p>Could you post a minimal, executable code snippet, which would show this behavior, so that we could debug it, please?</p>

Tuong_Lam: <p>You can see here: <a href="https://github.com/kan-bayashi/ParallelWaveGAN/issues/160#issuecomment-639850145" rel="nofollow noopener">https://github.com/kan-bayashi/ParallelWaveGAN/issues/160#issuecomment-639850145</a></p>

ptrblck: <p>The code is unfortunately not executable, as you are using private data.<br>
Could you post an executable code snippet using random input data?</p>