Why does the reuse of training data cause a stack of "Non-releasable Memory in GPU"?

Nardien: <p>Hi. I’ve used PyTorch 1.4 for my project.</p>
<p>I found the weird problem that the reuse of training data causes a stack of “Non-releasable Memory” in GPU.</p>
<p>Specifically, my program consists of several training processes.</p>
<p>Then, in every training process, a model, optimizer, and training data are initialized and trained.</p>
<p>However, since loading the training data using pickle causes too much time (each data instance is the class object consists of lists. It is a natural language dataset e.g. SQuAD v1.1), I try to load and ‘cache’ the training data before several training steps start rather than loading the training data at every training process.</p>
<p>Then, ‘cached’ training data is fed to the “train” function then it is wrapped with TensorDataset inside the “train” function.</p>
<p>However, by using cuda.memory_summary function, I found that this approach causes the stack of Non-releasable memory to GPU at every iteration and it ends up with a Cuda out of memory error.</p>
<p>When I load the training data or deep-copying training data in every training process, this problem did not happen but they cost too much time.</p>
<p>Does anybody have some ideas about why this problem occurs?</p>
<p>Furthermore, how can I solve this problem?</p>
<p>Thank you for reading my question.</p>

ptrblck: <p>Could you post a minimal code snippet to show how your caching function looks and works, so that we can reproduce this issue?</p>

Nardien: <p>Thank you for paying attention to my question.</p>
<p>Basically, our code is based on the following code:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/huggingface/transformers/blob/master/examples/run_squad.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/huggingface/transformers/blob/master/examples/run_squad.py" target="_blank" rel="nofollow noopener">huggingface/transformers/blob/master/examples/run_squad.py</a></h4>
<pre><code class="lang-py"># coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet)."""


import argparse
import glob
</code></pre>

  This file has been truncated. <a href="https://github.com/huggingface/transformers/blob/master/examples/run_squad.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Then, in our code, we call the main function of run_squad.py every iteration as follows:</p>
<pre><code class="lang-auto">from run_squad import main
...
for _ in range(100):
    main()
</code></pre>
<p>(this is not exact code since we slightly modified it that we can call this function iteratively, however, I think you can catch what we try to do by this code snippet)</p>
<p>Since this code loads the training dataset at every iteration like this (line 791),</p>
<pre><code class="lang-auto">        train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
</code></pre>
<p>it costs 15~30 seconds at every iteration for loading dataset.</p>
<p>Instead, we try to load the dataset before iteration instead of loading the dataset at every iteration as follows:</p>
<pre><code class="lang-auto">from run_squad import main, load_and_cache_examples
...
train_dataset = load_and_cache_examples(...)
for _ in range(100):
    main(train_dataset)
</code></pre>
<p>However, it ends up with a stack of “Non-releasable Memory in GPU” and finally the out of memory error occurs because of stacked non-releasable GPU memory.</p>
<p>This situation did not occur if we load the dataset inside every main function call.</p>
<p>I’m sorry for not providing the detailed code. If you need more information on inspecting this problem, please let me know.</p>
<p>Thanks.</p>

Molaison_zeng: <p>Hi, I encountered the exactly same problem. I’ve wrapped the train procedures into a function to keep the data destroyed outside of function calls. I even use following codes to clean up memory, but it seems not working:</p>
<pre data-code-wrap="python"><code class="lang-python">del loss_input
del loss
gc.collect()
torch.cuda.empty_cache()
torch._C._cuda_clearCublasWorkspaces()
torch._dynamo.reset()
</code></pre>
<p>Have you found any solutions?</p>