Is it possible to Free-up a Dataloader?

Paritosh: <p>I am using different Dataloaders for train set and test set, so in all, I have 2 Dataloaders.</p>
<p>I do training and testing in every epoch.</p>
<p>Is there a way I can free up the Dataloader not being used (for eg. free-up the train dataloader while testing; and free-up test dataloader while training) so as to be able to increase the batch size of the Dataloader being used?</p>

Deeply: <p>From my understanding, the dataloader is just a proxy between your train/test set, and these train and test sets are the variables that eat the memory. So, you will need to delete the train/test set to free up the allocated memory. I could be wrong, but I guess this is how it could be done:</p>
<pre><code class="lang-auto">test_loader = None
del test_loader
test_set  = None
del test_set
</code></pre>
<p>However, you will have to read the deleted (data)set and the associated loader before trying to use it again. More, you will have to compromise the reading time(s) of these sets with the gain you might get from deleting them.</p>

chenglu: <p>I think dataloader is just a generator. What takes memory is model weights and the data been yield. The memory of model weights will always be there since you are training them all the time. And about the yield data from dataloader, at end of every iteration, the yield data will be free since it’s a local variable. So I think there is no need to free dataloader.</p>

Paritosh: <p>Hi <a class="mention" href="/u/deeply">@Deeply</a>, thanks for the suggestion. I tried this but it is not working. GPU runs out of memory.</p>

Paritosh: <p>Hi <a class="mention" href="/u/chenglu">@chenglu</a>, thanks for the answer. I think that training and testing batch size matters. I tried out with larger test batch size and it didn’t work, but when I reduce the test batch size, it works. So it seems that even if the dataloader is emptied out at the end of every epoch, it occupies memory.</p>

ptrblck: <p>Have you used <code>with torch.no_grad()</code> during your test phase? This will avoid storing the intermediate variables needed for the backward pass, which is not necessary for testing.</p>

Paritosh: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks a lot, I am able to increase the batch size after using <code>with torch.no_grad()</code>!</p>

aksg87: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="21498">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>with torch.no_grad()</p>
</blockquote>
</aside>
<p>Hi all,</p>
<p>I am running into what I think is a memory leak? Not sure why my memory isn’t getting feed up while I iterate through my dataloader during inference. If anyone has ideas please let me know!</p>
<p>**More Details **</p>
<p>I noticed that the memory appears to accumulate based on the tracemalloc package at:</p>
<p><code>def __getitem__(self, index)</code></p>
<pre><code class="lang-auto">...
        image = (image / 255).astype(np.float32)
        mask = mask.astype(np.float32)
...
</code></pre>
<p>Iteration through inference DataLoader</p>
<pre><code class="lang-auto">...
    tracemalloc.start()

    snapshot1 = tracemalloc.take_snapshot()
    for (batch_idx, batch) in tqdm(enumerate(loader), total=len(loader)):
        try:
            if batch_idx % 10 == 0:
                snapshot1 = tracemalloc.take_snapshot()

            log.info(f"Running inference on {batch['image_path']}")

            batch_pred = inference_step(
                model, checkpoint_fp, batch, thresh, model_loaded=True
            )

            res = inference_ds.save_prediction_batch(
                batch_pred,
                out_folder,
                config=cfg,
                thresh=thresh,
                channel=channel,
            )
            successes.append(res)
            log.info("Inference complete")

            if batch_idx % 10 == 9:
                snapshot2 = tracemalloc.take_snapshot()
                top_stats = snapshot2.compare_to(snapshot1, "lineno")

                log.info("\n\n [ Top 10 differences ]")
                for stat in top_stats[:10]:
                    log.info(f"Snapshot difference: \n {stat}")
...
</code></pre>

ptrblck: <p>Are you storing tensors attached to the computation graph in e.g. a <code>list</code>?<br>
This would not be a leak, but the increased memory usage would be expected in this case (although this behavior is commonly referred to as a “memory leak”).<br>
Try to narrow down which part of your code is causing the increase in memory usage as e.g. <code>save_prediction_batch</code> and <code>successes.append(res)</code> might store the entire computation graph.</p>

aksg87: <p>I figured out the issue <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>I was using this package <a href="https://github.com/msamogh/nonechucks" class="inline-onebox" rel="noopener nofollow ugc">GitHub - msamogh/nonechucks: Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!</a></p>
<p>With this issue:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/msamogh/nonechucks/issues/13">
  <header class="source">

      <a href="https://github.com/msamogh/nonechucks/issues/13" target="_blank" rel="noopener nofollow ugc">github.com/msamogh/nonechucks</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/msamogh/nonechucks/issues/13" target="_blank" rel="noopener nofollow ugc">Memory leak</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-06-07" data-time="13:20:52" data-timezone="UTC">01:20PM - 07 Jun 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/shrubb" target="_blank" rel="noopener nofollow ugc">
          <img alt="shrubb" src="https://avatars.githubusercontent.com/u/9570420?v=4" class="onebox-avatar-inline" width="20" height="20">
          shrubb
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi,

when the below example is run, the RAM usage grows forever:

```python3<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">
import torch, torch.utils.data
import nonechucks

class DummyDataset(torch.utils.data.Dataset):
    def __len__(self):
        return 1_000_000

    def __getitem__(self, idx):
        return 666

dataset = nonechucks.SafeDataset(DummyDataset())

for _ in torch.utils.data.DataLoader(dataset):
    pass
```

Notes:

* Here the increase is quite slow; for a RAPID bug demonstration, replace `666` with `torch.empty(10_000)` (be careful to kill the process in time, before you're OOM!).
* No problems without `SafeDataset`.
* Without `torch.utils.data.DataLoader`, the leak is still there, although at a smaller scale, around 1 MB of RAM is lost per 30000-40000 `__getitem__` calls.
* PyTorch 1.0.1, nonechucks 0.3.1.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Subtle bug to track down <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>