Memory buildup or crash after many batches

username: thorstenwagner
cooked: <p>Another day, another thread <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m still working on getting distributed evaluation running.</p>
<p>This is the code:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee">
  <header class="source">

      <a href="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee</a></h4>

  <h5>embed.py</h5>
  <pre><code class="Python">    def embed(self, volume_data: VolumeDataset) -&gt; np.array:
        """Calculates the embeddings. The volumes showed have the dimension NxBSxBSxBS"""

        torch.backends.cudnn.benchmark = True

        dataset = TorchVolumeDataset(volumes=volume_data)
        sampler_data = torch.utils.data.DistributedSampler(dataset, rank=self.rank)
        volume_loader = DataLoader(
            dataset=dataset,
            batch_size=self.batchsize,</code></pre>
    This file has been truncated. <a href="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Now I’m facing problems which seem to be related to the <code>pin_memory</code> option of the data loader. Also <code>persistent_workers</code> seems to have an influence. The data that is loaded by the dataloader are just randomly generated arrays. No files are opened</p>
<p>Here is what is happening:</p>
<hr>
<p><strong>Case 1:</strong> <code>pin_memory</code>: True , <code>persistent_workers</code>: False</p>
<p>The system is running fine until it has processed like 4000 batches. Then suddenly the system memory starts to accumulate until the system crashes</p>
<p><strong>Case 2:</strong> <code>pin_memory</code>: False , <code>persistent_workers</code>: False</p>
<p>After 400-1000 batches I get the error:</p>
<pre><code class="lang-auto">RuntimeError: Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code
</code></pre>
<p><strong>Case 3:</strong> <code>pin_memory</code>: False , <code>persistent_workers</code>: True</p>
<p>Same as in case 2</p>
<p><strong>Case 4:</strong> <code>pin_memory</code>: False , <code>persistent_workers</code>: True + <code>set_sharing_strategy('file_system')</code></p>
<p>Same as in case 2</p>
<hr>
<p>Any idea what might happening?</p>
<p>I’m using pytorch 2.1.0.dev20230719 because of the problems written <a href="https://discuss.pytorch.org/t/combining-torch-compile-and-distributeddataparallel/184591/3">here</a>.</p>
<p>Best,<br>
Thorsten</p>

username: wconstab
cooked: <p>I would like to tag someone who is familiar with TorchVolumeDataset - is that a torch component or something you added?  (I’m unfamiliar but didn’t find it via a quick search).  I’d reroute the question to someone familiar with that if possible.</p>

username: thorstenwagner
cooked: <p>Its something I’ve added. But for the test I simplified it:</p>
<pre><code class="lang-auto">class TorchVolumeDataset(Dataset):
    """Implementation for the volume dataset"""

    def __init__(self, volumes: VolumeDataset):
        self.volumes = volumes

    def __getitem__(self, item_index):
        # vol = self.volumes[item_index]
        # vol = vol.astype(np.float32)
        # vol = pp.norm(vol)
        # vol = vol[np.newaxis]
        #
        vol = np.random.randn(1,37,37,37).astype(np.float32)
        torch_vol = torch.from_numpy(vol)
        input_triplet = {"volume": torch_vol}

        return input_triplet, item_index

    def __len__(self):
        return len(self.volumes)
</code></pre>

username: thorstenwagner
cooked: <p>Setting <code>ulimit -n 65000</code> solves the problem for case 2.</p>
<p>Case 1 is behaving the same as before, except that the accumulation starts now from the first batch.</p>
<p>But as it is now working for case 2 and not working for case 1, it seems to be clear that it must be bug with pin_memory?</p>
<p>EDIT: As case 2 is now working well enough, I would consider that as solution. However, something seems to be buggy with pin_memory</p>

username: wconstab
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you comment on whether pin_memory is being used correctly in case 2?  I think on some systems there is a limitation of how much cpu memory you can pin, i.e. not the whole system RAM.  I’m not sure if there is some config needed in the DataLoader to ensure that too much pinned memory is not used, or if explicit freeing is needed.</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/wconstab">@wconstab</a> I assume you meant case 1 as the others don’t use pinned memory.<br>
Generally, pinned memory should be used carefully, as it will disallow the OS to migrate these pages and you might run into memory thrashing or even instability issues when the OS needs to kill applications before running out of memory.<br>
The description in case 1 sounds a bit weird though. It seems using <code>pin_memory=True</code> works for some time until the host RAM usage suddenly spikes and crashes?<br>
The <code>DataLoader</code> should only pin the memory for the loaded batches (i.e. <code>num_workers</code>, <code>batch_size</code>, and <code>prefetch_factor</code> would be important here).<br>
<a class="mention" href="/u/thorstenwagner">@thorstenwagner</a> is this issue reproducible with any dataset (e.g. just random tensors in <code>TensorDataset</code>)?</p>

username: thorstenwagner
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>So far it happened for random tensors and for my data. So I guess yes, its reproducible</p>
<p>BTW: When the memory accumulation starts seems to be a bit random. At the time I opened the thread, it was around batch 4000. Now it starts basically from the beginning.</p>
<p>I also updated to the latest nightly build, which fixed other problems for me, but not this one.</p>